Docker gets you a browser you can pin. This page is about the month after you press start: what to persist, what to retry, what to watch, what it costs, and what to do at 3 a.m. on day four when the machine is gone and you have half a dataset.

This is the “Execution of experiment, deployment” step of the Research journey, and it is the step this corpus is quietest about. That silence is the page's first finding, not an excuse for it.

• Here: the crawl as a process that has to survive for weeks. The work queue and its outcomes, retry policy, what to monitor, concurrency and throughput, what it costs, and how to write up a run that was interrupted.
• Docker: the image digest, shm_size, ipc: host and Xvfb. That page pins the environment; this one runs it. The only overlap is the image pin, and it is not repeated here.
• Crawler: which library or patched browser drives the pages. Deployment does not pick the instrument.
• Longitudinal: *repeating* a measurement in separate waves months apart, and pinning the list, browser and classifier so wave two is comparable. This page is one continuous run; that page is several.
• Website selection: where the URL list comes from. Here it is already fixed, and the question is what happens to each entry on it.
• Crawling location: the vantage point. If it moves mid-run — because a VPN endpoint died and you replaced it — that is a deployment event with a measurement consequence.

Population: crawled papers with at least one temporal[] tuple whose mode is live-crawl and whose start and end are both day-precision. That is 230 of 1,120 crawled papers (20.5%); the rest give a month, a year, one endpoint, or nothing.

The raw span is not the answer, because the extraction's live-crawl mode also fires on tuples whose dates describe the *collected content* rather than the running apparatus. The longest “crawl” in the corpus is 9,028 days, and its sentence reads “90,153 disputes decided between December 1999 and August 2024”. Each of the 230 spans was classified by reading its own evidence quote:

Only the 182 campaign spans are used below. The full list of the 48 excluded ones, each with the sentence that decided it, is on deployment.

Median 30 days; 25th percentile 8 days; 75th percentile 98 days. 138 of 182 (75.8%) ran longer than a week and 90 (49.5%) longer than a month.

Two things follow. First, a week-long crawl is not an unusual ask and nobody will blink at it — the median paper that says how long it ran, ran for a month. Second, at those durations you are not running a script, you are operating a service, and the failure modes below are not edge cases. Konoth et al. ran 50 Docker containers for a week [1Konoth, Radhesh Krishnan; Vineti, Emanuele; Moonsamy, Veelasha; Lindorfer, Martina; Kruegel, Christopher; Bos, Herbert; Vigna, Giovanni (2018): "MineSweeper: An In-depth Look into Drive-by Cryptocurrency Mining and Its Defense", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)]; Rautenstrauch et al. ran 13 days for Chromium and Firefox and 47 days for WebKit on three Apple machines [2Rautenstrauch, Jannis; Pellegrino, Giancarlo; Stock, Ben (2023): "The Leaky Web: Automated Discovery of Cross-Site Information Leaks in Browsers and the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)]. A run that long will meet a reboot, a disk, or an expired credential.

vantage[].infrastructure is an enum, so it can carry a percentage. Against the 1,120 crawled papers:

The field is multi-valued, so the rows do not sum to 1,120. 445 of 1,120 (39.7%) name some non-sentinel infrastructure. A university machine is still the most common single answer, ahead of the cloud.

The operating knobs are almost invisible. These are full-text sweeps over paper.cols.txt with whitespace collapsed, paper-counted, over the same 1,120:

Probe width decides the claim. The bare word “retry” anywhere in the text matches 45 papers (4.0%); requiring it within 160 characters of a crawl noun halves that to 21 (1.9%). Neither is “the share of crawls that retry” — it is the share that wrote the word down. Read every row as an upper bound.

The last two rows are the interesting ones: a fifth of crawling papers name a page-load timeout, and an eighth say some pages failed, but almost none say what the process did about it. The timeout is a published parameter; the retry policy behind it usually is not.

The thing a week-long crawl must not lose is not the pages — you can fetch those again — it is the record of what happened to every URL you meant to visit. A crawl that resumes from “the files I already have” quietly redefines its own denominator: a URL that failed terminally and a URL the crash never reached look identical on disk, and the paper ends up reporting a percentage of a number that cannot be reconstructed.

So persist a row per URL, with exactly one terminal state, before you persist anything else. Three properties are worth getting right, and only the first is obvious:

1. Claim-then-commit with a lease and a named worker. Mark the row running with a deadline before touching the network, so a killed process leaves work reclaimable rather than lost or double-counted. Note what the lease alone does not buy you: five minutes after the crash it has not expired, so a “reclaim expired leases” pass returns zero and the row stays stranded. That is correct with other workers alive — you must not steal a live worker's row — which is why the row also carries a worker id. The restarting process is the one thing in the system that knows worker w1 is dead. Koop et al. describe the same shape from the other side: “nodes take jobs from the queue and execute them in individual Docker containers in parallel so that a problem in one of the tasks doesn't affect the other tasks” [3Koop, Martin; Tews, Erik; Katzenbeisser, Stefan (2020): "In-Depth Evaluation of Redirect Tracking and Link Usage", in: Proceedings on Privacy Enhancing Technologies. (DOI)].
2. Retryable and terminal errors are different. NXDOMAIN will still be NXDOMAIN in four days; a timeout or a 503 often will not be. Retrying everything spends the budget on domains that no longer exist; retrying nothing throws away real data (see the next section).
3. Fingerprint the seed list. A resumed run that seeds a different list is not a resumption, it is a second study. Hashing the list on first insert and refusing a changed one is four lines and it is the difference between a denominator you can name and one you cannot.

The script below is a working queue with those three properties, stdlib Python only (sqlite3), no network: the visit function is injected so the demo can kill the process deterministically and resume.

crawl_queue.py

#!/usr/bin/env python3
"""A crawl queue that survives the machine dying, and keeps its denominator.
 
The thing a week-long crawl must not lose is not the pages — those you can
fetch again — it is the *record of what happened to every URL you were going
to visit*. A crawl that resumes from "the files I have on disk" silently
redefines its own denominator: URLs that failed terminally look identical to
URLs the crash never reached, and the paper ends up reporting a percentage of
a number nobody can reconstruct.
 
So the unit of work here is a row, not a file, and every row ends in exactly
one terminal state:
 
    pending -> running -> done            the visit produced a result
                       -> failed          retries exhausted, or a terminal error
                       -> pending         retryable error, attempts left
    running -> pending                    lease expired (the worker died)
 
Three things this gets right that a dict-of-URLs in memory does not:
 
1.  **Claim-then-commit with a lease, and a named worker.** A worker marks a
    row `running` with its own id and a deadline before it touches the
    network. If the process is killed, the row is neither lost nor
    double-counted. But note what the lease alone does *not* buy you: five
    minutes after the crash the lease has not expired, so `reclaim_expired()`
    returns 0 and the row is stranded until the deadline passes. That is the
    correct behaviour with several workers running — you must not steal a live
    worker's row — and it is why the row also carries a worker id.
    `reclaim_worker("w1")` is the restart path: the process that is coming
    back up is the one thing in the system that *knows* w1 is dead. The demo
    below shows both calls, and the zero from the first one is the point.
 
2.  **Retryable and terminal errors are different.** NXDOMAIN will still be
    NXDOMAIN in four days; a timeout or a 503 often will not be. Retrying
    everything burns the budget on domains that no longer exist; retrying
    nothing throws away real data — Bouchet et al. (USENIX Security 2025)
    measured that nearly 40% of the sites that failed to load on the first
    attempt loaded on the second.
 
3.  **The failure classes are counted, not just the failures.** `progress()`
    returns the per-class breakdown, which is both the monitoring signal while
    the crawl runs and the attrition table the methods section owes a reviewer.
 
Stdlib only (sqlite3, no third-party driver). It talks to no network: the
visit function is injected, so the demo can simulate a crash deterministically.
 
    python3 crawl_queue.py --demo        crash-and-resume self-test
    python3 crawl_queue.py --help
"""
 
from __future__ import annotations
 
import argparse
import hashlib
import sqlite3
import sys
import time
from dataclasses import dataclass
 
SCHEMA = """
CREATE TABLE IF NOT EXISTS work (
    url          TEXT PRIMARY KEY,
    status       TEXT NOT NULL CHECK (status IN ('pending','running','done','failed')),
    attempts     INTEGER NOT NULL DEFAULT 0,
    worker       TEXT,
    lease_until  REAL,
    outcome      TEXT,
    error_class  TEXT,
    updated_at   REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS work_status ON work(status);
CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL);
"""
 
# Errors that will still be errors when you try again in four days. Anything
# not in here is treated as retryable. Getting this list wrong in the safe
# direction (calling a terminal error retryable) costs attempts; getting it
# wrong in the unsafe direction loses data.
TERMINAL_ERRORS = frozenset({"nxdomain", "http_404", "http_410", "invalid_url", "robots_denied"})
 
 
class Interrupted(Exception):
    """Raised by the demo's visit function to stand in for the machine dying."""
 
 
@dataclass(frozen=True)
class Progress:
    total: int
    pending: int
    running: int
    done: int
    failed: int
    by_error: dict
 
    @property
    def terminal(self) -> int:
        return self.done + self.failed
 
    def line(self) -> str:
        errs = ", ".join(f"{k}={v}" for k, v in sorted(self.by_error.items())) or "-"
        return (
            f"total={self.total} done={self.done} failed={self.failed} "
            f"running={self.running} pending={self.pending} | {errs}"
        )
 
 
class CrawlQueue:
    def __init__(self, path: str, max_attempts: int = 3, lease_seconds: float = 300.0,
                 clock=time.time):
        self.db = sqlite3.connect(path, isolation_level=None)
        self.db.execute("PRAGMA journal_mode=WAL")
        self.db.executescript(SCHEMA)
        self.max_attempts = max_attempts
        self.lease_seconds = lease_seconds
        self.clock = clock
 
    # -- population -------------------------------------------------------
    def seed(self, urls) -> int:
        """Insert the population once. Idempotent: re-seeding never resets a row.
 
        The seed is also fingerprinted. A resumed run that seeds a *different*
        list is not a resumption, it is a second study, and the fingerprint is
        what makes that visible instead of silent.
        """
        urls = list(urls)
        digest = hashlib.sha256("\n".join(urls).encode("utf-8")).hexdigest()
        row = self.db.execute("SELECT v FROM meta WHERE k='seed_sha256'").fetchone()
        if row is None:
            self.db.execute("INSERT INTO meta(k,v) VALUES('seed_sha256',?)", (digest,))
        elif row[0] != digest:
            raise ValueError(
                f"seed list changed since this queue was created "
                f"(stored {row[0][:12]}..., given {digest[:12]}...). "
                f"Use a new database, or you cannot name your denominator."
            )
        now = self.clock()
        before = self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0]
        self.db.executemany(
            "INSERT OR IGNORE INTO work(url,status,attempts,updated_at) VALUES(?,'pending',0,?)",
            [(u, now) for u in urls],
        )
        return self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0] - before
 
    def seed_fingerprint(self) -> str:
        return self.db.execute("SELECT v FROM meta WHERE k='seed_sha256'").fetchone()[0]
 
    # -- lease ------------------------------------------------------------
    def reclaim_expired(self) -> int:
        """Return rows whose lease has run out. Safe with other workers alive."""
        now = self.clock()
        cur = self.db.execute(
            "UPDATE work SET status='pending', worker=NULL, lease_until=NULL, updated_at=? "
            "WHERE status='running' AND (lease_until IS NULL OR lease_until <= ?)",
            (now, now),
        )
        return cur.rowcount
 
    def reclaim_worker(self, worker: str) -> int:
        """Return rows held by a worker you know is dead, lease or no lease.
 
        Only call this for a worker that is definitely not running. On a
        single-machine restart that is exactly the id the previous process
        used, and it is the difference between resuming now and resuming when
        the lease expires.
        """
        cur = self.db.execute(
            "UPDATE work SET status='pending', worker=NULL, lease_until=NULL, updated_at=? "
            "WHERE status='running' AND worker=?",
            (self.clock(), worker),
        )
        return cur.rowcount
 
    def claim(self, worker: str = "w0"):
        """Take one pending URL and mark it running. Returns None when drained."""
        now = self.clock()
        row = self.db.execute(
            "SELECT url, attempts FROM work WHERE status='pending' ORDER BY url LIMIT 1"
        ).fetchone()
        if row is None:
            return None
        url, attempts = row
        self.db.execute(
            "UPDATE work SET status='running', attempts=?, worker=?, lease_until=?, updated_at=? "
            "WHERE url=? AND status='pending'",
            (attempts + 1, worker, now + self.lease_seconds, now, url),
        )
        return url, attempts + 1
 
    # -- outcomes ---------------------------------------------------------
    def complete(self, url: str, outcome: str) -> None:
        self.db.execute(
            "UPDATE work SET status='done', outcome=?, error_class=NULL, "
            "worker=NULL, lease_until=NULL, updated_at=? WHERE url=?",
            (outcome, self.clock(), url),
        )
 
    def record_error(self, url: str, error_class: str) -> str:
        """Decide whether this URL comes back. Returns the new status."""
        attempts = self.db.execute("SELECT attempts FROM work WHERE url=?", (url,)).fetchone()[0]
        retryable = error_class not in TERMINAL_ERRORS and attempts < self.max_attempts
        status = "pending" if retryable else "failed"
        self.db.execute(
            "UPDATE work SET status=?, error_class=?, worker=NULL, lease_until=NULL, "
            "updated_at=? WHERE url=?",
            (status, error_class, self.clock(), url),
        )
        return status
 
    # -- reporting --------------------------------------------------------
    def progress(self) -> Progress:
        counts = dict(self.db.execute("SELECT status, COUNT(*) FROM work GROUP BY status").fetchall())
        by_error = dict(
            self.db.execute(
                "SELECT error_class, COUNT(*) FROM work WHERE status='failed' "
                "AND error_class IS NOT NULL GROUP BY error_class"
            ).fetchall()
        )
        return Progress(
            total=self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0],
            pending=counts.get("pending", 0),
            running=counts.get("running", 0),
            done=counts.get("done", 0),
            failed=counts.get("failed", 0),
            by_error=by_error,
        )
 
    def results(self) -> dict:
        return dict(self.db.execute("SELECT url, outcome FROM work WHERE status='done'").fetchall())
 
    def failures(self) -> dict:
        return dict(self.db.execute("SELECT url, error_class FROM work WHERE status='failed'").fetchall())
 
 
def run(queue: CrawlQueue, visit, worker: str = "w0", log_every: int = 0) -> int:
    """Drain the queue with one worker. Returns the number of visits attempted.
 
    `visit(url, attempt)` returns a result string, or raises. An exception
    carrying a `.error_class` attribute is classified; anything else is
    `unknown` and therefore retryable, which is the safe default.
    """
    attempted = 0
    while True:
        claimed = queue.claim(worker)
        if claimed is None:
            return attempted
        url, attempt = claimed
        attempted += 1
        try:
            queue.complete(url, visit(url, attempt))
        except Interrupted:
            raise
        except Exception as exc:  # a failed page must not stop the crawl
            queue.record_error(url, getattr(exc, "error_class", "unknown"))
        if log_every and attempted % log_every == 0:
            print(f"  [worker] {queue.progress().line()}")
 
 
# ---------------------------------------------------------------------------
# Demo: a deterministic fixture, a crash on the 87th visit, and a resume.
# ---------------------------------------------------------------------------
 
class FakeError(Exception):
    def __init__(self, error_class):
        super().__init__(error_class)
        self.error_class = error_class
 
 
def demo_sites(n=200):
    return [f"https://site{i:03d}.example/" for i in range(n)]
 
 
def make_visit(crash_at=None):
    """Deterministic outcomes keyed on the URL, so both runs agree.
 
    Buckets, by the last digit of the site number:
      0-5  succeed first time
      6    NXDOMAIN, terminal, never retried
      7    times out on attempts 1 and 2, succeeds on attempt 3
      8    404, terminal
      9    times out on every attempt, exhausts the retry budget
    """
    state = {"calls": 0}
 
    def visit(url, attempt):
        state["calls"] += 1
        if crash_at is not None and state["calls"] == crash_at:
            raise Interrupted(url)
        digit = int(url.rstrip("/").rsplit("site", 1)[1].split(".")[0]) % 10
        if digit == 6:
            raise FakeError("nxdomain")
        if digit == 8:
            raise FakeError("http_404")
        if digit == 9:
            raise FakeError("timeout")
        if digit == 7 and attempt < 3:
            raise FakeError("timeout")
        return f"ok:{len(url)}"
 
    return visit
 
 
def run_demo() -> int:
    import os
    import tempfile
 
    sites = demo_sites()
    tmp = tempfile.mkdtemp(prefix="crawlq")
    clean_db = os.path.join(tmp, "clean.sqlite")
    crash_db = os.path.join(tmp, "crash.sqlite")
 
    print(f"=== A. reference run, no crash ({len(sites)} URLs) ===")
    clean = CrawlQueue(clean_db)
    print(f"seeded {clean.seed(sites)} rows, seed sha256 {clean.seed_fingerprint()[:16]}...")
    calls_clean = make_visit()
    attempted_clean = run(clean, calls_clean, worker="w0", log_every=100)
    print(f"visits attempted: {attempted_clean}")
    print(f"final: {clean.progress().line()}")
 
    print("\n=== B. same population, machine dies on the 87th visit ===")
    crash = CrawlQueue(crash_db)
    crash.seed(sites)
    try:
        run(crash, make_visit(crash_at=87), worker="w1", log_every=100)
        print("FAILURE: the fixture did not crash")
        return 1
    except Interrupted as exc:
        print(f"killed while visiting {exc.args[0]}")
    print(f"state on disk: {crash.progress().line()}")
    stranded = crash.progress().running
    print(f"rows left in 'running' (the in-flight visit): {stranded}")
 
    print("\n=== C. resume: reopen the same database and drain it ===")
    resumed = CrawlQueue(crash_db)
    print(f"re-seeding the same list inserts {resumed.seed(sites)} new rows")
    by_lease = resumed.reclaim_expired()
    print(f"reclaim_expired() reclaimed {by_lease} row(s) -- the 300s lease has not run out yet")
    by_worker = resumed.reclaim_worker("w1")
    print(f"reclaim_worker('w1') reclaimed {by_worker} row(s) -- w1 is the process that died")
    attempted_resume = run(resumed, make_visit(), worker="w2", log_every=100)
    print(f"visits attempted after resume: {attempted_resume}")
    print(f"final: {resumed.progress().line()}")
 
    print("\n=== D. the seed guard ===")
    try:
        resumed.seed(sites + ["https://site200.example/"])
        print("FAILURE: seed guard did not fire")
        return 1
    except ValueError as exc:
        print(f"refused, as it should: {str(exc).splitlines()[0]}")
 
    print("\n=== E. checks ===")
    ok = True
    checks = [
        ("same successful set", clean.results() == resumed.results()),
        ("same failure set", clean.failures() == resumed.failures()),
        ("same terminal count", clean.progress().terminal == resumed.progress().terminal),
        ("nothing left pending", resumed.progress().pending == 0),
        ("nothing left running", resumed.progress().running == 0),
        ("every row terminal", resumed.progress().terminal == len(sites)),
        ("the lease alone did not free the stranded row", by_lease == 0),
        ("naming the dead worker did", by_worker == 1),
        ("in-flight URL was retried, not lost", stranded == 1),
    ]
    for name, passed in checks:
        print(f"  {'ok  ' if passed else 'FAIL'} {name}")
        ok = ok and passed
    print(f"  note  clean run: {attempted_clean} visits. crash+resume: 86 before the crash, "
          f"1 interrupted, {attempted_resume} after = {86 + 1 + attempted_resume} calls for the "
          f"same {len(sites)} terminal rows. The excess is the one interrupted visit.")
    print("\nself-test OK" if ok else "\nFAILURE: a check failed")
    return 0 if ok else 1
 
 
def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--demo", action="store_true", help="run the crash-and-resume self-test")
    ap.add_argument("--db", help="queue database to report on")
    args = ap.parse_args()
    if args.demo:
        return run_demo()
    if args.db:
        print(CrawlQueue(args.db).progress().line())
        return 0
    ap.error("pass --demo or --db")
 
 
if __name__ == "__main__":
    sys.exit(main())

Real output of python3 crawl_queue.py --demo (exit 0), 2026-08-28:

=== A. reference run, no crash (200 URLs) ===
seeded 200 rows, seed sha256 49ddaed11cb3d529...
  [worker] total=200 done=51 failed=21 running=0 pending=128 | http_404=7, nxdomain=7, timeout=7
  [worker] total=200 done=102 failed=42 running=0 pending=56 | http_404=14, nxdomain=14, timeout=14
visits attempted: 280
final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20

=== B. same population, machine dies on the 87th visit ===
killed while visiting https://site062.example/
state on disk: total=200 done=44 failed=18 running=1 pending=137 | http_404=6, nxdomain=6, timeout=6
rows left in 'running' (the in-flight visit): 1

=== C. resume: reopen the same database and drain it ===
re-seeding the same list inserts 0 new rows
reclaim_expired() reclaimed 0 row(s) -- the 300s lease has not run out yet
reclaim_worker('w1') reclaimed 1 row(s) -- w1 is the process that died
  [worker] total=200 done=95 failed=39 running=0 pending=66 | http_404=13, nxdomain=13, timeout=13
visits attempted after resume: 194
final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20

=== D. the seed guard ===
refused, as it should: seed list changed since this queue was created (stored 49ddaed11cb3..., given f1c74ecfee8e...). Use a new database, or you cannot name your denominator.

=== E. checks ===
  ok   same successful set
  ok   same failure set
  ok   same terminal count
  ok   nothing left pending
  ok   nothing left running
  ok   every row terminal
  ok   the lease alone did not free the stranded row
  ok   naming the dead worker did
  ok   in-flight URL was retried, not lost
  note  clean run: 280 visits. crash+resume: 86 before the crash, 1 interrupted, 194 after = 281 calls for the same 200 terminal rows. The excess is the one interrupted visit.

self-test OK

The check that matters is same successful set and same failure set: the crashed-and-resumed run ends with byte-identical outcomes to the run that was never interrupted, and the one visit that was in flight is retried rather than silently dropped. The two lines above it are the lesson: reclaim_expired() returns 0, reclaim_worker('w1') returns 1.

Retrying once is the single highest-return operational decision on this page, and one paper measured it. Hausladen et al. reload a site that failed on the first attempt, and report: “nearly 40% of sites that failed to load on the first attempt were loaded and successfully analyzed” on the second [4Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)]. Their first-attempt failure conditions are worth copying verbatim, because they are the three that matter: “a site failed to load in 35 seconds, had an insecure certificate, or led to an error page”.

What the retry is worth depends on what fraction of your list fails at all. Papers that report it:

Two patterns to take from that table. A top-1M crawl reaches roughly 88–94% of its list and a smaller, more popular list reaches 91–99%; if your number is far below that, the problem is your deployment, not the web. And the failure mix moves over time — 403 Forbidden dominating a 2024 crawl [8Annamalai, Meenatchi Sundaram Muthu Selva; Bilogrevic, Igor; Cristofaro, Emiliano De (2024): "FP-Fed: Privacy-Preserving Federated Detection of Browser Fingerprinting", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] is bot detection, not a network problem, and no amount of retrying fixes it.

Retries also come in shapes other than “do it again”. Ikram et al. retry with a changed parameter: “we checked if the rendering of any webpage stalled. If so, we reran the crawler on the webpage by increasing the waiting time until the webpage is rendered” [12Ikram, Muhammad; Asghar, Hassan Jameel; Kaafar, Mohamed Ali; Mahanti, Anirban; Krishnamurthy, Balachander (2017): "Towards Seamless Tracking-Free Web: Improved Detection of Trackers via One-class Learning", in: Proceedings on Privacy Enhancing Technologies. (DOI)]. Khodayari et al. simply bound it — “we made up to three repeated attempts for each failed crawling” [13Khodayari, Soheil; Barber, Thomas; Pellegrino, Giancarlo (2024): "The Great Request Robbery: An Empirical Study of Client-side Request Hijacking Vulnerabilities on the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)]. Senol et al. record what a retry-shaped fix was worth: a minor change “increased the successfully visited websites from 94,427 (EU pilot crawl) to 99,380 (EU final crawl)” [14Senol, Asuman; Acar, Gunes; Humbert, Mathias; Zuiderveen Borgesius, Frederik (2022): "Leaky Forms: A Study of Email and Password Exfiltration Before Form Submission", in: 31st USENIX Security Symposium (USENIX Security 22). (Link)].

29 of 1,120 crawled papers (2.6%) state how many browsers or workers they ran at once. That is the whole planning literature, and it is a probe result with a hand-read verdict per hit: a further 4 say they were parallel without a number, 5 parallelise an analysis stage rather than the crawl, and 6 are about something else entirely.

The one throughput figure anyone publishes is Bouhoula et al.'s: 300 websites per hour from 30 parallel OpenWPM browsers on 16 cores [18Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)] — about 10 sites per browser-hour, or 7,200 sites a day per machine. Take it as the order of magnitude for an instrumented, stateful, banner-interacting crawl, not as a target: a stateless landing-page fetch is much faster and a crawl that clicks through consent and visits subpages is much slower (Interaction).

That single number is enough to plan with. A one-million-site crawl at 300 sites per machine-hour is about 3,300 machine-hours: five days on 28 machines, or a month on four. Englehardt and Narayanan's ratio is the constraint behind it — “stateful parallel measurements are memory-limited while stateless parallel measurements are typically CPU-limited and can support a higher number of instances” [16Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]. Concurrency per machine is set by RAM if you keep profiles and by cores if you do not.

16 of 1,120 crawled papers (1.4%) name a monitoring stack, so there is no literature here to summarise. What there is, is a list of the things that went wrong in the ten papers that admitted something went wrong (next section), and every one of them is visible in a counter you could have been watching:

• Terminal rows per hour. Not “pages fetched” — rows that reached done or failed. A crawl whose throughput halves is usually a crawl where retries have started dominating, which is a signal about the target, not about your disk.
• The failure mix, by class. A run that shifts from timeout to http_403 has been detected as a bot; a run that shifts to nxdomain is reading a stale list. Both need a decision, and neither shows up in a total failure count. The queue in the section above returns exactly this breakdown from progress().
• Success rate against your own baseline. You measured it on day one. Kieserman et al.'s 94.51% and Annamalai et al.'s 91.5% are the shape of a healthy crawl [9Kieserman, Julia B.; Andreou, Athanasios; Geeng, Chris; Lauinger, Tobias; McCoy, Damon (2025): "Tracker Installations Are Not Created Equal: Understanding Tracker Configuration of Form Data Collection", in: Proceedings on Privacy Enhancing Technologies, pp. 679-695. (DOI)] [8Annamalai, Meenatchi Sundaram Muthu Selva; Bilogrevic, Igor; Cristofaro, Emiliano De (2024): "FP-Fed: Privacy-Preserving Federated Detection of Browser Fingerprinting", in: Proceedings of the Network and Distributed System Security Symposium. (Link)]; a sudden drop is an incident even if nothing crashed.
• Free disk, and the size of one visit's output. Multiply by the queue depth on day one, not on day four.
• A canary that is not part of the crawl. A page you control, fetched on the same schedule through the same stack. When the numbers move, it tells you whether the web changed or you did.

Two failure modes deserve naming because a dashboard of totals hides both. A crawl can produce a complete-looking empty database — OpenWPM documents this for OpenWPM specifically, where all fourteen tables exist and are empty after a failed run. And a crawl can keep running while measuring the wrong thing: Yeung et al.'s crawler “crashed on March 12th in both Ukraine and Russia, and again on March 20th and April 18th in Ukraine after running for some time” [20Yeung, Christina; Iqbal, Umar; O'Neil, Yekaterina Tsipenyuk; Kohno, Tadayoshi; Roesner, Franziska (2023): "Online Advertising in Ukraine and Russia During the 2022 Russian Invasion", in: Proceedings of the ACM Web Conference. (DOI)] — a per-vantage-point counter shows that, a global one does not.

5 of 1,120 crawled papers (0.4%) state an infrastructure bill. A further 3 state what they paid to acquire data — services, participants, annotation — so 8 (0.7%) state any cost they themselves paid. Eleven other papers put a currency figure next to a cost word in a measurement context, and in every one of those the price belongs to what was being measured, not to the measuring. This field reports the adversary's economics in detail and its own hardly at all.

The five, in full, are the entire published record:

Since nobody will tell you, work it out. The arithmetic has four terms and only the first is large:

1. Machine-hours = (sites × attempts) ÷ (workers per machine × sites per worker-hour). At Bouhoula et al.'s 300 sites/hour/machine [18Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)], a one-million-site pass is ~3,300 machine-hours. On-demand Linux in eu-central-1 on 2026-08-28 was USD 0.4074/hour for a c7i.2xlarge (8 vCPU) and USD 0.8148/hour for a c7i.4xlarge (16 vCPU), read out of the AWS bulk price list.¹⁾ So ~3,300 machine-hours on 16-core instances is roughly USD 2,700 for one pass — the same order as Thomas et al.'s $1,587/month fifteen years ago, which is the useful observation: the hardware got faster and the crawl got heavier.
2. A second pass. Budget for it. Every longitudinal design needs one and every first pass has a bug in it.
3. Egress and storage. Small for HTML and headers, not small for screenshots or full HAR files (Traffic files). Measure one visit's output on day one and multiply.
4. A university machine, which is free at the point of use and is what 134 of 1,120 papers (12.0%) used, against 123 (11.0%) on a cloud provider. It has no hourly bill and no autoscaling either; the trade is that a departmental power cut is your problem — which is exactly what happened to Agten et al. [25Agten, Pieter; Joosen, Wouter; Piessens, Frank; Nikiforakis, Nick (2015): "Seven Months' Worth of Mistakes: A Longitudinal Study of Typosquatting Abuse", in: Proceedings of the Network and Distributed System Security Symposium. (Link)].

Ten papers in this corpus — 0.9% of the 1,120 that crawled — say in print that their own measurement lost time. That is not a rate of machines dying. It is a rate of people writing it down, and these ten are the template for how to do it.

Only one paper in the corpus went the other way and claimed a clean run — “none of the three ROBOVIC instances experienced any downtime during our monitored period” [35Miramirkhani, Najmeh; Starov, Oleksii; Nikiforakis, Nick (2017): "Dial One for Scam: A Large-Scale Analysis of Technical Support Scams", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] — and it is worth noticing that they needed the claim because their result depended on the three instances being comparable. Agten et al. are the one case of an outage that was known in advance: they raised the crawl rate the week before “a planned power interruption of our crawling machines on August 27 and 28” [25Agten, Pieter; Joosen, Wouter; Piessens, Frank; Nikiforakis, Nick (2015): "Seven Months' Worth of Mistakes: A Longitudinal Study of Typosquatting Abuse", in: Proceedings of the Network and Distributed System Security Symposium. (Link)].

What to actually do, in order:

1. Do not restart the crawl. Resume it. If your queue has per-URL outcomes and a lease, the only work at risk is what was in flight, and the demo above shows the recovered run ending with the same outcome set as the uninterrupted one. If it does not, you now have to choose between a partial dataset and re-crawling a web that has moved under you — and re-crawling is a *different measurement*, not a repeat.
2. Record the gap in the same units as your results. Not “we had some downtime” — Leontiadis et al.'s “1 004 of 1 254 days” is the model, because a reader can divide it. If your unit is sites, say how many sites the gap cost. If your unit is days of a time series, say which days.
3. Decide what the gap does to your inference, and say so. Wang et al. quantified it as 4.5% of broadcasts missing [29Wang, Bolun; Zhang, Xinyi; Wang, Gang; Zheng, Haitao; Zhao, Ben Y. (2016): "Anatomy of a Personalized Livestreaming System", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]; Lee et al. pruned the 17 affected days out of the dataset entirely rather than analysing a series with holes in it [32Lee, Hyunwoo; Kim, Doowon; Kwon, Yonghwi (2021): "TLS 1.3 in Practice: How TLS 1.3 Contributes to the Internet", in: Proceedings of the ACM Web Conference. (DOI)]. Both are defensible. Neither is “we lost some data”.
4. Check that the gap is not correlated with what you measure. A crawl that dies overnight loses the sites that sort late in the list; a crawl that dies on Fridays loses whatever is different about weekends. If your queue orders by rank, a mid-run failure is a systematic loss, not a random one — shuffle the queue at seed time, or say the loss was ordered.
5. Check whether the environment changed while it was down. A restart after a package update is a new instrument. Longitudinal covers pinning; the deployment consequence is that “we restarted” and “we upgraded and restarted” are different sentences in a methods section.
6. Then decide whether to extend. Extending the window to recover lost coverage stretches the measurement period, which is a temporal confound, not a free fix. Say which you did.

A methods section a later run can reconstruct, and a reviewer can check:

1. The wall-clock window, with both endpoints at day precision. Only 230 of 1,120 crawled papers manage this, and it is the cheapest sentence on the list.
2. Whether that window is machine time or content coverage. A quarter of the day-precision spans in this corpus are ambiguous on exactly this point, and readers cannot tell them apart.
3. The population, and how much of it you reached — the numerator, the denominator, and the failure classes underneath. “We crawled the Tranco top 1M” is not a result; “we reached 88.1% after one retry, 7.7% did not resolve, 4.2% did not respond” is [6Murley, Paul; Ma, Zane; Mason, Joshua; Bailey, Michael D.; Kharraz, Amin (2021): "WebSocket Adoption and the Landscape of the Real-Time Web", in: Proceedings of the ACM Web Conference. (DOI)].
4. The retry policy: how many attempts, which errors were terminal, and whether the retry was immediate or at the end of the queue. If a retry pass changed your numbers, say by how much [4Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)].
5. Concurrency and hardware, in the same sentence: N browsers on M cores with K GB. 29 papers do this and they are the only ones another group can size a machine from.
6. The infrastructure: cloud provider and region, university network, or testbed. 61.4% of crawled papers say nothing here, which also makes their vantage point unverifiable (Crawling location).
7. Any interruption, in the units of your result, and what you did with the gap.
8. The cost, if you can. Five papers in sixteen years of seven venues have done it, and every student planning a crawl has to guess because of that.

• Docker — pinning the image, shm_size, ipc: host, Xvfb. Where this page starts.
• Crawler — the instrument; OpenWPM for the failed-crawl-looks-complete trap.
• Longitudinal — separate waves rather than one long run, and pinning across them.
• Stateful stateless — whether the profile persists, which sets your memory ceiling and therefore your concurrency.
• Interaction — depth and on-page actions, which set your seconds-per-site and therefore your wall clock.
• Crawling location — the vantage point and what it costs to hold several of them for a month.
• Ethics — a month-long crawl is a month of load on someone else's server.
• Corpus — seven-venue scope and the 2025–2026 edge.

Every query, the unedited report, the span-role and verdict maps with the sentence behind each one, the quote checks and the rejected probes are on deployment.

Corpus figures come from scripts/report_deployment.mjs against data/extract/run1 (5,859 papers; CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P; 2010–2026). The denominator throughout is the 1,120 papers that ran a crawl, never 5,859.

Three limits worth stating plainly. The duration figures rest on a hand classification, not on the extraction: temporal[].mode == “live-crawl” does not distinguish machine time from content coverage, and 48 of 230 spans had to be excluded by reading. Every operating figure is a full-text probe, so it counts papers that used a word, not papers that did the thing; the retry row moves by a factor of two depending on how wide the regex is, and both widths are printed above. The attrition probe is weak: it returns 75 papers, and a hand-read sample of 12 was right half the time, so no reporting rate is published from it. Per-paper figures were checked against paper.cols.txt with whitespace collapsed — 41 of 41 found verbatim. The AWS prices are a single fetch on 2026-08-28 and will be wrong soon. Year shares are not used on this page; where 2025–2026 papers appear they are individual citations, not a trend (Corpus).

[1]

Konoth, Radhesh Krishnan; Vineti, Emanuele; Moonsamy, Veelasha; Lindorfer, Martina; Kruegel, Christopher; Bos, Herbert; Vigna, Giovanni (2018): "MineSweeper: An In-depth Look into Drive-by Cryptocurrency Mining and Its Defense", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[2]

Rautenstrauch, Jannis; Pellegrino, Giancarlo; Stock, Ben (2023): "The Leaky Web: Automated Discovery of Cross-Site Information Leaks in Browsers and the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[3]

Koop, Martin; Tews, Erik; Katzenbeisser, Stefan (2020): "In-Depth Evaluation of Redirect Tracking and Link Usage", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[4]

Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)

[5]

Kumar, Deepak; Ma, Zane; Durumeric, Zakir; Mirian, Ariana; Mason, Joshua; Halderman, J. Alex; Bailey, Michael D. (2017): "Security Challenges in an Increasingly Tangled Web", in: Proceedings of the ACM Web Conference. (DOI)

[6]

Murley, Paul; Ma, Zane; Mason, Joshua; Bailey, Michael D.; Kharraz, Amin (2021): "WebSocket Adoption and the Landscape of the Real-Time Web", in: Proceedings of the ACM Web Conference. (DOI)

[7]

Musch, Marius; Johns, Martin (2021): "U Can't Debug This: Detecting JavaScript Anti-Debugging Techniques in the Wild", in: Proceedings of the USENIX Security Symposium. (Link)

[8]

Annamalai, Meenatchi Sundaram Muthu Selva; Bilogrevic, Igor; Cristofaro, Emiliano De (2024): "FP-Fed: Privacy-Preserving Federated Detection of Browser Fingerprinting", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[9]

Kieserman, Julia B.; Andreou, Athanasios; Geeng, Chris; Lauinger, Tobias; McCoy, Damon (2025): "Tracker Installations Are Not Created Equal: Understanding Tracker Configuration of Form Data Collection", in: Proceedings on Privacy Enhancing Technologies, pp. 679-695. (DOI)

[10]

Kuchhal, Dhruv; Li, Frank (2021): "Knock and talk: investigating local network communications on websites", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[11]

Demir, Nurullah; Urban, Tobias; Pohlmann, Norbert; Wressnegger, Christian (2024): "A Large-Scale Study of Cookie Banner Interaction Tools and their Impact on Users' Privacy", in: Proceedings on Privacy Enhancing Technologies, pp. 5-20. (DOI)

[12]

Ikram, Muhammad; Asghar, Hassan Jameel; Kaafar, Mohamed Ali; Mahanti, Anirban; Krishnamurthy, Balachander (2017): "Towards Seamless Tracking-Free Web: Improved Detection of Trackers via One-class Learning", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[13]

Khodayari, Soheil; Barber, Thomas; Pellegrino, Giancarlo (2024): "The Great Request Robbery: An Empirical Study of Client-side Request Hijacking Vulnerabilities on the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[14]

Senol, Asuman; Acar, Gunes; Humbert, Mathias; Zuiderveen Borgesius, Frederik (2022): "Leaky Forms: A Study of Email and Password Exfiltration Before Form Submission", in: 31st USENIX Security Symposium (USENIX Security 22). (Link)

[15]

Acar, Gunes; Eubank, Christian; Englehardt, Steven; Juarez, Marc; Narayanan, Arvind; Díaz, Claudia (2014): "The Web Never Forgets: Persistent Tracking Mechanisms in the Wild", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[16]

Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[17]

Bijmans, Hugo L.J.; Booij, Tim M.; Doerr, Christian (2019): "Inadvertently Making Cyber Criminals Rich: A Comprehensive Study of Cryptojacking Campaigns at Internet Scale", in: Proceedings of the USENIX Security Symposium. (Link)

[18]

Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)

[19]

Retana, Alberto Fernández de; Rautenstrauch, Jannis; Santos-Grueiro, Igor; Stock, Ben (2025): "A Permissions Odyssey: A Systematic Study of Browser Permissions on Modern Websites", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[20]

Yeung, Christina; Iqbal, Umar; O'Neil, Yekaterina Tsipenyuk; Kohno, Tadayoshi; Roesner, Franziska (2023): "Online Advertising in Ukraine and Russia During the 2022 Russian Invasion", in: Proceedings of the ACM Web Conference. (DOI)

[21]

Wondracek, Gilbert; Holz, Thorsten; Kirda, Engin; Kruegel, Christopher (2010): "A Practical Attack to De-anonymize Social Network Users", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[22]

Thomas, Kurt; Grier, Chris; Ma, Justin; Paxson, Vern; Song, Dawn (2011): "Design and Evaluation of a Real-Time URL Spam Filtering Service", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[23]

Zhang, Zhenrui; Hong, Geng; Li, Xiang; Fu, Zhuoqun; Zhang, Jia; Liu, Mingxuan; Wang, Chuhan; Chen, Jianjun; Liu, Baojun; Duan, Haixin; Zhang, Chao; Yang, Min (2023): "Under the Dark: A Systematical Study of Stealthy Mining Pools (Ab)use in the Wild", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[24]

Song, Chuxu; Mekala, Dheekshith Dev Manohar; Wang, Hao; Martin, Richard (2026): "Redefining Website Fingerprinting Attacks with Multi-Agent LLMs", in: Proceedings on Privacy Enhancing Technologies, pp. 688-702. (DOI)

[25]

Agten, Pieter; Joosen, Wouter; Piessens, Frank; Nikiforakis, Nick (2015): "Seven Months' Worth of Mistakes: A Longitudinal Study of Typosquatting Abuse", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[26]

Maggi, Federico; Frossi, Alessandro; Zanero, Stefano; Stringhini, Gianluca; Stone-Gross, Brett; Kruegel, Christopher; Vigna, Giovanni (2013): "Two years of short URLs internet measurement: security threats and countermeasures", in: Proceedings of the ACM Web Conference. (DOI)

[27]

Leontiadis, Nektarios; Moore, Tyler; Christin, Nicolas (2014): "A Nearly Four-Year Longitudinal Study of Search-Engine Poisoning", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[28]

Sabottke, Carl; Suciu, Octavian; Dumitraş, Tudor (2015): "Vulnerability Disclosure in the Age of Social Media: Exploiting Twitter for Predicting Real-World Exploits", in: Proceedings of the USENIX Security Symposium. (Link)

[29]

Wang, Bolun; Zhang, Xinyi; Wang, Gang; Zheng, Haitao; Zhao, Ben Y. (2016): "Anatomy of a Personalized Livestreaming System", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[30]

Chatterjee, Rahul; Doerfler, Periwinkle; Orgad, Hadas; Havron, Sam; Palmer, Jackeline; Freed, Diana; Levy, Karen; Dell, Nicola; McCoy, Damon; Ristenpart, Thomas (2018): "The Spyware Used in Intimate Partner Violence", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[31]

Rüth, Jan; Zimmermann, Torsten; Wolsing, Konrad; Hohlfeld, Oliver (2018): "Digging into Browser-based Crypto Mining", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[32]

Lee, Hyunwoo; Kim, Doowon; Kwon, Yonghwi (2021): "TLS 1.3 in Practice: How TLS 1.3 Contributes to the Internet", in: Proceedings of the ACM Web Conference. (DOI)

[33]

Poteat, Tara; Li, Frank (2021): "Who You Gonna Call? An Empirical Evaluation of Website security.txt Deployment", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[34]

Dahlberg, Rasmus; Pulls, Tobias (2023): "Timeless Timing Attacks and Preload Defenses in Tor's DNS Cache", in: Proceedings of the USENIX Security Symposium. (Link)

[35]

Miramirkhani, Najmeh; Starov, Oleksii; Nikiforakis, Nick (2017): "Dial One for Scam: A Large-Scale Analysis of Technical Support Scams", in: Proceedings of the Network and Distributed System Security Symposium. (Link)