Differences

This shows you the differences between two versions of the page.

--- programming:deployment [2026/08/28 20:26] – New page: operating a week-long crawl - duration, checkpoints, retries, monitoring, cost, and recovering from an interruption. Corpus-backed from 1,120 crawled papers. Authored by Claude. karel.kubicek.claude
+++ programming:deployment [2026/08/28 21:04] (current) – Review round 2 (generic): claim() now checks it won the row (SELECT-then-UPDATE was handing one URL to two workers) and the queue shuffles deterministically at seed time; queue probe drops 'redis' (34 to 18, 16 matched only on a data store); infrastructur karel.kubicek.claude
@@ Line 3: / Line 3: @@
 [[Programming:Docker]] gets you a browser you can pin. This page is about the month after you press start: what to persist, what to retry, what to watch, what it costs, and what to do at 3 a.m. on day four when the machine is gone and you have half a dataset.
-This is the "Execution of experiment, deployment" step of the [[Research journey]], and it is the step this corpus is quietest about. That silence is the page's first finding, not an excuse for it.
+This is the "Execution of experiment, deployment" step of the [[:Research journey]], and it is the step this corpus is quietest about. That silence is the page's first finding, not an excuse for it.
 <WRAP important>
-**Long runs are normal; saying how you ran them is not.** Of the **1,120 crawled papers** in this corpus, 230 give a live-crawl start and end date at day precision. Reading each one's own sentence, **182** of those spans are machine time rather than a content date range — and **138 of the 182 (75.8%)** ran for more than a week, **53 (29.1%)** for more than three months.
+**Long runs are normal; saying how you ran them is not.** Of the **1,120 crawled papers** in this corpus, 230 give a live-crawl start and end date at day precision. Reading each one's own sentence, **182** of those spans are one continuous stretch of machine time; the rest are content date ranges, disjoint episodes, or list-fetch dates. **138 of the 182 (75.8%)** of the continuous ones ran for more than a week, **53 (29.1%)** for more than three months.
-Against the same 1,120: **13 (1.2%)** mention checkpointing or resuming a crawl, **21 (1.9%)** name a cluster orchestrator, **16 (1.4%)** name a monitoring or logging stack, **8 (0.7%)** name anything that keeps a long process alive, and **10 (0.9%)** admit anywhere in the paper that their own measurement lost time. Every one of those is an upper bound: a match is a mention, not a use.
+Against the same 1,120: **13 (1.2%)** mention checkpointing or resuming a crawl, **21 (1.9%)** name a cluster orchestrator, **16 (1.4%)** name a monitoring or logging stack, **8 (0.7%)** name anything that keeps a long process alive, and **10 (0.9%)** are papers that a probe for interrupted measurements returns and a hand read confirms said their own run lost time. Every one of those is an upper bound: a match is a mention, not a use, and the last is a rate of //disclosure//, not of failure.
 </WRAP>
@@ Line 22: / Line 22: @@
 ===== How long a crawl actually runs =====
-Population: crawled papers with at least one ''temporal[]'' tuple whose mode is ''live-crawl'' and whose start **and** end are both day-precision. That is **230 of 1,120 crawled papers (20.5%)**; the rest give a month, a year, one endpoint, or nothing.
+Population: crawled papers with at least one ''temporal[]'' tuple whose mode is ''live-crawl'' and whose start **and** end are both day-precision. That is **230 of 1,120 crawled papers (20.5%)**; the rest give a month, a year, one endpoint, or nothing. Where a paper has several such tuples the **longest** is kept, so these figures describe the longest run each paper reports, not a typical one.
 The raw span is not the answer, because the extraction's ''live-crawl'' mode also fires on tuples whose dates describe the *collected content* rather than the running apparatus. The longest "crawl" in the corpus is 9,028 days, and its sentence reads "90,153 disputes decided between December 1999 and August 2024". Each of the 230 spans was classified by reading its own evidence quote:
@@ Line 69: / Line 69: @@
 ^ Probe ^ Papers ^ Share ^
 | ''checkpoint'' or resuming a crawl / scan / measurement | 13 | 1.2% |
-| a task or job queue by name (Celery, RabbitMQ, Redis, Kafka, SQS, ...) | 34 | 3.0% |
+| a task or job queue by name (Celery, RabbitMQ, Kafka, SQS, ...) | 18 | 1.6% |
 | a cluster orchestrator by name (Kubernetes, Swarm, Slurm, Nomad, Ansible, ...) | 21 | 1.9% |
 | a monitoring or logging stack by name (Prometheus, Grafana, Nagios, Kibana, ...) | 16 | 1.4% |
@@ Line 85: / Line 85: @@
 The thing a week-long crawl must not lose is not the pages — you can fetch those again — it is **the record of what happened to every URL you meant to visit**. A crawl that resumes from "the files I already have" quietly redefines its own denominator: a URL that failed terminally and a URL the crash never reached look identical on disk, and the paper ends up reporting a percentage of a number that cannot be reconstructed.
-So persist a row per URL, with exactly one terminal state, before you persist anything else. Three properties are worth getting right, and only the first is obvious:
+So persist a row per URL, with exactly one terminal state, before you persist anything else. Four properties are worth getting right, and only the first is obvious:
   - **Claim-then-commit with a lease and a named worker.** Mark the row ''running'' with a deadline before touching the network, so a killed process leaves work reclaimable rather than lost or double-counted. Note what the lease alone does //not// buy you: five minutes after the crash it has not expired, so a "reclaim expired leases" pass returns zero and the row stays stranded. That is correct with other workers alive — you must not steal a live worker's row — which is why the row also carries a worker id. The restarting process is the one thing in the system that //knows// worker ''w1'' is dead. Koop et al. describe the same shape from the other side: "nodes take jobs from the queue and execute them in individual Docker containers in parallel so that a problem in one of the tasks doesn't affect the other tasks" {[koop2020_redirect]}.
   - **Retryable and terminal errors are different.** NXDOMAIN will still be NXDOMAIN in four days; a timeout or a 503 often will not be. Retrying everything spends the budget on domains that no longer exist; retrying nothing throws away real data (see the next section).
+  - **A retry goes to the back of the queue, not the front.** This is one line of SQL and the obvious version gets it backwards. Ordering the pending rows by URL hands a just-failed row straight back on the next claim, because its position in the URL order has not changed and everything before it is already terminal — an immediate in-place retry against a host that refused you ten seconds ago. Ordering by attempt count first drains every first-attempt URL before any retry is taken. The published script asserts this, and the assertion fails if the ''ORDER BY'' is changed back.
   - **Fingerprint the seed list.** A resumed run that seeds a //different// list is not a resumption, it is a second study. Hashing the list on first insert and refusing a changed one is four lines and it is the difference between a denominator you can name and one you cannot.
-The script below is a working queue with those three properties, stdlib Python only (''sqlite3''), no network: the visit function is injected so the demo can kill the process deterministically and resume.
+The script below is a working queue with those four properties, stdlib Python only (''sqlite3''), no network: the visit function is injected so the demo can kill the process deterministically and resume. Its self-test also covers the two things that make a queue safe under the concurrency this page recommends — that ''claim()'' refuses a row whose UPDATE matched nothing, and that the claim order is a deterministic shuffle rather than the seed order.
 <file python crawl_queue.py>
@@ Line 112: / Line 113: @@
     running -> pending                    lease expired (the worker died)
-Three things this gets right that a dict-of-URLs in memory does not:
+Four things this gets right that a dict-of-URLs in memory does not:
 .  **Claim-then-commit with a lease, and a named worker.** A worker marks a
@@ Line 129: / Line 130: @@
     NXDOMAIN in four days; a timeout or a 503 often will not be. Retrying
     everything burns the budget on domains that no longer exist; retrying
-    nothing throws away real data — Bouchet et al. (USENIX Security 2025)
+    nothing throws away real data — Hausladen et al. (USENIX Security 2025)
     measured that nearly 40% of the sites that failed to load on the first
     attempt loaded on the second.
-.  **The failure classes are counted, not just the failures.** `progress()`
+.  **A retry goes to the back of the queue, not the front.** A URL that
+    returned 403 ten seconds ago will return 403 again; in ten hours, after
+    the bot-detection heuristic has forgotten you, it may not. That is one
+    line of SQL — ``ORDER BY attempts ASC, url ASC`` — and ordering by URL
+    alone silently gives you the opposite.
+.  **The failure classes are counted, not just the failures.** `progress()`
     returns the per-class breakdown, which is both the monitoring signal while
     the crawl runs and the attrition table the methods section owes a reviewer.
@@ Line 158: / Line 165: @@
     status       TEXT NOT NULL CHECK (status IN ('pending','running','done','failed')),
     attempts     INTEGER NOT NULL DEFAULT 0,
+    seq          TEXT NOT NULL,
     worker       TEXT,
     lease_until  REAL,
@@ Line 164: / Line 172: @@
     updated_at   REAL NOT NULL
 );
-CREATE INDEX IF NOT EXISTS work_status ON work(status);
+CREATE INDEX IF NOT EXISTS work_status ON work(status, attempts, seq);
 CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL);
 """
@@ Line 231: / Line 239: @@
         now = self.clock()
         before = self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0]
+        # seq is a deterministic shuffle: sha256(digest + url). Claim order is
+        # (attempts, seq), so the crawl walks the list in an order that is
+        # reproducible from the seed but uncorrelated with rank, TLD or name.
+        # It matters because a crawl that dies halfway through a rank-ordered
+        # queue loses the tail of the ranking systematically; the same crash on
+        # a shuffled queue loses a random subset, which is a far weaker claim
+        # to have to defend in a methods section.
         self.db.executemany(
-            "INSERT OR IGNORE INTO work(url,status,attempts,updated_at) VALUES(?,'pending',0,?)",
+            "INSERT OR IGNORE INTO work(url,status,attempts,seq,updated_at) "
-            [(u, now) for u in urls],
+            "VALUES(?,'pending',0,?,?)",
+            [(u, hashlib.sha256((digest + u).encode("utf-8")).hexdigest(), now) for u in urls],
         )
         return self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0] - before
@@ Line 267: / Line 283: @@
     def claim(self, worker: str = "w0"):
-        """Take one pending URL and mark it running. Returns None when drained."""
+        """Take one pending URL and mark it running. Returns None when drained.
-        now = self.clock()
-        row = self.db.execute(
+        Ordering is ``attempts ASC, url ASC``, not ``url ASC``. A row that
-            "SELECT url, attempts FROM work WHERE status='pending' ORDER BY url LIMIT 1"
+        failed retryably keeps its position in the URL order, so ordering by
-        ).fetchone()
+        URL alone hands it straight back on the next claim — an immediate
-        if row is None:
+        in-place retry, which is the worst time to try a host that just
-            return None
+        refused you. Ordering by attempts first drains every first-attempt URL
-        url, attempts = row
+        before any retry is taken.
-        self.db.execute(
+        """
-            "UPDATE work SET status='running', attempts=?, worker=?, lease_until=?, updated_at=? "
+        while True:
-            "WHERE url=? AND status='pending'",
+            now = self.clock()
-            (attempts + 1, worker, now + self.lease_seconds, now, url),
+            row = self.db.execute(
-        )
+                "SELECT url, attempts FROM work WHERE status='pending' "
-        return url, attempts + 1
+                "ORDER BY attempts ASC, seq ASC LIMIT 1"
+            ).fetchone()
+            if row is None:
+                return None
+            url, attempts = row
+            cur = self.db.execute(
+                "UPDATE work SET status='running', attempts=?, worker=?, lease_until=?, "
+                "updated_at=? WHERE url=? AND status='pending'",
+                (attempts + 1, worker, now + self.lease_seconds, now, url),
+            )
+            # SELECT-then-UPDATE is not atomic. Another worker can win the row
+            # between the two statements, and then this UPDATE matches zero
+            # rows. Returning the URL anyway would hand two workers the same
+            # site and let the second one overwrite the first's outcome — the
+            # exact double-count the lease exists to prevent. Loop instead.
+            if cur.rowcount == 1:
+                return url, attempts + 1
     # -- outcomes ---------------------------------------------------------
@@ Line 393: / Line 425: @@
     return visit
+class _FakeCursor:
+    """Just enough of a sqlite3 cursor for claim()'s SELECT."""
+    def __init__(self, row):
+        self._row = row
+    def fetchone(self):
+        return self._row
+class _RaceOnce:
+    """Connection wrapper that loses the SELECT/UPDATE race exactly once.
+    Used only by the demo. It passes everything through, except that the first
+    time claim() runs its SELECT it hands the row to a "rival" worker before
+    returning it — reproducing, deterministically, the interleaving that makes
+    an unchecked SELECT-then-UPDATE hand one URL to two workers.
+    """
+    def __init__(self, db):
+        self.db = db
+        self.armed = True
+        self.stolen = None
+    def execute(self, sql, params=()):
+        if self.armed and sql.startswith("SELECT url, attempts FROM work"):
+            self.armed = False
+            row = self.db.execute(sql, params).fetchone()
+            if row is not None:
+                self.stolen = row[0]
+                self.db.execute(
+                    "UPDATE work SET status='running', worker='rival' WHERE url=?",
+                    (row[0],),
+                )
+            return _FakeCursor(row)
+        return self.db.execute(sql, params)
+    def executemany(self, sql, params):
+        return self.db.executemany(sql, params)
+    def executescript(self, sql):
+        return self.db.executescript(sql)
@@ Line 436: / Line 512: @@
     print(f"final: {resumed.progress().line()}")
-    print("\n=== D. the seed guard ===")
+    print("\n=== D. does reclaim_expired() actually reclaim an expired lease? ===")
+    # The check in section C only shows that a FRESH lease is not reclaimed.
+    # A reclaim_expired() that did nothing at all would pass that. Drive a
+    # movable clock past the deadline and assert the row really comes back.
+    lease_db = os.path.join(tmp, "lease.sqlite")
+    fake = {"t": 1_000_000.0}
+    lq = CrawlQueue(lease_db, lease_seconds=300.0, clock=lambda: fake["t"])
+    lq.seed(sites[:3])
+    lq.claim("dead-worker")
+    before = lq.reclaim_expired()
+    fake["t"] += 301.0
+    after = lq.reclaim_expired()
+    print(f"lease still live  -> reclaim_expired() = {before}")
+    print(f"clock +301s       -> reclaim_expired() = {after}")
+    print("\n=== E. does a retry go to the back of the queue? ===")
+    # site007 fails retryably on attempts 1 and 2. With ORDER BY url it would
+    # be handed straight back; with ORDER BY attempts, url it must wait for
+    # every first-attempt URL.
+    order_db = os.path.join(tmp, "order.sqlite")
+    oq = CrawlQueue(order_db)
+    small = demo_sites(20)
+    oq.seed(small)
+    visit = make_visit()
+    order = []
+    while True:
+        claimed = oq.claim("w0")
+        if claimed is None:
+            break
+        url, attempt = claimed
+        order.append((url.rstrip("/").rsplit("site", 1)[1].split(".")[0], attempt))
+        try:
+            oq.complete(url, visit(url, attempt))
+        except Exception as exc:
+            oq.record_error(url, getattr(exc, "error_class", "unknown"))
+    first_pass = [u for u, a in order if a == 1]
+    retries = [i for i, (u, a) in enumerate(order) if a > 1]
+    print(f"claim order (site, attempt): {order[:12]} ...")
+    print(f"first-attempt claims: {len(first_pass)}; first retry at position {retries[0]}")
+    print("\n=== F. does claim() refuse a row it did not win? ===")
+    # SELECT-then-UPDATE is not atomic, and the window between them is the only
+    # place two workers can be handed the same URL. Marking the row running
+    # BEFORE claim() runs does not test that window — the SELECT simply skips
+    # it. _RaceOnce steals the row in the window itself: it lets the SELECT
+    # return normally and then marks that row running behind claim()'s back,
+    # so claim()'s own UPDATE matches zero rows. Without the rowcount check,
+    # claim() returns the stolen URL and two workers crawl the same site.
+    race_db = os.path.join(tmp, "race.sqlite")
+    rq = CrawlQueue(race_db)
+    rq.seed(sites[:5])
+    rq.db = _RaceOnce(rq.db)
+    got = rq.claim("w0")
+    stolen = rq.db.stolen
+    print(f"a rival took this row inside the window: {stolen}")
+    print(f"claim() returned:                       {got[0] if got else None}")
+    race_ok = got is not None and got[0] != stolen
+    print("\n=== G. is the claim order shuffled, and reproducible? ===")
+    order_a = [u for u, in oq.db.execute(
+        "SELECT url FROM work ORDER BY seq ASC").fetchall()]
+    rerun_db = os.path.join(tmp, "rerun.sqlite")
+    rq2 = CrawlQueue(rerun_db)
+    rq2.seed(small)
+    order_b = [u for u, in rq2.db.execute(
+        "SELECT url FROM work ORDER BY seq ASC").fetchall()]
+    print(f"seed order:    {[u[12:15] for u in small[:8]]} ...")
+    print(f"claim order:   {[u[12:15] for u in order_a[:8]]} ...")
+    print(f"same list, new database, same order: {order_a == order_b}")
+    print("\n=== H. the seed guard ===")
     try:
         resumed.seed(sites + ["https://site200.example/"])
@@ Line 444: / Line 590: @@
         print(f"refused, as it should: {str(exc).splitlines()[0]}")
-    print("\n=== E. checks ===")
+    print("\n=== I. checks ===")
     ok = True
     checks = [
@@ Line 455: / Line 601: @@
         ("the lease alone did not free the stranded row", by_lease == 0),
         ("naming the dead worker did", by_worker == 1),
+        ("a live lease is not reclaimed", before == 0),
+        ("an expired lease IS reclaimed", after == 1),
+        ("every first attempt precedes every retry", len(first_pass) == len(small)
+            and (not retries or retries[0] == len(small))),
+        ("claim() refuses a row it did not win", race_ok),
+        ("claim order is not the seed order", order_a != small),
+        ("claim order is reproducible from the seed", order_a == order_b),
         ("in-flight URL was retried, not lost", stranded == 1),
     ]
@@ Line 489: / Line 642: @@
 === A. reference run, no crash (200 URLs) ===
 seeded 200 rows, seed sha256 49ddaed11cb3d529...
-  [worker] total=200 done=51 failed=21 running=0 pending=128 | http_404=7, nxdomain=7, timeout=7
+  [worker] total=200 done=61 failed=21 running=0 pending=118 | http_404=9, nxdomain=12
-  [worker] total=200 done=102 failed=42 running=0 pending=56 | http_404=14, nxdomain=14, timeout=14
+  [worker] total=200 done=120 failed=40 running=0 pending=40 | http_404=20, nxdomain=20
 visits attempted: 280
 final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20
 === B. same population, machine dies on the 87th visit ===
-killed while visiting https://site062.example/
+killed while visiting https://site013.example/
-state on disk: total=200 done=44 failed=18 running=1 pending=137 | http_404=6, nxdomain=6, timeout=6
+state on disk: total=200 done=51 failed=19 running=1 pending=129 | http_404=7, nxdomain=12
 rows left in 'running' (the in-flight visit): 1
@@ Line 503: / Line 656: @@
 reclaim_expired() reclaimed 0 row(s) -- the 300s lease has not run out yet
 reclaim_worker('w1') reclaimed 1 row(s) -- w1 is the process that died
-  [worker] total=200 done=95 failed=39 running=0 pending=66 | http_404=13, nxdomain=13, timeout=13
+  [worker] total=200 done=113 failed=36 running=0 pending=51 | http_404=17, nxdomain=19
 visits attempted after resume: 194
 final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20
-=== D. the seed guard ===
+=== D. does reclaim_expired() actually reclaim an expired lease? ===
+lease still live  -> reclaim_expired() = 0
+clock +301s       -> reclaim_expired() = 1
+=== E. does a retry go to the back of the queue? ===
+claim order (site, attempt): [('010', 1), ('002', 1), ('008', 1), ('003', 1), ('017', 1), ('001', 1), ('006', 1), ('012', 1), ('016', 1), ('015', 1), ('009', 1), ('005', 1)] ...
+first-attempt claims: 20; first retry at position 20
+=== F. does claim() refuse a row it did not win? ===
+a rival took this row inside the window: https://site004.example/
+claim() returned:                       https://site000.example/
+=== G. is the claim order shuffled, and reproducible? ===
+seed order:    ['000', '001', '002', '003', '004', '005', '006', '007'] ...
+claim order:   ['010', '002', '008', '003', '017', '001', '006', '012'] ...
+same list, new database, same order: True
+=== H. the seed guard ===
 refused, as it should: seed list changed since this queue was created (stored 49ddaed11cb3..., given f1c74ecfee8e...). Use a new database, or you cannot name your denominator.
-=== E. checks ===
+=== I. checks ===
   ok   same successful set
   ok   same failure set
@@ Line 519: / Line 689: @@
   ok   the lease alone did not free the stranded row
   ok   naming the dead worker did
+  ok   a live lease is not reclaimed
+  ok   an expired lease IS reclaimed
+  ok   every first attempt precedes every retry
+  ok   claim() refuses a row it did not win
+  ok   claim order is not the seed order
+  ok   claim order is reproducible from the seed
   ok   in-flight URL was retried, not lost
   note  clean run: 280 visits. crash+resume: 86 before the crash, 1 interrupted, 194 after = 281 calls for the same 200 terminal rows. The excess is the one interrupted visit.
@@ Line 525: / Line 701: @@
 </code>
-The check that matters is ''same successful set'' and ''same failure set'': the crashed-and-resumed run ends with byte-identical outcomes to the run that was never interrupted, and the one visit that was in flight is retried rather than silently dropped. The two lines above it are the lesson: ''reclaim_expired()'' returns 0, ''reclaim_worker('w1')'' returns 1.
+The check that matters is ''same successful set'' and ''same failure set'': the crashed-and-resumed run ends with byte-identical outcomes to the run that was never interrupted, and the one visit that was in flight is retried rather than silently dropped. Section C is the lesson about leases — ''reclaim_expired()'' returns 0 and ''reclaim_worker('w1')'' returns 1 — and section D exists because a check that only ever sees a fresh lease would pass even if ''reclaim_expired()'' did nothing at all. Both of those checks were added after a reviewer replaced the function body with ''return 0'' and watched the self-test still print OK.
 ===== Retries: what to retry, and what it buys =====
-Retrying once is the single highest-return operational decision on this page, and one paper measured it. Hausladen et al. reload a site that failed on the first attempt, and report: "nearly 40% of sites that failed to load on the first attempt were loaded and successfully analyzed" on the second {[hausladen2025_websites]}. Their first-attempt failure conditions are worth copying verbatim, because they are the three that matter: "a site failed to load in 35 seconds, had an insecure certificate, or led to an error page".
+Retrying once is the single highest-return operational decision on this page, and one paper measured the retry itself. Hausladen et al. reload a site that failed on the first attempt, and report: "nearly 40% of sites that failed to load on the first attempt were loaded and successfully analyzed" on the second {[hausladen2025_websites]}. Their first-attempt failure conditions are worth copying verbatim, because they are the three that matter: "a site failed to load in 35 seconds, had an insecure certificate, or led to an error page".
-What the retry is worth depends on what fraction of your list fails at all. Papers that report it:
+What the retry is worth depends on what fraction of your list fails at all. Seven papers whose attrition figures were surfaced by the retry and attrition probes and then read and verified individually — **not** an exhaustive list; the attrition probe suggests of the order of 38 crawled papers report something of this kind:
 ^ Paper ^ Population ^ Reached ^ What the rest were ^
 | Kumar et al., TheWebConf 2017 {[kumar2017_security]} | Alexa top 1M | 944,000 | 15K did not resolve, 13K timed out, 24K returned an HTTP error, 5K would not render |
 | Murley et al., TheWebConf 2021 {[murley2021_websocket]} | top 1M, **after one retry** | 88.1% | "7.7% of the listed domain names failed to resolve, and the remaining 4.2% had web servers which failed to respond" |
-| Musch et al., USENIX Sec 2021 {[musch2021_debug]} | Tranco top 1M | ~846K sites | ~8% network/DNS, "In another 4%, the server returned an HTTP error code and the remaining 3% failed to load before our 30 seconds timeout hit" |
+| Musch and Johns, USENIX Sec 2021 {[musch2021_debug]} | Tranco top 1M | ~846K sites | ~8% network/DNS, "In another 4%, the server returned an HTTP error code and the remaining 3% failed to load before our 30 seconds timeout hit" |
 | Annamalai et al., NDSS 2024 {[annamalai2024_fpfed]} | "the 20k websites" | 18,300 (91.5%) | "the overwhelming majority (64.3%) of them due to HTTP 403 Forbidden errors" |
 | Kieserman et al., PoPETs 2025 {[kieserman2025_tracker]} | 42,481 sites visited | 40,150 (94.51%) | inaccessible domains, bot detection, general unreliability |
-| Kuchhal and Li, IMC 2021 {[kuchhal2021_knock]} | Tranco top lists, two waves | ~90% | not broken out |
+| Kuchhal and Li, IMC 2021 {[kuchhal2021_knock]} | Tranco top 100K domains | ~90% | "Nearly 90% of the failures were due to DNS resolution errors (NAME_NOT_RESOLVED)" |
 | Demir et al., PoPETs 2024 {[demir2024_bannertools]} | a Tranco sample | 29,660 (99%) | not broken out |
-Two patterns to take from that table. A top-1M crawl reaches roughly 88–94% of its list and a smaller, more popular list reaches 91–99%; if your number is far below that, the problem is your deployment, not the web. And the failure mix moves over time — 403 Forbidden dominating a 2024 crawl {[annamalai2024_fpfed]} is bot detection, not a network problem, and no amount of retrying fixes it.
+Two patterns to take from that table. A top-1M crawl reaches roughly 85–94% of its list — Musch and Johns's ~846K is 84.6%, the lowest here — and a smaller or more popular list reaches roughly 90–99%. Those are the published range, not a target: they come from seven hand-read papers, not from a census. And the failure mix moves over time — 403 Forbidden dominating a 2024 crawl {[annamalai2024_fpfed]} is bot detection, not a network problem, and no amount of retrying fixes it.
 Retries also come in shapes other than "do it again". Ikram et al. retry with a **changed parameter**: "we checked if the rendering of any webpage stalled. If so, we reran the crawler on the webpage by increasing the waiting time until the webpage is rendered" {[ikram2017_seamless]}. Khodayari et al. simply bound it — "we made up to three repeated attempts for each failed crawling" {[khodayari2024_great]}. Senol et al. record what a retry-shaped fix was worth: a minor change "increased the successfully visited websites from 94,427 (EU pilot crawl) to 99,380 (EU final crawl)" {[senol2022_leaky]}.
 <WRAP tip>
-Retry at the end, not in place. A URL that 403s now will 403 in ten seconds; it may not in ten hours, from a different exit, after the bot-detection heuristic has forgotten you. A queue with a retry budget gives you this for free — the row goes back to ''pending'' and gets picked up after everything else in the list.
+Retry at the end, not in place. A URL that 403s now will 403 in ten seconds; it may not in ten hours, from a different exit, after the bot-detection heuristic has forgotten you. You do not get this for free — a queue that hands out the lowest pending URL gives you the opposite, because a requeued row is still the lowest pending URL. Order by attempt count first.
 </WRAP>
 ===== Concurrency and throughput =====
-**29 of 1,120 crawled papers (2.6%)** state how many browsers or workers they ran at once. That is the whole planning literature, and it is a probe result with a hand-read verdict per hit: a further 4 say they were parallel without a number, 5 parallelise an analysis stage rather than the crawl, and 6 are about something else entirely.
+**29 of 1,120 crawled papers (2.6%)** state how many browsers or workers they ran at once. That is nearly all the published guidance there is on sizing a crawl machine, and it is a probe result with a hand-read verdict on every hit: a further 4 say they were parallel without a number, 5 parallelise an analysis stage rather than the crawl, and 6 are about something else entirely.
+Six of the 29, chosen to span the range of scale in the class — one desktop, one cloud VM, one departmental cluster, and the three papers that also give a throughput or a wall clock:
 ^ Paper ^ Hardware ^ Concurrency ^
@@ Line 564: / Line 742: @@
 The one throughput figure anyone publishes is Bouhoula et al.'s: **300 websites per hour from 30 parallel OpenWPM browsers on 16 cores** {[bouhoula2024_automated]} — about 10 sites per browser-hour, or 7,200 sites a day per machine. Take it as the order of magnitude for an instrumented, stateful, banner-interacting crawl, not as a target: a stateless landing-page fetch is much faster and a crawl that clicks through consent and visits subpages is much slower ([[Programming:Interaction]]).
-That single number is enough to plan with. A one-million-site crawl at 300 sites per machine-hour is about 3,300 machine-hours: five days on 28 machines, or a month on four. Englehardt and Narayanan's ratio is the constraint behind it — "stateful parallel measurements are memory-limited while stateless parallel measurements are typically CPU-limited and can support a higher number of instances" {[englehardt2016online]}. Concurrency per machine is set by RAM if you keep profiles and by cores if you do not.
+That single number is enough to plan with. A one-million-site crawl at 300 sites per machine-hour is about 3,300 machine-hours: five days on 28 machines, or five weeks on four. Englehardt and Narayanan's ratio is the constraint behind it — "stateful parallel measurements are memory-limited while stateless parallel measurements are typically CPU-limited and can support a higher number of instances" {[englehardt2016online]}. Concurrency per machine is set by RAM if you keep profiles and by cores if you do not.
 ===== Monitoring =====
-**16 of 1,120 crawled papers (1.4%)** name a monitoring stack, so there is no literature here to summarise. What there is, is a list of the things that went wrong in the ten papers that admitted something went wrong (next section), and every one of them is visible in a counter you could have been watching:
+**16 of 1,120 crawled papers (1.4%)** name a monitoring or logging product. That probe only sees brand names — a paper that describes watching its own crawl without naming a product is invisible to it — but 16 named-tool papers is not a literature to systematise either. What there is, is a list of the things that went wrong in the ten papers that admitted something went wrong (next section), and every one of them is visible in a counter you could have been watching:
   * **Terminal rows per hour.** Not "pages fetched" — rows that reached ''done'' or ''failed''. A crawl whose throughput halves is usually a crawl where retries have started dominating, which is a signal about the target, not about your disk.
@@ Line 576: / Line 754: @@
   * **A canary that is not part of the crawl.** A page you control, fetched on the same schedule through the same stack. When the numbers move, it tells you whether the web changed or you did.
-Two failure modes deserve naming because a dashboard of totals hides both. A crawl can produce a **complete-looking empty database** — [[Programming:Crawler:OpenWPM]] documents this for OpenWPM specifically, where all fourteen tables exist and are empty after a failed run. And a crawl can keep running while measuring the wrong thing: Yeung et al.'s crawler "crashed on March 12th in both Ukraine and Russia, and again on March 20th and April 18th in Ukraine after running for some time" {[yeung2023_online]} — a per-vantage-point counter shows that, a global one does not.
+Two failure modes deserve naming because a dashboard of totals hides both. A crawl can produce a **complete-looking empty database** — [[Programming:Crawler:OpenWPM]] documents this for OpenWPM specifically, where a run in which every browser failed to launch still produces a database with all of OpenWPM's tables present and empty. And a crawl can keep running while measuring the wrong thing: Yeung et al.'s crawler "crashed on March 12th in both Ukraine and Russia, and again on March 20th and April 18th in Ukraine after running for some time" {[yeung2023_online]} — a per-vantage-point counter shows that, a global one does not.
 ===== Cost =====
-**5 of 1,120 crawled papers (0.4%) state an infrastructure bill.** A further 3 state what they paid to acquire data — services, participants, annotation — so **8 (0.7%)** state any cost they themselves paid. Eleven other papers put a currency figure next to a cost word in a measurement context, and in every one of those the price belongs to what was being measured, not to the measuring. This field reports the adversary's economics in detail and its own hardly at all.
+A probe for a currency figure beside a cost word in a measurement context returns **19 of 1,120 crawled papers (1.7%)**. Reading them: **5 state an infrastructure bill**, 3 state what they paid to acquire data — services, participants, annotation — and in the other 11 the price belongs to what was being measured, not to the measuring. So **at least 8 (0.7%)** state a cost they themselves paid. Unlike the other probes on this page these counts are **lower** bounds, and demonstrably so: the first version of this regex found 2 of the 5 infrastructure papers and missed Englehardt and Narayanan's own line about the machine that ran their 1-million-site crawl. This field reports the adversary's economics in detail and its own hardly at all.
-The five, in full, are the entire published record:
+All eight, in full. The **Kind** column matters: only the first five are a machine bill, and the row that classifies a paper is the one whose sentence the probe matched, not the paper's subject.
-^ Paper ^ What it cost ^
+^ Paper ^ Kind ^ What it cost ^
-| Wondracek et al., IEEE S&P 2010 {[wondracek2010_practical]} | "The crawling service cost us $18.47" for "7.1GB of HTML data" of Facebook groups; they estimate $88 to crawl all 40 million public profiles |
+| Wondracek et al., IEEE S&P 2010 {[wondracek2010_practical]} | infrastructure | Two separate outsourced crawls: "The crawling service cost us $18.47" for "7.1GB of HTML data" of Facebook groups, and, for three million LinkedIn profiles, "The costs for the crawling were $6.57 ... we estimate overall costs of about $88 for crawling all 40 million public profiles" |
-| Thomas et al., IEEE S&P 2011 {[thomas2011_design]} | An itemised monthly AWS bill: "URL aggregation 1 Extra Large $178 Feature collection 20 High-CPU Medium $882", classification $527, "Storage 700GB on EBS $70 Total $1,587" |
+| Thomas et al., IEEE S&P 2011 {[thomas2011_design]} | infrastructure | An itemised monthly AWS bill: "URL aggregation 1 Extra Large $178 Feature collection 20 High-CPU Medium $882", classification $527, "Storage 700GB on EBS $70 Total $1,587" |
-| Englehardt and Narayanan, CCS 2016 {[englehardt2016online]} | "This virtual machine costs around $300 per month using price estimates from May 2016" — the machine that ran the 1-million-site crawl |
+| Englehardt and Narayanan, CCS 2016 {[englehardt2016online]} | infrastructure | "This virtual machine costs around $300 per month using price estimates from May 2016" — the machine that ran the 1-million-site crawl |
-| Zhang et al., CCS 2023 {[zhang2023_under]} | "our overall purchase cost is $44.538 for one month ($40 for renting servers and $4.538 for registering domains)" |
+| Genkin et al., USENIX Sec 2022 {[genkin2022_lend]} | infrastructure | An attack evaluation, not a crawl, but a real EC2 bill: the key was extracted "at a total computational cost of less than $1.3" on a ''c5.9xlarge'' |
-| Song et al., PoPETs 2026 {[song2026_wfpllm]} | "acquiring 1GB of authentic human browsing data costs approximately $35, primarily due to participant compensation and lab infrastructure" |
+| Zhang et al., CCS 2023 {[zhang2023_under]} | infrastructure | "our overall purchase cost is $44.538 for one month ($40 for renting servers and $4.538 for registering domains)" |
+| Qiu et al., USENIX Sec 2023 {[qiu2023_calpric]} | data | Crowdsourced labelling: prior work "reported a cost of $60 to label each privacy policy. Our system is able to further reduce the average cost of $13.5" |
+| Pu et al., IEEE S&P 2023 {[pu2023_deepfake]} | data | "We spent $586 to collect the articles from the services" |
+| Song et al., PoPETs 2026 {[song2026_wfpllm]} | data | "acquiring 1GB of authentic human browsing data costs approximately $35, primarily due to participant compensation and lab infrastructure" |
-Since nobody will tell you, work it out. The arithmetic has four terms and only the first is large:
+Eight figures spread over sixteen years and four pricing models will not size your grant. Work it out instead. The arithmetic has four terms and only the first is large:
-  - **Machine-hours** = (sites × attempts) ÷ (workers per machine × sites per worker-hour). At Bouhoula et al.'s 300 sites/hour/machine {[bouhoula2024_automated]}, a one-million-site pass is ~3,300 machine-hours. On-demand Linux in ''eu-central-1'' on 2026-08-28 was **USD 0.4074/hour** for a ''c7i.2xlarge'' (8 vCPU) and **USD 0.8148/hour** for a ''c7i.4xlarge'' (16 vCPU), read out of the AWS bulk price list.((AWS EC2 bulk price list, offer version ''20260828175757'', ''eu-central-1'', Linux, shared tenancy, on-demand. Fetched 2026-08-28 by ''scripts/ec2_price.py'' — the region offer file is about 431 MB, so the script streams it. Re-fetch before quoting: these move.)) So ~3,300 machine-hours on 16-core instances is roughly **USD 2,700** for one pass — the same order as Thomas et al.'s $1,587/month fifteen years ago, which is the useful observation: the hardware got faster and the crawl got heavier.
+  - **Machine-hours** = (sites × attempts) ÷ (workers per machine × sites per worker-hour). At Bouhoula et al.'s 300 sites/hour/machine {[bouhoula2024_automated]}, a one-million-site pass is ~3,300 machine-hours. On-demand Linux in ''eu-central-1'' on 2026-08-28 was **USD 0.4074/hour** for a ''c7i.2xlarge'' (8 vCPU) and **USD 0.8148/hour** for a ''c7i.4xlarge'' (16 vCPU), read out of the AWS bulk price list.((AWS EC2 bulk price list, offer version ''20260828175757'', ''eu-central-1'', Linux, shared tenancy, on-demand. Fetched 2026-08-28 by ''scripts/ec2_price.py'' — the region offer file is about 431 MB, so the script streams it. Re-fetch before quoting: these move.)) So ~3,300 machine-hours on ''c7i.4xlarge'' instances is roughly **USD 2,700** for one pass. Treat that as a floor: a ''c7i.4xlarge'' is 16 vCPU, which on a hyperthreaded Intel instance is 8 physical cores against the 16 physical cores of Bouhoula et al.'s Ryzen, so the same 300 sites per hour will want a bigger instance or a longer wall clock. Either way it is the same order as Thomas et al.'s $1,587/month fifteen years ago, which is the useful observation: the hardware got faster and the crawl got heavier.
   - **A second pass.** Budget for it. Every longitudinal design needs one and every first pass has a bug in it.
   - **Egress and storage.** Small for HTML and headers, not small for screenshots or full HAR files ([[Programming:Traffic files]]). Measure one visit's output on day one and multiply.
@@ Line 600: / Line 781: @@
 ===== The machine dies on day four =====
-Ten papers in this corpus — **0.9% of the 1,120 that crawled** — say in print that their own measurement lost time. That is not a rate of machines dying. It is a rate of people writing it down, and these ten are the template for how to do it.
+A probe for interrupted measurements returns 21 of the 1,120 crawled papers; reading each one leaves **10 (0.9%)** whose own measurement lost time, 3 where the //measured// site went down, 1 planned outage, 1 explicit claim of no downtime, and 6 off-topic. That 0.9% is not a rate of machines dying. It is a rate of people writing it down, and these ten are the template for how to do it.
 ^ Paper ^ What happened ^ What they wrote ^
@@ Line 611: / Line 792: @@
 | Lee et al., TheWebConf 2021 {[lee2021_practice]} | network outages | "There were network outages for 17 days, which are pruned out from the dataset"; "around 84% of the websites were consistently collected" |
 | Poteat and Li, IMC 2021 {[poteat2021_securitytxt]} | a six-week gap in a 15-month series | "we had a 1.5-month measurement interruption throughout November and the first half of December, 2020" |
-| Yeung et al., TheWebConf 2023 {[yeung2023_online]} | repeated crawler crashes, per vantage point | "Our crawler crashed on March 12th in both Ukraine and Russia, and again on March 20th and April 18th in Ukraine" |
+| Yeung et al., TheWebConf 2023 {[yeung2023_online]} | repeated crawler crashes, per vantage point | "Our crawler crashed on March 12th in both Ukraine and Russia, and again on March 20th and April 18th in Ukraine after running for some time" |
 | Dahlberg and Pulls, USENIX Sec 2023 {[dahlberg2023_timeless]} | Tor exit downtime plus a network-wide DDoS | the dips are visible in their own published counters |
-Only one paper in the corpus went the other way and claimed a clean run — "none of the three ROBOVIC instances experienced any downtime during our monitored period" {[miramirkhani2017_dial]} — and it is worth noticing that they needed the claim because their result depended on the three instances being comparable. Agten et al. are the one case of an outage that was known in advance: they raised the crawl rate the week before "a planned power interruption of our crawling machines on August 27 and 28" {[agten2015_seven]}.
+Only one of the 21 papers that probe returns went the other way and claimed a clean run — "none of the three ROBOVIC instances experienced any downtime during our monitored period" {[miramirkhani2017_dial]} — and it is worth noticing that they needed the claim because their result depended on the three instances being comparable. Agten et al. are the one case of an outage that was known in advance: they raised the crawl rate the week before "a planned power interruption of our crawling machines on August 27 and 28" {[agten2015_seven]}.
 **What to actually do, in order:**
   - **Do not restart the crawl.** Resume it. If your queue has per-URL outcomes and a lease, the only work at risk is what was in flight, and the demo above shows the recovered run ending with the same outcome set as the uninterrupted one. If it does not, you now have to choose between a partial dataset and re-crawling a web that has moved under you — and re-crawling is a *different measurement*, not a repeat.
-  - **Record the gap in the same units as your results.** Not "we had some downtime" — Leontiadis et al.'s "1 004 of 1 254 days" is the model, because a reader can divide it. If your unit is sites, say how many sites the gap cost. If your unit is days of a time series, say which days.
+  - **Record the gap in the same units as your results.** Not "we had some downtime". Leontiadis et al. report complete measurements for 1,004 of the 1,254 days in their measurement period {[leontiadis2014_nearly]}, and that is the model, because a reader can divide it. If your unit is sites, say how many sites the gap cost. If your unit is days of a time series, say which days.
   - **Decide what the gap does to your inference, and say so.** Wang et al. quantified it as 4.5% of broadcasts missing {[wang2016_anatomy]}; Lee et al. pruned the 17 affected days out of the dataset entirely rather than analysing a series with holes in it {[lee2021_practice]}. Both are defensible. Neither is "we lost some data".
-  - **Check that the gap is not correlated with what you measure.** A crawl that dies overnight loses the sites that sort late in the list; a crawl that dies on Fridays loses whatever is different about weekends. If your queue orders by rank, a mid-run failure is a systematic loss, not a random one — shuffle the queue at seed time, or say the loss was ordered.
+  - **Check that the gap is not correlated with what you measure.** A crawl that dies overnight loses the sites that sort late in the list; a crawl that dies on Fridays loses whatever is different about weekends. If your queue orders by rank, a mid-run failure is a systematic loss, not a random one — shuffle the queue at seed time, or say the loss was ordered. The script above shuffles deterministically, on a hash of the seed digest and the URL, so the order is reproducible from the seed list and uncorrelated with rank.
   - **Check whether the environment changed while it was down.** A restart after a package update is a new instrument. [[Design:Longitudinal]] covers pinning; the deployment consequence is that "we restarted" and "we upgraded and restarted" are different sentences in a methods section.
   - **Then decide whether to extend.** Extending the window to recover lost coverage stretches the measurement period, which is a temporal confound, not a free fix. Say which you did.
@@ Line 630: / Line 811: @@
   - **The wall-clock window**, with both endpoints at day precision. Only 230 of 1,120 crawled papers manage this, and it is the cheapest sentence on the list.
-  - **Whether that window is machine time or content coverage.** A quarter of the day-precision spans in this corpus are ambiguous on exactly this point, and readers cannot tell them apart.
+  - **Whether that window is machine time or content coverage.** A fifth of the day-precision spans in this corpus (48 of 230) turned out to measure something other than one continuous run, and only reading each paper's own sentence separated them.
-  - **The population, and how much of it you reached** — the numerator, the denominator, and the failure classes underneath. "We crawled the Tranco top 1M" is not a result; "we reached 88.1% after one retry, 7.7% did not resolve, 4.2% did not respond" is {[murley2021_websocket]}.
+  - **The population, and how much of it you reached** — the numerator, the denominator, and the failure classes underneath. "We crawled the Tranco top 1M" is not a result. Murley et al. reached 88.1% of the top million after one retry, with 7.7% failing to resolve and 4.2% whose servers did not respond — that is {[murley2021_websocket]}.
   - **The retry policy**: how many attempts, which errors were terminal, and whether the retry was immediate or at the end of the queue. If a retry pass changed your numbers, say by how much {[hausladen2025_websites]}.
-  - **Concurrency and hardware**, in the same sentence: N browsers on M cores with K GB. 29 papers do this and they are the only ones another group can size a machine from.
+  - **Concurrency and hardware**, in the same sentence: N browsers on M cores with K GB. 29 papers state a concurrency figure; fewer pair it with the hardware, and those are the only ones another group can size a machine from.
-  - **The infrastructure**: cloud provider and region, university network, or testbed. 61.4% of crawled papers say nothing here, which also makes their vantage point unverifiable ([[Design:Crawling location]]).
+  - **The infrastructure**: cloud provider and region, university network, or testbed. 675 of 1,120 crawled papers (60.3%) name none, which also makes their vantage point unverifiable ([[Design:Crawling location]]). (60.3% is the complement of the 39.7% that do; the ''not-stated'' row is 61.4% because 44 papers carry both a stated and a not-stated tuple. Subtracting a sentinel row from the population is the wrong way to get this number.)
   - **Any interruption, in the units of your result**, and what you did with the gap.
-  - **The cost**, if you can. Five papers in sixteen years of seven venues have done it, and every student planning a crawl has to guess because of that.
+  - **The cost**, if you can. Across seventeen publication years of seven venues, five papers state an infrastructure bill and three more state what their data acquisition cost, and every student planning a crawl has to guess because of that.
 ===== Related pages =====
@@ Line 655: / Line 836: @@
 Corpus figures come from ''scripts/report_deployment.mjs'' against ''data/extract/run1'' (5,859 papers; CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P; 2010–2026). The denominator throughout is the **1,120 papers that ran a crawl**, never 5,859.
-Three limits worth stating plainly. **The duration figures rest on a hand classification**, not on the extraction: ''temporal[].mode == "live-crawl"'' does not distinguish machine time from content coverage, and 48 of 230 spans had to be excluded by reading. **Every operating figure is a full-text probe**, so it counts papers that used a word, not papers that did the thing; the retry row moves by a factor of two depending on how wide the regex is, and both widths are printed above. **The attrition probe is weak**: it returns 75 papers, and a hand-read sample of 12 was right half the time, so no reporting rate is published from it. Per-paper figures were checked against ''paper.cols.txt'' with whitespace collapsed — 41 of 41 found verbatim. The AWS prices are a single fetch on 2026-08-28 and will be wrong soon. Year shares are not used on this page; where 2025–2026 papers appear they are individual citations, not a trend ([[Literature:Corpus]]).
+Three limits worth stating plainly. **The duration figures rest on a hand classification**, not on the extraction: ''temporal[].mode == "live-crawl"'' does not distinguish machine time from content coverage, and 48 of 230 spans had to be excluded by reading. **Every operating figure is a full-text probe**, so it counts papers that used a word, not papers that did the thing; the retry row moves by a factor of two depending on how wide the regex is, and both widths are printed above. **The attrition probe is weak**: it returns 75 papers, and a hand-read sample of 12 was right half the time, so no reporting rate is published from it. Per-paper figures were checked against the papers with whitespace collapsed — 52 of 52 found verbatim, 51 in ''paper.cols.txt'' and one only in the PDF, because the ''.cols'' rendering splits its thousands separators. The AWS prices are a single fetch on 2026-08-28 and will be wrong soon. Year shares are not used on this page; where 2025–2026 papers appear they are individual citations, not a trend ([[Literature:Corpus]]).
 <bibtex bibliography></bibtex>
 ~~DISCUSSION~~