Measure The Web

User Tools

Site Tools

Trace:
programming:deployment

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
programming:deployment [2026/08/28 20:40] – Review round 1: fix misattributed retry figure in the published script (Hausladen, not Bouchet); queue now retries at the back of the queue, with a mutation-tested self-check; lease-expiry check added; cost table corrected to all 8 own-cost papers with a karel.kubicek.claudeprogramming:deployment [2026/08/28 21:04] (current) – Review round 2 (generic): claim() now checks it won the row (SELECT-then-UPDATE was handing one URL to two workers) and the queue shuffles deterministically at seed time; queue probe drops 'redis' (34 to 18, 16 matched only on a data store); infrastructur karel.kubicek.claude
Line 6: Line 6:
  
 <WRAP important> <WRAP important>
-**Long runs are normal; saying how you ran them is not.** Of the **1,120 crawled papers** in this corpus, 230 give a live-crawl start and end date at day precision. Reading each one's own sentence, **182** of those spans are machine time rather than a content date range — and **138 of the 182 (75.8%)** ran for more than a week, **53 (29.1%)** for more than three months.+**Long runs are normal; saying how you ran them is not.** Of the **1,120 crawled papers** in this corpus, 230 give a live-crawl start and end date at day precision. Reading each one's own sentence, **182** of those spans are one continuous stretch of machine time; the rest are content date ranges, disjoint episodes, or list-fetch dates. **138 of the 182 (75.8%)** of the continuous ones ran for more than a week, **53 (29.1%)** for more than three months.
  
 Against the same 1,120: **13 (1.2%)** mention checkpointing or resuming a crawl, **21 (1.9%)** name a cluster orchestrator, **16 (1.4%)** name a monitoring or logging stack, **8 (0.7%)** name anything that keeps a long process alive, and **10 (0.9%)** are papers that a probe for interrupted measurements returns and a hand read confirms said their own run lost time. Every one of those is an upper bound: a match is a mention, not a use, and the last is a rate of //disclosure//, not of failure. Against the same 1,120: **13 (1.2%)** mention checkpointing or resuming a crawl, **21 (1.9%)** name a cluster orchestrator, **16 (1.4%)** name a monitoring or logging stack, **8 (0.7%)** name anything that keeps a long process alive, and **10 (0.9%)** are papers that a probe for interrupted measurements returns and a hand read confirms said their own run lost time. Every one of those is an upper bound: a match is a mention, not a use, and the last is a rate of //disclosure//, not of failure.
Line 22: Line 22:
 ===== How long a crawl actually runs ===== ===== How long a crawl actually runs =====
  
-Population: crawled papers with at least one ''temporal[]'' tuple whose mode is ''live-crawl'' and whose start **and** end are both day-precision. That is **230 of 1,120 crawled papers (20.5%)**; the rest give a month, a year, one endpoint, or nothing.+Population: crawled papers with at least one ''temporal[]'' tuple whose mode is ''live-crawl'' and whose start **and** end are both day-precision. That is **230 of 1,120 crawled papers (20.5%)**; the rest give a month, a year, one endpoint, or nothing. Where a paper has several such tuples the **longest** is kept, so these figures describe the longest run each paper reports, not a typical one.
  
 The raw span is not the answer, because the extraction's ''live-crawl'' mode also fires on tuples whose dates describe the *collected content* rather than the running apparatus. The longest "crawl" in the corpus is 9,028 days, and its sentence reads "90,153 disputes decided between December 1999 and August 2024". Each of the 230 spans was classified by reading its own evidence quote: The raw span is not the answer, because the extraction's ''live-crawl'' mode also fires on tuples whose dates describe the *collected content* rather than the running apparatus. The longest "crawl" in the corpus is 9,028 days, and its sentence reads "90,153 disputes decided between December 1999 and August 2024". Each of the 230 spans was classified by reading its own evidence quote:
Line 69: Line 69:
 ^ Probe ^ Papers ^ Share ^ ^ Probe ^ Papers ^ Share ^
 | ''checkpoint'' or resuming a crawl / scan / measurement | 13 | 1.2% | | ''checkpoint'' or resuming a crawl / scan / measurement | 13 | 1.2% |
-| a task or job queue by name (Celery, RabbitMQ, Redis, Kafka, SQS, ...) | 34 | 3.0% |+| a task or job queue by name (Celery, RabbitMQ, Kafka, SQS, ...) | 18 | 1.6% |
 | a cluster orchestrator by name (Kubernetes, Swarm, Slurm, Nomad, Ansible, ...) | 21 | 1.9% | | a cluster orchestrator by name (Kubernetes, Swarm, Slurm, Nomad, Ansible, ...) | 21 | 1.9% |
 | a monitoring or logging stack by name (Prometheus, Grafana, Nagios, Kibana, ...) | 16 | 1.4% | | a monitoring or logging stack by name (Prometheus, Grafana, Nagios, Kibana, ...) | 16 | 1.4% |
Line 92: Line 92:
   - **Fingerprint the seed list.** A resumed run that seeds a //different// list is not a resumption, it is a second study. Hashing the list on first insert and refusing a changed one is four lines and it is the difference between a denominator you can name and one you cannot.   - **Fingerprint the seed list.** A resumed run that seeds a //different// list is not a resumption, it is a second study. Hashing the list on first insert and refusing a changed one is four lines and it is the difference between a denominator you can name and one you cannot.
  
-The script below is a working queue with those three properties, stdlib Python only (''sqlite3''), no network: the visit function is injected so the demo can kill the process deterministically and resume.+The script below is a working queue with those four properties, stdlib Python only (''sqlite3''), no network: the visit function is injected so the demo can kill the process deterministically and resume. Its self-test also covers the two things that make a queue safe under the concurrency this page recommends — that ''claim()'' refuses a row whose UPDATE matched nothing, and that the claim order is a deterministic shuffle rather than the seed order.
  
 <file python crawl_queue.py> <file python crawl_queue.py>
Line 113: Line 113:
     running -> pending                    lease expired (the worker died)     running -> pending                    lease expired (the worker died)
  
-Three things this gets right that a dict-of-URLs in memory does not:+Four things this gets right that a dict-of-URLs in memory does not:
  
 1.  **Claim-then-commit with a lease, and a named worker.** A worker marks a 1.  **Claim-then-commit with a lease, and a named worker.** A worker marks a
Line 165: Line 165:
     status       TEXT NOT NULL CHECK (status IN ('pending','running','done','failed')),     status       TEXT NOT NULL CHECK (status IN ('pending','running','done','failed')),
     attempts     INTEGER NOT NULL DEFAULT 0,     attempts     INTEGER NOT NULL DEFAULT 0,
 +    seq          TEXT NOT NULL,
     worker       TEXT,     worker       TEXT,
     lease_until  REAL,     lease_until  REAL,
Line 171: Line 172:
     updated_at   REAL NOT NULL     updated_at   REAL NOT NULL
 ); );
-CREATE INDEX IF NOT EXISTS work_status ON work(status);+CREATE INDEX IF NOT EXISTS work_status ON work(status, attempts, seq);
 CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL); CREATE TABLE IF NOT EXISTS meta (k TEXT PRIMARY KEY, v TEXT NOT NULL);
 """ """
Line 238: Line 239:
         now = self.clock()         now = self.clock()
         before = self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0]         before = self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0]
 +        # seq is a deterministic shuffle: sha256(digest + url). Claim order is
 +        # (attempts, seq), so the crawl walks the list in an order that is
 +        # reproducible from the seed but uncorrelated with rank, TLD or name.
 +        # It matters because a crawl that dies halfway through a rank-ordered
 +        # queue loses the tail of the ranking systematically; the same crash on
 +        # a shuffled queue loses a random subset, which is a far weaker claim
 +        # to have to defend in a methods section.
         self.db.executemany(         self.db.executemany(
-            "INSERT OR IGNORE INTO work(url,status,attempts,updated_at) VALUES(?,'pending',0,?)", +            "INSERT OR IGNORE INTO work(url,status,attempts,seq,updated_at) " 
-            [(u, now) for u in urls],+            "VALUES(?,'pending',0,?,?)", 
 +            [(u, hashlib.sha256((digest + u).encode("utf-8")).hexdigest(), now) for u in urls],
         )         )
         return self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0] - before         return self.db.execute("SELECT COUNT(*) FROM work").fetchone()[0] - before
Line 283: Line 292:
         before any retry is taken.         before any retry is taken.
         """         """
-        now = self.clock() +        while True: 
-        row = self.db.execute( +            now = self.clock() 
-            "SELECT url, attempts FROM work WHERE status='pending' " +            row = self.db.execute( 
-            "ORDER BY attempts ASC, url ASC LIMIT 1" +                "SELECT url, attempts FROM work WHERE status='pending' " 
-        ).fetchone() +                "ORDER BY attempts ASC, seq ASC LIMIT 1" 
-        if row is None: +            ).fetchone() 
-            return None +            if row is None: 
-        url, attempts = row +                return None 
-        self.db.execute( +            url, attempts = row 
-            "UPDATE work SET status='running', attempts=?, worker=?, lease_until=?, updated_at=? " +            cur = self.db.execute( 
-            "WHERE url=? AND status='pending'", +                "UPDATE work SET status='running', attempts=?, worker=?, lease_until=?, " 
-            (attempts + 1, worker, now + self.lease_seconds, now, url), +                "updated_at=? WHERE url=? AND status='pending'", 
-        ) +                (attempts + 1, worker, now + self.lease_seconds, now, url), 
-        return url, attempts + 1+            ) 
 +            # SELECT-then-UPDATE is not atomic. Another worker can win the row 
 +            # between the two statements, and then this UPDATE matches zero 
 +            # rows. Returning the URL anyway would hand two workers the same 
 +            # site and let the second one overwrite the first's outcome — the 
 +            # exact double-count the lease exists to prevent. Loop instead. 
 +            if cur.rowcount == 1: 
 +                return url, attempts + 1
  
     # -- outcomes ---------------------------------------------------------     # -- outcomes ---------------------------------------------------------
Line 409: Line 425:
  
     return visit     return visit
 +
 +
 +class _FakeCursor:
 +    """Just enough of a sqlite3 cursor for claim()'s SELECT."""
 +
 +    def __init__(self, row):
 +        self._row = row
 +
 +    def fetchone(self):
 +        return self._row
 +
 +
 +class _RaceOnce:
 +    """Connection wrapper that loses the SELECT/UPDATE race exactly once.
 +
 +    Used only by the demo. It passes everything through, except that the first
 +    time claim() runs its SELECT it hands the row to a "rival" worker before
 +    returning it — reproducing, deterministically, the interleaving that makes
 +    an unchecked SELECT-then-UPDATE hand one URL to two workers.
 +    """
 +
 +    def __init__(self, db):
 +        self.db = db
 +        self.armed = True
 +        self.stolen = None
 +
 +    def execute(self, sql, params=()):
 +        if self.armed and sql.startswith("SELECT url, attempts FROM work"):
 +            self.armed = False
 +            row = self.db.execute(sql, params).fetchone()
 +            if row is not None:
 +                self.stolen = row[0]
 +                self.db.execute(
 +                    "UPDATE work SET status='running', worker='rival' WHERE url=?",
 +                    (row[0],),
 +                )
 +            return _FakeCursor(row)
 +        return self.db.execute(sql, params)
 +
 +    def executemany(self, sql, params):
 +        return self.db.executemany(sql, params)
 +
 +    def executescript(self, sql):
 +        return self.db.executescript(sql)
  
  
Line 492: Line 552:
     print(f"first-attempt claims: {len(first_pass)}; first retry at position {retries[0]}")     print(f"first-attempt claims: {len(first_pass)}; first retry at position {retries[0]}")
  
-    print("\n=== F. the seed guard ===")+    print("\n=== F. does claim() refuse a row it did not win? ===") 
 +    # SELECT-then-UPDATE is not atomic, and the window between them is the only 
 +    # place two workers can be handed the same URL. Marking the row running 
 +    # BEFORE claim() runs does not test that window — the SELECT simply skips 
 +    # it. _RaceOnce steals the row in the window itself: it lets the SELECT 
 +    # return normally and then marks that row running behind claim()'s back, 
 +    # so claim()'s own UPDATE matches zero rows. Without the rowcount check, 
 +    # claim() returns the stolen URL and two workers crawl the same site. 
 +    race_db = os.path.join(tmp, "race.sqlite") 
 +    rq = CrawlQueue(race_db) 
 +    rq.seed(sites[:5]) 
 +    rq.db = _RaceOnce(rq.db) 
 +    got = rq.claim("w0") 
 +    stolen = rq.db.stolen 
 +    print(f"a rival took this row inside the window: {stolen}") 
 +    print(f"claim() returned:                       {got[0] if got else None}") 
 +    race_ok = got is not None and got[0] != stolen 
 + 
 +    print("\n=== G. is the claim order shuffled, and reproducible? ===") 
 +    order_a = [u for u, in oq.db.execute( 
 +        "SELECT url FROM work ORDER BY seq ASC").fetchall()] 
 +    rerun_db = os.path.join(tmp, "rerun.sqlite") 
 +    rq2 = CrawlQueue(rerun_db) 
 +    rq2.seed(small) 
 +    order_b = [u for u, in rq2.db.execute( 
 +        "SELECT url FROM work ORDER BY seq ASC").fetchall()] 
 +    print(f"seed order:    {[u[12:15] for u in small[:8]]} ...") 
 +    print(f"claim order:   {[u[12:15] for u in order_a[:8]]} ...") 
 +    print(f"same list, new database, same order: {order_a == order_b}") 
 + 
 +    print("\n=== H. the seed guard ===")
     try:     try:
         resumed.seed(sites + ["https://site200.example/"])         resumed.seed(sites + ["https://site200.example/"])
Line 500: Line 590:
         print(f"refused, as it should: {str(exc).splitlines()[0]}")         print(f"refused, as it should: {str(exc).splitlines()[0]}")
  
-    print("\n=== G. checks ===")+    print("\n=== I. checks ===")
     ok = True     ok = True
     checks = [     checks = [
Line 515: Line 605:
         ("every first attempt precedes every retry", len(first_pass) == len(small)         ("every first attempt precedes every retry", len(first_pass) == len(small)
             and (not retries or retries[0] == len(small))),             and (not retries or retries[0] == len(small))),
 +        ("claim() refuses a row it did not win", race_ok),
 +        ("claim order is not the seed order", order_a != small),
 +        ("claim order is reproducible from the seed", order_a == order_b),
         ("in-flight URL was retried, not lost", stranded == 1),         ("in-flight URL was retried, not lost", stranded == 1),
     ]     ]
Line 549: Line 642:
 === A. reference run, no crash (200 URLs) === === A. reference run, no crash (200 URLs) ===
 seeded 200 rows, seed sha256 49ddaed11cb3d529... seeded 200 rows, seed sha256 49ddaed11cb3d529...
-  [worker] total=200 done=60 failed=20 running=0 pending=120 | http_404=10, nxdomain=10+  [worker] total=200 done=61 failed=21 running=0 pending=118 | http_404=9, nxdomain=12
   [worker] total=200 done=120 failed=40 running=0 pending=40 | http_404=20, nxdomain=20   [worker] total=200 done=120 failed=40 running=0 pending=40 | http_404=20, nxdomain=20
 visits attempted: 280 visits attempted: 280
Line 555: Line 648:
  
 === B. same population, machine dies on the 87th visit === === B. same population, machine dies on the 87th visit ===
-killed while visiting https://site086.example/ +killed while visiting https://site013.example/ 
-state on disk: total=200 done=54 failed=16 running=1 pending=129 | http_404=8, nxdomain=8+state on disk: total=200 done=51 failed=19 running=1 pending=129 | http_404=7, nxdomain=12
 rows left in 'running' (the in-flight visit): 1 rows left in 'running' (the in-flight visit): 1
  
Line 563: Line 656:
 reclaim_expired() reclaimed 0 row(s) -- the 300s lease has not run out yet reclaim_expired() reclaimed 0 row(s) -- the 300s lease has not run out yet
 reclaim_worker('w1') reclaimed 1 row(s) -- w1 is the process that died reclaim_worker('w1') reclaimed 1 row(s) -- w1 is the process that died
-  [worker] total=200 done=114 failed=36 running=0 pending=50 | http_404=18, nxdomain=18+  [worker] total=200 done=113 failed=36 running=0 pending=51 | http_404=17, nxdomain=19
 visits attempted after resume: 194 visits attempted after resume: 194
 final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20 final: total=200 done=140 failed=60 running=0 pending=0 | http_404=20, nxdomain=20, timeout=20
Line 572: Line 665:
  
 === E. does a retry go to the back of the queue? === === E. does a retry go to the back of the queue? ===
-claim order (site, attempt): [('000', 1), ('001', 1), ('002', 1), ('003', 1), ('004', 1), ('005', 1), ('006', 1), ('007', 1), ('008', 1), ('009', 1), ('010', 1), ('011', 1)] ...+claim order (site, attempt): [('010', 1), ('002', 1), ('008', 1), ('003', 1), ('017', 1), ('001', 1), ('006', 1), ('012', 1), ('016', 1), ('015', 1), ('009', 1), ('005', 1)] ...
 first-attempt claims: 20; first retry at position 20 first-attempt claims: 20; first retry at position 20
  
-=== F. the seed guard ===+=== F. does claim() refuse a row it did not win? === 
 +a rival took this row inside the window: https://site004.example/ 
 +claim() returned:                       https://site000.example/ 
 + 
 +=== G. is the claim order shuffled, and reproducible? === 
 +seed order:    ['000', '001', '002', '003', '004', '005', '006', '007'] ... 
 +claim order:   ['010', '002', '008', '003', '017', '001', '006', '012'] ... 
 +same list, new database, same order: True 
 + 
 +=== H. the seed guard ===
 refused, as it should: seed list changed since this queue was created (stored 49ddaed11cb3..., given f1c74ecfee8e...). Use a new database, or you cannot name your denominator. refused, as it should: seed list changed since this queue was created (stored 49ddaed11cb3..., given f1c74ecfee8e...). Use a new database, or you cannot name your denominator.
  
-=== G. checks ===+=== I. checks ===
   ok   same successful set   ok   same successful set
   ok   same failure set   ok   same failure set
Line 590: Line 692:
   ok   an expired lease IS reclaimed   ok   an expired lease IS reclaimed
   ok   every first attempt precedes every retry   ok   every first attempt precedes every retry
 +  ok   claim() refuses a row it did not win
 +  ok   claim order is not the seed order
 +  ok   claim order is reproducible from the seed
   ok   in-flight URL was retried, not lost   ok   in-flight URL was retried, not lost
   note  clean run: 280 visits. crash+resume: 86 before the crash, 1 interrupted, 194 after = 281 calls for the same 200 terminal rows. The excess is the one interrupted visit.   note  clean run: 280 visits. crash+resume: 86 before the crash, 1 interrupted, 194 after = 281 calls for the same 200 terminal rows. The excess is the one interrupted visit.
Line 600: Line 705:
 ===== Retries: what to retry, and what it buys ===== ===== Retries: what to retry, and what it buys =====
  
-Retrying once is the single highest-return operational decision on this page, and one paper measured it. Hausladen et al. reload a site that failed on the first attempt, and report: "nearly 40% of sites that failed to load on the first attempt were loaded and successfully analyzed" on the second {[hausladen2025_websites]}. Their first-attempt failure conditions are worth copying verbatim, because they are the three that matter: "a site failed to load in 35 seconds, had an insecure certificate, or led to an error page".+Retrying once is the single highest-return operational decision on this page, and one paper measured the retry itself. Hausladen et al. reload a site that failed on the first attempt, and report: "nearly 40% of sites that failed to load on the first attempt were loaded and successfully analyzed" on the second {[hausladen2025_websites]}. Their first-attempt failure conditions are worth copying verbatim, because they are the three that matter: "a site failed to load in 35 seconds, had an insecure certificate, or led to an error page".
  
-What the retry is worth depends on what fraction of your list fails at all. Papers that report it:+What the retry is worth depends on what fraction of your list fails at all. Seven papers whose attrition figures were surfaced by the retry and attrition probes and then read and verified individually — **not** an exhaustive list; the attrition probe suggests of the order of 38 crawled papers report something of this kind:
  
 ^ Paper ^ Population ^ Reached ^ What the rest were ^ ^ Paper ^ Population ^ Reached ^ What the rest were ^
Line 613: Line 718:
 | Demir et al., PoPETs 2024 {[demir2024_bannertools]} | a Tranco sample | 29,660 (99%) | not broken out | | Demir et al., PoPETs 2024 {[demir2024_bannertools]} | a Tranco sample | 29,660 (99%) | not broken out |
  
-Two patterns to take from that table. A top-1M crawl reaches roughly 88–94% of its list and a smaller, more popular list reaches 91–99%; if your number is far below that, the problem is your deployment, not the web. And the failure mix moves over time — 403 Forbidden dominating a 2024 crawl {[annamalai2024_fpfed]} is bot detection, not a network problem, and no amount of retrying fixes it.+Two patterns to take from that table. A top-1M crawl reaches roughly 85–94% of its list — Musch and Johns's ~846K is 84.6%, the lowest here — and a smaller or more popular list reaches roughly 90–99%. Those are the published range, not a target: they come from seven hand-read papers, not from a census. And the failure mix moves over time — 403 Forbidden dominating a 2024 crawl {[annamalai2024_fpfed]} is bot detection, not a network problem, and no amount of retrying fixes it.
  
 Retries also come in shapes other than "do it again". Ikram et al. retry with a **changed parameter**: "we checked if the rendering of any webpage stalled. If so, we reran the crawler on the webpage by increasing the waiting time until the webpage is rendered" {[ikram2017_seamless]}. Khodayari et al. simply bound it — "we made up to three repeated attempts for each failed crawling" {[khodayari2024_great]}. Senol et al. record what a retry-shaped fix was worth: a minor change "increased the successfully visited websites from 94,427 (EU pilot crawl) to 99,380 (EU final crawl)" {[senol2022_leaky]}. Retries also come in shapes other than "do it again". Ikram et al. retry with a **changed parameter**: "we checked if the rendering of any webpage stalled. If so, we reran the crawler on the webpage by increasing the waiting time until the webpage is rendered" {[ikram2017_seamless]}. Khodayari et al. simply bound it — "we made up to three repeated attempts for each failed crawling" {[khodayari2024_great]}. Senol et al. record what a retry-shaped fix was worth: a minor change "increased the successfully visited websites from 94,427 (EU pilot crawl) to 99,380 (EU final crawl)" {[senol2022_leaky]}.
Line 624: Line 729:
  
 **29 of 1,120 crawled papers (2.6%)** state how many browsers or workers they ran at once. That is nearly all the published guidance there is on sizing a crawl machine, and it is a probe result with a hand-read verdict on every hit: a further 4 say they were parallel without a number, 5 parallelise an analysis stage rather than the crawl, and 6 are about something else entirely. **29 of 1,120 crawled papers (2.6%)** state how many browsers or workers they ran at once. That is nearly all the published guidance there is on sizing a crawl machine, and it is a probe result with a hand-read verdict on every hit: a further 4 say they were parallel without a number, 5 parallelise an analysis stage rather than the crawl, and 6 are about something else entirely.
 +
 +Six of the 29, chosen to span the range of scale in the class — one desktop, one cloud VM, one departmental cluster, and the three papers that also give a throughput or a wall clock:
  
 ^ Paper ^ Hardware ^ Concurrency ^ ^ Paper ^ Hardware ^ Concurrency ^
Line 639: Line 746:
 ===== Monitoring ===== ===== Monitoring =====
  
-**16 of 1,120 crawled papers (1.4%)** name a monitoring stack, so there is no literature here to summarise. What there is, is a list of the things that went wrong in the ten papers that admitted something went wrong (next section), and every one of them is visible in a counter you could have been watching:+**16 of 1,120 crawled papers (1.4%)** name a monitoring or logging product. That probe only sees brand names — a paper that describes watching its own crawl without naming a product is invisible to it — but 16 named-tool papers is not a literature to systematise either. What there is, is a list of the things that went wrong in the ten papers that admitted something went wrong (next section), and every one of them is visible in a counter you could have been watching:
  
   * **Terminal rows per hour.** Not "pages fetched" — rows that reached ''done'' or ''failed''. A crawl whose throughput halves is usually a crawl where retries have started dominating, which is a signal about the target, not about your disk.   * **Terminal rows per hour.** Not "pages fetched" — rows that reached ''done'' or ''failed''. A crawl whose throughput halves is usually a crawl where retries have started dominating, which is a signal about the target, not about your disk.
Line 651: Line 758:
 ===== Cost ===== ===== Cost =====
  
-**5 of 1,120 crawled papers (0.4%) state an infrastructure bill.** A further 3 state what they paid to acquire data — services, participants, annotation — so **8 (0.7%)** state any cost they themselves paid. Eleven other papers put a currency figure next to a cost word in a measurement context, and in every one of those the price belongs to what was being measured, not to the measuring. This field reports the adversary's economics in detail and its own hardly at all.+A probe for a currency figure beside a cost word in a measurement context returns **19 of 1,120 crawled papers (1.7%)**. Reading them: **5 state an infrastructure bill**, 3 state what they paid to acquire data — services, participants, annotation — and in the other 11 the price belongs to what was being measured, not to the measuring. So **at least 8 (0.7%)** state a cost they themselves paid. Unlike the other probes on this page these counts are **lower** bounds, and demonstrably so: the first version of this regex found 2 of the 5 infrastructure papers and missed Englehardt and Narayanan's own line about the machine that ran their 1-million-site crawl. This field reports the adversary's economics in detail and its own hardly at all.
  
 All eight, in full. The **Kind** column matters: only the first five are a machine bill, and the row that classifies a paper is the one whose sentence the probe matched, not the paper's subject. All eight, in full. The **Kind** column matters: only the first five are a machine bill, and the row that classifies a paper is the one whose sentence the probe matched, not the paper's subject.
Line 665: Line 772:
 | Song et al., PoPETs 2026 {[song2026_wfpllm]} | data | "acquiring 1GB of authentic human browsing data costs approximately $35, primarily due to participant compensation and lab infrastructure" | | Song et al., PoPETs 2026 {[song2026_wfpllm]} | data | "acquiring 1GB of authentic human browsing data costs approximately $35, primarily due to participant compensation and lab infrastructure" |
  
-Since nobody will tell you, work it out. The arithmetic has four terms and only the first is large:+Eight figures spread over sixteen years and four pricing models will not size your grant. Work it out instead. The arithmetic has four terms and only the first is large:
  
   - **Machine-hours** = (sites × attempts) ÷ (workers per machine × sites per worker-hour). At Bouhoula et al.'s 300 sites/hour/machine {[bouhoula2024_automated]}, a one-million-site pass is ~3,300 machine-hours. On-demand Linux in ''eu-central-1'' on 2026-08-28 was **USD 0.4074/hour** for a ''c7i.2xlarge'' (8 vCPU) and **USD 0.8148/hour** for a ''c7i.4xlarge'' (16 vCPU), read out of the AWS bulk price list.((AWS EC2 bulk price list, offer version ''20260828175757'', ''eu-central-1'', Linux, shared tenancy, on-demand. Fetched 2026-08-28 by ''scripts/ec2_price.py'' — the region offer file is about 431 MB, so the script streams it. Re-fetch before quoting: these move.)) So ~3,300 machine-hours on ''c7i.4xlarge'' instances is roughly **USD 2,700** for one pass. Treat that as a floor: a ''c7i.4xlarge'' is 16 vCPU, which on a hyperthreaded Intel instance is 8 physical cores against the 16 physical cores of Bouhoula et al.'s Ryzen, so the same 300 sites per hour will want a bigger instance or a longer wall clock. Either way it is the same order as Thomas et al.'s $1,587/month fifteen years ago, which is the useful observation: the hardware got faster and the crawl got heavier.   - **Machine-hours** = (sites × attempts) ÷ (workers per machine × sites per worker-hour). At Bouhoula et al.'s 300 sites/hour/machine {[bouhoula2024_automated]}, a one-million-site pass is ~3,300 machine-hours. On-demand Linux in ''eu-central-1'' on 2026-08-28 was **USD 0.4074/hour** for a ''c7i.2xlarge'' (8 vCPU) and **USD 0.8148/hour** for a ''c7i.4xlarge'' (16 vCPU), read out of the AWS bulk price list.((AWS EC2 bulk price list, offer version ''20260828175757'', ''eu-central-1'', Linux, shared tenancy, on-demand. Fetched 2026-08-28 by ''scripts/ec2_price.py'' — the region offer file is about 431 MB, so the script streams it. Re-fetch before quoting: these move.)) So ~3,300 machine-hours on ''c7i.4xlarge'' instances is roughly **USD 2,700** for one pass. Treat that as a floor: a ''c7i.4xlarge'' is 16 vCPU, which on a hyperthreaded Intel instance is 8 physical cores against the 16 physical cores of Bouhoula et al.'s Ryzen, so the same 300 sites per hour will want a bigger instance or a longer wall clock. Either way it is the same order as Thomas et al.'s $1,587/month fifteen years ago, which is the useful observation: the hardware got faster and the crawl got heavier.
Line 695: Line 802:
   - **Record the gap in the same units as your results.** Not "we had some downtime". Leontiadis et al. report complete measurements for 1,004 of the 1,254 days in their measurement period {[leontiadis2014_nearly]}, and that is the model, because a reader can divide it. If your unit is sites, say how many sites the gap cost. If your unit is days of a time series, say which days.   - **Record the gap in the same units as your results.** Not "we had some downtime". Leontiadis et al. report complete measurements for 1,004 of the 1,254 days in their measurement period {[leontiadis2014_nearly]}, and that is the model, because a reader can divide it. If your unit is sites, say how many sites the gap cost. If your unit is days of a time series, say which days.
   - **Decide what the gap does to your inference, and say so.** Wang et al. quantified it as 4.5% of broadcasts missing {[wang2016_anatomy]}; Lee et al. pruned the 17 affected days out of the dataset entirely rather than analysing a series with holes in it {[lee2021_practice]}. Both are defensible. Neither is "we lost some data".   - **Decide what the gap does to your inference, and say so.** Wang et al. quantified it as 4.5% of broadcasts missing {[wang2016_anatomy]}; Lee et al. pruned the 17 affected days out of the dataset entirely rather than analysing a series with holes in it {[lee2021_practice]}. Both are defensible. Neither is "we lost some data".
-  - **Check that the gap is not correlated with what you measure.** A crawl that dies overnight loses the sites that sort late in the list; a crawl that dies on Fridays loses whatever is different about weekends. If your queue orders by rank, a mid-run failure is a systematic loss, not a random one — shuffle the queue at seed time, or say the loss was ordered.+  - **Check that the gap is not correlated with what you measure.** A crawl that dies overnight loses the sites that sort late in the list; a crawl that dies on Fridays loses whatever is different about weekends. If your queue orders by rank, a mid-run failure is a systematic loss, not a random one — shuffle the queue at seed time, or say the loss was ordered. The script above shuffles deterministically, on a hash of the seed digest and the URL, so the order is reproducible from the seed list and uncorrelated with rank.
   - **Check whether the environment changed while it was down.** A restart after a package update is a new instrument. [[Design:Longitudinal]] covers pinning; the deployment consequence is that "we restarted" and "we upgraded and restarted" are different sentences in a methods section.   - **Check whether the environment changed while it was down.** A restart after a package update is a new instrument. [[Design:Longitudinal]] covers pinning; the deployment consequence is that "we restarted" and "we upgraded and restarted" are different sentences in a methods section.
   - **Then decide whether to extend.** Extending the window to recover lost coverage stretches the measurement period, which is a temporal confound, not a free fix. Say which you did.   - **Then decide whether to extend.** Extending the window to recover lost coverage stretches the measurement period, which is a temporal confound, not a free fix. Say which you did.
Line 704: Line 811:
  
   - **The wall-clock window**, with both endpoints at day precision. Only 230 of 1,120 crawled papers manage this, and it is the cheapest sentence on the list.   - **The wall-clock window**, with both endpoints at day precision. Only 230 of 1,120 crawled papers manage this, and it is the cheapest sentence on the list.
-  - **Whether that window is machine time or content coverage.** A quarter of the day-precision spans in this corpus are ambiguous on exactly this point, and readers cannot tell them apart.+  - **Whether that window is machine time or content coverage.** A fifth of the day-precision spans in this corpus (48 of 230) turned out to measure something other than one continuous run, and only reading each paper's own sentence separated them.
   - **The population, and how much of it you reached** — the numerator, the denominator, and the failure classes underneath. "We crawled the Tranco top 1M" is not a result. Murley et al. reached 88.1% of the top million after one retry, with 7.7% failing to resolve and 4.2% whose servers did not respond — that is {[murley2021_websocket]}.   - **The population, and how much of it you reached** — the numerator, the denominator, and the failure classes underneath. "We crawled the Tranco top 1M" is not a result. Murley et al. reached 88.1% of the top million after one retry, with 7.7% failing to resolve and 4.2% whose servers did not respond — that is {[murley2021_websocket]}.
   - **The retry policy**: how many attempts, which errors were terminal, and whether the retry was immediate or at the end of the queue. If a retry pass changed your numbers, say by how much {[hausladen2025_websites]}.   - **The retry policy**: how many attempts, which errors were terminal, and whether the retry was immediate or at the end of the queue. If a retry pass changed your numbers, say by how much {[hausladen2025_websites]}.
-  - **Concurrency and hardware**, in the same sentence: N browsers on M cores with K GB. 29 papers do this and they are the only ones another group can size a machine from. +  - **Concurrency and hardware**, in the same sentence: N browsers on M cores with K GB. 29 papers state a concurrency figure; fewer pair it with the hardware, and those are the only ones another group can size a machine from. 
-  - **The infrastructure**: cloud provider and region, university network, or testbed. 61.4% of crawled papers say nothing here, which also makes their vantage point unverifiable ([[Design:Crawling location]]).+  - **The infrastructure**: cloud provider and region, university network, or testbed. 675 of 1,120 crawled papers (60.3%) name none, which also makes their vantage point unverifiable ([[Design:Crawling location]]). (60.3% is the complement of the 39.7% that do; the ''not-stated'' row is 61.4% because 44 papers carry both a stated and a not-stated tuple. Subtracting a sentinel row from the population is the wrong way to get this number.)
   - **Any interruption, in the units of your result**, and what you did with the gap.   - **Any interruption, in the units of your result**, and what you did with the gap.
-  - **The cost**, if you can. Five papers in sixteen years of seven venues have done it, and every student planning a crawl has to guess because of that.+  - **The cost**, if you can. Across seventeen publication years of seven venues, five papers state an infrastructure bill and three more state what their data acquisition cost, and every student planning a crawl has to guess because of that.
  
 ===== Related pages ===== ===== Related pages =====
Line 729: Line 836:
 Corpus figures come from ''scripts/report_deployment.mjs'' against ''data/extract/run1'' (5,859 papers; CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P; 2010–2026). The denominator throughout is the **1,120 papers that ran a crawl**, never 5,859. Corpus figures come from ''scripts/report_deployment.mjs'' against ''data/extract/run1'' (5,859 papers; CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P; 2010–2026). The denominator throughout is the **1,120 papers that ran a crawl**, never 5,859.
  
-Three limits worth stating plainly. **The duration figures rest on a hand classification**, not on the extraction: ''temporal[].mode == "live-crawl"'' does not distinguish machine time from content coverage, and 48 of 230 spans had to be excluded by reading. **Every operating figure is a full-text probe**, so it counts papers that used a word, not papers that did the thing; the retry row moves by a factor of two depending on how wide the regex is, and both widths are printed above. **The attrition probe is weak**: it returns 75 papers, and a hand-read sample of 12 was right half the time, so no reporting rate is published from it. Per-paper figures were checked against ''paper.cols.txt'' with whitespace collapsed — 41 of 41 found verbatim. The AWS prices are a single fetch on 2026-08-28 and will be wrong soon. Year shares are not used on this page; where 2025–2026 papers appear they are individual citations, not a trend ([[Literature:Corpus]]).+Three limits worth stating plainly. **The duration figures rest on a hand classification**, not on the extraction: ''temporal[].mode == "live-crawl"'' does not distinguish machine time from content coverage, and 48 of 230 spans had to be excluded by reading. **Every operating figure is a full-text probe**, so it counts papers that used a word, not papers that did the thing; the retry row moves by a factor of two depending on how wide the regex is, and both widths are printed above. **The attrition probe is weak**: it returns 75 papers, and a hand-read sample of 12 was right half the time, so no reporting rate is published from it. Per-paper figures were checked against the papers with whitespace collapsed — 52 of 52 found verbatim, 51 in ''paper.cols.txt'' and one only in the PDF, because the ''.cols'' rendering splits its thousands separators. The AWS prices are a single fetch on 2026-08-28 and will be wrong soon. Year shares are not used on this page; where 2025–2026 papers appear they are individual citations, not a trend ([[Literature:Corpus]]).
  
 <bibtex bibliography></bibtex> <bibtex bibliography></bibtex>
 ~~DISCUSSION~~ ~~DISCUSSION~~
  
programming/deployment.txt · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki