| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:programming:stateful_stateless [2026/09/17 07:24] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | provenance:programming:stateful_stateless [2026/09/29 11:39] (current) – Log Ramakrishnan 2026 additions, suggestion-list verdicts, review. Authored by Claude karel.kubicek.claude |
|---|
| === Statefulness against the other crawl-configuration fields it competes with === | === Statefulness against the other crawl-configuration fields it competes with === |
| |
| Field Papers stating it Share of 1120 crawling papers | Field Papers stating it Share of 1120 crawling papers |
| -------------------------- ----------------- ----------------------------- | ------------------------------------------------------------------- ----------------- ----------------------------- |
| Interaction depth 841 75.1% | Interaction depth 841 75.1% |
| Authentication 779 69.6% | Authentication 779 69.6% |
| At least one browser named 529 47.2% | At least one browser named 529 47.2% |
| Consent action 349 31.2% | Consent action field populated (schema; not an audited paper claim) 349 31.2% |
| **Stateful or stateless** 219 19.6% | **Stateful or stateless** 219 19.6% |
| Headless or headful 140 12.5% | Headless or headful 140 12.5% |
| |
| |
| === Every crawl-configuration field, reporting rate per bucket (of the crawling papers in each bucket) === | === Every crawl-configuration field, reporting rate per bucket (of the crawling papers in each bucket) === |
| |
| Field 2010–2013 2014–2017 2018–2021 2022–2024 2025–2026* max-min last - first | Field 2010–2013 2014–2017 2018–2021 2022–2024 2025–2026* max-min last - first |
| -------------------- -------------- --------------- --------------- --------------- --------------- ------- ------------ | ---------------------------------------------------------------- -------------- --------------- --------------- --------------- --------------- ------- ------------ |
| interactionDepth 80/102 = 78.4% 126/167 = 75.4% 237/308 = 76.9% 255/345 = 73.9% 143/198 = 72.2% 6.2 pp -6.2 pp | interactionDepth 80/102 = 78.4% 126/167 = 75.4% 237/308 = 76.9% 255/345 = 73.9% 143/198 = 72.2% 6.2 pp -6.2 pp |
| authentication 61/102 = 59.8% 117/167 = 70.1% 213/308 = 69.2% 251/345 = 72.8% 137/198 = 69.2% 12.9 pp +9.4 pp | authentication 61/102 = 59.8% 117/167 = 70.1% 213/308 = 69.2% 251/345 = 72.8% 137/198 = 69.2% 12.9 pp +9.4 pp |
| browsers (>=1 named) 33/102 = 32.4% 81/167 = 48.5% 153/308 = 49.7% 162/345 = 47.0% 100/198 = 50.5% 18.2 pp +18.2 pp | browsers (>=1 named) 33/102 = 32.4% 81/167 = 48.5% 153/308 = 49.7% 162/345 = 47.0% 100/198 = 50.5% 18.2 pp +18.2 pp |
| consentAction 25/102 = 24.5% 50/167 = 29.9% 99/308 = 32.1% 116/345 = 33.6% 59/198 = 29.8% 9.1 pp +5.3 pp | consentAction (schema field populated; not a paper-claim series) 25/102 = 24.5% 50/167 = 29.9% 99/308 = 32.1% 116/345 = 33.6% 59/198 = 29.8% 9.1 pp +5.3 pp |
| **statefulness** 16/102 = 15.7% 35/167 = 21.0% 61/308 = 19.8% 71/345 = 20.6% 36/198 = 18.2% 5.3 pp +2.5 pp | **statefulness** 16/102 = 15.7% 35/167 = 21.0% 61/308 = 19.8% 71/345 = 20.6% 36/198 = 18.2% 5.3 pp +2.5 pp |
| headless 1/102 = 1.0% 26/167 = 15.6% 41/308 = 13.3% 50/345 = 14.5% 22/198 = 11.1% 14.6 pp +10.1 pp | headless 1/102 = 1.0% 26/167 = 15.6% 41/308 = 13.3% 50/345 = 14.5% 22/198 = 11.1% 14.6 pp +10.1 pp |
| |
| The last two columns are the test of the page's claim: which field moved, and by how much. | The last two columns are the test of the page's claim: which field moved, and by how much. |
| | |
| | Audited consent-action papers by bucket (paper claims, not raw field population): |
| | Bucket Crawling papers Audited papers stating a consent action Share Of those, interacted |
| | ---------- --------------- --------------------------------------- ----- -------------------- |
| | 2010–2013 102 0 0.0% 0 |
| | 2014–2017 167 0 0.0% 0 |
| | 2018–2021 308 11 3.6% 3 |
| | 2022–2024 345 25 7.2% 16 |
| | 2025–2026* 198 19 9.6% 13 |
| | The schema row above is field population only; this audited row is the paper-claim series. |
| |
| |
| === Designs whose result is uninterpretable without the statefulness: do they state it? === | === Designs whose result is uninterpretable without the statefulness: do they state it? === |
| |
| Subset of crawling papers N State statefulness Share stating stateless stateful both | Subset of crawling papers N State statefulness Share stating stateless stateful both |
| ------------------------------------------------------------------------------ ---- ------------------ ------------- --------- -------- ---- | ------------------------------------------------------------------------- ---- ------------------ ------------- --------- -------- ---- |
| Repeat-visit designs (repeatVisits >= 2) 199 85 42.7% 49 21 15 | Repeat-visit designs (repeatVisits >= 2) 199 85 42.7% 49 21 15 |
| Crawls that acted on a consent notice 36 22 61.1% 11 4 7 | Crawls that an audit found acted on a consent notice 32 20 62.5% 10 4 6 |
| Crawls that logged in (authentication stated, not "none") 90 54 60.0% 7 39 8 | Crawls with an audited consent action (interaction or explicit no-action) 55 32 58.2% 18 8 6 |
| Deep crawls (interactionDepth = deep-crawl) 157 40 25.5% 8 26 6 | Crawls that logged in (authentication stated, not "none") 90 54 60.0% 7 39 8 |
| Crawls beyond the landing page (subpages or deep) 303 83 27.4% 30 40 13 | Deep crawls (interactionDepth = deep-crawl) 157 40 25.5% 8 26 6 |
| Crawls whose consent interaction was //verified// by the privacy:consent audit 28 16 57.1% 8 3 5 | Crawls beyond the landing page (subpages or deep) 303 83 27.4% 30 40 13 |
| //all crawling papers, for comparison// 1120 219 19.6% 113 77 29 | //all crawling papers, for comparison// 1120 219 19.6% 113 77 29 |
| |
| |
| - **The page originally compared statefulness reporting against artifact release "24.3% to 72.7%" and ethics review "8.3% to 45.2%", quoted from ''OVERVIEW.md''.** Those are computed on ''OVERVIEW.md''`s bucket boundaries (2010–2011 / 2012–2015 / 2016–2019 / 2020–2023 / 2024–2026), not on ''lib.mjs'' ''YEAR_BUCKETS'' (2010–2013 / 2014–2017 / 2018–2021 / 2022–2024 / 2025–2026) which the page's own trend table uses. Quoting across incompatible buckets in a sentence beginning "over the same sixteen years" is exactly the kind of borrowed figure the corpus refresh warns about. Both indicators are now recomputed in ''report_stateful_stateless.mjs'' on the page's own buckets — **23.7% → 76.5%** and **10.2% → 45.8%** — and published as a table beside the statefulness column. (Writing that code also caught a wrong field name: the artifacts schema uses ''links[].belongsToAuthors'', not ''isAuthorsOwn'', which had silently produced 0.0% in every bucket.) | - **The page originally compared statefulness reporting against artifact release "24.3% to 72.7%" and ethics review "8.3% to 45.2%", quoted from ''OVERVIEW.md''.** Those are computed on ''OVERVIEW.md''`s bucket boundaries (2010–2011 / 2012–2015 / 2016–2019 / 2020–2023 / 2024–2026), not on ''lib.mjs'' ''YEAR_BUCKETS'' (2010–2013 / 2014–2017 / 2018–2021 / 2022–2024 / 2025–2026) which the page's own trend table uses. Quoting across incompatible buckets in a sentence beginning "over the same sixteen years" is exactly the kind of borrowed figure the corpus refresh warns about. Both indicators are now recomputed in ''report_stateful_stateless.mjs'' on the page's own buckets — **23.7% → 76.5%** and **10.2% → 45.8%** — and published as a table beside the statefulness column. (Writing that code also caught a wrong field name: the artifacts schema uses ''links[].belongsToAuthors'', not ''isAuthorsOwn'', which had silently produced 0.0% in every bucket.) |
| - **"the only reporting rate on this site that has not improved since 2010"** was an unverifiable claim about every page on the wiki. It was narrowed to the crawl-configuration fields — but **that narrowing was still not computed**, and reviewer 4 caught it: the script had per-bucket trends for statefulness only. It is computed now; see reviewer 4, finding 2. | - **"the only reporting rate on this site that has not improved since 2010"** was an unverifiable claim about every page on the wiki. It was narrowed to the crawl-configuration fields — but **that narrowing was still not computed**, and reviewer 4 caught it: the script had per-bucket trends for statefulness only. It is computed now; see reviewer 4, finding 2. |
| - **The consent-interaction subset (36 papers, 61.1% reporting) is the same 36 papers [[privacy:consent]] audited**, where 7 (19.4%) are extraction false positives. Rather than repeat an unaudited figure, ''report_stateful_stateless.mjs'' now imports that page's ''VERDICTS'' map and prints the rate on the 28 verified papers too: **16/28 = 57.1%**, i.e. the finding survives. The page carries both rows and the caveat. | - **The consent-interaction subset is now taken from the complete 2026-09-05 audit:** **32** papers were found to have acted on a consent notice, and **55** papers were audited as stating a consent action. The audit found **279/313 = 89.1%** unsupported ''no-interaction'' labels and **7/36 = 19.4%** false positives among labels claiming interaction. ''report_stateful_stateless.mjs'' imports the exported audit sets and prints both audited rows. |
| |
| ===== Where this page's figures cross-check against neighbours ===== | ===== Where this page's figures cross-check against neighbours ===== |
| |
| ^ Figure ^ Here ^ Neighbour ^ Reconciled? ^ | ^ Figure ^ Here ^ Neighbour ^ Reconciled? ^ |
| | statefulness stated, of crawling papers | 219 / 1,120 = 19.6% | [[programming:crawler]]: "Stateful or stateless \| 219 \| 19.6%" | identical | | | statefulness stated, of crawling papers | 219 / 1,120 = 19.6% | [[programming:crawler]]: same 219 / 1,120 = 19.6% row | identical | |
| | OpenWPM papers stating it | 32/58 = 55.2% (category-restricted fold) | [[programming:crawler:openwpm]]: 33/60 = 55.0%, and 186/1,061 for everyone else | differ by one paper; both definitions printed by this page's script and named on the page | | | OpenWPM papers stating it | 32/58 = 55.2% (category-restricted fold) | [[programming:crawler:openwpm]]: 33/60 = 55.0%, and 186/1,061 for everyone else | differ by one paper; both definitions printed by this page's script and named on the page | |
| | Consent-interaction subset | 36 papers, 22 state statefulness | [[privacy:consent]]: same 36 papers, 7 false positives | this page prints both the raw and the audited row | | | Audited interaction subset | 32 papers, 20 state statefulness | [[privacy:consent]]: 7/36 interaction labels false positive | audited row printed by this page's report | |
| | Papers with a consent action stated | 349 / 1,120 = 31.2% | [[privacy:requests]] reports 52.3% on its own 197-paper subset | different populations; not compared on the page | | | Audited consent-action papers | 55 / 1,120 = 4.9% | [[privacy:requests]] reports 52.3% on its own 197-paper subset | different populations; not compared on the page | |
| |
| ===== Artefacts ===== | ===== Artefacts ===== |
| |
| ^ Path ^ What it does ^ | ^ Path ^ What it does ^ |
| | ''scripts/report_stateful_stateless.mjs'' | the page's tables, each with its denominator; ''%%--wiki%%'' emits DokuWiki markup. Imports ''VERDICTS'' from ''consent_action_audit.mjs'', whose printing is guarded behind an is-main check so the import has no side effect | | | ''scripts/report_stateful_stateless.mjs'' | the page's tables, each with its denominator; ''%%--wiki%%'' emits DokuWiki markup. Imports the audited paper-key sets from ''consent_action_noninteraction_audit.mjs'', whose printing is guarded behind an is-main check so the import has no side effect | |
| | ''scripts/statefulness_probe.mjs'' | extracts state-management sentences from the 219 stating papers' own text | | | ''scripts/statefulness_probe.mjs'' | extracts state-management sentences from the 219 stating papers' own text | |
| | ''scripts/statefulness_audit.mjs'' | the 29-paper hand adjudication and its effect on the page's figures | | | ''scripts/statefulness_audit.mjs'' | the 29-paper hand adjudication and its effect on the page's figures | |
| |
| Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== ConsentAction audit propagation, 2026-09-17 ===== |
| | |
| | **What was wrong.** The page's ''consentAction'' row and its 24.5% / 29.9% / 32.1% / 33.6% / 29.8% series were presented without saying that they measure raw field population, not paper claims; the old +5.3 pp reading was therefore not a valid consent-action trend. The 2026-09-05 full-text audit of all 313 ''no-interaction'' labels found 279/313 unsupported (89.1%). |
| | |
| | **Command and real output.** ''node scripts/report_stateful_stateless.mjs'' → raw schema row 349/1,120 = 31.2%; audited paper-claim buckets **0/102 = 0.0%, 0/167 = 0.0%, 11/308 = 3.6%, 25/345 = 7.2%, 19/198 = 9.6%**, with interacted counts 0, 0, 3, 16, 13. ''python3 scripts/consent_ni_significance.py'' → old raw-series adjacent p-values **0.4008, 0.679, 0.7387, 0.3911**, all not significant; audited-series p-values **1, 0.009883, 0.05751, 0.3324**, so only the 2014–2017 → 2018–2021 audited movement is significant and 2025–2026 remains provisional. |
| | |
| | **What changed.** The raw row is explicitly labelled schema field population, a separate audited paper-claim table is published, and the consent subset now uses **32** audited-interacting papers (20 state statefulness, 62.5%) and **55** audited-action papers (32 state statefulness, 58.2%). The page also records the 89.1% unsupported rate alongside the older 19.4% interacting-side false-positive rate. |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_stateful_stateless.mjs'' and ''consent_ni_significance.py'', checked the edited table and the figures immediately before and after it, and found no mismatch. No citation pass was needed: no citekey or quoted claim changed. |
| | |
| | **Rejected finding.** The raw schema series was not accepted as evidence that papers stated consent actions; the eight ''no interaction at all'' papers without consent vocabulary were not folded into the 55-action series. |
| | |
| | ===== Ramakrishnan et al. (IMC 2026) added, 2026-09-29 ===== |
| | |
| | **What.** Karel pointed to Ramakrishnan, Xie, Anand and Li, //Crawling in the Deep: Evaluating Web Crawling Configurations for Web Privacy Measurements//, IMC 2026 {[ramakrishnan2026_crawling]}, with a list of suggested guidance. The paper was read in full and the pages were extended from the paper's own text, not from the suggestion list. Edits are in one refuse-on-mismatch script, ''scripts/apply_ramakrishnan2026_edits.py'', run against a fresh ''export_raw'' of each page. |
| | |
| | **Source and its status.** PDF at ''karthikr.in/data/papers/privacy_diff_imc.pdf'' (15 pages), text extracted with ''pypdf''. **Not in the corpus**: IMC 2026 (12–16 October 2026) had not been held when the corpus was built, and there is no ''data/fulltext/2026/IMC'' directory. So none of its figures enter any corpus count, and the pages say so. Bibliography entry built from Crossref (''10.1145/3777912.3839794'': title, four authors, pages 374–388, //Proceedings of the 2026 ACM Internet Measurement Conference//), key ''ramakrishnan2026_crawling''; no DOI or key collision in ''literature:bibliography'' before adding. |
| | |
| | **The suggestion list, checked against the paper.** Kept here because it is what stops the next edit re-adding the rejected parts. |
| | |
| | ^ Suggested claim ^ Verdict ^ What the paper says ^ |
| | | cited as "IMC 2024" | **wrong** | IMC 2026, Karlsruhe; CrUX list of October 2024 is probably the source of the year | |
| | | stop after 8–13 consecutive pages with nothing new | **kept** | //k// = 11 (domains), 12–13 (APIs), 8 (cookies); //k// = 13 covers all six metrics (§4.3.2) | |
| | | … "yields 90%+ coverage" | **wrong unit** | the criterion is a 0% //median// per-site error relative to a 50-page crawl, not a coverage share | |
| | | … "reduces pages crawled by 30–50%" | **overstated** | fixed-depth crawls need 4.2–52.8% //more// pages; read the other way that is 4.0–34.6% //fewer// pages with adaptive stopping, and only about 5% at a looser error target (footnote 1) | |
| | | small or static sites "saturate after 2–3 pages" | **not in the paper** | dropped | |
| | | over 90% of tracking requests, cookie writes and fingerprinting fire within 15–20 s | **true for tracking domains only** | 93–95% of (site, tracking domain) pairs by 15 s; per site, 65.4% of sites miss no tracking cookie at 15 s and 70.9% at 20 s; fingerprinting APIs 83.8% / over 87% (§4.4) | |
| | | recommend a 15–20 s dwell | **kept, qualified** | the paper recommends 15 s for time-constrained crawls; the page adds that cookie studies should wait longer | |
| | | BFS beats DFS because DFS "gets trapped in homogeneous sub-trees … same tracking templates", finds more "CMP banners" | **mechanism not in the paper** | the paper's evidence is that deep links are more often dead (BFS 50 live pages on ~79% of sites, DFS short on ~half) and that BFS still leads on sites where all strategies found 50 pages; CMP banners were not measured and banners were never interacted with | |
| | | intra-site statefulness needed to see "cookie syncing, stateful retargeting, cookies set once consent state is established" | **not measured** | the paper measures tracking domains, fingerprinting APIs and cookies; state was kept only within a site; stateless crawls gave the //most// raw unique cookies (§4.2) | |
| | | interaction "non-negotiable", "scrolling and cursor movement", trackers defer execution "to verify human presence" | **overstated / not in the paper** | interaction was Tab keystrokes plus scroll-to-bottom every 5 s, no mouse movement; state + interaction gave 0.8–11.6% more tracking domains than other configurations of the same strategy; the paper makes no claim about human-presence checks | |
| | | a new "Configuring Crawl Budgets" section | **declined** | the material fits existing sections of [[programming:interaction]] (depth, page choice, on-page wait) and [[programming:stateful_stateless]]; a parallel page would split them | |
| | |
| | **Figures quoted and where.** All from the paper's text: §3.1 (50 live pages / 100 requests cap, breadth-limited BFS branching 3, 30 s max wait), §3.1.3 (interaction = Tab keystrokes + scroll every 5 s, no clicks), §3.3 (CrUX October 2024, 9,673 of 10K reached, OpenWPM 0.30.0, Firefox 130, headful, one Georgia academic vantage, no bot mitigation), §4.1 (79% / 67% / ~half), §4.2 (1–14% tracking domains, 1–26% tracking cookies, 0.8–11.6% state+interaction, 9.5% stateless cookie total, Wilcoxon signed-rank with Holm), §4.3.1 (fixed depths 33/39/23/24/37/43), §4.3.2 (//k//, median stopping depth and page totals per metric; footnote 1 for the 10%-error operating point), §4.4 (timeout figures), §4.5 (1K-site re-run five months later). The Stafeev and Pellegrino sentence on randomised BFS was checked verbatim in ''data/fulltext/2024/USENIX/sok-state-of-the-krawlers-…/paper.cols.txt''. |
| | |
| | **What this does not establish.** Everything past page 50 (the paper's ceiling); the long tail of the ranking; the post-consent state; state carried across sites; the effect of scrolling alone. |
| | |
| | **On this page's content page.** ''programming:stateful_stateless'' gains one bullet in //What a stateful design buys you// (the within-site comparison and the stateless cookie-count inversion), one reading-list entry, and an update note on the first open question. |
| | |
| | **Review.** One **model: 'sonnet'** pass checked every new figure and quote against the paper text, the Stafeev and Pellegrino ''paper.cols.txt'' and Crossref, then re-checked after fixes. All table figures, the recomputed "fewer" percentages and both verbatim quotes were correct on the first pass. Accepted, all 13 plus two conflicts: the Stafeev sentence overstated support for plain BFS (their Table 8 has BFS and DFS within about 1%, DFS marginally ahead on JS source) — reworded, citation dropped from the breadth-first row; the 9.5% cookie margin is stateless-with-interaction over stateless-without, not stateless over stateful; the 1–14% / 1–26% BFS margins are over all sites, not at equal page counts; "the only measurement at scale of interaction" contradicted the paper's own citation of Demir et al. 2022/2023 — removed; the state+interaction effect is a combination the paper never splits, significance is on per-site distributions, fingerprinting-API margin only 0.5–1.2%; "twice, nothing else changed" misdescribed twelve configurations; "the clean experiment now exists" softened to "closest so far"; per-site cookie medians are similar, only the 75th percentile is higher; the dead-deep-links explanation is the authors' untested reading; the depth analysis configuration, //k// scoping of the 95th-percentile and 10%-error points, and "inefficient on many sites"; the re-run covers adaptive thresholds only and BFS coverage fell 79% → 72%; the cookie-wait caution is labelled as our reading and the paper's single 15 s recommendation is stated; "no mouse movement" is not in the paper; "replication" heading renamed; two pre-existing "only paper" sentences on the interaction page scoped to the corpus; booktitle aligned with Crossref. Rejected: none. Optional qualifier not added: BFS stateful+interaction was not significantly different from BFS stateless+interaction for fingerprinting APIs at depth 50 — the 0.5–1.2% margin on the page already signals it. |
| |
