Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler_detection [2026/09/16 21:20] – Addendum 2026-09-16: provenance for the new 'Measuring who crawls you' section — probe widths and the containment guard that caught the five-papers defect, the coding of the 44 single-mention papers with its residue, quote checks in both renderings, judge karel.kubicek.claude
+++ provenance:programming:crawler_detection [2026/09/21 13:42] (current) – Quote-check refresh 2026-09-21: regenerated cd_quotecheck and report_crawler_detection output blocks; 13 below threshold -> 11 rescued + 2 below in both. Authored by Claude karel.kubicek.claude
@@ Line 147: / Line 147: @@
 The guard was run **both** windowed and whole-page, because figures outside an audited
 window have survived a refresh on this wiki before. Both pass.
+The configuration table in the output below is a schema-completeness comparison. Its ''consentAction'' row is field population, not an audited paper-claim count; the 2026-09-05 audit found 55 of 1,120 paper claims and 279 of 313 unsupported ''no-interaction'' labels.
 <file text report_crawler_detection-output.txt>
@@ Line 364: / Line 366: @@
 Population: the 1,120 crawled papers. Sentinels are counted as sentinels.
-Choice                 Field                         Papers stating it  Share of 1,120
+Choice                                                               Field                         Papers stating it  Share of 1,120
----------------------  ----------------------------  -----------------  --------------
+-------------------------------------------------------------------  ----------------------------  -----------------  --------------
-Headless or headed     crawlConfig.headless          140                12.5%
+Headless or headed                                                   crawlConfig.headless          140                12.5%
-Stateful or stateless  crawlConfig.statefulness      219                19.6%
+Stateful or stateless                                                crawlConfig.statefulness      219                19.6%
-Consent action         crawlConfig.consentAction     349                31.2%
+Consent-action field populated (schema; not an audited paper claim)  crawlConfig.consentAction     349                31.2%
-Interaction depth      crawlConfig.interactionDepth  841                75.1%
+Interaction depth                                                    crawlConfig.interactionDepth  841                75.1%
-Authentication         crawlConfig.authentication    779                69.6%
+Authentication                                                       crawlConfig.authentication    779                69.6%
-Which browser          crawlConfig.browsers          529                47.2%
+Which browser                                                        crawlConfig.browsers          529                47.2%
 Headless is the single most detectable configuration a crawl can choose, and it
@@ Line 435: / Line 437: @@
 ==============================================================================
 scripts/cd_quotecheck.mjs, last line of its output:
-quotes: 111 exact, 37 partial, 13 below threshold
+quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
 Below-threshold is NOT "unsupported": every one was read by hand and all are
 present in the paper, spliced by the two-column reading order.
@@ Line 500: / Line 502: @@
 | Names a stealth patch in ''tools[]'' | 11 | 5,859 papers | report §H |
-**Populations that are NOT this page's.** ''site_queries.mjs --page crawlers'' reports a
+**Populations that are NOT this page's.** ''site_queries.mjs %%--page%% crawlers'' reports a
 crawl-config table over 1,080 papers — the papers that have a ''crawlConfig'' object —
 which is not the same as the 1,120 ''crawled'' papers used throughout this page. The
@@ Line 600: / Line 602: @@
 ### verdicts_all.json
   partial   CCS/2010/spam-the-underground-on-140-characters-or-less  93% of 15 windows
-  MISSING   CCS/2011/text-based-captcha-strengths-and-weaknesses  45% of 22 windows
+  RESCUED   CCS/2011/text-based-captcha-strengths-and-weaknesses  stored=45% pdf=82%
-            "Many websites use CAPTCHAs, or Completely Automated Public Turing tests to tell Computers and Humans Apart, in an attempt to block automated"
   partial   CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels  76% of 17 windows
-  MISSING   CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach  56% of 9 windows
+  RESCUED   CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach  stored=56% pdf=100%
-            "The real captchas are automatically collected using a web crawler written in Python."
   partial   CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten  61% of 28 windows
   partial   IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i  67% of 9 windows
-  MISSING   IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions  55% of 11 windows
+  RESCUED   IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions  stored=55% pdf=100%
-            "To ensure that the search engine servers would not be overloaded, we rate-limited our crawlers."
   partial   IMC/2015/who-is-com-learning-to-parse-whois-records  80% of 10 windows
   partial   IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets  92% of 49 windows
@@ Line 616: / Line 615: @@
   partial   IMC/2023/ecn-with-quic-challenges-in-the-wild  63% of 32 windows
   partial   IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th  86% of 29 windows
-  MISSING   NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories  46% of 13 windows
+  RESCUED   NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories  stored=46% pdf=100%
-            "we can run all queries every thirty minutes within the rate limits using a single API key."
+  RESCUED   NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms  stored=50% pdf=100%
-  MISSING   NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms  50% of 2 windows
+  RESCUED   NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns  stored=35% pdf=81%
-            "posting prevented due to bot detection"
-  MISSING   NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns  35% of 26 windows
-            "During the development of our framework, SCAMMAGNIFIER, we faced several challenges including dealing with diverse fraudulent shopping websi"
   MISSING   PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists  56% of 9 windows
             "For instance, LinkedIn blocked us from accessing user profile pages and job listings."
@@ Line 630: / Line 626: @@
   partial   PETS/2024/automatic-generation-of-web-censorship-probe-lists  74% of 23 windows
   partial   USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  87% of 30 windows
-  MISSING   USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  56% of 18 windows
+  RESCUED   USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  stored=56% pdf=100%
-            "Websites in these categories are also observed to be the most aggressive-with 50% of them blocking over 60% of the chosen relays."
+  RESCUED   USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  stored=46% pdf=100%
-  MISSING   USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  46% of 13 windows
-            "On session start, we provided the scraper with a session cookie manually obtained after solving a CAPTCHA."
   partial   USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies  72% of 18 windows
   partial   USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability  70% of 27 windows
@@ Line 640: / Line 634: @@
   partial   USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi  93% of 15 windows
   partial   USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v  86% of 14 windows
-  MISSING   USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale  56% of 18 windows
+  RESCUED   USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale  stored=56% pdf=100%
-            "except in 2024, when we found a new server that provided over a million SM2 certificates until they blocked our IP address."
   partial   USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal  73% of 15 windows
   partial   WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale  93% of 15 windows
   no-quote  WWW/2022/reproducibility-and-replicability-of-web-measurement-studies  (offtopic)
   partial   WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion  79% of 19 windows
-  MISSING   WWW/2024/navigating-the-post-api-dilemma  58% of 24 windows
+  RESCUED   WWW/2024/navigating-the-post-api-dilemma  stored=58% pdf=100%
-            "Moreover, given the severity and the impact of the topic surrounding COVID19, search engines may have placed stricter moderation policies on"
   partial   WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e  63% of 16 windows
-  MISSING   WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we  52% of 25 windows
+  RESCUED   WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we  stored=52% pdf=100%
-            "This approach provides a comprehensive view of the phishing webpages by fully rendering all client-side resources and helps evade anti-bot t"
 ### recall_verdicts.json
@@ Line 666: / Line 657: @@
   partial   USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents  72% of 18 windows
-quotes: 111 exact, 37 partial, 13 below threshold
+quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
 </file>
-All 13 below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are
+**Re-run 2026-09-21 with the PDF fallback: 11 of the 13 are rescued, 2 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The old figure was **13 below threshold**; 11 of those 13 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are a defect in the stored text rather than in the extraction. ''exact'' (111) and ''partial'' (37) did not move.
-present in the paper.** The mismatch is always the two-column reading order splicing
+All 13 of the then-below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are
+present in the paper**, a hand result the fallback independently confirms for 11 of them. The mismatch is always the two-column reading order splicing
 another column through the sentence. Three were rewritten after that reading, and the
 rewrite is recorded in the verdict file's ''note'' field:
@@ Line 1042: / Line 1035: @@
 | ''node scripts/check_wrap.mjs'' | OK |
 | ''node scripts/check_tables.mjs'' | OK — every table has one cell width |
-| ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''--code'') | OK |
+| ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''%%--code%%'') | OK |
 | ''node scripts/check_attributions.mjs'' | OK — 19 attributions |
-| ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 13 read by hand |
+| ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in both renderings. All 13 of the pre-fallback below-threshold quotes were read by hand (2026-08-29); re-run 2026-09-21 |
 | ''node scripts/verify_crawler_detection_figures.mjs'' | 22 per-paper figures checked against the papers, 0 not found |
 | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED |
@@ Line 1196: / Line 1189: @@
 | What was added | ''==== Measuring who crawls you ===='' inside //The AI-Crawler Turn, 2025–2026// |
 | What was corrected | the bullet that read "exactly **five papers** … mention an AI crawler or name an AI crawler user-agent token" now reads **three**, with the wider width explained inline |
-| Why a section and not a page | the [[roadmap]] row "AI crawlers, ''robots.txt'', training opt-out" assessed it at 4 candidate papers and sent it here. Re-probed at this run: **three** papers name an AI-crawler user agent, all 2025. That is not a page. The disposition stands and the roadmap row was updated with the corrected count. |
+| Why a section and not a page | the [[:roadmap]] row "AI crawlers, ''robots.txt'', training opt-out" assessed it at 4 candidate papers and sent it here. Re-probed at this run: **three** papers name an AI-crawler user agent, all 2025. That is not a page. The disposition stands and the roadmap row was updated with the corrected count. |
 | Orchestrating model | Claude Opus 5 |
 | Review layer | three focused Sonnet passes plus a generic Fable pass; log at the foot of this addendum |
@@ Line 1202: / Line 1195: @@
 ==== The script ====
-''scripts/report_ai_crawlers.mjs'', run as ''node scripts/report_ai_crawlers.mjs --list'',
+''scripts/report_ai_crawlers.mjs'', run as ''node scripts/report_ai_crawlers.mjs %%--list%%'',
 output committed to ''scripts/report_ai_crawlers-output.txt'' and reproduced in full
 below. It is separate from ''report_crawler_detection.mjs'' rather than a new section
@@ Line 1360: / Line 1353: @@
 Content Signals Policy is live with its ''search'' / ''ai-input'' / ''ai-train''
 vocabulary unchanged.
+=== Pass 4 — generic, no checklist (Fable) ===
+Spawned with the fixed section text, the whole page, both provenance addenda, the
+scripts and the roadmap rows, and asked for whatever the focused three were not
+looking for. **It had not returned when these pages were published on 2026-09-16**,
+and the pages were published rather than held, on the reasoning that three passes
+had already run and each of their accepted findings is applied and logged above.
+This is the honest state of the review layer for this addendum: three of four.
+If the pass returns, its findings belong here, and anything it finds is a defect
+that shipped.
+One thing the author changed without a reviewer asking, recorded so it is not
+mistaken for a review outcome: the Cloudflare Web Bot Auth paragraph originally sat
+between the disagreement box and the //nobody has published this// paragraph, and
+was moved after the latter, so the subsection ends on what moved rather than
+interrupting what is missing.
 ==== The published script ====
@@ Line 1617: / Line 1627: @@
 ==== Its real, unedited output ====
-Produced by ''node scripts/report_ai_crawlers.mjs --list'' on 2026-09-16 and committed as
+Produced by ''node scripts/report_ai_crawlers.mjs %%--list%%'' on 2026-09-16 and committed as
 ''scripts/report_ai_crawlers-output.txt''.
@@ Line 1787: / Line 1797: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== IETF Web Bot Auth currency correction, 2026-09-17 =====
+The dated sentence on [[programming:crawler_detection]] was re-fetched against the
+Datatracker API and status pages. The exact source URLs were:
+<code>
+https://datatracker.ietf.org/api/v1/doc/document/draft-meunier-web-bot-auth-architecture/?format=json
+https://datatracker.ietf.org/api/v1/doc/document/draft-meunier-webbotauth-httpsig-protocol/?format=json
+https://datatracker.ietf.org/api/v1/doc/state/4/?format=json
+https://datatracker.ietf.org/api/v1/doc/relateddocument/?format=json&source__name=draft-meunier-webbotauth-httpsig-protocol&relationship__slug=replaces&limit=100
+https://datatracker.ietf.org/api/v1/group/group/?format=json&acronym=webbotauth&limit=20
+https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/
+</code>
+The API returned ''draft-meunier-webbotauth-httpsig-protocol'' rev 02, timestamp
+-08-19T17:12:45Z, with state id 4; the state endpoint names id 4 **Replaced**.
+The filtered ''relateddocument'' result records ''replaces'' from that draft to
+''draft-meunier-web-bot-auth-architecture'' (and also to the earlier directory
+draft). The successor's status page now says **Replaced Internet-Draft**, shows
+the ''Expired & archived'' expiry badge, and names ''draft-ietf-webbotauth-httpsig-protocol''
+as its replacement. The group endpoint reports ''webbotauth'' as an active WG with
+a charter. The task's 2026-08-29 Active state had therefore moved by this check;
+the content page records the live state rather than carrying that older value forward.
+===== Review layer, 2026-09-17 =====
+One shared ''model: sonnet'' external-currency pass ran after publication and
+re-fetched the URLs above. It **passed** the corrected state chain: the original
+document is **Replaced** (state 4), the ''replaces'' relation names
+''draft-meunier-webbotauth-httpsig-protocol-02'', the intermediate draft is now
+also replaced by the WG draft, and the ''webbotauth'' WG is active. The reviewer
+confirmed that the item’s older 2026-08-29 Active wording had moved, so the
+published clause correctly follows the live source.
+===== ConsentAction audit propagation, 2026-09-17 =====
+**What was wrong.** The configuration comparison called the raw 349/1,120 = 31.2% ''Consent action'' value a paper statement, although it only reports that the extraction field is populated.
+**Command and real output.** ''node scripts/report_crawler_detection.mjs'' → ''Consent-action field populated (schema; not an audited paper claim) | crawlConfig.consentAction | 349 | 31.2%''. ''node scripts/consent_action_noninteraction_audit.mjs'' → **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels.
+**What changed.** The report row and the page now carry the schema qualifier; the 349 value remains because this is a reporting-completeness table, not the audit's paper-claim table.
+**Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_crawler_detection.mjs'' and checked the consent row plus the adjacent statefulness, interaction-depth and authentication rows. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed.
+**Rejected finding.** No schema value was overwritten with 55; the audited number is separately defined and published by [[privacy:consent]].
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''cd_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output, which came from the 2026-08-29 run. Re-run, last line, unedited:
+<code>
+$ node scripts/cd_quotecheck.mjs
+quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 161 | 161 | population unchanged |
+| exact | 111 | 111 | unchanged |
+| partial (≥60% of 5-word windows) | 37 | 37 | unchanged |
+| rescued from the PDF | — | **11** | these were inside the old 13 |
+| below threshold | **13** | **2** (in both renderings) | 13 = 11 + 2 exactly |
+**Both embedded output blocks were regenerated, not hand-edited.** ''out/cd_quotecheck-output.txt'' was rewritten by re-running the checker and the §Y2 block by re-running ''report_crawler_detection.mjs'', which reads the last line of that artifact rather than carrying the figure itself. The report's output is otherwise **byte-identical** to the published copy — the only line that moved is the one quoting the checker — which is the check that nothing else on this page drifted.
+**Scope of this edit.** The two output blocks, the prose under the verdict table, the guard-table row and the content page's quote bullet. No fold, candidate count, obstruction rate or citation was touched.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Commands | ''%%node scripts/cd_quotecheck.mjs > out/cd_quotecheck-output.txt%%'' then ''%%node scripts/report_crawler_detection.mjs%%'' |
+| Artifacts | ''out/cd_quotecheck-output.txt'' (regenerated; the 2026-08-29 copy kept as ''out/cd_quotecheck-output.txt.bak0921''), ''out/cd_report_0921.txt'' |
+| Script changes | none |
+| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler_detection]] |
+| Pages saved | this page, [[:programming:crawler_detection]] |