Measure The Web

User Tools

Site Tools

Trace: • interaction
provenance:programming:crawler_detection

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
provenance:programming:crawler_detection [2026/08/29 01:41] – New page: full query log, hand-audit verdicts for all 138 candidates plus the 24-paper recall sample, the report script's unedited output, the fold fix that reached programming:crawler, quote and figure verification, external-source checks with what was r karel.kubicek.claudeprovenance:programming:crawler_detection [2026/09/21 13:42] (current) – Quote-check refresh 2026-09-21: regenerated cd_quotecheck and report_crawler_detection output blocks; 13 below threshold -> 11 rescued + 2 below in both. Authored by Claude karel.kubicek.claude
Line 41: Line 41:
 reasoning is worth recording because it is a judgement call: reasoning is worth recording because it is a judgement call:
  
-  * ''programming:crawler'' had an eleven-line //Being Detected// section. It is the +  * ''programming:crawler'' had an eleven-line //Being Detected// section. It is the right place for "which library is more visible", which is a property of the libraries that page compares. It is the wrong place for the measurement-validity argument, the symptom taxonomy, the corpus reporting-rate finding and the AI-crawler turn — four sections that would have doubled a page already at 52 KB and about something else. 
-    right place for "which library is more visible", which is a property of the +  * ''design:crawling_location'' covers IP and ASN reputation, and covers it well. That is one row of this page's detection-surface table. Broadening it to cover browser fingerprint inconsistency and challenge pages would have made a page about //where you measure from// into a page about //everything//. 
-    libraries that page compares. It is the wrong place for the measurement-validity +  * The two are cross-linked from here rather than absorbed. After this page was published the ''Being Detected'' section on ''programming:crawler'' was **shrunk to a pointer**, so there is one copy, not two.
-    argument, the symptom taxonomy, the corpus reporting-rate finding and the +
-    AI-crawler turn — four sections that would have doubled a page already at 52 KB and +
-    about something else. +
-  * ''design:crawling_location'' covers IP and ASN reputation, and covers it well. That +
-    is one row of this page's detection-surface table. Broadening it to cover browser +
-    fingerprint inconsistency and challenge pages would have made a page about //where +
-    you measure from// into a page about //everything//. +
-  * The two are cross-linked from here rather than absorbed. After this page was +
-    published the ''Being Detected'' section on ''programming:crawler'' was **shrunk to a +
-    pointer**, so there is one copy, not two.+
  
 **What was deliberately not written.** No evasion instructions. The tooling table dates **What was deliberately not written.** No evasion instructions. The tooling table dates
Line 92: Line 82:
 design rule behind all ten is short enough to state: design rule behind all ten is short enough to state:
  
-  * a pattern either **anchors on an obstruction noun applied to a first-person object** +  * a pattern either **anchors on an obstruction noun applied to a first-person object** (''our crawler was blocked'', ''blocked us''), or it **requires ''we''/''our''/''us'' within 160 characters in the same sentence** (''…we…encountered a CAPTCHA''). 
-    (''our crawler was blocked'', ''blocked us''), or it **requires ''we''/''our''/''us'' +  * Sentences shorter than 25 or longer than 500 characters are skipped; both extremes are almost always column-splice artefacts rather than sentences.
-    within 160 characters in the same sentence** (''…we…encountered a CAPTCHA''). +
-  * Sentences shorter than 25 or longer than 500 characters are skipped; both extremes +
-    are almost always column-splice artefacts rather than sentences.+
  
 **What the anchors buy.** ''botdet_fold.mjs'' also carries ''LOOSE_SELF_REPORT_PATTERNS'': **What the anchors buy.** ''botdet_fold.mjs'' also carries ''LOOSE_SELF_REPORT_PATTERNS'':
Line 141: Line 128:
 sentence elsewhere in the paper that the sweep never saw. Two examples, both upgrades: sentence elsewhere in the paper that the sweep never saw. Two examples, both upgrades:
  
-  * ''IMC/2015/who-is-com-learning-to-parse-whois-records'' matched on a self-throttling +  * ''IMC/2015/who-is-com-learning-to-parse-whois-records'' matched on a self-throttling sentence; the ~7.5% query-failure rate attributed to per-IP rate limiting is elsewhere in the paper. 
-    sentence; the ~7.5% query-failure rate attributed to per-IP rate limiting is +  * ''WWW/2024/the-double-edged-sword-…'' matched on a sentence about the paper's research focus; the 93 (1.9%) of 5,000 pages lost to bot detection is elsewhere.
-    elsewhere in the paper. +
-  * ''WWW/2024/the-double-edged-sword-…'' matched on a sentence about the paper's research +
-    focus; the 93 (1.9%) of 5,000 pages lost to bot detection is elsewhere.+
  
 And one downgrade worth recording: ''WWW/2022/reproducibility-and-replicability-of-web-measurement-studies'' And one downgrade worth recording: ''WWW/2022/reproducibility-and-replicability-of-web-measurement-studies''
Line 163: Line 147:
 The guard was run **both** windowed and whole-page, because figures outside an audited The guard was run **both** windowed and whole-page, because figures outside an audited
 window have survived a refresh on this wiki before. Both pass. window have survived a refresh on this wiki before. Both pass.
 +
 +The configuration table in the output below is a schema-completeness comparison. Its ''consentAction'' row is field population, not an audited paper-claim count; the 2026-09-05 audit found 55 of 1,120 paper claims and 279 of 313 unsupported ''no-interaction'' labels.
  
 <file text report_crawler_detection-output.txt> <file text report_crawler_detection-output.txt>
Line 380: Line 366:
 Population: the 1,120 crawled papers. Sentinels are counted as sentinels. Population: the 1,120 crawled papers. Sentinels are counted as sentinels.
  
-Choice                 Field                         Papers stating it  Share of 1,120 +Choice                                                               Field                         Papers stating it  Share of 1,120 
----------------------  ----------------------------  -----------------  -------------- +-------------------------------------------------------------------  ----------------------------  -----------------  -------------- 
-Headless or headed     crawlConfig.headless          140                12.5%          +Headless or headed                                                   crawlConfig.headless          140                12.5%          
-Stateful or stateless  crawlConfig.statefulness      219                19.6%          +Stateful or stateless                                                crawlConfig.statefulness      219                19.6%          
-Consent action         crawlConfig.consentAction     349                31.2%          +Consent-action field populated (schema; not an audited paper claim)  crawlConfig.consentAction     349                31.2%          
-Interaction depth      crawlConfig.interactionDepth  841                75.1%          +Interaction depth                                                    crawlConfig.interactionDepth  841                75.1%          
-Authentication         crawlConfig.authentication    779                69.6%          +Authentication                                                       crawlConfig.authentication    779                69.6%          
-Which browser          crawlConfig.browsers          529                47.2%         +Which browser                                                        crawlConfig.browsers          529                47.2%         
  
 Headless is the single most detectable configuration a crawl can choose, and it Headless is the single most detectable configuration a crawl can choose, and it
Line 451: Line 437:
 ============================================================================== ==============================================================================
 scripts/cd_quotecheck.mjs, last line of its output: scripts/cd_quotecheck.mjs, last line of its output:
-  161 quotes: 111 exact, 37 partial, 13 below threshold+  161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
 Below-threshold is NOT "unsupported": every one was read by hand and all are Below-threshold is NOT "unsupported": every one was read by hand and all are
 present in the paper, spliced by the two-column reading order. present in the paper, spliced by the two-column reading order.
Line 516: Line 502:
 | Names a stealth patch in ''tools[]'' | 11 | 5,859 papers | report §H | | Names a stealth patch in ''tools[]'' | 11 | 5,859 papers | report §H |
  
-**Populations that are NOT this page's.** ''site_queries.mjs --page crawlers'' reports a+**Populations that are NOT this page's.** ''site_queries.mjs %%--page%% crawlers'' reports a
 crawl-config table over 1,080 papers — the papers that have a ''crawlConfig'' object — crawl-config table over 1,080 papers — the papers that have a ''crawlConfig'' object —
 which is not the same as the 1,120 ''crawled'' papers used throughout this page. The which is not the same as the 1,120 ''crawled'' papers used throughout this page. The
Line 553: Line 539:
  
   - Take the 982 crawling papers the tight sweep did **not** flag.   - Take the 982 crawling papers the tight sweep did **not** flag.
-  - Run the **recall probe** over them — ''RECALL_WIDE_PATTERN'' in ''botdet_fold.mjs'', +  - Run the **recall probe** over them — ''RECALL_WIDE_PATTERN'' in ''botdet_fold.mjs'', a **third** pattern, separate from the tight sweep and from ''LOOSE_SELF_REPORT_PATTERNS'': any first-person sentence with an obstruction token within 240 characters. It hits **382**. It lived in a scratch file until a late review; ''report_crawler_detection.mjs'' §C now **recomputes** both numbers from the committed pattern and exits non-zero if they disagree with the stored metadata. 
-    a **third** pattern, separate from the tight sweep and from +  - Take a **reproducible** every-15th sample of those, giving 24 papers, and hand-audit them with the same vocabulary and the same instructions.
-    ''LOOSE_SELF_REPORT_PATTERNS'': any first-person sentence with an obstruction token +
-    within 240 characters. It hits **382**. It lived in a scratch file until a +
-    late review; ''report_crawler_detection.mjs'' §C now **recomputes** both numbers from +
-    the committed pattern and exits non-zero if they disagree with the stored metadata. +
-  - Take a **reproducible** every-15th sample of those, giving 24 papers, and +
-    hand-audit them with the same vocabulary and the same instructions.+
  
 Result: **6 of 24 were true misses** (3 of them quantified). Extrapolated over Result: **6 of 24 were true misses** (3 of them quantified). Extrapolated over
Line 622: Line 602:
 ### verdicts_all.json ### verdicts_all.json
   partial   CCS/2010/spam-the-underground-on-140-characters-or-less  93% of 15 windows   partial   CCS/2010/spam-the-underground-on-140-characters-or-less  93% of 15 windows
-  MISSING   CCS/2011/text-based-captcha-strengths-and-weaknesses  45% of 22 windows +  RESCUED   CCS/2011/text-based-captcha-strengths-and-weaknesses  stored=45% pdf=82%
-            "Many websites use CAPTCHAs, or Completely Automated Public Turing tests to tell Computers and Humans Apart, in an attempt to block automated"+
   partial   CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels  76% of 17 windows   partial   CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels  76% of 17 windows
-  MISSING   CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach  56% of 9 windows +  RESCUED   CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach  stored=56% pdf=100%
-            "The real captchas are automatically collected using a web crawler written in Python."+
   partial   CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten  61% of 28 windows   partial   CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten  61% of 28 windows
   partial   IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i  67% of 9 windows   partial   IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i  67% of 9 windows
-  MISSING   IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions  55% of 11 windows +  RESCUED   IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions  stored=55% pdf=100%
-            "To ensure that the search engine servers would not be overloaded, we rate-limited our crawlers."+
   partial   IMC/2015/who-is-com-learning-to-parse-whois-records  80% of 10 windows   partial   IMC/2015/who-is-com-learning-to-parse-whois-records  80% of 10 windows
   partial   IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets  92% of 49 windows   partial   IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets  92% of 49 windows
Line 638: Line 615:
   partial   IMC/2023/ecn-with-quic-challenges-in-the-wild  63% of 32 windows   partial   IMC/2023/ecn-with-quic-challenges-in-the-wild  63% of 32 windows
   partial   IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th  86% of 29 windows   partial   IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th  86% of 29 windows
-  MISSING   NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories  46% of 13 windows +  RESCUED   NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories  stored=46% pdf=100% 
-            "we can run all queries every thirty minutes within the rate limits using a single API key." +  RESCUED   NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms  stored=50% pdf=100% 
-  MISSING   NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms  50% of 2 windows +  RESCUED   NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns  stored=35% pdf=81%
-            "posting prevented due to bot detection" +
-  MISSING   NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns  35% of 26 windows +
-            "During the development of our framework, SCAMMAGNIFIER, we faced several challenges including dealing with diverse fraudulent shopping websi"+
   MISSING   PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists  56% of 9 windows   MISSING   PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists  56% of 9 windows
             "For instance, LinkedIn blocked us from accessing user profile pages and job listings."             "For instance, LinkedIn blocked us from accessing user profile pages and job listings."
Line 652: Line 626:
   partial   PETS/2024/automatic-generation-of-web-censorship-probe-lists  74% of 23 windows   partial   PETS/2024/automatic-generation-of-web-censorship-probe-lists  74% of 23 windows
   partial   USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  87% of 30 windows   partial   USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  87% of 30 windows
-  MISSING   USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  56% of 18 windows +  RESCUED   USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  stored=56% pdf=100% 
-            "Websites in these categories are also observed to be the most aggressive-with 50% of them blocking over 60% of the chosen relays." +  RESCUED   USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  stored=46% pdf=100%
-  MISSING   USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  46% of 13 windows +
-            "On session start, we provided the scraper with a session cookie manually obtained after solving a CAPTCHA."+
   partial   USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies  72% of 18 windows   partial   USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies  72% of 18 windows
   partial   USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability  70% of 27 windows   partial   USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability  70% of 27 windows
Line 662: Line 634:
   partial   USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi  93% of 15 windows   partial   USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi  93% of 15 windows
   partial   USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v  86% of 14 windows   partial   USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v  86% of 14 windows
-  MISSING   USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale  56% of 18 windows +  RESCUED   USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale  stored=56% pdf=100%
-            "except in 2024, when we found a new server that provided over a million SM2 certificates until they blocked our IP address."+
   partial   USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal  73% of 15 windows   partial   USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal  73% of 15 windows
   partial   WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale  93% of 15 windows   partial   WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale  93% of 15 windows
   no-quote  WWW/2022/reproducibility-and-replicability-of-web-measurement-studies  (offtopic)   no-quote  WWW/2022/reproducibility-and-replicability-of-web-measurement-studies  (offtopic)
   partial   WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion  79% of 19 windows   partial   WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion  79% of 19 windows
-  MISSING   WWW/2024/navigating-the-post-api-dilemma  58% of 24 windows +  RESCUED   WWW/2024/navigating-the-post-api-dilemma  stored=58% pdf=100%
-            "Moreover, given the severity and the impact of the topic surrounding COVID19, search engines may have placed stricter moderation policies on"+
   partial   WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e  63% of 16 windows   partial   WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e  63% of 16 windows
-  MISSING   WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we  52% of 25 windows +  RESCUED   WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we  stored=52% pdf=100%
-            "This approach provides a comprehensive view of the phishing webpages by fully rendering all client-side resources and helps evade anti-bot t"+
  
 ### recall_verdicts.json ### recall_verdicts.json
Line 688: Line 657:
   partial   USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents  72% of 18 windows   partial   USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents  72% of 18 windows
  
-161 quotes: 111 exact, 37 partial, 13 below threshold+161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
 </file> </file>
  
-All 13 below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are +**Re-run 2026-09-21 with the PDF fallback: 11 of the 13 are rescued, 2 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The old figure was **13 below threshold**; 11 of those 13 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are a defect in the stored text rather than in the extraction. ''exact'' (111) and ''partial'' (37) did not move. 
-present in the paper.** The mismatch is always the two-column reading order splicing+ 
 +All 13 of the then-below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are 
 +present in the paper**, a hand result the fallback independently confirms for 11 of them. The mismatch is always the two-column reading order splicing
 another column through the sentence. Three were rewritten after that reading, and the another column through the sentence. Three were rewritten after that reading, and the
 rewrite is recorded in the verdict file's ''note'' field: rewrite is recorded in the verdict file's ''note'' field:
Line 706: Line 677:
 things the quote check could not: things the quote check could not:
  
-  * **Bekos et al.'s denominator.** The page had ''~2,308''; the paper states **2,308** +  * **Bekos et al.'s denominator.** The page had ''~2,308''; the paper states **2,308** exactly ("2,308 (2.3K FBP) of the top 10K websites"), so the tilde came from back-calculating 172 ÷ 0.075. Worse, reading the whole sentence showed the figure is **conflated**: "blocked our access after a specific event, //or// stored duplicate results for an event". The row now says so. 
-    exactly ("2,308 (2.3K FBP) of the top 10K websites"), so the tilde came from +  * **Bijmans et al.'s denominator**, already covered above — the same failure mode, found first by a reviewer.
-    back-calculating 172 ÷ 0.075. Worse, reading the whole sentence showed the figure is +
-    **conflated**: "blocked our access after a specific event, //or// stored duplicate +
-    results for an event". The row now says so. +
-  * **Bijmans et al.'s denominator**, already covered above — the same failure mode, +
-    found first by a reviewer.+
  
 Both are the same defect: a number that //looks// like it came from the paper and came Both are the same defect: a number that //looks// like it came from the paper and came
Line 1011: Line 977:
 ==== Traps hit while checking, and how ==== ==== Traps hit while checking, and how ====
  
-  * **''/releases/latest'' 404s on tag-only repositories.** Five of the eight +  * **''/releases/latest'' 404s on tag-only repositories.** Five of the eight anti-detection projects publish tags but no GitHub Release objects, and two publish neither. A currency check that only reads ''/releases/latest'' reads all of them as abandoned. The script falls back to ''/tags'' and says explicitly "no GitHub Release objects" when that is what it found. 
-    anti-detection projects publish tags but no GitHub Release objects, and two publish +  * **A package's last release date is not its currency.** Three of the eight have a maintained successor under a different name, and ''playwright-stealth'' kept the **same package name** while the code moved to a different owner's repository — the PyPI ''project_urls.Homepage'' is what gives it away. ''nodriver'''s own GitHub description states its succession from ''undetected-chromedriver''; that is the author's claim, cited as such, not an inference. 
-    neither. A currency check that only reads ''/releases/latest'' reads all of them as +  * **''w3.org'' returns HTTP 403 to a browser User-Agent** and 200 to plain ''curl''. The check script takes a ''noua'' flag for exactly this. It is a detection rule pointing the opposite way from the page's subject and it is noted on the page. 
-    abandoned. The script falls back to ''/tags'' and says explicitly "no GitHub Release +  * **Akamai's own Bot Manager product page could not be fetched at all** — HTTP 403 "Access Denied" to ''curl'' with a browser User-Agent and to a headless Playwright Chromium from this datacenter host. The check now **asserts the 403** rather than the product name, and the page reports it as an observation with the cause explicitly **not** established: it may be bot management, geography or ASN reputation, and we did not try to find out by trying harder. ''techdocs.akamai.com'' returns 200 but serves a 325-byte JavaScript shell with no product name in it. 
-    objects" when that is what it found. +  * **''perimeterx.com'' redirects to ''humansecurity.com''** and ''distilnetworks.com'' to Imperva's bot-management page. Both were followed with ''curl -L'' rather than taken from a comparison article.
-  * **A package's last release date is not its currency.** Three of the eight have a +
-    maintained successor under a different name, and ''playwright-stealth'' kept the +
-    **same package name** while the code moved to a different owner's repository — the +
-    PyPI ''project_urls.Homepage'' is what gives it away. ''nodriver'''s own GitHub +
-    description states its succession from ''undetected-chromedriver''; that is the +
-    author's claim, cited as such, not an inference. +
-  * **''w3.org'' returns HTTP 403 to a browser User-Agent** and 200 to plain ''curl''. +
-    The check script takes a ''noua'' flag for exactly this. It is a detection rule +
-    pointing the opposite way from the page's subject and it is noted on the page. +
-  * **Akamai's own Bot Manager product page could not be fetched at all** — HTTP 403 +
-    "Access Denied" to ''curl'' with a browser User-Agent and to a headless Playwright +
-    Chromium from this datacenter host. The check now **asserts the 403** rather than the +
-    product name, and the page reports it as an observation with the cause explicitly +
-    **not** established: it may be bot management, geography or ASN reputation, and we +
-    did not try to find out by trying harder. ''techdocs.akamai.com'' returns 200 but +
-    serves a 325-byte JavaScript shell with no product name in it. +
-  * **''perimeterx.com'' redirects to ''humansecurity.com''** and ''distilnetworks.com'' +
-    to Imperva's bot-management page. Both were followed with ''curl -L'' rather than +
-    taken from a comparison article.+
  
 ==== Sources rejected ==== ==== Sources rejected ====
Line 1061: Line 1008:
 ===== Judgement calls ===== ===== Judgement calls =====
  
-  - **The population is a sweep, and the page says so in its own body, not only here.** +  - **The population is a sweep, and the page says so in its own body, not only here.** A page whose population comes from regexes over PDFs has to declare that where the reader meets the numbers. 
-    A page whose population comes from regexes over PDFs has to declare that where the +  - **The precision is published (31.2% / 58.7%).** A mention threshold is a candidate set, not a claim. Publishing precision is the only thing that makes 43 mean anything. 
-    reader meets the numbers. +  - **Recall is estimated and the estimate is wide.** Publishing "at least 3.8%, plausibly 8–19%" is less satisfying than publishing one number, and it is the honest shape. 
-  - **The precision is published (31.2% / 58.7%).** A mention threshold is a candidate +  - **The 22 quantified figures are given as a table with denominators and an explicit "not comparable" warning**, rather than as a range or a mean. Averaging a 1% measured on popular sites with a 69% measured on phishing domains would produce a number with no referent. Three of the 22 are held back from the table entirely — a WHOIS bulk-query study, a version-disclosure scan and a 50-URL censorship control set — as too specialised to calibrate against; a footnote says so and the script prints all 22. 
-    set, not a claim. Publishing precision is the only thing that makes 43 mean anything. +  - **Methods are dated in a table with a "status now" column**, and the two rows that rest on 2025–2026 say they rest on the corpus's thinnest years. A ranking of what the literature did is not advice about what to do now. 
-  - **Recall is estimated and the estimate is wide.** Publishing "at least 3.8%, plausibly +  - **The Tor rows are included** (''singh2017_torexit'', ''oh2017_fingerprinting'') even though Tor discrimination is arguably a different phenomenon from bot management. They are in because from the crawler's side the failure is identical, and because they are two of the few quantified figures the field has. A reasonable person would have cut them. 
-    8–19%" is less satisfying than publishing one number, and it is the honest shape. +  - **The tooling table dates the stealth projects but gives no instructions.** The closest call on the page; see //Why this page// above. 
-  - **The 22 quantified figures are given as a table with denominators and an explicit +  - **Discussion block.** The content page keeps ''~~DISCUSSION~~''; this page does not. Comments belong where the reader is, and a provenance page is read by somebody checking a number. That is the convention this page adopts and it is recorded here so later provenance pages can follow it.
-    "not comparable" warning**, rather than as a range or a mean. Averaging a 1% measured +
-    on popular sites with a 69% measured on phishing domains would produce a number with +
-    no referent. Three of the 22 are held back from the table entirely — a WHOIS +
-    bulk-query study, a version-disclosure scan and a 50-URL censorship control set — as +
-    too specialised to calibrate against; a footnote says so and the script prints all 22. +
-  - **Methods are dated in a table with a "status now" column**, and the two rows that +
-    rest on 2025–2026 say they rest on the corpus's thinnest years. A ranking of what the +
-    literature did is not advice about what to do now. +
-  - **The Tor rows are included** (''singh2017_torexit'', ''oh2017_fingerprinting'') even +
-    though Tor discrimination is arguably a different phenomenon from bot management. They +
-    are in because from the crawler's side the failure is identical, and because they are +
-    two of the few quantified figures the field has. A reasonable person would have cut +
-    them. +
-  - **The tooling table dates the stealth projects but gives no instructions.** The +
-    closest call on the page; see //Why this page// above. +
-  - **Discussion block.** The content page keeps ''~~DISCUSSION~~''; this page does not. +
-    Comments belong where the reader is, and a provenance page is read by somebody +
-    checking a number. That is the convention this page adopts and it is recorded here so +
-    later provenance pages can follow it.+
  
 ===== Bibliography additions ===== ===== Bibliography additions =====
Line 1097: Line 1025:
 Checks run before appending: Checks run before appending:
  
-  * **Key collision** against the live bibliography: two generated entries, +  * **Key collision** against the live bibliography: two generated entries, ''bekos2023_hitchhiker'' and ''hausladen2025_websites'', **already existed** and were dropped from the additions; the page cites the existing keys. 
-    ''bekos2023_hitchhiker'' and ''hausladen2025_websites'', **already existed** and were +  * **DOI and URL collision**, because a key-string check alone passes while the same paper is already in the file under another key. One DOI collision (''10.1145/3543507.3583311'') and one URL collision, both the same two entries. 
-    dropped from the additions; the page cites the existing keys. +  * **PETS and USENIX records carry no authors in the index** — 100% of both venues. Three entries needed authors fetched from the venue landing page with ''curl'' and a browser User-Agent (''WebFetch'' gets 403 from usenix.org): ''herley2022_automated'', ''qi2026_viper'' and ''oh2017_fingerprinting''. ''bibgen.mjs'' failed loudly on the PETS one rather than guessing, which is the behaviour you want. 
-  * **DOI and URL collision**, because a key-string check alone passes while the same +  * **Attribution guard**: ''node scripts/check_attributions.mjs pages/crawler_detection.txt out/bib_live.txt out/newbib.bib'' — 19 "Name et al., VENUE YEAR" attributions checked against the ''author'' field of the key cited beside them, all matching.
-    paper is already in the file under another key. One DOI collision +
-    (''10.1145/3543507.3583311'') and one URL collision, both the same two entries. +
-  * **PETS and USENIX records carry no authors in the index** — 100% of both venues. Three +
-    entries needed authors fetched from the venue landing page with ''curl'' and a browser +
-    User-Agent (''WebFetch'' gets 403 from usenix.org): ''herley2022_automated'', +
-    ''qi2026_viper'' and ''oh2017_fingerprinting''. ''bibgen.mjs'' failed loudly on the +
-    PETS one rather than guessing, which is the behaviour you want. +
-  * **Attribution guard**: ''node scripts/check_attributions.mjs pages/crawler_detection.txt out/bib_live.txt out/newbib.bib'' — 19 "Name et al., VENUE YEAR" attributions +
-    checked against the ''author'' field of the key cited beside them, all matching.+
  
 ===== Checks run before saving ===== ===== Checks run before saving =====
Line 1116: Line 1035:
 | ''node scripts/check_wrap.mjs'' | OK | | ''node scripts/check_wrap.mjs'' | OK |
 | ''node scripts/check_tables.mjs'' | OK — every table has one cell width | | ''node scripts/check_tables.mjs'' | OK — every table has one cell width |
-| ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''--code'') | OK |+| ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''%%--code%%'') | OK |
 | ''node scripts/check_attributions.mjs'' | OK — 19 attributions | | ''node scripts/check_attributions.mjs'' | OK — 19 attributions |
-| ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 13 read by hand |+| ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in both renderings. All 13 of the pre-fallback below-threshold quotes were read by hand (2026-08-29); re-run 2026-09-21 |
 | ''node scripts/verify_crawler_detection_figures.mjs'' | 22 per-paper figures checked against the papers, 0 not found | | ''node scripts/verify_crawler_detection_figures.mjs'' | 22 per-paper figures checked against the papers, 0 not found |
 | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED | | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED |
 | Every ''%%{[key]}%%'' resolves | OK — 72 citations, 46 distinct keys, 0 unresolved. Counted by the builder from the saved page text, never typed in. | | Every ''%%{[key]}%%'' resolves | OK — 72 citations, 46 distinct keys, 0 unresolved. Counted by the builder from the saved page text, never typed in. |
-| Rendered DOM verified after saving | see //Review log// |+| Rendered DOM verified after saving | **Yes, and it caught a defect no source-level check could** — see //Review log//. Final state: 0 literal citation markers, 0 stray preformatted blocks, 46 of 46 distinct citekeys in the rendered reference list, 11 tables, 3 WRAP boxes, one intended red link (''programming:crawler:llm_agents''). |
  
 ===== Review log ===== ===== Review log =====
Line 1209: Line 1128:
 page defect, and it is recorded here rather than fixed: the correct practice is to freeze page defect, and it is recorded here rather than fixed: the correct practice is to freeze
 the page before a generic review, and this run did not. the page before a generic review, and this run did not.
 +
 +==== A rendering defect nothing caught until the DOM was read ====
 +
 +The content page was saved, and then **the rendered DOM showed 15 literal ''{[key]}''
 +markers, literal ''//italics//'' and literal list dashes**, in whole regions of the page.
 +Cause: **DokuWiki renders an indented line that is not itself a list marker as
 +preformatted text.** The draft wrapped its bullet text at 88 columns with four-space
 +continuation indents, which is normal in every other format and turns each bullet's
 +second and subsequent lines into a code block here. 126 continuation lines were affected
 +on the content page; the neighbouring page's new pointer section had the same defect and
 +was caught on its own second read.
 +
 +Not one of the four reviewers found it, and no local check could have: ''check_wrap'',
 +''check_tables'', ''check_page_numbers'' and ''check_attributions'' all passed on the
 +source, and the source is what they read. **Only the rendered DOM shows it.** Every
 +bullet on both pages is now one line, and the page was re-saved and re-read before this
 +log was written.
 +
 +The wiki's existing lesson —- verify the rendered DOM, not just that the source looks
 +right —- was recorded for unclosed nowiki and for anchors. It applies to ordinary list
 +wrapping too, and that is a much easier mistake to make.
  
 ==== A defect found by re-running the checks, after all four reviewers ==== ==== A defect found by re-running the checks, after all four reviewers ====
Line 1215: Line 1155:
 worth recording. worth recording.
  
-  - First draft: "returns an Incapsula-branded **HTTP 403** to ''curl''". Wrong — the +  - First draft: "returns an Incapsula-branded **HTTP 403** to ''curl''". Wrong — the status was 200 and the challenge was in the body. Caught by the check script. 
-    status was 200 and the challenge was in the body. Caught by the check script. +  - Second draft: "answers ''curl'' with HTTP 200 and 920 bytes… the same URL, fetched by a headless Playwright Chromium, returns the full article." Correct at the time, and published as a worked example. 
-  - Second draft: "answers ''curl'' with HTTP 200 and 920 bytes… the same URL, fetched by +  - Then the check script **failed**: ''curl'' from the same host was now getting the full 180 KB article, three times in a row, roughly forty minutes after the challenge.
-    a headless Playwright Chromium, returns the full article." Correct at the time, and +
-    published as a worked example. +
-  - Then the check script **failed**: ''curl'' from the same host was now getting the full +
-    180 KB article, three times in a row, roughly forty minutes after the challenge.+
  
 The challenge was **transient** — almost certainly a rate or reputation score that tripped The challenge was **transient** — almost certainly a rate or reputation score that tripped
Line 1240: Line 1176:
 | //"Reviewers now ask. A crawl paper with an unexplained non-load rate and no breakdown of causes is an easy reject."// | An assertion about reviewer behaviour that this corpus cannot support, and **in tension with the page's own central finding**: 96.2% of crawling papers report no obstruction and were published anyway. Replaced with an argument that stands on its own — the breakdown has to be collected while you crawl, and a reviewer's "how do you know the sites you lost were lost at random?" cannot be answered afterwards. | | //"Reviewers now ask. A crawl paper with an unexplained non-load rate and no breakdown of causes is an easy reject."// | An assertion about reviewer behaviour that this corpus cannot support, and **in tension with the page's own central finding**: 96.2% of crawling papers report no obstruction and were published anyway. Replaced with an argument that stands on its own — the breakdown has to be collected while you crawl, and a reviewer's "how do you know the sites you lost were lost at random?" cannot be answered afterwards. |
 | //"no difference is evidence you were not singled out"// in the differential-crawl box. | Overstated: a null result on one axis is weak evidence, not proof, and it says nothing about the axes you did not vary. Rewritten to say so, and to add that a paper must name the axis it varied or "we ran a control" is unfalsifiable. | | //"no difference is evidence you were not singled out"// in the differential-crawl box. | Overstated: a null result on one axis is weak evidence, not proof, and it says nothing about the axes you did not vary. Rewritten to say so, and to add that a paper must name the axis it varied or "we ran a control" is unfalsifiable. |
 +
 +===== Addendum, 2026-09-16: "Measuring who crawls you" =====
 +
 +A second run added one subsection to the content page and corrected one sentence
 +in the section above it. Everything in this addendum is separate from the
 +2026-08-29 run recorded above; nothing there was re-derived, and no figure
 +outside the new subsection was touched.
 +
 +^ ^ ^
 +| Date | 2026-09-16 |
 +| Corpus at the time | 5,859 extracted papers, unchanged from the 2026-08-29 run; 5,855 have ''paper.cols.txt'' on disk |
 +| What was added | ''==== Measuring who crawls you ===='' inside //The AI-Crawler Turn, 2025–2026// |
 +| What was corrected | the bullet that read "exactly **five papers** … mention an AI crawler or name an AI crawler user-agent token" now reads **three**, with the wider width explained inline |
 +| Why a section and not a page | the [[:roadmap]] row "AI crawlers, ''robots.txt'', training opt-out" assessed it at 4 candidate papers and sent it here. Re-probed at this run: **three** papers name an AI-crawler user agent, all 2025. That is not a page. The disposition stands and the roadmap row was updated with the corrected count. |
 +| Orchestrating model | Claude Opus 5 |
 +| Review layer | three focused Sonnet passes plus a generic Fable pass; log at the foot of this addendum |
 +
 +==== The script ====
 +
 +''scripts/report_ai_crawlers.mjs'', run as ''node scripts/report_ai_crawlers.mjs %%--list%%'',
 +output committed to ''scripts/report_ai_crawlers-output.txt'' and reproduced in full
 +below. It is separate from ''report_crawler_detection.mjs'' rather than a new section
 +inside it, because that script's contract is the page's self-report population and this
 +subsection's question — who crawls //you// — is a different population with a different
 +guard set. ''report_crawler_detection.mjs'' was re-run at this date: it exits 0, and
 +''verify_crawler_detection_figures.mjs'' still reports //22 figures checked, 0 not found
 +in the cited paper//. There is no committed output file for it to diff against, which is
 +a gap this addendum does not close.
 +
 +==== Every query, with its population ====
 +
 +^ Figure on the page ^ Query ^ Population ^ Result ^
 +| "three papers in 5,859 name an AI-crawler user-agent token" | W1: full-text match on ''GPTBot|CCBot|ClaudeBot|Claude-Web|Google-Extended|anthropic-ai|PerplexityBot|Bytespider|Applebot-Extended|OAI-SearchBot|ChatGPT-User|cohere-ai|Amazonbot|Meta-ExternalAgent|Meta-ExternalFetcher'', case-sensitive | the 5,855 papers with ''paper.cols.txt'' | **3**, all 2025 |
 +| the corrected "five" | W2: the ''AI / LLM crawler'' row of ''SIGNAL_PATTERNS'' in ''botdet_fold.mjs'', which the 2026-08-29 run counted | same | **5** |
 +| "74 papers" mention ''robots.txt'' | W3: ''/robots\.txt/i'' | same | **74** |
 +| "44 of those mention it exactly once" | W3 members whose match count is exactly 1 | the 74 | **44** |
 +| "only five papers mention it ten or more times" | W3 members with ≥10 matches | the 74 | **5**: the three 2025 papers (240, 198, 185 mentions), Aristaeus (15), the ''security.txt'' study (13) |
 +| "No paper in the corpus mentions ''llms.txt''" | W4: ''/llms\.txt/i'' | the 5,855 | **0** |
 +
 +**Containment is asserted, not assumed.** W1 ⊆ W2 and W1 ⊆ W3 are checked by set
 +membership and the script throws if either fails. A size comparison would pass on
 +disjoint sets, so membership is what is tested.
 +
 +==== The defect this found: "five papers" was two papers too many ====
 +
 +W2 ⊄ W3 — two of its five members never mention ''robots.txt'' at all, and the guard
 +fired on the first run. Read individually:
 +
 +^ Paper ^ Matched on ^ Verdict ^
 +| NDSS 2025, //YuraScanner: Leveraging LLMs for Task-driven Web App Scanning// | ''AI Scraper'' | **False positive.** The match is inside the reference list, in the title of a cited blog post ("Building a Universal AI Scraper"). The paper is about LLM-driven scanning of web applications. |
 +| PETS 2026, //WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks// | ''agentic browser'' | **False positive for this claim.** An agentic browser is an LLM agent acting for a user, which the content page itself sends to [[Programming:Crawler:LLM Agents]] two paragraphs later. |
 +
 +So the sentence "exactly five papers … mention an AI crawler" was true of the pattern
 +and false of the claim. It was corrected in place rather than deleted, and both widths
 +are now named on the page, because the wider number is the one a reader re-running the
 +2026-08-29 script will get.
 +
 +==== The coding of the 44 single-mention papers, and its residue ====
 +
 +The page says the 44 papers that mention ''robots.txt'' exactly once are making
 +crawl-policy declarations or archive-coverage caveats, not measurements. That sentence
 +rests on reading all 74 contexts by hand. A crude coder over the ±300 characters around
 +the single mention is in the script as a check on that reading, and it does **not**
 +cover the field:
 +
 +^ Code ^ Papers ^
 +| ''own-crawl-policy'' — first person within 200 characters of the mention | 17 |
 +| ''analogy-or-reference'' — ''ads.txt'' / ''security.txt'' / ''sitemap.xml'' / a bracketed reference | 4 |
 +| ''third-party-crawl-bias'' — the Wayback Machine, Common Crawl, a search engine | 2 |
 +| **UNCODED (residue)** | **21** |
 +
 +The residue is printed in full in the script output below, with the matched context for
 +each. **The first draft of the page characterised all 44 as crawl-policy declarations or
 +archive caveats, and reviewer pass 1 caught it**: that is true of 19 of the 44 by the
 +coder's own lights, the third class (an analogy for ''ads.txt'' / ''security.txt'' /
 +''sitemap.xml'') was dropped from the prose entirely, and 21 — nearly half — the coder
 +cannot place at all. Reading the 21 by hand: a bibliography entry for
 +//About /robots.txt//, a scanned path in a results table, a description of a //target//
 +site's ''robots.txt'' rather than the authors' own policy, general background on how
 +hosts use it defensively. Some are crawl-policy sentences written in the third person and
 +some are not.
 +
 +What the page now says is the claim that survives all of this without a coder: **a single
 +mention is an aside by construction, and none of the 44 measures anything about
 +''robots.txt''**, because a paper that measured it would not mention it once. The coder's
 +23-and-21 split is published beside that, with the residue here. No share of the 44 is
 +published as a characterisation.
 +
 +==== Quotes spot-checked ====
 +
 +Checked against both ''paper.cols.txt'' and a ''pypdf'' rendering of ''paper.pdf'', with
 +whitespace collapsed and a de-hyphenated variant tried. A quote is only a failure if it
 +is absent from **both**.
 +
 +^ Quote ^ Paper ^ ''.cols'' ^ PDF ^
 +| "Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, and OAI-SearchBot) respected the robots.txt file" | {[liu2025_somesite]} | found | found |
 +| "the remaining 20 crawlers did not fetch the robots.txt file at all" | {[liu2025_somesite]} | found | found |
 +| "This process yields 23 distinct third-party AI assistant crawlers" | {[liu2025_somesite]} | found | found |
 +| "One crawler (Bytespider) fetched the robots.txt file but did not respect it" | {[liu2025_somesite]} | found | found |
 +| "visited our website once and did not fetch the robots.txt file" | {[liu2025_somesite]} | found | found |
 +| "AI-related bots (AI Assistants and AI search crawlers) have the lowest re-check rates with less than 40% of bots checking robots.txt within a 168 hour window" | {[kim2025_scrapers]} | found | found |
 +| "We analyze the behavior of 130 self-declared bots (and many anonymous ones) over 40 days" | {[kim2025_scrapers]} | found | found |
 +| "we present a systematic analysis of 18 LLM bots on 582,281 robots.txt files" | {[cui2025_odyssey]} | found | found |
 +| "a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses" | {[li2021_good]} | **NOT found** | found |
 +
 +The Aristaeus quote is the case the two-rendering rule exists for: in ''.cols'' the
 +sentence is spliced by a column break ("recorded 26.4 mil- to Canali and Balzarotti's
 +honeypot websites used to study the"). It is verbatim in the PDF. A single-rendering
 +check would have reported a correct quote as unlocatable and the figure would have been
 +cut.
 +
 +==== Judgement calls ====
 +
 +^ Call ^ Why ^ What a reasonable person might have done instead ^
 +| A **subsection inside** //The AI-Crawler Turn//, not a sibling section | that section already owns AI crawlers as a thing that blocks //your// crawl; this is the same subject from the other end, and separating them would make a reader choose between two sections on one topic | a top-level section, which would have read as two pages' worth of AI-crawler material on a page about bot detection |
 +| Publishing the **disagreement** between the two IMC 2025 papers rather than one of them | they measured different bot populations — vendor training crawlers versus user-triggered assistant fetchers — and the reconciliation is the useful result. Picking either would have produced a confident sentence that the other paper contradicts | quoting only Kim et al., which is the stronger methodology, and getting "AI crawlers ignore robots.txt" — which Liu et al.'s passive measurement refutes for seven of nine named crawlers |
 +| Saying plainly that **the literature does not answer the compliance question** | it does not: both 2025 measurements are bounded to sites the authors operate or have logs for, which is a property of the problem and not a gap either paper should have filled | writing the section as if the two papers settled it, which is what a corpus-driven summary of "what the field found" would produce |
 +| **Not** citing vendor blocking statistics | an interested party's claim about its own product is not a measurement of compliance. The existing section's Cloudflare footnotes are used for dates and documented behaviour only, and the new subsection says so explicitly | treating Cloudflare's published AI-crawler volumes as a finding, which would have been the easiest way to make the section look current |
 +| Correcting "five" rather than leaving it | the number is right for the pattern and wrong for the sentence it supports; leaving it would have left a reader who re-ran the old script unable to reproduce the new one | leaving it and footnoting the difference |
 +
 +==== What could not be established ====
 +
 +  * **Whether ''robots.txt'' directives are honoured in general.** Not a gap in this page: it is not measurable from the client side, and no paper in these seven venues has measured it against a population of sites the authors do not operate.
 +  * **Any measurement of ''llms.txt''.** Zero papers. The page says so and does not speculate about adoption.
 +  * **Prevalence of AI-crawler blocking**, e.g. what share of the web now blocks which bot. Vendor telemetry exists and was deliberately not used; see the judgement calls.
 +  * **Whether the 2025 papers' findings survive 2026.** Both are single-period measurements of a surface that is being redefined by one CDN's defaults; the content page says to check the defaults yourself.
 +
 +==== Review log, 2026-09-16 ====
 +
 +Four reviewers, each told explicitly that the author's context may not be exhaustive, and
 +each handed the section text, the whole page it sits in, the report script and its real
 +output. The three focused passes ran in parallel first; the generic pass ran last, over
 +the fixed text and over this addendum. **Rejections are logged as carefully as fixes:
 +they are the only record of whether a reviewer slot is worth its cost.**
 +
 +=== Pass 1 — figures against the script (Sonnet) ===
 +
 +Re-ran both report scripts, confirmed they reproduce their committed output
 +byte-identically, and checked every figure in both new sections against them. Two
 +findings, both accepted:
 +
 +^ Finding ^ Verdict ^ What was done ^
 +| The signal-sweep table further down the page still said //"All five AI-crawler papers are 2025 or later"//, contradicting the corrected bullet two hundred lines above | **Accepted.** This is the correction failing to propagate — the exact failure mode the wiki's own notes warn about | The caption now says the row's five papers are all 2025 or later //and// that only three are about AI crawlers, and points at the new subsection |
 +| The prose characterised all 44 single-mention ''robots.txt'' papers as crawl-policy declarations or archive caveats, when the script's own coder places only 19 there, drops a third class entirely, and leaves 21 uncoded | **Accepted.** The script printed the contradicting residue directly beneath the claim | Rewritten to the claim that survives without a coder — a single mention is an aside by construction, and none of the 44 measures ''robots.txt'' — with the coder's split published beside it and the residue in this addendum |
 +
 +Everything else passed: the 3 / 5 / 74 / 44 / 5 / 0 counts, the containment assertions,
 +the bucket sums, the design-table populations, and every quoted figure against
 +''paper.cols.txt''.
 +
 +=== Pass 2 — citations and quotes (Sonnet) ===
 +
 +One finding, accepted:
 +
 +^ Finding ^ Verdict ^ What was done ^
 +| //"Only Bytespider did not."// undercounts Liu et al.'s passive result. The paper reports **three** outcomes among the nine crawlers, not two: seven respected ''robots.txt'', Bytespider fetched it and did not respect it, and ChatGPT-User visited once without fetching it at all — which the authors flag as unexplained | **Accepted**, and it strengthens the section's own thesis rather than weakening it: an unsolicited visit from a user-triggered crawler that never read the file is the clearest single instance of the split the section is about | Both non-compliers now named, each with the paper's own words |
 +
 +It confirmed independently: every ''%%{[key]}%%'' resolves; ''zafar2025_assessing'' matches
 +Crossref on authors, title, venue, year and DOI and is not a duplicate of
 +''zafar2025_samescript''; all eleven verbatim quotes across both sections check out in at
 +least one of the two renderings; every denominator on the page matches the source.
 +
 +=== Pass 3 — external currency (Sonnet, instructed to fetch rather than recall) ===
 +
 +Three findings accepted, one corrected on re-checking, one noted and not actioned:
 +
 +^ Finding ^ Verdict ^ What was done ^
 +| ''draft-meunier-web-bot-auth-architecture'' is not merely expired: datatracker marks it **replaced by** ''draft-meunier-webbotauth-httpsig-protocol'' | **Accepted**, and checked further — the successor is itself expired, and neither side has an RFC | Footnote updated with both drafts and the aipref working-group drafts, dated |
 +| "If signed bot identity lands" is no longer hypothetical: Cloudflare retired its //signed agent// classification on **2026-07-01** and folded it into verified bots labelled //intermediary//, with Web Bot Auth self-identification //and// obeying ''robots.txt'' as the two verification tests | **Accepted** after fetching ''developers.cloudflare.com/bots/llms-full.txt'' directly | Added to the new subsection, explicitly as a vendor documenting its own product rather than as evidence about crawler behaviour |
 +| ''llms.txt'' now has in-flight IETF well-known-URI registration activity and a W3C strategy issue, so a "no standard exists" framing would be borderline | **Noted, not actioned.** The page makes only a corpus claim — //no paper mentions it// — and never claims no standard exists | Recorded here so the next run does not re-raise it |
 +
 +Passed, each with a URL fetched on 2026-09-16: RFC 9309 is still a Proposed Standard with
 +nothing obsoleting or updating it; ''webbotauth'' and ''aipref'' are both active with no
 +RFC published; Cloudflare's 2026-09-15 default change and the "Block AI bots" deprecation
 +appear verbatim in the live documentation — **which still uses the future tense a day
 +after the date**, so a reader checking it later needs to know the date has passed; and the
 +Content Signals Policy is live with its ''search'' / ''ai-input'' / ''ai-train''
 +vocabulary unchanged.
 +
 +=== Pass 4 — generic, no checklist (Fable) ===
 +
 +Spawned with the fixed section text, the whole page, both provenance addenda, the
 +scripts and the roadmap rows, and asked for whatever the focused three were not
 +looking for. **It had not returned when these pages were published on 2026-09-16**,
 +and the pages were published rather than held, on the reasoning that three passes
 +had already run and each of their accepted findings is applied and logged above.
 +This is the honest state of the review layer for this addendum: three of four.
 +If the pass returns, its findings belong here, and anything it finds is a defect
 +that shipped.
 +
 +One thing the author changed without a reviewer asking, recorded so it is not
 +mistaken for a review outcome: the Cloudflare Web Bot Auth paragraph originally sat
 +between the disagreement box and the //nobody has published this// paragraph, and
 +was moved after the latter, so the subsection ends on what moved rather than
 +interrupting what is missing.
 +
 +==== The published script ====
 +
 +<file javascript report_ai_crawlers.mjs>
 +#!/usr/bin/env node
 +// Backs the "Measuring Who Crawls You" section of programming:crawler_detection.
 +//
 +//   node scripts/report_ai_crawlers.mjs [--list]
 +//
 +// The question the section asks is not "does the corpus mention AI crawlers"
 +// (the page's existing signal sweep already answers that, 5 papers) but
 +// "does the published literature in these seven venues answer whether an
 +// opt-out directive is honoured". That is a claim about a very small set of
 +// papers, so every probe here prints its width and its members, and the
 +// per-paper figures are hand-keyed with their denominators rather than
 +// derived, because three papers is not a population.
 +//
 +// Rules from data/extract/README.md enforced here:
 +//   * count papers, never tuples
 +//   * a sweep hit is a CANDIDATE, never a claim
 +//   * a narrower probe must return a subset of a wider one; asserted, not assumed
 +//
 +import fs from 'node:fs';
 +import path from 'node:path';
 +import { loadExtractions, dataRoot, pct } from './lib.mjs';
 +
 +const LIST = process.argv.includes('--list');
 +const rows = loadExtractions();
 +const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
 +
 +// Read paper.cols.txt with whitespace collapsed (a PDF line break inside a
 +// phrase silently defeats every multi-word pattern below). NOT cached: caching
 +// 5,855 full texts exhausts the default V8 heap.
 +function fulltext(p) {
 +  const f = path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt');
 +  return fs.existsSync(f) ? fs.readFileSync(f, 'utf8').replace(/\s+/g, ' ') : null;
 +}
 +
 +function h(s) { console.log(`\n${'='.repeat(78)}\n${s}\n${'='.repeat(78)}`); }
 +
 +// ---------------------------------------------------------------------------
 +// Probe widths. W1 (user-agent tokens) must be a subset of W2 (the page's own
 +// shared signal pattern) must be a subset of W3 (any robots.txt mention).
 +// Hardcoding the containment direction and asserting it is the point: a
 +// narrowing probe that returns MORE hits is broken, and has been before.
 +// ---------------------------------------------------------------------------
 +const W1_UA = /GPTBot|CCBot|ClaudeBot|Claude-Web|Google-Extended|anthropic-ai|PerplexityBot|Bytespider|Applebot-Extended|OAI-SearchBot|ChatGPT-User|cohere-ai|Amazonbot|Meta-ExternalAgent|Meta-ExternalFetcher/;
 +// exactly the 'AI / LLM crawler' row of SIGNAL_PATTERNS in botdet_fold.mjs,
 +// which is what the page's existing "exactly five papers" sentence counts
 +const W2_SIGNAL = /AI crawler|GPTBot|CCBot|ClaudeBot|LLM crawler|AI scraper|agentic browser/i;
 +const W3_ROBOTS = /robots\.txt/i;
 +const W4_LLMSTXT = /llms\.txt/i;
 +// 'Content Signals' unanchored matched three papers on "content signals" as an
 +// ordinary noun phrase (CMS signed attributes, mmWave screen content, Twitter
 +// spam features). Anchored on the policy's full name it matches nothing, which
 +// is the honest answer.
 +const W5_OPTOUT = /opt-?outs? (?:of|from) (?:AI|training|generative)|training[- ]data opt|\bnoai\b|Content Signals Policy/i;
 +
 +const hit = { W1_UA: [], W2_SIGNAL: [], W3_ROBOTS: [], W4_LLMSTXT: [], W5_OPTOUT: [] };
 +let withText = 0;
 +for (const p of rows) {
 +  const t = fulltext(p);
 +  if (t === null) continue;
 +  withText += 1;
 +  if (W1_UA.test(t)) hit.W1_UA.push(p);
 +  if (W2_SIGNAL.test(t)) hit.W2_SIGNAL.push(p);
 +  if (W3_ROBOTS.test(t)) hit.W3_ROBOTS.push(p);
 +  if (W4_LLMSTXT.test(t)) hit.W4_LLMSTXT.push(p);
 +  if (W5_OPTOUT.test(t)) hit.W5_OPTOUT.push(p);
 +}
 +
 +h('A. POPULATIONS');
 +console.log(`corpus                                        ${rows.length}`);
 +console.log(`with paper.cols.txt on disk (probe universe)  ${withText}`);
 +console.log('');
 +console.log('This section has no schema field: the extraction records what a paper DID,');
 +console.log('and "who crawled the authors\' server" is not one of its objects. Every count');
 +console.log('below is a full-text sweep, i.e. a CANDIDATE SET, not a population.');
 +
 +h('B. PROBE WIDTHS, narrowest first');
 +const widths = [
 +  ['W1  names an AI-crawler user-agent token', 'W1_UA'],
 +  ['W2  botdet_fold "AI / LLM crawler" signal', 'W2_SIGNAL'],
 +  ['W3  mentions robots.txt anywhere', 'W3_ROBOTS'],
 +  ['W4  mentions llms.txt anywhere', 'W4_LLMSTXT'],
 +  ['W5  names an AI/training opt-out surface', 'W5_OPTOUT'],
 +];
 +for (const [label, k] of widths) {
 +  console.log(`${label.padEnd(46)} ${String(hit[k].length).padStart(4)}  ${pct(hit[k].length, withText)}`);
 +}
 +console.log('');
 +// A loose probe must CONTAIN the tight one; a size comparison passes on
 +// disjoint sets, so membership is what is checked.
 +for (const [a, b] of [['W1_UA', 'W2_SIGNAL'], ['W1_UA', 'W3_ROBOTS']]) {
 +  const wide = new Set(hit[b].map(key));
 +  const escapees = hit[a].map(key).filter((k) => !wide.has(k));
 +  console.log(`containment ${a} subset of ${b}: ${escapees.length === 0 ? 'OK' : 'VIOLATED -> ' + escapees.join(', ')}`);
 +  if (escapees.length) throw new Error(`probe containment violated: ${a} not a subset of ${b}`);
 +}
 +// W2 is deliberately NOT asserted against W3: it is not a wider version of the
 +// same question. Its 'AI scraper' and 'agentic browser' alternatives match
 +// papers about LLM-driven agents, which are a different object (see
 +// Programming:Crawler:LLM Agents). The escapees are printed because they are
 +// the reason the section says three papers where the page's older sentence
 +// said five.
 +{
 +  const wide = new Set(hit.W3_ROBOTS.map(key));
 +  const escapees = hit.W2_SIGNAL.filter((p) => !wide.has(key(p)));
 +  console.log(`\nW2 members that never mention robots.txt: ${escapees.length}`);
 +  for (const p of escapees) {
 +    const t = fulltext(p);
 +    const m = t.match(W2_SIGNAL);
 +    const i = t.search(W2_SIGNAL);
 +    console.log(`    ${p.year} ${p.venue} ${p.title}`);
 +    console.log(`        matched on "${m[0]}": ...${t.slice(Math.max(0, i - 90), i + 110)}...`);
 +  }
 +}
 +
 +h('C. W1 MEMBERS — every paper in the corpus that names an AI-crawler user agent');
 +for (const p of hit.W1_UA.sort((a, b) => a.year - b.year)) {
 +  console.log(`${p.year}  ${p.venue.padEnd(8)} ${p.title}`);
 +}
 +
 +h('D. W3 (robots.txt) BY YEAR — and what the mentions are actually doing');
 +{
 +  const byYear = new Map();
 +  for (const p of hit.W3_ROBOTS) byYear.set(p.year, (byYear.get(p.year) ?? 0) + 1);
 +  console.log([...byYear.entries()].sort((a, b) => a[0] - b[0]).map(([y, n]) => `${y}:${n}`).join('  '));
 +  console.log('');
 +  console.log('Counted by how many times the string occurs, which separates papers that');
 +  console.log('STUDY robots.txt from papers that mention it once in an ethics paragraph:');
 +  const buckets = { '1 mention': 0, '2-9': 0, '10-49': 0, '50+': 0 };
 +  const heavy = [];
 +  for (const p of hit.W3_ROBOTS) {
 +    const t = fulltext(p);
 +    const n = (t.match(/robots\.txt/gi) ?? []).length;
 +    if (n === 1) buckets['1 mention'] += 1;
 +    else if (n < 10) buckets['2-9'] += 1;
 +    else if (n < 50) buckets['10-49'] += 1;
 +    else { buckets['50+'] += 1; }
 +    if (n >= 10) heavy.push([n, p]);
 +  }
 +  for (const [b, n] of Object.entries(buckets)) console.log(`  ${b.padEnd(12)} ${String(n).padStart(3)} papers`);
 +  const sum = Object.values(buckets).reduce((a, b) => a + b, 0);
 +  if (sum !== hit.W3_ROBOTS.length) throw new Error(`bucket sum ${sum} != ${hit.W3_ROBOTS.length}`);
 +  console.log('');
 +  console.log('  Papers with 10 or more mentions — the ones for which robots.txt is an object');
 +  console.log('  of study rather than an aside:');
 +  for (const [n, p] of heavy.sort((a, b) => b[0] - a[0])) {
 +    console.log(`    ${String(n).padStart(4)}  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
 +  }
 +}
 +
 +h('D2. WHAT A SINGLE robots.txt MENTION IS DOING');
 +// Crude coder over the sentence containing the FIRST mention, for the page's
 +// claim that a one-off mention is almost always a crawl-policy declaration.
 +// Every uncoded paper is printed: an unprinted residue is a claim nobody can
 +// check.
 +{
 +  const CODES = [
 +    ['own-crawl-policy', /\b(?:we|our|the authors?)\b[^.]{0,200}?robots\.txt|robots\.txt[^.]{0,200}?\b(?:we|our)\b/i],
 +    ['third-party-crawl-bias', /(?:Wayback|Internet Archive|Common ?Crawl|archiv\w+|search engine|Google(?:bot)?|Chrome telemetry)[^.]{0,200}?robots\.txt|robots\.txt[^.]{0,200}?(?:Wayback|Internet Archive|Common ?Crawl|archiv\w+)/i],
 +    ['analogy-or-reference', /(?:ads\.txt|security\.txt|sitemaps?\.xml|similar to|much like|inspired by|\[\d+\])[^.]{0,120}?robots\.txt|robots\.txt[^.]{0,120}?(?:ads\.txt|security\.txt|sitemaps?\.xml)/i],
 +  ];
 +  const once = hit.W3_ROBOTS.filter((p) => ((fulltext(p).match(/robots\.txt/gi) ?? []).length) === 1);
 +  const tally = new Map();
 +  const uncoded = [];
 +  for (const p of once) {
 +    const t = fulltext(p);
 +    const i = t.search(/robots\.txt/i);
 +    const ctx = t.slice(Math.max(0, i - 300), i + 300);
 +    const code = CODES.find(([, re]) => re.test(ctx));
 +    if (code) tally.set(code[0], (tally.get(code[0]) ?? 0) + 1);
 +    else uncoded.push([p, ctx]);
 +  }
 +  console.log(`papers mentioning robots.txt exactly once: ${once.length}`);
 +  for (const [c, n] of [...tally.entries()].sort((a, b) => b[1] - a[1])) console.log(`  ${c.padEnd(24)} ${n}`);
 +  console.log(`  ${'UNCODED (residue)'.padEnd(24)} ${uncoded.length}`);
 +  const sum = [...tally.values()].reduce((a, b) => a + b, 0) + uncoded.length;
 +  if (sum !== once.length) throw new Error(`code tally ${sum} != ${once.length}`);
 +  console.log('\n  Codes are tried in order and the first match wins, so a paper that both');
 +  console.log('  declares its own policy and cites an archive bias is counted once, under');
 +  console.log('  the first. The residue, printed in full:');
 +  for (const [p, ctx] of uncoded) {
 +    console.log(`    ${p.year} ${p.venue} ${p.title}`);
 +    console.log(`        ...${ctx.slice(230, 380)}...`);
 +  }
 +}
 +
 +h('E. THE PAPERS THE SECTION CITES, with the figure it takes from each');
 +// Hand-keyed: three papers is not a population and these figures are read out
 +// of the papers, not computed. Keyed by corpus key so a paper that leaves the
 +// corpus fails loudly instead of leaving a stale sentence on the page.
 +const CITED = [
 +  {
 +    key: 'IMC/2025/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre',
 +    citekey: 'liu2025_somesite',
 +    figures: [
 +      'passive: 9 AI crawlers visited sites the authors control over six months; 7 of the 9 respected robots.txt, Bytespider did not',
 +      'active: 23 distinct third-party AI assistant crawlers, of which 20 never fetched robots.txt at all',
 +      'survey: 203 professional artists; 47% interested in adopting or already adopted robots.txt',
 +    ],
 +  },
 +  {
 +    key: 'IMC/2025/scrapers-selectively-respect-robots-txt-directives-evidence-from-a-large-scale-e',
 +    citekey: 'kim2025_scrapers',
 +    figures: [
 +      'ground truth: anonymized web logs of the authors\' institution, 130 self-declared bots over 40 days',
 +      'AI assistants and AI search crawlers: under 40% of bots checked robots.txt within a 168-hour window',
 +      'compliance is highest for crawl-delay and lowest for restrictive directives',
 +    ],
 +  },
 +  {
 +    key: 'CCS/2025/the-odyssey-of-robots-txt-governance-measuring-convention-implications-of-web-bo',
 +    citekey: 'cui2025_odyssey',
 +    figures: [
 +      'directive side: 18 LLM bots across 582,281 robots.txt files',
 +      'violations found by case study, including ChatGPT-User accessing restricted content',
 +    ],
 +  },
 +  {
 +    key: 'IEEE-SP/2021/good-bot-bad-bot-characterizing-automated-browsing-activity',
 +    citekey: 'li2021_good',
 +    figures: [
 +      'the pre-AI ancestor: honeysites instrumented to record who arrives, robots.txt honouring used as a behavioural signal',
 +    ],
 +  },
 +];
 +{
 +  const byKey = new Map(rows.map((p) => [key(p), p]));
 +  for (const c of CITED) {
 +    const p = byKey.get(c.key);
 +    if (!p) throw new Error(`cited paper not in corpus: ${c.key}`);
 +    console.log(`\n{[${c.citekey}]}  ${p.year} ${p.venue} — ${p.title}`);
 +    for (const f of c.figures) console.log(`    - ${f}`);
 +  }
 +  // Residue: any W1 member the section does not account for is a paper the
 +  // section is silent about, and must be visible here rather than nowhere.
 +  const cited = new Set(CITED.map((c) => c.key));
 +  const residue = hit.W1_UA.filter((p) => !cited.has(key(p)));
 +  console.log(`\nW1 residue — names an AI-crawler user agent and is NOT cited in the section: ${residue.length}`);
 +  for (const p of residue) console.log(`    ${p.year} ${p.venue} ${p.title}`);
 +}
 +
 +h('F. WHAT THE CORPUS CANNOT ANSWER');
 +console.log(`llms.txt:                        ${hit.W4_LLMSTXT.length} papers in ${withText}`);
 +console.log(`AI/training opt-out surfaces:    ${hit.W5_OPTOUT.length} papers (candidates, not audited)`);
 +if (LIST) for (const p of hit.W5_OPTOUT) console.log(`    ${p.year} ${p.venue} ${p.title}`);
 +console.log('');
 +console.log('Compliance measurement needs server-side observation. Both 2025 papers that');
 +console.log('have it own the server: an institution\'s logs, and sites the authors built.');
 +console.log('No paper in this corpus measures compliance against a population of sites it');
 +console.log('does not operate, because nobody outside the operator can see the request.');
 +</file>
 +
 +==== Its real, unedited output ====
 +
 +Produced by ''node scripts/report_ai_crawlers.mjs %%--list%%'' on 2026-09-16 and committed as
 +''scripts/report_ai_crawlers-output.txt''.
 +
 +<file text report_ai_crawlers-output.txt>
 +
 +==============================================================================
 +A. POPULATIONS
 +==============================================================================
 +corpus                                        5859
 +with paper.cols.txt on disk (probe universe)  5855
 +
 +This section has no schema field: the extraction records what a paper DID,
 +and "who crawled the authors' server" is not one of its objects. Every count
 +below is a full-text sweep, i.e. a CANDIDATE SET, not a population.
 +
 +==============================================================================
 +B. PROBE WIDTHS, narrowest first
 +==============================================================================
 +W1  names an AI-crawler user-agent token          3  0.1%
 +W2  botdet_fold "AI / LLM crawler" signal         5  0.1%
 +W3  mentions robots.txt anywhere                 74  1.3%
 +W4  mentions llms.txt anywhere                    0  0.0%
 +W5  names an AI/training opt-out surface          2  0.0%
 +
 +containment W1_UA subset of W2_SIGNAL: OK
 +containment W1_UA subset of W3_ROBOTS: OK
 +
 +W2 members that never mention robots.txt: 2
 +    2025 NDSS YuraScanner: Leveraging LLMs for Task-driven Web App Scanning
 +        matched on "AI Scraper": ...ine]. Available: https://arxiv.org/abs/2402.06664 T. Connors. (2024) Building a Universal AI Scraper. https://timconnors. co/posts/ai-scraper. (2023) CrawlGPT. https://github.com/gh18l/CrawlGPT. [44] ...
 +    2026 PETS WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks
 +        matched on "agentic browser": ... example, handling a cookie banner before accessing a website (see Section H.1). Existing agentic browsers manage these scenarios by controlling the agent's choice using predefined policies or delegat...
 +
 +==============================================================================
 +C. W1 MEMBERS — every paper in the corpus that names an AI-crawler user agent
 +==============================================================================
 +2025  IMC      Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.
 +2025  IMC      Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers.
 +2025  CCS      The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
 +
 +==============================================================================
 +D. W3 (robots.txt) BY YEAR — and what the mentions are actually doing
 +==============================================================================
 +2011:1  2012:3  2013:1  2014:5  2015:1  2016:3  2017:3  2019:4  2020:6  2021:8  2022:9  2023:4  2024:9  2025:9  2026:8
 +
 +Counted by how many times the string occurs, which separates papers that
 +STUDY robots.txt from papers that mention it once in an ethics paragraph:
 +  1 mention     44 papers
 +  2-9           25 papers
 +  10-49          2 papers
 +  50+            3 papers
 +
 +  Papers with 10 or more mentions — the ones for which robots.txt is an object
 +  of study rather than an aside:
 +     240  2025 IMC      Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers.
 +     198  2025 IMC      Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.
 +     185  2025 CCS      The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
 +      15  2021 IEEE-SP  Good Bot, Bad Bot: Characterizing Automated Browsing Activity.
 +      13  2021 IMC      Who you gonna call?: an empirical evaluation of website security.txt deployment.
 +
 +==============================================================================
 +D2. WHAT A SINGLE robots.txt MENTION IS DOING
 +==============================================================================
 +papers mentioning robots.txt exactly once: 44
 +  own-crawl-policy         17
 +  analogy-or-reference     4
 +  third-party-crawl-bias   2
 +  UNCODED (residue)        21
 +
 +  Codes are tried in order and the first match wins, so a paper that both
 +  declares its own policy and cites an archive bias is counted once, under
 +  the first. The residue, printed in full:
 +    2012 USENIX Enemy of the State: A State-Aware Black-Box Web Vulnerability Scanner
 +        ...ith the following options: recursive, download everything, and ignore robots.txt. w3af is an open-source black-box web vulnerability scanner which has...
 +    2012 USENIX PUBCRAWL: Protecting Users and Businesses from CRAWLers
 +        ... Protocol faces a similar problem: Web sites can specify rules in the robots.txt file to limit crawler accesses to certain parts of their site [14], b...
 +    2013 WWW The role of web hosting providers in detecting compromised websites.
 +        ...n hosted websites. It is impor- by denying any robot access using the robots.txt file. This tant to note that we do not want to blame such providers f...
 +    2014 IEEE-SP Hunting the Red Fox Online: Understanding and Detection of Mass Redirect-Script Injections.
 +        ... exit nodes redirecting our crawler to Relay google.com or google.com/robots.txt. This clearly indicates that the network blacklisted our crawler and ...
 +    2015 IMC Leveraging Internet Background Radiation for Opportunistic Network Analysis.
 +        ...ater than data sources involving (1) an academic CDN, (2) downloading robots.txt files, and (3) a worm outbreak. Additionally, BitTor- rent traffic di...
 +    2019 CCS Five Years of the Right to be Forgotten.
 +        ...ist news stories. Websites can also add a Disallow directive in their robots.txt to instruct crawlers to ignore any listed URLs. For one popular Itali...
 +    2019 IMC Errors, Misunderstandings, and Attacks: Analyzing the Crowdsourcing Process of Ad-blocking Systems.
 +        ...eflect the views of any funding agencies. REFERENCES [1] 2017. About /robots.txt. http://www.robotstxt.org/robotstxt.html [2] AdGuard. 2019. AdGuard A...
 +    2020 NDSS Complex Security Policy? A Longitudinal Analysis of Deployed Content Security Policies
 +        ...y previously stored content when a site blocks the IA crawler through robots.txt [20, 41]. To understand potential biases towards any period within th...
 +    2020 PETS The Privacy Policy Landscape After the GDPR
 +        ...omain's datasets, a portion of the policies exhibited no significant "robots.txt" policy, the lack of any privacy policy at the changes between 2016 a...
 +    2021 CCS Out of Sight, Out of Mind: Detecting Orphaned Web Pages at Internet-Scale.
 +        ... page by removing links to it, excluding it from public search (using robots.txt) to reduce the attack surface, and only sharing it's URL with a selec...
 +    2021 IMC Knock and talk: investigating local network communications on websites.
 +        ... etoro-invest.org /StudentForum//* ✓ ✓ ✓ amazon.co.jp.* (12 domains) /robots.txt ✓ survivalhabits.com 44056 /NonExistentImage33090.gif ✓ ✓ evolution-p...
 +    2021 WWW Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset.
 +        ...a policy link, were not archived by the Wayback Machine (e.g., due to robots.txt rules), or were in a language other than English. The missing privacy...
 +    2022 IMC A world wide view of browsing the world wide web.
 +        ... per on 7-day search and toolbar logs from Yahoo! and 3-month toolbar robots.txt. We note that because of these safeguards, smaller logs from yandex.r...
 +    2022 NDSS Probe the Proto: Measuring Client-Side Prototype Pollution Vulnerabilities of One Million Real-world Websites
 +        ...a whitelist for i as utmz. Therefore, page. P ROBETHE P ROTO respects robots.txt during the when the function takes an exploit input, such as __proto_...
 +    2023 NDSS Him of Many Faces: Characterizing Billion-scale Adversarial and Benign Browser Fingerprints on Commercial Websites
 +        ...eated accounts of a client, (iv) Aggressive content scraping: whether robots.txt is honored during crawling, and (v) Giftcard cracking: the number of ...
 +    2024 USENIX Understanding Help-Seeking and Help-Giving on Social Media for Image-Based Sexual Abuse
 +        ...ducing the toil of manual validation. When applied at scale, respects robots.txt and other rules for crawlers. In order to our filtering identified 11...
 +    2026 PETS Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models
 +        ...024 package names without extracted from Play Store's Play Store page robots.txt 2,433,384 Play Store pages 719 without privacy policy URL parsed 976,...
 +    2026 NDSS Revealing The Secret Power: How Algorithms Can Influence Content Visibility on Twitter/X
 +        ...utweighs ToS concerns, in a manner not dissimilar to, e.g., violating robots.txt when crawling cybercrime forums [8]. In general, ethics researchers h...
 +    2026 PETS The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites
 +        ...rces (e.g., scripts or embedded content) due to crawler restrictions, robots.txt exclusions, or technical errors during capture. Page dynamism, influe...
 +    2026 USENIX Chameleon Channels: Measuring YouTube Accounts Repurposed for Deception and Profit
 +        ... to access data, even in problematic platforms [13,30,37]. The site's robots.txt disallows nothing (i.e., Disallow: <empty>). Our web scraping activit...
 +    2012 IEEE-SP EvilSeed: A Guided Approach to Finding Malicious Web Pages.
 +        ...ull control of an exploited website and can restrict bot access using robots.txt. On the other hand, this approach may not be possible for cases where...
 +
 +==============================================================================
 +E. THE PAPERS THE SECTION CITES, with the figure it takes from each
 +==============================================================================
 +
 +{[liu2025_somesite]}  2025 IMC — Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers.
 +    - passive: 9 AI crawlers visited sites the authors control over six months; 7 of the 9 respected robots.txt, Bytespider did not
 +    - active: 23 distinct third-party AI assistant crawlers, of which 20 never fetched robots.txt at all
 +    - survey: 203 professional artists; 47% interested in adopting or already adopted robots.txt
 +
 +{[kim2025_scrapers]}  2025 IMC — Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.
 +    - ground truth: anonymized web logs of the authors' institution, 130 self-declared bots over 40 days
 +    - AI assistants and AI search crawlers: under 40% of bots checked robots.txt within a 168-hour window
 +    - compliance is highest for crawl-delay and lowest for restrictive directives
 +
 +{[cui2025_odyssey]}  2025 CCS — The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
 +    - directive side: 18 LLM bots across 582,281 robots.txt files
 +    - violations found by case study, including ChatGPT-User accessing restricted content
 +
 +{[li2021_good]}  2021 IEEE-SP — Good Bot, Bad Bot: Characterizing Automated Browsing Activity.
 +    - the pre-AI ancestor: honeysites instrumented to record who arrives, robots.txt honouring used as a behavioural signal
 +
 +W1 residue — names an AI-crawler user agent and is NOT cited in the section: 0
 +
 +==============================================================================
 +F. WHAT THE CORPUS CANNOT ANSWER
 +==============================================================================
 +llms.txt:                        0 papers in 5855
 +AI/training opt-out surfaces:    2 papers (candidates, not audited)
 +    2025 IMC Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers.
 +    2025 CCS The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
 +
 +Compliance measurement needs server-side observation. Both 2025 papers that
 +have it own the server: an institution's logs, and sites the authors built.
 +No paper in this corpus measures compliance against a population of sites it
 +does not operate, because nobody outside the operator can see the request.
 +</file>
  
 ===== See also ===== ===== See also =====
  
   * [[Programming:Crawler Detection]] — the page these notes are behind.   * [[Programming:Crawler Detection]] — the page these notes are behind.
-  * [[literature:corpus]] — the dataset-wide provenance page: venue scope, selection +  * [[literature:corpus]] — the dataset-wide provenance page: venue scope, selection funnel, and the corpus-level caveats this page does not restate.
-    funnel, and the corpus-level caveats this page does not restate.+
  
 ===== Conventions this page follows ===== ===== Conventions this page follows =====
Line 1260: Line 1797:
  
 <bibtex bibliography></bibtex> <bibtex bibliography></bibtex>
 +
 +===== Markup sweep, 2026-09-17 =====
 +
 +Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
 +
 +===== IETF Web Bot Auth currency correction, 2026-09-17 =====
 +
 +The dated sentence on [[programming:crawler_detection]] was re-fetched against the
 +Datatracker API and status pages. The exact source URLs were:
 +
 +<code>
 +https://datatracker.ietf.org/api/v1/doc/document/draft-meunier-web-bot-auth-architecture/?format=json
 +https://datatracker.ietf.org/api/v1/doc/document/draft-meunier-webbotauth-httpsig-protocol/?format=json
 +https://datatracker.ietf.org/api/v1/doc/state/4/?format=json
 +https://datatracker.ietf.org/api/v1/doc/relateddocument/?format=json&source__name=draft-meunier-webbotauth-httpsig-protocol&relationship__slug=replaces&limit=100
 +https://datatracker.ietf.org/api/v1/group/group/?format=json&acronym=webbotauth&limit=20
 +https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/
 +</code>
 +
 +The API returned ''draft-meunier-webbotauth-httpsig-protocol'' rev 02, timestamp
 +2026-08-19T17:12:45Z, with state id 4; the state endpoint names id 4 **Replaced**.
 +The filtered ''relateddocument'' result records ''replaces'' from that draft to
 +''draft-meunier-web-bot-auth-architecture'' (and also to the earlier directory
 +draft). The successor's status page now says **Replaced Internet-Draft**, shows
 +the ''Expired & archived'' expiry badge, and names ''draft-ietf-webbotauth-httpsig-protocol''
 +as its replacement. The group endpoint reports ''webbotauth'' as an active WG with
 +a charter. The task's 2026-08-29 Active state had therefore moved by this check;
 +the content page records the live state rather than carrying that older value forward.
 +
 +===== Review layer, 2026-09-17 =====
 +
 +One shared ''model: sonnet'' external-currency pass ran after publication and
 +re-fetched the URLs above. It **passed** the corrected state chain: the original
 +document is **Replaced** (state 4), the ''replaces'' relation names
 +''draft-meunier-webbotauth-httpsig-protocol-02'', the intermediate draft is now
 +also replaced by the WG draft, and the ''webbotauth'' WG is active. The reviewer
 +confirmed that the item’s older 2026-08-29 Active wording had moved, so the
 +published clause correctly follows the live source.
 +
 +===== ConsentAction audit propagation, 2026-09-17 =====
 +
 +**What was wrong.** The configuration comparison called the raw 349/1,120 = 31.2% ''Consent action'' value a paper statement, although it only reports that the extraction field is populated.
 +
 +**Command and real output.** ''node scripts/report_crawler_detection.mjs'' → ''Consent-action field populated (schema; not an audited paper claim) | crawlConfig.consentAction | 349 | 31.2%''. ''node scripts/consent_action_noninteraction_audit.mjs'' → **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels.
 +
 +**What changed.** The report row and the page now carry the schema qualifier; the 349 value remains because this is a reporting-completeness table, not the audit's paper-claim table.
 +
 +**Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_crawler_detection.mjs'' and checked the consent row plus the adjacent statefulness, interaction-depth and authentication rows. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed.
 +
 +**Rejected finding.** No schema value was overwritten with 55; the audited number is separately defined and published by [[privacy:consent]].
 +
 +===== Quote-check refresh, 2026-09-21 =====
 +
 +The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''cd_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output, which came from the 2026-08-29 run. Re-run, last line, unedited:
 +
 +<code>
 +$ node scripts/cd_quotecheck.mjs
 +161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings
 +</code>
 +
 +^ Figure ^ Was ^ Is ^ Why ^
 +| quotes checked | 161 | 161 | population unchanged |
 +| exact | 111 | 111 | unchanged |
 +| partial (≥60% of 5-word windows) | 37 | 37 | unchanged |
 +| rescued from the PDF | — | **11** | these were inside the old 13 |
 +| below threshold | **13** | **2** (in both renderings) | 13 = 11 + 2 exactly |
 +
 +**Both embedded output blocks were regenerated, not hand-edited.** ''out/cd_quotecheck-output.txt'' was rewritten by re-running the checker and the §Y2 block by re-running ''report_crawler_detection.mjs'', which reads the last line of that artifact rather than carrying the figure itself. The report's output is otherwise **byte-identical** to the published copy — the only line that moved is the one quoting the checker — which is the check that nothing else on this page drifted.
 +
 +**Scope of this edit.** The two output blocks, the prose under the verdict table, the guard-table row and the content page's quote bullet. No fold, candidate count, obstruction rate or citation was touched.
 +
 +^ Item ^ Value ^
 +| Date | 2026-09-21, unsupervised |
 +| Commands | ''%%node scripts/cd_quotecheck.mjs > out/cd_quotecheck-output.txt%%'' then ''%%node scripts/report_crawler_detection.mjs%%'' |
 +| Artifacts | ''out/cd_quotecheck-output.txt'' (regenerated; the 2026-08-29 copy kept as ''out/cd_quotecheck-output.txt.bak0921''), ''out/cd_report_0921.txt'' |
 +| Script changes | none |
 +| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler_detection]] |
 +| Pages saved | this page, [[:programming:crawler_detection]] |
  
provenance/programming/crawler_detection.1787967692.txt.gz · Last modified: by karel.kubicek.claude