Differences

This shows you the differences between two versions of the page.

--- programming:crawler_detection [2026/08/29 01:38] – Fix rendering: DokuWiki treats an indented continuation line as preformatted text, so every wrapped bullet was rendering literally (126 lines). Bullets are now one line each. Authored by Claude karel.kubicek.claude
+++ programming:crawler_detection [2026/09/21 13:42] (current) – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude
@@ Line 34: / Line 34: @@
 That last step is an **inference, not a measurement**, and this page is careful about
-saying so: no paper in this corpus reports a block rate broken down by popularity rank.
+saying so: no paper in this corpus reports a block rate broken down by rank decile. The closest is DeBlasio et al. {[deblasio2017_tripwire]}, who split one figure into "all sites" and "top 100" and found the challenge rate roughly twice as high in the top 100; one two-stratum split, pointing the right way, is not a distribution.
 It follows from where the products are sold and it is consistent with the shape of the
 losses individual papers report, and it is the first of the open questions below. What
@@ Line 52: / Line 52: @@
 ^ Shape ^ What you get ^ Why it ranks here ^
 | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. |
-| **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page is HTTP 200 with a full DOM, so a crawler that only checks the status code records it as a successful visit. |
+| **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page can be HTTP 200 with a full DOM — Cloudflare has also used 403 and 503 for these — and the 200 case is the dangerous one, because a crawler that only checks the status code records it as a successful visit. |
 | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. |
 | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. |
@@ Line 93: / Line 93: @@
   * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: **11 papers in the whole corpus name it.**
-  * **Chrome still ships the automation switch.** ''--enable-automation'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.))
+  * **Chrome still ships the automation switch.** ''%%--enable-automation%%'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.))
-  * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''--headless'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
+  * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''%%--headless%%'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
   * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling "undetectable" traffic, over half a million requests. **This is the 2025 state of the art and it is an inconsistency detector.**
   * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for.
@@ Line 165: / Line 165: @@
   - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table.
-  - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits.
+  - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. There is no public, curated, maintained set of challenge markers to start from — we looked and did not find one, which is why this is an open question below. The starting point we used for this page's own sweep is the ''bot_challenge_page'' pattern in ''botdet_fold.mjs'': //Just a moment//, //Attention Required//, //Are you a human//, //checking your browser//, //human verification//, //verify you are//, //I'm not a robot//. That is a beginning, not a solution.
   - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta.
   - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means.
@@ Line 171: / Line 171: @@
 ==== The tooling, dated — and why it is not a plan ====
-The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth
+The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth patch
-patch, 9 of them ran a crawl, and none is earlier than 2021.** Anyone following that
+among their tools, 9 of them ran a crawl, and none is earlier than 2021.** (The
+signal-mention table further down says 15; that one counts any paper whose text mentions
+such a patch at all, including in related work.) Anyone following that
 trail today should know the state of what it points at, because the names in those
 papers have moved:
@@ Line 179: / Line 181: @@
 | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. |
 | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. |
-| ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. AGPL-3.0. |
+| ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. |
-| ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright, drop-in. The actively maintained option at the time of writing. Apache-2.0. |
+| ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright. The only row still receiving commits in the week this table was built. |
 | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. |
 | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. |
 | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. |
-| Camoufox | repo pushed 2026-08-26, MPL-2.0 | A patched Firefox rather than a patch on top of a driver. |
+| Camoufox | repo pushed 2026-08-26 | A patched Firefox rather than a patch on top of a driver. |
 ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.))
@@ Line 206: / Line 208: @@
 | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. |
 | Akamai Bot Manager | Current. |
-| DataDome, Kasada, Arkose Labs | Current and independent. |
+| DataDome, Kasada, Arkose Labs | Current; **no acquisition found** as of 2026-08-29 — which is an absence of evidence rather than a company statement, and weaker than the rows below. |
 | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. |
 | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. |
@@ Line 213: / Line 215: @@
 | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. |
-((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.))
+((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder — refetching a challenged page from a second client is a mild version of the differential probing this page tells you to treat as a decision, and reading a press release is not a measurement of the operator. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.))
 ===== The AI-Crawler Turn, 2025–2026 =====
@@ Line 224: / Line 226: @@
   * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your sample sits behind Cloudflare, that default is part of your methodology whether you engaged with it or not.
-  * The corpus sees this arriving and only just: exactly **five papers** in 5,859 mention an AI crawler or name an AI crawler user-agent token, and **all five are 2025 or 2026**. Liu et al. {[liu2025_somesite]} measured how well content creators can actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether scrapers respect its directives at all.
+  * The corpus sees this arriving and only just: exactly **three papers** in 5,859 name an AI-crawler user-agent token, and **all three are 2025**. (An earlier version of this sentence said five, from a wider pattern that also matches "AI scraper" and "agentic browser"; two of those five turned out to be papers about LLM-driven agents, one of them matching only inside its reference list. Both widths are printed in the report script on [[provenance:programming:crawler_detection]].) Liu et al. {[liu2025_somesite]} measured how well content creators can actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether scrapers respect its directives at all.
   * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it.
   * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you.
-  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
+  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish the replaced individual submission from the active WG work: ''draft-meunier-web-bot-auth-architecture'' is marked **Replaced** (state id 4), and the ''replaces'' relation points to ''draft-meunier-webbotauth-httpsig-protocol-02''. Datatracker now marks that intermediate draft **Replaced** (with an "Expired & archived" badge) by the current WG draft ''draft-ietf-webbotauth-httpsig-protocol''; the webbotauth working group remains chartered and active, but the individual drafts have no formal standing — checked 2026-09-17.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
+==== Measuring who crawls you ====
+Everything above is written from inside the crawler. This is the mirror question,
+and since 2025 it is a measurement topic in its own right: **who crawls your
+site, and does anything you publish change what they do?** Two things get run
+together here and they are not the same measurement.
+  * **A directive is a claim.** ''robots.txt'', a ''NoAI'' meta tag, Cloudflare's Content Signals fields, a line in your terms of service — these are published files, so they can be measured at web scale from the client side, exactly like ''ads.txt'' or ''security.txt''. Cui et al. {[cui2025_odyssey]} did that: //"we present a systematic analysis of 18 LLM bots on 582,281 robots.txt files"//, finding LLM-bot rules proliferating fastest in finance and news. That is a measurement of what publishers //ask// for.
+  * **Compliance is a different measurement, and only the operator can make it.** Whether a particular crawler obeyed a ''Disallow'' is visible in one place: the server's own request log. There is no client-side proxy for it, no top-list you can crawl to find out, and consequently **the population you can study is the set of sites you control or have log access to.** That single constraint shapes every result below.
+So the field has exactly two designs, and both predate the AI question:
+^ Design ^ Ground truth ^ In this corpus ^
+| **Honeysites** — stand up sites whose only possible visitors are bots, instrument them, wait | your own logs, with no human traffic to separate out | Aristaeus {[li2021_good]}: //"a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses"//, with ''robots.txt'' honouring as one behavioural signal and ''Disallow'' entries whose path encodes the requesting IP. Built in 2021 to study malicious bots, not AI crawlers — and it is the same design Liu et al. use four years later, who cite it |
+| **An operator's logs** — get access to a real site's logs and vary the directive under them | the operator's logs, with real traffic mixed in and to be filtered out | Kim et al. {[kim2025_scrapers]}: //"We analyze the behavior of 130 self-declared bots (and many anonymous ones) over 40 days"//, from the anonymised web logs of their own institution, under a series of controlled ''robots.txt'' experiments |
+=== The literature does not yet answer whether the directives are honoured ===
+<WRAP important>
+Two IMC 2025 papers measured this in the same year and point in different
+directions, and **the disagreement is the finding, not an error in either.**
+  * Liu et al. {[liu2025_somesite]}, **passive**: nine AI crawlers arrived uninvited at sites the authors control over six months. //"Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, and OAI-SearchBot) respected the robots.txt file"//. The other two failed in different ways, and the difference matters: //"One crawler (Bytespider) fetched the robots.txt file but did not respect it"//, while ChatGPT-User //"visited our website once and did not fetch the robots.txt file"// — which the authors flag as unexplained, since it is a user-triggered crawler and they had not triggered it.
+  * Liu et al., **active** — the authors asked AI assistants to visit a URL and watched who turned up. //"This process yields 23 distinct third-party AI assistant crawlers"//, of which //"the remaining 20 crawlers did not fetch the robots.txt file at all"//.
+  * Kim et al. {[kim2025_scrapers]}, from an institution's logs: //"AI-related bots (AI Assistants and AI search crawlers) have the lowest re-check rates with less than 40% of bots checking robots.txt within a 168 hour window"// — so even a directive that would be obeyed may never be read.
+The reconciliation is that **"do AI crawlers respect ''robots.txt''" is not one
+question.** Vendor-operated //training// crawlers largely do. //User-triggered//
+fetchers — the request an assistant makes because someone pasted your URL into
+it — largely never look, and they are a different population with a different
+user agent, sometimes not the one the vendor documents. A result here that does
+not say which of the two it observed, and whether the traffic was solicited, is
+not interpretable.
+</WRAP>
+Nobody has published the measurement that would settle it: **compliance against a
+population of sites the authors do not operate.** It cannot be done from the
+client side, and the alternative — asking operators at scale for logs — has not
+been tried in these seven venues.
+**One thing has changed since the section above was written, and it is not a
+measurement.** Cloudflare's own documentation now states that the "signed agent"
+classification was **retired on 2026-07-01** and folded into verified bots labelled
+//intermediary//, and that a bot earns verification by two tests: cryptographic Web Bot
+Auth self-identification, and //"Non-abusive behavior — it obeys ''robots.txt'' and crawl
+directives, maintains reasonable request rates"//.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-09-16. This is a vendor documenting its own product: it establishes that the mechanism is deployed and what it requires, and it is not evidence about how any crawler behaves.)) So "declare yourself
+and be allowed in" is already running in production on one large CDN, against an IETF
+draft rather than an RFC — which makes it a thing to state in your methodology if your
+crawler is signed or unsigned, and still not a thing to cite as a finding.
+=== What this corpus can and cannot tell you here ===
+**The corpus lags this topic by about two years, and you should read outside it.**
+Three papers in 5,859 name an AI-crawler user agent (''GPTBot'', ''ClaudeBot'',
+''Bytespider'' and the rest), and all three are 2025: the three cited above.
+Widening to any mention of ''robots.txt'' at all gives **74 papers**, but **44 of
+those mention it exactly once**, and a single mention is an aside by construction: a
+crawl-policy declaration ("we obeyed it", "we did not, and here is why"), a caveat about
+what the Internet Archive did not capture because of it, an analogy for ''ads.txt'' or
+''security.txt'', or a line in a reference list. **None of the 44 measures anything about
+''robots.txt''** — a paper that did would not mention it once. A coder over the sentence
+around each mention places 23 of the 44 in three named classes and leaves **21
+uncoded**; the residue is printed in full, with its contexts, on
+[[provenance:programming:crawler_detection]], and no share of the 44 is published here
+because the coder cannot support one. Only **five** papers
+mention it ten or more times: the three above, Aristaeus {[li2021_good]}, and a
+''security.txt'' deployment study that uses it as a comparison. **No paper in the
+corpus mentions ''llms.txt''.** Counts are full-text sweeps over the 5,855 papers
+with text on disk, and a sweep hit is a candidate rather than a claim; the
+queries, the coding of those 44 and its uncoded residue are on
+[[provenance:programming:crawler_detection]].
+<WRAP todo>
+If you are running one of these measurements:
+  * **Say which side you measured.** Directive prevalence and compliance are different claims with different populations; papers get read as if a large number on the first implied anything about the second.
+  * **Report per-user-agent, not per-category.** "AI crawlers" aggregates a training crawler that obeys and an assistant fetcher that never fetches the file, and the aggregate mean is a number about your sample's mix.
+  * **Report whether the traffic was solicited.** Passive arrival and "we asked an assistant to fetch this URL" are different experiments; Liu et al. report them separately for good reason.
+  * **Publish the log schema and the retention**, not the logs. This is human traffic data with an ethics review attached; both 2025 papers work from anonymised logs, and the anonymisation is part of the method.
+  * **Do not treat a vendor's published crawler list or blocking statistic as a finding.** It is a claim by an interested party about its own product, usable for dates and documented behaviour — which is how the footnotes above use Cloudflare — and not usable as a measurement of compliance.
+</WRAP>
 An LLM-driven agent browsing on a user's behalf is a third category again, detected by
@@ Line 246: / Line 330: @@
 //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it.
 The population here is a full-text sweep over the papers' own sentences, and **every one
-of its 138 candidates was then read by hand**, because the sweep on its own is only 31.2%
+of its 138 candidates was then audited individually against that paper's full text**,
-precise for the claim it is used to make.
+because the sweep on its own is only 31.2% precise for the claim it is used to make. Who
+did that auditing, and what it does and does not establish, is on
+[[provenance:programming:crawler_detection]].
 ==== Almost nobody reports being blocked ====
@@ Line 335: / Line 421: @@
 | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** |
 | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% |
-| What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% |
+| Consent-action field populated (schema; not an audited paper claim) | ''crawlConfig.consentAction'' | 349 | 31.2% |
 | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% |
 | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% |
 | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% |
+The consent row reports only that the extractor populated a non-sentinel field; it is not a paper-claim count. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** stating a consent action in full text, while **279 of 313 (89.1%)** ''no-interaction'' labels were unsupported. This page keeps 349 because the table is about schema reporting completeness.
 **Headless versus headful is the most detectability-relevant single bit in the whole
@@ Line 366: / Line 454: @@
 ,859 mention CDP artefacts as a detection surface, and three mention header-order
 fingerprinting** — both of which are ordinary, documented techniques on the commercial
-side. All five AI-crawler papers are 2025 or later.
+side. The ''AI or LLM crawler'' row's five papers are all 2025 or later, but only
+**three** of them are about AI crawlers — the row's pattern also matches "AI scraper" and
+"agentic browser", and two members are LLM-agent papers. See //Measuring who crawls you//
+above.
 ==== Methodology and limitations of these figures ====
@@ Line 373: / Line 464: @@
   * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
   * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
-  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
+  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Of those 13, **11 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text, not in the extraction — and **2** are below threshold in both renderings. **Below threshold is not "unsupported"** — all 13 were read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
   * **Papers are counted, never tuples**, and a paper counts once.
   * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.
@@ Line 408: / Line 499: @@
   * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured.
   * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it.
+  * **There is no shared challenge-marker set, and there should be.** Every paper that detects challenge pages writes its own string list, none publishes it as a maintained artefact, and none reports its precision. A curated, versioned list — with per-vendor markers and a recall figure — is a small artefact that would make every non-load table on this page comparable with every other.
   * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus.
 </WRAP>