Differences

This shows you the differences between two versions of the page.

--- programming:crawler_detection [2026/09/16 21:08] – Add 'Measuring who crawls you' to the AI-crawler section: directive vs compliance, honeysite and operator-log designs, the two IMC 2025 papers that disagree, corpus thinness. Correct 'five papers' to three (two were agentic-browser false positives). Autho karel.kubicek.claude
+++ programming:crawler_detection [2026/09/21 13:42] (current) – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude
@@ Line 93: / Line 93: @@
   * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: **11 papers in the whole corpus name it.**
-  * **Chrome still ships the automation switch.** ''--enable-automation'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.))
+  * **Chrome still ships the automation switch.** ''%%--enable-automation%%'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.))
-  * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''--headless'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
+  * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''%%--headless%%'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
   * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling "undetectable" traffic, over half a million requests. **This is the 2025 state of the art and it is an inconsistency detector.**
   * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for.
@@ Line 229: / Line 229: @@
   * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it.
   * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you.
-  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
+  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish the replaced individual submission from the active WG work: ''draft-meunier-web-bot-auth-architecture'' is marked **Replaced** (state id 4), and the ''replaces'' relation points to ''draft-meunier-webbotauth-httpsig-protocol-02''. Datatracker now marks that intermediate draft **Replaced** (with an "Expired & archived" badge) by the current WG draft ''draft-ietf-webbotauth-httpsig-protocol''; the webbotauth working group remains chartered and active, but the individual drafts have no formal standing — checked 2026-09-17.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
 ==== Measuring who crawls you ====
@@ Line 244: / Line 244: @@
 ^ Design ^ Ground truth ^ In this corpus ^
-| **Honeysites** — stand up sites whose only possible visitors are bots, instrument them, wait | your own logs, with no human traffic to separate out | Aristaeus {[li2021_good]}: //"a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses"//, with ''robots.txt'' honouring as one behavioural signal and ''Disallow'' entries whose path encodes the requesting IP. Built to study malicious bots in 2021; it is the template the AI-crawler papers reuse |
+| **Honeysites** — stand up sites whose only possible visitors are bots, instrument them, wait | your own logs, with no human traffic to separate out | Aristaeus {[li2021_good]}: //"a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses"//, with ''robots.txt'' honouring as one behavioural signal and ''Disallow'' entries whose path encodes the requesting IP. Built in 2021 to study malicious bots, not AI crawlers — and it is the same design Liu et al. use four years later, who cite it |
 | **An operator's logs** — get access to a real site's logs and vary the directive under them | the operator's logs, with real traffic mixed in and to be filtered out | Kim et al. {[kim2025_scrapers]}: //"We analyze the behavior of 130 self-declared bots (and many anonymous ones) over 40 days"//, from the anonymised web logs of their own institution, under a series of controlled ''robots.txt'' experiments |
@@ Line 253: / Line 253: @@
 directions, and **the disagreement is the finding, not an error in either.**
-  * Liu et al. {[liu2025_somesite]}, **passive**: nine AI crawlers arrived uninvited at sites the authors control over six months. //"Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, and OAI-SearchBot) respected the robots.txt file"//. Only Bytespider did not.
+  * Liu et al. {[liu2025_somesite]}, **passive**: nine AI crawlers arrived uninvited at sites the authors control over six months. //"Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, and OAI-SearchBot) respected the robots.txt file"//. The other two failed in different ways, and the difference matters: //"One crawler (Bytespider) fetched the robots.txt file but did not respect it"//, while ChatGPT-User //"visited our website once and did not fetch the robots.txt file"// — which the authors flag as unexplained, since it is a user-triggered crawler and they had not triggered it.
   * Liu et al., **active** — the authors asked AI assistants to visit a URL and watched who turned up. //"This process yields 23 distinct third-party AI assistant crawlers"//, of which //"the remaining 20 crawlers did not fetch the robots.txt file at all"//.
   * Kim et al. {[kim2025_scrapers]}, from an institution's logs: //"AI-related bots (AI Assistants and AI search crawlers) have the lowest re-check rates with less than 40% of bots checking robots.txt within a 168 hour window"// — so even a directive that would be obeyed may never be read.
@@ Line 270: / Line 270: @@
 client side, and the alternative — asking operators at scale for logs — has not
 been tried in these seven venues.
+**One thing has changed since the section above was written, and it is not a
+measurement.** Cloudflare's own documentation now states that the "signed agent"
+classification was **retired on 2026-07-01** and folded into verified bots labelled
+//intermediary//, and that a bot earns verification by two tests: cryptographic Web Bot
+Auth self-identification, and //"Non-abusive behavior — it obeys ''robots.txt'' and crawl
+directives, maintains reasonable request rates"//.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-09-16. This is a vendor documenting its own product: it establishes that the mechanism is deployed and what it requires, and it is not evidence about how any crawler behaves.)) So "declare yourself
+and be allowed in" is already running in production on one large CDN, against an IETF
+draft rather than an RFC — which makes it a thing to state in your methodology if your
+crawler is signed or unsigned, and still not a thing to cite as a finding.
 === What this corpus can and cannot tell you here ===
@@ Line 277: / Line 287: @@
 ''Bytespider'' and the rest), and all three are 2025: the three cited above.
 Widening to any mention of ''robots.txt'' at all gives **74 papers**, but **44 of
-those mention it exactly once**, and reading those mentions they are crawl-policy
+those mention it exactly once**, and a single mention is an aside by construction: a
-declarations — "we obeyed it", "we did not, and here is why" — or caveats about
+crawl-policy declaration ("we obeyed it", "we did not, and here is why"), a caveat about
-what the Internet Archive did not capture because of it. Only **five** papers
+what the Internet Archive did not capture because of it, an analogy for ''ads.txt'' or
+''security.txt'', or a line in a reference list. **None of the 44 measures anything about
+''robots.txt''** — a paper that did would not mention it once. A coder over the sentence
+around each mention places 23 of the 44 in three named classes and leaves **21
+uncoded**; the residue is printed in full, with its contexts, on
+[[provenance:programming:crawler_detection]], and no share of the 44 is published here
+because the coder cannot support one. Only **five** papers
 mention it ten or more times: the three above, Aristaeus {[li2021_good]}, and a
 ''security.txt'' deployment study that uses it as a comparison. **No paper in the
@@ Line 405: / Line 421: @@
 | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** |
 | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% |
-| What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% |
+| Consent-action field populated (schema; not an audited paper claim) | ''crawlConfig.consentAction'' | 349 | 31.2% |
 | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% |
 | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% |
 | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% |
+The consent row reports only that the extractor populated a non-sentinel field; it is not a paper-claim count. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** stating a consent action in full text, while **279 of 313 (89.1%)** ''no-interaction'' labels were unsupported. This page keeps 349 because the table is about schema reporting completeness.
 **Headless versus headful is the most detectability-relevant single bit in the whole
@@ Line 436: / Line 454: @@
 ,859 mention CDP artefacts as a detection surface, and three mention header-order
 fingerprinting** — both of which are ordinary, documented techniques on the commercial
-side. All five AI-crawler papers are 2025 or later.
+side. The ''AI or LLM crawler'' row's five papers are all 2025 or later, but only
+**three** of them are about AI crawlers — the row's pattern also matches "AI scraper" and
+"agentic browser", and two members are LLM-agent papers. See //Measuring who crawls you//
+above.
 ==== Methodology and limitations of these figures ====
@@ Line 443: / Line 464: @@
   * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
   * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
-  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
+  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Of those 13, **11 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text, not in the extraction — and **2** are below threshold in both renderings. **Below threshold is not "unsupported"** — all 13 were read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
   * **Papers are counted, never tuples**, and a paper counts once.
   * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.