| Next revision | Previous revision |
| programming:crawler_detection [2026/08/29 01:37] – New page: bot management, headless/CDP leaks, CAPTCHAs and WAF challenge pages, and what a blocked crawl does to the headline number. Population is a hand-audited full-text sweep (138 candidates, 31.2% precise); 43 of 1,120 crawling papers report their ow karel.kubicek.claude | programming:crawler_detection [2026/09/21 13:42] (current) – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude |
|---|
| |
| That last step is an **inference, not a measurement**, and this page is careful about | That last step is an **inference, not a measurement**, and this page is careful about |
| saying so: no paper in this corpus reports a block rate broken down by popularity rank. | saying so: no paper in this corpus reports a block rate broken down by rank decile. The closest is DeBlasio et al. {[deblasio2017_tripwire]}, who split one figure into "all sites" and "top 100" and found the challenge rate roughly twice as high in the top 100; one two-stratum split, pointing the right way, is not a distribution. |
| It follows from where the products are sold and it is consistent with the shape of the | It follows from where the products are sold and it is consistent with the shape of the |
| losses individual papers report, and it is the first of the open questions below. What | losses individual papers report, and it is the first of the open questions below. What |
| ^ Shape ^ What you get ^ Why it ranks here ^ | ^ Shape ^ What you get ^ Why it ranks here ^ |
| | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | | | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | |
| | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page is HTTP 200 with a full DOM, so a crawler that only checks the status code records it as a successful visit. | | | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page can be HTTP 200 with a full DOM — Cloudflare has also used 403 and 503 for these — and the 200 case is the dangerous one, because a crawler that only checks the status code records it as a successful visit. | |
| | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | | | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | |
| | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | | | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | |
| This is the layer the literature has actually studied, and the layer that has moved most. | This is the layer the literature has actually studied, and the layer that has moved most. |
| |
| * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been | * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: **11 papers in the whole corpus name it.** |
| a W3C Recommendation since 05 June 2018 and defines the property as returning | * **Chrome still ships the automation switch.** ''%%--enable-automation%%'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.)) |
| ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a | * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''%%--headless%%'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all. |
| Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any | * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling "undetectable" traffic, over half a million requests. **This is the 2025 state of the art and it is an inconsistency detector.** |
| Selenium, Puppeteer or Playwright session sets it by default. It is the single | * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for. |
| cheapest automation check in existence and the field barely discusses it: **11 | |
| papers in the whole corpus name it.** | |
| * **Chrome still ships the automation switch.** ''--enable-automation'' and the | |
| "controlled by automated test software" infobar are both present in the Chromium | |
| tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.)) | |
| * **"Headless" no longer means one thing.** Since Chrome 132 the old headless | |
| implementation is available only as a separate ''chrome-headless-shell'' binary; | |
| plain ''--headless'' now selects a unified mode that shares its code with headful | |
| Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate | |
| headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an | |
| ambiguous sentence**, and which binary you meant changes how detectable you were. | |
| Only 12.5% of crawling papers say headless or headful at all. | |
| * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. | |
| {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed | |
| browser is identified not by any one attribute but by attributes that contradict | |
| each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at | |
| billion-request scale on commercial sites, and Venugopalan et al. | |
| {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey | |
| site behind two commercial anti-bot services (DataDome and BotD), fed by 20 | |
| commercial bot services selling "undetectable" traffic, over half a million | |
| requests. **This is the 2025 state of the art and it is an inconsistency detector.** | |
| * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} | |
| studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is | |
| deployed. A crawler that many papers share is worth writing a signature for. | |
| |
| ==== What is dated, and what is current ==== | ==== What is dated, and what is current ==== |
| to make the obstruction a measured quantity. Four reasons the second is better: | to make the obstruction a measured quantity. Four reasons the second is better: |
| |
| - **The arms race is unwinnable and, worse, silent.** You will not be told when a | - **The arms race is unwinnable and, worse, silent.** You will not be told when a patch stops working. A result that depends on evasion has no error bar and no way to acquire one. |
| patch stops working. A result that depends on evasion has no error bar and no way | - **It is an ethics decision.** Deliberately overriding an operator's expressed access preference needs a justification you are prepared to write in the paper. See [[Practices:Ethics]]. |
| to acquire one. | - **The obstruction is itself publishable.** Every quantified figure in the table below is a finding somebody got for free by counting what they lost. |
| - **It is an ethics decision.** Deliberately overriding an operator's expressed access | - **It is the cheapest possible defence against the obvious objection.** This page's own figures show the bar has been low — 96.2% of crawling papers report no obstruction at all — but a reviewer who asks "how do you know the sites you lost were lost at random?" is asking something you cannot answer after the crawl is over. The breakdown costs one table and has to be collected while you crawl. |
| preference needs a justification you are prepared to write in the paper. See | |
| [[Practices:Ethics]]. | |
| - **The obstruction is itself publishable.** Every quantified figure in the table | |
| below is a finding somebody got for free by counting what they lost. | |
| - **It is the cheapest possible defence against the obvious objection.** This page's | |
| own figures show the bar has been low — 96.2% of crawling papers report no | |
| obstruction at all — but a reviewer who asks "how do you know the sites you lost were | |
| lost at random?" is asking something you cannot answer after the crawl is over. The | |
| breakdown costs one table and has to be collected while you crawl. | |
| |
| The minimum viable version costs almost nothing: | The minimum viable version costs almost nothing: |
| |
| - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, | - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. |
| //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. | - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. There is no public, curated, maintained set of challenge markers to start from — we looked and did not find one, which is why this is an open question below. The starting point we used for this page's own sweep is the ''bot_challenge_page'' pattern in ''botdet_fold.mjs'': //Just a moment//, //Attention Required//, //Are you a human//, //checking your browser//, //human verification//, //verify you are//, //I'm not a robot//. That is a beginning, not a solution. |
| - Add a **challenge-page detector** to the success path, not just to the error path, | - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta. |
| so HTTP 200 challenges stop counting as visits. | - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means. |
| - Run a **control condition**: a small sample crawled under a second configuration, | |
| same day, same URLs. Report the delta. | |
| - Say in the paper **whether you attempted evasion, and what you did**. "We made no | |
| attempt to evade bot detection" is a perfectly good sentence and it tells the reader | |
| what the number means. | |
| |
| ==== The tooling, dated — and why it is not a plan ==== | ==== The tooling, dated — and why it is not a plan ==== |
| |
| The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth | The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth patch |
| patch, 9 of them ran a crawl, and none is earlier than 2021.** Anyone following that | among their tools, 9 of them ran a crawl, and none is earlier than 2021.** (The |
| | signal-mention table further down says 15; that one counts any paper whose text mentions |
| | such a patch at all, including in related work.) Anyone following that |
| trail today should know the state of what it points at, because the names in those | trail today should know the state of what it points at, because the names in those |
| papers have moved: | papers have moved: |
| | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | | | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | |
| | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | | | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | |
| | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. AGPL-3.0. | | | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. | |
| | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright, drop-in. The actively maintained option at the time of writing. Apache-2.0. | | | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright. The only row still receiving commits in the week this table was built. | |
| | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | | | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | |
| | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | | | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | |
| | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | | | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | |
| | Camoufox | repo pushed 2026-08-26, MPL-2.0 | A patched Firefox rather than a patch on top of a driver. | | | Camoufox | repo pushed 2026-08-26 | A patched Firefox rather than a patch on top of a driver. | |
| |
| ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) | ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) |
| | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | | | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | |
| | Akamai Bot Manager | Current. | | | Akamai Bot Manager | Current. | |
| | DataDome, Kasada, Arkose Labs | Current and independent. | | | DataDome, Kasada, Arkose Labs | Current; **no acquisition found** as of 2026-08-29 — which is an absence of evidence rather than a company statement, and weaker than the rows below. | |
| | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | | | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | |
| | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | | | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | |
| | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | | | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | |
| |
| ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) | ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder — refetching a challenged page from a second client is a mild version of the differential probing this page tells you to treat as a decision, and reading a press release is not a measurement of the operator. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) |
| |
| ===== The AI-Crawler Turn, 2025–2026 ===== | ===== The AI-Crawler Turn, 2025–2026 ===== |
| distinguish you from them.** | distinguish you from them.** |
| |
| * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they | * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your sample sits behind Cloudflare, that default is part of your methodology whether you engaged with it or not. |
| pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your | * The corpus sees this arriving and only just: exactly **three papers** in 5,859 name an AI-crawler user-agent token, and **all three are 2025**. (An earlier version of this sentence said five, from a wider pattern that also matches "AI scraper" and "agentic browser"; two of those five turned out to be papers about LLM-driven agents, one of them matching only inside its reference list. Both widths are printed in the report script on [[provenance:programming:crawler_detection]].) Liu et al. {[liu2025_somesite]} measured how well content creators can actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether scrapers respect its directives at all. |
| sample sits behind Cloudflare, that default is part of your methodology whether you | * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it. |
| engaged with it or not. | * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you. |
| * The corpus sees this arriving and only just: exactly **five papers** in 5,859 | * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish the replaced individual submission from the active WG work: ''draft-meunier-web-bot-auth-architecture'' is marked **Replaced** (state id 4), and the ''replaces'' relation points to ''draft-meunier-webbotauth-httpsig-protocol-02''. Datatracker now marks that intermediate draft **Replaced** (with an "Expired & archived" badge) by the current WG draft ''draft-ietf-webbotauth-httpsig-protocol''; the webbotauth working group remains chartered and active, but the individual drafts have no formal standing — checked 2026-09-17.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting. |
| mention an AI crawler or name an AI crawler user-agent token, and **all five are | |
| 2025 or 2026**. Liu et al. {[liu2025_somesite]} measured how well content creators can | ==== Measuring who crawls you ==== |
| actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM | |
| services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether | Everything above is written from inside the crawler. This is the mirror question, |
| scrapers respect its directives at all. | and since 2025 it is a measurement topic in its own right: **who crawls your |
| * **And the defaults are still moving.** Cloudflare has since replaced the single | site, and does anything you publish change what they do?** Two things get run |
| "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and | together here and they are not the same measurement. |
| **Training** — and its own documentation states that from **2026-09-15** new domains | |
| default to blocking Training //and Agent// on pages that display ads, while Search | * **A directive is a claim.** ''robots.txt'', a ''NoAI'' meta tag, Cloudflare's Content Signals fields, a line in your terms of service — these are published files, so they can be measured at web scale from the client side, exactly like ''ads.txt'' or ''security.txt''. Cui et al. {[cui2025_odyssey]} did that: //"we present a systematic analysis of 18 LLM bots on 582,281 robots.txt files"//, finding LLM-bot rules proliferating fastest in finance and news. That is a measurement of what publishers //ask// for. |
| stays allowed; the old "Block AI bots" option is marked deprecating on the same | * **Compliance is a different measurement, and only the operator can make it.** Whether a particular crawler obeyed a ''Disallow'' is visible in one place: the server's own request log. There is no client-side proxy for it, no top-list you can crawl to find out, and consequently **the population you can study is the set of sites you control or have log access to.** That single constraint shapes every result below. |
| date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that | |
| date.** If you are reading it later, check the current defaults yourself: an | So the field has exactly two designs, and both predate the AI question: |
| academic crawler that a classifier puts in //Agent// is on the blocked side of the | |
| new default, and nothing about being academic exempts it. | ^ Design ^ Ground truth ^ In this corpus ^ |
| * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and | | **Honeysites** — stand up sites whose only possible visitors are bots, instrument them, wait | your own logs, with no human traffic to separate out | Aristaeus {[li2021_good]}: //"a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses"//, with ''robots.txt'' honouring as one behavioural signal and ''Disallow'' entries whose path encodes the requesting IP. Built in 2021 to study malicious bots, not AI crawlers — and it is the same design Liu et al. use four years later, who cite it | |
| nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not | | **An operator's logs** — get access to a real site's logs and vary the directive under them | the operator's logs, with real traffic mixed in and to be filtered out | Kim et al. {[kim2025_scrapers]}: //"We analyze the behavior of 130 self-declared bots (and many anonymous ones) over 40 days"//, from the anonymised web logs of their own institution, under a series of controlled ''robots.txt'' experiments | |
| enforce one, and it is not what blocks you. | |
| * Two IETF working groups are now chartered in this space and are worth watching | === The literature does not yet answer whether the directives are honoured === |
| rather than citing: **webbotauth**, on cryptographically authenticating automated | |
| clients and identifying their operators, and **aipref**, on a vocabulary and | <WRAP important> |
| attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity | Two IMC 2025 papers measured this in the same year and point in different |
| lands, "declare yourself and be allowed in" becomes a real option for research | directions, and **the disagreement is the finding, not an error in either.** |
| crawlers, and this page will need rewriting. | |
| | * Liu et al. {[liu2025_somesite]}, **passive**: nine AI crawlers arrived uninvited at sites the authors control over six months. //"Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, and OAI-SearchBot) respected the robots.txt file"//. The other two failed in different ways, and the difference matters: //"One crawler (Bytespider) fetched the robots.txt file but did not respect it"//, while ChatGPT-User //"visited our website once and did not fetch the robots.txt file"// — which the authors flag as unexplained, since it is a user-triggered crawler and they had not triggered it. |
| | * Liu et al., **active** — the authors asked AI assistants to visit a URL and watched who turned up. //"This process yields 23 distinct third-party AI assistant crawlers"//, of which //"the remaining 20 crawlers did not fetch the robots.txt file at all"//. |
| | * Kim et al. {[kim2025_scrapers]}, from an institution's logs: //"AI-related bots (AI Assistants and AI search crawlers) have the lowest re-check rates with less than 40% of bots checking robots.txt within a 168 hour window"// — so even a directive that would be obeyed may never be read. |
| | |
| | The reconciliation is that **"do AI crawlers respect ''robots.txt''" is not one |
| | question.** Vendor-operated //training// crawlers largely do. //User-triggered// |
| | fetchers — the request an assistant makes because someone pasted your URL into |
| | it — largely never look, and they are a different population with a different |
| | user agent, sometimes not the one the vendor documents. A result here that does |
| | not say which of the two it observed, and whether the traffic was solicited, is |
| | not interpretable. |
| | </WRAP> |
| | |
| | Nobody has published the measurement that would settle it: **compliance against a |
| | population of sites the authors do not operate.** It cannot be done from the |
| | client side, and the alternative — asking operators at scale for logs — has not |
| | been tried in these seven venues. |
| | |
| | **One thing has changed since the section above was written, and it is not a |
| | measurement.** Cloudflare's own documentation now states that the "signed agent" |
| | classification was **retired on 2026-07-01** and folded into verified bots labelled |
| | //intermediary//, and that a bot earns verification by two tests: cryptographic Web Bot |
| | Auth self-identification, and //"Non-abusive behavior — it obeys ''robots.txt'' and crawl |
| | directives, maintains reasonable request rates"//.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-09-16. This is a vendor documenting its own product: it establishes that the mechanism is deployed and what it requires, and it is not evidence about how any crawler behaves.)) So "declare yourself |
| | and be allowed in" is already running in production on one large CDN, against an IETF |
| | draft rather than an RFC — which makes it a thing to state in your methodology if your |
| | crawler is signed or unsigned, and still not a thing to cite as a finding. |
| | |
| | === What this corpus can and cannot tell you here === |
| | |
| | **The corpus lags this topic by about two years, and you should read outside it.** |
| | Three papers in 5,859 name an AI-crawler user agent (''GPTBot'', ''ClaudeBot'', |
| | ''Bytespider'' and the rest), and all three are 2025: the three cited above. |
| | Widening to any mention of ''robots.txt'' at all gives **74 papers**, but **44 of |
| | those mention it exactly once**, and a single mention is an aside by construction: a |
| | crawl-policy declaration ("we obeyed it", "we did not, and here is why"), a caveat about |
| | what the Internet Archive did not capture because of it, an analogy for ''ads.txt'' or |
| | ''security.txt'', or a line in a reference list. **None of the 44 measures anything about |
| | ''robots.txt''** — a paper that did would not mention it once. A coder over the sentence |
| | around each mention places 23 of the 44 in three named classes and leaves **21 |
| | uncoded**; the residue is printed in full, with its contexts, on |
| | [[provenance:programming:crawler_detection]], and no share of the 44 is published here |
| | because the coder cannot support one. Only **five** papers |
| | mention it ten or more times: the three above, Aristaeus {[li2021_good]}, and a |
| | ''security.txt'' deployment study that uses it as a comparison. **No paper in the |
| | corpus mentions ''llms.txt''.** Counts are full-text sweeps over the 5,855 papers |
| | with text on disk, and a sweep hit is a candidate rather than a claim; the |
| | queries, the coding of those 44 and its uncoded residue are on |
| | [[provenance:programming:crawler_detection]]. |
| | |
| | <WRAP todo> |
| | If you are running one of these measurements: |
| | |
| | * **Say which side you measured.** Directive prevalence and compliance are different claims with different populations; papers get read as if a large number on the first implied anything about the second. |
| | * **Report per-user-agent, not per-category.** "AI crawlers" aggregates a training crawler that obeys and an assistant fetcher that never fetches the file, and the aggregate mean is a number about your sample's mix. |
| | * **Report whether the traffic was solicited.** Passive arrival and "we asked an assistant to fetch this URL" are different experiments; Liu et al. report them separately for good reason. |
| | * **Publish the log schema and the retention**, not the logs. This is human traffic data with an ethics review attached; both 2025 papers work from anonymised logs, and the anonymisation is part of the method. |
| | * **Do not treat a vendor's published crawler list or blocking statistic as a finding.** It is a claim by an interested party about its own product, usable for dates and documented behaviour — which is how the footnotes above use Cloudflare — and not usable as a measurement of compliance. |
| | </WRAP> |
| |
| An LLM-driven agent browsing on a user's behalf is a third category again, detected by | An LLM-driven agent browsing on a user's behalf is a third category again, detected by |
| //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. | //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. |
| The population here is a full-text sweep over the papers' own sentences, and **every one | The population here is a full-text sweep over the papers' own sentences, and **every one |
| of its 138 candidates was then read by hand**, because the sweep on its own is only 31.2% | of its 138 candidates was then audited individually against that paper's full text**, |
| precise for the claim it is used to make. | because the sweep on its own is only 31.2% precise for the claim it is used to make. Who |
| | did that auditing, and what it does and does not establish, is on |
| | [[provenance:programming:crawler_detection]]. |
| |
| ==== Almost nobody reports being blocked ==== | ==== Almost nobody reports being blocked ==== |
| | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** | | | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** | |
| | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% | | | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% | |
| | What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% | | | Consent-action field populated (schema; not an audited paper claim) | ''crawlConfig.consentAction'' | 349 | 31.2% | |
| | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% | | | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% | |
| | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% | | | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% | |
| | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% | | | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% | |
| | |
| | The consent row reports only that the extractor populated a non-sentinel field; it is not a paper-claim count. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** stating a consent action in full text, while **279 of 313 (89.1%)** ''no-interaction'' labels were unsupported. This page keeps 349 because the table is about schema reporting completeness. |
| |
| **Headless versus headful is the most detectability-relevant single bit in the whole | **Headless versus headful is the most detectability-relevant single bit in the whole |
| 5,859 mention CDP artefacts as a detection surface, and three mention header-order | 5,859 mention CDP artefacts as a detection surface, and three mention header-order |
| fingerprinting** — both of which are ordinary, documented techniques on the commercial | fingerprinting** — both of which are ordinary, documented techniques on the commercial |
| side. All five AI-crawler papers are 2025 or later. | side. The ''AI or LLM crawler'' row's five papers are all 2025 or later, but only |
| | **three** of them are about AI crawlers — the row's pattern also matches "AI scraper" and |
| | "agentic browser", and two members are LLM-agent papers. See //Measuring who crawls you// |
| | above. |
| |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| |
| * **How the population was built.** No schema field records being blocked, so the | * **How the population was built.** No schema field records being blocked, so the population is a full-text sweep over ''paper.cols.txt'' with whitespace collapsed — a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten ordered patterns, each either anchored on an obstruction noun or requiring a first-person subject in the same sentence. Take the same ten patterns with every first-person requirement removed and the sweep returns **406** papers instead of 138; the extra 268 were not hand-audited, but a read of a sample of them found adversaries evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking requests, which is what the anchors are there to exclude. |
| population is a full-text sweep over ''paper.cols.txt'' with whitespace collapsed — | * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone. |
| a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten | * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80. |
| ordered patterns, each either anchored on an obstruction noun or requiring a | * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Of those 13, **11 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text, not in the extraction — and **2** are below threshold in both renderings. **Below threshold is not "unsupported"** — all 13 were read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded. |
| first-person subject in the same sentence. Take the same ten patterns with every | |
| first-person requirement removed and the sweep returns **406** papers instead of 138; | |
| the extra 268 were not hand-audited, but a read of a sample of them found adversaries | |
| evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking | |
| requests, which is what the anchors are there to exclude. | |
| * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict | |
| with its deciding sentence; the report script fails loudly if any candidate is | |
| uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", | |
| 58.7% for "on topic at all" — is published above rather than hidden, and it is the | |
| reason no figure on this page is taken from the sweep alone. | |
| * **Recall is estimated, not measured.** The **recall probe** is a third pattern, | |
| separate from both sets above and much wider than either: any first-person sentence | |
| with an obstruction token within 240 characters. Run over the 982 crawling papers the | |
| tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible | |
| every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand | |
| map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable | |
| local benchmark app, not on the open web. Dropping it moves the point estimate from | |
| 96 missed papers to 80. | |
| * **Quotes were checked.** All 161 deciding quotes were verified against the rendering | |
| the auditors read: 111 match exactly, 37 match on at least 60% of their five-word | |
| windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one | |
| was read by hand, all 13 are present in the paper, and the mismatch is always a | |
| two-column reading order splicing another column through the sentence. Three quotes | |
| were rewritten after that reading and the rewrite is recorded. | |
| * **Papers are counted, never tuples**, and a paper counts once. | * **Papers are counted, never tuples**, and a paper counts once. |
| * **Silence is not absence.** "Does not report obstruction" means the paper did not | * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low. |
| say. These are reporting figures throughout, and the whole point of the page is that | * **Venue coverage.** Seven venues only. Several relevant papers were published outside them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the field. |
| the reporting rate is low. | * **Every query, the report script's unedited output, the full candidate list with its verdicts, the residue and the external-source checks are on [[provenance:programming:crawler_detection]].** Corpus-level caveats are on [[literature:corpus]]. |
| * **Venue coverage.** Seven venues only. Several relevant papers were published outside | |
| them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the | |
| field. | |
| * **Every query, the report script's unedited output, the full candidate list with its | |
| verdicts, the residue and the external-source checks are on | |
| [[provenance:programming:crawler_detection]].** Corpus-level caveats are on | |
| [[literature:corpus]]. | |
| |
| ===== What to Report ===== | ===== What to Report ===== |
| |
| - **Your non-load rate, broken down by cause.** One table. Network error, DNS, 4xx, | - **Your non-load rate, broken down by cause.** One table. Network error, DNS, 4xx, 5xx, timeout, challenge page, empty render. This is the single highest-value thing on the list, and the closest measurable proxy for how rare it is — the share of crawling papers that report obstruction at all — is 3.8%. |
| 5xx, timeout, challenge page, empty render. This is the single highest-value thing on | - **Whether you counted challenge pages as successes.** If your success test is the HTTP status code, say so, because it means you did. |
| the list, and the closest measurable proxy for how rare it is — the share of crawling | - **Headless or headful, and which binary.** Since Chrome 132 that is two questions, not one. 12.5% of papers answer even the first. |
| papers that report obstruction at all — is 3.8%. | - **Your vantage point and its address space** — cloud, university, residential, Tor. See [[Design:Crawling location]]. |
| - **Whether you counted challenge pages as successes.** If your success test is the | - **Whether you attempted evasion.** Either answer is fine; the absence of an answer is not. If you did, name the tool //and its version//, and justify it against [[Practices:Ethics]]. |
| HTTP status code, say so, because it means you did. | - **Any control condition you ran** — a second configuration, a second vantage point, a manual sample — and what the delta was. A null delta is a result worth a sentence. |
| - **Headless or headful, and which binary.** Since Chrome 132 that is two questions, | - **The date.** Bot-management deployment moved materially in 2025 (see above). A block rate without a date is not comparable with anything. |
| not one. 12.5% of papers answer even the first. | |
| - **Your vantage point and its address space** — cloud, university, residential, Tor. | |
| See [[Design:Crawling location]]. | |
| - **Whether you attempted evasion.** Either answer is fine; the absence of an answer is | |
| not. If you did, name the tool //and its version//, and justify it against | |
| [[Practices:Ethics]]. | |
| - **Any control condition you ran** — a second configuration, a second vantage point, a | |
| manual sample — and what the delta was. A null delta is a result worth a sentence. | |
| - **The date.** Bot-management deployment moved materially in 2025 (see above). A block | |
| rate without a date is not comparable with anything. | |
| |
| ===== Papers to Read First ===== | ===== Papers to Read First ===== |
| If you have an afternoon, in this order: | If you have an afternoon, in this order: |
| |
| - **Ahmad et al., TheWebConf 2020** {[ahmad2020_apophanies]} — why crawler design is a | - **Ahmad et al., TheWebConf 2020** {[ahmad2020_apophanies]} — why crawler design is a threat to validity and not an implementation detail. Read this one first even if you read nothing else. |
| threat to validity and not an implementation detail. Read this one first even if you | - **Jueckstock et al., TheWebConf 2021** {[jueckstock2021_realistic]} — the same argument made quantitatively, over crawl configuration and vantage point together. |
| read nothing else. | - **Invernizzi et al., IEEE S&P 2016** {[invernizzi2016_cloak]} — the canonical measurement of sites serving machines a different web. |
| - **Jueckstock et al., TheWebConf 2021** {[jueckstock2021_realistic]} — the same | - **Vastel et al., USENIX Sec 2018** {[vastel2018_scanner]} — how automation is actually detected in the browser, and why inconsistency beats any single attribute. |
| argument made quantitatively, over crawl configuration and vantage point together. | - **Venugopalan et al., IMC 2025** {[venugopalan2025_inconsistent]} — the current state of that method, measured against two commercial detectors. |
| - **Invernizzi et al., IEEE S&P 2016** {[invernizzi2016_cloak]} — the canonical | - **Liu et al., IMC 2025** {[liu2025_somesite]} — the AI-crawler turn, and a clean example of the control-condition design this page recommends. |
| measurement of sites serving machines a different web. | - **Krumnow et al., CoNEXT 2022** {[krumnow2022_gullible]} — what it looks like when somebody writes a detector for the tool //you// are using. |
| - **Vastel et al., USENIX Sec 2018** {[vastel2018_scanner]} — how automation is | |
| actually detected in the browser, and why inconsistency beats any single attribute. | |
| - **Venugopalan et al., IMC 2025** {[venugopalan2025_inconsistent]} — the current | |
| state of that method, measured against two commercial detectors. | |
| - **Liu et al., IMC 2025** {[liu2025_somesite]} — the AI-crawler turn, and a clean | |
| example of the control-condition design this page recommends. | |
| - **Krumnow et al., CoNEXT 2022** {[krumnow2022_gullible]} — what it looks like when | |
| somebody writes a detector for the tool //you// are using. | |
| |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| <WRAP todo> | <WRAP todo> |
| * **Nobody has measured bot-management deployment against a site-popularity | * **Nobody has measured bot-management deployment against a site-popularity distribution.** The bias this page argues for — that blocking concentrates on high-ranked sites — is inferred from where the products are sold and from the shape of individual papers' losses. A crawl that reports its block rate //by rank decile// would settle it, and would cost one extra column. |
| distribution.** The bias this page argues for — that blocking concentrates on | * **The published figures are not comparable and nobody has made them so.** Twenty-two quantified figures, twenty-two denominators, no two of them the same population. A single study crawling one sample under several configurations and reporting one block rate per configuration would give the field a reference point it does not have. |
| high-ranked sites — is inferred from where the products are sold and from the shape | * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured. |
| of individual papers' losses. A crawl that reports its block rate //by rank decile// | * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it. |
| would settle it, and would cost one extra column. | * **There is no shared challenge-marker set, and there should be.** Every paper that detects challenge pages writes its own string list, none publishes it as a maintained artefact, and none reports its precision. A curated, versioned list — with per-vendor markers and a recall figure — is a small artefact that would make every non-load table on this page comparable with every other. |
| * **The published figures are not comparable and nobody has made them so.** | * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus. |
| Twenty-two quantified figures, twenty-two denominators, no two of them the same | |
| population. A single study crawling one sample under several configurations and | |
| reporting one block rate per configuration would give the field a reference point it | |
| does not have. | |
| * **How much of a "difference between crawls" result is bot management** rather than | |
| the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} | |
| measured that repeated and differently configured crawls disagree; how much of that | |
| disagreement is the site discriminating, rather than the web changing, is unmeasured. | |
| * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare | |
| changed a default on a known date in July 2025. Whether academic crawlers' block | |
| rates moved across that date is answerable from any longitudinal crawl that spans it. | |
| * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could | |
| declare itself and be admitted on those terms. Whether operators would admit | |
| academic crawlers, and on what conditions, is a question for a survey of operators, | |
| not for this corpus. | |
| </WRAP> | </WRAP> |
| |
