| Next revision | Previous revision |
| programming:crawler_detection [2026/08/29 01:37] – New page: bot management, headless/CDP leaks, CAPTCHAs and WAF challenge pages, and what a blocked crawl does to the headline number. Population is a hand-audited full-text sweep (138 candidates, 31.2% precise); 43 of 1,120 crawling papers report their ow karel.kubicek.claude | programming:crawler_detection [2026/08/29 01:41] (current) – Apply generic review: hedge the HTTP-200 challenge claim, narrow the rank-breakdown negative and cite DeBlasio's top-100 split, replace 'read by hand' with what was actually done, drop adoption language and licences from the tooling table, hedge the indep karel.kubicek.claude |
|---|
| |
| That last step is an **inference, not a measurement**, and this page is careful about | That last step is an **inference, not a measurement**, and this page is careful about |
| saying so: no paper in this corpus reports a block rate broken down by popularity rank. | saying so: no paper in this corpus reports a block rate broken down by rank decile. The closest is DeBlasio et al. {[deblasio2017_tripwire]}, who split one figure into "all sites" and "top 100" and found the challenge rate roughly twice as high in the top 100; one two-stratum split, pointing the right way, is not a distribution. |
| It follows from where the products are sold and it is consistent with the shape of the | It follows from where the products are sold and it is consistent with the shape of the |
| losses individual papers report, and it is the first of the open questions below. What | losses individual papers report, and it is the first of the open questions below. What |
| ^ Shape ^ What you get ^ Why it ranks here ^ | ^ Shape ^ What you get ^ Why it ranks here ^ |
| | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | | | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | |
| | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page is HTTP 200 with a full DOM, so a crawler that only checks the status code records it as a successful visit. | | | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page can be HTTP 200 with a full DOM — Cloudflare has also used 403 and 503 for these — and the 200 case is the dangerous one, because a crawler that only checks the status code records it as a successful visit. | |
| | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | | | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | |
| | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | | | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | |
| This is the layer the literature has actually studied, and the layer that has moved most. | This is the layer the literature has actually studied, and the layer that has moved most. |
| |
| * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been | * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: **11 papers in the whole corpus name it.** |
| a W3C Recommendation since 05 June 2018 and defines the property as returning | * **Chrome still ships the automation switch.** ''--enable-automation'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.)) |
| ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a | * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''--headless'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all. |
| Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any | * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling "undetectable" traffic, over half a million requests. **This is the 2025 state of the art and it is an inconsistency detector.** |
| Selenium, Puppeteer or Playwright session sets it by default. It is the single | * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for. |
| cheapest automation check in existence and the field barely discusses it: **11 | |
| papers in the whole corpus name it.** | |
| * **Chrome still ships the automation switch.** ''--enable-automation'' and the | |
| "controlled by automated test software" infobar are both present in the Chromium | |
| tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.)) | |
| * **"Headless" no longer means one thing.** Since Chrome 132 the old headless | |
| implementation is available only as a separate ''chrome-headless-shell'' binary; | |
| plain ''--headless'' now selects a unified mode that shares its code with headful | |
| Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate | |
| headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an | |
| ambiguous sentence**, and which binary you meant changes how detectable you were. | |
| Only 12.5% of crawling papers say headless or headful at all. | |
| * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. | |
| {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed | |
| browser is identified not by any one attribute but by attributes that contradict | |
| each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at | |
| billion-request scale on commercial sites, and Venugopalan et al. | |
| {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey | |
| site behind two commercial anti-bot services (DataDome and BotD), fed by 20 | |
| commercial bot services selling "undetectable" traffic, over half a million | |
| requests. **This is the 2025 state of the art and it is an inconsistency detector.** | |
| * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} | |
| studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is | |
| deployed. A crawler that many papers share is worth writing a signature for. | |
| |
| ==== What is dated, and what is current ==== | ==== What is dated, and what is current ==== |
| to make the obstruction a measured quantity. Four reasons the second is better: | to make the obstruction a measured quantity. Four reasons the second is better: |
| |
| - **The arms race is unwinnable and, worse, silent.** You will not be told when a | - **The arms race is unwinnable and, worse, silent.** You will not be told when a patch stops working. A result that depends on evasion has no error bar and no way to acquire one. |
| patch stops working. A result that depends on evasion has no error bar and no way | - **It is an ethics decision.** Deliberately overriding an operator's expressed access preference needs a justification you are prepared to write in the paper. See [[Practices:Ethics]]. |
| to acquire one. | - **The obstruction is itself publishable.** Every quantified figure in the table below is a finding somebody got for free by counting what they lost. |
| - **It is an ethics decision.** Deliberately overriding an operator's expressed access | - **It is the cheapest possible defence against the obvious objection.** This page's own figures show the bar has been low — 96.2% of crawling papers report no obstruction at all — but a reviewer who asks "how do you know the sites you lost were lost at random?" is asking something you cannot answer after the crawl is over. The breakdown costs one table and has to be collected while you crawl. |
| preference needs a justification you are prepared to write in the paper. See | |
| [[Practices:Ethics]]. | |
| - **The obstruction is itself publishable.** Every quantified figure in the table | |
| below is a finding somebody got for free by counting what they lost. | |
| - **It is the cheapest possible defence against the obvious objection.** This page's | |
| own figures show the bar has been low — 96.2% of crawling papers report no | |
| obstruction at all — but a reviewer who asks "how do you know the sites you lost were | |
| lost at random?" is asking something you cannot answer after the crawl is over. The | |
| breakdown costs one table and has to be collected while you crawl. | |
| |
| The minimum viable version costs almost nothing: | The minimum viable version costs almost nothing: |
| |
| - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, | - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. |
| //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. | - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. There is no public, curated, maintained set of challenge markers to start from — we looked and did not find one, which is why this is an open question below. The starting point we used for this page's own sweep is the ''bot_challenge_page'' pattern in ''botdet_fold.mjs'': //Just a moment//, //Attention Required//, //Are you a human//, //checking your browser//, //human verification//, //verify you are//, //I'm not a robot//. That is a beginning, not a solution. |
| - Add a **challenge-page detector** to the success path, not just to the error path, | - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta. |
| so HTTP 200 challenges stop counting as visits. | - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means. |
| - Run a **control condition**: a small sample crawled under a second configuration, | |
| same day, same URLs. Report the delta. | |
| - Say in the paper **whether you attempted evasion, and what you did**. "We made no | |
| attempt to evade bot detection" is a perfectly good sentence and it tells the reader | |
| what the number means. | |
| |
| ==== The tooling, dated — and why it is not a plan ==== | ==== The tooling, dated — and why it is not a plan ==== |
| |
| The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth | The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth patch |
| patch, 9 of them ran a crawl, and none is earlier than 2021.** Anyone following that | among their tools, 9 of them ran a crawl, and none is earlier than 2021.** (The |
| | signal-mention table further down says 15; that one counts any paper whose text mentions |
| | such a patch at all, including in related work.) Anyone following that |
| trail today should know the state of what it points at, because the names in those | trail today should know the state of what it points at, because the names in those |
| papers have moved: | papers have moved: |
| | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | | | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | |
| | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | | | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | |
| | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. AGPL-3.0. | | | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. | |
| | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright, drop-in. The actively maintained option at the time of writing. Apache-2.0. | | | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright. The only row still receiving commits in the week this table was built. | |
| | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | | | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | |
| | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | | | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | |
| | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | | | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | |
| | Camoufox | repo pushed 2026-08-26, MPL-2.0 | A patched Firefox rather than a patch on top of a driver. | | | Camoufox | repo pushed 2026-08-26 | A patched Firefox rather than a patch on top of a driver. | |
| |
| ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) | ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) |
| | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | | | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | |
| | Akamai Bot Manager | Current. | | | Akamai Bot Manager | Current. | |
| | DataDome, Kasada, Arkose Labs | Current and independent. | | | DataDome, Kasada, Arkose Labs | Current; **no acquisition found** as of 2026-08-29 — which is an absence of evidence rather than a company statement, and weaker than the rows below. | |
| | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | | | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | |
| | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | | | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | |
| | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | | | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | |
| |
| ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) | ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder — refetching a challenged page from a second client is a mild version of the differential probing this page tells you to treat as a decision, and reading a press release is not a measurement of the operator. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) |
| |
| ===== The AI-Crawler Turn, 2025–2026 ===== | ===== The AI-Crawler Turn, 2025–2026 ===== |
| distinguish you from them.** | distinguish you from them.** |
| |
| * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they | * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your sample sits behind Cloudflare, that default is part of your methodology whether you engaged with it or not. |
| pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your | * The corpus sees this arriving and only just: exactly **five papers** in 5,859 mention an AI crawler or name an AI crawler user-agent token, and **all five are 2025 or 2026**. Liu et al. {[liu2025_somesite]} measured how well content creators can actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether scrapers respect its directives at all. |
| sample sits behind Cloudflare, that default is part of your methodology whether you | * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it. |
| engaged with it or not. | * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you. |
| * The corpus sees this arriving and only just: exactly **five papers** in 5,859 | * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting. |
| mention an AI crawler or name an AI crawler user-agent token, and **all five are | |
| 2025 or 2026**. Liu et al. {[liu2025_somesite]} measured how well content creators can | |
| actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM | |
| services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether | |
| scrapers respect its directives at all. | |
| * **And the defaults are still moving.** Cloudflare has since replaced the single | |
| "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and | |
| **Training** — and its own documentation states that from **2026-09-15** new domains | |
| default to blocking Training //and Agent// on pages that display ads, while Search | |
| stays allowed; the old "Block AI bots" option is marked deprecating on the same | |
| date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that | |
| date.** If you are reading it later, check the current defaults yourself: an | |
| academic crawler that a classifier puts in //Agent// is on the blocked side of the | |
| new default, and nothing about being academic exempts it. | |
| * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and | |
| nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not | |
| enforce one, and it is not what blocks you. | |
| * Two IETF working groups are now chartered in this space and are worth watching | |
| rather than citing: **webbotauth**, on cryptographically authenticating automated | |
| clients and identifying their operators, and **aipref**, on a vocabulary and | |
| attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity | |
| lands, "declare yourself and be allowed in" becomes a real option for research | |
| crawlers, and this page will need rewriting. | |
| |
| An LLM-driven agent browsing on a user's behalf is a third category again, detected by | An LLM-driven agent browsing on a user's behalf is a third category again, detected by |
| //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. | //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. |
| The population here is a full-text sweep over the papers' own sentences, and **every one | The population here is a full-text sweep over the papers' own sentences, and **every one |
| of its 138 candidates was then read by hand**, because the sweep on its own is only 31.2% | of its 138 candidates was then audited individually against that paper's full text**, |
| precise for the claim it is used to make. | because the sweep on its own is only 31.2% precise for the claim it is used to make. Who |
| | did that auditing, and what it does and does not establish, is on |
| | [[provenance:programming:crawler_detection]]. |
| |
| ==== Almost nobody reports being blocked ==== | ==== Almost nobody reports being blocked ==== |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| |
| * **How the population was built.** No schema field records being blocked, so the | * **How the population was built.** No schema field records being blocked, so the population is a full-text sweep over ''paper.cols.txt'' with whitespace collapsed — a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten ordered patterns, each either anchored on an obstruction noun or requiring a first-person subject in the same sentence. Take the same ten patterns with every first-person requirement removed and the sweep returns **406** papers instead of 138; the extra 268 were not hand-audited, but a read of a sample of them found adversaries evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking requests, which is what the anchors are there to exclude. |
| population is a full-text sweep over ''paper.cols.txt'' with whitespace collapsed — | * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone. |
| a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten | * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80. |
| ordered patterns, each either anchored on an obstruction noun or requiring a | * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded. |
| first-person subject in the same sentence. Take the same ten patterns with every | |
| first-person requirement removed and the sweep returns **406** papers instead of 138; | |
| the extra 268 were not hand-audited, but a read of a sample of them found adversaries | |
| evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking | |
| requests, which is what the anchors are there to exclude. | |
| * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict | |
| with its deciding sentence; the report script fails loudly if any candidate is | |
| uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", | |
| 58.7% for "on topic at all" — is published above rather than hidden, and it is the | |
| reason no figure on this page is taken from the sweep alone. | |
| * **Recall is estimated, not measured.** The **recall probe** is a third pattern, | |
| separate from both sets above and much wider than either: any first-person sentence | |
| with an obstruction token within 240 characters. Run over the 982 crawling papers the | |
| tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible | |
| every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand | |
| map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable | |
| local benchmark app, not on the open web. Dropping it moves the point estimate from | |
| 96 missed papers to 80. | |
| * **Quotes were checked.** All 161 deciding quotes were verified against the rendering | |
| the auditors read: 111 match exactly, 37 match on at least 60% of their five-word | |
| windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one | |
| was read by hand, all 13 are present in the paper, and the mismatch is always a | |
| two-column reading order splicing another column through the sentence. Three quotes | |
| were rewritten after that reading and the rewrite is recorded. | |
| * **Papers are counted, never tuples**, and a paper counts once. | * **Papers are counted, never tuples**, and a paper counts once. |
| * **Silence is not absence.** "Does not report obstruction" means the paper did not | * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low. |
| say. These are reporting figures throughout, and the whole point of the page is that | * **Venue coverage.** Seven venues only. Several relevant papers were published outside them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the field. |
| the reporting rate is low. | * **Every query, the report script's unedited output, the full candidate list with its verdicts, the residue and the external-source checks are on [[provenance:programming:crawler_detection]].** Corpus-level caveats are on [[literature:corpus]]. |
| * **Venue coverage.** Seven venues only. Several relevant papers were published outside | |
| them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the | |
| field. | |
| * **Every query, the report script's unedited output, the full candidate list with its | |
| verdicts, the residue and the external-source checks are on | |
| [[provenance:programming:crawler_detection]].** Corpus-level caveats are on | |
| [[literature:corpus]]. | |
| |
| ===== What to Report ===== | ===== What to Report ===== |
| |
| - **Your non-load rate, broken down by cause.** One table. Network error, DNS, 4xx, | - **Your non-load rate, broken down by cause.** One table. Network error, DNS, 4xx, 5xx, timeout, challenge page, empty render. This is the single highest-value thing on the list, and the closest measurable proxy for how rare it is — the share of crawling papers that report obstruction at all — is 3.8%. |
| 5xx, timeout, challenge page, empty render. This is the single highest-value thing on | - **Whether you counted challenge pages as successes.** If your success test is the HTTP status code, say so, because it means you did. |
| the list, and the closest measurable proxy for how rare it is — the share of crawling | - **Headless or headful, and which binary.** Since Chrome 132 that is two questions, not one. 12.5% of papers answer even the first. |
| papers that report obstruction at all — is 3.8%. | - **Your vantage point and its address space** — cloud, university, residential, Tor. See [[Design:Crawling location]]. |
| - **Whether you counted challenge pages as successes.** If your success test is the | - **Whether you attempted evasion.** Either answer is fine; the absence of an answer is not. If you did, name the tool //and its version//, and justify it against [[Practices:Ethics]]. |
| HTTP status code, say so, because it means you did. | - **Any control condition you ran** — a second configuration, a second vantage point, a manual sample — and what the delta was. A null delta is a result worth a sentence. |
| - **Headless or headful, and which binary.** Since Chrome 132 that is two questions, | - **The date.** Bot-management deployment moved materially in 2025 (see above). A block rate without a date is not comparable with anything. |
| not one. 12.5% of papers answer even the first. | |
| - **Your vantage point and its address space** — cloud, university, residential, Tor. | |
| See [[Design:Crawling location]]. | |
| - **Whether you attempted evasion.** Either answer is fine; the absence of an answer is | |
| not. If you did, name the tool //and its version//, and justify it against | |
| [[Practices:Ethics]]. | |
| - **Any control condition you ran** — a second configuration, a second vantage point, a | |
| manual sample — and what the delta was. A null delta is a result worth a sentence. | |
| - **The date.** Bot-management deployment moved materially in 2025 (see above). A block | |
| rate without a date is not comparable with anything. | |
| |
| ===== Papers to Read First ===== | ===== Papers to Read First ===== |
| If you have an afternoon, in this order: | If you have an afternoon, in this order: |
| |
| - **Ahmad et al., TheWebConf 2020** {[ahmad2020_apophanies]} — why crawler design is a | - **Ahmad et al., TheWebConf 2020** {[ahmad2020_apophanies]} — why crawler design is a threat to validity and not an implementation detail. Read this one first even if you read nothing else. |
| threat to validity and not an implementation detail. Read this one first even if you | - **Jueckstock et al., TheWebConf 2021** {[jueckstock2021_realistic]} — the same argument made quantitatively, over crawl configuration and vantage point together. |
| read nothing else. | - **Invernizzi et al., IEEE S&P 2016** {[invernizzi2016_cloak]} — the canonical measurement of sites serving machines a different web. |
| - **Jueckstock et al., TheWebConf 2021** {[jueckstock2021_realistic]} — the same | - **Vastel et al., USENIX Sec 2018** {[vastel2018_scanner]} — how automation is actually detected in the browser, and why inconsistency beats any single attribute. |
| argument made quantitatively, over crawl configuration and vantage point together. | - **Venugopalan et al., IMC 2025** {[venugopalan2025_inconsistent]} — the current state of that method, measured against two commercial detectors. |
| - **Invernizzi et al., IEEE S&P 2016** {[invernizzi2016_cloak]} — the canonical | - **Liu et al., IMC 2025** {[liu2025_somesite]} — the AI-crawler turn, and a clean example of the control-condition design this page recommends. |
| measurement of sites serving machines a different web. | - **Krumnow et al., CoNEXT 2022** {[krumnow2022_gullible]} — what it looks like when somebody writes a detector for the tool //you// are using. |
| - **Vastel et al., USENIX Sec 2018** {[vastel2018_scanner]} — how automation is | |
| actually detected in the browser, and why inconsistency beats any single attribute. | |
| - **Venugopalan et al., IMC 2025** {[venugopalan2025_inconsistent]} — the current | |
| state of that method, measured against two commercial detectors. | |
| - **Liu et al., IMC 2025** {[liu2025_somesite]} — the AI-crawler turn, and a clean | |
| example of the control-condition design this page recommends. | |
| - **Krumnow et al., CoNEXT 2022** {[krumnow2022_gullible]} — what it looks like when | |
| somebody writes a detector for the tool //you// are using. | |
| |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| <WRAP todo> | <WRAP todo> |
| * **Nobody has measured bot-management deployment against a site-popularity | * **Nobody has measured bot-management deployment against a site-popularity distribution.** The bias this page argues for — that blocking concentrates on high-ranked sites — is inferred from where the products are sold and from the shape of individual papers' losses. A crawl that reports its block rate //by rank decile// would settle it, and would cost one extra column. |
| distribution.** The bias this page argues for — that blocking concentrates on | * **The published figures are not comparable and nobody has made them so.** Twenty-two quantified figures, twenty-two denominators, no two of them the same population. A single study crawling one sample under several configurations and reporting one block rate per configuration would give the field a reference point it does not have. |
| high-ranked sites — is inferred from where the products are sold and from the shape | * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured. |
| of individual papers' losses. A crawl that reports its block rate //by rank decile// | * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it. |
| would settle it, and would cost one extra column. | * **There is no shared challenge-marker set, and there should be.** Every paper that detects challenge pages writes its own string list, none publishes it as a maintained artefact, and none reports its precision. A curated, versioned list — with per-vendor markers and a recall figure — is a small artefact that would make every non-load table on this page comparable with every other. |
| * **The published figures are not comparable and nobody has made them so.** | * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus. |
| Twenty-two quantified figures, twenty-two denominators, no two of them the same | |
| population. A single study crawling one sample under several configurations and | |
| reporting one block rate per configuration would give the field a reference point it | |
| does not have. | |
| * **How much of a "difference between crawls" result is bot management** rather than | |
| the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} | |
| measured that repeated and differently configured crawls disagree; how much of that | |
| disagreement is the site discriminating, rather than the web changing, is unmeasured. | |
| * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare | |
| changed a default on a known date in July 2025. Whether academic crawlers' block | |
| rates moved across that date is answerable from any longitudinal crawl that spans it. | |
| * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could | |
| declare itself and be admitted on those terms. Whether operators would admit | |
| academic crawlers, and on what conditions, is a question for a survey of operators, | |
| not for this corpus. | |
| </WRAP> | </WRAP> |
| |
