| Both sides previous revisionPrevious revision | |
| programming:crawler_detection [2026/08/29 01:38] – Fix rendering: DokuWiki treats an indented continuation line as preformatted text, so every wrapped bullet was rendering literally (126 lines). Bullets are now one line each. Authored by Claude karel.kubicek.claude | programming:crawler_detection [2026/08/29 01:41] (current) – Apply generic review: hedge the HTTP-200 challenge claim, narrow the rank-breakdown negative and cite DeBlasio's top-100 split, replace 'read by hand' with what was actually done, drop adoption language and licences from the tooling table, hedge the indep karel.kubicek.claude |
|---|
| |
| That last step is an **inference, not a measurement**, and this page is careful about | That last step is an **inference, not a measurement**, and this page is careful about |
| saying so: no paper in this corpus reports a block rate broken down by popularity rank. | saying so: no paper in this corpus reports a block rate broken down by rank decile. The closest is DeBlasio et al. {[deblasio2017_tripwire]}, who split one figure into "all sites" and "top 100" and found the challenge rate roughly twice as high in the top 100; one two-stratum split, pointing the right way, is not a distribution. |
| It follows from where the products are sold and it is consistent with the shape of the | It follows from where the products are sold and it is consistent with the shape of the |
| losses individual papers report, and it is the first of the open questions below. What | losses individual papers report, and it is the first of the open questions below. What |
| ^ Shape ^ What you get ^ Why it ranks here ^ | ^ Shape ^ What you get ^ Why it ranks here ^ |
| | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | | | **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. | |
| | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page is HTTP 200 with a full DOM, so a crawler that only checks the status code records it as a successful visit. | | | **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page can be HTTP 200 with a full DOM — Cloudflare has also used 403 and 503 for these — and the 200 case is the dangerous one, because a crawler that only checks the status code records it as a successful visit. | |
| | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | | | **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. | |
| | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | | | **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. | |
| |
| - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. | - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table. |
| - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. | - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. There is no public, curated, maintained set of challenge markers to start from — we looked and did not find one, which is why this is an open question below. The starting point we used for this page's own sweep is the ''bot_challenge_page'' pattern in ''botdet_fold.mjs'': //Just a moment//, //Attention Required//, //Are you a human//, //checking your browser//, //human verification//, //verify you are//, //I'm not a robot//. That is a beginning, not a solution. |
| - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta. | - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta. |
| - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means. | - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means. |
| ==== The tooling, dated — and why it is not a plan ==== | ==== The tooling, dated — and why it is not a plan ==== |
| |
| The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth | The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth patch |
| patch, 9 of them ran a crawl, and none is earlier than 2021.** Anyone following that | among their tools, 9 of them ran a crawl, and none is earlier than 2021.** (The |
| | signal-mention table further down says 15; that one counts any paper whose text mentions |
| | such a patch at all, including in related work.) Anyone following that |
| trail today should know the state of what it points at, because the names in those | trail today should know the state of what it points at, because the names in those |
| papers have moved: | papers have moved: |
| | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | | | ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. | |
| | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | | | ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. | |
| | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. AGPL-3.0. | | | ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. | |
| | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright, drop-in. The actively maintained option at the time of writing. Apache-2.0. | | | ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright. The only row still receiving commits in the week this table was built. | |
| | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | | | ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. | |
| | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | | | ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. | |
| | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | | | ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. | |
| | Camoufox | repo pushed 2026-08-26, MPL-2.0 | A patched Firefox rather than a patch on top of a driver. | | | Camoufox | repo pushed 2026-08-26 | A patched Firefox rather than a patch on top of a driver. | |
| |
| ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) | ((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.)) |
| | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | | | Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. | |
| | Akamai Bot Manager | Current. | | | Akamai Bot Manager | Current. | |
| | DataDome, Kasada, Arkose Labs | Current and independent. | | | DataDome, Kasada, Arkose Labs | Current; **no acquisition found** as of 2026-08-29 — which is an absence of evidence rather than a company statement, and weaker than the rows below. | |
| | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | | | Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. | |
| | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | | | HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. | |
| | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | | | Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. | |
| |
| ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) | ((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder — refetching a challenged page from a second client is a mild version of the differential probing this page tells you to treat as a decision, and reading a press release is not a measurement of the operator. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.)) |
| |
| ===== The AI-Crawler Turn, 2025–2026 ===== | ===== The AI-Crawler Turn, 2025–2026 ===== |
| //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. | //did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it. |
| The population here is a full-text sweep over the papers' own sentences, and **every one | The population here is a full-text sweep over the papers' own sentences, and **every one |
| of its 138 candidates was then read by hand**, because the sweep on its own is only 31.2% | of its 138 candidates was then audited individually against that paper's full text**, |
| precise for the claim it is used to make. | because the sweep on its own is only 31.2% precise for the claim it is used to make. Who |
| | did that auditing, and what it does and does not establish, is on |
| | [[provenance:programming:crawler_detection]]. |
| |
| ==== Almost nobody reports being blocked ==== | ==== Almost nobody reports being blocked ==== |
| * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured. | * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured. |
| * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it. | * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it. |
| | * **There is no shared challenge-marker set, and there should be.** Every paper that detects challenge pages writes its own string list, none publishes it as a maintained artefact, and none reports its precision. A curated, versioned list — with per-vendor markers and a recall figure — is a small artefact that would make every non-load table on this page comparable with every other. |
| * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus. | * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus. |
| </WRAP> | </WRAP> |
