Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== When the Website Notices Your Crawler ======

A crawl that is blocked does not fail loudly. It returns a page, a status code and a
timestamp, and your pipeline records all three. What it does not record is that the
page you got is not the page a person gets — and every number you compute downstream
inherits that difference without carrying a flag for it.

This page is about that difference: what makes an automated client identifiable, what
happens when it is identified, and what a reviewer will expect you to have measured.
It is **not** an anti-detection cookbook. Evading a site's access controls is a
research-ethics decision before it is an engineering one, and it belongs on
[[Practices:Ethics]]. The advice here points the other way: **measure the obstruction
and report it**, because the obstruction is data about your sample, and the sample is
what your paper is actually about.

Three neighbours cover adjacent ground. [[Programming:Crawler]] compares the libraries
that drive a browser and how visible each is. [[Design:Crawling location]] covers the
one detection input you choose before you write any code — where your packets come
from. [[Programming:Crawler:LLM Agents]] covers a different surface entirely: an
LLM-driven agent is detected, and blocked, by mechanisms aimed at a category that did
not exist when most of this literature was written.

===== What a Blocked Crawl Does to Your Number =====

The reason to care is not that you lose sites. It is **which** sites you lose.

Bot management is not deployed at random. It is bought by sites with something to
protect and the budget to protect it: large e-commerce, banking, ticketing, travel,
social platforms, anything behind an enterprise CDN. Those are disproportionately the
high-ranked, high-traffic sites — the ones your top-//N// list oversamples on purpose,
and the ones whose behaviour your paper is most likely to be about. A 10% non-load rate
that falls entirely on the top 2,000 sites is not 10% noise; it is a systematic hole
exactly where your signal is.

That last step is an **inference, not a measurement**, and this page is careful about
saying so: no paper in this corpus reports a block rate broken down by rank decile. The closest is DeBlasio et al. {[deblasio2017_tripwire]}, who split one figure into "all sites" and "top 100" and found the challenge rate roughly twice as high in the top 100; one two-stratum split, pointing the right way, is not a distribution.
It follows from where the products are sold and it is consistent with the shape of the
losses individual papers report, and it is the first of the open questions below. What
//is// measured is everything after it.

Ahmad et al. {[ahmad2020_apophanies]} make the general form of this argument: what a
crawler sees is a function of how the crawler is built, and conclusions drawn from
crawl data can be artefacts of the instrument. Jueckstock et al. {[jueckstock2021_realistic]}
measured the specific version of it — crawl configuration and vantage point change what
the web shows you — and Zeber et al. {[zeber2020representativeness]} measured how far
automated crawls diverge from real human browsing. Annamalai et al. {[annamalai2025_beyond]}
ran the comparison again in 2025 with real users alongside a crawler and found the
crawler's failures concentrated in 4XX responses tied to bot detection.

**Four shapes of obstruction, in increasing order of danger:**

^ Shape ^ What you get ^ Why it ranks here ^
| **Hard block** | HTTP 403, 429, a connection reset, or an "Access Denied" page | Loud. Your pipeline sees a non-200 and you can count it. |
| **Interactive challenge** | A CAPTCHA, a JavaScript challenge, an interstitial | Loud //if you look//. A challenge page can be HTTP 200 with a full DOM — Cloudflare has also used 403 and 503 for these — and the 200 case is the dangerous one, because a crawler that only checks the status code records it as a successful visit. |
| **Degraded response** | The page loads, but third-party scripts, ads or personalisation do not | Quiet. You measure "this site has fewer trackers" and publish it. |
| **Cloaking** | A different, plausible page, served only to you | Silent and indistinguishable from a finding. This is the one that ends up in the abstract. |

Cloaking is not hypothetical and it is not confined to phishing. Wang et al.
{[wang2011_cloak]} characterised search cloaking as far back as 2011; Invernizzi et al.
{[invernizzi2016_cloak]} built the canonical measurement of it, showing systematic
serving of different content to machines; Zhang et al. {[zhang2021_crawlphish]} measured
client-side cloaking specifically aimed at security crawlers; and Szurdi et al.
{[szurdi2021_where]} showed the quantitative form directly — driving the same traffic
distribution systems with a crawler persona and a human persona, the crawler was
**explicitly blocked 5 percentage points more often, and //covertly// blocked — sent to
a parked or error page — at least 8 points more often.** The covert number is larger
than the overt one. That is the whole problem in one measurement.

<WRAP important>
**The practical consequence.** A status-code check is not a load check. If your
pipeline's definition of "successfully crawled" is ''response.status == 200'', you are
counting challenge pages, interstitials and cloaked pages as successes, and their
content is flowing into your denominators.
</WRAP>

===== The Detection Surface, Layer by Layer =====

A bot-management product does not run one test. It scores a request against signals
from several layers at once, and the layers differ in how cheap they are to read and
how much you can do about them. Ordered from the network up:

^ Layer ^ What it reads ^ How much it costs the site ^ What you can honestly do ^
| **Network** | ASN and IP reputation, datacenter vs residential vs mobile address space, request rate per address | Nothing — it is a lookup | Choose the vantage point deliberately and report it. See [[Design:Crawling location]]. |
| **Transport** | The TLS ClientHello fingerprint (JA3/JA4): cipher order, extension order, ALPN | Nothing — the handshake is already parsed | Nothing, if you use a normal browser. A raw HTTP client fingerprints as a raw HTTP client. |
| **HTTP** | Header set, header //order//, HTTP/2 SETTINGS and priority frames, User-Agent coherence | Nothing | Drive a real browser rather than an HTTP library. |
| **Browser runtime** | ''navigator.webdriver'', missing or extra JS APIs, headless-specific values, CDP artefacts, fingerprint self-contradictions | One script execution | This is where a headless/headful choice and a driver choice actually land. |
| **Behaviour** | Mouse movement, scroll, timing between actions, dwell time, navigation graph | Telemetry over a session | Interact realistically if your research question needs interaction — see [[Programming:Interaction]]. |
| **Challenge** | Whatever the client does when handed a CAPTCHA, a proof-of-work puzzle or a JS challenge | User friction, so it is reserved for suspicious scores | Detect it and count it. |

==== The browser-runtime layer, dated ====

This is the layer the literature has actually studied, and the layer that has moved most.

  * **''navigator.webdriver'' is a specification, not a bug.** WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning ''true'' when the browser's //webdriver-active// flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.(([[https://www.w3.org/TR/webdriver1/|www.w3.org/TR/webdriver1]] and [[https://www.w3.org/TR/webdriver2/|www.w3.org/TR/webdriver2]], both fetched 2026-08-29. Note that ''w3.org'' returns HTTP 403 to a browser User-Agent string and 200 to plain ''curl'' — a detection rule pointed the opposite way from the ones this page is about.)) Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: **11 papers in the whole corpus name it.**
  * **Chrome still ships the automation switch.** ''--enable-automation'' and the "controlled by automated test software" infobar are both present in the Chromium tree today.((''content/public/common/content_switches.cc'' defines ''kEnableAutomation'' with the comment "Enable indication that browser is controlled by automation"; ''chrome/browser/ui/startup/automation_infobar_delegate.cc'' defines the infobar and its ''IDS_CONTROLLED_BY_AUTOMATION'' string. Both read from ''chromium.googlesource.com/chromium/src/+/HEAD'' on 2026-08-29.))
  * **"Headless" no longer means one thing.** Since Chrome 132 the old headless implementation is available only as a separate ''chrome-headless-shell'' binary; plain ''--headless'' now selects a unified mode that shares its code with headful Chrome.(([[https://developer.chrome.com/docs/chromium/headless|developer.chrome.com/docs/chromium/headless]], fetched 2026-08-29.)) Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.(([[https://playwright.dev/docs/browsers|playwright.dev/docs/browsers]], fetched 2026-08-29.)) So **"we ran headless Chrome" is now an ambiguous sentence**, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
  * **Fingerprint //inconsistency// beat fingerprint //content//.** Vastel et al. {[vastel2018_scanner]} established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. {[wu2023_manyfaces]} characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. {[venugopalan2025_inconsistent]} ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling "undetectable" traffic, over half a million requests. **This is the 2025 state of the art and it is an inconsistency detector.**
  * **Framework-specific detection exists.** Krumnow et al. {[krumnow2022_gullible]} studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for.

==== What is dated, and what is current ====

The corpus rewards methods that were written up, which biases any ranking toward
whatever was fashionable mid-window. Dating each strand explicitly:

^ Strand ^ Peak in this corpus ^ Status now ^
| Breaking text CAPTCHAs with a bespoke solver | 2010–2020 | **Superseded as a research question.** Shi et al. {[shi2020_text]} asked whether text CAPTCHA was dead in 2020; the answer settled. |
| Cloaking measurement against security crawlers | 2011–2022 | **Still current, still under-cited outside phishing research.** {[wang2011_cloak]} {[invernizzi2016_cloak]} {[zhang2021_crawlphish]} {[zhang2022_spartacus]} |
| Using a //human-verification// step as the cloak | 2020–2024 | **Current, and the reason a CAPTCHA is not only your problem.** Maroofi et al. {[maroofi2020_human]} showed phishing pages putting a human-verification step in front of content specifically to defeat detection crawlers; Teoh et al. {[teoh2024_phishdecloaker]} built the counter-measurement. |
| Server-side bot classification from request logs | 2020–2022 | **Current but industrial.** {[jan2020_throwing]} {[li2021_good]} {[herley2022_automated]} |
| Browser-fingerprint inconsistency detection | 2018–2025 | **Current state of the art.** {[vastel2018_scanner]} {[wu2023_manyfaces]} {[venugopalan2025_inconsistent]} |
| CAPTCHA solved by a general-purpose model | 2023–2026 | **The live front.** {[searles2023_captcha]} {[teoh2025_captcha]} {[qi2026_viper]} |
| Blocking crawlers because they are AI training crawlers | 2025–2026 | **New, and orthogonal to everything above.** {[liu2025_somesite]} {[cui2025_odyssey]} {[kim2025_scrapers]} |

The last two rows rest on the corpus's thinnest years — 2025 and 2026 are provisional
here (see [[#Use in Publications]] below) — so treat them as a direction of travel rather
than as settled counts. But the direction is not ambiguous: Teoh et al.
{[teoh2025_captcha]} solved visual CAPTCHAs with an agentic vision-language model, and
Qi et al. {[qi2026_viper]} did the same for visual //reasoning// CAPTCHAs. Meanwhile
Searles et al. {[searles2023_captcha]} measured what CAPTCHAs cost the //humans// —
1,400 participants solving 14,000 CAPTCHAs, with solving time and preference varying
sharply by type. Read together, those are the two halves of why a site reserves a
challenge for a suspicious score rather than showing it to everyone. Nguyen et al.
{[nguyen2024_frame]} measured in-the-wild CAPTCHA abuse from the other direction.
**Designing a study around "the CAPTCHA will stop the adversary" is dated; designing one
around "the CAPTCHA will stop //us//" is still true, and is the reason it costs you
sites.**

===== What You Actually See, and How to Tell It Apart =====

Your crawler cannot ask why it failed. It can only classify what came back. These are
the discriminable cases:

^ Symptom ^ Likely cause ^ How to tell ^
| HTTP 403 with a short body | Edge block on IP or ASN reputation | Re-request the same URL from a second vantage point. If it succeeds, it is you, not the site. See [[Design:Crawling location]]. |
| HTTP 429, or 403 after //n// requests | Rate limiting | Back off and retry with a longer interval. If it succeeds, it is rate, not identity. |
| HTTP 200, small DOM, a challenge phrase, a meta-refresh or a script that reloads | JavaScript or proof-of-work challenge | Match the body against a small set of challenge markers, and check whether the final URL equals the requested one. |
| HTTP 200, a visible CAPTCHA widget | Interactive challenge | Look for the widget's own script origin or iframe rather than for the word "CAPTCHA", which appears on plenty of ordinary pages. |
| HTTP 200, plausible page, but no third-party requests at all | Degraded or cloaked response | Compare against a headful run, or a run from a different vantage point, on the same URL on the same day. |
| HTTP 200, plausible page, //different// from the human one | Cloaking | Only a differential comparison finds this. Nothing in the response says so. |

<WRAP tip>
The general instrument for the bottom three rows is the same: **the differential crawl.**
Fetch the same URL under two configurations that differ in one deliberate respect —
headless vs headful, datacenter vs residential, your crawler's User-Agent vs a stock one
— and diff the results. A systematic difference is a measurement of the site's
discrimination on that axis; no difference is evidence, though not proof, that you were
not singled out on it. Say which axis you varied, because "we ran a control" without it
is unfalsifiable. Szurdi et al.
{[szurdi2021_where]} and Liu et al. {[liu2025_somesite]} both use exactly this design,
and both use a **control condition** — sites that block //everything// — so that
"blocks our crawler" is separated from "blocks this particular identity".
</WRAP>

===== Measure It, Do Not Fight It =====

The engineering instinct is to become harder to detect. The research instinct should be
to make the obstruction a measured quantity. Four reasons the second is better:

  - **The arms race is unwinnable and, worse, silent.** You will not be told when a patch stops working. A result that depends on evasion has no error bar and no way to acquire one.
  - **It is an ethics decision.** Deliberately overriding an operator's expressed access preference needs a justification you are prepared to write in the paper. See [[Practices:Ethics]].
  - **The obstruction is itself publishable.** Every quantified figure in the table below is a finding somebody got for free by counting what they lost.
  - **It is the cheapest possible defence against the obvious objection.** This page's own figures show the bar has been low — 96.2% of crawling papers report no obstruction at all — but a reviewer who asks "how do you know the sites you lost were lost at random?" is asking something you cannot answer after the crawl is over. The breakdown costs one table and has to be collected while you crawl.

The minimum viable version costs almost nothing:

  - Classify every non-success into //network error//, //DNS failure//, //HTTP 4xx//, //HTTP 5xx//, //timeout//, //challenge page//, //empty render//. Publish the table.
  - Add a **challenge-page detector** to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits. There is no public, curated, maintained set of challenge markers to start from — we looked and did not find one, which is why this is an open question below. The starting point we used for this page's own sweep is the ''bot_challenge_page'' pattern in ''botdet_fold.mjs'': //Just a moment//, //Attention Required//, //Are you a human//, //checking your browser//, //human verification//, //verify you are//, //I'm not a robot//. That is a beginning, not a solution.
  - Run a **control condition**: a small sample crawled under a second configuration, same day, same URLs. Report the delta.
  - Say in the paper **whether you attempted evasion, and what you did**. "We made no attempt to evade bot detection" is a perfectly good sentence and it tells the reader what the number means.

==== The tooling, dated — and why it is not a plan ====

The corpus's own stealth-tool trail is short and recent: **11 papers name a stealth patch
among their tools, 9 of them ran a crawl, and none is earlier than 2021.** (The
signal-mention table further down says 15; that one counts any paper whose text mentions
such a patch at all, including in related work.) Anyone following that
trail today should know the state of what it points at, because the names in those
papers have moved:

^ Project ^ State on 2026-08-29 ^ What to know ^
| ''puppeteer-extra-plugin-stealth'' | newest npm release **2.11.2**, published **2023-03-01**; repo not archived, no code release since | The name most cited in the corpus, and the most stale. |
| ''undetected-chromedriver'' | PyPI **3.5.5** (2024-02-17); repo last pushed 2025-07-05; publishes **no** GitHub Releases and no tags | Its own author has moved on — see the next row. |
| ''nodriver'' | PyPI **0.50.3** (2026-05-13) | Its GitHub description reads "Successor of Undetected-Chromedriver" — the same author's own succession claim. |
| ''patchright'' | tag **v1.62.0**, repo pushed 2026-08-19 | A patched Playwright. The only row still receiving commits in the week this table was built. |
| ''rebrowser-patches'' | tag **1.0.19** (2025-05-09), no commits since | Patches upstream Puppeteer and Playwright rather than forking them. |
| ''selenium-stealth'' | PyPI **1.0.6** (2020-11-05); repo last pushed 2021-12-30 | Abandoned. No successor found. |
| ''playwright-stealth'' | PyPI **2.0.3** (2026-04-04) | **The package name stayed put and the code moved.** The PyPI Homepage now points at ''Mattwmaster58/playwright_stealth''; the original ''AtuboDad'' repo was last pushed 2024-07-29. |
| Camoufox | repo pushed 2026-08-26 | A patched Firefox rather than a patch on top of a driver. |

((All repository, tag, npm and PyPI states in this table were read from ''api.github.com'', ''registry.npmjs.org'' and ''pypi.org'' on 2026-08-29 by ''scripts/external_checks_crawler_detection.sh''; the full output is on [[provenance:programming:crawler_detection]]. Note that several of these repositories publish //tags// but no GitHub Release objects, so ''/releases/latest'' returns HTTP 404 for them and a naive currency check reads them as abandoned.))

Two things this table is here to teach, neither of them about stealth. First, **a
package's last release date is not its currency**, and it fails in two different
directions here: ''undetected-chromedriver'' looks alive on GitHub but its own author's
successor is a differently-named project, while ''playwright-stealth'' looks stale under
its original repository and is in fact maintained — the package name stayed put and the
code moved to a different owner. Checking the name you already had would have got both
wrong. Second, **if you cite one of these in a paper, cite the version you ran**, because
the detection landscape it was written against no longer exists.

===== Who Is On the Other Side =====

Useful mostly so you can recognise a challenge page and name the vendor in your
methodology section. The market has consolidated hard, and papers that cite a vendor by
its 2019 name are citing something that no longer exists under it:

^ Product ^ Status on 2026-08-29 ^
| Cloudflare Bot Management, Turnstile | Current. Turnstile went generally available 2023-09-29, with its "Managed" mode free for unlimited use; Cloudflare calls it a CAPTCHA //alternative//, not a CAPTCHA. |
| Akamai Bot Manager | Current. |
| DataDome, Kasada, Arkose Labs | Current; **no acquisition found** as of 2026-08-29 — which is an absence of evidence rather than a company statement, and weaker than the rows below. |
| Imperva Advanced Bot Protection | Current — and this is where **Distil Networks** went: ''distilnetworks.com'' now redirects to Imperva's bot-management product page. Imperva itself has been part of **Thales** since 2023-12-04. |
| HUMAN Security | This is where **PerimeterX** went: ''perimeterx.com'' now redirects to ''humansecurity.com''. |
| F5 Distributed Cloud Bot Defense | This is where **Shape Security** went; F5 announced completion of the acquisition in its own press release. |
| BotD (open source, MIT) | Current, from the Fingerprint team. The one you can actually read the source of, and the one Venugopalan et al. {[venugopalan2025_inconsistent]} used as one of their two detectors. |
| Anubis (open source, MIT) | A proof-of-work interstitial aimed specifically at AI scrapers; its own repository describes it as weighing "the soul of incoming HTTP requests to stop AI crawlers". Newest tag v1.27.0. |

((Vendor states verified on 2026-08-29: ''perimeterx.com'' and ''distilnetworks.com'' redirect chains followed with ''curl''; ''f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security''; ''developers.cloudflare.com/bots/''; ''blog.cloudflare.com/turnstile-ga/''; ''api.github.com/repos/fingerprintjs/BotD'' and ''api.github.com/repos/TecharoHQ/anubis''. One check could not be completed and the failure is the point: Akamai's own Bot Manager product page returned **HTTP 403 "Access Denied"** to ''curl'' with a browser User-Agent //and// to a headless Playwright Chromium from our datacenter host, consistently, on every attempt across several hours. We did not establish the cause — it may be bot management, geography or ASN reputation — and we are not going to find out by trying harder — refetching a challenged page from a second client is a mild version of the differential probing this page tells you to treat as a decision, and reading a press release is not a measurement of the operator. A second check turned into the page's other worked example, and then into a better one. Imperva's own press-release page for the Thales acquisition first answered ''curl'' with **HTTP 200 and 920 bytes** — a ''NOINDEX, NOFOLLOW'' meta tag and an iframe reading "Request unsuccessful. Incapsula incident ID: 429000330232386336-…". A crawler whose success test is the status code would have recorded that as a visited page. A **headless** Playwright Chromium from the same host got the full article at the same time. Roughly forty minutes later, after we had stopped hammering it, plain ''curl'' from the same host got the full 180 KB article too, three times in a row. Three lessons, and the third is the one that matters: a challenge can be HTTP 200; "headless, therefore blocked" is not what these products score; and **the same client can be challenged and then admitted within the hour**, so a block rate measured once is a measurement of a moment. We are recording what we observed with its timing rather than presenting it as a standing property of that host.))

===== The AI-Crawler Turn, 2025–2026 =====

Everything above is about a site deciding whether a client is a //human//. Since 2025
there is a second question layered on top, with different economics: whether a client is
an **AI training or inference crawler**, and whether it has paid. This matters to an
academic crawler for a blunt reason — **the defences aimed at AI crawlers do not
distinguish you from them.**

  * On **2025-07-01** Cloudflare changed its default to blocking AI crawlers unless they pay for the content, and introduced a pay-per-crawl mechanism.(([[https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/|blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation]], fetched 2026-08-29.)) If any of your sample sits behind Cloudflare, that default is part of your methodology whether you engaged with it or not.
  * The corpus sees this arriving and only just: exactly **five papers** in 5,859 mention an AI crawler or name an AI crawler user-agent token, and **all five are 2025 or 2026**. Liu et al. {[liu2025_somesite]} measured how well content creators can actually keep AI crawlers out; Cui et al. {[cui2025_odyssey]} measured what LLM services do with ''robots.txt''; Kim et al. {[kim2025_scrapers]} measured whether scrapers respect its directives at all.
  * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it.
  * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you.
  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired and archived, with no formal standing.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.

An LLM-driven agent browsing on a user's behalf is a third category again, detected by
different signals and blocked under different rules. That is
[[Programming:Crawler:LLM Agents]], not this page.

===== Use in Publications =====

The figures below come from a structured extraction over **5,859 full-text papers** from
CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Unless stated
otherwise the population is the **1,120 papers that ran a crawl**. The 2025 and 2026
venue-years are provisional — CCS and IMC 2026 have not been held, and IEEE S&P and WWW
2026 abstracts are not yet in the selection source — so any per-year row reaching them is
under-represented by construction. Methodology and limitations are at the end of this
section.

**This page's population is not a schema field.** The extraction records what a paper
//did//, not what was done //to// it, so "our crawler was blocked" appears nowhere in it.
The population here is a full-text sweep over the papers' own sentences, and **every one
of its 138 candidates was then audited individually against that paper's full text**,
because the sweep on its own is only 31.2% precise for the claim it is used to make. Who
did that auditing, and what it does and does not establish, is on
[[provenance:programming:crawler_detection]].

==== Almost nobody reports being blocked ====

^ Verdict ^ Papers ^ Share of the 138 candidates ^ Share of the 1,120 crawled papers ^
| **Reports its own crawl obstructed, with a number** | 22 | 15.9% | **2.0%** |
| Reports its own crawl obstructed, no number | 21 | 15.2% | 1.9% |
| Describes mitigations, never says whether it happened | 38 | 27.5% | 3.4% |
| Sweep false positive — not about the authors' own collection | 57 | 41.3% | — |

So **43 of 1,120 crawling papers (3.8%) say their own crawl was obstructed, and 22
(2.0%) put a number on it.** Counting generously, 81 papers (7.2%) engage with the
question at all — the other 92.8% neither report obstruction nor say they took steps
against it.

Those are floors, not estimates, because a sweep can only find a paper that wrote the
sentence. To bound the gap, 24 crawling papers that the tight sweep did //not// flag but
a separate, much wider **recall probe** did were audited the same way: **6 of the 24 were misses**
(3 of them quantified). Extrapolating over the 382 papers the recall probe hits gives
roughly 96 missed papers, with a Wilson 95% interval of 46–172. **So the honest statement
is: at least 3.8% of crawling papers report obstruction, and the true share is plausibly
8–19%.** Even the top of that range leaves four in five crawling papers silent, and even
19% is a floor — a crawl that was blocked and never wrote a sentence about it is
invisible to both sweeps.

==== The papers that put a number on it ====

**Nineteen of the 22**, plus two more (marked †) that the recall audit turned up outside
the 138 candidates and that are therefore //not// counted in the 22. Read the
denominators. These are **not comparable with one another** — they are answers to
different questions on different populations, and averaging them would be meaningless.
What they are useful for is calibration: the plausible range on a general web crawl is
roughly 1–15%, and the outliers are populations that are adversarial by construction.

^ Paper ^ What was obstructed ^ Figure, with its denominator ^
| Kaizer and Gupta, IMC 2016 {[kaizer2016_characterizing]} | crawler blocked, or login required CAPTCHA | 75 of 420 selected sites (~17.9%) dropped |
| DeBlasio et al., IMC 2017 {[deblasio2017_tripwire]} | registration forms behind a human check | 19% of sites with a registration form (37% within the top-100 subset) |
| Ardi and Calder, IMC 2023 {[ardi2023_prevalence]} | bot-detection service prevented page load | 8% of 994 responsive sites in a CrUX top-1K crawl |
| Liu et al., IMC 2025 {[liu2025_somesite]} | site blocks the crawler regardless of user-agent | **1,487 of the top 10,000 sites (15%)** |
| Annamalai et al., NDSS 2024 {[annamalai2024_fpfed]} | crawl failed to collect traces | 1,700 of 20,000 sites (8.5%) failed; 64.3% of those were HTTP 403 |
| Senol et al., TheWebConf 2024 {[senol2024_double]} | feature collection incomplete | 93 of 5,000 labelled pages (1.9%) |
| Annamalai et al., TheWebConf 2025 {[annamalai2025_beyond]} | automated crawler failed to visit | 15 of ~3,000 sites (~1%); 86.7% of those returned 4XX tied to bot detection |
| Senol et al., USENIX Sec 2022 {[senol2022_leaky]} | CAPTCHA page blocked the crawler | 3 pages in a 1,000-site pilot crawl |
| Aziz et al., PETS 2024 {[aziz2024_johnny]} | CAPTCHA prevented normal load | 2 of 200 re-visited sites (1%); the paper estimates ~1% of the full sample |
| † Hausladen et al., USENIX Sec 2025 {[hausladen2025_websites]} | human-check page | 1.6–2.6% of the crawl set, across three crawls |
| † Bekos et al., TheWebConf 2023 {[bekos2023_hitchhiker]} | access blocked after an event | 172 of 2,308 sites (7.5%) — but the paper's own sentence reads "blocked our access after a specific event, //or// stored duplicate results", so blocking is not isolated |
| Kieserman et al., PETS 2025 {[kieserman2025_tracker]} | bot detection caused missed detections | 8.6% of the manually validated false-negative sample |
| Al Roomi and Li, USENIX Sec 2023 {[alroomi2023_login]} | CAPTCHA-solving failures during login | **52% of domains where an account had been created** could not be analysed |
| Gavazzi et al., USENIX Sec 2023 {[gavazzi2023_multi]} | login CAPTCHAs; web driver detected | 43 of 56 sites needed manual login; 3 of 56 blocked the driver |
| Campobasso and Allodi, CCS 2020 {[campobasso2020_impersonation]} | scraping accounts banned by the marketplace | 5 accounts; the paper does not state how many it held in total |
| Chehade et al., IEEE S&P 2025 {[chehade2025_forbidden]} | accounts blocked, sessions invalidated | 3 of 110 sites with test accounts |
| McDonald et al., IMC 2018 {[mcdonald2018_forbidden]} | crawler flagged as a bot, denied access | ~30% of Akamai 403s were **false positives** — the crawler was blocked where a browser was not |
| Szurdi et al., TheWebConf 2021 {[szurdi2021_where]} | crawler persona vs human persona | explicitly blocked +5 points, covertly blocked +8 points |
| Bijmans et al., USENIX Sec 2021 {[bijmans2021_catching]} | phishing sites cloaked against the crawler | **946 of 1,363 detected phishing domains (69%)** served a blank page |
| Oh et al., PETS 2017 {[oh2017_fingerprinting]} | Google CAPTCHA during Tor collection | 54% of fingerprintable keyword groups vs 35% of others |
| Singh et al., USENIX Sec 2017 {[singh2017_torexit]} | Tor exit relays discriminated against | 50% of the most aggressive site categories blocked over 60% of relays |

((The three members of the 22 that are **not** in the table are left out because their populations are too specialised to calibrate against: a WHOIS bulk-query study (~7.5% of queried domains failed after three attempts, attributed to per-IP rate limiting), a version-disclosure scan of Joomla and phpMyAdmin servers (17 of 98 and 18 of ~100 blocked the disclosing paths), and a censorship-probe-list study whose ground-truth control set was 50 URLs (3 of them 403'd a ''curl'' client). ''report_crawler_detection.mjs'' §D prints all 22 with their quotes and §D2 prints the recall-audit ones separately.))

Two entries in that table deserve to be read twice. Al Roomi and Li lost **52%** of the
domains they had already created accounts on, mostly to CAPTCHA-solving failures, and
said so — which is why their prevalence figure comes with the honest caveat that the
true value may be higher. Bijmans et al.'s **69%** is what a fully adversarial population
looks like: on phishing infrastructure, cloaking against the crawler is the default, not
the exception.

==== Over time: not much of a trend ====

^ Period ^ Crawling papers ^ Report obstruction ^ Share ^
| 2010–2019 | 427 | 14 | 3.3% |
| 2020–2024 | 495 | 19 | 3.8% |
| 2025–2026 //(provisional)// | 198 | 10 | 5.1% |

**Resist reading that as a rise.** Per year the series is noisy on small bases: 2015 is
the corpus's highest at 9.8% (4 of 41 crawling papers), 2017 and 2025 tie at 7.8%, and
2010–2013 are flat zero. The 2025–2026 row is also the provisional one, and 2025 is the
year the sweep flags the most candidates (29 of 129) — which is as consistent with the
field //discussing// this more as with anyone being blocked more. The corpus cannot
separate those two explanations and this page does not claim to. What the series does
show, unambiguously, is that no year in seventeen exceeded one paper in ten.

==== The choices that decide detectability are the least-reported ones ====

Population: the 1,120 crawling papers. ''not-stated'' is counted as what it is.

^ Choice ^ Field ^ Papers stating it ^ Share ^
| Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** |
| Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% |
| What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% |
| Which browser | ''crawlConfig.browsers'' | 529 | 47.2% |
| How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% |
| How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% |

**Headless versus headful is the most detectability-relevant single bit in the whole
table and the least often reported.** Of the 1,080 crawling papers that have a crawl
configuration recorded at all, 940 do not say; 93 say headless, 35 headful, 12 both.

==== The literature on each signal is thinner than you would guess ====

A full-text sweep over all 5,859 papers for the signal's name. These are **mention**
counts, not usage counts — a paper naming ''navigator.webdriver'' may be describing it,
defending against it or checking for it — and unlike the table above they are **not**
hand-audited. They are here to show where there is a literature and where there is not.

^ Signal named anywhere in the paper ^ Papers ^ Of which ran a crawl ^
| a CAPTCHA or challenge product by name | 118 | 41 |
| a bot-management vendor by name | 106 | 38 |
| TLS fingerprinting (JA3/JA4) | 37 | 11 |
| fingerprint inconsistency | 24 | 10 |
| headless detection | 20 | 19 |
| a stealth patch | 15 | 13 |
| ''navigator.webdriver'' | 11 | 11 |
| an AI or LLM crawler | 5 | 3 |
| CDP or DevTools artefacts | 4 | 1 |
| HTTP header-order fingerprinting | 3 | 2 |

Read the bottom half of that table as a map of what nobody has written. **Four papers in
5,859 mention CDP artefacts as a detection surface, and three mention header-order
fingerprinting** — both of which are ordinary, documented techniques on the commercial
side. All five AI-crawler papers are 2025 or later.

==== Methodology and limitations of these figures ====

  * **How the population was built.** No schema field records being blocked, so the population is a full-text sweep over ''paper.cols.txt'' with whitespace collapsed — a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten ordered patterns, each either anchored on an obstruction noun or requiring a first-person subject in the same sentence. Take the same ten patterns with every first-person requirement removed and the sweep returns **406** papers instead of 138; the extra 268 were not hand-audited, but a read of a sample of them found adversaries evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking requests, which is what the anchors are there to exclude.
  * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
  * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
  * **Papers are counted, never tuples**, and a paper counts once.
  * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.
  * **Venue coverage.** Seven venues only. Several relevant papers were published outside them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the field.
  * **Every query, the report script's unedited output, the full candidate list with its verdicts, the residue and the external-source checks are on [[provenance:programming:crawler_detection]].** Corpus-level caveats are on [[literature:corpus]].

===== What to Report =====

  - **Your non-load rate, broken down by cause.** One table. Network error, DNS, 4xx, 5xx, timeout, challenge page, empty render. This is the single highest-value thing on the list, and the closest measurable proxy for how rare it is — the share of crawling papers that report obstruction at all — is 3.8%.
  - **Whether you counted challenge pages as successes.** If your success test is the HTTP status code, say so, because it means you did.
  - **Headless or headful, and which binary.** Since Chrome 132 that is two questions, not one. 12.5% of papers answer even the first.
  - **Your vantage point and its address space** — cloud, university, residential, Tor. See [[Design:Crawling location]].
  - **Whether you attempted evasion.** Either answer is fine; the absence of an answer is not. If you did, name the tool //and its version//, and justify it against [[Practices:Ethics]].
  - **Any control condition you ran** — a second configuration, a second vantage point, a manual sample — and what the delta was. A null delta is a result worth a sentence.
  - **The date.** Bot-management deployment moved materially in 2025 (see above). A block rate without a date is not comparable with anything.

===== Papers to Read First =====

If you have an afternoon, in this order:

  - **Ahmad et al., TheWebConf 2020** {[ahmad2020_apophanies]} — why crawler design is a threat to validity and not an implementation detail. Read this one first even if you read nothing else.
  - **Jueckstock et al., TheWebConf 2021** {[jueckstock2021_realistic]} — the same argument made quantitatively, over crawl configuration and vantage point together.
  - **Invernizzi et al., IEEE S&P 2016** {[invernizzi2016_cloak]} — the canonical measurement of sites serving machines a different web.
  - **Vastel et al., USENIX Sec 2018** {[vastel2018_scanner]} — how automation is actually detected in the browser, and why inconsistency beats any single attribute.
  - **Venugopalan et al., IMC 2025** {[venugopalan2025_inconsistent]} — the current state of that method, measured against two commercial detectors.
  - **Liu et al., IMC 2025** {[liu2025_somesite]} — the AI-crawler turn, and a clean example of the control-condition design this page recommends.
  - **Krumnow et al., CoNEXT 2022** {[krumnow2022_gullible]} — what it looks like when somebody writes a detector for the tool //you// are using.

===== Open Questions =====

<WRAP todo>
  * **Nobody has measured bot-management deployment against a site-popularity distribution.** The bias this page argues for — that blocking concentrates on high-ranked sites — is inferred from where the products are sold and from the shape of individual papers' losses. A crawl that reports its block rate //by rank decile// would settle it, and would cost one extra column.
  * **The published figures are not comparable and nobody has made them so.** Twenty-two quantified figures, twenty-two denominators, no two of them the same population. A single study crawling one sample under several configurations and reporting one block rate per configuration would give the field a reference point it does not have.
  * **How much of a "difference between crawls" result is bot management** rather than the thing being studied. Demir et al. {[demir2022_reproducibility]} {[demir2023_similarity]} measured that repeated and differently configured crawls disagree; how much of that disagreement is the site discriminating, rather than the web changing, is unmeasured.
  * **The AI-crawler defences are a natural experiment nobody has run.** Cloudflare changed a default on a known date in July 2025. Whether academic crawlers' block rates moved across that date is answerable from any longitudinal crawl that spans it.
  * **There is no shared challenge-marker set, and there should be.** Every paper that detects challenge pages writes its own string list, none publishes it as a maintained artefact, and none reports its precision. A curated, versioned list — with per-vendor markers and a recall figure — is a small artefact that would make every non-load table on this page comparable with every other.
  * **Signed bot identity.** If the IETF webbotauth work lands, a research crawler could declare itself and be admitted on those terms. Whether operators would admit academic crawlers, and on what conditions, is a question for a survey of operators, not for this corpus.
</WRAP>

===== Related Pages =====

  * [[Programming:Crawler]] — which library drives the browser, and how visible each one is.
  * [[Programming:Crawler:OpenWPM]] — the specific tool with a specific published detector.
  * [[Programming:Crawler:LLM Agents]] — a different detection surface, different rules.
  * [[Design:Crawling location]] — the vantage point, which is the first detection input.
  * [[Programming:Interaction]] — behavioural signals, from the other direction.
  * [[Programming:Stateful Stateless]] — statefulness is both a design choice and a signal.
  * [[Practices:Ethics]] — where the evasion decision belongs.
  * [[Design:Automated Measurements]] — the surrounding methodology.
  * [[Privacy:Fingerprinting]] — the same techniques, pointed at users instead of crawlers.
  * [[provenance:programming:crawler_detection]] — every query, script output and source check behind this page.

====== References ======

<bibtex bibliography></bibtex>

~~DISCUSSION~~