A crawl that is blocked does not fail loudly. It returns a page, a status code and a timestamp, and your pipeline records all three. What it does not record is that the page you got is not the page a person gets — and every number you compute downstream inherits that difference without carrying a flag for it.

This page is about that difference: what makes an automated client identifiable, what happens when it is identified, and what a reviewer will expect you to have measured. It is not an anti-detection cookbook. Evading a site's access controls is a research-ethics decision before it is an engineering one, and it belongs on Ethics. The advice here points the other way: measure the obstruction and report it, because the obstruction is data about your sample, and the sample is what your paper is actually about.

Three neighbours cover adjacent ground. Crawler compares the libraries that drive a browser and how visible each is. Crawling location covers the one detection input you choose before you write any code — where your packets come from. LLM Agents covers a different surface entirely: an LLM-driven agent is detected, and blocked, by mechanisms aimed at a category that did not exist when most of this literature was written.

The reason to care is not that you lose sites. It is which sites you lose.

Bot management is not deployed at random. It is bought by sites with something to protect and the budget to protect it: large e-commerce, banking, ticketing, travel, social platforms, anything behind an enterprise CDN. Those are disproportionately the high-ranked, high-traffic sites — the ones your top-N list oversamples on purpose, and the ones whose behaviour your paper is most likely to be about. A 10% non-load rate that falls entirely on the top 2,000 sites is not 10% noise; it is a systematic hole exactly where your signal is.

That last step is an inference, not a measurement, and this page is careful about saying so: no paper in this corpus reports a block rate broken down by popularity rank. It follows from where the products are sold and it is consistent with the shape of the losses individual papers report, and it is the first of the open questions below. What is measured is everything after it.

Ahmad et al. [1Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)] make the general form of this argument: what a crawler sees is a function of how the crawler is built, and conclusions drawn from crawl data can be artefacts of the instrument. Jueckstock et al. [2Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)] measured the specific version of it — crawl configuration and vantage point change what the web shows you — and Zeber et al. [3Zeber, David; Bird, Sarah; Oliveira, Camila; Rudametkin, Walter; Segall, Ilana; Wolls´en, Fredrik; Lopatka, Martin (2020): "The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing", in: Proceedings of The Web Conference 2020, pp. 167–178. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] measured how far automated crawls diverge from real human browsing. Annamalai et al. [4Annamalai, Meenatchi Sundaram Muthu Selva; De Cristofaro, Emiliano; Bilogrevic, Igor (2025): "Beyond the Crawl: Unmasking Browser Fingerprinting in Real User Interactions", in: Proceedings of the ACM Web Conference. (DOI)] ran the comparison again in 2025 with real users alongside a crawler and found the crawler's failures concentrated in 4XX responses tied to bot detection.

Four shapes of obstruction, in increasing order of danger:

Cloaking is not hypothetical and it is not confined to phishing. Wang et al. [5Wang, David Y.; Savage, Stefan; Voelker, Geoffrey M. (2011): "Cloak and dagger: dynamics of web search cloaking", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] characterised search cloaking as far back as 2011; Invernizzi et al. [6Invernizzi, Luca; Thomas, Kurt; Kapravelos, Alexandros; Comanescu, Oxana; Picod, Jean-Michel; Bursztein, Elie (2016): "Cloak of Visibility: Detecting When Machines Browse a Different Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] built the canonical measurement of it, showing systematic serving of different content to machines; Zhang et al. [7Zhang, Penghui; Oest, Adam; Cho, Haehyun; Sun, Zhibo; Johnson, RC; Wardman, Brad; Sarker, Shaown; Kapravelos, Alexandros; Bao, Tiffany; Wang, Ruoyu; Shoshitaishvili, Yan; Doupé, Adam; Ahn, Gail-Joon (2021): "CrawlPhish: Large-scale Analysis of Client-side Cloaking Techniques in Phishing", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] measured client-side cloaking specifically aimed at security crawlers; and Szurdi et al. [8Szurdi, Janos; Luo, Meng; Kondracki, Brian; Nikiforakis, Nick; Christin, Nicolas (2021): "Where are you taking me?Understanding Abusive Traffic Distribution Systems", in: Proceedings of the ACM Web Conference. (DOI)] showed the quantitative form directly — driving the same traffic distribution systems with a crawler persona and a human persona, the crawler was explicitly blocked 5 percentage points more often, and covertly blocked — sent to a parked or error page — at least 8 points more often. The covert number is larger than the overt one. That is the whole problem in one measurement.

A bot-management product does not run one test. It scores a request against signals from several layers at once, and the layers differ in how cheap they are to read and how much you can do about them. Ordered from the network up:

This is the layer the literature has actually studied, and the layer that has moved most.

• navigator.webdriver is a specification, not a bug. WebDriver Level 1 has been a W3C Recommendation since 05 June 2018 and defines the property as returning true when the browser's webdriver-active flag is set; Level 2 is still a Working Draft, dated 02 July 2026, and carries the same definition.¹⁾ Any Selenium, Puppeteer or Playwright session sets it by default. It is the single cheapest automation check in existence and the field barely discusses it: 11 papers in the whole corpus name it.
• Chrome still ships the automation switch. –enable-automation and the “controlled by automated test software” infobar are both present in the Chromium tree today.²⁾
• “Headless” no longer means one thing. Since Chrome 132 the old headless implementation is available only as a separate chrome-headless-shell binary; plain –headless now selects a unified mode that shares its code with headful Chrome.³⁾ Playwright still ships a regular Chromium for headed runs and a separate headless shell for headless ones.⁴⁾ So “we ran headless Chrome” is now an ambiguous sentence, and which binary you meant changes how detectable you were. Only 12.5% of crawling papers say headless or headful at all.
• Fingerprint inconsistency beat fingerprint content. Vastel et al. [9Vastel, Antoine; Laperdrix, Pierre; Rudametkin, Walter; Rouvoy, Romain (2018): "Fp-Scanner: The Privacy Implications of Browser Fingerprint Inconsistencies", in: Proceedings of the USENIX Security Symposium. (Link)] established the method in 2018: an instrumented or spoofed browser is identified not by any one attribute but by attributes that contradict each other. Wu et al. [10Wu, Shujiang; Sun, Pengfei; Zhao, Yao; Cao, Yinzhi (2023): "Him of Many Faces: Characterizing Billion-scale Adversarial and Benign Browser Fingerprints on Commercial Websites", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] characterised adversarial fingerprints at billion-request scale on commercial sites, and Venugopalan et al. [11Venugopalan, Hari; Munir, Shaoor; Ahmed, Shuaib; Wang, Tangbaihe; King, Samuel T.; Shafiq, Zubair (2025): "FP-Inconsistent: Measurement and Analysis of Fingerprint Inconsistencies in Evasive Bot Traffic", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] ran the current version of the measurement: a honey site behind two commercial anti-bot services (DataDome and BotD), fed by 20 commercial bot services selling “undetectable” traffic, over half a million requests. This is the 2025 state of the art and it is an inconsistency detector.
• Framework-specific detection exists. Krumnow et al. [12Krumnow, Benjamin; Jonker, Hugo; Karsch, Stefan (2022): "How gullible are web measurement tools?", in: Proceedings of the 18th International Conference on Emerging Networking EXperiments and Technologies, pp. 171-186. (DOI)] studied how detectable OpenWPM specifically is, and how widely OpenWPM detection is deployed. A crawler that many papers share is worth writing a signature for.

The corpus rewards methods that were written up, which biases any ranking toward whatever was fashionable mid-window. Dating each strand explicitly:

The last two rows rest on the corpus's thinnest years — 2025 and 2026 are provisional here (see Use in Publications below) — so treat them as a direction of travel rather than as settled counts. But the direction is not ambiguous: Teoh et al. [21Teoh, Xiwen; Lin, Yun; Li, Siqi; Liu, Ruofan; Sollomoni, Avi; Harel, Yaniv; Dong, Jin Song (2025): "Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model", in: Proceedings of the USENIX Security Symposium. (Link)] solved visual CAPTCHAs with an agentic vision-language model, and Qi et al. [22Qi, Minfeng; He, Dongyang; Wang, Qin; Zhang, Lefeng (2026): "VIPER Strike: Defeating Visual Reasoning CAPTCHAs via Structured Vision–Language Inference", in: Proceedings of the USENIX Security Symposium. (Link)] did the same for visual reasoning CAPTCHAs. Meanwhile Searles et al. [20Searles, Andrew; Nakatsuka, Yoshimichi; Ozturk, Ercan; Paverd, Andrew; Tsudik, Gene; Enkoji, Ai (2023): "An Empirical Study & Evaluation of Modern CAPTCHAs", in: Proceedings of the USENIX Security Symposium. (Link)] measured what CAPTCHAs cost the humans — 1,400 participants solving 14,000 CAPTCHAs, with solving time and preference varying sharply by type. Read together, those are the two halves of why a site reserves a challenge for a suspicious score rather than showing it to everyone. Nguyen et al. [26Nguyen, Hoang Dai; Subramani, Karthika; Acharya, Bhupendra; Perdisci, Roberto; Vadrevu, Phani (2024): "C-Frame: Characterizing and measuring in-the-wild CAPTCHA attacks", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] measured in-the-wild CAPTCHA abuse from the other direction. Designing a study around “the CAPTCHA will stop the adversary” is dated; designing one around “the CAPTCHA will stop us” is still true, and is the reason it costs you sites.

Your crawler cannot ask why it failed. It can only classify what came back. These are the discriminable cases:

The engineering instinct is to become harder to detect. The research instinct should be to make the obstruction a measured quantity. Four reasons the second is better:

1. The arms race is unwinnable and, worse, silent. You will not be told when a patch stops working. A result that depends on evasion has no error bar and no way to acquire one.
2. It is an ethics decision. Deliberately overriding an operator's expressed access preference needs a justification you are prepared to write in the paper. See Ethics.
3. The obstruction is itself publishable. Every quantified figure in the table below is a finding somebody got for free by counting what they lost.
4. It is the cheapest possible defence against the obvious objection. This page's own figures show the bar has been low — 96.2% of crawling papers report no obstruction at all — but a reviewer who asks “how do you know the sites you lost were lost at random?” is asking something you cannot answer after the crawl is over. The breakdown costs one table and has to be collected while you crawl.

The minimum viable version costs almost nothing:

1. Classify every non-success into network error, DNS failure, HTTP 4xx, HTTP 5xx, timeout, challenge page, empty render. Publish the table.
2. Add a challenge-page detector to the success path, not just to the error path, so HTTP 200 challenges stop counting as visits.
3. Run a control condition: a small sample crawled under a second configuration, same day, same URLs. Report the delta.
4. Say in the paper whether you attempted evasion, and what you did. “We made no attempt to evade bot detection” is a perfectly good sentence and it tells the reader what the number means.

The corpus's own stealth-tool trail is short and recent: 11 papers name a stealth patch, 9 of them ran a crawl, and none is earlier than 2021. Anyone following that trail today should know the state of what it points at, because the names in those papers have moved:

⁵⁾

Two things this table is here to teach, neither of them about stealth. First, a package's last release date is not its currency, and it fails in two different directions here: undetected-chromedriver looks alive on GitHub but its own author's successor is a differently-named project, while playwright-stealth looks stale under its original repository and is in fact maintained — the package name stayed put and the code moved to a different owner. Checking the name you already had would have got both wrong. Second, if you cite one of these in a paper, cite the version you ran, because the detection landscape it was written against no longer exists.

Useful mostly so you can recognise a challenge page and name the vendor in your methodology section. The market has consolidated hard, and papers that cite a vendor by its 2019 name are citing something that no longer exists under it:

⁶⁾

Everything above is about a site deciding whether a client is a human. Since 2025 there is a second question layered on top, with different economics: whether a client is an AI training or inference crawler, and whether it has paid. This matters to an academic crawler for a blunt reason — the defences aimed at AI crawlers do not distinguish you from them.

• On 2025-07-01 Cloudflare changed its default to blocking AI crawlers unless they pay for the content, and introduced a pay-per-crawl mechanism.⁷⁾ If any of your sample sits behind Cloudflare, that default is part of your methodology whether you engaged with it or not.
• The corpus sees this arriving and only just: exactly five papers in 5,859 mention an AI crawler or name an AI crawler user-agent token, and all five are 2025 or 2026. Liu et al. [23Liu, Enze; Luo, Elisa; Shan, Shawn; Voelker, Geoffrey M.; Zhao, Ben Y.; Savage, Stefan (2025): "Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] measured how well content creators can actually keep AI crawlers out; Cui et al. [24Cui, Jian; Zha, Mingming; Wang, XiaoFeng; Liao, Xiaojing (2025): "The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] measured what LLM services do with robots.txt; Kim et al. [25Kim, Taein; Bock, Karstan; Luo, Claire; Liswood, Amanda; Poroslay, Chloe; Wenger, Emily (2025): "Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] measured whether scrapers respect its directives at all.
• And the defaults are still moving. Cloudflare has since replaced the single “Block AI bots” toggle with three behaviour categories — Search, Agent and Training — and its own documentation states that from 2026-09-15 new domains default to blocking Training and Agent on pages that display ads, while Search stays allowed; the old “Block AI bots” option is marked deprecating on the same date.⁸⁾ This page was written 17 days before that date. If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in Agent is on the blocked side of the new default, and nothing about being academic exempts it.
• robots.txt itself is RFC 9309, a Proposed Standard from September 2022, and nothing obsoletes or updates it.⁹⁾ It expresses a preference; it does not enforce one, and it is not what blocks you.
• Two IETF working groups are now chartered in this space and are worth watching rather than citing: webbotauth, on cryptographically authenticating automated clients and identifying their operators, and aipref, on a vocabulary and attachment mechanism for AI-usage preferences.¹⁰⁾ If signed bot identity lands, “declare yourself and be allowed in” becomes a real option for research crawlers, and this page will need rewriting.

An LLM-driven agent browsing on a user's behalf is a third category again, detected by different signals and blocked under different rules. That is LLM Agents, not this page.

The figures below come from a structured extraction over 5,859 full-text papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Unless stated otherwise the population is the 1,120 papers that ran a crawl. The 2025 and 2026 venue-years are provisional — CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source — so any per-year row reaching them is under-represented by construction. Methodology and limitations are at the end of this section.

This page's population is not a schema field. The extraction records what a paper did, not what was done to it, so “our crawler was blocked” appears nowhere in it. The population here is a full-text sweep over the papers' own sentences, and every one of its 138 candidates was then read by hand, because the sweep on its own is only 31.2% precise for the claim it is used to make.

So 43 of 1,120 crawling papers (3.8%) say their own crawl was obstructed, and 22 (2.0%) put a number on it. Counting generously, 81 papers (7.2%) engage with the question at all — the other 92.8% neither report obstruction nor say they took steps against it.

Those are floors, not estimates, because a sweep can only find a paper that wrote the sentence. To bound the gap, 24 crawling papers that the tight sweep did not flag but a separate, much wider recall probe did were audited the same way: 6 of the 24 were misses (3 of them quantified). Extrapolating over the 382 papers the recall probe hits gives roughly 96 missed papers, with a Wilson 95% interval of 46–172. So the honest statement is: at least 3.8% of crawling papers report obstruction, and the true share is plausibly 8–19%. Even the top of that range leaves four in five crawling papers silent, and even 19% is a floor — a crawl that was blocked and never wrote a sentence about it is invisible to both sweeps.

Nineteen of the 22, plus two more (marked †) that the recall audit turned up outside the 138 candidates and that are therefore not counted in the 22. Read the denominators. These are not comparable with one another — they are answers to different questions on different populations, and averaging them would be meaningless. What they are useful for is calibration: the plausible range on a general web crawl is roughly 1–15%, and the outliers are populations that are adversarial by construction.

¹¹⁾

Two entries in that table deserve to be read twice. Al Roomi and Li lost 52% of the domains they had already created accounts on, mostly to CAPTCHA-solving failures, and said so — which is why their prevalence figure comes with the honest caveat that the true value may be higher. Bijmans et al.'s 69% is what a fully adversarial population looks like: on phishing infrastructure, cloaking against the crawler is the default, not the exception.

Resist reading that as a rise. Per year the series is noisy on small bases: 2015 is the corpus's highest at 9.8% (4 of 41 crawling papers), 2017 and 2025 tie at 7.8%, and 2010–2013 are flat zero. The 2025–2026 row is also the provisional one, and 2025 is the year the sweep flags the most candidates (29 of 129) — which is as consistent with the field discussing this more as with anyone being blocked more. The corpus cannot separate those two explanations and this page does not claim to. What the series does show, unambiguously, is that no year in seventeen exceeded one paper in ten.

Population: the 1,120 crawling papers. not-stated is counted as what it is.

Headless versus headful is the most detectability-relevant single bit in the whole table and the least often reported. Of the 1,080 crawling papers that have a crawl configuration recorded at all, 940 do not say; 93 say headless, 35 headful, 12 both.

A full-text sweep over all 5,859 papers for the signal's name. These are mention counts, not usage counts — a paper naming navigator.webdriver may be describing it, defending against it or checking for it — and unlike the table above they are not hand-audited. They are here to show where there is a literature and where there is not.

Read the bottom half of that table as a map of what nobody has written. Four papers in 5,859 mention CDP artefacts as a detection surface, and three mention header-order fingerprinting — both of which are ordinary, documented techniques on the commercial side. All five AI-crawler papers are 2025 or later.

• How the population was built. No schema field records being blocked, so the population is a full-text sweep over paper.cols.txt with whitespace collapsed — a PDF line break inside a phrase defeats every multi-word pattern otherwise. Ten ordered patterns, each either anchored on an obstruction noun or requiring a first-person subject in the same sentence. Take the same ten patterns with every first-person requirement removed and the sweep returns 406 papers instead of 138; the extra 268 were not hand-audited, but a read of a sample of them found adversaries evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking requests, which is what the anchors are there to exclude.
• A sweep hit is a candidate, not a claim. All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for “reports obstruction”, 58.7% for “on topic at all” — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
• Recall is estimated, not measured. The recall probe is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside bWAPP, a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
• Quotes were checked. All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Below threshold is not “unsupported” — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
• Papers are counted, never tuples, and a paper counts once.
• Silence is not absence. “Does not report obstruction” means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.
• Venue coverage. Seven venues only. Several relevant papers were published outside them — Krumnow et al. at CoNEXT, for one — so any count here is a lower bound on the field.
• Every query, the report script's unedited output, the full candidate list with its verdicts, the residue and the external-source checks are on crawler_detection. Corpus-level caveats are on corpus.

1. Your non-load rate, broken down by cause. One table. Network error, DNS, 4xx, 5xx, timeout, challenge page, empty render. This is the single highest-value thing on the list, and the closest measurable proxy for how rare it is — the share of crawling papers that report obstruction at all — is 3.8%.
2. Whether you counted challenge pages as successes. If your success test is the HTTP status code, say so, because it means you did.
3. Headless or headful, and which binary. Since Chrome 132 that is two questions, not one. 12.5% of papers answer even the first.
4. Your vantage point and its address space — cloud, university, residential, Tor. See Crawling location.
5. Whether you attempted evasion. Either answer is fine; the absence of an answer is not. If you did, name the tool and its version, and justify it against Ethics.
6. Any control condition you ran — a second configuration, a second vantage point, a manual sample — and what the delta was. A null delta is a result worth a sentence.
7. The date. Bot-management deployment moved materially in 2025 (see above). A block rate without a date is not comparable with anything.

If you have an afternoon, in this order:

1. Ahmad et al., TheWebConf 2020 [1Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)] — why crawler design is a threat to validity and not an implementation detail. Read this one first even if you read nothing else.
2. Jueckstock et al., TheWebConf 2021 [2Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)] — the same argument made quantitatively, over crawl configuration and vantage point together.
3. Invernizzi et al., IEEE S&P 2016 [6Invernizzi, Luca; Thomas, Kurt; Kapravelos, Alexandros; Comanescu, Oxana; Picod, Jean-Michel; Bursztein, Elie (2016): "Cloak of Visibility: Detecting When Machines Browse a Different Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] — the canonical measurement of sites serving machines a different web.
4. Vastel et al., USENIX Sec 2018 [9Vastel, Antoine; Laperdrix, Pierre; Rudametkin, Walter; Rouvoy, Romain (2018): "Fp-Scanner: The Privacy Implications of Browser Fingerprint Inconsistencies", in: Proceedings of the USENIX Security Symposium. (Link)] — how automation is actually detected in the browser, and why inconsistency beats any single attribute.
5. Venugopalan et al., IMC 2025 [11Venugopalan, Hari; Munir, Shaoor; Ahmed, Shuaib; Wang, Tangbaihe; King, Samuel T.; Shafiq, Zubair (2025): "FP-Inconsistent: Measurement and Analysis of Fingerprint Inconsistencies in Evasive Bot Traffic", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] — the current state of that method, measured against two commercial detectors.
6. Liu et al., IMC 2025 [23Liu, Enze; Luo, Elisa; Shan, Shawn; Voelker, Geoffrey M.; Zhao, Ben Y.; Savage, Stefan (2025): "Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] — the AI-crawler turn, and a clean example of the control-condition design this page recommends.
7. Krumnow et al., CoNEXT 2022 [12Krumnow, Benjamin; Jonker, Hugo; Karsch, Stefan (2022): "How gullible are web measurement tools?", in: Proceedings of the 18th International Conference on Emerging Networking EXperiments and Technologies, pp. 171-186. (DOI)] — what it looks like when somebody writes a detector for the tool you are using.

• Crawler — which library drives the browser, and how visible each one is.
• OpenWPM — the specific tool with a specific published detector.
• LLM Agents — a different detection surface, different rules.
• Crawling location — the vantage point, which is the first detection input.
• Interaction — behavioural signals, from the other direction.
• Stateful Stateless — statefulness is both a design choice and a signal.
• Ethics — where the evasion decision belongs.
• Automated Measurements — the surrounding methodology.
• Fingerprinting — the same techniques, pointed at users instead of crawlers.
• crawler_detection — every query, script output and source check behind this page.

[1]

Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)

[2]

Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)

[3]

Zeber, David; Bird, Sarah; Oliveira, Camila; Rudametkin, Walter; Segall, Ilana; Wolls´en, Fredrik; Lopatka, Martin (2020): "The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing", in: Proceedings of The Web Conference 2020, pp. 167–178. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[4]

Annamalai, Meenatchi Sundaram Muthu Selva; De Cristofaro, Emiliano; Bilogrevic, Igor (2025): "Beyond the Crawl: Unmasking Browser Fingerprinting in Real User Interactions", in: Proceedings of the ACM Web Conference. (DOI)

[5]

Wang, David Y.; Savage, Stefan; Voelker, Geoffrey M. (2011): "Cloak and dagger: dynamics of web search cloaking", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[6]

Invernizzi, Luca; Thomas, Kurt; Kapravelos, Alexandros; Comanescu, Oxana; Picod, Jean-Michel; Bursztein, Elie (2016): "Cloak of Visibility: Detecting When Machines Browse a Different Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[7]

Zhang, Penghui; Oest, Adam; Cho, Haehyun; Sun, Zhibo; Johnson, RC; Wardman, Brad; Sarker, Shaown; Kapravelos, Alexandros; Bao, Tiffany; Wang, Ruoyu; Shoshitaishvili, Yan; Doupé, Adam; Ahn, Gail-Joon (2021): "CrawlPhish: Large-scale Analysis of Client-side Cloaking Techniques in Phishing", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[8]

Szurdi, Janos; Luo, Meng; Kondracki, Brian; Nikiforakis, Nick; Christin, Nicolas (2021): "Where are you taking me?Understanding Abusive Traffic Distribution Systems", in: Proceedings of the ACM Web Conference. (DOI)

[9]

Vastel, Antoine; Laperdrix, Pierre; Rudametkin, Walter; Rouvoy, Romain (2018): "Fp-Scanner: The Privacy Implications of Browser Fingerprint Inconsistencies", in: Proceedings of the USENIX Security Symposium. (Link)

[10]

Wu, Shujiang; Sun, Pengfei; Zhao, Yao; Cao, Yinzhi (2023): "Him of Many Faces: Characterizing Billion-scale Adversarial and Benign Browser Fingerprints on Commercial Websites", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[11]

Venugopalan, Hari; Munir, Shaoor; Ahmed, Shuaib; Wang, Tangbaihe; King, Samuel T.; Shafiq, Zubair (2025): "FP-Inconsistent: Measurement and Analysis of Fingerprint Inconsistencies in Evasive Bot Traffic", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[12]

Krumnow, Benjamin; Jonker, Hugo; Karsch, Stefan (2022): "How gullible are web measurement tools?", in: Proceedings of the 18th International Conference on Emerging Networking EXperiments and Technologies, pp. 171-186. (DOI)

[13]

Shi, Chenghui; Ji, Shouling; Liu, Qianjun; Liu, Changchang; Chen, Yuefeng; He, Yuan; Liu, Zhe; Beyah, Raheem; Wang, Ting (2020): "Text Captcha Is Dead? A Large Scale Deployment and Empirical Study", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[14]

Zhang, Penghui; Sun, Zhibo; Kyung, Sukwha; Behrens, Hans Walter; Basque, Zion Leonahenahe; Cho, Haehyun; Oest, Adam; Wang, Ruoyu; Bao, Tiffany; Shoshitaishvili, Yan; Ahn, Gail-Joon; Doupé, Adam (2022): "I'm SPARTACUS, No, I'm SPARTACUS: Proactively Protecting Users from Phishing by Intentionally Triggering Cloaking Behavior", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[15]

Maroofi, Sourena; Korczynski, Maciej; Duda, Andrzej (2020): "Are You Human?: Resilience of Phishing Detection to Evasion Techniques Based on Human Verification", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[16]

Teoh, Xiwen; Lin, Yun; Liu, Ruofan; Huang, Zhiyong; Dong, Jin Song (2024): "PhishDecloaker: Detecting CAPTCHA-cloaked Phishing Websites via Hybrid Vision-based Interactive Models", in: Proceedings of the USENIX Security Symposium. (Link)

[17]

Jan, Steve T. K.; Hao, Qingying; Hu, Tianrui; Pu, Jiameng; Oswal, Sonal; Wang, Gang; Viswanath, Bimal (2020): "Throwing Darts in the Dark? Detecting Bots with Limited Data using Neural Data Augmentation", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[18]

Li, Xigao; Azad, Babak Amin; Rahmati, Amir; Nikiforakis, Nick (2021): "Good Bot, Bad Bot: Characterizing Automated Browsing Activity", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[19]

Herley, Cormac (2022): "Automated Detection of Automated Traffic", in: Proceedings of the USENIX Security Symposium. (Link)

[20]

Searles, Andrew; Nakatsuka, Yoshimichi; Ozturk, Ercan; Paverd, Andrew; Tsudik, Gene; Enkoji, Ai (2023): "An Empirical Study & Evaluation of Modern CAPTCHAs", in: Proceedings of the USENIX Security Symposium. (Link)

[21]

Teoh, Xiwen; Lin, Yun; Li, Siqi; Liu, Ruofan; Sollomoni, Avi; Harel, Yaniv; Dong, Jin Song (2025): "Are CAPTCHAs Still Bot-hard? Generalized Visual CAPTCHA Solving with Agentic Vision Language Model", in: Proceedings of the USENIX Security Symposium. (Link)

[22]

Qi, Minfeng; He, Dongyang; Wang, Qin; Zhang, Lefeng (2026): "VIPER Strike: Defeating Visual Reasoning CAPTCHAs via Structured Vision–Language Inference", in: Proceedings of the USENIX Security Symposium. (Link)

[23]

Liu, Enze; Luo, Elisa; Shan, Shawn; Voelker, Geoffrey M.; Zhao, Ben Y.; Savage, Stefan (2025): "Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[24]

Cui, Jian; Zha, Mingming; Wang, XiaoFeng; Liao, Xiaojing (2025): "The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[25]

Kim, Taein; Bock, Karstan; Luo, Claire; Liswood, Amanda; Poroslay, Chloe; Wenger, Emily (2025): "Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[26]

Nguyen, Hoang Dai; Subramani, Karthika; Acharya, Bhupendra; Perdisci, Roberto; Vadrevu, Phani (2024): "C-Frame: Characterizing and measuring in-the-wild CAPTCHA attacks", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[27]

Kaizer, Andrew J.; Gupta, Minaxi (2016): "Characterizing Website Behaviors Across Logged-in and Not-logged-in Users", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[28]

DeBlasio, Joe; Savage, Stefan; Voelker, Geoffrey M.; Snoeren, Alex C. (2017): "Tripwire: inferring internet site compromise", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[29]

Ardi, Calvin; Calder, Matt (2023): "The Prevalence of Single Sign-On on the Web: Towards the Next Generation of Web Content Measurement", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[30]

Annamalai, Meenatchi Sundaram Muthu Selva; Bilogrevic, Igor; Cristofaro, Emiliano De (2024): "FP-Fed: Privacy-Preserving Federated Detection of Browser Fingerprinting", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[31]

Senol, Asuman; Ukani, Alisha; Cutler, Dylan; Bilogrevic, Igor (2024): "The Double Edged Sword: Identifying Authentication Pages and their Fingerprinting Behavior", in: Proceedings of the ACM Web Conference 2024. (DOI)

[32]

Senol, Asuman; Acar, Gunes; Humbert, Mathias; Zuiderveen Borgesius, Frederik (2022): "Leaky Forms: A Study of Email and Password Exfiltration Before Form Submission", in: 31st USENIX Security Symposium (USENIX Security 22). (Link)

[33]

Aziz, Muhammad Abu Bakar; Wilson, Christo (2024): "Johnny Still Can't Opt-out: Assessing the IAB CCPA Compliance Framework", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[34]

Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)

[35]

Bekos, Paschalis; Papadopoulos, Panagiotis; Markatos, Evangelos P.; Kourtellis, Nicolas (2023): "The Hitchhiker's Guide to Facebook Web Tracking with Invisible Pixels and Click IDs", in: Proceedings of the ACM Web Conference. (DOI)

[36]

Kieserman, Julia B.; Andreou, Athanasios; Geeng, Chris; Lauinger, Tobias; McCoy, Damon (2025): "Tracker Installations Are Not Created Equal: Understanding Tracker Configuration of Form Data Collection", in: Proceedings on Privacy Enhancing Technologies, pp. 679-695. (DOI)

[37]

Al Roomi, Suood; Li, Frank (2023): "A Large-Scale Measurement of Website Login Policies", in: Proceedings of the USENIX Security Symposium. (Link)

[38]

Gavazzi, Anthony; Williams, Ryan; Kirda, Engin; Lu, Long; King, Andre; Davis, Andy; Leek, Tim (2023): "A Study of Multi-Factor and Risk-Based Authentication Availability", in: Proceedings of the USENIX Security Symposium. (Link)

[39]

Campobasso, Michele; Allodi, Luca (2020): "Impersonation-as-a-Service: Characterizing the Emerging Criminal Infrastructure for User Impersonation at Scale", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[40]

Chehade, Saiid El Hajj; Hantke, Florian; Stock, Ben (2025): "403 Forbidden? Ethically Evaluating Broken Access Control in the Wild", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[41]

McDonald, Allison; Bernhard, Matthew; Valenta, Luke; VanderSloot, Benjamin; Scott, Will; Sullivan, Nick; Halderman, J. Alex; Ensafi, Roya (2018): "403 Forbidden: A Global View of CDN Geoblocking", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[42]

Bijmans, Hugo; Booij, Tim; Schwedersky, Anneke; Nedgabat, Aria; Wegberg, Rolf van (2021): "Catching Phishers By Their Bait: Investigating the Dutch Phishing Landscape through Phishing Kit Detection", in: Proceedings of the USENIX Security Symposium. (Link)

[43]

Oh, Se Eun; Li, Shuai; Hopper, Nicholas (2017): "Fingerprinting Keywords in Search Queries over Tor", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[44]

Singh, Rachee; Nithyanand, Rishab; Afroz, Sadia; Pearce, Paul; Tschantz, Michael Carl; Gill, Phillipa; Paxson, Vern (2017): "Characterizing the Nature and Dynamics of Tor Exit Blocking", in: 26th USENIX Security Symposium (USENIX Security 17), pp. 325-341. USENIX Association. (Link)

[45]

Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)

[46]

Demir, Nurullah; Hörnemann, Jan; Große-Kampmann, Matteo; Urban, Tobias; Pohlmann, Norbert; Holz, Thorsten; Wressnegger, Christian (2023): "On the Similarity of Web Measurements Under Different Experimental Setups", in: Proceedings of the ACM Internet Measurement Conference, pp. 356-369. ACM DOI 10.1145/3618257.3624795 is listed by DBLP but was not registered with the DOI resolver as of 2026-08-12; the DOI above resolves to the authors' institutional record of the same paper (DOI)