Working notes behind Crawler Detection. Every figure on that page has its query here, with the population it was counted over. Corpus-level caveats — the venue scope, the selection funnel, what each stage costs — are on corpus and are not restated. This page adds no bibliography entries of its own; it cites the same keys as the content page, from the same shared bibliography.

This is a log, not prose. It is for somebody checking a number.

Scripts committed for this page:

The item asked for a new page and the check confirmed it was the right call, but the reasoning is worth recording because it is a judgement call:

• programming:crawler had an eleven-line Being Detected section. It is the

right place for “which library is more visible”, which is a property of the

  libraries that page compares. It is the wrong place for the measurement-validity
  argument, the symptom taxonomy, the corpus reporting-rate finding and the
  AI-crawler turn — four sections that would have doubled a page already at 52 KB and
  about something else.
* ''design:crawling_location'' covers IP and ASN reputation, and covers it well. That
  is one row of this page's detection-surface table. Broadening it to cover browser
  fingerprint inconsistency and challenge pages would have made a page about //where
  you measure from// into a page about //everything//.
* The two are cross-linked from here rather than absorbed. After this page was
  published the ''Being Detected'' section on ''programming:crawler'' was **shrunk to a
  pointer**, so there is one copy, not two.

What was deliberately not written. No evasion instructions. The tooling table dates the stealth projects and says which are abandoned; it does not say how to use any of them, and the page says twice that evasion is an ethics decision that belongs on Ethics. A reviewer who thinks that table still reads as a cookbook should say so — it was the closest call on the page.

The schema has no field for this. extractions.jsonl records what a paper did — its tools, its crawl configuration, its detections, its statistics. Nothing records what was done to it. There is no crawlConfig.wasBlocked, and detection[] carries the paper's own measured phenomena, not its incidents. Every schema-side query tried in the first hour returned the wrong thing:

So the page population is a full-text sweep, hand-audited. That is the central methodological decision on this page and everything below is a consequence of it.

Text is read from paper.cols.txt with whitespace collapsed to single spaces before any pattern runs. Without that, a PDF line break inside a phrase silently defeats every multi-word pattern; this has bitten this wiki before.

Ten ordered patterns, in scripts/botdet_fold.mjs. Each sentence is tested against them in order and attributed to the first that matches, so the per-pattern counts in section A of the report sum to more than 138 only because a paper can contain several matching sentences.

The patterns are in the script rather than reproduced here, because a DokuWiki table cell cannot hold a literal | and these are regular expressions full of them. The design rule behind all ten is short enough to state:

• a pattern either anchors on an obstruction noun applied to a first-person object

(our crawler was blocked, blocked us), or it requires we/our/us within 160 characters in the same sentence (…we…encountered a CAPTCHA).

• Sentences shorter than 25 or longer than 500 characters are skipped; both extremes

are almost always column-splice artefacts rather than sentences.

What the anchors buy. botdet_fold.mjs also carries LOOSE_SELF_REPORT_PATTERNS: the same ten patterns with every first-person requirement removed, written out by hand. They return 406 crawling papers instead of 138. The extra 268 were not hand-audited — a read of a sample found adversaries evading detection, Cloudflare and Akamai appearing as CDNs, and ad-blockers blocking requests.

The hand-written loose set exists because of a review finding. The first implementation derived the loose patterns by string-editing the regex sources of the tight ones, and that only stripped the anchor from four of the ten — the other six spell their person-requirement differently (our alone, an embedded our|us alternation, we alone). It reported 249 and the page said 249; the true figure is 406, so the page had been understating what the anchors buy by more than half. Earlier, looser drafts of the sweep were worse still: a first draft that only required an obstruction verb and a crawl-ish noun in the same sentence returned 382 papers with a precision low enough that reading twelve of them found one true positive. (That 382 and the recall probe's 382 are an unrelated coincidence — different patterns over different populations. Do not read the 1-in-12 anecdote as bearing on the recall audit's 6-in-24.)

Each of the 138 candidates got exactly one verdict, the strongest that applies:

The instruction to every auditor was explicit that a false obstructed-quantified is the expensive error, and that a paper whose research subject is bot detection is offtopic unless it separately reports its own crawl being obstructed. That rule decided several borderline cases in both directions and each is recorded in the note column of the full table below.

The 138 candidates were split into 12 batches of 12 (the last has 6) and handed to 12 independent Sonnet sub-agents. Each got: the paper key, the title, the absolute path to the paper's own paper.cols.txt, and the matched sentences. Each was instructed to open the full text and grep around the match before deciding, because the matched sentence alone is frequently misleading — and it was: several verdicts were decided by a sentence elsewhere in the paper that the sweep never saw. Two examples, both upgrades:

• IMC/2015/who-is-com-learning-to-parse-whois-records matched on a self-throttling

sentence; the ~7.5% query-failure rate attributed to per-IP rate limiting is

  elsewhere in the paper.
* ''WWW/2024/the-double-edged-sword-…'' matched on a sentence about the paper's research
  focus; the 93 (1.9%) of 5,000 pages lost to bot detection is elsewhere.

And one downgrade worth recording: WWW/2022/reproducibility-and-replicability-of-web-measurement-studies matched a string the auditor could not locate in any rendering of the paper, and the paper contains no block/CAPTCHA/bot/403/429/rate-limit mentions at all. It is recorded as a sweep false positive with no quote — the only candidate of 138 in that state.

The report script prints all of them. Its real, unedited output follows; the page carries no number that is not in it, which the number guard asserts.

node scripts/report_crawler_detection.mjs
node scripts/check_page_numbers.mjs pages/crawler_detection.txt out/report_crawler_detection-output.txt
node scripts/check_page_numbers.mjs pages/crawler_detection.txt out/report_crawler_detection-output.txt --code

The guard was run both windowed and whole-page, because figures outside an audited window have survived a refresh on this wiki before. Both pass.

report_crawler_detection-output.txt

==============================================================================
A. POPULATIONS
==============================================================================
corpus                                         5859
crawled  (crawlConfig != null OR studyTypes has automated-web-crawl)  1120
web-crawled (crawled AND platforms has web)    857
 
This page has no schema field. The extraction schema records what a paper
DID, not what was done TO it, so "our crawler was blocked" appears nowhere in
extractions.jsonl. The page population is therefore a full-text sweep over
paper.cols.txt with whitespace collapsed, hand-audited paper by paper.
 
self-report sweep candidates                   138  (12.3% of crawled)
crawled papers with no full text on disk       0
same ten patterns with every first-person requirement removed  406  (36.3% of crawled)
  -> the anchors remove 268 papers. Not hand-audited; a read of a
     sample of them found adversaries evading detection, CDN mentions and
     ad-blocker filtering, which is why the anchors are there.
 
Which pattern fired (a paper may fire several):
pattern             papers
------------------  ------
active              37    
avoid_detect        34    
captcha_served      23    
excluded_blocked    17    
antibot_stopped     15    
bot_challenge_page  15    
blockrate           13    
passive             11    
identified_as_bot   8     
weblocked           3     
 
==============================================================================
B. HAND VERDICTS, AND WHAT THE SWEEP IS WORTH
==============================================================================
verdict                papers  share of 138 candidates  share of 1,120 crawled
---------------------  ------  -----------------------  ----------------------
obstructed-quantified  22      15.9%                    2.0%                  
obstructed-narrative   21      15.2%                    1.9%                  
mitigated-only         38      27.5%                    3.4%                  
offtopic               57      41.3%                    5.1%                  
 
reports its OWN crawl obstructed (quantified + narrative)  43  = 3.8% of the 1,120 crawled papers
puts a NUMBER on it                                       22  = 2.0% of the 1,120 crawled papers
describes mitigations but never says whether it happened  38  = 3.4% of the 1,120 crawled papers
 
sweep precision for "reports obstruction"   43/138 = 31.2%
sweep precision for "on topic at all"       81/138 = 58.7%
 
Every count above is a LOWER BOUND on the field: the sweep can only find a
paper that WROTE the sentence. Section C estimates how many it missed.
 
==============================================================================
C. RECALL ESTIMATE
==============================================================================
recall probe: any first-person sentence with an obstruction token within
240 characters (RECALL_WIDE_PATTERN in botdet_fold.mjs). A third pattern,
separate from the tight sweep and from the loose set in section A.
non-candidate crawled papers                    982
of those, hit by the recall probe               382
hand-audited sample (every 15th)              24
verdict                papers in sample
---------------------  ----------------
obstructed-quantified  3               
obstructed-narrative   3               
mitigated-only         3               
offtopic               15              
 
self-reports the tight sweep MISSED, in the sample   6 of 24
extrapolated over the 382 recall-probe hits          ~96 papers
so the true count of self-reporting papers is roughly 43-139
i.e. between 3.8% and 12.4% of the 1,120 crawled papers.
Wilson 95% interval on 6/24 = [12.0%, 44.9%]
  -> 46-172 missed papers
  -> 89-215 papers total = 7.9%-19.2% of the 1,120 crawled papers
 
This is an estimate from a 24-paper sample, not a measurement. And even the
upper end is a lower bound, because the WIDE pattern has unknown recall too:
a paper that was blocked and never wrote a sentence about it is invisible to
both. Publish the 43 as the measured floor and this range as its shape.
 
One of the 3 quantified misses is borderline and is flagged in the hand map:
IEEE-SP/2025 predator hit CAPTCHAs in bWAPP, a deliberately vulnerable local
benchmark app, not on the open web. Dropping it moves the point estimate from
96 to 80 missed papers.
 
==============================================================================
D. THE QUANTIFIED FIGURES, EACH WITH ITS DENOMINATOR
==============================================================================
22 of the 1,120 crawled papers put a number on their own obstruction.
Read the denominators: they are not comparable with one another. A "1%"
measured on 3,000 popular sites and a "69%" measured on detected phishing
domains are answers to different questions.
 
* CCS/2020/impersonation-as-a-service-characterizing-the-emerging-criminal-infrastructure-f
    figure: 5 accounts the authors held on ImpaaS.ru were banned or blacklisted; the paper does not state the total number of accounts obtained
    quote : "In the process, an additional three accounts were banned, for a total of five banned accounts."
* IEEE-SP/2025/403-forbidden-ethically-evaluating-broken-access-control-in-the-wild
    figure: 3 of 110 websites with two accounts (3 of the 10 failed visits) lost sessions to blocking/bot detection
    quote : "Further investigation shows our accounts were either blocked and required an account recovery process or bot detection invalidated our sessions."
* IMC/2015/who-is-com-learning-to-parse-whois-records
    figure: ~7.5% of domains queried failed after 3 attempts, out of 102M WHOIS records (~90% of .com); attributed to per-IP rate limiting
    quote : "Roughly 7.5% of domains we queries resulted in a failure after all 3 attempts."
* IMC/2016/characterizing-website-behaviors-across-logged-in-and-not-logged-in-users
    figure: 345 of 420 selected sites retained; 75 (~17.9%) dropped, attributed to blocking or CAPTCHA
    quote : "Note that the total number of websites per each category is less than 30 as several websites in each category were excluded due to technical problems during one of the crawls: the website blocked the crawler or the login process started to require human verification via captcha."
* IMC/2017/tripwire-inferring-internet-site-compromise
    figure: 19% of sites with registration forms (37% of the top-100 subset) presented a human-verification/CAPTCHA challenge
    quote : "In our manual study above, 19% of sites (37% of the top-100) with registration forms used some kind of test to ensure the registration form was being filled out by a human actor."
* IMC/2018/403-forbidden-a-global-view-of-cdn-geoblocking
    figure: ~30% of Akamai 403 responses were false positives from the crawler being flagged as a bot; 286 of 1,068 (27%) automated block-page detections across the full scan were false positives, all Akamai
    quote : "We found that on the order of 30% of the Akamai 403s appeared to be false positives: the crawler request was flagged as a bot or otherwise was denied access while a real web browser request was able to load the page."
* IMC/2023/the-prevalence-of-single-sign-on-on-the-web-towards-the-next-generation-of-web-c
    figure: 8% of 994 responsive sites (Top 1K CrUX U.S. crawl, Table 2 'Blocked' row) blocked by a bot-detection service
    quote : "8% of sites which are labeled blocked used some kind of bot-detection service preventing our Crawler from loading the page."
* IMC/2025/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre
    figure: 1,487 (15%) of the top 10,000 popular sites (October 2024) inherently blocked the authors' crawler
    quote : "Among the top 10k popular sites in October 2024, 1,487 (15%) of them inherently block our crawler."
* NDSS/2024/fp-fed-privacy-preserving-federated-detection-of-browser-fingerprinting
    figure: 1,700 of 20,000 websites (8.5%) failed; 64.3% of those failures were HTTP 403, attributed to login walls or bot detection
    quote : "our crawl fails to collect traces from 1,700 websites, with the overwhelming majority (64.3%) of them due to HTTP 403 Forbidden errors."
* PETS/2017/fingerprinting-keywords-in-search-queries-over-tor
    figure: 54% of fingerprintable keyword trace groups hit a Google CAPTCHA during collection vs 35% of non-FP groups
    quote : "In addition, 54% of FP keywords received a CAPTCHA during collection and the Tor process had been restarted 60 times on average (35% and 23 times for Non-FP keywords)."
* PETS/2024/automatic-generation-of-web-censorship-probe-lists
    figure: 3 of 50 (6%) ground-truth URLs returned 403 from curl due to server-side anti-bot blocking, across all vantage points
    quote : "At least 3 of the 50 "ground truth" URLs implement some server-side blocking against automated headless browsers like curl, resulting in Error 403 in all vantage points."
* PETS/2024/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework
    figure: 2 of 200 (1%) manually re-visited websites received CAPTCHA challenges preventing normal load; the paper estimates ~1% of the crawled sample was impacted by anti-crawler countermeasures
    quote : "We received CAPTCHA challenges on two of the websites that prevented them from loading normally."
* PETS/2025/tracker-installations-are-not-created-equal-understanding-tracker-configuration
    figure: 8.6% of false negatives attributed to bot detection (denominator: the manually validated false-negative sample, ~52 cases)
    quote : "We generally found that a subset of the false negatives could be attributed to cookie consent or other dialog boxes that blocked page load (17.3%), bot detection (8.6%), and websites we had not successfully visited (4.9%)."
* USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking
    figure: 50% of the most aggressive site categories (social networking, online shopping) blocked the authors' crawl over more than 60% of their Tor exit relays
    quote : "Websites in these categories are also observed to be the most aggressive-with 50% of them blocking over 60% of the chosen relays."
* USENIX/2021/catching-phishers-by-their-bait-investigating-the-dutch-phishing-landscape-throu
    figure: 946 (69%) of the paper's 1,363 verified Dutch phishing domains served a blank screen to the authors' crawler via cloaking
    quote : "In fact, 946 (69%) of the detected phishing domains returned a blank screen - and no favicon - to our crawler when we visited the domain, meaning that the phishing website detected us and deployed cloaking techniques."
* USENIX/2022/leaky-forms-a-study-of-email-and-password-exfiltration-before-form-submission
    figure: 3 CloudFlare CAPTCHA pages blocked the crawler, out of a 1,000-website pilot crawl
    quote : "identified only three CloudFlare CAPTCHA pages that"
* USENIX/2023/a-large-scale-measurement-of-website-login-policies
    figure: 52% of domains on which a test account was created could not be analysed, primarily due to CAPTCHA-solving failures (18.0K of 37.3K fully evaluated)
    quote : "However, given that we failed to analyze 52% of domains on which we created an account, the true prevalence may be higher."
* USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability
    figure: 43 of 56 'manual' sites (of 208 audited) needed manual login due to login CAPTCHAs; 3 of 56 detected and blocked the web driver
    quote : "Of these manual sites, 43 could not be scripted because they used CAPTCHAs at the login page, 10 required MFA on every login, and 3 detected and blocked our web driver."
* USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v
    figure: 17 of 98 Joomla servers and 18 of ~100 phpMyAdmin servers blocked version-disclosure paths
    quote : "For instance, 17 websites blocked access to version-revealing paths like "language/en-GB/en-GB.xml", causing Metasploit-Joomla to fail in version identification."
* WWW/2021/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems
    figure: the automated-crawler persona was explicitly blocked 5 percentage points more often and covertly blocked at least 8 points more often than the human persona
    quote : "We observe that automated crawlers are explicitly blocked 5% more often than other users and covertly blocked (by sending users to parked or other error pages) at least 8% more frequently."
* WWW/2024/the-double-edged-sword-identifying-authentication-pages-and-their-fingerprinting
    figure: 93 (1.9%) of 5,000 labelled pages could not have all 88 features collected due to bot detection
    quote : "we were unable to collect all 88 features for our model for 93 (1.9%) pages due to bot detection mechanisms"
* WWW/2025/beyond-the-crawl-unmasking-browser-fingerprinting-in-real-user-interactions
    figure: 15 of ~3,000 websites (~1%) failed, of which 86.7% returned 4XX errors associated with bot detection
    quote : "First, we find that the automated crawler failed to visit 15 out of the 3K websites (≈ 1%)."
 
==============================================================================
D2. QUANTIFIED FIGURES FOUND BY THE RECALL AUDIT
==============================================================================
These are NOT part of the 22 above (they come from the 24-paper recall sample,
not from the 138 candidates) but they are real published figures and the page
uses them, so they are printed here with their denominators.
 
* WWW/2023/the-hitchhikers-guide-to-facebook-web-tracking-with-invisible-pixels-and-click-i
    figure: 172 of ~2,308 websites (7.5%)
    quote : "Interestingly, there are 172 (7.5%) websites that blocked our access after a specifc event, or stored duplicate results for an event."
* IEEE-SP/2025/predator-directed-web-application-fuzzing-for-efficient-vulnerability-validation
    figure: 3 of 6 undetected SQLi vulnerabilities in bWAPP
    quote : "The remaining 3 vulnerabilities could not be triggered by our current prototype due to the presence of CAPTCHAs and the requirement of specific User Agent (UA) header values."
* USENIX/2025/websites-global-privacy-control-compliance-at-scale-and-over-time
    figure: 1.6-2.6% of crawl-set sites (307, 307 and 186 sites across three crawls)
    quote : "We identified that 1.6-2.6% of sites in our crawl set had a human check."
 
==============================================================================
E. OBSTRUCTION REPORTING OVER TIME
==============================================================================
Year    Crawled papers  Sweep candidates  Reports obstruction  Quantifies it  Share of crawled
------  --------------  ----------------  -------------------  -------------  ----------------
2010    25              2                 0                    0              0.0%            
2011    22              3                 0                    0              0.0%            
2012    29              3                 0                    0              0.0%            
2013    26              1                 0                    0              0.0%            
2014    35              5                 1                    0              2.9%            
2015    41              7                 4                    1              9.8%            
2016    40              5                 2                    1              5.0%            
2017    51              8                 4                    3              7.8%            
2018    61              6                 2                    1              3.3%            
2019    97              9                 1                    0              1.0%            
2020    75              7                 2                    1              2.7%            
2021    75              8                 2                    2              2.7%            
2022    110             16                4                    1              3.6%            
2023    125             12                5                    3              4.0%            
2024    110             15                6                    4              5.5%            
2025 *  129             29                10                   5              7.8%            
2026 *  69              2                 0                    0              0.0%            
* 2025 and 2026 are provisional: CCS 2026 and IMC 2026 have not been held, and
  IEEE S&P 2026 and WWW 2026 abstracts are not in the selection source, so those
  venue-years are under-represented by construction.
 
2010-2019: 14 of 427 crawled papers report obstruction (3.3%)
2020-2024: 19 of 495 crawled papers report obstruction (3.8%)
2025-2026: 10 of 198 crawled papers report obstruction (5.1%)  [PROVISIONAL venue-years]
 
==============================================================================
F. THE CONFIGURATION CHOICES THAT DECIDE DETECTABILITY, AND HOW OFTEN THEY ARE STATED
==============================================================================
Population: the 1,120 crawled papers. Sentinels are counted as sentinels.
 
Choice                 Field                         Papers stating it  Share of 1,120
---------------------  ----------------------------  -----------------  --------------
Headless or headed     crawlConfig.headless          140                12.5%         
Stateful or stateless  crawlConfig.statefulness      219                19.6%         
Consent action         crawlConfig.consentAction     349                31.2%         
Interaction depth      crawlConfig.interactionDepth  841                75.1%         
Authentication         crawlConfig.authentication    779                69.6%         
Which browser          crawlConfig.browsers          529                47.2%         
 
Headless is the single most detectable configuration a crawl can choose, and it
is the least reported of these.
 
crawlConfig.headless values, over the 1080 crawled papers that have a crawlConfig object:
value       papers
----------  ------
not-stated  940   
headless    93    
headful     35    
both        12    
 
==============================================================================
G. OBSERVABLE DETECTION SIGNALS — FULL-TEXT SWEEP OVER THE WHOLE CORPUS
==============================================================================
Population: all 5859 papers with full text on disk. These are MENTION
counts, not usage counts: a paper that names navigator.webdriver may be
describing it, defending against it, or checking for it. They are here to show
how thin the literature on each signal is, and are not hand-audited.
 
Signal named anywhere in the paper  Papers (of 5859)  Of which crawled (of 1,120)
----------------------------------  ----------------  ---------------------------
CAPTCHA / challenge product         118               41                         
named bot-management vendor         106               38                         
TLS fingerprint (JA3/JA4)           37                11                         
fingerprint inconsistency           24                10                         
headless detection                  20                19                         
stealth patches                     15                13                         
navigator.webdriver                 11                11                         
AI / LLM crawler                    5                 3                          
CDP / DevTools artefacts            4                 1                          
HTTP header-order fingerprint       3                 2                          
 
==============================================================================
H. ANTI-DETECTION TOOLING IN tools[]
==============================================================================
papers naming a stealth patch in tools[]:  11  (of 5859 in the corpus)
of which ran a crawl:                      9  (of 1120 crawled)
years:                                     2021-2026
Year  Papers
----  ------
2021  1     
2022  1     
2023  2     
2024  2     
2025  4     
2026  1     
 
==============================================================================
Y. ARITHMETIC DERIVED ON THIS PAGE
==============================================================================
obstructed = quantified + narrative = 22 + 21 = 43
share of crawled = 43/1120 = 3.8%
quantified share = 22/1120 = 2.0%
candidates share = 138/1120 = 12.3%
mitigated-only + obstructed = 38 + 43 = 81 papers engage with the problem at all = 7.2% of crawled
papers that do NOT report obstruction = 1120 - 43 = 1077 = 96.2% of crawled
papers that do NOT engage at all = 1120 - 81 = 1039 = 92.8% of crawled
 
==============================================================================
Y2. QUOTE VERIFICATION
==============================================================================
scripts/cd_quotecheck.mjs, last line of its output:
  161 quotes: 111 exact, 37 partial, 13 below threshold
Below-threshold is NOT "unsupported": every one was read by hand and all are
present in the paper, spliced by the two-column reading order.
 
==============================================================================
Z. EVERY NON-CORPUS NUMBER ON THE PAGE
==============================================================================
  2.11.2 / 2023-03-01            puppeteer-extra-plugin-stealth's newest npm release and its publish date; the repo is not archived but has had no code release since
      source: registry.npmjs.org/puppeteer-extra-plugin-stealth; api.github.com/repos/berstend/puppeteer-extra (checked 2026-08-29)
  3.5.5 / 2024-02-17 / 2025-07-05 undetected-chromedriver's newest PyPI release, its upload date, and the repo's last push; the repo publishes no GitHub Releases and no tags
      source: pypi.org/pypi/undetected-chromedriver/json; api.github.com/repos/ultrafunkamsterdam/undetected-chromedriver (checked 2026-08-29)
  0.50.3 / 2026-05-13            nodriver's newest PyPI release and upload date. Its GitHub description reads 'Successor of Undetected-Chromedriver' — the same author's own succession claim
      source: pypi.org/pypi/nodriver/json; api.github.com/repos/ultrafunkamsterdam/nodriver (checked 2026-08-29)
  v1.62.0 / 2026-08-19           patchright-python's newest tag and the repo's last push
      source: api.github.com/repos/Kaliiiiiiiiii-Vinyzu/patchright-python (checked 2026-08-29)
  1.0.19 / 2025-05-09            rebrowser-patches' newest tag and last push
      source: api.github.com/repos/rebrowser/rebrowser-patches (checked 2026-08-29)
  1.0.6 / 2020-11-05 / 2021-12-30 selenium-stealth's newest PyPI release, its upload date and the repo's last push
      source: pypi.org/pypi/selenium-stealth/json; api.github.com/repos/diprajpatra/selenium-stealth (checked 2026-08-29)
  2.0.3 / 2026-04-04 / 2024-07-29 the playwright-stealth PyPI package's newest release and upload date, whose Homepage now points at Mattwmaster58/playwright_stealth; the original AtuboDad/playwright_stealth repo was last pushed 2024-07-29
      source: pypi.org/pypi/playwright-stealth/json; api.github.com/repos/{AtuboDad,Mattwmaster58}/playwright_stealth (checked 2026-08-29)
  MPL-2.0 / 2026-08-26           Camoufox's licence and last push
      source: api.github.com/repos/daijro/camoufox (checked 2026-08-29)
  MIT / v2.0.0 / 2026-08-11      BotD's licence, newest tag and last push
      source: api.github.com/repos/fingerprintjs/BotD (checked 2026-08-29)
  MIT / v1.27.0 / 2026-08-29     Anubis's licence, newest tag and last push. Its GitHub description reads 'Weighs the soul of incoming HTTP requests to stop AI crawlers'
      source: api.github.com/repos/TecharoHQ/anubis (checked 2026-08-29)
  403                            the HTTP status Akamai's own Bot Manager product page returned to curl with a browser User-Agent, and to a headless Playwright Chromium, from this datacentre host. Cause not established: it may be bot management, geography or ASN reputation
      source: curl and Playwright Chromium against https://www.akamai.com/products/bot-manager from a European datacentre host (checked 2026-08-29)
  2023-09-29                     the date Cloudflare announced Turnstile general availability, with Managed mode free for unlimited use. Cloudflare calls it a 'CAPTCHA alternative', not a CAPTCHA
      source: blog.cloudflare.com/turnstile-ga/ (checked 2026-08-29)
  2025-07-01                     the date Cloudflare changed its default to blocking AI crawlers unless they pay, and introduced pay-per-crawl
      source: blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ (checked 2026-08-29)
  05 June 2018                   the date WebDriver Level 1, the W3C Recommendation that defines navigator.webdriver as returning true when the webdriver-active flag is set, was published. Level 2 is still a Working Draft, dated 02 July 2026
      source: w3.org/TR/webdriver1/; w3.org/TR/webdriver2/ (checked 2026-08-29)
  RFC 9309 / September 2022      the Robots Exclusion Protocol RFC and its date. Proposed Standard; nothing obsoletes or updates it
      source: datatracker.ietf.org/doc/rfc9309/; rfc-editor.org/info/rfc9309 (checked 2026-08-29)
  132                            the Chrome version from which the old headless mode is available only as a separate chrome-headless-shell binary; --headless now selects the unified mode that shares code with headful Chrome
      source: developer.chrome.com/docs/chromium/headless (checked 2026-08-29)
  perimeterx.com -> humansecurity.com PerimeterX now redirects to HUMAN Security; distilnetworks.com redirects to imperva.com/products/bot-management; F5 completed its acquisition of Shape Security
      source: curl redirect chains; f5.com/company/news/press-releases/f5-completes-acquisition-of-shape-security (checked 2026-08-29)
  2026-09-15                     the date Cloudflare's new AI-bot defaults take effect for new domains: Training and Agent blocked on pages that display ads, Search still allowed; the legacy 'Block AI bots' option is marked deprecating on the same date
      source: developers.cloudflare.com/bots/llms-full.txt (checked 2026-08-29)
  2025-09-24                     the date Cloudflare announced its Content Signals Policy, adding search / ai-input / ai-train directives to robots.txt
      source: blog.cloudflare.com/content-signals-policy/ (checked 2026-08-29)
  2023-12-04                     the date Thales completed its acquisition of Imperva, per Imperva's own press release. Read first with a headless Playwright Chromium because curl was being served a 920-byte Incapsula challenge with HTTP 200; ~40 minutes later curl got the full 180 KB article three times in a row. The challenge was transient, and the page says so.
      source: imperva.com/company/press_releases/thales-completes-acquisition-of-imperva/ (checked 2026-08-29)

Populations that are NOT this page's. site_queries.mjs –page crawlers reports a crawl-config table over 1,080 papers — the papers that have a crawlConfig object — which is not the same as the 1,120 crawled papers used throughout this page. The distinction is visible in report §F: the crawlConfig.headless value breakdown sums to 1,080, while every share on the page is out of 1,120. Do not mix them.

scripts/tool_fold.mjs is shared with Crawler. Its Anti-detection patches family was /stealth|undetected-chromedriver/i, which is a bare substring test on a free-text tool name, and it matched two things that are not browser-automation patches at all:

The family now requires either an automation library within 20 characters of the word or an explicit patch noun, and also covers the successors named on the content page (nodriver, patchright, rebrowser-patches). The 20-character window is what keeps Puppeteer/Playwright stealth mode in while keeping the laptop out; both transitions were checked one by one against the previous regex, and those two strings are the only two that moved.

This changed a figure on a page that is not this one. report_crawler.mjs now prints 11 / 1 where it printed 12 / 2 for the anti-detection row, so Crawler's Which specialised crawlers actually get used table was corrected in the same sitting. Its 9 | 0.8% row is unaffected, because neither false positive is a crawling paper. The number guard was run over that page before and after the edit and the set of unaccounted figures is identical, so nothing else moved.

The sweep can only find a paper that wrote the sentence, so 43 is a floor. To bound the gap:

1. Take the 982 crawling papers the tight sweep did not flag.
2. Run the recall probe over them — RECALL_WIDE_PATTERN in botdet_fold.mjs,

a third pattern, separate from the tight sweep and from

  ''LOOSE_SELF_REPORT_PATTERNS'': any first-person sentence with an obstruction token
  within 240 characters. It hits **382**. It lived in a scratch file until a
  late review; ''report_crawler_detection.mjs'' §C now **recomputes** both numbers from
  the committed pattern and exits non-zero if they disagree with the stored metadata.
- Take a **reproducible** every-15th sample of those, giving 24 papers, and
  hand-audit them with the same vocabulary and the same instructions.

Result: 6 of 24 were true misses (3 of them quantified). Extrapolated over 382, that is ~96 missed papers, Wilson 95% interval 46–172, so 89–215 papers total, or 7.9%–19.2% of the 1,120. The page publishes “at least 3.8%, plausibly 8–19%” and says explicitly that even 19% is a floor, because the wide pattern has unknown recall too.

A borderline that is flagged rather than buried. IEEE-SP/2025/predator-directed-web-application-fuzzing-… is one of the three quantified misses, and its CAPTCHAs were inside bWAPP, a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 to 80 missed papers. The report prints both. It is kept in because the auditor's reasoning — a CAPTCHA stopped an automated tool from completing its measurement — is the same reasoning applied elsewhere, and excluding it after seeing the number would be the wrong order of operations.

Two figures on the content page come from this recall sample, not from the 138. They are Hausladen et al.'s 1.6–2.6% human-check rate and Bekos et al.'s 172 of ~2,308. Section D2 of the report prints them separately for exactly that reason; they are not counted in the 22.

The full recall sample with verdicts:

Every deciding quote in both audits was checked against the rendering the auditors read, with whitespace, quote marks and dashes normalised. Three verdicts, and below-threshold is not “unsupported”: it means read it by hand.

node scripts/cd_quotecheck.mjs

cd_quotecheck-output.txt

### verdicts_all.json
  partial   CCS/2010/spam-the-underground-on-140-characters-or-less  93% of 15 windows
  MISSING   CCS/2011/text-based-captcha-strengths-and-weaknesses  45% of 22 windows
            "Many websites use CAPTCHAs, or Completely Automated Public Turing tests to tell Computers and Humans Apart, in an attempt to block automated"
  partial   CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels  76% of 17 windows
  MISSING   CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach  56% of 9 windows
            "The real captchas are automatically collected using a web crawler written in Python."
  partial   CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten  61% of 28 windows
  partial   IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i  67% of 9 windows
  MISSING   IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions  55% of 11 windows
            "To ensure that the search engine servers would not be overloaded, we rate-limited our crawlers."
  partial   IMC/2015/who-is-com-learning-to-parse-whois-records  80% of 10 windows
  partial   IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets  92% of 49 windows
  partial   IMC/2019/what-you-see-is-not-what-you-get-discovering-and-tracking-social-engineering-att  97% of 37 windows
  partial   IMC/2022/measuring-uid-smuggling-in-the-wild  90% of 50 windows
  partial   IMC/2022/phishinpatterns-measuring-elicited-user-interactions-at-scale-on-phishing-websit  69% of 13 windows
  partial   IMC/2023/ecn-with-quic-challenges-in-the-wild  63% of 32 windows
  partial   IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th  86% of 29 windows
  MISSING   NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories  46% of 13 windows
            "we can run all queries every thirty minutes within the rate limits using a single API key."
  MISSING   NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms  50% of 2 windows
            "posting prevented due to bot detection"
  MISSING   NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns  35% of 26 windows
            "During the development of our framework, SCAMMAGNIFIER, we faced several challenges including dealing with diverse fraudulent shopping websi"
  MISSING   PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists  56% of 9 windows
            "For instance, LinkedIn blocked us from accessing user profile pages and job listings."
  partial   PETS/2020/missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix  95% of 19 windows
  MISSING   PETS/2022/a-global-survey-of-android-dual-use-applications-used-in-intimate-partner-survei  38% of 16 windows
            "Google Play blocks some IPS-specific queries, such as "track your husband", probably to prevent potential abusers from finding dual-use apps"
  partial   PETS/2022/setting-the-bar-low-are-websites-complying-with-the-minimum-requirements-of-the  86% of 28 windows
  partial   PETS/2024/automatic-generation-of-web-censorship-probe-lists  74% of 23 windows
  partial   USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  87% of 30 windows
  MISSING   USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  56% of 18 windows
            "Websites in these categories are also observed to be the most aggressive-with 50% of them blocking over 60% of the chosen relays."
  MISSING   USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  46% of 13 windows
            "On session start, we provided the scraper with a session cookie manually obtained after solving a CAPTCHA."
  partial   USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies  72% of 18 windows
  partial   USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability  70% of 27 windows
  partial   USENIX/2023/trident-towards-detecting-and-mitigating-web-based-social-engineering-attacks  81% of 27 windows
  partial   USENIX/2024/fv8-a-forced-execution-javascript-engine-for-detecting-evasive-techniques  63% of 19 windows
  partial   USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi  93% of 15 windows
  partial   USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v  86% of 14 windows
  MISSING   USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale  56% of 18 windows
            "except in 2024, when we found a new server that provided over a million SM2 certificates until they blocked our IP address."
  partial   USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal  73% of 15 windows
  partial   WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale  93% of 15 windows
  no-quote  WWW/2022/reproducibility-and-replicability-of-web-measurement-studies  (offtopic)
  partial   WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion  79% of 19 windows
  MISSING   WWW/2024/navigating-the-post-api-dilemma  58% of 24 windows
            "Moreover, given the severity and the impact of the topic surrounding COVID19, search engines may have placed stricter moderation policies on"
  partial   WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e  63% of 16 windows
  MISSING   WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we  52% of 25 windows
            "This approach provides a comprehensive view of the phishing webpages by fully rendering all client-side resources and helps evade anti-bot t"
 
### recall_verdicts.json
  partial   IMC/2016/a-multi-perspective-analysis-of-carrier-grade-nat-deployment  94% of 16 windows
  partial   IMC/2019/ddos-hide-seek-on-the-effectiveness-of-a-booter-services-takedown  76% of 17 windows
  partial   PETS/2020/a-comparative-measurement-study-of-web-tracking-on-mobile-and-desktop-environmen  95% of 19 windows
  partial   CCS/2021/sugarcoat-programmatically-generating-privacy-preserving-web-compatible-resource  100% of 16 windows
  partial   USENIX/2021/fingerprinting-in-style-detecting-browser-extensions-via-injected-style-sheets  90% of 20 windows
  partial   PETS/2022/fingerprintv-fingerprinting-smart-tv-apps  94% of 16 windows
  partial   NDSS/2023/doitrust-dissecting-on-chain-compromised-internet-domains-via-graph-learning  92% of 13 windows
  partial   WWW/2023/the-hitchhikers-guide-to-facebook-web-tracking-with-invisible-pixels-and-click-i  71% of 17 windows
  partial   USENIX/2024/does-online-anonymous-market-vendor-reputation-matter  80% of 5 windows
  partial   USENIX/2024/understanding-help-seeking-and-help-giving-on-social-media-for-image-based-sexua  94% of 16 windows
  partial   USENIX/2025/websites-global-privacy-control-compliance-at-scale-and-over-time  80% of 10 windows
  partial   USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents  72% of 18 windows
 
161 quotes: 111 exact, 37 partial, 13 below threshold

All 13 below-threshold quotes were read by hand against paper.cols.txt. All 13 are present in the paper. The mismatch is always the two-column reading order splicing another column through the sentence. Three were rewritten after that reading, and the rewrite is recorded in the verdict file's note field:

A second, independent check on the same figures. cd_quotecheck.mjs verifies the auditors' quotes; verify_crawler_detection_figures.mjs verifies the page's figures, against the papers, without consulting the audit at all. It found and fixed two things the quote check could not:

• Bekos et al.'s denominator. The page had ~2,308; the paper states 2,308

exactly (“2,308 (2.3K FBP) of the top 10K websites”), so the tilde came from

  back-calculating 172 ÷ 0.075. Worse, reading the whole sentence showed the figure is
  **conflated**: "blocked our access after a specific event, //or// stored duplicate
  results for an event". The row now says so.
* **Bijmans et al.'s denominator**, already covered above — the same failure mode,
  found first by a reviewer.

Both are the same defect: a number that looks like it came from the paper and came from arithmetic on the page instead. Two of 22 rows had it.

verify_crawler_detection_figures-output.txt

ok       IMC/2016/characterizing-website-behaviors-across-logged-in-and-not-logged-in-users  --  75 of 420 selected sites (~17.9%) dropped
ok       IMC/2017/tripwire-inferring-internet-site-compromise  --  19% of sites with a registration form (37% within the top-100 subset)
ok       IMC/2023/the-prevalence-of-single-sign-on-on-the-web-towards-the-next-generation-of-web-c  --  8% of 994 responsive sites
ok       IMC/2025/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre  --  1,487 of the top 10,000 sites (15%)
ok       NDSS/2024/fp-fed-privacy-preserving-federated-detection-of-browser-fingerprinting  --  1,700 of 20,000 sites (8.5%); 64.3% of those HTTP 403
ok       WWW/2024/the-double-edged-sword-identifying-authentication-pages-and-their-fingerprinting  --  93 of 5,000 labelled pages (1.9%)
ok       WWW/2025/beyond-the-crawl-unmasking-browser-fingerprinting-in-real-user-interactions  --  15 of ~3,000 sites (~1%); 86.7% returned 4XX
ok       USENIX/2022/leaky-forms-a-study-of-email-and-password-exfiltration-before-form-submission  --  3 pages in a 1,000-site pilot crawl
ok       PETS/2024/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework  --  2 of 200 re-visited sites (1%)
ok       USENIX/2025/websites-global-privacy-control-compliance-at-scale-and-over-time  --  1.6-2.6% of the crawl set
ok       WWW/2023/the-hitchhikers-guide-to-facebook-web-tracking-with-invisible-pixels-and-click-i  --  172 of 2,308 sites (7.5%)
ok       PETS/2025/tracker-installations-are-not-created-equal-understanding-tracker-configuration  --  8.6% of the manually validated false-negative sample
ok       USENIX/2023/a-large-scale-measurement-of-website-login-policies  --  52% of domains where an account had been created
ok       USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability  --  43 of 56 needed manual login; 3 of 56 blocked the driver
ok       IEEE-SP/2025/403-forbidden-ethically-evaluating-broken-access-control-in-the-wild  --  3 of 110 sites with test accounts
ok       IMC/2018/403-forbidden-a-global-view-of-cdn-geoblocking  --  ~30% of Akamai 403s were false positives
ok       WWW/2021/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  --  explicitly blocked +5 points, covertly blocked +8 points
ok       USENIX/2021/catching-phishers-by-their-bait-investigating-the-dutch-phishing-landscape-throu  --  946 of 1,363 detected phishing domains (69%)
ok       PETS/2017/fingerprinting-keywords-in-search-queries-over-tor  --  54% of FP keyword groups vs 35% of others
ok       USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking  --  50% of the most aggressive categories blocked over 60% of relays
ok       CCS/2020/impersonation-as-a-service-characterizing-the-emerging-criminal-infrastructure-f  --  5 accounts banned; total not stated
ok       USENIX/2023/an-empirical-study-evaluation-of-modern-captchas  --  1,400 participants solving 14,000 CAPTCHAs
 
22 figures checked, 0 not found in the cited paper.

One further figure was checked by hand because the page leans on it: Searles et al.'s 1,400 participants — verified verbatim in the paper's own abstract (“user studies in which 1,400 participants collectively solved 14,000 CAPTCHAs”). A draft of the page had misdescribed that paper as showing CAPTCHAs are no obstacle to a solver; it is a usability study of what CAPTCHAs cost humans, and the sentence was rewritten before publication.

All 138 candidates, with the verdict, the deciding sentence and the reasoning. This is the residue as well as the result: the 57 offtopic rows are the sweep's false positives, printed in full rather than discarded, so the next run can see what the patterns catch that they should not.

Training data is stale by construction on this topic, so every external fact was fetched, not recalled, on 2026-08-29, and every one is re-checkable:

bash scripts/external_checks_crawler_detection.sh

external_checks_crawler_detection-output.txt

== Anti-detection tooling: is it maintained, and what replaced it ==
  berstend/puppeteer-extra           archived=False  pushed_at=2024-07-18T12:28:02Z  license=MIT
      desc: 💯  Teach puppeteer new tricks through plugins.
      newest tag puppeteer-extra-plugin-user-preferences@2.4.1 (no GitHub Release objects)
  ultrafunkamsterdam/undetected-chromedriver archived=False  pushed_at=2025-07-05T10:42:02Z  license=GPL-3.0
      desc: Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
      no releases and no tags
  ultrafunkamsterdam/nodriver        archived=False  pushed_at=2026-05-13T13:58:15Z  license=AGPL-3.0
      desc: Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are 
      no releases and no tags
  Kaliiiiiiiiii-Vinyzu/patchright-python archived=False  pushed_at=2026-08-19T17:20:12Z  license=Apache-2.0
      desc: Undetected Python version of the Playwright testing and automation library. 
      newest tag v1.62.0 (no GitHub Release objects)
  rebrowser/rebrowser-patches        archived=False  pushed_at=2025-05-09T05:24:53Z  license=none
      desc: Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy
      newest tag 1.0.19 (no GitHub Release objects)
  diprajpatra/selenium-stealth       archived=False  pushed_at=2021-12-30T14:57:26Z  license=MIT
      desc: Trying to make python selenium more stealthy.
      newest tag v1.0.6 (no GitHub Release objects)
  AtuboDad/playwright_stealth        archived=False  pushed_at=2024-07-29T17:34:50Z  license=MIT
      desc: playwright stealth
      no releases and no tags
  Mattwmaster58/playwright_stealth   archived=False  pushed_at=2026-04-04T03:02:13Z  license=MIT
      desc: playwright stealth
      newest tag rc4 (no GitHub Release objects)
  daijro/camoufox                    archived=False  pushed_at=2026-08-26T01:40:45Z  license=MPL-2.0
      desc: 🦊 Anti-detect browser
      newest tag v152.0.4-beta.29 (no GitHub Release objects)
      npm puppeteer-extra-plugin-stealth 2.11.2 published 2023-03-01T12:07:26.660Z
      pypi undetected-chromedriver 3.5.5 uploaded 2024-02-17T17:18:36.855607Z
      project_urls: {'Homepage': 'https://github.com/ultrafunkamsterdam/undetected-chromedriver'}
      pypi nodriver 0.50.3 uploaded 2026-05-13T13:57:51.877457Z
      project_urls: {'Bug Reports': 'https://github.com/UltrafunkAmsterdam/nodriver/issues', 'Homepage': 'https://github.com/UltrafunkAmsterdam/nodriver', 'Source': 'https://github.com/UltrafunkAmsterdam/nodriver'}
      pypi playwright-stealth 2.0.3 uploaded 2026-04-04T02:50:35.246953Z
      project_urls: {'Homepage': 'https://github.com/Mattwmaster58/playwright_stealth', 'changelog': 'https://github.com/Mattwmaster58/playwright_stealth/raw/main/CHANGELOG.md'}
      pypi selenium-stealth 1.0.6 uploaded 2020-11-05T15:26:05.240256Z
      project_urls: {'Homepage': 'https://github.com/diprajpatra/selenium-stealth'}
 
== Bot-management vendors: who absorbed whom ==
  perimeterx.com     -> https://www.humansecurity.com/
  distilnetworks.com -> https://www.imperva.com/products/bot-management/?redirect=Distil&
  OK      F5 completed the Shape Security acquisition
  OK      akamai.com/products/bot-manager still 403s a datacentre client (got 403)
  OK      Cloudflare bot products
  fingerprintjs/BotD                 archived=False  pushed_at=2026-08-11T06:30:36Z  license=MIT
      desc: Free bot detection library that runs in the browser. Detects automation tools and frameworks. No server required, runs 100% on the client. MIT license
      newest tag v2.0.0 (no GitHub Release objects)
 
== Challenge products ==
  OK      Turnstile GA post (2023-09-29)
  OK      Cloudflare default-block AI crawlers (2025-07-01)
  OK      reCAPTCHA v2 and v3 still documented
  TecharoHQ/anubis                   archived=False  pushed_at=2026-08-29T00:29:37Z  license=MIT
      desc: Weighs the soul of incoming HTTP requests to stop AI crawlers
      newest tag v1.27.0 (no GitHub Release objects)
  OK      Cloudflare new AI-bot defaults, 2026-09-15
  OK      legacy Block AI bots option deprecating
  OK      Cloudflare Content Signals Policy (2025-09-24)
  OK      imperva.com press release readable by curl (HTTP 200); Thales acquisition claim stands
 
== Standards ==
  OK      WebDriver Level 1 is a W3C Recommendation
  OK      WebDriver Level 1 REC date is 05 June 2018
  OK      navigator.webdriver defined there
  OK      WebDriver Level 2 is still a Working Draft
  OK      RFC 9309 exists
  OK      RFC 9309 is a Proposed Standard
  OK      IETF webbotauth WG is chartered
  OK      IETF aipref WG is chartered
 
== Browser facts that decide detectability ==
  OK      Chrome unified headless / chrome-headless-shell
  OK      Playwright still ships a separate headless shell
  OK      chromium HEAD: content/public/common/content_switches.cc
  OK      chromium HEAD: chrome/browser/ui/startup/automation_infobar_delegate.cc

The Z block — every non-corpus number on the content page, with its claim, its primary source and the date it was checked:

• /releases/latest 404s on tag-only repositories. Five of the eight

anti-detection projects publish tags but no GitHub Release objects, and two publish

  neither. A currency check that only reads ''/releases/latest'' reads all of them as
  abandoned. The script falls back to ''/tags'' and says explicitly "no GitHub Release
  objects" when that is what it found.
* **A package's last release date is not its currency.** Three of the eight have a
  maintained successor under a different name, and ''playwright-stealth'' kept the
  **same package name** while the code moved to a different owner's repository — the
  PyPI ''project_urls.Homepage'' is what gives it away. ''nodriver'''s own GitHub
  description states its succession from ''undetected-chromedriver''; that is the
  author's claim, cited as such, not an inference.
* **''w3.org'' returns HTTP 403 to a browser User-Agent** and 200 to plain ''curl''.
  The check script takes a ''noua'' flag for exactly this. It is a detection rule
  pointing the opposite way from the page's subject and it is noted on the page.
* **Akamai's own Bot Manager product page could not be fetched at all** — HTTP 403
  "Access Denied" to ''curl'' with a browser User-Agent and to a headless Playwright
  Chromium from this datacenter host. The check now **asserts the 403** rather than the
  product name, and the page reports it as an observation with the cause explicitly
  **not** established: it may be bot management, geography or ASN reputation, and we
  did not try to find out by trying harder. ''techdocs.akamai.com'' returns 200 but
  serves a 325-byte JavaScript shell with no product name in it.
* **''perimeterx.com'' redirects to ''humansecurity.com''** and ''distilnetworks.com''
  to Imperva's bot-management page. Both were followed with ''curl -L'' rather than
  taken from a comparison article.

Recorded so the next run does not re-add them:

1. The population is a sweep, and the page says so in its own body, not only here.

A page whose population comes from regexes over PDFs has to declare that where the

  reader meets the numbers.
- **The precision is published (31.2% / 58.7%).** A mention threshold is a candidate
  set, not a claim. Publishing precision is the only thing that makes 43 mean anything.
- **Recall is estimated and the estimate is wide.** Publishing "at least 3.8%, plausibly
  8–19%" is less satisfying than publishing one number, and it is the honest shape.
- **The 22 quantified figures are given as a table with denominators and an explicit
  "not comparable" warning**, rather than as a range or a mean. Averaging a 1% measured
  on popular sites with a 69% measured on phishing domains would produce a number with
  no referent. Three of the 22 are held back from the table entirely — a WHOIS
  bulk-query study, a version-disclosure scan and a 50-URL censorship control set — as
  too specialised to calibrate against; a footnote says so and the script prints all 22.
- **Methods are dated in a table with a "status now" column**, and the two rows that
  rest on 2025–2026 say they rest on the corpus's thinnest years. A ranking of what the
  literature did is not advice about what to do now.
- **The Tor rows are included** (''singh2017_torexit'', ''oh2017_fingerprinting'') even
  though Tor discrimination is arguably a different phenomenon from bot management. They
  are in because from the crawler's side the failure is identical, and because they are
  two of the few quantified figures the field has. A reasonable person would have cut
  them.
- **The tooling table dates the stealth projects but gives no instructions.** The
  closest call on the page; see //Why this page// above.
- **Discussion block.** The content page keeps ''~~DISCUSSION~~''; this page does not.
  Comments belong where the reader is, and a provenance page is read by somebody
  checking a number. That is the convention this page adopts and it is recorded here so
  later provenance pages can follow it.

16 entries were appended to bibliography, generated by scripts/bibgen.mjs from the corpus index (real DOIs where the index has them): li2021_good, herley2022_automated, venugopalan2025_inconsistent, jan2020_throwing, liu2025_somesite, cui2025_odyssey, kim2025_scrapers, qi2026_viper, nguyen2024_frame, ardi2023_prevalence, wang2011_cloak, maroofi2020_human, szurdi2021_where, shi2020_text, oh2017_fingerprinting, campobasso2020_impersonation.

Checks run before appending:

• Key collision against the live bibliography: two generated entries,

bekos2023_hitchhiker and hausladen2025_websites, already existed and were

  dropped from the additions; the page cites the existing keys.
* **DOI and URL collision**, because a key-string check alone passes while the same
  paper is already in the file under another key. One DOI collision
  (''10.1145/3543507.3583311'') and one URL collision, both the same two entries.
* **PETS and USENIX records carry no authors in the index** — 100% of both venues. Three
  entries needed authors fetched from the venue landing page with ''curl'' and a browser
  User-Agent (''WebFetch'' gets 403 from usenix.org): ''herley2022_automated'',
  ''qi2026_viper'' and ''oh2017_fingerprinting''. ''bibgen.mjs'' failed loudly on the
  PETS one rather than guessing, which is the behaviour you want.
* **Attribution guard**: ''node scripts/check_attributions.mjs pages/crawler_detection.txt out/bib_live.txt out/newbib.bib'' — 19 "Name et al., VENUE YEAR" attributions
  checked against the ''author'' field of the key cited beside them, all matching.

Four reviewers, all told explicitly that the author's context may not be exhaustive, all handed the page text, the report script and its unedited output, and these notes. The three focused reviewers ran in parallel first; the generic one ran after their findings were applied.

Instructed to fetch rather than recall, given today's date, and to reject listicles and vendor comparison blogs.

A defect the reviewer did not find, caught by re-running the check script after applying its findings. A first draft of the Imperva footnote said the page “returns an Incapsula-branded HTTP 403 to curl”. It does not: it returns HTTP 200 with a 920-byte Incapsula challenge body. The status code had been misread from a two-column curl -w output. The footnote was rewritten, and the check was rewritten to assert the body, not the status — asserting the status is precisely the mistake the page is about. The corrected observation is a better example than the original claim.

Handed the page, the provenance draft, the report script and its output, the fold script and the hand-audit JSON, and told to re-run the script and to look for bugs inside it rather than only for page/script disagreements.

Told explicitly that the three focused reviewers had already run, that its job was whatever they were not looking for, and that the author's context may not be exhaustive. It also reported that the files changed under it mid-review — true, and a real defect in how this run was sequenced: two of its drafted findings had already been fixed by a concurrent edit. Freezing the files before a generic review is the lesson.

One finding recorded and not acted on. The reviewer noted the run's own sequencing problem — files edited while a reviewer was reading them. That is a process defect, not a page defect, and it is recorded here rather than fixed: the correct practice is to freeze the page before a generic review, and this run did not.

The Imperva observation had to be rewritten twice, and the second rewrite is the one worth recording.

1. First draft: “returns an Incapsula-branded HTTP 403 to curl”. Wrong — the

status was 200 and the challenge was in the body. Caught by the check script.

1. Second draft: “answers curl with HTTP 200 and 920 bytes… the same URL, fetched by

a headless Playwright Chromium, returns the full article.” Correct at the time, and

  published as a worked example.
- Then the check script **failed**: ''curl'' from the same host was now getting the full
  180 KB article, three times in a row, roughly forty minutes after the challenge.

The challenge was transient — almost certainly a rate or reputation score that tripped under our repeated requests and then decayed. The footnote now records the whole sequence with its timing and says explicitly that it is not a standing property of that host, and the check script no longer asserts the transient: it asserts the claim (the article's Thales date) and merely reports which of the two responses it got today. The Akamai 403, by contrast, was consistent on every attempt across several hours, and the page now says so, so the two observations are not read as the same kind of thing.

This is the most useful thing the run produced about its own subject: a block rate measured once is a measurement of a moment, and the page says so where the reader meets the example.

• Crawler Detection — the page these notes are behind.
• corpus — the dataset-wide provenance page: venue scope, selection

funnel, and the corpus-level caveats this page does not restate.

No ~~DISCUSSION~~ block, following the existing provenance: pages: comments belong on the content page, where the reader is. No bibliography entries of its own — it cites the same {[key]} keys as Crawler Detection against the shared bibliography. The keys it names are [1Venugopalan, Hari; Munir, Shaoor; Ahmed, Shuaib; Wang, Tangbaihe; King, Samuel T.; Shafiq, Zubair (2025): "FP-Inconsistent: Measurement and Analysis of Fingerprint Inconsistencies in Evasive Bot Traffic", in: Proceedings of the ACM Internet Measurement Conference. (DOI)], [2Liu, Enze; Luo, Elisa; Shan, Shawn; Voelker, Geoffrey M.; Zhao, Ben Y.; Savage, Stefan (2025): "Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers", in: Proceedings of the ACM Internet Measurement Conference. (DOI)], [3Searles, Andrew; Nakatsuka, Yoshimichi; Ozturk, Ercan; Paverd, Andrew; Tsudik, Gene; Enkoji, Ai (2023): "An Empirical Study & Evaluation of Modern CAPTCHAs", in: Proceedings of the USENIX Security Symposium. (Link)], [4Singh, Rachee; Nithyanand, Rishab; Afroz, Sadia; Pearce, Paul; Tschantz, Michael Carl; Gill, Phillipa; Paxson, Vern (2017): "Characterizing the Nature and Dynamics of Tor Exit Blocking", in: 26th USENIX Security Symposium (USENIX Security 17), pp. 325-341. USENIX Association. (Link)], [5Oh, Se Eun; Li, Shuai; Hopper, Nicholas (2017): "Fingerprinting Keywords in Search Queries over Tor", in: Proceedings on Privacy Enhancing Technologies. (DOI)], [6Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)] and [7Bekos, Paschalis; Papadopoulos, Panagiotis; Markatos, Evangelos P.; Kourtellis, Nicolas (2023): "The Hitchhiker's Guide to Facebook Web Tracking with Invisible Pixels and Click IDs", in: Proceedings of the ACM Web Conference. (DOI)].

[1]

Venugopalan, Hari; Munir, Shaoor; Ahmed, Shuaib; Wang, Tangbaihe; King, Samuel T.; Shafiq, Zubair (2025): "FP-Inconsistent: Measurement and Analysis of Fingerprint Inconsistencies in Evasive Bot Traffic", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[2]

Liu, Enze; Luo, Elisa; Shan, Shawn; Voelker, Geoffrey M.; Zhao, Ben Y.; Savage, Stefan (2025): "Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[3]

Searles, Andrew; Nakatsuka, Yoshimichi; Ozturk, Ercan; Paverd, Andrew; Tsudik, Gene; Enkoji, Ai (2023): "An Empirical Study & Evaluation of Modern CAPTCHAs", in: Proceedings of the USENIX Security Symposium. (Link)

[4]

Singh, Rachee; Nithyanand, Rishab; Afroz, Sadia; Pearce, Paul; Tschantz, Michael Carl; Gill, Phillipa; Paxson, Vern (2017): "Characterizing the Nature and Dynamics of Tor Exit Blocking", in: 26th USENIX Security Symposium (USENIX Security 17), pp. 325-341. USENIX Association. (Link)

[5]

Oh, Se Eun; Li, Shuai; Hopper, Nicholas (2017): "Fingerprinting Keywords in Search Queries over Tor", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[6]

Hausladen, Katherine; Wang, Oliver; Eng, Sophie; Wang, Jocelyn; Wijaya, Francisca; May, Matthew; Zimmeck, Sebastian (2025): "Websites' Global Privacy Control Compliance at Scale and over Time", in: Proceedings of the USENIX Security Symposium. (Link)

[7]

Bekos, Paschalis; Papadopoulos, Panagiotis; Markatos, Evangelos P.; Kourtellis, Nicolas (2023): "The Hitchhiker's Guide to Facebook Web Tracking with Invisible Pixels and Click IDs", in: Proceedings of the ACM Web Conference. (DOI)