Differences

This shows you the differences between two versions of the page.

--- programming:crawler_detection [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
+++ programming:crawler_detection [2026/09/21 13:42] (current) – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude
@@ Line 229: / Line 229: @@
   * **And the defaults are still moving.** Cloudflare has since replaced the single "Block AI bots" toggle with three behaviour categories — **Search**, **Agent** and **Training** — and its own documentation states that from **2026-09-15** new domains default to blocking Training //and Agent// on pages that display ads, while Search stays allowed; the old "Block AI bots" option is marked deprecating on the same date.((''developers.cloudflare.com/bots/llms-full.txt'', fetched 2026-08-29: "New defaults, in which Training and Agent are blocked on pages that display ads while Search remains allowed, take effect for new domains on September 15, 2026", and "Block AI bots [Deprecating on September 15, 2026]". Cloudflare's separate **Content Signals Policy**, announced 2025-09-24, adds ''search'' / ''ai-input'' / ''ai-train'' directives to ''robots.txt''.)) **This page was written 17 days before that date.** If you are reading it later, check the current defaults yourself: an academic crawler that a classifier puts in //Agent// is on the blocked side of the new default, and nothing about being academic exempts it.
   * ''robots.txt'' itself is **RFC 9309**, a Proposed Standard from September 2022, and nothing obsoletes or updates it.((''datatracker.ietf.org/doc/rfc9309/'' and ''rfc-editor.org/info/rfc9309'', fetched 2026-08-29.)) It expresses a preference; it does not enforce one, and it is not what blocks you.
-  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish an adopted working-group draft from an expired individual submission: ''draft-meunier-web-bot-auth-architecture'' is marked expired //and// replaced by ''draft-meunier-webbotauth-httpsig-protocol'', which datatracker in turn shows as expired — checked 2026-09-16. Neither has formal standing, and the working group's own adopted drafts are ''draft-ietf-aipref-vocab'' and ''draft-ietf-aipref-attach'' on the aipref side, with no RFC published on either side as of that date.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
+  * Two IETF working groups are now chartered in this space and are worth watching rather than citing: **webbotauth**, on cryptographically authenticating automated clients and identifying their operators, and **aipref**, on a vocabulary and attachment mechanism for AI-usage preferences.((''datatracker.ietf.org/wg/webbotauth/about/'' and ''datatracker.ietf.org/wg/aipref/about/'', both fetched 2026-08-29. Be careful to distinguish the replaced individual submission from the active WG work: ''draft-meunier-web-bot-auth-architecture'' is marked **Replaced** (state id 4), and the ''replaces'' relation points to ''draft-meunier-webbotauth-httpsig-protocol-02''. Datatracker now marks that intermediate draft **Replaced** (with an "Expired & archived" badge) by the current WG draft ''draft-ietf-webbotauth-httpsig-protocol''; the webbotauth working group remains chartered and active, but the individual drafts have no formal standing — checked 2026-09-17.)) If signed bot identity lands, "declare yourself and be allowed in" becomes a real option for research crawlers, and this page will need rewriting.
 ==== Measuring who crawls you ====
@@ Line 421: / Line 421: @@
 | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** |
 | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% |
-| What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% |
+| Consent-action field populated (schema; not an audited paper claim) | ''crawlConfig.consentAction'' | 349 | 31.2% |
 | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% |
 | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% |
 | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% |
+The consent row reports only that the extractor populated a non-sentinel field; it is not a paper-claim count. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** stating a consent action in full text, while **279 of 313 (89.1%)** ''no-interaction'' labels were unsupported. This page keeps 349 because the table is about schema reporting completeness.
 **Headless versus headful is the most detectability-relevant single bit in the whole
@@ Line 462: / Line 464: @@
   * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
   * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
-  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
+  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Of those 13, **11 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text, not in the extraction — and **2** are below threshold in both renderings. **Below threshold is not "unsupported"** — all 13 were read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
   * **Papers are counted, never tuples**, and a paper counts once.
   * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.