Differences

This shows you the differences between two versions of the page.

--- programming:crawler_detection [2026/09/17 09:18] – Correct IETF Web Bot Auth state and replacement clause; Authored by Claude karel.kubicek.claude
+++ programming:crawler_detection [2026/09/21 13:42] (current) – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude
@@ Line 421: / Line 421: @@
 | Headless or headed | ''crawlConfig.headless'' | 140 | **12.5%** |
 | Stateful or stateless | ''crawlConfig.statefulness'' | 219 | 19.6% |
-| What it did with the consent banner | ''crawlConfig.consentAction'' | 349 | 31.2% |
+| Consent-action field populated (schema; not an audited paper claim) | ''crawlConfig.consentAction'' | 349 | 31.2% |
 | Which browser | ''crawlConfig.browsers'' | 529 | 47.2% |
 | How authentication was handled | ''crawlConfig.authentication'' | 779 | 69.6% |
 | How deep it went | ''crawlConfig.interactionDepth'' | 841 | 75.1% |
+The consent row reports only that the extractor populated a non-sentinel field; it is not a paper-claim count. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** stating a consent action in full text, while **279 of 313 (89.1%)** ''no-interaction'' labels were unsupported. This page keeps 349 because the table is about schema reporting completeness.
 **Headless versus headful is the most detectability-relevant single bit in the whole
@@ Line 462: / Line 464: @@
   * **A sweep hit is a candidate, not a claim.** All 138 candidates carry a hand verdict with its deciding sentence; the report script fails loudly if any candidate is uncovered. The precision that hand audit measured — 31.2% for "reports obstruction", 58.7% for "on topic at all" — is published above rather than hidden, and it is the reason no figure on this page is taken from the sweep alone.
   * **Recall is estimated, not measured.** The **recall probe** is a third pattern, separate from both sets above and much wider than either: any first-person sentence with an obstruction token within 240 characters. Run over the 982 crawling papers the tight sweep did not flag, it hits 382, and the 24-paper sample is a reproducible every-15th selection from those. One of its three quantified misses is borderline and flagged in the hand map: the CAPTCHAs that stopped it were inside ''bWAPP'', a deliberately vulnerable local benchmark app, not on the open web. Dropping it moves the point estimate from 96 missed papers to 80.
-  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. **Below threshold is not "unsupported"** — every one was read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
+  * **Quotes were checked.** All 161 deciding quotes were verified against the rendering the auditors read: 111 match exactly, 37 match on at least 60% of their five-word windows, and 13 fall below that. Of those 13, **11 are located in an independent ''pypdf'' rendering of the same PDF** — a defect in the stored text, not in the extraction — and **2** are below threshold in both renderings. **Below threshold is not "unsupported"** — all 13 were read by hand, all 13 are present in the paper, and the mismatch is always a two-column reading order splicing another column through the sentence. Three quotes were rewritten after that reading and the rewrite is recorded.
   * **Papers are counted, never tuples**, and a paper counts once.
   * **Silence is not absence.** "Does not report obstruction" means the paper did not say. These are reporting figures throughout, and the whole point of the page is that the reporting rate is low.