provenance:programming:crawler_detection
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| provenance:programming:crawler_detection [2026/08/29 01:43] – Record the generic review, the rendered-DOM verification and the wrapped-bullet defect it caught; unwrap this page's own bullets (the builder now does it). Authored by Claude karel.kubicek.claude | provenance:programming:crawler_detection [2026/09/21 13:42] (current) – Quote-check refresh 2026-09-21: regenerated cd_quotecheck and report_crawler_detection output blocks; 13 below threshold -> 11 rescued + 2 below in both. Authored by Claude karel.kubicek.claude | ||
|---|---|---|---|
| Line 147: | Line 147: | ||
| The guard was run **both** windowed and whole-page, because figures outside an audited | The guard was run **both** windowed and whole-page, because figures outside an audited | ||
| window have survived a refresh on this wiki before. Both pass. | window have survived a refresh on this wiki before. Both pass. | ||
| + | |||
| + | The configuration table in the output below is a schema-completeness comparison. Its '' | ||
| <file text report_crawler_detection-output.txt> | <file text report_crawler_detection-output.txt> | ||
| Line 364: | Line 366: | ||
| Population: the 1,120 crawled papers. Sentinels are counted as sentinels. | Population: the 1,120 crawled papers. Sentinels are counted as sentinels. | ||
| - | Choice | + | Choice |
| - | --------------------- | + | ------------------------------------------------------------------- |
| - | Headless or headed | + | Headless or headed |
| - | Stateful or stateless | + | Stateful or stateless |
| - | Consent action | + | Consent-action |
| - | Interaction depth crawlConfig.interactionDepth | + | Interaction depth crawlConfig.interactionDepth |
| - | Authentication | + | Authentication |
| - | Which browser | + | Which browser |
| Headless is the single most detectable configuration a crawl can choose, and it | Headless is the single most detectable configuration a crawl can choose, and it | ||
| Line 435: | Line 437: | ||
| ============================================================================== | ============================================================================== | ||
| scripts/ | scripts/ | ||
| - | 161 quotes: 111 exact, 37 partial, | + | 161 quotes: 111 exact, 37 partial, |
| Below-threshold is NOT " | Below-threshold is NOT " | ||
| present in the paper, spliced by the two-column reading order. | present in the paper, spliced by the two-column reading order. | ||
| Line 500: | Line 502: | ||
| | Names a stealth patch in '' | | Names a stealth patch in '' | ||
| - | **Populations that are NOT this page' | + | **Populations that are NOT this page' |
| crawl-config table over 1,080 papers — the papers that have a '' | crawl-config table over 1,080 papers — the papers that have a '' | ||
| which is not the same as the 1,120 '' | which is not the same as the 1,120 '' | ||
| Line 600: | Line 602: | ||
| ### verdicts_all.json | ### verdicts_all.json | ||
| partial | partial | ||
| - | | + | |
| - | "Many websites use CAPTCHAs, or Completely Automated Public Turing tests to tell Computers and Humans Apart, in an attempt to block automated" | + | |
| partial | partial | ||
| - | | + | |
| - | "The real captchas are automatically collected using a web crawler written in Python." | + | |
| partial | partial | ||
| partial | partial | ||
| - | | + | |
| - | "To ensure that the search engine servers would not be overloaded, we rate-limited our crawlers." | + | |
| partial | partial | ||
| partial | partial | ||
| Line 616: | Line 615: | ||
| partial | partial | ||
| partial | partial | ||
| - | | + | |
| - | "we can run all queries every thirty minutes within the rate limits using a single API key." | + | |
| - | | + | |
| - | " | + | |
| - | | + | |
| - | " | + | |
| MISSING | MISSING | ||
| "For instance, LinkedIn blocked us from accessing user profile pages and job listings." | "For instance, LinkedIn blocked us from accessing user profile pages and job listings." | ||
| Line 630: | Line 626: | ||
| partial | partial | ||
| partial | partial | ||
| - | | + | |
| - | " | + | |
| - | | + | |
| - | "On session start, we provided the scraper with a session cookie manually obtained after solving a CAPTCHA." | + | |
| partial | partial | ||
| partial | partial | ||
| Line 640: | Line 634: | ||
| partial | partial | ||
| partial | partial | ||
| - | | + | |
| - | " | + | |
| partial | partial | ||
| partial | partial | ||
| no-quote | no-quote | ||
| partial | partial | ||
| - | | + | |
| - | " | + | |
| partial | partial | ||
| - | | + | |
| - | "This approach provides a comprehensive view of the phishing webpages by fully rendering all client-side resources and helps evade anti-bot t" | + | |
| ### recall_verdicts.json | ### recall_verdicts.json | ||
| Line 666: | Line 657: | ||
| partial | partial | ||
| - | 161 quotes: 111 exact, 37 partial, | + | 161 quotes: 111 exact, 37 partial, |
| </ | </ | ||
| - | All 13 below-threshold quotes were read by hand against '' | + | **Re-run 2026-09-21 with the PDF fallback: 11 of the 13 are rescued, 2 are below threshold in both renderings.** See // |
| - | present in the paper.** The mismatch is always the two-column reading order splicing | + | |
| + | All 13 of the then-below-threshold quotes were read by hand against '' | ||
| + | present in the paper**, a hand result the fallback independently confirms for 11 of them. The mismatch is always the two-column reading order splicing | ||
| another column through the sentence. Three were rewritten after that reading, and the | another column through the sentence. Three were rewritten after that reading, and the | ||
| rewrite is recorded in the verdict file's '' | rewrite is recorded in the verdict file's '' | ||
| Line 1042: | Line 1035: | ||
| | '' | | '' | ||
| | '' | | '' | ||
| - | | '' | + | | '' |
| | '' | | '' | ||
| - | | '' | + | | '' |
| | '' | | '' | ||
| | '' | | '' | ||
| Line 1183: | Line 1176: | ||
| | //" | | //" | ||
| | //"no difference is evidence you were not singled out"// in the differential-crawl box. | Overstated: a null result on one axis is weak evidence, not proof, and it says nothing about the axes you did not vary. Rewritten to say so, and to add that a paper must name the axis it varied or "we ran a control" | | //"no difference is evidence you were not singled out"// in the differential-crawl box. | Overstated: a null result on one axis is weak evidence, not proof, and it says nothing about the axes you did not vary. Rewritten to say so, and to add that a paper must name the axis it varied or "we ran a control" | ||
| + | |||
| + | ===== Addendum, 2026-09-16: " | ||
| + | |||
| + | A second run added one subsection to the content page and corrected one sentence | ||
| + | in the section above it. Everything in this addendum is separate from the | ||
| + | 2026-08-29 run recorded above; nothing there was re-derived, and no figure | ||
| + | outside the new subsection was touched. | ||
| + | |||
| + | ^ ^ ^ | ||
| + | | Date | 2026-09-16 | | ||
| + | | Corpus at the time | 5,859 extracted papers, unchanged from the 2026-08-29 run; 5,855 have '' | ||
| + | | What was added | '' | ||
| + | | What was corrected | the bullet that read " | ||
| + | | Why a section and not a page | the [[: | ||
| + | | Orchestrating model | Claude Opus 5 | | ||
| + | | Review layer | three focused Sonnet passes plus a generic Fable pass; log at the foot of this addendum | | ||
| + | |||
| + | ==== The script ==== | ||
| + | |||
| + | '' | ||
| + | output committed to '' | ||
| + | below. It is separate from '' | ||
| + | inside it, because that script' | ||
| + | subsection' | ||
| + | guard set. '' | ||
| + | '' | ||
| + | in the cited paper//. There is no committed output file for it to diff against, which is | ||
| + | a gap this addendum does not close. | ||
| + | |||
| + | ==== Every query, with its population ==== | ||
| + | |||
| + | ^ Figure on the page ^ Query ^ Population ^ Result ^ | ||
| + | | "three papers in 5,859 name an AI-crawler user-agent token" | W1: full-text match on '' | ||
| + | | the corrected " | ||
| + | | "74 papers" | ||
| + | | "44 of those mention it exactly once" | W3 members whose match count is exactly 1 | the 74 | **44** | | ||
| + | | "only five papers mention it ten or more times" | W3 members with ≥10 matches | the 74 | **5**: the three 2025 papers (240, 198, 185 mentions), Aristaeus (15), the '' | ||
| + | | "No paper in the corpus mentions '' | ||
| + | |||
| + | **Containment is asserted, not assumed.** W1 ⊆ W2 and W1 ⊆ W3 are checked by set | ||
| + | membership and the script throws if either fails. A size comparison would pass on | ||
| + | disjoint sets, so membership is what is tested. | ||
| + | |||
| + | ==== The defect this found: "five papers" | ||
| + | |||
| + | W2 ⊄ W3 — two of its five members never mention '' | ||
| + | fired on the first run. Read individually: | ||
| + | |||
| + | ^ Paper ^ Matched on ^ Verdict ^ | ||
| + | | NDSS 2025, // | ||
| + | | PETS 2026, // | ||
| + | |||
| + | So the sentence " | ||
| + | and false of the claim. It was corrected in place rather than deleted, and both widths | ||
| + | are now named on the page, because the wider number is the one a reader re-running the | ||
| + | 2026-08-29 script will get. | ||
| + | |||
| + | ==== The coding of the 44 single-mention papers, and its residue ==== | ||
| + | |||
| + | The page says the 44 papers that mention '' | ||
| + | crawl-policy declarations or archive-coverage caveats, not measurements. That sentence | ||
| + | rests on reading all 74 contexts by hand. A crude coder over the ±300 characters around | ||
| + | the single mention is in the script as a check on that reading, and it does **not** | ||
| + | cover the field: | ||
| + | |||
| + | ^ Code ^ Papers ^ | ||
| + | | '' | ||
| + | | '' | ||
| + | | '' | ||
| + | | **UNCODED (residue)** | **21** | | ||
| + | |||
| + | The residue is printed in full in the script output below, with the matched context for | ||
| + | each. **The first draft of the page characterised all 44 as crawl-policy declarations or | ||
| + | archive caveats, and reviewer pass 1 caught it**: that is true of 19 of the 44 by the | ||
| + | coder' | ||
| + | '' | ||
| + | cannot place at all. Reading the 21 by hand: a bibliography entry for | ||
| + | //About / | ||
| + | site's '' | ||
| + | hosts use it defensively. Some are crawl-policy sentences written in the third person and | ||
| + | some are not. | ||
| + | |||
| + | What the page now says is the claim that survives all of this without a coder: **a single | ||
| + | mention is an aside by construction, | ||
| + | '' | ||
| + | 23-and-21 split is published beside that, with the residue here. No share of the 44 is | ||
| + | published as a characterisation. | ||
| + | |||
| + | ==== Quotes spot-checked ==== | ||
| + | |||
| + | Checked against both '' | ||
| + | whitespace collapsed and a de-hyphenated variant tried. A quote is only a failure if it | ||
| + | is absent from **both**. | ||
| + | |||
| + | ^ Quote ^ Paper ^ '' | ||
| + | | "Seven crawlers (Amazonbot, Applebot, CCBot, ClaudeBot, GPTBot, Meta-ExternalAgent, | ||
| + | | "the remaining 20 crawlers did not fetch the robots.txt file at all" | {[liu2025_somesite]} | found | found | | ||
| + | | "This process yields 23 distinct third-party AI assistant crawlers" | ||
| + | | "One crawler (Bytespider) fetched the robots.txt file but did not respect it" | {[liu2025_somesite]} | found | found | | ||
| + | | " | ||
| + | | " | ||
| + | | "We analyze the behavior of 130 self-declared bots (and many anonymous ones) over 40 days" | {[kim2025_scrapers]} | found | found | | ||
| + | | "we present a systematic analysis of 18 LLM bots on 582,281 robots.txt files" | {[cui2025_odyssey]} | found | found | | ||
| + | | "a seven-month-long experiment with 100 dedicated honeysites, Aristaeus recorded 26.4 million requests sent by more than 287K unique IP addresses" | ||
| + | |||
| + | The Aristaeus quote is the case the two-rendering rule exists for: in '' | ||
| + | sentence is spliced by a column break (" | ||
| + | honeypot websites used to study the"). It is verbatim in the PDF. A single-rendering | ||
| + | check would have reported a correct quote as unlocatable and the figure would have been | ||
| + | cut. | ||
| + | |||
| + | ==== Judgement calls ==== | ||
| + | |||
| + | ^ Call ^ Why ^ What a reasonable person might have done instead ^ | ||
| + | | A **subsection inside** //The AI-Crawler Turn//, not a sibling section | that section already owns AI crawlers as a thing that blocks //your// crawl; this is the same subject from the other end, and separating them would make a reader choose between two sections on one topic | a top-level section, which would have read as two pages' worth of AI-crawler material on a page about bot detection | | ||
| + | | Publishing the **disagreement** between the two IMC 2025 papers rather than one of them | they measured different bot populations — vendor training crawlers versus user-triggered assistant fetchers — and the reconciliation is the useful result. Picking either would have produced a confident sentence that the other paper contradicts | quoting only Kim et al., which is the stronger methodology, | ||
| + | | Saying plainly that **the literature does not answer the compliance question** | it does not: both 2025 measurements are bounded to sites the authors operate or have logs for, which is a property of the problem and not a gap either paper should have filled | writing the section as if the two papers settled it, which is what a corpus-driven summary of "what the field found" would produce | | ||
| + | | **Not** citing vendor blocking statistics | an interested party' | ||
| + | | Correcting " | ||
| + | |||
| + | ==== What could not be established ==== | ||
| + | |||
| + | * **Whether '' | ||
| + | * **Any measurement of '' | ||
| + | * **Prevalence of AI-crawler blocking**, e.g. what share of the web now blocks which bot. Vendor telemetry exists and was deliberately not used; see the judgement calls. | ||
| + | * **Whether the 2025 papers' | ||
| + | |||
| + | ==== Review log, 2026-09-16 ==== | ||
| + | |||
| + | Four reviewers, each told explicitly that the author' | ||
| + | each handed the section text, the whole page it sits in, the report script and its real | ||
| + | output. The three focused passes ran in parallel first; the generic pass ran last, over | ||
| + | the fixed text and over this addendum. **Rejections are logged as carefully as fixes: | ||
| + | they are the only record of whether a reviewer slot is worth its cost.** | ||
| + | |||
| + | === Pass 1 — figures against the script (Sonnet) === | ||
| + | |||
| + | Re-ran both report scripts, confirmed they reproduce their committed output | ||
| + | byte-identically, | ||
| + | findings, both accepted: | ||
| + | |||
| + | ^ Finding ^ Verdict ^ What was done ^ | ||
| + | | The signal-sweep table further down the page still said //"All five AI-crawler papers are 2025 or later"//, | ||
| + | | The prose characterised all 44 single-mention '' | ||
| + | |||
| + | Everything else passed: the 3 / 5 / 74 / 44 / 5 / 0 counts, the containment assertions, | ||
| + | the bucket sums, the design-table populations, | ||
| + | '' | ||
| + | |||
| + | === Pass 2 — citations and quotes (Sonnet) === | ||
| + | |||
| + | One finding, accepted: | ||
| + | |||
| + | ^ Finding ^ Verdict ^ What was done ^ | ||
| + | | //" | ||
| + | |||
| + | It confirmed independently: | ||
| + | Crossref on authors, title, venue, year and DOI and is not a duplicate of | ||
| + | '' | ||
| + | least one of the two renderings; every denominator on the page matches the source. | ||
| + | |||
| + | === Pass 3 — external currency (Sonnet, instructed to fetch rather than recall) === | ||
| + | |||
| + | Three findings accepted, one corrected on re-checking, | ||
| + | |||
| + | ^ Finding ^ Verdict ^ What was done ^ | ||
| + | | '' | ||
| + | | "If signed bot identity lands" is no longer hypothetical: | ||
| + | | '' | ||
| + | |||
| + | Passed, each with a URL fetched on 2026-09-16: RFC 9309 is still a Proposed Standard with | ||
| + | nothing obsoleting or updating it; '' | ||
| + | RFC published; Cloudflare' | ||
| + | appear verbatim in the live documentation — **which still uses the future tense a day | ||
| + | after the date**, so a reader checking it later needs to know the date has passed; and the | ||
| + | Content Signals Policy is live with its '' | ||
| + | vocabulary unchanged. | ||
| + | |||
| + | === Pass 4 — generic, no checklist (Fable) === | ||
| + | |||
| + | Spawned with the fixed section text, the whole page, both provenance addenda, the | ||
| + | scripts and the roadmap rows, and asked for whatever the focused three were not | ||
| + | looking for. **It had not returned when these pages were published on 2026-09-16**, | ||
| + | and the pages were published rather than held, on the reasoning that three passes | ||
| + | had already run and each of their accepted findings is applied and logged above. | ||
| + | This is the honest state of the review layer for this addendum: three of four. | ||
| + | If the pass returns, its findings belong here, and anything it finds is a defect | ||
| + | that shipped. | ||
| + | |||
| + | One thing the author changed without a reviewer asking, recorded so it is not | ||
| + | mistaken for a review outcome: the Cloudflare Web Bot Auth paragraph originally sat | ||
| + | between the disagreement box and the //nobody has published this// paragraph, and | ||
| + | was moved after the latter, so the subsection ends on what moved rather than | ||
| + | interrupting what is missing. | ||
| + | |||
| + | ==== The published script ==== | ||
| + | |||
| + | <file javascript report_ai_crawlers.mjs> | ||
| + | # | ||
| + | // Backs the " | ||
| + | // | ||
| + | // node scripts/ | ||
| + | // | ||
| + | // The question the section asks is not "does the corpus mention AI crawlers" | ||
| + | // (the page's existing signal sweep already answers that, 5 papers) but | ||
| + | // "does the published literature in these seven venues answer whether an | ||
| + | // opt-out directive is honoured" | ||
| + | // papers, so every probe here prints its width and its members, and the | ||
| + | // per-paper figures are hand-keyed with their denominators rather than | ||
| + | // derived, because three papers is not a population. | ||
| + | // | ||
| + | // Rules from data/ | ||
| + | // * count papers, never tuples | ||
| + | // * a sweep hit is a CANDIDATE, never a claim | ||
| + | // * a narrower probe must return a subset of a wider one; asserted, not assumed | ||
| + | // | ||
| + | import fs from ' | ||
| + | import path from ' | ||
| + | import { loadExtractions, | ||
| + | |||
| + | const LIST = process.argv.includes(' | ||
| + | const rows = loadExtractions(); | ||
| + | const key = (p) => `${p.venue}/ | ||
| + | |||
| + | // Read paper.cols.txt with whitespace collapsed (a PDF line break inside a | ||
| + | // phrase silently defeats every multi-word pattern below). NOT cached: caching | ||
| + | // 5,855 full texts exhausts the default V8 heap. | ||
| + | function fulltext(p) { | ||
| + | const f = path.join(dataRoot(), | ||
| + | return fs.existsSync(f) ? fs.readFileSync(f, | ||
| + | } | ||
| + | |||
| + | function h(s) { console.log(`\n${' | ||
| + | |||
| + | // --------------------------------------------------------------------------- | ||
| + | // Probe widths. W1 (user-agent tokens) must be a subset of W2 (the page's own | ||
| + | // shared signal pattern) must be a subset of W3 (any robots.txt mention). | ||
| + | // Hardcoding the containment direction and asserting it is the point: a | ||
| + | // narrowing probe that returns MORE hits is broken, and has been before. | ||
| + | // --------------------------------------------------------------------------- | ||
| + | const W1_UA = / | ||
| + | // exactly the 'AI / LLM crawler' | ||
| + | // which is what the page's existing " | ||
| + | const W2_SIGNAL = /AI crawler|GPTBot|CCBot|ClaudeBot|LLM crawler|AI scraper|agentic browser/i; | ||
| + | const W3_ROBOTS = / | ||
| + | const W4_LLMSTXT = / | ||
| + | // ' | ||
| + | // ordinary noun phrase (CMS signed attributes, mmWave screen content, Twitter | ||
| + | // spam features). Anchored on the policy' | ||
| + | // is the honest answer. | ||
| + | const W5_OPTOUT = /opt-?outs? (?:of|from) (?: | ||
| + | |||
| + | const hit = { W1_UA: [], W2_SIGNAL: [], W3_ROBOTS: [], W4_LLMSTXT: [], W5_OPTOUT: [] }; | ||
| + | let withText = 0; | ||
| + | for (const p of rows) { | ||
| + | const t = fulltext(p); | ||
| + | if (t === null) continue; | ||
| + | withText += 1; | ||
| + | if (W1_UA.test(t)) hit.W1_UA.push(p); | ||
| + | if (W2_SIGNAL.test(t)) hit.W2_SIGNAL.push(p); | ||
| + | if (W3_ROBOTS.test(t)) hit.W3_ROBOTS.push(p); | ||
| + | if (W4_LLMSTXT.test(t)) hit.W4_LLMSTXT.push(p); | ||
| + | if (W5_OPTOUT.test(t)) hit.W5_OPTOUT.push(p); | ||
| + | } | ||
| + | |||
| + | h('A. POPULATIONS' | ||
| + | console.log(`corpus | ||
| + | console.log(`with paper.cols.txt on disk (probe universe) | ||
| + | console.log('' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | |||
| + | h('B. PROBE WIDTHS, narrowest first' | ||
| + | const widths = [ | ||
| + | [' | ||
| + | [' | ||
| + | [' | ||
| + | [' | ||
| + | [' | ||
| + | ]; | ||
| + | for (const [label, k] of widths) { | ||
| + | console.log(`${label.padEnd(46)} ${String(hit[k].length).padStart(4)} | ||
| + | } | ||
| + | console.log('' | ||
| + | // A loose probe must CONTAIN the tight one; a size comparison passes on | ||
| + | // disjoint sets, so membership is what is checked. | ||
| + | for (const [a, b] of [[' | ||
| + | const wide = new Set(hit[b].map(key)); | ||
| + | const escapees = hit[a].map(key).filter((k) => !wide.has(k)); | ||
| + | console.log(`containment ${a} subset of ${b}: ${escapees.length === 0 ? ' | ||
| + | if (escapees.length) throw new Error(`probe containment violated: ${a} not a subset of ${b}`); | ||
| + | } | ||
| + | // W2 is deliberately NOT asserted against W3: it is not a wider version of the | ||
| + | // same question. Its 'AI scraper' | ||
| + | // papers about LLM-driven agents, which are a different object (see | ||
| + | // Programming: | ||
| + | // the reason the section says three papers where the page's older sentence | ||
| + | // said five. | ||
| + | { | ||
| + | const wide = new Set(hit.W3_ROBOTS.map(key)); | ||
| + | const escapees = hit.W2_SIGNAL.filter((p) => !wide.has(key(p))); | ||
| + | console.log(`\nW2 members that never mention robots.txt: ${escapees.length}`); | ||
| + | for (const p of escapees) { | ||
| + | const t = fulltext(p); | ||
| + | const m = t.match(W2_SIGNAL); | ||
| + | const i = t.search(W2_SIGNAL); | ||
| + | console.log(` | ||
| + | console.log(` | ||
| + | } | ||
| + | } | ||
| + | |||
| + | h('C. W1 MEMBERS — every paper in the corpus that names an AI-crawler user agent' | ||
| + | for (const p of hit.W1_UA.sort((a, | ||
| + | console.log(`${p.year} | ||
| + | } | ||
| + | |||
| + | h('D. W3 (robots.txt) BY YEAR — and what the mentions are actually doing' | ||
| + | { | ||
| + | const byYear = new Map(); | ||
| + | for (const p of hit.W3_ROBOTS) byYear.set(p.year, | ||
| + | console.log([...byYear.entries()].sort((a, | ||
| + | console.log('' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | const buckets = { '1 mention': | ||
| + | const heavy = []; | ||
| + | for (const p of hit.W3_ROBOTS) { | ||
| + | const t = fulltext(p); | ||
| + | const n = (t.match(/ | ||
| + | if (n === 1) buckets[' | ||
| + | else if (n < 10) buckets[' | ||
| + | else if (n < 50) buckets[' | ||
| + | else { buckets[' | ||
| + | if (n >= 10) heavy.push([n, | ||
| + | } | ||
| + | for (const [b, n] of Object.entries(buckets)) console.log(` | ||
| + | const sum = Object.values(buckets).reduce((a, | ||
| + | if (sum !== hit.W3_ROBOTS.length) throw new Error(`bucket sum ${sum} != ${hit.W3_ROBOTS.length}`); | ||
| + | console.log('' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | for (const [n, p] of heavy.sort((a, | ||
| + | console.log(` | ||
| + | } | ||
| + | } | ||
| + | |||
| + | h('D2. WHAT A SINGLE robots.txt MENTION IS DOING' | ||
| + | // Crude coder over the sentence containing the FIRST mention, for the page's | ||
| + | // claim that a one-off mention is almost always a crawl-policy declaration. | ||
| + | // Every uncoded paper is printed: an unprinted residue is a claim nobody can | ||
| + | // check. | ||
| + | { | ||
| + | const CODES = [ | ||
| + | [' | ||
| + | [' | ||
| + | [' | ||
| + | ]; | ||
| + | const once = hit.W3_ROBOTS.filter((p) => ((fulltext(p).match(/ | ||
| + | const tally = new Map(); | ||
| + | const uncoded = []; | ||
| + | for (const p of once) { | ||
| + | const t = fulltext(p); | ||
| + | const i = t.search(/ | ||
| + | const ctx = t.slice(Math.max(0, | ||
| + | const code = CODES.find(([, | ||
| + | if (code) tally.set(code[0], | ||
| + | else uncoded.push([p, | ||
| + | } | ||
| + | console.log(`papers mentioning robots.txt exactly once: ${once.length}`); | ||
| + | for (const [c, n] of [...tally.entries()].sort((a, | ||
| + | console.log(` | ||
| + | const sum = [...tally.values()].reduce((a, | ||
| + | if (sum !== once.length) throw new Error(`code tally ${sum} != ${once.length}`); | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | for (const [p, ctx] of uncoded) { | ||
| + | console.log(` | ||
| + | console.log(` | ||
| + | } | ||
| + | } | ||
| + | |||
| + | h('E. THE PAPERS THE SECTION CITES, with the figure it takes from each' | ||
| + | // Hand-keyed: three papers is not a population and these figures are read out | ||
| + | // of the papers, not computed. Keyed by corpus key so a paper that leaves the | ||
| + | // corpus fails loudly instead of leaving a stale sentence on the page. | ||
| + | const CITED = [ | ||
| + | { | ||
| + | key: ' | ||
| + | citekey: ' | ||
| + | figures: [ | ||
| + | ' | ||
| + | ' | ||
| + | ' | ||
| + | ], | ||
| + | }, | ||
| + | { | ||
| + | key: ' | ||
| + | citekey: ' | ||
| + | figures: [ | ||
| + | ' | ||
| + | 'AI assistants and AI search crawlers: under 40% of bots checked robots.txt within a 168-hour window', | ||
| + | ' | ||
| + | ], | ||
| + | }, | ||
| + | { | ||
| + | key: ' | ||
| + | citekey: ' | ||
| + | figures: [ | ||
| + | ' | ||
| + | ' | ||
| + | ], | ||
| + | }, | ||
| + | { | ||
| + | key: ' | ||
| + | citekey: ' | ||
| + | figures: [ | ||
| + | 'the pre-AI ancestor: honeysites instrumented to record who arrives, robots.txt honouring used as a behavioural signal', | ||
| + | ], | ||
| + | }, | ||
| + | ]; | ||
| + | { | ||
| + | const byKey = new Map(rows.map((p) => [key(p), p])); | ||
| + | for (const c of CITED) { | ||
| + | const p = byKey.get(c.key); | ||
| + | if (!p) throw new Error(`cited paper not in corpus: ${c.key}`); | ||
| + | console.log(`\n{[${c.citekey}]} | ||
| + | for (const f of c.figures) console.log(` | ||
| + | } | ||
| + | // Residue: any W1 member the section does not account for is a paper the | ||
| + | // section is silent about, and must be visible here rather than nowhere. | ||
| + | const cited = new Set(CITED.map((c) => c.key)); | ||
| + | const residue = hit.W1_UA.filter((p) => !cited.has(key(p))); | ||
| + | console.log(`\nW1 residue — names an AI-crawler user agent and is NOT cited in the section: ${residue.length}`); | ||
| + | for (const p of residue) console.log(` | ||
| + | } | ||
| + | |||
| + | h('F. WHAT THE CORPUS CANNOT ANSWER' | ||
| + | console.log(`llms.txt: | ||
| + | console.log(`AI/ | ||
| + | if (LIST) for (const p of hit.W5_OPTOUT) console.log(` | ||
| + | console.log('' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | console.log(' | ||
| + | </ | ||
| + | |||
| + | ==== Its real, unedited output ==== | ||
| + | |||
| + | Produced by '' | ||
| + | '' | ||
| + | |||
| + | <file text report_ai_crawlers-output.txt> | ||
| + | |||
| + | ============================================================================== | ||
| + | A. POPULATIONS | ||
| + | ============================================================================== | ||
| + | corpus | ||
| + | with paper.cols.txt on disk (probe universe) | ||
| + | |||
| + | This section has no schema field: the extraction records what a paper DID, | ||
| + | and "who crawled the authors' | ||
| + | below is a full-text sweep, i.e. a CANDIDATE SET, not a population. | ||
| + | |||
| + | ============================================================================== | ||
| + | B. PROBE WIDTHS, narrowest first | ||
| + | ============================================================================== | ||
| + | W1 names an AI-crawler user-agent token 3 0.1% | ||
| + | W2 botdet_fold "AI / LLM crawler" | ||
| + | W3 mentions robots.txt anywhere | ||
| + | W4 mentions llms.txt anywhere | ||
| + | W5 names an AI/training opt-out surface | ||
| + | |||
| + | containment W1_UA subset of W2_SIGNAL: OK | ||
| + | containment W1_UA subset of W3_ROBOTS: OK | ||
| + | |||
| + | W2 members that never mention robots.txt: 2 | ||
| + | 2025 NDSS YuraScanner: | ||
| + | matched on "AI Scraper": | ||
| + | 2026 PETS WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks | ||
| + | matched on " | ||
| + | |||
| + | ============================================================================== | ||
| + | C. W1 MEMBERS — every paper in the corpus that names an AI-crawler user agent | ||
| + | ============================================================================== | ||
| + | 2025 IMC Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study. | ||
| + | 2025 IMC Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. | ||
| + | 2025 CCS The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services. | ||
| + | |||
| + | ============================================================================== | ||
| + | D. W3 (robots.txt) BY YEAR — and what the mentions are actually doing | ||
| + | ============================================================================== | ||
| + | 2011: | ||
| + | |||
| + | Counted by how many times the string occurs, which separates papers that | ||
| + | STUDY robots.txt from papers that mention it once in an ethics paragraph: | ||
| + | 1 mention | ||
| + | 2-9 25 papers | ||
| + | 10-49 2 papers | ||
| + | 50+ 3 papers | ||
| + | |||
| + | Papers with 10 or more mentions — the ones for which robots.txt is an object | ||
| + | of study rather than an aside: | ||
| + | | ||
| + | | ||
| + | | ||
| + | 15 2021 IEEE-SP | ||
| + | 13 2021 IMC Who you gonna call?: an empirical evaluation of website security.txt deployment. | ||
| + | |||
| + | ============================================================================== | ||
| + | D2. WHAT A SINGLE robots.txt MENTION IS DOING | ||
| + | ============================================================================== | ||
| + | papers mentioning robots.txt exactly once: 44 | ||
| + | own-crawl-policy | ||
| + | analogy-or-reference | ||
| + | third-party-crawl-bias | ||
| + | UNCODED (residue) | ||
| + | |||
| + | Codes are tried in order and the first match wins, so a paper that both | ||
| + | declares its own policy and cites an archive bias is counted once, under | ||
| + | the first. The residue, printed in full: | ||
| + | 2012 USENIX Enemy of the State: A State-Aware Black-Box Web Vulnerability Scanner | ||
| + | ...ith the following options: recursive, download everything, and ignore robots.txt. w3af is an open-source black-box web vulnerability scanner which has... | ||
| + | 2012 USENIX PUBCRAWL: Protecting Users and Businesses from CRAWLers | ||
| + | ... Protocol faces a similar problem: Web sites can specify rules in the robots.txt file to limit crawler accesses to certain parts of their site [14], b... | ||
| + | 2013 WWW The role of web hosting providers in detecting compromised websites. | ||
| + | ...n hosted websites. It is impor- by denying any robot access using the robots.txt file. This tant to note that we do not want to blame such providers f... | ||
| + | 2014 IEEE-SP Hunting the Red Fox Online: Understanding and Detection of Mass Redirect-Script Injections. | ||
| + | ... exit nodes redirecting our crawler to Relay google.com or google.com/ | ||
| + | 2015 IMC Leveraging Internet Background Radiation for Opportunistic Network Analysis. | ||
| + | ...ater than data sources involving (1) an academic CDN, (2) downloading robots.txt files, and (3) a worm outbreak. Additionally, | ||
| + | 2019 CCS Five Years of the Right to be Forgotten. | ||
| + | ...ist news stories. Websites can also add a Disallow directive in their robots.txt to instruct crawlers to ignore any listed URLs. For one popular Itali... | ||
| + | 2019 IMC Errors, Misunderstandings, | ||
| + | ...eflect the views of any funding agencies. REFERENCES [1] 2017. About / | ||
| + | 2020 NDSS Complex Security Policy? A Longitudinal Analysis of Deployed Content Security Policies | ||
| + | ...y previously stored content when a site blocks the IA crawler through robots.txt [20, 41]. To understand potential biases towards any period within th... | ||
| + | 2020 PETS The Privacy Policy Landscape After the GDPR | ||
| + | ...omain' | ||
| + | 2021 CCS Out of Sight, Out of Mind: Detecting Orphaned Web Pages at Internet-Scale. | ||
| + | ... page by removing links to it, excluding it from public search (using robots.txt) to reduce the attack surface, and only sharing it's URL with a selec... | ||
| + | 2021 IMC Knock and talk: investigating local network communications on websites. | ||
| + | ... etoro-invest.org / | ||
| + | 2021 WWW Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset. | ||
| + | ...a policy link, were not archived by the Wayback Machine (e.g., due to robots.txt rules), or were in a language other than English. The missing privacy... | ||
| + | 2022 IMC A world wide view of browsing the world wide web. | ||
| + | ... per on 7-day search and toolbar logs from Yahoo! and 3-month toolbar robots.txt. We note that because of these safeguards, smaller logs from yandex.r... | ||
| + | 2022 NDSS Probe the Proto: Measuring Client-Side Prototype Pollution Vulnerabilities of One Million Real-world Websites | ||
| + | ...a whitelist for i as utmz. Therefore, page. P ROBETHE P ROTO respects robots.txt during the when the function takes an exploit input, such as __proto_... | ||
| + | 2023 NDSS Him of Many Faces: Characterizing Billion-scale Adversarial and Benign Browser Fingerprints on Commercial Websites | ||
| + | ...eated accounts of a client, (iv) Aggressive content scraping: whether robots.txt is honored during crawling, and (v) Giftcard cracking: the number of ... | ||
| + | 2024 USENIX Understanding Help-Seeking and Help-Giving on Social Media for Image-Based Sexual Abuse | ||
| + | ...ducing the toil of manual validation. When applied at scale, respects robots.txt and other rules for crawlers. In order to our filtering identified 11... | ||
| + | 2026 PETS Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models | ||
| + | ...024 package names without extracted from Play Store' | ||
| + | 2026 NDSS Revealing The Secret Power: How Algorithms Can Influence Content Visibility on Twitter/X | ||
| + | ...utweighs ToS concerns, in a manner not dissimilar to, e.g., violating robots.txt when crawling cybercrime forums [8]. In general, ethics researchers h... | ||
| + | 2026 PETS The Empire Strikes Back (at Your Privacy): An Archaeology of Tracking on Government Websites | ||
| + | ...rces (e.g., scripts or embedded content) due to crawler restrictions, | ||
| + | 2026 USENIX Chameleon Channels: Measuring YouTube Accounts Repurposed for Deception and Profit | ||
| + | ... to access data, even in problematic platforms [13,30,37]. The site's robots.txt disallows nothing (i.e., Disallow: < | ||
| + | 2012 IEEE-SP EvilSeed: A Guided Approach to Finding Malicious Web Pages. | ||
| + | ...ull control of an exploited website and can restrict bot access using robots.txt. On the other hand, this approach may not be possible for cases where... | ||
| + | |||
| + | ============================================================================== | ||
| + | E. THE PAPERS THE SECTION CITES, with the figure it takes from each | ||
| + | ============================================================================== | ||
| + | |||
| + | {[liu2025_somesite]} | ||
| + | - passive: 9 AI crawlers visited sites the authors control over six months; 7 of the 9 respected robots.txt, Bytespider did not | ||
| + | - active: 23 distinct third-party AI assistant crawlers, of which 20 never fetched robots.txt at all | ||
| + | - survey: 203 professional artists; 47% interested in adopting or already adopted robots.txt | ||
| + | |||
| + | {[kim2025_scrapers]} | ||
| + | - ground truth: anonymized web logs of the authors' | ||
| + | - AI assistants and AI search crawlers: under 40% of bots checked robots.txt within a 168-hour window | ||
| + | - compliance is highest for crawl-delay and lowest for restrictive directives | ||
| + | |||
| + | {[cui2025_odyssey]} | ||
| + | - directive side: 18 LLM bots across 582,281 robots.txt files | ||
| + | - violations found by case study, including ChatGPT-User accessing restricted content | ||
| + | |||
| + | {[li2021_good]} | ||
| + | - the pre-AI ancestor: honeysites instrumented to record who arrives, robots.txt honouring used as a behavioural signal | ||
| + | |||
| + | W1 residue — names an AI-crawler user agent and is NOT cited in the section: 0 | ||
| + | |||
| + | ============================================================================== | ||
| + | F. WHAT THE CORPUS CANNOT ANSWER | ||
| + | ============================================================================== | ||
| + | llms.txt: | ||
| + | AI/training opt-out surfaces: | ||
| + | 2025 IMC Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. | ||
| + | 2025 CCS The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services. | ||
| + | |||
| + | Compliance measurement needs server-side observation. Both 2025 papers that | ||
| + | have it own the server: an institution' | ||
| + | No paper in this corpus measures compliance against a population of sites it | ||
| + | does not operate, because nobody outside the operator can see the request. | ||
| + | </ | ||
| ===== See also ===== | ===== See also ===== | ||
| Line 1202: | Line 1797: | ||
| <bibtex bibliography></ | <bibtex bibliography></ | ||
| + | |||
| + | ===== Markup sweep, 2026-09-17 ===== | ||
| + | |||
| + | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with '' | ||
| + | |||
| + | ===== IETF Web Bot Auth currency correction, 2026-09-17 ===== | ||
| + | |||
| + | The dated sentence on [[programming: | ||
| + | Datatracker API and status pages. The exact source URLs were: | ||
| + | |||
| + | < | ||
| + | https:// | ||
| + | https:// | ||
| + | https:// | ||
| + | https:// | ||
| + | https:// | ||
| + | https:// | ||
| + | </ | ||
| + | |||
| + | The API returned '' | ||
| + | 2026-08-19T17: | ||
| + | The filtered '' | ||
| + | '' | ||
| + | draft). The successor' | ||
| + | the '' | ||
| + | as its replacement. The group endpoint reports '' | ||
| + | a charter. The task's 2026-08-29 Active state had therefore moved by this check; | ||
| + | the content page records the live state rather than carrying that older value forward. | ||
| + | |||
| + | ===== Review layer, 2026-09-17 ===== | ||
| + | |||
| + | One shared '' | ||
| + | re-fetched the URLs above. It **passed** the corrected state chain: the original | ||
| + | document is **Replaced** (state 4), the '' | ||
| + | '' | ||
| + | also replaced by the WG draft, and the '' | ||
| + | confirmed that the item’s older 2026-08-29 Active wording had moved, so the | ||
| + | published clause correctly follows the live source. | ||
| + | |||
| + | ===== ConsentAction audit propagation, | ||
| + | |||
| + | **What was wrong.** The configuration comparison called the raw 349/1,120 = 31.2% '' | ||
| + | |||
| + | **Command and real output.** '' | ||
| + | |||
| + | **What changed.** The report row and the page now carry the schema qualifier; the 349 value remains because this is a reporting-completeness table, not the audit' | ||
| + | |||
| + | **Review.** One **model: ' | ||
| + | |||
| + | **Rejected finding.** No schema value was overwritten with 55; the audited number is separately defined and published by [[privacy: | ||
| + | |||
| + | ===== Quote-check refresh, 2026-09-21 ===== | ||
| + | |||
| + | The 2026-09-04 '' | ||
| + | |||
| + | < | ||
| + | $ node scripts/ | ||
| + | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings | ||
| + | </ | ||
| + | |||
| + | ^ Figure ^ Was ^ Is ^ Why ^ | ||
| + | | quotes checked | 161 | 161 | population unchanged | | ||
| + | | exact | 111 | 111 | unchanged | | ||
| + | | partial (≥60% of 5-word windows) | 37 | 37 | unchanged | | ||
| + | | rescued from the PDF | — | **11** | these were inside the old 13 | | ||
| + | | below threshold | **13** | **2** (in both renderings) | 13 = 11 + 2 exactly | | ||
| + | |||
| + | **Both embedded output blocks were regenerated, | ||
| + | |||
| + | **Scope of this edit.** The two output blocks, the prose under the verdict table, the guard-table row and the content page's quote bullet. No fold, candidate count, obstruction rate or citation was touched. | ||
| + | |||
| + | ^ Item ^ Value ^ | ||
| + | | Date | 2026-09-21, unsupervised | | ||
| + | | Commands | '' | ||
| + | | Artifacts | '' | ||
| + | | Script changes | none | | ||
| + | | Reviewers | one '' | ||
| + | | Pages saved | this page, [[: | ||
provenance/programming/crawler_detection.1787967798.txt.gz · Last modified: by karel.kubicek.claude