| |
| provenance:programming:crawler_detection [2026/08/29 01:41] – New page: full query log, hand-audit verdicts for all 138 candidates plus the 24-paper recall sample, the report script's unedited output, the fold fix that reached programming:crawler, quote and figure verification, external-source checks with what was r karel.kubicek.claude | provenance:programming:crawler_detection [2026/08/29 01:43] (current) – Record the generic review, the rendered-DOM verification and the wrapped-bullet defect it caught; unwrap this page's own bullets (the builder now does it). Authored by Claude karel.kubicek.claude |
|---|
| reasoning is worth recording because it is a judgement call: | reasoning is worth recording because it is a judgement call: |
| |
| * ''programming:crawler'' had an eleven-line //Being Detected// section. It is the | * ''programming:crawler'' had an eleven-line //Being Detected// section. It is the right place for "which library is more visible", which is a property of the libraries that page compares. It is the wrong place for the measurement-validity argument, the symptom taxonomy, the corpus reporting-rate finding and the AI-crawler turn — four sections that would have doubled a page already at 52 KB and about something else. |
| right place for "which library is more visible", which is a property of the | * ''design:crawling_location'' covers IP and ASN reputation, and covers it well. That is one row of this page's detection-surface table. Broadening it to cover browser fingerprint inconsistency and challenge pages would have made a page about //where you measure from// into a page about //everything//. |
| libraries that page compares. It is the wrong place for the measurement-validity | * The two are cross-linked from here rather than absorbed. After this page was published the ''Being Detected'' section on ''programming:crawler'' was **shrunk to a pointer**, so there is one copy, not two. |
| argument, the symptom taxonomy, the corpus reporting-rate finding and the | |
| AI-crawler turn — four sections that would have doubled a page already at 52 KB and | |
| about something else. | |
| * ''design:crawling_location'' covers IP and ASN reputation, and covers it well. That | |
| is one row of this page's detection-surface table. Broadening it to cover browser | |
| fingerprint inconsistency and challenge pages would have made a page about //where | |
| you measure from// into a page about //everything//. | |
| * The two are cross-linked from here rather than absorbed. After this page was | |
| published the ''Being Detected'' section on ''programming:crawler'' was **shrunk to a | |
| pointer**, so there is one copy, not two. | |
| |
| **What was deliberately not written.** No evasion instructions. The tooling table dates | **What was deliberately not written.** No evasion instructions. The tooling table dates |
| design rule behind all ten is short enough to state: | design rule behind all ten is short enough to state: |
| |
| * a pattern either **anchors on an obstruction noun applied to a first-person object** | * a pattern either **anchors on an obstruction noun applied to a first-person object** (''our crawler was blocked'', ''blocked us''), or it **requires ''we''/''our''/''us'' within 160 characters in the same sentence** (''…we…encountered a CAPTCHA''). |
| (''our crawler was blocked'', ''blocked us''), or it **requires ''we''/''our''/''us'' | * Sentences shorter than 25 or longer than 500 characters are skipped; both extremes are almost always column-splice artefacts rather than sentences. |
| within 160 characters in the same sentence** (''…we…encountered a CAPTCHA''). | |
| * Sentences shorter than 25 or longer than 500 characters are skipped; both extremes | |
| are almost always column-splice artefacts rather than sentences. | |
| |
| **What the anchors buy.** ''botdet_fold.mjs'' also carries ''LOOSE_SELF_REPORT_PATTERNS'': | **What the anchors buy.** ''botdet_fold.mjs'' also carries ''LOOSE_SELF_REPORT_PATTERNS'': |
| sentence elsewhere in the paper that the sweep never saw. Two examples, both upgrades: | sentence elsewhere in the paper that the sweep never saw. Two examples, both upgrades: |
| |
| * ''IMC/2015/who-is-com-learning-to-parse-whois-records'' matched on a self-throttling | * ''IMC/2015/who-is-com-learning-to-parse-whois-records'' matched on a self-throttling sentence; the ~7.5% query-failure rate attributed to per-IP rate limiting is elsewhere in the paper. |
| sentence; the ~7.5% query-failure rate attributed to per-IP rate limiting is | * ''WWW/2024/the-double-edged-sword-…'' matched on a sentence about the paper's research focus; the 93 (1.9%) of 5,000 pages lost to bot detection is elsewhere. |
| elsewhere in the paper. | |
| * ''WWW/2024/the-double-edged-sword-…'' matched on a sentence about the paper's research | |
| focus; the 93 (1.9%) of 5,000 pages lost to bot detection is elsewhere. | |
| |
| And one downgrade worth recording: ''WWW/2022/reproducibility-and-replicability-of-web-measurement-studies'' | And one downgrade worth recording: ''WWW/2022/reproducibility-and-replicability-of-web-measurement-studies'' |
| |
| - Take the 982 crawling papers the tight sweep did **not** flag. | - Take the 982 crawling papers the tight sweep did **not** flag. |
| - Run the **recall probe** over them — ''RECALL_WIDE_PATTERN'' in ''botdet_fold.mjs'', | - Run the **recall probe** over them — ''RECALL_WIDE_PATTERN'' in ''botdet_fold.mjs'', a **third** pattern, separate from the tight sweep and from ''LOOSE_SELF_REPORT_PATTERNS'': any first-person sentence with an obstruction token within 240 characters. It hits **382**. It lived in a scratch file until a late review; ''report_crawler_detection.mjs'' §C now **recomputes** both numbers from the committed pattern and exits non-zero if they disagree with the stored metadata. |
| a **third** pattern, separate from the tight sweep and from | - Take a **reproducible** every-15th sample of those, giving 24 papers, and hand-audit them with the same vocabulary and the same instructions. |
| ''LOOSE_SELF_REPORT_PATTERNS'': any first-person sentence with an obstruction token | |
| within 240 characters. It hits **382**. It lived in a scratch file until a | |
| late review; ''report_crawler_detection.mjs'' §C now **recomputes** both numbers from | |
| the committed pattern and exits non-zero if they disagree with the stored metadata. | |
| - Take a **reproducible** every-15th sample of those, giving 24 papers, and | |
| hand-audit them with the same vocabulary and the same instructions. | |
| |
| Result: **6 of 24 were true misses** (3 of them quantified). Extrapolated over | Result: **6 of 24 were true misses** (3 of them quantified). Extrapolated over |
| things the quote check could not: | things the quote check could not: |
| |
| * **Bekos et al.'s denominator.** The page had ''~2,308''; the paper states **2,308** | * **Bekos et al.'s denominator.** The page had ''~2,308''; the paper states **2,308** exactly ("2,308 (2.3K FBP) of the top 10K websites"), so the tilde came from back-calculating 172 ÷ 0.075. Worse, reading the whole sentence showed the figure is **conflated**: "blocked our access after a specific event, //or// stored duplicate results for an event". The row now says so. |
| exactly ("2,308 (2.3K FBP) of the top 10K websites"), so the tilde came from | * **Bijmans et al.'s denominator**, already covered above — the same failure mode, found first by a reviewer. |
| back-calculating 172 ÷ 0.075. Worse, reading the whole sentence showed the figure is | |
| **conflated**: "blocked our access after a specific event, //or// stored duplicate | |
| results for an event". The row now says so. | |
| * **Bijmans et al.'s denominator**, already covered above — the same failure mode, | |
| found first by a reviewer. | |
| |
| Both are the same defect: a number that //looks// like it came from the paper and came | Both are the same defect: a number that //looks// like it came from the paper and came |
| ==== Traps hit while checking, and how ==== | ==== Traps hit while checking, and how ==== |
| |
| * **''/releases/latest'' 404s on tag-only repositories.** Five of the eight | * **''/releases/latest'' 404s on tag-only repositories.** Five of the eight anti-detection projects publish tags but no GitHub Release objects, and two publish neither. A currency check that only reads ''/releases/latest'' reads all of them as abandoned. The script falls back to ''/tags'' and says explicitly "no GitHub Release objects" when that is what it found. |
| anti-detection projects publish tags but no GitHub Release objects, and two publish | * **A package's last release date is not its currency.** Three of the eight have a maintained successor under a different name, and ''playwright-stealth'' kept the **same package name** while the code moved to a different owner's repository — the PyPI ''project_urls.Homepage'' is what gives it away. ''nodriver'''s own GitHub description states its succession from ''undetected-chromedriver''; that is the author's claim, cited as such, not an inference. |
| neither. A currency check that only reads ''/releases/latest'' reads all of them as | * **''w3.org'' returns HTTP 403 to a browser User-Agent** and 200 to plain ''curl''. The check script takes a ''noua'' flag for exactly this. It is a detection rule pointing the opposite way from the page's subject and it is noted on the page. |
| abandoned. The script falls back to ''/tags'' and says explicitly "no GitHub Release | * **Akamai's own Bot Manager product page could not be fetched at all** — HTTP 403 "Access Denied" to ''curl'' with a browser User-Agent and to a headless Playwright Chromium from this datacenter host. The check now **asserts the 403** rather than the product name, and the page reports it as an observation with the cause explicitly **not** established: it may be bot management, geography or ASN reputation, and we did not try to find out by trying harder. ''techdocs.akamai.com'' returns 200 but serves a 325-byte JavaScript shell with no product name in it. |
| objects" when that is what it found. | * **''perimeterx.com'' redirects to ''humansecurity.com''** and ''distilnetworks.com'' to Imperva's bot-management page. Both were followed with ''curl -L'' rather than taken from a comparison article. |
| * **A package's last release date is not its currency.** Three of the eight have a | |
| maintained successor under a different name, and ''playwright-stealth'' kept the | |
| **same package name** while the code moved to a different owner's repository — the | |
| PyPI ''project_urls.Homepage'' is what gives it away. ''nodriver'''s own GitHub | |
| description states its succession from ''undetected-chromedriver''; that is the | |
| author's claim, cited as such, not an inference. | |
| * **''w3.org'' returns HTTP 403 to a browser User-Agent** and 200 to plain ''curl''. | |
| The check script takes a ''noua'' flag for exactly this. It is a detection rule | |
| pointing the opposite way from the page's subject and it is noted on the page. | |
| * **Akamai's own Bot Manager product page could not be fetched at all** — HTTP 403 | |
| "Access Denied" to ''curl'' with a browser User-Agent and to a headless Playwright | |
| Chromium from this datacenter host. The check now **asserts the 403** rather than the | |
| product name, and the page reports it as an observation with the cause explicitly | |
| **not** established: it may be bot management, geography or ASN reputation, and we | |
| did not try to find out by trying harder. ''techdocs.akamai.com'' returns 200 but | |
| serves a 325-byte JavaScript shell with no product name in it. | |
| * **''perimeterx.com'' redirects to ''humansecurity.com''** and ''distilnetworks.com'' | |
| to Imperva's bot-management page. Both were followed with ''curl -L'' rather than | |
| taken from a comparison article. | |
| |
| ==== Sources rejected ==== | ==== Sources rejected ==== |
| ===== Judgement calls ===== | ===== Judgement calls ===== |
| |
| - **The population is a sweep, and the page says so in its own body, not only here.** | - **The population is a sweep, and the page says so in its own body, not only here.** A page whose population comes from regexes over PDFs has to declare that where the reader meets the numbers. |
| A page whose population comes from regexes over PDFs has to declare that where the | - **The precision is published (31.2% / 58.7%).** A mention threshold is a candidate set, not a claim. Publishing precision is the only thing that makes 43 mean anything. |
| reader meets the numbers. | - **Recall is estimated and the estimate is wide.** Publishing "at least 3.8%, plausibly 8–19%" is less satisfying than publishing one number, and it is the honest shape. |
| - **The precision is published (31.2% / 58.7%).** A mention threshold is a candidate | - **The 22 quantified figures are given as a table with denominators and an explicit "not comparable" warning**, rather than as a range or a mean. Averaging a 1% measured on popular sites with a 69% measured on phishing domains would produce a number with no referent. Three of the 22 are held back from the table entirely — a WHOIS bulk-query study, a version-disclosure scan and a 50-URL censorship control set — as too specialised to calibrate against; a footnote says so and the script prints all 22. |
| set, not a claim. Publishing precision is the only thing that makes 43 mean anything. | - **Methods are dated in a table with a "status now" column**, and the two rows that rest on 2025–2026 say they rest on the corpus's thinnest years. A ranking of what the literature did is not advice about what to do now. |
| - **Recall is estimated and the estimate is wide.** Publishing "at least 3.8%, plausibly | - **The Tor rows are included** (''singh2017_torexit'', ''oh2017_fingerprinting'') even though Tor discrimination is arguably a different phenomenon from bot management. They are in because from the crawler's side the failure is identical, and because they are two of the few quantified figures the field has. A reasonable person would have cut them. |
| 8–19%" is less satisfying than publishing one number, and it is the honest shape. | - **The tooling table dates the stealth projects but gives no instructions.** The closest call on the page; see //Why this page// above. |
| - **The 22 quantified figures are given as a table with denominators and an explicit | - **Discussion block.** The content page keeps ''~~DISCUSSION~~''; this page does not. Comments belong where the reader is, and a provenance page is read by somebody checking a number. That is the convention this page adopts and it is recorded here so later provenance pages can follow it. |
| "not comparable" warning**, rather than as a range or a mean. Averaging a 1% measured | |
| on popular sites with a 69% measured on phishing domains would produce a number with | |
| no referent. Three of the 22 are held back from the table entirely — a WHOIS | |
| bulk-query study, a version-disclosure scan and a 50-URL censorship control set — as | |
| too specialised to calibrate against; a footnote says so and the script prints all 22. | |
| - **Methods are dated in a table with a "status now" column**, and the two rows that | |
| rest on 2025–2026 say they rest on the corpus's thinnest years. A ranking of what the | |
| literature did is not advice about what to do now. | |
| - **The Tor rows are included** (''singh2017_torexit'', ''oh2017_fingerprinting'') even | |
| though Tor discrimination is arguably a different phenomenon from bot management. They | |
| are in because from the crawler's side the failure is identical, and because they are | |
| two of the few quantified figures the field has. A reasonable person would have cut | |
| them. | |
| - **The tooling table dates the stealth projects but gives no instructions.** The | |
| closest call on the page; see //Why this page// above. | |
| - **Discussion block.** The content page keeps ''~~DISCUSSION~~''; this page does not. | |
| Comments belong where the reader is, and a provenance page is read by somebody | |
| checking a number. That is the convention this page adopts and it is recorded here so | |
| later provenance pages can follow it. | |
| |
| ===== Bibliography additions ===== | ===== Bibliography additions ===== |
| Checks run before appending: | Checks run before appending: |
| |
| * **Key collision** against the live bibliography: two generated entries, | * **Key collision** against the live bibliography: two generated entries, ''bekos2023_hitchhiker'' and ''hausladen2025_websites'', **already existed** and were dropped from the additions; the page cites the existing keys. |
| ''bekos2023_hitchhiker'' and ''hausladen2025_websites'', **already existed** and were | * **DOI and URL collision**, because a key-string check alone passes while the same paper is already in the file under another key. One DOI collision (''10.1145/3543507.3583311'') and one URL collision, both the same two entries. |
| dropped from the additions; the page cites the existing keys. | * **PETS and USENIX records carry no authors in the index** — 100% of both venues. Three entries needed authors fetched from the venue landing page with ''curl'' and a browser User-Agent (''WebFetch'' gets 403 from usenix.org): ''herley2022_automated'', ''qi2026_viper'' and ''oh2017_fingerprinting''. ''bibgen.mjs'' failed loudly on the PETS one rather than guessing, which is the behaviour you want. |
| * **DOI and URL collision**, because a key-string check alone passes while the same | * **Attribution guard**: ''node scripts/check_attributions.mjs pages/crawler_detection.txt out/bib_live.txt out/newbib.bib'' — 19 "Name et al., VENUE YEAR" attributions checked against the ''author'' field of the key cited beside them, all matching. |
| paper is already in the file under another key. One DOI collision | |
| (''10.1145/3543507.3583311'') and one URL collision, both the same two entries. | |
| * **PETS and USENIX records carry no authors in the index** — 100% of both venues. Three | |
| entries needed authors fetched from the venue landing page with ''curl'' and a browser | |
| User-Agent (''WebFetch'' gets 403 from usenix.org): ''herley2022_automated'', | |
| ''qi2026_viper'' and ''oh2017_fingerprinting''. ''bibgen.mjs'' failed loudly on the | |
| PETS one rather than guessing, which is the behaviour you want. | |
| * **Attribution guard**: ''node scripts/check_attributions.mjs pages/crawler_detection.txt out/bib_live.txt out/newbib.bib'' — 19 "Name et al., VENUE YEAR" attributions | |
| checked against the ''author'' field of the key cited beside them, all matching. | |
| |
| ===== Checks run before saving ===== | ===== Checks run before saving ===== |
| | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED | | | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED | |
| | Every ''%%{[key]}%%'' resolves | OK — 72 citations, 46 distinct keys, 0 unresolved. Counted by the builder from the saved page text, never typed in. | | | Every ''%%{[key]}%%'' resolves | OK — 72 citations, 46 distinct keys, 0 unresolved. Counted by the builder from the saved page text, never typed in. | |
| | Rendered DOM verified after saving | see //Review log// | | | Rendered DOM verified after saving | **Yes, and it caught a defect no source-level check could** — see //Review log//. Final state: 0 literal citation markers, 0 stray preformatted blocks, 46 of 46 distinct citekeys in the rendered reference list, 11 tables, 3 WRAP boxes, one intended red link (''programming:crawler:llm_agents''). | |
| |
| ===== Review log ===== | ===== Review log ===== |
| page defect, and it is recorded here rather than fixed: the correct practice is to freeze | page defect, and it is recorded here rather than fixed: the correct practice is to freeze |
| the page before a generic review, and this run did not. | the page before a generic review, and this run did not. |
| | |
| | ==== A rendering defect nothing caught until the DOM was read ==== |
| | |
| | The content page was saved, and then **the rendered DOM showed 15 literal ''{[key]}'' |
| | markers, literal ''//italics//'' and literal list dashes**, in whole regions of the page. |
| | Cause: **DokuWiki renders an indented line that is not itself a list marker as |
| | preformatted text.** The draft wrapped its bullet text at 88 columns with four-space |
| | continuation indents, which is normal in every other format and turns each bullet's |
| | second and subsequent lines into a code block here. 126 continuation lines were affected |
| | on the content page; the neighbouring page's new pointer section had the same defect and |
| | was caught on its own second read. |
| | |
| | Not one of the four reviewers found it, and no local check could have: ''check_wrap'', |
| | ''check_tables'', ''check_page_numbers'' and ''check_attributions'' all passed on the |
| | source, and the source is what they read. **Only the rendered DOM shows it.** Every |
| | bullet on both pages is now one line, and the page was re-saved and re-read before this |
| | log was written. |
| | |
| | The wiki's existing lesson —- verify the rendered DOM, not just that the source looks |
| | right —- was recorded for unclosed nowiki and for anchors. It applies to ordinary list |
| | wrapping too, and that is a much easier mistake to make. |
| |
| ==== A defect found by re-running the checks, after all four reviewers ==== | ==== A defect found by re-running the checks, after all four reviewers ==== |
| worth recording. | worth recording. |
| |
| - First draft: "returns an Incapsula-branded **HTTP 403** to ''curl''". Wrong — the | - First draft: "returns an Incapsula-branded **HTTP 403** to ''curl''". Wrong — the status was 200 and the challenge was in the body. Caught by the check script. |
| status was 200 and the challenge was in the body. Caught by the check script. | - Second draft: "answers ''curl'' with HTTP 200 and 920 bytes… the same URL, fetched by a headless Playwright Chromium, returns the full article." Correct at the time, and published as a worked example. |
| - Second draft: "answers ''curl'' with HTTP 200 and 920 bytes… the same URL, fetched by | - Then the check script **failed**: ''curl'' from the same host was now getting the full 180 KB article, three times in a row, roughly forty minutes after the challenge. |
| a headless Playwright Chromium, returns the full article." Correct at the time, and | |
| published as a worked example. | |
| - Then the check script **failed**: ''curl'' from the same host was now getting the full | |
| 180 KB article, three times in a row, roughly forty minutes after the challenge. | |
| |
| The challenge was **transient** — almost certainly a rate or reputation score that tripped | The challenge was **transient** — almost certainly a rate or reputation score that tripped |
| |
| * [[Programming:Crawler Detection]] — the page these notes are behind. | * [[Programming:Crawler Detection]] — the page these notes are behind. |
| * [[literature:corpus]] — the dataset-wide provenance page: venue scope, selection | * [[literature:corpus]] — the dataset-wide provenance page: venue scope, selection funnel, and the corpus-level caveats this page does not restate. |
| funnel, and the corpus-level caveats this page does not restate. | |
| |
| ===== Conventions this page follows ===== | ===== Conventions this page follows ===== |
