Old Revisions

These are the older revisons of the current document. To revert to an old revision, select it from below, click Edit this page and save it.

2026/09/21 13:42 programming:crawler_detection – Quote bullet: the 13 below-threshold quotes split 11 rescued from the PDF / 2 below in both. Authored by Claude karel.kubicek.claude +193 B (current)
2026/09/17 10:45 programming:crawler_detection – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude +400 B
2026/09/17 09:18 programming:crawler_detection – Correct IETF Web Bot Auth state and replacement clause; Authored by Claude karel.kubicek.claude +51 B
2026/09/17 07:22 programming:crawler_detection – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude +8 B
2026/09/16 21:22 programming:crawler_detection – Reorder: the Cloudflare Web Bot Auth note now follows the 'nobody has published it' paragraph instead of interrupting it. Authored by Claude karel.kubicek.claude ±0 B
2026/09/16 21:20 programming:crawler_detection – Review fixes: name both non-compliers in Liu et al.'s passive result; propagate the five-to-three correction to the signal-table caption; restate the 44 single-mention robots.txt papers without overclaiming the coder; add Cloudflare's now-deployed Web Bot karel.kubicek.claude +2.3 KB
2026/09/16 21:08 programming:crawler_detection – Add 'Measuring who crawls you' to the AI-crawler section: directive vs compliance, honeysite and operator-log designs, the two IMC 2025 papers that disagree, corpus thinness. Correct 'five papers' to three (two were agentic-browser false positives). Autho karel.kubicek.claude +6.6 KB
2026/08/29 01:41 programming:crawler_detection – Apply generic review: hedge the HTTP-200 challenge claim, narrow the rank-breakdown negative and cite DeBlasio's top-100 split, replace 'read by hand' with what was actually done, drop adoption language and licences from the tooling table, hedge the indep karel.kubicek.claude +1.8 KB
2026/08/29 01:38 programming:crawler_detection – Fix rendering: DokuWiki treats an indented continuation line as preformatted text, so every wrapped bullet was rendering literally (126 lines). Bullets are now one line each. Authored by Claude karel.kubicek.claude -504 B
2026/08/29 01:37 programming:crawler_detection – New page: bot management, headless/CDP leaks, CAPTCHAs and WAF challenge pages, and what a blocked crawl does to the headline number. Population is a hand-audited full-text sweep (138 candidates, 31.2% precise); 43 of 1,120 crawling papers report their ow karel.kubicek.claude +44.7 KB