Old Revisions

These are the older revisons of the current document. To revert to an old revision, select it from below, click Edit this page and save it.

2026/09/17 10:45 programming:crawler – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude +427 B (current)
2026/09/17 08:12 programming:crawler – Remove stale live Artifacts caveat; Authored by Claude karel.kubicek.claude -33 B
2026/09/17 07:22 programming:crawler – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude +8 B
2026/09/11 17:55 programming:crawler – webXray bullet: repoint the ownership-comparison pointer to the new Design:Ownership resolution, which the material moved to on 2026-09-11. The comparison-row and licence corrections this page owed were already applied (revs 1786953266 and 1788636934); se karel.kubicek.claude +202 B
2026/09/05 19:35 programming:crawler – Correct the webXray licence claim: thezedwards is the most mirrored copy, not the most complete, and webXray ended MIT-licensed, so a lawful route to the code does exist. Authored by Claude karel.kubicek.claude +283 B
2026/09/03 21:06 programming:crawler – Review fixes (generic pass): shorten the agent detection bullet to a pointer, drop the dangling ordinal and the Browser-Use search claim generalised from a non-crawl paper, name the referent for the vendor defaults; 'Since 2025' -> 'Since 2026', which is karel.kubicek.claude -455 B
2026/09/03 21:00 programming:crawler – Recommendations: 75 -> 74 unnamed bespoke crawlers, a figure the 2026-08-29 LLM-agent fold moved outside the Use in Publications window. Authored by Claude karel.kubicek.claude ±0 B
2026/09/03 20:56 programming:crawler – Being Detected: add third bullet on the agent-driven detection surface, cross-linking Programming:Crawler:LLM Agents and Crawler Detection (llm-agent-crawlers item). Authored by Claude karel.kubicek.claude +1.1 KB
2026/09/03 08:42 programming:crawler – Fix two Tracker Radar claims found while reviewing the TRC subpage: footnote the 21-paper row with its 10 crawler / 11 dataset split (the tool_fold family is named after the crawler but matches the dataset), and replace the stale README-derived 'caps conc karel.kubicek.claude +1.4 KB
2026/08/29 06:37 programming:crawler – LLM-agent fold: new family row (4 papers), bespoke own-name 184->181, union 321->318 (28.7%->28.4%), 75->74, residue 204/210->199/205; new Agent-Driven Crawling section pointing at programming:crawler:llm_agents. Authored by Claude karel.kubicek.claude +2.2 KB
2026/08/29 01:42 programming:crawler – Fix rendering of the new pointer section: DokuWiki treats an indented continuation line as preformatted, so the two wrapped bullets rendered literally. Authored by Claude karel.kubicek.claude -28 B
2026/08/29 01:41 programming:crawler – Shrink 'Being Detected' to a pointer at the new programming:crawler_detection, which now covers it in full; keep the two points specific to choosing a library. Also correct the anti-detection patches row from 12/2 to 11/1: tool_fold.mjs matched 'Stealth A karel.kubicek.claude -739 B
2026/08/21 08:34 programming:crawler – [Open Questions] karelkubicek ±0 B
2026/08/17 21:45 programming:crawler – Update the PanoptiChrome row and taint-tracking note now that Programming:Crawler:PanoptiChrome exists: real output format, the published patch does not compile, and start with Foxhound between the two. Authored by Claude karel.kubicek.claude +544 B
2026/08/17 07:54 programming:crawler – Correct the webXray paragraph: the newest surviving copy is 2021 and drives consumer Chrome over CDP, not a 2015 PhantomJS mirror; webXray is now the commercial webxray.ai, which closes the open TODO; add its PolyForm Strict no-redistribution licence and karel.kubicek.claude +288 B
2026/08/12 15:45 programming:crawler – Methodology section: replaced the generic corpus caveats (absent venues, provisional 2025-2026) with a pointer to the new literature:corpus page, keeping the page-specific consequence. Authored by Claude. karel.kubicek.claude +1 B
2026/08/12 10:05 programming:crawler – Review pass (Fable): fix stale figures that sat OUTSIDE the corpus section and so were missed by the windowed staleness guard. 11.6->12.0%, 21.8->21.6%, 6.3->6.8%, 4.1->3.8%, 12.9->12.5%, 34->75 unnamed bespoke crawlers; Playwright rewritten (10 -> 34 pap karel.kubicek.claude +324 B
2026/08/12 09:16 programming:crawler – Refresh all corpus figures for the extended 2010-2026 extraction (4,322 -> 5,859 papers; crawled 859 -> 1,120). Adds a provisional 2025-2026 column; Playwright overtakes Puppeteer there. Browser fold extended (15 unmapped strings -> 1). Report script now karel.kubicek.claude +1.6 KB
2026/08/06 07:44 programming:crawler – Correct the BiDi finding: responseCompleted DOES carry the status; the real defect is that registering two network handlers cross-delivers both event types (8/8 events, only 4 with response data), which makes URL-keyed deduplication lose the status. Also: karel.kubicek.claude +2 KB
2026/08/06 07:37 programming:crawler – Two wording fixes: Puppeteer is no longer Chromium-only, and Selenium is the top off-the-shelf library rather than the top tool. Authored by Claude. karel.kubicek.claude +20 B