Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler [2026/08/12 09:17] – Create provenance page for programming:crawler: queries, denominators, fold changes and residue, quote verification, and the 4,322 -> 5,859 refresh diff. Partly reconstructed after the fact; marked as such. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:crawler [2026/09/21 14:40] (current) – Citations pass: do not invent the paper's dataset notation; describe the 42,288-webpage subset and quote the sentence that defines it. Authored by Claude karel.kubicek.claude
@@ Line 16: / Line 16: @@
 | Runnable comparison | ''sandbox/fixture.mjs'', ''sandbox/run_{playwright,puppeteer,cdp,selenium}.mjs'', ''sandbox/compare.sh'' |
 | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
-| Refreshed | 2026-08-12 |
+| Refreshed | 2026-08-12; extended 2026-08-29 (§5.3) and 2026-09-03 (§12) |
 ===== 2. Populations and denominators =====
@@ Line 37: / Line 37: @@
 node scripts/report_crawler.mjs                       # every figure on the page
 node scripts/report_crawler.mjs --wiki                # the same, as DokuWiki tables
-node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
+node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
 node scripts/check_page_numbers.mjs \
   pages/programming_crawler.txt out/new/report_crawler.txt \
@@ Line 43: / Line 43: @@
 </code>
-''check_page_numbers.mjs'' left four figures unaccounted on 2026-08-12, all deliberate: ''100'' ("not against 100%"), ''19.1'' (the crawled share of the corpus, from ''data/extract/OVERVIEW.md''), ''53'' (the **old** OpenWPM count, quoted on the page as history), and ''93'' — which was a stale number from an intermediate run of the quote check and was corrected to 106 before saving. That last one is the guard doing exactly the job it exists for.
+''check_page_numbers.mjs'' left four figures unaccounted on 2026-08-12. **Three are permanent and legitimate** and are named here so the next run does not re-investigate them: ''100'' ("not against 100%"), ''19.1'' (the crawled share of the corpus, from ''data/extract/OVERVIEW.md'') and ''106'' (the quote-check total, from ''out/new/quote_check_crawler.txt'', not from this report). A fourth, ''53'', is the **old** OpenWPM count quoted on the page as history. The guard also caught a real error: the page briefly carried ''93'' from an intermediate run of the quote check, corrected to 106 before saving. Run it whole-page too (§10).
 ===== 4. What the refresh changed =====
@@ Line 55: / Line 55: @@
 | Names neither | 248 (28.9%) | 311 (27.8%) |
 | Selenium | 187 (21.8%) | 242 (21.6%) |
-| Bespoke, own name | 98 (11.4%) | 184 (16.4%) |
+| Bespoke, own name | 98 (11.4%) | 184 (16.4%) → **181 (16.2%)** on 2026-08-29((The LLM-agent fold, §5.3.)) |
 | Bespoke, generic | 130 (15.1%) | 147 (13.1%) |
-| Bespoke, union | 221 (25.7%) | 321 (28.7%) |
+| Bespoke, union | 221 (25.7%) | 321 (28.7%) → **318 (28.4%)** on 2026-08-29 |
-| …of which named nothing else | 35 | 75 |
+| …of which named nothing else | 35 | 75 → **74** on 2026-08-29 |
 | Puppeteer | 54 (6.3%) | 76 (6.8%) |
 | OpenWPM | 51 (5.9%) | 58 (5.2%) |
@@ Line 71: / Line 71: @@
 | Any tool states its version | 63 of 544 (11.6%) | 87 of 723 (12.0%) |
 | Distinct tool strings / mentions | 378 / 789 | 501 / 1,075 |
-| Unmapped residue | 104 strings / 106 mentions | 204 / 210 |
+| Unmapped residue | 104 strings / 106 mentions | 204 / 210 → **199 / 205** on 2026-08-29 |
 | Exact string "Selenium" vs folded | 147 vs 187 | 194 vs 242 |
 | Artifact share, 2010–2013 → 2022–2024 | 23.6% → 62.1% | 21.6% → 61.2% |
@@ Line 87: / Line 87: @@
   * **Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table.** The old page's closing sentence for that table already warned that "the 2024 boundary cuts them off mid-growth". It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it.
   * **"Names a library, not a browser" 23 → 49**, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write "a Playwright-controlled browser", "Playwright-derived browser", "Puppeteer/TRC desktop browser".
-  * **Bespoke, own name 98 → 184.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names.
+  * **Bespoke, own name 98 → 184, then 181.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names. On **2026-08-29** the LLM-agent family (§5.3) claimed five of those strings, moving the row to 181 and the union to 318.
   * **Artifact share 23.6% → 21.6% for 2010–2013.** A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets.
@@ Line 104: / Line 104: @@
 Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers
   (exact-string counting would undercount by 19.8%).
-Bespoke: 147 generic + 184 own-name, 10 in both, union 321 of 1120 (28.7%).
+Bespoke: 147 generic + 181 own-name, 10 in both, union 318 of 1120 (28.4%).
 Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct),
   Scrapy, OpenWPM: 215.
 </code>
-The 204-string residue is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below.
+The residue — 204 strings until 2026-08-29, 199 after the LLM-agent fold — is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below.
 ==== 5.2 Browser families — the fold that had gone stale ====
@@ Line 130: / Line 130: @@
 ==== 5.3 What is new in the framework residue, and what was deliberately not done with it ====
-//Recorded.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' (two spellings, two papers), ''AgentLab'', ''BrowserGym'', ''MCP Server Crawler''. A sixth paper, ''USENIX/2026/the-state-of-passkeys…'', drives Playwright //through// Browser Use.
+//Recorded 2026-08-12; **resolved 2026-08-29**.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' / ''Browser Use'' (three spellings, three papers), ''AgentLab'', ''BrowserGym''. A sixth string, ''MCP Server Crawler'', was listed with them on 2026-08-12 and **that was wrong** — see below.
-**This was deliberately not folded into a family and not written up on the page.** The refresh brief for this run is a mechanical re-derivation, and "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers — five or six papers in the thinnest, provisional years of the corpus is not enough to call it current practice. It is logged as a separate work item instead. Anyone picking it up should start from this residue list and from ''programming:crawler''’s //Being Detected// section, since an agent-driven browser is a different detection surface.
+The 2026-08-12 run deliberately did not fold it, on the grounds that "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers. That work was done on **2026-08-29**. The outcome:
+  * A family **''LLM browser agents''** now exists in ''tool_fold.mjs'', ordered //before// Playwright and Puppeteer, because Browser Use and Skyvern drive Playwright underneath and would otherwise be counted as Playwright.
+  * It claims **4 of the 1,120 crawling papers (0.4%)**, all of them 2026. That is the row now on [[programming:crawler]], and the five figures it moved are in §4 above.
+  * ''MCP Server Crawler'' was **excluded**, and the 2026-08-12 note that grouped it with the agents is corrected here. Reading IEEE-SP/2026 //Parasites in the Toolchain// shows it is an ordinary crawler over MCP directory listings and GitHub links — "we implemented the MCP Server Crawler to leverage these links as the entry point for data collection". The name describes what it crawls, not what drives it. It stays in the printed residue. ''WebAgentDriver'' (NDSS/2026 //ChameleoScan//), the Appium XCUITest driver on jailbroken iOS, is excluded for the same kind of reason.
+  * The dating question was answered rather than deferred: **9 papers in 5,859 name an agent framework at all, 1 in 2025 and 8 in 2026**, and six of the nine //measure// agents rather than crawl with them. It is not current practice and the new page says so.
+  * The page is [[programming:crawler:llm_agents]] and its own provenance is [[provenance:programming:crawler:llm_agents]], which carries the full query log, the fold rule, the residue and the quote adjudication.
 A second, smaller cluster in the same residue: consent-automation tools used as instruments — ''autoconsent'', ''DuckDuckGo autoconsent'', ''Super Agent-Automatic Cookie Consent'', ''ConsentChk''. Those belong to [[privacy:consent]] rather than here.
@@ Line 139: / Line 145: @@
 //Recorded, 2026-08-12.// ''scripts/quote_check.mjs'' is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure ("66 quotes, 44 exact") cannot be reproduced.
+**Re-run 2026-09-21** with the PDF fallback — see //Quote-check refresh, 2026-09-21// at the foot of this page:
 <code>
-$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
+$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
-quotes checked: 58 exact, 32 partial (>=60% of 5-word windows),
+quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.
- below threshold, 0 with no full text on disk.
 </code>
-All **16** below-threshold quotes were then read by hand against ''paper.cols.txt'' with whitespace normalised. **All sixteen are present in the paper. None was unsupported.** Two representative failures:
+**The old figure was 16 below threshold.** 13 of the 16 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are therefore a defect in the stored text, not in the extraction; **3** are below threshold in both renderings. ''exact'' (58) and ''partial'' (32) did not move, and the below-threshold-in-both rate is **2.8%** of 106.
+All **16** of the then-below-threshold quotes had already been read by hand against ''paper.cols.txt'' with whitespace normalised, before the fallback existed. **All sixteen are present in the paper. None was unsupported** — a hand result the fallback independently confirms for 13 of them. Two representative failures, one from each bucket:
+  * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker. This one is **still below threshold in both renderings** (50% of windows in the stored text), because a dropped marker is in the extraction's own string and no rendering of the PDF can repair it. It is one of the 3.
+  * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word. This one is **rescued**: 33% of windows in the stored text, 67% in the PDF, which is what a column-repair failure looks like when the second rendering happens not to splice the same sentence.
-  * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker.
+**Those two examples are the distinction the fallback draws.** A quote the extraction itself mangled fails in every rendering; a quote the //de-columner// mangled usually does not. That is why 13 of 16 moved and 3 did not.
-  * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word.
 The lesson for anyone reusing ''evidence.quote'': a failed exact match on ''.cols.txt'' is weak evidence of anything. Check the distinctive noun phrase, not the sentence.
@@ Line 174: / Line 185: @@
   * **The sandbox measurements were not re-run.** The Selenium/Puppeteer/Playwright/CDP comparison, the WebDriver BiDi ''responseCompleted'' finding and the "two network handlers cross-deliver" bug are from 2026-08-06 against ''selenium-webdriver'' 4.46. Library versions have moved since. Nothing in this refresh touched them, and nothing in this refresh verified them.
-===== 9. Run log =====
+===== 10. Review pass, 2026-08-12 =====
+//Recorded.// The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a **systematic defect, not a scatter of typos**, and it is worth stating because it will recur on the next refresh:
+  * ''check_page_numbers.mjs'' was run with a **heading window** — normally ''Use in Publications'' to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's **intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block** was outside the window and stayed at its 4,322-corpus value. Across the six pages **29 such figures** survived the first pass.
+  * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason.
+Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''%%--code%%'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean.
+Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced:
+  * The Playwright section still said "still rare in the literature — 10 papers", two screens from a corpus table saying 34 and a sentence saying Playwright overtakes Puppeteer. Rewritten.
+  * "OpenWPM users are more than twice as good as average … Playwright's cohort is next best" **ranked the two backwards** against the table directly above it (Playwright 26.5%, OpenWPM 25.9%). On the old corpus Playwright led 50.0% to 25.5% on a base of ten papers; with 34 they are effectively tied, and the sentence now says so.
+  * The framework table had drifted out of descending order after the refresh moved three rows. Restored.
+**''106'' is a permanent legitimate unaccounted figure** on this page: it is the quote-check total, from ''out/new/quote_check_crawler.txt'', not from ''report_crawler.mjs''. Recorded here so the next run does not re-investigate it. §3 above previously said "four … all deliberate" while listing one figure (''93'') that was a real error — corrected.
+===== 11. Run log =====
 ^ ^ ^
@@ Line 185: / Line 212: @@
 | Claim weakened on purpose | The old page said the bespoke rows together were "more than used every off-the-shelf library on this page combined". That was not true of the old numbers either (187+54+35+10 = 286 > 221) and is not true of the new ones. Replaced with the two comparisons the report actually prints: more than Selenium (242), and more than the 215 papers naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM. |
 | Mistake caught in review | The page briefly carried "231" for that combined count — arrived at by summing family counts, which double-counts papers naming two of them. The real union is 215. ''check_page_numbers.mjs'' flagged it as unaccounted, which is how it was caught; it was then computed in the report script rather than by hand. |
-| Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. |
+| Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. **Resolved 2026-08-29** — see §5.3 and §12. |
+| Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 7 stale figures plus a mis-ordered table and a backwards ranking sentence on this page. All fixes were applied and re-saved the same day. |
+===== 12. Follow-up, 2026-09-03: the LLM-agent fold re-verified, and Being Detected closed =====
+//Recorded as the run went, tidied at the end.// Work item ''llm-agent-crawlers'' asked three
+things of this page: whether the 2025–2026 agent strings should become a family in
+''tool_fold.mjs'', whether [[programming:crawler]] should carry it, and what it does to the
+//Being Detected// section. The first two were answered on 2026-08-29 (§5.3 above, and §4.3
+of [[provenance:programming:crawler:llm_agents]]). This run **verified** them instead of
+taking that note at face value, and closed the third.
+==== 12.1 Verification of the fold, against the current run1 ====
+''node scripts/report_crawler.mjs'' re-run on 2026-09-03 against ''data/extract/run1''
+(5,859 papers; population ''crawled'' = 1,120). Exit 0. Every figure in the content page's
+//Use in Publications// section matches the fresh output, including the rows the 2026-08-29
+fold moved:
+^ Checked ^ Fresh script output ^ On the content page ^
+| ''LLM browser agents'' family | 4 papers, 0.4% of 1,120 | 4 / 0.4%, framework table |
+| Bespoke, own name (residue row) | 181, 16.2% | 181 / 16.2%, framework table |
+| Bespoke union / of which named nothing else | 318 of 1,120 (28.4%) / 74 | 318 (28.4%) / 74, the ''WRAP important'' box |
+| Selenium | 242, 21.6% | 242 / 21.6% |
+| Puppeteer / Playwright | 76 (6.8%) / 34 (3.0%) | same |
+| Framework residue | 199 distinct strings, 205 paper-mentions | 199 / 205, //Methodology and limitations// |
+| Browser residue | 1 distinct, ''Ghostery'' | same, //Methodology and limitations// |
+Grepping the fresh residue listing for every agent-shaped string
+(''browser.?use'', ''agentlab'', ''browsergym'', ''skyvern'', ''stagehand'', ''crawl4ai'',
+''firecrawl'', ''computer.use'', ''operator'', ''mcp'', ''webagent'') returns exactly one
+line: ''MCP Server Crawler''. That is the deliberate exclusion recorded in §5.3, so the fold
+is behaving as documented and nothing new has arrived in the residue since 2026-08-29.
+The full output is now written to ''scripts/report_crawler-output.txt'' in the wiki workdir,
+next to the script. §5.1 above still points at ''out/new/report_crawler.txt'', a path in a
+working directory; that reference was **not** edited, so both now exist and the committed
+one is the readable one.
+==== 12.2 The one edit: a bullet in //Being Detected// ====
+The content page gained an //Agent-Driven Crawling// section on 2026-08-29, but
+//Being Detected// still opened with "Two points from there bear on the choice made on this
+page" and both points were about scripted automation. Neither neighbour is at fault —
+[[programming:crawler_detection]] says in three places that an agent is a different surface
+(its intro, the close of //What a Blocked Crawl Does to Your Number//, and //Related Pages//),
+and [[programming:crawler:llm_agents]] has a full section on it — but a reader who takes the
+agent option on //this// page was given no signal on the way past. Added one bullet and
+changed "Two points from there" to "Three points". Revisions ''1788424922'' →
+''1788468979'' → (after review) ''1788469575''.
+**The first version of that bullet was wrong in four ways and was rewritten**, see §12.5.
+It is worth recording what it contained, because the rewrite deleted a citation:
+^ Dropped ^ Why ^
+| A verbatim quote from Jeong et al. (''%%{[jeong2026_network]}%%'') on Browser-Use issuing Google queries via ''search_web'' and triggering reCAPTCHA | The quote is real and was checked against ''data/fulltext/2026/USENIX/network-level-prompt-and-trait-leakage-in-local-research-agents/paper.cols.txt'' (verbatim; note the ''.cols'' rendering interleaves the caption of Table 14 mid-sentence, so a plain ''grep'' for the phrase fails and whitespace must be collapsed first). What was wrong was the **use**: Jeong et al. run Browser-Use as a research agent on open queries, whereas a measurement crawl hands the agent a URL, and the two actual agent-crawl papers report no search-engine challenge at entry. Generalising it to "the open-source agents make it worse by default" was not supported. It also duplicated the child page's own first detection bullet almost word for word. |
+''jeong2026_network'' is therefore **no longer cited on this content page**; it remains cited
+on [[programming:crawler:llm_agents]], where the claim is correctly hedged. No bibliography
+entry was added, changed or removed at any point in this run.
+Rendered checks after each save, with ''purge=true'': after the first save, references
+→ 15 with Jeong present and citekey spans 36 → 38 (the plugin emits two per marker);
+after the rewrite, back to **14** with Jeong absent. Elements matching
+''%%<h[123] … id=%%'' stayed at **25**, ''%%<table%%'' at **12** and ''%%<pre%%'' at **5**
+across all three revisions, and ''wikilink2'' red links stayed at **0**. (Those are counts
+of that regex, not of every heading in the DOM — a reviewer counting all ''h1''/''h2''/''h3''
+gets 29. The point of the count is that it did not change, not its absolute value.) The
+anchor ''#being_detected_and_the_ethics_of_not_being'' was taken from the child page's
+**rendered** heading id, not guessed, and the saved page's ''href'' matches it exactly.
+==== 12.3 A stale figure the 2026-08-29 fold left behind ====
+//Found on 2026-09-03, after the edit above, by running the number guard whole-page.//
+The save summary for revision ''1787985453'' says the fold moved **75 → 74** unnamed bespoke
+crawlers, and it did — in the ''WRAP important'' box under the framework table. The **same
+figure in the //Recommendations// section was not moved** and shipped as ''75'' for five days
+across three subsequent saves. Fixed in revision ''1788469206''.
+This is the exact failure §10 above describes, one refresh later, and it had a specific
+disguise: the whole-page guard //does// list ''75'' as unaccounted, but the context it prints
+is the **first** occurrence of the string, which on this page is the sandbox sentence "a
+silent 75% data loss". Anyone reading the guard output and recognising that sentence as a
+legitimate non-corpus figure clears the line and never reaches the second occurrence. The
+same trap is still there today: after the fix, ''75'' is still reported, still with the
+sandbox context, and now legitimately.
+Command:
+<code>
+node scripts/check_page_numbers.mjs \
+  pages_crawler_live.txt scripts/report_crawler-output.txt
+</code>
+with **no** heading markers. The whole-page run also reports 23 other unaccounted figures
+and 3 version identifiers. All were read; **none is a corpus figure.** They are the sandbox
+measurements (779 / 877 / 2038 / 1329 / 1838 ms, HTTP 200 and 204, ''selenium-webdriver''
+.46, Node 22.23, Firefox 153, Chromium 151.0.7922.34), figures quoted from other papers
+(PanoptiChrome 50% and 36.7× against Foxhound's 95% and 1.4×), version constraints (Brave
+≥ 1.46, Chrome 116, PolyForm 1.0.0, TRC ''maxLoadTimeMs || 60000'', OpenWPM issue #557) and
+prose numbers ("Chromium 121 driven by Playwright 1.41", "nearly twice", "100%"). The
+windowed run leaves four, all already recorded as permanent in §3 and §10: ''100'', ''106'',
+''19.1'' and ''53''.
+==== 12.4 Judgement calls and what was not done ====
+  * **Did not broaden or re-open [[programming:crawler:llm_agents]].** The work item says the page belongs to the sibling item and was explicitly out of scope for this one. It is published and reviewed; re-editing it here would have put an unreviewed change on a reviewed page.
+  * **Did not touch [[programming:crawler_detection]].** Its own open work item covers the ''draft-meunier-web-bot-auth-architecture'' state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made.
+  * **Did not re-run the sandbox measurements or the quote check.** §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against ''selenium-webdriver'' 4.46.
+  * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%&lt;file&gt;%%'' and §10 renders in full. Left alone rather than nowiki-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. ((**This bullet carried a latent render defect that only fired when a later section was added, which is the part worth recording.** The sentence above ended with the word "escaped" preceded by a **bare, unpaired pair of nowiki delimiters**, so this line's own delimiter count was odd: two for the file tag, two for its escaped-entity form, one stray. At revision ''1789641906'' the page still rendered — 24 headings, 9 tables, 4 preformatted blocks — because a later delimiter elsewhere on the page happened to close it. Appending the quote-check section of 2026-09-21 changed the pairing, and the rendered page collapsed to **19 headings, 8 tables**, with everything from §12.5 to the end shown as raw DokuWiki markup. ''check_wrap.mjs'' and ''check_typography.mjs'' passed on both revisions and on the repair. Removing the stray delimiter gives **25 headings, 11 tables, 5 preformatted blocks** — exactly 24 + 1, 9 + 2, 4 + 1, which is what the new section adds, so the structure reconciles. Found by counting literal bold and monospace markup in the **rendered** DOM outside any preformatted block (''scripts/_qc0921_dom2.mjs''); no source-level check on this wiki can see this class of defect. **The lesson is that an odd delimiter count is a defect even while the page renders**, because the next edit to the page is what detonates it.))
+==== 12.5 Review ====
+Two reviewers, both told explicitly that the brief they were given might not be exhaustive,
+both handed the page text, the report script and its output, and these notes. A one-bullet
+addition would not normally get a review layer; it got one because the bullet asserts a
+framing rather than a number, and that is the kind of claim this wiki gets wrong quietly.
+**The generic pass earned its slot several times over.**
+^ Pass ^ Finding ^ Verdict ^
+| Citations and quotes (''sonnet'') | Jeong quote verbatim; authors, title and venue match the bibliography; ''jeong2026_network'' resolves with no colliding duplicate; all three unquoted claims supported by the two neighbour pages; anchor resolves; §12's DOM and script figures all reproduce | **Accepted, no change needed.** It re-derived every check independently rather than reading this log. |
+| Citations and quotes (''sonnet'') | Soft note: the bullet collapsed "AI crawler" and "agent" into one "AI clients" bucket where the neighbours keep them as separate tiers | **Accepted** — folded into the rewrite below. |
+| Generic (''fable'') | The ''75'' in //Recommendations// is stale | **Accepted.** Already found independently and fixed as ''1788469206'' before the review landed, but the reviewer's diagnosis of //why// it survived — the guard printing the first, decoy occurrence — is new and is now §12.3. |
+| Generic (''fable'') | §12.4 claimed the whole-page guard was not run, which had stopped being true | **Accepted.** The claim was written before the guard was run and not revisited. Rewritten; this is precisely the "provenance page overstates its own rigour" failure it was told to hunt for. |
+| Generic (''fable'') | The bullet's "not a harder case of //the second//" is a dangling ordinal — this page never states the human / AI-crawler taxonomy, so "the second" reads as the anti-detection bullet above it | **Accepted**, rewritten. |
+| Generic (''fable'') | The bullet generalised "the open-source agents make it worse by default" from Jeong et al., who ran Browser-Use as a research agent on open queries; the two actual agent-crawl papers hand the agent URLs and report no search-engine challenge at entry | **Accepted**, and it is the most substantive finding of the run. Quote and sentence deleted (§12.2). |
+| Generic (''fable'') | The bullet duplicated the child page's prose and was the longest of the three for a 4-of-1,120 instrument; "one whose vendor defaults … dates" has no stated referent | **Accepted.** Rewritten to one framing sentence plus a pointer, ~90 words, no quote. |
+| Generic (''fable'') | //Agent-Driven Crawling// says "Since **2025** there is a **fifth** option", but all four counted papers are 2026 and "fifth" is unanchored | **Accepted**, changed to "Since 2026 there is an option". |
+| Generic (''fable'') | The agent material is structurally over-weighted for 4 papers (own H2, table row with a 60-word footnote, a detection bullet) where Scrapy at 21 papers gets one row | **Partly accepted.** Shortening the bullet removes some of it. The H2 and the footnote were written and reviewed on 2026-08-29 and are not re-opened here — but the observation is recorded so the next editor of this page can weigh it. |
+| Generic (''fable'') | §12.1 said §5.1 "referenced it only as a path in a working directory", while §5.1 itself was not edited; a residue row cited §5.1 rather than the content page; "headings 25" is not reproducible from the DOM without the regex; "says twice" is three places | **All accepted**, all corrected above. |
+| Generic (''fable'') | Housekeeping: §1's "Refreshed" date is stale, §11's "Deferred" row is resolved but unannotated, the section numbering skips §9 | **First two accepted** and fixed. The §9 gap is pre-existing and renumbering would break every reference to §10 and §11 from other pages; left alone and recorded here. |
+Neither reviewer was re-run after the fixes. That is a deliberate call on a change this size
+and it is the weakest point of this section: the rewritten bullet has been read by nobody but
+its author.
+===== 13. Run log, 2026-09-03 =====
+^ ^ ^
+| Date | 2026-09-03 |
+| Corpus at the time | ''data/extract/run1'', 5,859 papers, 2010–2026; unchanged since 2026-08-12 |
+| Work item | ''llm-agent-crawlers'' |
+| Model | Claude Opus 5; reviewers Claude Sonnet 5 (citations and quotes) and Claude Fable 5 (generic) |
+| Scope | Verification of the 2026-08-29 fold, one bullet, one stale figure, one date. No figure was re-derived by hand and no new query was written. |
+| Script changes | ''tool_fold.mjs'' — a re-verification comment only, no rule changed. ''scripts/report_crawler-output.txt'' — new, the script's real output. |
+| Content page revisions | ''1788468979'' (bullet), ''1788469206'' (75 → 74), ''1788469575'' (review rewrite) |
+| Mistake caught in review | This page claimed a check it had not run (§12.5, row 4). It was written before the check and not revisited when the check happened. |
 [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]]
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene sweep, 2026-09-17 =====
+The paired content page's obsolete live-page caveat was removed after the fresh DOM confirmed that the root ''Artifacts'' link is live. The post-save DOM was re-checked for red links. No figures or citations changed.
+===== ConsentAction audit propagation, 2026-09-17 =====
+**What was wrong.** The crawler configuration table and family comparison called the 349/1,120 = 31.2% raw field population ''States consent action'', which could be read as 349 papers making that claim. The schema statistic itself is correct.
+**Command and real output.** ''node scripts/report_crawler.mjs'' → ''Consent action field populated (schema; not an audited paper claim) | 349 | 31.2%''; the family table's corresponding column is ''Consent field populated (schema)''. ''node scripts/consent_action_noninteraction_audit.mjs'' → **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels.
+**What changed.** The page and report output now distinguish field population from paper-claim support; the family percentages remain schema statistics and are labelled accordingly.
+**Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_crawler.mjs'' and checked the edited row, the adjacent statefulness/interaction-depth rows, and the family table. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed.
+**Rejected finding.** The 349 schema statistic was retained rather than rewritten as 55, because this page's table measures extraction-field completeness. The audited claim result remains on [[privacy:consent]].
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. §6's figure predates the patch. Re-run, unedited first line:
+<code>
+$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
+quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 106 | 106 | population unchanged — the corpus has not moved |
+| exact | 58 | 58 | unchanged |
+| partial (≥60% of 5-word windows) | 32 | 32 | unchanged |
+| rescued from the PDF | — | **13** | new verdict; these were inside the old 16 |
+| below threshold | **16** | **3** (in both renderings) | 16 = 13 + 3 exactly; nothing else moved |
+| below-threshold rate | 15% | **2.8%** | 3 of 106 |
+**This page is the one where the refresh changes nothing about the conclusion.** All 16 had already been read by hand and all 16 were present, so the page never claimed a quote failure. What moves is the figure other pages compare themselves against: [[:provenance:design:ip_classification]] quoted "the ''%%--tools%%'' checks produce 15%" as its baseline, and that baseline is now 2.8%. That row was corrected on the same day.
+**Scope of this edit.** §6 and the content page's spot-check bullet. ''report_crawler.mjs'' and ''tool_fold.mjs'' were **not** re-run in this pass; no fold, framework count, trend or citation was touched, and every other figure on [[:programming:crawler]] and this page stands as published.
+The command is the code block above — it contains a regex alternation, and a DokuWiki table cell cannot hold a ''%%|%%''.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Artifact | ''out/qc0921/tools_openwpm_playwright.txt'' (full run, 13 RESCUED rows and 3 below-threshold rows listed) |
+| Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 |
+| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler]] |
+| Pages saved | this page, [[:programming:crawler]] |
+===== Foxhound count errata, 2026-09-21 =====
+**What was wrong.** The //Which specialised crawlers actually get used// table read ''SAP Project Foxhound | 8 | 1'', and the comparison table above it read **8**. The deeper per-paper audit on [[:programming:crawler:foxhound]] (''scripts/report_foxhound.mjs'' + ''scripts/fh_fold.mjs'', 2026-08-17) finds **9 papers that actually ran the browser**, and its own footnote said so and called these two tables queued for an errata edit.
+**Why the two disagreed — and it is not a denominator.** This table's two columns come from ''tools[].usedOrMentioned'': ''used'' or ''produced'' go left, everything else goes right. Khodayari et al.'s NDSS 2025 open-redirection paper is extracted as ''compared'' — //"We considered the following state-of-the-art detectors as baselines: dynamic taint-tracking [24, 36, 45] using Foxhound [44, 45]"// — and its comparison ran over the paper's 42,288-webpage subset of 50 randomly chosen applications((//"We chose 50 applications at random from the P2 dataset in Table I, encompassing a total of 42,288 webpages"//. The paper's own name for that subset is a subscripted P2 that the extracted text renders as ''P20'', so it is described here rather than named.)). Running a tool as a baseline is running it, so ''compared'' put a user of the browser in the //cites only// column.
+**The fix is in the script, not on the page.** ''report_crawler.mjs'' §6 now carries a ''ROLE_AUDITED'' map, currently one entry, ''SAP Project Foxhound'' → the ''ROLE'' map exported by ''fh_fold.mjs''. For a row in that map the enum verdict is replaced by the published hand verdict (''used'' / ''extended'' / ''baseline'' count as ran it; ''citation'' does not), and the script **throws** if any paper the row matches has no verdict in the audit, so the audit and the extraction cannot drift apart silently. Rows with no audit are untouched: this is deliberately not a general rule that ''compared'' means "ran it", because deciding that needs the sentence.
+<code>
+$ node scripts/report_crawler.mjs > scripts/report_crawler-output.txt
+$ diff <old> <new>
+c130
+< SAP Project Foxhound                         8             1                       2024–2026
+---
+> SAP Project Foxhound                         9             0                       2024–2026
+</code>
+**That is the whole diff of the report.** No other row, fold, residue, denominator or year range moved, and the folded-framework table above is untouched — this paper's Foxhound tuple carries ''category: "program-analysis"'', not a crawler-framework category, so it was never in that population.
+^ Figure ^ Was ^ Is ^ Where ^
+| Foxhound, papers using or producing it | 8 | **9** | //Which specialised crawlers actually get used// |
+| Foxhound, papers only citing it | 1 | **0** | same row |
+| Foxhound, //Papers// column | **8** | **9** | //Specialised Measurement Crawlers// comparison table |
+| growth sentence | "quadrupled Foxhound (2 → 8)" | "more than quadrupled Foxhound (2 → 9)" | same section |
+| Years used | 2024–2026 | 2024–2026 | unchanged |
+The **2** in that sentence is the pre-extension corpus figure and is not re-derived here; the paper this errata adds is NDSS 2025 and was not in that corpus, so the comparison is still like-for-like.
+**A finding rejected.** The //cites only// column now reads 0, while [[:programming:crawler:foxhound]] reports **2 citation-only papers**. That is not a new inconsistency and was not "fixed": those two papers have no ''tools[]'' tuple naming Foxhound at all, so this table — which is built from ''tools[]'' — cannot see them, and inventing a 2 here would mean this row alone was full-text-swept while every other row was not. The child page's reconciling footnote now says exactly this, and was edited in the same run.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Script changes | ''scripts/report_crawler.mjs'' (''ROLE_AUDITED'' block in §6, import of ''fh_fold.mjs''); committed output regenerated |
+| Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass |
+| Pages saved | this page, [[:programming:crawler]], [[:programming:crawler:foxhound]] |