Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler [2026/09/03 20:58] – Section 12: re-verified the 2026-08-29 LLM-agent fold against a fresh report_crawler.mjs run, logged the Being Detected bullet added to the content page, its quote check and rendered checks (llm-agent-crawlers item). Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler [2026/09/21 14:40] (current) – Citations pass: do not invent the paper's dataset notation; describe the 42,288-webpage subset and quote the sentence that defines it. Authored by Claude karel.kubicek.claude
@@ Line 16: / Line 16: @@
 | Runnable comparison | ''sandbox/fixture.mjs'', ''sandbox/run_{playwright,puppeteer,cdp,selenium}.mjs'', ''sandbox/compare.sh'' |
 | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
-| Refreshed | 2026-08-12 |
+| Refreshed | 2026-08-12; extended 2026-08-29 (§5.3) and 2026-09-03 (§12) |
 ===== 2. Populations and denominators =====
@@ Line 37: / Line 37: @@
 node scripts/report_crawler.mjs                       # every figure on the page
 node scripts/report_crawler.mjs --wiki                # the same, as DokuWiki tables
-node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
+node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
 node scripts/check_page_numbers.mjs \
   pages/programming_crawler.txt out/new/report_crawler.txt \
@@ Line 145: / Line 145: @@
 //Recorded, 2026-08-12.// ''scripts/quote_check.mjs'' is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure ("66 quotes, 44 exact") cannot be reproduced.
+**Re-run 2026-09-21** with the PDF fallback — see //Quote-check refresh, 2026-09-21// at the foot of this page:
 <code>
-$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
+$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
-quotes checked: 58 exact, 32 partial (>=60% of 5-word windows),
+quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.
- below threshold, 0 with no full text on disk.
 </code>
-All **16** below-threshold quotes were then read by hand against ''paper.cols.txt'' with whitespace normalised. **All sixteen are present in the paper. None was unsupported.** Two representative failures:
+**The old figure was 16 below threshold.** 13 of the 16 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are therefore a defect in the stored text, not in the extraction; **3** are below threshold in both renderings. ''exact'' (58) and ''partial'' (32) did not move, and the below-threshold-in-both rate is **2.8%** of 106.
+All **16** of the then-below-threshold quotes had already been read by hand against ''paper.cols.txt'' with whitespace normalised, before the fallback existed. **All sixteen are present in the paper. None was unsupported** — a hand result the fallback independently confirms for 13 of them. Two representative failures, one from each bucket:
+  * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker. This one is **still below threshold in both renderings** (50% of windows in the stored text), because a dropped marker is in the extraction's own string and no rendering of the PDF can repair it. It is one of the 3.
+  * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word. This one is **rescued**: 33% of windows in the stored text, 67% in the PDF, which is what a column-repair failure looks like when the second rendering happens not to splice the same sentence.
-  * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker.
+**Those two examples are the distinction the fallback draws.** A quote the extraction itself mangled fails in every rendering; a quote the //de-columner// mangled usually does not. That is why 13 of 16 moved and 3 did not.
-  * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word.
 The lesson for anyone reusing ''evidence.quote'': a failed exact match on ''.cols.txt'' is weak evidence of anything. Check the distinctive noun phrase, not the sentence.
@@ Line 187: / Line 192: @@
   * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason.
-Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''<file>'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean.
+Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''%%--code%%'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean.
 Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced:
@@ Line 207: / Line 212: @@
 | Claim weakened on purpose | The old page said the bespoke rows together were "more than used every off-the-shelf library on this page combined". That was not true of the old numbers either (187+54+35+10 = 286 > 221) and is not true of the new ones. Replaced with the two comparisons the report actually prints: more than Selenium (242), and more than the 215 papers naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM. |
 | Mistake caught in review | The page briefly carried "231" for that combined count — arrived at by summing family counts, which double-counts papers naming two of them. The real union is 215. ''check_page_numbers.mjs'' flagged it as unaccounted, which is how it was caught; it was then computed in the report script rather than by hand. |
-| Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. |
+| Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. **Resolved 2026-08-29** — see §5.3 and §12. |
 | Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 7 stale figures plus a mis-ordered table and a backwards ranking sentence on this page. All fixes were applied and re-saved the same day. |
-===== 12. Follow-up, 2026-09-03: the LLM-agent fold re-verified, and one bullet added =====
+===== 12. Follow-up, 2026-09-03: the LLM-agent fold re-verified, and Being Detected closed =====
-//Recorded.// Work item ''llm-agent-crawlers'' asked three things of this page: whether the
+//Recorded as the run went, tidied at the end.// Work item ''llm-agent-crawlers'' asked three
-–2026 agent strings should become a family in ''tool_fold.mjs'', whether
+things of this page: whether the 2025–2026 agent strings should become a family in
-[[programming:crawler]] should carry it, and what it does to the //Being Detected//
+''tool_fold.mjs'', whether [[programming:crawler]] should carry it, and what it does to the
-section. The first two were answered on 2026-08-29 (§5.3, §4.3 of
+//Being Detected// section. The first two were answered on 2026-08-29 (§5.3 above, and §4.3
-[[provenance:programming:crawler:llm_agents]]). This run **verified** them rather than
+of [[provenance:programming:crawler:llm_agents]]). This run **verified** them instead of
-taking the note at face value, and closed the third.
+taking that note at face value, and closed the third.
 ==== 12.1 Verification of the fold, against the current run1 ====
-''node scripts/report_crawler.mjs'' re-run on 2026-09-03 against
+''node scripts/report_crawler.mjs'' re-run on 2026-09-03 against ''data/extract/run1''
-''data/extract/run1'' (5,859 papers; population ''crawled'' = 1,120). Exit 0. Every figure
+(5,859 papers; population ''crawled'' = 1,120). Exit 0. Every figure in the content page's
-in the content page's //Use in Publications// section matches the fresh output, including
+//Use in Publications// section matches the fresh output, including the rows the 2026-08-29
-the rows the 2026-08-29 fold moved:
+fold moved:
-^ Checked ^ Fresh script output ^ On the page ^
+^ Checked ^ Fresh script output ^ On the content page ^
-| ''LLM browser agents'' family | 4 papers, 0.4% of 1,120 | 4 / 0.4% |
+| ''LLM browser agents'' family | 4 papers, 0.4% of 1,120 | 4 / 0.4%, framework table |
-| Bespoke, own name (residue row) | 181, 16.2% | 181 / 16.2% |
+| Bespoke, own name (residue row) | 181, 16.2% | 181 / 16.2%, framework table |
-| Bespoke union / of which named nothing else | 318 of 1,120 (28.4%) / 74 | 318 (28.4%) / 74 |
+| Bespoke union / of which named nothing else | 318 of 1,120 (28.4%) / 74 | 318 (28.4%) / 74, the ''WRAP important'' box |
 | Selenium | 242, 21.6% | 242 / 21.6% |
 | Puppeteer / Playwright | 76 (6.8%) / 34 (3.0%) | same |
-| Framework residue | 199 distinct strings, 205 paper-mentions | "199 after the LLM-agent fold" (§5.1) |
+| Framework residue | 199 distinct strings, 205 paper-mentions | 199 / 205, //Methodology and limitations// |
-| Browser residue | 1 distinct, ''Ghostery'' | same (§5.2) |
+| Browser residue | 1 distinct, ''Ghostery'' | same, //Methodology and limitations// |
 Grepping the fresh residue listing for every agent-shaped string
 (''browser.?use'', ''agentlab'', ''browsergym'', ''skyvern'', ''stagehand'', ''crawl4ai'',
 ''firecrawl'', ''computer.use'', ''operator'', ''mcp'', ''webagent'') returns exactly one
-line: ''MCP Server Crawler''. That is the deliberate exclusion recorded in §5.3, so the
+line: ''MCP Server Crawler''. That is the deliberate exclusion recorded in §5.3, so the fold
-fold is behaving as documented and nothing new has arrived in the residue since.
+is behaving as documented and nothing new has arrived in the residue since 2026-08-29.
-The full output is now committed as ''scripts/report_crawler-output.txt'' — §5.1 above
+The full output is now written to ''scripts/report_crawler-output.txt'' in the wiki workdir,
-referenced it only as a path in a working directory, which is a residue nobody could look
+next to the script. §5.1 above still points at ''out/new/report_crawler.txt'', a path in a
-at without repo access.
+working directory; that reference was **not** edited, so both now exist and the committed
+one is the readable one.
-==== 12.2 The one edit: a third bullet in //Being Detected// ====
+==== 12.2 The one edit: a bullet in //Being Detected// ====
-The page gained an //Agent-Driven Crawling// section on 2026-08-29, but //Being Detected//
+The content page gained an //Agent-Driven Crawling// section on 2026-08-29, but
-still opened with "Two points from there bear on the choice made on this page" and both
+//Being Detected// still opened with "Two points from there bear on the choice made on this
-points were about scripted automation. Neither neighbour is at fault —
+page" and both points were about scripted automation. Neither neighbour is at fault —
-[[programming:crawler_detection]] says twice that an agent is a different surface, and
+[[programming:crawler_detection]] says in three places that an agent is a different surface
-[[programming:crawler:llm_agents]] has a full section on it — but a reader who takes the
+(its intro, the close of //What a Blocked Crawl Does to Your Number//, and //Related Pages//),
-agent option on //this// page was given no signal on the way past. Added one bullet,
+and [[programming:crawler:llm_agents]] has a full section on it — but a reader who takes the
-changed "Two points from there" to "Three points", no other change. Content page
+agent option on //this// page was given no signal on the way past. Added one bullet and
-1788424922 → 1788468979.
+changed "Two points from there" to "Three points". Revisions ''1788424922'' →
+''1788468979'' → (after review) ''1788469575''.
-The bullet's one load-bearing quote was checked against the source, not carried over from
+**The first version of that bullet was wrong in four ways and was rewritten**, see §12.5.
-the child page:
+It is worth recording what it contained, because the rewrite deleted a citation:
-^ Citekey ^ Quote ^ Checked against ^ Result ^
+^ Dropped ^ Why ^
-| ''jeong2026_network'' | "By default, it issues Google queries via ''search_web'', which frequently triggers reCAPTCHA; we therefore configure it to use Bing with automatic fallback" | ''data/fulltext/2026/USENIX/network-level-prompt-and-trait-leakage-in-local-research-agents/paper.cols.txt'' | **Verbatim.** Note the ''.cols'' rendering interleaves the caption of Table 14 into the middle of the sentence, so it is only found after collapsing whitespace and reading around the caption — a plain ''grep'' for the whole phrase fails. |
+| A verbatim quote from Jeong et al. (''%%{[jeong2026_network]}%%'') on Browser-Use issuing Google queries via ''search_web'' and triggering reCAPTCHA | The quote is real and was checked against ''data/fulltext/2026/USENIX/network-level-prompt-and-trait-leakage-in-local-research-agents/paper.cols.txt'' (verbatim; note the ''.cols'' rendering interleaves the caption of Table 14 mid-sentence, so a plain ''grep'' for the phrase fails and whitespace must be collapsed first). What was wrong was the **use**: Jeong et al. run Browser-Use as a research agent on open queries, whereas a measurement crawl hands the agent a URL, and the two actual agent-crawl papers report no search-engine challenge at entry. Generalising it to "the open-source agents make it worse by default" was not supported. It also duplicated the child page's own first detection bullet almost word for word. |
-''jeong2026_network'' is new to this content page and was already in
+''jeong2026_network'' is therefore **no longer cited on this content page**; it remains cited
-[[literature:bibliography]]; **no bibliography entry was added or changed.** Rendered
+on [[programming:crawler:llm_agents]], where the claim is correctly hedged. No bibliography
-check after saving, with ''purge=true'': references 14 → 15 with Jeong present, citekey
+entry was added, changed or removed at any point in this run.
-spans 36 → 38 (the plugin emits two per marker), and headings 25, tables 12 and ''pre''
-blocks 5 all unchanged before and after. Zero ''wikilink2'' red links. The anchor
-''#being_detected_and_the_ethics_of_not_being'' was taken from the child page's **rendered**
-heading id, not guessed, and the saved page's ''href'' matches it exactly.
-==== 12.3 Judgement calls and what was not done ====
+Rendered checks after each save, with ''purge=true'': after the first save, references
+→ 15 with Jeong present and citekey spans 36 → 38 (the plugin emits two per marker);
+after the rewrite, back to **14** with Jeong absent. Elements matching
+''%%<h[123] … id=%%'' stayed at **25**, ''%%<table%%'' at **12** and ''%%<pre%%'' at **5**
+across all three revisions, and ''wikilink2'' red links stayed at **0**. (Those are counts
+of that regex, not of every heading in the DOM — a reviewer counting all ''h1''/''h2''/''h3''
+gets 29. The point of the count is that it did not change, not its absolute value.) The
+anchor ''#being_detected_and_the_ethics_of_not_being'' was taken from the child page's
+**rendered** heading id, not guessed, and the saved page's ''href'' matches it exactly.
+==== 12.3 A stale figure the 2026-08-29 fold left behind ====
+//Found on 2026-09-03, after the edit above, by running the number guard whole-page.//
+The save summary for revision ''1787985453'' says the fold moved **75 → 74** unnamed bespoke
+crawlers, and it did — in the ''WRAP important'' box under the framework table. The **same
+figure in the //Recommendations// section was not moved** and shipped as ''75'' for five days
+across three subsequent saves. Fixed in revision ''1788469206''.
+This is the exact failure §10 above describes, one refresh later, and it had a specific
+disguise: the whole-page guard //does// list ''75'' as unaccounted, but the context it prints
+is the **first** occurrence of the string, which on this page is the sandbox sentence "a
+silent 75% data loss". Anyone reading the guard output and recognising that sentence as a
+legitimate non-corpus figure clears the line and never reaches the second occurrence. The
+same trap is still there today: after the fix, ''75'' is still reported, still with the
+sandbox context, and now legitimately.
+Command:
+<code>
+node scripts/check_page_numbers.mjs \
+  pages_crawler_live.txt scripts/report_crawler-output.txt
+</code>
+with **no** heading markers. The whole-page run also reports 23 other unaccounted figures
+and 3 version identifiers. All were read; **none is a corpus figure.** They are the sandbox
+measurements (779 / 877 / 2038 / 1329 / 1838 ms, HTTP 200 and 204, ''selenium-webdriver''
+.46, Node 22.23, Firefox 153, Chromium 151.0.7922.34), figures quoted from other papers
+(PanoptiChrome 50% and 36.7× against Foxhound's 95% and 1.4×), version constraints (Brave
+≥ 1.46, Chrome 116, PolyForm 1.0.0, TRC ''maxLoadTimeMs || 60000'', OpenWPM issue #557) and
+prose numbers ("Chromium 121 driven by Playwright 1.41", "nearly twice", "100%"). The
+windowed run leaves four, all already recorded as permanent in §3 and §10: ''100'', ''106'',
+''19.1'' and ''53''.
+==== 12.4 Judgement calls and what was not done ====
   * **Did not broaden or re-open [[programming:crawler:llm_agents]].** The work item says the page belongs to the sibling item and was explicitly out of scope for this one. It is published and reviewed; re-editing it here would have put an unreviewed change on a reviewed page.
   * **Did not touch [[programming:crawler_detection]].** Its own open work item covers the ''draft-meunier-web-bot-auth-architecture'' state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made.
-  * **Did not re-run the sandbox measurements**, the quote check, or ''check_page_numbers.mjs'' whole-page. The edit adds prose and one citation and changes no figure, so no figure on the page moved; the verification above confirms the existing ones still match the script. §8's open questions are unchanged.
+  * **Did not re-run the sandbox measurements or the quote check.** §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against ''selenium-webdriver'' 4.46.
-  * **No new reviewer passes were run for a one-bullet addition.** That is a deliberate proportionality call and it is the thing on this page most open to disagreement: the bullet asserts a framing ("a third detection surface") that the two neighbouring pages already assert, with one verified quote, and no number.
+  * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%&lt;file&gt;%%'' and §10 renders in full. Left alone rather than nowiki-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. ((**This bullet carried a latent render defect that only fired when a later section was added, which is the part worth recording.** The sentence above ended with the word "escaped" preceded by a **bare, unpaired pair of nowiki delimiters**, so this line's own delimiter count was odd: two for the file tag, two for its escaped-entity form, one stray. At revision ''1789641906'' the page still rendered — 24 headings, 9 tables, 4 preformatted blocks — because a later delimiter elsewhere on the page happened to close it. Appending the quote-check section of 2026-09-21 changed the pairing, and the rendered page collapsed to **19 headings, 8 tables**, with everything from §12.5 to the end shown as raw DokuWiki markup. ''check_wrap.mjs'' and ''check_typography.mjs'' passed on both revisions and on the repair. Removing the stray delimiter gives **25 headings, 11 tables, 5 preformatted blocks** — exactly 24 + 1, 9 + 2, 4 + 1, which is what the new section adds, so the structure reconciles. Found by counting literal bold and monospace markup in the **rendered** DOM outside any preformatted block (''scripts/_qc0921_dom2.mjs''); no source-level check on this wiki can see this class of defect. **The lesson is that an odd delimiter count is a defect even while the page renders**, because the next edit to the page is what detonates it.))
-  * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''&lt;file&gt;'' and §§10–11 render in full. Left alone rather than %%-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it.
+==== 12.5 Review ====
+Two reviewers, both told explicitly that the brief they were given might not be exhaustive,
+both handed the page text, the report script and its output, and these notes. A one-bullet
+addition would not normally get a review layer; it got one because the bullet asserts a
+framing rather than a number, and that is the kind of claim this wiki gets wrong quietly.
+**The generic pass earned its slot several times over.**
+^ Pass ^ Finding ^ Verdict ^
+| Citations and quotes (''sonnet'') | Jeong quote verbatim; authors, title and venue match the bibliography; ''jeong2026_network'' resolves with no colliding duplicate; all three unquoted claims supported by the two neighbour pages; anchor resolves; §12's DOM and script figures all reproduce | **Accepted, no change needed.** It re-derived every check independently rather than reading this log. |
+| Citations and quotes (''sonnet'') | Soft note: the bullet collapsed "AI crawler" and "agent" into one "AI clients" bucket where the neighbours keep them as separate tiers | **Accepted** — folded into the rewrite below. |
+| Generic (''fable'') | The ''75'' in //Recommendations// is stale | **Accepted.** Already found independently and fixed as ''1788469206'' before the review landed, but the reviewer's diagnosis of //why// it survived — the guard printing the first, decoy occurrence — is new and is now §12.3. |
+| Generic (''fable'') | §12.4 claimed the whole-page guard was not run, which had stopped being true | **Accepted.** The claim was written before the guard was run and not revisited. Rewritten; this is precisely the "provenance page overstates its own rigour" failure it was told to hunt for. |
+| Generic (''fable'') | The bullet's "not a harder case of //the second//" is a dangling ordinal — this page never states the human / AI-crawler taxonomy, so "the second" reads as the anti-detection bullet above it | **Accepted**, rewritten. |
+| Generic (''fable'') | The bullet generalised "the open-source agents make it worse by default" from Jeong et al., who ran Browser-Use as a research agent on open queries; the two actual agent-crawl papers hand the agent URLs and report no search-engine challenge at entry | **Accepted**, and it is the most substantive finding of the run. Quote and sentence deleted (§12.2). |
+| Generic (''fable'') | The bullet duplicated the child page's prose and was the longest of the three for a 4-of-1,120 instrument; "one whose vendor defaults … dates" has no stated referent | **Accepted.** Rewritten to one framing sentence plus a pointer, ~90 words, no quote. |
+| Generic (''fable'') | //Agent-Driven Crawling// says "Since **2025** there is a **fifth** option", but all four counted papers are 2026 and "fifth" is unanchored | **Accepted**, changed to "Since 2026 there is an option". |
+| Generic (''fable'') | The agent material is structurally over-weighted for 4 papers (own H2, table row with a 60-word footnote, a detection bullet) where Scrapy at 21 papers gets one row | **Partly accepted.** Shortening the bullet removes some of it. The H2 and the footnote were written and reviewed on 2026-08-29 and are not re-opened here — but the observation is recorded so the next editor of this page can weigh it. |
+| Generic (''fable'') | §12.1 said §5.1 "referenced it only as a path in a working directory", while §5.1 itself was not edited; a residue row cited §5.1 rather than the content page; "headings 25" is not reproducible from the DOM without the regex; "says twice" is three places | **All accepted**, all corrected above. |
+| Generic (''fable'') | Housekeeping: §1's "Refreshed" date is stale, §11's "Deferred" row is resolved but unannotated, the section numbering skips §9 | **First two accepted** and fixed. The §9 gap is pre-existing and renumbering would break every reference to §10 and §11 from other pages; left alone and recorded here. |
+Neither reviewer was re-run after the fixes. That is a deliberate call on a change this size
+and it is the weakest point of this section: the rewritten bullet has been read by nobody but
+its author.
+===== 13. Run log, 2026-09-03 =====
+^ ^ ^
+| Date | 2026-09-03 |
+| Corpus at the time | ''data/extract/run1'', 5,859 papers, 2010–2026; unchanged since 2026-08-12 |
+| Work item | ''llm-agent-crawlers'' |
+| Model | Claude Opus 5; reviewers Claude Sonnet 5 (citations and quotes) and Claude Fable 5 (generic) |
+| Scope | Verification of the 2026-08-29 fold, one bullet, one stale figure, one date. No figure was re-derived by hand and no new query was written. |
+| Script changes | ''tool_fold.mjs'' — a re-verification comment only, no rule changed. ''scripts/report_crawler-output.txt'' — new, the script's real output. |
+| Content page revisions | ''1788468979'' (bullet), ''1788469206'' (75 → 74), ''1788469575'' (review rewrite) |
+| Mistake caught in review | This page claimed a check it had not run (§12.5, row 4). It was written before the check and not revisited when the check happened. |
 [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]]
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene sweep, 2026-09-17 =====
+The paired content page's obsolete live-page caveat was removed after the fresh DOM confirmed that the root ''Artifacts'' link is live. The post-save DOM was re-checked for red links. No figures or citations changed.
+===== ConsentAction audit propagation, 2026-09-17 =====
+**What was wrong.** The crawler configuration table and family comparison called the 349/1,120 = 31.2% raw field population ''States consent action'', which could be read as 349 papers making that claim. The schema statistic itself is correct.
+**Command and real output.** ''node scripts/report_crawler.mjs'' → ''Consent action field populated (schema; not an audited paper claim) | 349 | 31.2%''; the family table's corresponding column is ''Consent field populated (schema)''. ''node scripts/consent_action_noninteraction_audit.mjs'' → **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels.
+**What changed.** The page and report output now distinguish field population from paper-claim support; the family percentages remain schema statistics and are labelled accordingly.
+**Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_crawler.mjs'' and checked the edited row, the adjacent statefulness/interaction-depth rows, and the family table. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed.
+**Rejected finding.** The 349 schema statistic was retained rather than rewritten as 55, because this page's table measures extraction-field completeness. The audited claim result remains on [[privacy:consent]].
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. §6's figure predates the patch. Re-run, unedited first line:
+<code>
+$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
+quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 106 | 106 | population unchanged — the corpus has not moved |
+| exact | 58 | 58 | unchanged |
+| partial (≥60% of 5-word windows) | 32 | 32 | unchanged |
+| rescued from the PDF | — | **13** | new verdict; these were inside the old 16 |
+| below threshold | **16** | **3** (in both renderings) | 16 = 13 + 3 exactly; nothing else moved |
+| below-threshold rate | 15% | **2.8%** | 3 of 106 |
+**This page is the one where the refresh changes nothing about the conclusion.** All 16 had already been read by hand and all 16 were present, so the page never claimed a quote failure. What moves is the figure other pages compare themselves against: [[:provenance:design:ip_classification]] quoted "the ''%%--tools%%'' checks produce 15%" as its baseline, and that baseline is now 2.8%. That row was corrected on the same day.
+**Scope of this edit.** §6 and the content page's spot-check bullet. ''report_crawler.mjs'' and ''tool_fold.mjs'' were **not** re-run in this pass; no fold, framework count, trend or citation was touched, and every other figure on [[:programming:crawler]] and this page stands as published.
+The command is the code block above — it contains a regex alternation, and a DokuWiki table cell cannot hold a ''%%|%%''.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Artifact | ''out/qc0921/tools_openwpm_playwright.txt'' (full run, 13 RESCUED rows and 3 below-threshold rows listed) |
+| Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 |
+| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler]] |
+| Pages saved | this page, [[:programming:crawler]] |
+===== Foxhound count errata, 2026-09-21 =====
+**What was wrong.** The //Which specialised crawlers actually get used// table read ''SAP Project Foxhound | 8 | 1'', and the comparison table above it read **8**. The deeper per-paper audit on [[:programming:crawler:foxhound]] (''scripts/report_foxhound.mjs'' + ''scripts/fh_fold.mjs'', 2026-08-17) finds **9 papers that actually ran the browser**, and its own footnote said so and called these two tables queued for an errata edit.
+**Why the two disagreed — and it is not a denominator.** This table's two columns come from ''tools[].usedOrMentioned'': ''used'' or ''produced'' go left, everything else goes right. Khodayari et al.'s NDSS 2025 open-redirection paper is extracted as ''compared'' — //"We considered the following state-of-the-art detectors as baselines: dynamic taint-tracking [24, 36, 45] using Foxhound [44, 45]"// — and its comparison ran over the paper's 42,288-webpage subset of 50 randomly chosen applications((//"We chose 50 applications at random from the P2 dataset in Table I, encompassing a total of 42,288 webpages"//. The paper's own name for that subset is a subscripted P2 that the extracted text renders as ''P20'', so it is described here rather than named.)). Running a tool as a baseline is running it, so ''compared'' put a user of the browser in the //cites only// column.
+**The fix is in the script, not on the page.** ''report_crawler.mjs'' §6 now carries a ''ROLE_AUDITED'' map, currently one entry, ''SAP Project Foxhound'' → the ''ROLE'' map exported by ''fh_fold.mjs''. For a row in that map the enum verdict is replaced by the published hand verdict (''used'' / ''extended'' / ''baseline'' count as ran it; ''citation'' does not), and the script **throws** if any paper the row matches has no verdict in the audit, so the audit and the extraction cannot drift apart silently. Rows with no audit are untouched: this is deliberately not a general rule that ''compared'' means "ran it", because deciding that needs the sentence.
+<code>
+$ node scripts/report_crawler.mjs > scripts/report_crawler-output.txt
+$ diff <old> <new>
+c130
+< SAP Project Foxhound                         8             1                       2024–2026
+---
+> SAP Project Foxhound                         9             0                       2024–2026
+</code>
+**That is the whole diff of the report.** No other row, fold, residue, denominator or year range moved, and the folded-framework table above is untouched — this paper's Foxhound tuple carries ''category: "program-analysis"'', not a crawler-framework category, so it was never in that population.
+^ Figure ^ Was ^ Is ^ Where ^
+| Foxhound, papers using or producing it | 8 | **9** | //Which specialised crawlers actually get used// |
+| Foxhound, papers only citing it | 1 | **0** | same row |
+| Foxhound, //Papers// column | **8** | **9** | //Specialised Measurement Crawlers// comparison table |
+| growth sentence | "quadrupled Foxhound (2 → 8)" | "more than quadrupled Foxhound (2 → 9)" | same section |
+| Years used | 2024–2026 | 2024–2026 | unchanged |
+The **2** in that sentence is the pre-extension corpus figure and is not re-derived here; the paper this errata adds is NDSS 2025 and was not in that corpus, so the comparison is still like-for-like.
+**A finding rejected.** The //cites only// column now reads 0, while [[:programming:crawler:foxhound]] reports **2 citation-only papers**. That is not a new inconsistency and was not "fixed": those two papers have no ''tools[]'' tuple naming Foxhound at all, so this table — which is built from ''tools[]'' — cannot see them, and inventing a 2 here would mean this row alone was full-text-swept while every other row was not. The child page's reconciling footnote now says exactly this, and was edited in the same run.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Script changes | ''scripts/report_crawler.mjs'' (''ROLE_AUDITED'' block in §6, import of ''fh_fold.mjs''); committed output regenerated |
+| Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass |
+| Pages saved | this page, [[:programming:crawler]], [[:programming:crawler:foxhound]] |