Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler [2026/09/21 13:35] – Quote-check refresh 2026-09-21: re-ran quote_check.mjs --tools openwpm/playwright with the pypdf fallback; 16 below threshold -> 13 rescued + 3 below in both; new provenance section. Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler [2026/09/21 14:40] (current) – Citations pass: do not invent the paper's dataset notation; describe the 42,288-webpage subset and quote the sentence that defines it. Authored by Claude karel.kubicek.claude
@@ Line 321: / Line 321: @@
   * **Did not touch [[programming:crawler_detection]].** Its own open work item covers the ''draft-meunier-web-bot-auth-architecture'' state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made.
   * **Did not re-run the sandbox measurements or the quote check.** §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against ''selenium-webdriver'' 4.46.
-  * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%&lt;file&gt;%%'' and §§10–12 render in full. Left alone rather than %%-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it.
+  * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%&lt;file&gt;%%'' and §10 renders in full. Left alone rather than nowiki-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. ((**This bullet carried a latent render defect that only fired when a later section was added, which is the part worth recording.** The sentence above ended with the word "escaped" preceded by a **bare, unpaired pair of nowiki delimiters**, so this line's own delimiter count was odd: two for the file tag, two for its escaped-entity form, one stray. At revision ''1789641906'' the page still rendered — 24 headings, 9 tables, 4 preformatted blocks — because a later delimiter elsewhere on the page happened to close it. Appending the quote-check section of 2026-09-21 changed the pairing, and the rendered page collapsed to **19 headings, 8 tables**, with everything from §12.5 to the end shown as raw DokuWiki markup. ''check_wrap.mjs'' and ''check_typography.mjs'' passed on both revisions and on the repair. Removing the stray delimiter gives **25 headings, 11 tables, 5 preformatted blocks** — exactly 24 + 1, 9 + 2, 4 + 1, which is what the new section adds, so the structure reconciles. Found by counting literal bold and monospace markup in the **rendered** DOM outside any preformatted block (''scripts/_qc0921_dom2.mjs''); no source-level check on this wiki can see this class of defect. **The lesson is that an odd delimiter count is a defect even while the page renders**, because the next edit to the page is what detonates it.))
 ==== 12.5 Review ====
@@ Line 412: / Line 412: @@
 | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler]] |
 | Pages saved | this page, [[:programming:crawler]] |
+===== Foxhound count errata, 2026-09-21 =====
+**What was wrong.** The //Which specialised crawlers actually get used// table read ''SAP Project Foxhound | 8 | 1'', and the comparison table above it read **8**. The deeper per-paper audit on [[:programming:crawler:foxhound]] (''scripts/report_foxhound.mjs'' + ''scripts/fh_fold.mjs'', 2026-08-17) finds **9 papers that actually ran the browser**, and its own footnote said so and called these two tables queued for an errata edit.
+**Why the two disagreed — and it is not a denominator.** This table's two columns come from ''tools[].usedOrMentioned'': ''used'' or ''produced'' go left, everything else goes right. Khodayari et al.'s NDSS 2025 open-redirection paper is extracted as ''compared'' — //"We considered the following state-of-the-art detectors as baselines: dynamic taint-tracking [24, 36, 45] using Foxhound [44, 45]"// — and its comparison ran over the paper's 42,288-webpage subset of 50 randomly chosen applications((//"We chose 50 applications at random from the P2 dataset in Table I, encompassing a total of 42,288 webpages"//. The paper's own name for that subset is a subscripted P2 that the extracted text renders as ''P20'', so it is described here rather than named.)). Running a tool as a baseline is running it, so ''compared'' put a user of the browser in the //cites only// column.
+**The fix is in the script, not on the page.** ''report_crawler.mjs'' §6 now carries a ''ROLE_AUDITED'' map, currently one entry, ''SAP Project Foxhound'' → the ''ROLE'' map exported by ''fh_fold.mjs''. For a row in that map the enum verdict is replaced by the published hand verdict (''used'' / ''extended'' / ''baseline'' count as ran it; ''citation'' does not), and the script **throws** if any paper the row matches has no verdict in the audit, so the audit and the extraction cannot drift apart silently. Rows with no audit are untouched: this is deliberately not a general rule that ''compared'' means "ran it", because deciding that needs the sentence.
+<code>
+$ node scripts/report_crawler.mjs > scripts/report_crawler-output.txt
+$ diff <old> <new>
+c130
+< SAP Project Foxhound                         8             1                       2024–2026
+---
+> SAP Project Foxhound                         9             0                       2024–2026
+</code>
+**That is the whole diff of the report.** No other row, fold, residue, denominator or year range moved, and the folded-framework table above is untouched — this paper's Foxhound tuple carries ''category: "program-analysis"'', not a crawler-framework category, so it was never in that population.
+^ Figure ^ Was ^ Is ^ Where ^
+| Foxhound, papers using or producing it | 8 | **9** | //Which specialised crawlers actually get used// |
+| Foxhound, papers only citing it | 1 | **0** | same row |
+| Foxhound, //Papers// column | **8** | **9** | //Specialised Measurement Crawlers// comparison table |
+| growth sentence | "quadrupled Foxhound (2 → 8)" | "more than quadrupled Foxhound (2 → 9)" | same section |
+| Years used | 2024–2026 | 2024–2026 | unchanged |
+The **2** in that sentence is the pre-extension corpus figure and is not re-derived here; the paper this errata adds is NDSS 2025 and was not in that corpus, so the comparison is still like-for-like.
+**A finding rejected.** The //cites only// column now reads 0, while [[:programming:crawler:foxhound]] reports **2 citation-only papers**. That is not a new inconsistency and was not "fixed": those two papers have no ''tools[]'' tuple naming Foxhound at all, so this table — which is built from ''tools[]'' — cannot see them, and inventing a 2 here would mean this row alone was full-text-swept while every other row was not. The child page's reconciling footnote now says exactly this, and was edited in the same run.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Script changes | ''scripts/report_crawler.mjs'' (''ROLE_AUDITED'' block in §6, import of ''fh_fold.mjs''); committed output regenerated |
+| Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass |
+| Pages saved | this page, [[:programming:crawler]], [[:programming:crawler:foxhound]] |