| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:programming:crawler [2026/09/03 21:08] – Section 12 rewritten after the generic review pass: adds the stale 75->74 in Recommendations and why the guard's decoy context hid it, corrects a claim that a check had been run, records both reviewers' findings and the four fixes to the Being Detected bu karel.kubicek.claude | provenance:programming:crawler [2026/09/21 14:40] (current) – Citations pass: do not invent the paper's dataset notation; describe the 42,288-webpage subset and quote the sentence that defines it. Authored by Claude karel.kubicek.claude |
|---|
| node scripts/report_crawler.mjs # every figure on the page | node scripts/report_crawler.mjs # every figure on the page |
| node scripts/report_crawler.mjs --wiki # the same, as DokuWiki tables | node scripts/report_crawler.mjs --wiki # the same, as DokuWiki tables |
| node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20 | node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400 |
| node scripts/check_page_numbers.mjs \ | node scripts/check_page_numbers.mjs \ |
| pages/programming_crawler.txt out/new/report_crawler.txt \ | pages/programming_crawler.txt out/new/report_crawler.txt \ |
| |
| //Recorded, 2026-08-12.// ''scripts/quote_check.mjs'' is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure ("66 quotes, 44 exact") cannot be reproduced. | //Recorded, 2026-08-12.// ''scripts/quote_check.mjs'' is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure ("66 quotes, 44 exact") cannot be reproduced. |
| | |
| | **Re-run 2026-09-21** with the PDF fallback — see //Quote-check refresh, 2026-09-21// at the foot of this page: |
| |
| <code> | <code> |
| $ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20 | $ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400 |
| 106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), | 106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk. |
| 16 below threshold, 0 with no full text on disk. | |
| </code> | </code> |
| |
| All **16** below-threshold quotes were then read by hand against ''paper.cols.txt'' with whitespace normalised. **All sixteen are present in the paper. None was unsupported.** Two representative failures: | **The old figure was 16 below threshold.** 13 of the 16 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are therefore a defect in the stored text, not in the extraction; **3** are below threshold in both renderings. ''exact'' (58) and ''partial'' (32) did not move, and the below-threshold-in-both rate is **2.8%** of 106. |
| | |
| | All **16** of the then-below-threshold quotes had already been read by hand against ''paper.cols.txt'' with whitespace normalised, before the fallback existed. **All sixteen are present in the paper. None was unsupported** — a hand result the fallback independently confirms for 13 of them. Two representative failures, one from each bucket: |
| | |
| | * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker. This one is **still below threshold in both renderings** (50% of windows in the stored text), because a dropped marker is in the extraction's own string and no rendering of the PDF can repair it. It is one of the 3. |
| | * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word. This one is **rescued**: 33% of windows in the stored text, 67% in the PDF, which is what a column-repair failure looks like when the second rendering happens not to splice the same sentence. |
| |
| * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker. | **Those two examples are the distinction the fallback draws.** A quote the extraction itself mangled fails in every rendering; a quote the //de-columner// mangled usually does not. That is why 13 of 16 moved and 3 did not. |
| * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word. | |
| |
| The lesson for anyone reusing ''evidence.quote'': a failed exact match on ''.cols.txt'' is weak evidence of anything. Check the distinctive noun phrase, not the sentence. | The lesson for anyone reusing ''evidence.quote'': a failed exact match on ''.cols.txt'' is weak evidence of anything. Check the distinctive noun phrase, not the sentence. |
| * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. | * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. |
| |
| Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''<file>'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. | Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''%%--code%%'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. |
| Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced: | Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced: |
| |
| * **Did not touch [[programming:crawler_detection]].** Its own open work item covers the ''draft-meunier-web-bot-auth-architecture'' state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made. | * **Did not touch [[programming:crawler_detection]].** Its own open work item covers the ''draft-meunier-web-bot-auth-architecture'' state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made. |
| * **Did not re-run the sandbox measurements or the quote check.** §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against ''selenium-webdriver'' 4.46. | * **Did not re-run the sandbox measurements or the quote check.** §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against ''selenium-webdriver'' 4.46. |
| * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%<file>%%'' and §§10–12 render in full. Left alone rather than %%-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. | * **''check_wrap.mjs'' false-positives on this page, and did so before this edit.** It flags the ''%%<file>%%'' inside inline code in §10 as a tag that "WILL fire". It does not: the rendered page escapes it to ''%%<file>%%'' and §10 renders in full. Left alone rather than nowiki-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. ((**This bullet carried a latent render defect that only fired when a later section was added, which is the part worth recording.** The sentence above ended with the word "escaped" preceded by a **bare, unpaired pair of nowiki delimiters**, so this line's own delimiter count was odd: two for the file tag, two for its escaped-entity form, one stray. At revision ''1789641906'' the page still rendered — 24 headings, 9 tables, 4 preformatted blocks — because a later delimiter elsewhere on the page happened to close it. Appending the quote-check section of 2026-09-21 changed the pairing, and the rendered page collapsed to **19 headings, 8 tables**, with everything from §12.5 to the end shown as raw DokuWiki markup. ''check_wrap.mjs'' and ''check_typography.mjs'' passed on both revisions and on the repair. Removing the stray delimiter gives **25 headings, 11 tables, 5 preformatted blocks** — exactly 24 + 1, 9 + 2, 4 + 1, which is what the new section adds, so the structure reconciles. Found by counting literal bold and monospace markup in the **rendered** DOM outside any preformatted block (''scripts/_qc0921_dom2.mjs''); no source-level check on this wiki can see this class of defect. **The lesson is that an odd delimiter count is a defect even while the page renders**, because the next edit to the page is what detonates it.)) |
| |
| ==== 12.5 Review ==== | ==== 12.5 Review ==== |
| |
| [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Link-hygiene sweep, 2026-09-17 ===== |
| | |
| | The paired content page's obsolete live-page caveat was removed after the fresh DOM confirmed that the root ''Artifacts'' link is live. The post-save DOM was re-checked for red links. No figures or citations changed. |
| | |
| | ===== ConsentAction audit propagation, 2026-09-17 ===== |
| | |
| | **What was wrong.** The crawler configuration table and family comparison called the 349/1,120 = 31.2% raw field population ''States consent action'', which could be read as 349 papers making that claim. The schema statistic itself is correct. |
| | |
| | **Command and real output.** ''node scripts/report_crawler.mjs'' → ''Consent action field populated (schema; not an audited paper claim) | 349 | 31.2%''; the family table's corresponding column is ''Consent field populated (schema)''. ''node scripts/consent_action_noninteraction_audit.mjs'' → **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels. |
| | |
| | **What changed.** The page and report output now distinguish field population from paper-claim support; the family percentages remain schema statistics and are labelled accordingly. |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_crawler.mjs'' and checked the edited row, the adjacent statefulness/interaction-depth rows, and the family table. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed. |
| | |
| | **Rejected finding.** The 349 schema statistic was retained rather than rewritten as 55, because this page's table measures extraction-field completeness. The audited claim result remains on [[privacy:consent]]. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. §6's figure predates the patch. Re-run, unedited first line: |
| | |
| | <code> |
| | $ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400 |
| | 106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk. |
| | </code> |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | quotes checked | 106 | 106 | population unchanged — the corpus has not moved | |
| | | exact | 58 | 58 | unchanged | |
| | | partial (≥60% of 5-word windows) | 32 | 32 | unchanged | |
| | | rescued from the PDF | — | **13** | new verdict; these were inside the old 16 | |
| | | below threshold | **16** | **3** (in both renderings) | 16 = 13 + 3 exactly; nothing else moved | |
| | | below-threshold rate | 15% | **2.8%** | 3 of 106 | |
| | |
| | **This page is the one where the refresh changes nothing about the conclusion.** All 16 had already been read by hand and all 16 were present, so the page never claimed a quote failure. What moves is the figure other pages compare themselves against: [[:provenance:design:ip_classification]] quoted "the ''%%--tools%%'' checks produce 15%" as its baseline, and that baseline is now 2.8%. That row was corrected on the same day. |
| | |
| | **Scope of this edit.** §6 and the content page's spot-check bullet. ''report_crawler.mjs'' and ''tool_fold.mjs'' were **not** re-run in this pass; no fold, framework count, trend or citation was touched, and every other figure on [[:programming:crawler]] and this page stands as published. |
| | |
| | The command is the code block above — it contains a regex alternation, and a DokuWiki table cell cannot hold a ''%%|%%''. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Artifact | ''out/qc0921/tools_openwpm_playwright.txt'' (full run, 13 RESCUED rows and 3 below-threshold rows listed) | |
| | | Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler]] | |
| | | Pages saved | this page, [[:programming:crawler]] | |
| | |
| | ===== Foxhound count errata, 2026-09-21 ===== |
| | |
| | **What was wrong.** The //Which specialised crawlers actually get used// table read ''SAP Project Foxhound | 8 | 1'', and the comparison table above it read **8**. The deeper per-paper audit on [[:programming:crawler:foxhound]] (''scripts/report_foxhound.mjs'' + ''scripts/fh_fold.mjs'', 2026-08-17) finds **9 papers that actually ran the browser**, and its own footnote said so and called these two tables queued for an errata edit. |
| | |
| | **Why the two disagreed — and it is not a denominator.** This table's two columns come from ''tools[].usedOrMentioned'': ''used'' or ''produced'' go left, everything else goes right. Khodayari et al.'s NDSS 2025 open-redirection paper is extracted as ''compared'' — //"We considered the following state-of-the-art detectors as baselines: dynamic taint-tracking [24, 36, 45] using Foxhound [44, 45]"// — and its comparison ran over the paper's 42,288-webpage subset of 50 randomly chosen applications((//"We chose 50 applications at random from the P2 dataset in Table I, encompassing a total of 42,288 webpages"//. The paper's own name for that subset is a subscripted P2 that the extracted text renders as ''P20'', so it is described here rather than named.)). Running a tool as a baseline is running it, so ''compared'' put a user of the browser in the //cites only// column. |
| | |
| | **The fix is in the script, not on the page.** ''report_crawler.mjs'' §6 now carries a ''ROLE_AUDITED'' map, currently one entry, ''SAP Project Foxhound'' → the ''ROLE'' map exported by ''fh_fold.mjs''. For a row in that map the enum verdict is replaced by the published hand verdict (''used'' / ''extended'' / ''baseline'' count as ran it; ''citation'' does not), and the script **throws** if any paper the row matches has no verdict in the audit, so the audit and the extraction cannot drift apart silently. Rows with no audit are untouched: this is deliberately not a general rule that ''compared'' means "ran it", because deciding that needs the sentence. |
| | |
| | <code> |
| | $ node scripts/report_crawler.mjs > scripts/report_crawler-output.txt |
| | $ diff <old> <new> |
| | 130c130 |
| | < SAP Project Foxhound 8 1 2024–2026 |
| | --- |
| | > SAP Project Foxhound 9 0 2024–2026 |
| | </code> |
| | |
| | **That is the whole diff of the report.** No other row, fold, residue, denominator or year range moved, and the folded-framework table above is untouched — this paper's Foxhound tuple carries ''category: "program-analysis"'', not a crawler-framework category, so it was never in that population. |
| | |
| | ^ Figure ^ Was ^ Is ^ Where ^ |
| | | Foxhound, papers using or producing it | 8 | **9** | //Which specialised crawlers actually get used// | |
| | | Foxhound, papers only citing it | 1 | **0** | same row | |
| | | Foxhound, //Papers// column | **8** | **9** | //Specialised Measurement Crawlers// comparison table | |
| | | growth sentence | "quadrupled Foxhound (2 → 8)" | "more than quadrupled Foxhound (2 → 9)" | same section | |
| | | Years used | 2024–2026 | 2024–2026 | unchanged | |
| | |
| | The **2** in that sentence is the pre-extension corpus figure and is not re-derived here; the paper this errata adds is NDSS 2025 and was not in that corpus, so the comparison is still like-for-like. |
| | |
| | **A finding rejected.** The //cites only// column now reads 0, while [[:programming:crawler:foxhound]] reports **2 citation-only papers**. That is not a new inconsistency and was not "fixed": those two papers have no ''tools[]'' tuple naming Foxhound at all, so this table — which is built from ''tools[]'' — cannot see them, and inventing a 2 here would mean this row alone was full-text-swept while every other row was not. The child page's reconciling footnote now says exactly this, and was edited in the same run. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Script changes | ''scripts/report_crawler.mjs'' (''ROLE_AUDITED'' block in §6, import of ''fh_fold.mjs''); committed output regenerated | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass | |
| | | Pages saved | this page, [[:programming:crawler]], [[:programming:crawler:foxhound]] | |
| |
