Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:openwpm [2026/09/17 09:24] – Record Sonnet currency review of Firefox pin; Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler:openwpm [2026/09/21 13:36] (current) – Quote-check refresh 2026-09-21: re-ran quote_check.mjs --tools open[ -]?wpm with the pypdf fallback; 12 below threshold -> 10 rescued + 2 below in both; new provenance section. Authored by Claude karel.kubicek.claude
@@ Line 1482: / Line 1482: @@
 All **65** evidence quotes attached to an OpenWPM ''tools[]'' tuple anywhere in the corpus were checked, twice, by two different methods.
-**First pass, ''scripts/quote_check.mjs %%--tools%% "open[ -]?wpm"'':** 34 exact, 19 partial (≥60% of five-word windows), 12 below threshold, 0 with no full text on disk.
+**First pass, ''scripts/quote_check.mjs %%--tools%% "open[ -]?wpm"'', re-run 2026-09-21 with the PDF fallback:** 34 exact, 19 partial (≥60% of five-word windows), **10 rescued from the PDF**, **2 below threshold in both renderings**, 0 with no full text on disk. See //Quote-check refresh, 2026-09-21// at the foot of this page.
-**The 12 below threshold are not fabrications, and the method was the problem.** Hand-checking with whitespace normalisation found 3 of them immediately; the rest failed because the repaired two-column text splices the other column into the middle of a sentence. With punctuation and spaces stripped, PETS 2025 ''intractable-cookie-crumbs-…'' reads:
+**The old figure was 12 below threshold, and the script now agrees with what this page worked out by hand.** 10 of the 12 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' — the stored text is what was wrong — and only **2** are below threshold in both, which is 3.1% of the 65. ''exact'' (34) and ''partial'' (19) did not move.
+**The 12 below threshold were not fabrications, and the method was the problem.** Hand-checking with whitespace normalisation found 3 of them immediately; the rest failed because the repaired two-column text splices the other column into the middle of a sentence. With punctuation and spaces stripped, PETS 2025 ''intractable-cookie-crumbs-…'' reads:
 <code>
@@ Line 1490: / Line 1492: @@
 </code>
-"Firefox v121.0" //is// in the paper; the word ''firefox'' is cut in half. A whole-quote containment test scores that quote as absent.
+"Firefox v121.0" //is// in the paper; the word ''firefox'' is cut in half. A whole-quote containment test scores that quote as absent. **That quote is one of the 10 the PDF fallback now rescues**: 46% of its five-word windows in the stored rendering, 62% in ''pypdf'''s. The two still below threshold in both are ''PETS/2022/my-cookie-is-a-phoenix…'' (50%) and ''USENIX/2022/automating-cookie-consent-and-gdpr-violation-detection'' (58%), and in both the mismatch is inside the extraction's own string — a dropped citation marker — which no rendering of the PDF can repair.
 **Second pass, ''scripts/openwpm_quotecheck.py'', written for this page.** Skeletonise (lowercase, drop everything that is not a letter or digit), cut each quote into overlapping 4-word windows, and report the share of the quote's **word positions** covered by a present window. Coverage, not window count: one splice in a twelve-word quote kills up to six windows but only one word position. An earlier version of this script counted windows and reported 19 quotes below 60%, all of them short quotes with a single splice — a metric artefact, and it is recorded here because the first draft of this page nearly published it.
@@ Line 2376: / Line 2378: @@
 the v0.35.0 and v0.36.0 pins returned HTTP 200 from their resolved Mozilla build
 URLs.
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. §5's first-pass figure predates the patch. Re-run, unedited first line:
+<code>
+$ node scripts/quote_check.mjs --tools 'open[ -]?wpm' --show 400
+quotes checked: 34 exact, 19 partial (>=60% of 5-word windows), 10 rescued from the PDF, 2 below threshold in both renderings, 0 with no full text on disk.
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 65 | 65 | population unchanged — the corpus has not moved |
+| exact | 34 | 34 | unchanged |
+| partial (≥60% of 5-word windows) | 19 | 19 | unchanged |
+| rescued from the PDF | — | **10** | new verdict; these were inside the old 12 |
+| below threshold | **12** | **2** (in both renderings) | 12 = 10 + 2 exactly; nothing else moved |
+**This page argued for the fallback before the fallback existed, and the second pass is not superseded by it.** §5's ''openwpm_quotecheck.py'' answers a different question — how much of each quote's //word positions// a present window covers — and its verdict on the 65 (52 verbatim, 13 split by a splice, 0 below 60%, none unsupported) is unchanged and was not re-derived here. The shared checker's ''%%--tools%%'' pass has simply stopped disagreeing with it about 10 of the 12.
+**Scope of this edit.** §5's first-pass figure and the sentence naming the ''intractable-cookie-crumbs'' example. ''openwpm_quotecheck.py'' was **not** re-run, ''report_*'' scripts were not re-run, and no other figure, fold or citation on this page was touched.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Artifact | ''out/qc0921/tools_openwpm.txt'' (full run, 10 RESCUED rows and 2 below-threshold rows listed) |
+| Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Pages saved | this page only — [[:programming:crawler:openwpm]] carries no quote-check figure |