Differences

This shows you the differences between two versions of the page.

--- provenance:design:longitudinal [2026/09/07 18:21] – Generic (Fable) pass returned after all: log its findings in 18.11, correct 18.4's kappa defence (the coders DO disagree on 3 of 5 all-four candidates — the earlier text said the opposite), delete an unverifiable claim about coders opening full text, reso karel.kubicek.claude
+++ provenance:design:longitudinal [2026/09/22 04:33] (current) – Figures-vs-script finding accepted: the 18.5 heading still named a three-generations-stale count (43); renamed to carry no count. No inbound anchor links across 188 pages. Authored by Claude karel.kubicek.claude
@@ Line 59: / Line 59: @@
 </code>
-Both default to ''/workspace/publications_dataset/data/extract/run1'' and ''.../data/fulltext''; ''--run'' and ''--fulltext'' override.
+Both default to ''/workspace/publications_dataset/data/extract/run1'' and ''.../data/fulltext''; ''%%--run%%'' and ''%%--fulltext%%'' override.
 ===== 5. The fold, and its residue in full =====
@@ Line 1473: / Line 1473: @@
 ^ Item ^ Value ^
 | Content page | [[design:longitudinal]], §"The frame moves even when you pin it" |
-| Saves | rev **1788465991** (''--if-rev 1787798449''), rev **1788467004** (''--if-rev 1788465991''), rev **1788467915** (''--if-rev 1788467004'') |
+| Saves | rev **1788465991** (''%%--if-rev%% 1787798449''), rev **1788467004** (''%%--if-rev%% 1788465991''), rev **1788467915** (''%%--if-rev%% 1788467004'') |
 | Corpus queried | none — this is a vendor date |
 | External source | Tranco's HTTP API and front page, fetched 2026-09-03 |
@@ Line 1595: / Line 1595: @@
 Every disagreement is printed paper by paper in the unedited output in §18.12.
-==== 18.5 The quote check, and the 43 that did not locate ====
+==== 18.5 The quote check, and what "unlocated" actually means ====
 Every non-empty ''quote'' must appear in the paper it is attributed to. 1,000 pin codings, 464 with no quote (all of them ''not-stated'', which the codebook permits and the script verifies: **0 codings carry a value other than ''not-stated'' with no quote**), 536 checked.
@@ Line 1604: / Line 1604: @@
   * what ''.cols'' actually holds: ''%%We implemented FourthParty as an extension to Mozilla \n B. Design Principles Firefox. It currently instruments%%''
-The two-column repair still splices a quarter of pages, and a sentence crossing a column boundary comes back with the neighbouring column injected into the middle of it. **A coder who de-splices that correctly writes a quote that is a substring of nothing.** Failing it would call a correct reading a hallucination. So the check now searches ''paper.cols.txt'', ''paper.norm.txt'', ''paper.txt'' and the packet, in four escalating modes, the last of which cuts the quote into 6-word shingles and passes at 80% found.
+The two-column repair still splices a quarter of pages, and a sentence crossing a column boundary comes back with the neighbouring column injected into the middle of it. **A coder who de-splices that correctly writes a quote that is a substring of nothing.** Failing it would call a correct reading a hallucination. So the check now searches ''paper.cols.txt'', ''paper.norm.txt'', ''paper.txt'' and the packet, in four escalating modes, the last of which cuts the quote into 6-word shingles and passes at 80% found. **And since 2026-09-22 a fifth text:** a quote located in none of the stored renderings is re-checked, in the same four modes, against a ''pypdf'' rendering of the same ''paper.pdf'' (''scripts/pdftext.py''). That verdict is reported as **RESCUED** — the quote is in the paper and the stored text is what is wrong. It is the fallback ''scripts/quote_check.mjs'' has had since 2026-09-04; this checker did not have it, and until it did, its unlocated bucket was not comparable with the rest of the site's.
 ^ Mode ^ Codings ^ Share of 536 ^
@@ Line 1611: / Line 1611: @@
 | Loose (de-hyphenated, quotes and dashes folded, case-insensitive) | 2 | 0.4% |
 | De-spliced (6-word shingles, ≥80%) | 3 | 0.6% |
-| **Located by no mode** | **42** | **7.8%** |
+| Rescued from the ''pypdf'' rendering of ''paper.pdf'' | 35 | 6.5% |
+| **Located by no mode, in any of the five texts** | **7** | **1.3%** |
-//The counts above are after the one correction described below. Before it: 488 exact, 43 unlocated (8.0%).//
+//The counts above are after the one correction described below **and** after the ''pypdf'' fallback added on 2026-09-22. Before the correction and before the fallback: 488 exact, 43 unlocated (8.0%). With the correction and still without the fallback — which is what this table published until 2026-09-22 — 489 exact, 42 unlocated (7.8%). The 42 did not become 7 because any coding changed; 35 of them are the same quotes, mechanically located in a second rendering of the same PDF.//
-**All 43 were then read by hand** — three sub-agents, each given the paper's three renderings and the PDF, asked to classify each quote as a faithful de-splice, a supported paraphrase, an unsupported paraphrase, or a quote lifted from the wrong part of the paper, and //independently// to say whether the paper supports the coded pin value at all.
+**All 43 were then read by hand** — three sub-agents, each given the paper's three renderings and the PDF, asked to classify each quote as a faithful de-splice, a supported paraphrase, an unsupported paraphrase, or a quote lifted from the wrong part of the paper, and //independently// to say whether the paper supports the coded pin value at all. **The 2026-09-22 fallback is an independent, mechanical check on that hand audit, and it agrees.** 35 of the 42 the hand audit called faithful de-splices are now located verbatim or near-verbatim in ''pypdf'''s rendering, by a script with no access to the hand verdicts. The 7 that remain unlocated are all inside the hand-audited set and were all given the same verdict — faithful de-splice — so the fallback has not contradicted a single one of the 43.
 ^ Verdict ^ Codings ^
@@ Line 1628: / Line 1629: @@
   * **What moved on the content page:** the vantage row over the 124 went from 25/44/10 to **26/45/9** (words / estimate / moved), and the all-pins fixed-versus-not agreement from 80.8% to 81.7%. The all-four figure, the 124, and the 6-of-23 survivor count did not move.
-The verdict files are ''out_pins/qaudit/verdict_1.tsv'' … ''verdict_3.tsv''. **The lesson is the check, not the count.** A plain substring test on ''paper.cols.txt'' alone would have reported a 9.1% fabrication rate here, of which the real rate is **0.2% (1 of 536)** — and the one real defect was not a fabrication at all but a quote from the wrong section, which no substring test can catch. One reviewer flagged a further boundary case worth recording rather than changing: two codings use ''same-explicit'' on the strength of a pinned identifier in papers whose ''repeatKind'' is ''single-crawl'', where the codebook's rule 4 would suggest ''same-implied''. Both readings are inside the codebook as written; the rule needs tightening before a rerun, and neither paper is in the 124.
+The verdict files are ''out_pins/qaudit/verdict_1.tsv'' … ''verdict_3.tsv''. **The lesson is the check, not the count.** A plain substring test on ''paper.cols.txt'' alone would have reported a 9.1% fabrication rate here, of which the real rate is **0.2% (1 of 536)** — and the one real defect was not a fabrication at all but a quote from the wrong section, which no substring test can catch. The 2026-09-22 fallback moves the //automated// bucket from 7.8% to **1.3%**, and that 1.3% is still not a fabrication rate: the real rate is the hand-audited 0.2%, unchanged, and the seven remaining are the hand audit's own faithful de-splices. An automated unlocated count is an upper bound on a fabrication rate and has never been an estimate of one. One reviewer flagged a further boundary case worth recording rather than changing: two codings use ''same-explicit'' on the strength of a pinned identifier in papers whose ''repeatKind'' is ''single-crawl'', where the codebook's rule 4 would suggest ''same-implied''. Both readings are inside the codebook as written; the rule needs tightening before a rerun, and neither paper is in the 124.
 ==== 18.6 What changed on the content page ====
@@ Line 1684: / Line 1685: @@
 Re-ran all six scripts and confirmed every committed ''-output.txt'', ''population.jsonl'' and ''valsample.tsv'' reproduces byte-identically, and that ''pins_population.mjs'' lines 17 and 19 are character-for-character the predicate in ''report_longitudinal.mjs''.
-  - **Stale table cell.** The 23-row exemplar table's vantage cell for {[singanamalla2022_respect]} still read ''**changed**''. The table had been pasted from ''--wiki'' output generated //before// the §18.5 correction was applied. **Accepted and fixed** by regenerating the table; it was the only divergent cell of 207.
+  - **Stale table cell.** The 23-row exemplar table's vantage cell for {[singanamalla2022_respect]} still read ''**changed**''. The table had been pasted from ''%%--wiki%%'' output generated //before// the §18.5 correction was applied. **Accepted and fixed** by regenerating the table; it was the only divergent cell of 207.
   - **Wrong hand-written count.** The prose said //"nine turn out not to be repeated crawls at all"//. Recounting from the coding files: of the 23, eight are ''parallel-not-temporal'', one of which is a survivor, so among the 17 non-survivors **seven** are not repeat crawls and **ten** are. The script never prints this aggregate, which is exactly why it was wrong. **Accepted and fixed.**
   - **The structural guard in ''build_provenance_pins.py'' asserted much less than it claimed.** It compared heading, table and file-block counts over the //whole// output against the whole live page. Because the appended section adds eleven headings of its own, the check had eleven headings of slack: the reviewer mutation-tested it by deleting a heading from the **preserved** live content and got ''headings: 27 -> 37'' and **exit 0**. **Accepted and fixed** — the counts are now compared on the preserved text alone, which must equal the live page byte for byte, and the appended section is separately required to be non-empty. This is a page-matches-its-own-script failure of the kind the guard existed to prevent, and reading the script would not have found it; mutating it did.
-  - Mutation-tested and **confirmed sound**: the partial-population refusal in ''report_pins_audit.mjs'' (deleting one of 250 codings gives ''FATAL: coded 249 of 250''), the citekey-map guard (removing one slug exits 1), ''pins_quotecheck.mjs'' (an injected fabricated quote raises NOT FOUND from 42 to 43), ''pins_agreement.mjs'' (a removed coder-B file drops //n// to 29 and prints ''MISSING'' rather than silently continuing at 30), and the missing-file and empty-file guards in the builder.
+  - Mutation-tested and **confirmed sound**: the partial-population refusal in ''report_pins_audit.mjs'' (deleting one of 250 codings gives ''FATAL: coded 249 of 250''), the citekey-map guard (removing one slug exits 1), ''pins_quotecheck.mjs'' (an injected fabricated quote raises NOT FOUND from 7 to 8 and is printed in full; re-run on 2026-09-22 against the patched script — the pin was 42 → 43 before the ''pypdf'' fallback), ''pins_agreement.mjs'' (a removed coder-B file drops //n// to 29 and prints ''MISSING'' rather than silently continuing at 30), and the missing-file and empty-file guards in the builder.
   - Hand-computed Cohen's //κ// for the list pin from the raw pairs: observed 0.667, expected 0.351, //κ// = 0.486 → 0.49, matching the script. No bug.
   - Checked the figures **outside** the edited window against ''report_longitudinal-output.txt'' and found none stale.
@@ Line 1695: / Line 1696: @@
   - All 32 keys on the content page and all 8 on this page resolve against the **live** ''literature:bibliography'' (850 entries, fetched today, not a local copy). The content page's key set is unchanged from the live version; this page gains exactly one key, {[singanamalla2022_respect]}, which is already in the bibliography and already cited on the content page. No entry added, none changed.
   - Verified all 23 rows of ''out_pins/citekeys.json'' against the corpus index — title, venue, year and first-author surname — including that ''fouad2020_missed'' and ''fouad2024_devil'' are two distinct papers and are not conflated. No swapped keys.
-  - **Independently found the same stale vantage cell as Pass A**, by diffing the published table against a fresh ''--wiki'' run, and confirmed all three quotes in the §18.5 account by locating them in the paper itself.
+  - **Independently found the same stale vantage cell as Pass A**, by diffing the published table against a fresh ''%%--wiki%%'' run, and confirmed all three quotes in the §18.5 account by locating them in the paper itself.
   - Spot-checked **24 evidence quotes** (10 ''changed'', 14 ''same-explicit'') against all three renderings: **24 of 24** located and supporting their coded value. Flagged one as loose rather than wrong — the Zoom-apps ''browser'' quote is about a crawler and parser rebuild rather than a browser version. **Noted, not changed**: the coding is ''changed'' either way and the codebook's browser pin covers the driven stack.
   - Verified the "12 further papers outside this 23" arithmetic (18 all-four minus the 6 in the table) and every external figure quoted from a cited paper.
@@ Line 2413: / Line 2414: @@
 //              de-splicing mode and it is reported separately and listed in full,
 //              because it is the weakest evidence and has to be read by a human.
-// Anything failing all four is printed in full.
+// A quote that fails all four against every stored rendering is then tried
+// against a FIFTH text: a pypdf rendering of the same paper.pdf, produced on
+// demand by `scripts/pdftext.py` via `scripts/pdftext_cache.mjs`. `.cols` and
+// pypdf splice different sentences, so a coder who read the PDF correctly can
+// produce a quote that is absent from every stored rendering and verbatim in
+// the PDF. The same four modes are tried there and a pass is reported as
+//   RESCUED  located in the PDF but in none of the stored renderings. The
+//            quote is in the paper; the stored text is what is wrong.
+// This is the fallback `scripts/quote_check.mjs` and `scripts/report_webxray.mjs`
+// already use; it was added here on 2026-09-22, and before that this checker's
+// NOT FOUND bucket was not comparable with theirs.
+// Anything failing all four modes against all five texts is printed in full.
 import fs from 'node:fs';
 import path from 'node:path';
+import { pdfText } from './pdftext_cache.mjs';
 const argv = process.argv.slice(2);
@@ Line 2433: / Line 2446: @@
     .replace(/[ ]/g, ' ')
     .toLowerCase();
+// Fraction of the quote's overlapping SHINGLE_N-word windows present in a
+// loose-folded haystack; null when the quote is too short to shingle.
+const shingleFrac = (q, hayLoose) => {
+  const w = loose(q).split(' ').filter(Boolean);
+  if (w.length < SHINGLE_N) return null;
+  const sh = [];
+  for (let i = 0; i + SHINGLE_N <= w.length; i++) sh.push(w.slice(i, i + SHINGLE_N).join(' '));
+  return sh.filter((x) => hayLoose.includes(x)).length / sh.length;
+};
 const pop = fs.readFileSync(POP, 'utf8').trim().split('\n').map(JSON.parse);
@@ Line 2439: / Line 2462: @@
 const fails = [];
 const splicedList = [];
+const rescuedList = [];
 let missing = 0;
@@ Line 2468: / Line 2492: @@
     if (hayL.includes(loose(q))) { modes.loose++; continue; }
     // de-splicing mode
-    const w = loose(q).split(' ').filter(Boolean);
+    const frac = shingleFrac(q, hayL);
-    if (w.length >= SHINGLE_N) {
+    if (frac !== null && frac >= SHINGLE_MIN) {
-      const sh = [];
+      modes.spliced++;
-      for (let i = 0; i + SHINGLE_N <= w.length; i++) sh.push(w.slice(i, i + SHINGLE_N).join(' '));
+      splicedList.push({ key: p.key, pin, value: v.value, quote: q, frac });
-      const hit = sh.filter((s) => hayL.includes(s)).length;
+      continue;
-      if (hit / sh.length >= SHINGLE_MIN) {
+    }
-        modes.spliced++;
+    // Fifth text: pypdf. Only consulted for a quote that is in none of the
-        splicedList.push({ key: p.key, pin, value: v.value, quote: q, frac: hit / sh.length });
+    // stored renderings, so it costs nothing on a clean run.
+    const pdf = pdfText(p.key);
+    if (pdf !== null) {
+      const pdfC = collapse(pdf);
+      const pdfL = loose(pdf);
+      let mode = null;
+      if (pdf.includes(q)) mode = 'exact';
+      else if (pdfC.includes(collapse(q))) mode = 'collapsed';
+      else if (pdfL.includes(loose(q))) mode = 'loose';
+      else {
+        const pdfFrac = shingleFrac(q, pdfL);
+        if (pdfFrac !== null && pdfFrac >= SHINGLE_MIN) mode = `spliced ${(100 * pdfFrac).toFixed(0)}%`;
+      }
+      if (mode !== null) {
+        rescuedList.push({ key: p.key, pin, value: v.value, quote: q, frac, mode });
         continue;
       }
-      fails.push({ key: p.key, pin, value: v.value, quote: q, frac: hit / sh.length });
-    } else {
-      fails.push({ key: p.key, pin, value: v.value, quote: q, frac: null });
     }
+    fails.push({ key: p.key, pin, value: v.value, quote: q, frac, pdf: pdf !== null });
   }
 }
@@ Line 2490: / Line 2526: @@
 console.log(`coded dir: ${CODED}`);
 console.log(`renderings searched per paper: paper.cols.txt, paper.norm.txt, paper.txt, and the excerpt packet`);
+console.log(`fallback for a quote found in none of those: a pypdf rendering of paper.pdf (scripts/pdftext.py), same four modes, reported as RESCUED`);
 console.log(`de-splicing mode: ${SHINGLE_N}-word shingles, pass at >= ${(100 * SHINGLE_MIN).toFixed(0)}% found`);
 console.log(`\npapers with no coding file (skipped): ${missing}`);
@@ Line 2499: / Line 2536: @@
 console.log(`      loose match:                 ${pc(modes.loose)}`);
 console.log(`      spliced match (de-spliced):  ${pc(modes.spliced)}`);
+console.log(`      RESCUED from the PDF:        ${pc(rescuedList.length)}`);
 console.log(`      NOT FOUND:                   ${pc(fails.length)}`);
+console.log(`   (papers with no pypdf rendering available: ${[...new Set(fails.filter((f) => !f.pdf).map((f) => f.key))].length})`);
 // A quote-less coding that is not `not-stated` is a coding with no evidence.
@@ Line 2517: / Line 2556: @@
 for (const f of splicedList.sort((a, b) => a.frac - b.frac)) {
   console.log(`   ${f.key}  [${f.pin}=${f.value}]  shingles ${(100 * f.frac).toFixed(0)}%\n      ${f.quote}`);
+}
+console.log(`\nRESCUED — in none of the stored renderings, located in the pypdf rendering of paper.pdf (${rescuedList.length}).`);
+console.log(`These are defects in the stored text, not in the hand coding:`);
+for (const f of rescuedList) {
+  console.log(`   ${f.key}  [${f.pin}=${f.value}]  stored shingles ${f.frac == null ? 'n/a' : (100 * f.frac).toFixed(0) + '%'} -> pdf ${f.mode}\n      ${f.quote}`);
 }
@@ Line 3243: / Line 3288: @@
 coded dir: out_pins/coded
 renderings searched per paper: paper.cols.txt, paper.norm.txt, paper.txt, and the excerpt packet
+fallback for a quote found in none of those: a pypdf rendering of paper.pdf (scripts/pdftext.py), same four modes, reported as RESCUED
 de-splicing mode: 6-word shingles, pass at >= 80% found
@@ Line 3253: / Line 3299: @@
       loose match:                 2  0.4%
       spliced match (de-spliced):  3  0.6%
-      NOT FOUND:                   42  7.8%
+      RESCUED from the PDF:        35  6.5%
+      NOT FOUND:                   7  1.3%
+   (papers with no pypdf rendering available: 0)
 codings with no quote and a value other than not-stated: 0
@@ Line 3265: / Line 3313: @@
       we sampled 100 domains from each parking service and setup an additional crawler which crawled them daily for a week.
-QUOTES NOT LOCATED, printed in full (42):
+RESCUED — in none of the stored renderings, located in the pypdf rendering of paper.pdf (35).
-/IEEE-SP/third-party-web-tracking-policy-and-technology  [browser=same-implied]  shingles 75%
+These are defects in the stored text, not in the hand coding:
+/IEEE-SP/third-party-web-tracking-policy-and-technology  [browser=same-implied]  stored shingles 75% -> pdf exact
       We implemented FourthParty as an extension to Mozilla Firefox.
-/WWW/understanding-and-combating-link-farming-in-the-twitter-social-network  [list=same-explicit]  shingles 50%
+/NDSS/parking-sensors-analyzing-and-detecting-parked-domains  [vantage=varied-deliberate]  stored shingles 47% -> pdf exact
-      We re-crawled the profile page of each user in the above dataset in February 2011
-/NDSS/parking-sensors-analyzing-and-detecting-parked-domains  [vantage=varied-deliberate]  shingles 47%
       To assess any geographical differences, we performed this crawl in parallel from the United States and one country from Europe.
-/USENIX/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  [state=same-implied]  shingles 38%
+/USENIX/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  [state=same-implied]  stored shingles 38% -> pdf exact
       We managed to emulate typical browser behavior in all but one case (BlueSky).
-/WWW/stress-testing-the-booters-understanding-and-undermining-the-business-of-ddos-se  [list=same-implied]  shingles 58%
+/CCS/poster-probing-tor-hidden-service-with-dockers  [list=same-explicit]  stored shingles 69% -> pdf exact
-      In order to measure the stability of these amplifiers we probed them periodically for 13 weeks to understand how many were still located at the same IP and misconfiged.
-/CCS/poster-probing-tor-hidden-service-with-dockers  [list=same-explicit]  shingles 69%
       The number of Onion addresses in this experiment was 16,683, and the addresses were crawled twice a day at 12-hour intervals.
-/CCS/how-you-get-shot-in-the-back-a-systematical-study-about-cryptojacking-in-the-rea  [list=same-implied]  shingles 50%
+/CCS/how-you-get-shot-in-the-back-a-systematical-study-about-cryptojacking-in-the-rea  [list=same-implied]  stored shingles 50% -> pdf exact
       we can only continuously monitor 1,000 samples. Besides, we re-visit these samples every 3 days
-/IMC/visiblev8-in-browser-monitoring-of-javascript-in-the-wild  [list=same-explicit]  shingles 0%
+/IMC/visiblev8-in-browser-monitoring-of-javascript-in-the-wild  [list=same-explicit]  stored shingles 0% -> pdf exact
       to visit the Alexa top 50k web domains
-/WWW/a-first-look-at-deep-learning-apps-on-smartphones  [list=changed]  shingles 52%
+/WWW/a-first-look-at-deep-learning-apps-on-smartphones  [list=changed]  stored shingles 52% -> pdf exact
       We have crawled two datasets at different moments, June 2018 and September 2018, which are three months apart. The two app datasets have more than 2/3 overlapped apps.
-/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [browser=same-implied]  shingles 43%
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [browser=same-implied]  stored shingles 43% -> pdf exact
       We installed the widely used OpenWPM framework and configured it to use four vanilla Mozilla Firefox browsers in parallel
-/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [vantage=same-implied]  shingles 50%
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [vantage=same-implied]  stored shingles 50% -> pdf exact
       the physical machine on which the VM was deployed is located in the EU, meaning that the visited websites must comply with GDPR
-/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [state=same-implied]  shingles 43%
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [state=same-implied]  stored shingles 43% -> pdf exact
       we did not install any extension on the browser that would stop the websites from storing or reading cookies
-/WWW/mobile-app-risk-ranking-via-exclusive-sparse-coding  [list=changed]  shingles 77%
+/WWW/mobile-app-risk-ranking-via-exclusive-sparse-coding  [list=changed]  stored shingles 77% -> pdf spliced 91%
       There are no overlaps for the above datasets, and the apps are from a wide range of functionalities, including game, news, music, social, movies/TV, travel, life, etc.
-/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [vantage=varied-deliberate]  shingles 53%
+/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [vantage=varied-deliberate]  stored shingles 53% -> pdf exact
       we conduct the measurements from three different locations (Europe (DE), North America (US), and Asia (JP)) to account for possible geographical differences
-/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [state=same-implied]  shingles 53%
+/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [state=same-implied]  stored shingles 53% -> pdf exact
       We used the same method to create the browser profile for our experiment crawls that we utilized in the pre-study.
-/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [list=same-explicit]  shingles 0%
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [list=same-explicit]  stored shingles 0% -> pdf exact
       Tranco list of top sites [18] (snapshot 77PX)
-/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [browser=same-implied]  shingles 33%
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [browser=same-implied]  stored shingles 33% -> pdf exact
       We crawl using two variants of Puppeteer [4] controlling Chromium 80: a lower-realism naive variant and a higher-realism stealth variant.
-/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [state=same-implied]  shingles 70%
+/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [browser=same-implied]  stored shingles 50% -> pdf exact
-      the browser is launched with a clean user profile (i.e., no cookies or cached content)
-/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [browser=same-implied]  shingles 50%
       we rely on a fully-featured, headless Chrome browser, governed by Selenium.
-/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [vantage=same-implied]  shingles 47%
+/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [vantage=same-implied]  stored shingles 47% -> pdf exact
       ODIN uses university IP addresses (one per profile) and a /24 subnet from a research-friendly, but less well-known VPS provider
-/IMC/muzeel-assessing-the-impact-of-javascript-dead-code-elimination-on-mobile-web-pe  [browser=varied-deliberate]  shingles 17%
+/IMC/muzeel-assessing-the-impact-of-javascript-dead-code-elimination-on-mobile-web-pe  [browser=varied-deliberate]  stored shingles 17% -> pdf exact
       We now evaluate Muzeel across browsers: Chrome, Edge, Firefox, and Brave.
-/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [list=same-explicit]  shingles 57%
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [list=same-explicit]  stored shingles 57% -> pdf loose
       Our initial sample set was the 5000 domains with the most requests to the third party, as indicated by the Referer field in the CDN's third party logs.
-/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [browser=changed]  shingles 38%
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [browser=changed]  stored shingles 38% -> pdf exact
       Results shown are for Firefox (v91) for later comparison, since only Firefox has client-side support for ORIGIN Frame.
-/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [state=same-explicit]  shingles 57%
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [state=same-explicit]  stored shingles 57% -> pdf exact
       Client caches are intentionally deleted between measurements in all of our active measurements to prevent bias from caching effects
-/PETS/a-global-survey-of-android-dual-use-applications-used-in-intimate-partner-survei  [list=same-explicit]  shingles 52%
+/USENIX/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  [list=same-implied]  stored shingles 59% -> pdf exact
-      we crawled Google Play again in all fifteen languages for one day in February 2022 with the queries we obtained and used in our 2020 crawling.
-/PETS/atom-ad-network-tomography  [state=same-implied]  shingles 79%
-      Each persona was associated with a unique browser running on an isolated virtual machine with a unique IP address.
-/USENIX/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  [list=same-implied]  shingles 59%
       We scraped Hansa 17 times between late 2015 and mid-2017, collecting a total of 332,795 pages amounting to 39.5 GB of data.
-/WWW/leveraging-googles-publisher-specific-ids-to-detect-website-administration  [list=changed]  shingles 58%
+/WWW/leveraging-googles-publisher-specific-ids-to-detect-website-administration  [list=changed]  stored shingles 58% -> pdf exact
       We perform our analysis on websites which have been crawled in all snapshots (i.e., their intersection) and contain at least one Publisher ID. There are over 191,000 such websites.
-/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [list=same-explicit]  shingles 63%
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [list=same-explicit]  stored shingles 63% -> pdf exact
       We successfully conducted the four parallel crawls for 99.31% of the 20K websites.
-/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [vantage=same-explicit]  shingles 71%
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [vantage=same-explicit]  stored shingles 71% -> pdf exact
       We run all crawls in the US to minimize the impact of the EU GDPR and do not interact with cookie banners.
-/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [state=varied-deliberate]  shingles 46%
+/WWW/automatic-discovery-of-emerging-browser-fingerprinting-techniques  [list=same-explicit]  stored shingles 59% -> pdf exact
-      We conduct four parallel crawls: two with third-party cookies enabled (3P-Allowed) and two with third-party cookies blocked (3P-Blocked).
-/WWW/automatic-discovery-of-emerging-browser-fingerprinting-techniques  [list=same-explicit]  shingles 59%
       we redid the crawling on the same 10K domains in September 2022 (11-month gap) and applied our system to this data set by following the same procedure.
-/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [list=same-implied]  shingles 57%
+/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [list=same-implied]  stored shingles 57% -> pdf exact
       clients repeatedly accessed onion services and the top 150 accessed websites according to the Tranco ranking [68], in parallel.
-/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [browser=same-explicit]  shingles 61%
+/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [browser=same-explicit]  stored shingles 61% -> pdf exact
       Clients also run on an isolated Docker container that executes a 0.4.7.10 Tor process and issues requests to Tor via the Python libraries selenium [58] and tbselenium [3]
-/IMC/learning-as-to-organization-mappings-with-borges  [list=same-implied]  shingles 64%
+/IMC/learning-as-to-organization-mappings-with-borges  [list=same-implied]  stored shingles 64% -> pdf loose
       we used the PeeringDB snapshot from July 24, 2024, and CAIDA's AS2Org from July 1, 2024.
-/IMC/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre  [list=same-explicit]  shingles 77%
+/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [browser=varied-deliberate]  stored shingles 48% -> pdf loose
-      We restrict the list to sites that appear in all of the top 100k lists over this period to avoid having our results affected by list churn
-/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [browser=varied-deliberate]  shingles 48%
       Since OpenWPM only supports Firefox and, unlike Chrome, Firefox does not expose the partitioned_key field in its cookie storage, we modified BannerClick to operate in Chrome's default mode.
-/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [state=varied-deliberate]  shingles 70%
+/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [state=varied-deliberate]  stored shingles 70% -> pdf exact
       We use OpenWPM [24] to conduct a combination of stateful and stateless crawls aiming to observe the interplay of the setter and the sender website
-/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [list=same-implied]  shingles 55%
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [list=same-implied]  stored shingles 55% -> pdf exact
       we joined the top one million websites from Tranco [49] with SimilarWeb, a web traffic estimator
-/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [browser=same-implied]  shingles 56%
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [browser=same-implied]  stored shingles 56% -> pdf exact
       Each website was opened in a Chrome instance with two installed custom Chrome extensions.
-/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [vantage=same-implied]  shingles 47%
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [vantage=same-implied]  stored shingles 47% -> pdf loose
       All website visits took place on Linux virtual machines in Google Cloud's US Central region between September and November 2024.
-/PETS/who-s-watching-you-zoom-investigating-privacy-of-third-party-zoom-apps  [list=changed]  shingles 40%
+/PETS/who-s-watching-you-zoom-investigating-privacy-of-third-party-zoom-apps  [list=changed]  stored shingles 40% -> pdf exact
       between December 2023 and December 2024, 667 new apps were added, 212 apps were removed, and 49 apps were renamed
+QUOTES NOT LOCATED, printed in full (7):
+/WWW/understanding-and-combating-link-farming-in-the-twitter-social-network  [list=same-explicit]  shingles 50%
+      We re-crawled the profile page of each user in the above dataset in February 2011
+/WWW/stress-testing-the-booters-understanding-and-undermining-the-business-of-ddos-se  [list=same-implied]  shingles 58%
+      In order to measure the stability of these amplifiers we probed them periodically for 13 weeks to understand how many were still located at the same IP and misconfiged.
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [state=same-implied]  shingles 70%
+      the browser is launched with a clean user profile (i.e., no cookies or cached content)
+/PETS/a-global-survey-of-android-dual-use-applications-used-in-intimate-partner-survei  [list=same-explicit]  shingles 52%
+      we crawled Google Play again in all fifteen languages for one day in February 2022 with the queries we obtained and used in our 2020 crawling.
+/PETS/atom-ad-network-tomography  [state=same-implied]  shingles 79%
+      Each persona was associated with a unique browser running on an isolated virtual machine with a unique IP address.
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [state=varied-deliberate]  shingles 46%
+      We conduct four parallel crawls: two with third-party cookies enabled (3P-Allowed) and two with third-party cookies blocked (3P-Blocked).
+/IMC/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre  [list=same-explicit]  shingles 77%
+      We restrict the list to sites that appear in all of the top 100k lists over this period to avoid having our results affected by list churn
 </file>
@@ Line 3487: / Line 3538: @@
 [[design:longitudinal|← back to the content page]] · [[provenance:design:website_selection|the 2026-09-03 Alexa date audit]] · [[literature:corpus|corpus-level provenance]]
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Quote-check refresh, 2026-09-22 =====
+**What was wrong.** ''scripts/pins_quotecheck.mjs'' searched four texts — the three
+stored renderings and the excerpt packet — and had **no ''pypdf'' fallback**. The
+-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] measured that
+.1% of quotes unlocatable in ''paper.cols.txt'' are present in an independent
+''pypdf'' rendering of the same ''paper.pdf'', and ''scripts/quote_check.mjs'' was
+patched that day to report a **RESCUED** verdict. This checker was never patched, so
+§18.5's unlocated bucket was an artefact of a narrower search than the rest of the
+site was using. It has now been patched the same way and re-run.
+<code>
+$ node scripts/pins_quotecheck.mjs
+      exact match:                 489  91.2%
+      whitespace-collapsed match:  0  0.0%
+      loose match:                 2  0.4%
+      spliced match (de-spliced):  3  0.6%
+      RESCUED from the PDF:        35  6.5%
+      NOT FOUND:                   7  1.3%
+   (papers with no pypdf rendering available: 0)
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| pin codings | 1,000 | 1,000 | population unchanged — no coding was touched |
+| checked (non-empty quote) | 536 | 536 | unchanged |
+| exact | 489 | 489 | unchanged |
+| whitespace-collapsed | 0 | 0 | unchanged |
+| loose | 2 | 2 | unchanged |
+| de-spliced (6-word shingles, ≥80%) | 3 | 3 | unchanged |
+| rescued from the PDF | — | **35** | new verdict; all 35 were inside the old 42 |
+| located by no mode | **42** (7.8%) | **7** (1.3%) | 42 = 35 + 7 exactly; nothing else moved |
+**The hand-audited fabrication rate is unchanged at 0.2%, and this run is evidence
+for it rather than a replacement of it.** The 42 had already been read by three
+sub-agents, who called 42 of the 43 faithful de-splices of real sentences and found
+**one** real defect — a quote from the wrong part of {[singanamalla2022_respect]} —
+and **zero** fabrications. The fallback now locates **35 of those same 42** in a
+second rendering, by a script that has never seen the hand verdicts. It contradicts
+none of them. The 7 that remain are all inside the hand-audited set and all carry the
+verdict //faithful de-splice//. **1.3% is not a fabrication rate either**; like 7.8%
+and 9.1% before it, it is an upper bound produced by an instrument, and the rate is
+still the hand-read **0.2% (1 of 536)**.
+**The mutation test was re-run and its pin moved.** The guard recorded in §18.11 was
+//"an injected fabricated quote raises NOT FOUND from 42 to 43"//, which pinned the
+literal 42. Injecting a fabricated sentence into one coding of
+''2012/IEEE-SP/third-party-web-tracking-policy-and-technology'' against the patched
+script raises NOT FOUND **from 7 to 8**, drops ''exact'' from 489 to 488, and prints
+the fabricated sentence in full in the unlocated list. The pin in §18.11 has been
+updated to 7 → 8 and the old value recorded beside it, because a guard that still
+asserts a superseded number is a guard that will pass for the wrong reason.
+**The fallback was verified to be doing work, not failing silently.** ''pdfText()''
+swallows a subprocess failure and returns ''null'', which would produce ''0 rescued''
+and look like a clean result. The run therefore prints the number of papers for which
+no ''pypdf'' rendering could be produced — **0** — and two rescues were confirmed by
+hand against ''scripts/pdftext.py'' output directly: ''2019/IMC/visiblev8-…''
+(//"to visit the Alexa top 50k web domains"//, 0% of shingles in the stored text,
+verbatim in the PDF) and ''2021/WWW/towards-realistic-and-reproducibleweb-crawl-measurements''
+(//"Tranco list of top sites [18] (snapshot 77PX)"//, likewise). A ''0 rescued'' from
+this script would now be a measured zero.
+**Scope of this edit.** §18.5's mode table and the four sentences around it, the
+§18.11 mutation-test pin, and the two ''%%<file>%%'' blocks — the published script and
+its published output — both regenerated from the committed files rather than edited.
+''report_pins_audit.mjs'', ''pins_agreement.mjs'' and ''report_longitudinal.mjs'' were
+**not** re-run: no coding, no population and no agreement figure is touched by this
+change, and the quote check feeds none of them. Nothing on
+[[:design:longitudinal]] carries a quote-check figure, so no content page was saved.
+No ''%%{[citekey]}%%'' was added or moved.
+^ Item ^ Value ^
+| Date | 2026-09-22, unsupervised |
+| Command | ''node scripts/pins_quotecheck.mjs'' |
+| Artifacts | ''out/qc0922/pins_quotecheck-output.txt'' (the run published above), ''out/qc0922/pins_quotecheck_mutation.txt'' (the mutation run, NOT FOUND 8) |
+| Script change | ''scripts/pins_quotecheck.mjs'': imports ''pdfText'' from ''scripts/pdftext_cache.mjs''; a quote in none of the four stored texts is retried in all four modes against the ''pypdf'' rendering and reported as ''RESCUED''; the shingle test was factored into one ''shingleFrac()'' used by both passes; the summary gained a RESCUED line and a count of papers with no ''pypdf'' rendering |
+| Source fragment | ''prov/pins_section.txt'' carries the same §18.5 and §18.11 prose changes, so ''build_provenance_pins.py'' regenerates this section rather than reverting it. This appended refresh section lives on the page only. ''pages/provenance_design_longitudinal.LIVE.txt'' is a 2026-09-07 snapshot and was **not** refreshed — the builder's ''%%--live%%'' input is stale for reasons that predate this edit |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Reviewer finding accepted | The pass caught that this section's **heading** still read //"the 43 that did not locate"// — a count three generations stale (43 before the 2026-09-07 correction, 42 before today's fallback, 7 now). The prose below it narrates all three, so no reader was misled, but a heading is also an anchor and a TOC entry. It has been renamed to carry **no count at all**, because a heading that names a figure is a figure nobody re-runs. No page links to the old anchor — checked across all 188 pages before renaming |
+| Pages saved | this page only |