Differences

This shows you the differences between two versions of the page.

--- provenance:design:longitudinal [2026/09/04 17:29] – Citekey consolidation 2026-09-04: repoint duplicate bibliography keys to the kept key and add a dated amendment; no prose or figure changed. Authored by Claude karel.kubicek.claude
+++ provenance:design:longitudinal [2026/09/22 04:33] (current) – Figures-vs-script finding accepted: the 18.5 heading still named a three-generations-stale count (43); renamed to carry no count. No inbound anchor links across 188 pages. Authored by Claude karel.kubicek.claude
@@ Line 59: / Line 59: @@
 </code>
-Both default to ''/workspace/publications_dataset/data/extract/run1'' and ''.../data/fulltext''; ''--run'' and ''--fulltext'' override.
+Both default to ''/workspace/publications_dataset/data/extract/run1'' and ''.../data/fulltext''; ''%%--run%%'' and ''%%--fulltext%%'' override.
 ===== 5. The fold, and its residue in full =====
@@ Line 211: / Line 211: @@
 ===== 9. What could not be established =====
-  * **Whether a paper held a pin //constant// across waves.** The extraction records what a paper stated, once. Every pin figure on the content page is a ceiling and is labelled as one. Closing this needs a hand-coded sample of the 250, reading each paper's per-wave methods description. That is a day of work and would turn every ceiling into an estimate; it is the single most valuable follow-up.
+  * **Whether a paper held a pin //constant// across waves.** **Closed on 2026-09-07** — see **§18** below. All 250 were hand-coded rather than a sample, and the content page now publishes estimates beside the stated-value figures. What is left open by that work is in §18.8, and the largest residue is that ''same-implied'' — one configuration described and never contradicted — cannot be distinguished from a pin that silently moved.
   * **Whether the 2025–2026 slice changes the trend.** The repeat rate is flat at roughly one crawling paper in five across four buckets, and the last bucket is provisional by construction. Nothing in the 2025–2026 material suggests a change, but those venue-years are the thinnest in the corpus.
   * **Whether pinning actually changes a published answer.** The whole case rests on single-wave variation studies ({[demir2022_reproducibility]}, {[demir2023_similarity]}, {[jueckstock2021_realistic]}, {[ahmad2020_apophanies]}). No study repeats one measurement under pinned and unpinned conditions over a year. Listed as an open question on the content page.
@@ Line 1473: / Line 1473: @@
 ^ Item ^ Value ^
 | Content page | [[design:longitudinal]], §"The frame moves even when you pin it" |
-| Saves | rev **1788465991** (''--if-rev 1787798449''), rev **1788467004** (''--if-rev 1788465991''), rev **1788467915** (''--if-rev 1788467004'') |
+| Saves | rev **1788465991** (''%%--if-rev%% 1787798449''), rev **1788467004** (''%%--if-rev%% 1788465991''), rev **1788467915** (''%%--if-rev%% 1788467004'') |
 | Corpus queried | none — this is a vendor date |
 | External source | Tranco's HTTP API and front page, fetched 2026-09-03 |
@@ Line 1512: / Line 1512: @@
   * ''ahmad2026_ipfp'' was one of two keys for the same paper in [[:literature:bibliography]]. The wiki-wide consolidation of 2026-09-04 (drain item ''dedup-regional-filter-lists-bibkey'') kept ''ahmad2026_more'' and deleted the other entry.
   * 1 citation marker on [[:design:longitudinal]] was repointed to the kept key. No prose on either page changed, and no figure moved. Statements above that name the deleted key describe the state when they were written. Full query log and the invariants checked before saving: [[:provenance:literature:bibliography]].
+===== 18. The four-pin hand-coding, 2026-09-07 =====
+§9 above named one follow-up as the most valuable: //"Whether a paper held a pin constant across waves. The extraction records what a paper stated, once. Every pin figure on the content page is a ceiling … Closing this needs a hand-coded sample of the 250, reading each paper's per-wave methods description."// This section is that work. It was not a sample: **all 250 were read.**
+==== 18.1 What was built, and what each piece is for ====
+^ Artefact ^ What it does ^ Committed as ^
+| Population | Re-derives the 250 from the extraction with the predicate copied verbatim out of ''report_longitudinal.mjs'' | ''scripts/pins_population.mjs'' → ''out_pins/population.jsonl'' |
+| Reading packet | Per-paper keyword-window excerpt of the ''.cols'' text, with the full-text path in its header | ''scripts/pins_excerpt.mjs'' → ''out_pins/excerpts/'' |
+| Codebook | ''repeatKind'' + four pins, six values each, a verbatim quote per pin | ''out_pins/CODEBOOK.md'' |
+| Coding | 250 JSON files, one per paper | ''out_pins/coded/'' |
+| Reliability sample | Deterministic draw of 30 from the 250 | ''scripts/pins_valsample.mjs'' → ''out_pins/valsample.tsv'' |
+| Blind second coding | The same 30, re-coded from the **full text** by a coder who never saw the first coding | ''out_pins/coded_val/'' |
+| Aggregation | Every published figure, with its denominator, plus the DokuWiki tables | ''scripts/report_pins_audit.mjs'' |
+| Quote check | Every evidence quote located in the source, in four escalating modes | ''scripts/pins_quotecheck.mjs'' |
+| Agreement | Per-pin exact and collapsed agreement, Cohen's //κ//, every disagreement printed | ''scripts/pins_agreement.mjs'' |
+| Citekey map | Slug → the citekey **already on the content page**, so the exemplar table is emitted, not retyped | ''out_pins/citekeys.json'' |
+| Quote adjudication | Hand verdicts on every quote no matching mode could locate | ''out_pins/qaudit/verdict_1..3.tsv'' |
+==== 18.2 The population, and that it reproduces ====
+<code bash>
+cd /workspace/artifacts/wiki
+node scripts/pins_population.mjs > out_pins/population.jsonl
+node scripts/pins_excerpt.mjs
+node scripts/pins_valsample.mjs > out_pins/valsample.tsv
+node scripts/report_pins_audit.mjs        > scripts/report_pins_audit-output.txt
+node scripts/report_pins_audit.mjs --wiki                       # the page's tables
+node scripts/pins_quotecheck.mjs          > scripts/pins_quotecheck-output.txt
+node scripts/pins_agreement.mjs           > scripts/pins_agreement-output.txt
+</code>
+Both derived files were re-run on 2026-09-07 and diffed against the committed copies: ''population.jsonl'' and ''valsample.tsv'' reproduce byte-identically. ''node scripts/report_longitudinal.mjs'' was also re-run and diffed against ''scripts/report_longitudinal-output.txt'' — **zero lines differ**, so nothing on the rest of the content page moved under this edit.
+The predicate is ''crawled'' AND ''max(temporal[].snapshots) > 1'', copied out of ''report_longitudinal.mjs'' rather than re-implemented, so §3's ''repeated'' population and this one agree by construction rather than by luck.
+''report_pins_audit.mjs'' **refuses to print anything** if fewer than 250 codings are present, rather than silently shrinking its denominator:
+<code>
+if (coded.length !== pop.length) {
+  console.error(`\nFATAL: coded ${coded.length} of ${pop.length}. Figures would be computed over the wrong population.`);
+</code>
+It has the same guard on the citekey map: if the map and the all-four-stated set disagree by one slug, the exemplar table would silently lose a row, so it exits instead.
+==== 18.3 The reading packet, and what it costs ====
+papers is 20.9 MB of ''.cols'' text. Each coder was given a packet instead: the first 3,000 characters, plus context windows around two tiers of keyword matches (wave/repetition language, "same/unchanged/pinned", statefulness, the named ranking lists and filter lists, browsers, cloud regions), merged and capped at 32,000 characters — **and the full-text path in the packet header, with an instruction to open it whenever the packet is not enough.** The packets instruct the coder to open the full text whenever the windows are not enough. **Whether they did is not logged**, so the recall of the packet rule is bounded only by the blind full-text pass below, not by any record of what each coder read.
+The packet is a recall risk, and it was measured rather than assumed — that is what the blind full-text second coder in §18.4 is for. The answer, in short: the packet does not lose "fixed" evidence, it loses the //absence// of it. See the direction-of-disagreement table below.
+==== 18.4 The blind second coder, and the agreement ====
+of the 250 were drawn with a seeded shuffle (''mulberry32'', seed ''20260907'') and re-coded from the **full ''.cols'' text** by a second coder given ''CODEBOOK_VALIDATION.md'', told explicitly not to open ''out_pins/coded/'' or ''out_pins/excerpts/''. Ten agents, three papers each. No coder saw another's output.
+^ Measure ^ Exact ^ //κ// ^ Collapsed to fixed / not fixed ^ //κ// ^
+| Population list | 20/30 (66.7%) | 0.49 | 26/30 (86.7%) | 0.59 |
+| Browser | 21/30 (70.0%) | 0.57 | 22/30 (73.3%) | 0.48 |
+| Vantage point | 21/30 (70.0%) | 0.56 | 25/30 (83.3%) | 0.67 |
+| Statefulness | 24/30 (80.0%) | 0.64 | 25/30 (83.3%) | 0.64 |
+| **All 120 pin codings** | **86/120 (71.7%)** | **0.60** | **98/120 (81.7%)** | **0.64** |
+| ''repeatKind'' | 25/30 (83.3%) | 0.77 | own repeat crawl vs everything else: 27/30 (90.0%) | 0.80 |
+| "all four fixed" | 27/30 (90.0%) | 0.52 | A: 3/30, B: 4/30 | — |
+An earlier draft of this section said //κ// on "all four fixed" is low only because the class is rare, //"not because the coders disagree about which papers they are"//. **That was wrong and the files say the opposite.** Five papers are called all-four by at least one coder; the two agree on **two** of the five, and disagree on three. The class is indeed rare at //n// = 30, but the disagreement is real, and the 15-of-124 headline should be read with that in mind.
+**Direction of disagreement.** This is the part that changes how the content page is written:
+^ Pin ^ Both call it fixed ^ Only the excerpt coder ^ Only the full-text coder ^ Neither ^
+| Population list | 22 | 4 | 0 | 4 |
+| Browser | 9 | **7** | 1 | 13 |
+| Vantage point | 11 | 1 | **4** | 14 |
+| Statefulness | 8 | 4 | 1 | 17 |
+The excerpt coder calls the **browser** fixed seven times where the full-text coder finds it never stated, and only once the other way; statefulness leans the same way, four to one. The **vantage point** goes the opposite way — the full-text coder finds four vantage statements the packet's windows missed, because vantage is often mentioned once, far from any wave language, in an infrastructure or ethics paragraph.
+The most likely cause of the browser lean is a design mistake that is worth recording: **the packet header prints the structured extraction's own values** (''browsers=["Selenium"]'', ''statefulness=not-stated'', …) so the coder can see what a machine thought. The codebook's first rule tells the coder not to infer from it. The agreement data suggests that instruction was not fully obeyed. A rerun should put that line at the end of the packet, or drop it.
+Consequence for the content page: **the browser row of the estimate column is more likely too high than too low**, and the page says so.
+Every disagreement is printed paper by paper in the unedited output in §18.12.
+==== 18.5 The quote check, and what "unlocated" actually means ====
+Every non-empty ''quote'' must appear in the paper it is attributed to. 1,000 pin codings, 464 with no quote (all of them ''not-stated'', which the codebook permits and the script verifies: **0 codings carry a value other than ''not-stated'' with no quote**), 536 checked.
+The naive substring test is wrong here, and getting it wrong once already produced a scare. Run against ''paper.cols.txt'' alone it reported **49 unlocated quotes (9.1%)**, which reads as a fabrication rate. It is not. The first one inspected was:
+  * coder's quote: //"We implemented FourthParty as an extension to Mozilla Firefox."//
+  * what ''.cols'' actually holds: ''%%We implemented FourthParty as an extension to Mozilla \n B. Design Principles Firefox. It currently instruments%%''
+The two-column repair still splices a quarter of pages, and a sentence crossing a column boundary comes back with the neighbouring column injected into the middle of it. **A coder who de-splices that correctly writes a quote that is a substring of nothing.** Failing it would call a correct reading a hallucination. So the check now searches ''paper.cols.txt'', ''paper.norm.txt'', ''paper.txt'' and the packet, in four escalating modes, the last of which cuts the quote into 6-word shingles and passes at 80% found. **And since 2026-09-22 a fifth text:** a quote located in none of the stored renderings is re-checked, in the same four modes, against a ''pypdf'' rendering of the same ''paper.pdf'' (''scripts/pdftext.py''). That verdict is reported as **RESCUED** — the quote is in the paper and the stored text is what is wrong. It is the fallback ''scripts/quote_check.mjs'' has had since 2026-09-04; this checker did not have it, and until it did, its unlocated bucket was not comparable with the rest of the site's.
+^ Mode ^ Codings ^ Share of 536 ^
+| Exact substring | 489 | 91.2% |
+| Whitespace-collapsed | 0 | 0.0% |
+| Loose (de-hyphenated, quotes and dashes folded, case-insensitive) | 2 | 0.4% |
+| De-spliced (6-word shingles, ≥80%) | 3 | 0.6% |
+| Rescued from the ''pypdf'' rendering of ''paper.pdf'' | 35 | 6.5% |
+| **Located by no mode, in any of the five texts** | **7** | **1.3%** |
+//The counts above are after the one correction described below **and** after the ''pypdf'' fallback added on 2026-09-22. Before the correction and before the fallback: 488 exact, 43 unlocated (8.0%). With the correction and still without the fallback — which is what this table published until 2026-09-22 — 489 exact, 42 unlocated (7.8%). The 42 did not become 7 because any coding changed; 35 of them are the same quotes, mechanically located in a second rendering of the same PDF.//
+**All 43 were then read by hand** — three sub-agents, each given the paper's three renderings and the PDF, asked to classify each quote as a faithful de-splice, a supported paraphrase, an unsupported paraphrase, or a quote lifted from the wrong part of the paper, and //independently// to say whether the paper supports the coded pin value at all. **The 2026-09-22 fallback is an independent, mechanical check on that hand audit, and it agrees.** 35 of the 42 the hand audit called faithful de-splices are now located verbatim or near-verbatim in ''pypdf'''s rendering, by a script with no access to the hand verdicts. The 7 that remain unlocated are all inside the hand-audited set and were all given the same verdict — faithful de-splice — so the fallback has not contradicted a single one of the 43.
+^ Verdict ^ Codings ^
+| Faithful de-splice of a real sentence | 42 |
+| Quote is real but from the wrong part of the paper | **1** |
+| Unsupported paraphrase / fabrication | **0** |
+The one bad coding, and what was done about it:
+  * **{[singanamalla2022_respect]}, ''vantage'' = ''changed''.** The evidence quote — //"the ORIGIN Frame implementation was deployed globally at over 275 points of presence"// — is a real sentence, correctly de-spliced, and describes the **CDN's own** deployment footprint, not where the authors' crawler ran. The paper's methods section says the opposite: //"We used the same machines throughout our evaluations for consistency, and saw no observable change in datacenter provisioning or capacity over time."// The blind second coder had independently coded that pin ''same-implied'', which is the same direction.
+  * **Corrected to ''same-explicit''** with the methods-section sentence as its quote, and the correction is recorded in the paper's ''notes'' field. The same pass offered a stronger quote for that paper's ''state'' pin (//"Client caches are intentionally deleted between measurements in all of our active measurements"//); the quote was swapped, the value was already ''same-explicit'' and did not move.
+  * **What moved on the content page:** the vantage row over the 124 went from 25/44/10 to **26/45/9** (words / estimate / moved), and the all-pins fixed-versus-not agreement from 80.8% to 81.7%. The all-four figure, the 124, and the 6-of-23 survivor count did not move.
+The verdict files are ''out_pins/qaudit/verdict_1.tsv'' … ''verdict_3.tsv''. **The lesson is the check, not the count.** A plain substring test on ''paper.cols.txt'' alone would have reported a 9.1% fabrication rate here, of which the real rate is **0.2% (1 of 536)** — and the one real defect was not a fabrication at all but a quote from the wrong section, which no substring test can catch. The 2026-09-22 fallback moves the //automated// bucket from 7.8% to **1.3%**, and that 1.3% is still not a fabrication rate: the real rate is the hand-audited 0.2%, unchanged, and the seven remaining are the hand audit's own faithful de-splices. An automated unlocated count is an upper bound on a fabrication rate and has never been an estimate of one. One reviewer flagged a further boundary case worth recording rather than changing: two codings use ''same-explicit'' on the strength of a pinned identifier in papers whose ''repeatKind'' is ''single-crawl'', where the codebook's rule 4 would suggest ''same-implied''. Both readings are inside the codebook as written; the rule needs tightening before a rerun, and neither paper is in the 124.
+==== 18.6 What changed on the content page ====
+^ Where ^ Before ^ After ^
+| Lead paragraph | "Of those 250, **23 (9.2%)** state all four" | 124 are really repeat crawls; **15 of 124 (12.1%)** plausibly held all four fixed, **2** say so in words |
+| ''The four pins, measured'' | title, and //"every figure here is an upper bound on comparability"// | retitled ''The four pins: how often they are stated''; the upper-bound sentence now points forward instead of standing as the last word. **No figure in that table changed** — it is the reporting-rate table and it is still correct as such |
+| — | — | **new section** ''Were the pins actually held fixed?'': the ''repeatKind'' table, the four-pin table over the 124, the divergence between "stated" and "held fixed", the span and period splits, and the agreement figures |
+| ''Papers that pin all four'' | 23 papers, //"These are the methods sections to copy from"// | retitled ''Papers that stated all four, and what they actually did''; same 23 rows, each with its hand-coded verdict, and **6 survive** |
+| ''Use in Publications'' intro | the 250 rule, no caveat | adds that the rule is right about half the time and names the 124 |
+| Methodology bullets | one bullet: treat the pin figures as ceilings | three bullets: what the two columns each measure, the coder-agreement caveat, and that every other share-of-250 figure on the page inherits the false-positive rate and has **not** been recomputed |
+**Also corrected in the same edit, outside the four-pin work.** Four external claims the currency reviewer found stale — Quantcast's exit date, the OpenWPM version, the Playwright ''browsers.json'' source, and ICLab's maintenance status. Each is described in §18.10 Pass C with what was re-fetched.
+**Not changed, on purpose.** The cadence, span, probe and Tranco-ID figures are still shares of 250 rather than of 124. Recomputing them is a different piece of work with its own review obligation; folding it into this edit would have meant republishing a dozen tables under a population change that nothing in this hand-coding validates for them. The content page says plainly that they were not recomputed, which is the honest state, and it is filed below as the follow-up.
+==== 18.7 Judgement calls ====
+  * **Two populations are published, and the 124 is the one the prose leans on.** Publishing only the 250 would keep the page consistent with itself and answer the wrong question; publishing only the 124 would break every cross-reference to [[statistics:biases]]. Both tables are emitted by the script; the content page carries the 124 and points at the 250 version here.
+  * **''same-implied'' counts toward the estimate.** It is the common case — one methods paragraph, no statement that anything changed — and excluding it would give a "floor" of 2 papers in 250, which is a measurement of how rarely people write //"unchanged"//, not of how often things were unchanged. It is published as a separate column so a reader can take the strict reading, and the page calls the estimate generous.
+  * **A list re-downloaded each wave is ''changed'', not ''same''.** Scientifically that is often the right thing to do; the codebook measures whether the frame is constant, not whether the authors were wise. 28 of the 124 are coded ''changed'' on the list pin, and several are deliberate and defensible.
+  * **The ceiling is not a ceiling for the list pin.** The hand-coded estimate for the population list (95 of 124) is //above// the structured figure (86). They measure different things: the extraction looks for ''population.listVersion'', a version string; the hand-coding asks whether the frame was constant, which "the same 200 sites throughout" satisfies without any version string. 62 of the 250 are in that gap. Presenting the structured column as a ceiling without saying this would be wrong, so the content page's column is now labelled "stated at all" and the divergence has its own paragraph.
+  * **The 23-paper exemplar table was kept, not deleted.** It is the clearest possible demonstration of what "stated" was hiding, and deleting it would erase the correction along with the error.
+  * **''repeatKind'' was added to the codebook** after a pilot batch showed the mechanical predicate misfiring. Without it the four-pin figures would be computed over a population half of which cannot have waves. This turned out to be the largest single finding.
+==== 18.8 What still cannot be established ====
+  * **Whether a pin was //actually// held fixed, as opposed to described once and never contradicted.** ''same-implied'' is 47 of the 124 on the list pin and 55 on the browser. Only the authors know. Nothing short of asking them closes this, and the page publishes the strict and generous readings side by side rather than picking one.
+  * **Whether the reliability figures hold at n=250.** They were measured on 30. //κ// = 0.60 on exact values is moderate; the confidence interval at n=30 is wide and is not published because it would imply a precision the design does not have.
+  * **The other share-of-250 figures on the content page.** Cadence, span bands, the probe table and the Tranco-ID rate are all still computed over the 250. The 124 exists now, so they //could// be recomputed; they have not been, and the page says so.
+  * **Whether the pin figures differ by venue.** n per venue-year is small and the 124 splits into groups of two and three. Not attempted.
+  * **Whether a paper that pinned got a different answer than one that did not.** Unchanged from §9: this remains the open question the whole page rests on and no study in the corpus answers it.
+==== 18.9 Run log ====
+^ Item ^ Value ^
+| Date | 2026-09-07 |
+| Corpus | ''data/extract/run1'', 5,859 papers, 7 venues, 2010–2026. Unchanged since the page was written on 2026-08-27. |
+| Coder A | Claude Sonnet, 63 batches of ≤5 papers, excerpt packet plus full-text access, one paper coded once |
+| Coder B | Claude Sonnet, 10 batches of 3, full text only, blind |
+| Quote adjudication | Claude Sonnet, 3 batches, over the 43 quotes no matching mode located; 1 correction applied |
+| Review | Three focused passes (Claude Sonnet: figures-vs-script, citations-and-quotes, external currency) and one generic pass (Claude Fable), all handed the page text, the scripts, their unedited output and these notes; findings and rejections in §18.10 and §18.11 |
+| Orchestration, scripts, page text | Claude Opus 5 |
+| Interruption | The run hit an API quota limit with 185 of 250 coded and resumed the same day. Same model, same codebook, same prompt; the resumed batches are ''out_pins/rebatch/''. Nothing had been published at the point of interruption. |
+| Bibliography | **No entries added and none changed.** The exemplar table reuses the 23 citekeys already on the content page, via ''out_pins/citekeys.json''; the builder exits if that map and the population disagree. |
+| Guards run before saving | ''node scripts/check_wrap.mjs'' on both pages; heading, table and %%<file>%%-block counts compared before and after by ''scripts/build_provenance_pins.py'', which exits non-zero if any of the three falls |
+| Discussion block | Still none on this page, following the convention in §16. |
+==== 18.10 Review passes, 2026-09-07 ====
+Four reviewers, each told explicitly that the author's context may not be exhaustive, each handed both draft pages, all six scripts, their unedited output and these notes. **Every finding was accepted except one, marked "noted, not changed" in Pass B.** Two of the four independently found the same defect, which is the only evidence there is that the layer is worth its cost — and the generic pass, which nearly did not run at all, produced the most serious finding of the four (§18.11).
+=== Pass A — figures against the script (Claude Sonnet) ===
+Re-ran all six scripts and confirmed every committed ''-output.txt'', ''population.jsonl'' and ''valsample.tsv'' reproduces byte-identically, and that ''pins_population.mjs'' lines 17 and 19 are character-for-character the predicate in ''report_longitudinal.mjs''.
+  - **Stale table cell.** The 23-row exemplar table's vantage cell for {[singanamalla2022_respect]} still read ''**changed**''. The table had been pasted from ''%%--wiki%%'' output generated //before// the §18.5 correction was applied. **Accepted and fixed** by regenerating the table; it was the only divergent cell of 207.
+  - **Wrong hand-written count.** The prose said //"nine turn out not to be repeated crawls at all"//. Recounting from the coding files: of the 23, eight are ''parallel-not-temporal'', one of which is a survivor, so among the 17 non-survivors **seven** are not repeat crawls and **ten** are. The script never prints this aggregate, which is exactly why it was wrong. **Accepted and fixed.**
+  - **The structural guard in ''build_provenance_pins.py'' asserted much less than it claimed.** It compared heading, table and file-block counts over the //whole// output against the whole live page. Because the appended section adds eleven headings of its own, the check had eleven headings of slack: the reviewer mutation-tested it by deleting a heading from the **preserved** live content and got ''headings: 27 -> 37'' and **exit 0**. **Accepted and fixed** — the counts are now compared on the preserved text alone, which must equal the live page byte for byte, and the appended section is separately required to be non-empty. This is a page-matches-its-own-script failure of the kind the guard existed to prevent, and reading the script would not have found it; mutating it did.
+  - Mutation-tested and **confirmed sound**: the partial-population refusal in ''report_pins_audit.mjs'' (deleting one of 250 codings gives ''FATAL: coded 249 of 250''), the citekey-map guard (removing one slug exits 1), ''pins_quotecheck.mjs'' (an injected fabricated quote raises NOT FOUND from 7 to 8 and is printed in full; re-run on 2026-09-22 against the patched script — the pin was 42 → 43 before the ''pypdf'' fallback), ''pins_agreement.mjs'' (a removed coder-B file drops //n// to 29 and prints ''MISSING'' rather than silently continuing at 30), and the missing-file and empty-file guards in the builder.
+  - Hand-computed Cohen's //κ// for the list pin from the raw pairs: observed 0.667, expected 0.351, //κ// = 0.486 → 0.49, matching the script. No bug.
+  - Checked the figures **outside** the edited window against ''report_longitudinal-output.txt'' and found none stale.
+=== Pass B — citations and quotes (Claude Sonnet) ===
+  - All 32 keys on the content page and all 8 on this page resolve against the **live** ''literature:bibliography'' (850 entries, fetched today, not a local copy). The content page's key set is unchanged from the live version; this page gains exactly one key, {[singanamalla2022_respect]}, which is already in the bibliography and already cited on the content page. No entry added, none changed.
+  - Verified all 23 rows of ''out_pins/citekeys.json'' against the corpus index — title, venue, year and first-author surname — including that ''fouad2020_missed'' and ''fouad2024_devil'' are two distinct papers and are not conflated. No swapped keys.
+  - **Independently found the same stale vantage cell as Pass A**, by diffing the published table against a fresh ''%%--wiki%%'' run, and confirmed all three quotes in the §18.5 account by locating them in the paper itself.
+  - Spot-checked **24 evidence quotes** (10 ''changed'', 14 ''same-explicit'') against all three renderings: **24 of 24** located and supporting their coded value. Flagged one as loose rather than wrong — the Zoom-apps ''browser'' quote is about a crawler and parser rebuild rather than a browser version. **Noted, not changed**: the coding is ''changed'' either way and the codebook's browser pin covers the driven stack.
+  - Verified the "12 further papers outside this 23" arithmetic (18 all-four minus the 6 in the table) and every external figure quoted from a cited paper.
+=== Pass C — external currency (Claude Sonnet) ===
+This pass reviewed the **whole** of both pages, not the diff, and found four stale external claims — all of them in text this edit did not touch. All four accepted:
+  - **Quantcast's exit date was imprecise.** The page said //"dropped out after 1 April 2020"//, which was only as precise as the rows sampled in the table above it. Bisecting the Tranco API on 2026-09-07 puts it between **14 and 15 August 2020** (''date/20200814'' → ''GK9K'', providers include ''quantcast''; ''date/20200815'' → ''Z7NG'', providers do not). Re-verified independently before the edit. Every other row and date in that table and its footnote was checked against the live API and is exact.
+  - **OpenWPM had moved on.** The page cited v0.36.0 / Firefox 154.0 as its worked example; **v0.37.0 shipped on 6 September 2026** pinning **Firefox 155.0.1**. Confirmed via ''releases/latest'' and by reading ''install-firefox.sh'' (''# FIREFOX_155_0_1_RELEASE''). Updated, with the drift itself now noted in the footnote, since it is the paragraph's own point.
+  - **The Playwright footnote quoted ''main'', not a release.** It cited ''browsers.json'' from the development branch — ''revision 1241'', ''browserVersion 152.0.7977.54'' — as evidence that pinning Playwright pins a browser. That snapshot shipped in no release, so it was the wrong evidence for the claim. Re-fetched from the tagged release ''v1.63.0'': ''revision 1243'', ''browserVersion 153.0.8010.12''. The reviewer reported different values for that tag than the author's own fetch did; a third fetch during the generic pass returned ''revision 1243'', ''browserVersion 153.0.8010.12'', matching the author's, so the published numbers stand.
+  - **ICLab is dormant.** The page called Censored Planet and ICLab //"continuous, longitudinal and public"// in one breath. Censored Planet is active. ICLab's site serves an **expired TLS certificate** as of 2026-09-07 (''curl'' fails with exit 60 without ''-k''), its data page returns **404**, and its news stops in 2019 — all re-verified before the edit. The sentence now recommends the paper and warns off the feed.
+  - Confirmed accurate and unchanged: the HTTP Archive changelog 404 (still 12 entries, last 2017-06-01, on ''master'' after the repo rename), the HTTP Archive cadence text, all seven Tranco list ids, the EasyList commit cadence, the Public Suffix List footnote, the Chrome for Testing endpoints, and the WebREC repository facts. Neither page names an RFC or W3C specification, so that category is empty.
+=== Pass D — generic (Claude Fable) ===
+Run last, on the published pages, after A–C were applied. Findings and their disposition are in §18.11.
+==== 18.11 Generic review pass ====
+**Commissioned, nearly missed, and completed late.** A Claude Fable pass was given both published pages, the scripts and their output, and asked for whatever the three focused passes were not looking for — overstated claims, structure, a section that does not answer its own question, and specifically whether ''same-implied'' is doing too much work, whether the split between the 250 and the 124 is legible, whether //n// = 30 and //κ// = 0.60 are being asked to carry more than they can, and whether the page states plainly enough that both coders and the adjudicator were LLMs reading LLM-extracted data. The run's budget all but ran out before it returned, and this section was published saying it never would; it arrived afterwards and its findings are below.
+This is worth stating rather than quietly leaving a gap, because the generic pass is the one that catches overstatement, and §18 is the section most at risk of it. The specific things a reader should discount accordingly:
+  * The **estimate** column rests on ''same-implied'', which is a judgement that silence about change means no change. It is 47 of 124 on the list pin and 55 of 124 on the browser. The strict reading is published beside it and is much smaller.
+  * The reliability figures are **//n// = 30**, and the //κ// for "all four fixed" (0.52) is computed on three and four positives respectively.
+  * **Every coder in this exercise was a language model**, reading text extracted from PDFs by a pipeline, coding a schema produced by another language model. The blind second coder and the quote adjudication bound the error; they do not remove it, and no human read any of the 250 papers.
+Re-running this pass was filed as follow-up work — and then it returned.
+**Findings, and what was done.** The pass returned after the first save of this section. Applied to the content page: the disclosure that **both coders and the adjudicator were language-model agents** reading LLM-extracted text, which the content page had nowhere stated and which "hand-coded" actively obscures; the softening of //"two say so in words"//, after the reviewer read the two ''same-explicit'' quote sets and found them to be permanent identifiers and one-off constructions rather than statements that anything was unchanged; **Fisher exact //p//-values** on the span reversal (0.10, 0.08 and 0.69 across three cuts, none significant) in place of //"yes, and more sharply"//; the floor-versus-false-positive contradiction in the lead; and a flag that one of the 15 all-four papers is, by its own coder's note, a cellular walk test rather than a web crawl. Applied here: the corrected //κ// defence in §18.4, the deletion of an unverifiable claim about coders opening the full text, and the resolution of the Playwright value discrepancy.
+**Accepted and not done, for budget:** re-adjudicating the 73 ''same-explicit'' codings against the codebook's own wording; re-coding the 5G paper and recomputing 15/124 as 14/123; reading the 12 all-four papers outside the exemplar table and printing their notes; adjudicating the seven browser disagreements, which the reviewer argues may be the full-text coder's misses rather than the excerpt coder's over-calls, making §18.4's directional claim unsafe; publishing the //κ// confidence interval; recomputing the repeated-versus-single-shot comparison over the 124; per-table "of 250, unaudited" markers; and the ''waveCount''-stratified middle reading of ''same-implied''. Each is recorded as follow-up rather than silently dropped. **Until they are done, §18.4's "the browser row is more likely too high than too low" and the content page's browser sentence rest on an untested attribution of the disagreement to one coder.**
+==== 18.12 The scripts, and their unedited output ====
+Everything below is read from the committed file by ''scripts/build_provenance_pins.py'' at build time. None of it is retyped.
+<file  CODEBOOK.md>
+# Four-pin hand-coding codebook — design:longitudinal
+You are hand-coding papers for a systematization page on measuretheweb.org. The
+page currently publishes, for 250 repeated-crawl papers, how many **stated** each
+of four crawl parameters. "Stated once" is only a CEILING on "held the same in
+every wave". Your job is to turn the ceiling into an estimate by reading what the
+paper actually says about its repeated measurement.
+## Step 0 — is this really a repeated crawl by the authors?
+The 250 were selected mechanically (`temporal.snapshots > 1` in a structured
+extraction). That predicate has false positives. Code `repeatKind` as exactly one:
+| value | meaning |
+|---|---|
+| `own-repeat-crawl` | the authors themselves ran their crawler more than once over time and the waves are compared or aggregated |
+| `own-repeat-nonweb` | repeated measurement, but not a web crawl the four pins apply to (e.g. repeated DNS probing, repeated app scraping, repeated API pulls, network scans) |
+| `parallel-not-temporal` | the "snapshots" are parallel conditions run at one time (browsers, vantage points, profiles), not waves separated in time |
+| `reused-waves` | the multiple time points come from a dataset or archive somebody else collected (web archive, existing corpus), not from the authors' own repeated crawling |
+| `single-crawl` | the paper ran one crawl; the snapshot count is a mis-extraction (page counts, record counts, number of sites, number of runs of an unrelated thing) |
+| `unclear` | you genuinely cannot tell after reading |
+Also record `waveCount`: the number of distinct time-separated waves of the
+authors' OWN crawl, as an integer, or `null` if not determinable. A continuous
+daily/weekly crawl over N periods counts as N waves if the paper says so;
+otherwise `null`.
+## Step 1 — the four pins
+Code each of the four independently. **The question is not "did they say it" but
+"was it the same in every wave".**
+* `list` — the **population / seed list**: which sites or targets were crawled
+  (Tranco, Alexa, CrUX, Majestic, a custom seed set …). NOT the filter list.
+* `browser` — the browser (and version, if given) the crawler drove.
+* `vantage` — the network location the crawl ran from (country, cloud region,
+  university network, VPN exit, residential proxy …).
+* `state` — stateful vs stateless crawling: fresh profile per site, persistent
+  cookie jar, shared profile, cleared between waves, etc.
+Each pin takes exactly one value:
+| value | when to use |
+|---|---|
+| `same-explicit` | the paper says in words that this was the same / unchanged / the same identifier across the waves. A single permanent identifier used for all waves (one Tranco list ID, one pinned browser version, one container image) counts. |
+| `same-implied` | the paper gives ONE value in a single methods description that plainly covers the whole repeated measurement, and never says it changed. This is the common case. It is weaker evidence than `same-explicit` — do not upgrade it. |
+| `varied-deliberate` | the value differs across waves ON PURPOSE, as part of the design (rotating vantage points, comparing browsers across waves). |
+| `changed` | the value changed or was refreshed between waves — the top list re-downloaded per wave, the filter list or browser updated, a move to a different host — whether stated as deliberate or as drift. |
+| `not-stated` | the paper never gives the value at all. |
+| `unclear` | the value is given somewhere but you cannot tell which wave(s) it applies to, or the description is contradictory. |
+Rules that matter:
+. **Do not infer from the structured extraction line at the top of the packet.**
+   It is shown so you can see what a machine thought; it records a value stated
+   *once*. If the paper's text does not support it, code what the text says.
+. **A list that is re-downloaded each wave is `changed`, not `same-implied`** —
+   even though that is often the right thing to do scientifically. We are
+   measuring whether the frame is constant, not whether the authors were wise.
+. **"We used Chrome" with no version, across a two-year study, is
+   `same-implied` for the browser pin** only if the paper describes one crawler
+   setup for the whole study. If the paper says the crawler was rebuilt, updated,
+   or migrated, it is `changed`.
+. If `repeatKind` is `single-crawl` or `parallel-not-temporal`, still code the
+   four pins, but the "across waves" question is vacuous — use `same-implied`
+   only if one configuration is described, and note it in `notes`.
+. `not-stated` is a finding, not a failure. Do not guess.
+## Step 2 — evidence
+For each pin, give `quote`: a **verbatim** substring of the packet, ≤ 220
+characters, that is your evidence. Copy it exactly — it will be checked against
+the source file automatically, and a paraphrase will fail. If your value is
+`not-stated`, set `quote` to `""`.
+Set `confidence` per paper: `high` | `medium` | `low`.
+## Output
+For EACH paper write one file `out_pins/coded/<KEY_WITH_SLASHES_AS___>.json`
+(same base name as its packet, `.json` instead of `.txt`), containing:
+```json
+{
+  "key": "2024/PETS/some-slug",
+  "repeatKind": "own-repeat-crawl",
+  "waveCount": 3,
+  "pins": {
+    "list":    {"value": "same-explicit",  "quote": "..."},
+    "browser": {"value": "same-implied",   "quote": "..."},
+    "vantage": {"value": "not-stated",     "quote": ""},
+    "state":   {"value": "changed",        "quote": "..."}
+  },
+  "confidence": "high",
+  "notes": "one or two sentences: what the paper actually did across waves"
+}
+```
+Write valid JSON. No markdown fences in the file.
+</file>
+<file  CODEBOOK_VALIDATION.md>
+# Blind re-coding — second coder
+You are the SECOND coder in a two-coder reliability check. A first coder has
+already coded these papers; you will not see their answers, and you must not go
+looking for them. Files under `/workspace/artifacts/wiki/out_pins/coded/` are the
+first coder's output — **do not open that directory**.
+Read `/workspace/artifacts/wiki/out_pins/CODEBOOK.md` and apply it exactly.
+Two differences from the first pass:
+. **You read the FULL paper text**, not a keyword excerpt. The path is given to
+   you. Read the whole methodology and any limitations section.
+. **Write your output to `/workspace/artifacts/wiki/out_pins/coded_val/`**, same
+   file naming (`<year>__<venue>__<slug>.json`), same JSON schema.
+Everything else — the `repeatKind` values, the six pin values, the verbatim
+`quote` requirement, `confidence`, `notes` — is unchanged. Code what the paper
+says, not what you think it probably did.
+</file>
+<file javascript pins_population.mjs>
+#!/usr/bin/env node
+// Emit the 250 repeated-crawl papers that design:longitudinal's four-pin table
+// is computed over, with the extraction's own value for each pin, as JSONL.
+//
+//   node scripts/pins_population.mjs [--run <dir>] > out_pins/population.jsonl
+//
+// The population predicate is copied verbatim from scripts/report_longitudinal.mjs
+// (`crawled` AND max stated temporal.snapshots > 1) so the two agree by construction.
+import fs from 'node:fs';
+import path from 'node:path';
+const argv = process.argv.slice(2);
+const RUN = argv.includes('--run') ? argv[argv.indexOf('--run') + 1] : '/workspace/publications_dataset/data/extract/run1';
+const FULLTEXT = '/workspace/publications_dataset/data/fulltext';
+const P = fs.readFileSync(path.join(RUN, 'extractions.jsonl'), 'utf8').split('\n').filter(Boolean).map((s) => JSON.parse(s));
+const crawled = P.filter((p) => p.crawlConfig != null || p.studyTypes.includes('automated-web-crawl'));
+const snapOf = (p) => Math.max(0, ...p.temporal.map((t) => (t.snapshots == null ? 0 : t.snapshots)));
+const repeated = crawled.filter((p) => snapOf(p) > 1);
+const SENT = /^(?:not[- ]stated|unknown|n\/a)$/i;
+const statedListVersion = (p) => p.population.some((x) => x.listVersion != null && !SENT.test(x.listVersion.trim()));
+const namedBrowser = (p) => p.crawlConfig != null && p.crawlConfig.browsers.length > 0;
+const statedVantage = (p) => p.vantage.some((v) => v.locations.some((l) => !SENT.test(String(l).trim())));
+const ccStated = (p, f) => p.crawlConfig != null && !['not-stated', 'not-applicable'].includes(p.crawlConfig[f]);
+for (const p of repeated.sort((a, b) => a.year - b.year || a.venue.localeCompare(b.venue) || a.slug.localeCompare(b.slug))) {
+  const key = `${p.year}/${p.venue}/${p.slug}`;
+  const f = path.join(FULLTEXT, String(p.year), p.venue, p.slug, 'paper.cols.txt');
+  process.stdout.write(JSON.stringify({
+    key, year: p.year, venue: p.venue, slug: p.slug, title: p.title,
+    snapshots: snapOf(p),
+    cadence: p.temporal.map((t) => t.cadence).filter(Boolean),
+    spans: p.temporal.map((t) => [t.spanStart, t.spanEnd]).filter(([a, b]) => a || b),
+    modes: [...new Set(p.temporal.map((t) => t.mode))],
+    x_listVersion: statedListVersion(p),
+    x_browser: namedBrowser(p),
+    x_vantage: statedVantage(p),
+    x_statefulness: ccStated(p, 'statefulness'),
+    v_listVersion: [...new Set(p.population.map((x) => x.listVersion).filter((s) => s && !SENT.test(s.trim())))],
+    v_sourceList: [...new Set(p.population.map((x) => x.sourceList).filter(Boolean))],
+    v_browsers: p.crawlConfig ? p.crawlConfig.browsers : [],
+    v_statefulness: p.crawlConfig ? p.crawlConfig.statefulness : null,
+    v_vantage: [...new Set(p.vantage.flatMap((v) => v.locations).filter((l) => l && !SENT.test(String(l).trim())))],
+    textPath: f,
+    textBytes: fs.existsSync(f) ? fs.statSync(f).size : 0,
+  }) + '\n');
+}
+</file>
+<file javascript pins_excerpt.mjs>
+#!/usr/bin/env node
+// Build the per-paper reading packet a hand-coder gets for the four-pin audit.
+//
+//   node scripts/pins_excerpt.mjs [--out out_pins/excerpts] [--budget 32000]
+//
+// Why an excerpt and not the whole paper: 250 papers is 20.9 MB of `.cols` text.
+// The coder needs the parts that describe the repeated measurement, not the
+// results or the related work. The rule below is deliberately over-inclusive —
+// two keyword tiers with context windows, merged — and the coder is also given
+// the full-text path and told to open it when the packet is not enough. The
+// recall of this rule is measured, not assumed: see the validation pass in
+// provenance:design:longitudinal.
+import fs from 'node:fs';
+import path from 'node:path';
+const argv = process.argv.slice(2);
+const OUT = argv.includes('--out') ? argv[argv.indexOf('--out') + 1] : 'out_pins/excerpts';
+const BUDGET = argv.includes('--budget') ? +argv[argv.indexOf('--budget') + 1] : 32000;
+const POP = 'out_pins/population.jsonl';
+// Tier 1: the sentence is about a wave, a repetition, or a pin being held.
+const CORE = [
+  /\b(?:re-?crawl|re-?ran|re-?run|repeat(?:ed|ing)?|second|third|fourth|fifth|subsequent|follow-?up|later|earlier)\b[^.]{0,60}\b(?:crawl|measur|experiment|round|wave|run|snapshot|iteration|visit|scan)/i,
+  /\b(?:crawl|measur\w+|experiment|round|wave|run|snapshot|iteration|scan)s?\b[^.]{0,60}\b(?:re-?crawl|repeat|again|monthly|weekly|daily|bi-?weekly|quarterly|yearly|annual|every|each)\b/i,
+  /\b(?:each|every|all|both|per|the (?:first|second|third|last|final))\s+(?:crawl|wave|round|run|iteration|snapshot|batch|measurement|visit|scan)\b/i,
+  /\b(?:same|identical|unchanged|consistent(?:ly)?|kept|keep|held|fixed|pinn?ed|froze[nd]?|constant)\b[^.]{0,80}\b(?:version|list|browser|profile|configuration|setup|vantage|location|machine|instance|seed|snapshot|settings?|parameters?)\b/i,
+  /\b(?:version|list|browser|profile|configuration|setup|vantage|location|machine|instance|seed)\b[^.]{0,60}\b(?:same|identical|unchanged|kept|held|fixed|pinn?ed|froze[nd]?|constant|updated|upgraded|refreshed|re-?download)/i,
+  /\bstate(?:less|ful)\b|\bcookie jar\b|\b(?:fresh|clean|new|persistent|shared|separate)\s+(?:browser\s+)?profiles?\b/i,
+  /\b(?:Tranco|Alexa|Majestic|Umbrella|CrUX|Chrome UX Report|Quantcast|BuiltWith|Cloudflare Radar)\b/i,
+  /\b(?:EasyList|EasyPrivacy|Disconnect|uBlock|AdBlock|Ghostery|filter\s+lists?|block\s?lists?|Whotracks)\b/i,
+  /\b(?:OpenWPM|Selenium|Puppeteer|Playwright|WebDriver|Xvfb|Chromedriver|Geckodriver|Docker|container)\b/i,
+  /\b(?:Chrome|Chromium|Firefox|Brave|Edge|Safari|Tor Browser|Opera)\b[^.]{0,30}\b(?:v(?:ersion)?\.?\s*)?\d{1,3}(?:\.\d+)*\b/i,
+  /\bvantage\b|\bVPN\b|\bproxy\b|\b(?:EC2|AWS|Azure|GCP|Google Cloud|DigitalOcean|Hetzner|Linode|OVH|Amazon)\b|\bdata\s?cent(?:er|re)\b|\bresidential\s+(?:IP|proxy|network)\b|\bexit node\b/i,
+  /\bsnapshots?\b|\bwaves?\b|\blongitudinal\b|\btime series\b|\bover\s+(?:a|the|\d+)\s+(?:period|year|month|week|day)/i,
+];
+// Tier 2: methodology furniture — useful surroundings, cheaper windows.
+const CTX = [
+  /\b(?:we|our)\s+(?:crawl|collect|gather|visit|measure|configure|deploy|instrument|run|download|use[d]?|set\s?up)/i,
+  /\b(?:methodolog|data collection|experimental setup|measurement setup|crawl(?:er|ing) setup|apparatus|infrastructure|limitations|threats to validity)\b/i,
+  /\b(?:January|February|March|April|May|June|July|August|September|October|November|December)\b[^.]{0,20}\b(?:19|20)\d\d\b/,
+  /\b(?:19|20)\d\d[-/](?:0?[1-9]|1[0-2])\b/,
+  /\b(?:headless|headful|user[- ]agent|screen resolution|viewport|GDPR|consent|banner|accept|reject)\b/i,
+  /\btop[- ](?:\d[\d,.]*\s*[km]?|million|thousand)\b|\bseed (?:list|set)\b|\bsite list\b|\bdomain list\b/i,
+];
+const CORE_W = 700;
+const CTX_W = 300;
+function windows(text) {
+  const spans = [];
+  for (const [res, w, tier] of [[CORE, CORE_W, 1], [CTX, CTX_W, 2]]) {
+    for (const re of res) {
+      const g = new RegExp(re.source, re.flags.includes('g') ? re.flags : re.flags + 'g');
+      let m;
+      while ((m = g.exec(text)) !== null) {
+        spans.push({ a: Math.max(0, m.index - w), b: Math.min(text.length, m.index + m[0].length + w), tier });
+        if (m.index === g.lastIndex) g.lastIndex++;
+      }
+    }
+  }
+  return spans;
+}
+function merge(spans) {
+  spans.sort((x, y) => x.a - y.a);
+  const out = [];
+  for (const s of spans) {
+    const last = out[out.length - 1];
+    if (last && s.a <= last.b + 40) { last.b = Math.max(last.b, s.b); last.tier = Math.min(last.tier, s.tier); }
+    else out.push({ ...s });
+  }
+  return out;
+}
+fs.mkdirSync(OUT, { recursive: true });
+const pop = fs.readFileSync(POP, 'utf8').trim().split('\n').map(JSON.parse);
+const stats = [];
+for (const p of pop) {
+  const raw = fs.readFileSync(p.textPath, 'utf8');
+  const text = raw.replace(/\r/g, '');
+  const HEAD = text.slice(0, 3000); // title + abstract, always included
+  let sp = merge(windows(text));
+  let kept = sp;
+  let dropped = 0;
+  const total = (s) => s.reduce((a, c) => a + (c.b - c.a), 0);
+  if (total(kept) > BUDGET) {
+    // Drop tier-2-only windows first, longest-shortfall last: keep every core window.
+    const core = kept.filter((s) => s.tier === 1);
+    dropped = kept.length - core.length;
+    kept = core;
+  }
+  if (total(kept) > BUDGET) {
+    // Still over: shrink core windows symmetrically rather than cutting the tail,
+    // because limitations sections sit at the end and carry the pin admissions.
+    const f = BUDGET / total(kept);
+    kept = kept.map((s) => { const mid = (s.a + s.b) / 2, h = ((s.b - s.a) * f) / 2; return { a: Math.round(mid - h), b: Math.round(mid + h), tier: s.tier }; });
+    kept = merge(kept);
+  }
+  const body = kept.map((s) => `[…${s.a}] ` + text.slice(s.a, s.b).replace(/\s+/g, ' ').trim()).join('\n\n');
+  const packet = `# ${p.key}\n# title: ${p.title}\n# full text: ${p.textPath} (${p.textBytes} bytes) — OPEN IT if this packet is not enough\n# extraction recorded: snapshots=${p.snapshots} browsers=${JSON.stringify(p.v_browsers)} statefulness=${p.v_statefulness} listVersion=${JSON.stringify(p.v_listVersion)} sourceList=${JSON.stringify(p.v_sourceList)} vantage=${JSON.stringify(p.v_vantage)}\n\n== HEAD (first 3000 chars) ==\n${HEAD.replace(/\s+/g, ' ').trim()}\n\n== KEYWORD WINDOWS (offsets in […N]) ==\n${body}\n`;
+  const f = path.join(OUT, p.key.replace(/\//g, '__') + '.txt');
+  fs.writeFileSync(f, packet);
+  stats.push({ key: p.key, src: text.length, packet: packet.length, windows: kept.length, droppedCtx: dropped });
+}
+const ps = stats.map((s) => s.packet).sort((a, b) => a - b);
+console.log(`wrote ${stats.length} packets to ${OUT}`);
+console.log(`packet chars  min ${ps[0]}  median ${ps[Math.floor(ps.length / 2)]}  p90 ${ps[Math.floor(ps.length * 0.9)]}  max ${ps[ps.length - 1]}`);
+console.log(`total packet MB ${(ps.reduce((a, c) => a + c, 0) / 1e6).toFixed(2)} (source 20.9 MB)`);
+console.log(`coverage: median packet is ${(100 * stats.map((s) => s.packet / s.src).sort((a, b) => a - b)[Math.floor(stats.length / 2)]).toFixed(0)}% of its source`);
+console.log(`packets where tier-2 windows had to be dropped: ${stats.filter((s) => s.droppedCtx > 0).length}`);
+fs.writeFileSync(path.join(OUT, '_stats.json'), JSON.stringify(stats, null, 1));
+</file>
+<file javascript pins_valsample.mjs>
+#!/usr/bin/env node
+// Draw the blind re-coding sample for the four-pin audit.
+//
+//   node scripts/pins_valsample.mjs [--n 30] [--seed 20260907]
+//
+// A second coder re-codes these papers from the FULL `.cols` text, without
+// seeing the first coding and without the keyword packet. That measures two
+// things at once: how reliable the coding is, and how much the excerpting rule
+// in scripts/pins_excerpt.mjs loses. Deterministic PRNG so the draw is
+// reproducible from the seed alone.
+import fs from 'node:fs';
+const argv = process.argv.slice(2);
+const N = argv.includes('--n') ? +argv[argv.indexOf('--n') + 1] : 30;
+const SEED = argv.includes('--seed') ? +argv[argv.indexOf('--seed') + 1] : 20260907;
+// mulberry32
+let s = SEED >>> 0;
+const rnd = () => { s |= 0; s = (s + 0x6d2b79f5) | 0; let t = Math.imul(s ^ (s >>> 15), 1 | s); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; };
+const pop = fs.readFileSync('out_pins/population.jsonl', 'utf8').trim().split('\n').map(JSON.parse);
+const idx = pop.map((_, i) => i);
+for (let i = idx.length - 1; i > 0; i--) { const j = Math.floor(rnd() * (i + 1)); [idx[i], idx[j]] = [idx[j], idx[i]]; }
+const pick = idx.slice(0, N).map((i) => pop[i]).sort((a, b) => a.key.localeCompare(b.key));
+console.error(`seed ${SEED}, n=${N} of ${pop.length}`);
+for (const p of pick) console.log(`${p.key}\t${p.textPath}`);
+</file>
+<file javascript report_pins_audit.mjs>
+#!/usr/bin/env node
+// The four-pin HAND-CODED audit for design:longitudinal.
+//
+//   node scripts/report_pins_audit.mjs [--coded out_pins/coded] [--run <dir>]
+//
+// design:longitudinal used to publish, for each of four crawl parameters, how
+// many of the 250 repeated-crawl papers STATED it. Stating a value once is only a
+// ceiling on holding it fixed across waves. This script aggregates a hand-coding
+// of all 250 papers against out_pins/CODEBOOK.md and turns each ceiling into a
+// three-number estimate:
+//
+//   floor    same-explicit                    the paper says it was unchanged
+//   estimate same-explicit + same-implied     one configuration described, no change stated
+//   ceiling  the old "stated at all" figure   what report_longitudinal.mjs prints
+//
+// Rules obeyed: papers are counted, never tuples; sentinels are never an answer;
+// every denominator is named on the line above the numbers; every value in the
+// codebook is printed even when it is zero, so nothing vanishes.
+import fs from 'node:fs';
+import path from 'node:path';
+const argv = process.argv.slice(2);
+const CODED = argv.includes('--coded') ? argv[argv.indexOf('--coded') + 1] : 'out_pins/coded';
+const RUN = argv.includes('--run') ? argv[argv.indexOf('--run') + 1] : '/workspace/publications_dataset/data/extract/run1';
+const POP = argv.includes('--pop') ? argv[argv.indexOf('--pop') + 1] : 'out_pins/population.jsonl';
+const pct = (a, b) => (b ? ((100 * a) / b).toFixed(1) : '0.0');
+const rule = (t) => console.log(`\n${'='.repeat(78)}\n${t}\n${'='.repeat(78)}`);
+const head = (t, d) => console.log(`\n-- ${t}\n   population: ${d}`);
+const pop = fs.readFileSync(POP, 'utf8').trim().split('\n').map(JSON.parse);
+const byKey = new Map(pop.map((p) => [p.key, p]));
+// ------------------------------------------------------------------ load
+const PINS = ['list', 'browser', 'vantage', 'state'];
+const VALUES = ['same-explicit', 'same-implied', 'varied-deliberate', 'changed', 'not-stated', 'unclear'];
+const KINDS = ['own-repeat-crawl', 'own-repeat-nonweb', 'parallel-not-temporal', 'reused-waves', 'single-crawl', 'unclear'];
+const coded = [];
+const problems = [];
+for (const p of pop) {
+  const f = path.join(CODED, p.key.replace(/\//g, '__') + '.json');
+  if (!fs.existsSync(f)) { problems.push(`MISSING coding: ${p.key}`); continue; }
+  const raw = fs.readFileSync(f, 'utf8').replace(/^\s*```(?:json)?\s*/i, '').replace(/\s*```\s*$/, '');
+  let c;
+  try { c = JSON.parse(raw); } catch (e) { problems.push(`UNPARSABLE: ${p.key}: ${e.message}`); continue; }
+  if (c.key !== p.key) problems.push(`KEY MISMATCH in ${f}: says ${c.key}`);
+  if (!KINDS.includes(c.repeatKind)) problems.push(`BAD repeatKind "${c.repeatKind}" in ${p.key}`);
+  let ok = true;
+  for (const k of PINS) {
+    if (!c.pins || !c.pins[k] || !VALUES.includes(c.pins[k].value)) {
+      problems.push(`BAD/MISSING pin ${k} in ${p.key}: ${JSON.stringify(c.pins && c.pins[k])}`);
+      ok = false;
+    }
+  }
+  if (!ok) continue;
+  coded.push({ ...c, key: p.key, year: p.year, venue: p.venue, slug: p.slug, ex: p });
+}
+// The whole audit rests on every paper being coded. Refuse to print figures
+// computed over a partial population rather than quietly shrinking the
+// denominator — a shelled-out denominator is how percentages go wrong silently.
+if (coded.length !== pop.length) {
+  console.error(`\nFATAL: coded ${coded.length} of ${pop.length}. Figures would be computed over the wrong population.`);
+  for (const s of problems) console.error('   ' + s);
+  process.exit(1);
+}
+console.log(`design:longitudinal — the four pins, HAND-CODED`);
+console.log(`coded dir: ${CODED}`);
+console.log(`extraction run: ${RUN}`);
+console.log(`corpus: 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P), 2010-2026`);
+if (problems.length) { console.log(`\nSCHEMA PROBLEMS (${problems.length}):`); for (const s of problems) console.log('   ' + s); }
+else console.log(`\nschema: all ${coded.length} codings parse and use only codebook values.`);
+// ------------------------------------------------- 1. is it a repeat crawl
+rule('1. IS THE MECHANICALLY SELECTED POPULATION REALLY REPEATED CRAWLING');
+head('repeatKind, hand-coded', `${coded.length} papers selected by temporal.snapshots > 1 AND crawled`);
+for (const k of KINDS) {
+  const n = coded.filter((c) => c.repeatKind === k).length;
+  console.log(`   ${k.padEnd(24)} ${String(n).padStart(4)}  ${pct(n, coded.length).padStart(5)}%`);
+}
+const OWN = coded.filter((c) => c.repeatKind === 'own-repeat-crawl');
+console.log(`\n   TRUE POSITIVES for "the authors re-crawled the web over time": ${OWN.length} / ${coded.length} = ${pct(OWN.length, coded.length)}%`);
+console.log(`   The remaining ${coded.length - OWN.length} are the mechanical predicate's false positives for this question.`);
+// ------------------------------------------------------- 2. the four pins
+rule('2. THE FOUR PINS — CEILING, ESTIMATE, FLOOR');
+for (const [POPNAME, G] of [['all 250 mechanically selected', coded], [`the ${OWN.length} hand-confirmed own repeat crawls`, OWN]]) {
+  head('hand-coded pin values', `${G.length} papers — ${POPNAME}`);
+  console.log(`   ${'pin'.padEnd(9)} ${VALUES.map((v) => v.padStart(17)).join('')}`);
+  for (const k of PINS) {
+    const row = VALUES.map((v) => G.filter((c) => c.pins[k].value === v).length);
+    console.log(`   ${k.padEnd(9)} ${row.map((n) => `${n} (${pct(n, G.length)}%)`.padStart(17)).join('')}`);
+  }
+  console.log(`\n   ${'pin'.padEnd(9)} ${'floor'.padStart(14)} ${'estimate'.padStart(16)} ${'ceiling'.padStart(16)}`);
+  console.log(`   ${''.padEnd(9)} ${'explicit'.padStart(14)} ${'expl+implied'.padStart(16)} ${'stated at all'.padStart(16)}`);
+  const XF = { list: 'x_listVersion', browser: 'x_browser', vantage: 'x_vantage', state: 'x_statefulness' };
+  for (const k of PINS) {
+    const fl = G.filter((c) => c.pins[k].value === 'same-explicit').length;
+    const es = G.filter((c) => ['same-explicit', 'same-implied'].includes(c.pins[k].value)).length;
+    const ce = G.filter((c) => c.ex[XF[k]]).length;
+    console.log(`   ${k.padEnd(9)} ${`${fl} (${pct(fl, G.length)}%)`.padStart(14)} ${`${es} (${pct(es, G.length)}%)`.padStart(16)} ${`${ce} (${pct(ce, G.length)}%)`.padStart(16)}`);
+  }
+  const four = (c, set) => PINS.every((k) => set.includes(c.pins[k].value));
+  const f4fl = G.filter((c) => four(c, ['same-explicit'])).length;
+  const f4es = G.filter((c) => four(c, ['same-explicit', 'same-implied'])).length;
+  const f4ce = G.filter((c) => c.ex.x_listVersion && c.ex.x_browser && c.ex.x_vantage && c.ex.x_statefulness).length;
+  console.log(`\n   ALL FOUR held fixed`);
+  console.log(`      floor    (all four same-explicit)              ${f4fl} / ${G.length} = ${pct(f4fl, G.length)}%`);
+  console.log(`      estimate (all four explicit or implied)        ${f4es} / ${G.length} = ${pct(f4es, G.length)}%`);
+  console.log(`      ceiling  (all four merely stated — old figure) ${f4ce} / ${G.length} = ${pct(f4ce, G.length)}%`);
+  const none = G.filter((c) => PINS.every((k) => c.pins[k].value === 'not-stated')).length;
+  console.log(`      none of the four stated at all                 ${none} / ${G.length} = ${pct(none, G.length)}%`);
+  const anyMoved = G.filter((c) => PINS.some((k) => ['changed', 'varied-deliberate'].includes(c.pins[k].value))).length;
+  console.log(`      at least one pin demonstrably NOT fixed        ${anyMoved} / ${G.length} = ${pct(anyMoved, G.length)}%`);
+}
+// -------------------------------------------- 3. where the ceiling overstates
+rule('3. HOW FAR THE OLD CEILING OVERSTATED EACH PIN');
+head('ceiling minus estimate, per pin', `${coded.length} papers (the population the published table used)`);
+const XF = { list: 'x_listVersion', browser: 'x_browser', vantage: 'x_vantage', state: 'x_statefulness' };
+console.log(`   ${'pin'.padEnd(9)} ${'ceiling'.padStart(9)} ${'estimate'.padStart(9)} ${'over by'.padStart(9)}  what the difference is`);
+for (const k of PINS) {
+  const ce = coded.filter((c) => c.ex[XF[k]]).length;
+  const es = coded.filter((c) => ['same-explicit', 'same-implied'].includes(c.pins[k].value)).length;
+  const why = coded.filter((c) => c.ex[XF[k]] && !['same-explicit', 'same-implied'].includes(c.pins[k].value));
+  const brk = new Map();
+  for (const c of why) brk.set(c.pins[k].value, (brk.get(c.pins[k].value) || 0) + 1);
+  console.log(`   ${k.padEnd(9)} ${String(ce).padStart(9)} ${String(es).padStart(9)} ${String(ce - es).padStart(9)}  ${[...brk.entries()].sort((a, b) => b[1] - a[1]).map(([v, n]) => `${v} ${n}`).join(', ') || '-'}`);
+}
+console.log(`\n   The reverse direction — hand-coding finds the pin fixed where the extraction`);
+console.log(`   recorded nothing (the paper says it in prose the extractor did not tuple):`);
+for (const k of PINS) {
+  const n = coded.filter((c) => !c.ex[XF[k]] && ['same-explicit', 'same-implied'].includes(c.pins[k].value)).length;
+  console.log(`   ${k.padEnd(9)} ${String(n).padStart(4)}`);
+}
+// ------------------------------------------------- 4. by span, by year
+rule('4. DOES THE LONGER PANEL STILL PIN LESS');
+{
+  const MONTHS = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec'];
+  // Same parser as report_longitudinal.mjs, including the textual-month fix.
+  const parseDate = (s) => {
+    if (!s) return null;
+    const t = String(s).toLowerCase();
+    const ym = /(?:^|[^\d])(\d{4})(?:[-/](0?[1-9]|1[0-2]))(?![\d])/.exec(t);
+    if (ym) return { y: +ym[1], m: +ym[2] };
+    const yO = /(?:^|[^\d])(\d{4})(?![\d])/.exec(t);
+    if (!yO) return null;
+    const y = +yO[1];
+    if (y < 1990 || y > 2027) return null;
+    const nm = new RegExp(`\\b(${MONTHS.join('|')})[a-z]*\\b`).exec(t);
+    return { y, m: nm ? MONTHS.indexOf(nm[1]) + 1 : 1 };
+  };
+  const longestSpan = (p) => {
+    let best = null;
+    for (const [a, b] of p.spans) {
+      const pa = parseDate(a), pb = parseDate(b);
+      if (!pa || !pb) continue;
+      const d = (pb.y - pa.y) * 12 + (pb.m - pa.m);
+      if (d >= 0 && (best == null || d > best)) best = d;
+    }
+    return best;
+  };
+  const four = (c) => PINS.every((k) => ['same-explicit', 'same-implied'].includes(c.pins[k].value));
+  head('all four fixed (estimate), by longest stated span', `${OWN.length} hand-confirmed own repeat crawls`);
+  const multi = OWN.filter((c) => { const m = longestSpan(c.ex); return m != null && m >= 12; });
+  const sub = OWN.filter((c) => { const m = longestSpan(c.ex); return m != null && m < 12; });
+  const unk = OWN.length - multi.length - sub.length;
+  for (const [name, g] of [['span >= 12 months', multi], ['span <  12 months', sub]]) {
+    console.log(`   ${name}  n=${String(g.length).padStart(3)}   all four ${g.filter(four).length} (${pct(g.filter(four).length, g.length)}%)   ` +
+      PINS.map((k) => `${k} ${g.filter((c) => ['same-explicit', 'same-implied'].includes(c.pins[k].value)).length}`).join('  '));
+  }
+  console.log(`   (${unk} have no parsable end date and are excluded from this split)`);
+  head('all four fixed (estimate), by four-year bucket', `${OWN.length} hand-confirmed own repeat crawls`);
+  for (const [a, b] of [[2010, 2011], [2012, 2015], [2016, 2019], [2020, 2023], [2024, 2026]]) {
+    const g = OWN.filter((c) => c.year >= a && c.year <= b);
+    const flag = b === 2026 ? '   << provisional' : '';
+    console.log(`   ${a}-${b}  n=${String(g.length).padStart(3)}   all four ${g.filter(four).length} (${pct(g.filter(four).length, g.length)}%)${flag}`);
+  }
+}
+// ------------------------------------------------- 5. explicit-pin exemplars
+rule('5. PAPERS THAT SAY IN WORDS THAT A PIN WAS HELD FIXED');
+{
+  for (const k of PINS) {
+    const g = coded.filter((c) => c.pins[k].value === 'same-explicit').sort((a, b) => b.year - a.year);
+    console.log(`\n-- ${k}: ${g.length} papers state explicitly that it was unchanged across waves`);
+    for (const c of g) console.log(`   ${c.year} ${c.venue.padEnd(9)} ${c.slug.slice(0, 62).padEnd(62)} "${(c.pins[k].quote || '').slice(0, 110)}"`);
+  }
+}
+rule('6. PAPERS WHERE A PIN DEMONSTRABLY MOVED BETWEEN WAVES');
+{
+  for (const k of PINS) {
+    const g = coded.filter((c) => c.pins[k].value === 'changed').sort((a, b) => b.year - a.year);
+    console.log(`\n-- ${k}: ${g.length} papers state the value changed or was refreshed between waves`);
+    for (const c of g) console.log(`   ${c.year} ${c.venue.padEnd(9)} ${c.slug.slice(0, 62).padEnd(62)} "${(c.pins[k].quote || '').slice(0, 110)}"`);
+  }
+  for (const k of PINS) {
+    const g = coded.filter((c) => c.pins[k].value === 'varied-deliberate');
+    console.log(`\n-- ${k}: ${g.length} papers vary it across waves BY DESIGN`);
+    for (const c of g) console.log(`   ${c.year} ${c.venue.padEnd(9)} ${c.slug.slice(0, 62)}`);
+  }
+}
+// ------------------------------------------------------- 7. coder confidence
+rule('7. CODER CONFIDENCE AND THE ALL-FOUR EXEMPLAR LIST');
+{
+  head('self-reported confidence', `${coded.length} papers`);
+  for (const v of ['high', 'medium', 'low']) {
+    const n = coded.filter((c) => c.confidence === v).length;
+    console.log(`   ${v.padEnd(8)} ${String(n).padStart(4)}  ${pct(n, coded.length).padStart(5)}%`);
+  }
+  const bad = coded.filter((c) => !['high', 'medium', 'low'].includes(c.confidence));
+  if (bad.length) console.log(`   OTHER/UNSET: ${bad.length} -> ${bad.map((c) => c.key).join(', ')}`);
+  const four = (c) => PINS.every((k) => ['same-explicit', 'same-implied'].includes(c.pins[k].value));
+  const g = coded.filter(four).sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue));
+  console.log(`\n-- the ${g.length} papers estimated to hold all four fixed (any repeatKind), most recent first`);
+  console.log(`   year | venue | slug | repeatKind | waves | explicit pins`);
+  for (const c of g) {
+    const ex = PINS.filter((k) => c.pins[k].value === 'same-explicit');
+    console.log(`   ${c.year} | ${c.venue} | ${c.slug} | ${c.repeatKind} | ${c.waveCount == null ? '?' : c.waveCount} | ${ex.join(',') || '-'}`);
+  }
+  const ownFour = OWN.filter(four);
+  console.log(`\n   of those, hand-confirmed own repeat crawls: ${ownFour.length}`);
+  for (const c of ownFour.sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue))) console.log(`   ${c.year} | ${c.venue} | ${c.slug}`);
+}
+// ---------------------------------------------- 8. old exemplar list, rechecked
+rule('8. WHAT HAPPENED TO THE 23 PAPERS THE OLD CEILING TABLE LISTED');
+{
+  const old = coded.filter((c) => c.ex.x_listVersion && c.ex.x_browser && c.ex.x_vantage && c.ex.x_statefulness);
+  const four = (c) => PINS.every((k) => ['same-explicit', 'same-implied'].includes(c.pins[k].value));
+  console.log(`   ${old.length} papers stated all four. Hand-coding each:\n`);
+  console.log(`   year venue      slug                                        kind                 list/browser/vantage/state`);
+  for (const c of old.sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue))) {
+    console.log(`   ${c.year} ${c.venue.padEnd(10)} ${c.slug.slice(0, 43).padEnd(43)} ${c.repeatKind.padEnd(20)} ${PINS.map((k) => c.pins[k].value).join(' / ')}`);
+  }
+  console.log(`\n   survive as all-four-fixed (estimate): ${old.filter(four).length} / ${old.length}`);
+}
+console.log();
+// ============================================================ wiki emitter
+// The page's tables are pasted from here, not retyped. A retyped table is how a
+// stale cell survives a refresh.
+if (argv.includes('--wiki')) {
+  const four = (c, set) => PINS.every((k) => set.includes(c.pins[k].value));
+  const XW = { list: 'x_listVersion', browser: 'x_browser', vantage: 'x_vantage', state: 'x_statefulness' };
+  const LBL = { list: 'Population list', browser: 'Browser', vantage: 'Vantage point', state: 'Statefulness' };
+  rule('WIKI MARKUP');
+  console.log(`\n---- table: repeatKind over the ${coded.length} mechanically selected papers ----`);
+  const KL = {
+    'own-repeat-crawl': 'The authors re-crawled the web over time',
+    'own-repeat-nonweb': 'Repeated, but not a web crawl (DNS probes, app stores, API pulls)',
+    'parallel-not-temporal': 'Parallel conditions at one time, not waves',
+    'reused-waves': 'Time points come from an archive or somebody else\'s dataset',
+    'single-crawl': 'One crawl — the snapshot count is a mis-extraction',
+    unclear: 'Undecidable after reading',
+  };
+  console.log(`^ What the "more than one snapshot" really is ^ Papers ^ Share of ${coded.length} ^`);
+  for (const k of KINDS) {
+    const n = coded.filter((c) => c.repeatKind === k).length;
+    console.log(`| ${KL[k]} | ${n} | ${pct(n, coded.length)}% |`);
+  }
+  for (const [G, name] of [[coded, `all ${coded.length}`], [OWN, `${OWN.length} hand-confirmed own repeat crawls`]]) {
+    console.log(`\n---- table: four pins over ${name} ----`);
+    console.log(`^ Pin ^ Held fixed, stated in words ^ Held fixed, estimate ^ Stated at all (old ceiling) ^ Moved between waves ^ Never stated ^`);
+    for (const k of PINS) {
+      const fl = G.filter((c) => c.pins[k].value === 'same-explicit').length;
+      const es = G.filter((c) => ['same-explicit', 'same-implied'].includes(c.pins[k].value)).length;
+      const ce = G.filter((c) => c.ex[XW[k]]).length;
+      const mv = G.filter((c) => ['changed', 'varied-deliberate'].includes(c.pins[k].value)).length;
+      const ns = G.filter((c) => c.pins[k].value === 'not-stated').length;
+      console.log(`| ${LBL[k]} | ${fl} (${pct(fl, G.length)}%) | **${es} (${pct(es, G.length)}%)** | ${ce} (${pct(ce, G.length)}%) | ${mv} (${pct(mv, G.length)}%) | ${ns} (${pct(ns, G.length)}%) |`);
+    }
+    const f4fl = G.filter((c) => four(c, ['same-explicit'])).length;
+    const f4es = G.filter((c) => four(c, ['same-explicit', 'same-implied'])).length;
+    const f4ce = G.filter((c) => c.ex.x_listVersion && c.ex.x_browser && c.ex.x_vantage && c.ex.x_statefulness).length;
+    const f4mv = G.filter((c) => PINS.some((k) => ['changed', 'varied-deliberate'].includes(c.pins[k].value))).length;
+    const f4ns = G.filter((c) => PINS.every((k) => c.pins[k].value === 'not-stated')).length;
+    console.log(`| **All four** | ${f4fl} (${pct(f4fl, G.length)}%) | **${f4es} (${pct(f4es, G.length)}%)** | ${f4ce} (${pct(f4ce, G.length)}%) | ${f4mv} (${pct(f4mv, G.length)}%) | ${f4ns} (${pct(f4ns, G.length)}%) |`);
+    console.log(`   (last two columns of the All-four row are "at least one pin moved" and "none of the four stated")`);
+  }
+}
+// ------------------------------------------------- the exemplar table, regenerated
+// The content page's "papers that pin all four" table is emitted here rather than
+// edited in place, so a hand-coded verdict cannot drift from the table cell that
+// reports it. Citekeys come from out_pins/citekeys.json, which is a copy of the
+// keys ALREADY on design:longitudinal — this adds no bibliography entries.
+if (argv.includes('--wiki')) {
+  const MAPF = argv.includes('--citekeys') ? argv[argv.indexOf('--citekeys') + 1] : 'out_pins/citekeys.json';
+  const map = JSON.parse(fs.readFileSync(MAPF, 'utf8'));
+  delete map._comment;
+  const old = coded.filter((c) => c.ex.x_listVersion && c.ex.x_browser && c.ex.x_vantage && c.ex.x_statefulness);
+  // If the map and the population disagree the table would silently lose a row.
+  const mapped = new Set(Object.keys(map));
+  const inPop = new Set(old.map((c) => c.slug));
+  const onlyMap = [...mapped].filter((s) => !inPop.has(s));
+  const onlyPop = [...inPop].filter((s) => !mapped.has(s));
+  if (onlyMap.length || onlyPop.length) {
+    console.error(`\nFATAL: citekey map does not match the all-four-stated set.`);
+    for (const s of onlyMap) console.error(`   in map, not in set: ${s}`);
+    for (const s of onlyPop) console.error(`   in set, not in map: ${s}`);
+    process.exit(1);
+  }
+  const VEN = { 'IEEE-SP': 'IEEE S&P', USENIX: 'USENIX', WWW: 'WWW', PETS: 'PETS', IMC: 'IMC', NDSS: 'NDSS', CCS: 'CCS' };
+  const KIND = {
+    'own-repeat-crawl': 'own repeat crawl',
+    'own-repeat-nonweb': 'repeated, not a web crawl',
+    'parallel-not-temporal': 'parallel, not waves',
+    'reused-waves': 'reused waves',
+    'single-crawl': 'single crawl',
+    unclear: 'unclear',
+  };
+  const SH = { 'same-explicit': 'fixed, in words', 'same-implied': 'fixed, implied', 'varied-deliberate': 'varied by design', changed: '**changed**', 'not-stated': 'not stated', unclear: 'unclear' };
+  const four = (c) => PINS.every((k) => ['same-explicit', 'same-implied'].includes(c.pins[k].value));
+  console.log(`\n---- table: the ${old.length} papers that STATED all four, with what hand-coding found ----`);
+  console.log(`^ Year ^ Venue ^ Paper ^ Really a repeat crawl? ^ List ^ Browser ^ Vantage ^ State ^ All four fixed? ^`);
+  for (const c of old.sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue) || a.slug.localeCompare(b.slug))) {
+    const m = map[c.slug];
+    console.log(`| ${c.year} | ${VEN[c.venue]} | ${m.name} {[${m.key}]} | ${KIND[c.repeatKind]} | ${PINS.map((k) => SH[c.pins[k].value]).join(' | ')} | ${four(c) ? '**yes**' : 'no'} |`);
+  }
+  console.log(`\n   survivors: ${old.filter(four).length} of ${old.length}`);
+}
+</file>
+<file javascript pins_quotecheck.mjs>
+#!/usr/bin/env node
+// Verify every evidence quote the four-pin hand-coding produced.
+//
+//   node scripts/pins_quotecheck.mjs [--coded out_pins/coded] [--pop ...] [--exc ...]
+//
+// A hand-coder that paraphrases is a hand-coder that hallucinates. Each non-empty
+// `quote` must appear in the paper it is attributed to.
+//
+// WHY THIS IS NOT A PLAIN SUBSTRING TEST. The corpus stores three renderings of
+// each PDF and they disagree sentence by sentence:
+//   paper.cols.txt   two-column reading order repaired  (what the coders were given)
+//   paper.norm.txt   the unrepaired extraction
+//   paper.txt        the raw pdftotext dump
+// Even `.cols` still splices a quarter of pages: a sentence that runs across a
+// column boundary comes back with the neighbouring column's text injected into
+// the middle of it, e.g.
+//     "We implemented FourthParty as an extension to Mozilla
+//      B. Design Principles Firefox. It currently instruments ..."
+// A coder who reads that correctly and writes down the real sentence produces a
+// quote that is NOT a substring of any rendering. Failing it as unlocated would
+// call a correct reading a hallucination.
+//
+// So matching is tried in four escalating modes, against all three renderings
+// and the excerpt packet:
+//   exact      raw substring
+//   collapsed  runs of whitespace -> one space on both sides
+//   loose      collapsed + de-hyphenation across line breaks, unicode quotes and
+//              dashes folded to ASCII, case ignored
+//   spliced    the quote is cut into overlapping 6-word shingles and a quote
+//              passes if SHINGLE_MIN of them are found individually. This is the
+//              de-splicing mode and it is reported separately and listed in full,
+//              because it is the weakest evidence and has to be read by a human.
+// A quote that fails all four against every stored rendering is then tried
+// against a FIFTH text: a pypdf rendering of the same paper.pdf, produced on
+// demand by `scripts/pdftext.py` via `scripts/pdftext_cache.mjs`. `.cols` and
+// pypdf splice different sentences, so a coder who read the PDF correctly can
+// produce a quote that is absent from every stored rendering and verbatim in
+// the PDF. The same four modes are tried there and a pass is reported as
+//   RESCUED  located in the PDF but in none of the stored renderings. The
+//            quote is in the paper; the stored text is what is wrong.
+// This is the fallback `scripts/quote_check.mjs` and `scripts/report_webxray.mjs`
+// already use; it was added here on 2026-09-22, and before that this checker's
+// NOT FOUND bucket was not comparable with theirs.
+// Anything failing all four modes against all five texts is printed in full.
+import fs from 'node:fs';
+import path from 'node:path';
+import { pdfText } from './pdftext_cache.mjs';
+const argv = process.argv.slice(2);
+const CODED = argv.includes('--coded') ? argv[argv.indexOf('--coded') + 1] : 'out_pins/coded';
+const POP = argv.includes('--pop') ? argv[argv.indexOf('--pop') + 1] : 'out_pins/population.jsonl';
+const EXC = argv.includes('--exc') ? argv[argv.indexOf('--exc') + 1] : 'out_pins/excerpts';
+const SHINGLE_N = 6;
+const SHINGLE_MIN = 0.8;
+const collapse = (s) => s.replace(/\s+/g, ' ').trim();
+const loose = (s) =>
+  collapse(s)
+    .replace(/(\w)-\s+(\w)/g, '$1$2')
+    .replace(/[‘’ʼ′]/g, "'")
+    .replace(/[“”″]/g, '"')
+    .replace(/[‐-―−]/g, '-')
+    .replace(/[ ]/g, ' ')
+    .toLowerCase();
+// Fraction of the quote's overlapping SHINGLE_N-word windows present in a
+// loose-folded haystack; null when the quote is too short to shingle.
+const shingleFrac = (q, hayLoose) => {
+  const w = loose(q).split(' ').filter(Boolean);
+  if (w.length < SHINGLE_N) return null;
+  const sh = [];
+  for (let i = 0; i + SHINGLE_N <= w.length; i++) sh.push(w.slice(i, i + SHINGLE_N).join(' '));
+  return sh.filter((x) => hayLoose.includes(x)).length / sh.length;
+};
+const pop = fs.readFileSync(POP, 'utf8').trim().split('\n').map(JSON.parse);
+let total = 0, empty = 0;
+const modes = { exact: 0, collapsed: 0, loose: 0, spliced: 0 };
+const fails = [];
+const splicedList = [];
+const rescuedList = [];
+let missing = 0;
+for (const p of pop) {
+  const f = path.join(CODED, p.key.replace(/\//g, '__') + '.json');
+  if (!fs.existsSync(f)) { missing++; continue; }
+  const raw = fs.readFileSync(f, 'utf8').replace(/^\s*```(?:json)?\s*/i, '').replace(/\s*```\s*$/, '');
+  const c = JSON.parse(raw);
+  // Every rendering the corpus holds, plus the packet the coder was handed.
+  const dir = path.dirname(p.textPath);
+  const parts = [];
+  for (const name of ['paper.cols.txt', 'paper.norm.txt', 'paper.txt']) {
+    const q = path.join(dir, name);
+    if (fs.existsSync(q)) parts.push(fs.readFileSync(q, 'utf8'));
+  }
+  const packet = path.join(EXC, p.key.replace(/\//g, '__') + '.txt');
+  if (fs.existsSync(packet)) parts.push(fs.readFileSync(packet, 'utf8'));
+  const hay = parts.join('\n');
+  const hayC = collapse(hay);
+  const hayL = loose(hay);
+  for (const [pin, v] of Object.entries(c.pins)) {
+    total++;
+    const q = (v.quote || '').trim();
+    if (!q) { empty++; continue; }
+    if (hay.includes(q)) { modes.exact++; continue; }
+    if (hayC.includes(collapse(q))) { modes.collapsed++; continue; }
+    if (hayL.includes(loose(q))) { modes.loose++; continue; }
+    // de-splicing mode
+    const frac = shingleFrac(q, hayL);
+    if (frac !== null && frac >= SHINGLE_MIN) {
+      modes.spliced++;
+      splicedList.push({ key: p.key, pin, value: v.value, quote: q, frac });
+      continue;
+    }
+    // Fifth text: pypdf. Only consulted for a quote that is in none of the
+    // stored renderings, so it costs nothing on a clean run.
+    const pdf = pdfText(p.key);
+    if (pdf !== null) {
+      const pdfC = collapse(pdf);
+      const pdfL = loose(pdf);
+      let mode = null;
+      if (pdf.includes(q)) mode = 'exact';
+      else if (pdfC.includes(collapse(q))) mode = 'collapsed';
+      else if (pdfL.includes(loose(q))) mode = 'loose';
+      else {
+        const pdfFrac = shingleFrac(q, pdfL);
+        if (pdfFrac !== null && pdfFrac >= SHINGLE_MIN) mode = `spliced ${(100 * pdfFrac).toFixed(0)}%`;
+      }
+      if (mode !== null) {
+        rescuedList.push({ key: p.key, pin, value: v.value, quote: q, frac, mode });
+        continue;
+      }
+    }
+    fails.push({ key: p.key, pin, value: v.value, quote: q, frac, pdf: pdf !== null });
+  }
+}
+const checked = total - empty;
+const pc = (n) => `${n}  ${((100 * n) / (checked || 1)).toFixed(1)}%`;
+console.log(`design:longitudinal four-pin audit — evidence quote check`);
+console.log(`coded dir: ${CODED}`);
+console.log(`renderings searched per paper: paper.cols.txt, paper.norm.txt, paper.txt, and the excerpt packet`);
+console.log(`fallback for a quote found in none of those: a pypdf rendering of paper.pdf (scripts/pdftext.py), same four modes, reported as RESCUED`);
+console.log(`de-splicing mode: ${SHINGLE_N}-word shingles, pass at >= ${(100 * SHINGLE_MIN).toFixed(0)}% found`);
+console.log(`\npapers with no coding file (skipped): ${missing}`);
+console.log(`pin codings:            ${total}`);
+console.log(`   no quote given:      ${empty}   (permitted only for not-stated; checked below)`);
+console.log(`   checked:             ${checked}`);
+console.log(`      exact match:                 ${pc(modes.exact)}`);
+console.log(`      whitespace-collapsed match:  ${pc(modes.collapsed)}`);
+console.log(`      loose match:                 ${pc(modes.loose)}`);
+console.log(`      spliced match (de-spliced):  ${pc(modes.spliced)}`);
+console.log(`      RESCUED from the PDF:        ${pc(rescuedList.length)}`);
+console.log(`      NOT FOUND:                   ${pc(fails.length)}`);
+console.log(`   (papers with no pypdf rendering available: ${[...new Set(fails.filter((f) => !f.pdf).map((f) => f.key))].length})`);
+// A quote-less coding that is not `not-stated` is a coding with no evidence.
+const noEvidence = [];
+for (const p of pop) {
+  const f = path.join(CODED, p.key.replace(/\//g, '__') + '.json');
+  if (!fs.existsSync(f)) continue;
+  const c = JSON.parse(fs.readFileSync(f, 'utf8').replace(/^\s*```(?:json)?\s*/i, '').replace(/\s*```\s*$/, ''));
+  for (const [pin, v] of Object.entries(c.pins)) {
+    if (!(v.quote || '').trim() && v.value !== 'not-stated') noEvidence.push(`${p.key} ${pin}=${v.value}`);
+  }
+}
+console.log(`\ncodings with no quote and a value other than not-stated: ${noEvidence.length}`);
+for (const s of noEvidence) console.log('   ' + s);
+console.log(`\nMATCHED ONLY AFTER DE-SPLICING (${splicedList.length}) — these are read by hand, the source page is two-column spliced:`);
+for (const f of splicedList.sort((a, b) => a.frac - b.frac)) {
+  console.log(`   ${f.key}  [${f.pin}=${f.value}]  shingles ${(100 * f.frac).toFixed(0)}%\n      ${f.quote}`);
+}
+console.log(`\nRESCUED — in none of the stored renderings, located in the pypdf rendering of paper.pdf (${rescuedList.length}).`);
+console.log(`These are defects in the stored text, not in the hand coding:`);
+for (const f of rescuedList) {
+  console.log(`   ${f.key}  [${f.pin}=${f.value}]  stored shingles ${f.frac == null ? 'n/a' : (100 * f.frac).toFixed(0) + '%'} -> pdf ${f.mode}\n      ${f.quote}`);
+}
+console.log(`\nQUOTES NOT LOCATED, printed in full (${fails.length}):`);
+for (const f of fails) console.log(`   ${f.key}  [${f.pin}=${f.value}]  shingles ${f.frac == null ? 'n/a' : (100 * f.frac).toFixed(0) + '%'}\n      ${f.quote}`);
+</file>
+<file javascript pins_agreement.mjs>
+#!/usr/bin/env node
+// Inter-coder agreement for the design:longitudinal four-pin hand-coding.
+//
+//   node scripts/pins_agreement.mjs [--a out_pins/coded] [--b out_pins/coded_val]
+//
+// Coder A read a keyword-window excerpt packet; coder B read the whole paper and
+// never saw A's output. The sample is out_pins/valsample.tsv. Agreement is
+// reported three ways per pin, because the three answer different questions:
+//
+//   exact        the same codebook value
+//   fixed/not    both collapse to {fixed = same-explicit|same-implied} vs
+//                {not = varied-deliberate|changed|not-stated|unclear} — this is
+//                the partition the published "estimate" column actually uses
+//   Cohen kappa  chance-corrected, on the exact values
+//
+// A high exact agreement with a low kappa means the coders agree because one
+// value dominates, not because they are reading the same thing. Both are printed.
+import fs from 'node:fs';
+import path from 'node:path';
+const argv = process.argv.slice(2);
+const A = argv.includes('--a') ? argv[argv.indexOf('--a') + 1] : 'out_pins/coded';
+const B = argv.includes('--b') ? argv[argv.indexOf('--b') + 1] : 'out_pins/coded_val';
+const SAMPLE = argv.includes('--sample') ? argv[argv.indexOf('--sample') + 1] : 'out_pins/valsample.tsv';
+const PINS = ['list', 'browser', 'vantage', 'state'];
+const VALUES = ['same-explicit', 'same-implied', 'varied-deliberate', 'changed', 'not-stated', 'unclear'];
+const FIXED = new Set(['same-explicit', 'same-implied']);
+const pct = (a, b) => (b ? ((100 * a) / b).toFixed(1) : '0.0');
+const keys = fs.readFileSync(SAMPLE, 'utf8').trim().split('\n').map((l) => l.split('\t')[0]);
+const load = (dir, key) => {
+  const f = path.join(dir, key.replace(/\//g, '__') + '.json');
+  if (!fs.existsSync(f)) return null;
+  const raw = fs.readFileSync(f, 'utf8').replace(/^\s*```(?:json)?\s*/i, '').replace(/\s*```\s*$/, '');
+  return JSON.parse(raw);
+};
+const pairs = [];
+const missing = [];
+for (const k of keys) {
+  const a = load(A, k), b = load(B, k);
+  if (!a || !b) { missing.push(`${k}  ${a ? '' : 'A missing '}${b ? '' : 'B missing'}`); continue; }
+  pairs.push({ key: k, a, b });
+}
+// Refuse to report an agreement rate over a shrunken sample without saying so.
+console.log('design:longitudinal four-pin hand-coding — inter-coder agreement');
+console.log(`coder A (excerpt packets): ${A}`);
+console.log(`coder B (full text, blind): ${B}`);
+console.log(`\nreliability sample: ${keys.length} papers drawn from the 250`);
+console.log(`pairs available:    ${pairs.length}`);
+if (missing.length) { console.log(`MISSING (${missing.length}):`); for (const m of missing) console.log('   ' + m); }
+const kappa = (rows) => {
+  const n = rows.length;
+  if (!n) return NaN;
+  const obs = rows.filter(([x, y]) => x === y).length / n;
+  const ma = new Map(), mb = new Map();
+  for (const [x, y] of rows) { ma.set(x, (ma.get(x) || 0) + 1); mb.set(y, (mb.get(y) || 0) + 1); }
+  let exp = 0;
+  for (const v of new Set([...ma.keys(), ...mb.keys()])) exp += ((ma.get(v) || 0) / n) * ((mb.get(v) || 0) / n);
+  return exp === 1 ? NaN : (obs - exp) / (1 - exp);
+};
+console.log(`\n${'='.repeat(78)}\nPER-PIN AGREEMENT (n=${pairs.length} papers)\n${'='.repeat(78)}`);
+console.log(`   ${'pin'.padEnd(9)} ${'exact'.padStart(14)} ${'fixed/not'.padStart(14)} ${'kappa(exact)'.padStart(14)} ${'kappa(fixed)'.padStart(14)}`);
+for (const k of PINS) {
+  const rows = pairs.map((p) => [p.a.pins[k].value, p.b.pins[k].value]);
+  const bin = rows.map(([x, y]) => [FIXED.has(x) ? 'fixed' : 'not', FIXED.has(y) ? 'fixed' : 'not']);
+  const ex = rows.filter(([x, y]) => x === y).length;
+  const bi = bin.filter(([x, y]) => x === y).length;
+  const ke = kappa(rows), kb = kappa(bin);
+  console.log(`   ${k.padEnd(9)} ${`${ex}/${rows.length} (${pct(ex, rows.length)}%)`.padStart(14)} ${`${bi}/${rows.length} (${pct(bi, rows.length)}%)`.padStart(14)} ${(Number.isNaN(ke) ? 'n/a' : ke.toFixed(2)).padStart(14)} ${(Number.isNaN(kb) ? 'n/a' : kb.toFixed(2)).padStart(14)}`);
+}
+{
+  const rows = pairs.flatMap((p) => PINS.map((k) => [p.a.pins[k].value, p.b.pins[k].value]));
+  const bin = rows.map(([x, y]) => [FIXED.has(x) ? 'fixed' : 'not', FIXED.has(y) ? 'fixed' : 'not']);
+  const ex = rows.filter(([x, y]) => x === y).length;
+  const bi = bin.filter(([x, y]) => x === y).length;
+  console.log(`   ${'ALL PINS'.padEnd(9)} ${`${ex}/${rows.length} (${pct(ex, rows.length)}%)`.padStart(14)} ${`${bi}/${rows.length} (${pct(bi, rows.length)}%)`.padStart(14)} ${kappa(rows).toFixed(2).padStart(14)} ${kappa(bin).toFixed(2).padStart(14)}`);
+}
+console.log(`\n-- repeatKind`);
+{
+  const rows = pairs.map((p) => [p.a.repeatKind, p.b.repeatKind]);
+  const ex = rows.filter(([x, y]) => x === y).length;
+  console.log(`   exact ${ex}/${rows.length} (${pct(ex, rows.length)}%)   kappa ${kappa(rows).toFixed(2)}`);
+  const own = (v) => (v === 'own-repeat-crawl' ? 'own' : 'other');
+  const bin = rows.map(([x, y]) => [own(x), own(y)]);
+  const bi = bin.filter(([x, y]) => x === y).length;
+  console.log(`   own-repeat-crawl vs everything else: ${bi}/${bin.length} (${pct(bi, bin.length)}%)   kappa ${kappa(bin).toFixed(2)}`);
+}
+console.log(`\n-- "all four fixed", the figure the page publishes`);
+{
+  const f = (c) => PINS.every((k) => FIXED.has(c.pins[k].value));
+  const rows = pairs.map((p) => [String(f(p.a)), String(f(p.b))]);
+  const ex = rows.filter(([x, y]) => x === y).length;
+  console.log(`   A says all four fixed: ${rows.filter(([x]) => x === 'true').length}/${rows.length}`);
+  console.log(`   B says all four fixed: ${rows.filter(([, y]) => y === 'true').length}/${rows.length}`);
+  console.log(`   agree:                 ${ex}/${rows.length} (${pct(ex, rows.length)}%)   kappa ${kappa(rows).toFixed(2)}`);
+}
+console.log(`\n${'='.repeat(78)}\nDIRECTION OF DISAGREEMENT — does the excerpt coder over- or under-call "fixed"\n${'='.repeat(78)}`);
+for (const k of PINS) {
+  let aOnly = 0, bOnly = 0, both = 0, neither = 0;
+  for (const p of pairs) {
+    const x = FIXED.has(p.a.pins[k].value), y = FIXED.has(p.b.pins[k].value);
+    if (x && y) both++; else if (x) aOnly++; else if (y) bOnly++; else neither++;
+  }
+  console.log(`   ${k.padEnd(9)} both fixed ${String(both).padStart(3)}   only A (excerpt) ${String(aOnly).padStart(3)}   only B (full text) ${String(bOnly).padStart(3)}   neither ${String(neither).padStart(3)}`);
+}
+console.log(`\n${'='.repeat(78)}\nEVERY DISAGREEMENT, PRINTED IN FULL\n${'='.repeat(78)}`);
+let nd = 0;
+for (const p of pairs) {
+  const diffs = PINS.filter((k) => p.a.pins[k].value !== p.b.pins[k].value);
+  const kindDiff = p.a.repeatKind !== p.b.repeatKind;
+  if (!diffs.length && !kindDiff) continue;
+  nd++;
+  console.log(`\n${p.key}`);
+  if (kindDiff) console.log(`   repeatKind   A=${p.a.repeatKind}   B=${p.b.repeatKind}`);
+  for (const k of diffs) console.log(`   ${k.padEnd(9)}    A=${p.a.pins[k].value.padEnd(18)} B=${p.b.pins[k].value}`);
+}
+console.log(`\npapers with at least one disagreement: ${nd}/${pairs.length} (${pct(nd, pairs.length)}%)`);
+console.log(`\n${'='.repeat(78)}\nVALUE DISTRIBUTION, EACH CODER (so a dominant value is visible)\n${'='.repeat(78)}`);
+console.log(`   ${'value'.padEnd(20)} ${'A'.padStart(6)} ${'B'.padStart(6)}`);
+for (const v of VALUES) {
+  const a = pairs.flatMap((p) => PINS.map((k) => p.a.pins[k].value)).filter((x) => x === v).length;
+  const b = pairs.flatMap((p) => PINS.map((k) => p.b.pins[k].value)).filter((x) => x === v).length;
+  console.log(`   ${v.padEnd(20)} ${String(a).padStart(6)} ${String(b).padStart(6)}`);
+}
+</file>
+<file python build_provenance_pins.py>
+#!/usr/bin/env python3
+"""Append the four-pin hand-coding section to provenance:design:longitudinal.
+Reads the live page text and the prose fragment, and splices in <file> blocks
+read from the committed scripts and their committed outputs — never retyped, so
+the published listing is the code that produced the published figures.
+  python3 scripts/build_provenance_pins.py \
+      --live pages/provenance_design_longitudinal.LIVE.txt \
+      --prose prov/pins_section.txt \
+      --out  pages/provenance_design_longitudinal.NEW.txt
+The prose fragment carries {{BLOCK:<relpath>|<lang>|<name>}} placeholders. Each is
+replaced by the file's real contents. A placeholder naming a missing file, or a
+file containing a literal file-closing tag (which would end the block early and
+render the rest of the page as markup), is a hard error.
+"""
+import argparse, pathlib, re, sys
+CLOSER = "</" + "file>"
+ap = argparse.ArgumentParser()
+ap.add_argument("--live", required=True)
+ap.add_argument("--prose", required=True)
+ap.add_argument("--out", required=True)
+ap.add_argument("--anchor", default="[[design:longitudinal|← back to the content page]]")
+a = ap.parse_args()
+root = pathlib.Path(__file__).resolve().parent.parent
+live = pathlib.Path(a.live).read_text(encoding="utf8")
+prose = pathlib.Path(a.prose).read_text(encoding="utf8")
+def expand(m):
+    rel, lang, name = m.group(1), m.group(2), m.group(3)
+    p = root / rel
+    if not p.exists():
+        sys.exit(f"FATAL: {rel} does not exist; the section would publish a placeholder.")
+    src = p.read_text(encoding="utf8").rstrip()
+    if CLOSER in src:
+        sys.exit(f"FATAL: {rel} contains a literal file-closing tag; the block would end early.")
+    if not src.strip():
+        sys.exit(f"FATAL: {rel} is empty.")
+    return f"<file {lang} {name}>\n{src}\n{CLOSER}"
+prose, n = re.subn(r"\{\{BLOCK:([^|]+)\|([^|]*)\|([^}]+)\}\}", expand, prose)
+if n == 0:
+    sys.exit("FATAL: no {{BLOCK:...}} placeholders expanded; the section would carry no code.")
+# §9 said the hand-coding "needs" doing. It is done, and leaving the old wording
+# would leave the page contradicting its own §18. Patched here rather than by hand
+# so the edit is regenerable and fails loudly if the target text ever moves.
+PATCHES = [(
+    "  * **Whether a paper held a pin //constant// across waves.** The extraction records what a "
+    "paper stated, once. Every pin figure on the content page is a ceiling and is labelled as one. "
+    "Closing this needs a hand-coded sample of the 250, reading each paper's per-wave methods "
+    "description. That is a day of work and would turn every ceiling into an estimate; it is the "
+    "single most valuable follow-up.",
+    "  * **Whether a paper held a pin //constant// across waves.** **Closed on 2026-09-07** — see "
+    "**§18** below. All 250 were hand-coded rather than a sample, and the content page now publishes "
+    "estimates beside the stated-value figures. What is left open by that work is in §18.8, and the "
+    "largest residue is that ''same-implied'' — one configuration described and never contradicted — "
+    "cannot be distinguished from a pin that silently moved."
+)]
+for _old, _new in PATCHES:
+    if _old not in live:
+        sys.exit("FATAL: patch target not found in the live page; §9 would keep contradicting §18.")
+    live = live.replace(_old, _new, 1)
+if a.anchor not in live:
+    sys.exit(f"FATAL: anchor not found in the live page: {a.anchor!r}")
+head, sep, tail = live.rpartition(a.anchor)
+out = head + prose.rstrip() + "\n\n" + sep + tail
+# Structural guard: an in-place generator that silently drops sections is the
+# failure mode this check exists for.
+#
+# A whole-output count is NOT enough. The appended section adds ~11 headings of
+# its own, so a total-count check has that much slack: drop one heading from the
+# preserved live content and the total still exceeds the original. Mutation
+# testing on 2026-09-07 confirmed exactly that — a heading deleted from `head`
+# gave "27 -> 37" and exit 0. So the counts are compared on the PRESERVED live
+# text alone (head + sep + tail), which must match the live page exactly, and the
+# appended section is counted separately and required to be non-empty.
+preserved = head + sep + tail
+for tag, pat in (("headings", r"(?m)^=====? "), ("tables", r"(?m)^\^ "), ("file blocks", r"<file ")):
+    b, keep, add = len(re.findall(pat, live)), len(re.findall(pat, preserved)), len(re.findall(pat, prose))
+    if keep != b:
+        sys.exit(f"FATAL: {tag} in the preserved live text went from {b} to {keep}; the append altered content it must not touch.")
+    print(f"{tag}: live {b} kept {keep} + new {add} = {keep + add}")
+    if add == 0 and tag != "tables":
+        sys.exit(f"FATAL: the appended section contains no {tag}; it is not the section that was written.")
+# Byte-level check that nothing in the live page was silently rewritten, beyond
+# the patches applied above, which are themselves verified to have a target.
+if preserved != live:
+    sys.exit("FATAL: the preserved text is not byte-identical to the (patched) live page.")
+pathlib.Path(a.out).write_text(out, encoding="utf8")
+print(f"expanded {n} code blocks; wrote {a.out} ({len(out)} bytes)")
+</file>
+**Unedited output of the audit** — ''node scripts/report_pins_audit.mjs'', 2026-09-07. Sections 5 and 6 list, paper by paper with its evidence quote, every pin coded as explicitly fixed and every pin coded as having moved; section 8 is the 23-paper recheck.
+<file  report_pins_audit-output.txt>
+design:longitudinal — the four pins, HAND-CODED
+coded dir: out_pins/coded
+extraction run: /workspace/publications_dataset/data/extract/run1
+corpus: 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P), 2010-2026
+schema: all 250 codings parse and use only codebook values.
+==============================================================================
+. IS THE MECHANICALLY SELECTED POPULATION REALLY REPEATED CRAWLING
+==============================================================================
+-- repeatKind, hand-coded
+   population: 250 papers selected by temporal.snapshots > 1 AND crawled
+   own-repeat-crawl          124   49.6%
+   own-repeat-nonweb          44   17.6%
+   parallel-not-temporal      30   12.0%
+   reused-waves               21    8.4%
+   single-crawl               31   12.4%
+   unclear                     0    0.0%
+   TRUE POSITIVES for "the authors re-crawled the web over time": 124 / 250 = 49.6%
+   The remaining 126 are the mechanical predicate's false positives for this question.
+==============================================================================
+. THE FOUR PINS — CEILING, ESTIMATE, FLOOR
+==============================================================================
+-- hand-coded pin values
+   population: 250 papers — all 250 mechanically selected
+   pin           same-explicit     same-impliedvaried-deliberate          changed       not-stated          unclear
+   list             73 (29.2%)      117 (46.8%)         6 (2.4%)       38 (15.2%)        16 (6.4%)         0 (0.0%)
+   browser           23 (9.2%)       88 (35.2%)        14 (5.6%)         7 (2.8%)      116 (46.4%)         2 (0.8%)
+   vantage          32 (12.8%)       38 (15.2%)        20 (8.0%)         2 (0.8%)      155 (62.0%)         3 (1.2%)
+   state             24 (9.6%)       38 (15.2%)         8 (3.2%)         0 (0.0%)      177 (70.8%)         3 (1.2%)
+   pin                floor         estimate          ceiling
+                   explicit     expl+implied    stated at all
+   list          73 (29.2%)      190 (76.0%)      167 (66.8%)
+   browser        23 (9.2%)      111 (44.4%)      143 (57.2%)
+   vantage       32 (12.8%)       70 (28.0%)       89 (35.6%)
+   state          24 (9.6%)       62 (24.8%)       70 (28.0%)
+   ALL FOUR held fixed
+      floor    (all four same-explicit)              2 / 250 = 0.8%
+      estimate (all four explicit or implied)        18 / 250 = 7.2%
+      ceiling  (all four merely stated — old figure) 23 / 250 = 9.2%
+      none of the four stated at all                 10 / 250 = 4.0%
+      at least one pin demonstrably NOT fixed        79 / 250 = 31.6%
+-- hand-coded pin values
+   population: 124 papers — the 124 hand-confirmed own repeat crawls
+   pin           same-explicit     same-impliedvaried-deliberate          changed       not-stated          unclear
+   list             48 (38.7%)       47 (37.9%)         2 (1.6%)       26 (21.0%)         1 (0.8%)         0 (0.0%)
+   browser          14 (11.3%)       55 (44.4%)         4 (3.2%)         6 (4.8%)       44 (35.5%)         1 (0.8%)
+   vantage          26 (21.0%)       19 (15.3%)         8 (6.5%)         1 (0.8%)       69 (55.6%)         1 (0.8%)
+   state            16 (12.9%)       22 (17.7%)         2 (1.6%)         0 (0.0%)       81 (65.3%)         3 (2.4%)
+   pin                floor         estimate          ceiling
+                   explicit     expl+implied    stated at all
+   list          48 (38.7%)       95 (76.6%)       86 (69.4%)
+   browser       14 (11.3%)       69 (55.6%)       83 (66.9%)
+   vantage       26 (21.0%)       45 (36.3%)       50 (40.3%)
+   state         16 (12.9%)       38 (30.6%)       45 (36.3%)
+   ALL FOUR held fixed
+      floor    (all four same-explicit)              2 / 124 = 1.6%
+      estimate (all four explicit or implied)        15 / 124 = 12.1%
+      ceiling  (all four merely stated — old figure) 15 / 124 = 12.1%
+      none of the four stated at all                 0 / 124 = 0.0%
+      at least one pin demonstrably NOT fixed        42 / 124 = 33.9%
+==============================================================================
+. HOW FAR THE OLD CEILING OVERSTATED EACH PIN
+==============================================================================
+-- ceiling minus estimate, per pin
+   population: 250 papers (the population the published table used)
+   pin         ceiling  estimate   over by  what the difference is
+   list            167       190       -23  changed 27, not-stated 9, varied-deliberate 3
+   browser         143       111        32  not-stated 20, varied-deliberate 12, changed 7, unclear 2
+   vantage          89        70        19  varied-deliberate 19, not-stated 18, unclear 1
+   state            70        62         8  not-stated 8, varied-deliberate 6, unclear 3
+   The reverse direction — hand-coding finds the pin fixed where the extraction
+   recorded nothing (the paper says it in prose the extractor did not tuple):
+   list        62
+   browser      9
+   vantage     19
+   state        9
+==============================================================================
+. DOES THE LONGER PANEL STILL PIN LESS
+==============================================================================
+-- all four fixed (estimate), by longest stated span
+   population: 124 hand-confirmed own repeat crawls
+   span >= 12 months  n= 33   all four 1 (3.0%)   list 21  browser 12  vantage 11  state 5
+   span <  12 months  n= 77   all four 12 (15.6%)   list 64  browser 51  vantage 31  state 28
+   (14 have no parsable end date and are excluded from this split)
+-- all four fixed (estimate), by four-year bucket
+   population: 124 hand-confirmed own repeat crawls
+-2011  n=  2   all four 0 (0.0%)
+-2015  n= 14   all four 0 (0.0%)
+-2019  n= 32   all four 3 (9.4%)
+-2023  n= 41   all four 8 (19.5%)
+-2026  n= 35   all four 4 (11.4%)   << provisional
+==============================================================================
+. PAPERS THAT SAY IN WORDS THAT A PIN WAS HELD FIXED
+==============================================================================
+-- list: 73 papers state explicitly that it was unchanged across waves
+PETS      more-space-less-privacy-measuring-the-effectiveness-of-ip-base "Tranco list: https://tranco-list.eu/list/YXQ4G"
+PETS      overcoming-language-barriers-multilingual-analysis-of-the-2023 "The set of websites is held constant across both snapshots;"
+IEEE-SP   sok-decoding-the-enigma-of-encrypted-network-traffic-classifie "load 660 urls of shortlisted domains"
+IMC       canvassing-the-fingerprinters-characterizing-canvas-fingerprin "we revisited the top 20k and tail 20k sites with our crawler twice more"
+IMC       exploration-of-the-dynamics-of-buy-and-sale-of-social-media-ac "Out of 58 trading markets, 11 contain selling offers with publicly visible social media account handles, which"
+IMC       somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-prot "We restrict the list to sites that appear in all of the top 100k lists over this period to avoid having our re"
+IMC       toward-better-efficiency-vs-fidelity-tradeoffs-in-web-archives "Starting in January 2025, we crawled 10K pages each week for 16 consecutive weeks."
+IMC       towards-a-non-binary-view-of-ipv6-adoption                     "the Tranco [31] top 1M list dated 2024-10-16 (ID:"
+PETS      buy-it-now-track-me-later-attacking-user-privacy-via-wi-fi-ap- "We continued to look these MAC addresses up every day for a month, from mid-January to mid-February 2025."
+PETS      intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-bann "This list, generated on 07 December 2023 with ID K2NZW, is available at: https:"
+PETS      measuring-the-accuracy-and-effectiveness-of-pii-removal-servic "we use a script to automatically crawl the data broker coverage list on the official website of the PII remova"
+USENIX    double-edged-shield-on-the-fingerprintability-of-customized-ad "Available on https://tranco-list.eu/list/24P99/1000"
+USENIX    evaluating-the-effectiveness-and-robustness-of-visual-similari "For each domain, we collect monthly"
+USENIX    navigating-cookie-consent-violations-across-the-globe          "this initial crawl, we recrawled the union of the sites across all"
+USENIX    no-way-to-sign-out-unpacking-non-compliance-with-google-plays- "291 apps missing an account deletion link after one month."
+USENIX    websites-global-privacy-control-compliance-at-scale-and-over-t "crawled our crawl set of 11,708 sites three times"
+WWW       welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud- "we perform a temporal analysis and re-crawl the same list of 2,600 domains with sellers.json files from Sectio"
+IMC       browsing-without-third-party-cookies-what-do-you-see           "We deploy our crawler on the top 10,000 domains of the Tranco [35] list generated on Feb. 18th, 2024."
+PETS      a-large-scale-study-of-cookie-banner-interaction-tools-and-the "we resort to the widely used quasi-standard Tranco list [30] generated on 29/08/2022"
+PETS      johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-fr "We use the Tranco list dated April 17th, 2023, with ID GZ7NK."
+PETS      two-steps-forward-and-one-step-back-the-right-to-opt-out-of-sa "the October 29, 2022 Tranco top websites list"
+USENIX    darkfleece-probing-the-dark-side-of-android-subscription-apps  "apps (75 suspected and 25 benign) that were collected in June 2021. We downloaded their versions in August 202"
+CCS       cookiegraph-understanding-and-detecting-first-party-tracking-c "We successfully conducted the four parallel crawls for 99.31% of the 20K websites."
+IMC       a-longitudinal-study-of-vulnerable-client-side-resources-and-w "We utilize the single snapshot of the Alexa Top 1M domains of Mar. 2018."
+IMC       tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-sp "(from 09/07/2021) [49] and probe for prebid.js version. We treat"
+PETS      blocking-javascript-without-breaking-the-web-an-empirical-inve "In 2021, we crawled 100K webpages and classified the collected JS"
+USENIX    a-study-of-multi-factor-and-risk-based-authentication-availabi "generated on June 21, 2021, and then chose 50 random sites"
+USENIX    know-your-cybercriminal-evaluating-attacker-preferences-by-mea "Crawling started in Nov 2020, but as IMPaaS.ru went offline for an infrastructural upgrade from 11 Dec 2020 to"
+WWW       automatic-discovery-of-emerging-browser-fingerprinting-techniq "we redid the crawling on the same 10K domains in September 2022 (11-month gap) and applied our system to this "
+WWW       scan-me-if-you-can-understanding-and-detecting-unwanted-vulner "we run each scanner for 10 rounds, against both our WordPress and Joomla web applications."
+CCS       helping-or-hindering-how-browser-extensions-undermine-security "we employ the Tranco Top 100 domains based on the list of November 1, 2021 (ID: Y3JG)"
+CCS       understanding-security-issues-in-the-nft-ecosystem             "we crawled the same set of assets three times with a three-month interval between two subsequent crawls: in Ju"
+IEEE-SP   surakav-generating-realistic-traces-for-a-strong-website-finge "We collect data on the Tranco top 1 million [43] sites. The list was generated on 21st January 2021."
+IEEE-SP   wtagraph-web-tracking-and-advertising-detection-using-graph-ne "We selected the top 10K websites from the Alexa top one million list dated on June 8th, 2020 for data collecti"
+IMC       html-violations-and-where-to-find-them-a-longitudinal-analysis "To get a reproducible list of popular websites, we rely on the Tranco lists [44]. From these lists, we take th"
+IMC       respect-the-origin-a-best-case-evaluation-of-connection-coales "Our initial sample set was the 5000 domains with the most requests to the third party, as indicated by the Ref"
+PETS      a-global-survey-of-android-dual-use-applications-used-in-intim "we crawled Google Play again in all fifteen languages for one day in February 2022 with the queries we obtaine"
+PETS      atom-ad-network-tomography                                     "We create one interest group with 45 manually curated websites (cf. §3.2.1)."
+WWW       measuring-the-privacy-vs-compatibility-trade-off-in-preventing "But each of our 8 parallel crawls must visit the same sequence of page URLs to produce comparable results."
+NDSS      whos-hosting-the-block-party-studying-third-party-blockage-of- "For each crawl, our crawlers visited the start pages from the fixed list and followed every same-site link."
+PETS      deja-vu-abusing-browser-cache-headers-to-identify-and-track-on "websites from the Tranco list [33] on the 2nd and the"
+PETS      domain-name-encryption-is-not-enough-privacy-leakage-via-ip-ba "The list was created on March 3rd 2020, and is available at https://tranco-list.eu/list/J2KY."
+WWW       tls-1-3-in-practice-how-tls-1-3-contributes-to-the-internet    "We utilize a single snapshot of the Alexa 1M websites generated in April 2018 during our observation period"
+WWW       towards-realistic-and-reproducibleweb-crawl-measurements       "Tranco list of top sites [18] (snapshot 77PX)"
+IMC       analyzing-third-party-service-dependencies-in-modern-web-servi "we use a snapshot of the rankings from December 2016 and collect data for these websites in 2016 and 2020 from"
+IMC       out-of-sight-not-out-of-mind-a-user-view-on-the-criticality-of "2020, with the same list of countries and websites."
+NDSS      complex-security-policy-a-longitudinal-analysis-of-deployed-co "Given that our list of Web sites remains stable, we can attribute the increased CSP adoption to Web developers"
+PETS      a-comparative-measurement-study-of-web-tracking-on-mobile-and- "in both experiments, we collected data from all the 116,000 websites, but the data for each website in Experim"
+WWW       beyond-the-front-page-measuring-third-party-dynamics-in-the-fi "We used the list generated on 03/26/2019 (ID: W9L9)."
+WWW       facebook-ads-monitor-an-independent-auditing-system-for-politi "we repeated the task of querying for an empty string periodically during the period of Aug 2018 to December 20"
+WWW       the-representativeness-of-automated-web-crawls-as-a-surrogate- "All of the crawls described in this section used the AlexaTop1k [4] as their seed list and were performed betw"
+IMC       no-more-chasing-waterfalls-a-measurement-study-of-the-header-b "we crawled these 5k websites every day for a period of 34 days in Feb'19, collecting metadata about the HB auc"
+IMC       visiblev8-in-browser-monitoring-of-javascript-in-the-wild      "to visit the Alexa top 50k web domains"
+WWW       before-and-after-gdpr-the-changes-in-third-party-presence-at-p "from the 1,363 websites for each of the 21 harvests, capturing all"
+WWW       outguard-detecting-in-browser-covert-cryptocurrency-mining-in- "We continuously monitored all cryptojacking websites from our first deployment, which observed 5,873 websites "
+WWW       revisiting-user-mobility-and-social-relationships-in-lbsns-a-h "we keep (a) Node (user) degree (b) Number of check-ins per user only users appearing in both social network sn"
+WWW       who-watches-the-watchmen-exploring-complaints-on-the-web       "Each week, we exclude URLs that have already been deleted."
+USENIX    plug-and-prey-measuring-the-commoditization-of-cybercrime-via- "We use longitudinal data from eight online anonymous marketplaces over six years, from the original Silk Road "
+WWW       uncovering-http-header-inconsistencies-and-the-impact-on-deskt "using the top 70,000 website as ranked by Alexa [1] as of"
+CCS       poster-probing-tor-hidden-service-with-dockers                 "The number of Onion addresses in this experiment was 16,683, and the addresses were crawled twice a day at 12-"
+IEEE-SP   xhound-quantifying-the-fingerprintability-of-browser-extension "we waited for four months since the original collection of Chrome extensions and examined the updated versions"
+PETS      cross-device-tracking-measurement-and-disclosures              "browse the same 100 popular websites - the top 20 sites for Games, Sports, News, Shopping, and Reference accor"
+USENIX    characterizing-the-nature-and-dynamics-of-tor-exit-blocking    "We crawled the Alexa Top 500 web pages from a control host and a subset of Tor exit relays."
+USENIX    measuring-the-insecurity-of-mobile-deep-links-of-android       "In May 2017, we scan these domains again."
+CCS       cspautogen-black-box-enforcement-of-content-security-policy-up "we use these templates to match the same website captured on 02/01/2016, 03/01/2016 and 04/01/2016"
+IEEE-SP   cloak-of-visibility-detecting-when-machines-browse-a-different "we crawl each URL three times per browsing profile which we denote C1 , C2 , C3 for clarity."
+USENIX    the-ever-changing-labyrinth-a-large-scale-analysis-of-wildcard "we monitored the structural changes of SSP for 25 days within Jan 2016."
+USENIX    tracing-information-flows-between-ad-exchanges-using-retargete "During each round of crawling, the personas visit their associated e-commerce sites, then visit the 2,250 publ"
+IMC       who-is-com-learning-to-parse-whois-records                     "Some domains in the February 2015 zone file snapshot were expired by the time we crawled them"
+NDSS      i-do-not-know-what-you-visited-last-summer-protecting-users-fr "This process is repeated twice: once starting with a clean browser and once more after priming the client-side"
+WWW       early-detection-of-spam-mobile-apps                            "For instance, all apps in Crawl 1 were checked again during Crawl 2."
+IEEE-SP   lastor-a-low-latency-as-aware-tor-client                       "To quantify the extent of this latency overhead, we measured
+the latency of visiting the top 200 websites [17]"
+WWW       understanding-and-combating-link-farming-in-the-twitter-social "We re-crawled the profile page of each user in the above dataset in February 2011"
+-- browser: 23 papers state explicitly that it was unchanged across waves
+NDSS      eviction-notice-reviving-and-advancing-page-cache-attacks      "We target Firefox 133.0, which uses the shared library"
+IEEE-SP   sok-decoding-the-enigma-of-encrypted-network-traffic-classifie "we chose two opensource web browsers: Firefox [22] and Chromium [11]."
+USENIX    double-edged-shield-on-the-fingerprintability-of-customized-ad "we use Chrome v.113"
+NDSS      flow-correlation-attacks-on-tor-onion-service-sessions-with-sl "Clients also run on an isolated Docker container that executes a 0.4.7.10 Tor process and issues requests to T"
+PETS      a-large-scale-study-of-cookie-banner-interaction-tools-and-the "Each profile is based on OpenWPM [12] (v0.20.0), uses the Firefox browser (v100.0 with the default user agent1"
+PETS      johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-fr "Each of our four crawls used the same baseline configuration."
+PETS      two-steps-forward-and-one-step-back-the-right-to-opt-out-of-sa "All data collection was conducted using the Chrome web browser."
+CCS       cookiegraph-understanding-and-detecting-first-party-tracking-c "We use OpenWPM (v0.17.0) and Firefox (v102) [52] to crawl a sample of 20K out of the top-million websites."
+IMC       tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-sp "personas. Specifically, we use identical hardware/software, collect"
+WWW       automatic-discovery-of-emerging-browser-fingerprinting-techniq "applied our system to this data set by following the same procedure."
+IEEE-SP   the-state-of-the-samesite-studying-the-usage-effectiveness-and "The exact user-agent strings used by our crawler are detailed below. • Chrome Desktop: Mozilla/5.0 (Windows NT"
+IMC       polls-clickbait-and-commemorative-2-bills-problematic-politica "we ran 312 daily crawls, on 4 machines, using Chromium 88.0.4298.0, on a Debian 9 Docker image."
+PETS      deja-vu-abusing-browser-cache-headers-to-identify-and-track-on "3rd of June, using Headless Chrome instrumented with"
+PETS      domain-name-encryption-is-not-enough-privacy-leakage-via-ip-ba "we repeatedly crawled the 220K websites from our test list curated in §5.1, using the Chrome browser (desktop "
+PETS      a-comparative-measurement-study-of-web-tracking-on-mobile-and- "The version of both Firefox for Android and desktop Firefox is 53.0 without the tracking protection function."
+WWW       beyond-the-front-page-measuring-third-party-dynamics-in-the-fi "For each visit, we use the same user agent (Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52"
+WWW       the-representativeness-of-automated-web-crawls-as-a-surrogate- "44 crawls using Firefox 68 were performed, with a variable cadence, over 54 days, using a single OSX machine l"
+IMC       visiblev8-in-browser-monitoring-of-javascript-in-the-wild      "Chrome 64, which was used for primary data collection"
+PETS      p1-fp-extraction-classification-and-prediction-of-website-fing "Tor version 0.4.0.8 to collect our datasets"
+CCS       rendered-insecure-gpu-side-channel-attacks-are-practical       "the Chrome browser version is 63.0.3239.84"
+IMC       needle-in-a-haystack-tracking-down-elite-phishing-domains-in-t "We set "User-Agent" for iPhone 6 and Chrome 65 to obtain the mobile and web pages respectively."
+IMC       browser-feature-usage-on-the-modern-web                        "browser (version 46.0.1) with a custom browser extension"
+CCS       surf-detecting-and-measuring-search-poisoning                  "we implemented SURF on top of an instrumented version of Internet Explorer 8."
+-- vantage: 32 papers state explicitly that it was unchanged across waves
+PETS      more-space-less-privacy-measuring-the-effectiveness-of-ip-base "crawler machine used a distinct, static public IP address throughout"
+PETS      privacy-vs-profit-the-impact-of-googles-manifest-version-3-mv3 "held geography constant so that any differences reflect"
+IEEE-SP   sok-decoding-the-enigma-of-encrypted-network-traffic-classifie "All data was collected using a university network, providing a realistic environment from January 13, 2024, to"
+IMC       canvassing-the-fingerprinters-characterizing-canvas-fingerprin "We performed the crawls in May 2025 using a UCSD IP address."
+PETS      intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-bann "executed from a server located in the EU, and each crawl within"
+PETS      more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settin "we run five such virtual machines simultaneously with local IP addresses in Sydney (Australia), Toronto (Canad"
+USENIX    navigating-cookie-consent-violations-across-the-globe          "Each measurement iteration involved crawling all 8 regions"
+USENIX    websites-global-privacy-control-compliance-at-scale-and-over-t "16 GB RAM set to a Los Angeles IP address using Mullvad"
+PETS      a-bilingual-longitudinal-analysis-of-privacy-policies-measurin "For all crawls, we used a server located in California to simulate the geolocation of Californian residents in"
+PETS      automatic-generation-of-web-censorship-probe-lists             "we repeat our measurements 50 times in each vantage point (apart from US-West 1 and US-West 2, where measureme"
+PETS      johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-fr "All crawls were conducted using virtual machines from Amazon Web Services with IP addresses in California."
+PETS      the-devil-is-in-the-details-detection-measurement-and-lawfulne "Location EU EU"
+PETS      two-steps-forward-and-one-step-back-the-right-to-opt-out-of-sa "All of the data collection was done on AWS EC2 c6i.8xlarge instances that were located in a California data ce"
+USENIX    fledging-will-continue-until-privacy-improves-empirical-analys "While all the measurements were performed through US-based IP addresses"
+CCS       cookiegraph-understanding-and-detecting-first-party-tracking-c "We run all crawls in the US to minimize the impact of the EU GDPR and do not interact with cookie banners."
+IMC       tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-sp "bids at the same time (simultaneously), from the same location, and"
+WWW       automatic-discovery-of-emerging-browser-fingerprinting-techniq "applied our system to this data set by following the same procedure."
+IMC       respect-the-origin-a-best-case-evaluation-of-connection-coales "We used the same machines throughout our evaluations for consistency, and saw no observable change in datacent"
+PETS      a-global-survey-of-android-dual-use-applications-used-in-intim "The EC2 servers we used for our crawling in 2020 and 2022 were all located in the US."
+WWW       measuring-the-privacy-vs-compatibility-trade-off-in-preventing "All crawls were performed in parallel and simultaneously from a single network vantage point."
+PETS      deja-vu-abusing-browser-cache-headers-to-identify-and-track-on "For our evaluation, we performed crawls from a single address in our research lab."
+IMC       analyzing-third-party-service-dependencies-in-modern-web-servi "We conduct our measurements from a single vantage point on the US East Coast."
+IMC       out-of-sight-not-out-of-mind-a-user-view-on-the-criticality-of "2020, with the same list of countries and websites."
+WWW       the-representativeness-of-automated-web-crawls-as-a-surrogate- "44 crawls using Firefox 68 were performed, with a variable cadence, over 54 days, using a single OSX machine l"
+WWW       measuring-political-personalization-of-google-news-search      "we use a static DNS entry for Google News to ensure that our search queries are routed to the same datacenter"
+IEEE-SP   tracking-certificate-misissuance-in-the-wild                   "from the University of Illinois Urbana- Champaign campus between September 1-20, 2017"
+WWW       auditing-the-personalization-and-composition-of-politically-re "we utilized built in features to restrict the visibility of our recruitment ads to participants within the US"
+NDSS      dial-one-for-scam-a-large-scale-analysis-of-technical-support- "We deployed the ROBOVIC Crawler on three different sites (our campus, Amazon's Elastic Compute Cloud [25], and"
+NDSS      thou-shalt-not-depend-on-me-analysing-the-use-of-outdated-java "We conducted the two crawls in May 2016 from IP addresses in a /24 range in the US."
+USENIX    investigating-commercial-pay-per-install-and-the-distribution- "our pipeline runs exclusively from United States IP addresses."
+USENIX    tracing-information-flows-between-ad-exchanges-using-retargete "All crawls were done from Northeastern University's IP addresses in Boston."
+IEEE-SP   lastor-a-low-latency-as-aware-tor-client                       "To study this, we measure path latencies on
+the Tor network when visiting the top 200 websites from 50
+PlanetL"
+-- state: 24 papers state explicitly that it was unchanged across waves
+NDSS      characterizing-the-implementation-of-censorship-policies-in-ch "For the majority of services, we keep the browser session
+persistent across a given test run."
+PETS      more-space-less-privacy-measuring-the-effectiveness-of-ip-base "interference between successive website visits, ensuring consistency across crawl batches."
+IEEE-SP   sok-decoding-the-enigma-of-encrypted-network-traffic-classifie "the browser is quit to clean up session data and free up system resources (Line 11)."
+NDSS      evocrawl-exploring-web-application-code-and-state-using-evolut "reset all tested web application instances before each crawling
+session. This ensures that all scanners commen"
+PETS      unmasking-the-shadows-a-cross-country-study-of-online-tracking "in stateless mode, where no session data or cookies were preserved between page loads"
+WWW       welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud- "only issue a single HTTP(S) request to fetch either the ads.txt"
+PETS      a-large-scale-study-of-cookie-banner-interaction-tools-and-the "In our experiment, we perform a stateful crawl for each visited site, meaning we keep the browser's state when"
+PETS      the-devil-is-in-the-details-detection-measurement-and-lawfulne "Crawl state Stateless Stateless"
+IMC       tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-sp "same browser profiles that are logged into the Amazon account and"
+PETS      blocking-javascript-without-breaking-the-web-an-empirical-inve "and stateless crawls."
+PETS      raven-stateless-rapid-ip-address-variation-for-enterprise-netw "The browser cache was cleaned"
+USENIX    a-study-of-multi-factor-and-risk-based-authentication-availabi "performed with an empty browser cache and cleared cookies."
+IMC       respect-the-origin-a-best-case-evaluation-of-connection-coales "Client caches are intentionally deleted between measurements in all of our active measurements to prevent bias"
+USENIX    a-large-scale-investigation-into-geodifferences-in-mobile-apps "We also clear the Play Store cache on the phones between downloads."
+WWW       measuring-the-privacy-vs-compatibility-trade-off-in-preventing "The crawlers maintained independent, persistent user profiles for each policy instance to maintain state acros"
+PETS      a-comparative-measurement-study-of-web-tracking-on-mobile-and- "Each data collec- tion experiment includes two stateless2 measurements on the mobile and desktop environments,"
+WWW       a-first-look-at-commercial-5g-performance-on-smartphones       "clear the browser's cache before each loading"
+IMC       an-empirical-study-of-the-cost-of-dns-over-https               "Caches of both Firefox and the page load times, suggesting that it is possible to obtain the improved DNS stub"
+IMC       no-more-chasing-waterfalls-a-measurement-study-of-the-header-b "We initiated a clean slate instance before visiting each website, in order to keep the crawling process statel"
+PETS      p1-fp-extraction-classification-and-prediction-of-website-fing "reset the Tor process after each website visit, but otherwise used the default options"
+NDSS      thou-shalt-not-depend-on-me-analysing-the-use-of-outdated-java "our crawler remains on each page for a fixed delay of 60 seconds before clearing its entire state, restarting,"
+PETS      cross-device-tracking-measurement-and-disclosures              "much like a browser would with a consumer that has reopened her browser without clearing browsing data"
+IEEE-SP   cloak-of-visibility-detecting-when-machines-browse-a-different "we tear down our environment and clear all cookies between fetches"
+IMC       location-location-location-the-impact-of-geolocation-on-web-se "all of our treatments were configured and behaved identically."
+==============================================================================
+. PAPERS WHERE A PIN DEMONSTRABLY MOVED BETWEEN WAVES
+==============================================================================
+-- list: 38 papers state the value changed or was refreshed between waves
+PETS      quicstep-evaluating-connection-migration-based-quic-censorship "daily Tranco top 1M websites"
+NDSS      the-unusual-suspects-studying-reasons-for-lacking-updates-in-w "as described for Sampling Round 1 (May 9-19, 2023)"
+PETS      beyond-the-request-harnessing-http-response-headers-for-cross- "using T.EX and visiting the landing pages with Chrome to explore the top 10K websites from the Tranco ranking "
+PETS      the-effect-of-platform-policies-on-app-privacy-compliance-a-st "we re-queued all of the previous apps for testing and added 302 additional apps that we subsequently identifie"
+PETS      who-s-watching-you-zoom-investigating-privacy-of-third-party-z "between December 2023 and December 2024, 667 new apps were added, 212 apps were removed, and 49 apps were rena"
+WWW       detecting-and-understanding-the-promotion-of-illicit-goods-and "The newly captured PIPs will be further fed into the keyword generator so as to extract new keywords which in "
+PETS      a-bilingual-longitudinal-analysis-of-privacy-policies-measurin "crawls in February 2021 we visited the homepages of the top 10K Tranco domains using the list from January 31,"
+USENIX    unpacking-privacy-labels-a-measurement-and-developer-perspecti "we observed a total of 2.72M unique apps and 2.17M com- mon apps."
+USENIX    policycomp-counterpart-comparison-of-privacy-policies-uncovers "we compared the results from the dataset described in Sec. 5.1 (collected in August 2021) with those from anot"
+USENIX    timeless-timing-attacks-and-preload-defenses-in-tors-dns-cache "the weekend before each of the first four weeks of data collection (see Section 3), we downloaded fresh popula"
+WWW       leveraging-googles-publisher-specific-ids-to-detect-website-ad "We perform our analysis on websites which have been crawled in all snapshots (i.e., their intersection) and co"
+IEEE-SP   crawlphish-large-scale-analysis-of-client-side-cloaking-techni "CrawlPhish collected the source code of 28,973 phishing websites from June to December 2018 and 100,000 websit"
+PETS      privacy-preference-signals-past-present-and-future             "Netograph continuously ingests a live feed of social media posts, extracts all URLs, and visits them from craw"
+USENIX    can-i-take-your-subdomain-exploring-same-site-attacks-in-the-m "Concerning discontinued services, we focused on the 3 largest providers (WordPress, Shopify and Tumblr), repre"
+WWW       privacy-policies-over-time-curation-and-analysis-of-a-million- "We obtained 541,616 websites by combining all domains that appear in the top 100K of these 22 Alexa lists (two"
+WWW       where-are-you-taking-me-understanding-abusive-traffic-distribu "We generate a new set of target URLs for every run of an experiment. The only exception is the URL shortening "
+IMC       demystifying-the-messaging-platforms-ecosystem-through-the-len "We commence the metadata collection for each group from the date when we discovered it and repeat it every day"
+WWW       read-between-the-lines-an-empirical-measurement-of-sensitive-a "Note that even for the same store, the different snapshots do not necessarily contain the same set of skills."
+IEEE-SP   empoweb-empowering-web-applications-with-browser-extensions    "In the middle of May 2018, we did another crawl and analysis. The results presented here are for this second d"
+IEEE-SP   phishfarm-a-scalable-framework-for-measuring-the-effectiveness "we solely used nondeceptive paths and hostnames (i.e. with randomly-chosen English words) in order to remove U"
+IMC       a-longitudinal-analysis-of-the-ads-txt-standard                "before each crawl, we fetched the latest Alexa Top-100K list, computed the union of it and our existing list o"
+IMC       an-empirical-study-of-the-cost-of-dns-over-https               "The Alexa list was retrieved on 15 September 2018."
+NDSS      we-value-your-privacy-now-take-some-cookies-measuring-the-gdpr "To extend the scope of our study, we retrieved updated top lists once per month."
+WWW       a-first-look-at-deep-learning-apps-on-smartphones              "We have crawled two datasets at different moments, June 2018 and September 2018, which are three months apart."
+WWW       mobile-app-risk-ranking-via-exclusive-sparse-coding            "There are no overlaps for the above datasets, and the apps are from a wide range of functionalities, including"
+WWW       understanding-the-evolution-of-mobile-app-ecosystems-a-longitu "we repeated the same process as described above to crawl Google Play apps, except that we take the previous 1."
+IEEE-SP   the-spyware-used-in-intimate-partner-violence                  "The set of queries retrieved every day changed over time even though the seed queries were the same."
+IMC       needle-in-a-haystack-tracking-down-elite-phishing-domains-in-t "From April 09 to April 29 in 2018, we collect three additional snapshots only for the detected squatting phish"
+WWW       betrayed-by-your-dashboard-discovering-malicious-campaigns-via "We use our system to crawl 145K malicious URLs provided by VirusTotal on a daily basis for a period of two wee"
+NDSS      dial-one-for-scam-a-large-scale-analysis-of-technical-support- "Note that ROBOVIC was originally relying just on domain parking in order to find technical support scams and w"
+WWW       extended-tracking-powers-measuring-the-privacy-diffusion-enabl "we collect a second set of top 10K browser extensions (Enew ) four months after our first one (Eold )"
+USENIX    internet-jones-and-the-raiders-of-the-lost-trackers-an-archaeo "For 2010-2016, we use Wayback Machine archives of Alexa's top million sites list (a csv file)."
+CCS       caronte-detecting-location-leaks-for-deanonymizing-tor-hidden- "as C ARONTE explores the hidden services, it adds any new onion URLs found in their content to the list, so th"
+USENIX    measuring-the-longitudinal-evolution-of-the-online-anonymous-m "We started collecting data from Evolution in July, so it is possible that we miss quite a bit in the early par"
+CCS       a-nearly-four-year-longitudinal-study-of-search-engine-poisoni "The size and composition of the query set varies 4.3 over time, but, on average, consists of 64 queries."
+CCS       poster-proactive-blacklist-update-for-anti-phishing            "We hourly downloaded the brand-new blacklist containing phishing URLs belonging to PayPal target on 1449 6th J"
+USENIX    deseo-combating-search-result-poisoning                        "We collect three sampled sets of URLs from Bing. These"
+IEEE-SP   a-practical-attack-to-de-anonymize-social-network-users        "entire network, we only downloaded the group directory and"
+-- browser: 7 papers state the value changed or was refreshed between waves
+PETS      who-s-watching-you-zoom-investigating-privacy-of-third-party-z "We created another crawler and parser to handle a major change in the app category listing after May 2024."
+PETS      the-devil-is-in-the-details-detection-measurement-and-lawfulne "Browser FF86.0.1 FF95"
+IMC       respect-the-origin-a-best-case-evaluation-of-connection-coales "Results shown are for Firefox (v91) for later comparison, since only Firefox has client-side support for ORIGI"
+IEEE-SP   phishfarm-a-scalable-framework-for-measuring-the-effectiveness "We freshly installed the latest stable version of each browser at the time of each test and kept default secur"
+NDSS      we-value-your-privacy-now-take-some-cookies-measuring-the-gdpr "using the latest version of Firefox (version 57 onward)"
+WWW       uncovering-http-header-inconsistencies-and-the-impact-on-deskt "there are notable differences in the HSTS header and the HTTPS Redirection configuration where the newer versi"
+CCS       a-nearly-four-year-longitudinal-study-of-search-engine-poisoni "we use an automated (command-line) script, increasing the level of automation in collecting search results."
+-- vantage: 2 papers state the value changed or was refreshed between waves
+USENIX    characterizing-the-nature-and-dynamics-of-tor-exit-blocking    "several of the selected exit relays intermittently went offline, with a total of 0, 12, 19, and 28 offline dur"
+CCS       seeing-through-network-protocol-obfuscation                    "In addition, we employed two other campus network traces, which we call CloudDataset and WifiDataset. These we"
+-- state: 0 papers state the value changed or was refreshed between waves
+-- list: 6 papers vary it across waves BY DESIGN
+CCS       surf-detecting-and-measuring-search-poisoning
+NDSS      thou-shalt-not-depend-on-me-analysing-the-use-of-outdated-java
+USENIX    how-the-web-tangled-itself-uncovering-the-history-of-client-si
+USENIX    web-cache-deception-escalates
+PETS      website-data-transparency-in-the-browser
+PETS      more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settin
+-- browser: 14 papers vary it across waves BY DESIGN
+IEEE-SP   cloak-of-visibility-detecting-when-machines-browse-a-different
+NDSS      time-does-not-heal-all-wounds-a-longitudinal-analysis-of-secur
+USENIX    phishtime-continuous-longitudinal-measurement-of-the-effective
+WWW       apophanies-or-epiphanies-how-crawlers-impact-our-understanding
+IMC       muzeel-assessing-the-impact-of-javascript-dead-code-eliminatio
+USENIX    leaky-forms-a-study-of-email-and-password-exfiltration-before-
+USENIX    a-study-of-multi-factor-and-risk-based-authentication-availabi
+USENIX    glowing-in-the-dark-uncovering-ipv6-address-discovery-and-scan
+IEEE-SP   targeted-and-troublesome-tracking-and-advertising-on-childrens
+NDSS      cross-origin-web-attacks-via-http-2-server-push-and-signed-htt
+PETS      intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-bann
+PETS      clicking-into-exposure-uncovering-privacy-risks-of-google-clic
+PETS      privacy-vs-profit-the-impact-of-googles-manifest-version-3-mv3
+USENIX    bridges-to-self-silent-web-to-app-tracking-on-mobile-via-local
+-- vantage: 20 papers vary it across waves BY DESIGN
+CCS       a-nearly-four-year-longitudinal-study-of-search-engine-poisoni
+NDSS      parking-sensors-analyzing-and-detecting-parked-domains
+IEEE-SP   cloak-of-visibility-detecting-when-machines-browse-a-different
+IMC       recommended-for-you-a-first-look-at-content-recommendation-net
+WWW       beyond-the-front-page-measuring-third-party-dynamics-in-the-fi
+IMC       polls-clickbait-and-commemorative-2-bills-problematic-politica
+USENIX    scalable-detection-of-promotional-website-defacements-in-black
+IMC       muzeel-assessing-the-impact-of-javascript-dead-code-eliminatio
+PETS      from-onion-not-found-to-guard-discovery
+USENIX    a-large-scale-investigation-into-geodifferences-in-mobile-apps
+USENIX    leaky-forms-a-study-of-email-and-password-exfiltration-before-
+USENIX    a-study-of-multi-factor-and-risk-based-authentication-availabi
+USENIX    know-your-cybercriminal-evaluating-attacker-preferences-by-mea
+USENIX    timeless-timing-attacks-and-preload-defenses-in-tors-dns-cache
+IEEE-SP   targeted-and-troublesome-tracking-and-advertising-on-childrens
+PETS      a-large-scale-study-of-cookie-banner-interaction-tools-and-the
+PETS      website-data-transparency-in-the-browser
+PETS      unmasking-the-shadows-a-cross-country-study-of-online-tracking
+NDSS      characterizing-the-implementation-of-censorship-policies-in-ch
+PETS      clicking-into-exposure-uncovering-privacy-risks-of-google-clic
+-- state: 8 papers vary it across waves BY DESIGN
+NDSS      i-do-not-know-what-you-visited-last-summer-protecting-users-fr
+USENIX    investigating-commercial-pay-per-install-and-the-distribution-
+WWW       cookie-swap-party-abusing-first-party-cookies-for-web-tracking
+USENIX    web-cache-deception-escalates
+CCS       cookiegraph-understanding-and-detecting-first-party-tracking-c
+IMC       browsing-without-third-party-cookies-what-do-you-see
+PETS      intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-bann
+PETS      clicking-into-exposure-uncovering-privacy-risks-of-google-clic
+==============================================================================
+. CODER CONFIDENCE AND THE ALL-FOUR EXEMPLAR LIST
+==============================================================================
+-- self-reported confidence
+   population: 250 papers
+   high      116   46.4%
+   medium    131   52.4%
+   low         3    1.2%
+-- the 18 papers estimated to hold all four fixed (any repeatKind), most recent first
+   year | venue | slug | repeatKind | waves | explicit pins
+| PETS | more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr | own-repeat-crawl | 5 | list,vantage,state
+| IEEE-SP | sok-decoding-the-enigma-of-encrypted-network-traffic-classifiers | own-repeat-crawl | 100 | list,browser,vantage,state
+| USENIX | navigating-cookie-consent-violations-across-the-globe | own-repeat-crawl | 10 | list,vantage
+| USENIX | fledging-will-continue-until-privacy-improves-empirical-analysis-of-googles-priv | own-repeat-crawl | 3 | vantage
+| IMC | tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem | own-repeat-crawl | 31 | list,browser,vantage,state
+| PETS | blocking-javascript-without-breaking-the-web-an-empirical-investigation | own-repeat-crawl | 2 | list,state
+| PETS | raven-stateless-rapid-ip-address-variation-for-enterprise-networks | own-repeat-nonweb | ? | state
+| WWW | measuring-the-privacy-vs-compatibility-trade-off-in-preventing-third-party-state | own-repeat-crawl | 2 | list,vantage,state
+| WWW | towards-realistic-and-reproducibleweb-crawl-measurements | own-repeat-crawl | 3 | list
+| NDSS | encrypted-dns-privacy-a-traffic-analysis-perspective | own-repeat-crawl | 5 | -
+| PETS | a-comparative-measurement-study-of-web-tracking-on-mobile-and-desktop-environmen | own-repeat-crawl | 2 | list,browser,state
+| PETS | missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix | parallel-not-temporal | ? | -
+| PETS | no-boundaries-data-exfiltration-by-third-parties-embedded-on-web-pages | single-crawl | ? | -
+| WWW | a-first-look-at-commercial-5g-performance-on-smartphones | own-repeat-crawl | ? | state
+| WWW | the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing | own-repeat-crawl | 44 | list,browser,vantage
+| WWW | before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private | own-repeat-crawl | 21 | list
+| WWW | measuring-political-personalization-of-google-news-search | own-repeat-crawl | 7 | vantage
+| WWW | auditing-the-personalization-and-composition-of-politically-related-search-engin | own-repeat-crawl | 6 | vantage
+   of those, hand-confirmed own repeat crawls: 15
+| PETS | more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
+| IEEE-SP | sok-decoding-the-enigma-of-encrypted-network-traffic-classifiers
+| USENIX | navigating-cookie-consent-violations-across-the-globe
+| USENIX | fledging-will-continue-until-privacy-improves-empirical-analysis-of-googles-priv
+| IMC | tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem
+| PETS | blocking-javascript-without-breaking-the-web-an-empirical-investigation
+| WWW | measuring-the-privacy-vs-compatibility-trade-off-in-preventing-third-party-state
+| WWW | towards-realistic-and-reproducibleweb-crawl-measurements
+| NDSS | encrypted-dns-privacy-a-traffic-analysis-perspective
+| PETS | a-comparative-measurement-study-of-web-tracking-on-mobile-and-desktop-environmen
+| WWW | a-first-look-at-commercial-5g-performance-on-smartphones
+| WWW | the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing
+| WWW | before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private
+| WWW | measuring-political-personalization-of-google-news-search
+| WWW | auditing-the-personalization-and-composition-of-politically-related-search-engin
+==============================================================================
+. WHAT HAPPENED TO THE 23 PAPERS THE OLD CEILING TABLE LISTED
+==============================================================================
+papers stated all four. Hand-coding each:
+   year venue      slug                                        kind                 list/browser/vantage/state
+PETS       clicking-into-exposure-uncovering-privacy-r parallel-not-temporal same-implied / varied-deliberate / varied-deliberate / varied-deliberate
+PETS       more-space-less-privacy-measuring-the-effec own-repeat-crawl     same-explicit / same-implied / same-explicit / same-explicit
+PETS       privacy-vs-profit-the-impact-of-googles-man own-repeat-crawl     same-implied / varied-deliberate / same-explicit / same-implied
+PETS       beyond-the-request-harnessing-http-response own-repeat-crawl     changed / same-implied / unclear / unclear
+PETS       intractable-cookie-crumbs-unveiling-the-nex own-repeat-crawl     same-explicit / varied-deliberate / same-explicit / varied-deliberate
+PETS       more-and-scammier-ads-the-perils-of-youtube own-repeat-crawl     varied-deliberate / same-implied / same-explicit / same-implied
+USENIX     navigating-cookie-consent-violations-across own-repeat-crawl     same-explicit / same-implied / same-explicit / same-implied
+WWW        before-after-the-effect-of-eus-2022-code-of own-repeat-crawl     same-implied / not-stated / not-stated / not-stated
+IEEE-SP    targeted-and-troublesome-tracking-and-adver parallel-not-temporal same-implied / varied-deliberate / varied-deliberate / same-implied
+PETS       a-large-scale-study-of-cookie-banner-intera parallel-not-temporal same-explicit / same-explicit / varied-deliberate / same-explicit
+PETS       the-devil-is-in-the-details-detection-measu own-repeat-crawl     same-implied / changed / same-explicit / same-explicit
+USENIX     a-study-of-multi-factor-and-risk-based-auth own-repeat-crawl     same-explicit / varied-deliberate / varied-deliberate / same-explicit
+IMC        respect-the-origin-a-best-case-evaluation-o own-repeat-crawl     same-explicit / changed / same-explicit / same-explicit
+USENIX     a-large-scale-investigation-into-geodiffere parallel-not-temporal same-implied / same-implied / varied-deliberate / same-explicit
+WWW        reproducibility-and-replicability-of-web-me own-repeat-crawl     same-implied / not-stated / not-stated / unclear
+NDSS       encrypted-dns-privacy-a-traffic-analysis-pe own-repeat-crawl     same-implied / same-implied / same-implied / same-implied
+PETS       missed-by-filter-lists-detecting-unknown-th parallel-not-temporal same-implied / same-implied / same-implied / same-implied
+WWW        beyond-the-front-page-measuring-third-party parallel-not-temporal same-explicit / same-explicit / varied-deliberate / same-implied
+WWW        the-representativeness-of-automated-web-cra own-repeat-crawl     same-explicit / same-explicit / same-explicit / same-implied
+WWW        measuring-political-personalization-of-goog own-repeat-crawl     same-implied / same-implied / same-explicit / same-implied
+NDSS       thou-shalt-not-depend-on-me-analysing-the-u parallel-not-temporal varied-deliberate / same-implied / same-explicit / same-explicit
+IEEE-SP    cloak-of-visibility-detecting-when-machines parallel-not-temporal same-explicit / varied-deliberate / varied-deliberate / same-explicit
+USENIX     tracing-information-flows-between-ad-exchan own-repeat-crawl     same-explicit / same-implied / same-explicit / not-stated
+   survive as all-four-fixed (estimate): 6 / 23
+</file>
+**Unedited output of the quote check.**
+<file  pins_quotecheck-output.txt>
+design:longitudinal four-pin audit — evidence quote check
+coded dir: out_pins/coded
+renderings searched per paper: paper.cols.txt, paper.norm.txt, paper.txt, and the excerpt packet
+fallback for a quote found in none of those: a pypdf rendering of paper.pdf (scripts/pdftext.py), same four modes, reported as RESCUED
+de-splicing mode: 6-word shingles, pass at >= 80% found
+papers with no coding file (skipped): 0
+pin codings:            1000
+   no quote given:      464   (permitted only for not-stated; checked below)
+   checked:             536
+      exact match:                 489  91.2%
+      whitespace-collapsed match:  0  0.0%
+      loose match:                 2  0.4%
+      spliced match (de-spliced):  3  0.6%
+      RESCUED from the PDF:        35  6.5%
+      NOT FOUND:                   7  1.3%
+   (papers with no pypdf rendering available: 0)
+codings with no quote and a value other than not-stated: 0
+MATCHED ONLY AFTER DE-SPLICING (3) — these are read by hand, the source page is two-column spliced:
+/WWW/automatic-discovery-of-emerging-browser-fingerprinting-techniques  [browser=same-explicit]  shingles 86%
+      applied our system to this data set by following the same procedure.
+/WWW/automatic-discovery-of-emerging-browser-fingerprinting-techniques  [vantage=same-explicit]  shingles 86%
+      applied our system to this data set by following the same procedure.
+/NDSS/parking-sensors-analyzing-and-detecting-parked-domains  [list=same-implied]  shingles 93%
+      we sampled 100 domains from each parking service and setup an additional crawler which crawled them daily for a week.
+RESCUED — in none of the stored renderings, located in the pypdf rendering of paper.pdf (35).
+These are defects in the stored text, not in the hand coding:
+/IEEE-SP/third-party-web-tracking-policy-and-technology  [browser=same-implied]  stored shingles 75% -> pdf exact
+      We implemented FourthParty as an extension to Mozilla Firefox.
+/NDSS/parking-sensors-analyzing-and-detecting-parked-domains  [vantage=varied-deliberate]  stored shingles 47% -> pdf exact
+      To assess any geographical differences, we performed this crawl in parallel from the United States and one country from Europe.
+/USENIX/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst  [state=same-implied]  stored shingles 38% -> pdf exact
+      We managed to emulate typical browser behavior in all but one case (BlueSky).
+/CCS/poster-probing-tor-hidden-service-with-dockers  [list=same-explicit]  stored shingles 69% -> pdf exact
+      The number of Onion addresses in this experiment was 16,683, and the addresses were crawled twice a day at 12-hour intervals.
+/CCS/how-you-get-shot-in-the-back-a-systematical-study-about-cryptojacking-in-the-rea  [list=same-implied]  stored shingles 50% -> pdf exact
+      we can only continuously monitor 1,000 samples. Besides, we re-visit these samples every 3 days
+/IMC/visiblev8-in-browser-monitoring-of-javascript-in-the-wild  [list=same-explicit]  stored shingles 0% -> pdf exact
+      to visit the Alexa top 50k web domains
+/WWW/a-first-look-at-deep-learning-apps-on-smartphones  [list=changed]  stored shingles 52% -> pdf exact
+      We have crawled two datasets at different moments, June 2018 and September 2018, which are three months apart. The two app datasets have more than 2/3 overlapped apps.
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [browser=same-implied]  stored shingles 43% -> pdf exact
+      We installed the widely used OpenWPM framework and configured it to use four vanilla Mozilla Firefox browsers in parallel
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [vantage=same-implied]  stored shingles 50% -> pdf exact
+      the physical machine on which the VM was deployed is located in the EU, meaning that the visited websites must comply with GDPR
+/WWW/before-and-after-gdpr-the-changes-in-third-party-presence-at-public-and-private  [state=same-implied]  stored shingles 43% -> pdf exact
+      we did not install any extension on the browser that would stop the websites from storing or reading cookies
+/WWW/mobile-app-risk-ranking-via-exclusive-sparse-coding  [list=changed]  stored shingles 77% -> pdf spliced 91%
+      There are no overlaps for the above datasets, and the apps are from a wide range of functionalities, including game, news, music, social, movies/TV, travel, life, etc.
+/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [vantage=varied-deliberate]  stored shingles 53% -> pdf exact
+      we conduct the measurements from three different locations (Europe (DE), North America (US), and Asia (JP)) to account for possible geographical differences
+/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field  [state=same-implied]  stored shingles 53% -> pdf exact
+      We used the same method to create the browser profile for our experiment crawls that we utilized in the pre-study.
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [list=same-explicit]  stored shingles 0% -> pdf exact
+      Tranco list of top sites [18] (snapshot 77PX)
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [browser=same-implied]  stored shingles 33% -> pdf exact
+      We crawl using two variants of Puppeteer [4] controlling Chromium 80: a lower-realism naive variant and a higher-realism stealth variant.
+/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [browser=same-implied]  stored shingles 50% -> pdf exact
+      we rely on a fully-featured, headless Chrome browser, governed by Selenium.
+/WWW/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems  [vantage=same-implied]  stored shingles 47% -> pdf exact
+      ODIN uses university IP addresses (one per profile) and a /24 subnet from a research-friendly, but less well-known VPS provider
+/IMC/muzeel-assessing-the-impact-of-javascript-dead-code-elimination-on-mobile-web-pe  [browser=varied-deliberate]  stored shingles 17% -> pdf exact
+      We now evaluate Muzeel across browsers: Chrome, Edge, Firefox, and Brave.
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [list=same-explicit]  stored shingles 57% -> pdf loose
+      Our initial sample set was the 5000 domains with the most requests to the third party, as indicated by the Referer field in the CDN's third party logs.
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [browser=changed]  stored shingles 38% -> pdf exact
+      Results shown are for Firefox (v91) for later comparison, since only Firefox has client-side support for ORIGIN Frame.
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild  [state=same-explicit]  stored shingles 57% -> pdf exact
+      Client caches are intentionally deleted between measurements in all of our active measurements to prevent bias from caching effects
+/USENIX/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements  [list=same-implied]  stored shingles 59% -> pdf exact
+      We scraped Hansa 17 times between late 2015 and mid-2017, collecting a total of 332,795 pages amounting to 39.5 GB of data.
+/WWW/leveraging-googles-publisher-specific-ids-to-detect-website-administration  [list=changed]  stored shingles 58% -> pdf exact
+      We perform our analysis on websites which have been crawled in all snapshots (i.e., their intersection) and contain at least one Publisher ID. There are over 191,000 such websites.
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [list=same-explicit]  stored shingles 63% -> pdf exact
+      We successfully conducted the four parallel crawls for 99.31% of the 20K websites.
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [vantage=same-explicit]  stored shingles 71% -> pdf exact
+      We run all crawls in the US to minimize the impact of the EU GDPR and do not interact with cookie banners.
+/WWW/automatic-discovery-of-emerging-browser-fingerprinting-techniques  [list=same-explicit]  stored shingles 59% -> pdf exact
+      we redid the crawling on the same 10K domains in September 2022 (11-month gap) and applied our system to this data set by following the same procedure.
+/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [list=same-implied]  stored shingles 57% -> pdf exact
+      clients repeatedly accessed onion services and the top 150 accessed websites according to the Tranco ranking [68], in parallel.
+/NDSS/flow-correlation-attacks-on-tor-onion-service-sessions-with-sliding-subset-sum  [browser=same-explicit]  stored shingles 61% -> pdf exact
+      Clients also run on an isolated Docker container that executes a 0.4.7.10 Tor process and issues requests to Tor via the Python libraries selenium [58] and tbselenium [3]
+/IMC/learning-as-to-organization-mappings-with-borges  [list=same-implied]  stored shingles 64% -> pdf loose
+      we used the PeeringDB snapshot from July 24, 2024, and CAIDA's AS2Org from July 1, 2024.
+/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [browser=varied-deliberate]  stored shingles 48% -> pdf loose
+      Since OpenWPM only supports Firefox and, unlike Chrome, Firefox does not expose the partitioned_key field in its cookie storage, we modified BannerClick to operate in Chrome's default mode.
+/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and  [state=varied-deliberate]  stored shingles 70% -> pdf exact
+      We use OpenWPM [24] to conduct a combination of stateful and stateless crawls aiming to observe the interplay of the setter and the sender website
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [list=same-implied]  stored shingles 55% -> pdf exact
+      we joined the top one million websites from Tranco [49] with SimilarWeb, a web traffic estimator
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [browser=same-implied]  stored shingles 56% -> pdf exact
+      Each website was opened in a Chrome instance with two installed custom Chrome extensions.
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration  [vantage=same-implied]  stored shingles 47% -> pdf loose
+      All website visits took place on Linux virtual machines in Google Cloud's US Central region between September and November 2024.
+/PETS/who-s-watching-you-zoom-investigating-privacy-of-third-party-zoom-apps  [list=changed]  stored shingles 40% -> pdf exact
+      between December 2023 and December 2024, 667 new apps were added, 212 apps were removed, and 49 apps were renamed
+QUOTES NOT LOCATED, printed in full (7):
+/WWW/understanding-and-combating-link-farming-in-the-twitter-social-network  [list=same-explicit]  shingles 50%
+      We re-crawled the profile page of each user in the above dataset in February 2011
+/WWW/stress-testing-the-booters-understanding-and-undermining-the-business-of-ddos-se  [list=same-implied]  shingles 58%
+      In order to measure the stability of these amplifiers we probed them periodically for 13 weeks to understand how many were still located at the same IP and misconfiged.
+/WWW/towards-realistic-and-reproducibleweb-crawl-measurements  [state=same-implied]  shingles 70%
+      the browser is launched with a clean user profile (i.e., no cookies or cached content)
+/PETS/a-global-survey-of-android-dual-use-applications-used-in-intimate-partner-survei  [list=same-explicit]  shingles 52%
+      we crawled Google Play again in all fifteen languages for one day in February 2022 with the queries we obtained and used in our 2020 crawling.
+/PETS/atom-ad-network-tomography  [state=same-implied]  shingles 79%
+      Each persona was associated with a unique browser running on an isolated virtual machine with a unique IP address.
+/CCS/cookiegraph-understanding-and-detecting-first-party-tracking-cookies  [state=varied-deliberate]  shingles 46%
+      We conduct four parallel crawls: two with third-party cookies enabled (3P-Allowed) and two with third-party cookies blocked (3P-Blocked).
+/IMC/somesite-i-used-to-crawl-awareness-agency-and-efficacy-in-protecting-content-cre  [list=same-explicit]  shingles 77%
+      We restrict the list to sites that appear in all of the top 100k lists over this period to avoid having our results affected by list churn
+</file>
+**Unedited output of the agreement check**, including every disagreement between the two coders.
+<file  pins_agreement-output.txt>
+design:longitudinal four-pin hand-coding — inter-coder agreement
+coder A (excerpt packets): out_pins/coded
+coder B (full text, blind): out_pins/coded_val
+reliability sample: 30 papers drawn from the 250
+pairs available:    30
+==============================================================================
+PER-PIN AGREEMENT (n=30 papers)
+==============================================================================
+   pin                exact      fixed/not   kappa(exact)   kappa(fixed)
+   list       20/30 (66.7%)  26/30 (86.7%)           0.49           0.59
+   browser    21/30 (70.0%)  22/30 (73.3%)           0.57           0.48
+   vantage    21/30 (70.0%)  25/30 (83.3%)           0.56           0.67
+   state      24/30 (80.0%)  25/30 (83.3%)           0.64           0.64
+   ALL PINS  86/120 (71.7%) 98/120 (81.7%)           0.60           0.64
+-- repeatKind
+   exact 25/30 (83.3%)   kappa 0.77
+   own-repeat-crawl vs everything else: 27/30 (90.0%)   kappa 0.80
+-- "all four fixed", the figure the page publishes
+   A says all four fixed: 3/30
+   B says all four fixed: 4/30
+   agree:                 27/30 (90.0%)   kappa 0.52
+==============================================================================
+DIRECTION OF DISAGREEMENT — does the excerpt coder over- or under-call "fixed"
+==============================================================================
+   list      both fixed  22   only A (excerpt)   4   only B (full text)   0   neither   4
+   browser   both fixed   9   only A (excerpt)   7   only B (full text)   1   neither  13
+   vantage   both fixed  11   only A (excerpt)   1   only B (full text)   4   neither  14
+   state     both fixed   8   only A (excerpt)   4   only B (full text)   1   neither  17
+==============================================================================
+EVERY DISAGREEMENT, PRINTED IN FULL
+==============================================================================
+/CCS/protocol-misidentification-made-easy-with-format-transforming-encryption
+   repeatKind   A=single-crawl   B=own-repeat-nonweb
+   list         A=same-implied       B=unclear
+   browser      A=same-implied       B=not-stated
+   vantage      A=not-stated         B=same-implied
+   state        A=same-implied       B=not-stated
+/CCS/seeing-through-network-protocol-obfuscation
+   list         A=not-stated         B=changed
+/IMC/location-location-location-the-impact-of-geolocation-on-web-search-personalizati
+   browser      A=same-implied       B=same-explicit
+   vantage      A=not-stated         B=same-explicit
+/IMC/a-first-look-at-quality-of-mobile-live-streaming-experience-the-case-of-periscop
+   list         A=same-implied       B=not-stated
+   vantage      A=same-implied       B=not-stated
+   state        A=same-implied       B=not-stated
+/IEEE-SP/the-spyware-used-in-intimate-partner-violence
+   repeatKind   A=own-repeat-crawl   B=own-repeat-nonweb
+   browser      A=same-implied       B=not-stated
+   state        A=same-implied       B=not-stated
+/WWW/spiders-like-onions-on-the-network-of-tor-hidden-services
+   browser      A=same-implied       B=not-stated
+/WWW/apophanies-or-epiphanies-how-crawlers-impact-our-understanding-of-the-web
+   vantage      A=same-implied       B=same-explicit
+/IMC/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild
+   list         A=same-explicit      B=same-implied
+   vantage      A=same-explicit      B=same-implied
+/USENIX/a-large-scale-investigation-into-geodifferences-in-mobile-apps
+   repeatKind   A=parallel-not-temporal   B=own-repeat-nonweb
+   browser      A=same-implied       B=not-stated
+   vantage      A=varied-deliberate  B=same-implied
+   state        A=same-explicit      B=not-stated
+/IEEE-SP/fashion-faux-pas-implicit-stylistic-fingerprints-for-bypassing-browsers-anti-fin
+   list         A=same-implied       B=not-stated
+   state        A=not-stated         B=same-implied
+/IMC/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem
+   browser      A=same-explicit      B=not-stated
+/PETS/two-steps-forward-and-one-step-back-the-right-to-opt-out-of-sale-under-cpra
+   browser      A=same-explicit      B=unclear
+   vantage      A=same-explicit      B=same-implied
+/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts
+   list         A=same-explicit      B=same-implied
+/IMC/toward-better-efficiency-vs-fidelity-tradeoffs-in-web-archives
+   list         A=same-explicit      B=same-implied
+/PETS/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and
+   list         A=same-explicit      B=varied-deliberate
+/PETS/tracker-installations-are-not-created-equal-understanding-tracker-configuration
+   repeatKind   A=own-repeat-crawl   B=single-crawl
+/IEEE-SP/parasites-in-the-toolchain-a-large-scale-analysis-of-attacks-on-the-mcp-ecosyste
+   vantage      A=not-stated         B=same-implied
+/NDSS/cellshift-rtt-aware-trace-transduction-for-real-world-website-fingerprinting
+   list         A=same-implied       B=same-explicit
+   browser      A=same-implied       B=not-stated
+/NDSS/characterizing-the-implementation-of-censorship-policies-in-chinese-llm-services
+   repeatKind   A=own-repeat-crawl   B=own-repeat-nonweb
+   vantage      A=varied-deliberate  B=unclear
+/PETS/privacy-vs-profit-the-impact-of-googles-manifest-version-3-mv3-update-on-ad-bloc
+   list         A=same-implied       B=same-explicit
+   browser      A=varied-deliberate  B=same-implied
+   state        A=same-implied       B=same-explicit
+papers with at least one disagreement: 20/30 (66.7%)
+==============================================================================
+VALUE DISTRIBUTION, EACH CODER (so a dominant value is visible)
+==============================================================================
+   value                     A      B
+   same-explicit            28     25
+   same-implied             38     31
+   varied-deliberate         6      4
+   changed                   4      5
+   not-stated               44     52
+   unclear                   0      3
+</file>
 [[design:longitudinal|← back to the content page]] · [[provenance:design:website_selection|the 2026-09-03 Alexa date audit]] · [[literature:corpus|corpus-level provenance]]
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Quote-check refresh, 2026-09-22 =====
+**What was wrong.** ''scripts/pins_quotecheck.mjs'' searched four texts — the three
+stored renderings and the excerpt packet — and had **no ''pypdf'' fallback**. The
+-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] measured that
+.1% of quotes unlocatable in ''paper.cols.txt'' are present in an independent
+''pypdf'' rendering of the same ''paper.pdf'', and ''scripts/quote_check.mjs'' was
+patched that day to report a **RESCUED** verdict. This checker was never patched, so
+§18.5's unlocated bucket was an artefact of a narrower search than the rest of the
+site was using. It has now been patched the same way and re-run.
+<code>
+$ node scripts/pins_quotecheck.mjs
+      exact match:                 489  91.2%
+      whitespace-collapsed match:  0  0.0%
+      loose match:                 2  0.4%
+      spliced match (de-spliced):  3  0.6%
+      RESCUED from the PDF:        35  6.5%
+      NOT FOUND:                   7  1.3%
+   (papers with no pypdf rendering available: 0)
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| pin codings | 1,000 | 1,000 | population unchanged — no coding was touched |
+| checked (non-empty quote) | 536 | 536 | unchanged |
+| exact | 489 | 489 | unchanged |
+| whitespace-collapsed | 0 | 0 | unchanged |
+| loose | 2 | 2 | unchanged |
+| de-spliced (6-word shingles, ≥80%) | 3 | 3 | unchanged |
+| rescued from the PDF | — | **35** | new verdict; all 35 were inside the old 42 |
+| located by no mode | **42** (7.8%) | **7** (1.3%) | 42 = 35 + 7 exactly; nothing else moved |
+**The hand-audited fabrication rate is unchanged at 0.2%, and this run is evidence
+for it rather than a replacement of it.** The 42 had already been read by three
+sub-agents, who called 42 of the 43 faithful de-splices of real sentences and found
+**one** real defect — a quote from the wrong part of {[singanamalla2022_respect]} —
+and **zero** fabrications. The fallback now locates **35 of those same 42** in a
+second rendering, by a script that has never seen the hand verdicts. It contradicts
+none of them. The 7 that remain are all inside the hand-audited set and all carry the
+verdict //faithful de-splice//. **1.3% is not a fabrication rate either**; like 7.8%
+and 9.1% before it, it is an upper bound produced by an instrument, and the rate is
+still the hand-read **0.2% (1 of 536)**.
+**The mutation test was re-run and its pin moved.** The guard recorded in §18.11 was
+//"an injected fabricated quote raises NOT FOUND from 42 to 43"//, which pinned the
+literal 42. Injecting a fabricated sentence into one coding of
+''2012/IEEE-SP/third-party-web-tracking-policy-and-technology'' against the patched
+script raises NOT FOUND **from 7 to 8**, drops ''exact'' from 489 to 488, and prints
+the fabricated sentence in full in the unlocated list. The pin in §18.11 has been
+updated to 7 → 8 and the old value recorded beside it, because a guard that still
+asserts a superseded number is a guard that will pass for the wrong reason.
+**The fallback was verified to be doing work, not failing silently.** ''pdfText()''
+swallows a subprocess failure and returns ''null'', which would produce ''0 rescued''
+and look like a clean result. The run therefore prints the number of papers for which
+no ''pypdf'' rendering could be produced — **0** — and two rescues were confirmed by
+hand against ''scripts/pdftext.py'' output directly: ''2019/IMC/visiblev8-…''
+(//"to visit the Alexa top 50k web domains"//, 0% of shingles in the stored text,
+verbatim in the PDF) and ''2021/WWW/towards-realistic-and-reproducibleweb-crawl-measurements''
+(//"Tranco list of top sites [18] (snapshot 77PX)"//, likewise). A ''0 rescued'' from
+this script would now be a measured zero.
+**Scope of this edit.** §18.5's mode table and the four sentences around it, the
+§18.11 mutation-test pin, and the two ''%%<file>%%'' blocks — the published script and
+its published output — both regenerated from the committed files rather than edited.
+''report_pins_audit.mjs'', ''pins_agreement.mjs'' and ''report_longitudinal.mjs'' were
+**not** re-run: no coding, no population and no agreement figure is touched by this
+change, and the quote check feeds none of them. Nothing on
+[[:design:longitudinal]] carries a quote-check figure, so no content page was saved.
+No ''%%{[citekey]}%%'' was added or moved.
+^ Item ^ Value ^
+| Date | 2026-09-22, unsupervised |
+| Command | ''node scripts/pins_quotecheck.mjs'' |
+| Artifacts | ''out/qc0922/pins_quotecheck-output.txt'' (the run published above), ''out/qc0922/pins_quotecheck_mutation.txt'' (the mutation run, NOT FOUND 8) |
+| Script change | ''scripts/pins_quotecheck.mjs'': imports ''pdfText'' from ''scripts/pdftext_cache.mjs''; a quote in none of the four stored texts is retried in all four modes against the ''pypdf'' rendering and reported as ''RESCUED''; the shingle test was factored into one ''shingleFrac()'' used by both passes; the summary gained a RESCUED line and a count of papers with no ''pypdf'' rendering |
+| Source fragment | ''prov/pins_section.txt'' carries the same §18.5 and §18.11 prose changes, so ''build_provenance_pins.py'' regenerates this section rather than reverting it. This appended refresh section lives on the page only. ''pages/provenance_design_longitudinal.LIVE.txt'' is a 2026-09-07 snapshot and was **not** refreshed — the builder's ''%%--live%%'' input is stale for reasons that predate this edit |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Reviewer finding accepted | The pass caught that this section's **heading** still read //"the 43 that did not locate"// — a count three generations stale (43 before the 2026-09-07 correction, 42 before today's fallback, 7 now). The prose below it narrates all three, so no reader was misled, but a heading is also an anchor and a TOC entry. It has been renamed to carry **no count at all**, because a heading that names a figure is a figure nobody re-runs. No page links to the old anchor — checked across all 188 pages before renaming |
+| Pages saved | this page only |