Working notes behind longitudinal — every query with its population and denominator, the report script and its unedited output, the fold and its full residue, the quotes that were checked, the probes that were hand-audited, the external sources that were verified or rejected, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Contemporaneous. Written during the run that produced the content page, 2026-08-27, not reconstructed afterwards.

Four existing pages own a piece of this subject, and the judgement was that none of them owns the whole:

The gap none of them fills is the one between “I know how to draw a sample” and “wave two is comparable to wave one”: what has to be pinned, what cannot be pinned however hard you try, and what the literature actually reports. That is the page. Every overlapping claim links out rather than restating — the four papers whose variation figures appear in the noise-floor table are all already cited on statistics:biases, and the EasyList-anachronism figure is already on design:archives; both are re-used with a pointer because the longitudinal reading of them is different from the bias reading.

One consequence accepted deliberately: the content page's noise-floor table duplicates rows that appear on statistics:biases. A reader arriving from a search engine at either page should not have to visit the other to learn the number, and the framing differs (there: “your sample is not the population”; here: “this is your wave-to-wave error bar”).

Every figure on the content page is a share of one of these. crawled uses the same definition as data/extract/OVERVIEW.md so the numbers line up with the rest of the site.

The repeated rule is deliberately identical to the one biases uses, so the two pages agree on 250 rather than publishing two different counts of the same thing.

Sentinels. not-stated, not-applicable, none-mentioned, unknown are never counted as a stated value. This matters most for two rows: counting not-applicable as an answer would move consent action stated from 105 to 148 on the repeated population and statefulness stated from 149 to 166 on the single-shot one, which would have put this page out of step with OVERVIEW.md (219 and 349 over crawled respectively — reproduced exactly here as 70+149 and 105+244).

“Stated” is not “held fixed.” Every pin figure counts papers that mentioned the thing once. The extraction cannot see whether a paper used the same browser in wave one and wave two. Every pin figure is therefore a ceiling on comparability, and the content page says so in the same table.

cd /workspace/artifacts/wiki
node scripts/report_longitudinal.mjs > scripts/report_longitudinal-output.txt
node scripts/quotecheck_longitudinal.mjs > scripts/quotecheck_longitudinal-output.txt

Both default to /workspace/publications_dataset/data/extract/run1 and …/data/fulltext; –run and –fulltext override.

temporal.cadence is the only free-text field this page aggregates. 334 distinct strings fire on the 1,120 crawling papers. The fold is ordered fastest-interval-first, so “hourly and daily” folds to hourly — the fastest repeat a paper states is the one that bounds its time resolution.

The first rule is the interesting one. A large minority of the values are durations, not cadences: “one week”, “5-day experiment”, “month-long”, “single crawl lasting 12 days”. Those papers have not stated how often they repeated, and folding them into weekly or monthly would have converted a reporting gap into a finding about cadence. They get their own family and the content page reports them as silence.

After folding, 16 distinct strings (16 papers, 4.1% of the 389) remain unmapped. They are printed in full in the report output below, under UNMAPPED RESIDUE, and reproduced here so they are readable without running anything:

four increasing cumulative periods for threshold selection
four APKs per year per browser family
July and December
15-minute trend monitoring
one hour per website
two six-month intervals per year for 2009–2019
longitudinal measurement
eleven-minute time slots
Crawler B ran at the middle of each month
irregular confirmation crawls
collection cycles over six months
during the day, paused overnight
December 2023, May 2024, and December 2024
sequential experimental windows
four-month interval
before and after the second rollout

Most of those are genuinely ambiguous rather than mis-folded: “four-month interval” could be a cadence or a span, and the paper has to be read to tell. They are left unmapped rather than guessed at.

One value in the field is a sentinel. A single paper has cadence: “not-stated”. It is folded to sentinel and excluded from every cadence figure.

The double-counting trap, and how the published figure avoids it. The families are multi-valued: a paper stating both “daily” and “for three weeks” lands in two. Summing the non-cadence families (58 + 29 + 31 + 1 = 119) therefore over-counts. The figure published on the content page instead counts papers whose cadence values are all non-cadences: 88 of 389 (22.6%), with unmapped generously treated as a possible real cadence. Counting unmapped as a non-cadence would give a larger number; the smaller one is published.

Every quotation and every figure the content page takes from a cited paper is in scripts/quotecheck_longitudinal.mjs, whose unedited output is in §14. The checker reads four renderings of each paper: paper.cols.txt (two-column reading order repaired), paper.norm.txt, paper.txt, and paper.cols.txt with end-of-line hyphenation joined.

The verdicts mean:

No quote came back MISSING. Seven came back PARTIAL, every one for the same mechanical reason — a two-column interleave, a hyphen-split word, or both — and every one was read by hand against the source:

A correction the checker forced. An earlier draft of the lead said Demir et al. ran the twelve-day experiment “from the same place”. The paper's C4 paragraph says only “we visit the top 1k sites from our website corpus (18,377 distinct pages) daily (starting at midnight) throughout our experiment (from 07/08/2021 to 07/19/2021)”, and the study as a whole crawls from three geolocations. It does not say the temporal sub-analysis holds the vantage point fixed, so the claim was removed rather than assumed.

A figure the extraction offered and the page does not use. temporal.snapshots has nine papers reporting more than 1,000 snapshots — the largest is 550,000. Spot-reading them shows page counts and record counts mis-extracted into the field (an NDSS 2019 longitudinal browser study, a CCS 2025 robots.txt study). No maximum, mean or median of snapshots is published anywhere; only bands, and the band table carries the warning.

All probes collapse whitespace before matching, because a PDF line break inside a phrase would otherwise hide it. Every probe is a mention count and therefore an upper bound. Two probes carry a load-bearing claim on the content page and were therefore read in full.

The structured field crawlConfig.browsers contains a digit for 17 of 250 repeated-crawl papers (6.8%). That undercounts, because a paper can write “Chrome” in its setup table and “Chrome 88.0.4298.0” in a footnote. So a deliberately wide probe was run for any <browser> <number> string anywhere in the text: 58 hits (23.2%).

All 58 were read. 26 state the version of a browser the study's own measurement drove. The 32 rejected split into:

• bibliography entries — “Firefox 85 Cracks Down on Supercookies”, “About the security content of Safari 11.1”;
• ecosystem history — “Full support for this CSP feature first landed in browsers around March 2015 (with Chrome 43)”;
• the version of a browser being measured rather than driven — the NDSS 2019 mobile-browser study reports “Firefox v59.0 on Android 7.1” because those browsers are the subject;
• outright false positives — footnote markers (Chromium 1 and Firefox 2), citation markers (Firefox22), table cells whose numbers are accuracies or counts, and one “visited from a Chrome browser 100 times”.

So the honest range published on the content page is 17 (structured field) to 26 (text), of 250.

A wide probe for auto-update or browser-version drift returns 14 of 250. All 14 were read. Not one discusses pinning or reporting the crawler's own browser version across waves. The hits are: Windows Update traffic, phishing blacklist auto-updating, a cited paper title, mobile users auto-updating apps, WordPress auto-update, an ad-blocker list auto-update proposal, and Chromium's internal frame-process mapping being “updated”. The closest to the topic is a limitations note in a PETS 2020 paper: “Our measurement tools require constant maintenance to keep up with browser updates and interface changes.” That is about tool maintenance, not about wave comparability.

The content page states this as “not one of the 14 hits”, never as “nobody”, because a probe cannot support “nobody”.

These three are corpus-wide probes (all 5,855 readable papers), not restricted to repeated, because the question is whether the mechanism has any footprint in these seven venues at all. They live in §11 of the report script, and the script prints each regex beside its count — a different regex gives a different number, and hiding the regex would hide that.

• Chrome for Testing. The exact phrase returns 0. Widened to googlechromelabs, chrome labs, known-good-versions and the bare token CfT, it returns 10 — and reading all 10 shows every one is a false positive: Combating the Financing of Terrorism (three papers), Control Flow Trimming, Call Flow Tree, Crash-Fault Tolerant, a certificate organisation name O=CFT, a CFt variable in an algorithm listing, CFT-IDF, and a fitness-test app named CFT & PFT.
• Docker. /\bdocker\b/i gives 293 (5.0%); adding containerised/containerized gives 308 (5.3%). Both are printed, and the content page names which is which — an earlier draft published 308 as “mention Docker”, which is the wider figure under the narrower label.
• Image digest. /\bimage digest\b|\bsha256:[0-9a-f]{12}/i returns 0.
• WebREC. /\bWebREC\b|\bweb execution bundle/i returns 2, and reading both shows neither is a use: one is the WebREC paper, and the other cites it in its reference list only. An earlier draft said “two papers mention it” without that distinction, which reads as two adopters.

Every one of these is a mention count. A paper can pin a container digest in its repository and never say so in the text; these are reporting figures, not practice figures, and the content page says so.

Two wide probes were run over the 266 Tranco papers; their union was 37 candidates, and all 37 were read. 23 genuinely cite a Tranco list ID. The other 14 matched on a legal section number near the word Tranco (CAL. CIV. Code § 1798.120©), a reference-list code, or nothing identifiable. Two of the 23 were found only by the looser probe and would have been missed by the tighter one — mobileatlas writes “Tranco [25] list V78N” with no “ID” token, and autofr cites it only in the reference list. That is why the report script carries a written-out allowlist rather than recomputing the figure from a regex: no regex found all 23, and the script warns loudly if an allowlist entry stops matching a Tranco paper.

The 23, by year:

Everything below was fetched on 2026-08-27, from a primary source, not recalled.

• Whether a paper held a pin constant across waves. Closed on 2026-09-07 — see §18 below. All 250 were hand-coded rather than a sample, and the content page now publishes estimates beside the stated-value figures. What is left open by that work is in §18.8, and the largest residue is that same-implied — one configuration described and never contradicted — cannot be distinguished from a pin that silently moved.
• Whether the 2025–2026 slice changes the trend. The repeat rate is flat at roughly one crawling paper in five across four buckets, and the last bucket is provisional by construction. Nothing in the 2025–2026 material suggests a change, but those venue-years are the thinnest in the corpus.
• Whether pinning actually changes a published answer. The whole case rests on single-wave variation studies ([1Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)], [2Demir, Nurullah; Hörnemann, Jan; Große-Kampmann, Matteo; Urban, Tobias; Pohlmann, Norbert; Holz, Thorsten; Wressnegger, Christian (2023): "On the Similarity of Web Measurements Under Different Experimental Setups", in: Proceedings of the ACM Internet Measurement Conference, pp. 356-369. ACM DOI 10.1145/3618257.3624795 is listed by DBLP but was not registered with the DOI resolver as of 2026-08-12; the DOI above resolves to the authors' institutional record of the same paper (DOI)], [3Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)], [4Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)]). No study repeats one measurement under pinned and unpinned conditions over a year. Listed as an open question on the content page.
• A denominator for “how many longitudinal studies got their trend wrong”. Unknowable from this data and not attempted.

Every figure on the content page comes from this script. It prints each population with its denominator, folds the one free-text field it aggregates, and prints that fold's unmapped residue in full.

report_longitudinal.mjs

#!/usr/bin/env node
// Every figure on measuretheweb.org/design:longitudinal, with its denominator.
//
//   node scripts/report_longitudinal.mjs [--run <dir>]
//
// Default run dir: /workspace/publications_dataset/data/extract/run1
//
// Rules this script obeys, from the site's own methodology:
//   * paper-counted, never tuple-counted;
//   * sentinels (`not-stated`, `not-applicable`, ...) are never an answer;
//   * every free-text field is folded before aggregating and the unmapped
//     residue is printed in full;
//   * every block names its denominator on the line above the numbers.
 
import fs from 'node:fs';
import path from 'node:path';
import { foldCadence } from './cadence_fold.mjs';
 
const argv = process.argv.slice(2);
const RUN = argv.includes('--run')
  ? argv[argv.indexOf('--run') + 1]
  : '/workspace/publications_dataset/data/extract/run1';
const FULLTEXT = argv.includes('--fulltext')
  ? argv[argv.indexOf('--fulltext') + 1]
  : '/workspace/publications_dataset/data/fulltext';
 
const P = fs
  .readFileSync(path.join(RUN, 'extractions.jsonl'), 'utf8')
  .split('\n')
  .filter(Boolean)
  .map((s) => JSON.parse(s));
 
const pct = (a, b) => (b ? ((100 * a) / b).toFixed(1) : '0.0');
const rule = (t) => console.log(`\n${'='.repeat(74)}\n${t}\n${'='.repeat(74)}`);
const head = (t, d) => console.log(`\n-- ${t}\n   population: ${d}`);
 
// -------------------------------------------------------------- populations
// `crawled` is the same definition extract_overview.mjs uses, so the counts on
// this page line up with the ones in OVERVIEW.md.
const crawled = P.filter((p) => p.crawlConfig != null || p.studyTypes.includes('automated-web-crawl'));
const empirical = P.filter((p) => p.isEmpirical);
 
// A paper is `repeated` when any temporal tuple states more than one snapshot.
// This is the same rule statistics:biases uses, so the two pages agree.
const snapOf = (p) => Math.max(0, ...p.temporal.map((t) => (t.snapshots == null ? 0 : t.snapshots)));
const repeated = crawled.filter((p) => snapOf(p) > 1);
const singleShot = crawled.filter((p) => snapOf(p) <= 1);
 
// ------------------------------------------------------------------- pins
// Four things a repeat crawl has to hold constant to be comparable with its own
// earlier waves. Each predicate asks only "did the paper state it at all" —
// the extraction cannot tell us whether the value was the SAME across waves, so
// every figure below is an upper bound on comparability, not a measure of it.
const statedListVersion = (p) =>
  p.population.some((x) => x.listVersion != null && !/^(?:not[- ]stated|unknown|n\/a)$/i.test(x.listVersion.trim()));
const namedBrowser = (p) => p.crawlConfig != null && p.crawlConfig.browsers.length > 0;
const browserWithVersion = (p) => p.crawlConfig != null && p.crawlConfig.browsers.some((b) => /\d/.test(b));
const statedVantage = (p) =>
  p.vantage.some((v) => v.locations.some((l) => !/^(?:not[- ]stated|unknown|n\/a)$/i.test(String(l).trim())));
const statedInfra = (p) => p.vantage.some((v) => v.infrastructure !== 'not-stated');
const ccStated = (p, f) => p.crawlConfig != null && !['not-stated', 'not-applicable'].includes(p.crawlConfig[f]);
const allFourPins = (p) => statedListVersion(p) && namedBrowser(p) && statedVantage(p) && ccStated(p, 'statefulness');
 
const PINS = [
  ['population list version stated', statedListVersion],
  ['browser named', namedBrowser],
  ['   ... with a version number in the name', browserWithVersion],
  ['vantage location stated', statedVantage],
  ['vantage infrastructure stated', statedInfra],
  ['statefulness stated', (p) => ccStated(p, 'statefulness')],
  ['consent action stated', (p) => ccStated(p, 'consentAction')],
  ['interaction depth stated', (p) => ccStated(p, 'interactionDepth')],
  ['headless stated', (p) => ccStated(p, 'headless')],
  ['authentication stated', (p) => ccStated(p, 'authentication')],
  ['span start AND end stated', (p) => p.temporal.some((t) => t.spanStart != null) && p.temporal.some((t) => t.spanEnd != null)],
  ['cadence stated', (p) => p.temporal.some((t) => t.cadence != null)],
];
 
// ---------------------------------------------------------------- span
// Longest stated span on any temporal tuple, in months. CAVEAT, and it matters:
// a paper's longest span is often a REUSED dataset arm, not the span of its own
// repeated crawl. The "multi-year" group below therefore mixes both, which is
// itself part of why it pins less — see the mode breakdown printed with it.
const MONTH_NAMES = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec'];
 
// Parse a stated date into {year, month}. spanStart/spanEnd are free-text as the
// paper wrote them, so BOTH forms occur and both must be handled:
//   numeric  "2019-03", "2019/03", "2019"
//   textual  "March 2019", "Mar. 2019", "26 March 2019", "April 17th, 2023"
// An earlier version of this function only understood the numeric form, so every
// textual month silently became January. That put a same-year span like
// "February 2011 -> July 2011" into the "<1 month" band and a cross-year span
// like "June 2010 -> January 2011" into "1-2 years". 129 stated dates in the
// corpus are textual, so this was not a rare edge case; it moved five bands and
// the multi-year/sub-year split. Caught by review, fixed, and both tables re-run.
//
// The year must be a standalone token, not a run of digits inside a larger
// number: "2000 domains, March 2020" would otherwise parse 2000 as the year, and
// 2000 sits inside the sanity window so nothing would catch it.
function parseDate(s) {
  if (!s) return null;
  const t = String(s).toLowerCase();
  const ym = /(?:^|[^\d])(\d{4})(?:[-/](0?[1-9]|1[0-2]))(?![\d])/.exec(t);
  if (ym) return { y: +ym[1], m: +ym[2] };
  const yOnly = /(?:^|[^\d])(\d{4})(?![\d])/.exec(t);
  if (!yOnly) return null;
  const y = +yOnly[1];
  if (y < 1990 || y > 2027) return null;
  const name = new RegExp(`\\b(${MONTH_NAMES.join('|')})[a-z]*\\b`).exec(t);
  return { y, m: name ? MONTH_NAMES.indexOf(name[1]) + 1 : 1 };
}
 
function spanMonths(a, b) {
  const pa = parseDate(a);
  const pb = parseDate(b);
  if (!pa || !pb) return null;
  if (pa.y < 1990 || pb.y < 1990 || pa.y > 2027 || pb.y > 2027) return null;
  const d = (pb.y - pa.y) * 12 + (pb.m - pa.m);
  return d < 0 ? null : d;
}
const longestSpan = (p) => {
  let best = null;
  for (const t of p.temporal) {
    const m = spanMonths(t.spanStart, t.spanEnd);
    if (m != null && (best == null || m > best)) best = m;
  }
  return best;
};
 
// ================================================================== output
console.log(`design:longitudinal — figures with denominators`);
console.log(`run: ${RUN}`);
console.log(`corpus: ${P.length.toLocaleString()} papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P), 2010-2026`);
 
rule('1. POPULATIONS');
console.log(`all papers                                        ${P.length}`);
console.log(`empirical (measured the world as it is)           ${empirical.length}`);
console.log(`crawled  (crawlConfig != null OR studyType crawl) ${crawled.length}`);
console.log(`repeated (crawled AND temporal.snapshots > 1)     ${repeated.length}  = ${pct(repeated.length, crawled.length)}% of crawled`);
console.log(`single-shot or snapshot count unstated            ${singleShot.length}  = ${pct(singleShot.length, crawled.length)}% of crawled`);
const anyRepeat = P.filter((p) => snapOf(p) > 1);
console.log(`\nany paper (not only crawls) with snapshots > 1    ${anyRepeat.length}  = ${pct(anyRepeat.length, P.length)}% of all papers`);
 
rule('2. IS THE MEASUREMENT PERIOD EVEN STATED');
head('temporal reporting fields', `${empirical.length} empirical papers`);
for (const [label, f] of [
  ['spanStart stated', (p) => p.temporal.some((t) => t.spanStart != null)],
  ['spanEnd stated', (p) => p.temporal.some((t) => t.spanEnd != null)],
  ['both stated', (p) => p.temporal.some((t) => t.spanStart != null) && p.temporal.some((t) => t.spanEnd != null)],
  ['snapshots stated', (p) => p.temporal.some((t) => t.snapshots != null)],
  ['cadence stated', (p) => p.temporal.some((t) => t.cadence != null)],
]) {
  const n = empirical.filter(f).length;
  console.log(`   ${label.padEnd(22)} ${String(n).padStart(5)}  ${pct(n, empirical.length).padStart(5)}%`);
}
head('the same fields', `${crawled.length} crawling papers`);
for (const [label, f] of [
  ['spanStart stated', (p) => p.temporal.some((t) => t.spanStart != null)],
  ['spanEnd stated', (p) => p.temporal.some((t) => t.spanEnd != null)],
  ['both stated', (p) => p.temporal.some((t) => t.spanStart != null) && p.temporal.some((t) => t.spanEnd != null)],
  ['snapshots stated', (p) => p.temporal.some((t) => t.snapshots != null)],
  ['cadence stated', (p) => p.temporal.some((t) => t.cadence != null)],
]) {
  const n = crawled.filter(f).length;
  console.log(`   ${label.padEnd(22)} ${String(n).padStart(5)}  ${pct(n, crawled.length).padStart(5)}%`);
}
 
rule('3. HOW MANY WAVES, AND HOW FAR APART');
head('stated snapshot count, banded', `${crawled.length} crawling papers`);
{
  const bands = new Map();
  const order = ['not stated', '1', '2-3', '4-6', '7-12', '13-52', '>52'];
  for (const p of crawled) {
    const s = snapOf(p);
    const b = s === 0 ? 'not stated' : s === 1 ? '1' : s <= 3 ? '2-3' : s <= 6 ? '4-6' : s <= 12 ? '7-12' : s <= 52 ? '13-52' : '>52';
    bands.set(b, (bands.get(b) || 0) + 1);
  }
  for (const b of order) console.log(`   ${b.padEnd(12)} ${String(bands.get(b) || 0).padStart(5)}  ${pct(bands.get(b) || 0, crawled.length).padStart(5)}%`);
  const wild = crawled.filter((p) => snapOf(p) > 1000).length;
  console.log(`   NOTE: ${wild} papers carry a snapshot count above 1,000. Spot-checking these finds`);
  console.log(`         page counts and record counts mis-extracted as snapshots, so the top band is`);
  console.log(`         unreliable as a count of waves. Bands are used on the page, never the raw number.`);
}
head('longest stated span, banded', `${repeated.length} repeated crawling papers`);
{
  const bands = new Map();
  const order = ['<1 month', '1-2 months', '3-5 months', '6-11 months', '1-2 years', '2-5 years', '5+ years', 'no parsable end date'];
  for (const p of repeated) {
    const m = longestSpan(p);
    const b = m == null ? 'no parsable end date' : m < 1 ? '<1 month' : m < 3 ? '1-2 months' : m < 6 ? '3-5 months' : m < 12 ? '6-11 months' : m < 24 ? '1-2 years' : m < 60 ? '2-5 years' : '5+ years';
    bands.set(b, (bands.get(b) || 0) + 1);
  }
  for (const b of order) console.log(`   ${b.padEnd(22)} ${String(bands.get(b) || 0).padStart(5)}  ${pct(bands.get(b) || 0, repeated.length).padStart(5)}%`);
}
head('cadence, folded (scripts/cadence_fold.mjs)', `${crawled.filter((p) => p.temporal.some((t) => t.cadence != null)).length} crawling papers that state any cadence`);
{
  const denom = crawled.filter((p) => p.temporal.some((t) => t.cadence != null)).length;
  const fam = new Map();
  const residue = new Map();
  for (const p of crawled) {
    const seen = new Set();
    for (const t of p.temporal) {
      if (t.cadence == null) continue;
      const f = foldCadence(t.cadence);
      if (f === 'unmapped') residue.set(t.cadence.trim(), (residue.get(t.cadence.trim()) || 0) + 1);
      if (!seen.has(f)) { seen.add(f); fam.set(f, (fam.get(f) || 0) + 1); }
    }
  }
  for (const [f, n] of [...fam.entries()].sort((a, b) => b[1] - a[1])) {
    console.log(`   ${f.padEnd(26)} ${String(n).padStart(5)}  ${pct(n, denom).padStart(5)}%`);
  }
  // The families are multi-valued, so summing the non-cadence rows double-counts.
  // The publishable figure is the number of papers whose cadence values are ALL
  // non-cadences — a duration, a bare round count, or "periodically".
  const NOTACADENCE = new Set(['duration-not-cadence', 'n-rounds-no-interval', 'irregular-or-unspecified', 'sentinel']);
  let noRealCadence = 0;
  for (const p of crawled) {
    const fams = new Set(p.temporal.filter((t) => t.cadence != null).map((t) => foldCadence(t.cadence)));
    if (fams.size && [...fams].every((f) => NOTACADENCE.has(f))) noRealCadence++;
  }
  console.log(`\n   papers that fill the cadence slot but never state a cadence:`);
  console.log(`      ${noRealCadence} / ${denom} = ${pct(noRealCadence, denom)}%   (counting 'unmapped' as a possible cadence, i.e. conservative)`);
  console.log(`\n   A paper can land in several families (it may state more than one cadence), so`);
  console.log(`   the column does not sum to ${denom}. 'duration-not-cadence' is the field being used`);
  console.log(`   to state how long the study ran rather than how often it repeated — those papers`);
  console.log(`   have NOT stated a cadence and must not be counted as if they had.`);
  console.log(`\n   UNMAPPED RESIDUE — ${residue.size} distinct strings, printed in full:`);
  for (const [s, n] of [...residue.entries()].sort((a, b) => b[1] - a[1])) console.log(`      ${n} x  ${s}`);
}
 
rule('4. THE FOUR PINS: WHAT A REPEATED CRAWL HOLDS FIXED');
head('stated at all (an upper bound on holding it fixed)', `${repeated.length} repeated vs ${singleShot.length} other crawling papers`);
console.log(`   ${'pin'.padEnd(42)} ${'repeated'.padStart(14)} ${'other crawls'.padStart(14)}`);
for (const [label, f] of PINS) {
  const a = repeated.filter(f).length;
  const b = singleShot.filter(f).length;
  console.log(`   ${label.padEnd(42)} ${`${a} (${pct(a, repeated.length)}%)`.padStart(14)} ${`${b} (${pct(b, singleShot.length)}%)`.padStart(14)}`);
}
{
  const a = repeated.filter(allFourPins).length;
  const b = singleShot.filter(allFourPins).length;
  const none = repeated.filter((p) => !statedListVersion(p) && !namedBrowser(p) && !statedVantage(p) && !ccStated(p, 'statefulness')).length;
  const strict = repeated.filter((p) => statedListVersion(p) && browserWithVersion(p) && statedVantage(p) && ccStated(p, 'statefulness')).length;
  console.log(`\n   all four (list version + browser + vantage + statefulness)`);
  console.log(`      repeated      ${a} / ${repeated.length} = ${pct(a, repeated.length)}%`);
  console.log(`      other crawls  ${b} / ${singleShot.length} = ${pct(b, singleShot.length)}%`);
  console.log(`   all four, with a browser VERSION rather than a browser name`);
  console.log(`      repeated      ${strict} / ${repeated.length} = ${pct(strict, repeated.length)}%`);
  console.log(`   none of the four`);
  console.log(`      repeated      ${none} / ${repeated.length} = ${pct(none, repeated.length)}%`);
}
 
rule('5. THE LONGER THE PANEL, THE LESS IT PINS');
{
  const multi = repeated.filter((p) => { const m = longestSpan(p); return m != null && m >= 12; });
  const sub = repeated.filter((p) => { const m = longestSpan(p); return m != null && m < 12; });
  const unk = repeated.length - multi.length - sub.length;
  head('by longest stated span', `${repeated.length} repeated crawling papers (${unk} have no parsable end date and are excluded)`);
  for (const [name, g] of [['span >= 12 months', multi], ['span <  12 months', sub]]) {
    const f = g.filter(allFourPins).length;
    console.log(`   ${name}  n=${String(g.length).padStart(4)}   list ${String(g.filter(statedListVersion).length).padStart(3)}  browser ${String(g.filter(namedBrowser).length).padStart(3)}  vantage ${String(g.filter(statedVantage).length).padStart(3)}  statefulness ${String(g.filter((p) => ccStated(p, 'statefulness')).length).padStart(3)}   all four ${f} (${pct(f, g.length)}%)`);
  }
  console.log(`\n   Why the multi-year group pins less — its data provenance is different.`);
  console.log(`   temporal.mode, paper-counted, multi-valued:`);
  const modes = (g) => { const c = new Map(); for (const p of g) for (const m of new Set(p.temporal.map((t) => t.mode))) c.set(m, (c.get(m) || 0) + 1); return c; };
  for (const [name, g] of [['span >= 12 months', multi], ['span <  12 months', sub]]) {
    const c = modes(g);
    console.log(`   ${name}  ` + [...c.entries()].sort((a, b) => b[1] - a[1]).map(([m, n]) => `${m} ${n} (${pct(n, g.length)}%)`).join(', '));
  }
  const liveOnly = (g) => g.filter((p) => p.temporal.every((t) => ['live-crawl', 'active-probing', 'not-stated'].includes(t.mode)));
  console.log(`\n   Restricted to papers whose every temporal tuple is a live crawl or active probe`);
  console.log(`   (no archive, no reused dataset), so the pins were at least available:`);
  for (const [name, g] of [['span >= 12 months', liveOnly(multi)], ['span <  12 months', liveOnly(sub)]]) {
    const f = g.filter(allFourPins).length;
    console.log(`   ${name}  n=${String(g.length).padStart(4)}   all four ${f} (${pct(f, g.length)}%)   << small n, report as a count`);
  }
}
 
rule('6. IS REPEATING GETTING MORE COMMON');
head('share of crawling papers that repeat, by year', 'crawling papers in each year');
{
  const years = [...new Set(crawled.map((p) => p.year))].sort();
  for (const y of years) {
    const g = crawled.filter((p) => p.year === y);
    const r = g.filter((p) => snapOf(p) > 1).length;
    const flag = y >= 2025 ? '   << provisional venue-year, under-represented by construction' : '';
    console.log(`   ${y}   crawled ${String(g.length).padStart(4)}   repeated ${String(r).padStart(3)}  ${pct(r, g.length).padStart(5)}%${flag}`);
  }
  const bucket = (a, b) => { const g = crawled.filter((p) => p.year >= a && p.year <= b); const r = g.filter((p) => snapOf(p) > 1).length; return `${a}-${b}: ${r}/${g.length} = ${pct(r, g.length)}%`; };
  console.log(`\n   four-year buckets: ${bucket(2010, 2011)} | ${bucket(2012, 2015)} | ${bucket(2016, 2019)} | ${bucket(2020, 2023)} | ${bucket(2024, 2026)} (last bucket provisional)`);
}
 
rule('7. DATA PROVENANCE OF EMPIRICAL WORK');
head('temporal.mode', `${empirical.length} empirical papers, multi-valued, paper-counted`);
{
  const c = new Map();
  for (const p of empirical) for (const m of new Set(p.temporal.map((t) => t.mode))) c.set(m, (c.get(m) || 0) + 1);
  for (const [m, n] of [...c.entries()].sort((a, b) => b[1] - a[1])) console.log(`   ${m.padEnd(22)} ${String(n).padStart(5)}  ${pct(n, empirical.length).padStart(5)}%`);
}
 
rule('8. FULL-TEXT PROBES OVER THE REPEATED-CRAWL PAPERS');
console.log(`   Whitespace is collapsed before matching, because a PDF line break inside a`);
console.log(`   phrase would otherwise hide it. Every probe is a MENTION count and therefore`);
console.log(`   an upper bound; the two load-bearing ones were hand-read in full and the`);
console.log(`   audited counts are printed beside them.`);
head('probe hits', `${repeated.length} repeated crawling papers, all with readable full text`);
{
  const PROBES = [
    ['self-describes as longitudinal', /\blongitudinal\b/i, null],
    ['re-crawl / repeat crawl / second crawl', /\bre-?crawl|\brepeat(?:ed)? (?:the )?(?:crawl|measurement|experiment)|\bsecond crawl\b|\bfollow-?up crawl\b/i, null],
    ['wave / round / iteration terminology', /\b(?:each|every|first|second|third|per) (?:wave|round|run|iteration)\b|\bcrawl(?:ing)? (?:wave|round)s?\b/i, null],
    ['comparable / comparability', /\bcomparab(?:le|ility)\b/i, null],
    ['reproducib* / replicab* / replicat*', /\breproducib\w*|\breplicab\w*|\breplicat\w*/i, null],
    ['same version / list / snapshot / configuration', /\bsame (?:version|list|snapshot|configuration|setup|browser|profile|seed list|instance)\b/i, null],
    ['pinned / froze / fixed the version (wide)', /\bpinn?ed\b|\bfroze[nd]?\b|\bfixed\s+(?:the\s+)?(?:version|browser|list|snapshot|commit|configuration|seed)|\bversion[- ]controll?ed\b/i, null],
    ['Docker / container / VM image', /\bdocker\b|\bcontaineri[sz]ed\b|\bVM image\b|\bvirtual machine image\b/i, null],
    ['names a filter list or blocklist', /\bEasyList\b|\bEasyPrivacy\b|\bDisconnect\b|\bfilter\s+lists?\b|\bblock\s?lists?\b/i, null],
    ['filter list pinned to a version or date', /\b(?:EasyList|EasyPrivacy|Disconnect|filter list|blocklist|block list)\b.{0,80}\b(?:version|as of|snapshot|dated|commit|downloaded on)\b/i, null],
    ['attrition or survivorship', /\battrition\b|\bsurvivorship\b/i, null],
    ['churn', /\bchurn\b/i, null],
    ['<browser> <number> anywhere in the text', /\b(?:Chrome|Chromium|Firefox|Edge|Safari|Tor Browser|Brave)\s*(?:browser\s*)?(?:v(?:ersion)?\.?\s*)?\d{1,3}(?:\.\d+)*\b/i, 26],
    ['browser auto-update / version drift (wide)', /\bauto-?updat\w*|\bautomatic(?:ally)?\s+updat\w*|\bbrowser\s+(?:version\s+)?(?:updates?|upgrad\w*)|\bdisabl\w+\s+updates?/i, 0],
  ];
  const texts = repeated.map((p) => {
    const f = path.join(FULLTEXT, String(p.year), p.venue, p.slug, 'paper.cols.txt');
    return fs.existsSync(f) ? fs.readFileSync(f, 'utf8').replace(/\s+/g, ' ') : null;
  });
  const readable = texts.filter((t) => t != null).length;
  console.log(`   full text readable for ${readable} of ${repeated.length}\n`);
  for (const [label, re, audited] of PROBES) {
    const n = texts.filter((t) => t != null && re.test(t)).length;
    const aud = audited == null ? '' : `   -> hand-read all ${n}: ${audited} genuine (${pct(audited, readable)}% of the population)`;
    console.log(`   ${label.padEnd(46)} ${String(n).padStart(4)}  ${pct(n, readable).padStart(5)}%${aud}`);
  }
  console.log(`\n   The two audited probes, in words:`);
  console.log(`     * "<browser> <number>" is a candidate set, not an answer. Reading all of them,`);
  console.log(`       26 state the version of a browser the study's OWN measurement drove. The rest`);
  console.log(`       are bibliography entries, ecosystem history ("Chrome 43 shipped ..."), the`);
  console.log(`       version of a browser being measured rather than driven, and outright false`);
  console.log(`       positives (footnote markers, table cells, "Chrome browser 100 times").`);
  console.log(`     * The auto-update probe: reading all hits, NOT ONE discusses pinning or`);
  console.log(`       reporting the crawler's own browser version across waves. The closest is`);
  console.log(`       a note that measurement tools need maintenance as browsers change.`);
}
 
rule('9. PINNING THE LIST: DOES ANYONE CITE A PERMANENT TRANCO ID');
{
  // Tranco has published permanent, retrievable list IDs since 2019, and a list
  // ID is the only mechanism in this literature that makes a population frame
  // recoverable years later (verified 2026-08-27: the 2019 id GVWK still
  // resolves). So: of the papers that use Tranco at all, how many cite one?
  //
  // Two wide probes were run over the full text of every Tranco paper; their
  // union was 37 candidates; ALL 37 were read by hand. The 23 below are the ones
  // that genuinely cite an identifier. The rest matched legal section numbers
  // and reference codes sitting near the word "Tranco". The allowlist is written
  // out rather than recomputed because a probe cannot make this call.
  const CITES_TRANCO_ID = new Set([
    '2020/WWW/beyond-the-front-page-measuring-third-party-dynamics-in-the-field',
    '2021/NDSS/reining-in-the-webs-inconsistencies-with-site-policy',
    '2021/PETS/privacy-preference-signals-past-present-and-future',
    '2022/CCS/helping-or-hindering-how-browser-extensions-undermine-security',
    '2022/PETS/setting-the-bar-low-are-websites-complying-with-the-minimum-requirements-of-the',
    '2023/CCS/you-call-this-archaeology-evaluating-web-archives-for-reproducible-web-security',
    '2023/IEEE-SP/its-dom-clobbering-time-attack-techniques-prevalence-and-defenses',
    '2023/USENIX/autofr-automated-filter-rule-generation-for-adblocking',
    '2023/USENIX/mobileatlas-geographically-decoupled-measurements-in-cellular-networks-for-secur',
    '2023/WWW/the-hitchhikers-guide-to-facebook-web-tracking-with-invisible-pixels-and-click-i',
    '2024/IEEE-SP/to-auth-or-not-to-auth-a-comparative-analysis-of-the-pre-and-post-login-security',
    '2024/IEEE-SP/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac',
    '2024/PETS/a-bilingual-longitudinal-analysis-of-privacy-policies-measuring-the-impacts-of-t',
    '2024/PETS/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework',
    '2025/CCS/in-the-dom-we-trust-exploring-the-hidden-dangers-of-reading-from-the-dom-on-the',
    '2025/IEEE-SP/a-wall-behind-a-wall-emerging-regional-censorship-in-china',
    '2025/IMC/chaos-in-the-chain-evaluate-deployment-and-construction-compliance-of-web-pki-ce',
    '2025/IMC/towards-a-non-binary-view-of-ipv6-adoption',
    '2025/PETS/johnny-can-t-revoke-consent-either-measuring-compliance-of-consent-revocation-on',
    '2025/PETS/understanding-privacy-norms-through-web-forms',
    '2025/USENIX/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch',
    '2025/USENIX/navigating-cookie-consent-violations-across-the-globe',
    '2026/USENIX/inconsistent-incomplete-and-insecure-a-survey-of-account-security-interfaces',
  ]);
  const usesTranco = (p) => p.population.some((x) => /\btranco\b/i.test(x.sourceList || ''));
  const T = P.filter(usesTranco);
  head('Tranco users', `${T.length} papers naming Tranco as a population source`);
  const withVersion = T.filter((p) => p.population.some((x) => /\btranco\b/i.test(x.sourceList || '') && x.listVersion != null));
  console.log(`   states any list version or date (extraction field)   ${withVersion.length}  ${pct(withVersion.length, T.length)}%`);
  const cited = T.filter((p) => CITES_TRANCO_ID.has(`${p.year}/${p.venue}/${p.slug}`));
  console.log(`   cites a permanent Tranco list ID (hand-audited)      ${cited.length}  ${pct(cited.length, T.length)}%`);
  const years = [...new Set(T.map((p) => p.year))].sort();
  console.log(`\n   by year (Tranco users / of them, citing an ID):`);
  for (const y of years) {
    const g = T.filter((p) => p.year === y);
    const c = cited.filter((p) => p.year === y).length;
    console.log(`      ${y}   ${String(g.length).padStart(3)}   ${String(c).padStart(2)}  ${pct(c, g.length).padStart(5)}%`);
  }
  const stray = [...CITES_TRANCO_ID].filter((k) => !T.some((p) => `${p.year}/${p.venue}/${p.slug}` === k));
  if (stray.length) console.log(`\n   WARNING: ${stray.length} allowlist entries no longer match a Tranco paper: ${stray.join(', ')}`);
}
 
rule('10. EXEMPLARS — REPEATED CRAWLS THAT STATE ALL FOUR PINS');
console.log(`   ${repeated.filter(allFourPins).length} papers. Most recent first. These are the papers to copy a methods section from.`);
for (const p of repeated.filter(allFourPins).sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue))) {
  const cad = p.temporal.map((t) => t.cadence).filter(Boolean).join(' / ') || '-';
  console.log(`\n   ${p.year} ${p.venue}  ${p.slug}`);
  console.log(`        snapshots=${snapOf(p)}  cadence=${cad}  longest span=${longestSpan(p) == null ? 'n/a' : longestSpan(p) + 'mo'}`);
  console.log(`        browsers=${JSON.stringify(p.crawlConfig.browsers)}  statefulness=${p.crawlConfig.statefulness}  consent=${p.crawlConfig.consentAction}`);
  console.log(`        listVersion=${JSON.stringify(p.population.map((x) => x.listVersion).filter(Boolean))}`);
  console.log(`        vantage=${JSON.stringify([...new Set(p.vantage.flatMap((v) => v.locations))])}`);
}
// Compact form of the same 23 papers, in the column order the wiki table uses,
// so the page is copied from real output rather than retyped.
console.log(`\n   -- compact table (year | venue | slug | snapshots | distinct stated vantage locations)`);
for (const p of repeated.filter(allFourPins).sort((a, b) => b.year - a.year || a.venue.localeCompare(b.venue))) {
  const locs = new Set(p.vantage.flatMap((v) => v.locations).filter((l) => !/^(?:not[- ]stated|unknown|n\/a)$/i.test(String(l).trim())));
  console.log(`   ${p.year} | ${p.venue} | ${p.slug} | ${snapOf(p)} | ${locs.size}`);
}
console.log();
 
rule('11. CORPUS-WIDE ADOPTION PROBES FOR THE MODERN PINNING MECHANISMS');
{
  // These four probes are corpus-wide, not restricted to `repeated`, because the
  // question is "has this mechanism any footprint in these seven venues at all".
  // Each regex is printed with its count so a reader can re-run the exact query;
  // a different regex gives a different number, and that is the point of showing it.
  //
  // Every one of these is a MENTION count. A paper can pin a container digest in
  // its repository and never say so in the text. Read them as reporting figures.
  const PROBES = [
    ['Docker OR containerised', /\bdocker\b|\bcontaineri[sz]ed\b/i, null],
    ['   ...Docker alone', /\bdocker\b/i, null],
    ['image digest or sha256: image reference', /\bimage digest\b|\bsha256:[0-9a-f]{12}/i, null],
    ['"Chrome for Testing", exact phrase', /chrome[-\s]?for[-\s]?testing/i, null],
    ['Chrome for Testing, widened to the tooling and the acronym', /chrome[-\s]?for[-\s]?testing|googlechromelabs|chrome[-\s]?labs|known[-\s]good[-\s]versions|\bCfT\b/i, 0],
    ['WebREC or "web execution bundle"', /\bWebREC\b|\bweb execution bundle/i, 0],
  ];
  let readable = 0;
  const counts = PROBES.map(() => 0);
  for (const p of P) {
    const f = path.join(FULLTEXT, String(p.year), p.venue, p.slug, 'paper.cols.txt');
    if (!fs.existsSync(f)) continue;
    readable++;
    const t = fs.readFileSync(f, 'utf8').replace(/\s+/g, ' ');
    PROBES.forEach(([, re], i) => { if (re.test(t)) counts[i]++; });
  }
  head('mentions anywhere in the full text', `${readable} papers with readable full text, of ${P.length}`);
  PROBES.forEach(([label, re, audited], i) => {
    const aud = audited == null ? '' : `   -> hand-read all ${counts[i]}: ${audited} genuine`;
    console.log(`   ${label.padEnd(58)} ${String(counts[i]).padStart(4)}  ${pct(counts[i], readable).padStart(5)}%${aud}`);
    console.log(`   ${' '.repeat(58)} ${String(re)}`);
  });
  console.log(`\n   Hand audits:`);
  console.log(`     * The widened Chrome-for-Testing probe: every hit is a false positive —`);
  console.log(`       CFT as "Combating the Financing of Terrorism" (three papers), "Control Flow`);
  console.log(`       Trimming", "Call Flow Tree", "Crash-Fault Tolerant", a certificate subject`);
  console.log(`       O=CFT, a CFt variable in an algorithm listing, CFT-IDF, and a fitness app`);
  console.log(`       named "CFT & PFT". The exact-phrase probe is 0, and so is the audited count.`);
  console.log(`     * WebREC: one hit is the WebREC paper itself; the other cites it in its`);
  console.log(`       reference list only. No paper in the corpus USES it.`);
}

temporal.cadence is free text: 334 distinct strings fire on the 1,120 crawling papers. This is the fold, its rule order, and the reasoning behind the first rule.

cadence_fold.mjs

// Folds the free-text `temporal.cadence` string into a small family vocabulary.
//
// WHY THIS EXISTS. `temporal.cadence` is free text and 334 distinct strings fire
// on the 1,120 crawling papers alone. Counting exact strings would report
// "daily" at 66 papers and bury the other 268 values in a tail nobody reads.
// It is also *contaminated*: a large minority of the values state how LONG the
// measurement ran ("one week", "5-day experiment", "month-long") rather than how
// OFTEN it repeated. Those are not cadences and are folded into their own family
// so they cannot be counted as one.
//
// Rules, in order. The FIRST rule that matches wins, and rules are ordered from
// fastest interval to slowest, so "hourly and daily" folds to `hourly` — the
// fastest repeat the paper states is the one that bounds its time resolution.
//
// Anything no rule matches lands in `unmapped` and is PRINTED IN FULL by the
// report. A residue that only exists inside a script is a residue nobody reads.
 
const SENTINEL = /^(?:not[- ]stated|none[- ]mentioned|unclear|unknown|n\/?a)$/i;
 
const RULES = [
  // A duration is not a cadence. This rule runs FIRST for the bare forms
  // ("one week", "15 days", "month-long", "5-day experiment") because they
  // would otherwise be swept up by the weekly/monthly rules below. It is
  // deliberately anchored: "every 5 minutes for 8 days" states a real cadence
  // and must not land here, so a leading `every|per|once` disqualifies it.
  // "seven days", "over one month", "two-week measurement period", "single crawl
  // lasting 12 days", "24-hour runs", "one-week collection period" — the paper is
  // stating how long it ran, not how often it repeated.
  // A string containing "between" or "apart" states an INTERVAL, not a duration
  // ("15 days between batch starts"), so it must never reach the duration rules.
  // The interval rules below pick it up by unit.
  ['duration-not-cadence', /^(?!.*\b(?:between|apart)\b)(?:.*\b(?:collection|measurement|study|crawl(?:ing)?|campaign|experiment|observation)\s+(?:period|window|cycle)\b|\b(?:period|window)\s+of\s+\d+|\bsingle\s+(?:day|crawl|scrape|run|visit|snapshot)\b|\bpilot\s+study\b|\blasting\s+\d+|\b\d+-hour\s+runs?\b|^(?:about|approximately|around|over|nearly|spread\s+across|during)\s+(?:a\s+)?(?:couple\s+of\s+|few\s+|multiple\s+|one\s+|two\s+|three\s+|several\s+|\d+\s*)?(?:day|week|month|year)s?\b|^(?:several|multiple)\s+(?:day|week|month|year)s?\b|^\d+\s*(?:day|week|month|year)s?$|^(?:one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)\s+(?:consecutive\s+)?(?:day|week|month|year)s?$|\b(?:one|two|three|four|five|six|\d+)-(?:day|week|month|year)\s+(?:campaign|collection|trace|slot|phishing))/i],
 
  ['duration-not-cadence', /^(?!.*\b(?:every|per|once|between|apart|times?\s+(?:a|per)|twice|daily|hourly|weekly|monthly|yearly|annual|continuous|periodic|regular)\b)(?:(?:about|over|nearly|one|two|three|four|five|six|\d+)[- ]?)?(?:\d+[- ])?(?:second|minute|hour|day|week|month|year)s?[- ]?(?:long|period|experiment|crawl|campaign|study|snapshot|window|trace)?$|^\d+\s*(?:second|minute|hour|day|week|month|year)s?\b(?!.*\b(?:every|between|apart)\b)|(?:^|\b)(?:one|two|three|four|five|a)?[- ]?(?:day|week|month|year)-long\b/i],
 
  ['sub-minute', /\bmillisecond|\b\d+\s*(?:ms|milliseconds)\b|every\s+\d+(?:\.\d+)?\s*seconds?\b|every\s+(?:few|five|ten|fifteen|thirty)\s+seconds?\b|per\s+second\b/i],
  ['minutes', /\bevery\s+(?:\d+|a|one|two|three|five|ten|twelve|fifteen|thirty|sixty)\s*minutes?\b|\b\d+-minute\s+(?:interval|snapshot)|\bminute\s+intervals?\b|every\s+minute\b|once\s+per\s+minute\b|\bper\s+minute\b/i],
  ['hourly', /\bhourly\b|\bevery\s+(?:\d+|a|an|one|two|three|four|six|eight|twelve|24)\s*hours?\b|\b\d+-hour(?:ly)?\s+(?:interval|window)|\bhours?\s+apart\b|\bfour-hourly\b|\bevery\s+few\s+hours\b|\b(?:twice|two|three)\s+(?:times\s+)?(?:a|per)\s+day\b|\bmultiple\s+(?:scrapes|observations)\s+per\s+day\b|\btwice\s+daily\b|\bfive\s+times\s+daily\b|\b12\s+hours\b/i],
  ['daily', /\bdaily\b|\bnightly\b|\bevery\s+day\b|\bper\s+day\b|\bday\b.*\bintervals?\b|\bat\s+least\s+24\s+hours\s+between\b|\bmultiple\s+times\s+a\s+day\b/i],
  ['every-few-days', /\bevery\s+(?:\d+|a\s+few|two|three|four|five|six|ten|other)\s*days?\b|\bevery\s+two\s+to\s+three\s+days\b|\b(?:two|three|\d+)-day\s+(?:retry\s+)?interval\b|\bonce\s+every\s+\d+\s*days?\b|\bevery\s+\d+(?:\.\d+)?\s*days?\b|\bone\s+day\s+between\b|\b\d+\s*days?\s+between\b/i],
  ['weekly', /\bweekly\b|\bevery\s+week\b|\bper\s+week\b|\bweekdays?\b|\bevery\s+seven\s+days\b|\bat\s+least\s+one\s+week\s+between\b|\bone\s+week\s+between\b/i],
  ['fortnightly', /\bfortnight|\bbi-?weekly\b|\bevery\s+(?:two\s+weeks|14\s*days|15\s*days)\b|\btwo-week\s+intervals?\b|\btwo\s+weeks\s+between\b/i],
  ['monthly', /\bmonthly\b|\bbimonthly\b|\bevery\s+(?:four|twelve)\s+(?:months|weeks)\b|\bevery\s+month\b|\bper\s+month\b|\bevery\s+30\s*days\b|\bmonths?\s+apart\b|\bat\s+least\s+one\s+month\s+between\b|\bmonths?\s+between\b/i],
  ['quarterly', /\bquarterly\b|\btrimester\b|\bevery\s+three\s+months\b|\bevery\s+quarter\b|\bthree-month\s+intervals?\b/i],
  ['yearly', /\byearly\b|\bannual\w*\b|\bevery\s+year\b|\bone\s+year\s+apart\b/i],
  ['continuous', /\bcontinuous\w*\b|\breal[- ]?time\b|\biteratively\s+without\s+sleep\b|\bstreaming\b/i],
  ['irregular-or-unspecified', /\bperiodic\w*\b|\bsporadic\b|\bsimultaneous\b|\btemporally\s+close\b|\bas\s+domains\s+appeared\b|\bincreasing\s+(?:historical\s+)?intervals?\b|\bregular\w*\b|\badaptive\b|\bon[- ]demand\b|\bas\s+needed\b|\bwhen(?:ever)?\b|\bvariable\b|\bad\s?hoc\b/i],
  ['n-rounds-no-interval', /\b(?:one|two|three|four|five|six|seven|eight|nine|ten|\d+)\s+(?:repeated\s+|sequential\s+|distinct\s+|measurement\s+|querying\s+|collection\s+)*(?:rounds?|repetitions?|iterations?|visits?|scans?|scrapes?|queries|campaigns?|snapshots?|periods?|timestamps?|crawls?|groups?|runs?|windows?|days?|weeks?|intervals?)\b|\bonce\b|\brepeated\s+(?:experiments?|collection\s+iterations?)\b|\bup\s+to\s+\w+\s+sequential\s+visits\b|\bconsecutive\s+crawls\b/i],
];
 
export function foldCadence(raw) {
  const s = String(raw).trim().toLowerCase().replace(/\s+/g, ' ');
  if (SENTINEL.test(s)) return 'sentinel';
  for (const [family, re] of RULES) if (re.test(s)) return family;
  return 'unmapped';
}
 
export const FAMILIES = RULES.map(([f]) => f).concat('unmapped');
export const REAL_FAMILIES = FAMILIES.filter((f) => f !== 'duration-not-cadence' && f !== 'unmapped');

quotecheck_longitudinal.mjs

#!/usr/bin/env node
// Verifies every quotation used on design:longitudinal against the paper text.
//
//   node scripts/quotecheck_longitudinal.mjs
//
// Two-column PDFs are repaired into paper.cols.txt but not perfectly, so a
// sentence can arrive spliced with text from the neighbouring column, and a word
// can be hyphen-split across a line break. Each quote is checked against FOUR
// renderings: paper.cols.txt, paper.norm.txt, paper.txt, and paper.cols.txt with
// end-of-line hyphenation joined. The verdicts:
//   EXACT     — the quote, whitespace-collapsed, appears verbatim
//   SPLICED   — every word is covered by contiguous runs of >= MIN_RUN words,
//               so only the reading order was broken
//   PARTIAL   — some words matched only in shorter runs. Read it by hand.
//   MISSING   — a word of the quote does not appear at all. A real failure.
import fs from 'node:fs';
import path from 'node:path';
const ROOT = '/workspace/publications_dataset/data/fulltext';
const norm = (s) => s.replace(/[‐-―−]/g, '-').replace(/[‘’]/g, "'").replace(/[“”]/g, '"').replace(/\s+/g, ' ').trim();
 
const QUOTES = [
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'Over twelve days, we saw a variation of up to 27% (max on day 3-80,274; min on day 9-58,951) in observed tracking requests.'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'the number of distinct tracking domains remains almost stable during the experiment (variation of 3.5%)'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'studies that analyze the ecosystem will find similar results, while studies that aim to analyze the extent of a tracking phenomenon might see different results based on the measurement day'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'we find that the identified trackers on pages can vary by 25% based on the used browser configuration'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'even the same setup measures different levels of tracking on different days, which might lead to different conclusions of a study'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)'],
  ['2023/IMC/on-the-similarity-of-web-measurements-under-different-experimental-setups',
    'when comparing two different profiles, 48% of the underlying data varies'],
  ['2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law',
    'we restrict the analysis to the balanced panel of websites with an observed policy in both snapshots'],
  ['2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law',
    'Standard errors are clustered at the website level to account for repeated observations of the same policy across snapshots.'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'The standard deviation of such requests is 8,203.'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'For our continuous measurement, we visit the top 1k sites from our website corpus (18,377 distinct pages) daily'],
  ['2022/WWW/reproducibility-and-replicability-of-web-measurement-studies',
    'privacy measurements and analyses can vary up to 65% depending on the region'],
  ['2023/IMC/on-the-similarity-of-web-measurements-under-different-experimental-setups',
    'only 32% of the cookies appear in all profiles and 42% only in one profile'],
  ['2021/WWW/towards-realistic-and-reproducibleweb-crawl-measurements',
    'Around 5% of content-providing domains show significant measurement bias across VP'],
  ['2020/WWW/apophanies-or-epiphanies-how-crawlers-impact-our-understanding-of-the-web',
    'variation of over 16% in the number of successful page loads'],
  ['2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law',
    'The balanced panel comprises N = 11,800 websites observed in both August and October 2023'],
  ['2025/USENIX/web-execution-bundles-reproducible-accurate-and-archivable-web-measurements',
    'could be conducted using WebREC as is'],
  ['2026/PETS/the-empire-strikes-back-at-your-privacy-an-archaeology-of-tracking-on-government',
    'We therefore label trackers using the June 2025 EasyList and EasyPrivacy snapshot for all years, keeping the labeling rule fixed'],
];
 
// A two-column splice interleaves the neighbouring column every dozen words or
// so, which destroys a naive substring test and most n-gram tests too. The check
// that survives it: greedily segment the quote into the LONGEST contiguous word
// runs that each appear in the text. If every word of the quote is covered by a
// run of at least MIN_RUN words, the quote is present in the source and only the
// reading order was broken.
//
//   EXACT    the whole quote appears verbatim
//   SPLICED  every word covered by runs of >= MIN_RUN words
//   PARTIAL  some words only matched in short runs -> read it by hand
//   MISSING  a run of the quote does not appear at all
const MIN_RUN = 4;
 
function segment(words, text) {
  const runs = [];
  let i = 0;
  while (i < words.length) {
    let j = words.length;
    while (j > i && !text.includes(words.slice(i, j).join(' '))) j--;
    if (j === i) { runs.push({ len: 0, text: words[i] }); i++; }
    else { runs.push({ len: j - i, text: words.slice(i, j).join(' ') }); i = j; }
  }
  return runs;
}
 
let bad = 0;
for (const [slug, q] of QUOTES) {
  // Fourth rendering: paper.cols.txt with PDF end-of-line hyphenation repaired
  // ("varia- tion" -> "variation"). Without it a quote that straddles a line
  // break inside a word reports MISSING even though it is plainly in the paper.
  const files = ['paper.cols.txt', 'paper.norm.txt', 'paper.txt', 'paper.cols.txt (de-hyphenated)'].map((f) =>
    path.join(ROOT, slug, f.replace(' (de-hyphenated)', '')));
  const texts = files.map((f, i) => {
    if (!fs.existsSync(f)) return null;
    const raw = fs.readFileSync(f, 'utf8');
    return norm(i === 3 ? raw.replace(/(\w)-\s+(\w)/g, '$1$2') : raw);
  });
  const LABEL = ['paper.cols.txt', 'paper.norm.txt', 'paper.txt', 'paper.cols.txt (de-hyphenated)'];
  const nq = norm(q);
  const words = nq.split(' ');
  let verdict = 'MISSING', where = '', detail = '';
  for (let i = 0; i < files.length; i++) {
    if (texts[i] != null && texts[i].includes(nq)) { verdict = 'EXACT'; where = LABEL[i]; break; }
  }
  if (verdict !== 'EXACT') {
    let bestScore = -Infinity;
    for (let i = 0; i < files.length; i++) {
      if (texts[i] == null) continue;
      const runs = segment(words, texts[i]);
      const zero = runs.filter((r) => r.len === 0).length;
      const short = runs.filter((r) => r.len > 0 && r.len < MIN_RUN).length;
      const score = -zero * 100 - short;
      if (score > bestScore) {
        bestScore = score;
        where = LABEL[i];
        detail = `${runs.length} run(s): ${runs.map((r) => r.len).join('+')} words` + (zero ? `, ${zero} word(s) NOT FOUND` : '');
        verdict = zero > 0 ? 'MISSING' : short > 0 ? 'PARTIAL' : 'SPLICED';
      }
    }
  }
  if (verdict === 'MISSING' || verdict === 'PARTIAL') bad++;
  console.log(`${verdict.padEnd(8)} ${slug}\n         "${q.slice(0, 110)}${q.length > 110 ? '...' : ''}"\n         ${where}${detail ? ' — ' + detail : ''}\n`);
}
console.log(bad === 0 ? 'every quote located in its source (EXACT or SPLICED)' : `${bad} quote(s) need a hand read`);

node scripts/report_longitudinal.mjs, run on 2026-08-27 against data/extract/run1. Nothing below is edited.

design:longitudinal — figures with denominators
run: /workspace/publications_dataset/data/extract/run1
corpus: 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P), 2010-2026
 
==========================================================================
1. POPULATIONS
==========================================================================
all papers                                        5859
empirical (measured the world as it is)           5118
crawled  (crawlConfig != null OR studyType crawl) 1120
repeated (crawled AND temporal.snapshots > 1)     250  = 22.3% of crawled
single-shot or snapshot count unstated            870  = 77.7% of crawled
 
any paper (not only crawls) with snapshots > 1    818  = 14.0% of all papers
 
==========================================================================
2. IS THE MEASUREMENT PERIOD EVEN STATED
==========================================================================
 
-- temporal reporting fields
   population: 5118 empirical papers
   spanStart stated        2882   56.3%
   spanEnd stated          2888   56.4%
   both stated             2773   54.2%
   snapshots stated        1237   24.2%
   cadence stated          1304   25.5%
 
-- the same fields
   population: 1120 crawling papers
   spanStart stated         803   71.7%
   spanEnd stated           795   71.0%
   both stated              776   69.3%
   snapshots stated         409   36.5%
   cadence stated           389   34.7%
 
==========================================================================
3. HOW MANY WAVES, AND HOW FAR APART
==========================================================================
 
-- stated snapshot count, banded
   population: 1120 crawling papers
   not stated     711   63.5%
   1              159   14.2%
   2-3             93    8.3%
   4-6             41    3.7%
   7-12            33    2.9%
   13-52           46    4.1%
   >52             37    3.3%
   NOTE: 9 papers carry a snapshot count above 1,000. Spot-checking these finds
         page counts and record counts mis-extracted as snapshots, so the top band is
         unreliable as a count of waves. Bands are used on the page, never the raw number.
 
-- longest stated span, banded
   population: 250 repeated crawling papers
   <1 month                  35   14.0%
   1-2 months                35   14.0%
   3-5 months                41   16.4%
   6-11 months               26   10.4%
   1-2 years                 27   10.8%
   2-5 years                 23    9.2%
   5+ years                  18    7.2%
   no parsable end date      45   18.0%
 
-- cadence, folded (scripts/cadence_fold.mjs)
   population: 389 crawling papers that state any cadence
   daily                        102   26.2%
   duration-not-cadence          58   14.9%
   hourly                        46   11.8%
   monthly                       44   11.3%
   weekly                        43   11.1%
   irregular-or-unspecified      31    8.0%
   n-rounds-no-interval          29    7.5%
   minutes                       27    6.9%
   continuous                    27    6.9%
   every-few-days                20    5.1%
   unmapped                      16    4.1%
   sub-minute                     9    2.3%
   yearly                         9    2.3%
   quarterly                      4    1.0%
   fortnightly                    4    1.0%
   sentinel                       1    0.3%
 
   papers that fill the cadence slot but never state a cadence:
      88 / 389 = 22.6%   (counting 'unmapped' as a possible cadence, i.e. conservative)
 
   A paper can land in several families (it may state more than one cadence), so
   the column does not sum to 389. 'duration-not-cadence' is the field being used
   to state how long the study ran rather than how often it repeated — those papers
   have NOT stated a cadence and must not be counted as if they had.
 
   UNMAPPED RESIDUE — 16 distinct strings, printed in full:
      1 x  four increasing cumulative periods for threshold selection
      1 x  four APKs per year per browser family
      1 x  July and December
      1 x  15-minute trend monitoring
      1 x  one hour per website
      1 x  two six-month intervals per year for 2009–2019
      1 x  longitudinal measurement
      1 x  eleven-minute time slots
      1 x  Crawler B ran at the middle of each month
      1 x  irregular confirmation crawls
      1 x  collection cycles over six months
      1 x  during the day, paused overnight
      1 x  December 2023, May 2024, and December 2024
      1 x  sequential experimental windows
      1 x  four-month interval
      1 x  before and after the second rollout
 
==========================================================================
4. THE FOUR PINS: WHAT A REPEATED CRAWL HOLDS FIXED
==========================================================================
 
-- stated at all (an upper bound on holding it fixed)
   population: 250 repeated vs 870 other crawling papers
   pin                                              repeated   other crawls
   population list version stated                167 (66.8%)    493 (56.7%)
   browser named                                 143 (57.2%)    386 (44.4%)
      ... with a version number in the name        17 (6.8%)      49 (5.6%)
   vantage location stated                        89 (35.6%)    212 (24.4%)
   vantage infrastructure stated                 121 (48.4%)    324 (37.2%)
   statefulness stated                            70 (28.0%)    149 (17.1%)
   consent action stated                         105 (42.0%)    244 (28.0%)
   interaction depth stated                      208 (83.2%)    633 (72.8%)
   headless stated                                42 (16.8%)     98 (11.3%)
   authentication stated                         179 (71.6%)    600 (69.0%)
   span start AND end stated                     210 (84.0%)    566 (65.1%)
   cadence stated                                158 (63.2%)    231 (26.6%)
 
   all four (list version + browser + vantage + statefulness)
      repeated      23 / 250 = 9.2%
      other crawls  32 / 870 = 3.7%
   all four, with a browser VERSION rather than a browser name
      repeated      6 / 250 = 2.4%
   none of the four
      repeated      28 / 250 = 11.2%
 
==========================================================================
5. THE LONGER THE PANEL, THE LESS IT PINS
==========================================================================
 
-- by longest stated span
   population: 250 repeated crawling papers (45 have no parsable end date and are excluded)
   span >= 12 months  n=  68   list  53  browser  36  vantage  22  statefulness  14   all four 3 (4.4%)
   span <  12 months  n= 137   list  94  browser  84  vantage  55  statefulness  43   all four 18 (13.1%)
 
   Why the multi-year group pins less — its data provenance is different.
   temporal.mode, paper-counted, multi-valued:
   span >= 12 months  live-crawl 52 (76.5%), existing-dataset 27 (39.7%), web-archive 20 (29.4%), active-probing 15 (22.1%), passive-collection 12 (17.6%)
   span <  12 months  live-crawl 132 (96.4%), existing-dataset 35 (25.5%), active-probing 32 (23.4%), passive-collection 12 (8.8%), web-archive 3 (2.2%)
 
   Restricted to papers whose every temporal tuple is a live crawl or active probe
   (no archive, no reused dataset), so the pins were at least available:
   span >= 12 months  n=  16   all four 1 (6.3%)   << small n, report as a count
   span <  12 months  n=  94   all four 13 (13.8%)   << small n, report as a count
 
==========================================================================
6. IS REPEATING GETTING MORE COMMON
==========================================================================
 
-- share of crawling papers that repeat, by year
   population: crawling papers in each year
   2010   crawled   25   repeated   2    8.0%
   2011   crawled   22   repeated   2    9.1%
   2012   crawled   29   repeated   6   20.7%
   2013   crawled   26   repeated   6   23.1%
   2014   crawled   35   repeated   3    8.6%
   2015   crawled   41   repeated  12   29.3%
   2016   crawled   40   repeated  12   30.0%
   2017   crawled   51   repeated  16   31.4%
   2018   crawled   61   repeated  14   23.0%
   2019   crawled   97   repeated  23   23.7%
   2020   crawled   75   repeated  19   25.3%
   2021   crawled   75   repeated  17   22.7%
   2022   crawled  110   repeated  27   24.5%
   2023   crawled  125   repeated  20   16.0%
   2024   crawled  110   repeated  22   20.0%
   2025   crawled  129   repeated  34   26.4%   << provisional venue-year, under-represented by construction
   2026   crawled   69   repeated  15   21.7%   << provisional venue-year, under-represented by construction
 
   four-year buckets: 2010-2011: 4/47 = 8.5% | 2012-2015: 27/131 = 20.6% | 2016-2019: 65/249 = 26.1% | 2020-2023: 83/385 = 21.6% | 2024-2026: 71/308 = 23.1% (last bucket provisional)
 
==========================================================================
7. DATA PROVENANCE OF EMPIRICAL WORK
==========================================================================
 
-- temporal.mode
   population: 5118 empirical papers, multi-valued, paper-counted
   existing-dataset        2142   41.9%
   active-probing          1629   31.8%
   live-crawl              1236   24.2%
   passive-collection       988   19.3%
   not-stated               338    6.6%
   web-archive               65    1.3%
 
==========================================================================
8. FULL-TEXT PROBES OVER THE REPEATED-CRAWL PAPERS
==========================================================================
   Whitespace is collapsed before matching, because a PDF line break inside a
   phrase would otherwise hide it. Every probe is a MENTION count and therefore
   an upper bound; the two load-bearing ones were hand-read in full and the
   audited counts are printed beside them.
 
-- probe hits
   population: 250 repeated crawling papers, all with readable full text
   full text readable for 250 of 250
 
   self-describes as longitudinal                  123   49.2%
   re-crawl / repeat crawl / second crawl           42   16.8%
   wave / round / iteration terminology             54   21.6%
   comparable / comparability                       80   32.0%
   reproducib* / replicab* / replicat*              78   31.2%
   same version / list / snapshot / configuration   33   13.2%
   pinned / froze / fixed the version (wide)        13    5.2%
   Docker / container / VM image                    18    7.2%
   names a filter list or blocklist                 84   33.6%
   filter list pinned to a version or date           5    2.0%
   attrition or survivorship                         2    0.8%
   churn                                            19    7.6%
   <browser> <number> anywhere in the text          58   23.2%   -> hand-read all 58: 26 genuine (10.4% of the population)
   browser auto-update / version drift (wide)       14    5.6%   -> hand-read all 14: 0 genuine (0.0% of the population)
 
   The two audited probes, in words:
     * "<browser> <number>" is a candidate set, not an answer. Reading all of them,
       26 state the version of a browser the study's OWN measurement drove. The rest
       are bibliography entries, ecosystem history ("Chrome 43 shipped ..."), the
       version of a browser being measured rather than driven, and outright false
       positives (footnote markers, table cells, "Chrome browser 100 times").
     * The auto-update probe: reading all hits, NOT ONE discusses pinning or
       reporting the crawler's own browser version across waves. The closest is
       a note that measurement tools need maintenance as browsers change.
 
==========================================================================
9. PINNING THE LIST: DOES ANYONE CITE A PERMANENT TRANCO ID
==========================================================================
 
-- Tranco users
   population: 266 papers naming Tranco as a population source
   states any list version or date (extraction field)   159  59.8%
   cites a permanent Tranco list ID (hand-audited)      23  8.6%
 
   by year (Tranco users / of them, citing an ID):
      2020     7    1   14.3%
      2021    28    2    7.1%
      2022    29    2    6.9%
      2023    48    5   10.4%
      2024    56    4    7.1%
      2025    76    8   10.5%
      2026    22    1    4.5%
 
==========================================================================
10. EXEMPLARS — REPEATED CRAWLS THAT STATE ALL FOUR PINS
==========================================================================
   23 papers. Most recent first. These are the papers to copy a methods section from.
 
   2026 PETS  clicking-into-exposure-uncovering-privacy-risks-of-google-click-identifier-in-yo
        snapshots=5  cadence=five repeated iterations  longest span=0mo
        browsers=["Chrome","Firefox","Edge"]  statefulness=stateless  consent=accept-and-reject
        listVersion=["v3"]
        vantage=["Germany","Australia","India","Brazil","USA (California)","USA (New York)"]
 
   2026 PETS  privacy-vs-profit-the-impact-of-googles-manifest-version-3-mv3-update-on-ad-bloc
        snapshots=5  cadence=-  longest span=0mo
        browsers=["Google Chrome","Firefox"]  statefulness=stateless  consent=accept-all
        listVersion=["January 9, 2024","July 12, 2025"]
        vantage=["Europe"]
 
   2026 PETS  more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
        snapshots=5  cadence=15 days between batch starts / hourly refreshes  longest span=4mo
        browsers=["Google Chrome"]  statefulness=stateless  consent=no-interaction
        listVersion=["January 20, 2024","January 20, 2024","January 20, 2024","January 20, 2024"]
        vantage=["US"]
 
   2025 PETS  beyond-the-request-harnessing-http-response-headers-for-cross-browser-web-tracke
        snapshots=18  cadence=-  longest span=0mo
        browsers=["Chrome","Brave","Firefox"]  statefulness=stateful  consent=not-stated
        listVersion=["16th of May 2022","March 29th, 2023"]
        vantage=["Frankfurt, Germany"]
 
   2025 PETS  intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and
        snapshots=4  cadence=-  longest span=0mo
        browsers=["Firefox v121.0","Chrome"]  statefulness=both  consent=accept-and-reject
        listVersion=["ID K2NZW, generated 07 December 2023","ID K2NZW, generated 07 December 2023","ID K2NZW, generated 07 December 2023"]
        vantage=["EU"]
 
   2025 PETS  more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings
        snapshots=6  cadence=at least one week between consecutive runs / at least 24 hours between runs  longest span=2mo
        browsers=["Google Chrome"]  statefulness=stateful  consent=not-stated
        listVersion=["data from September 9, 2022","March 14, 2024","November 2023","March 20, 2024 availability check"]
        vantage=["Sydney (Australia)","Toronto (Canada)","Dublin (Ireland)","London (the United Kingdom)","Oregon (the United States)"]
 
   2025 USENIX  navigating-cookie-consent-violations-across-the-globe
        snapshots=10  cadence=over one week  longest span=0mo
        browsers=["Google Chrome"]  statefulness=stateless  consent=reject-all
        listVersion=["November 2023, ID: 5Y3LN","July 2021, ID: 9QK2"]
        vantage=["Ireland","United Kingdom","California, USA","Michigan, USA","Canada","South Africa","Singapore","Australia"]
 
   2025 WWW  before-after-the-effect-of-eus-2022-code-of-practice-on-disinformation
        snapshots=2  cadence=-  longest span=24mo
        browsers=["Playwright-controlled browser"]  statefulness=stateful  consent=accept-all
        listVersion=["accessed October 2023","compiled October 2023 and earlier"]
        vantage=["Netherlands","Greece","USA"]
 
   2024 IEEE-SP  targeted-and-troublesome-tracking-and-advertising-on-childrens-websites
        snapshots=7  cadence=-  longest span=1mo
        browsers=["Puppeteer/TRC desktop browser","Puppeteer/TRC emulated mobile browser"]  statefulness=stateless  consent=accept-all
        listVersion=["May 2022","June-July 2022 crawl snapshot"]
        vantage=["Frankfurt","Amsterdam","London","San Francisco","New York City"]
 
   2024 PETS  the-devil-is-in-the-details-detection-measurement-and-lawfulness-of-server-side
        snapshots=3  cadence=-  longest span=26mo
        browsers=["Firefox"]  statefulness=stateless  consent=no-interaction
        listVersion=["2021"]
        vantage=["Europe","EU"]
 
   2024 PETS  a-large-scale-study-of-cookie-banner-interaction-tools-and-their-impact-on-users
        snapshots=3  cadence=temporally close repeated measurements  longest span=1mo
        browsers=["Firefox 100.0"]  statefulness=stateful  consent=accept-and-reject
        listVersion=["generated 29/08/2022; X568N","not-stated"]
        vantage=["Frankfurt am Main, DEU","Council Bluffs, IA, USA","Frankfurt, DEU","Paris, FRA","Stockholm, SWE"]
 
   2023 USENIX  a-study-of-multi-factor-and-risk-based-authentication-availability
        snapshots=2  cadence=two months between suspicious-login attempts  longest span=n/a
        browsers=["Chrome 89.0","Firefox 91.0","Safari 15.0"]  statefulness=stateless  consent=not-stated
        listVersion=["generated June 21, 2021"]
        vantage=["Boston, Massachusetts","Sofia, Bulgaria","San Francisco, California","Auckland, New Zealand","Dallas, Texas"]
 
   2022 IMC  respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild
        snapshots=2  cadence=-  longest span=5mo
        browsers=["Google Chrome v88.0"]  statefulness=stateless  consent=not-stated
        listVersion=["14 February 2021","14–18 February 2021","August 2021"]
        vantage=["East US","not-stated"]
 
   2022 USENIX  a-large-scale-investigation-into-geodifferences-in-mobile-apps
        snapshots=31  cadence=daily  longest span=0mo
        browsers=["Selenium"]  statefulness=stateful  consent=no-interaction
        listVersion=["June 2020"]
        vantage=["Canada","Germany","USA","UK","Australia","Japan","Hungary","Kenya","Colombia","South Korea","Tunisia","Mexico","Singapore","Ukraine","India","Zimbabwe","Turkey","Russia","Venezuela","Bahrain","UAE","Egypt","Iran","Hong Kong","Ireland","Israel","not-stated"]
 
   2022 WWW  reproducibility-and-replicability-of-web-measurement-studies
        snapshots=12  cadence=daily  longest span=60mo
        browsers=["Chrome","Chrome headless","Firefox","Firefox headless"]  statefulness=stateless  consent=no-interaction
        listVersion=["2016-2021","2016-2021","2021-06-23","2021-06-24"]
        vantage=["Germany (EU)","Japan (AS)","United States (NA)"]
 
   2020 NDSS  encrypted-dns-privacy-a-traffic-analysis-perspective
        snapshots=5  cadence=two-week intervals  longest span=3mo
        browsers=["Firefox","Chrome"]  statefulness=stateless  consent=not-stated
        listVersion=["26 March 2018","26 March 2018","accessed 2019-09-11"]
        vantage=["Lausanne","Leuven","Singapore"]
 
   2020 PETS  missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix
        snapshots=6  cadence=simultaneous crawls / simultaneous measurements  longest span=0mo
        browsers=["Firefox"]  statefulness=stateful  consent=no-interaction
        listVersion=["February 2019","November 2019","November 2019"]
        vantage=["France"]
 
   2020 WWW  beyond-the-front-page-measuring-third-party-dynamics-in-the-field
        snapshots=3  cadence=-  longest span=0mo
        browsers=["Firefox 52"]  statefulness=stateful  consent=no-interaction
        listVersion=["2019-03-26, ID W9L9"]
        vantage=["Europe (DE)","North America (US)","Asia (JP)"]
 
   2020 WWW  the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing
        snapshots=2  cadence=variable / two seven-day periods separated by seven days  longest span=3mo
        browsers=["Firefox 68","Firefox 69"]  statefulness=stateless  consent=not-stated
        listVersion=["2019; accessed 29-July-2019","top 10,000; accessed 29-July-2019","top 10,000","data collected July 16–August 13, 2019"]
        vantage=["not-stated","Brazil","Texas, USA"]
 
   2019 WWW  measuring-political-personalization-of-google-news-search
        snapshots=7  cadence=daily  longest span=n/a
        browsers=["Firefox"]  statefulness=stateful  consent=not-stated
        listVersion=["two weeks in March 2017"]
        vantage=["United States","Oregon"]
 
   2017 NDSS  thou-shalt-not-depend-on-me-analysing-the-use-of-outdated-javascript-libraries-o
        snapshots=2  cadence=-  longest span=0mo
        browsers=["Chromium"]  statefulness=stateless  consent=no-interaction
        listVersion=["May 2016"]
        vantage=["US"]
 
   2016 IEEE-SP  cloak-of-visibility-detecting-when-machines-browse-a-different-web
        snapshots=3  cadence=-  longest span=3mo
        browsers=["Chrome","basic HTTP fetcher"]  statefulness=stateless  consent=no-interaction
        listVersion=["collected February, 2015-May, 2015"]
        vantage=["not-stated","United States"]
 
   2016 USENIX  tracing-information-flows-between-ad-exchanges-using-retargeted-ads
        snapshots=9  cadence=-  longest span=0mo
        browsers=["instrumented Chromium"]  statefulness=stateful  consent=not-stated
        listVersion=["January 2015"]
        vantage=["Boston, United States"]
 
   -- compact table (year | venue | slug | snapshots | distinct stated vantage locations)
   2026 | PETS | clicking-into-exposure-uncovering-privacy-risks-of-google-click-identifier-in-yo | 5 | 6
   2026 | PETS | privacy-vs-profit-the-impact-of-googles-manifest-version-3-mv3-update-on-ad-bloc | 5 | 1
   2026 | PETS | more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr | 5 | 1
   2025 | PETS | beyond-the-request-harnessing-http-response-headers-for-cross-browser-web-tracke | 18 | 1
   2025 | PETS | intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and | 4 | 1
   2025 | PETS | more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings | 6 | 5
   2025 | USENIX | navigating-cookie-consent-violations-across-the-globe | 10 | 8
   2025 | WWW | before-after-the-effect-of-eus-2022-code-of-practice-on-disinformation | 2 | 3
   2024 | IEEE-SP | targeted-and-troublesome-tracking-and-advertising-on-childrens-websites | 7 | 5
   2024 | PETS | the-devil-is-in-the-details-detection-measurement-and-lawfulness-of-server-side | 3 | 2
   2024 | PETS | a-large-scale-study-of-cookie-banner-interaction-tools-and-their-impact-on-users | 3 | 5
   2023 | USENIX | a-study-of-multi-factor-and-risk-based-authentication-availability | 2 | 5
   2022 | IMC | respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild | 2 | 1
   2022 | USENIX | a-large-scale-investigation-into-geodifferences-in-mobile-apps | 31 | 26
   2022 | WWW | reproducibility-and-replicability-of-web-measurement-studies | 12 | 3
   2020 | NDSS | encrypted-dns-privacy-a-traffic-analysis-perspective | 5 | 3
   2020 | PETS | missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix | 6 | 1
   2020 | WWW | beyond-the-front-page-measuring-third-party-dynamics-in-the-field | 3 | 3
   2020 | WWW | the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing | 2 | 2
   2019 | WWW | measuring-political-personalization-of-google-news-search | 7 | 2
   2017 | NDSS | thou-shalt-not-depend-on-me-analysing-the-use-of-outdated-javascript-libraries-o | 2 | 1
   2016 | IEEE-SP | cloak-of-visibility-detecting-when-machines-browse-a-different-web | 3 | 1
   2016 | USENIX | tracing-information-flows-between-ad-exchanges-using-retargeted-ads | 9 | 1
 
 
==========================================================================
11. CORPUS-WIDE ADOPTION PROBES FOR THE MODERN PINNING MECHANISMS
==========================================================================
 
-- mentions anywhere in the full text
   population: 5855 papers with readable full text, of 5859
   Docker OR containerised                                     308    5.3%
                                                              /\bdocker\b|\bcontaineri[sz]ed\b/i
      ...Docker alone                                          293    5.0%
                                                              /\bdocker\b/i
   image digest or sha256: image reference                       0    0.0%
                                                              /\bimage digest\b|\bsha256:[0-9a-f]{12}/i
   "Chrome for Testing", exact phrase                            0    0.0%
                                                              /chrome[-\s]?for[-\s]?testing/i
   Chrome for Testing, widened to the tooling and the acronym   10    0.2%   -> hand-read all 10: 0 genuine
                                                              /chrome[-\s]?for[-\s]?testing|googlechromelabs|chrome[-\s]?labs|known[-\s]good[-\s]versions|\bCfT\b/i
   WebREC or "web execution bundle"                              2    0.0%   -> hand-read all 2: 0 genuine
                                                              /\bWebREC\b|\bweb execution bundle/i
 
   Hand audits:
     * The widened Chrome-for-Testing probe: every hit is a false positive —
       CFT as "Combating the Financing of Terrorism" (three papers), "Control Flow
       Trimming", "Call Flow Tree", "Crash-Fault Tolerant", a certificate subject
       O=CFT, a CFt variable in an algorithm listing, CFT-IDF, and a fitness app
       named "CFT & PFT". The exact-phrase probe is 0, and so is the audited count.
     * WebREC: one hit is the WebREC paper itself; the other cites it in its
       reference list only. No paper in the corpus USES it.

PARTIAL  2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "Over twelve days, we saw a variation of up to 27% (max on day 3-80,274; min on day 9-58,951) in observed track..."
         paper.cols.txt — 4 run(s): 6+2+12+3 words
 
EXACT    2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "the number of distinct tracking domains remains almost stable during the experiment (variation of 3.5%)"
         paper.cols.txt
 
PARTIAL  2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "studies that analyze the ecosystem will find similar results, while studies that aim to analyze the extent of ..."
         paper.cols.txt — 4 run(s): 6+11+10+3 words
 
EXACT    2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "we find that the identified trackers on pages can vary by 25% based on the used browser configuration"
         paper.cols.txt
 
SPLICED  2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "even the same setup measures different levels of tracking on different days, which might lead to different con..."
         paper.cols.txt — 3 run(s): 6+11+4 words
 
EXACT    2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)"
         paper.cols.txt
 
PARTIAL  2023/IMC/on-the-similarity-of-web-measurements-under-different-experimental-setups
         "when comparing two different profiles, 48% of the underlying data varies"
         paper.cols.txt — 2 run(s): 2+9 words
 
PARTIAL  2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law
         "we restrict the analysis to the balanced panel of websites with an observed policy in both snapshots"
         paper.cols.txt — 3 run(s): 6+10+1 words
 
SPLICED  2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law
         "Standard errors are clustered at the website level to account for repeated observations of the same policy acr..."
         paper.cols.txt — 2 run(s): 11+8 words
 
PARTIAL  2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "The standard deviation of such requests is 8,203."
         paper.cols.txt — 2 run(s): 7+1 words
 
EXACT    2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "For our continuous measurement, we visit the top 1k sites from our website corpus (18,377 distinct pages) dail..."
         paper.cols.txt
 
PARTIAL  2022/WWW/reproducibility-and-replicability-of-web-measurement-studies
         "privacy measurements and analyses can vary up to 65% depending on the region"
         paper.cols.txt — 3 run(s): 2+10+1 words
 
EXACT    2023/IMC/on-the-similarity-of-web-measurements-under-different-experimental-setups
         "only 32% of the cookies appear in all profiles and 42% only in one profile"
         paper.cols.txt
 
SPLICED  2021/WWW/towards-realistic-and-reproducibleweb-crawl-measurements
         "Around 5% of content-providing domains show significant measurement bias across VP"
         paper.cols.txt — 2 run(s): 6+5 words
 
EXACT    2020/WWW/apophanies-or-epiphanies-how-crawlers-impact-our-understanding-of-the-web
         "variation of over 16% in the number of successful page loads"
         paper.cols.txt
 
PARTIAL  2026/PETS/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law
         "The balanced panel comprises N = 11,800 websites observed in both August and October 2023"
         paper.cols.txt — 3 run(s): 4+1+10 words
 
EXACT    2025/USENIX/web-execution-bundles-reproducible-accurate-and-archivable-web-measurements
         "could be conducted using WebREC as is"
         paper.cols.txt
 
EXACT    2026/PETS/the-empire-strikes-back-at-your-privacy-an-archaeology-of-tracking-on-government
         "We therefore label trackers using the June 2025 EasyList and EasyPrivacy snapshot for all years, keeping the l..."
         paper.cols.txt
 
7 quote(s) need a hand read

Four reviewers, all told explicitly that the author's context might not be exhaustive, all handed the page text, the report script and its unedited output, and these notes. The three focused passes ran in parallel first; the generic pass ran afterwards, on the corrected text.

Contemporaneous. Made during a different work item — a site-wide audit of how this wiki dates Alexa's retirement, whose full notes, queries, script and review log are in website_selection §13. Nothing else on longitudinal was touched: no corpus query was re-run and no corpus figure changed.

What was wrong. Not a wrong date — an imprecise one, on the page whose whole argument is that a pinned list id does not pin the frame. The provider-set table sampled 1 January of each year, so it showed Alexa present on 1 Jan 2023 and absent on 1 Jan 2024, and the sentence under it read:

//"Quantcast dropped out, Farsight came in, and then in 2023–24 Alexa was replaced by CrUX and Cloudflare Radar."//

“2023–24” is the gap between two sampled rows, not the date of the change. The footnote then gave Amazon's API retirement (15 December 2022) as the cause without giving the swap date at all — so the page put two different organisations' events side by side with neither dated to the day. That is the shape the “Alexa was discontinued on 1 August 2023” error on website_selection grew out of.

What it says now. All three provider transitions are dated in the sentence, and the Alexa one is dated to the day from the Tranco API rather than bracketed:

All four read from https://tranco-list.eu/api/lists/date/YYYYMMDD on 2026-09-03. The 31 July and 1 August rows are on the content page; 30 July and 2 August were read as controls, to check the change is a single-day step and not a run of churn. The prefix change from full to one million lands on the same day, which independently corroborates the claim tranco already makes about the prefix.

Whose date is whose. The distinction the amendment exists to enforce:

Eight months separate the second from the third, so Alexa's removal from the list was forced long before it happened. Any page naming one of these dates has to say which organisation it belongs to; the audit in website_selection §13.3 is the site-wide check that all 160 pages still do.

Verified after the last save. Counted from the wiki source and confirmed in the rendered DOM: 32 headings, 11 ((…)) footnotes matching 11 distinct fn__N ids, 14 tables, 0 wikilink2 (red links), 45 citation markers over 32 distinct keys rendering as 90 bibtex_citekey spans and 32 numbered references — all unchanged from before the edit except the table and the sentence. node scripts/check_wrap.mjs clean before each save. An earlier draft of this section quoted “34 headings, 33 footnotes”; both were miscounts of rendered elements rather than source constructs, and are explained in website_selection §13.7.

Not changed, on purpose. The four 1 January rows stay: they are what the API returns for those dates, and re-sampling the whole table around each transition would be a different table answering a different question. The pre-swap id W9ZN9 is only on longitudinal — the natural second home is tranco's id table, and adding it there is filed as separate work rather than folded into this edit, because that page has its own provenance page and its own review obligation.

• ahmad2026_ipfp was one of two keys for the same paper in bibliography. The wiki-wide consolidation of 2026-09-04 (drain item dedup-regional-filter-lists-bibkey) kept ahmad2026_more and deleted the other entry.
• 1 citation marker on longitudinal was repointed to the kept key. No prose on either page changed, and no figure moved. Statements above that name the deleted key describe the state when they were written. Full query log and the invariants checked before saving: bibliography.

§9 above named one follow-up as the most valuable: “Whether a paper held a pin constant across waves. The extraction records what a paper stated, once. Every pin figure on the content page is a ceiling … Closing this needs a hand-coded sample of the 250, reading each paper's per-wave methods description.” This section is that work. It was not a sample: all 250 were read.

cd /workspace/artifacts/wiki
node scripts/pins_population.mjs > out_pins/population.jsonl
node scripts/pins_excerpt.mjs
node scripts/pins_valsample.mjs > out_pins/valsample.tsv
node scripts/report_pins_audit.mjs        > scripts/report_pins_audit-output.txt
node scripts/report_pins_audit.mjs --wiki                       # the page's tables
node scripts/pins_quotecheck.mjs          > scripts/pins_quotecheck-output.txt
node scripts/pins_agreement.mjs           > scripts/pins_agreement-output.txt

Both derived files were re-run on 2026-09-07 and diffed against the committed copies: population.jsonl and valsample.tsv reproduce byte-identically. node scripts/report_longitudinal.mjs was also re-run and diffed against scripts/report_longitudinal-output.txt — zero lines differ, so nothing on the rest of the content page moved under this edit.

The predicate is crawled AND max(temporal[].snapshots) > 1, copied out of report_longitudinal.mjs rather than re-implemented, so §3's repeated population and this one agree by construction rather than by luck.

report_pins_audit.mjs refuses to print anything if fewer than 250 codings are present, rather than silently shrinking its denominator:

if (coded.length !== pop.length) {
  console.error(`\nFATAL: coded ${coded.length} of ${pop.length}. Figures would be computed over the wrong population.`);

It has the same guard on the citekey map: if the map and the all-four-stated set disagree by one slug, the exemplar table would silently lose a row, so it exits instead.

250 papers is 20.9 MB of .cols text. Each coder was given a packet instead: the first 3,000 characters, plus context windows around two tiers of keyword matches (wave/repetition language, “same/unchanged/pinned”, statefulness, the named ranking lists and filter lists, browsers, cloud regions), merged and capped at 32,000 characters — and the full-text path in the packet header, with an instruction to open it whenever the packet is not enough. The packets instruct the coder to open the full text whenever the windows are not enough. Whether they did is not logged, so the recall of the packet rule is bounded only by the blind full-text pass below, not by any record of what each coder read.

The packet is a recall risk, and it was measured rather than assumed — that is what the blind full-text second coder in §18.4 is for. The answer, in short: the packet does not lose “fixed” evidence, it loses the absence of it. See the direction-of-disagreement table below.

30 of the 250 were drawn with a seeded shuffle (mulberry32, seed 20260907) and re-coded from the full .cols text by a second coder given CODEBOOK_VALIDATION.md, told explicitly not to open out_pins/coded/ or out_pins/excerpts/. Ten agents, three papers each. No coder saw another's output.

An earlier draft of this section said κ on “all four fixed” is low only because the class is rare, “not because the coders disagree about which papers they are”. That was wrong and the files say the opposite. Five papers are called all-four by at least one coder; the two agree on two of the five, and disagree on three. The class is indeed rare at n = 30, but the disagreement is real, and the 15-of-124 headline should be read with that in mind.

Direction of disagreement. This is the part that changes how the content page is written:

The excerpt coder calls the browser fixed seven times where the full-text coder finds it never stated, and only once the other way; statefulness leans the same way, four to one. The vantage point goes the opposite way — the full-text coder finds four vantage statements the packet's windows missed, because vantage is often mentioned once, far from any wave language, in an infrastructure or ethics paragraph.

The most likely cause of the browser lean is a design mistake that is worth recording: the packet header prints the structured extraction's own values (browsers=[“Selenium”], statefulness=not-stated, …) so the coder can see what a machine thought. The codebook's first rule tells the coder not to infer from it. The agreement data suggests that instruction was not fully obeyed. A rerun should put that line at the end of the packet, or drop it.

Consequence for the content page: the browser row of the estimate column is more likely too high than too low, and the page says so.

Every disagreement is printed paper by paper in the unedited output in §18.12.

Every non-empty quote must appear in the paper it is attributed to. 1,000 pin codings, 464 with no quote (all of them not-stated, which the codebook permits and the script verifies: 0 codings carry a value other than not-stated with no quote), 536 checked.

The naive substring test is wrong here, and getting it wrong once already produced a scare. Run against paper.cols.txt alone it reported 49 unlocated quotes (9.1%), which reads as a fabrication rate. It is not. The first one inspected was:

• coder's quote: “We implemented FourthParty as an extension to Mozilla Firefox.”
• what .cols actually holds: We implemented FourthParty as an extension to Mozilla \n B. Design Principles Firefox. It currently instruments

The two-column repair still splices a quarter of pages, and a sentence crossing a column boundary comes back with the neighbouring column injected into the middle of it. A coder who de-splices that correctly writes a quote that is a substring of nothing. Failing it would call a correct reading a hallucination. So the check now searches paper.cols.txt, paper.norm.txt, paper.txt and the packet, in four escalating modes, the last of which cuts the quote into 6-word shingles and passes at 80% found.

The counts above are after the one correction described below. Before it: 488 exact, 43 unlocated (8.0%).

All 43 were then read by hand — three sub-agents, each given the paper's three renderings and the PDF, asked to classify each quote as a faithful de-splice, a supported paraphrase, an unsupported paraphrase, or a quote lifted from the wrong part of the paper, and independently to say whether the paper supports the coded pin value at all.

The one bad coding, and what was done about it:

• [8Singanamalla, Sudheesh; Paracha, Muhammad Talha; Ahmad, Suleman; Hoyland, Jonathan; Valenta, Luke; Safronov, Yevgen; Wu, Peter; Galloni, Andrew; Heimerl, Kurtis; Sullivan, Nick; Wood, Christopher A.; Fayed, Marwan (2022): "Respect the ORIGIN!: a best-case evaluation of connection coalescing in the wild", in: Proceedings of the ACM Internet Measurement Conference. (DOI)], vantage = changed. The evidence quote — “the ORIGIN Frame implementation was deployed globally at over 275 points of presence” — is a real sentence, correctly de-spliced, and describes the CDN's own deployment footprint, not where the authors' crawler ran. The paper's methods section says the opposite: “We used the same machines throughout our evaluations for consistency, and saw no observable change in datacenter provisioning or capacity over time.” The blind second coder had independently coded that pin same-implied, which is the same direction.
• Corrected to same-explicit with the methods-section sentence as its quote, and the correction is recorded in the paper's notes field. The same pass offered a stronger quote for that paper's state pin (“Client caches are intentionally deleted between measurements in all of our active measurements”); the quote was swapped, the value was already same-explicit and did not move.
• What moved on the content page: the vantage row over the 124 went from 25/44/10 to 26/45/9 (words / estimate / moved), and the all-pins fixed-versus-not agreement from 80.8% to 81.7%. The all-four figure, the 124, and the 6-of-23 survivor count did not move.

The verdict files are out_pins/qaudit/verdict_1.tsv … verdict_3.tsv. The lesson is the check, not the count. A plain substring test on paper.cols.txt alone would have reported a 9.1% fabrication rate here, of which the real rate is 0.2% (1 of 536) — and the one real defect was not a fabrication at all but a quote from the wrong section, which no substring test can catch. One reviewer flagged a further boundary case worth recording rather than changing: two codings use same-explicit on the strength of a pinned identifier in papers whose repeatKind is single-crawl, where the codebook's rule 4 would suggest same-implied. Both readings are inside the codebook as written; the rule needs tightening before a rerun, and neither paper is in the 124.

Also corrected in the same edit, outside the four-pin work. Four external claims the currency reviewer found stale — Quantcast's exit date, the OpenWPM version, the Playwright browsers.json source, and ICLab's maintenance status. Each is described in §18.10 Pass C with what was re-fetched.

Not changed, on purpose. The cadence, span, probe and Tranco-ID figures are still shares of 250 rather than of 124. Recomputing them is a different piece of work with its own review obligation; folding it into this edit would have meant republishing a dozen tables under a population change that nothing in this hand-coding validates for them. The content page says plainly that they were not recomputed, which is the honest state, and it is filed below as the follow-up.

• Two populations are published, and the 124 is the one the prose leans on. Publishing only the 250 would keep the page consistent with itself and answer the wrong question; publishing only the 124 would break every cross-reference to biases. Both tables are emitted by the script; the content page carries the 124 and points at the 250 version here.
• same-implied counts toward the estimate. It is the common case — one methods paragraph, no statement that anything changed — and excluding it would give a “floor” of 2 papers in 250, which is a measurement of how rarely people write “unchanged”, not of how often things were unchanged. It is published as a separate column so a reader can take the strict reading, and the page calls the estimate generous.
• A list re-downloaded each wave is changed, not same. Scientifically that is often the right thing to do; the codebook measures whether the frame is constant, not whether the authors were wise. 28 of the 124 are coded changed on the list pin, and several are deliberate and defensible.
• The ceiling is not a ceiling for the list pin. The hand-coded estimate for the population list (95 of 124) is above the structured figure (86). They measure different things: the extraction looks for population.listVersion, a version string; the hand-coding asks whether the frame was constant, which “the same 200 sites throughout” satisfies without any version string. 62 of the 250 are in that gap. Presenting the structured column as a ceiling without saying this would be wrong, so the content page's column is now labelled “stated at all” and the divergence has its own paragraph.
• The 23-paper exemplar table was kept, not deleted. It is the clearest possible demonstration of what “stated” was hiding, and deleting it would erase the correction along with the error.
• repeatKind was added to the codebook after a pilot batch showed the mechanical predicate misfiring. Without it the four-pin figures would be computed over a population half of which cannot have waves. This turned out to be the largest single finding.

• Whether a pin was actually held fixed, as opposed to described once and never contradicted. same-implied is 47 of the 124 on the list pin and 55 on the browser. Only the authors know. Nothing short of asking them closes this, and the page publishes the strict and generous readings side by side rather than picking one.
• Whether the reliability figures hold at n=250. They were measured on 30. κ = 0.60 on exact values is moderate; the confidence interval at n=30 is wide and is not published because it would imply a precision the design does not have.
• The other share-of-250 figures on the content page. Cadence, span bands, the probe table and the Tranco-ID rate are all still computed over the 250. The 124 exists now, so they could be recomputed; they have not been, and the page says so.
• Whether the pin figures differ by venue. n per venue-year is small and the 124 splits into groups of two and three. Not attempted.
• Whether a paper that pinned got a different answer than one that did not. Unchanged from §9: this remains the open question the whole page rests on and no study in the corpus answers it.

Four reviewers, each told explicitly that the author's context may not be exhaustive, each handed both draft pages, all six scripts, their unedited output and these notes. Every finding was accepted except one, marked “noted, not changed” in Pass B. Two of the four independently found the same defect, which is the only evidence there is that the layer is worth its cost — and the generic pass, which nearly did not run at all, produced the most serious finding of the four (§18.11).

Re-ran all six scripts and confirmed every committed -output.txt, population.jsonl and valsample.tsv reproduces byte-identically, and that pins_population.mjs lines 17 and 19 are character-for-character the predicate in report_longitudinal.mjs.

1. Stale table cell. The 23-row exemplar table's vantage cell for [8Singanamalla, Sudheesh; Paracha, Muhammad Talha; Ahmad, Suleman; Hoyland, Jonathan; Valenta, Luke; Safronov, Yevgen; Wu, Peter; Galloni, Andrew; Heimerl, Kurtis; Sullivan, Nick; Wood, Christopher A.; Fayed, Marwan (2022): "Respect the ORIGIN!: a best-case evaluation of connection coalescing in the wild", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] still read changed. The table had been pasted from –wiki output generated before the §18.5 correction was applied. Accepted and fixed by regenerating the table; it was the only divergent cell of 207.
2. Wrong hand-written count. The prose said “nine turn out not to be repeated crawls at all”. Recounting from the coding files: of the 23, eight are parallel-not-temporal, one of which is a survivor, so among the 17 non-survivors seven are not repeat crawls and ten are. The script never prints this aggregate, which is exactly why it was wrong. Accepted and fixed.
3. The structural guard in build_provenance_pins.py asserted much less than it claimed. It compared heading, table and file-block counts over the whole output against the whole live page. Because the appended section adds eleven headings of its own, the check had eleven headings of slack: the reviewer mutation-tested it by deleting a heading from the preserved live content and got headings: 27 → 37 and exit 0. Accepted and fixed — the counts are now compared on the preserved text alone, which must equal the live page byte for byte, and the appended section is separately required to be non-empty. This is a page-matches-its-own-script failure of the kind the guard existed to prevent, and reading the script would not have found it; mutating it did.
4. Mutation-tested and confirmed sound: the partial-population refusal in report_pins_audit.mjs (deleting one of 250 codings gives FATAL: coded 249 of 250), the citekey-map guard (removing one slug exits 1), pins_quotecheck.mjs (an injected fabricated quote raises NOT FOUND from 42 to 43), pins_agreement.mjs (a removed coder-B file drops n to 29 and prints MISSING rather than silently continuing at 30), and the missing-file and empty-file guards in the builder.
5. Hand-computed Cohen's κ for the list pin from the raw pairs: observed 0.667, expected 0.351, κ = 0.486 → 0.49, matching the script. No bug.
6. Checked the figures outside the edited window against report_longitudinal-output.txt and found none stale.

1. All 32 keys on the content page and all 8 on this page resolve against the live literature:bibliography (850 entries, fetched today, not a local copy). The content page's key set is unchanged from the live version; this page gains exactly one key, [8Singanamalla, Sudheesh; Paracha, Muhammad Talha; Ahmad, Suleman; Hoyland, Jonathan; Valenta, Luke; Safronov, Yevgen; Wu, Peter; Galloni, Andrew; Heimerl, Kurtis; Sullivan, Nick; Wood, Christopher A.; Fayed, Marwan (2022): "Respect the ORIGIN!: a best-case evaluation of connection coalescing in the wild", in: Proceedings of the ACM Internet Measurement Conference. (DOI)], which is already in the bibliography and already cited on the content page. No entry added, none changed.
2. Verified all 23 rows of out_pins/citekeys.json against the corpus index — title, venue, year and first-author surname — including that fouad2020_missed and fouad2024_devil are two distinct papers and are not conflated. No swapped keys.
3. Independently found the same stale vantage cell as Pass A, by diffing the published table against a fresh –wiki run, and confirmed all three quotes in the §18.5 account by locating them in the paper itself.
4. Spot-checked 24 evidence quotes (10 changed, 14 same-explicit) against all three renderings: 24 of 24 located and supporting their coded value. Flagged one as loose rather than wrong — the Zoom-apps browser quote is about a crawler and parser rebuild rather than a browser version. Noted, not changed: the coding is changed either way and the codebook's browser pin covers the driven stack.
5. Verified the “12 further papers outside this 23” arithmetic (18 all-four minus the 6 in the table) and every external figure quoted from a cited paper.

This pass reviewed the whole of both pages, not the diff, and found four stale external claims — all of them in text this edit did not touch. All four accepted:

1. Quantcast's exit date was imprecise. The page said “dropped out after 1 April 2020”, which was only as precise as the rows sampled in the table above it. Bisecting the Tranco API on 2026-09-07 puts it between 14 and 15 August 2020 (date/20200814 → GK9K, providers include quantcast; date/20200815 → Z7NG, providers do not). Re-verified independently before the edit. Every other row and date in that table and its footnote was checked against the live API and is exact.
2. OpenWPM had moved on. The page cited v0.36.0 / Firefox 154.0 as its worked example; v0.37.0 shipped on 6 September 2026 pinning Firefox 155.0.1. Confirmed via releases/latest and by reading install-firefox.sh (# FIREFOX_155_0_1_RELEASE). Updated, with the drift itself now noted in the footnote, since it is the paragraph's own point.
3. The Playwright footnote quoted main, not a release. It cited browsers.json from the development branch — revision 1241, browserVersion 152.0.7977.54 — as evidence that pinning Playwright pins a browser. That snapshot shipped in no release, so it was the wrong evidence for the claim. Re-fetched from the tagged release v1.63.0: revision 1243, browserVersion 153.0.8010.12. The reviewer reported different values for that tag than the author's own fetch did; a third fetch during the generic pass returned revision 1243, browserVersion 153.0.8010.12, matching the author's, so the published numbers stand.
4. ICLab is dormant. The page called Censored Planet and ICLab “continuous, longitudinal and public” in one breath. Censored Planet is active. ICLab's site serves an expired TLS certificate as of 2026-09-07 (curl fails with exit 60 without -k), its data page returns 404, and its news stops in 2019 — all re-verified before the edit. The sentence now recommends the paper and warns off the feed.
5. Confirmed accurate and unchanged: the HTTP Archive changelog 404 (still 12 entries, last 2017-06-01, on master after the repo rename), the HTTP Archive cadence text, all seven Tranco list ids, the EasyList commit cadence, the Public Suffix List footnote, the Chrome for Testing endpoints, and the WebREC repository facts. Neither page names an RFC or W3C specification, so that category is empty.

Run last, on the published pages, after A–C were applied. Findings and their disposition are in §18.11.

Commissioned, nearly missed, and completed late. A Claude Fable pass was given both published pages, the scripts and their output, and asked for whatever the three focused passes were not looking for — overstated claims, structure, a section that does not answer its own question, and specifically whether same-implied is doing too much work, whether the split between the 250 and the 124 is legible, whether n = 30 and κ = 0.60 are being asked to carry more than they can, and whether the page states plainly enough that both coders and the adjudicator were LLMs reading LLM-extracted data. The run's budget all but ran out before it returned, and this section was published saying it never would; it arrived afterwards and its findings are below.

This is worth stating rather than quietly leaving a gap, because the generic pass is the one that catches overstatement, and §18 is the section most at risk of it. The specific things a reader should discount accordingly:

• The estimate column rests on same-implied, which is a judgement that silence about change means no change. It is 47 of 124 on the list pin and 55 of 124 on the browser. The strict reading is published beside it and is much smaller.
• The reliability figures are n = 30, and the κ for “all four fixed” (0.52) is computed on three and four positives respectively.
• Every coder in this exercise was a language model, reading text extracted from PDFs by a pipeline, coding a schema produced by another language model. The blind second coder and the quote adjudication bound the error; they do not remove it, and no human read any of the 250 papers.

Re-running this pass was filed as follow-up work — and then it returned.

Findings, and what was done. The pass returned after the first save of this section. Applied to the content page: the disclosure that both coders and the adjudicator were language-model agents reading LLM-extracted text, which the content page had nowhere stated and which “hand-coded” actively obscures; the softening of “two say so in words”, after the reviewer read the two same-explicit quote sets and found them to be permanent identifiers and one-off constructions rather than statements that anything was unchanged; Fisher exact p-values on the span reversal (0.10, 0.08 and 0.69 across three cuts, none significant) in place of “yes, and more sharply”; the floor-versus-false-positive contradiction in the lead; and a flag that one of the 15 all-four papers is, by its own coder's note, a cellular walk test rather than a web crawl. Applied here: the corrected κ defence in §18.4, the deletion of an unverifiable claim about coders opening the full text, and the resolution of the Playwright value discrepancy.

Accepted and not done, for budget: re-adjudicating the 73 same-explicit codings against the codebook's own wording; re-coding the 5G paper and recomputing 15/124 as 14/123; reading the 12 all-four papers outside the exemplar table and printing their notes; adjudicating the seven browser disagreements, which the reviewer argues may be the full-text coder's misses rather than the excerpt coder's over-calls, making §18.4's directional claim unsafe; publishing the κ confidence interval; recomputing the repeated-versus-single-shot comparison over the 124; per-table “of 250, unaudited” markers; and the waveCount-stratified middle reading of same-implied. Each is recorded as follow-up rather than silently dropped. Until they are done, §18.4's “the browser row is more likely too high than too low” and the content page's browser sentence rest on an untested attribution of the disagreement to one coder.

Everything below is read from the committed file by scripts/build_provenance_pins.py at build time. None of it is retyped.

# Four-pin hand-coding codebook — design:longitudinal
 
You are hand-coding papers for a systematization page on measuretheweb.org. The
page currently publishes, for 250 repeated-crawl papers, how many **stated** each
of four crawl parameters. "Stated once" is only a CEILING on "held the same in
every wave". Your job is to turn the ceiling into an estimate by reading what the
paper actually says about its repeated measurement.
 
## Step 0 — is this really a repeated crawl by the authors?
 
The 250 were selected mechanically (`temporal.snapshots > 1` in a structured
extraction). That predicate has false positives. Code `repeatKind` as exactly one:
 
| value | meaning |
|---|---|
| `own-repeat-crawl` | the authors themselves ran their crawler more than once over time and the waves are compared or aggregated |
| `own-repeat-nonweb` | repeated measurement, but not a web crawl the four pins apply to (e.g. repeated DNS probing, repeated app scraping, repeated API pulls, network scans) |
| `parallel-not-temporal` | the "snapshots" are parallel conditions run at one time (browsers, vantage points, profiles), not waves separated in time |
| `reused-waves` | the multiple time points come from a dataset or archive somebody else collected (web archive, existing corpus), not from the authors' own repeated crawling |
| `single-crawl` | the paper ran one crawl; the snapshot count is a mis-extraction (page counts, record counts, number of sites, number of runs of an unrelated thing) |
| `unclear` | you genuinely cannot tell after reading |
 
Also record `waveCount`: the number of distinct time-separated waves of the
authors' OWN crawl, as an integer, or `null` if not determinable. A continuous
daily/weekly crawl over N periods counts as N waves if the paper says so;
otherwise `null`.
 
## Step 1 — the four pins
 
Code each of the four independently. **The question is not "did they say it" but
"was it the same in every wave".**
 
* `list` — the **population / seed list**: which sites or targets were crawled
  (Tranco, Alexa, CrUX, Majestic, a custom seed set …). NOT the filter list.
* `browser` — the browser (and version, if given) the crawler drove.
* `vantage` — the network location the crawl ran from (country, cloud region,
  university network, VPN exit, residential proxy …).
* `state` — stateful vs stateless crawling: fresh profile per site, persistent
  cookie jar, shared profile, cleared between waves, etc.
 
Each pin takes exactly one value:
 
| value | when to use |
|---|---|
| `same-explicit` | the paper says in words that this was the same / unchanged / the same identifier across the waves. A single permanent identifier used for all waves (one Tranco list ID, one pinned browser version, one container image) counts. |
| `same-implied` | the paper gives ONE value in a single methods description that plainly covers the whole repeated measurement, and never says it changed. This is the common case. It is weaker evidence than `same-explicit` — do not upgrade it. |
| `varied-deliberate` | the value differs across waves ON PURPOSE, as part of the design (rotating vantage points, comparing browsers across waves). |
| `changed` | the value changed or was refreshed between waves — the top list re-downloaded per wave, the filter list or browser updated, a move to a different host — whether stated as deliberate or as drift. |
| `not-stated` | the paper never gives the value at all. |
| `unclear` | the value is given somewhere but you cannot tell which wave(s) it applies to, or the description is contradictory. |
 
Rules that matter:
 
1. **Do not infer from the structured extraction line at the top of the packet.**
   It is shown so you can see what a machine thought; it records a value stated
   *once*. If the paper's text does not support it, code what the text says.
2. **A list that is re-downloaded each wave is `changed`, not `same-implied`** —
   even though that is often the right thing to do scientifically. We are
   measuring whether the frame is constant, not whether the authors were wise.
3. **"We used Chrome" with no version, across a two-year study, is
   `same-implied` for the browser pin** only if the paper describes one crawler
   setup for the whole study. If the paper says the crawler was rebuilt, updated,
   or migrated, it is `changed`.
4. If `repeatKind` is `single-crawl` or `parallel-not-temporal`, still code the
   four pins, but the "across waves" question is vacuous — use `same-implied`
   only if one configuration is described, and note it in `notes`.
5. `not-stated` is a finding, not a failure. Do not guess.
 
## Step 2 — evidence
 
For each pin, give `quote`: a **verbatim** substring of the packet, ≤ 220
characters, that is your evidence. Copy it exactly — it will be checked against
the source file automatically, and a paraphrase will fail. If your value is
`not-stated`, set `quote` to `""`.
 
Set `confidence` per paper: `high` | `medium` | `low`.
 
## Output
 
For EACH paper write one file `out_pins/coded/<KEY_WITH_SLASHES_AS___>.json`
(same base name as its packet, `.json` instead of `.txt`), containing:
 
```json
{
  "key": "2024/PETS/some-slug",
  "repeatKind": "own-repeat-crawl",
  "waveCount": 3,
  "pins": {
    "list":    {"value": "same-explicit",  "quote": "..."},
    "browser": {"value": "same-implied",   "quote": "..."},
    "vantage": {"value": "not-stated",     "quote": ""},
    "state":   {"value": "changed",        "quote": "..."}
  },
  "confidence": "high",
  "notes": "one or two sentences: what the paper actually did across waves"
}
```
 
Write valid JSON. No markdown fences in the file.

# Blind re-coding — second coder
 
You are the SECOND coder in a two-coder reliability check. A first coder has
already coded these papers; you will not see their answers, and you must not go
looking for them. Files under `/workspace/artifacts/wiki/out_pins/coded/` are the
first coder's output — **do not open that directory**.
 
Read `/workspace/artifacts/wiki/out_pins/CODEBOOK.md` and apply it exactly.
 
Two differences from the first pass:
 
1. **You read the FULL paper text**, not a keyword excerpt. The path is given to
   you. Read the whole methodology and any limitations section.
2. **Write your output to `/workspace/artifacts/wiki/out_pins/coded_val/`**, same
   file naming (`<year>__<venue>__<slug>.json`), same JSON schema.
 
Everything else — the `repeatKind` values, the six pin values, the verbatim
`quote` requirement, `confidence`, `notes` — is unchanged. Code what the paper
says, not what you think it probably did.

pins_population.mjs

#!/usr/bin/env node
// Emit the 250 repeated-crawl papers that design:longitudinal's four-pin table
// is computed over, with the extraction's own value for each pin, as JSONL.
//
//   node scripts/pins_population.mjs [--run <dir>] > out_pins/population.jsonl
//
// The population predicate is copied verbatim from scripts/report_longitudinal.mjs
// (`crawled` AND max stated temporal.snapshots > 1) so the two agree by construction.
import fs from 'node:fs';
import path from 'node:path';
 
const argv = process.argv.slice(2);
const RUN = argv.includes('--run') ? argv[argv.indexOf('--run') + 1] : '/workspace/publications_dataset/data/extract/run1';
const FULLTEXT = '/workspace/publications_dataset/data/fulltext';
 
const P = fs.readFileSync(path.join(RUN, 'extractions.jsonl'), 'utf8').split('\n').filter(Boolean).map((s) => JSON.parse(s));
const crawled = P.filter((p) => p.crawlConfig != null || p.studyTypes.includes('automated-web-crawl'));
const snapOf = (p) => Math.max(0, ...p.temporal.map((t) => (t.snapshots == null ? 0 : t.snapshots)));
const repeated = crawled.filter((p) => snapOf(p) > 1);
 
const SENT = /^(?:not[- ]stated|unknown|n\/a)$/i;
const statedListVersion = (p) => p.population.some((x) => x.listVersion != null && !SENT.test(x.listVersion.trim()));
const namedBrowser = (p) => p.crawlConfig != null && p.crawlConfig.browsers.length > 0;
const statedVantage = (p) => p.vantage.some((v) => v.locations.some((l) => !SENT.test(String(l).trim())));
const ccStated = (p, f) => p.crawlConfig != null && !['not-stated', 'not-applicable'].includes(p.crawlConfig[f]);
 
for (const p of repeated.sort((a, b) => a.year - b.year || a.venue.localeCompare(b.venue) || a.slug.localeCompare(b.slug))) {
  const key = `${p.year}/${p.venue}/${p.slug}`;
  const f = path.join(FULLTEXT, String(p.year), p.venue, p.slug, 'paper.cols.txt');
  process.stdout.write(JSON.stringify({
    key, year: p.year, venue: p.venue, slug: p.slug, title: p.title,
    snapshots: snapOf(p),
    cadence: p.temporal.map((t) => t.cadence).filter(Boolean),
    spans: p.temporal.map((t) => [t.spanStart, t.spanEnd]).filter(([a, b]) => a || b),
    modes: [...new Set(p.temporal.map((t) => t.mode))],
    x_listVersion: statedListVersion(p),
    x_browser: namedBrowser(p),
    x_vantage: statedVantage(p),
    x_statefulness: ccStated(p, 'statefulness'),
    v_listVersion: [...new Set(p.population.map((x) => x.listVersion).filter((s) => s && !SENT.test(s.trim())))],
    v_sourceList: [...new Set(p.population.map((x) => x.sourceList).filter(Boolean))],
    v_browsers: p.crawlConfig ? p.crawlConfig.browsers : [],
    v_statefulness: p.crawlConfig ? p.crawlConfig.statefulness : null,
    v_vantage: [...new Set(p.vantage.flatMap((v) => v.locations).filter((l) => l && !SENT.test(String(l).trim())))],
    textPath: f,
    textBytes: fs.existsSync(f) ? fs.statSync(f).size : 0,
  }) + '\n');
}