Working log behind Measuring Privacy Policies and Terms. Corpus-wide caveats — which venues are in, how papers were selected, what the extraction gets wrong — are on corpus and are not restated here. Citations use the shared bibliography; this page adds no entries of its own. No ~~DISCUSSION~~ block: comments belong on the content page.

Voice here is a working log, not prose. It is read by somebody checking a number.

Three populations. They are not interchangeable and no table on the content page mixes them.

Overlap: POLICY ∩ PROBE = 56; POLICY only = 46; PROBE only = 21. The 46 enum-only papers are the reason a title probe alone is not enough — they are Alexa-skill, IoT-companion-app and VR audits whose policy analysis is one component of a wider study.

report_policies.mjs asserts three invariants at load and throws rather than drifting: POLICY ⊆ UNION, PROBE ⊆ UNION, and UNION has no duplicate venue/year/slug keys.

Title+summary probe (defines PROBE, and with it UNION). Nine alternatives, case-insensitive, tested against title • summary:

privacy polic|privacy notice|terms of service|terms and conditions|privacy label|data safety|nutrition label|policy text|privacy statement

The 2026-09-02 gap analysis that proposed this page used four alternatives (privacy polic|privacy notice|terms of service|privacy label) and reported a union of 121. The nine-alternative form gives 123. Both are recorded because probe width decides the claim; the page quotes the nine-alternative figure and says so.

Whole-corpus lineage scan (added 2026-09-10, same script). For each of the fifteen artefacts in the lineage, a case-insensitive scan of all 5,869 readable paper.cols.txt files, counting papers whose text names it. This is a different question from the extraction fold in §4 of the report and gives systematically larger answers (Polisis 13 by extraction, 68 in text; PolicyLint 18 and 63). The distinction is the fix for this run's worst error: “no corpus paper's extraction records PrivaSeer” is true, “nobody in these venues cites PrivaSeer” is false, and only the full-text scan can tell the two apart. One artefact needs a different pattern in full text than in the extraction — MAPS collides with Google Maps and the Play Store's Maps & Navigation category, so the full-text pattern is the paper's title. The override and its reason are printed by the script; the shared regex list lives in policy_fold.mjs so the two scripts cannot drift.

Full-text probes (policies_fulltext_probe.mjs): thirteen questions, each with a narrow and a wide pattern, run over the 123 UNION papers' paper.cols.txt. The page quotes the narrow form throughout and labels it an upper bound on a candidate set — a probe counts papers whose text contains a phrase, not papers that did the thing. The wide form is printed beside it in the output below so a reader can see how far the answer moves: “reports dead policy links” is 7 narrow and 64 wide, and the page's “only 7 of 123” claim would be a different claim at the wide width. Three of the thirteen are not narrow/wide pairs but different questions, and the output marks them so.

scripts/policy_fold.mjs. Two folds, both ordered — first match wins — and both leaving anything unmatched as raw residue that report_policies.mjs prints in full.

• population[].sourceList → 13 families (app stores, ranking lists, the four annotated corpora, archives, participant panels, other app ecosystems, and an honest custom / hand-built list bucket). Residue: 138 distinct strings, printed in full in §3 of the output below.
• tools[].name + otherToolsMentioned[].name → 23 families (the policy-analysis lineage, the NLP substrate, browser automation, boilerplate strippers, language detection, readability metrics). Residue: 572 distinct names, printed in full in §4.

Known limits of the fold, recorded rather than hidden:

• The tool fold is deliberately narrow. It maps the policy lineage and the NLP substrate and leaves general-purpose tools unmapped, because guessing at families for 572 names would produce a table nobody could audit. That is why the residue is large.
• participant panel is folded as its own family and kept out of every “where policies come from” table — Prolific is where the participants came from, not where the policies came from.
• Alexa list (the ranking) is separated from Alexa Skills Store (the voice-app marketplace) by a negative lookahead. Strings like Alexa skill marketplaces satisfy neither rule and land in the residue, where they are visible.
• MAPS is matched with an anchored, case-sensitive /^maps$/ to avoid folding the word “maps”; a paper writing “MAPS pipeline” falls to the residue.
• No folded free-text count is published as a percentage anywhere on the page. They appear as rankings only.

scripts/policies_quotecheck.mjs. Every measured figure the page prints from a paper is a needle taken from the paper's own words, not from the extraction's evidence.quote — the point is to catch an extraction error, so checking the extraction against itself would prove nothing. Each needle is searched in four renderings: paper.cols.txt, paper.norm.txt, paper.txt and a pypdf extraction of paper.pdf cached under cache/pypdf/. Whitespace, soft hyphens, curly quotes and the several Unicode dashes are normalised before matching.

Result: 70 needles, 70 located, 0 missing. 69 in paper.cols.txt; 1 only outside it — [2Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]'s “corpus are from 2009-2019” is spliced by the de-columning and was found only by pypdf. A .cols-only check would have reported a false MISS on it. Full output below.

Seventeen of the 70 needles were added on 2026-09-10. Ten came from the whole-page number guard, which flagged ten figures the page printed that no earlier needle covered: [5Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)]'s 10,375/46,472, 9,523/46,472 and 15.7%; [6Andow, Benjamin; Mahmud, Samin Yaseer; Whitaker, Justin; Enck, William; Reaves, Bradley; Singh, Kapil; Egelman, Serge (2020): "Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck", in: Proceedings of the USENIX Security Symposium. (Link)]'s 37.1% and 31.1% (14,409/45,603); [1Ali, Mir Masood; Balash, David G.; Kodwani, Monica; Kanich, Chris; Aviv, Adam J. (2024): "Honesty is the Best Policy: On the Accuracy of Apple Privacy Labels Compared to Apps' Privacy Policies", in: Proceedings on Privacy Enhancing Technologies. (DOI)]'s 228,539 and n=306,404; [7Xiang, Anhao; Pei, Weiping; Yue, Chuan (2023): "PolicyChecker: Analyzing the GDPR Completeness of Mobile Apps' Privacy Policies", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)]'s 98.1%; [8Zimmeck, Sebastian; Wang, Ziqi; Zou, Lieyong; Iyengar, Roger; Liu, Bin; Schaub, Florian; Wilson, Shomir; Sadeh, Norman; Bellovin, Steven M.; Reidenberg, Joel (2017): "Automated Analysis of Privacy Requirements for Mobile Apps", in: Proceedings of the Network and Distributed System Security Symposium. (Link)]'s mean of 1.83; and the two [2Amos, Ryan; Acar, Gunes; Lucherini, Eli; Kshirsagar, Mihir; Narayanan, Arvind; Mayer, Jonathan (2021): "Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset", in: Proceedings of the Web Conference 2021, pp. 2165–2176. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] year-span needles. Five more came from the citations review, which found two availability rows naming a different population from the paper's own (see Review). Two more came from the generic review's back-calculated-denominator finding. All seventeen check out. This is the guard doing its job: the figures were right, but nothing had verified them.

Needle specificity, and what this check does not prove. Locating a needle proves the string is in the right PDF. It does not prove the string is the sentence the page is quoting: a bare 26% is in nine of these papers, and the generic review showed that changing the Degeling needle from 84.5 % to 84.9 % still passed, because both appear in that paper's tables. The check now prints a specificity report — how many other check papers contain each needle, and how many needles are numeric-only or under twelve characters — and nine of the thirteen weak needles it found were rewritten to include the paper's surrounding words. Four remain weak and are printed in the output: 1,071,488, 84.7, 54.5% and 1,035,853 each also occur in one other check paper. Each was read in context; none is load-bearing on its own.

Two things the quote check specifically caught or settled:

• [5Pan, Shidong; Zhang, Dawen; Staples, Mark; Xing, Zhenchang; Chen, Jieshan; Xu, Xiwei; Hoang, Thong (2024): "Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps", in: Proceedings of the USENIX Security Symposium. (Link)] states both “37.5% (37,150/99,194) of privacy policy links lead to unavailable websites” and “15.7% (15,572/99,194) of apps do not provide a privacy policy” — 99,194 is used as a denominator of links in one sentence and of apps in another. The page uses the paper's own wording for each figure and does not reconcile them.
• An earlier draft of the page claimed “two needles needed a pypdf rendering”. The committed output showed zero. The claim was removed on 2026-09-10; after the ten new needles it is genuinely one, and the page now says one. A sentence about a check is a figure like any other.

The page cites 38 keys. 14 were already in bibliography and were reused unchanged; 24 were added in two rounds — 22 with the first publication, and 2 more after the generic review asked for citekeys on four papers the page had described in prose. Two of those four turned out to be in the bibliography already under other keys ([9Khatun, Mst Eshita; Noureddine, Lamine; Bello, Sideeq; Ali-Gombe, Aisha (2026): "Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale", Proceedings on Privacy Enhancing Technologies 2026(4):213-231. (DOI)] and [10Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)]); the key-string check passed them and bib_dedup_scan.py caught them on DOI and title. They are in scripts/bib_additions_policies.bib, appended before the closing </bibtex> of a freshly exported copy of the live page (not a local snapshot — the local copy in this workdir was already stale by one page's worth of additions on the morning of 2026-09-10).

Authors for PETS records. The corpus index carries no authors and no DOI for any PETS or USENIX record (100% of both venues). scripts/fetch_authors.py fails on every PETS landing page as of 2026-09-09 — the pages return HTTP 200 but its parser expects a byline layout petsymposium.org no longer serves. scripts/policies_fetch_pets_authors.py was written for this run: it reads the citation_author meta tags from the landing page with curl and a browser User-Agent, refuses to overwrite a cached entry, and fails loudly rather than guessing when no meta tag is present. USENIX authors were taken from the paper PDF, not from usenix.org's own metadata or DBLP — both are known to drop authors from long author lists.

What the number guard does not prove: it checks that each numeral on the page appears somewhere in the script output, not that it appears in the right sentence, and it prints only the first occurrence's context. A stale figure that happens to collide with a live one passes it — demonstrated, not assumed: changing the llm row from 12 to 77 passes the number guard. policies_table_check.mjs was written to close that hole for the four tables it can re-derive, and it catches that mutation and six others. Everything outside those four tables still rests on reading; that is how the availability-denominator findings were caught, not by a guard.

The number guard's ALLOW map is shared across every page on this wiki, so whitelisting a value here would silently bless it elsewhere. Nothing was added to it for this page. The four figures that would have needed an entry were instead given real evidence: two became quote-check needles against the paper text, and two are now printed by policies_external_checks.sh.

scripts/policies_external_checks.sh re-fetches every external fact the content page states; its unedited output is at the foot of this page. Two rules it encodes, both from earlier mistakes on this wiki: print %{http_code} and the effective URL before any byte count, because a 302 recorded as an empty body has been published here as “empty 200”; and never date a repository from /releases/latest, /tags or pushed_at — ask the commits API on the repository's own default branch.

• Google's own current pages do not state when Data safety became mandatory. The 20 July 2022 date rests on contemporaneous third-party reporting. The quote the page uses is on the live primary page; the date is not. Closing this needs an archived snapshot of the Play Console announcement.
• Whether the visible text of w3.org/P3P/ has changed since 2007. Only the server's Last-Modified (2018-02-02) and the page's own newest news item (2007-10-03) could be established; the Wayback Machine rate-limited both attempts to diff them.
• How much of the 16%–94.2% availability spread is ecosystem and how much is method. Nobody has run five link-detection heuristics over one site sample. This is on the page as an open question, not resolved.
• Whether LLM extraction actually beats PoliGraph or a PrivBERT baseline. No head-to-head evaluation on the same documents with the same ground truth was found in these seven venues. The page says the switch is currently a preference, not a finding.
• Anything about venues outside the seven. OPP-115 (ACL 2016), APP-350 and PrivaSeer (ACL 2021) are cited as artefacts because corpus papers use them, but ACL, EMNLP, CHI, SOUPS and the law reviews are not in the corpus and no count here covers them.
• The ~20% free-text run-to-run stability and 0.9% unlocatable-quote rates quoted in this project's documentation were measured on the previous 4,322-paper extraction run and have not been re-measured on this one. They are used here only as an order of magnitude, and are the reason nothing on the content page publishes a percentage over a folded free-text field.
• Figures deliberately not published: no single “fraction of the web with a privacy policy”; no per-year method trend (the era buckets are used instead); no percentage over any folded free-text field; no rate over the 123-paper UNION except the full-text probe table and the two prose sentences that quote its largest row, all three labelled.

Four reviewers, all told explicitly that the author's context might not be exhaustive, and all handed the page text, every script and its unedited output. The three focused passes ran in parallel on the first revision; the generic pass ran afterwards on the corrected page and on this log.

Re-ran all four scripts and diffed them against the committed outputs: byte-identical, no drift. Then checked every numeral on the page.

The pass also independently re-derived a long list of figures it found correct, including the whole tool-lineage table, the per-venue table, the platform split and the probe table.

Independently re-extracted the page's citekeys and quoted spans rather than working from a supplied list, and verified all 22 new BibTeX entries against the corpus index, the paper PDFs and the PETS landing pages.

It also confirmed: 34 of 34 citekeys resolve, 0 key collisions, 0 DOI or title duplicates against the live bibliography, and all 22 new entries correct in author order, venue, year and identifier — including three where the PDF's front matter splits the byline across columns and a careless reader would drop authors.

Fetched rather than recalled. Its findings and their verdicts are in the External sources table above rather than duplicated here; in summary it corrected the P3P currency claim, the PoliGraph GPU claim, the APP-350 licence symmetry and the artifact-discoverability footnote, confirmed every dataset host, every repository date and licence, and both platform-policy quotes, and could not verify two things now listed under What could not be established.

One of its corrections was itself incomplete and was corrected in turn: it proposed replacing the discoverability footnote with a claim about Calpric and Lalaine, but a direct GitHub name search showed the genuinely unfindable pair is PolicyLint and PoliCheck, whose repository is named after neither tool. A reviewer finding is a lead, not a verdict.

No checklist. It ran on the corrected page and on this log, and it was the most productive of the four. Its own disclosures, recorded because they matter: one of its mutation-test commands was denied, so a later sed ran against the committed scripts/policies_quotecheck.mjs instead of a copy; it reverted the change and said so. Verified independently afterwards — the script reproduces its committed output byte for byte. It also noted that the content page was edited while it read, which is true and is a process failure of mine, recorded below.

Reading a guard does not tell you whether it asserts anything. Both the generic reviewer and I broke things on purpose in /tmp copies and checked that the guard failed.

Seven of seventeen mutations survived the guards as first written. That ratio is the argument for mutation-testing every published check rather than reading it.

Every block below is the file itself, inserted by scripts/build_provenance_policies.py at build time — not a sample, not an abridgement. Re-running the generator re-inserts whatever is on disk.

report_policies.mjs

// Every figure on privacy:policies, with its denominator.
//
//   node scripts/report_policies.mjs > scripts/report_policies-output.txt
//
// THREE populations are used and they are NOT interchangeable. Every table
// says which one it is on, and no table mixes them:
//
//   POLICY   classification[].target === 'privacy-policy'   — a STRUCTURED ENUM.
//            "the extractor recorded this paper as classifying privacy-policy
//            text". No folding needed, stable between runs. This is the page's
//            primary population.
//   PROBE    /privacy polic|.../ over title + summary only  — a CANDIDATE SET,
//            recall-oriented, NOT a population. Reported so the page can say how
//            much the enum misses and vice versa. Never published as a rate.
//   UNION    POLICY ∪ PROBE — used only for "which tools/sources appear in this
//            literature at all" rankings, never for a percentage.
//
// Counts are of PAPERS, never tuples. Sentinels are printed but never counted
// as an answer. Free-text sourceList and tool names are folded through
// policy_fold.mjs and the unmapped residue is printed in full at the end.
 
import { loadExtractions, pct, table, isSentinel } from './lib.mjs';
import { foldSource, foldTool, LINEAGE } from './policy_fold.mjs';
 
const P = loadExtractions();
const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
 
// The probe. Widened from the 2026-09-02 brainstorm's four alternatives to nine:
// 'privacy label', 'data safety', 'nutrition label' and 'privacy statement' are
// the store-declaration half of this literature and the narrower probe missed
// them. Recorded here because probe width decides the claim.
const PROBE_RE =
  /privacy polic|privacy notice|terms of service|terms and conditions|privacy label|data safety|nutrition label|policy text|privacy statement/i;
const titleSummary = (p) => `${p.title ?? ''} • ${p.summary ?? ''}`;
 
const POLICY = P.filter((p) => p.classification.some((c) => c.target === 'privacy-policy'));
const PROBE = P.filter((p) => PROBE_RE.test(titleSummary(p)));
const UNION = [...new Set([...POLICY, ...PROBE])];
const CLASSIFIED = P.filter((p) => p.classification.length > 0);
 
const h = (s) => console.log(`\n${'='.repeat(78)}\n${s}\n${'='.repeat(78)}`);
const sub = (s) => console.log(`\n--- ${s}`);
 
// Sanity invariant: every published population must be reproducible from the
// definitions above. If the corpus moves, this throws rather than drifting.
if (POLICY.some((p) => !UNION.includes(p))) throw new Error('POLICY not a subset of UNION');
if (PROBE.some((p) => !UNION.includes(p))) throw new Error('PROBE not a subset of UNION');
if (UNION.length !== new Set(UNION.map(key)).size) throw new Error('UNION has duplicate keys');
 
// ------------------------------------------------------------ 1. populations
h('1. POPULATIONS');
console.log(`corpus                                                   ${P.length}`);
console.log(`classified   classification[] non-empty                  ${CLASSIFIED.length}   ${pct(CLASSIFIED.length, P.length)} of corpus`);
console.log(`POLICY       classification[].target=='privacy-policy'   ${POLICY.length}   ${pct(POLICY.length, CLASSIFIED.length)} of classified`);
console.log(`PROBE        title+summary regex (candidate set)         ${PROBE.length}`);
console.log(`UNION        POLICY u PROBE                              ${UNION.length}`);
console.log(`  POLICY n PROBE                                         ${POLICY.filter((p) => PROBE.includes(p)).length}`);
console.log(`  POLICY only (enum fires, title silent)                 ${POLICY.filter((p) => !PROBE.includes(p)).length}`);
console.log(`  PROBE only (title fires, enum silent)                  ${PROBE.filter((p) => !POLICY.includes(p)).length}`);
console.log(`\nprivacy-policy tuples in POLICY                          ${POLICY.reduce((n, p) => n + p.classification.filter((c) => c.target === 'privacy-policy').length, 0)}`);
 
// The enum-only papers are the argument for not using a title probe alone, so
// print them rather than characterising them from memory. A 2026-09-10 review
// found the page describing this set as "mostly Alexa-skill, IoT and VR studies"
// on no evidence; the list below is the evidence.
sub('POLICY-only: the enum fires and the title probe is silent (why a title probe is not enough)');
{
  const only = POLICY.filter((p) => !PROBE.includes(p))
    .sort((a, b) => a.year - b.year || a.venue.localeCompare(b.venue));
  console.log(`${only.length} papers. Platform measured (enum, multi-valued):`);
  const plat = {};
  for (const p of only) for (const x of new Set(p.platforms)) plat[x] = (plat[x] ?? 0) + 1;
  console.log(table(['platform', 'papers of the enum-only set'],
    Object.entries(plat).sort((a, b) => b[1] - a[1])));
  if (!Object.keys(plat).length) throw new Error('enum-only platform table came out empty');
  for (const p of only) console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
}
 
sub('classification[].target, whole corpus, papers (enum — publishable)');
{
  const m = {};
  for (const p of CLASSIFIED) for (const t of new Set(p.classification.map((c) => c.target))) m[t] = (m[t] ?? 0) + 1;
  console.log(table(['target', 'papers', 'share of 4,439 classified'],
    Object.entries(m).sort((a, b) => b[1] - a[1]).map(([k, v]) => [k, v, pct(v, CLASSIFIED.length)])));
}
 
sub('POLICY and UNION per year (2026 PROVISIONAL: CCS/IMC 2026 not held, IEEE S&P/WWW 2026 under-selected)');
{
  const y = {};
  for (const p of UNION) {
    y[p.year] ??= { u: 0, e: 0, t: 0 };
    y[p.year].u += 1;
    if (POLICY.includes(p)) y[p.year].e += 1;
    if (PROBE.includes(p)) y[p.year].t += 1;
  }
  const allY = {};
  for (const p of P) allY[p.year] = (allY[p.year] ?? 0) + 1;
  // Both shares are printed. UNION/corpus is a share of a regex-widened
  // candidate set and must be labelled as such wherever it is published;
  // POLICY/corpus is the share of the stable enum.
  console.log(table(['Year', 'corpus', 'POLICY', 'PROBE', 'UNION', 'POLICY share of corpus', 'UNION share of corpus'],
    Object.keys(y).sort().map((k) => [k + (k >= '2025' ? '*' : ''), allY[k], y[k].e, y[k].t, y[k].u,
      pct(y[k].e, allY[k]), pct(y[k].u, allY[k])])));
}
 
sub('UNION per venue (denominator: that venue\'s whole corpus slice)');
{
  const v = {}, vAll = {};
  for (const p of P) vAll[p.venue] = (vAll[p.venue] ?? 0) + 1;
  for (const p of UNION) v[p.venue] = (v[p.venue] ?? 0) + 1;
  console.log(table(['Venue', 'UNION', 'POLICY', 'venue papers', 'POLICY share of venue', 'UNION share of venue'],
    Object.entries(v).sort((a, b) => b[1] - a[1]).map(([k, n]) => {
      const pol = POLICY.filter((p) => p.venue === k).length;
      return [k, n, pol, vAll[k], pct(pol, vAll[k]), pct(n, vAll[k])];
    })));
  // The page says "PETS by a factor of five over the next venue"; print the ratio
  // rather than leaving the reader to divide two percentages by eye.
  {
    const byPolicy = Object.keys(vAll)
      .map((k) => [k, POLICY.filter((p) => p.venue === k).length / vAll[k]])
      .sort((a, b) => b[1] - a[1]);
    const byUnion = Object.entries(v).map(([k, n]) => [k, n / vAll[k]]).sort((a, b) => b[1] - a[1]);
    console.log(`top venue over the next, POLICY share: ${byPolicy[0][0]} / ${byPolicy[1][0]} = ${(byPolicy[0][1] / byPolicy[1][1]).toFixed(1)}x`);
    console.log(`top venue over the next, UNION share : ${byUnion[0][0]} / ${byUnion[1][0]} = ${(byUnion[0][1] / byUnion[1][1]).toFixed(1)}x`);
  }
}
 
sub('UNION by platform measured (multi-valued: an app+web paper is in two rows)');
{
  const m = {};
  for (const p of UNION) for (const x of new Set(p.platforms)) m[x] = (m[x] ?? 0) + 1;
  console.log(table(['platform', 'papers', 'share of UNION'],
    Object.entries(m).sort((a, b) => b[1] - a[1]).map(([k, v]) => [k, v, pct(v, UNION.length)])));
  const web = UNION.filter((p) => p.platforms.includes('web'));
  const mob = UNION.filter((p) => p.platforms.includes('mobile'));
  console.log(`\nweb only   ${web.filter((p) => !p.platforms.includes('mobile')).length}`);
  console.log(`mobile only ${mob.filter((p) => !p.platforms.includes('web')).length}`);
  console.log(`both        ${web.filter((p) => p.platforms.includes('mobile')).length}`);
  console.log(`neither     ${UNION.filter((p) => !p.platforms.includes('web') && !p.platforms.includes('mobile')).length}`);
}
 
// ------------------------------------------------- 2. how policies are labelled
h('2. HOW THE FIELD LABELS POLICY TEXT  (population: POLICY, n=' + POLICY.length + ')');
console.log('classification[].method is an ENUM. Multi-valued: a paper with two');
console.log('privacy-policy tuples using two methods appears in two rows.');
 
const polTuples = (p) => p.classification.filter((c) => c.target === 'privacy-policy');
const methodSet = (p) => new Set(polTuples(p).map((c) => c.method));
 
sub('method, all years');
{
  const m = {};
  for (const p of POLICY) for (const x of methodSet(p)) m[x] = (m[x] ?? 0) + 1;
  console.log(table(['method', 'papers', 'share of POLICY'],
    Object.entries(m).sort((a, b) => b[1] - a[1]).map(([k, v]) => [k, v, pct(v, POLICY.length)])));
}
 
// ERAS. 2025-2026 is starred everywhere it appears.
const ERAS = [
  ['2014-2018', (y) => y <= 2018],
  ['2019-2021', (y) => y >= 2019 && y <= 2021],
  ['2022-2024', (y) => y >= 2022 && y <= 2024],
  ['2025-2026*', (y) => y >= 2025],
];
 
sub('method by era — this is the CURRENCY table the page leans on');
{
  const methods = [...new Set(POLICY.flatMap((p) => [...methodSet(p)]))];
  const rows = methods.map((mm) => {
    const cells = ERAS.map(([, f]) => {
      const sub2 = POLICY.filter((p) => f(p.year));
      const n = sub2.filter((p) => methodSet(p).has(mm)).length;
      return `${n} (${pct(n, sub2.length)})`;
    });
    const total = POLICY.filter((p) => methodSet(p).has(mm)).length;
    return [mm, total, ...cells];
  }).sort((a, b) => b[1] - a[1]);
  console.log(table(['method', 'all', ...ERAS.map(([l]) => `${l} n=${POLICY.filter((p) => ERAS.find(([ll]) => ll === l)[1](p.year)).length}`)], rows));
}
 
sub('classification[].validation on privacy-policy tuples (enum; none-reported is a REAL value, not a gap in the data)');
{
  const m = {};
  for (const p of POLICY) for (const x of new Set(polTuples(p).map((c) => c.validation))) m[x] = (m[x] ?? 0) + 1;
  console.log(table(['validation', 'papers', 'share of POLICY'],
    Object.entries(m).sort((a, b) => b[1] - a[1]).map(([k, v]) => [k, v, pct(v, POLICY.length)])));
  // Same row for the whole classified corpus, so the page can print the base rate
  // beside the subgroup share instead of implying the subgroup is unusual.
  const base = {};
  for (const p of CLASSIFIED) for (const x of new Set(p.classification.map((c) => c.validation))) base[x] = (base[x] ?? 0) + 1;
  console.log('\nBASE RATE, all 4,439 papers that classified anything:');
  console.log(table(['validation', 'papers', 'share of classified'],
    Object.entries(base).sort((a, b) => b[1] - a[1]).map(([k, v]) => [k, v, pct(v, CLASSIFIED.length)])));
}
 
sub('LLM as the labelling method — POLICY papers whose privacy-policy tuple has method=="llm", listed in full');
{
  const llm = POLICY.filter((p) => methodSet(p).has('llm')).sort((a, b) => a.year - b.year);
  console.log(`n=${llm.length} of ${POLICY.length} POLICY papers`);
  for (const p of llm) console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
}
 
// --------------------------------------------------- 3. where policies come from
h('3. WHERE THE POLICIES COME FROM  (population: UNION, n=' + UNION.length + ')');
 
sub('population[].unit (enum)');
{
  const m = {};
  for (const p of UNION) for (const x of new Set(p.population.map((x2) => x2.unit))) if (!isSentinel(x)) m[x] = (m[x] ?? 0) + 1;
  console.log(table(['unit', 'papers'], Object.entries(m).sort((a, b) => b[1] - a[1])));
}
 
sub('population[].sourceList, FOLDED through policy_fold.mjs (free text — a ranking, not percentages)');
{
  const m = {}, residue = {};
  for (const p of UNION) {
    const seen = new Set();
    for (const s of p.population) {
      if (isSentinel(s.sourceList) || !s.sourceList) continue;
      const f = foldSource(s.sourceList);
      if (f === null) { residue[s.sourceList] = (residue[s.sourceList] ?? 0) + 1; continue; }
      seen.add(f);
    }
    for (const f of seen) m[f] = (m[f] ?? 0) + 1;
  }
  console.log(table(['folded source', 'papers'], Object.entries(m).sort((a, b) => b[1] - a[1])));
  console.log(`\nRESIDUE — sourceList strings matching no fold rule (${Object.keys(residue).length} distinct, printed in full):`);
  for (const [k, v] of Object.entries(residue).sort((a, b) => b[1] - a[1])) console.log(`  ${String(v).padStart(2)}  ${k}`);
}
 
// ------------------------------------------------------------- 4. the lineage
h('4. THE TOOL LINEAGE  (population: whole corpus, so a tool used outside UNION is visible)');
console.log('Matched against tools[].name, otherToolsMentioned[].name,');
console.log('classification[].resourceName/targetDetail, population[].sourceList,');
console.log('detection[].phenomenon/technique. Counts are PAPERS.');
{
  const NAMED = LINEAGE;   // defined in policy_fold.mjs, shared with the full-text probe
  const hay = (p) => {
    const a = [];
    for (const t of [...p.tools, ...p.otherToolsMentioned]) a.push(t.name ?? '', t.purpose ?? '');
    for (const c of p.classification) a.push(c.resourceName ?? '', c.targetDetail ?? '');
    for (const s of p.population) a.push(s.sourceList ?? '');
    for (const d of p.detection) a.push(d.phenomenon ?? '', d.technique ?? '');
    return a.join(' | ');
  };
  const rows = NAMED.map(([label, re]) => {
    const all = P.filter((p) => re.test(hay(p)));
    const yrs = all.map((p) => p.year).sort();
    return [label, all.length, all.filter((p) => UNION.includes(p)).length,
      yrs.length ? `${yrs[0]}-${yrs[yrs.length - 1]}` : '—',
      all.filter((p) => p.year >= 2024).length];
  });
  console.log(table(['artefact (first paper)', 'papers, corpus', 'of those in UNION', 'year range of use', 'used 2024+'], rows));
}
 
sub('tools[] + otherToolsMentioned in UNION, FOLDED (free text — ranking only)');
{
  const m = {}, residue = {};
  for (const p of UNION) {
    const seen = new Set();
    for (const t of [...p.tools, ...p.otherToolsMentioned]) {
      if (!t.name) continue;
      const f = foldTool(t.name);
      if (f === null) { residue[t.name] = (residue[t.name] ?? 0) + 1; continue; }
      seen.add(f);
    }
    for (const f of seen) m[f] = (m[f] ?? 0) + 1;
  }
  console.log(table(['folded tool family', 'papers in UNION'], Object.entries(m).sort((a, b) => b[1] - a[1])));
  const res = Object.entries(residue).sort((a, b) => b[1] - a[1]);
  console.log(`\nRESIDUE — ${res.length} distinct tool names matching no fold rule. Top 40 by frequency:`);
  for (const [k, v] of res.slice(0, 40)) console.log(`  ${String(v).padStart(2)}  ${k}`);
  console.log(`  … and ${Math.max(0, res.length - 40)} more, each in 1-2 papers.`);
}
 
// --------------------------------------------------- 5. measured results
h('5. MEASURED RESULTS: POLICY AVAILABILITY BY ECOSYSTEM');
console.log('Hand-keyed from detection[].prevalence tuples in UNION papers. The map');
console.log('below is INSIDE this script on purpose: the page must not carry a');
console.log('per-paper figure the script cannot print. Every row was read back');
console.log('against the paper\'s own quote; see report_policies_quotecheck.mjs.');
{
  // [citekey, venue/year/slug, ecosystem, denominator as the paper states it, figure]
  const AVAIL = [
    ['degeling2019_value', 'NDSS/2019/we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy', 'web, EU', '6,357 (the total of the paper\'s own availability table); the paper also states 6,759 domains in its January lists and 6,579 in its abstract', '84.5% had a policy after 25 May 2018, up from 79.6% in January'],
    ['vallina2019_porn', 'IMC/2019/tales-from-the-porn-a-comprehensive-privacy-analysis-of-the-web-porn-ecosystem', 'web, adult sites', '6,843 pornographic websites', 'only 16% had an accessible privacy policy'],
    ['cui2025_privacy', 'PETS/2025/understanding-privacy-norms-through-web-forms', 'web, sites with a PI-collecting form', '10,143 websites', '94.2% (9,559) had a privacy-policy link'],
    ['zimmeck2019_maps', 'PETS/2019/maps-scaling-privacy-compliance-analysis-to-a-million-apps', 'Android', '1,035,853 analysed apps, of 1,049,790 retrieved', 'only 50.5% had a policy link on the Play Store page'],
    ['pan2024_trap', 'USENIX/2024/is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin', 'Android', '99,194 usable apps — the paper divides link failures by its app count', '37.5% (37,150/99,194) of policy links led to an unavailable page; separately 15.7% (15,572/99,194) of apps have no link at all'],
    ['manandhar2022_smart', 'USENIX/2022/smart-home-privacy-policies-demystified-a-study-of-availability-content-and-cove', 'smart-home vendors', '596 vendors on 7 platforms', '48.99% had a device-applicable policy; 10.57% had none at all'],
    ['lentzsch2021_alexa', 'NDSS/2021/hey-alexa-is-this-skill-safe-taking-a-closer-look-at-the-alexa-skill-ecosystem', 'Alexa skills', '150,708 skills across 7 country stores', '36,475 (24.2%) provided a policy link'],
    ['yan2024_quality', 'PETS/2024/on-the-quality-of-privacy-policy-documents-of-virtual-personal-assistant-applica', 'Alexa skills', '65,195 skills', '21,063 of 65,195 provided a policy link'],
    ['zhan2024_vpvet', 'CCS/2024/vpvet-vetting-privacy-policies-of-virtual-reality-apps', 'VR apps', '11,923 apps on 10 VR platforms', '29.5% had a findable privacy policy'],
    ['edu2022_exploring', 'IMC/2022/exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services', 'Discord chatbots requesting permissions', "15,525 unique active chatbots (the paper's own Table 2 total); its 14,852-without figure does not reconcile with it", '676 (4.35%) had a policy; 14,852 (95.67%) did not'],
    ['wu2025_depth', 'IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems', 'GPT Actions', 'Actions declaring a legal_info_url', '93.96% of those policies were reachable'],
  ];
  const byKey = new Map(P.map((p) => [key(p), p]));
  const rows = AVAIL.map(([ck, k, eco, den, fig]) => {
    if (!byKey.has(k)) throw new Error(`AVAIL row names a paper not in the corpus: ${k}`);
    const p = byKey.get(k);
    return [ck, `${p.venue} ${p.year}`, eco, den, fig];
  });
  console.log(table(['citekey', 'venue', 'ecosystem', 'denominator (paper\'s own)', 'policy availability'], rows));
  console.log(`\nrows: ${rows.length}; every one resolves to a corpus paper.`);
}
 
h('6. MEASURED RESULTS: POLICY-VERSUS-BEHAVIOUR CONSISTENCY');
{
  const CONS = [
    ['zimmeck2017_automated', 'NDSS/2017/automated-analysis-of-privacy-requirements-for-mobile-apps', '9,050 apps with policies', 'mean 1.83 potential inconsistencies per app'],
    ['libert2018_automated', 'WWW/2018/an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w', '1,807,491 identified third-party transmissions', 'only 14.80% were disclosed in the policy'],
    ['andow2019_policylint', 'USENIX/2019/policylint-investigating-internal-privacy-policy-contradictions-on-google-play', '11,430 policies', '14.2% (1,618) contained logical contradictions; 17.7% (2,028) contradictions or narrowing definitions'],
    ['andow2020_actions', 'USENIX/2020/actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an', '13,796 applications / 45,603 data flows', '42.4% of apps had an omitted or incorrect disclosure; 31.1% of flows were omitted; only 0.5% of flows were clearly disclosed'],
    ['trimananda2022_ovrseen', 'USENIX/2022/ovrseen-auditing-network-traffic-and-privacy-policies-in-oculus-vr', '1,135 data flows in Oculus VR apps', '68% (776) inconsistent disclosures'],
    ['cui2023_poligraph', 'USENIX/2023/poligraph-automated-privacy-policy-analysis-using-knowledge-graphs', '1,566 mapped statement pairs', '13.5% (211) conflicting; 25.5% (1,339/5,255) of policies define a term differently from the CCPA-based ontology'],
    ['xiang2023_policychecker', 'CCS/2023/policychecker-analyzing-the-gdpr-completeness-of-mobile-apps-privacy-policies', '163,068 analysable policies', '99.3% incomplete under GDPR; 98.1% had at least one mandatory-requirement violation'],
    ['xiao2023_lalaine', 'USENIX/2023/lalaine-measuring-and-characterizing-non-compliance-of-apple-privacy-labels', '5,102 fully tested iOS apps', '3,423 (67.1%) non-compliant privacy labels'],
    ['samarin2023_lessons', 'PETS/2023/lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c', '69 apps with CCPA disclosures', '80% (55) collected an identifier they did not disclose'],
    ['ali2024_honesty', 'PETS/2024/honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a', 'iOS apps labelled "Data Not Collected"', '97% had policy statements indicating data collection'],
    ['cui2025_privacy', 'PETS/2025/understanding-privacy-norms-through-web-forms', 'websites collecting PI via web forms', 'phi coefficient between observed collection and PoliGraph-er disclosure was < 0.20 for every PI type'],
  ];
  const byKey = new Map(P.map((p) => [key(p), p]));
  const rows = CONS.map(([ck, k, den, fig]) => {
    if (!byKey.has(k)) throw new Error(`CONS row names a paper not in the corpus: ${k}`);
    const p = byKey.get(k);
    return [ck, `${p.venue} ${p.year}`, den, fig];
  });
  console.log(table(['citekey', 'venue', 'denominator (paper\'s own)', 'finding'], rows));
  console.log(`\nrows: ${rows.length}; every one resolves to a corpus paper.`);
}
 
// ------------------------------------------------------- 7. reading / language
h('7. WHAT ELSE THE UNION MEASURES');
sub('UNION papers that are also in the `legal` population (legal[] non-empty)');
{
  const L = UNION.filter((p) => p.legal.length > 0);
  const LC = P.filter((p) => p.legal.length > 0);
  console.log(`${L.length} of ${UNION.length} UNION papers assess a law; base rate ${LC.length} of ${P.length} (${pct(LC.length, P.length)}) corpus-wide.`);
  const laws = {};
  for (const p of L) for (const x of new Set(p.legal.map((l) => l.law).filter((x2) => x2 && !isSentinel(x2)))) laws[x] = (laws[x] ?? 0) + 1;
  console.log(table(['law (free text, unfolded — ranking only)', 'papers'],
    Object.entries(laws).sort((a, b) => b[1] - a[1]).slice(0, 15)));
}
 
sub('human annotation in UNION — policy labelling is a hand-coding literature');
{
  const ann = UNION.filter((p) => p.humanAnnotation.length > 0);
  const withMetric = ann.filter((p) => p.humanAnnotation.some((a) => !isSentinel(a.agreementMetric)));
  const withCount = ann.filter((p) => p.humanAnnotation.some((a) => !isSentinel(a.annotatorCount)));
  console.log(`UNION papers with humanAnnotation[]            ${ann.length} of ${UNION.length}  ${pct(ann.length, UNION.length)}`);
  console.log(`  …stating an agreement metric                 ${withMetric.length}  ${pct(withMetric.length, ann.length)} of annotated`);
  console.log(`  …stating an annotator count                  ${withCount.length}  ${pct(withCount.length, ann.length)} of annotated`);
  const ANN = P.filter((p) => p.humanAnnotation.length > 0);
  const bMetric = ANN.filter((p) => p.humanAnnotation.some((a) => !isSentinel(a.agreementMetric)));
  console.log(`\nBASE RATE, all ${ANN.length} papers that coded data by hand:`);
  console.log(`  …stating an agreement metric                 ${bMetric.length}  ${pct(bMetric.length, ANN.length)}`);
}
 
sub('temporal[].mode in UNION (enum) vs corpus base rate — is this a longitudinal literature?');
{
  const count = (set) => {
    const m = {};
    for (const p of set) for (const x of new Set(p.temporal.map((t) => t.mode))) if (!isSentinel(x)) m[x] = (m[x] ?? 0) + 1;
    return m;
  };
  const u = count(UNION), b = count(P);
  const uT = UNION.filter((p) => p.temporal.length > 0).length;
  const bT = P.filter((p) => p.temporal.length > 0).length;
  console.log(`UNION papers with a temporal[] tuple ${uT} of ${UNION.length}; corpus ${bT} of ${P.length}`);
  console.log(table(['temporal.mode', 'UNION', 'share of UNION w/ temporal', 'corpus', 'share of corpus w/ temporal'],
    [...new Set([...Object.keys(u), ...Object.keys(b)])].sort((x, y) => (u[y] ?? 0) - (u[x] ?? 0))
      .map((k) => [k, u[k] ?? 0, pct(u[k] ?? 0, uT), b[k] ?? 0, pct(b[k] ?? 0, bT)])));
}
 
sub('artifact availability in UNION vs corpus base rate (enum)');
{
  const a = {}, b = {};
  for (const p of UNION) if (p.artifacts) a[p.artifacts.availability] = (a[p.artifacts.availability] ?? 0) + 1;
  for (const p of P) if (p.artifacts) b[p.artifacts.availability] = (b[p.artifacts.availability] ?? 0) + 1;
  const withArt = P.filter((p) => p.artifacts);
  const uWithArt = UNION.filter((p) => p.artifacts);
  console.log(table(['availability', 'UNION', 'share', 'corpus', 'share'],
    [...new Set([...Object.keys(a), ...Object.keys(b)])].sort((x, y) => (b[y] ?? 0) - (b[x] ?? 0))
      .map((k) => [k, a[k] ?? 0, pct(a[k] ?? 0, uWithArt.length), b[k] ?? 0, pct(b[k] ?? 0, withArt.length)])));
}
 
// --------------------------------------- 7b. inputs for the significance test
//
// policies_significance.py PARSES these lines. They are emitted here, from the
// same population objects every other table uses, so the p-values can never be
// computed over a hand-keyed count that has drifted from the report.
//
// Each comparison is emitted TWICE: once over POLICY (the stable enum, 102) and
// once over UNION (the regex-widened candidate set, 123). The page publishes the
// POLICY rows; the UNION rows are published on the provenance page so a reader
// can see whether the choice of population changes any conclusion.
h('7b. SIGNIFICANCE INPUTS — parsed by policies_significance.py');
{
  const hasTemporalArchive = (p) => p.temporal.some((x) => x.mode === 'web-archive');
  const hasTemporal = (p) => p.temporal.length > 0;
  const annotated = (p) => p.humanAnnotation.length > 0;
  const statesMetric = (p) => p.humanAnnotation.some((a) => !isSentinel(a.agreementMetric));
  const hasArtifacts = (p) => Boolean(p.artifacts);
  const publicArtifacts = (p) => Boolean(p.artifacts) && p.artifacts.availability === 'public';
  const assessesLaw = (p) => p.legal.length > 0;
  const policyTuples = (p) => p.classification.filter((c) => c.target === 'privacy-policy');
  const noValidation = (p) => policyTuples(p).some((c) => c.validation === 'none-reported');
  const hasPolicyTuple = (p) => policyTuples(p).length > 0;
 
  // The validation comparison is the one asymmetric case: on the subgroup side
  // the question is about the paper's PRIVACY-POLICY tuples, on the corpus side
  // it is about ANY classification tuple. Giving each comparison its own pair of
  // predicates keeps that visible instead of hiding it in one shared predicate.
  const anyNoValidation = (p) => p.classification.some((c) => c.validation === 'none-reported');
  const classifiedAnything = (p) => p.classification.length > 0;
 
  // [label, subgroup restrict, subgroup property, base restrict, base property]
  const COMPARISONS = [
    ['temporal.mode == web-archive', hasTemporal, hasTemporalArchive, hasTemporal, hasTemporalArchive],
    ['humanAnnotation states an agreement metric', annotated, statesMetric, annotated, statesMetric],
    ['artifacts.availability == public', hasArtifacts, publicArtifacts, hasArtifacts, publicArtifacts],
    ['assesses a law (legal[] non-empty)', () => true, assessesLaw, () => true, assessesLaw],
    ['classification.validation == none-reported', hasPolicyTuple, noValidation, classifiedAnything, anyNoValidation],
  ];
  for (const [label, sRestrict, sProp, bRestrict, bProp] of COMPARISONS) {
    const base = P.filter(bRestrict);
    for (const [popName, pop] of [['POLICY', POLICY], ['UNION', UNION]]) {
      const sub2 = pop.filter(sRestrict);
      // The base rate must EXCLUDE the subgroup's own papers, otherwise the
      // subgroup is compared against a population that contains it.
      const rest = base.filter((p) => !pop.includes(p));
      console.log(`SIG|${label}|${popName}|${sub2.filter(sProp).length}|${sub2.length}|`
        + `${rest.filter(bProp).length}|${rest.length}|`
        + `${base.filter(bProp).length}|${base.length}`);
    }
  }
  console.log('\nColumns: SIG|comparison|population|sub hits|sub n|rest-of-corpus hits|rest n|whole-corpus hits|whole n');
  console.log('"rest" excludes the subgroup itself; "whole" is the figure a naive base rate would use.');
}
 
// --------------------------------------------------------------- 8. full list
h('8. THE UNION, IN FULL — the audit surface for every count above');
for (const p of [...UNION].sort((a, b) => a.year - b.year || a.venue.localeCompare(b.venue))) {
  const tag = POLICY.includes(p) ? (PROBE.includes(p) ? 'BOTH ' : 'ENUM ') : 'PROBE';
  console.log(`${tag} ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
}

policy_fold.mjs

// Fold the free-text names that privacy:policies aggregates, and nothing else.
//
// WHY: two fields on that page are free text and ~20% stable run-to-run by exact
// string (data/extract/README.md), so aggregating them raw undercounts:
//
//   population[].sourceList  — "Google Play Store" (15 papers) and "Google Play"
//     (15) are one source under two spellings; "OPP-115" (11) and "OPP-115
//     corpus" (2) are one dataset; five different Alexa spellings are one list.
//   tools[].name / otherToolsMentioned[].name — "Beautiful Soup" / "BeautifulSoup"
//     / "bs4", "PoliGraph" / "PoliGraph-er", "Selenium" / "Selenium WebDriver".
//
// Rules are ordered; the first match wins. Anything matching no rule keeps its
// raw name and is printed as residue by report_policies.mjs, so the part this
// file cannot classify stays visible instead of vanishing.
 
// [canonical family, matcher tested against the raw name]
export const SOURCE_FAMILIES = [
  // --- app stores and app-set sources
  ['Google Play', /google ?play|play ?store|androzoo|playdrone|google-play-scraper/i],
  ['Apple App Store', /app ?store(?! optimization)|apple ?app|ios app store|app annie|appfigures/i],
  ['Alexa list', /\balexa\b(?!.*skill)/i],
  ['Alexa Skills Store', /alexa (skill|store).*skill|skill store|alexa skills/i],
  ['Tranco', /\btranco\b/i],
  ['Majestic / Umbrella / Quantcast', /majestic|umbrella|quantcast|chrome ux|\bcrux\b/i],
 
  // --- annotated policy corpora
  ['OPP-115', /opp-?115|usable ?privacy|acl\/coling 2014|ramanath/i],
  ['APP-350', /app-?350/i],
  ['PrivaSeer', /privaseer/i],
  ['Princeton Policies-over-Time', /princeton.*polic|policies over time|million-document/i],
 
  // --- archives
  ['Wayback Machine', /wayback|internet archive|common ?crawl/i],
 
  // --- participant panels (these are NOT policy sources; kept separate so they
  // cannot be silently mixed into a "where policies come from" table)
  ['participant panel', /prolific|mechanical turk|\bmturk\b|qualtrics panel|clickworker|respondi/i],
 
  // --- other app / extension / package ecosystems
  ['other app ecosystem', /sidequest|oculus|meta quest|steam|chrome web store|cocoapods|rapidapi|wordpress|github|npm|pypi/i],
 
  // --- the honest bucket: the paper made its own list
  ['custom / hand-built list', /custom|hand-?(built|picked|curated)|manual(ly)? (selected|compiled)|own (list|corpus|selection)|seed list/i],
];
 
export const TOOL_FAMILIES = [
  ['Polisis / PriBot', /polisis|pribot/i],
  ['PolicyLint', /policylint/i],
  ['PoliCheck', /policheck/i],
  ['PoliGraph', /poligraph/i],
  ['PolicyChecker', /policychecker/i],
  ['PurPliance', /purpliance/i],
  ['PrivBERT', /privbert/i],
  ['MAPS', /^maps$/],
  ['spaCy', /^spacy/i],
  ['NLTK', /^nltk/i],
  ['Stanford CoreNLP / AllenNLP', /corenlp|allennlp|stanza/i],
  ['BERT family (non-privacy)', /^bert|roberta|distilbert|legal-?bert|^albert/i],
  ['LLM (commercial API)', /gpt-?[345]|chatgpt|openai|claude|gemini|\bbard\b/i],
  ['LLM (open weights)', /llama|mistral|qwen|vicuna|falcon|deepseek/i],
  ['boilerplate stripper', /boilerpipe|readability|trafilatura|html2text|justext|goose/i],
  ['BeautifulSoup', /beautiful ?soup|^bs4$/i],
  ['Selenium', /^selenium/i],
  ['Playwright', /^playwright/i],
  ['Puppeteer', /^puppeteer/i],
  ['OpenWPM', /openwpm/i],
  ['language detection', /langdetect|langid|\bcld[23]?\b|fasttext.*lang/i],
  ['readability metric', /flesch|kincaid|gunning|smog|coleman|dale-?chall|textstat/i],
];
 
function fold(raw, families) {
  const s = String(raw ?? '').trim();
  if (!s) return null;
  for (const [canon, re] of families) if (re.test(s)) return canon;
  return null; // residue: caller keeps the raw string and prints it
}
 
export const foldSource = (raw) => fold(raw, SOURCE_FAMILIES);
export const foldTool = (raw) => fold(raw, TOOL_FAMILIES);
 
// The policy-analysis lineage, one regex per artefact. Exported so that
// report_policies.mjs (which matches it against EXTRACTION fields) and
// policies_fulltext_probe.mjs (which matches it against the paper's FULL TEXT)
// cannot drift apart. The two ask different questions and give different
// answers: PrivaSeer is named as a tool or data source by no corpus paper and
// cited in the text of twelve.
export const LINEAGE = [
  ['Privee (USENIX 2014)', /\bPrivee\b/],
  ['OPP-115 corpus (ACL 2016)', /OPP-?115/i],
  ['Polisis / PriBot (USENIX 2018)', /polisis|pribot/i],
  ['PolicyLint (USENIX 2019)', /policylint/i],
  ['MAPS (PETS 2019)', /\bMAPS\b/],
  ['APP-350 corpus (2019)', /APP-?350/i],
  ['PoliCheck (USENIX 2020)', /policheck/i],
  ['PurPliance (2021)', /purpliance/i],
  ['PrivBERT (2021)', /privbert/i],
  ['Calpric (USENIX 2023)', /calpric/i],
  ['PoliGraph / PoliGraph-er (USENIX 2023)', /poligraph/i],
  ['PolicyChecker (CCS 2023)', /policychecker/i],
  ['Lalaine (USENIX 2023)', /lalaine/i],
  ['PolicyComp (USENIX 2023)', /policycomp/i],
  ['PrivaSeer', /privaseer/i],
];
 
// Matching a tool name against a paper's FULL TEXT is not the same problem as
// matching it against the extraction's tool fields. In full text an acronym
// collides with unrelated uses: /\bMAPS\b/ matches "Google MAPS abuse" and the
// Play Store's "MAPS & NAVIGATION" category. Where that happens, the full-text
// scan uses the artefact's own title instead of its acronym, and the override is
// recorded here rather than buried in the scanning script.
export const LINEAGE_FULLTEXT_OVERRIDE = new Map([
  ['MAPS (PETS 2019)', {
    re: /MAPS:\s*Scaling/i,
    why: '/\\bMAPS\\b/ also matches "Google MAPS" and the Play category "MAPS & NAVIGATION"; the paper is always cited by its title',
  }],
]);

policies_fulltext_probe.mjs

// Full-text probes for the methodological questions the extraction schema has no
// field for: how a paper FINDS a privacy policy, what it does about language, and
// what it does about the fact that the policy is a moving target.
//
//   node scripts/policies_fulltext_probe.mjs > scripts/policies_fulltext_probe-output.txt
//
// Population: the 123 UNION papers from report_policies.mjs, re-derived here from
// the same two definitions so the two scripts cannot drift apart.
//
// A probe count is a CANDIDATE SET, not a measurement: it says the phrase is in
// the paper, not that the paper did the thing. Two forms are printed for every
// probe, because probe width decides the claim. Whitespace is collapsed and
// hyphenation rejoined first — a PDF line break inside a phrase otherwise
// silently undercounts.
//
// CAVEAT ON THE SECOND COLUMN: 'wide' is a looser phrasing of the same question
// for most rows, and there it is a superset. For three rows it is a DIFFERENT
// question (a tool-name probe rather than a phrase probe), so it can be smaller
// than 'narrow'. Those rows are flagged in the output; do not read them as a
// narrow-vs-wide comparison.
 
import fs from 'node:fs';
import path from 'node:path';
import { loadExtractions, dataRoot, pct, table } from './lib.mjs';
import { LINEAGE, LINEAGE_FULLTEXT_OVERRIDE } from './policy_fold.mjs';
 
const ROOT = path.join(dataRoot(), 'fulltext');
const P = loadExtractions();
const PROBE_RE =
  /privacy polic|privacy notice|terms of service|terms and conditions|privacy label|data safety|nutrition label|policy text|privacy statement/i;
const ts = (p) => `${p.title ?? ''} • ${p.summary ?? ''}`;
const UNION = [...new Set([
  ...P.filter((p) => p.classification.some((c) => c.target === 'privacy-policy')),
  ...P.filter((p) => PROBE_RE.test(ts(p))),
])];
 
const norm = (s) => s.replace(/­/g, '').replace(/-\n/g, '').replace(/\s+/g, ' ');
function text(p) {
  const f = path.join(ROOT, String(p.year), p.venue, p.slug, 'paper.cols.txt');
  if (!fs.existsSync(f)) return null;
  return norm(fs.readFileSync(f, 'utf8'));
}
 
const TEXTS = new Map();
let noText = 0;
for (const p of UNION) {
  const t = text(p);
  if (t === null) { noText += 1; continue; }
  TEXTS.set(p, t);
}
console.log(`UNION = ${UNION.length} papers; full text present for ${TEXTS.size}; missing ${noText}.`);
console.log('Every count below is over the ' + TEXTS.size + ' papers with full text.\n');
 
// [label, narrow regex, wide regex]. Narrow is what the page quotes.
const PROBES = [
  ['finds the policy by LINK TEXT / anchor keyword',
    /(link|anchor)s? (text|label)|matching the (link|anchor)|keyword(s)? (in|on) the (link|footer)|footer link/i,
    /link text|anchor text|keyword/i],
  ['names a link-detection SEED PHRASE list',
    /seed phrase|seed keyword|list of (candidate )?(keywords|phrases) (used )?to (find|locate|identify)/i,
    /seed (phrase|term|keyword)/i],
  ['follows the policy link and reports FAILURES (404 / dead / unreachable)',
    /polic(y|ies)[^.]{0,80}(404|dead link|broken link|unreachable|did not resolve|failed to (load|download|retrieve))/i,
    /(404|broken link|dead link|unreachable)/i],
  ['handles a policy served as a PDF',
    /polic(y|ies)[^.]{0,60}\bPDF\b|\bPDF\b[^.]{0,60}polic(y|ies)/i,
    /\bPDF\b/],
  ['states the LANGUAGE of the policies it analysed',
    /english[- ]language (privacy )?polic|polic(y|ies)[^.]{0,60}in english|non-?english polic|language of the polic/i,
    /langdetect|langid|\bcld[23]\b|language detection/i],
  ['analyses policies in more than one language',
    /multilingual|bilingual|(translat(e|ed|ing|ion))[^.]{0,50}polic|polic(y|ies)[^.]{0,50}translat/i,
    /multilingual|bilingual|translat/i],
  ['strips BOILERPLATE / extracts the main content of the policy page',
    /boilerpipe|trafilatura|readability(-lxml| library|\.js)|justext|main content extraction|html to text|html2text/i,
    /boilerplate|extract(ed|ing)? the (main )?(text|content)/i],
  ['measures READABILITY of the policy',
    /flesch|kincaid|gunning fog|\bSMOG\b|coleman-liau|dale-chall|reading (ease|grade|level|time)/i,
    /readab(le|ility)/i],
  ['compares the policy against OBSERVED BEHAVIOUR (traffic, code, or storage)',
    /(consisten|inconsisten|discrepan|mismatch|divergen|misalign)[a-z]*[^.]{0,90}(polic|label|disclos)/i,
    /consistency analysis|policy[- ]to[- ]?(code|flow|traffic)/i],
  ['handles the policy being VERSIONED / changing under it',
    /polic(y|ies)[^.]{0,80}(version|snapshot|revision|updated?|chang(e|ed|es))|wayback|internet archive/i,
    /longitudinal|over time|snapshot/i],
  ['reports the sentence/segment SEGMENTATION step',
    /segment(ation|ed|ing)? (the )?polic|polic[^.]{0,40}(into|by) (sentences|segments|paragraphs)|sentence[- ]level/i,
    /segment/i],
  ['says which policy applies (app vs developer vs platform vs layered)',
    /(which|applicable|relevant|correct) privacy polic|polic(y|ies) (that )?appl(y|ies)|layered polic|multiple privacy polic|generic (company|corporate) polic/i,
    /applicable polic|multiple polic/i],
  ['deduplicates identical / templated policies',
    /(duplicate|template|boilerplate|reuse[d]?|identical)[^.]{0,60}polic|polic[^.]{0,60}(duplicat|templat|reus)/i,
    /duplicat|templat/i],
];
 
const rows = [];
for (const [label, narrow, wide] of PROBES) {
  let n = 0, w = 0;
  for (const [, t] of TEXTS) {
    if (narrow.test(t)) n += 1;
    if (wide.test(t)) w += 1;
  }
  const flag = w < n ? '  <-- not a superset: different question' : '';
  rows.push([label + flag, n, pct(n, TEXTS.size), w, pct(w, TEXTS.size)]);
}
console.log(table(['probe (narrow form is what the page quotes)', 'narrow', 'share', 'wide', 'share'], rows));
 
console.log('\n--- The two probes the page leans on hardest, with the matching papers named');
for (const [label, narrow] of [
  ['follows the policy link and reports FAILURES (404 / dead / unreachable)', PROBES[2][1]],
  ['analyses policies in more than one language', PROBES[5][1]],
]) {
  const hits = [...TEXTS.entries()].filter(([, t]) => narrow.test(t)).map(([p]) => p)
    .sort((a, b) => a.year - b.year);
  console.log(`\n${label}  —  ${hits.length} papers`);
  for (const p of hits) console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title.slice(0, 92)}`);
}
 
// ---------------------------------------------------------------------------
// The lineage, counted a second way: WHOLE CORPUS full text rather than the
// extraction's tool/source fields.
//
// These are different questions and they give different answers. "No corpus
// paper's extraction names PrivaSeer" is a fact about the extraction. "Nobody
// in these seven venues cites PrivaSeer" would be a fact about the literature,
// and it is false. A fold over a structured field cannot support an "at all"
// claim; only a full-text scan can even try.
//
// Scans every paper.cols.txt in the corpus, not just the UNION.
{
  const ROOTDIR = ROOT;
  const all = [];
  for (const y of fs.readdirSync(ROOTDIR)) {
    const yd = path.join(ROOTDIR, y);
    if (!fs.statSync(yd).isDirectory()) continue;
    for (const v of fs.readdirSync(yd)) {
      const vd = path.join(yd, v);
      if (!fs.statSync(vd).isDirectory()) continue;
      for (const s of fs.readdirSync(vd)) {
        const f = path.join(vd, s, 'paper.cols.txt');
        if (fs.existsSync(f)) all.push([`${y} ${v.padEnd(8)} ${s}`, f]);
      }
    }
  }
  console.log(`\n\n--- LINEAGE ARTEFACTS: extraction fold vs FULL-TEXT mention, whole corpus`);
  console.log(`Full-text denominator: ${all.length} papers with a readable paper.cols.txt.`);
  console.log(`"extraction" is the column report_policies.mjs section 4 prints.`);
  const PATTERNS = LINEAGE.map(([label, re]) => {
    const o = LINEAGE_FULLTEXT_OVERRIDE.get(label);
    return [label, o ? o.re : re, o ? o.why : ''];
  });
  const seen = new Map(PATTERNS.map(([label]) => [label, []]));
  for (const [id, f] of all) {
    const t2 = norm(fs.readFileSync(f, 'utf8').replace(/\0/g, ''));
    for (const [label, re] of PATTERNS) if (re.test(t2)) seen.get(label).push(id);
  }
  console.log(table(['artefact', 'full-text pattern', 'papers whose FULL TEXT names it'],
    PATTERNS.map(([label, re]) => [label, String(re), seen.get(label).length])));
  const overridden = PATTERNS.filter(([, , why]) => why);
  console.log('\nFull-text patterns that differ from the extraction pattern, and why:');
  for (const [label, re, why] of overridden) console.log(`  ${label}: ${re} — ${why}`);
  console.log('\nThe two artefacts where the gap changes what may be said:');
  for (const label of ['PrivaSeer', 'Calpric (USENIX 2023)']) {
    console.log(`\n${label} — ${seen.get(label).length} papers:`);
    for (const id of seen.get(label)) console.log('  ' + id);
  }
}

policies_quotecheck.mjs

// Verify every measured figure privacy:policies publishes against the paper's
// own text — not against the extraction's evidence.quote, which can be a
// different sentence from the one carrying the number.
//
//   node scripts/policies_quotecheck.mjs > scripts/policies_quotecheck-output.txt
//
// Checked against FOUR renderings of each PDF, because they fail on different
// sentences: paper.cols.txt (column order repaired — the one to quote from),
// paper.norm.txt, paper.txt, and a pypdf extraction cached under cache/pypdf/.
// A needle matching any of them is LOCATED, and the rendering that matched is
// named, so a .cols-only miss is visible. The pypdf pass is not decoration:
// PETS 2024 'Honesty is the Best Policy' states its 65% template figure in a
// sentence that .cols splices with an unrelated one, so .cols alone reports a
// false MISS on a figure the paper does state.
//
// Whitespace is collapsed and hyphenation rejoined before matching: a PDF line
// break inside a phrase otherwise produces a false MISS.
//
// Exit status is non-zero if any needle is unlocated, so this cannot pass by
// being ignored.
 
import fs from 'node:fs';
import { spawnSync } from 'node:child_process';
import path from 'node:path';
import { dataRoot } from './lib.mjs';
 
const ROOT = path.join(dataRoot(), 'fulltext');
 
const norm = (s) =>
  s
    .replace(/­/g, '')
    .replace(/-\n/g, '')
    .replace(/[‘’ʼ]/g, "'")
    .replace(/[“”]/g, '"')
    .replace(/[‐-―−]/g, '-')
    .replace(/\s+/g, ' ')
    .trim();
 
const RENDERINGS = ['paper.cols.txt', 'paper.norm.txt', 'paper.txt'];
const PYCACHE = 'cache/pypdf'; // the dataset mount is read-only, so cache locally
 
function pypdfText(dir, venue, year, slug) {
  const out = path.join(PYCACHE, `${year}_${venue}_${slug}.txt`);
  if (!fs.existsSync(out)) {
    const pdf = path.join(dir, 'paper.pdf');
    if (!fs.existsSync(pdf)) return null;
    fs.mkdirSync(PYCACHE, { recursive: true });
    const r = spawnSync('python3', ['-c',
      'import sys,pypdf;print("".join(p.extract_text() or "" for p in pypdf.PdfReader(sys.argv[1]).pages))',
      pdf], { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
    if (r.status !== 0) return null;
    fs.writeFileSync(out, r.stdout);
  }
  return fs.readFileSync(out, 'utf8');
}
 
const cache = new Map();
function renderings(venue, year, slug) {
  const k = `${venue}/${year}/${slug}`;
  if (!cache.has(k)) {
    const dir = path.join(ROOT, String(year), venue, slug);
    const out = {};
    for (const r of RENDERINGS) {
      const f = path.join(dir, r);
      if (fs.existsSync(f)) out[r] = norm(fs.readFileSync(f, 'utf8'));
    }
    const py = pypdfText(dir, venue, year, slug);
    if (py !== null) out['pypdf'] = norm(py);
    if (Object.keys(out).length === 0) throw new Error(`no full text at all: ${dir}`);
    cache.set(k, out);
  }
  return cache.get(k);
}
 
// [venue, year, slug, label as it appears on the page, needle from the PAPER]
// The needle is the paper's own words carrying the number the page prints.
const CHECKS = [
  // --- section: how many sites/apps have a policy at all
  ['NDSS', 2019, 'we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy',
    'web policy availability 84.5% / 79.6%', '84.5 %'],
  ['NDSS', 2019, 'we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy',
    'Degeling sample 6,579 sites, 500 per member state', '500 most popular websites'],
  ['IMC', 2019, 'tales-from-the-porn-a-comprehensive-privacy-analysis-of-the-web-porn-ecosystem',
    'adult sites: only 16% have an accessible policy, of 6,843', 'only 16% of the analyzed websites have an accessible privacy policy'],
  ['PETS', 2025, 'understanding-privacy-norms-through-web-forms',
    'web forms 94.2% policy link', '94.2% (9,559)'],
  ['PETS', 2019, 'maps-scaling-privacy-compliance-analysis-to-a-million-apps',
    'Play policy links 50.5%', '50.5%'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG: 37.5% of links unavailable', 'we found that 37.5%'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG: 20.5% non-English', '20.5%'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG: 22.3% low quality under 2KB/200 words', 'we identify 22.3% (10,375/46,472)'],
  ['USENIX', 2022, 'smart-home-privacy-policies-demystified-a-study-of-availability-content-and-cove',
    'smart-home vendors 48.99% (292/596)', '48.99%'],
  ['NDSS', 2021, 'hey-alexa-is-this-skill-safe-taking-a-closer-look-at-the-alexa-skill-ecosystem',
    'Alexa skills 24.2% policy link', '36,475 (24.2 %)'],
  ['CCS', 2024, 'vpvet-vetting-privacy-policies-of-virtual-reality-apps',
    'VR apps 29.5% have a policy', '(i.e., 29.5%) of privacy policies were successfully found'],
  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
    'chatbots 95.67% lack a policy', '95.67%'],
 
  ['IMC', 2019, 'tales-from-the-porn-a-comprehensive-privacy-analysis-of-the-web-porn-ecosystem',
    'adult sample 6,843 sites', '6,843 pornographic websites'],
  ['PETS', 2025, 'understanding-privacy-norms-through-web-forms',
    'web-forms denominator 10,143', 'leaving 10,143 websites for this analysis'],
  ['PETS', 2019, 'maps-scaling-privacy-compliance-analysis-to-a-million-apps',
    'MAPS denominator 1,049,790 retrieved / 1,035,853 analysed', '1,049,790 retrieved apps, 1,035,853'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG link denominator 37,150/99,194', '(37,150/99,194)'],
  ['USENIX', 2022, 'smart-home-privacy-policies-demystified-a-study-of-availability-content-and-cove',
    'smart home 10.57% no policy at all', '10.57% not providing privacy policies at all'],
  ['NDSS', 2021, 'hey-alexa-is-this-skill-safe-taking-a-closer-look-at-the-alexa-skill-ecosystem',
    'Alexa denominator 150,708 skills, 36,475 with a link', 'Combined 150,708 36,475 (24.2 %)'],
  ['PETS', 2024, 'on-the-quality-of-privacy-policy-documents-of-virtual-personal-assistant-applica',
    'Alexa 21,063 of 65,195', '65,195 skills, of which 21,063'],
  ['CCS', 2024, 'vpvet-vetting-privacy-policies-of-virtual-reality-apps',
    'VR denominator 11,923 apps', '11,923 VR apps'],
  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
    'Discord chatbots 676 (4.35%) have a policy', '676 (4.35%)'],
 
  // --- section: policy versus behaviour
  ['WWW', 2018, 'an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w',
    'only 14.80% of transmissions disclosed', '14.80%'],
  ['USENIX', 2019, 'policylint-investigating-internal-privacy-policy-contradictions-on-google-play',
    'PolicyLint 14.2% (1,618/11,430) contradictions', '1,618/11,430'],
  ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
    'PoliCheck 42.4% of apps', '42.4%'],
  ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
    'PoliCheck only 0.5% of flows clearly disclosed', 'Only 0.5% of data flows were explicitly discussed'],
  ['USENIX', 2022, 'ovrseen-auditing-network-traffic-and-privacy-policies-in-oculus-vr',
    'OVRseen 68% inconsistent', '68% (776/1,135)'],
  ['USENIX', 2023, 'poligraph-automated-privacy-policy-analysis-using-knowledge-graphs',
    'PoliGraph 70.6% recall / 96.9% precision', '70.6% recall'],
  ['USENIX', 2023, 'poligraph-automated-privacy-policy-analysis-using-knowledge-graphs',
    'PoliGraph 25.5% of policies redefine a term', '25.5%'],
  ['CCS', 2023, 'policychecker-analyzing-the-gdpr-completeness-of-mobile-apps-privacy-policies',
    'PolicyChecker 99.3% incomplete', '99.3%'],
  ['CCS', 2023, 'policychecker-analyzing-the-gdpr-completeness-of-mobile-apps-privacy-policies',
    'PolicyChecker 163,068 analysable of 205,973', '163,068'],
  ['USENIX', 2023, 'lalaine-measuring-and-characterizing-non-compliance-of-apple-privacy-labels',
    'Lalaine 3,423 of 5,102 apps', '3,423'],
  ['PETS', 2023, 'lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c',
    'CCPA VCR 80% undisclosed identifier', '(55 apps, 80%)'],
  ['PETS', 2024, 'honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a',
    'Apple labels: 97% of "Data Not Collected" contradicted', 'almost all (97%) apps that indicate in their privacy'],
  ['PETS', 2025, 'understanding-privacy-norms-through-web-forms',
    'phi < 0.20 policy-vs-form association', 'the association appears to be weak (< 0.20) for all PI types'],
 
  // --- section: what the policy text itself looks like
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'Princeton corpus 1,071,488 policies / 130,000 sites', '1,071,488'],
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'median length 876 -> 1,522 words', '1,522'],
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'FKGL 11.9 -> 13.2', 'to 2019B (13.2)'],
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'beacons: 25.8% of policies vs 94.6% of top-10K sites', '25.8%'],
  ['WWW', 2018, 'an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w',
    '84.7 minutes to read applicable policies', '84.7'],
  ['PETS', 2020, 'the-privacy-policy-landscape-after-the-gdpr',
    'EU policies gained 35% words / 33% sentences, Global 25% / 22%', '(+35%, +25%)'],
  ['CCS', 2024, 'vpvet-vetting-privacy-policies-of-virtual-reality-apps',
    'VR policy reuse 54.5% (1,919/3,521)', '54.5%'],
  ['PETS', 2024, 'honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a',
    'template reuse 65%', 'privacy policies of 65%'],
 
  // --- section: the annotated corpora
  ['USENIX', 2018, 'polisis-automated-analysis-and-presentation-of-privacy-policies-using-deep-learn',
    'Polisis trained on 65 of the OPP-115 policies, 50 held out', 'we used the data from 65 policies in the OPP-115 dataset, and we kept 50 policies as a testing set'],
  ['USENIX', 2018, 'polisis-automated-analysis-and-presentation-of-privacy-policies-using-deep-learn',
    'Polisis average F1 0.84', 'Average 0.87 0.83 0.84 0.84'],
  ['USENIX', 2023, 'calpric-inclusive-and-fine-grain-labeling-of-privacy-policies-with-crowdsourcing',
    'Calpric 16,856 labelled segments', '16856 labeled text segments'],
 
  ['PETS', 2023, 'researchers-experiences-in-analyzing-privacy-policies-challenges-and-opportuniti',
    'no best practices have emerged (26 interviews)', 'no clear best practices or methodologies for privacy policy analysis have emerged'],
  ['PETS', 2023, 'researchers-experiences-in-analyzing-privacy-policies-challenges-and-opportuniti',
    '26 researchers interviewed', 'semi-structured interviews with 26 researchers'],
  ['PETS', 2023, 'evolution-of-composition-readability-and-structure-of-privacy-policies-over-two',
    'User Choice/Control semantic change 26%', 'semantic modifications are made to 26% of the policies'],
 
  // --- section: LLM extraction (2024-2026)
  ['IMC', 2024, 'analyzing-corporate-privacy-policies-using-ai-chatbots',
    'GPT-4 chatbot annotation of corporate policies', 'GPT-4'],
  ['IMC', 2025, 'an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems',
    'LLM disclosure classifier 87.44% accuracy', '87.44%'],
  ['IMC', 2025, 'an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems',
    'only 5.8% of Actions clearly disclose', 'only 5.8% of Actions clearly disclosing'],
 
  // --- added 2026-09-10 after the whole-page number guard flagged these as
  //     unaccounted: figures the page prints that no earlier needle covered.
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG low-quality count 10,375/46,472', '22.3% (10,375/46,472)'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG non-English count 9,523/46,472', '(9,523/46,472)'],
  ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
    'APPG 15.7% of apps provide no policy', '15.7% (15,572/99,194)'],
  ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
    'PoliCheck entity-insensitive false-consistency 37.1%', '37.1% of inconsistent third-party flows as consistent'],
  ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
    'PoliCheck 31.1% (14,409/45,603) omitted flows', '31.1% (14,409/45,603)'],
  ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
    'PoliCheck 14,409 omitted disclosures (the 31.1% numerator)', 'Of the 14,409 omitted'],
  ['PETS', 2024, 'honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a',
    'Apple labels: 228,539 apps policy-but-not-label', 'an additional 228,539 apps'],
  ['PETS', 2024, 'honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a',
    'template count n=306,404 behind the 65%', '306, 404'],
  ['CCS', 2023, 'policychecker-analyzing-the-gdpr-completeness-of-mobile-apps-privacy-policies',
    'PolicyChecker 98.1% mandatory-requirement violation', '98.1% of them had at least one'],
  ['NDSS', 2017, 'automated-analysis-of-privacy-requirements-for-mobile-apps',
    'Zimmeck mean 1.83 inconsistencies per app', 'mean of 1.83'],
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'Princeton-Leuven corpus reaches back to 1997', 'policies from as early as 1997'],
  ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
    'the length/readability series is 2009-2019', 'corpus are from 2009-2019'],
 
  // --- added 2026-09-10 after the citations reviewer found two availability
  //     denominators that named a different population from the paper's own.
  ['NDSS', 2019, 'we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy',
    'Degeling: 6357 is the total of the availability table', 'Total 6357 79.6 % 84.5 %'],
  ['NDSS', 2019, 'we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy',
    'Degeling: the same paper also says 6,759 domains', 'contained 6,759 different domains'],
  ['PETS', 2019, 'maps-scaling-privacy-compliance-analysis-to-a-million-apps',
    'MAPS 50.5% is over the analysed set, not the retrieved set', 'our analysis reveals that only 50.5% of apps have links to privacy policies'],
  ['PETS', 2019, 'maps-scaling-privacy-compliance-analysis-to-a-million-apps',
    'MAPS analysed 1,035,853 of 1,049,790 retrieved', '1,035,853'],
  ['PETS', 2026, 'word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using',
    'cory2026 is word-level GDPR transparency annotation by LLM', 'word-level'],
 
  // --- added 2026-09-10 after the generic review found a back-calculated
  //     denominator: 15,528 was 14,852 + 676, not a number the paper states.
  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
    'Discord: the paper states 15,525 unique active chatbots', 'Unique active chatbots 15,525 100%'],
  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
    'Discord: and separately 14,852 (95.67%) without a policy', 'the remaining 14,852 (95.67%)'],
];
 
let miss = 0;
let colsOnlyMiss = 0;
for (const [venue, year, slug, label, needle] of CHECKS) {
  const rs = renderings(venue, year, slug);
  const n = norm(needle).toLowerCase();
  const hits = Object.entries(rs).filter(([, t]) => t.toLowerCase().includes(n)).map(([r]) => r);
  if (hits.length === 0) {
    miss += 1;
    console.log(`MISS      ${venue} ${year}  ${label}\n            needle: ${needle}`);
  } else {
    if (!hits.includes('paper.cols.txt')) colsOnlyMiss += 1;
    const tag = hits.includes('paper.cols.txt') ? 'OK  cols' : `OK  ${hits[0]} (NOT in .cols)`;
    console.log(`${tag.padEnd(28)} ${venue} ${year}  ${label}`);
  }
}
// SPECIFICITY. Locating a needle proves the string is in the right PDF; it does
// not prove the string is the sentence the page is quoting. A bare "26%" is in
// nine of these papers. A 2026-09-10 review showed that changing the Degeling
// needle from "84.5 %" to "84.9 %" still passed, because both appear in that
// paper's tables. So: report, for every needle, how many OTHER check papers also
// contain it, and how long it is. A needle that is short, numeric and shared is
// weak evidence and the page should not lean on it alone.
{
  const papers = [...new Set(CHECKS.map(([v, y, s]) => `${v}/${y}/${s}`))];
  const texts = new Map(papers.map((k) => {
    const [v, y, s] = k.split('/');
    return [k, Object.values(renderings(v, Number(y), s)).join(' \n ').toLowerCase()];
  }));
  const rows = [];
  for (const [venue, year, slug, label, needle] of CHECKS) {
    const n = norm(needle).toLowerCase();
    const own = `${venue}/${year}/${slug}`;
    const elsewhere = papers.filter((k) => k !== own && texts.get(k).includes(n));
    const numericOnly = !/[a-z]/i.test(needle);
    rows.push([needle.length, numericOnly, elsewhere.length, label, needle]);
  }
  const weak = rows.filter(([len, num, el]) => num && el > 0);
  console.log(`\nSPECIFICITY of the ${CHECKS.length} needles`);
  console.log(`  needles with no letters (pure number/punctuation): ${rows.filter((r) => r[1]).length}`);
  console.log(`  of those, also present in another check paper     : ${weak.length}`);
  console.log(`  needles under 12 characters                       : ${rows.filter((r) => r[0] < 12).length}`);
  if (weak.length) {
    console.log('\n  WEAK — numeric-only and not unique to the paper they are attributed to:');
    for (const [len, , el, label, needle] of weak.sort((a, b) => b[2] - a[2])) {
      console.log(`    ${String(el).padStart(2)} other check papers also contain ${JSON.stringify(needle)}  (${label})`);
    }
    console.log('\n  These are not wrong — each was read in context — but they are the needles');
    console.log('  a future edit could break without this check noticing.');
  }
}
 
console.log(`\n${CHECKS.length} needles, ${CHECKS.length - miss} located, ${miss} MISSING.`);
console.log(`${CHECKS.length - miss - colsOnlyMiss} located in paper.cols.txt; ${colsOnlyMiss} located only outside paper.cols.txt.`);
if (miss > 0) process.exitCode = 1;