Differences

This shows you the differences between two versions of the page.

--- provenance:privacy:policies [2026/09/10 17:43] – new page: working log behind privacy:policies — queries, folds and residue, quote check, external checks, review log, every script and its unedited output (Authored by Claude) karel.kubicek.claude
+++ provenance:privacy:policies [2026/09/10 18:09] (current) – publish policies_table_check.mjs and its output; record the guard-floor fix (Authored by Claude) karel.kubicek.claude
@@ Line 9: / Line 9: @@
 ^ Item ^ Value ^
 | Dates | Drafted 2026-09-09; verification, review and publication 2026-09-10. The 2026-09-09 sitting was cut off mid-draft by an API quota stop, so parts of this log were reconstructed on 2026-09-10 from the committed scripts, their outputs and the page draft rather than written as the decisions were made. Where a decision's reasoning could not be recovered, this page says so rather than inventing one |
+| Process failures in this run | Two, both mine. **The pages were published before the generic review returned**, so the first published revision carried the twelve defects that pass listed (including the PrivaSeer error) for about half an hour. **And I edited the content page while that reviewer was reading it**, which is why its report opens by saying the page moved under it. Neither is how this should go: publish after the last pass, and freeze the file while a reviewer holds it |
 | Corpus | ''data/extract/run1/extractions.jsonl'', 5,859 papers with extracted full text; 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026 |
 | Page status | **New page.** Before writing: a wiki search for "privacy policy", "OPP-115", "Polisis" and "privacy label" found **no content page with a policy section**. OPP-115 and Polisis occurred only inside provenance logs. [[privacy:consent]] covers the banner, [[design:mobile_and_app_measurement]] covers store listings, [[practices:legal_enforcement]] covers what regulators did — none covers the document. So this is a creation, not an extension |
-| Page id | ''privacy:policies'', **not** ''privacy:privacy_policies''. Decided 2026-09-07 on [[roadmap]] (see ''provenance:roadmap'' §3) because ''privacy_policies'' sorts next to the site's own ''privacy_policy'' page in search. ''scripts/sitemap.mjs'' already gated on ''privacy:policies'' and the roadmap's Queued table already promised it, so any other id would have left a dangling promise |
+| Page id | ''privacy:policies'', **not** ''privacy:privacy_policies''. Decided 2026-09-07 on [[:roadmap]] (see [[:provenance:roadmap]] §3) because ''privacy_policies'' sorts next to the site's own ''privacy_policy'' page in search. ''scripts/sitemap.mjs'' already gated on ''privacy:policies'' and the roadmap's Queued table already promised it, so any other id would have left a dangling promise |
 | Models | Page, scripts and this log: Claude (Opus 5, with the 2026-09-09 draft written by the same model in an earlier session). Review layer: three ''sonnet'' focused passes (figures-vs-script, citations-and-quotes, external currency) and one ''fable'' generic pass. Findings and verdicts below |
-| Scripts added | ''scripts/report_policies.mjs'' (+ ''-output.txt''), ''scripts/policy_fold.mjs'', ''scripts/policies_fulltext_probe.mjs'' (+ ''-output.txt''), ''scripts/policies_quotecheck.mjs'' (+ ''-output.txt''), ''scripts/policies_significance.py'' (+ ''-output.txt''), ''scripts/policies_external_checks.sh'' (+ ''-output.txt''), ''scripts/policies_fetch_pets_authors.py'', ''scripts/bib_additions_policies.bib'', ''scripts/build_provenance_policies.py'' |
+| Scripts added | ''scripts/report_policies.mjs'' (+ ''-output.txt''), ''scripts/policy_fold.mjs'', ''scripts/policies_fulltext_probe.mjs'' (+ ''-output.txt''), ''scripts/policies_quotecheck.mjs'' (+ ''-output.txt''), ''scripts/policies_significance.py'' (+ ''-output.txt''), ''scripts/policies_external_checks.sh'' (+ ''-output.txt''), ''scripts/policies_gh_search.py'', ''scripts/policies_w3c_p3p_check.mjs'' (+ ''-output.txt''), ''scripts/policies_table_check.mjs'' (+ ''-output.txt''), ''scripts/policies_fetch_pets_authors.py'', ''scripts/bib_additions_policies.bib'', ''scripts/build_provenance_policies.py'' |
 | Write path | ''node scripts/dw.mjs put'' (JSON-RPC) with ''--if-rev'' on every save |
 | Accidental exposure | None. Credentials stayed in ''.env'' and were never echoed. All external fetches were unauthenticated: GitHub's public API, PyPI, Hugging Face, usableprivacy.org, privaseer.ist.psu.edu, developer.apple.com, support.google.com, w3.org, secartifacts.github.io |
@@ Line 22: / Line 23: @@
 | **New page** rather than a section on [[privacy:consent]] | A cookie banner is a UI measurement — you click it and watch what changes. A privacy policy is a document-retrieval and NLP measurement. The two literatures share almost no method and barely cite each other; the corpus enum separates them too (''consent-notice'' 39 papers, ''privacy-policy'' 102, and the overlap is small) | Fold policies into the consent page as "the long version of the notice". Rejected: the reader who needs to retrieve and label 100,000 documents would find a page about clicking buttons |
 | **Scope the page to web //and// mobile**, and say so in the first screen | The platform split is ''mobile'' 60 / ''web'' 56 of the 123 UNION papers (43 mobile-only, 39 web-only, 17 both). A web-only page would silently drop the tool lineage, which was built for Android, and half the availability table | Write a web-only page and send app policies to [[design:mobile_and_app_measurement]]. Rejected: PolicyLint→PoliCheck→PoliGraph is the spine of this literature and it is Android work |
-| **''POLICY'' (102) is the population; ''UNION'' (123) is a candidate set** | ''classification[].target'' is a structured enum, so 102 needs no folding and is stable between extraction runs. The union depends on a regex I chose. Every rate on the page is over 102, 4,439 or 5,859; the only table that quotes shares of 123 is the full-text probe table, which is labelled a candidate set in its own header | Publish rates over 123 because it is the larger, more intuitive number. Rejected — that is the "mention threshold is a candidate set" failure |
+| **''POLICY'' (102) is the population; ''UNION'' (123) is a candidate set** | ''classification[].target'' is a structured enum, so 102 needs no folding and is stable between extraction runs. The union depends on a regex I chose. **Figures denominated on 123 do appear on the page** — the ''UNION'' share columns of the year and venue tables, the full-text probe table, and the sentences quoting probe rows — and every one is either labelled ''UNION'' in its column header or says it is a candidate set. No //rate about the field// is over 123: the method, validation and significance tables are all over the 102. An earlier draft claimed the page carried no 123-denominated figures at all, which was false; the generic review caught it | Publish rates over 123 because it is the larger, more intuitive number. Rejected — that is the "mention threshold is a candidate set" failure |
 | **Widened the title probe from four alternatives to nine** | The 2026-09-02 gap analysis used ''privacy polic|privacy notice|terms of service|privacy label'' and got a union of 121. Adding ''terms and conditions'', ''data safety'', ''nutrition label'', ''policy text'' and ''privacy statement'' takes it to 123, and the papers it adds are the store-declaration half of the literature ({[ali2024_honesty]}-adjacent Data-safety and nutrition-label work). Probe width decides the claim, so both widths are recorded below | Keep the narrow probe for comparability with the gap analysis. Rejected: the narrow probe misses a family the page is about, and the gap analysis was a scoping exercise, not a published figure |
 | **Dated the method table by era, not by year** | 102 papers over 12 years cannot carry a per-year method trend; four era buckets (2014–2018 n=7, 2019–2021 n=21, 2022–2024 n=50, 2025–2026 n=24) each have enough papers to read | Publish per-year percentages. Rejected: n=1 and n=3 years would produce 100% and 0% cells |
@@ Line 52: / Line 53: @@
 | 123, 56, 46, 21 | — | set arithmetic printed by §1 of the report |
 | 179 tuples | ''POLICY'' | tuple count, printed so the page can say it counts papers not tuples |
-| per-year table (2019–2026) | ''corpus'' per year | §1 of the report |
+| per-year table, 2014–2026, both share columns | ''corpus'' per year | §1 of the report. The ''UNION'' share column is a share of a candidate set and its header says so |
-| per-venue table | that venue's corpus slice | §1 of the report |
+| per-venue table, both share columns | that venue's corpus slice | §1 of the report, which also prints the top-venue ratio (PETS is 4.8x the next venue on the ''POLICY'' share and 5.5x on the ''UNION'' share) so the page does not have to divide two percentages by eye |
 | platform split 39 web / 43 mobile / 17 both | ''UNION'' | ''population[].platform'', multi-valued |
 | method-by-era table | ''POLICY'', split into four era buckets | §2 of the report; era n printed in the column header, and the four buckets partition the 102 |
@@ Line 71: / Line 72: @@
 The 2026-09-02 gap analysis that proposed this page used four alternatives (''privacy polic|privacy notice|terms of service|privacy label'') and reported a union of 121. The nine-alternative form gives **123**. Both are recorded because probe width decides the claim; the page quotes the nine-alternative figure and says so.
+**Whole-corpus lineage scan** (added 2026-09-10, same script). For each of the fifteen artefacts in the lineage, a case-insensitive scan of **all 5,869** readable ''paper.cols.txt'' files, counting papers whose text names it. This is a //different question// from the extraction fold in §4 of the report and gives systematically larger answers (Polisis 13 by extraction, 68 in text; PolicyLint 18 and 63). The distinction is the fix for this run's worst error: "no corpus paper's extraction records PrivaSeer" is true, "nobody in these venues cites PrivaSeer" is false, and only the full-text scan can tell the two apart. One artefact needs a different pattern in full text than in the extraction — ''MAPS'' collides with Google Maps and the Play Store's //Maps & Navigation// category, so the full-text pattern is the paper's title. The override and its reason are printed by the script; the shared regex list lives in ''policy_fold.mjs'' so the two scripts cannot drift.
 **Full-text probes** (''policies_fulltext_probe.mjs''): thirteen questions, each with a **narrow** and a **wide** pattern, run over the 123 UNION papers' ''paper.cols.txt''. The page quotes the narrow form throughout and labels it an upper bound on a candidate set — a probe counts papers whose text contains a phrase, not papers that did the thing. The wide form is printed beside it in the output below so a reader can see how far the answer moves: "reports dead policy links" is 7 narrow and 64 wide, and the page's "only 7 of 123" claim would be a different claim at the wide width. Three of the thirteen are **not** narrow/wide pairs but different questions, and the output marks them so.
@@ Line 93: / Line 96: @@
 ''scripts/policies_quotecheck.mjs''. Every measured figure the page prints from a paper is a needle taken from **the paper's own words**, not from the extraction's ''evidence.quote'' — the point is to catch an extraction error, so checking the extraction against itself would prove nothing. Each needle is searched in four renderings: ''paper.cols.txt'', ''paper.norm.txt'', ''paper.txt'' and a ''pypdf'' extraction of ''paper.pdf'' cached under ''cache/pypdf/''. Whitespace, soft hyphens, curly quotes and the several Unicode dashes are normalised before matching.
-**Result: 63 needles, 63 located, 0 missing. 62 in ''paper.cols.txt''; 1 only outside it** — {[amos2021privacy]}'s //"corpus are from 2009-2019"// is spliced by the de-columning and was found only by ''pypdf''. A ''.cols''-only check would have reported a false MISS on it. Full output below.
+**Result: 70 needles, 70 located, 0 missing. 69 in ''paper.cols.txt''; 1 only outside it** — {[amos2021privacy]}'s //"corpus are from 2009-2019"// is spliced by the de-columning and was found only by ''pypdf''. A ''.cols''-only check would have reported a false MISS on it. Full output below.
+Seventeen of the 70 needles were added on 2026-09-10. Ten came from the whole-page number guard, which flagged ten figures the page printed that no earlier needle covered: {[pan2024_trap]}'s 10,375/46,472, 9,523/46,472 and 15.7%; {[andow2020_actions]}'s 37.1% and 31.1% (14,409/45,603); {[ali2024_honesty]}'s 228,539 and n=306,404; {[xiang2023_policychecker]}'s 98.1%; {[zimmeck2017_automated]}'s mean of 1.83; and the two {[amos2021privacy]} year-span needles. Five more came from the citations review, which found two availability rows naming a different population from the paper's own (see //Review//). Two more came from the generic review's back-calculated-denominator finding. All seventeen check out. This is the guard doing its job: the figures were right, but nothing had verified them.
-Ten of the 63 needles were added on 2026-09-10, after the whole-page number guard flagged ten figures the page printed that no earlier needle covered: {[pan2024_trap]}'s 10,375/46,472, 9,523/46,472 and 15.7%; {[andow2020_actions]}'s 37.1% and 31.1% (14,409/45,603); {[ali2024_honesty]}'s 228,539 and n=306,404; {[xiang2023_policychecker]}'s 98.1%; {[zimmeck2017_automated]}'s mean of 1.83; and the two {[amos2021privacy]} year-span needles. All ten check out. This is the guard doing its job: the figures were right, but nothing had verified them.
+**Needle specificity, and what this check does not prove.** Locating a needle proves the string is in the right PDF. It does not prove the string is the sentence the page is quoting: a bare ''26%'' is in nine of these papers, and the generic review showed that changing the Degeling needle from ''84.5 %'' to ''84.9 %'' still passed, because both appear in that paper's tables. The check now prints a specificity report — how many //other// check papers contain each needle, and how many needles are numeric-only or under twelve characters — and nine of the thirteen weak needles it found were rewritten to include the paper's surrounding words. **Four remain weak** and are printed in the output: ''1,071,488'', ''84.7'', ''54.5%'' and ''1,035,853'' each also occur in one other check paper. Each was read in context; none is load-bearing on its own.
 Two things the quote check specifically caught or settled:
@@ Line 104: / Line 109: @@
 ===== Bibliography =====
-The page cites **34** keys. **12** were already in [[literature:bibliography]] and were reused unchanged; **22** were added, in ''scripts/bib_additions_policies.bib'', appended before the closing ''</bibtex>'' of a **freshly exported** copy of the live page (not a local snapshot — the local copy in this workdir was already stale by one page's worth of additions on the morning of 2026-09-10).
+The page cites **38** keys. **14** were already in [[literature:bibliography]] and were reused unchanged; **24** were added in two rounds — 22 with the first publication, and 2 more after the generic review asked for citekeys on four papers the page had described in prose. Two of those four turned out to be in the bibliography already under other keys ({[khatun2026_disclosure]} and {[nenadic2026_swiss]}); the key-string check passed them and ''bib_dedup_scan.py'' caught them on DOI and title. They are in ''scripts/bib_additions_policies.bib'', appended before the closing ''</bibtex>'' of a **freshly exported** copy of the live page (not a local snapshot — the local copy in this workdir was already stale by one page's worth of additions on the morning of 2026-09-10).
 ^ Check ^ Result ^
-| Keys used on the page that resolve after the append | 34 of 34 |
+| Keys used on the page that resolve after both appends | 38 of 38 |
 | New keys colliding with an existing key string | 0 |
-| Definite duplicates by DOI or squashed title (''bib_dedup_scan.py'' over the merged file, 909 entries) | **0** |
+| Definite duplicates by DOI or squashed title (''bib_dedup_scan.py'' over the merged file) | **0** in the first round of 22 (909 entries). **2** in the second round of 4 — both caught and dropped, the page repointed at the existing keys (911 entries) |
 | Candidate duplicate pairs (rule C/D) touching a new key | 4, all judged distinct by hand: ''cui2025_odyssey''/''cui2025_privacy'' (Cui, Jian vs Cui, Hao), ''wu2025_appprivacyreport''/''wu2025_depth'' (Wu, Xiaoyuan vs Wu, Yuhao), ''wu2025_depth''/''wu2025_revealing'' (Wu, Yuhao vs Wu, Mengying), ''zimmeck2017_automated''/''zimmeck2017_privacy'' (same first author, two different 2017 papers) |
 | Literal ASCII ''@'' inside any field of a new entry | 0 — one would silently drop the entry and every marker to it |
-| DOI or landing URL present | 12 entries carry a DOI (CCS, IMC, PoPETs), 10 carry a USENIX or NDSS landing URL |
+| Stray non-BibTeX lines in the additions file | 0. Checked by stripping every ''@entry{…}'' block and asserting the remainder is blank — a generator's QA chatter has gone live on the bibliography page before |
+| DOI or landing URL present | 14 entries carry a DOI (CCS, IMC, PoPETs), 10 carry a USENIX or NDSS landing URL. The four PoPETs 2026 DOIs were each resolved through ''doi.org'' and each returned HTTP 200 at its ''petsymposium.org'' landing page |
 **Authors for PETS records.** The corpus index carries no authors and no DOI for any PETS or USENIX record (100% of both venues). ''scripts/fetch_authors.py'' fails on every PETS landing page as of 2026-09-09 — the pages return HTTP 200 but its parser expects a byline layout petsymposium.org no longer serves. ''scripts/policies_fetch_pets_authors.py'' was written for this run: it reads the ''citation_author'' meta tags from the landing page with ''curl'' and a browser User-Agent, refuses to overwrite a cached entry, and **fails loudly** rather than guessing when no meta tag is present. USENIX authors were taken from the paper PDF, not from usenix.org's own metadata or DBLP — both are known to drop authors from long author lists.
@@ Line 124: / Line 130: @@
 | ''node scripts/check_attributions.mjs'' | content page + merged bibliography | **0 attributions checked, in both table and prose mode — which is NOT a pass.** This page cites by citekey without naming authors in prose, so the guard has nothing to match. Recorded rather than reported as green |
 | ''node scripts/check_page_numbers.mjs'' | the **whole** content page against the concatenation of all five outputs | OK — every figure traces. Run whole-page, not windowed: a windowed run cannot see figures in the introduction or the //Related Pages// section, which is how 29 stale figures once survived a refresh |
+| ''node scripts/policies_table_check.mjs'' | the page's four corpus tables, **cell by cell**, against the report and the significance output | OK — 11 method rows, 12 year rows, 7 venue rows, 5 significance rows. Written on 2026-09-10 because the number guard is a membership test: a mutation changing the ''llm'' count from 12 to 77 passed it, since 77 occurs elsewhere in the output |
 | ''bib_dedup_scan.py'' | live bibliography + the 22 new entries (909 entries) | 0 definite duplicates; 4 candidate pairs touching a new key, all judged distinct by hand |
 | ''build_provenance_policies.py'' structural assertions | this page, at generation time | 16 opening file tags = 16 closing, an even number of inline nowiki delimiters in the prose, no unescaped discussion macro, ≥10 level-2 headings |
-What the number guard does **not** prove: it checks that each numeral on the page appears somewhere in the script output, not that it appears in the //right// sentence, and it prints only the first occurrence's context. A stale figure that happens to collide with a live one passes it. That is why finding 1 of the figures review, and the availability-denominator findings, were caught by a human reading and not by this guard.
+What the number guard does **not** prove: it checks that each numeral on the page appears somewhere in the script output, not that it appears in the //right// sentence, and it prints only the first occurrence's context. A stale figure that happens to collide with a live one passes it — demonstrated, not assumed: changing the ''llm'' row from 12 to 77 passes the number guard. ''policies_table_check.mjs'' was written to close that hole for the four tables it can re-derive, and it catches that mutation and six others. Everything outside those four tables still rests on reading; that is how the availability-denominator findings were caught, not by a guard.
 The number guard's ALLOW map is shared across every page on this wiki, so whitelisting a value here would silently bless it elsewhere. **Nothing was added to it for this page.** The four figures that would have needed an entry were instead given real evidence: two became quote-check needles against the paper text, and two are now printed by ''policies_external_checks.sh''.
@@ Line 200: / Line 207: @@
 One of its corrections was itself incomplete and was corrected in turn: it proposed replacing the discoverability footnote with a claim about Calpric and Lalaine, but a direct GitHub name search showed the genuinely unfindable pair is **PolicyLint and PoliCheck**, whose repository is named after neither tool. A reviewer finding is a lead, not a verdict.
+==== Pass 4 — generic (''fable'') ====
+No checklist. It ran on the corrected page and on this log, and it was the most productive of the four. Its own disclosures, recorded because they matter: one of its mutation-test commands was denied, so a later ''sed'' ran against the committed ''scripts/policies_quotecheck.mjs'' instead of a copy; it reverted the change and said so. Verified independently afterwards — the script reproduces its committed output byte for byte. It also noted that the content page was edited while it read, which is true and is a process failure of mine, recorded below.
+^ # ^ Finding ^ Verdict ^
+| 1 | //"PrivaSeer is named by **zero** corpus papers"// and //"Nobody in these seven venues cites it"// are false: 12 corpus papers name it in their full text, including four the page itself cites | **Accepted, and it is the worst error the four reviews found.** Re-derived independently: a scan of all 5,869 ''paper.cols.txt'' finds exactly 12. The zero was a fold over ''tools''/''sourceList'' — a fact about the //extraction//, published as a fact about the //literature//. Fixed by adding a **second count column** to the lineage table (extraction vs full text) and a whole-corpus full-text scan to the probe script, and by rewording all three places. The scan also found a homonym: ''/\bMAPS\b/'' matches "Google MAPS" and the Play category "Maps & Navigation", so the full-text pattern for MAPS is its title, and the override is printed |
+| 2 | The page publishes ''UNION''-denominated shares (the "share of corpus" and "share of venue" columns, and 70/123 in an open question) while claiming it never does | **Accepted.** The report now prints a ''POLICY'' share beside every ''UNION'' share, both are on the page with ''UNION'' named in the column header, the open question uses 64/102, and the methodology bullet now enumerates the 123-denominated figures instead of denying they exist. The earlier claim was written when the Fisher table was the only offender and was not revisited after pass 1 fixed that one |
+| 3 | Probe counts stated as facts, several with "at all" | **Accepted.** Six sentences rewritten to name the probe and its width. "Only 7 papers say anything at all" became "a narrow probe matches 7 of 123, the wide form 64, and the gap is how much probe width decides the answer" |
+| 4 | The Discord denominator 15,528 is arithmetic on the paper's own non-reconciling numbers; the paper states 15,525 | **Accepted.** The page now gives 15,525 as the paper states it and says the paper's own 14,852 does not reconcile. This is a back-calculated denominator, which is a named failure mode here |
+| 5 | //"''heuristic-rules'' peaked at 40% in 2022–2024"// contradicts the table two screens above (42.9% in 2014–2018) | **Accepted**, reworded |
+| 6 | "three of the four corpora are still downloadable" while every row says live | **Accepted**, "all four were still reachable on 2026-09-10" |
+| 7 | This log was stale after pass 2 (63 needles, "per-year 2019–2026") | **Accepted**, corrected |
+| 8 | ''%%[[roadmap]]%%'' on this page resolves to ''provenance:privacy:roadmap'' and renders red | **Accepted**, and confirmed in the rendered DOM of the first published revision: exactly one ''wikilink2'' span. Now ''%%[[:roadmap]]%%'' |
+| 9 | The Apple December 2020 date is not primary-sourced while the parallel Google date is flagged | **Accepted**, flagged in the same way |
+| 10 | Generator guards: deleting two whole sections passed (floor of ≥10 headings), emptying an output passed, and the file-block count assertion was tautological | **Accepted, all three.** The heading check now names the thirteen expected sections; each output must be over 200 bytes, contain a known terminal string, and be **newer than the script that produced it**; and ''OUTPUTS'' is compared against ''glob('scripts/policies_*-output.txt')'' rather than against itself. All five mutations are now caught |
+| 11 | Quote-check needles are weak: 26 of 68 are bare numbers, and a wrong digit passes | **Accepted.** See //Quotes// above: a specificity report was added and nine needles rewritten. Four remain weak and are printed |
+| 12 | The 46 enum-only papers are characterised from memory as "mostly Alexa-skill, IoT and VR" | **Accepted.** The report now lists all 46 with their platform enum (''other-online-service'' 21, ''mobile'' 20, ''web'' 18, ''iot'' 4, ''offline'' 1) and the page says what is actually there |
+| 13 | "Half of this literature is about **Android** apps" — the store-label half is iOS-heavy | **Accepted**, "mobile apps" |
+| 14 | "Both are current and both are enforced" — no source for enforcement | **Accepted.** The page now says both requirements are current, that neither store publishes enforcement figures, and to treat "required" as a rule rather than a fact about the population |
+| 15 | "every tool in the lineage is abandoned code" overstates last-commit dates | **Accepted**, softened to unmaintained code, with the distinction stated |
+| 16 | Four corpus papers referred to by description with no citekey, so a reader cannot find them | **Accepted.** Keys added for all four. Two of them turned out to be **already in the bibliography under other keys** ({[khatun2026_disclosure]}, {[nenadic2026_swiss]}) — caught by ''bib_dedup_scan.py'', not by the key-string check, which is exactly what that scan exists for |
+| 17 | "four in 2024–2026" store-label comparison papers is a reading, not a count | **Accepted**, the number removed and the reason given |
+| 18 | "the two most-starred" reproductions — the search is by relevance, not stars | **Accepted**, "the top two hits" |
+| 19 | Licence wordings are tighter than the sources: OPP-115 says //"in the spirit of"// CC BY-NC and its commercial licence covers the annotation files | **Accepted** for OPP-115, APP-350 and PrivaSeer. The PolicyLint and PurPliance licence descriptions were left as they are: the external-check output prints the first lines of both ''LICENSE.txt'' files and the page says what they are |
+| 20 | {[chanenson2025_automating]}'s comparison arm is a custom RNN, not an OPP-115-trained classifier, so "the two to copy" misdescribes it | **Accepted**, split into one model and one companion read |
+| 21 | A footnote published this page's own draft history | **Accepted**, moved here |
+| 22 | P3P did define a well-known location, which the page's "no fixed address" bullet invites as an objection | **Accepted**, one clause added |
+| — | Two gaps it named but did not call defects: no reusable //retrieval// code is pointed at, and nothing on the cost of LLM extraction at scale | **Not fixed.** Both are real. The retrieval gap is already the page's first open question; the cost question has no corpus source and would be a vendor-price claim with a shelf life of months. Recorded here rather than guessed at |
+==== Mutation tests of the guards this page publishes ====
+Reading a guard does not tell you whether it asserts anything. Both the generic reviewer and I broke things on purpose in ''/tmp'' copies and checked that the guard failed.
+^ Mutation ^ Guard ^ Result ^
+| A needle that is nowhere in the paper | ''policies_quotecheck.mjs'' | caught |
+| A needle attributed to the wrong paper | ''policies_quotecheck.mjs'' | caught |
+| A needle with the wrong digit (''84.5'' → ''84.9'') | ''policies_quotecheck.mjs'' | **not caught** — both strings are in that paper. Hence the specificity report |
+| An empty needle, or ''%'' | ''policies_quotecheck.mjs'' | **not caught**. Left as it is: the specificity report now makes short needles visible |
+| ''llm'' count 12 → 77 (collides with a live value) | ''check_page_numbers.mjs'' | **not caught** → ''policies_table_check.mjs'' written, which catches it |
+| An era percentage changed to another value from the same table | ''policies_table_check.mjs'' | caught |
+| A method row silently deleted | ''policies_table_check.mjs'' | caught |
+| A venue share changed to a value used elsewhere | ''policies_table_check.mjs'' | caught |
+| A significance base swapped for the naive base rate | ''policies_table_check.mjs'' | caught |
+| A p-value exponent moved from 10⁻⁸ to 10⁻⁵ | ''policies_table_check.mjs'' | **not caught** at first — the check compared three digits and ignored the exponent. Now compares the numbers, and catches it |
+| Odd inline nowiki count in the prose | ''build_provenance_policies.py'' | caught |
+| An unescaped discussion macro | ''build_provenance_policies.py'' | caught |
+| A published script containing a closing file tag | ''build_provenance_policies.py'' | caught |
+| Two whole sections deleted from the prose | ''build_provenance_policies.py'' | **not caught** (floor of ≥10 headings) → now names all thirteen, and catches it |
+| An output file emptied, or truncated | ''build_provenance_policies.py'' | **not caught** → now caught |
+| An output dropped from the published list | ''build_provenance_policies.py'' | **not caught** (tautological) → now compared against the filesystem, and caught |
+| A script edited without re-running it | ''build_provenance_policies.py'' | **not caught** → now caught by an mtime comparison |
+Seven of seventeen mutations survived the guards as first written. That ratio is the argument for mutation-testing every published check rather than reading it.
 ===== The scripts, as committed =====
@@ Line 230: / Line 291: @@
 import { loadExtractions, pct, table, isSentinel } from './lib.mjs';
-import { foldSource, foldTool } from './policy_fold.mjs';
+import { foldSource, foldTool, LINEAGE } from './policy_fold.mjs';
 const P = loadExtractions();
@@ Line 268: / Line 329: @@
 console.log(`  PROBE only (title fires, enum silent)                  ${PROBE.filter((p) => !POLICY.includes(p)).length}`);
 console.log(`\nprivacy-policy tuples in POLICY                          ${POLICY.reduce((n, p) => n + p.classification.filter((c) => c.target === 'privacy-policy').length, 0)}`);
+// The enum-only papers are the argument for not using a title probe alone, so
+// print them rather than characterising them from memory. A 2026-09-10 review
+// found the page describing this set as "mostly Alexa-skill, IoT and VR studies"
+// on no evidence; the list below is the evidence.
+sub('POLICY-only: the enum fires and the title probe is silent (why a title probe is not enough)');
+{
+  const only = POLICY.filter((p) => !PROBE.includes(p))
+    .sort((a, b) => a.year - b.year || a.venue.localeCompare(b.venue));
+  console.log(`${only.length} papers. Platform measured (enum, multi-valued):`);
+  const plat = {};
+  for (const p of only) for (const x of new Set(p.platforms)) plat[x] = (plat[x] ?? 0) + 1;
+  console.log(table(['platform', 'papers of the enum-only set'],
+    Object.entries(plat).sort((a, b) => b[1] - a[1])));
+  if (!Object.keys(plat).length) throw new Error('enum-only platform table came out empty');
+  for (const p of only) console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
+}
 sub('classification[].target, whole corpus, papers (enum — publishable)');
@@ Line 288: / Line 366: @@
   const allY = {};
   for (const p of P) allY[p.year] = (allY[p.year] ?? 0) + 1;
-  console.log(table(['Year', 'corpus', 'POLICY', 'PROBE', 'UNION', 'UNION share of corpus'],
+  // Both shares are printed. UNION/corpus is a share of a regex-widened
-    Object.keys(y).sort().map((k) => [k + (k >= '2025' ? '*' : ''), allY[k], y[k].e, y[k].t, y[k].u, pct(y[k].u, allY[k])])));
+  // candidate set and must be labelled as such wherever it is published;
+  // POLICY/corpus is the share of the stable enum.
+  console.log(table(['Year', 'corpus', 'POLICY', 'PROBE', 'UNION', 'POLICY share of corpus', 'UNION share of corpus'],
+    Object.keys(y).sort().map((k) => [k + (k >= '2025' ? '*' : ''), allY[k], y[k].e, y[k].t, y[k].u,
+      pct(y[k].e, allY[k]), pct(y[k].u, allY[k])])));
 }
@@ Line 297: / Line 379: @@
   for (const p of P) vAll[p.venue] = (vAll[p.venue] ?? 0) + 1;
   for (const p of UNION) v[p.venue] = (v[p.venue] ?? 0) + 1;
-  console.log(table(['Venue', 'UNION', 'POLICY', 'venue papers', 'UNION share of venue'],
+  console.log(table(['Venue', 'UNION', 'POLICY', 'venue papers', 'POLICY share of venue', 'UNION share of venue'],
-    Object.entries(v).sort((a, b) => b[1] - a[1]).map(([k, n]) => [k, n, POLICY.filter((p) => p.venue === k).length, vAll[k], pct(n, vAll[k])])));
+    Object.entries(v).sort((a, b) => b[1] - a[1]).map(([k, n]) => {
+      const pol = POLICY.filter((p) => p.venue === k).length;
+      return [k, n, pol, vAll[k], pct(pol, vAll[k]), pct(n, vAll[k])];
+    })));
+  // The page says "PETS by a factor of five over the next venue"; print the ratio
+  // rather than leaving the reader to divide two percentages by eye.
+  {
+    const byPolicy = Object.keys(vAll)
+      .map((k) => [k, POLICY.filter((p) => p.venue === k).length / vAll[k]])
+      .sort((a, b) => b[1] - a[1]);
+    const byUnion = Object.entries(v).map(([k, n]) => [k, n / vAll[k]]).sort((a, b) => b[1] - a[1]);
+    console.log(`top venue over the next, POLICY share: ${byPolicy[0][0]} / ${byPolicy[1][0]} = ${(byPolicy[0][1] / byPolicy[1][1]).toFixed(1)}x`);
+    console.log(`top venue over the next, UNION share : ${byUnion[0][0]} / ${byUnion[1][0]} = ${(byUnion[0][1] / byUnion[1][1]).toFixed(1)}x`);
+  }
 }
@@ Line 410: / Line 505: @@
 console.log('detection[].phenomenon/technique. Counts are PAPERS.');
 {
-  const NAMED = [
+  const NAMED = LINEAGE;   // defined in policy_fold.mjs, shared with the full-text probe
-    ['Privee (USENIX 2014)', /\bPrivee\b/],
-    ['OPP-115 corpus (ACL 2016)', /OPP-?115/i],
-    ['Polisis / PriBot (USENIX 2018)', /polisis|pribot/i],
-    ['PolicyLint (USENIX 2019)', /policylint/i],
-    ['MAPS (PETS 2019)', /\bMAPS\b/],
-    ['APP-350 corpus (2019)', /APP-?350/i],
-    ['PoliCheck (USENIX 2020)', /policheck/i],
-    ['PurPliance (2021)', /purpliance/i],
-    ['PrivBERT (2021)', /privbert/i],
-    ['Calpric (USENIX 2023)', /calpric/i],
-    ['PoliGraph / PoliGraph-er (USENIX 2023)', /poligraph/i],
-    ['PolicyChecker (CCS 2023)', /policychecker/i],
-    ['Lalaine (USENIX 2023)', /lalaine/i],
-    ['PolicyComp (USENIX 2023)', /policycomp/i],
-    ['PrivaSeer', /privaseer/i],
-  ];
   const hay = (p) => {
     const a = [];
@@ Line 483: / Line 562: @@
     ['yan2024_quality', 'PETS/2024/on-the-quality-of-privacy-policy-documents-of-virtual-personal-assistant-applica', 'Alexa skills', '65,195 skills', '21,063 of 65,195 provided a policy link'],
     ['zhan2024_vpvet', 'CCS/2024/vpvet-vetting-privacy-policies-of-virtual-reality-apps', 'VR apps', '11,923 apps on 10 VR platforms', '29.5% had a findable privacy policy'],
-    ['edu2022_exploring', 'IMC/2022/exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services', 'Discord chatbots requesting permissions', '15,528 chatbots', '676 (4.35%) had a policy; 14,852 (95.67%) did not'],
+    ['edu2022_exploring', 'IMC/2022/exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services', 'Discord chatbots requesting permissions', "15,525 unique active chatbots (the paper's own Table 2 total); its 14,852-without figure does not reconcile with it", '676 (4.35%) had a policy; 14,852 (95.67%) did not'],
     ['wu2025_depth', 'IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems', 'GPT Actions', 'Actions declaring a legal_info_url', '93.96% of those policies were reachable'],
   ];
@@ Line 720: / Line 799: @@
 export const foldSource = (raw) => fold(raw, SOURCE_FAMILIES);
 export const foldTool = (raw) => fold(raw, TOOL_FAMILIES);
+// The policy-analysis lineage, one regex per artefact. Exported so that
+// report_policies.mjs (which matches it against EXTRACTION fields) and
+// policies_fulltext_probe.mjs (which matches it against the paper's FULL TEXT)
+// cannot drift apart. The two ask different questions and give different
+// answers: PrivaSeer is named as a tool or data source by no corpus paper and
+// cited in the text of twelve.
+export const LINEAGE = [
+  ['Privee (USENIX 2014)', /\bPrivee\b/],
+  ['OPP-115 corpus (ACL 2016)', /OPP-?115/i],
+  ['Polisis / PriBot (USENIX 2018)', /polisis|pribot/i],
+  ['PolicyLint (USENIX 2019)', /policylint/i],
+  ['MAPS (PETS 2019)', /\bMAPS\b/],
+  ['APP-350 corpus (2019)', /APP-?350/i],
+  ['PoliCheck (USENIX 2020)', /policheck/i],
+  ['PurPliance (2021)', /purpliance/i],
+  ['PrivBERT (2021)', /privbert/i],
+  ['Calpric (USENIX 2023)', /calpric/i],
+  ['PoliGraph / PoliGraph-er (USENIX 2023)', /poligraph/i],
+  ['PolicyChecker (CCS 2023)', /policychecker/i],
+  ['Lalaine (USENIX 2023)', /lalaine/i],
+  ['PolicyComp (USENIX 2023)', /policycomp/i],
+  ['PrivaSeer', /privaseer/i],
+];
+// Matching a tool name against a paper's FULL TEXT is not the same problem as
+// matching it against the extraction's tool fields. In full text an acronym
+// collides with unrelated uses: /\bMAPS\b/ matches "Google MAPS abuse" and the
+// Play Store's "MAPS & NAVIGATION" category. Where that happens, the full-text
+// scan uses the artefact's own title instead of its acronym, and the override is
+// recorded here rather than buried in the scanning script.
+export const LINEAGE_FULLTEXT_OVERRIDE = new Map([
+  ['MAPS (PETS 2019)', {
+    re: /MAPS:\s*Scaling/i,
+    why: '/\\bMAPS\\b/ also matches "Google MAPS" and the Play category "MAPS & NAVIGATION"; the paper is always cited by its title',
+  }],
+]);
 </file>
@@ Line 749: / Line 865: @@
 import path from 'node:path';
 import { loadExtractions, dataRoot, pct, table } from './lib.mjs';
+import { LINEAGE, LINEAGE_FULLTEXT_OVERRIDE } from './policy_fold.mjs';
 const ROOT = path.join(dataRoot(), 'fulltext');
@@ Line 841: / Line 958: @@
   console.log(`\n${label}  —  ${hits.length} papers`);
   for (const p of hits) console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title.slice(0, 92)}`);
+}
+// ---------------------------------------------------------------------------
+// The lineage, counted a second way: WHOLE CORPUS full text rather than the
+// extraction's tool/source fields.
+//
+// These are different questions and they give different answers. "No corpus
+// paper's extraction names PrivaSeer" is a fact about the extraction. "Nobody
+// in these seven venues cites PrivaSeer" would be a fact about the literature,
+// and it is false. A fold over a structured field cannot support an "at all"
+// claim; only a full-text scan can even try.
+//
+// Scans every paper.cols.txt in the corpus, not just the UNION.
+{
+  const ROOTDIR = ROOT;
+  const all = [];
+  for (const y of fs.readdirSync(ROOTDIR)) {
+    const yd = path.join(ROOTDIR, y);
+    if (!fs.statSync(yd).isDirectory()) continue;
+    for (const v of fs.readdirSync(yd)) {
+      const vd = path.join(yd, v);
+      if (!fs.statSync(vd).isDirectory()) continue;
+      for (const s of fs.readdirSync(vd)) {
+        const f = path.join(vd, s, 'paper.cols.txt');
+        if (fs.existsSync(f)) all.push([`${y} ${v.padEnd(8)} ${s}`, f]);
+      }
+    }
+  }
+  console.log(`\n\n--- LINEAGE ARTEFACTS: extraction fold vs FULL-TEXT mention, whole corpus`);
+  console.log(`Full-text denominator: ${all.length} papers with a readable paper.cols.txt.`);
+  console.log(`"extraction" is the column report_policies.mjs section 4 prints.`);
+  const PATTERNS = LINEAGE.map(([label, re]) => {
+    const o = LINEAGE_FULLTEXT_OVERRIDE.get(label);
+    return [label, o ? o.re : re, o ? o.why : ''];
+  });
+  const seen = new Map(PATTERNS.map(([label]) => [label, []]));
+  for (const [id, f] of all) {
+    const t2 = norm(fs.readFileSync(f, 'utf8').replace(/\0/g, ''));
+    for (const [label, re] of PATTERNS) if (re.test(t2)) seen.get(label).push(id);
+  }
+  console.log(table(['artefact', 'full-text pattern', 'papers whose FULL TEXT names it'],
+    PATTERNS.map(([label, re]) => [label, String(re), seen.get(label).length])));
+  const overridden = PATTERNS.filter(([, , why]) => why);
+  console.log('\nFull-text patterns that differ from the extraction pattern, and why:');
+  for (const [label, re, why] of overridden) console.log(`  ${label}: ${re} — ${why}`);
+  console.log('\nThe two artefacts where the gap changes what may be said:');
+  for (const label of ['PrivaSeer', 'Calpric (USENIX 2023)']) {
+    console.log(`\n${label} — ${seen.get(label).length} papers:`);
+    for (const id of seen.get(label)) console.log('  ' + id);
+  }
 }
 </file>
@@ Line 932: / Line 1099: @@
     'adult sites: only 16% have an accessible policy, of 6,843', 'only 16% of the analyzed websites have an accessible privacy policy'],
   ['PETS', 2025, 'understanding-privacy-norms-through-web-forms',
-    'web forms 94.2% policy link', '94.2%'],
+    'web forms 94.2% policy link', '94.2% (9,559)'],
   ['PETS', 2019, 'maps-scaling-privacy-compliance-analysis-to-a-million-apps',
     'Play policy links 50.5%', '50.5%'],
   ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
-    'APPG: 37.5% of links unavailable', '37.5%'],
+    'APPG: 37.5% of links unavailable', 'we found that 37.5%'],
   ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
     'APPG: 20.5% non-English', '20.5%'],
   ['USENIX', 2024, 'is-it-a-trap-a-large-scale-empirical-study-and-comprehensive-assessment-of-onlin',
-    'APPG: 22.3% low quality under 2KB/200 words', '22.3%'],
+    'APPG: 22.3% low quality under 2KB/200 words', 'we identify 22.3% (10,375/46,472)'],
   ['USENIX', 2022, 'smart-home-privacy-policies-demystified-a-study-of-availability-content-and-cove',
     'smart-home vendors 48.99% (292/596)', '48.99%'],
   ['NDSS', 2021, 'hey-alexa-is-this-skill-safe-taking-a-closer-look-at-the-alexa-skill-ecosystem',
-    'Alexa skills 24.2% policy link', '24.2%'],
+    'Alexa skills 24.2% policy link', '36,475 (24.2 %)'],
   ['CCS', 2024, 'vpvet-vetting-privacy-policies-of-virtual-reality-apps',
-    'VR apps 29.5% have a policy', '29.5%'],
+    'VR apps 29.5% have a policy', '(i.e., 29.5%) of privacy policies were successfully found'],
   ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
     'chatbots 95.67% lack a policy', '95.67%'],
@@ Line 977: / Line 1144: @@
     'PoliCheck 42.4% of apps', '42.4%'],
   ['USENIX', 2020, 'actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an',
-    'PoliCheck only 0.5% of flows clearly disclosed', '0.5%'],
+    'PoliCheck only 0.5% of flows clearly disclosed', 'Only 0.5% of data flows were explicitly discussed'],
   ['USENIX', 2022, 'ovrseen-auditing-network-traffic-and-privacy-policies-in-oculus-vr',
     'OVRseen 68% inconsistent', '68% (776/1,135)'],
@@ Line 1003: / Line 1170: @@
     'median length 876 -> 1,522 words', '1,522'],
   ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
-    'FKGL 11.9 -> 13.2', '13.2'],
+    'FKGL 11.9 -> 13.2', 'to 2019B (13.2)'],
   ['WWW', 2021, 'privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset',
     'beacons: 25.8% of policies vs 94.6% of top-10K sites', '25.8%'],
@@ Line 1028: / Line 1195: @@
     '26 researchers interviewed', 'semi-structured interviews with 26 researchers'],
   ['PETS', 2023, 'evolution-of-composition-readability-and-structure-of-privacy-policies-over-two',
-    'User Choice/Control semantic change 26%', '26%'],
+    'User Choice/Control semantic change 26%', 'semantic modifications are made to 26% of the policies'],
   // --- section: LLM extraction (2024-2026)
@@ Line 1036: / Line 1203: @@
     'LLM disclosure classifier 87.44% accuracy', '87.44%'],
   ['IMC', 2025, 'an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems',
-    'only 5.8% of Actions clearly disclose', '5.8%'],
+    'only 5.8% of Actions clearly disclose', 'only 5.8% of Actions clearly disclosing'],
   // --- added 2026-09-10 after the whole-page number guard flagged these as
@@ Line 1077: / Line 1244: @@
   ['PETS', 2026, 'word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using',
     'cory2026 is word-level GDPR transparency annotation by LLM', 'word-level'],
+  // --- added 2026-09-10 after the generic review found a back-calculated
+  //     denominator: 15,528 was 14,852 + 676, not a number the paper states.
+  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
+    'Discord: the paper states 15,525 unique active chatbots', 'Unique active chatbots 15,525 100%'],
+  ['IMC', 2022, 'exploring-the-security-and-privacy-risks-of-chatbots-in-messaging-services',
+    'Discord: and separately 14,852 (95.67%) without a policy', 'the remaining 14,852 (95.67%)'],
 ];
@@ Line 1094: / Line 1268: @@
   }
 }
+// SPECIFICITY. Locating a needle proves the string is in the right PDF; it does
+// not prove the string is the sentence the page is quoting. A bare "26%" is in
+// nine of these papers. A 2026-09-10 review showed that changing the Degeling
+// needle from "84.5 %" to "84.9 %" still passed, because both appear in that
+// paper's tables. So: report, for every needle, how many OTHER check papers also
+// contain it, and how long it is. A needle that is short, numeric and shared is
+// weak evidence and the page should not lean on it alone.
+{
+  const papers = [...new Set(CHECKS.map(([v, y, s]) => `${v}/${y}/${s}`))];
+  const texts = new Map(papers.map((k) => {
+    const [v, y, s] = k.split('/');
+    return [k, Object.values(renderings(v, Number(y), s)).join(' \n ').toLowerCase()];
+  }));
+  const rows = [];
+  for (const [venue, year, slug, label, needle] of CHECKS) {
+    const n = norm(needle).toLowerCase();
+    const own = `${venue}/${year}/${slug}`;
+    const elsewhere = papers.filter((k) => k !== own && texts.get(k).includes(n));
+    const numericOnly = !/[a-z]/i.test(needle);
+    rows.push([needle.length, numericOnly, elsewhere.length, label, needle]);
+  }
+  const weak = rows.filter(([len, num, el]) => num && el > 0);
+  console.log(`\nSPECIFICITY of the ${CHECKS.length} needles`);
+  console.log(`  needles with no letters (pure number/punctuation): ${rows.filter((r) => r[1]).length}`);
+  console.log(`  of those, also present in another check paper     : ${weak.length}`);
+  console.log(`  needles under 12 characters                       : ${rows.filter((r) => r[0] < 12).length}`);
+  if (weak.length) {
+    console.log('\n  WEAK — numeric-only and not unique to the paper they are attributed to:');
+    for (const [len, , el, label, needle] of weak.sort((a, b) => b[2] - a[2])) {
+      console.log(`    ${String(el).padStart(2)} other check papers also contain ${JSON.stringify(needle)}  (${label})`);
+    }
+    console.log('\n  These are not wrong — each was read in context — but they are the needles');
+    console.log('  a future edit could break without this check noticing.');
+  }
+}
 console.log(`\n${CHECKS.length} needles, ${CHECKS.length - miss} located, ${miss} MISSING.`);
 console.log(`${CHECKS.length - miss - colsOnlyMiss} located in paper.cols.txt; ${colsOnlyMiss} located only outside paper.cols.txt.`);
@@ Line 1168: / Line 1378: @@
 print('\n"rest of corpus" removes the subgroup\'s own papers from the base; the')
 print('bracketed column is the naive base rate that leaves them in.')
+</file>
+==== The cell-by-cell table check — ''policies_table_check.mjs'' ====
+<file javascript policies_table_check.mjs>
+// Assert that privacy:policies' four corpus tables match report_policies.mjs
+// CELL BY CELL, not merely that each numeral occurs somewhere in the output.
+//
+//   node scripts/policies_table_check.mjs
+//
+// WHY this exists in addition to check_page_numbers.mjs: that guard tests
+// whether every numeral on the page appears anywhere in the concatenated script
+// output. A mutation test on 2026-09-10 changed the `llm` row's count from 12 to
+// 77 and the guard still passed, because 77 occurs elsewhere in the output (the
+// PROBE size, and the public-artifacts count). A wrong figure that collides with
+// a live one is invisible to a membership test. This one re-derives each row
+// from the report and compares positionally.
+//
+// Exits non-zero on the first mismatch, printing the row it disagrees with.
+import fs from 'node:fs';
+const PAGE = process.argv[2] ?? 'pages/privacy_policies.txt';
+const REPORT = process.argv[3] ?? 'scripts/report_policies-output.txt';
+const SIG = process.argv[4] ?? 'scripts/policies_significance-output.txt';
+const page = fs.readFileSync(PAGE, 'utf8');
+const report = fs.readFileSync(REPORT, 'utf8');
+const sig = fs.readFileSync(SIG, 'utf8');
+let failures = 0;
+const fail = (what, expected, got) => {
+  failures += 1;
+  console.log(`MISMATCH  ${what}\n  report: ${expected}\n  page  : ${got}`);
+};
+const ok = (what) => console.log(`OK        ${what}`);
+// A page table row -> array of trimmed cells, markup stripped.
+const cells = (line) =>
+  line.replace(/^\||\|$/g, '').split('|')
+    .map((c) => c.replace(/\*\*|''|\/\//g, '').trim());
+const pageRows = (headerMatch) => {
+  const lines = page.split('\n');
+  const i = lines.findIndex((l) => l.startsWith('^') && headerMatch.test(l));
+  if (i < 0) throw new Error(`no page table whose header matches ${headerMatch}`);
+  const out = [];
+  for (let j = i + 1; j < lines.length && lines[j].startsWith('|'); j += 1) out.push(cells(lines[j]));
+  if (!out.length) throw new Error(`page table ${headerMatch} has no rows`);
+  return out;
+};
+const reportSection = (marker) => {
+  const i = report.indexOf(marker);
+  if (i < 0) throw new Error(`report has no section ${JSON.stringify(marker)}`);
+  const rest = report.slice(i + marker.length);
+  // A section ends at the next "--- " subsection header or "===" banner. The
+  // column-rule line under each table is also dashes, so match the space.
+  const end = rest.search(/\n(--- |=====)/);
+  return (end < 0 ? rest : rest.slice(0, end)).split('\n').filter((l) => l.trim());
+};
+// ---------------------------------------------------------------- 1. by era
+{
+  const want = new Map();
+  for (const l of reportSection('--- method by era')) {
+    const m = l.match(/^(\S+)\s+(\d+)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)/);
+    if (m) want.set(m[1], m.slice(2));
+  }
+  if (want.size < 5) throw new Error('parsed too few method-by-era rows from the report');
+  const rows = pageRows(/classification\.method/);
+  if (rows.length !== want.size) fail('method-by-era row count', want.size, rows.length);
+  for (const r of rows) {
+    const [name, all, ...eras] = r;
+    if (!want.has(name)) { fail(`method row "${name}"`, '(no such method in the report)', r.join(' | ')); continue; }
+    const w = want.get(name);
+    const got = [all, ...eras.map((e) => e.replace('%', ''))];
+    const exp = [w[0], ...w.slice(1).map((x) => String(parseFloat(x)))];
+    const norm = got.map((x) => String(parseFloat(x)));
+    if (norm.join(',') !== exp.join(',')) fail(`method row "${name}"`, exp.join(' | '), norm.join(' | '));
+  }
+  ok(`method-by-era table: ${rows.length} rows`);
+}
+// ---------------------------------------------------------------- 2. by year
+{
+  const want = new Map();
+  for (const l of reportSection('--- POLICY and UNION per year')) {
+    const m = l.match(/^(\d{4})\*?\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+([\d.]+)%/);
+    if (m) want.set(m[1], [m[2], m[3], m[5], m[6]]); // corpus, POLICY, UNION, share
+  }
+  const rows = pageRows(/\^ Year \^/);
+  for (const r of rows) {
+    const year = r[0].replace('*', '');
+    if (!want.has(year)) { fail(`year row ${year}`, '(not in the report)', r.join(' | ')); continue; }
+    const w = want.get(year);
+    const got = [r[1].replace(/,/g, ''), r[2], r[3], r[4].replace('%', '')];
+    const exp = [w[0], w[1], w[2], String(parseFloat(w[3]))];
+    if (got.map((x) => String(parseFloat(x))).join(',') !== exp.join(',')) {
+      fail(`year row ${year}`, exp.join(' | '), got.join(' | '));
+    }
+  }
+  if (rows.length !== want.size) fail('per-year row count', want.size, rows.length);
+  ok(`per-year table: ${rows.length} rows`);
+}
+// --------------------------------------------------------------- 3. by venue
+{
+  const want = new Map();
+  for (const l of reportSection('--- UNION per venue')) {
+    const m = l.match(/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+([\d.]+)%/);
+    if (m) want.set(m[1], [m[2], m[3], m[4], m[5]]);
+  }
+  const alias = { 'USENIX Sec': 'USENIX', TheWebConf: 'WWW', 'IEEE S&P': 'IEEE-SP' };
+  const rows = pageRows(/\^ Venue \^/);
+  for (const r of rows) {
+    const v = alias[r[0]] ?? r[0];
+    if (!want.has(v)) { fail(`venue row ${r[0]}`, '(not in the report)', r.join(' | ')); continue; }
+    const w = want.get(v);
+    const got = [r[1], r[2], r[3].replace(/,/g, ''), r[4].replace('%', '')];
+    const exp = [w[0], w[1], w[2], String(parseFloat(w[3]))];
+    if (got.map((x) => String(parseFloat(x))).join(',') !== exp.join(',')) {
+      fail(`venue row ${r[0]}`, exp.join(' | '), got.join(' | '));
+    }
+  }
+  if (rows.length !== want.size) fail('per-venue row count', want.size, rows.length);
+  ok(`per-venue table: ${rows.length} rows`);
+}
+// ---------------------------------------------------------- 4. Fisher's exact
+{
+  // Only the POLICY rows are published; the UNION rows stay on the provenance page.
+  const want = [];
+  for (const l of sig.split('\n')) {
+    const m = l.match(/^(.+?)\s+POLICY\s+(\d+)\/(\d+) \(\s*[\d.]+%\)\s+(\d+)\/(\d+) \(\s*[\d.]+%\)\s+(\S+)/);
+    if (m) want.push({ sub: `${m[2]}/${m[3]}`, base: `${m[4]}/${m[5]}`, p: m[6] });
+  }
+  if (want.length !== 5) throw new Error(`expected 5 POLICY significance rows, parsed ${want.length}`);
+  const rows = pageRows(/\^ Property \^/);
+  if (rows.length !== want.length) fail('significance row count', want.length, rows.length);
+  // The page orders rows for reading; match on the subgroup fraction, not position.
+  const bySub = new Map(want.map((w) => [w.sub, w]));
+  for (const r of rows) {
+    const sub = r[1].replace(/,/g, '').split(' ')[0];
+    if (!bySub.has(sub)) { fail(`significance row "${r[0]}"`, '(no POLICY row with that numerator/denominator)', r[1]); continue; }
+    const w = bySub.get(sub);
+    const base = r[2].replace(/,/g, '').split(' ')[0];
+    if (base !== w.base) fail(`significance base for "${r[0]}"`, w.base, base);
+    // p is printed on the page in scientific form with superscript digits
+    // ("2.65 x 10^-8"); the script prints "2.65e-08". Normalise BOTH to a number
+    // and compare the value, exponent included. An earlier form of this check
+    // compared only the first three digits and a mutation test on 2026-09-10
+    // showed it could not see an exponent moved from -8 to -5.
+    const SUP = { '⁰': '0', '¹': '1', '²': '2', '³': '3', '⁴': '4', '⁵': '5', '⁶': '6', '⁷': '7', '⁸': '8', '⁹': '9', '⁻': '-' };
+    const pageP = (() => {
+      const s = r[3].split('—')[0].trim();
+      const sci = s.match(/^([\d.]+)\s*×\s*10([⁻⁰¹²³⁴⁵⁶⁷⁸⁹]+)$/);
+      if (sci) {
+        const exp = [...sci[2]].map((c) => SUP[c] ?? c).join('');
+        return Number(sci[1]) * 10 ** Number(exp);
+      }
+      return Number(s);
+    })();
+    const repP = Number(w.p);
+    if (!Number.isFinite(pageP)) {
+      fail(`significance p for "${r[0]}" is unparseable`, w.p, r[3]);
+    } else if (Math.abs(pageP - repP) > Math.abs(repP) * 0.02) {
+      fail(`significance p for "${r[0]}"`, `${w.p} (${repP})`, `${r[3]} (${pageP})`);
+    }
+  }
+  ok(`significance table: ${rows.length} rows`);
+}
+console.log(failures ? `\n${failures} MISMATCHES` : '\nAll four tables match the report cell by cell.');
+process.exitCode = failures ? 1 : 0;
 </file>
@@ Line 1506: / Line 1890: @@
 in-place generator can silently drop a section.
 """
+import glob
 import os
 import re
@@ Line 1520: / Line 1905: @@
     ('The quote check', 'scripts/policies_quotecheck.mjs', 'javascript'),
     ('The significance test', 'scripts/policies_significance.py', 'python'),
+    ('The cell-by-cell table check', 'scripts/policies_table_check.mjs', 'javascript'),
     ('The external checks', 'scripts/policies_external_checks.sh', 'bash'),
     ('The GitHub name-search check', 'scripts/policies_gh_search.py', 'python'),
@@ Line 1528: / Line 1914: @@
 ]
+# (producing script, output path, a substring the output MUST end with).
+# The third field exists because an empty or truncated output file is the exact
+# way this generator can publish nothing and still pass: a 2026-09-10 mutation
+# test emptied one of these files and the page shipped an empty <file> block.
 OUTPUTS = [
-    ('report_policies.mjs', 'scripts/report_policies-output.txt'),
+    ('report_policies.mjs', 'scripts/report_policies-output.txt',
-    ('policies_fulltext_probe.mjs', 'scripts/policies_fulltext_probe-output.txt'),
+     'PROBE 2026 PETS'),
-    ('policies_significance.py', 'scripts/policies_significance-output.txt'),
+    ('policies_fulltext_probe.mjs', 'scripts/policies_fulltext_probe-output.txt',
-    ('policies_quotecheck.mjs', 'scripts/policies_quotecheck-output.txt'),
+     'Calpric (USENIX 2023) —'),
-    ('policies_external_checks.sh', 'scripts/policies_external_checks-output.txt'),
+    ('policies_significance.py', 'scripts/policies_significance-output.txt',
-    ('policies_w3c_p3p_check.mjs', 'scripts/policies_w3c_p3p_check-output.txt'),
+     'bracketed column is the naive base rate that leaves them in.'),
+    ('policies_table_check.mjs', 'scripts/policies_table_check-output.txt',
+     'All four tables match the report cell by cell.'),
+    ('policies_quotecheck.mjs', 'scripts/policies_quotecheck-output.txt',
+     'located only outside paper.cols.txt.'),
+    ('policies_external_checks.sh', 'scripts/policies_external_checks-output.txt',
+     'All repository lookups returned a date, not an API error.'),
+    ('policies_w3c_p3p_check.mjs', 'scripts/policies_w3c_p3p_check-output.txt',
+     'Last updated'),
 ]
@@ Line 1567: / Line 1965: @@
 parts.append('===== The outputs, unedited =====\n')
-for name, path in OUTPUTS:
+for name, path, tail in OUTPUTS:
     parts.append(f"==== Output of ''{name}'' ====\n")
     parts.append(block('text', os.path.basename(path), read(path)))
@@ Line 1573: / Line 1971: @@
 parts.append('====== References ======\n')
 parts.append('<bibtex bibliography></bibtex>\n')
+# --- Is every output actually an output? An empty or truncated file publishes an
+# empty <file> block and every other assertion still passes.
+for _name, _path, _tail in OUTPUTS:
+    _body = read(_path)
+    # 100 bytes is a floor against an emptied file, not a quality bar — one of
+    # these outputs is legitimately six lines long. The terminal-string check
+    # below is what catches truncation.
+    if len(_body.strip()) < 100:
+        sys.exit(f'{_path}: {len(_body)} bytes — that is not an output, it is a stub')
+    if _tail not in _body:
+        sys.exit(f'{_path}: does not contain {_tail!r} — truncated, or the script it '
+                 f'came from changed and this expectation was not updated')
+    if os.path.getmtime(_path) < os.path.getmtime(f'scripts/{_name}'):
+        sys.exit(f'{_path} is older than scripts/{_name} — re-run the script before publishing')
+# --- Does OUTPUTS cover everything on disk? Comparing len(OUTPUTS) against a
+# count derived from OUTPUTS is tautological; compare it against the filesystem.
+_on_disk = set(glob.glob('scripts/policies_*-output.txt')) | {'scripts/report_policies-output.txt'}
+_listed = {o[1] for o in OUTPUTS}
+if _on_disk != _listed:
+    sys.exit('OUTPUTS does not match the outputs on disk.\n'
+             f'  not published: {sorted(_on_disk - _listed)}\n'
+             f'  listed but absent: {sorted(_listed - _on_disk)}')
 page = '\n'.join(parts).rstrip() + '\n'
@@ Line 1595: / Line 2017: @@
 assert not re.search(r'(?<!%%)' + re.escape(DISCUSSION), prose_only), \
     'an unescaped discussion macro would put a comment box on a provenance page'
-assert h2 >= 10, f'only {h2} level-2 headings — a section went missing'
+# `h2 >= 10` was the first form of this check and a mutation test on 2026-09-10
+# showed it asserts almost nothing: deleting the whole second half of the prose
+# still leaves ten headings. Name them instead.
+EXPECTED_H2 = [
+    'The run',
+    'Scope and judgement calls',
+    'Populations, and every query behind a figure',
+    'The probes, at both widths',
+    'Folding, and the complete residue',
+    'Quotes and figures checked against the papers',
+    'Bibliography',
+    'Guards run before publication',
+    'External sources',
+    'What could not be established',
+    'Review',
+    'The scripts, as committed',
+    'The outputs, unedited',
+]
+found = re.findall(r'(?m)^=====\s*(.+?)\s*=====$', page)
+missing_h2 = [h for h in EXPECTED_H2 if h not in found]
+assert not missing_h2, f'sections missing from the page: {missing_h2}'
+assert h2 == len(EXPECTED_H2), f'{h2} level-2 headings, expected {len(EXPECTED_H2)}: {found}'
 with open(OUT, 'w', encoding='utf-8') as f:
@@ Line 1627: / Line 2070: @@
 privacy-policy tuples in POLICY                          179
+--- POLICY-only: the enum fires and the title probe is silent (why a title probe is not enough)
+papers. Platform measured (enum, multi-valued):
+platform              papers of the enum-only set
+--------------------  ---------------------------
+other-online-service  21
+mobile                20
+web                   18
+iot                   4
+offline               1
+NDSS     Automated Analysis of Privacy Requirements for Mobile Apps
+PETS     Analyzing Remote Server Locations for Personal Data Transfers in Mobile Apps
+IMC      Tales from the Porn: A Comprehensive Privacy Analysis of the Web Porn Ecosystem.
+PETS     MAPS: Scaling Privacy Compliance Analysis to a Million Apps
+WWW      Understanding the Evolution of Mobile App Ecosystems: A Longitudinal Measurement Study of Google Play.
+PETS     Angel or Devil? A Privacy Study of Mobile Parental Control Apps
+PETS     CanaryTrap: Detecting Data Misuse by Third-Party Apps on Online Social Networks
+PETS     The Price is (Not) Right: Comparing Privacy in Free and Paid Apps
+USENIX   SkillExplorer: Understanding the Behavior of Skills in Large Scale
+NDSS     Hey Alexa, is this Skill Safe?: Taking a Closer Look at the Alexa Skill Ecosystem
+IEEE-SP  Scraping Sticky Leftovers: App User Information Left on Servers After Account Deletion.
+IMC      Exploring the security and privacy risks of chatbots in messaging services.
+PETS     Checking Websites’ GDPR Consent Compliance for Marketing Emails
+PETS     Developers Say the Darnedest Things: Privacy Compliance Processes Followed by Developers of Child-Directed Apps
+PETS     How Can and Would People Protect From Online Tracking?
+PETS     “We may share the number of diaper changes”: A Privacy and Security Analysis of Mobile Child Care Applications
+USENIX   SkillDetective: Automated Policy-Violation Detection of Voice Assistant Applications in the Wild
+WWW      Et tu, Brute? Privacy Analysis of Government Websites and Mobile Apps.
+WWW      Measuring Alexa Skill Privacy Practices across Three Years.
+CCS      SkillScanner: Detecting Policy-Violating Voice Applications Through Static Analysis at the Development Phase.
+IMC      Tracking, Profiling, and Ad Targeting in the Alexa Echo Smart Speaker Ecosystem.
+NDSS     CHKPLUG: Checking GDPR Compliance of WordPress Plugins via Cross-language Code Property Graph
+PETS     Comparing Large-Scale Privacy and Security Notifications
+USENIX   Are You Spying on Me? Large-Scale Analysis on IoT Data Exposure through Companion Apps
+USENIX   The Digital-Safety Risks of Financial Technologies for Survivors of Intimate Partner Violence
+CCS      A First Look at Security and Privacy Risks in the RapidAPI Ecosystem.
+IEEE-SP  Wear's my Data? Understanding the Cross-Device Runtime Permission Model in Wearables.
+IEEE-SP  Understanding the Privacy Practices of Political Campaigns: A Perspective from the 2020 US Election Websites.
+NDSS     MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots
+PETS     The Medium is the Message: How Secure Messaging Apps Leak Sensitive Data to Push Notification Services
+PETS     Two Steps Forward and One Step Back: The Right to Opt-out of Sale under CPRA
+PETS     Connecting the Dots: Tracing Data Endpoints in IoT Devices
+USENIX   Arcanum: Detecting and Evaluating the Privacy Risks of Browser Extensions on Web Pages and Web Content
+CCS      The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
+IEEE-SP  SoK: A Privacy Framework for Security Research Using Social Media Data.
+IEEE-SP  On the (In)Security of LLM App Stores.
+IMC      An In-Depth Investigation of Data Collection in LLM App Ecosystems.
+PETS     Understanding Privacy Norms through Web Forms
+PETS     The Effect of Platform Policies on App Privacy Compliance: A Study of Child-Directed Apps
+PETS     Privacy Settings of Third-Party Libraries in Android Apps: A Study of Facebook SDKs
+PETS     Who’s Watching You Zoom? Investigating Privacy of Third-Party Zoom Apps
+PETS     Surveillance Disguised as Protection: A Comparative Analysis of Sideloaded and In-Store Parental Control Apps
+USENIX   AUTOVR: Automated UI Exploration for Detecting Sensitive Data Flow Exposures in Virtual Reality Apps
+USENIX   I Can Tell Your Secrets: Inferring Privacy Attributes from Mini-app Interaction History in Super-apps
+PETS     Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law’s Impact
+PETS     Chatbot Confessions:~Large-Scale Analysis of Private Data Disclosure in Shared AI Chatbot Conversations
 --- classification[].target, whole corpus, papers (enum — publishable)
@@ Line 1652: / Line 2151: @@
 --- POLICY and UNION per year (2026 PROVISIONAL: CCS/IMC 2026 not held, IEEE S&P/WWW 2026 under-selected)
-Year   corpus  POLICY  PROBE  UNION  UNION share of corpus
+Year   corpus  POLICY  PROBE  UNION  POLICY share of corpus  UNION share of corpus
------  ------  ------  -----  -----  ---------------------
+-----  ------  ------  -----  -----  ----------------------  ---------------------
-   166     1       1      1      0.6%
+   166     1       1      1      0.6%                    0.6%
-   182     1       2      2      1.1%
+   182     1       2      2      0.5%                    1.1%
-   231     3       1      3      1.3%
+   231     3       1      3      1.3%                    1.3%
-   254     2       2      2      0.8%
+   254     2       2      2      0.8%                    0.8%
-   402     6       3      6      1.5%
+   402     6       3      6      1.5%                    1.5%
-   404     8       5      9      2.2%
+   404     8       5      9      2.0%                    2.2%
-   379     7       8      9      2.4%
+   379     7       8      9      1.8%                    2.4%
-   546     17      10     19     3.5%
+   546     17      10     19     3.1%                    3.5%
-   719     14      10     16     2.2%
+   719     14      10     16     1.9%                    2.2%
-   690     19      16     24     3.5%
+   690     19      16     24     2.8%                    3.5%
-*  770     16      9      20     2.6%
+*  770     16      9      20     2.1%                    2.6%
-*  415     8       10     12     2.9%
+*  415     8       10     12     1.9%                    2.9%
 --- UNION per venue (denominator: that venue's whole corpus slice)
-Venue    UNION  POLICY  venue papers  UNION share of venue
+Venue    UNION  POLICY  venue papers  POLICY share of venue  UNION share of venue
--------  -----  ------  ------------  --------------------
+-------  -----  ------  ------------  ---------------------  --------------------
-PETS     54     43      510           10.6%
+PETS     54     43      510           8.4%                   10.6%
-USENIX   27     25      1410          1.9%
+USENIX   27     25      1410          1.8%                   1.9%
-CCS      12     8       990           1.2%
+CCS      12     8       990           0.8%                   1.2%
-NDSS     9      8       701           1.3%
+NDSS     9      8       701           1.1%                   1.3%
-WWW      9      7       843           1.1%
+WWW      9      7       843           0.8%                   1.1%
-IEEE-SP  7      6       767           0.9%
+IEEE-SP  7      6       767           0.8%                   0.9%
-IMC      5      5       638           0.8%
+IMC      5      5       638           0.8%                   0.8%
+top venue over the next, POLICY share: PETS / USENIX = 4.8x
+top venue over the next, UNION share : PETS / USENIX = 5.5x
 --- UNION by platform measured (multi-valued: an app+web paper is in two rows)
@@ Line 2051: / Line 2552: @@
 yan2024_quality      PETS 2024    Alexa skills                             65,195 skills                                                                                                                                21,063 of 65,195 provided a policy link
 zhan2024_vpvet       CCS 2024     VR apps                                  11,923 apps on 10 VR platforms                                                                                                               29.5% had a findable privacy policy
-edu2022_exploring    IMC 2022     Discord chatbots requesting permissions  15,528 chatbots                                                                                                                              676 (4.35%) had a policy; 14,852 (95.67%) did not
+edu2022_exploring    IMC 2022     Discord chatbots requesting permissions  15,525 unique active chatbots (the paper's own Table 2 total); its 14,852-without figure does not reconcile with it                          676 (4.35%) had a policy; 14,852 (95.67%) did not
 wu2025_depth         IMC 2025     GPT Actions                              Actions declaring a legal_info_url                                                                                                           93.96% of those policies were reachable
@@ Line 2336: / Line 2837: @@
 PETS     Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale
 PETS     From Lines of Code to Lines of Policy? Exploring Software Developers’ Perceptions of Their P
+--- LINEAGE ARTEFACTS: extraction fold vs FULL-TEXT mention, whole corpus
+Full-text denominator: 5869 papers with a readable paper.cols.txt.
+"extraction" is the column report_policies.mjs section 4 prints.
+artefact                                full-text pattern   papers whose FULL TEXT names it
+--------------------------------------  ------------------  -------------------------------
+Privee (USENIX 2014)                    /\bPrivee\b/        25
+OPP-115 corpus (ACL 2016)               /OPP-?115/i         30
+Polisis / PriBot (USENIX 2018)          /polisis|pribot/i   68
+PolicyLint (USENIX 2019)                /policylint/i       63
+MAPS (PETS 2019)                        /MAPS:\s*Scaling/i  50
+APP-350 corpus (2019)                   /APP-?350/i         9
+PoliCheck (USENIX 2020)                 /policheck/i        63
+PurPliance (2021)                       /purpliance/i       14
+PrivBERT (2021)                         /privbert/i         3
+Calpric (USENIX 2023)                   /calpric/i          3
+PoliGraph / PoliGraph-er (USENIX 2023)  /poligraph/i        14
+PolicyChecker (CCS 2023)                /policychecker/i    9
+Lalaine (USENIX 2023)                   /lalaine/i          28
+PolicyComp (USENIX 2023)                /policycomp/i       10
+PrivaSeer                               /privaseer/i        12
+Full-text patterns that differ from the extraction pattern, and why:
+  MAPS (PETS 2019): /MAPS:\s*Scaling/i — /\bMAPS\b/ also matches "Google MAPS" and the Play category "MAPS & NAVIGATION"; the paper is always cited by its title
+The two artefacts where the gap changes what may be said:
+PrivaSeer — 12 papers:
+PETS     automated-extraction-and-presentation-of-data-practices-in-privacy-policies
+WWW      privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset
+PETS     setting-the-bar-low-are-websites-complying-with-the-minimum-requirements-of-the
+PETS     researchers-experiences-in-analyzing-privacy-policies-challenges-and-opportuniti
+CCS      vpvet-vetting-privacy-policies-of-virtual-reality-apps
+PETS     connecting-the-dots-tracing-data-endpoints-in-iot-devices
+PETS     honesty-is-the-best-policy-on-the-accuracy-of-apple-privacy-labels-compared-to-a
+PETS     on-the-quality-of-privacy-policy-documents-of-virtual-personal-assistant-applica
+CCS      layered-overlapping-and-inconsistent-a-large-scale-analysis-of-the-multiple-priv
+CCS      the-odyssey-of-robots-txt-governance-measuring-convention-implications-of-web-bo
+USENIX   evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto
+PETS     word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using
+Calpric (USENIX 2023) — 3 papers:
+USENIX   calpric-inclusive-and-fine-grain-labeling-of-privacy-policies-with-crowdsourcing
+PETS     understanding-privacy-norms-through-web-forms
+USENIX   evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto
 </file>
@@ Line 2358: / Line 2905: @@
 "rest of corpus" removes the subgroup's own papers from the base; the
 bracketed column is the naive base rate that leaves them in.
+</file>
+==== Output of ''policies_table_check.mjs'' ====
+<file text policies_table_check-output.txt>
+OK        method-by-era table: 11 rows
+OK        per-year table: 12 rows
+OK        per-venue table: 7 rows
+OK        significance table: 5 rows
+All four tables match the report cell by cell.
 </file>
@@ Line 2431: / Line 2989: @@
 OK  cols                     PETS 2019  MAPS analysed 1,035,853 of 1,049,790 retrieved
 OK  cols                     PETS 2026  cory2026 is word-level GDPR transparency annotation by LLM
+OK  cols                     IMC 2022  Discord: the paper states 15,525 unique active chatbots
+OK  cols                     IMC 2022  Discord: and separately 14,852 (95.67%) without a policy
+SPECIFICITY of the 70 needles
+  needles with no letters (pure number/punctuation): 30
+  of those, also present in another check paper     : 4
+  needles under 12 characters                       : 22
+  WEAK — numeric-only and not unique to the paper they are attributed to:
+other check papers also contain "1,071,488"  (Princeton corpus 1,071,488 policies / 130,000 sites)
+other check papers also contain "84.7"  (84.7 minutes to read applicable policies)
+other check papers also contain "54.5%"  (VR policy reuse 54.5% (1,919/3,521))
+other check papers also contain "1,035,853"  (MAPS analysed 1,035,853 of 1,049,790 retrieved)
+  These are not wrong — each was read in context — but they are the needles
+  a future edit could break without this check noticing.
- needles, 68 located, 0 MISSING.
+ needles, 70 located, 0 MISSING.
  located in paper.cols.txt; 1 located only outside paper.cols.txt.
 </file>