Differences

This shows you the differences between two versions of the page.

--- provenance:privacy:policies [2026/09/10 18:06] – add pass-4 generic review log and the mutation-test table; document the whole-corpus lineage scan, needle specificity and the strengthened generator guards (Authored by Claude) karel.kubicek.claude
+++ provenance:privacy:policies [2026/09/10 18:09] (current) – publish policies_table_check.mjs and its output; record the guard-floor fix (Authored by Claude) karel.kubicek.claude
@@ Line 14: / Line 14: @@
 | Page id | ''privacy:policies'', **not** ''privacy:privacy_policies''. Decided 2026-09-07 on [[:roadmap]] (see [[:provenance:roadmap]] §3) because ''privacy_policies'' sorts next to the site's own ''privacy_policy'' page in search. ''scripts/sitemap.mjs'' already gated on ''privacy:policies'' and the roadmap's Queued table already promised it, so any other id would have left a dangling promise |
 | Models | Page, scripts and this log: Claude (Opus 5, with the 2026-09-09 draft written by the same model in an earlier session). Review layer: three ''sonnet'' focused passes (figures-vs-script, citations-and-quotes, external currency) and one ''fable'' generic pass. Findings and verdicts below |
-| Scripts added | ''scripts/report_policies.mjs'' (+ ''-output.txt''), ''scripts/policy_fold.mjs'', ''scripts/policies_fulltext_probe.mjs'' (+ ''-output.txt''), ''scripts/policies_quotecheck.mjs'' (+ ''-output.txt''), ''scripts/policies_significance.py'' (+ ''-output.txt''), ''scripts/policies_external_checks.sh'' (+ ''-output.txt''), ''scripts/policies_gh_search.py'', ''scripts/policies_w3c_p3p_check.mjs'' (+ ''-output.txt''), ''scripts/policies_table_check.mjs'', ''scripts/policies_fetch_pets_authors.py'', ''scripts/bib_additions_policies.bib'', ''scripts/build_provenance_policies.py'' |
+| Scripts added | ''scripts/report_policies.mjs'' (+ ''-output.txt''), ''scripts/policy_fold.mjs'', ''scripts/policies_fulltext_probe.mjs'' (+ ''-output.txt''), ''scripts/policies_quotecheck.mjs'' (+ ''-output.txt''), ''scripts/policies_significance.py'' (+ ''-output.txt''), ''scripts/policies_external_checks.sh'' (+ ''-output.txt''), ''scripts/policies_gh_search.py'', ''scripts/policies_w3c_p3p_check.mjs'' (+ ''-output.txt''), ''scripts/policies_table_check.mjs'' (+ ''-output.txt''), ''scripts/policies_fetch_pets_authors.py'', ''scripts/bib_additions_policies.bib'', ''scripts/build_provenance_policies.py'' |
 | Write path | ''node scripts/dw.mjs put'' (JSON-RPC) with ''--if-rev'' on every save |
 | Accidental exposure | None. Credentials stayed in ''.env'' and were never echoed. All external fetches were unauthenticated: GitHub's public API, PyPI, Hugging Face, usableprivacy.org, privaseer.ist.psu.edu, developer.apple.com, support.google.com, w3.org, secartifacts.github.io |
@@ Line 1378: / Line 1378: @@
 print('\n"rest of corpus" removes the subgroup\'s own papers from the base; the')
 print('bracketed column is the naive base rate that leaves them in.')
+</file>
+==== The cell-by-cell table check — ''policies_table_check.mjs'' ====
+<file javascript policies_table_check.mjs>
+// Assert that privacy:policies' four corpus tables match report_policies.mjs
+// CELL BY CELL, not merely that each numeral occurs somewhere in the output.
+//
+//   node scripts/policies_table_check.mjs
+//
+// WHY this exists in addition to check_page_numbers.mjs: that guard tests
+// whether every numeral on the page appears anywhere in the concatenated script
+// output. A mutation test on 2026-09-10 changed the `llm` row's count from 12 to
+// 77 and the guard still passed, because 77 occurs elsewhere in the output (the
+// PROBE size, and the public-artifacts count). A wrong figure that collides with
+// a live one is invisible to a membership test. This one re-derives each row
+// from the report and compares positionally.
+//
+// Exits non-zero on the first mismatch, printing the row it disagrees with.
+import fs from 'node:fs';
+const PAGE = process.argv[2] ?? 'pages/privacy_policies.txt';
+const REPORT = process.argv[3] ?? 'scripts/report_policies-output.txt';
+const SIG = process.argv[4] ?? 'scripts/policies_significance-output.txt';
+const page = fs.readFileSync(PAGE, 'utf8');
+const report = fs.readFileSync(REPORT, 'utf8');
+const sig = fs.readFileSync(SIG, 'utf8');
+let failures = 0;
+const fail = (what, expected, got) => {
+  failures += 1;
+  console.log(`MISMATCH  ${what}\n  report: ${expected}\n  page  : ${got}`);
+};
+const ok = (what) => console.log(`OK        ${what}`);
+// A page table row -> array of trimmed cells, markup stripped.
+const cells = (line) =>
+  line.replace(/^\||\|$/g, '').split('|')
+    .map((c) => c.replace(/\*\*|''|\/\//g, '').trim());
+const pageRows = (headerMatch) => {
+  const lines = page.split('\n');
+  const i = lines.findIndex((l) => l.startsWith('^') && headerMatch.test(l));
+  if (i < 0) throw new Error(`no page table whose header matches ${headerMatch}`);
+  const out = [];
+  for (let j = i + 1; j < lines.length && lines[j].startsWith('|'); j += 1) out.push(cells(lines[j]));
+  if (!out.length) throw new Error(`page table ${headerMatch} has no rows`);
+  return out;
+};
+const reportSection = (marker) => {
+  const i = report.indexOf(marker);
+  if (i < 0) throw new Error(`report has no section ${JSON.stringify(marker)}`);
+  const rest = report.slice(i + marker.length);
+  // A section ends at the next "--- " subsection header or "===" banner. The
+  // column-rule line under each table is also dashes, so match the space.
+  const end = rest.search(/\n(--- |=====)/);
+  return (end < 0 ? rest : rest.slice(0, end)).split('\n').filter((l) => l.trim());
+};
+// ---------------------------------------------------------------- 1. by era
+{
+  const want = new Map();
+  for (const l of reportSection('--- method by era')) {
+    const m = l.match(/^(\S+)\s+(\d+)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)\s+\d+ \(\s*([\d.]+)%\)/);
+    if (m) want.set(m[1], m.slice(2));
+  }
+  if (want.size < 5) throw new Error('parsed too few method-by-era rows from the report');
+  const rows = pageRows(/classification\.method/);
+  if (rows.length !== want.size) fail('method-by-era row count', want.size, rows.length);
+  for (const r of rows) {
+    const [name, all, ...eras] = r;
+    if (!want.has(name)) { fail(`method row "${name}"`, '(no such method in the report)', r.join(' | ')); continue; }
+    const w = want.get(name);
+    const got = [all, ...eras.map((e) => e.replace('%', ''))];
+    const exp = [w[0], ...w.slice(1).map((x) => String(parseFloat(x)))];
+    const norm = got.map((x) => String(parseFloat(x)));
+    if (norm.join(',') !== exp.join(',')) fail(`method row "${name}"`, exp.join(' | '), norm.join(' | '));
+  }
+  ok(`method-by-era table: ${rows.length} rows`);
+}
+// ---------------------------------------------------------------- 2. by year
+{
+  const want = new Map();
+  for (const l of reportSection('--- POLICY and UNION per year')) {
+    const m = l.match(/^(\d{4})\*?\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+([\d.]+)%/);
+    if (m) want.set(m[1], [m[2], m[3], m[5], m[6]]); // corpus, POLICY, UNION, share
+  }
+  const rows = pageRows(/\^ Year \^/);
+  for (const r of rows) {
+    const year = r[0].replace('*', '');
+    if (!want.has(year)) { fail(`year row ${year}`, '(not in the report)', r.join(' | ')); continue; }
+    const w = want.get(year);
+    const got = [r[1].replace(/,/g, ''), r[2], r[3], r[4].replace('%', '')];
+    const exp = [w[0], w[1], w[2], String(parseFloat(w[3]))];
+    if (got.map((x) => String(parseFloat(x))).join(',') !== exp.join(',')) {
+      fail(`year row ${year}`, exp.join(' | '), got.join(' | '));
+    }
+  }
+  if (rows.length !== want.size) fail('per-year row count', want.size, rows.length);
+  ok(`per-year table: ${rows.length} rows`);
+}
+// --------------------------------------------------------------- 3. by venue
+{
+  const want = new Map();
+  for (const l of reportSection('--- UNION per venue')) {
+    const m = l.match(/^(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s+([\d.]+)%/);
+    if (m) want.set(m[1], [m[2], m[3], m[4], m[5]]);
+  }
+  const alias = { 'USENIX Sec': 'USENIX', TheWebConf: 'WWW', 'IEEE S&P': 'IEEE-SP' };
+  const rows = pageRows(/\^ Venue \^/);
+  for (const r of rows) {
+    const v = alias[r[0]] ?? r[0];
+    if (!want.has(v)) { fail(`venue row ${r[0]}`, '(not in the report)', r.join(' | ')); continue; }
+    const w = want.get(v);
+    const got = [r[1], r[2], r[3].replace(/,/g, ''), r[4].replace('%', '')];
+    const exp = [w[0], w[1], w[2], String(parseFloat(w[3]))];
+    if (got.map((x) => String(parseFloat(x))).join(',') !== exp.join(',')) {
+      fail(`venue row ${r[0]}`, exp.join(' | '), got.join(' | '));
+    }
+  }
+  if (rows.length !== want.size) fail('per-venue row count', want.size, rows.length);
+  ok(`per-venue table: ${rows.length} rows`);
+}
+// ---------------------------------------------------------- 4. Fisher's exact
+{
+  // Only the POLICY rows are published; the UNION rows stay on the provenance page.
+  const want = [];
+  for (const l of sig.split('\n')) {
+    const m = l.match(/^(.+?)\s+POLICY\s+(\d+)\/(\d+) \(\s*[\d.]+%\)\s+(\d+)\/(\d+) \(\s*[\d.]+%\)\s+(\S+)/);
+    if (m) want.push({ sub: `${m[2]}/${m[3]}`, base: `${m[4]}/${m[5]}`, p: m[6] });
+  }
+  if (want.length !== 5) throw new Error(`expected 5 POLICY significance rows, parsed ${want.length}`);
+  const rows = pageRows(/\^ Property \^/);
+  if (rows.length !== want.length) fail('significance row count', want.length, rows.length);
+  // The page orders rows for reading; match on the subgroup fraction, not position.
+  const bySub = new Map(want.map((w) => [w.sub, w]));
+  for (const r of rows) {
+    const sub = r[1].replace(/,/g, '').split(' ')[0];
+    if (!bySub.has(sub)) { fail(`significance row "${r[0]}"`, '(no POLICY row with that numerator/denominator)', r[1]); continue; }
+    const w = bySub.get(sub);
+    const base = r[2].replace(/,/g, '').split(' ')[0];
+    if (base !== w.base) fail(`significance base for "${r[0]}"`, w.base, base);
+    // p is printed on the page in scientific form with superscript digits
+    // ("2.65 x 10^-8"); the script prints "2.65e-08". Normalise BOTH to a number
+    // and compare the value, exponent included. An earlier form of this check
+    // compared only the first three digits and a mutation test on 2026-09-10
+    // showed it could not see an exponent moved from -8 to -5.
+    const SUP = { '⁰': '0', '¹': '1', '²': '2', '³': '3', '⁴': '4', '⁵': '5', '⁶': '6', '⁷': '7', '⁸': '8', '⁹': '9', '⁻': '-' };
+    const pageP = (() => {
+      const s = r[3].split('—')[0].trim();
+      const sci = s.match(/^([\d.]+)\s*×\s*10([⁻⁰¹²³⁴⁵⁶⁷⁸⁹]+)$/);
+      if (sci) {
+        const exp = [...sci[2]].map((c) => SUP[c] ?? c).join('');
+        return Number(sci[1]) * 10 ** Number(exp);
+      }
+      return Number(s);
+    })();
+    const repP = Number(w.p);
+    if (!Number.isFinite(pageP)) {
+      fail(`significance p for "${r[0]}" is unparseable`, w.p, r[3]);
+    } else if (Math.abs(pageP - repP) > Math.abs(repP) * 0.02) {
+      fail(`significance p for "${r[0]}"`, `${w.p} (${repP})`, `${r[3]} (${pageP})`);
+    }
+  }
+  ok(`significance table: ${rows.length} rows`);
+}
+console.log(failures ? `\n${failures} MISMATCHES` : '\nAll four tables match the report cell by cell.');
+process.exitCode = failures ? 1 : 0;
 </file>
@@ Line 1731: / Line 1905: @@
     ('The quote check', 'scripts/policies_quotecheck.mjs', 'javascript'),
     ('The significance test', 'scripts/policies_significance.py', 'python'),
+    ('The cell-by-cell table check', 'scripts/policies_table_check.mjs', 'javascript'),
     ('The external checks', 'scripts/policies_external_checks.sh', 'bash'),
     ('The GitHub name-search check', 'scripts/policies_gh_search.py', 'python'),
@@ Line 1750: / Line 1925: @@
     ('policies_significance.py', 'scripts/policies_significance-output.txt',
      'bracketed column is the naive base rate that leaves them in.'),
+    ('policies_table_check.mjs', 'scripts/policies_table_check-output.txt',
+     'All four tables match the report cell by cell.'),
     ('policies_quotecheck.mjs', 'scripts/policies_quotecheck-output.txt',
      'located only outside paper.cols.txt.'),
@@ Line 1799: / Line 1976: @@
 for _name, _path, _tail in OUTPUTS:
     _body = read(_path)
-    if len(_body.strip()) < 200:
+    # 100 bytes is a floor against an emptied file, not a quality bar — one of
+    # these outputs is legitimately six lines long. The terminal-string check
+    # below is what catches truncation.
+    if len(_body.strip()) < 100:
         sys.exit(f'{_path}: {len(_body)} bytes — that is not an output, it is a stub')
     if _tail not in _body:
@@ Line 2725: / Line 2905: @@
 "rest of corpus" removes the subgroup's own papers from the base; the
 bracketed column is the naive base rate that leaves them in.
+</file>
+==== Output of ''policies_table_check.mjs'' ====
+<file text policies_table_check-output.txt>
+OK        method-by-era table: 11 rows
+OK        per-year table: 12 rows
+OK        per-venue table: 7 rows
+OK        significance table: 5 rows
+All four tables match the report cell by cell.
 </file>