| Next revision | Previous revision |
| provenance:statistics:study_preregistration [2026/08/13 03:04] – New provenance page for statistics:study_preregistration: every query with its denominator, the unedited report and external-check output, the hand classifications and their residue, bugs found in shared scripts, and the four review passes. Authored by Cl karel.kubicek.claude | provenance:statistics:study_preregistration [2026/09/17 08:14] (current) – Qualify root start links; Authored by Claude karel.kubicek.claude |
|---|
| | External-fact re-check | ''scripts/external_checks_preregistration.sh'' | | | External-fact re-check | ''scripts/external_checks_preregistration.sh'' | |
| | Runnable code published on the page | ''pages/check_prereg_link.py'' | | | Runnable code published on the page | ''pages/check_prereg_link.py'' | |
| | Stale-number guard | ''scripts/check_page_numbers.mjs'', run whole-page, windowed and with ''--code'' | | | Stale-number guard | ''scripts/check_page_numbers.mjs'', run whole-page, windowed and with ''%%--code%%'' | |
| | Data | ''data/extract/run1/extractions.jsonl'' — 5,859 papers, 7 venues, 2010–2026 — plus a regex sweep over all 5,869 ''paper.cols.txt'' full texts | | | Data | ''data/extract/run1/extractions.jsonl'' — 5,859 papers, 7 venues, 2010–2026 — plus a regex sweep over all 5,869 ''paper.cols.txt'' full texts | |
| | Written | 2026-08-13, against the corpus as extended on 2026-08-11 (commit ''8a6b843'') | | | Written | 2026-08-13, against the corpus as extended on 2026-08-11 (commit ''8a6b843'') | |
| |
| **Creating, not extending.** The wiki had no ''statistics:'' page at all: ''statistics:study_preregistration'', ''statistics:hypothesis_testing'', ''statistics:pvalue_corrections'', ''statistics:regression'' and ''statistics:biases'' were all red links promised from [[start]]. This is the first of the five. Nothing was overwritten and nothing on a neighbouring page was contradicted; the page links to the other four as red links, which the site already does elsewhere. | **Creating, not extending.** The wiki had no ''statistics:'' page at all: ''statistics:study_preregistration'', ''statistics:hypothesis_testing'', ''statistics:pvalue_corrections'', ''statistics:regression'' and ''statistics:biases'' were all red links promised from [[:start]]. This is the first of the five. Nothing was overwritten and nothing on a neighbouring page was contradicted; the page links to the other four as red links, which the site already does elsewhere. |
| |
| **Reachability** needed no work: [[start]] already links ''[[Statistics:Study preregistration]]'' in its outline. | **Reachability** needed no work: [[:start]] already links ''[[Statistics:Study preregistration]]'' in its outline. |
| |
| ===== 2. The number the task started from, and what happened to it ===== | ===== 2. The number the task started from, and what happened to it ===== |
| ===== 11. Judgement calls ===== | ===== 11. Judgement calls ===== |
| |
| - **New page, not a section of an existing one.** The alternative was a section of a broader ''statistics:'' page. Rejected because [[start]] already promises this exact page id, and because the measurement-specific content — the OSF template mapping, the registry-versus-project distinction, the registered-reports venue table — does not fit under a heading in a general statistics page without swamping it. | - **New page, not a section of an existing one.** The alternative was a section of a broader ''statistics:'' page. Rejected because [[:start]] already promises this exact page id, and because the measurement-specific content — the OSF template mapping, the registry-versus-project distinction, the registered-reports venue table — does not fit under a heading in a general statistics page without swamping it. |
| - **The population is 15, not 14 and not 7.** The schema union is 14 and ''statistics.kind'' alone is 7. The published population is the full-text sweep's ''study'' sense, which is 15, because the schema demonstrably misses 9 of them. This means **the page's headline number cannot be reproduced from the extraction alone** — it needs the full text — and §4 says so. | - **The population is 15, not 14 and not 7.** The schema union is 14 and ''statistics.kind'' alone is 7. The published population is the full-text sweep's ''study'' sense, which is 15, because the schema demonstrably misses 9 of them. This means **the page's headline number cannot be reproduced from the extraction alone** — it needs the full text — and §4 says so. |
| - **The three ''analysis-plan-only'' papers are excluded from the 15.** All three pre-specified an analysis plan and registered nothing. Including them would make the count 18 and the page's central distinction — that a commitment nobody else can check is not a preregistration — incoherent. A reasonable person could include them with a footnote. They are not dropped: they have their own section on the content page, because a reader who already writes an analysis plan is the reader most likely to act on this page. | - **The three ''analysis-plan-only'' papers are excluded from the 15.** All three pre-specified an analysis plan and registered nothing. Including them would make the count 18 and the page's central distinction — that a commitment nobody else can check is not a preregistration — incoherent. A reasonable person could include them with a footnote. They are not dropped: they have their own section on the content page, because a reader who already writes an analysis plan is the reader most likely to act on this page. |
| ===== 12. Guard runs ===== | ===== 12. Guard runs ===== |
| |
| ''check_page_numbers.mjs'' was run whole-page, windowed on ''Use in Publications'', and with ''--code''. | ''check_page_numbers.mjs'' was run whole-page, windowed on ''Use in Publications'', and with ''%%--code%%''. |
| |
| ^ Run ^ Unaccounted ^ What it found ^ | ^ Run ^ Unaccounted ^ What it found ^ |
| | 1 | 31 | Mostly external figures the report never printed. It also surfaced two real gaps: the collapsed 2010–2020 row (2,340 / 462) and the 67.7% descriptive-only figure were **the page's own arithmetic and ''OVERVIEW.md'''s number**, not the report's. Both are now computed by the report script | | | 1 | 31 | Mostly external figures the report never printed. It also surfaced two real gaps: the collapsed 2010–2020 row (2,340 / 462) and the 67.7% descriptive-only figure were **the page's own arithmetic and ''OVERVIEW.md'''s number**, not the report's. Both are now computed by the report script | |
| | 2 | 4 | The 2.9%-versus-1.9% sentence — the page's own arithmetic on the wrong-numerator bug — plus digit fragments of the AsPredicted registration numbers and the ''timeout=30'' inside the embedded Python. An ''F. EXTERNAL FIGURES'' section was added to the report, listing each non-corpus number with its primary source, so they are accounted for without polluting the guard's shared ''ALLOW'' map | | | 2 | 4 | The 2.9%-versus-1.9% sentence — the page's own arithmetic on the wrong-numerator bug — plus digit fragments of the AsPredicted registration numbers and the ''timeout=30'' inside the embedded Python. An ''F. EXTERNAL FIGURES'' section was added to the report, listing each non-corpus number with its primary source, so they are accounted for without polluting the guard's shared ''ALLOW'' map | |
| | 3 | 0 (whole-page, ''--code'', and windowed) | Clean | | | 3 | 0 (whole-page, ''%%--code%%'', and windowed) | Clean | |
| |
| The embedded ''%%<file python check_prereg_link.py>%%'' block was compared line-by-line against ''pages/check_prereg_link.py'': identical. The ''%%<code>%%'' block is the verbatim output of a real run, including the ''$ echo $?'' showing exit status 1. | The embedded ''%%<file python check_prereg_link.py>%%'' block was compared line-by-line against ''pages/check_prereg_link.py'': identical. The ''%%<code>%%'' block is the verbatim output of a real run, including the ''$ echo $?'' showing exit status 1. |
| | 3 | The footnote said ''aspredicted.org'' "returns HTTP 403 to some automated clients". A plain ''curl'' gets **406**. | **Accepted.** Measured: plain ''curl'' 406, browser User-Agent 200. Footnote and script comment corrected. | | | 3 | The footnote said ''aspredicted.org'' "returns HTTP 403 to some automated clients". A plain ''curl'' gets **406**. | **Accepted.** Measured: plain ''curl'' 406, browser User-Agent 200. Footnote and script comment corrected. | |
| | 4 | "11 papers, 8 distinct public registrations verified live" conflates the private one into the public count: 7 are public, 1 is private (HTTP 401), 2 are projects. | **Accepted.** The mechanics table now spells out 7 public + 1 private + 2 projects, and notes that the 11th paper writes "on OSF" with the URL only in a PDF hyperlink. | | | 4 | "11 papers, 8 distinct public registrations verified live" conflates the private one into the public count: 7 are public, 1 is private (HTTP 401), 2 are projects. | **Accepted.** The mechanics table now spells out 7 public + 1 private + 2 projects, and notes that the 11th paper writes "on OSF" with the URL only in a PDF hyperlink. | |
| | 5 | Every sibling ''statistics:'' link, plus ''design:user_studies'', ''design:automated_measurements'' and ''artifacts'', is a red link today. | **Accepted as a fact, rejected as a change.** All of them are already promised from [[start]], and this site links promised-but-missing pages as a matter of course — ''privacy:requests'' ships red links to ''programming:crawler:pagegraph'' and ''programming:crawler:tracker_radar_collector''. Removing them would make this page //less// consistent with the site, not more. It is recorded here so the next run does not "fix" it. | | | 5 | Every sibling ''statistics:'' link, plus ''design:user_studies'', ''design:automated_measurements'' and ''artifacts'', is a red link today. | **Accepted as a fact, rejected as a change.** All of them are already promised from [[:start]], and this site links promised-but-missing pages as a matter of course — ''privacy:requests'' ships red links to ''programming:crawler:pagegraph'' and ''programming:crawler:tracker_radar_collector''. Removing them would make this page //less// consistent with the site, not more. It is recorded here so the next run does not "fix" it. | |
| | — | Could not verify: the two Klemmer et al. quotes (CISPA's figshare and the S3 mirror both refuse non-JS fetches) and IMC 2026's "mandatory" ethics appendix (the CfP defers to a separate instructions page). It said so rather than guessing, and it explicitly reported pulling the **wrong** Klemmer paper first and confirming that it was not the source. | **Noted, and both gaps were closed elsewhere**: the Klemmer quotes were verified verbatim against ''paper.cols.txt'' by pass 13.2 and by the author. The IMC "mandatory ethics appendix" was then confirmed verbatim on IMC 2026's own submission-instructions page — //"All papers must include, in a clearly marked appendix section with the heading 'Ethics', a statement about ethical issues; papers that do not include such a statement may be rejected"// — and the artifact-availability declaration on the CfP itself. | | | — | Could not verify: the two Klemmer et al. quotes (CISPA's figshare and the S3 mirror both refuse non-JS fetches) and IMC 2026's "mandatory" ethics appendix (the CfP defers to a separate instructions page). It said so rather than guessing, and it explicitly reported pulling the **wrong** Klemmer paper first and confirming that it was not the source. | **Noted, and both gaps were closed elsewhere**: the Klemmer quotes were verified verbatim against ''paper.cols.txt'' by pass 13.2 and by the author. The IMC "mandatory ethics appendix" was then confirmed verbatim on IMC 2026's own submission-instructions page — //"All papers must include, in a clearly marked appendix section with the heading 'Ethics', a statement about ethical issues; papers that do not include such a statement may be rejected"// — and the artifact-availability declaration on the CfP itself. | |
| |
| | Date | 2026-08-13 | | | Date | 2026-08-13 | |
| | Corpus at the time | 5,859 extracted papers and 5,869 full texts, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026; ''data/extract/run1'' | | | Corpus at the time | 5,859 extracted papers and 5,869 full texts, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026; ''data/extract/run1'' | |
| | Page before | did not exist — a red link from [[start]] | | | Page before | did not exist — a red link from [[:start]] | |
| | Scripts written | ''prereg_fold.mjs'', ''report_preregistration.mjs'', ''external_checks_preregistration.sh'', ''pages/check_prereg_link.py'' | | | Scripts written | ''prereg_fold.mjs'', ''report_preregistration.mjs'', ''external_checks_preregistration.sh'', ''pages/check_prereg_link.py'' | |
| | Shared scripts fixed | ''fetch_authors.py'' (two author-dropping bugs), ''bibgen.mjs'' (PoPETs DOI prefix) — see §8 | | | Shared scripts fixed | ''fetch_authors.py'' (two author-dropping bugs), ''bibgen.mjs'' (PoPETs DOI prefix) — see §8 | |
| |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Link-hygiene sweep, 2026-09-17 ===== |
| | |
| | The fresh rendered-DOM sweep found five bare ''start'' links resolving to ''provenance:statistics:start''. They are now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed. |
| |
