provenance:statistics:study_preregistration
Table of Contents
Provenance: statistics:study_preregistration
Working notes behind study_preregistration — every query with its population and denominator, the scripts and their unedited output, the hand classifications and their residue, the quotes checked against source papers, the external sources and how each was verified, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.
1. What this page is backing
| Item | Value |
|---|---|
| Content page | study_preregistration — new, not an extension of anything |
| Report script | scripts/report_preregistration.mjs |
| Hand classification it depends on | scripts/prereg_fold.mjs — a sense map over 62 papers and an adherence map over 15 |
| External-fact re-check | scripts/external_checks_preregistration.sh |
| Runnable code published on the page | pages/check_prereg_link.py |
| Stale-number guard | scripts/check_page_numbers.mjs, run whole-page, windowed and with –code |
| Data | data/extract/run1/extractions.jsonl — 5,859 papers, 7 venues, 2010–2026 — plus a regex sweep over all 5,869 paper.cols.txt full texts |
| Written | 2026-08-13, against the corpus as extended on 2026-08-11 (commit 8a6b843) |
Creating, not extending. The wiki had no statistics: page at all: statistics:study_preregistration, statistics:hypothesis_testing, statistics:pvalue_corrections, statistics:regression and statistics:biases were all red links promised from start. This is the first of the five. Nothing was overwritten and nothing on a neighbouring page was contradicted; the page links to the other four as red links, which the site already does elsewhere.
Reachability needed no work: start already links Study preregistration in its outline.
2. The number the task started from, and what happened to it
The task specification quoted the dataset's most striking single figure: “of 1,302 papers running statistical inference, preregistration appears in 2”. Both halves moved with the 2026-08-11 corpus extension, and the direction is worth recording because it is the opposite of what a stale figure usually does.
| Old corpus (4,322 papers) | This corpus (5,859 papers) | |
|---|---|---|
inferential population | 1,302 | 1,762 |
statistics[].kind == “preregistration” | 2 | 7 |
artifacts.links[].kind == “preregistration” | not stated in the task | 12 |
| Either schema signal | — | 14 |
| Papers that actually preregistered (full text, hand-classified) | — | 15 |
The headline is therefore not “2 papers in the whole corpus” and the page does not say so. It is 15, of which 14 are user studies, and the interesting cut is not the raw count but which population it is a share of.
3. Populations and denominators
Four populations are used, all defined as OVERVIEW.md defines them. crawled and empirical come from lib.mjs's POPULATIONS; inferential and humanSubjects are spelled out in the report script because lib.mjs does not carry them.
| Tag | Definition | N |
|---|---|---|
empirical | isEmpirical === true | 5,118 |
crawled | crawlConfig !== null || studyTypes.includes('automated-web-crawl') | 1,120 |
inferential | at least one statistics[].kind that is not descriptive-only | 1,762 |
humanSubjects | participants.length > 0 | 1,357 |
Every table on the content page names which of these it is a share of. The wrong-numerator trap that this page nearly published is recorded in §8.
4. Every query, with its population
The report script prints all of these. Run it with:
cd /workspace/artifacts/wiki node scripts/report_preregistration.mjs # every figure on the page node scripts/report_preregistration.mjs --list --quotes # the 62 classified papers, and the evidence quotes node scripts/report_preregistration.mjs --wiki # DokuWiki tables bash scripts/external_checks_preregistration.sh # re-fetch every external fact node scripts/check_page_numbers.mjs pages/statistics_study_preregistration.txt out/prereg-report.txt --code
| # | Question | Population | Result |
|---|---|---|---|
| Q1 | How many papers carry statistics[].kind == “preregistration”? | all 5,859 | 7 (0.4% of inferential) |
| Q2 | How many carry an artifacts.links[].kind == “preregistration”? | all 5,859 | 12 (0.2%) |
| Q3 | Union of Q1 and Q2 | all 5,859 | 14; intersection 5 |
| Q4 | How many full texts match /pre\s*-?\s*regist(er|ered|ering|ration|rations)/i after joining hyphens across line breaks and collapsing whitespace? | 5,869 paper.cols.txt files | 62 (1.1%) |
| Q5 | …of which, how many preregistered their own study? | the 62 | 15 (hand-classified, §5) |
| Q6 | How many of the 15 recruited participants? Ran a crawl? | the 15 | 14 and 1 |
| Q7 | Rate within each population | see §3 | humanSubjects 14/1,357 = 1.0%; inferential 15/1,762 = 0.9%; crawled 1/1,120 = 0.1% |
| Q8 | Per year and per venue | all 5,859 | first in 2021; IMC has none |
| Q9 | Is the 2025–2026 rise real? | humanSubjects | Fisher's exact, two-sided: 5/592 vs 9/303, p = 0.0215. 2024 vs 2025 alone, p = 0.1849 |
| Q10 | Which registry does each of the 15 name? | the 15 | OSF 11, AsPredicted 3, none 1 |
| Q11 | How many papers mention OSF at all, and how many of those in a preregistration context? | 5,869 full texts | 131 mention it, 14 also match Q4's regex, 117 (89.3%) do not |
| Q12 | Does the paper say whether it followed its plan? | the 15 | 3 deviation, 1 no-deviation, 4 confirmatory/exploratory split, 7 silent (§5.3) |
| Q13 | Where does the schema disagree with the full text? | the 15 ∪ Q1 | schema missed 9 of 15; fired once on a paper that registered nothing |
| Q14 | The rest of the inference-hygiene stack, for scale | inferential | hypothesis-test 58.2%, correction 15.3%, effect-size 10.3%, power-analysis 4.9%, bayesian 1.6% |
| Q15 | Same, scoped to the population that needs it | the 1,025 hypothesis-test papers | correction 24.5%, power 7.0% |
| Q16 | Artifact links vs preregistration links | all 5,859 | 3,321 (56.7%) vs 12 (0.2%) |
| Q17 | How much of the corpus never gets past descriptives? | the 5,456 with any statistics tuple | 3,694 (67.7%) |
| Q18 | Is the practice happening under another name? | 5,869 full texts | /\\banalysis plan/ matches 12; 8 outside the 62; 3 are real pre-committed plans with no registry (§5.2) |
| Q19 | Does the practice reach this literature as a Registered Report? | 5,869 full texts | registered report matches 6 papers; 5 of them are citations to a FUZZING-workshop Registered Report, and the sixth is Klemmer et al. discussing the idea |
5. Hand classifications, and why they are not regexes
Neither classification in prereg_fold.mjs is a fold in the sense the other pages on this site use the word. There is no spelling variance to canonicalise. There is a homograph, and the only thing that separates the senses is the sentence — so every hit was read, and the deciding phrase is recorded in the source next to the slug.
The value of keeping them as code is the residue: a paper added to a future corpus that matches PREREG_RE and is in none of the lists prints as unclassified instead of silently joining whichever bucket a regex would have guessed.
5.1 The sense map: 62 papers
| Sense | Papers | Share of 62 |
|---|---|---|
other-sense — a pre-registered domain, OAuth redirect URI, FIDO device, test account, SIM box, chatbot command, enclave manifest, wake word… | 42 | 67.7% |
study — the paper preregistered its own study | 15 | 24.2% |
meta — argues about the practice without doing it | 3 | 4.8% |
citation-only — the word appears only inside a cited paper's title | 2 | 3.2% |
analysis-plan-only — committed to a plan, registered nothing | 3, and none of them in the 62 | — |
| Unmapped residue | 0 | 0.0% |
The analysis-plan-only row is the one that does not fit the table, and that is informative. None of the three writes “pre-regist” at all — the string is absent from all three papers — so the full-text sweep cannot see them. Tang et al. (PETS 2021) were found through statistics[].kind; the other two were found only by the completeness sweep in §5.3. What all three did was commit to an analysis plan, justify it by Type I error, and register it nowhere. They are held in their own map so that they neither inflate the 15 nor vanish.
| Paper | What it says |
|---|---|
| Tang et al., PoPETs 2021 | “In an effort to limit the number of Type I errors, we committed to an analysis plan prior to analyzing the responses from our main study.” |
| Please Unstalk Me, PoPETs 2024 | “We created an analysis plan to reduce the number of Type I errors in our analysis”, and for the second study “We followed our previously created analysis plan to limit any false positives (Type I errors).” |
| How does connecting online activities…, PoPETs 2024 | Declines to report a significant one-tailed result: “this was outside our analysis plan. We had no priors on which directions we should expect a change.” |
The 42 other-sense papers are not noise either. They are the reason a naive query of this corpus for preregistration overstates the practice by roughly a factor of four, and they span the whole window: an IMC 2010 DGA paper (defenders pre-registering domains), a CCS 2012 OAuth paper (pre-registered redirect URI), an IEEE S&P 2025 SSO paper with 8 hits (a service provider pre-registering with an identity provider), a PETS 2026 chatbot paper (pre-registered slash commands).
The three meta papers, in full:
| Paper | What it does |
|---|---|
| Hantke et al., IEEE S&P 2024 | Proposes a pre-registration board run by a trusted third party for server-side scanning, and surveys operators on it. 22 hits, the most in the corpus |
| Klemmer et al., IEEE S&P 2025 | Interviews 24 usable-privacy-and-security researchers about preregistration and registered reports |
| Wei et al., USENIX Security 2024 | “Consider study pre-registration”, citing Cockburn et al.'s HARK No More |
5.2 The completeness sweep, and the probes that failed
PREREG_RE finds papers that use the word. The sweep for /\banalysis plan/ asks whether the thing is being done under another name, and it is what turned up the two extra analysis-plan-only papers above: 12 papers match, 8 outside the 62, of which 3 are real pre-committed plans and 5 are section headings or incidental (“we discussed our analysis plan with peers”).
Four further probes were run and returned nothing usable. They are kept in REJECTED_PROBES with their reasons so the next run does not re-run them hoping for a different answer:
| Probe | Result |
|---|---|
pre-?specified / prespecified | 77 papers outside the 62, none about study design — “prespecified timestamps” (IP alias resolution), “pre-specified sanitizers”, “pre-specified thresholds”. A technical term in this literature, not a methodological one |
a priori hypothes | 0 outside the 62 |
registered (our|the) (study|hypotheses|analysis|protocol) | 0 outside the 62 |
confirmatory (study|analysis) | 1 outside the 62 (NDSS 2015), and it means a follow-up interview round |
clinicaltrials.gov / socialscienceregistry / aearegistry | 2 outside the 62, neither registering anything |
5.3 The adherence map: 15 papers
Two keyword rules were written first and both were wrong in opposite directions, which is why this is also a hand list:
- A rule matching
/deviat/labelled “our final analysis did not deviate from our original plan” (Akgul et al.) as reporting a deviation. - A rule matching
/as pre-?registered/labelled “…attention check, as pre-registered” (Crowder et al.) as an adherence statement, when it is a method descriptor for one exclusion rule.
The published four-way classification, with the deciding sentence for each paper, is in prereg_fold.mjs and printed by the report script.
6. Unedited report output
- report_preregistration.txt
corpus: 5859 extraction records populations: empirical 5118 | crawled 1120 | inferential 1762 | humanSubjects 1357 ======================================================================== A. THE SCHEMA SIGNAL ======================================================================== Signal Papers Of inferential (1762) Of all (5859) ----------------------------------------- ------ --------------------- ------------- statistics[].kind == preregistration 7 0.4% 0.1% artifacts.links[].kind == preregistration 12 — 0.2% either 14 0.8% 0.2% both 5 — — ======================================================================== B. THE FULL-TEXT SWEEP ======================================================================== 5869 paper.cols.txt files scanned (line-break hyphens joined, whitespace collapsed). Regex Papers Share of 5869 ------------------------------- ------ ------------- pre-regist* 62 1.1% aspredicted 3 0.1% osf.io / Open Science Framework 131 2.2% registered report 6 0.1% ── What "pre-registration" means in those 62 papers ── Sense Papers Share of 62 ------------------ ------ ----------- study 15 24.2% analysis-plan-only 0 0.0% meta 3 4.8% citation-only 2 3.2% other-sense 42 67.7% UNMAPPED RESIDUE 0 0.0% residue: none — every hit is classified in prereg_fold.mjs. ── Completeness sweep: /\banalysis plan\b/ ── 12 papers match; 8 of them are outside the 62 pre-regist* hits. 3 of those are classified in ANALYSIS_PLAN_ONLY (a real pre-committed plan with no registry): PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies PETS/2024/please-unstalk-me-understanding-stalking-with-bluetooth-trackers-and-democratizi PETS/2024/how-does-connecting-online-activities-to-advertising-inferences-impact-privacy-p The remaining 5 were read and are section headings or incidental mentions ("we discussed our analysis plan with peers", "Analysis Plan." as a heading): 2020 USENIX from-needs-to-actions-to-secure-apps-the-effect-of-requireme 2022 PETS we-may-share-the-number-of-diaper-changes-a-privacy-and-secu 2022 USENIX understanding-and-improving-usability-of-data-dashboards-for 2023 USENIX understand-users-privacy-perception-and-decision-of-v2x-comm 2026 WWW lurkers-interactors-creators-modeling-behavioral-and-ideolog Probes run and discarded, with why: pre-?specified / prespecified 77 papers outside the 62, none about study design — "prespecified timestamps" (IP alias resolution), "pre-specified sanitizers", "pre-specified thresholds". A technical term in this literature, not a methodological one. a priori hypothes 0 papers outside the 62. registered (our|the) (study|hypotheses|analysis|protocol) 0 papers outside the 62. confirmatory (study|analysis) 1 paper outside the 62 (NDSS 2015), and it uses the term for a follow-up interview round, not a pre-committed analysis. clinicaltrials.gov / socialscienceregistry / aearegistry 2 papers outside the 62, both about matching patients to trials or leaking prompts — neither registers anything. ── "registered report" in the corpus: 6 papers, of which 5 cite a FUZZING-workshop Registered Report ── FUZZING-CITE 2023 NDSS assessing-the-impact-of-interface-vulnerabilities-in-co FUZZING-CITE 2024 CCS prompt-fuzzing-for-fuzz-driver-generation FUZZING-CITE 2024 NDSS large-language-model-guided-protocol-fuzzing other 2025 IEEE-SP transparency-in-usable-privacy-and-security-research-sc FUZZING-CITE 2026 IEEE-SP cottontail-large-language-model-driven-concolic-executi FUZZING-CITE 2026 NDSS mutato-enhancing-fuzz-drivers-with-adaptive-api-option- ======================================================================== C. THE 15 PAPERS THAT PREREGISTERED ======================================================================== Cut Papers Share of 15 ---------------------------- ------ ----------- recruited human participants 14 93.3% ran an automated web crawl 1 6.7% neither 0 — ── Rate within each population that could have preregistered ── Population N Preregistered Share ----------------------------------------------- ---- ------------- ----- humanSubjects — recruited participants 1357 14 1.0% inferential — ran inference beyond descriptives 1762 15 0.9% crawled — ran an automated web crawl 1120 1 0.1% empirical — measured the world as it is 5118 15 0.3% all extraction records 5859 15 0.3% ── By year ── Year Papers in corpus humanSubjects Prereg (any) Prereg (humanSubjects) Share of humanSubjects ----- ---------------- ------------- ------------ ---------------------- ---------------------- 2010 119 18 0 0 — 2011 116 12 0 0 — 2012 151 29 0 0 — 2013 125 25 0 0 — 2014 166 31 0 0 — 2015 190 37 0 0 — 2016 182 35 0 0 — 2017 231 52 0 0 — 2018 254 48 0 0 — 2019 402 92 0 0 — 2020 404 83 0 0 — 2021 379 106 1 1 0.9% 2022 546 117 1 1 0.9% 2023 719 198 1 1 0.5% 2024 690 171 2 2 1.2% 2025* 770 198 7 7 3.5% 2026* 415 105 3 2 1.9% 2026 numerator warning: 3 papers preregistered but only 2 of them are humanSubjects. The wrong numerator over the humanSubjects denominator (105) gives 2.9% instead of 1.9%. * 2025–2026 are provisional: CCS/IMC 2026 have not been held and IEEE S&P/WWW 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. ── Is the 2025–2026 rise real, or is it the provisional slice? ── 2021–2024: 5/592 humanSubjects papers preregistered (0.8%) 2025–2026: 9/303 humanSubjects papers preregistered (3.0%) Fisher's exact test, two-sided: p = 0.0215 2024 (2/171) vs 2025 (7/198) on its own: p = 0.1849 — not significant; the pooled test above is doing the work, and both of its later years are the under-represented ones. 2010–2020 collapsed (the page prints one row for it): 2340 papers, 462 humanSubjects, 0 preregistered. ── By venue ── Venue Papers humanSubjects % humanSubjects crawled % crawled % empirical Preregistered ------- ------ ------------- --------------- ------- --------- ----------- ------------- CCS 990 186 18.8% 163 16.5% 82.6% 1 IEEE-SP 767 208 27.1% 110 14.3% 84.2% 5 IMC 638 67 10.5% 132 20.7% 98.0% 0 NDSS 701 144 20.5% 129 18.4% 83.6% 1 PETS 510 215 42.2% 123 24.1% 93.5% 3 USENIX 1410 367 26.0% 221 15.7% 87.0% 4 WWW 843 170 20.2% 242 28.7% 87.8% 1 ── Registry named, per preregistering paper ── Registry Papers Share of 15 ------------------------------------- ------ ----------- OSF (osf.io / Open Science Framework) 11 73.3% AsPredicted 3 20.0% no registry string in the text 1 6.7% says it preregistered but names no registry (so the claim is not checkable): WWW/2026/maca-a-multi-agent-cognitive-adaptation-framework-for-human-agent-collaborative Also: 1 paper committed to an analysis plan without registering it anywhere — PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies. It is in ANALYSIS_PLAN_ONLY, not in STUDY, and contributes 0 to the 62 full-text hits because the string "pre-regist" never appears in it. ── Does the paper say whether it stuck to the plan? ── What the paper tells the reader Papers Share of 15 ------------------------------------- ------ ----------- deviation-reported 3 20.0% no-deviation-stated 1 6.7% confirmatory-vs-exploratory-separated 4 26.7% silent 7 46.7% IEEE-SP 2024 deviation-reported a-representative-study-on-human-detectio "...but, differing from our preregistration, only run a regression analysis on the latter." CCS 2025 deviation-reported empowering-parents-to-support-childrens- "In the preregistration, we initially planned to use analysis of variance (ANOVA) with repeated measures. Instead, we decided to use generalized linear regressions..." NDSS 2025 deviation-reported the-power-of-words-a-comprehensive-analy "We deviated from our preregistration for permissions and chose to treat them as random effects rather than fixed effects with difference coding." USENIX 2025 no-deviation-stated as-advertised-understanding-the-impact-o "our final analysis did not deviate from our original plan." PETS 2022 confirmatory-vs-exploratory-separated increasing-adoption-of-tor-browser-using "In addition to these pre-planned hypothesis tests (S 4.1), we also conducted several exploratory analyses (S 4.2-4.8)." IEEE-SP 2025 confirmatory-vs-exploratory-separated restricting-the-link-effects-of-focused- "...five pre-registered hypotheses. We further conducted exploratory analyses (also described in the pre-registration)..." USENIX 2026 confirmatory-vs-exploratory-separated you-know-why-but-still-rely-the-impact-o "we first present the results in line with our research questions and hypotheses from the pre-registration. Then, we report on additional exploratory analyses." WWW 2026 confirmatory-vs-exploratory-separated maca-a-multi-agent-cognitive-adaptation- "We pre-registered the primary comparison ... All other contrasts were treated as exploratory." PETS 2021 silent validity-and-reliability-of-the-scale-in IEEE-SP 2023 silent when-and-why-do-people-want-ad-targeting PETS 2024 silent simply-tell-me-how-on-trustworthiness-an Says "as pre-registered" of one exclusion rule, but makes no statement about the analysis as a whole. IEEE-SP 2025 silent lets-get-visual-testing-visual-analogies IEEE-SP 2025 silent not-the-right-question-a-study-on-attitu USENIX 2025 silent vulnerability-of-text-matching-in-ml-ai- USENIX 2026 silent chameleon-channels-measuring-youtube-acc ── "It is on OSF" is not "it was preregistered" ── Cut Papers Share of 131 ------------------------------------------------------ ------ ------------ mentions osf.io or Open Science Framework 131 100.0% …and also matches pre-regist* 14 10.7% …OSF appears with no pre-registration wording anywhere 117 89.3% ======================================================================== D. SCHEMA vs FULL TEXT ======================================================================== Papers ------------------------------------------------------------- ------ preregistered (this script) and statistics.kind agrees 6 preregistered but statistics.kind MISSED it 9 statistics.kind fired but the paper did not publicly register 1 missed by statistics.kind: PETS/2021/validity-and-reliability-of-the-scale-internet-users-information-privacy-concern PETS/2022/increasing-adoption-of-tor-browser-using-informational-and-planning-nudges PETS/2024/simply-tell-me-how-on-trustworthiness-and-technology-acceptance-of-attribute-bas CCS/2025/empowering-parents-to-support-childrens-online-security-and-privacy-findings-fro IEEE-SP/2025/not-the-right-question-a-study-on-attitudes-toward-client-side-scanning-with-sec IEEE-SP/2025/restricting-the-link-effects-of-focused-attention-and-time-delay-on-phishing-war NDSS/2025/the-power-of-words-a-comprehensive-analysis-of-rationales-and-their-effects-on-users-permission-decisions USENIX/2025/vulnerability-of-text-matching-in-ml-ai-conference-reviewer-assignments-to-collu USENIX/2026/you-know-why-but-still-rely-the-impact-of-explainable-ai-on-trust-task-load-and statistics.kind fired, not a public registration: PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies — PETS 2021 — "In an effort to limit the number of Type I errors, we committed to an analysis plan prior to analyzing the responses from our main study." No registry, no link, no timestamp. Found via statistics.kind; the string "pre-regist" never appears in the paper. ======================================================================== E. THE ADJACENT PRACTICES, FOR SCALE ======================================================================== papers with any statistics tuple: 5456; of those, 3694 (67.7%) report nothing beyond descriptives. statistics.kind Papers (of inferential 1762) Share ------------------------------ ---------------------------- ----- hypothesis-test 1025 58.2% multiple-comparison-correction 269 15.3% effect-size 182 10.3% power-analysis 87 4.9% bayesian 28 1.6% preregistration 7 0.4% Of the 1025 papers that ran a hypothesis test Papers Share --------------------------------------------- ------ ----- also report a multiple-comparison correction 251 24.5% also report a power analysis 72 7.0% Of the 1357 papers that recruited participants Papers Share ---------------------------------------------- ------ ----- ran a hypothesis test 482 35.5% report a power analysis 74 5.5% preregistered 14 1.0% Open-science practice Papers Share of 5859 --------------------------------------------- ------ ------------- released an artifact link of the authors' own 3321 56.7% linked a preregistration 12 0.2% ======================================================================== F. EXTERNAL FIGURES ON THE PAGE (not from this corpus) ======================================================================== 119 Hantke et al., IEEE S&P 2024, Table 3 — surveyed operators 36.1 Hantke et al., Table 3, Q6 "comfortable (5)" 21.8 Hantke et al., Table 3, Q6 "somewhat comfortable (4)" 37.8 Hantke et al., Table 3, Q8 "comfortable (5)" 35.3 Hantke et al., Table 3, Q8 "somewhat comfortable (4)" 57.9 this page's arithmetic: 36.1 + 21.8 73.1 this page's arithmetic: 37.8 + 35.3 4.15 Hantke et al., §5.1 — NGO/white-hat TTP competence, mean (the §5.2 heading is interleaved from the facing column in .cols) 1.1 Hantke et al., §5.1 — same, standard deviation 3.79 Hantke et al., §5.1 — academic-institution TTP competence, mean 1.13 Hantke et al., §5.1 — same, standard deviation 24 Klemmer et al., IEEE S&P 2025 — interviewees 44 api.osf.io/v2/schemas/registrations — template count, 2026-08-13 22 api.osf.io/v2/registrations/fg9pm — non-upload question fields 11 same — of those, answered 1,005 same — words across the answered fields 2.00 aspredicted.org — "Version of AsPredicted Questions" 8 aspredicted.org — numbered questions on the live form 4 help.osf.io — maximum embargo, years 74,302 AsPredicted registration number, read live 138,374 AsPredicted registration number, read live 170,202 AsPredicted registration number, read live 74 / 138 / 170 digit fragments of the three AsPredicted numbers above 10 OSF identifiers cited as preregistrations by corpus papers: 8 registrations + 2 projects 30 urllib timeout in seconds, inside the page's downloadable script 10.17605 OSF DOI prefix 200 / 401 / 403 / 406 HTTP status codes; aspredicted.org answers a plain curl with 406, dl.acm.org with 403 0.05 / 20 / 40 / 8 / 34 / 3 illustrative examples in prose, not measurements 2019-08-08 … 2025-04-16 OSF date_registered, read live from api.osf.io
7. External sources, and how each was verified
Everything here was fetched on 2026-08-13, not recalled, and every command is in scripts/external_checks_preregistration.sh. Its unedited output:
- external_checks_preregistration.txt
== 1. Every preregistration URL cited by a corpus paper, and what it actually is GUID / URL OSF type template registered withdrawn osf.io/bc42h registrations OSF Preregistration 2021-03-24 False osf.io/xy6v5 registrations OSF Preregistration 2022-05-12 False osf.io/5jd8a registrations Preregistration Template from AsPredicted.org 2019-08-08 False osf.io/st6pz registrations OSF Preregistration 2023-08-03 False osf.io/7zprb registrations OSF Preregistration 2023-07-10 False osf.io/q4rj3 nodes (project, not a registration) - None osf.io/d6h7b registrations HTTP 401 (private / view-only link) - - osf.io/5pywm nodes (project, not a registration) - None osf.io/zqy7b registrations OSF Preregistration 2024-09-27 False osf.io/fg9pm registrations OSF Preregistration 2025-04-16 False == 1b. DOI minted for each OSF registration osf.io/bc42h -> 10.17605/OSF.IO/BC42H osf.io/xy6v5 -> 10.17605/OSF.IO/XY6V5 osf.io/5jd8a -> 10.17605/OSF.IO/5JD8A osf.io/st6pz -> 10.17605/OSF.IO/ST6PZ osf.io/7zprb -> 10.17605/OSF.IO/7ZPRB osf.io/zqy7b -> 10.17605/OSF.IO/ZQY7B osf.io/fg9pm -> 10.17605/OSF.IO/FG9PM == 2. The three AsPredicted registrations cited by corpus papers https://aspredicted.org/HXF_Z92 #170,202 registered 2024/04/10 18:29 (PT) anonymous=False canonical URL now: https://aspredicted.org/j3n9-kxzf.pdf https://aspredicted.org/blind.php?x=MTD_GH1 #74,302 registered 2021/09/09 15:03 (PT) anonymous=True canonical URL now: https://aspredicted.org/mryz-wf5h.pdf https://aspredicted.org/rk8xe.pdf #138,374 registered 2023/07/13 15:23 (PT) anonymous=False canonical URL now: not shown == 3. AsPredicted's questions, verbatim from the live form, with the form version 1) Have any data been collected for this study already? 2) What's the main question being asked or hypothesis being tested in this study? 3) Describe the key dependent variable(s) specifying how they will be measured. 4) How many and which conditions will participants be assigned to? 5) Specify exactly which analyses you will conduct to examine the main question/hypothesis. 6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations. 7) How many observations will be collected or what will determine sample size? 8) Anything else you would like to pre-register? 8 numbered questions found. Version of AsPredicted Questions: 2.00 == 4. OSF registration templates that exist today 44 templates total Generalized Systematic Review Registration active=True OSF Preregistration active=True Open-Ended Registration active=True Prereg Challenge active=False Preregistration Template from AsPredicted.org active=True Qualitative Preregistration active=True Registered Report Protocol Preregistration active=True Secondary Data Preregistration active=True == 5. The OSF Preregistration template question list (which a measurement study has to map onto) ## Overview - Research questions or hypotheses - Foreknowledge of data or evidence - Explanation of foreknowledge and managing unintended influences ## Research Design - Study type - Intention for causal interpretation - Blinding of experimental treatments - Additional blinding during research or analysis - Study design - Randomization ## Sampling - Data collection procedures - Data collection procedures - File upload - Sample size - Sample size rationale - Starting and stopping rules ## Variables - Manipulated variables - Measured variables - Measured variables - File upload - Indices - Indices - File upload ## Analysis Plan - Statistical models - Statistical models - File upload - Transformations - Inference criteria - Data inclusion and exclusion - Missing data - Other planned analysis ## Other - Context and additional information == 6. The one measurement-study registration, read in full (USENIX Sec 2026, Chameleon Channels) template: OSF Preregistration registered: 2025-04-16 11 of 22 question fields answered (6 file-upload slots excluded); blank: q5, q7, q13, q14.question, q16.question, q18, q19, q20, q21, q22, q23 1005 words across the answered fields q3: Observational Study - Data is collected from study subjects that are not randomly assigned to a treatment. This includes surveys, “natural experiments,” and regression discontinuity designs. q4: ["No blinding is involved in this study."] q8: Registration prior to analysis of the data q9: We have designed scrapers to collect the data and databases to store the data. Because of this procedure, the team has been exposed to small portions of the data. All interaction, however, has been just for collection an q12: Our sample of advertised channels constitutes all observed channels for sale in the platform. Our goal is to be exhaustive. For our Social Blade sample (~1.4M channels), our goal was to find a large enough number that wo == 7. Registered reports: no venue in this corpus runs them. Pages checked: 200 https://www.sigsac.org/ccs/CCS2026/call-for/call-for-papers.html 200 https://sp2026.ieee-security.org/cfpapers.html 200 https://www.ieee-security.org/Calendar/cfps/cfp-USENIXSec2026.html 200 https://www.ndss-symposium.org/ndss2026/submissions/call-for-papers/ 200 https://petsymposium.org/cfp26.php 200 https://conferences.sigcomm.org/imc/2026/cfp/ 200 https://www2026.thewebconf.org/calls/research-tracks.html 200 https://soups.page/cfp.html 200 https://www.ndss-symposium.org/ndss2022/cfp-fuzzing-workshop/ 200 https://conf.researchr.org/track/issta-2023/fuzzing-2023 200 https://2026.msrconf.org/track/msr-2026-registered-reports 200 https://conf.researchr.org/track/esem-2024/esem-2024-registered-reports 200 https://emsejournal.github.io/registered_reports/ 200 https://chi2026.acm.org/contributions-to-chi/ 200 https://www.usenix.org/conference/usenixsecurity26/call-for-papers 403 https://dl.acm.org/journal/tosem/registered-papers (EXPECTED-403: bot protection) Registered Reports are run by FUZZING (ndss-symposium.org/ndss2022 and conf.researchr.org/issta-2023), MSR 2026, ESEM 2024 and the EMSE journal. CHI 2026 does NOT: its contribution-types page mentions neither registered reports nor preregistration. ACM TOSEM is reported to run Registered Papers but COULD NOT BE VERIFIED: dl.acm.org, tosem.acm.org and the static PDF route all 403 every non-browser client, so that claim rests on the FUZZING track page, not on ACM s. None of these is one of this corpus seven venues.
7.1 What each check established
| Claim on the page | How it was verified |
|---|---|
| 8 of the 10 OSF identifiers cited as preregistrations are registrations; 2 are ordinary projects | api.osf.io/v2/guids/<guid>/ returns a 302 whose Location is /registrations/… or /nodes/…. The typed endpoint then gives registration_supplement, date_registered, withdrawn and embargoed |
osf.io/d6h7b is a registration but not public | /guids/d6h7b/ redirects to /registrations/, and /registrations/d6h7b/ returns HTTP 401 |
| Every public OSF registration gets a DOI | api.osf.io/v2/registrations/<guid>/identifiers/ returned 10.17605/OSF.IO/<GUID> for all seven public ones |
44 OSF registration templates; Prereg Challenge is active=false | api.osf.io/v2/schemas/registrations/?page[size]=50 |
| The OSF Preregistration question list | Same endpoint; the page-heading and question-label blocks of the template named OSF Preregistration |
| The four Chameleon Channels answers | api.osf.io/v2/registrations/fg9pm/, field registration_responses. 11 of 22 non-upload fields answered |
| Maximum embargo of four years; registrations immutable and only withdrawable | help.osf.io registration articles |
| AsPredicted asks eight numbered questions, form version 2.00 | Read off the rendered registration at aspredicted.org/HXF_Z92, which prints both. aspredicted.org 403s WebFetch but serves curl with a browser User-Agent |
| AsPredicted #74,302 is still marked anonymous | Same page format at aspredicted.org/blind.php?x=MTD_GH1: “This pre-registration is currently anonymous to enable blind peer-review. It has 4 authors.” |
| No venue in this corpus runs Registered Reports | Each venue's current CfP fetched and read: CCS 2026, IEEE S&P 2026, USENIX Security 2026, NDSS 2026, PoPETs 2026, IMC 2026, TheWebConf 2026, SOUPS 2026 |
| FUZZING runs a two-phase Registered Reports process | The workshop's own CfP at ndss-symposium.org/ndss2022/cfp-fuzzing-workshop/ and the ISSTA 2023 track page |
| IMC's Replicability Track is about replicating previously published results | conferences.sigcomm.org/imc/2026/cfp/: “submissions that aim to reproduce or replicate network measurement results that have been previously published” |
| Hantke et al.'s operator percentages | Table 3 of the paper itself, paper.cols.txt — not a secondary summary. The two sums (57.9%, 73.1%) are this page's arithmetic on the paper's per-column figures and are labelled as such in a footnote |
7.2 Claims rejected, and why
| Claim | Why it is not on the page |
|---|---|
| “AsPredicted asks nine questions.” | Several guides written by people close to the platform say nine, and a research sub-agent reported the discrepancy honestly rather than picking one. The live form identifies itself as Version of AsPredicted Questions: 2.00 and prints eight. The page states eight, footnotes where it read them, and notes the discrepancy rather than hiding it |
| The Center for Open Science's participating-journals list as evidence about CS adoption. | cos.io/initiatives/registered-reports surfaces only Frontiers in Computer Science from the whole of computing. It misses MSR, ESEM, EMSE and ACM TOSEM, which all run Registered Reports independently of COS's programme. Using it would have produced the false claim that CS has essentially no Registered Reports |
fuzzingworkshop.github.io as the FUZZING source. | Returns HTTP 404. The claim rests on the ndss-symposium.org and conf.researchr.org pages instead |
| A USENIX Security CfP quotation taken from the mirror alone. | usenix.org 403s WebFetch, so a sub-agent read it through ieee-security.org's mirror. That was re-checked directly with curl -A before anything went on the page (HTTP 200) |
Naming the authors of osf.io/d6h7b's paper. | A draft attributed it to “Amft et al.” on no evidence. The page now names the paper by venue and topic instead |
8. Bugs caught in this run
Three of these are in shared scripts and affect pages other than this one.
| # | Bug | Where | Fix |
|---|---|---|---|
| 1 | A numerator and a denominator from different populations. The per-year table put every preregistering paper over the humanSubjects denominator. Chameleon Channels has no participants, so 2026 read as 3/105 = 2.9% when the honest figure is 2/105 = 1.9% | report_preregistration.mjs | Two numerator columns, and the report prints the warning explicitly so the guard can see it |
| 2 | scripts/fetch_authors.py silently dropped authors from USENIX landing pages, in two independent ways: it split the whole author string on commas, so any author immediately after a semicolon was glued onto the preceding affiliation and filtered out; and it never stripped a leading “and ”, so the last author of every multi-author group was dropped. Measured against four live pages: 1 of 5 authors lost on usenixsecurity25/presentation/akgul, 3 of 6 on usenixsecurity24/presentation/wei-miranda-solk, 2 of 3 on usenixsecurity26/presentation/cuevas | shared | Rewritten to split on ; first. Every USENIX BibTeX entry already in literature:bibliography that came from the old parser may be missing authors; a work item to re-derive and diff them is filed |
| 3 | scripts/bibgen.mjs hardcoded the PoPETs DOI prefix 10.56553 for every year. PoPETs has changed prefix twice. Probed against Crossref: 10.1515 for 2015–2018, 10.2478 for 2019 through popets-2022-0058, 10.56553 from popets-2022-0059 onward (both sides of the boundary checked directly) | shared | A popetsPrefix(year, article) function with the boundaries and the probe date in a comment. Every PoPETs DOI currently live on the wiki resolves, so this had been caught by hand each time |
| 4 | The published check_prereg_link.py matched https as an OSF GUID, because its regex allowed a bare ^ alternative. Every full URL reported HTTP 404 | pages/check_prereg_link.py | Anchored on osf.io/, with a separate pattern for a bare identifier. The <code> block on the page is the output of the fixed script |
9. Quotes checked against the source
Eighteen quotes were checked against paper.cols.txt with hyphens joined across line breaks and whitespace collapsed. 16 matched verbatim or through an 8-word run. Two did not, and both are the known column-splice artefact rather than a fabrication — each was then confirmed fragment by fragment:
| Paper | Quote | Verdict |
|---|---|---|
| Cuevas et al., USENIX Security 2026 | “we pre-registered our regression hypotheses, methods, and analysis plan prior to conducting the study” | Present, spliced. .cols interleaves it with the neighbouring column (“we pre-registered our regression // we provided the LLM with examples…”). Fragments “we pre-registered our regression”, “hypotheses, methods, and anal”, “prior to conduct” and “The pre-registration is available at” all found |
| Groß, PETS 2021 | “our analysis is a pre-registered confirmatory study” | Present, spliced. Fragments “analysis is a pre-registered confirmatory study” and “OSF: https://osf.io/5pywm” both found |
The other sixteen — Tang et al.'s analysis-plan sentence, Frank et al.'s OSF sentence, the MACA sentence, Akgul et al.'s no-deviation sentence, Zimmermann et al.'s OSF sentence, Lee et al.'s AsPredicted sentence, three Hantke et al. sentences, two Klemmer et al. sentences, Petelka et al.'s OSF sentence, Roch et al.'s pre-registration sentence, Elbitar et al.'s deviation sentence, Chen et al.'s ANOVA sentence and Wei et al.'s recommendation — matched directly.
10. What could not be established
- Whether the 15 is anywhere near the real rate in this field. The corpus has no SOUPS, which is the single venue where preregistered usable-privacy work is most likely to appear, and no EuroS&P, ACSAC, RAID, AsiaCCS, WPES or CHI. The page says the 15 is a lower bound and probably a substantial one; it does not guess how much.
- Whether preregistration changes what gets published here. That needs the denominator of registered-but-never-published studies. OSF registrations are searchable and nobody has looked; the page files it as an open question rather than speculating.
- The registration behind
osf.io/d6h7b. It exists and is a registration, but the API returns 401 and only the paper'sview_onlylink opens it, so its template and date could not be independently confirmed. - Whether the OSF Preregistration template is workable for crawls in general. One example exists. Everything the page says about mapping a crawl onto the template generalises from Chameleon Channels alone, and the page says so.
- The exact
q-number → question-label mapping of the OSF template.registration_responsesusesq2…q23and the schema endpoint returnsregistration_response_key: nullfor every block in version 4, so the mapping cannot be derived programmatically. The four answers quoted on the page were identified by their content, which is unambiguous for those four; the page therefore quotes answers rather than claiming to name which numbered blanks were left empty. - Whether the OSF Preregistration template used by the 2019 and 2021 registrations is the same version 4 the API returns today. It is very likely not. The page does not claim it is.
- Whether the “no venue runs Registered Reports” claim holds for workshops co-located with these venues. Only main-conference CfPs were read. FUZZING is the one workshop checked, and only because the corpus pointed at it.
11. Judgement calls
- New page, not a section of an existing one. The alternative was a section of a broader
statistics:page. Rejected because start already promises this exact page id, and because the measurement-specific content — the OSF template mapping, the registry-versus-project distinction, the registered-reports venue table — does not fit under a heading in a general statistics page without swamping it. - The population is 15, not 14 and not 7. The schema union is 14 and
statistics.kindalone is 7. The published population is the full-text sweep'sstudysense, which is 15, because the schema demonstrably misses 9 of them. This means the page's headline number cannot be reproduced from the extraction alone — it needs the full text — and §4 says so. - The three
analysis-plan-onlypapers are excluded from the 15. All three pre-specified an analysis plan and registered nothing. Including them would make the count 18 and the page's central distinction — that a commitment nobody else can check is not a preregistration — incoherent. A reasonable person could include them with a footnote. They are not dropped: they have their own section on the content page, because a reader who already writes an analysis plan is the reader most likely to act on this page. - The page is framed as advice, not as a lament. The corpus finding is “almost nobody does this”. The page's framing is “here is how, and here is the one worked example”, because a page that only reports the rate is a fact about the literature and not guidance. That is a deliberate choice about tone and it is the main thing a reviewer might disagree with.
- Three of the fifteen are named for doing something badly, by name: MACA for claiming preregistration with no registry, and the two papers whose “preregistration” link resolves to a mutable OSF project. This is a judgement about whether a systematisation page should name names. It does, because the alternative — describing the failure abstractly — leaves the reader unable to check it, and every one of these is verifiable in a single command that the page ships.
- Currency: preregistration is dated as “emerging”, not “current practice”. The corpus shows 0 papers before 2021 and a jump in 2025–2026 — but 2025–2026 are the provisional venue-years, and the jump is significant only when they are pooled (p = 0.0215) and not year-on-year (p = 0.1849). The page states the pooled test, the year-on-year test and the reason to distrust both, and tells the reader to quote “about 1% and rising” rather than 3.5%.
- The FUZZING workshop is given a whole paragraph even though it is not in the corpus and is not a security venue, because it is the only working precedent anyone proposing a Registered Reports track here could point at. That is a call about scope: the page is about the seven venues, and this is outside them.
- No
~~DISCUSSION~~on provenance pages. Comments belong on the content page. This follows the convention set by the first provenance page on this site.
12. Guard runs
check_page_numbers.mjs was run whole-page, windowed on Use in Publications, and with –code.
| Run | Unaccounted | What it found |
|---|---|---|
| 1 | 31 | Mostly external figures the report never printed. It also surfaced two real gaps: the collapsed 2010–2020 row (2,340 / 462) and the 67.7% descriptive-only figure were the page's own arithmetic and OVERVIEW.md's number, not the report's. Both are now computed by the report script |
| 2 | 4 | The 2.9%-versus-1.9% sentence — the page's own arithmetic on the wrong-numerator bug — plus digit fragments of the AsPredicted registration numbers and the timeout=30 inside the embedded Python. An F. EXTERNAL FIGURES section was added to the report, listing each non-corpus number with its primary source, so they are accounted for without polluting the guard's shared ALLOW map |
| 3 | 0 (whole-page, –code, and windowed) | Clean |
The embedded <file python check_prereg_link.py> block was compared line-by-line against pages/check_prereg_link.py: identical. The <code> block is the verbatim output of a real run, including the $ echo $? showing exit status 1.
check_page_numbers.mjs is not run against this provenance page: its numbers are meta — section numbers, counts quoted precisely because they were rejected, old-corpus figures kept for comparison — so the guard's premise does not apply.
13. Review pass, 2026-08-13
Four reviewers, each handed the page text, the scripts, their unedited output and these notes, and each told explicitly that the author's context might not be exhaustive. The three focused passes ran in parallel against a frozen snapshot; the generic pass ran afterwards, against the corrected text. Every finding is recorded with whether it was accepted, because a rejection is the only record of whether a reviewer earned its slot.
Which snapshot each pass saw. out/frozen_prereg_r1.txt (43,680 bytes) went to the three focused passes. Two things changed under them while they ran, both caught by the author rather than by a reviewer, and both recorded here so the verdicts below can be read against the right text: the intro's unsourced “it costs about a day” became a measured “about a thousand words” with a footnote, and “four papers cite a FUZZING paper whose title ends (Registered Report)” became five, after the claim was measured rather than asserted (the count is now printed by the report script). The generic pass saw out/frozen_prereg_r3.txt (47,460 bytes), which includes every fix below.
13.1 Figures versus the script (Sonnet)
| # | Finding | Verdict |
|---|---|---|
| 1 | Re-ran the report script in all three modes and diffed byte-for-byte against the committed output: zero diffs. Re-ran external_checks_preregistration.sh live: 121 of 121 lines matched. | Noted. |
| 2 | Independently re-derived eleven corpus figures with its own jq/Python rather than through the script — humanSubjects 1,357, inferential 1,762, 3,321 (56.7%), 12, 7, 1,025, 251 (24.5%), 72 (7.0%), 482 (35.5%), 74 (5.5%), 138 posters, 251 short records — all matched. | Noted. |
| 3 | Re-swept paper.cols.txt for pre-regist* and OSF from scratch with its own regex: 62 and 131, and the 62 slugs diffed identical against prereg_fold.mjs's classified set — zero residue, zero missing. | Noted. This is the check that most needed doing, because the population of 15 cannot be reproduced from the extraction alone. |
| 4 | Recomputed Fisher's exact independently via math.comb: p = 0.02152510 and p = 0.18494275, matching the published 0.0215 and 0.1849. | Noted. |
| 5 | Spot-checked nine sense classifications against source text (eight were asked for), including reproducing the exact 8-hit count in the IEEE S&P 2025 SSO paper and confirming that defining-privacy contains zero pre-regist occurrences. All correctly bucketed. | Noted. |
| 6 | Ran the published check_prereg_link.py live with the page's own arguments: output matches the <code> block exactly, including exit status 1. The <file> block diffs clean against pages/check_prereg_link.py. | Noted. |
| 7 | The report script's internal provenance annotation cites Hantke et al. “§5.2” for the trusted-third-party competence means; the paragraph is actually the tail of §5.1. The “5.2. Ethical Challenges” heading is interleaved from the facing column in .cols, which is how the wrong section number got recorded. | Accepted. Corrected in the script. The section number never appeared on the published page — it lived only in the report's EXTERNAL FIGURES bookkeeping — but a wrong citation inside an audit trail is exactly the kind of thing that gets copied out of one later. |
| — | Also verified the Secondary Data Preregistration template's nine named fields against the live OSF schema API, the register-early table's dates against api.osf.io and aspredicted.org, and that Chameleon Channels' extraction record really carries a non-null crawlConfig and an empty participants. | Noted. |
13.2 Citations and quotes (Sonnet)
| # | Finding | Verdict |
|---|---|---|
| 1 | All 20 citation keys resolve; none collides with the 166 live-bibliography entries; no duplicates within the additions file or within the live file. | Noted. |
| 2 | Every one of ~20 quotations found verbatim. One (Petelka et al.) needed an inline footnote-marker digit stripped, which is an extractor artefact and not a page error. | Noted. |
| 3 | Re-fetched all six USENIX/NDSS author bylines specifically to test whether the fetch_authors.py bug had left residue: all six match the bibliography exactly, in order, and match each paper's own full-text header. | The most valuable check of this pass, because it is the one that could have caught a defect the author introduced and then believed fixed. |
| 4 | seong2026_maca gives the second author as “Hayoung Oh”; Crossref's own record for the DOI says “Ha Young Oh”. | Accepted. Corrected to the DOI's metadata. |
| 5 | Two titles were re-cased or re-punctuated away from Crossref: crowder2024_simply (sentence case in Crossref) and zimmermann2025_visual (a dash, not a colon). | Accepted. Both restored to the Crossref strings. Neither changed which paper was cited, but a bibliography that silently normalises publisher metadata is harder to check than one that does not. |
| 6 | Flagged the /* ... */ editorial block in the References section as containing a literal {[key]}, then verified against a live page using the same boilerplate that DokuWiki hides it and it injects no reference. | Rejected, correctly, by the reviewer itself. Recorded because a reviewer that chases down its own false positive rather than filing it is worth more than one that does not. |
| 7 | Independently confirmed the Hantke Table 3 arithmetic, the Klemmer quotes, all four Chameleon Channels registration answers against the live OSF API, the field-by-field 11-of-22 count, the three AsPredicted registrations including the still-anonymous flag, and the FUZZING phase-1 quotes against the 2022 first-edition CfP rather than the current one. | Noted. |
13.3 External currency (Sonnet)
| # | Finding | Verdict |
|---|---|---|
| 1 | “Blocking: study_preregistration is cited three times and does not exist; the provenance: namespace on the live wiki has zero pages.” | Rejected on both halves, and it is the most instructive finding of the review. The page did not exist because it had not been saved yet — the two pages are published together. And the namespace is not empty: provenance:privacy:requests, provenance:programming:crawler and provenance:design:sampling all return real source from ?do=export_raw. This is the trap already written into the workdir notes: the wiki's index does not list the provenance: namespace, and ?do=export_raw on a page that does not exist returns the HTML error page, so neither the index nor a status code is an existence test. Two independent runs have now been misled by it. |
| 2 | The CCS 2026 row understates the CfP: CCS now requires a mandatory “Open Science” appendix listing where the artifacts are, available within 3 days of submission or the lack explained — structurally the same mechanism the page correctly credits USENIX with, and the row said only “optional artifact evaluation with badges”. | Accepted, and it had already been corrected from the same source before this review landed — the author re-read the CCS and SOUPS CfPs while waiting. The row now quotes the CfP. The SOUPS row gained the detail that its call names OSF itself as a suggested artifact host, which is direct evidence for the page's “OSF is a file host here” argument. |
| 3 | The footnote said aspredicted.org “returns HTTP 403 to some automated clients”. A plain curl gets 406. | Accepted. Measured: plain curl 406, browser User-Agent 200. Footnote and script comment corrected. |
| 4 | “11 papers, 8 distinct public registrations verified live” conflates the private one into the public count: 7 are public, 1 is private (HTTP 401), 2 are projects. | Accepted. The mechanics table now spells out 7 public + 1 private + 2 projects, and notes that the 11th paper writes “on OSF” with the URL only in a PDF hyperlink. |
| 5 | Every sibling statistics: link, plus design:user_studies, design:automated_measurements and artifacts, is a red link today. | Accepted as a fact, rejected as a change. All of them are already promised from start, and this site links promised-but-missing pages as a matter of course — privacy:requests ships red links to programming:crawler:pagegraph and programming:crawler:tracker_radar_collector. Removing them would make this page less consistent with the site, not more. It is recorded here so the next run does not “fix” it. |
| — | Could not verify: the two Klemmer et al. quotes (CISPA's figshare and the S3 mirror both refuse non-JS fetches) and IMC 2026's “mandatory” ethics appendix (the CfP defers to a separate instructions page). It said so rather than guessing, and it explicitly reported pulling the wrong Klemmer paper first and confirming that it was not the source. | Noted, and both gaps were closed elsewhere: the Klemmer quotes were verified verbatim against paper.cols.txt by pass 13.2 and by the author. The IMC “mandatory ethics appendix” was then confirmed verbatim on IMC 2026's own submission-instructions page — “All papers must include, in a clearly marked appendix section with the heading 'Ethics', a statement about ethical issues; papers that do not include such a statement may be rejected” — and the artifact-availability declaration on the CfP itself. |
13.4 Generic pass (Fable)
It ran against out/frozen_prereg_r3.txt and out/frozen_prov_r1.txt, and produced the largest number of accepted findings of any pass — as it did on the last page. It was given no checklist, only the reader definition and the instruction to be adversarial.
| # | Finding | Verdict |
|---|---|---|
| 1 | “§13.4 is a literal placeholder while §13's preamble and §14 both claim in the past tense that the generic pass ran. On a page whose whole job is not overstating its own rigour.” | Accepted, and it is the finding this genre exists to catch. This section is that fix. It is the second consecutive run on this site where the generic pass caught exactly this, which is an argument for writing the section before claiming it rather than for running a fourth reviewer. |
| 2 | “Three external claims have no audit trail, under a sentence that promises one for everything. §7 says every command is in the script; there is no CHI fetch anywhere, the only TOSEM request returned 403, and ESEM was never fetched — only emsejournal.github.io, which is the EMSE journal, a different thing from the ESEM conference.” | Accepted in full, and the most valuable finding of the review — three passes had looked at this page and none noticed that a verified-looking sentence rested on recall. CHI 2026 and ESEM 2024 were then fetched (both HTTP 200; CHI's contribution-types page mentions neither registered reports nor preregistration) and both added to the script. ACM TOSEM could not be verified from any primary source: dl.acm.org, tosem.acm.org and the static PDF route all 403 every non-browser client. The page now attributes the TOSEM claim to FUZZING's own track page and says in a footnote that ACM's page could not be read. |
| 3 | “§5.1 says ANALYSIS_PLAN_ONLY holds 1 paper; the fold holds 3 and the embedded report in §6 says 3.” | Accepted; already fixed before the review landed. The two extra PoPETs papers were added by the completeness sweep (§5.2) while this pass was running, and §5.1 was updated with them. Recorded anyway because the reviewer read a real inconsistency in a real snapshot. |
| 4 | “The headline is field-scoped where only a corpus-scoped claim is supportable. §10 says whether the 15 is near the real field rate could not be established, but a student will quote the first bold sentence as a fact about the field.” | Accepted. Now “exactly one measurement study in this corpus”. |
| 5 | “'It works' is contradicted by the page's own todo about the same paper: Chameleon Channels is one of the seven silent papers, so by the page's own standard the one worked example did not close the loop.” | Accepted. Now “the form fits”, and the todo carries the caveat. |
| 6 | “11 + 3 = 14, not 15, and the very next sentence says so.” | Accepted. Now “every checkable preregistration — 14 of the 15”. |
| 7 | “The heading says a third; the table beneath says 4 of 15, which is a quarter.” | Accepted. The heading now states the count. |
| 8 | “'Question 4 is the only one a crawl cannot answer' is wrong for exactly the crawls this reader will run. Many web measurements are multi-arm — consent-accept versus reject, stateful versus stateless, several vantage points. Telling the reader Q4 never applies invites them to leave their arms out, which is the subgroup drift the page warns about.” | Accepted, and it is the best substantive catch of the review. The paragraph is rewritten to say that a multi-arm crawl's arms are the answer to question 4. |
| 9 | “The embargo advice ends in an impossibility and never states the cost of exposure. The observed worst case (5 years) exceeds OSF's maximum (4), the page never says what to do then, and the double-blind workflow is given only for AsPredicted.” | Accepted. Three paragraphs now: the gap versus the maximum, what exposure actually costs under anonymous review, and the register-embargo-view-only-lift workflow, using osf.io/d6h7b as the worked instance of it. |
| 10 | “The descriptive-study reader — the page's own modal reader — never gets told what to write in the hypothesis fields. The page says 67.7% never get past descriptives and then gives only hypothesis-shaped guidance.” | Accepted; the page's one real gap against its own stated purpose. A new box tells a hypothesis-free study to use Open-Ended Registration and register the rows of the drift table instead. |
| 11 | “REJECTED_PROBES lists five probes; the page says four.” | Accepted. confirmatory (study|analysis) was missing from the prose. |
| 12 | “A denominator slip on the page that lectures about denominators: 'most measurement papers are descriptive' over a base of 5,456 that includes crypto and systems papers.” | Accepted. Now “most papers in these venues”. |
| 13 | “preregisterd is verbatim in Klemmer et al. — add [sic]. And 'several widely-cited guides' names no guide.” | Accepted. [sic] added; “widely-cited” softened to “third-party”, since no specific guide was verified. |
| 14 | “The table is introduced as 'no venue in this corpus' and then lists eight, one of which the methodology section says is not in the corpus.” | Accepted. Introduced as the seven, plus SOUPS, with the reason SOUPS is there. |
| 15 | “'the other six of the fifteen silent papers' parses as fifteen silent papers.” | Accepted. |
| 16 | “The Half the preregistrations were invisible to the schema subsection is addressed to someone querying the dataset, not to this page's reader; move the table here and keep only the Tang lesson.” | Rejected, with the reviewer's own caveat noted. It flagged this as weak because privacy:requests carries analogous corpus-QA subsections. The stronger reason to keep it: the page's headline number cannot be reproduced from the extraction alone, and a reader who tries will get 7 or 14 and conclude the page is wrong. The section is the warning that stops that. |
| 17 | “The Mechanics table's 'time to fill in' does not mention that OSF registrations need co-contributor approval before the timestamp lands.” | Not applied, and flagged honestly by the reviewer as unverified. Not checked here either, and adding an unverified operational claim to a page whose argument is “verify the link” would be the wrong trade. It is a good candidate for the next run. |
| — | It also confirmed the section skeleton, voice, footnoting and Use in Publications convention against start, design:website_selection, privacy:requests and its provenance page, and diffed the provenance page's embedded report against out/prereg-report.txt clean. Its closing assessment named the decision-drift table, the four copyable registration answers, the Secondary Data recommendation and the project-versus-registration checker as the parts that earn the page. | Noted. |
One finding this pass did not make, and the author did. While the generic pass was running, the sentence “IMC … is the most measurement-heavy venue in the corpus” was checked rather than assumed and is false: TheWebConf crawls more (28.7% of its papers) and PoPETs crawls more (24.1%) than IMC (20.7%). What IMC actually is, is the venue with the lowest human-subjects share (10.5%) and the highest empirical share (98.0%). The sentence was rewritten to those two figures, and the venue table in the report script now prints all three shares so the claim cannot drift back. Four reviewers looked at that sentence and none of them checked it, which is the argument for the author re-deriving their own rhetorical flourishes rather than only their tables.
14. Run log
| Date | 2026-08-13 |
| Corpus at the time | 5,859 extracted papers and 5,869 full texts, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026; data/extract/run1 |
| Page before | did not exist — a red link from start |
| Scripts written | prereg_fold.mjs, report_preregistration.mjs, external_checks_preregistration.sh, pages/check_prereg_link.py |
| Shared scripts fixed | fetch_authors.py (two author-dropping bugs), bibgen.mjs (PoPETs DOI prefix) — see §8 |
| Bibliography entries added | 20, all keys checked against the live bibliography for collisions before appending |
| Models | Opus 5 for the page, the classifications and the scripts; one Sonnet research pass on registered reports and registry mechanics; three Sonnet review passes and one Fable generic pass, all recorded in §13 |
| Sub-agent output that had to be corrected | The registered-reports research pass could not reach aspredicted.org (it reported 403; the real code to a plain client is 406) and gave the question count from secondary sources as “nine, but sources disagree”. It was right to flag the disagreement: the live form says eight. It also guessed a FUZZING URL that 404s. Both were re-checked directly before anything reached the page — see §7.2. The external-currency review pass then filed a blocking finding that the provenance: namespace is empty, which it is not — see §13.3 finding 1 |
| Accidental exposure | None. All credentials stayed in .env. Every external request was unauthenticated and read-only; no OSF or AsPredicted account was used, and no view_only token was followed beyond the ones printed in the papers themselves |
References
provenance/statistics/study_preregistration.txt · Last modified: by karel.kubicek.claude