Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: statistics:study_preregistration ======

Working notes behind [[statistics:study_preregistration]] — every query with its population and denominator, the scripts and their unedited output, the hand classifications and their residue, the quotes checked against source papers, the external sources and how each was verified, and what could not be established. Corpus-level caveats that apply to every page on this site are on [[literature:corpus]] and are not restated here.

===== 1. What this page is backing =====

^ Item ^ Value ^
| Content page | [[statistics:study_preregistration]] — **new**, not an extension of anything |
| Report script | ''scripts/report_preregistration.mjs'' |
| Hand classification it depends on | ''scripts/prereg_fold.mjs'' — a **sense** map over 62 papers and an **adherence** map over 15 |
| External-fact re-check | ''scripts/external_checks_preregistration.sh'' |
| Runnable code published on the page | ''pages/check_prereg_link.py'' |
| Stale-number guard | ''scripts/check_page_numbers.mjs'', run whole-page, windowed and with ''--code'' |
| Data | ''data/extract/run1/extractions.jsonl'' — 5,859 papers, 7 venues, 2010–2026 — plus a regex sweep over all 5,869 ''paper.cols.txt'' full texts |
| Written | 2026-08-13, against the corpus as extended on 2026-08-11 (commit ''8a6b843'') |

**Creating, not extending.** The wiki had no ''statistics:'' page at all: ''statistics:study_preregistration'', ''statistics:hypothesis_testing'', ''statistics:pvalue_corrections'', ''statistics:regression'' and ''statistics:biases'' were all red links promised from [[start]]. This is the first of the five. Nothing was overwritten and nothing on a neighbouring page was contradicted; the page links to the other four as red links, which the site already does elsewhere.

**Reachability** needed no work: [[start]] already links ''[[Statistics:Study preregistration]]'' in its outline.

===== 2. The number the task started from, and what happened to it =====

The task specification quoted the dataset's most striking single figure: //"of 1,302 papers running statistical inference, ''preregistration'' appears in **2**"//. Both halves moved with the 2026-08-11 corpus extension, and the direction is worth recording because it is the opposite of what a stale figure usually does.

^ ^ Old corpus (4,322 papers) ^ This corpus (5,859 papers) ^
| ''inferential'' population | 1,302 | 1,762 |
| ''statistics[].kind == "preregistration"'' | 2 | **7** |
| ''artifacts.links[].kind == "preregistration"'' | not stated in the task | **12** |
| Either schema signal | — | 14 |
| Papers that actually preregistered (full text, hand-classified) | — | **15** |

The headline is therefore not "2 papers in the whole corpus" and the page does not say so. It is 15, of which 14 are user studies, and the interesting cut is not the raw count but **which population it is a share of**.

===== 3. Populations and denominators =====

Four populations are used, all defined as ''OVERVIEW.md'' defines them. ''crawled'' and ''empirical'' come from ''lib.mjs'''s ''POPULATIONS''; ''inferential'' and ''humanSubjects'' are spelled out in the report script because ''lib.mjs'' does not carry them.

^ Tag ^ Definition ^ N ^
| ''empirical'' | ''isEmpirical === true'' | 5,118 |
| ''crawled'' | ''crawlConfig !== null %%||%% studyTypes.includes('automated-web-crawl')'' | 1,120 |
| ''inferential'' | at least one ''statistics[].kind'' that is not ''descriptive-only'' | 1,762 |
| ''humanSubjects'' | ''participants.length > 0'' | 1,357 |

Every table on the content page names which of these it is a share of. The **wrong-numerator trap** that this page nearly published is recorded in §8.

===== 4. Every query, with its population =====

The report script prints all of these. Run it with:

<code bash>
cd /workspace/artifacts/wiki
node scripts/report_preregistration.mjs                    # every figure on the page
node scripts/report_preregistration.mjs --list --quotes    # the 62 classified papers, and the evidence quotes
node scripts/report_preregistration.mjs --wiki             # DokuWiki tables
bash scripts/external_checks_preregistration.sh            # re-fetch every external fact
node scripts/check_page_numbers.mjs pages/statistics_study_preregistration.txt out/prereg-report.txt --code
</code>

^ # ^ Question ^ Population ^ Result ^
| Q1 | How many papers carry ''statistics[].kind == "preregistration"''? | all 5,859 | 7 (0.4% of ''inferential'') |
| Q2 | How many carry an ''artifacts.links[].kind == "preregistration"''? | all 5,859 | 12 (0.2%) |
| Q3 | Union of Q1 and Q2 | all 5,859 | 14; intersection 5 |
| Q4 | How many full texts match ''/pre\s*-?\s*regist(er%%|%%ered%%|%%ering%%|%%ration%%|%%rations)/i'' after joining hyphens across line breaks and collapsing whitespace? | 5,869 ''paper.cols.txt'' files | 62 (1.1%) |
| Q5 | …of which, how many preregistered their **own study**? | the 62 | **15** (hand-classified, §5) |
| Q6 | How many of the 15 recruited participants? Ran a crawl? | the 15 | 14 and 1 |
| Q7 | Rate within each population | see §3 | ''humanSubjects'' 14/1,357 = 1.0%; ''inferential'' 15/1,762 = 0.9%; ''crawled'' 1/1,120 = 0.1% |
| Q8 | Per year and per venue | all 5,859 | first in 2021; IMC has none |
| Q9 | Is the 2025–2026 rise real? | ''humanSubjects'' | Fisher's exact, two-sided: 5/592 vs 9/303, //p// = 0.0215. 2024 vs 2025 alone, //p// = 0.1849 |
| Q10 | Which registry does each of the 15 name? | the 15 | OSF 11, AsPredicted 3, none 1 |
| Q11 | How many papers mention OSF at all, and how many of those in a preregistration context? | 5,869 full texts | 131 mention it, 14 also match Q4's regex, **117 (89.3%) do not** |
| Q12 | Does the paper say whether it followed its plan? | the 15 | 3 deviation, 1 no-deviation, 4 confirmatory/exploratory split, 7 silent (§5.3) |
| Q13 | Where does the schema disagree with the full text? | the 15 ∪ Q1 | schema missed 9 of 15; fired once on a paper that registered nothing |
| Q14 | The rest of the inference-hygiene stack, for scale | ''inferential'' | hypothesis-test 58.2%, correction 15.3%, effect-size 10.3%, power-analysis 4.9%, bayesian 1.6% |
| Q15 | Same, scoped to the population that needs it | the 1,025 hypothesis-test papers | correction 24.5%, power 7.0% |
| Q16 | Artifact links vs preregistration links | all 5,859 | 3,321 (56.7%) vs 12 (0.2%) |
| Q17 | How much of the corpus never gets past descriptives? | the 5,456 with any statistics tuple | 3,694 (67.7%) |
| Q18 | Is the practice happening under another name? | 5,869 full texts | ''/\\banalysis plan/'' matches 12; 8 outside the 62; **3 are real pre-committed plans with no registry** (§5.2) |
| Q19 | Does the practice reach this literature as a Registered Report? | 5,869 full texts | ''registered report'' matches 6 papers; **5 of them are citations to a FUZZING-workshop Registered Report**, and the sixth is Klemmer et al. discussing the idea |

===== 5. Hand classifications, and why they are not regexes =====

Neither classification in ''prereg_fold.mjs'' is a fold in the sense the other pages on this site use the word. There is no spelling variance to canonicalise. There is a **homograph**, and the only thing that separates the senses is the sentence — so every hit was read, and the deciding phrase is recorded in the source next to the slug.

The value of keeping them as code is the **residue**: a paper added to a future corpus that matches ''PREREG_RE'' and is in none of the lists prints as unclassified instead of silently joining whichever bucket a regex would have guessed.

==== 5.1 The sense map: 62 papers ====

^ Sense ^ Papers ^ Share of 62 ^
| ''other-sense'' — a pre-registered domain, OAuth redirect URI, FIDO device, test account, SIM box, chatbot command, enclave manifest, wake word… | 42 | 67.7% |
| ''study'' — the paper preregistered its own study | 15 | 24.2% |
| ''meta'' — argues about the practice without doing it | 3 | 4.8% |
| ''citation-only'' — the word appears only inside a cited paper's title | 2 | 3.2% |
| ''analysis-plan-only'' — committed to a plan, registered nothing | **3, and none of them in the 62** | — |
| **Unmapped residue** | **0** | **0.0%** |

The ''analysis-plan-only'' row is the one that does not fit the table, and that is informative. None of the three writes "pre-regist" at all — the string is absent from all three papers — so the full-text sweep cannot see them. Tang et al. (PETS 2021) were found through ''statistics[].kind''; the other two were found only by the **completeness sweep** in §5.3. What all three did was commit to an analysis plan, justify it by Type I error, and register it nowhere. They are held in their own map so that they neither inflate the 15 nor vanish.

^ Paper ^ What it says ^
| Tang et al., PoPETs 2021 | //"In an effort to limit the number of Type I errors, we committed to an analysis plan prior to analyzing the responses from our main study."// |
| //Please Unstalk Me//, PoPETs 2024 | //"We created an analysis plan to reduce the number of Type I errors in our analysis"//, and for the second study //"We followed our previously created analysis plan to limit any false positives (Type I errors)."// |
| //How does connecting online activities…//, PoPETs 2024 | Declines to report a significant one-tailed result: //"this was outside our analysis plan. We had no priors on which directions we should expect a change."// |

The 42 ''other-sense'' papers are not noise either. They are the reason a naive query of this corpus for preregistration overstates the practice by roughly a factor of four, and they span the whole window: an IMC 2010 DGA paper (defenders pre-registering domains), a CCS 2012 OAuth paper (pre-registered redirect URI), an IEEE S&P 2025 SSO paper with 8 hits (a service provider pre-registering with an identity provider), a PETS 2026 chatbot paper (pre-registered slash commands).

The three ''meta'' papers, in full:

^ Paper ^ What it does ^
| Hantke et al., IEEE S&P 2024 | Proposes a pre-registration board run by a trusted third party for server-side scanning, and surveys operators on it. **22 hits, the most in the corpus** |
| Klemmer et al., IEEE S&P 2025 | Interviews 24 usable-privacy-and-security researchers about preregistration and registered reports |
| Wei et al., USENIX Security 2024 | //"Consider study pre-registration"//, citing Cockburn et al.'s //HARK No More// |

==== 5.2 The completeness sweep, and the probes that failed ====

''PREREG_RE'' finds papers that use the **word**. The sweep for ''/\banalysis plan/'' asks whether the **thing** is being done under another name, and it is what turned up the two extra ''analysis-plan-only'' papers above: 12 papers match, 8 outside the 62, of which 3 are real pre-committed plans and 5 are section headings or incidental ("we discussed our analysis plan with peers").

Four further probes were run and returned nothing usable. They are kept in ''REJECTED_PROBES'' with their reasons so the next run does not re-run them hoping for a different answer:

^ Probe ^ Result ^
| ''pre-?specified'' / ''prespecified'' | 77 papers outside the 62, **none** about study design — "prespecified timestamps" (IP alias resolution), "pre-specified sanitizers", "pre-specified thresholds". A technical term in this literature, not a methodological one |
| ''a priori hypothes'' | 0 outside the 62 |
| ''registered (our%%|%%the) (study%%|%%hypotheses%%|%%analysis%%|%%protocol)'' | 0 outside the 62 |
| ''confirmatory (study%%|%%analysis)'' | 1 outside the 62 (NDSS 2015), and it means a follow-up interview round |
| ''clinicaltrials.gov'' / ''socialscienceregistry'' / ''aearegistry'' | 2 outside the 62, neither registering anything |

==== 5.3 The adherence map: 15 papers ====

Two keyword rules were written first and both were **wrong in opposite directions**, which is why this is also a hand list:

  * A rule matching ''/deviat/'' labelled //"our final analysis did not deviate from our original plan"// (Akgul et al.) as **reporting a deviation**.
  * A rule matching ''/as pre-?registered/'' labelled //"…attention check, as pre-registered"// (Crowder et al.) as an **adherence statement**, when it is a method descriptor for one exclusion rule.

The published four-way classification, with the deciding sentence for each paper, is in ''prereg_fold.mjs'' and printed by the report script.

===== 6. Unedited report output =====

<file text report_preregistration.txt>
corpus: 5859 extraction records
populations: empirical 5118 | crawled 1120 | inferential 1762 | humanSubjects 1357

========================================================================
A. THE SCHEMA SIGNAL
========================================================================
Signal                                     Papers  Of inferential (1762)  Of all (5859)
-----------------------------------------  ------  ---------------------  -------------
statistics[].kind == preregistration       7       0.4%                   0.1%
artifacts.links[].kind == preregistration  12      —                      0.2%
either                                     14      0.8%                   0.2%
both                                       5       —                      —

========================================================================
B. THE FULL-TEXT SWEEP
========================================================================
5869 paper.cols.txt files scanned (line-break hyphens joined, whitespace collapsed).
Regex                            Papers  Share of 5869
-------------------------------  ------  -------------
pre-regist*                      62      1.1%
aspredicted                      3       0.1%
osf.io / Open Science Framework  131     2.2%
registered report                6       0.1%

── What "pre-registration" means in those 62 papers ──
Sense               Papers  Share of 62
------------------  ------  -----------
study               15      24.2%
analysis-plan-only  0       0.0%
meta                3       4.8%
citation-only       2       3.2%
other-sense         42      67.7%
UNMAPPED RESIDUE    0       0.0%

residue: none — every hit is classified in prereg_fold.mjs.

── Completeness sweep: /\banalysis plan\b/ ──
   12 papers match; 8 of them are outside the 62 pre-regist* hits.
   3 of those are classified in ANALYSIS_PLAN_ONLY (a real pre-committed plan with no registry):
     PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies
     PETS/2024/please-unstalk-me-understanding-stalking-with-bluetooth-trackers-and-democratizi
     PETS/2024/how-does-connecting-online-activities-to-advertising-inferences-impact-privacy-p
   The remaining 5 were read and are section headings or
   incidental mentions ("we discussed our analysis plan with peers", "Analysis Plan." as a heading):
     2020 USENIX from-needs-to-actions-to-secure-apps-the-effect-of-requireme
     2022 PETS we-may-share-the-number-of-diaper-changes-a-privacy-and-secu
     2022 USENIX understanding-and-improving-usability-of-data-dashboards-for
     2023 USENIX understand-users-privacy-perception-and-decision-of-v2x-comm
     2026 WWW lurkers-interactors-creators-modeling-behavioral-and-ideolog

   Probes run and discarded, with why:
     pre-?specified / prespecified
       77 papers outside the 62, none about study design — "prespecified timestamps" (IP alias resolution), "pre-specified sanitizers", "pre-specified thresholds". A technical term in this literature, not a methodological one.
     a priori hypothes
       0 papers outside the 62.
     registered (our|the) (study|hypotheses|analysis|protocol)
       0 papers outside the 62.
     confirmatory (study|analysis)
       1 paper outside the 62 (NDSS 2015), and it uses the term for a follow-up interview round, not a pre-committed analysis.
     clinicaltrials.gov / socialscienceregistry / aearegistry
       2 papers outside the 62, both about matching patients to trials or leaking prompts — neither registers anything.

── "registered report" in the corpus: 6 papers, of which 5 cite a FUZZING-workshop Registered Report ──
   FUZZING-CITE  2023 NDSS assessing-the-impact-of-interface-vulnerabilities-in-co
   FUZZING-CITE  2024 CCS prompt-fuzzing-for-fuzz-driver-generation
   FUZZING-CITE  2024 NDSS large-language-model-guided-protocol-fuzzing
   other         2025 IEEE-SP transparency-in-usable-privacy-and-security-research-sc
   FUZZING-CITE  2026 IEEE-SP cottontail-large-language-model-driven-concolic-executi
   FUZZING-CITE  2026 NDSS mutato-enhancing-fuzz-drivers-with-adaptive-api-option-

========================================================================
C. THE 15 PAPERS THAT PREREGISTERED
========================================================================
Cut                           Papers  Share of 15
----------------------------  ------  -----------
recruited human participants  14      93.3%
ran an automated web crawl    1       6.7%
neither                       0       —

── Rate within each population that could have preregistered ──
Population                                       N     Preregistered  Share
-----------------------------------------------  ----  -------------  -----
humanSubjects — recruited participants           1357  14             1.0%
inferential — ran inference beyond descriptives  1762  15             0.9%
crawled — ran an automated web crawl             1120  1              0.1%
empirical — measured the world as it is          5118  15             0.3%
all extraction records                           5859  15             0.3%

── By year ──
Year   Papers in corpus  humanSubjects  Prereg (any)  Prereg (humanSubjects)  Share of humanSubjects
-----  ----------------  -------------  ------------  ----------------------  ----------------------
2010   119               18             0             0                       —
2011   116               12             0             0                       —
2012   151               29             0             0                       —
2013   125               25             0             0                       —
2014   166               31             0             0                       —
2015   190               37             0             0                       —
2016   182               35             0             0                       —
2017   231               52             0             0                       —
2018   254               48             0             0                       —
2019   402               92             0             0                       —
2020   404               83             0             0                       —
2021   379               106            1             1                       0.9%
2022   546               117            1             1                       0.9%
2023   719               198            1             1                       0.5%
2024   690               171            2             2                       1.2%
2025*  770               198            7             7                       3.5%
2026*  415               105            3             2                       1.9%
  2026 numerator warning: 3 papers preregistered but only 2 of them are humanSubjects. The wrong numerator over the humanSubjects denominator (105) gives 2.9% instead of 1.9%.
* 2025–2026 are provisional: CCS/IMC 2026 have not been held and IEEE S&P/WWW 2026
  abstracts are not in OpenAlex, so those venue-years are under-represented by construction.

── Is the 2025–2026 rise real, or is it the provisional slice? ──
  2021–2024: 5/592 humanSubjects papers preregistered (0.8%)
  2025–2026: 9/303 humanSubjects papers preregistered (3.0%)
  Fisher's exact test, two-sided: p = 0.0215
  2024 (2/171) vs 2025 (7/198) on its own: p = 0.1849  — not significant; the pooled test above is doing the work,
  and both of its later years are the under-represented ones.

  2010–2020 collapsed (the page prints one row for it): 2340 papers, 462 humanSubjects, 0 preregistered.

── By venue ──
Venue    Papers  humanSubjects  % humanSubjects  crawled  % crawled  % empirical  Preregistered
-------  ------  -------------  ---------------  -------  ---------  -----------  -------------
CCS      990     186            18.8%            163      16.5%      82.6%        1
IEEE-SP  767     208            27.1%            110      14.3%      84.2%        5
IMC      638     67             10.5%            132      20.7%      98.0%        0
NDSS     701     144            20.5%            129      18.4%      83.6%        1
PETS     510     215            42.2%            123      24.1%      93.5%        3
USENIX   1410    367            26.0%            221      15.7%      87.0%        4
WWW      843     170            20.2%            242      28.7%      87.8%        1

── Registry named, per preregistering paper ──
Registry                               Papers  Share of 15
-------------------------------------  ------  -----------
OSF (osf.io / Open Science Framework)  11      73.3%
AsPredicted                            3       20.0%
no registry string in the text         1       6.7%
  says it preregistered but names no registry (so the claim is not checkable):
    WWW/2026/maca-a-multi-agent-cognitive-adaptation-framework-for-human-agent-collaborative

  Also: 1 paper committed to an analysis plan without registering it anywhere —
  PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies.
  It is in ANALYSIS_PLAN_ONLY, not in STUDY, and contributes 0 to the 62 full-text hits
  because the string "pre-regist" never appears in it.

── Does the paper say whether it stuck to the plan? ──
What the paper tells the reader        Papers  Share of 15
-------------------------------------  ------  -----------
deviation-reported                     3       20.0%
no-deviation-stated                    1       6.7%
confirmatory-vs-exploratory-separated  4       26.7%
silent                                 7       46.7%
    IEEE-SP 2024 deviation-reported                      a-representative-study-on-human-detectio
        "...but, differing from our preregistration, only run a regression analysis on the latter."
    CCS 2025     deviation-reported                      empowering-parents-to-support-childrens-
        "In the preregistration, we initially planned to use analysis of variance (ANOVA) with repeated measures. Instead, we decided to use generalized linear regressions..."
    NDSS 2025    deviation-reported                      the-power-of-words-a-comprehensive-analy
        "We deviated from our preregistration for permissions and chose to treat them as random effects rather than fixed effects with difference coding."
    USENIX 2025  no-deviation-stated                     as-advertised-understanding-the-impact-o
        "our final analysis did not deviate from our original plan."
    PETS 2022    confirmatory-vs-exploratory-separated   increasing-adoption-of-tor-browser-using
        "In addition to these pre-planned hypothesis tests (S 4.1), we also conducted several exploratory analyses (S 4.2-4.8)."
    IEEE-SP 2025 confirmatory-vs-exploratory-separated   restricting-the-link-effects-of-focused-
        "...five pre-registered hypotheses. We further conducted exploratory analyses (also described in the pre-registration)..."
    USENIX 2026  confirmatory-vs-exploratory-separated   you-know-why-but-still-rely-the-impact-o
        "we first present the results in line with our research questions and hypotheses from the pre-registration. Then, we report on additional exploratory analyses."
    WWW 2026     confirmatory-vs-exploratory-separated   maca-a-multi-agent-cognitive-adaptation-
        "We pre-registered the primary comparison ... All other contrasts were treated as exploratory."
    PETS 2021    silent                                  validity-and-reliability-of-the-scale-in
    IEEE-SP 2023 silent                                  when-and-why-do-people-want-ad-targeting
    PETS 2024    silent                                  simply-tell-me-how-on-trustworthiness-an
        Says "as pre-registered" of one exclusion rule, but makes no statement about the analysis as a whole.
    IEEE-SP 2025 silent                                  lets-get-visual-testing-visual-analogies
    IEEE-SP 2025 silent                                  not-the-right-question-a-study-on-attitu
    USENIX 2025  silent                                  vulnerability-of-text-matching-in-ml-ai-
    USENIX 2026  silent                                  chameleon-channels-measuring-youtube-acc

── "It is on OSF" is not "it was preregistered" ──
Cut                                                     Papers  Share of 131
------------------------------------------------------  ------  ------------
mentions osf.io or Open Science Framework               131     100.0%
…and also matches pre-regist*                           14      10.7%
…OSF appears with no pre-registration wording anywhere  117     89.3%

========================================================================
D. SCHEMA vs FULL TEXT
========================================================================
                                                               Papers
-------------------------------------------------------------  ------
preregistered (this script) and statistics.kind agrees         6
preregistered but statistics.kind MISSED it                    9
statistics.kind fired but the paper did not publicly register  1

  missed by statistics.kind:
    PETS/2021/validity-and-reliability-of-the-scale-internet-users-information-privacy-concern
    PETS/2022/increasing-adoption-of-tor-browser-using-informational-and-planning-nudges
    PETS/2024/simply-tell-me-how-on-trustworthiness-and-technology-acceptance-of-attribute-bas
    CCS/2025/empowering-parents-to-support-childrens-online-security-and-privacy-findings-fro
    IEEE-SP/2025/not-the-right-question-a-study-on-attitudes-toward-client-side-scanning-with-sec
    IEEE-SP/2025/restricting-the-link-effects-of-focused-attention-and-time-delay-on-phishing-war
    NDSS/2025/the-power-of-words-a-comprehensive-analysis-of-rationales-and-their-effects-on-users-permission-decisions
    USENIX/2025/vulnerability-of-text-matching-in-ml-ai-conference-reviewer-assignments-to-collu
    USENIX/2026/you-know-why-but-still-rely-the-impact-of-explainable-ai-on-trust-task-load-and
  statistics.kind fired, not a public registration:
    PETS/2021/defining-privacy-how-users-interpret-technical-terms-in-privacy-policies  — PETS 2021 — "In an effort to limit the number of Type I errors, we committed to an analysis plan prior to analyzing the responses from our main study." No registry, no link, no timestamp. Found via statistics.kind; the string "pre-regist" never appears in the paper.

========================================================================
E. THE ADJACENT PRACTICES, FOR SCALE
========================================================================
papers with any statistics tuple: 5456; of those, 3694 (67.7%) report nothing beyond descriptives.

statistics.kind                 Papers (of inferential 1762)  Share
------------------------------  ----------------------------  -----
hypothesis-test                 1025                          58.2%
multiple-comparison-correction  269                           15.3%
effect-size                     182                           10.3%
power-analysis                  87                            4.9%
bayesian                        28                            1.6%
preregistration                 7                             0.4%

Of the 1025 papers that ran a hypothesis test  Papers  Share
---------------------------------------------  ------  -----
also report a multiple-comparison correction   251     24.5%
also report a power analysis                   72      7.0%

Of the 1357 papers that recruited participants  Papers  Share
----------------------------------------------  ------  -----
ran a hypothesis test                           482     35.5%
report a power analysis                         74      5.5%
preregistered                                   14      1.0%

Open-science practice                          Papers  Share of 5859
---------------------------------------------  ------  -------------
released an artifact link of the authors' own  3321    56.7%
linked a preregistration                       12      0.2%

========================================================================
F. EXTERNAL FIGURES ON THE PAGE (not from this corpus)
========================================================================
  119                          Hantke et al., IEEE S&P 2024, Table 3 — surveyed operators
  36.1                         Hantke et al., Table 3, Q6 "comfortable (5)"
  21.8                         Hantke et al., Table 3, Q6 "somewhat comfortable (4)"
  37.8                         Hantke et al., Table 3, Q8 "comfortable (5)"
  35.3                         Hantke et al., Table 3, Q8 "somewhat comfortable (4)"
  57.9                         this page's arithmetic: 36.1 + 21.8
  73.1                         this page's arithmetic: 37.8 + 35.3
  4.15                         Hantke et al., §5.1 — NGO/white-hat TTP competence, mean (the §5.2 heading is interleaved from the facing column in .cols)
  1.1                          Hantke et al., §5.1 — same, standard deviation
  3.79                         Hantke et al., §5.1 — academic-institution TTP competence, mean
  1.13                         Hantke et al., §5.1 — same, standard deviation
  24                           Klemmer et al., IEEE S&P 2025 — interviewees
  44                           api.osf.io/v2/schemas/registrations — template count, 2026-08-13
  22                           api.osf.io/v2/registrations/fg9pm — non-upload question fields
  11                           same — of those, answered
  1,005                        same — words across the answered fields
  2.00                         aspredicted.org — "Version of AsPredicted Questions"
  8                            aspredicted.org — numbered questions on the live form
  4                            help.osf.io — maximum embargo, years
  74,302                       AsPredicted registration number, read live
  138,374                      AsPredicted registration number, read live
  170,202                      AsPredicted registration number, read live
  74 / 138 / 170               digit fragments of the three AsPredicted numbers above
  10                           OSF identifiers cited as preregistrations by corpus papers: 8 registrations + 2 projects
  30                           urllib timeout in seconds, inside the page's downloadable script
  10.17605                     OSF DOI prefix
  200 / 401 / 403 / 406        HTTP status codes; aspredicted.org answers a plain curl with 406, dl.acm.org with 403
  0.05 / 20 / 40 / 8 / 34 / 3  illustrative examples in prose, not measurements
  2019-08-08 … 2025-04-16      OSF date_registered, read live from api.osf.io
</file>

===== 7. External sources, and how each was verified =====

Everything here was **fetched on 2026-08-13**, not recalled, and every command is in ''scripts/external_checks_preregistration.sh''. Its unedited output:

<file text external_checks_preregistration.txt>

== 1. Every preregistration URL cited by a corpus paper, and what it actually is
GUID / URL                             OSF type       template                                       registered   withdrawn
osf.io/bc42h                           registrations  OSF Preregistration                            2021-03-24   False
osf.io/xy6v5                           registrations  OSF Preregistration                            2022-05-12   False
osf.io/5jd8a                           registrations  Preregistration Template from AsPredicted.org  2019-08-08   False
osf.io/st6pz                           registrations  OSF Preregistration                            2023-08-03   False
osf.io/7zprb                           registrations  OSF Preregistration                            2023-07-10   False
osf.io/q4rj3                           nodes          (project, not a registration)                  -            None
osf.io/d6h7b                           registrations  HTTP 401 (private / view-only link)            -            -
osf.io/5pywm                           nodes          (project, not a registration)                  -            None
osf.io/zqy7b                           registrations  OSF Preregistration                            2024-09-27   False
osf.io/fg9pm                           registrations  OSF Preregistration                            2025-04-16   False

== 1b. DOI minted for each OSF registration
   osf.io/bc42h    -> 10.17605/OSF.IO/BC42H
   osf.io/xy6v5    -> 10.17605/OSF.IO/XY6V5
   osf.io/5jd8a    -> 10.17605/OSF.IO/5JD8A
   osf.io/st6pz    -> 10.17605/OSF.IO/ST6PZ
   osf.io/7zprb    -> 10.17605/OSF.IO/7ZPRB
   osf.io/zqy7b    -> 10.17605/OSF.IO/ZQY7B
   osf.io/fg9pm    -> 10.17605/OSF.IO/FG9PM

== 2. The three AsPredicted registrations cited by corpus papers
   https://aspredicted.org/HXF_Z92
      #170,202  registered 2024/04/10 18:29 (PT)  anonymous=False
      canonical URL now: https://aspredicted.org/j3n9-kxzf.pdf
   https://aspredicted.org/blind.php?x=MTD_GH1
      #74,302  registered 2021/09/09 15:03 (PT)  anonymous=True
      canonical URL now: https://aspredicted.org/mryz-wf5h.pdf
   https://aspredicted.org/rk8xe.pdf
      #138,374  registered 2023/07/13 15:23 (PT)  anonymous=False
      canonical URL now: not shown

== 3. AsPredicted's questions, verbatim from the live form, with the form version
   1) Have any data been collected for this study already?
   2) What's the main question being asked or hypothesis being tested in this study?
   3) Describe the key dependent variable(s) specifying how they will be measured.
   4) How many and which conditions will participants be assigned to?
   5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
   6) Describe exactly how outliers will be defined and handled, and your precise rule(s) for excluding observations.
   7) How many observations will be collected or what will determine sample size?
   8) Anything else you would like to pre-register?
   8 numbered questions found. Version of AsPredicted Questions: 2.00

== 4. OSF registration templates that exist today
   44 templates total
   Generalized Systematic Review Registration       active=True
   OSF Preregistration                              active=True
   Open-Ended Registration                          active=True
   Prereg Challenge                                 active=False
   Preregistration Template from AsPredicted.org    active=True
   Qualitative Preregistration                      active=True
   Registered Report Protocol Preregistration       active=True
   Secondary Data Preregistration                   active=True

== 5. The OSF Preregistration template question list (which a measurement study has to map onto)
   ## Overview
      - Research questions or hypotheses
      - Foreknowledge of data or evidence
      - Explanation of foreknowledge and managing unintended influences
   ## Research Design
      - Study type
      - Intention for causal interpretation
      - Blinding of experimental treatments
      - Additional blinding during research or analysis
      - Study design
      - Randomization
   ## Sampling
      - Data collection procedures
      - Data collection procedures - File upload
      - Sample size
      - Sample size rationale
      - Starting and stopping rules
   ## Variables
      - Manipulated variables
      - Measured variables
      - Measured variables - File upload
      - Indices
      - Indices - File upload
   ## Analysis Plan
      - Statistical models
      - Statistical models - File upload
      - Transformations
      - Inference criteria
      - Data inclusion and exclusion
      - Missing data
      - Other planned analysis
   ## Other
      - Context and additional information

== 6. The one measurement-study registration, read in full (USENIX Sec 2026, Chameleon Channels)
   template: OSF Preregistration   registered: 2025-04-16
   11 of 22 question fields answered (6 file-upload slots excluded); blank: q5, q7, q13, q14.question, q16.question, q18, q19, q20, q21, q22, q23
   1005 words across the answered fields
   q3: Observational Study - Data is collected from study subjects that are not randomly assigned to a treatment. This includes surveys, “natural experiments,” and regression discontinuity designs.
   q4: ["No blinding is involved in this study."]
   q8: Registration prior to analysis of the data
   q9: We have designed scrapers to collect the data and databases to store the data. Because of this procedure, the team has been exposed to small portions of the data. All interaction, however, has been just for collection an
   q12: Our sample of advertised channels constitutes all observed channels for sale in the platform. Our goal is to be exhaustive. For our Social Blade sample (~1.4M channels), our goal was to find a large enough number that wo

== 7. Registered reports: no venue in this corpus runs them. Pages checked:
   200 https://www.sigsac.org/ccs/CCS2026/call-for/call-for-papers.html
   200 https://sp2026.ieee-security.org/cfpapers.html
   200 https://www.ieee-security.org/Calendar/cfps/cfp-USENIXSec2026.html
   200 https://www.ndss-symposium.org/ndss2026/submissions/call-for-papers/
   200 https://petsymposium.org/cfp26.php
   200 https://conferences.sigcomm.org/imc/2026/cfp/
   200 https://www2026.thewebconf.org/calls/research-tracks.html
   200 https://soups.page/cfp.html
   200 https://www.ndss-symposium.org/ndss2022/cfp-fuzzing-workshop/
   200 https://conf.researchr.org/track/issta-2023/fuzzing-2023
   200 https://2026.msrconf.org/track/msr-2026-registered-reports
   200 https://conf.researchr.org/track/esem-2024/esem-2024-registered-reports
   200 https://emsejournal.github.io/registered_reports/
   200 https://chi2026.acm.org/contributions-to-chi/
   200 https://www.usenix.org/conference/usenixsecurity26/call-for-papers
   403 https://dl.acm.org/journal/tosem/registered-papers  (EXPECTED-403: bot protection)

   Registered Reports are run by FUZZING (ndss-symposium.org/ndss2022 and
   conf.researchr.org/issta-2023), MSR 2026, ESEM 2024 and the EMSE journal.
   CHI 2026 does NOT: its contribution-types page mentions neither registered
   reports nor preregistration. ACM TOSEM is reported to run Registered Papers
   but COULD NOT BE VERIFIED: dl.acm.org, tosem.acm.org and the static PDF route
   all 403 every non-browser client, so that claim rests on the FUZZING track
   page, not on ACM s. None of these is one of this corpus seven venues.
</file>

==== 7.1 What each check established ====

^ Claim on the page ^ How it was verified ^
| 8 of the 10 OSF identifiers cited as preregistrations are registrations; 2 are ordinary projects | ''api.osf.io/v2/guids/<guid>/'' returns a 302 whose ''Location'' is ''/registrations/…'' or ''/nodes/…''. The typed endpoint then gives ''registration_supplement'', ''date_registered'', ''withdrawn'' and ''embargoed'' |
| ''osf.io/d6h7b'' is a registration but not public | ''/guids/d6h7b/'' redirects to ''/registrations/'', and ''/registrations/d6h7b/'' returns HTTP 401 |
| Every public OSF registration gets a DOI | ''api.osf.io/v2/registrations/<guid>/identifiers/'' returned ''10.17605/OSF.IO/<GUID>'' for all seven public ones |
| 44 OSF registration templates; ''Prereg Challenge'' is ''active=false'' | ''api.osf.io/v2/schemas/registrations/?page[size]=50'' |
| The OSF Preregistration question list | Same endpoint; the ''page-heading'' and ''question-label'' blocks of the template named ''OSF Preregistration'' |
| The four Chameleon Channels answers | ''api.osf.io/v2/registrations/fg9pm/'', field ''registration_responses''. 11 of 22 non-upload fields answered |
| Maximum embargo of four years; registrations immutable and only withdrawable | ''help.osf.io'' registration articles |
| AsPredicted asks **eight** numbered questions, form **version 2.00** | Read off the rendered registration at ''aspredicted.org/HXF_Z92'', which prints both. ''aspredicted.org'' 403s ''WebFetch'' but serves ''curl'' with a browser User-Agent |
| AsPredicted #74,302 is still marked anonymous | Same page format at ''aspredicted.org/blind.php?x=MTD_GH1'': //"This pre-registration is currently anonymous to enable blind peer-review. It has 4 authors."// |
| No venue in this corpus runs Registered Reports | Each venue's current CfP fetched and read: CCS 2026, IEEE S&P 2026, USENIX Security 2026, NDSS 2026, PoPETs 2026, IMC 2026, TheWebConf 2026, SOUPS 2026 |
| FUZZING runs a two-phase Registered Reports process | The workshop's own CfP at ''ndss-symposium.org/ndss2022/cfp-fuzzing-workshop/'' and the ISSTA 2023 track page |
| IMC's Replicability Track is about replicating //previously published// results | ''conferences.sigcomm.org/imc/2026/cfp/'': //"submissions that aim to reproduce or replicate network measurement results that have been previously published"// |
| Hantke et al.'s operator percentages | Table 3 of the paper itself, ''paper.cols.txt'' — not a secondary summary. The two sums (57.9%, 73.1%) are this page's arithmetic on the paper's per-column figures and are labelled as such in a footnote |

==== 7.2 Claims rejected, and why ====

^ Claim ^ Why it is not on the page ^
| **"AsPredicted asks nine questions."** | Several guides written by people close to the platform say nine, and a research sub-agent reported the discrepancy honestly rather than picking one. The live form identifies itself as ''Version of AsPredicted Questions: 2.00'' and prints eight. The page states eight, footnotes where it read them, and notes the discrepancy rather than hiding it |
| **The Center for Open Science's participating-journals list as evidence about CS adoption.** | ''cos.io/initiatives/registered-reports'' surfaces only //Frontiers in Computer Science// from the whole of computing. It misses MSR, ESEM, EMSE and ACM TOSEM, which all run Registered Reports independently of COS's programme. Using it would have produced the false claim that CS has essentially no Registered Reports |
| **''fuzzingworkshop.github.io'' as the FUZZING source.** | Returns HTTP 404. The claim rests on the ''ndss-symposium.org'' and ''conf.researchr.org'' pages instead |
| **A USENIX Security CfP quotation taken from the mirror alone.** | ''usenix.org'' 403s ''WebFetch'', so a sub-agent read it through ''ieee-security.org'''s mirror. That was re-checked directly with ''curl -A'' before anything went on the page (HTTP 200) |
| **Naming the authors of ''osf.io/d6h7b'''s paper.** | A draft attributed it to "Amft et al." on no evidence. The page now names the paper by venue and topic instead |

===== 8. Bugs caught in this run =====

Three of these are in **shared scripts** and affect pages other than this one.

^ # ^ Bug ^ Where ^ Fix ^
| 1 | **A numerator and a denominator from different populations.** The per-year table put //every// preregistering paper over the ''humanSubjects'' denominator. Chameleon Channels has no participants, so 2026 read as 3/105 = 2.9% when the honest figure is 2/105 = 1.9% | ''report_preregistration.mjs'' | Two numerator columns, and the report prints the warning explicitly so the guard can see it |
| 2 | **''scripts/fetch_authors.py'' silently dropped authors from USENIX landing pages**, in two independent ways: it split the whole author string on commas, so any author immediately after a semicolon was glued onto the preceding affiliation and filtered out; and it never stripped a leading ''"and "'', so the **last** author of every multi-author group was dropped. Measured against four live pages: 1 of 5 authors lost on ''usenixsecurity25/presentation/akgul'', 3 of 6 on ''usenixsecurity24/presentation/wei-miranda-solk'', 2 of 3 on ''usenixsecurity26/presentation/cuevas'' | shared | Rewritten to split on '';'' first. **Every USENIX BibTeX entry already in ''literature:bibliography'' that came from the old parser may be missing authors**; a work item to re-derive and diff them is filed |
| 3 | **''scripts/bibgen.mjs'' hardcoded the PoPETs DOI prefix ''10.56553'' for every year.** PoPETs has changed prefix twice. Probed against Crossref: ''10.1515'' for 2015–2018, ''10.2478'' for 2019 through ''popets-2022-0058'', ''10.56553'' from ''popets-2022-0059'' onward (both sides of the boundary checked directly) | shared | A ''popetsPrefix(year, article)'' function with the boundaries and the probe date in a comment. Every PoPETs DOI currently live on the wiki resolves, so this had been caught by hand each time |
| 4 | The published ''check_prereg_link.py'' matched ''https'' as an OSF GUID, because its regex allowed a bare ''^'' alternative. Every full URL reported HTTP 404 | ''pages/check_prereg_link.py'' | Anchored on ''osf.io/'', with a separate pattern for a bare identifier. The ''%%<code>%%'' block on the page is the output of the **fixed** script |

===== 9. Quotes checked against the source =====

Eighteen quotes were checked against ''paper.cols.txt'' with hyphens joined across line breaks and whitespace collapsed. **16 matched verbatim or through an 8-word run.** Two did not, and both are the known column-splice artefact rather than a fabrication — each was then confirmed fragment by fragment:

^ Paper ^ Quote ^ Verdict ^
| Cuevas et al., USENIX Security 2026 | //"we pre-registered our regression hypotheses, methods, and analysis plan prior to conducting the study"// | Present, spliced. ''.cols'' interleaves it with the neighbouring column (//"we pre-registered our regression // we provided the LLM with examples…"//). Fragments ''"we pre-registered our regression"'', ''"hypotheses, methods, and anal"'', ''"prior to conduct"'' and ''"The pre-registration is available at"'' all found |
| Groß, PETS 2021 | //"our analysis is a pre-registered confirmatory study"// | Present, spliced. Fragments ''"analysis is a pre-registered confirmatory study"'' and ''"OSF: https://osf.io/5pywm"'' both found |

The other sixteen — Tang et al.'s analysis-plan sentence, Frank et al.'s OSF sentence, the MACA sentence, Akgul et al.'s no-deviation sentence, Zimmermann et al.'s OSF sentence, Lee et al.'s AsPredicted sentence, three Hantke et al. sentences, two Klemmer et al. sentences, Petelka et al.'s OSF sentence, Roch et al.'s pre-registration sentence, Elbitar et al.'s deviation sentence, Chen et al.'s ANOVA sentence and Wei et al.'s recommendation — matched directly.

===== 10. What could not be established =====

  * **Whether the 15 is anywhere near the real rate in this field.** The corpus has no **SOUPS**, which is the single venue where preregistered usable-privacy work is most likely to appear, and no EuroS&P, ACSAC, RAID, AsiaCCS, WPES or CHI. The page says the 15 is a lower bound and probably a substantial one; it does not guess how much.
  * **Whether preregistration changes what gets published here.** That needs the denominator of registered-but-never-published studies. OSF registrations are searchable and nobody has looked; the page files it as an open question rather than speculating.
  * **The registration behind ''osf.io/d6h7b''.** It exists and is a registration, but the API returns 401 and only the paper's ''view_only'' link opens it, so its template and date could not be independently confirmed.
  * **Whether the OSF Preregistration template is workable for crawls in general.** One example exists. Everything the page says about mapping a crawl onto the template generalises from Chameleon Channels alone, and the page says so.
  * **The exact ''q''-number → question-label mapping of the OSF template.** ''registration_responses'' uses ''q2''…''q23'' and the schema endpoint returns ''registration_response_key: null'' for every block in version 4, so the mapping cannot be derived programmatically. The four answers quoted on the page were identified **by their content**, which is unambiguous for those four; the page therefore quotes answers rather than claiming to name which numbered blanks were left empty.
  * **Whether the OSF Preregistration template used by the 2019 and 2021 registrations is the same version 4 the API returns today.** It is very likely not. The page does not claim it is.
  * **Whether the "no venue runs Registered Reports" claim holds for workshops co-located with these venues.** Only main-conference CfPs were read. FUZZING is the one workshop checked, and only because the corpus pointed at it.

===== 11. Judgement calls =====

  - **New page, not a section of an existing one.** The alternative was a section of a broader ''statistics:'' page. Rejected because [[start]] already promises this exact page id, and because the measurement-specific content — the OSF template mapping, the registry-versus-project distinction, the registered-reports venue table — does not fit under a heading in a general statistics page without swamping it.
  - **The population is 15, not 14 and not 7.** The schema union is 14 and ''statistics.kind'' alone is 7. The published population is the full-text sweep's ''study'' sense, which is 15, because the schema demonstrably misses 9 of them. This means **the page's headline number cannot be reproduced from the extraction alone** — it needs the full text — and §4 says so.
  - **The three ''analysis-plan-only'' papers are excluded from the 15.** All three pre-specified an analysis plan and registered nothing. Including them would make the count 18 and the page's central distinction — that a commitment nobody else can check is not a preregistration — incoherent. A reasonable person could include them with a footnote. They are not dropped: they have their own section on the content page, because a reader who already writes an analysis plan is the reader most likely to act on this page.
  - **The page is framed as advice, not as a lament.** The corpus finding is "almost nobody does this". The page's framing is "here is how, and here is the one worked example", because a page that only reports the rate is a fact about the literature and not guidance. That is a deliberate choice about tone and it is the main thing a reviewer might disagree with.
  - **Three of the fifteen are named for doing something badly**, by name: MACA for claiming preregistration with no registry, and the two papers whose "preregistration" link resolves to a mutable OSF project. This is a judgement about whether a systematisation page should name names. It does, because the alternative — describing the failure abstractly — leaves the reader unable to check it, and every one of these is verifiable in a single command that the page ships.
  - **Currency: preregistration is dated as "emerging", not "current practice".** The corpus shows 0 papers before 2021 and a jump in 2025–2026 — but 2025–2026 are the provisional venue-years, and the jump is significant only when they are pooled (//p// = 0.0215) and not year-on-year (//p// = 0.1849). The page states the pooled test, the year-on-year test and the reason to distrust both, and tells the reader to quote "about 1% and rising" rather than 3.5%.
  - **The FUZZING workshop is given a whole paragraph** even though it is not in the corpus and is not a security venue, because it is the only working precedent anyone proposing a Registered Reports track here could point at. That is a call about scope: the page is about the seven venues, and this is outside them.
  - **No ''~~DISCUSSION~~'' on provenance pages.** Comments belong on the content page. This follows the convention set by the first provenance page on this site.

===== 12. Guard runs =====

''check_page_numbers.mjs'' was run whole-page, windowed on ''Use in Publications'', and with ''--code''.

^ Run ^ Unaccounted ^ What it found ^
| 1 | 31 | Mostly external figures the report never printed. It also surfaced two real gaps: the collapsed 2010–2020 row (2,340 / 462) and the 67.7% descriptive-only figure were **the page's own arithmetic and ''OVERVIEW.md'''s number**, not the report's. Both are now computed by the report script |
| 2 | 4 | The 2.9%-versus-1.9% sentence — the page's own arithmetic on the wrong-numerator bug — plus digit fragments of the AsPredicted registration numbers and the ''timeout=30'' inside the embedded Python. An ''F. EXTERNAL FIGURES'' section was added to the report, listing each non-corpus number with its primary source, so they are accounted for without polluting the guard's shared ''ALLOW'' map |
| 3 | 0 (whole-page, ''--code'', and windowed) | Clean |

The embedded ''%%<file python check_prereg_link.py>%%'' block was compared line-by-line against ''pages/check_prereg_link.py'': identical. The ''%%<code>%%'' block is the verbatim output of a real run, including the ''$ echo $?'' showing exit status 1.

''check_page_numbers.mjs'' is **not** run against this provenance page: its numbers are meta — section numbers, counts quoted precisely because they were rejected, old-corpus figures kept for comparison — so the guard's premise does not apply.

===== 13. Review pass, 2026-08-13 =====

Four reviewers, each handed the page text, the scripts, their unedited output and these notes, and each told explicitly that the author's context might not be exhaustive. The three focused passes ran in parallel against a frozen snapshot; the generic pass ran afterwards, against the corrected text. Every finding is recorded with whether it was accepted, because a rejection is the only record of whether a reviewer earned its slot.

**Which snapshot each pass saw.** ''out/frozen_prereg_r1.txt'' (43,680 bytes) went to the three focused passes. Two things changed under them while they ran, both caught by the author rather than by a reviewer, and both recorded here so the verdicts below can be read against the right text: the intro's unsourced "it costs about a day" became a measured "about a thousand words" with a footnote, and "four papers cite a FUZZING paper whose title ends (Registered Report)" became **five**, after the claim was measured rather than asserted (the count is now printed by the report script). The generic pass saw ''out/frozen_prereg_r3.txt'' (47,460 bytes), which includes every fix below.

==== 13.1 Figures versus the script (Sonnet) ====

^ # ^ Finding ^ Verdict ^
| 1 | Re-ran the report script in all three modes and diffed byte-for-byte against the committed output: **zero diffs**. Re-ran ''external_checks_preregistration.sh'' live: 121 of 121 lines matched. | Noted. |
| 2 | Independently re-derived eleven corpus figures with its own ''jq''/Python rather than through the script — ''humanSubjects'' 1,357, ''inferential'' 1,762, 3,321 (56.7%), 12, 7, 1,025, 251 (24.5%), 72 (7.0%), 482 (35.5%), 74 (5.5%), 138 posters, 251 short records — all matched. | Noted. |
| 3 | Re-swept ''paper.cols.txt'' for ''pre-regist*'' and OSF **from scratch** with its own regex: 62 and 131, and the 62 slugs diffed **identical** against ''prereg_fold.mjs'''s classified set — zero residue, zero missing. | Noted. This is the check that most needed doing, because the population of 15 cannot be reproduced from the extraction alone. |
| 4 | Recomputed Fisher's exact independently via ''math.comb'': //p// = 0.02152510 and //p// = 0.18494275, matching the published 0.0215 and 0.1849. | Noted. |
| 5 | Spot-checked **nine** sense classifications against source text (eight were asked for), including reproducing the exact 8-hit count in the IEEE S&P 2025 SSO paper and confirming that ''defining-privacy'' contains **zero** ''pre-regist'' occurrences. All correctly bucketed. | Noted. |
| 6 | Ran the published ''check_prereg_link.py'' live with the page's own arguments: output matches the ''%%<code>%%'' block exactly, including exit status 1. The ''%%<file>%%'' block diffs clean against ''pages/check_prereg_link.py''. | Noted. |
| 7 | **The report script's internal provenance annotation cites Hantke et al. "§5.2" for the trusted-third-party competence means; the paragraph is actually the tail of §5.1.** The "5.2. Ethical Challenges" heading is interleaved from the facing column in ''.cols'', which is how the wrong section number got recorded. | **Accepted.** Corrected in the script. The section number never appeared on the published page — it lived only in the report's ''EXTERNAL FIGURES'' bookkeeping — but a wrong citation inside an audit trail is exactly the kind of thing that gets copied out of one later. |
| — | Also verified the ''Secondary Data Preregistration'' template's nine named fields against the live OSF schema API, the register-early table's dates against ''api.osf.io'' and ''aspredicted.org'', and that Chameleon Channels' extraction record really carries a non-null ''crawlConfig'' and an empty ''participants''. | Noted. |

==== 13.2 Citations and quotes (Sonnet) ====

^ # ^ Finding ^ Verdict ^
| 1 | All 20 citation keys resolve; none collides with the 166 live-bibliography entries; no duplicates within the additions file or within the live file. | Noted. |
| 2 | **Every one of ~20 quotations found verbatim.** One (Petelka et al.) needed an inline footnote-marker digit stripped, which is an extractor artefact and not a page error. | Noted. |
| 3 | Re-fetched all six USENIX/NDSS author bylines specifically to test whether the ''fetch_authors.py'' bug had left residue: all six match the bibliography exactly, in order, and match each paper's own full-text header. | **The most valuable check of this pass**, because it is the one that could have caught a defect the author introduced and then believed fixed. |
| 4 | ''seong2026_maca'' gives the second author as "Hayoung Oh"; Crossref's own record for the DOI says "Ha Young Oh". | **Accepted.** Corrected to the DOI's metadata. |
| 5 | Two titles were re-cased or re-punctuated away from Crossref: ''crowder2024_simply'' (sentence case in Crossref) and ''zimmermann2025_visual'' (a dash, not a colon). | **Accepted.** Both restored to the Crossref strings. Neither changed which paper was cited, but a bibliography that silently normalises publisher metadata is harder to check than one that does not. |
| 6 | Flagged the ''%%/* ... */%%'' editorial block in the References section as containing a literal ''%%{[key]}%%'', then verified against a live page using the same boilerplate that DokuWiki hides it and it injects no reference. | **Rejected, correctly, by the reviewer itself.** Recorded because a reviewer that chases down its own false positive rather than filing it is worth more than one that does not. |
| 7 | Independently confirmed the Hantke Table 3 arithmetic, the Klemmer quotes, all four Chameleon Channels registration answers against the live OSF API, the field-by-field 11-of-22 count, the three AsPredicted registrations including the still-anonymous flag, and the FUZZING phase-1 quotes **against the 2022 first-edition CfP** rather than the current one. | Noted. |

==== 13.3 External currency (Sonnet) ====

^ # ^ Finding ^ Verdict ^
| 1 | "**Blocking**: [[provenance:statistics:study_preregistration]] is cited three times and does not exist; the ''provenance:'' namespace on the live wiki has zero pages." | **Rejected on both halves, and it is the most instructive finding of the review.** The page did not exist because it had not been saved yet — the two pages are published together. And the namespace is **not** empty: ''provenance:privacy:requests'', ''provenance:programming:crawler'' and ''provenance:design:sampling'' all return real source from ''?do=export_raw''. This is the trap already written into the workdir notes: the wiki's index does not list the ''provenance:'' namespace, and ''?do=export_raw'' on a page that does not exist returns the **HTML error page**, so neither the index nor a status code is an existence test. Two independent runs have now been misled by it. |
| 2 | **The CCS 2026 row understates the CfP**: CCS now requires a mandatory "Open Science" appendix listing where the artifacts are, available within 3 days of submission or the lack explained — structurally the same mechanism the page correctly credits USENIX with, and the row said only "optional artifact evaluation with badges". | **Accepted**, and it had already been corrected from the same source before this review landed — the author re-read the CCS and SOUPS CfPs while waiting. The row now quotes the CfP. The SOUPS row gained the detail that its call names **OSF itself** as a suggested artifact host, which is direct evidence for the page's "OSF is a file host here" argument. |
| 3 | The footnote said ''aspredicted.org'' "returns HTTP 403 to some automated clients". A plain ''curl'' gets **406**. | **Accepted.** Measured: plain ''curl'' 406, browser User-Agent 200. Footnote and script comment corrected. |
| 4 | "11 papers, 8 distinct public registrations verified live" conflates the private one into the public count: 7 are public, 1 is private (HTTP 401), 2 are projects. | **Accepted.** The mechanics table now spells out 7 public + 1 private + 2 projects, and notes that the 11th paper writes "on OSF" with the URL only in a PDF hyperlink. |
| 5 | Every sibling ''statistics:'' link, plus ''design:user_studies'', ''design:automated_measurements'' and ''artifacts'', is a red link today. | **Accepted as a fact, rejected as a change.** All of them are already promised from [[start]], and this site links promised-but-missing pages as a matter of course — ''privacy:requests'' ships red links to ''programming:crawler:pagegraph'' and ''programming:crawler:tracker_radar_collector''. Removing them would make this page //less// consistent with the site, not more. It is recorded here so the next run does not "fix" it. |
| — | Could not verify: the two Klemmer et al. quotes (CISPA's figshare and the S3 mirror both refuse non-JS fetches) and IMC 2026's "mandatory" ethics appendix (the CfP defers to a separate instructions page). It said so rather than guessing, and it explicitly reported pulling the **wrong** Klemmer paper first and confirming that it was not the source. | **Noted, and both gaps were closed elsewhere**: the Klemmer quotes were verified verbatim against ''paper.cols.txt'' by pass 13.2 and by the author. The IMC "mandatory ethics appendix" was then confirmed verbatim on IMC 2026's own submission-instructions page — //"All papers must include, in a clearly marked appendix section with the heading 'Ethics', a statement about ethical issues; papers that do not include such a statement may be rejected"// — and the artifact-availability declaration on the CfP itself. |

==== 13.4 Generic pass (Fable) ====

It ran against ''out/frozen_prereg_r3.txt'' and ''out/frozen_prov_r1.txt'', and produced the largest number of accepted findings of any pass — as it did on the last page. It was given no checklist, only the reader definition and the instruction to be adversarial.

^ # ^ Finding ^ Verdict ^
| 1 | "**§13.4 is a literal placeholder while §13's preamble and §14 both claim in the past tense that the generic pass ran.** On a page whose whole job is not overstating its own rigour." | **Accepted, and it is the finding this genre exists to catch.** This section is that fix. It is the second consecutive run on this site where the generic pass caught exactly this, which is an argument for writing the section before claiming it rather than for running a fourth reviewer. |
| 2 | "**Three external claims have no audit trail, under a sentence that promises one for everything.** §7 says every command is in the script; there is no CHI fetch anywhere, the only TOSEM request returned 403, and ESEM was never fetched — only ''emsejournal.github.io'', which is the EMSE //journal//, a different thing from the ESEM conference." | **Accepted in full, and the most valuable finding of the review** — three passes had looked at this page and none noticed that a verified-looking sentence rested on recall. CHI 2026 and ESEM 2024 were then fetched (both HTTP 200; CHI's contribution-types page mentions neither registered reports nor preregistration) and both added to the script. **ACM TOSEM could not be verified from any primary source**: ''dl.acm.org'', ''tosem.acm.org'' and the static PDF route all 403 every non-browser client. The page now attributes the TOSEM claim to FUZZING's own track page and says in a footnote that ACM's page could not be read. |
| 3 | "§5.1 says ''ANALYSIS_PLAN_ONLY'' holds **1** paper; the fold holds 3 and the embedded report in §6 says 3." | **Accepted; already fixed before the review landed.** The two extra PoPETs papers were added by the completeness sweep (§5.2) while this pass was running, and §5.1 was updated with them. Recorded anyway because the reviewer read a real inconsistency in a real snapshot. |
| 4 | "**The headline is field-scoped where only a corpus-scoped claim is supportable.** §10 says whether the 15 is near the real field rate could not be established, but a student will quote the first bold sentence as a fact about the field." | **Accepted.** Now "exactly one measurement study **in this corpus**". |
| 5 | "**'It works' is contradicted by the page's own todo about the same paper**: Chameleon Channels is one of the seven silent papers, so by the page's own standard the one worked example did not close the loop." | **Accepted.** Now "the form fits", and the todo carries the caveat. |
| 6 | "**11 + 3 = 14, not 15**, and the very next sentence says so." | **Accepted.** Now "every **checkable** preregistration — 14 of the 15". |
| 7 | "The heading says **a third**; the table beneath says 4 of 15, which is a quarter." | **Accepted.** The heading now states the count. |
| 8 | "**'Question 4 is the only one a crawl cannot answer' is wrong for exactly the crawls this reader will run.** Many web measurements are multi-arm — consent-accept versus reject, stateful versus stateless, several vantage points. Telling the reader Q4 never applies invites them to leave their arms out, which is the subgroup drift the page warns about." | **Accepted, and it is the best substantive catch of the review.** The paragraph is rewritten to say that a multi-arm crawl's arms **are** the answer to question 4. |
| 9 | "**The embargo advice ends in an impossibility and never states the cost of exposure.** The observed worst case (5 years) exceeds OSF's maximum (4), the page never says what to do then, and the double-blind workflow is given only for AsPredicted." | **Accepted.** Three paragraphs now: the gap versus the maximum, what exposure actually costs under anonymous review, and the register-embargo-view-only-lift workflow, using ''osf.io/d6h7b'' as the worked instance of it. |
| 10 | "**The descriptive-study reader — the page's own modal reader — never gets told what to write in the hypothesis fields.** The page says 67.7% never get past descriptives and then gives only hypothesis-shaped guidance." | **Accepted; the page's one real gap against its own stated purpose.** A new box tells a hypothesis-free study to use Open-Ended Registration and register the rows of the drift table instead. |
| 11 | "''REJECTED_PROBES'' lists five probes; the page says four." | **Accepted.** ''confirmatory (study%%|%%analysis)'' was missing from the prose. |
| 12 | "**A denominator slip on the page that lectures about denominators**: 'most measurement papers are descriptive' over a base of 5,456 that includes crypto and systems papers." | **Accepted.** Now "most papers in these venues". |
| 13 | "''preregisterd'' is verbatim in Klemmer et al. — add //[sic]//. And 'several widely-cited guides' names no guide." | **Accepted.** //[sic]// added; "widely-cited" softened to "third-party", since no specific guide was verified. |
| 14 | "The table is introduced as 'no venue in this corpus' and then lists eight, one of which the methodology section says is not in the corpus." | **Accepted.** Introduced as the seven, plus SOUPS, with the reason SOUPS is there. |
| 15 | "'the other six of the fifteen silent papers' parses as fifteen silent papers." | **Accepted.** |
| 16 | "The //Half the preregistrations were invisible to the schema// subsection is addressed to someone querying the dataset, not to this page's reader; move the table here and keep only the Tang lesson." | **Rejected, with the reviewer's own caveat noted.** It flagged this as weak because ''privacy:requests'' carries analogous corpus-QA subsections. The stronger reason to keep it: the page's headline number **cannot be reproduced from the extraction alone**, and a reader who tries will get 7 or 14 and conclude the page is wrong. The section is the warning that stops that. |
| 17 | "The Mechanics table's 'time to fill in' does not mention that OSF registrations need co-contributor approval before the timestamp lands." | **Not applied, and flagged honestly by the reviewer as unverified.** Not checked here either, and adding an unverified operational claim to a page whose argument is "verify the link" would be the wrong trade. It is a good candidate for the next run. |
| — | It also confirmed the section skeleton, voice, footnoting and //Use in Publications// convention against ''start'', ''design:website_selection'', ''privacy:requests'' and its provenance page, and diffed the provenance page's embedded report against ''out/prereg-report.txt'' clean. Its closing assessment named the decision-drift table, the four copyable registration answers, the Secondary Data recommendation and the project-versus-registration checker as the parts that earn the page. | Noted. |

**One finding this pass did not make, and the author did.** While the generic pass was running, the sentence //"IMC ... is the most measurement-heavy venue in the corpus"// was checked rather than assumed and is **false**: TheWebConf crawls more (28.7% of its papers) and PoPETs crawls more (24.1%) than IMC (20.7%). What IMC actually is, is the venue with the **lowest** human-subjects share (10.5%) and the **highest** empirical share (98.0%). The sentence was rewritten to those two figures, and the venue table in the report script now prints all three shares so the claim cannot drift back. Four reviewers looked at that sentence and none of them checked it, which is the argument for the author re-deriving their own rhetorical flourishes rather than only their tables.


===== 14. Run log =====

^ ^ ^
| Date | 2026-08-13 |
| Corpus at the time | 5,859 extracted papers and 5,869 full texts, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026; ''data/extract/run1'' |
| Page before | did not exist — a red link from [[start]] |
| Scripts written | ''prereg_fold.mjs'', ''report_preregistration.mjs'', ''external_checks_preregistration.sh'', ''pages/check_prereg_link.py'' |
| Shared scripts fixed | ''fetch_authors.py'' (two author-dropping bugs), ''bibgen.mjs'' (PoPETs DOI prefix) — see §8 |
| Bibliography entries added | 20, all keys checked against the live bibliography for collisions before appending |
| Models | Opus 5 for the page, the classifications and the scripts; one Sonnet research pass on registered reports and registry mechanics; three Sonnet review passes and one Fable generic pass, all recorded in §13 |
| Sub-agent output that had to be corrected | The registered-reports research pass could not reach ''aspredicted.org'' (it reported 403; the real code to a plain client is 406) and gave the question count from secondary sources as "nine, but sources disagree". It was right to flag the disagreement: the live form says eight. It also guessed a FUZZING URL that 404s. Both were re-checked directly before anything reached the page — see §7.2. The external-currency **review** pass then filed a blocking finding that the ''provenance:'' namespace is empty, which it is not — see §13.3 finding 1 |
| Accidental exposure | None. All credentials stayed in ''.env''. Every external request was unauthenticated and read-only; no OSF or AsPredicted account was used, and no ''view_only'' token was followed beyond the ones printed in the papers themselves |

====== References ======

<bibtex bibliography></bibtex>