Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: statistics:pvalue_corrections ======

Working notes behind [[statistics:pvalue_corrections]] — every query with its population and denominator, the scripts and their unedited output, the fold and its residue, the quotes checked against source papers, the external sources and how each was verified or rejected, the review log, and what could not be established. Corpus-level caveats that apply to every page on this site are on [[literature:corpus]] and are not restated here.

===== 1. What this page is backing =====

^ Item ^ Value ^
| Content page | [[statistics:pvalue_corrections]] — **new**, not an extension of anything |
| Report script | ''scripts/report_pvalue_corrections.mjs'' |
| Free-text fold it depends on | ''scripts/mcc_fold.mjs'' — 11 procedure families plus two non-correction buckets, with a self-test |
| Quote verification | ''scripts/quote_check.mjs --statistics multiple-comparison-correction'' (the ''--statistics'' filter was added for this page) |
| Stale-number guard | ''scripts/check_page_numbers.mjs'', whole-page, with ''--code'' |
| Non-corpus number ledger | ''out/pvalue-external-facts.txt'' |
| Runnable code published on the page | ''out/adjust_pvalues.py'' |
| Data | ''data/extract/run1/extractions.jsonl'' — 5,859 papers, 7 venues, 2010–2026 — plus a regex sweep over all 5,869 ''paper.cols.txt'' full texts |
| Written | 2026-08-13, against the corpus as extended on 2026-08-11 (commit ''8a6b843'') |

**Creating, not extending, and not overlapping.** ''statistics:pvalue_corrections'' was a red link promised from [[start]]. The only neighbouring page that existed when this was written is [[statistics:study_preregistration]], which touches multiplicity in one subsection ("For scale: the rest of the inference-hygiene stack is thin too") and cites this page forward. That subsection's figures (15.3% of ''inferential'', 24.5% of hypothesis-test papers) agree with this page's; this page keeps the ''hypothesisTest'' denominator throughout and adds the corrected-vs-tuple distinction the preregistration page did not need. **Nothing was overwritten and no neighbouring figure was contradicted.**

**Reachability** needed no work: [[start]] already lists ''[[Statistics:Pvalue corrections|P-value corrections]]'' in its outline, and [[statistics:study_preregistration]] links it twice.

===== 2. The number the task started from, and what happened to it =====

The task specification quoted a figure from the old corpus. Both the numerator and the denominator moved, and one distinction the old figure did not make turned out to matter.

^ ^ Old corpus (4,322 papers) ^ This corpus (5,859 papers) ^
| ''inferential'' population | 1,302 | 1,762 |
| papers with a ''multiple-comparison-correction'' tuple | 203 (15.6%) | 269 (15.3%) |
| …of which actually corrected | not distinguished | **265** |
| "Bonferroni named in…" | 84 | **141 papers, folded** (105 tuples say exactly ''Bonferroni correction''; 90 distinct strings fold to 11 families) |

Two things the task's framing got right and one it did not. Right: the correction is skipped in the setting that needs it, and the page's central table quantifies that (12.9% of crawl papers vs 38.6% of user studies). Not right: **''inferential'' is the wrong denominator.** A paper whose only statistics tuple is ''correlation'' or ''regression'' may have no //p//-value family at all, and 41.5% of the corpus never gets past descriptives. The population that needs a correction is the 1,025 papers with a ''hypothesis-test'' tuple, and the page uses that throughout. Over ''inferential'' the rate is 15.0%; over ''hypothesisTest'' it is 24.1%. **Publishing 15.0% as "how often the field corrects" would have understated it by a third.**

===== 3. Populations and denominators =====

^ Tag ^ Definition ^ //N// ^
| ''all'' | every extraction record | 5,859 |
| ''withAnyStats'' | ''statistics.length > 0'' | 5,456 |
| ''inferential'' | at least one ''statistics[].kind'' that is not ''descriptive-only'' | 1,762 |
| ''hypothesisTest'' | at least one ''statistics[].kind == "hypothesis-test"'' | **1,025** |
| ''regression'' / ''correlation'' | at least one tuple of that kind | 391 / 458 |
| ''humanSubjects'' | ''participants.length > 0'' | 1,357 |
| ''crawled'' | ''crawlConfig !== null %%||%% studyTypes.includes('automated-web-crawl')'' | 1,120 |

''inferential'', ''crawled'' and ''humanSubjects'' match ''OVERVIEW.md'' exactly (1,762 / 1,120 / 1,357), which is the check that the report's population code is the same code as the dossier's.

===== 4. Every query, with its population =====

<code bash>
cd /workspace/artifacts/wiki
node scripts/mcc_fold.mjs                                   # fold self-test
node scripts/report_pvalue_corrections.mjs                  # every figure on the page
node scripts/report_pvalue_corrections.mjs --wiki           # DokuWiki tables
node scripts/quote_check.mjs --statistics multiple-comparison-correction --show 62
python3 out/adjust_pvalues.py --demo                        # the published code's output
cat out/pvalue-report.txt out/adjust_pvalues_demo.txt \
    out/pvalue-external-facts.txt out/pvalue-quotecheck.txt > out/pvalue-check-source.txt
node scripts/check_page_numbers.mjs pages/statistics_pvalue_corrections.txt \
    out/pvalue-check-source.txt --code
</code>

^ # ^ Question ^ Population ^ Result ^
| Q1 | How many papers carry a ''multiple-comparison-correction'' tuple? | all 5,859 | 269 papers, 292 tuples |
| Q2 | …of which report a real correction rather than the absence of one? | the 269 | **265**; 3 declare they did not correct; 2 more are a sphericity correction (§5) |
| Q3 | Rate within each population | §3 | ''hypothesisTest'' 247/1,025 = **24.1%**; ''inferential'' 265/1,762 = 15.0%; ''regression'' 92/391 = 23.5%; ''correlation'' 56/458 = 12.2%; ''humanSubjects'' 194/1,357 = 14.3%; ''crawled'' 36/1,120 = 3.2%; all 265/5,859 = 4.5% |
| Q4 | Crossing crawl against participants, inside ''hypothesisTest'' | the 1,025 | participants-no-crawl 167/433 = **38.6%**; crawl-no-participants 18/140 = **12.9%**; both 17/49 = 34.7%; neither 45/403 = 11.2% |
| Q5 | Per venue, inside ''hypothesisTest'' | the 1,025 | PoPETs 68/149 = 45.6%; USENIX 69/242 = 28.5%; IEEE S&P 33/124 = 26.6%; CCS 23/110 = 20.9%; WWW 34/236 = 14.4%; IMC 10/71 = 14.1%; NDSS 10/93 = 10.8% |
| Q6 | Which procedure, folded? | the 269 with a tuple | Bonferroni 141, Holm 66, BH 43, Tukey 13, unnamed 8, BY 4, Šidák 3, Games–Howell 2, simultaneous bands 1; plus 3 NEGATIVE and 2 not-a-MCC (§5) |
| Q7 | Which family pairs overlap, so they are not added up? | the 269 | Tukey ∩ Games–Howell = 1 (union 14, 5.2%); Bonferroni ∩ Holm = 5 (union 202); Bonferroni ∩ BH = 3; BH ∩ BY = 0 |
| Q8 | Per period, all columns scoped to ''hypothesisTest'' | the 1,025 | 2010–2014 9/97 = 9.3%; 2015–2019 53/227 = 23.3%; 2020–2024 140/487 = 28.7%; 2025–2026* 45/214 = 21.0%. Procedures in 2025–2026: Bonferroni 19, Holm 14, BH 9, BY 0 |
| Q9 | Per year, uncollapsed | the 1,025 | ranges 0.0% (2010–2011) to 39.1% (2021) on denominators of 8–128. **Not published as a series**; the four buckets are |
| Q10 | Does the paper say how many hypotheses were in the family? | the 287 real correction tuples | **14 (4.9%)** state a count; 57 (19.9%) state an adjusted α; 161 (56.1%) have any ''detail'' at all |
| Q11 | The rest of the inference-hygiene stack | the 1,025 | correction 24.5%, regression 20.8%, correlation 20.2%, effect-size 15.5%, resampling 9.1%, power 7.0%, bayesian 1.2%, preregistration 0.6% |
| Q12 | Effect size and correction together | the 1,025 | both 69; correction without an effect size **178** |
| Q13 | Power analysis and correction together | the 1,025 | **38** papers; all 38 slugs are in §9 |
| Q14 | How large is the population a crawl paper tests over? | the 189 that ran both a crawl and a hypothesis test | all 189 state a ''population.n''; **median of each paper's largest //n// = 38,090**; ≥1,000: 161 (85.2%); ≥10,000: 133 (70.4%); ≥100,000: 85 (45.0%); ≥1,000,000: 50 (26.5%) |
| Q15 | Full-text sweep for named procedures | 5,869 ''paper.cols.txt'' | Bonferroni 189, Holm 67, BH 53, FDR spelled out 38, Tukey 27, BY 5, Šidák 5; **any named procedure: 279** |
| Q16 | Full-text sweep for the surrounding vocabulary | 5,869 | "multiple comparison(s)/test(s)/hypotheses" phrase 232 (4.0%); "family-wise error" 15; "p-hack"/"HARKing" 12 |
| Q17 | Schema recall against the sweep | Q15 ∩ Q2 | 246 in both; 33 named in text and missed by the schema; 19 with a tuple whose procedure is not in the named list. **88.2% recall** |
| Q18 | Has the FWER→FDR shift happened? | papers naming ≥1 of Bonferroni/Holm/BH | BH's share of that set: 2010–2014 2/9 = 22.2%; 2015–2019 8/51 = 15.7%; 2020–2024 30/141 = 21.3%; 2025–2026* 13/45 = 28.9% |
| Q19 | Is Storey's //q//-value, e-BH or a knockoff used? | 5,869 | q-value 11 hits, **all Q-learning**; e-value 1 hit, an ''E-VALUE'' semantics rule; knockoff 17 hits, all **Knockoff Nets** or counterfeit goods. Zero real uses of any of the three |
| Q20 | Does the literature engage with the debate about correcting? | 5,869 | "ASA statement" **0**; Rothman/Perneger 1 hit, an unrelated Perneger paper on questionnaire pre-test sample size; "forking paths" 2 hits, packet-forwarding paths and symbolic-execution paths. **Zero** |
| Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 |
| Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment |
| Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | unguarded regex: 41 hits, **mostly χ² notation** (see §5.4). With three guards: 20. **Hand-reading all 20: 15 genuine = 5.7%.** Q10's tuple figure is 4.9%; the two agree to within a point |
| Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** |
| Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets |
| Q26 | Preregistration among hypothesis-test papers: field vs hand count | the 1,025 | ''statistics.kind'' fires on **6 (0.6%)**; the sibling page's hand classification (''prereg_fold.mjs'' ''STUDY'', 15 papers) puts **14 (1.4%)** in this population. The page uses 14 |

===== 5. The fold, and its residue =====

''statistics[].method'' is free text and agrees run-to-run on ~0.2 of exact strings (''data/extract/README.md''), so it is folded before any aggregation. ''mcc_fold.mjs'' is an ordered rule list, first match wins, and a string matching no rule is returned as ''null'' and printed.

**Unmapped residue: 0 distinct strings, 0 papers.** All 90 distinct method strings across the 292 tuples fold.

==== 5.1 Rule order is load-bearing ====

Three orderings would silently mis-file entries if reversed, and the self-test asserts each:

^ Rule that must fire first ^ Why ^ Example string ^
| NEGATIVE, before everything | ''"No multiple-testing correction"'' contains ''multiple-testing'' and would fold to the unnamed-adjustment family | ''No adjustment for multiple testing'' |
| Holm, before Bonferroni | Every Holm string in the corpus also says "Bonferroni" | ''Holm-Bonferroni correction'', ''Bonferroni-Holm'', ''Bon Ferroni-Holm correction'', ''Holm's sequential Bonferroni procedure'' |
| Benjamini–Yekutieli, before Benjamini–Hochberg | The BY rule matches ''yekutieli''; the BH rule matches ''benjamini'' | ''Benjamini-Yekutieli False Discovery Rate procedure'' |

''node scripts/mcc_fold.mjs'' runs 34 cases:

<code>
34/34 fold self-tests passed
</code>

''Dunn'', ''Scheffé'' and ''Nemenyi'' have rules that match nothing in this corpus on their own — every corpus string naming Dunn also names Bonferroni or Holm (''Dunn post-hoc tests with Holm-Bonferroni method'' → Holm; ''Dunn-Bonferroni post hoc comparison test'' → Bonferroni), and Scheffé and Nemenyi never appear. The rules are kept with self-test cases so they stay tested rather than rotting, and this is recorded here rather than left as an unexplained empty family.

==== 5.2 Two buckets that are not corrections ====

The extraction scores a tuple as ''multiple-comparison-correction'' when the //sentence// is about multiple comparisons, which catches two kinds of thing that are not one. Both are excluded from every "corrected" figure on the page and both are shown in the page's own procedure table, because deleting them would be the mistake.

^ Bucket ^ Papers ^ What it actually is ^
| NEGATIVE | 3 | The paper states it did **not** correct. All three justify it as exploratory: {[pu2016_model]}, {[goetzen2022_ctrl]}, {[naji2025_responsibility]} |
| NOT-A-MCC | 2 | Greenhouse–Geisser, a **sphericity** correction to the //F//-test's degrees of freedom. One is {[boettger2025_regional]} |

A query that counts ''kind == "multiple-comparison-correction"'' and stops there reports 269 where the answer is 265, and reports three papers that explicitly declined as having complied.

==== 5.3 Free-text strings that are corrections but name no procedure ====

Eight papers describe an adjustment without naming a published procedure. They are folded to ''Unnamed / ad-hoc adjustment'' rather than guessed at, and they are listed here because the wording is the interesting part:

  * ''multiple hypothesis correction'' (USENIX 2021), ''correcting for multiple hypothesis testing'' (USENIX 2023), ''multiple-comparison corrected'' (PoPETs 2024) — a correction was applied and the reader cannot tell which
  * ''significance threshold divided by the number of groups'' (IEEE S&P 2024) — a described Bonferroni, deliberately not folded to Bonferroni because the string does not name it
  * ''correction factor of two'' (USENIX 2024), ''p-value threshold adjustment for six repeated tests'' (IEEE S&P 2024), ''conservative α = 0.002'' (PoPETs 2024) — a numerator and no procedure
  * ''multiple test procedures'' (NDSS 2019) — feature selection, arguably a false positive of the schema, kept in the bucket rather than removed by hand

==== 5.4 The family-size guards and their hand list ====

Q23's full-text pass needs its own fold, for the same reason ''mcc_fold.mjs'' does: the naive pattern is dominated by a homograph. ''/\d+\s*tests?/'' matches the degrees-of-freedom digit in **χ² notation**, which is everywhere in this literature.

^ Guard ^ Pattern applied to the 40 characters before the digit ^ Matches rejected ^
| chi-squared notation | ''/[χ𝜒Xx]\s*$/'' | **77** |
| exponent | ''/[eE]\s*[-−+]?\s*$/'' | 3 |
| venue page furniture | ''/(Symposium%%|%%Proceedings%%|%%Association%%|%%Conference%%|%%USENIX%%|%%pp\.)\s*$/i'' | 1 |

Twenty matches survive the guards and all twenty were read. Four are still artefacts and are a hand list in the report script, keyed on slug so a future corpus prints an unclassified paper rather than silently bucketing it:

^ Paper ^ Matched ^ Why it is not a family size ^
| ''how-does-your-password-measure-up…'' | ''5 tests'' | χ² degrees of freedom as a subscript: //"χ 2 5 tests"// |
| ''measuring-password-guessability-for-an-entire-university'' | ''1 test'' | a test name: //"the G1 test"// |
| ''sunlight-fine-grained-targeting-detection…'' | ''836 hypothesis'' | counts //discoveries//, not the family: //"836 hypothesis below 5%"// |
| ''e-vote-your-conscience…'' | ''2 Pairwise test'' | figure axis labels spliced together: //"3 3 2 Pairwise test"// |
| ''a-multi-region-investigation-of-the-perceptions…'' | ''12 comparisons'' | running header and table cells interleaved. The sentence is //"Dunn's tests (multiple pairwise comparisons)"// with no count; the 12 is page furniture. **Found by reading the source, after the re-review, not by a reviewer** |

One further hit is genuine as a //paper// and wrong as a //number//, and is annotated rather than rejected: ''deliberate-exposure-to-opposing-views…'' states its family as //"we make 3 comparisons with each population"// and the regex matched a later ''12 comparisons'', the total across populations. The paper does state //k//, so it counts.

**15 genuine of 265 papers = 5.7%.** The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust.

===== 6. Quotes checked against source =====

Bulk pass over every correction quote, with the ''--statistics'' filter added to ''quote_check.mjs'' for this page:

<code>
$ node scripts/quote_check.mjs --statistics multiple-comparison-correction
292 quotes checked: 167 exact, 63 partial (>=60% of 5-word windows), 62 below threshold, 0 with no full text on disk.
</code>

62 below threshold is 21% and higher than this tool usually reports. **The first 40 of the 62 were read** (the tool's ''--show 40'' default output, kept in ''out/pvalue-quotecheck.txt''); the remaining 22 were **not** read. In all 40 read, the cause is not fabrication: correction quotes are short (median well under 20 words), so one dropped citation marker or one extractor ellipsis destroys a large fraction of the five-word windows. ''"Fisher's binomial proportion test ... with a Bonferroni correction."'' scores 0% and is real. **The 22 unread ones are an open item** — nothing on the content page depends on them, since every quote the page uses was checked by hand (below), but they are not evidence for anything either.

**Every quote used on the content page was then located by hand** in ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt''. All 18 checked out:

^ Paper ^ Quote fragment searched ^ Verdict ^
| {[datta2015_automated]} | "uniformly more powerful" | found, spliced across a column break ("commonly used Bon-" / "ferroni correction") |
| {[lecuyer2015_sunlight]} | "the proverbial haystack", "88% to 61%", "75% to 21%", "46% of its p-values", "widely accepted in applied statistics" | all found, §4.4 and §6.5 |
| {[weinberg2017_topics]} | "585 of the topic" | found; the sentence is spliced, and "2,904 such combinations" is in the preceding sentence |
| {[zimmeck2024_website]} | "Benjamini-Yekutieli" | found; "59 were significant" is upstream in the same paragraph |
| {[maass2021_effective]} | "45 significance tests" | found verbatim |
| {[valapu2025_binary]} | "family-wise error rate" | found verbatim |
| {[nenadic2026_swiss]} | "Benjamini-Hochberg" (4 occurrences) | found; the "parallel hypothesis tests increases the risk of false positives" sentence and the two-families split both verbatim |
| {[liu2024_opted]} | "16 personas" | found verbatim, including "original value multiplied by 16" |
| {[bouhoula2024_automated]} | "Holm-Bonferroni" (2 occurrences) | found verbatim |
| {[goetzen2022_ctrl]} | "multiple comparison correction" | found verbatim |
| {[naji2025_responsibility]} | "multiple testing correction" | found verbatim; the extraction dropped the "[78]" citation marker, which is why it scored low in the bulk pass |
| {[pu2016_model]} | "multiple testing problem" | **the extractor's quote contains an ellipsis and does not match**. Located by searching "exploratory" instead: the full sentence adds "…where multiplicity adjustments are neither mandatory, nor important [7]", and the page quotes the expanded version |
| {[boettger2025_regional]} | "Greenhouse-Geisser" | found verbatim |
| {[bobek2026_community]} | "simultaneous confidence band" | found verbatim |
| {[despres2024_best]} | "conservative" | found verbatim, including "(based on a maximum of 25 hypotheses tested per outcome)" |
| {[kablo2023_privacy]} | "106 test" | found verbatim |
| {[ho2025_efficacy]} | "multiple comparison" | found verbatim, including "N = 2,693 per group" |

<wrap todo>
One thing found while checking and **not** put on the page. {[kablo2023_privacy]} writes //"we adjusted the standard threshold of 0.05 to a new threshold of 0.00047 (0.005 / 106)"//. 0.05/106 = 0.00047, so the printed threshold is right and the parenthetical divisor has a typo, or is a PDF-extraction artefact of "0.05". It is a one-character slip in an otherwise exemplary paper and calling it out on the page would be a gotcha, not a lesson. Recorded here instead.
</wrap>

===== 7. External sources, and how each was verified =====

Everything below was fetched on **2026-08-13**, not recalled.

==== 7.1 DOIs, through doi.org content negotiation ====

<code bash>
curl -sL -H "Accept: application/vnd.citationstyles.csl+json" "https://doi.org/$DOI"
</code>

Nine DOIs were checked this way. Eight are the external statistics references; the ninth, CONSORT 2025, was added after the currency review (§11.3) and is listed at the bottom of the table.

^ DOI ^ HTTP ^ Title, journal, year, volume(issue):pages returned ^
| 10.1111/j.2517-6161.1995.tb02031.x | 200 | Controlling the False Discovery Rate… / JRSS-B / 1995 / 57(1):289–300 |
| 10.1214/aos/1013699998 | 200 | The control of the false discovery rate… / Annals of Statistics / 2001 / 29(4) |
| 10.1111/1467-9868.00346 | 200 | A Direct Approach to False Discovery Rates / JRSS-B / 2002 / 64(3):479–498 |
| 10.1080/19345747.2011.618213 | 200 | Why We (Usually) Don't Have to Worry… / JREE / 2012 / 5(2):189–211 |
| 10.1511/2014.111.460 | 200 | The Statistical Crisis in Science / American Scientist / 2014 / 102(6):460 |
| 10.1080/00031305.2016.1154108 | 200 | The ASA Statement on //p//-Values… / The American Statistician / 2016 / 70(2):129–133 |
| 10.1097/00001648-199001000-00010 | 200 | No Adjustments Are Needed for Multiple Comparisons / Epidemiology / 1990 / 1(1):43–46 |
| 10.1136/bmj.316.7139.1236 | 200 | What's wrong with Bonferroni adjustments / BMJ / 1998 / 316(7139):1236–1238 |
| 10.1136/bmj-2024-081124 | 200 | CONSORT 2025 explanation and elaboration… / BMJ / 2025 / 389:e081124, 39 authors |

The Annals of Statistics record returns no page range; ''1165--1188'' in the bibliography entry is the standard citation and is **not** confirmed by Crossref. Every other volume, issue and page range above is as returned.

==== 7.2 Holm 1979 has no DOI, and OpenAlex reports one that does not exist ====

Worth recording because it is exactly the kind of thing that gets copied into a bibliography and never checked.

<code bash>
$ curl -sI "https://doi.org/10.2307/4615733" | head -1
HTTP/2 404
$ curl -s "https://doi.org/api/handles/10.2307/4615733"
{"responseCode":100,"handle":"10.2307/4615733"}          # 100 = handle not found
$ curl -s "https://api.crossref.org/works/10.2307/4615733" -o /dev/null -w "%{http_code}\n"
404
</code>

OpenAlex nonetheless reports ''doi: https://doi.org/10.2307/4615733'' for the work (''W2121044470'', 21,859 citations), with ''biblio: {volume: 6, issue: null, first_page: 65, last_page: 70}''. The DOI is a MAG-derived artefact: **10.2307 is JSTOR's prefix, and JSTOR did not register a DOI for this article.** The bibliography entry therefore carries a ''url'' and a ''note'' saying so, not a ''doi''.

''jstor.org'' could not be read either: HTTP 403 to ''curl'', and a reCAPTCHA wall to Playwright's own Chromium from this network ("JSTOR: Access Check", block reference recorded). So the URL is confirmed to be served by JSTOR and the article page itself was **not** read. Volume and pages come from OpenAlex; DBLP has no record; Europe PMC does not index the journal; the Semantic Scholar API returned HTTP 429. **This is the weakest-sourced entry in the bibliography and it is the most-cited paper on the page.**

==== 7.3 Software, from the projects' own documentation ====

^ Claim on the page ^ Source ^ Verified ^
| ''scipy.stats.false_discovery_control'' exists, takes ''method={'bh','by'}'' | ''docs.scipy.org/doc/scipy/reference/generated/scipy.stats.false_discovery_control.html'' | HTTP 200; page self-identifies as **SciPy v1.18.0 Manual** |
| //"tend to offer higher power than familywise error rate control procedures (e.g. Bonferroni correction)"// | same page, Notes section | quoted verbatim from the fetched HTML |
| ''by'' //"is guaranteed to control the FDR even when the p-values are not from independent tests"// | same page, ''method'' parameter | quoted verbatim |
| latest SciPy release | ''pypi.org/pypi/scipy/json'' | 1.18.0 |
| latest statsmodels release | ''pypi.org/pypi/statsmodels/json'' | 0.14.6 |
| ''statsmodels.stats.multitest.multipletests'' does all four plus Šidák | the function was **run locally** (§7.4), not read from the docs page, whose parameter table did not survive HTML-to-text extraction |

==== 7.4 The published code was run, and its Holm implementation cross-checked ====

''numpy'' 2.4.6, ''scipy'' 1.17.1, ''statsmodels'' 0.14.6, installed into the sandbox with ''pip install --break-system-packages''. The page publishes the unedited ''--demo'' output.

<code>
$ python3 -c "import numpy,scipy,statsmodels; print(numpy.__version__, scipy.__version__, statsmodels.__version__)"
2.4.6 1.17.1 0.14.6

# holm() and bonferroni() vs statsmodels, five random 50-value families:
0 holm max abs diff 0.0     0 bonf max abs diff 0.0
1 holm max abs diff 0.0     1 bonf max abs diff 0.0
2 holm max abs diff 0.0     2 bonf max abs diff 0.0
3 holm max abs diff 0.0     3 bonf max abs diff 0.0
4 holm max abs diff 0.0     4 bonf max abs diff 0.0

p = [0.001 0.02  0.03 ]
bonferroni -> [0.003 0.06  0.09 ] survivors 1
holm       -> [0.003 0.04  0.04 ] survivors 3
statsmodels holm -> [0.003 0.04  0.04 ]
</code>

Error paths were exercised too: a non-numeric line raises ''ValueError: could not convert string to float: 'not-a-number''' rather than being skipped, which is the behaviour the docstring promises.

Note that the demo output on the page was produced with SciPy **1.17.1** while the documentation quoted on the page is **1.18.0**. ''false_discovery_control'' has not changed between them as far as the release notes show, but the two version numbers on the page are deliberately different and neither is a typo.

==== 7.5 Rejected, and why ====

^ Source considered ^ Rejected because ^
| OpenAlex DOI for Holm 1979 | does not resolve (§7.2). This is the one that would have shipped a broken DOI |
| The "Bonferroni 1936" primary reference (//Teoria statistica delle classi e calcolo delle probabilità//) | an Italian-language monograph nobody on this page's audience will read, and the operative modern reference is Dunn 1961. The page gives both dates and cites neither, rather than citing something unverifiable |
| Any "top 10 statistical mistakes" style listicle for the currency claim | the currency claim is decidable from the corpus counts plus the Holm dominance argument, so no secondary source is needed and none is used |
| Storey 2002 as a recommendation | it is in the procedure table and the bibliography, but it is **not** recommended: nothing in this corpus uses it and its assumptions are less clearly met than BH's. Cited as "worth knowing", not "use this" |
| e-values / e-BH (Wang & Ramdas 2022) as a current-practice claim | genuinely current in the statistics literature, and **zero** presence in this corpus or in web-measurement practice. Mentioning it would have made the page's currency section look better-informed and told the reader nothing they can use. Left out entirely, and recorded here so the next run does not add it as decoration |
| Semantic Scholar API | HTTP 429 without a key; not retried |
| DBLP, Europe PMC, DataCite for Holm 1979 | zero hits, zero hits, and not the right registry |

===== 8. What could not be established =====

  * **Whether any of the 65 mixed-effects papers frames its model as a multiplicity strategy.** The page says "not one frames it as an answer to multiplicity" in the body and immediately qualifies it in a footnote: none surfaced while reading correction passages, and the 65 were not read one by one. **This is the softest claim on the page** and it is flagged as an open question there too. Closing it is 65 paper-reads.
  * **Whether the other 37 power-analysis-plus-correction papers computed the correction //before// sizing the study.** The extraction carries no ordering between two ''statistics'' tuples, so only {[ho2025_efficacy]}, whose sentence states the sequence, can be claimed. The page says so.
  * **Why NDSS is lowest.** 10.8% is the lowest venue rate and the page reports it without an explanation. A plausible one — NDSS publishes proportionally more systems papers whose statistics are classifier metrics rather than hypothesis tests — is not tested here. Note that classifier metrics live in ''detection.metric'', not ''statistics'', by design, so they are not what depresses this number.
  * **Whether the 33 schema-missed papers are real misses.** They are listed in full in the report output. They were **not** read individually; the 88.2% recall figure treats all 33 as misses, which is the conservative direction — a paper mentioning Bonferroni only in related work is a sweep false positive, not a schema miss, and would raise the recall figure.
  * **The true correction rate.** Every figure here is a rate of //reporting//, not of practice. A paper that corrected and did not write it down is indistinguishable from one that did not correct. ''statistics'' has no ''not-stated'' sentinel, so silence and absence are the same value.
  * **Whether uncorrected findings in this literature are wrong.** The obvious study — take papers that ran many tests without correcting, recompute their //p//-values from released data, see how many claims survive — needs released //p//-values, which is an [[:Artifacts]] problem. Listed as an open question on the page.

===== 9. The 38 papers with both a power analysis and a correction =====

Q13's full list, since the page quotes only the count and one paper. Population: the 1,025 ''hypothesisTest'' papers.

<code>
PETS/2017/to-permit-or-not-to-permit-that-is-the-usability-question-crowdsourcing-mobile-a
PETS/2020/the-best-of-both-worlds-mitigating-trade-offs-between-accuracy-and-user-burden-i
USENIX/2020/i-am-uncomfortable-sharing-what-i-cant-see-privacy-concerns-of-the-visually-impa
PETS/2021/warn-them-or-just-block-them-investigating-privacy-concerns-among-older-and-work
PETS/2021/awareness-adoption-and-misconceptions-of-web-privacy-tools
USENIX/2021/effect-of-mood-location-trust-and-presence-of-others-on-video-based-social-authe
USENIX/2021/evaluating-in-workflow-messages-for-improving-mental-models-of-end-to-end-encryp
USENIX/2021/on-the-usability-of-authenticity-checks-for-hardware-security-tokens
WWW/2021/an-experimental-study-to-understand-user-experience-and-perception-bias-occurred
IEEE-SP/2020/influencing-photo-sharing-decisions-on-social-media-a-case-of-paradoxical-findin
IEEE-SP/2020/are-anonymity-seekers-just-like-everybody-else-an-analysis-of-contributions-to-w
IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement
PETS/2022/increasing-adoption-of-tor-browser-using-informational-and-planning-nudges
PETS/2022/personal-information-inference-from-voice-recordings-user-awareness-and-privacy
USENIX/2022/where-to-recruit-for-security-development-studies-comparing-six-software-develop
USENIX/2023/exploring-tenants-preferences-of-privacy-negotiation-in-airbnb
USENIX/2023/if-sighted-people-know-i-should-be-able-to-know-privacy-perceptions-of-bystander
PETS/2024/how-does-connecting-online-activities-to-advertising-inferences-impact-privacy-p
PETS/2024/supporting-informed-choices-about-browser-cookies-the-impact-of-personalised-coo
IEEE-SP/2024/a-picture-is-worth-500-labels-a-case-study-of-demographic-disparities-in-local-m
PETS/2024/simply-tell-me-how-on-trustworthiness-and-technology-acceptance-of-attribute-bas
USENIX/2024/engaging-company-developers-in-security-research-studies-a-comprehensive-literat
PETS/2024/what-does-it-mean-to-be-creepy-responses-to-visualizations-of-personal-browsing
USENIX/2024/simulated-stress-a-case-study-of-the-effects-of-a-simulated-phishing-campaign-on
IEEE-SP/2025/not-the-right-question-a-study-on-attitudes-toward-client-side-scanning-with-sec
IEEE-SP/2025/its-time-time-for-digital-security-an-end-user-study-on-actionable-security-and
PETS/2025/ai-is-from-the-devil-behaviors-and-concerns-toward-personal-data-sharing-with-ll
PETS/2025/privacy-perceptions-and-behaviors-towards-targeted-advertising-on-social-media-a
USENIX/2025/vulnerability-of-text-matching-in-ml-ai-conference-reviewer-assignments-to-collu
PETS/2026/the-city-isn-t-uploading-me-to-tiktok-exploring-privacy-attitudes-towards-data-c
USENIX/2026/you-know-why-but-still-rely-the-impact-of-explainable-ai-on-trust-task-load-and
PETS/2026/are-bite-size-data-safety-details-a-healthy-diet-for-android-telehealth-app-user
NDSS/2025/the-kids-are-all-right-investigating-the-susceptibility-of-teens-and-adults-to-youtube-giveaway-scams
PETS/2025/who-cares-contextual-privacy-judgments-from-owner-and-bystander-perspectives-in
IEEE-SP/2025/lets-get-visual-testing-visual-analogies-and-metaphors-for-conveying-privacy-pol
USENIX/2025/malicious-llm-based-conversational-ai-makes-users-reveal-personal-information
IEEE-SP/2024/casual-users-and-rational-choices-within-differential-privacy
IEEE-SP/2024/investigating-voter-perceptions-of-printed-physical-audit-trails-for-online-voti
</code>

**36 of the 38 recruited human participants.** The two that did not are ''IEEE-SP/2020/are-anonymity-seekers-just-like-everybody-else'' and ''IEEE-SP/2024/a-picture-is-worth-500-labels''. **Exactly one of the 38 ran a crawl**: ''PETS/2024/what-does-it-mean-to-be-creepy'', whose ''studyTypes'' includes ''automated-web-crawl'' and which also recruited participants. So the population that sizes a study with a correction in the calculation is, with one partial exception, a user-study population.

<wrap todo>
An earlier version of this section said "37 of the 38" and named ''IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising'' as the exception and as a crawl. **All three parts of that were wrong** — the count is 36, that paper has ''participants'' tuples, and its ''studyTypes'' are ''['user-study', 'manual-audit']'' with ''crawlConfig === null'', so it is not a crawl by the definition in §3. It was caught by the figures-vs-script reviewer and is recorded here rather than silently corrected, because it is the clearest example on this page of a claim written from memory instead of from a query.
</wrap>

===== 10. Judgement calls =====

^ Call ^ Why ^ A reasonable person could have ^
| Denominator is ''hypothesisTest'' (1,025), not ''inferential'' (1,762) | a paper with no //p//-values cannot correct, so ''inferential'' answers a question nobody asked | used ''inferential'' for continuity with the task specification and [[statistics:study_preregistration]]. Both are on the page, with ''hypothesisTest'' first |
| Bonferroni is called **historical / superseded**, while the page also says it is still modal here | the two statements are about different things and the page separates them explicitly. Holm's dominance is a theorem, not a fashion, and Datta et al. state it inside this literature | refused to date it at all, on the grounds that a valid procedure is never superseded. That reading is defensible and it is why the verdict column says "Never wrong, just never optimal" |
| The three NEGATIVE papers are shown, not deleted | they are the page's only examples of correctly declining, and hiding them would misrepresent the extraction | dropped them as extraction noise. That would have lost the best content in the section |
| Per-year series is **not** published; four buckets are | the per-year series moves 17.4%–39.1% on denominators of 64–128 and would read as a trend | published the series with a caveat. The uncollapsed series is in the report output and in Q9 for anyone who wants it |
| The 2025–2026 procedure counts are scoped to ''hypothesisTest'' | so that every column in that table shares one population | left them over all 269 MCC papers, which is what the first draft did (24/14/10 instead of 19/14/9). That mixed two denominators inside one row and was caught before publication — the report now prints both, with the unscoped version labelled "do not publish" |
| Tukey and Games–Howell share one table row | they answer the same design question | given them separate rows. The row states "13 + 2, union 14 (5.2%)" rather than 15, because one paper uses both |
| The page teaches FWER-vs-FDR **as an editorial choice**, not a statistical one | it is the only framing that a measurement person can actually apply, and the textbook framing (independence, PRDS) does not decide anything for them | led with the assumptions. The assumptions are in the table; the decision rule is in prose |
| No ''~~DISCUSSION~~'' on this provenance page | comments belong on the content page. This follows the precedent set by [[provenance:statistics:study_preregistration]] | added one |

===== 11. Review log =====

Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the report script, its unedited output, and these notes. The three focused passes ran in parallel first; the generic pass ran after their findings were applied.

==== 11.1 ''sonnet'' — figures against the script ====

Re-ran all three scripts (byte-identical to the committed output), re-derived the 1,025 / 269 / 141 anchor counts from ''extractions.jsonl'' in an independent Python re-implementation of the fold, and cross-checked ''holm()''/''bonferroni()'' against ''statsmodels'' on 20 random families plus edge cases (all-ones, single value, ties): max abs diff 0.0.

^ Finding ^ Accepted? ^ What was done ^
| //"The phrase appears in zero of the 5,869 full texts"// is false twice on the page: the literal string matches 2, and the page's own table says so two paragraphs later | **yes** | Both sentences rewritten to say the string matches twice and both are false positives, with a pointer to the table. The intended claim (nobody cites the idea) survives; the falsifiable one did not |
| The footnote //"so the row sums exceed 45"// is wrong for the row it is attached to: 19+14+9+0 = 42 | **yes** | Footnote rewritten. The four figures are neither disjoint nor exhaustive, so they can sum either side of 45; the footnote now says which and gives 42 |
| //"moves between 17.4% (2026) and 39.1% (2021) on denominators of 64–128"// — the real range is 0.0% (2010, 2011) to 39.1% (2021) on 8–128, which §4 Q9 of this page states correctly | **yes** | Corrected on the content page. The content page contradicted its own provenance page, which is the worst version of this defect |
| Provenance §9's //"37 of the 38 recruited human participants, the exception is IMC/2022…"// is wrong on the count, the named paper, and the "is also a crawl" claim | **yes** | Re-queried: 36 of 38, and the two without participants are both IEEE S&P. §9 rewritten and the error left visible in a box |
| ''out/pvalue-external-facts.txt'' still carried //"0.4 illustrative percentage-point difference"// from the draft the page no longer uses | **yes** | Line deleted. The reviewer independently recomputed the replacement //z//-test (//z// = 7.294, //p// = 3.00 × 10⁻¹³) and confirmed it |
| //"eight of them are reportable at k = 200"// is only true for Benjamini–Hochberg; under Holm it is 2 | **yes** | Sentence now names both procedures. It read as procedure-agnostic and was not |

One correction to the reviewer, recorded for the record: it reported that ''IMC/2022/what-factors-affect-targeting-and-bids'' //"has 286 participants (recruited via Prolific)"//. The ''participants'' array on that record has **2 tuples**, not 286 people; the reviewer conflated a tuple count with a headcount. Its substantive point — that the paper has participants and is not a crawl — is correct and was the basis for the fix.

==== 11.2 ''sonnet'' — citations and quotes ====

Verified all 27 citekeys resolve with no collisions, all 17 corpus entries against ''data/corpus2/.meta'' and the papers' own front matter, all 8 external DOIs through ''doi.org'', and all 18 quotations against ''paper.cols.txt''.

^ Finding ^ Accepted? ^ What was done ^
| The Sunlight quote //"the proportion of high p-value hypothesis play an important role"// should be //"hypotheses"// — the singular came from mis-resolving the column-split word ''hypothe-'' / ''ses'' | **yes** | Fixed to the plural |
| //"apply Holm–Bonferroni to Fisher's exact tests across violation types and CMP subsets"// merges two distinct applications in Bouhoula et al. — §5.3 pairwise violation comparisons across popularity ranks, and §6 Fisher's exact tests on CMP subsets | **yes** | The table cell now describes both, which is a better example of //k// growing anyway |
| ''gelman2014_statistical'' has ''pages = {460}'', which matches Crossref, OpenAlex and Semantic Scholar, but the article is a multi-page //American Scientist// feature (secondary tools show "460+" or 460–465) | **partly** | Left as ''460''. Every queryable registry gives a single page and no primary source for the end page could be found; inventing 465 would be worse than under-specifying. Recorded here |
| ''benjamini2001_control'' pages ''1165--1188'' are not in the Crossref record | **already recorded** | §7.1 already says so. The reviewer independently confirmed them against Semantic Scholar, which is a stronger position than before |
| Every substantive claim attributed to Gelman/Hill/Yajima, Gelman & Loken, Rothman, Perneger and the ASA statement checks out against those papers' own abstracts and text; the Holm-dominates-Bonferroni claim is correct as statistics **and** is stated by Datta et al. inside the corpus | **no change needed** | — |
| Kablo & Cabarcos's //"(0.005 / 106)"// is confirmed as an error in the paper itself, not an extraction artefact | **no change needed** | Stays off the content page for the reason in §6 |

==== 11.3 ''sonnet'' — external currency, as of 2026-08-13 ====

^ Finding ^ Accepted? ^ What was done ^
| **CONSORT 2025** {[hopewell2025_consort]} added a multiplicity-reporting requirement, published 2025 and therefore newer than anything else on the page | **yes, then corrected in the re-review** | The DOI and the quote were verified before writing (''10.1136/bmj-2024-081124'' resolves; the sentence was read verbatim from the E&E document at ''pmc.ncbi.nlm.nih.gov/articles/PMC11995452''). **The novelty claim was not verified and was wrong** — see §11.5 |
| ''multipletests'' accepts ''bonferroni, sidak, holm-sidak, holm, simes-hochberg, hommel, fdr_bh, fdr_by, fdr_tsbh, fdr_tsbky'' as of 0.14.6 | **yes** | The exact method strings are now on the page, which is more useful than "does all four plus Šidák" |
| e-values / e-BH and selective inference are active research but have no adopted standing and zero presence in this corpus, so //"nothing has been overturned"// holds | **yes, sharpened** | The currency box now says so explicitly instead of leaving it implicit. Rejecting e-BH as a recommendation is recorded in §7.5 |
| The DokuWiki code exporter renders ''Šidák'' as ''Sidak'' in the downloadable ''adjust_pvalues.py'' | **noted, not fixed** | Cosmetic, in a docstring, does not affect execution. The page's source deliberately ASCII-ises it for this reason |
| Holm 1979 has no real DOI anywhere — Crossref by title and by container, DataCite and OpenAlex all searched; Wiley's issue page returns HTTP 402 | **confirms §7.2** | No change. The independent search is worth more than the original check |
| All 17 corpus DOIs and all 3 USENIX URLs return HTTP 200; SciPy 1.18.0 and statsmodels 0.14.6 are current; ''false_discovery_control'' is unchanged between 1.17.1 and 1.18.0 | **no change needed** | — |

==== 11.4 ''fable'' — generic ====

No checklist; run after the three focused passes were applied. It returned fifteen findings and **fourteen were accepted**, which is the highest accept rate of the four reviewers and the reason this slot is worth keeping.

^ Finding ^ Accepted? ^ What was done ^
| The headline //"the papers with the most hypotheses are the least likely to"// is false twice: the lowest rate is the "neither" group at 11.2%, not crawl papers at 12.9%; and //"most hypotheses"// was never measured — population size is not family size, and 4.9% of papers state //k// at all | **yes** | Headline rewritten to the claim the data supports: whether a paper corrects tracks **whether it recruited people**, not how many hypotheses it had. The population-size figures moved out of the box and into the section that argues about large //n//, where they belong |
| The page reports **6** hypothesis-test papers as having preregistered, using ''statistics.kind''; the sibling page's headline is **15**, hand-classified, and says the field misses 9 of them. §1 of this page simultaneously claims //"no neighbouring figure was contradicted"// | **yes** | Q26 added: 14 of the 15 hand-classified preregistrations ran a hypothesis test, so the page now says **14 (1.4%)** and footnotes why it is not 6. The stack table keeps 6 but labels it as the schema field. This was a real cross-page contradiction |
| //"the one procedure with no remaining reason to use"// is overstated: Bonferroni still gives **simultaneous confidence intervals**, which Holm does not do straightforwardly, and it gives a **threshold you can pre-declare** from //k// alone, which Holm cannot because it needs the whole sorted family | **yes** | The best finding of the review. The universal is gone from the box, the procedure table and //What to Report//; all three now name the two surviving uses. §10 of this page had already recorded that the claim was contestable, which makes publishing it worse, not better |
| The Benjamini–Yekutieli recommendation is undercut by the page's own demo: at //k// = 200 BY leaves **1** survivor where Holm leaves **2**, and the commentary never says so | **yes** | The verdict box now tells the reader to run all four and look, and states the inversion with its numbers. A reader following the old box could have ended up strictly worse off |
| //"14 of 287 correction tuples (4.9%)"// is measured over the extracted snippet, not the paper, so it is a lower bound presented as a rate | **yes** | Q23 added — a full-text pass for a count within ±1,500 characters of a procedure mention gives **41 of 265 (15.5%)** as the upper bound. The page now gives both and calls the truth "between them" |
| The heading //"It is rising, and Bonferroni is not losing"// contradicts its own last bucket (28.7% → 21.0%), and the provisionality caveat is used to excuse a dip it does not explain | **yes** | Q24 and Q25 added. The rise is //p// = 2.1 × 10⁻⁵; the fall is //p// = 0.0328; and the venue mix predicts 25.4% then 25.6%, so composition explains nothing. Section retitled and the box rewritten to say so. **The reviewer's specific mechanism was wrong** — it argued the mix over-weights PoPETs and USENIX, but NDSS's hypothesis-test count rose from 26 to 40 and offsets it — while its conclusion was right, and the arithmetic is now on the page either way |
| //"Of the 269 papers with a correction"// commits the error the page itself warns about: 269 is papers with a //tuple//, and 4 of them did not correct | **yes** | Now "with a correction tuple" |
| Two universals with no recorded basis: //"no paper in this corpus does it and then corrects"// (per-site tests) and //"no… review form in these seven venues requests it"// | **yes** | The first is now scoped to the 269 papers with a tuple, with an explicit note that papers testing per-site //without// correcting were not queried. The second is dropped — review forms are not public and were never checked |
| The question the reader arrives with — **what counts as one family?** — is promised in the intro and never answered head-on | **yes** | New section **Deciding What Counts as One Family**, placed before the procedure table, with a five-step rule and a box on why one correction across the whole paper is the wrong answer. This is the largest single improvement the review produced |
| The //"What you should do"// box lists procedures and omits the page's own strongest advice: fix the family first, split confirmatory from exploratory, report effect sizes | **yes** | Box rewritten as five ordered steps with the procedure menu third, not first |
| //"The only mechanism that addresses it"// is contradicted twice on the same page (the "cheap version" paragraph, and the multiverse open question) | **yes** | Now "the standard mechanism", with the alternatives named |
| Provenance §11.4 was published as an empty placeholder while §12 counted a ''fable'' pass among the reviewers | **yes** | This section. The placeholder should not have been saved; recording a review before it exists is the same defect as publishing a figure before checking it |
| Provenance §6 says //"Reading them, the cause is not fabrication"// about the 62 below-threshold quotes without saying how many were read, on the one bucket where fabrication would hide | **yes** | §6 now states 40 read, 22 not, and says the 22 are evidence for nothing |
| Provenance arithmetic drift: the content page said "nine DOIs" where §7.1 listed eight, and §12 said "26 entries" next to "27 keys" | **yes** | CONSORT's DOI added to §7.1; §12 corrected to 27 entries and 28 keys |
| The headline box's three bullets omit the //both// cell (34.7%), which is the one that sharpens the story | **yes** | Fourth bullet added, and it is now the point the box makes: the two participant rows are the two high ones |
| //"11.0% against 10.0% is not a finding"// is a normative overreach — 1 percentage point across a top-100k list is thousands of sites and may be the whole story | **yes** | Reworded to the claim the surrounding text actually makes: the //p//-value can no longer tell you whether it matters |
| //"where 38.6% of the hypothesis-test papers that do it live"// in Related Pages reads as a share of correcting papers; the real share is 167/247 ≈ 68% | **yes** | Fixed to two-thirds, with the rate given separately |

**Rejected: none outright.** The one partial is the ''gelman2014_statistical'' page range from §11.2, left at ''460'' because no primary source for an end page could be found.

==== 11.5 Re-review, both ''sonnet'' passes, after the fixes ====

The two focused reviewers whose findings were acted on were re-run against the edited pages, as the workflow requires. Both found real defects in the **new** material, which is the argument for re-running them.

^ Finding ^ Accepted? ^ What was done ^
| **Q23's regex is dominated by false positives.** Of the 41 unguarded hits, 24 are χ²-test notation — the digit in ''χ2 tests'' sits immediately before the word — plus exponents (''1.82e−06 Pairwise test'') and USENIX page furniture (''…Symposium 3589 test''). 15.5% is inflated roughly 2.4× | **yes** | The worst defect in this run and it was introduced //by a fix// for a previous finding. Three guards added, each inspecting the 40 characters before the digit, with **every rejected match counted by guard** (77 χ², 3 exponent, 1 page furniture). That leaves 20, and all 20 were then **read**: 15 genuine, 5 artefacts named with their reason in a hand list keyed on slug. The figure is now **15 of 265 = 5.7%**, and the page states it alongside the tuple figure's 4.9% instead of presenting a range. Every accepted match is printed with its sentence in the report output |
| The complement, //"84.5% of corrections come without a nearby k"//, inherits the same error | **yes** | Now 94.3%, and the Open Questions bullet no longer implies CONSORT asks for //k// |
| The page's //"upper bound"// framing was wrong in both directions: the window misses a count stated in a distant table caption //and// can admit an unrelated number | **yes** | The script and the methodology bullet now say explicitly that **neither pass is a bound** |
| **CONSORT 2025 did not "add multiplicity".** CONSORT 2010's elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"//, item 7b already required stating adjustment for interim analyses, and item 20 already named //"multiplicity of analyses"// in Limitations. What 2025 adds is the explicit obligation to report that //no// method was used | **yes** | Rewritten to claim only the negative obligation, with the CONSORT 2010 history stated. **This is the clearest overclaim of the run**: the DOI and the quote were checked, and the word "added" was not |
| The quoted sentence is in the **explanation and elaboration** (''…081124''), not in the statement/checklist (''…081123''), which contains no occurrence of "multiplicity" in its checklist text | **yes** | The citation was already to the E&E, which is correct; a footnote now says so and says the statement does not carry the sentence |
| //"Nenadic et al. correct separately to the two families because they make two claims"// puts an inference in the paper's mouth: the two families are two contrast definitions (CH vs. EU, CH & EU vs. EU) of **one** research question, and the paper's stated reason is that it estimates separate models | **yes** | Both mentions rewritten to the paper's own reason and its own description of the split. The sentence is now more useful as well as more accurate |
| **''nenadic2026_overcoming'' duplicates ''nenadic2026_swiss''** — same paper, same DOI, two keys, the second already in the bibliography from [[design:sampling]] | **yes** | The duplicate was removed from ''literature:bibliography'' and both pages now cite ''nenadic2026_swiss''. ''bibgen.mjs'' derives a key from title words and does not check the live bibliography for the same DOI; **checking the DOI, not just the key, is the lesson** |
| §11.3 presented the CONSORT novelty claim as verified when only the DOI and the quote were checked | **yes** | §11.3 amended to say which parts were verified and to point here |
| The Bonferroni surviving-uses claims were checked against the literature: Holm-compatible simultaneous confidence regions were an open problem until Strassburger & Bretz (2008) and Guilbaud (2008), and are noted there as often non-informative, while Bonferroni gives intervals at 1 − α/k by a union bound; and Bonferroni's threshold depends only on //k// where Holm's depends on each //p//-value's rank | **no change needed** | Both claims stand as written. This was the fix most likely to have overreached and it did not |
| The Fisher implementation, the venue-mix control, the preregistration cross-check, the family-arithmetic, the headline box, the demo cross-references and every previously-checked figure reproduce exactly, including on 30 random 2×2 tables against ''scipy.stats.fisher_exact'' (max abs diff ≈ 5 × 10⁻¹⁴) | **no change needed** | — |

===== 12. The run itself =====

^ ^ ^
| Date | 2026-08-13 |
| Corpus at the time | 5,859 extracted papers / 5,869 full texts, 7 venues, 2010–2026, as extended 2026-08-11 (''8a6b843'') |
| Author | Claude (Opus 5), single session, no human supervision |
| Reviewers | three ''sonnet'' passes (figures-vs-script, citations-and-quotes, external currency) and one ''fable'' generic pass |
| New scripts | ''scripts/mcc_fold.mjs'', ''scripts/report_pvalue_corrections.mjs'' (which carries its own two-sided Fisher exact test, checked against ''scipy.stats.fisher_exact'' to four significant figures) |
| Modified scripts | ''scripts/quote_check.mjs'' — added a ''--statistics <kind>'' filter, which did not exist |
| New artifacts | ''out/pvalue-report.txt'', ''out/pvalue-quotecheck.txt'', ''out/pvalue-external-facts.txt'', ''out/adjust_pvalues.py'', ''out/adjust_pvalues_demo.txt'', ''out/bib_additions_pvalue.bib'' |
| Bibliography | **27 entries appended**: 17 corpus papers via ''bibgen.mjs'', 9 external statistics references, and ''hopewell2025_consort'' added after the currency review. No duplicate keys; all 28 keys used on the page resolve and the rendered page reports zero bibtex warnings |
| Mistakes caught in review of my own work | (1) the per-period procedure columns were scoped to all 269 MCC papers while the rest of the row was scoped to 1,025 hypothesis-test papers — two denominators in one row, fixed before publication; (2) "one paper in this corpus puts the correction in the power analysis" was wrong: 38 papers report both, and only the sequencing claim is defensible; (3) the Tukey/Games–Howell row summed two overlapping families to 15 where the union is 14; (4) the "BH share of the three" column divided by a sum of overlapping paper counts rather than by the union; (5) the first ''adjust_pvalues.py'' demo labelled case 3 as adding 1,780 nulls when the code adds 1,980; (6) the large-//n// example claimed 0.4 percentage points at //n// = 100,000 gives //p// < 10⁻⁴ — the real value is 0.003, and the example was replaced with a computed one |
| Mistakes the reviewers caught that I did not | (7) two "appears in zero of the 5,869 full texts" claims that the page's own table contradicted; (8) a footnote asserting 42 exceeds 45; (9) a per-year range of "17.4%–39.1% on 64–128" where the truth is 0.0%–39.1% on 8–128, contradicting this page's own Q9; (10) §9 of this page naming the wrong paper, the wrong count and a non-existent crawl; (11) reporting 6 preregistrations where the sibling page's hand count gives 14, while §1 claimed no neighbouring figure was contradicted; (12) "no remaining reason to use Bonferroni", which ignores simultaneous confidence intervals and pre-declared thresholds; (13) a §11.4 placeholder saved before the review it describes had run. Full list with dispositions in §11 |
| Publishing gotcha worth recording | **A new bibliography entry does not resolve until ''literature:bibliography'' is purged.** After appending entries, the content page rendered ''BibTeX key '…' could not be found'' for every new key. ''?purge=true'' on the content page alone does not help — the plugin caches the //bibliography// page's parse. ''curl "https://measuretheweb.org/literature/bibliography?purge=true"'' first, then purge the citing pages. This cost two rounds of wrongly suspecting a malformed BibTeX entry, and one unnecessary edit to the bibliography (unescaping ''\&'' in a ''series'' field, which was harmless either way but is now consistent with the rest of the file) |
| Accidental exposure | none. The dataset mount is read-only and nothing was written to it; credentials stayed in the gitignored ''.env'' |

===== Related =====

  * [[statistics:pvalue_corrections]] — the page these notes back.
  * [[literature:corpus]] — the corpus-level provenance: how the 5,859 papers were selected, extracted and validated.
  * [[provenance:statistics:study_preregistration]] — the sibling ''statistics:'' provenance page, and the source of this page's format.

====== References ======

<bibtex bibliography></bibtex>