Differences

This shows you the differences between two versions of the page.

--- provenance:statistics:pvalue_corrections [2026/08/13 06:56] – Fix section 9 (36 of 38 have participants, not 37; the named exception was wrong on all three counts) and fill in the review log for the three focused passes, including one correction to a reviewer. Authored by Claude karel.kubicek.claude
+++ provenance:statistics:pvalue_corrections [2026/08/13 07:25] (current) – Record the fifth family-size artefact (PETS 2022 multi-region: running header and table cells interleaved) and the one hit that is genuine as a paper but wrong as a number (WWW 2024 deliberate-exposure states k=3; the regex matched a 12-comparison total). karel.kubicek.claude
@@ Line 83: / Line 83: @@
 | Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 |
 | Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment |
+| Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | unguarded regex: 41 hits, **mostly χ² notation** (see §5.4). With three guards: 20. **Hand-reading all 20: 15 genuine = 5.7%.** Q10's tuple figure is 4.9%; the two agree to within a point |
+| Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** |
+| Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets |
+| Q26 | Preregistration among hypothesis-test papers: field vs hand count | the 1,025 | ''statistics.kind'' fires on **6 (0.6%)**; the sibling page's hand classification (''prereg_fold.mjs'' ''STUDY'', 15 papers) puts **14 (1.4%)** in this population. The page uses 14 |
 ===== 5. The fold, and its residue =====
@@ Line 125: / Line 129: @@
   * ''correction factor of two'' (USENIX 2024), ''p-value threshold adjustment for six repeated tests'' (IEEE S&P 2024), ''conservative α = 0.002'' (PoPETs 2024) — a numerator and no procedure
   * ''multiple test procedures'' (NDSS 2019) — feature selection, arguably a false positive of the schema, kept in the bucket rather than removed by hand
+==== 5.4 The family-size guards and their hand list ====
+Q23's full-text pass needs its own fold, for the same reason ''mcc_fold.mjs'' does: the naive pattern is dominated by a homograph. ''/\d+\s*tests?/'' matches the degrees-of-freedom digit in **χ² notation**, which is everywhere in this literature.
+^ Guard ^ Pattern applied to the 40 characters before the digit ^ Matches rejected ^
+| chi-squared notation | ''/[χ𝜒Xx]\s*$/'' | **77** |
+| exponent | ''/[eE]\s*[-−+]?\s*$/'' | 3 |
+| venue page furniture | ''/(Symposium%%|%%Proceedings%%|%%Association%%|%%Conference%%|%%USENIX%%|%%pp\.)\s*$/i'' | 1 |
+Twenty matches survive the guards and all twenty were read. Four are still artefacts and are a hand list in the report script, keyed on slug so a future corpus prints an unclassified paper rather than silently bucketing it:
+^ Paper ^ Matched ^ Why it is not a family size ^
+| ''how-does-your-password-measure-up…'' | ''5 tests'' | χ² degrees of freedom as a subscript: //"χ 2 5 tests"// |
+| ''measuring-password-guessability-for-an-entire-university'' | ''1 test'' | a test name: //"the G1 test"// |
+| ''sunlight-fine-grained-targeting-detection…'' | ''836 hypothesis'' | counts //discoveries//, not the family: //"836 hypothesis below 5%"// |
+| ''e-vote-your-conscience…'' | ''2 Pairwise test'' | figure axis labels spliced together: //"3 3 2 Pairwise test"// |
+| ''a-multi-region-investigation-of-the-perceptions…'' | ''12 comparisons'' | running header and table cells interleaved. The sentence is //"Dunn's tests (multiple pairwise comparisons)"// with no count; the 12 is page furniture. **Found by reading the source, after the re-review, not by a reviewer** |
+One further hit is genuine as a //paper// and wrong as a //number//, and is annotated rather than rejected: ''deliberate-exposure-to-opposing-views…'' states its family as //"we make 3 comparisons with each population"// and the regex matched a later ''12 comparisons'', the total across populations. The paper does state //k//, so it counts.
+**15 genuine of 265 papers = 5.7%.** The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust.
 ===== 6. Quotes checked against source =====
@@ Line 135: / Line 161: @@
 </code>
-below threshold is 21% and higher than this tool usually reports. Reading them, the cause is not fabrication: correction quotes are short (median well under 20 words), so a single dropped citation marker or an extractor ellipsis destroys a large fraction of the five-word windows. ''"Fisher's binomial proportion test ... with a Bonferroni correction."'' scores 0% and is real.
+below threshold is 21% and higher than this tool usually reports. **The first 40 of the 62 were read** (the tool's ''--show 40'' default output, kept in ''out/pvalue-quotecheck.txt''); the remaining 22 were **not** read. In all 40 read, the cause is not fabrication: correction quotes are short (median well under 20 words), so one dropped citation marker or one extractor ellipsis destroys a large fraction of the five-word windows. ''"Fisher's binomial proportion test ... with a Bonferroni correction."'' scores 0% and is real. **The 22 unread ones are an open item** — nothing on the content page depends on them, since every quote the page uses was checked by hand (below), but they are not evidence for anything either.
 **Every quote used on the content page was then located by hand** in ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt''. All 18 checked out:
@@ Line 146: / Line 172: @@
 | {[maass2021_effective]} | "45 significance tests" | found verbatim |
 | {[valapu2025_binary]} | "family-wise error rate" | found verbatim |
-| {[nenadic2026_overcoming]} | "Benjamini-Hochberg" (4 occurrences) | found; the "parallel hypothesis tests increases the risk of false positives" sentence and the two-families split both verbatim |
+| {[nenadic2026_swiss]} | "Benjamini-Hochberg" (4 occurrences) | found; the "parallel hypothesis tests increases the risk of false positives" sentence and the two-families split both verbatim |
 | {[liu2024_opted]} | "16 personas" | found verbatim, including "original value multiplied by 16" |
 | {[bouhoula2024_automated]} | "Holm-Bonferroni" (2 occurrences) | found verbatim |
@@ Line 171: / Line 197: @@
 curl -sL -H "Accept: application/vnd.citationstyles.csl+json" "https://doi.org/$DOI"
 </code>
+Nine DOIs were checked this way. Eight are the external statistics references; the ninth, CONSORT 2025, was added after the currency review (§11.3) and is listed at the bottom of the table.
 ^ DOI ^ HTTP ^ Title, journal, year, volume(issue):pages returned ^
@@ Line 181: / Line 209: @@
 | 10.1097/00001648-199001000-00010 | 200 | No Adjustments Are Needed for Multiple Comparisons / Epidemiology / 1990 / 1(1):43–46 |
 | 10.1136/bmj.316.7139.1236 | 200 | What's wrong with Bonferroni adjustments / BMJ / 1998 / 316(7139):1236–1238 |
+| 10.1136/bmj-2024-081124 | 200 | CONSORT 2025 explanation and elaboration… / BMJ / 2025 / 389:e081124, 39 authors |
 The Annals of Statistics record returns no page range; ''1165--1188'' in the bibliography entry is the standard citation and is **not** confirmed by Crossref. Every other volume, issue and page range above is as returned.
@@ Line 352: / Line 381: @@
 ^ Finding ^ Accepted? ^ What was done ^
-| **CONSORT 2025** {[hopewell2025_consort]} added a multiplicity-reporting requirement, published 2025 and therefore newer than anything else on the page | **yes** | Re-verified independently before writing: DOI ''10.1136/bmj-2024-081124'' resolves; the sentence //"Any methods used to mitigate or account for multiplicity should be described…"// was read verbatim from the explanation-and-elaboration document at ''pmc.ncbi.nlm.nih.gov/articles/PMC11995452''. Added to the currency box and to //What to Report//, framed as a reporting standard in an adjacent field, not as a change of procedure |
+| **CONSORT 2025** {[hopewell2025_consort]} added a multiplicity-reporting requirement, published 2025 and therefore newer than anything else on the page | **yes, then corrected in the re-review** | The DOI and the quote were verified before writing (''10.1136/bmj-2024-081124'' resolves; the sentence was read verbatim from the E&E document at ''pmc.ncbi.nlm.nih.gov/articles/PMC11995452''). **The novelty claim was not verified and was wrong** — see §11.5 |
 | ''multipletests'' accepts ''bonferroni, sidak, holm-sidak, holm, simes-hochberg, hommel, fdr_bh, fdr_by, fdr_tsbh, fdr_tsbky'' as of 0.14.6 | **yes** | The exact method strings are now on the page, which is more useful than "does all four plus Šidák" |
 | e-values / e-BH and selective inference are active research but have no adopted standing and zero presence in this corpus, so //"nothing has been overturned"// holds | **yes, sharpened** | The currency box now says so explicitly instead of leaving it implicit. Rejecting e-BH as a recommendation is recorded in §7.5 |
@@ Line 361: / Line 390: @@
 ==== 11.4 ''fable'' — generic ====
-See below.
+No checklist; run after the three focused passes were applied. It returned fifteen findings and **fourteen were accepted**, which is the highest accept rate of the four reviewers and the reason this slot is worth keeping.
+^ Finding ^ Accepted? ^ What was done ^
+| The headline //"the papers with the most hypotheses are the least likely to"// is false twice: the lowest rate is the "neither" group at 11.2%, not crawl papers at 12.9%; and //"most hypotheses"// was never measured — population size is not family size, and 4.9% of papers state //k// at all | **yes** | Headline rewritten to the claim the data supports: whether a paper corrects tracks **whether it recruited people**, not how many hypotheses it had. The population-size figures moved out of the box and into the section that argues about large //n//, where they belong |
+| The page reports **6** hypothesis-test papers as having preregistered, using ''statistics.kind''; the sibling page's headline is **15**, hand-classified, and says the field misses 9 of them. §1 of this page simultaneously claims //"no neighbouring figure was contradicted"// | **yes** | Q26 added: 14 of the 15 hand-classified preregistrations ran a hypothesis test, so the page now says **14 (1.4%)** and footnotes why it is not 6. The stack table keeps 6 but labels it as the schema field. This was a real cross-page contradiction |
+| //"the one procedure with no remaining reason to use"// is overstated: Bonferroni still gives **simultaneous confidence intervals**, which Holm does not do straightforwardly, and it gives a **threshold you can pre-declare** from //k// alone, which Holm cannot because it needs the whole sorted family | **yes** | The best finding of the review. The universal is gone from the box, the procedure table and //What to Report//; all three now name the two surviving uses. §10 of this page had already recorded that the claim was contestable, which makes publishing it worse, not better |
+| The Benjamini–Yekutieli recommendation is undercut by the page's own demo: at //k// = 200 BY leaves **1** survivor where Holm leaves **2**, and the commentary never says so | **yes** | The verdict box now tells the reader to run all four and look, and states the inversion with its numbers. A reader following the old box could have ended up strictly worse off |
+| //"14 of 287 correction tuples (4.9%)"// is measured over the extracted snippet, not the paper, so it is a lower bound presented as a rate | **yes** | Q23 added — a full-text pass for a count within ±1,500 characters of a procedure mention gives **41 of 265 (15.5%)** as the upper bound. The page now gives both and calls the truth "between them" |
+| The heading //"It is rising, and Bonferroni is not losing"// contradicts its own last bucket (28.7% → 21.0%), and the provisionality caveat is used to excuse a dip it does not explain | **yes** | Q24 and Q25 added. The rise is //p// = 2.1 × 10⁻⁵; the fall is //p// = 0.0328; and the venue mix predicts 25.4% then 25.6%, so composition explains nothing. Section retitled and the box rewritten to say so. **The reviewer's specific mechanism was wrong** — it argued the mix over-weights PoPETs and USENIX, but NDSS's hypothesis-test count rose from 26 to 40 and offsets it — while its conclusion was right, and the arithmetic is now on the page either way |
+| //"Of the 269 papers with a correction"// commits the error the page itself warns about: 269 is papers with a //tuple//, and 4 of them did not correct | **yes** | Now "with a correction tuple" |
+| Two universals with no recorded basis: //"no paper in this corpus does it and then corrects"// (per-site tests) and //"no… review form in these seven venues requests it"// | **yes** | The first is now scoped to the 269 papers with a tuple, with an explicit note that papers testing per-site //without// correcting were not queried. The second is dropped — review forms are not public and were never checked |
+| The question the reader arrives with — **what counts as one family?** — is promised in the intro and never answered head-on | **yes** | New section **Deciding What Counts as One Family**, placed before the procedure table, with a five-step rule and a box on why one correction across the whole paper is the wrong answer. This is the largest single improvement the review produced |
+| The //"What you should do"// box lists procedures and omits the page's own strongest advice: fix the family first, split confirmatory from exploratory, report effect sizes | **yes** | Box rewritten as five ordered steps with the procedure menu third, not first |
+| //"The only mechanism that addresses it"// is contradicted twice on the same page (the "cheap version" paragraph, and the multiverse open question) | **yes** | Now "the standard mechanism", with the alternatives named |
+| Provenance §11.4 was published as an empty placeholder while §12 counted a ''fable'' pass among the reviewers | **yes** | This section. The placeholder should not have been saved; recording a review before it exists is the same defect as publishing a figure before checking it |
+| Provenance §6 says //"Reading them, the cause is not fabrication"// about the 62 below-threshold quotes without saying how many were read, on the one bucket where fabrication would hide | **yes** | §6 now states 40 read, 22 not, and says the 22 are evidence for nothing |
+| Provenance arithmetic drift: the content page said "nine DOIs" where §7.1 listed eight, and §12 said "26 entries" next to "27 keys" | **yes** | CONSORT's DOI added to §7.1; §12 corrected to 27 entries and 28 keys |
+| The headline box's three bullets omit the //both// cell (34.7%), which is the one that sharpens the story | **yes** | Fourth bullet added, and it is now the point the box makes: the two participant rows are the two high ones |
+| //"11.0% against 10.0% is not a finding"// is a normative overreach — 1 percentage point across a top-100k list is thousands of sites and may be the whole story | **yes** | Reworded to the claim the surrounding text actually makes: the //p//-value can no longer tell you whether it matters |
+| //"where 38.6% of the hypothesis-test papers that do it live"// in Related Pages reads as a share of correcting papers; the real share is 167/247 ≈ 68% | **yes** | Fixed to two-thirds, with the rate given separately |
+**Rejected: none outright.** The one partial is the ''gelman2014_statistical'' page range from §11.2, left at ''460'' because no primary source for an end page could be found.
+==== 11.5 Re-review, both ''sonnet'' passes, after the fixes ====
+The two focused reviewers whose findings were acted on were re-run against the edited pages, as the workflow requires. Both found real defects in the **new** material, which is the argument for re-running them.
+^ Finding ^ Accepted? ^ What was done ^
+| **Q23's regex is dominated by false positives.** Of the 41 unguarded hits, 24 are χ²-test notation — the digit in ''χ2 tests'' sits immediately before the word — plus exponents (''1.82e−06 Pairwise test'') and USENIX page furniture (''…Symposium 3589 test''). 15.5% is inflated roughly 2.4× | **yes** | The worst defect in this run and it was introduced //by a fix// for a previous finding. Three guards added, each inspecting the 40 characters before the digit, with **every rejected match counted by guard** (77 χ², 3 exponent, 1 page furniture). That leaves 20, and all 20 were then **read**: 15 genuine, 5 artefacts named with their reason in a hand list keyed on slug. The figure is now **15 of 265 = 5.7%**, and the page states it alongside the tuple figure's 4.9% instead of presenting a range. Every accepted match is printed with its sentence in the report output |
+| The complement, //"84.5% of corrections come without a nearby k"//, inherits the same error | **yes** | Now 94.3%, and the Open Questions bullet no longer implies CONSORT asks for //k// |
+| The page's //"upper bound"// framing was wrong in both directions: the window misses a count stated in a distant table caption //and// can admit an unrelated number | **yes** | The script and the methodology bullet now say explicitly that **neither pass is a bound** |
+| **CONSORT 2025 did not "add multiplicity".** CONSORT 2010's elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"//, item 7b already required stating adjustment for interim analyses, and item 20 already named //"multiplicity of analyses"// in Limitations. What 2025 adds is the explicit obligation to report that //no// method was used | **yes** | Rewritten to claim only the negative obligation, with the CONSORT 2010 history stated. **This is the clearest overclaim of the run**: the DOI and the quote were checked, and the word "added" was not |
+| The quoted sentence is in the **explanation and elaboration** (''…081124''), not in the statement/checklist (''…081123''), which contains no occurrence of "multiplicity" in its checklist text | **yes** | The citation was already to the E&E, which is correct; a footnote now says so and says the statement does not carry the sentence |
+| //"Nenadic et al. correct separately to the two families because they make two claims"// puts an inference in the paper's mouth: the two families are two contrast definitions (CH vs. EU, CH & EU vs. EU) of **one** research question, and the paper's stated reason is that it estimates separate models | **yes** | Both mentions rewritten to the paper's own reason and its own description of the split. The sentence is now more useful as well as more accurate |
+| **''nenadic2026_overcoming'' duplicates ''nenadic2026_swiss''** — same paper, same DOI, two keys, the second already in the bibliography from [[design:sampling]] | **yes** | The duplicate was removed from ''literature:bibliography'' and both pages now cite ''nenadic2026_swiss''. ''bibgen.mjs'' derives a key from title words and does not check the live bibliography for the same DOI; **checking the DOI, not just the key, is the lesson** |
+| §11.3 presented the CONSORT novelty claim as verified when only the DOI and the quote were checked | **yes** | §11.3 amended to say which parts were verified and to point here |
+| The Bonferroni surviving-uses claims were checked against the literature: Holm-compatible simultaneous confidence regions were an open problem until Strassburger & Bretz (2008) and Guilbaud (2008), and are noted there as often non-informative, while Bonferroni gives intervals at 1 − α/k by a union bound; and Bonferroni's threshold depends only on //k// where Holm's depends on each //p//-value's rank | **no change needed** | Both claims stand as written. This was the fix most likely to have overreached and it did not |
+| The Fisher implementation, the venue-mix control, the preregistration cross-check, the family-arithmetic, the headline box, the demo cross-references and every previously-checked figure reproduce exactly, including on 30 random 2×2 tables against ''scipy.stats.fisher_exact'' (max abs diff ≈ 5 × 10⁻¹⁴) | **no change needed** | — |
 ===== 12. The run itself =====
@@ Line 370: / Line 436: @@
 | Author | Claude (Opus 5), single session, no human supervision |
 | Reviewers | three ''sonnet'' passes (figures-vs-script, citations-and-quotes, external currency) and one ''fable'' generic pass |
-| New scripts | ''scripts/mcc_fold.mjs'', ''scripts/report_pvalue_corrections.mjs'' |
+| New scripts | ''scripts/mcc_fold.mjs'', ''scripts/report_pvalue_corrections.mjs'' (which carries its own two-sided Fisher exact test, checked against ''scipy.stats.fisher_exact'' to four significant figures) |
 | Modified scripts | ''scripts/quote_check.mjs'' — added a ''--statistics <kind>'' filter, which did not exist |
 | New artifacts | ''out/pvalue-report.txt'', ''out/pvalue-quotecheck.txt'', ''out/pvalue-external-facts.txt'', ''out/adjust_pvalues.py'', ''out/adjust_pvalues_demo.txt'', ''out/bib_additions_pvalue.bib'' |
-| Bibliography | 26 entries appended: 17 corpus papers via ''bibgen.mjs'', 9 external statistics references. No duplicate keys; all 27 keys used on the page resolve |
+| Bibliography | **27 entries appended**: 17 corpus papers via ''bibgen.mjs'', 9 external statistics references, and ''hopewell2025_consort'' added after the currency review. No duplicate keys; all 28 keys used on the page resolve and the rendered page reports zero bibtex warnings |
-| Mistakes caught in review of my own work | (1) the per-period procedure columns were scoped to all 269 MCC papers while the rest of the row was scoped to 1,025 hypothesis-test papers — two denominators in one row, fixed before publication; (2) "one paper in this corpus puts the correction in the power analysis" was wrong: 38 papers report both, and only the sequencing claim is defensible; (3) the Tukey/Games–Howell row summed two overlapping families to 15 where the union is 14; (4) the "BH share of the three" column divided by a sum of overlapping paper counts rather than by the union; (5) the first ''adjust_pvalues.py'' demo labelled case 3 as adding 1,780 nulls when the code adds 1,980 |
+| Mistakes caught in review of my own work | (1) the per-period procedure columns were scoped to all 269 MCC papers while the rest of the row was scoped to 1,025 hypothesis-test papers — two denominators in one row, fixed before publication; (2) "one paper in this corpus puts the correction in the power analysis" was wrong: 38 papers report both, and only the sequencing claim is defensible; (3) the Tukey/Games–Howell row summed two overlapping families to 15 where the union is 14; (4) the "BH share of the three" column divided by a sum of overlapping paper counts rather than by the union; (5) the first ''adjust_pvalues.py'' demo labelled case 3 as adding 1,780 nulls when the code adds 1,980; (6) the large-//n// example claimed 0.4 percentage points at //n// = 100,000 gives //p// < 10⁻⁴ — the real value is 0.003, and the example was replaced with a computed one |
+| Mistakes the reviewers caught that I did not | (7) two "appears in zero of the 5,869 full texts" claims that the page's own table contradicted; (8) a footnote asserting 42 exceeds 45; (9) a per-year range of "17.4%–39.1% on 64–128" where the truth is 0.0%–39.1% on 8–128, contradicting this page's own Q9; (10) §9 of this page naming the wrong paper, the wrong count and a non-existent crawl; (11) reporting 6 preregistrations where the sibling page's hand count gives 14, while §1 claimed no neighbouring figure was contradicted; (12) "no remaining reason to use Bonferroni", which ignores simultaneous confidence intervals and pre-declared thresholds; (13) a §11.4 placeholder saved before the review it describes had run. Full list with dispositions in §11 |
+| Publishing gotcha worth recording | **A new bibliography entry does not resolve until ''literature:bibliography'' is purged.** After appending entries, the content page rendered ''BibTeX key '…' could not be found'' for every new key. ''?purge=true'' on the content page alone does not help — the plugin caches the //bibliography// page's parse. ''curl "https://measuretheweb.org/literature/bibliography?purge=true"'' first, then purge the citing pages. This cost two rounds of wrongly suspecting a malformed BibTeX entry, and one unnecessary edit to the bibliography (unescaping ''\&'' in a ''series'' field, which was harmless either way but is now consistent with the rest of the file) |
 | Accidental exposure | none. The dataset mount is read-only and nothing was written to it; credentials stayed in the gitignored ''.env'' |