| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:statistics:pvalue_corrections [2026/08/13 07:09] – Add Q23-Q26 (full-text bound on stating k, Fisher tests on the period trend, venue-mix control, preregistration field-vs-hand-count), fill in the generic review log with all 15 findings and their dispositions, state how many of the 62 below-threshold quot karel.kubicek.claude | provenance:statistics:pvalue_corrections [2026/08/13 07:25] (current) – Record the fifth family-size artefact (PETS 2022 multi-region: running header and table cells interleaved) and the one hit that is genuine as a paper but wrong as a number (WWW 2024 deliberate-exposure states k=3; the regex matched a 12-comparison total). karel.kubicek.claude |
|---|
| | Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 | | | Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 | |
| | Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment | | | Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment | |
| | Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | a count of comparisons/tests/hypotheses within ±1,500 characters of a procedure mention: **41 (15.5%)**. This is the **upper** bound; Q10's 4.9% of tuples is the lower one | | | Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | unguarded regex: 41 hits, **mostly χ² notation** (see §5.4). With three guards: 20. **Hand-reading all 20: 15 genuine = 5.7%.** Q10's tuple figure is 4.9%; the two agree to within a point | |
| | Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** | | | Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** | |
| | Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets | | | Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets | |
| * ''correction factor of two'' (USENIX 2024), ''p-value threshold adjustment for six repeated tests'' (IEEE S&P 2024), ''conservative α = 0.002'' (PoPETs 2024) — a numerator and no procedure | * ''correction factor of two'' (USENIX 2024), ''p-value threshold adjustment for six repeated tests'' (IEEE S&P 2024), ''conservative α = 0.002'' (PoPETs 2024) — a numerator and no procedure |
| * ''multiple test procedures'' (NDSS 2019) — feature selection, arguably a false positive of the schema, kept in the bucket rather than removed by hand | * ''multiple test procedures'' (NDSS 2019) — feature selection, arguably a false positive of the schema, kept in the bucket rather than removed by hand |
| | |
| | ==== 5.4 The family-size guards and their hand list ==== |
| | |
| | Q23's full-text pass needs its own fold, for the same reason ''mcc_fold.mjs'' does: the naive pattern is dominated by a homograph. ''/\d+\s*tests?/'' matches the degrees-of-freedom digit in **χ² notation**, which is everywhere in this literature. |
| | |
| | ^ Guard ^ Pattern applied to the 40 characters before the digit ^ Matches rejected ^ |
| | | chi-squared notation | ''/[χ𝜒Xx]\s*$/'' | **77** | |
| | | exponent | ''/[eE]\s*[-−+]?\s*$/'' | 3 | |
| | | venue page furniture | ''/(Symposium%%|%%Proceedings%%|%%Association%%|%%Conference%%|%%USENIX%%|%%pp\.)\s*$/i'' | 1 | |
| | |
| | Twenty matches survive the guards and all twenty were read. Four are still artefacts and are a hand list in the report script, keyed on slug so a future corpus prints an unclassified paper rather than silently bucketing it: |
| | |
| | ^ Paper ^ Matched ^ Why it is not a family size ^ |
| | | ''how-does-your-password-measure-up…'' | ''5 tests'' | χ² degrees of freedom as a subscript: //"χ 2 5 tests"// | |
| | | ''measuring-password-guessability-for-an-entire-university'' | ''1 test'' | a test name: //"the G1 test"// | |
| | | ''sunlight-fine-grained-targeting-detection…'' | ''836 hypothesis'' | counts //discoveries//, not the family: //"836 hypothesis below 5%"// | |
| | | ''e-vote-your-conscience…'' | ''2 Pairwise test'' | figure axis labels spliced together: //"3 3 2 Pairwise test"// | |
| | | ''a-multi-region-investigation-of-the-perceptions…'' | ''12 comparisons'' | running header and table cells interleaved. The sentence is //"Dunn's tests (multiple pairwise comparisons)"// with no count; the 12 is page furniture. **Found by reading the source, after the re-review, not by a reviewer** | |
| | |
| | One further hit is genuine as a //paper// and wrong as a //number//, and is annotated rather than rejected: ''deliberate-exposure-to-opposing-views…'' states its family as //"we make 3 comparisons with each population"// and the regex matched a later ''12 comparisons'', the total across populations. The paper does state //k//, so it counts. |
| | |
| | **15 genuine of 265 papers = 5.7%.** The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust. |
| |
| ===== 6. Quotes checked against source ===== | ===== 6. Quotes checked against source ===== |
| | {[maass2021_effective]} | "45 significance tests" | found verbatim | | | {[maass2021_effective]} | "45 significance tests" | found verbatim | |
| | {[valapu2025_binary]} | "family-wise error rate" | found verbatim | | | {[valapu2025_binary]} | "family-wise error rate" | found verbatim | |
| | {[nenadic2026_overcoming]} | "Benjamini-Hochberg" (4 occurrences) | found; the "parallel hypothesis tests increases the risk of false positives" sentence and the two-families split both verbatim | | | {[nenadic2026_swiss]} | "Benjamini-Hochberg" (4 occurrences) | found; the "parallel hypothesis tests increases the risk of false positives" sentence and the two-families split both verbatim | |
| | {[liu2024_opted]} | "16 personas" | found verbatim, including "original value multiplied by 16" | | | {[liu2024_opted]} | "16 personas" | found verbatim, including "original value multiplied by 16" | |
| | {[bouhoula2024_automated]} | "Holm-Bonferroni" (2 occurrences) | found verbatim | | | {[bouhoula2024_automated]} | "Holm-Bonferroni" (2 occurrences) | found verbatim | |
| |
| ^ Finding ^ Accepted? ^ What was done ^ | ^ Finding ^ Accepted? ^ What was done ^ |
| | **CONSORT 2025** {[hopewell2025_consort]} added a multiplicity-reporting requirement, published 2025 and therefore newer than anything else on the page | **yes** | Re-verified independently before writing: DOI ''10.1136/bmj-2024-081124'' resolves; the sentence //"Any methods used to mitigate or account for multiplicity should be described…"// was read verbatim from the explanation-and-elaboration document at ''pmc.ncbi.nlm.nih.gov/articles/PMC11995452''. Added to the currency box and to //What to Report//, framed as a reporting standard in an adjacent field, not as a change of procedure | | | **CONSORT 2025** {[hopewell2025_consort]} added a multiplicity-reporting requirement, published 2025 and therefore newer than anything else on the page | **yes, then corrected in the re-review** | The DOI and the quote were verified before writing (''10.1136/bmj-2024-081124'' resolves; the sentence was read verbatim from the E&E document at ''pmc.ncbi.nlm.nih.gov/articles/PMC11995452''). **The novelty claim was not verified and was wrong** — see §11.5 | |
| | ''multipletests'' accepts ''bonferroni, sidak, holm-sidak, holm, simes-hochberg, hommel, fdr_bh, fdr_by, fdr_tsbh, fdr_tsbky'' as of 0.14.6 | **yes** | The exact method strings are now on the page, which is more useful than "does all four plus Šidák" | | | ''multipletests'' accepts ''bonferroni, sidak, holm-sidak, holm, simes-hochberg, hommel, fdr_bh, fdr_by, fdr_tsbh, fdr_tsbky'' as of 0.14.6 | **yes** | The exact method strings are now on the page, which is more useful than "does all four plus Šidák" | |
| | e-values / e-BH and selective inference are active research but have no adopted standing and zero presence in this corpus, so //"nothing has been overturned"// holds | **yes, sharpened** | The currency box now says so explicitly instead of leaving it implicit. Rejecting e-BH as a recommendation is recorded in §7.5 | | | e-values / e-BH and selective inference are active research but have no adopted standing and zero presence in this corpus, so //"nothing has been overturned"// holds | **yes, sharpened** | The currency box now says so explicitly instead of leaving it implicit. Rejecting e-BH as a recommendation is recorded in §7.5 | |
| |
| **Rejected: none outright.** The one partial is the ''gelman2014_statistical'' page range from §11.2, left at ''460'' because no primary source for an end page could be found. | **Rejected: none outright.** The one partial is the ''gelman2014_statistical'' page range from §11.2, left at ''460'' because no primary source for an end page could be found. |
| | |
| | ==== 11.5 Re-review, both ''sonnet'' passes, after the fixes ==== |
| | |
| | The two focused reviewers whose findings were acted on were re-run against the edited pages, as the workflow requires. Both found real defects in the **new** material, which is the argument for re-running them. |
| | |
| | ^ Finding ^ Accepted? ^ What was done ^ |
| | | **Q23's regex is dominated by false positives.** Of the 41 unguarded hits, 24 are χ²-test notation — the digit in ''χ2 tests'' sits immediately before the word — plus exponents (''1.82e−06 Pairwise test'') and USENIX page furniture (''…Symposium 3589 test''). 15.5% is inflated roughly 2.4× | **yes** | The worst defect in this run and it was introduced //by a fix// for a previous finding. Three guards added, each inspecting the 40 characters before the digit, with **every rejected match counted by guard** (77 χ², 3 exponent, 1 page furniture). That leaves 20, and all 20 were then **read**: 15 genuine, 5 artefacts named with their reason in a hand list keyed on slug. The figure is now **15 of 265 = 5.7%**, and the page states it alongside the tuple figure's 4.9% instead of presenting a range. Every accepted match is printed with its sentence in the report output | |
| | | The complement, //"84.5% of corrections come without a nearby k"//, inherits the same error | **yes** | Now 94.3%, and the Open Questions bullet no longer implies CONSORT asks for //k// | |
| | | The page's //"upper bound"// framing was wrong in both directions: the window misses a count stated in a distant table caption //and// can admit an unrelated number | **yes** | The script and the methodology bullet now say explicitly that **neither pass is a bound** | |
| | | **CONSORT 2025 did not "add multiplicity".** CONSORT 2010's elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"//, item 7b already required stating adjustment for interim analyses, and item 20 already named //"multiplicity of analyses"// in Limitations. What 2025 adds is the explicit obligation to report that //no// method was used | **yes** | Rewritten to claim only the negative obligation, with the CONSORT 2010 history stated. **This is the clearest overclaim of the run**: the DOI and the quote were checked, and the word "added" was not | |
| | | The quoted sentence is in the **explanation and elaboration** (''…081124''), not in the statement/checklist (''…081123''), which contains no occurrence of "multiplicity" in its checklist text | **yes** | The citation was already to the E&E, which is correct; a footnote now says so and says the statement does not carry the sentence | |
| | | //"Nenadic et al. correct separately to the two families because they make two claims"// puts an inference in the paper's mouth: the two families are two contrast definitions (CH vs. EU, CH & EU vs. EU) of **one** research question, and the paper's stated reason is that it estimates separate models | **yes** | Both mentions rewritten to the paper's own reason and its own description of the split. The sentence is now more useful as well as more accurate | |
| | | **''nenadic2026_overcoming'' duplicates ''nenadic2026_swiss''** — same paper, same DOI, two keys, the second already in the bibliography from [[design:sampling]] | **yes** | The duplicate was removed from ''literature:bibliography'' and both pages now cite ''nenadic2026_swiss''. ''bibgen.mjs'' derives a key from title words and does not check the live bibliography for the same DOI; **checking the DOI, not just the key, is the lesson** | |
| | | §11.3 presented the CONSORT novelty claim as verified when only the DOI and the quote were checked | **yes** | §11.3 amended to say which parts were verified and to point here | |
| | | The Bonferroni surviving-uses claims were checked against the literature: Holm-compatible simultaneous confidence regions were an open problem until Strassburger & Bretz (2008) and Guilbaud (2008), and are noted there as often non-informative, while Bonferroni gives intervals at 1 − α/k by a union bound; and Bonferroni's threshold depends only on //k// where Holm's depends on each //p//-value's rank | **no change needed** | Both claims stand as written. This was the fix most likely to have overreached and it did not | |
| | | The Fisher implementation, the venue-mix control, the preregistration cross-check, the family-arithmetic, the headline box, the demo cross-references and every previously-checked figure reproduce exactly, including on 30 random 2×2 tables against ''scipy.stats.fisher_exact'' (max abs diff ≈ 5 × 10⁻¹⁴) | **no change needed** | — | |
| |
| ===== 12. The run itself ===== | ===== 12. The run itself ===== |
