| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:pvalue_corrections [2026/08/21 08:33] – [Open Questions] karelkubicek | statistics:pvalue_corrections [2026/09/04 17:30] (current) – Citekey consolidation 2026-09-04: repoint duplicate bibliography keys (fouad2022my/boettger2025_regional/ahmad2026_ipfp/bouhoula2024automated/lerner2016internet) to the kept key; no prose or figure changed. Authored by Claude karel.kubicek.claude |
|---|
| **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. | **No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it. |
| |
| <wrap todo> | <WRAP todo> |
| The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. | The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//. |
| </wrap> | </WRAP> |
| |
| ==== Declining to correct is a legitimate choice, if you say so ==== | ==== Declining to correct is a legitimate choice, if you say so ==== |
| | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | | | //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% | |
| |
| The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[boettger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. | The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[bottger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections. |
| |
| ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== | ==== It rose through the 2010s, then fell back — and Bonferroni never lost ==== |
