| Both sides previous revisionPrevious revision | |
| provenance:statistics:pvalue_corrections [2026/08/13 07:23] – Add section 5.4 (the family-size guards, their rejection counts and the four-paper hand list) and section 11.5 (the re-review log for both sonnet passes, including the guard bug that a previous fix introduced, the CONSORT novelty overclaim, and the duplic karel.kubicek.claude | provenance:statistics:pvalue_corrections [2026/08/13 07:25] (current) – Record the fifth family-size artefact (PETS 2022 multi-region: running header and table cells interleaved) and the one hit that is genuine as a paper but wrong as a number (WWW 2024 deliberate-exposure states k=3; the regex matched a 12-comparison total). karel.kubicek.claude |
|---|
| | Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 | | | Q21 | Is the partial-pooling alternative present? | 5,869 | 65 papers fit a mixed-effects/multilevel/hierarchical model. **Not** checked paper by paper for whether any frames it as a multiplicity strategy — see §8 | |
| | Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment | | | Q22 | Permutation tests as a multiplicity device? | 5,869 | 26 papers, 8 read; every one uses it as the test itself, not as a max-//T// adjustment | |
| | Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | unguarded regex: 41 hits, **mostly χ² notation** (see §5.4). With three guards: 20. **Hand-reading all 20: 16 genuine = 6.0%.** Q10's tuple figure is 4.9%; the two agree to about a point | | | Q23 | Does the **full text** state a family size near the correction? | the 265 that corrected | unguarded regex: 41 hits, **mostly χ² notation** (see §5.4). With three guards: 20. **Hand-reading all 20: 15 genuine = 5.7%.** Q10's tuple figure is 4.9%; the two agree to within a point | |
| | Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** | | | Q24 | Is the period trend real? | the 1,025 | Fisher's exact, two-sided, hand-implemented in the report script and matched against ''scipy.stats.fisher_exact'': 2010–2014 vs 2020–2024 //p// = **2.1 × 10⁻⁵**; 2020–2024 vs 2025–2026 //p// = **0.0328**; 2015–2019 vs 2025–2026 //p// = **0.5689** | |
| | Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets | | | Q25 | Does venue composition explain the 2025–2026 fall? | the 1,025 | **No.** Expected rate from each period's venue mix alone: 18.8% / 22.1% / 25.4% / **25.6%**, against observed 9.3% / 23.3% / 28.7% / **21.0%**. The mix barely moved between the last two buckets | |
| | ''sunlight-fine-grained-targeting-detection…'' | ''836 hypothesis'' | counts //discoveries//, not the family: //"836 hypothesis below 5%"// | | | ''sunlight-fine-grained-targeting-detection…'' | ''836 hypothesis'' | counts //discoveries//, not the family: //"836 hypothesis below 5%"// | |
| | ''e-vote-your-conscience…'' | ''2 Pairwise test'' | figure axis labels spliced together: //"3 3 2 Pairwise test"// | | | ''e-vote-your-conscience…'' | ''2 Pairwise test'' | figure axis labels spliced together: //"3 3 2 Pairwise test"// | |
| | | ''a-multi-region-investigation-of-the-perceptions…'' | ''12 comparisons'' | running header and table cells interleaved. The sentence is //"Dunn's tests (multiple pairwise comparisons)"// with no count; the 12 is page furniture. **Found by reading the source, after the re-review, not by a reviewer** | |
| |
| **16 genuine of 265 papers = 6.0%.** The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust. | One further hit is genuine as a //paper// and wrong as a //number//, and is annotated rather than rejected: ''deliberate-exposure-to-opposing-views…'' states its family as //"we make 3 comparisons with each population"// and the regex matched a later ''12 comparisons'', the total across populations. The paper does state //k//, so it counts. |
| | |
| | **15 genuine of 265 papers = 5.7%.** The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust. |
| |
| ===== 6. Quotes checked against source ===== | ===== 6. Quotes checked against source ===== |
| |
| ^ Finding ^ Accepted? ^ What was done ^ | ^ Finding ^ Accepted? ^ What was done ^ |
| | **Q23's regex is dominated by false positives.** Of the 41 unguarded hits, 24 are χ²-test notation — the digit in ''χ2 tests'' sits immediately before the word — plus exponents (''1.82e−06 Pairwise test'') and USENIX page furniture (''…Symposium 3589 test''). 15.5% is inflated roughly 2.4× | **yes** | The worst defect in this run and it was introduced //by a fix// for a previous finding. Three guards added, each inspecting the 40 characters before the digit, with **every rejected match counted by guard** (77 χ², 3 exponent, 1 page furniture). That leaves 20, and all 20 were then **read**: 16 genuine, 4 artefacts named with their reason in a hand list keyed on slug. The figure is now **16 of 265 = 6.0%**, and the page states it alongside the tuple figure's 4.9% instead of presenting a range. Every accepted match is printed with its sentence in the report output | | | **Q23's regex is dominated by false positives.** Of the 41 unguarded hits, 24 are χ²-test notation — the digit in ''χ2 tests'' sits immediately before the word — plus exponents (''1.82e−06 Pairwise test'') and USENIX page furniture (''…Symposium 3589 test''). 15.5% is inflated roughly 2.4× | **yes** | The worst defect in this run and it was introduced //by a fix// for a previous finding. Three guards added, each inspecting the 40 characters before the digit, with **every rejected match counted by guard** (77 χ², 3 exponent, 1 page furniture). That leaves 20, and all 20 were then **read**: 15 genuine, 5 artefacts named with their reason in a hand list keyed on slug. The figure is now **15 of 265 = 5.7%**, and the page states it alongside the tuple figure's 4.9% instead of presenting a range. Every accepted match is printed with its sentence in the report output | |
| | The complement, //"84.5% of corrections come without a nearby k"//, inherits the same error | **yes** | Now 94.0%, and the Open Questions bullet no longer implies CONSORT asks for //k// | | | The complement, //"84.5% of corrections come without a nearby k"//, inherits the same error | **yes** | Now 94.3%, and the Open Questions bullet no longer implies CONSORT asks for //k// | |
| | The page's //"upper bound"// framing was wrong in both directions: the window misses a count stated in a distant table caption //and// can admit an unrelated number | **yes** | The script and the methodology bullet now say explicitly that **neither pass is a bound** | | | The page's //"upper bound"// framing was wrong in both directions: the window misses a count stated in a distant table caption //and// can admit an unrelated number | **yes** | The script and the methodology bullet now say explicitly that **neither pass is a bound** | |
| | **CONSORT 2025 did not "add multiplicity".** CONSORT 2010's elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"//, item 7b already required stating adjustment for interim analyses, and item 20 already named //"multiplicity of analyses"// in Limitations. What 2025 adds is the explicit obligation to report that //no// method was used | **yes** | Rewritten to claim only the negative obligation, with the CONSORT 2010 history stated. **This is the clearest overclaim of the run**: the DOI and the quote were checked, and the word "added" was not | | | **CONSORT 2025 did not "add multiplicity".** CONSORT 2010's elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"//, item 7b already required stating adjustment for interim analyses, and item 20 already named //"multiplicity of analyses"// in Limitations. What 2025 adds is the explicit obligation to report that //no// method was used | **yes** | Rewritten to claim only the negative obligation, with the CONSORT 2010 history stated. **This is the clearest overclaim of the run**: the DOI and the quote were checked, and the word "added" was not | |
