Working notes behind pvalue_corrections — every query with its population and denominator, the scripts and their unedited output, the fold and its residue, the quotes checked against source papers, the external sources and how each was verified or rejected, the review log, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Creating, not extending, and not overlapping. statistics:pvalue_corrections was a red link promised from start. The only neighbouring page that existed when this was written is study_preregistration, which touches multiplicity in one subsection (“For scale: the rest of the inference-hygiene stack is thin too”) and cites this page forward. That subsection's figures (15.3% of inferential, 24.5% of hypothesis-test papers) agree with this page's; this page keeps the hypothesisTest denominator throughout and adds the corrected-vs-tuple distinction the preregistration page did not need. Nothing was overwritten and no neighbouring figure was contradicted.

Reachability needed no work: start already lists P-value corrections in its outline, and study_preregistration links it twice.

The task specification quoted a figure from the old corpus. Both the numerator and the denominator moved, and one distinction the old figure did not make turned out to matter.

Two things the task's framing got right and one it did not. Right: the correction is skipped in the setting that needs it, and the page's central table quantifies that (12.9% of crawl papers vs 38.6% of user studies). Not right: inferential is the wrong denominator. A paper whose only statistics tuple is correlation or regression may have no p-value family at all, and 41.5% of the corpus never gets past descriptives. The population that needs a correction is the 1,025 papers with a hypothesis-test tuple, and the page uses that throughout. Over inferential the rate is 15.0%; over hypothesisTest it is 24.1%. Publishing 15.0% as “how often the field corrects” would have understated it by a third.

inferential, crawled and humanSubjects match OVERVIEW.md exactly (1,762 / 1,120 / 1,357), which is the check that the report's population code is the same code as the dossier's.

cd /workspace/artifacts/wiki
node scripts/mcc_fold.mjs                                   # fold self-test
node scripts/report_pvalue_corrections.mjs                  # every figure on the page
node scripts/report_pvalue_corrections.mjs --wiki           # DokuWiki tables
node scripts/quote_check.mjs --statistics multiple-comparison-correction --show 62
python3 out/adjust_pvalues.py --demo                        # the published code's output
cat out/pvalue-report.txt out/adjust_pvalues_demo.txt \
    out/pvalue-external-facts.txt out/pvalue-quotecheck.txt > out/pvalue-check-source.txt
node scripts/check_page_numbers.mjs pages/statistics_pvalue_corrections.txt \
    out/pvalue-check-source.txt --code

statistics[].method is free text and agrees run-to-run on ~0.2 of exact strings (data/extract/README.md), so it is folded before any aggregation. mcc_fold.mjs is an ordered rule list, first match wins, and a string matching no rule is returned as null and printed.

Unmapped residue: 0 distinct strings, 0 papers. All 90 distinct method strings across the 292 tuples fold.

Three orderings would silently mis-file entries if reversed, and the self-test asserts each:

node scripts/mcc_fold.mjs runs 34 cases:

34/34 fold self-tests passed

Dunn, Scheffé and Nemenyi have rules that match nothing in this corpus on their own — every corpus string naming Dunn also names Bonferroni or Holm (Dunn post-hoc tests with Holm-Bonferroni method → Holm; Dunn-Bonferroni post hoc comparison test → Bonferroni), and Scheffé and Nemenyi never appear. The rules are kept with self-test cases so they stay tested rather than rotting, and this is recorded here rather than left as an unexplained empty family.

The extraction scores a tuple as multiple-comparison-correction when the sentence is about multiple comparisons, which catches two kinds of thing that are not one. Both are excluded from every “corrected” figure on the page and both are shown in the page's own procedure table, because deleting them would be the mistake.

A query that counts kind == “multiple-comparison-correction” and stops there reports 269 where the answer is 265, and reports three papers that explicitly declined as having complied.

Eight papers describe an adjustment without naming a published procedure. They are folded to Unnamed / ad-hoc adjustment rather than guessed at, and they are listed here because the wording is the interesting part:

• multiple hypothesis correction (USENIX 2021), correcting for multiple hypothesis testing (USENIX 2023), multiple-comparison corrected (PoPETs 2024) — a correction was applied and the reader cannot tell which
• significance threshold divided by the number of groups (IEEE S&P 2024) — a described Bonferroni, deliberately not folded to Bonferroni because the string does not name it
• correction factor of two (USENIX 2024), p-value threshold adjustment for six repeated tests (IEEE S&P 2024), conservative α = 0.002 (PoPETs 2024) — a numerator and no procedure
• multiple test procedures (NDSS 2019) — feature selection, arguably a false positive of the schema, kept in the bucket rather than removed by hand

Q23's full-text pass needs its own fold, for the same reason mcc_fold.mjs does: the naive pattern is dominated by a homograph. /\d+\s*tests?/ matches the degrees-of-freedom digit in χ² notation, which is everywhere in this literature.

Twenty matches survive the guards and all twenty were read. Four are still artefacts and are a hand list in the report script, keyed on slug so a future corpus prints an unclassified paper rather than silently bucketing it:

One further hit is genuine as a paper and wrong as a number, and is annotated rather than rejected: deliberate-exposure-to-opposing-views… states its family as “we make 3 comparisons with each population” and the regex matched a later 12 comparisons, the total across populations. The paper does state k, so it counts.

15 genuine of 265 papers = 5.7%. The report prints all twenty with their surrounding sentences, so the figure can be discounted line by line rather than taken on trust.

Bulk pass over every correction quote, with the –statistics filter added to quote_check.mjs for this page:

$ node scripts/quote_check.mjs --statistics multiple-comparison-correction
292 quotes checked: 167 exact, 63 partial (>=60% of 5-word windows), 62 below threshold, 0 with no full text on disk.

62 below threshold is 21% and higher than this tool usually reports. The first 40 of the 62 were read (the tool's –show 40 default output, kept in out/pvalue-quotecheck.txt); the remaining 22 were not read. In all 40 read, the cause is not fabrication: correction quotes are short (median well under 20 words), so one dropped citation marker or one extractor ellipsis destroys a large fraction of the five-word windows. “Fisher's binomial proportion test … with a Bonferroni correction.” scores 0% and is real. The 22 unread ones are an open item — nothing on the content page depends on them, since every quote the page uses was checked by hand (below), but they are not evidence for anything either.

Every quote used on the content page was then located by hand in data/fulltext/<year>/<venue>/<slug>/paper.cols.txt. All 18 checked out:

One thing found while checking and not put on the page. [16Kablo, Emiram; Cabarcos, Patricia Arias (2023): "Privacy in the Age of Neurotechnology: Investigating Public Attitudes towards Brain Data Collection and Use", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] writes “we adjusted the standard threshold of 0.05 to a new threshold of 0.00047 (0.005 / 106)”. 0.05/106 = 0.00047, so the printed threshold is right and the parenthetical divisor has a typo, or is a PDF-extraction artefact of “0.05”. It is a one-character slip in an otherwise exemplary paper and calling it out on the page would be a gotcha, not a lesson. Recorded here instead.

Everything below was fetched on 2026-08-13, not recalled.

curl -sL -H "Accept: application/vnd.citationstyles.csl+json" "https://doi.org/$DOI"

Nine DOIs were checked this way. Eight are the external statistics references; the ninth, CONSORT 2025, was added after the currency review (§11.3) and is listed at the bottom of the table.

The Annals of Statistics record returns no page range; 1165–1188 in the bibliography entry is the standard citation and is not confirmed by Crossref. Every other volume, issue and page range above is as returned.

Worth recording because it is exactly the kind of thing that gets copied into a bibliography and never checked.

$ curl -sI "https://doi.org/10.2307/4615733" | head -1
HTTP/2 404
$ curl -s "https://doi.org/api/handles/10.2307/4615733"
{"responseCode":100,"handle":"10.2307/4615733"}          # 100 = handle not found
$ curl -s "https://api.crossref.org/works/10.2307/4615733" -o /dev/null -w "%{http_code}\n"
404

OpenAlex nonetheless reports doi: https://doi.org/10.2307/4615733 for the work (W2121044470, 21,859 citations), with biblio: {volume: 6, issue: null, first_page: 65, last_page: 70}. The DOI is a MAG-derived artefact: 10.2307 is JSTOR's prefix, and JSTOR did not register a DOI for this article. The bibliography entry therefore carries a url and a note saying so, not a doi.

jstor.org could not be read either: HTTP 403 to curl, and a reCAPTCHA wall to Playwright's own Chromium from this network (“JSTOR: Access Check”, block reference recorded). So the URL is confirmed to be served by JSTOR and the article page itself was not read. Volume and pages come from OpenAlex; DBLP has no record; Europe PMC does not index the journal; the Semantic Scholar API returned HTTP 429. This is the weakest-sourced entry in the bibliography and it is the most-cited paper on the page.

numpy 2.4.6, scipy 1.17.1, statsmodels 0.14.6, installed into the sandbox with pip install –break-system-packages. The page publishes the unedited –demo output.

$ python3 -c "import numpy,scipy,statsmodels; print(numpy.__version__, scipy.__version__, statsmodels.__version__)"
2.4.6 1.17.1 0.14.6

# holm() and bonferroni() vs statsmodels, five random 50-value families:
0 holm max abs diff 0.0     0 bonf max abs diff 0.0
1 holm max abs diff 0.0     1 bonf max abs diff 0.0
2 holm max abs diff 0.0     2 bonf max abs diff 0.0
3 holm max abs diff 0.0     3 bonf max abs diff 0.0
4 holm max abs diff 0.0     4 bonf max abs diff 0.0

p = [0.001 0.02  0.03 ]
bonferroni -> [0.003 0.06  0.09 ] survivors 1
holm       -> [0.003 0.04  0.04 ] survivors 3
statsmodels holm -> [0.003 0.04  0.04 ]

Error paths were exercised too: a non-numeric line raises ValueError: could not convert string to float: 'not-a-number' rather than being skipped, which is the behaviour the docstring promises.

Note that the demo output on the page was produced with SciPy 1.17.1 while the documentation quoted on the page is 1.18.0. false_discovery_control has not changed between them as far as the release notes show, but the two version numbers on the page are deliberately different and neither is a typo.

• Whether any of the 65 mixed-effects papers frames its model as a multiplicity strategy. The page says “not one frames it as an answer to multiplicity” in the body and immediately qualifies it in a footnote: none surfaced while reading correction passages, and the 65 were not read one by one. This is the softest claim on the page and it is flagged as an open question there too. Closing it is 65 paper-reads.
• Whether the other 37 power-analysis-plus-correction papers computed the correction before sizing the study. The extraction carries no ordering between two statistics tuples, so only [17Ho, Grant; Mirian, Ariana; Luo, Elisa; Tong, Khang; Lee, Euyhyun; Liu, Lin; Longhurst, Christopher A.; Dameff, Christian; Savage, Stefan; Voelker, Geoffrey M. (2025): "Understanding the Efficacy of Phishing Training in Practice", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)], whose sentence states the sequence, can be claimed. The page says so.
• Why NDSS is lowest. 10.8% is the lowest venue rate and the page reports it without an explanation. A plausible one — NDSS publishes proportionally more systems papers whose statistics are classifier metrics rather than hypothesis tests — is not tested here. Note that classifier metrics live in detection.metric, not statistics, by design, so they are not what depresses this number.
• Whether the 33 schema-missed papers are real misses. They are listed in full in the report output. They were not read individually; the 88.2% recall figure treats all 33 as misses, which is the conservative direction — a paper mentioning Bonferroni only in related work is a sweep false positive, not a schema miss, and would raise the recall figure.
• The true correction rate. Every figure here is a rate of reporting, not of practice. A paper that corrected and did not write it down is indistinguishable from one that did not correct. statistics has no not-stated sentinel, so silence and absence are the same value.
• Whether uncorrected findings in this literature are wrong. The obvious study — take papers that ran many tests without correcting, recompute their p-values from released data, see how many claims survive — needs released p-values, which is an Artifacts problem. Listed as an open question on the page.

Q13's full list, since the page quotes only the count and one paper. Population: the 1,025 hypothesisTest papers.

PETS/2017/to-permit-or-not-to-permit-that-is-the-usability-question-crowdsourcing-mobile-a
PETS/2020/the-best-of-both-worlds-mitigating-trade-offs-between-accuracy-and-user-burden-i
USENIX/2020/i-am-uncomfortable-sharing-what-i-cant-see-privacy-concerns-of-the-visually-impa
PETS/2021/warn-them-or-just-block-them-investigating-privacy-concerns-among-older-and-work
PETS/2021/awareness-adoption-and-misconceptions-of-web-privacy-tools
USENIX/2021/effect-of-mood-location-trust-and-presence-of-others-on-video-based-social-authe
USENIX/2021/evaluating-in-workflow-messages-for-improving-mental-models-of-end-to-end-encryp
USENIX/2021/on-the-usability-of-authenticity-checks-for-hardware-security-tokens
WWW/2021/an-experimental-study-to-understand-user-experience-and-perception-bias-occurred
IEEE-SP/2020/influencing-photo-sharing-decisions-on-social-media-a-case-of-paradoxical-findin
IEEE-SP/2020/are-anonymity-seekers-just-like-everybody-else-an-analysis-of-contributions-to-w
IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement
PETS/2022/increasing-adoption-of-tor-browser-using-informational-and-planning-nudges
PETS/2022/personal-information-inference-from-voice-recordings-user-awareness-and-privacy
USENIX/2022/where-to-recruit-for-security-development-studies-comparing-six-software-develop
USENIX/2023/exploring-tenants-preferences-of-privacy-negotiation-in-airbnb
USENIX/2023/if-sighted-people-know-i-should-be-able-to-know-privacy-perceptions-of-bystander
PETS/2024/how-does-connecting-online-activities-to-advertising-inferences-impact-privacy-p
PETS/2024/supporting-informed-choices-about-browser-cookies-the-impact-of-personalised-coo
IEEE-SP/2024/a-picture-is-worth-500-labels-a-case-study-of-demographic-disparities-in-local-m
PETS/2024/simply-tell-me-how-on-trustworthiness-and-technology-acceptance-of-attribute-bas
USENIX/2024/engaging-company-developers-in-security-research-studies-a-comprehensive-literat
PETS/2024/what-does-it-mean-to-be-creepy-responses-to-visualizations-of-personal-browsing
USENIX/2024/simulated-stress-a-case-study-of-the-effects-of-a-simulated-phishing-campaign-on
IEEE-SP/2025/not-the-right-question-a-study-on-attitudes-toward-client-side-scanning-with-sec
IEEE-SP/2025/its-time-time-for-digital-security-an-end-user-study-on-actionable-security-and
PETS/2025/ai-is-from-the-devil-behaviors-and-concerns-toward-personal-data-sharing-with-ll
PETS/2025/privacy-perceptions-and-behaviors-towards-targeted-advertising-on-social-media-a
USENIX/2025/vulnerability-of-text-matching-in-ml-ai-conference-reviewer-assignments-to-collu
PETS/2026/the-city-isn-t-uploading-me-to-tiktok-exploring-privacy-attitudes-towards-data-c
USENIX/2026/you-know-why-but-still-rely-the-impact-of-explainable-ai-on-trust-task-load-and
PETS/2026/are-bite-size-data-safety-details-a-healthy-diet-for-android-telehealth-app-user
NDSS/2025/the-kids-are-all-right-investigating-the-susceptibility-of-teens-and-adults-to-youtube-giveaway-scams
PETS/2025/who-cares-contextual-privacy-judgments-from-owner-and-bystander-perspectives-in
IEEE-SP/2025/lets-get-visual-testing-visual-analogies-and-metaphors-for-conveying-privacy-pol
USENIX/2025/malicious-llm-based-conversational-ai-makes-users-reveal-personal-information
IEEE-SP/2024/casual-users-and-rational-choices-within-differential-privacy
IEEE-SP/2024/investigating-voter-perceptions-of-printed-physical-audit-trails-for-online-voti

36 of the 38 recruited human participants. The two that did not are IEEE-SP/2020/are-anonymity-seekers-just-like-everybody-else and IEEE-SP/2024/a-picture-is-worth-500-labels. Exactly one of the 38 ran a crawl: PETS/2024/what-does-it-mean-to-be-creepy, whose studyTypes includes automated-web-crawl and which also recruited participants. So the population that sizes a study with a correction in the calculation is, with one partial exception, a user-study population.

An earlier version of this section said “37 of the 38” and named IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising as the exception and as a crawl. All three parts of that were wrong — the count is 36, that paper has participants tuples, and its studyTypes are ['user-study', 'manual-audit'] with crawlConfig === null, so it is not a crawl by the definition in §3. It was caught by the figures-vs-script reviewer and is recorded here rather than silently corrected, because it is the clearest example on this page of a claim written from memory instead of from a query.

Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the report script, its unedited output, and these notes. The three focused passes ran in parallel first; the generic pass ran after their findings were applied.

Re-ran all three scripts (byte-identical to the committed output), re-derived the 1,025 / 269 / 141 anchor counts from extractions.jsonl in an independent Python re-implementation of the fold, and cross-checked holm()/bonferroni() against statsmodels on 20 random families plus edge cases (all-ones, single value, ties): max abs diff 0.0.

One correction to the reviewer, recorded for the record: it reported that IMC/2022/what-factors-affect-targeting-and-bids “has 286 participants (recruited via Prolific)”. The participants array on that record has 2 tuples, not 286 people; the reviewer conflated a tuple count with a headcount. Its substantive point — that the paper has participants and is not a crawl — is correct and was the basis for the fix.

Verified all 27 citekeys resolve with no collisions, all 17 corpus entries against data/corpus2/.meta and the papers' own front matter, all 8 external DOIs through doi.org, and all 18 quotations against paper.cols.txt.

No checklist; run after the three focused passes were applied. It returned fifteen findings and fourteen were accepted, which is the highest accept rate of the four reviewers and the reason this slot is worth keeping.

Rejected: none outright. The one partial is the gelman2014_statistical page range from §11.2, left at 460 because no primary source for an end page could be found.

The two focused reviewers whose findings were acted on were re-run against the edited pages, as the workflow requires. Both found real defects in the new material, which is the argument for re-running them.

• pvalue_corrections — the page these notes back.
• corpus — the corpus-level provenance: how the 5,859 papers were selected, extracted and validated.
• study_preregistration — the sibling statistics: provenance page, and the source of this page's format.

[1]

Pu, Yu; Grossklags, Jens (2016): "Towards a Model on the Factors Influencing Social App Users' Valuation of Interdependent Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[2]

Goetzen, Angelica; Dooley, Samuel; Redmiles, Elissa M. (2022): "Ctrl-Shift: How Privacy Sentiment Changed from 2019 to 2021", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[3]

Naji, Houda; Reichmann, Felix; Bruns, Tobias; Sasse, M. Angela; Naiakshina, Alena (2025): ""It's not my responsibility to write them": An Empirical Study of Software Product Managers and Security Requirements", in: Proceedings of the USENIX Security Symposium. (Link)

[4]

Böttger, Christian; Demir, Nurullah; Hörnemann, Jan; Acharya, Bhupendra; Pohlmann, Norbert; Holz, Thorsten; Grosse-Kampmann, Matteo; Urban, Tobias (2025): "Understanding Regional Filter Lists: Efficacy and Impact", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[5]

Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[6]

Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[7]

Weinberg, Zachary; Sharif, Mahmood; Szurdi, Janos; Christin, Nicolas (2017): "Topics of Controversy: An Empirical Analysis of Web Censorship Lists", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[8]

Zimmeck, Sebastian; Goldelman, Daniel; Kaplan, Owen; Brown, Logan; Casler, Justin; Jean-Charles, Judeley; Champeau, Joe; Harkous, Hamza (2024): "Website Data Transparency in the Browser", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[9]

Maass, Max; Stöver, Alina; Pridöhl, Henning; Bretthauer, Sebastian; Herrmann, Dominik; Hollick, Matthias; Spiecker, Indra (2021): "Effective Notification Campaigns on the Web: A Matter of Trust, Framing, and Support", in: Proceedings of the USENIX Security Symposium. (Link)

[10]

Valapu, Sulyab Thottungal; Heidemann, John S. (2025): "Towards a Non-Binary View of IPv6 Adoption", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[11]

Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)

[12]

Liu, Zengrui; Iqbal, Umar; Saxena, Nitesh (2024): "Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[13]

Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)

[14]

Bobek, Michelle; Pröllochs, Nicolas (2026): "Community Fact-Checks Do Not Break Follower Loyalty", in: Proceedings of the ACM Web Conference. (DOI)

[15]

Despres, Tess; Constantino, Marcelino Ayala; Lizola, Naomi Zacarias; Romero, Gerardo Sánchez; He, Shijing; Zhan, Xiao; Abdi, Noura; Abu-Salma, Ruba; Such, Jose; Bernd, Julia (2024): ""My Best Friend's Husband Sees and Knows Everything": A Cross-Contextual and Cross-Country Approach to Understanding Smart Home Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[16]

Kablo, Emiram; Cabarcos, Patricia Arias (2023): "Privacy in the Age of Neurotechnology: Investigating Public Attitudes towards Brain Data Collection and Use", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[17]

Ho, Grant; Mirian, Ariana; Luo, Elisa; Tong, Khang; Lee, Euyhyun; Liu, Lin; Longhurst, Christopher A.; Dameff, Christian; Savage, Stefan; Voelker, Geoffrey M. (2025): "Understanding the Efficacy of Phishing Training in Practice", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[18]

Hopewell, Sally; Chan, An-Wen; Collins, Gary S.; Hróbjartsson, Asbjørn; Moher, David; Schulz, Kenneth F.; others, (2025): "CONSORT 2025 explanation and elaboration: updated guideline for reporting randomised trials", BMJ 389:e081124. (DOI)