Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== P-value Corrections ======

A web measurement generates hypotheses the way a crawl generates rows: one test per country, per tracker, per site category, per CMP, per browser profile, per consent arm — and then all the pairs of those. The number of tests is a **property of the data you collected**, not a decision you made, and it is usually in the hundreds before you have noticed. That is what makes multiplicity a measurement problem rather than a statistics-course problem, and it is why this page exists: the arithmetic of Bonferroni is in every textbook, but nothing in a textbook tells you what the family //is// when your unit of analysis is a site and your strata are whatever the ranking list happened to contain.

This page assumes you know what a //p//-value and a type-I error are. It is about three things a textbook will not tell you: where the multiplicity comes from in a crawl, which procedure is current and which is merely traditional, and what to report so a reviewer can check you.

<WRAP important>
**Roughly a quarter of the papers that run a hypothesis test report a correction, and whether they do tracks whether they recruited people — not how many hypotheses they had.**

Of the 5,859 papers extracted from seven security, privacy and measurement venues (2010–2026), **1,025 ran a hypothesis test**. Of those, **247 (24.1%) report a multiple-comparison correction.** Split by the shape of the study:

  * **38.6%** of the 433 that recruited participants and ran no crawl
  * **34.7%** of the 49 that did both
  * **12.9%** of the 140 that ran a crawl and recruited nobody
  * **11.2%** of the 403 that did neither

The two participant rows are the two high ones. **Correcting is a habit that arrived with human-subjects methodology, and a crawl paper that runs a hypothesis test corrects at a third the rate of a user study that does** — in the setting where the number of candidate hypotheses is set by the data rather than by a protocol. By venue: **PoPETs 45.6%**, USENIX Security 28.5%, IEEE S&P 26.6%, CCS 20.9%, TheWebConf 14.4%, **IMC 14.1%**, **NDSS 10.8%**.

**Bonferroni is still the modal choice in this field, including in 2025–2026, and for testing there is no longer a reason to prefer it.** Of the 269 papers with a correction tuple, **141 (52.4%) use Bonferroni**, 66 (24.5%) Holm, 43 (16.0%) Benjamini–Hochberg, 4 (1.5%) Benjamini–Yekutieli. Holm {[holm1979_sequentially]} needs exactly the same assumptions as Bonferroni and can never reject fewer hypotheses — Datta et al. said so in this corpus in 2015 {[datta2015_automated]} — so //"we applied a Bonferroni correction"// should read //"we applied a Holm correction"// almost everywhere it is written. It survives for two things Holm does not do: **simultaneous confidence intervals**, and a **threshold you can pre-declare** before any //p//-value exists. See [[#Which Procedure, and Which Are Current]].

And **hardly anyone says how many hypotheses were in the family: about one corrected paper in twenty.** Two independent measurements agree. 14 of 287 correction tuples (4.9%) put a count in the extracted method, detail or evidence sentence. Searching the whole paper instead — any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention, then reading every hit — gives **15 of 265 papers (5.7%)**. Without //k// the adjusted //p// is not reconstructible and the correction is not checkable.
</WRAP>

===== What to Read First =====

  * **Sunlight** {[lecuyer2015_sunlight]} (CCS 2015) — the only paper found in this corpus that treats multiple-testing correction as a first-class stage of a web-measurement pipeline rather than a line in the results section — it offers both a FWER and an FDR procedure and **measures what the choice costs at scale**.(("Only" here means: of the 269 papers with a correction tuple, this is the one whose correction is a named architectural stage. The claim comes from reading all 269 method strings and the correction passages of the 37 crawl papers among them, not from a search that could rule out a paper with no correction tuple at all.)) Read §4.4 and §6.4–6.5. Everything in [[#The Power Cost Is Not Abstract, and Somebody Measured It]] comes from it.
  * **Automated Experiments on Ad Privacy Settings** {[datta2015_automated]} (PoPETs 2015) — the AdFisher paper. Its statistical apparatus is the model for an automated web experiment, and it is where the //"uniformly more powerful"// argument for Holm over Bonferroni appears inside this literature rather than in a statistics journal.
  * **Holm 1979** {[holm1979_sequentially]} — four pages, and the only reason you need to read it is to be able to say in a rebuttal that the procedure makes no extra assumption.
  * **Benjamini & Hochberg 1995** {[benjamini1995_controlling]} and **Benjamini & Yekutieli 2001** {[benjamini2001_control]} — FDR, and FDR when the tests are dependent. The second is the one a crawl usually needs and almost nobody in this corpus uses.
  * **Gelman, Hill & Yajima 2012** {[gelman2012_multiple]} — the argument for modelling the strata jointly instead of correcting across them. It is the strongest alternative to everything else on this page and it is invisible in these venues.
  * **Gelman & Loken 2014** {[gelman2014_statistical]} — the garden of forking paths. Correcting the tests you report does nothing about the tests you decided not to run, and in a crawl pipeline that is where most of the multiplicity lives. **No paper in this corpus cites the idea:** the literal string //forking paths// matches two of the 5,869 full texts and both are false positives (packet-forwarding paths, symbolic-execution paths).

===== Where the Multiplicity Actually Comes From =====

The reason a measurement paper ends up with a large //k// without anyone choosing one is that the strata are inherited from the data source. Each row below is a real design from this corpus, with its own arithmetic.

^ Design ^ How //k// is set ^ Real example ^
| **Per-country or per-vantage comparison** | one test per country, or //k//(//k//−1)/2 if you report all pairs. 13 vantage points is 78 pairs | Valapu & Heidemann {[valapu2025_binary]} compare 13 clouds pairwise for IPv6 adoption and correct with Holm–Bonferroni //"to control the family-wise error rate"// |
| **Per-persona or per-profile arm** | one test per synthetic profile | Liu et al. {[liu2024_opted]}: //"compare bid values for all 16 personas"//, //p// //"multiplied by 16"// |
| **Per-outcome × per-condition grid** | rows × columns, minus the cells that do not apply. This is the one that surprises people | Kablo & Cabarcos {[kablo2023_privacy]}: 10 recipients × 15 transmission principles, //"due to the inapplicability of some principles to certain recipients, a total of 106 tests were conducted"// |
| **Per-list or per-language correlation matrix** | every list against every topic | Weinberg et al. {[weinberg2017_topics]}: //"a total of 2,904 such combinations"//; after Bonferroni, //"585 of the topic-list correlations and 580 of the topic-language correlations are significant"// |
| **Per-treatment survival comparison over time** | arms × time points | Maass et al. {[maass2021_effective]}: //"a single Holm-Bonferroni correction [24] for all 45 significance tests"// |
| **Per-violation-type compliance test** | one test per legal requirement per subgroup, and again per subgroup comparison | Bouhoula et al. {[bouhoula2024_automated]} apply Holm–Bonferroni twice: to pairwise violation comparisons across popularity ranks, and to Fisher's exact tests comparing each CMP's subset against all websites |
| **Per-regression-coefficient interaction term** | one test per coefficient, and the family is the model, not the paper | Nenadic et al. {[nenadic2026_swiss]}: //"Because we estimate separate models for multiple disclosure obligations, conducting parallel hypothesis tests increases the risk of false positives"// — BH //"applied separately to the two families"// |
| **Per-site test** | one test per site. This is where //k// stops being tractable and the answer is a model, not a correction | none of the 269 papers with a correction tuple does this. Whether papers run per-site tests //without// correcting is not something this corpus was queried for |

Two consequences a reviewer will pick up on.

**Nesting the strata multiplies, it does not add.** Three consent arms × four vantage points × eight site categories is 96 cells, and every pairwise contrast you draw from them is another test. The natural thing to write in a paper — //"we compared each pair"// — is the thing that makes //k// quadratic.

**The last row of the table is the honest one.** Once the natural unit of the test is the site, //k// is the size of the crawl, the correction is annihilating, and the procedure is not the answer. See [[#When Correcting Is the Wrong Lever]].

===== Deciding What Counts as One Family =====

This is the decision the procedures cannot make for you, it is the one a textbook does not cover, and it is where a crawl-shaped study goes wrong. Three consent arms × 40 countries is 120 cells. Is that one family of 120 tests, three families of 40, 40 families of 3, or — if you report every pairwise country contrast inside each arm — one family of 3 × 780 = 2,340? The answer changes every number in your results table, and **any of those answers is defensible if you fixed it before you looked.** None of them is defensible if you chose after.

The rule the corpus's better papers follow, stated as a rule:

  - **One family per confirmatory claim you will make in the abstract.** If the paper's claim is "consent rejection increases tracking", the family is the tests that support that claim — not every test in the paper. Nenadic et al. {[nenadic2026_swiss]} do this at the model level: because they //"estimate separate models for multiple disclosure obligations"//, they apply Benjamini–Hochberg //"separately to the two families of interaction terms (CH vs. EU and CH & EU vs. EU)"// — two contrast definitions of one research question, corrected apart rather than pooled.
  - **Then, per outcome variable.** Despres et al. {[despres2024_best]} fix α from //"a maximum of 25 hypotheses tested per outcome"//; Zimmeck et al. {[zimmeck2024_website]} correct across 231 comparisons drawn from one comparison set. Both state the grouping, which is what makes them checkable.
  - **A model is one family, not one test per coefficient across the paper.** If you fit six regressions, that is six families, each the size of its own coefficient set — not one family of all the coefficients you happen to report.
  - **Everything else is exploratory.** Say so, report unadjusted //p//-values, and drop the word "significant". Correcting a 400-test screen and a 3-test confirmatory claim in one family is not conservative, it is meaningless: it destroys the confirmatory claim to protect a screen you were going to follow up anyway.
  - **Write the grouping down before the data arrive**, in the paper's methodology and, better, in a registration {[hopewell2025_consort]}. This is the whole point of [[Statistics:Study preregistration]] and it costs one paragraph.

<WRAP important>
**One correction across the whole paper is the wrong answer, and it is the one people reach for.** It is not the conservative choice. It inflates //k// with tests nobody was going to interpret, which is exactly the cost the worked example below measures: the same twenty real effects go from eight reportable at //k// = 200 to one at //k// = 2,000, purely because the family got wider. **Splitting into stated families is not //p//-hacking as long as the split is fixed in advance and reported** — and reporting the split is what distinguishes it.
</WRAP>

===== Which Procedure, and Which Are Current =====

Dated, with what it assumes and whether it is current. "Papers" is out of the 269 in this corpus with a multiple-comparison-correction tuple; the fold behind those counts is described in [[#Use in Publications]].

^ Procedure ^ Dated ^ Controls ^ Assumes about dependence ^ Papers ^ Verdict ^
| **Bonferroni** | 1936 / Dunn 1961 | FWER | nothing — valid under arbitrary dependence | **141 (52.4%)** | **Historical for testing: dominated by Holm since 1979, still modal here.** Keep it for the two jobs Holm does not do — **simultaneous confidence intervals** at level 1 − α/k, and a **threshold you can state in advance** (α/k needs only //k//, where Holm needs every //p//-value in the family) |
| **Holm** (step-down) | 1979 {[holm1979_sequentially]} | FWER | nothing | 66 (24.5%) | **Current. The default for a small confirmatory family.** Strictly dominates Bonferroni |
| **Šidák** | 1967 | FWER | independence | 3 (1.1%) | Marginal. Buys a fraction of a percent over Bonferroni and needs an assumption a crawl cannot support |
| **Tukey HSD / Games–Howell** | 1949 / 1976 | FWER over all pairs | normality; Games–Howell relaxes equal variance | 13 + 2, union 14 (5.2%) | Fine //if// your design really is an ANOVA over group means. Most crawl comparisons are not |
| **Benjamini–Hochberg** | 1995 {[benjamini1995_controlling]} | FDR | independence or positive regression dependence (PRDS) | 43 (16.0%) | **Current. The default for a large exploratory family** — screening trackers, categories, coefficients |
| **Benjamini–Yekutieli** | 2001 {[benjamini2001_control]} | FDR | nothing — arbitrary dependence | **4 (1.5%)** | **Current, and the under-used one.** What you want when the tests share sites, third parties or a model |
| **Storey //q//-value** | 2002 {[storey2002_direct]} | FDR, adaptive | independence-ish; estimates the null proportion | **0** | Legitimate; absent from this literature. Worth knowing so you can read a genomics paper |
| **Resampling / max-//T// / simultaneous bands** | — | FWER, empirically | learns the dependence from the data | 1 explicit | Current and a good fit: it does not need a dependence assumption because it estimates one. Bobek & Pröllochs {[bobek2026_community]} report //"simultaneous confidence bands based on 5,000 bootstrap replications, which account for multiple testing concerns"// |
| **Hierarchical / partial-pooling model** | 2012 {[gelman2012_multiple]} | shrinkage instead of a correction | models the dependence explicitly | 0 framed as such | **The strongest alternative, and structurally absent here.** See [[#When Correcting Is the Wrong Lever]] |

<WRAP important>
**The currency verdict, stated plainly, because the corpus and the statistics disagree.**

**What this field does:** Bonferroni remains the modal procedure in the most recent years available. Of the 45 hypothesis-test papers that corrected in 2025–2026, **19 used Bonferroni, 14 Holm, 9 Benjamini–Hochberg and none Benjamini–Yekutieli**.((Papers, not tuples, and restricted to papers that ran a hypothesis test so that every column shares one denominator. A paper naming two procedures is counted in two columns and a paper using an unlisted procedure in none, so the four figures neither sum to 45 nor are disjoint — here they sum to 42. From ''report_pvalue_corrections.mjs''.)) The FDR share is rising — of the papers whose full text names Bonferroni, Holm or Benjamini–Hochberg, BH's share goes from **15.7% in 2015–2019 (8 of 51) to 28.9% in 2025–2026 (13 of 45)** — but it is still a minority, and 2025–2026 are the provisional years of this corpus.

**What you should do**, in this order:

  - **Decide what the family is, and write it down, before you look at the data.** This matters more than which procedure you pick, and it is the one step no procedure can repair. See [[#Deciding What Counts as One Family]].
  - **Split confirmatory from exploratory**, correct inside the confirmatory family only, and label the rest as exploratory with unadjusted //p//-values and no significance claims.
  - **Then pick a procedure:** **Holm** for a small confirmatory family, **Benjamini–Hochberg** for a large exploratory one, **Benjamini–Yekutieli** instead of BH when you cannot argue the tests are positively dependent — which in a site-level crawl you usually cannot. For a test, prefer Holm over plain Bonferroni: same assumptions, never fewer rejections.
  - **Check the choice against your own numbers.** BY pays a factor of about ln(//k//)+0.58, so it beats Holm only when //k// is large //and// discoveries are plentiful. In the worked example below, at //k// = 200 with sparse signal BY leaves **1** survivor and Holm leaves **2** — the recommendation inverts. Run all four and look before committing.
  - **Report an effect size with an interval for every corrected comparison.** Past about 10,000 units per group the //p//-value stops being the binding constraint, and no correction fixes that. See [[#When Correcting Is the Wrong Lever]].

**Nothing here has been overturned recently.** Multiplicity correction is one of the few methodological areas in web measurement where no 2020s development — LLM classification included — changed the answer. The procedures date from 1979, 1995 and 2001, and they are still the right ones. Where the field has moved is in //which// of them it reaches for, and slowly. Research-frontier work does exist — e-values and e-BH since 2022, selective inference — and **none of it has any presence in this corpus or any adopted standing**, so this page does not recommend it.

What //has// moved is **reporting**, in an adjacent field, and by a smaller step than it first looks. Clinical trials have had multiplicity in their reporting guideline since CONSORT 2010, whose elaboration already discouraged multiple primary outcomes //"because of the problems of interpretation associated with multiplicity of analyses"// and whose limitations item already named //"multiplicity of analyses"//. What the **CONSORT 2025** elaboration {[hopewell2025_consort]} adds is the //explicit negative// obligation: //"Any methods used to mitigate or account for multiplicity should be described. If no methods have been used to account for multiplicity (eg, not applicable, or not considered), then this should also be reported, particularly when a large number of analyses has been carried out."//((The sentence is in the CONSORT 2025 **explanation and elaboration** document (BMJ 389:e081124, ''10.1136/bmj-2024-081124''), under the elaboration for item 21. It is **not** in the statement/checklist paper (''10.1136/bmj-2024-081123''), which contains no occurrence of "multiplicity" in its checklist text. Both fetched 2026-08-13.)) That is the standard the **Declining to correct** section of this page argues for, and **none of the seven venues here asks for anything like it** — nor does any of them ask for //k//, which CONSORT does not ask for either.
</WRAP>

==== FWER or FDR: the question is what a false claim costs you ====

The choice is not statistical, it is editorial, and it is decidable in one sentence: **does one wrong row in your table discredit the paper, or does it cost you a footnote?**

  * **A compliance or enforcement claim** — "this CMP violates Article 7(3)" — is FWER. One false accusation is a problem in itself, and the audience includes people who will read exactly one row. Correct so that the probability of //any// false claim is bounded: Holm.
  * **A screen** — "which of these 400 third parties set an identifier before consent, so we can go look at them" — is FDR. You expect some of the shortlist to be wrong, you will follow up, and being unable to find anything is the real failure. Correct so that the //expected fraction// of wrong rows is bounded: Benjamini–Hochberg, or Benjamini–Yekutieli if you cannot argue positive dependence.
  * **A paper usually has both**, and the two families should be adjusted separately with the split stated. Nenadic et al. {[nenadic2026_swiss]} do this for two contrast definitions of one question: BH //"applied separately to the two families of interaction terms"//. That is the pattern to copy — one correction across the whole paper is not more conservative, it is less meaningful.

SciPy's own documentation states the trade-off in one line: FDR procedures //"tend to offer higher power than familywise error rate control procedures (e.g. Bonferroni correction)"//, and the ''by'' method //"is guaranteed to control the FDR even when the p-values are not from independent tests"//.((''scipy.stats.false_discovery_control'', SciPy 1.18.0 documentation, fetched 2026-08-13.))

===== The Power Cost Is Not Abstract, and Somebody Measured It =====

Sunlight {[lecuyer2015_sunlight]} is a targeting-detection system for ad ecosystems whose pipeline has four stages, and the fourth is //"Multiple Testing Correction"//. It implements both Holm–Bonferroni and Benjamini–Yekutieli and recommends BY as its default. §6.5 reports the same analysis on a small and a large website experiment, and the numbers are the clearest statement of what FWER costs at web-measurement scale that this literature contains:

^ Experiment ^ Hypotheses with //p// < 0.05, uncorrected ^ after Holm ^ after Benjamini–Yekutieli ^
| Website (small) | 88% | 61% | 74% |
| Website-large | 75% | **21%** | 46% |

The paper's own reading:

> We conclude that the Holm correction is very hard on experiments with a lot hypotheses. The larger the proverbial haystack, the harder the needles will be to find.

and, on why it ships BY as the default:

> Although this guarantee on the expected false discovery rate is weaker than what is provided by the Holm-Bonferroni method, it is widely accepted in applied statistics as an appropriate and preferred correction for exploratory studies.

Two things to take from this table rather than from the procedure names. First, **the cost of a FWER correction grows with //k//, so the same method that is reasonable on a 200-site pilot can wipe out a 100,000-site crawl** — and the crawl is not a bigger version of the pilot, it is a different statistical situation. Second, Sunlight also finds an **inversion**: the Stage-1 algorithm that generates the most hypotheses wins before correction and loses after it, because //"the number of hypotheses, and the proportion of high p-value hypotheses play an important role in the correction, both increasing the penalty applied to each p-value"//. Generating more candidate hypotheses makes your pipeline look better and your paper weaker.

You can reproduce the shape of this on your own numbers:

<file python adjust_pvalues.py>
#!/usr/bin/env python3
"""How many of your findings survive, under each correction, and at what k.

Written for a crawl-shaped family of hypotheses: one test per country, per
tracker, per site category, per browser profile. The point is not to pick a
winner but to make the two numbers a reviewer will ask for impossible to omit:
**how large the family is** and **how the answer moves between procedures**.

    python3 adjust_pvalues.py pvalues.txt              # one p-value per line
    python3 adjust_pvalues.py pvalues.txt --alpha 0.01
    python3 adjust_pvalues.py --demo                   # worked example, no input

Requires scipy >= 1.11 for `false_discovery_control` (Benjamini-Hochberg and
Benjamini-Yekutieli). Bonferroni and Holm are four lines each and are written
out here so the arithmetic is visible; statsmodels' `multipletests` does all
four plus Sidak if you would rather have one call.

Crashes on a malformed p-value rather than dropping it: a family whose size is
wrong makes every adjusted p-value wrong.
"""

import argparse
import sys

import numpy as np
from scipy.stats import false_discovery_control


def bonferroni(p):
    """Single-step FWER. Valid under arbitrary dependence. Dominated by Holm."""
    return np.minimum(1.0, p * len(p))


def holm(p):
    """Step-down FWER (Holm 1979). Same assumptions as Bonferroni, never
    rejects fewer hypotheses, so there is no situation in which plain
    Bonferroni is preferable on power grounds."""
    n = len(p)
    order = np.argsort(p)
    adj = np.empty(n)
    running = 0.0
    for rank, i in enumerate(order):
        running = max(running, (n - rank) * p[i])
        adj[i] = min(1.0, running)
    return adj


def read_pvalues(path):
    """One p-value per line, blank lines and #-comments ignored.

    A line that is not a float in [0, 1] raises: silently skipping it would
    shrink the family and inflate every adjusted p-value in it.
    """
    values = []
    with open(path) as fh:
        for lineno, line in enumerate(fh, 1):
            s = line.strip()
            if not s or s.startswith("#"):
                continue
            v = float(s)  # ValueError here is the intended failure
            if not 0.0 <= v <= 1.0:
                raise ValueError(f"{path}:{lineno}: {v} is not a p-value in [0, 1]")
            values.append(v)
    if not values:
        raise ValueError(f"{path}: no p-values found")
    return np.array(values)


def report(p, alpha):
    k = len(p)
    procedures = [
        ("none (nominal alpha)", p, "FWER not controlled"),
        ("Bonferroni", bonferroni(p), "FWER, any dependence"),
        ("Holm", holm(p), "FWER, any dependence"),
        ("Benjamini-Hochberg", false_discovery_control(p, method="bh"), "FDR, independent or PRDS"),
        ("Benjamini-Yekutieli", false_discovery_control(p, method="by"), "FDR, any dependence"),
    ]
    print(f"family size k = {k}    alpha = {alpha}")
    print(f"raw p-values below alpha: {int((p < alpha).sum())}")
    print()
    print(f"{'procedure':<22}{'controls':<28}{'survive':>8}{'share':>8}   smallest adjusted p")
    print("-" * 88)
    for name, adj, controls in procedures:
        s = int((adj < alpha).sum())
        print(f"{name:<22}{controls:<28}{s:>8}{s / k:>8.1%}   {adj.min():.3g}")
    print()
    print("Report k, the procedure, and the adjusted p-values -- not just the count of")
    print("survivors. A reader cannot reconstruct any of the rows above from the count.")


def demo():
    """Three cases, in the order the reader needs them.

    Deterministic seed, so every number printed here is reproducible.
    """
    print("=" * 88)
    print("Case 1: why Holm rather than Bonferroni. Three tests, alpha = 0.05.")
    print("=" * 88)
    p = np.array([0.001, 0.02, 0.03])
    print(f"p                    = {p}")
    print(f"Bonferroni adjusted  = {bonferroni(p)}  -> 1 survives")
    print(f"Holm adjusted        = {holm(p)}  -> 3 survive")
    print()
    print("Same assumptions, same alpha, three findings instead of one. Holm can never")
    print("reject fewer than Bonferroni. In a real family the two usually agree -- the")
    print("reason to use Holm is that it is never worse and costs one function call.")

    print()
    print("=" * 88)
    print("Case 2: a crawl-shaped family. 200 tests, 20 with a real effect.")
    print("=" * 88)
    rng = np.random.default_rng(20260813)
    nulls = rng.uniform(0.0, 1.0, 180)
    signal = rng.beta(1.0, 250.0, 20)
    report(np.concatenate([signal, nulls]), 0.05)

    print()
    print("=" * 88)
    print("Case 3: the same 20 real effects, inside a family of 2,000.")
    print("1,980 nulls this time instead of 180 -- the study got wider, not better.")
    print("=" * 88)
    report(np.concatenate([signal, rng.uniform(0.0, 1.0, 1980)]), 0.05)
    print()
    print("Widening the family is not free: every procedure above loses power, while the")
    print("unadjusted count rises and is now almost all noise. Adding strata you did not")
    print("need costs you the findings you did. Decide the family before you look.")


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("path", nargs="?", help="file of p-values, one per line")
    ap.add_argument("--alpha", type=float, default=0.05)
    ap.add_argument("--demo", action="store_true")
    args = ap.parse_args()
    if args.demo:
        demo()
        return 0
    if not args.path:
        ap.error("give a file of p-values, or --demo")
    report(read_pvalues(args.path), args.alpha)
    return 0


if __name__ == "__main__":
    sys.exit(main())
</file>

Real output, ''numpy'' 2.4.6 / ''scipy'' 1.17.1, run 2026-08-13. The ''holm'' and ''bonferroni'' functions above were checked against ''statsmodels.stats.multitest.multipletests'' (0.14.6) on five random 50-value families: maximum absolute difference 0.0. If you would rather have one call than four functions, ''multipletests'' takes ''method'' in ''bonferroni'', ''sidak'', ''holm-sidak'', ''holm'', ''simes-hochberg'', ''hommel'', ''fdr_bh'', ''fdr_by'', ''fdr_tsbh'' and ''fdr_tsbky'' as of 0.14.6, which covers every procedure in the table above except Tukey and Games–Howell (''statsmodels.stats.multicomp.pairwise_tukeyhsd'' for the former).((Method list read from the live ''statsmodels.stats.multitest.multipletests'' documentation on 2026-08-13; version 0.14.6 confirmed as current from ''pypi.org/pypi/statsmodels/json''.))

<code>
$ python3 adjust_pvalues.py --demo
========================================================================================
Case 1: why Holm rather than Bonferroni. Three tests, alpha = 0.05.
========================================================================================
p                    = [0.001 0.02  0.03 ]
Bonferroni adjusted  = [0.003 0.06  0.09 ]  -> 1 survives
Holm adjusted        = [0.003 0.04  0.04 ]  -> 3 survive

Same assumptions, same alpha, three findings instead of one. Holm can never
reject fewer than Bonferroni. In a real family the two usually agree -- the
reason to use Holm is that it is never worse and costs one function call.

========================================================================================
Case 2: a crawl-shaped family. 200 tests, 20 with a real effect.
========================================================================================
family size k = 200    alpha = 0.05
raw p-values below alpha: 28

procedure             controls                     survive   share   smallest adjusted p
----------------------------------------------------------------------------------------
none (nominal alpha)  FWER not controlled               28   14.0%   2.92e-06
Bonferroni            FWER, any dependence               2    1.0%   0.000584
Holm                  FWER, any dependence               2    1.0%   0.000584
Benjamini-Hochberg    FDR, independent or PRDS           8    4.0%   0.000584
Benjamini-Yekutieli   FDR, any dependence                1    0.5%   0.00343

Report k, the procedure, and the adjusted p-values -- not just the count of
survivors. A reader cannot reconstruct any of the rows above from the count.

========================================================================================
Case 3: the same 20 real effects, inside a family of 2,000.
1,980 nulls this time instead of 180 -- the study got wider, not better.
========================================================================================
family size k = 2000    alpha = 0.05
raw p-values below alpha: 103

procedure             controls                     survive   share   smallest adjusted p
----------------------------------------------------------------------------------------
none (nominal alpha)  FWER not controlled              103    5.1%   2.92e-06
Bonferroni            FWER, any dependence               1    0.1%   0.00584
Holm                  FWER, any dependence               1    0.1%   0.00584
Benjamini-Hochberg    FDR, independent or PRDS           1    0.1%   0.00584
Benjamini-Yekutieli   FDR, any dependence                1    0.1%   0.0478

Report k, the procedure, and the adjusted p-values -- not just the count of
survivors. A reader cannot reconstruct any of the rows above from the count.

Widening the family is not free: every procedure above loses power, while the
unadjusted count rises and is now almost all noise. Adding strata you did not
need costs you the findings you did. Decide the family before you look.
</code>

Case 1 is the whole argument for Holm in three numbers. Cases 2 and 3 are the argument for deciding //k// before you look at the data: the twenty real effects never change, and under Benjamini–Hochberg eight of them are reportable at //k// = 200 and one at //k// = 2,000. Under Holm it is 2 and 1 — the widening costs BH the most because BH had the most to lose.

===== Two Things That Go Wrong Specifically in Web Measurement =====

==== Your tests are not independent, and BH's guarantee needs them to be (roughly) ====

Benjamini–Hochberg controls FDR under independence or positive regression dependence. Site-level web measurement produces tests that are dependent in ways that are neither of those and that you cannot inspect:

  * sites share **third parties** — one Google Tag Manager deployment moves hundreds of sites at once
  * sites share **a CMS or a CMP** — a WordPress-plus-Cookiebot cohort behaves as one unit, and the size of that cohort is not something you chose
  * sites share **hosting, CDN and IP space**, so a network-level failure correlates rows that look independent (see [[Design:IP classification]])
  * repeated crawls of the same site are **the same site**, so a longitudinal panel's tests are correlated by construction (see [[Design:Archives]])

The correlation is positive in some of these and unsigned in others, so the honest position is that you do not know the dependence structure. Two defensible responses, and one indefensible one:

  * **Benjamini–Yekutieli** — valid under arbitrary dependence, at the cost of a factor of about ln(//k//)+0.58. **Four papers in this corpus use it.** Zimmeck et al. {[zimmeck2024_website]} report //"59 were significant (p<=0.05 corrected for multiple tests with the Benjamini-Yekutieli procedure)"// out of 231 comparisons; Sunlight ships it as its default.
  * **Resample the dependence** — permute or bootstrap at the level of the unit that is actually exchangeable (site, or site-cluster, or user), which estimates the joint distribution instead of assuming it. Bobek & Pröllochs {[bobek2026_community]} cluster standard errors at the user level //and// report bootstrap simultaneous confidence bands.
  * **Not defensible:** applying BH and saying nothing about dependence. It may well be fine; the point is that the paper gives a reader no way to tell, and the fix is one sentence.

==== When correcting is the wrong lever ====

<WRAP important>
**At crawl scale, the binding constraint is almost never the //p//-value.** Compare two groups of 100,000 sites each, base rate 10%, and a difference of **1.0 percentage point** gives //p// = 3 × 10⁻¹³ on a two-proportion //z//-test. Bonferroni at //k// = 1,000 leaves it at 3 × 10⁻¹⁰. It survives every procedure on this page. Whether 11.0% against 10.0% //matters// is a question the //p//-value can no longer answer — only the effect size and its interval can, and at the top of a ranking list one percentage point is thousands of sites, which may be the whole story or may be nothing.((Two-proportion //z//-test with a pooled standard error, //n// = 100,000 per group: //z// = 7.29. The same 1.0-point difference at //n// = 10,000 per group gives //p// = 0.021 and does not survive //k// = 1,000. Computed with ''scipy.stats.norm'' 1.17.1 on 2026-08-13; the two lines are in ''out/pvalue-external-facts.txt''.)) **45.0% of the papers in this corpus that ran both a crawl and a hypothesis test state a population of 100,000 or more.**

Correcting such a //p//-value is arithmetic performed on a quantity that stopped carrying information. What a reader needs is the **effect size and its interval**: the difference in percentage points, the risk ratio, the //r// or Cramér's //V//, with a confidence interval. **Only 159 of the 1,025 hypothesis-test papers (15.5%) report an effect size, and 178 report a correction without one.** That ratio is the wrong way round: a correction with no effect size protects a reader from a false positive and tells them nothing about whether the true positive matters.
</WRAP>

The structural alternative is to stop treating the strata as separate hypotheses. Gelman, Hill & Yajima {[gelman2012_multiple]} argue that a hierarchical model over the strata — countries, categories, trackers as levels of a random effect — shrinks the extreme estimates toward the mean automatically, which is what a correction is trying to approximate, and gives you an estimate per stratum instead of a survive/die verdict. This is a very good fit for web measurement: the strata are exchangeable-ish, there are many of them, and most have small //n//. **65 papers in this corpus fit a mixed-effects or multilevel model, and not one frames it as an answer to multiplicity.**((''report_pvalue_corrections.mjs'' probes the 5,869 full texts for ''multilevel model'', ''hierarchical model'' and ''mixed-effects model'' and finds 65. Whether any of them frames the model as a multiplicity strategy was **not** checked paper by paper — the claim here is only that none surfaced while reading the correction passages, and it is listed as an open question below.))

===== Correcting What You Report Does Not Fix What You Ran =====

A correction is computed over the tests in the paper. The multiplicity that actually threatens a crawl-based finding happened earlier, in the pipeline, and is invisible to any procedure on this page:

  * which seed list, and which rank cut (see [[Design:Website selection]], [[Design:Sampling|Representative sampling methods]])
  * which exclusion rule for sites that failed to load, and how many each candidate rule removed
  * which denominator each headline percentage sits over
  * which classifier, filter list, or confidence threshold, and which version of it (see [[Design:Website classification]], [[Privacy:Requests]])
  * which crawl run counts, after the first one broke

Each of those was a choice made with the data in front of you, and each is a test you effectively ran and did not report. Gelman & Loken {[gelman2014_statistical]} call this the garden of forking paths, and their point is precisely that it does not require any dishonesty — a single analysis path, chosen after seeing the data, has an uncontrolled error rate even though no multiple comparison was ever computed.

**No paper in this corpus engages with the idea.** The literal string //forking paths// matches two of the 5,869 full texts, and reading both shows they are packet-forwarding paths and symbolic-execution paths — see [[#What is missing entirely]]. The ASA statement on //p//-values {[wasserstein2016_asa]} matches **zero**. The standard mechanism that addresses it is committing to the analysis before you look — which is [[Statistics:Study preregistration]], and which **14 of these 1,025 hypothesis-test papers (1.4%) did**.((14, not the 6 in the ''statistics.kind'' table below. The schema field catches fewer than half the preregistrations in this corpus; the sibling page hand-classified all 62 full-text matches for ''pre-regist*'' and found 15 real study preregistrations, of which 14 also ran a hypothesis test. Use the hand count, not the field.)) It is not the only one: the multiverse analysis in [[#Open Questions]] addresses the same problem after the fact, and a held-out split of the crawl addresses part of it.

<WRAP todo>
The cheap version, if you will not preregister: **name your confirmatory tests in the paper**, correct within that family only, and label everything else exploratory with uncorrected //p//-values and no significance claims. Nenadic et al. {[nenadic2026_swiss]} split their families; nobody in this corpus splits confirmatory from exploratory in a //crawl//.
</WRAP>

==== Declining to correct is a legitimate choice, if you say so ====

Three papers in this corpus have a correction tuple that records the //absence// of one, and all three give the same reason — the analysis is exploratory — and all three say it in one sentence. That is a defensible position, and it is much better than silence:

  * Pu & Grossklags {[pu2016_model]}: //"we did not adjust p-values for the multiple testing problem since we consider our preliminary tests of the impact of collection context on privacy valuation as exploratory analysis, where multiplicity adjustments are neither mandatory, nor important"//
  * Goetzen et al. {[goetzen2022_ctrl]}: //"because our data are organized from three similar, non-identical populations and the comparisons we make are purely complementary, we do not make any multiple comparison corrections in our regressions"//
  * Naji et al. {[naji2025_responsibility]}: //"We did not apply multiple testing corrections, as our analysis is exploratory in nature"//

If you declare exploratory, the obligation that comes with it is to stop using the word "significant" and to stop reporting stars. The declaration and the significance claims cannot both stand.

===== Use in Publications =====

Every figure below comes from the extraction over 5,859 papers from CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, plus a regex sweep over the 5,869 ''paper.cols.txt'' full texts. The script is ''report_pvalue_corrections.mjs'', the free-text fold is ''mcc_fold.mjs'', and both live with the full query log, the residue and the sources rejected on [[provenance:statistics:pvalue_corrections]].

==== How often, and in which kind of study ====

Each row is its own population. **"MCC tuple" counts papers with a ''statistics[].kind == "multiple-comparison-correction"'' tuple; "corrected" removes the ones whose tuple records the //absence// of a correction.**

^ Population ^ //N// ^ MCC tuple ^ Share ^ Corrected ^ Share ^
| ''hypothesisTest'' — ran a hypothesis test | 1025 | 251 | 24.5% | **247** | **24.1%** |
| ''inferential'' — any non-descriptive statistic | 1762 | 269 | 15.3% | 265 | 15.0% |
| ''regression'' — fitted a regression | 391 | 94 | 24.0% | 92 | 23.5% |
| ''correlation'' — reported a correlation | 458 | 58 | 12.7% | 56 | 12.2% |
| ''humanSubjects'' — recruited participants | 1357 | 197 | 14.5% | 194 | 14.3% |
| ''crawled'' — ran an automated web crawl | 1120 | 37 | 3.3% | 36 | 3.2% |
| all extraction records | 5859 | 269 | 4.6% | 265 | 4.5% |

''hypothesisTest'' is the denominator that matters: a paper that reports only descriptive statistics has no //p//-values to correct. The ''crawled'' row understates the gap because most crawl papers never run an inferential test at all — cross the two:

^ Study shape ^ Ran a hypothesis test ^ Corrected ^ Share ^
| recruited participants, no crawl | 433 | 167 | **38.6%** |
| ran a crawl, no participants | 140 | 18 | **12.9%** |
| both | 49 | 17 | 34.7% |
| neither | 403 | 45 | 11.2% |

**A crawl paper that runs a hypothesis test is a third as likely to correct as a user study that does** — in the setting where //k// is set by the data rather than by the protocol. The venue breakdown says the same thing from another angle:

^ Venue ^ Papers ^ Ran a hypothesis test ^ Corrected ^ Share ^
| PoPETs | 510 | 149 | 68 | **45.6%** |
| USENIX Security | 1410 | 242 | 69 | 28.5% |
| IEEE S&P | 767 | 124 | 33 | 26.6% |
| CCS | 990 | 110 | 23 | 20.9% |
| TheWebConf | 843 | 236 | 34 | 14.4% |
| IMC | 638 | 71 | 10 | **14.1%** |
| NDSS | 701 | 93 | 10 | **10.8%** |

IMC and NDSS are the two lowest. IMC is the most empirical venue in the corpus (98.0% of its papers measure something) and it has the second-lowest correction rate — the same pattern [[Statistics:Study preregistration]] finds for preregistration, where IMC has none at all.

==== Which procedure, folded ====

''statistics[].method'' is free text and agrees run-to-run on about 20% of exact strings, so it must be folded before it is counted. **90 distinct strings describe eleven procedures**; the fold is ''mcc_fold.mjs'' and its **unmapped residue is 0 strings and 0 papers**. Denominator: the 269 papers with an MCC tuple. Papers are counted once, and a paper naming two procedures appears in two rows, so the shares exceed 100%.

^ Family ^ Controls ^ Papers ^ Share of 269 ^
| Bonferroni (single-step) | FWER | 141 | 52.4% |
| Holm (step-down) | FWER | 66 | 24.5% |
| Benjamini–Hochberg | FDR | 43 | 16.0% |
| Tukey HSD | FWER | 13 | 4.8% |
| Unnamed / ad-hoc adjustment | unstated | 8 | 3.0% |
| Benjamini–Yekutieli | FDR | 4 | 1.5% |
| Šidák | FWER | 3 | 1.1% |
| Games–Howell | FWER | 2 | 0.7% |
| Simultaneous confidence bands | FWER | 1 | 0.4% |
| //(not a correction)// **NEGATIVE — the paper says it did not correct** | — | 3 | 1.1% |
| //(not a correction)// **Greenhouse–Geisser — a sphericity correction** | — | 2 | 0.7% |

The last two rows are the reason the fold exists rather than a ''method'' histogram. Böttger et al. {[bottger2025_regional]} write //"due to a violation of sphericity, we applied the Greenhouse-Geisser correction"// — a correction, but of the //F//-test's degrees of freedom, not of multiplicity. Any query that counts ''kind == "multiple-comparison-correction"'' and stops there reports both of these as corrections.

==== It rose through the 2010s, then fell back — and Bonferroni never lost ====

^ Period ^ Ran a hypothesis test ^ Corrected ^ Share ^ Bonferroni ^ Holm ^ B–H ^ B–Y ^
| 2010–2014 | 97 | 9 | 9.3% | 4 | 3 | 1 | 0 |
| 2015–2019 | 227 | 53 | 23.3% | 32 | 17 | 5 | 1 |
| 2020–2024 | 487 | 140 | 28.7% | 75 | 31 | 25 | 2 |
| 2025–2026* | 214 | 45 | 21.0% | 19 | 14 | 9 | 0 |

<WRAP important>
**The rise is solid; the last bucket's fall is real too, and provisionality does not explain it away.** 9/97 in 2010–2014 against 140/487 in 2020–2024 is Fisher's exact //p// = 2.1 × 10⁻⁵ — the practice genuinely tripled. Then 140/487 against 45/214 in 2025–2026 is //p// = 0.0328 in the other direction, putting the current rate back at its 2015–2019 level (53/227 against 45/214 gives //p// = 0.5689, i.e. indistinguishable).

The obvious excuse does not work. 2025–2026 are the provisional years — CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are absent from OpenAlex — so one might expect the venue mix to explain the dip. It does not: applying each venue's own correction rate to the 2025–2026 venue mix predicts **25.6%**, and to the 2020–2024 mix **25.4%**. The composition barely moved.((Expected rate = Σ(venue rate × venue's hypothesis-test papers in the period) / period total. NDSS's hypothesis-test count rose from 26 to 40 between the two buckets, which offsets the extra PoPETs weight. Computed by ''report_pvalue_corrections.mjs''.)) So either the dip is sampling noise on 214 papers, or the practice has stopped spreading. **Treat "about a quarter, and no longer climbing" as the claim**, and do not quote the per-year series: it moves between 0.0% (2010, 2011) and 39.1% (2021) on denominators of 8 to 128.

What is stable across the buckets is that **Bonferroni has never stopped being modal.** Every column above is restricted to hypothesis-test papers, so the four procedure counts and the "corrected" count share one denominator; a paper naming two procedures is counted in two columns, which is why the procedure columns can sum above the corrected count.

The FDR share does rise. Of the papers whose full text names Bonferroni, Holm or Benjamini–Hochberg, BH's share of that set is 22.2% in 2010–2014 (2 of 9 — too small to read), **15.7% in 2015–2019 (8 of 51), 21.3% in 2020–2024 (30 of 141) and 28.9% in 2025–2026 (13 of 45)**. Read from 2015 onward that is close to a doubling over a decade, and it is still a minority position.
</WRAP>

==== Almost nobody states the size of the family ====

An adjusted //p// cannot be checked, reproduced or compared without //k//. Two independent passes, agreeing to about a point.

**Pass 1, over the 287 tuples that record a real correction** — what landed in ''statistics[].method'', ''.detail'' or the evidence sentence:

^ What the tuple states ^ Tuples ^ Share of 287 ^
| a number of comparisons, tests or hypotheses in the family | **14** | **4.9%** |
| an adjusted α or significance threshold | 57 | 19.9% |
| any reported detail at all (''statistics[].detail'' non-null) | 161 | 56.1% |

**Pass 2, over the whole text of the 265 papers that corrected.** A tuple only carries one sentence, so this pass searches the full paper for any count of comparisons, tests or hypotheses within ±1,500 characters of a procedure mention. Doing that naively is useless in this literature: the unguarded regex returned 41 hits, of which the majority were **χ² notation** — the digit in "χ² tests" sits immediately before the word — plus exponents and USENIX page furniture. With three guards the count is 20, and **reading all 20 leaves 15 genuine: 5.7% of the 265**. The five rejected, and every accepted match with its surrounding sentence, are printed in the report output on [[provenance:statistics:pvalue_corrections]].

**So: 4.9% by tuple, 5.7% by paper. Roughly one corrected paper in twenty states the size of its family.**

The fourteen that do state it are the ones a reader can check, and they read like this:

  * Kablo & Cabarcos {[kablo2023_privacy]}: //"a total of 106 tests were conducted"//, threshold adjusted to 0.00047
  * Maass et al. {[maass2021_effective]}: //"a single Holm-Bonferroni correction for all 45 significance tests"//
  * Weinberg et al. {[weinberg2017_topics]}: //"a total of 2,904 such combinations"//
  * Liu et al. {[liu2024_opted]}: //"compare bid values for all 16 personas"//, //p// //"multiplied by 16"//
  * Despres et al. {[despres2024_best]}: //"a conservative α of p = 0.002 as the criterion for significance (based on a maximum of 25 hypotheses tested per outcome)"//
  * Zimmeck et al. {[zimmeck2024_website]}: 59 of 231 comparisons significant after Benjamini–Yekutieli

Note the last two: **the family is "per outcome" or "per comparison set", not "per paper".** That is the right granularity and it is also why //k// has to be stated — a reader cannot infer which grouping you used.

==== The correction belongs in the power analysis, not only in the results ====

**38 of the 1,025 hypothesis-test papers report both a power analysis and a correction**, and the one whose wording makes the sequencing unambiguous is Ho et al. {[ho2025_efficacy]}, who fold the correction into the sample-size calculation: //"After adjusting for multiple comparisons by taking α* = α/3 to compare three different types of training (interactive, static, and control), we would need N = 2,693 per group, which our study greatly exceeds."// The correction determines how large the study has to be, and computing it afterwards means discovering that the study was too small.((Whether the other 37 also ran the correction //before// sizing the study cannot be read off the extraction — the two tuples carry no ordering — and the 37 were not read one by one. The claim here is about the one quotation, not about a count. All 38 slugs are listed on [[provenance:statistics:pvalue_corrections]].))

Doing it at all is uncommon, because power analysis is uncommon: **72 of the 1,025 hypothesis-test papers (7.0%) report one.** The rest of the inference-hygiene stack, over the same denominator:

^ ''statistics.kind'' ^ Papers ^ Share of 1025 ^
| hypothesis-test | 1025 | 100.0% |
| multiple-comparison-correction | 251 | 24.5% |
| regression | 213 | 20.8% |
| correlation | 207 | 20.2% |
| effect-size | 159 | 15.5% |
| resampling | 93 | 9.1% |
| power-analysis | 72 | 7.0% |
| bayesian | 12 | 1.2% |
| preregistration //(schema field; the hand count is 14)// | 6 | 0.6% |

**69 papers report both an effect size and a correction. 178 report a correction and no effect size.**

==== What is missing entirely ====

Probes over all 5,869 full texts. Each row is a concept that sits next to a correction in any statistics course, and each is absent or a homograph — worth knowing before you grep this corpus yourself.

^ Concept ^ Papers matching the probe ^ What the hits actually are ^
| **Storey //q//-value** | 11 | all 11 read: every one is a **Q-learning Q-value**. Zero uses. The fold has no //q//-value string either |
| **statistical knockoffs** | 17 | all 17 read: **Knockoff Nets** model stealing, or counterfeit goods |
| **e-values / e-BH** | 1 | an ''E-VALUE'' rule name in an operational semantics |
| **the ASA statement on //p//-values** | 0 | absent |
| **Rothman/Perneger "no adjustments are needed"** | 1 | an unrelated Perneger paper on questionnaire pre-test sample size |
| **garden of forking paths** | 2 | packet-forwarding paths, and symbolic-execution paths |
| permutation test | 26 | real, 8 read: each uses it as the test itself, never as a max-//T// multiplicity adjustment |
| mixed-effects / multilevel model | 65 | real; whether any frames it as a multiplicity strategy was not checked paper by paper |

The pattern is that this literature **applies corrections without ever engaging with the argument about them.** The case against routine correction (Rothman {[rothman1990_adjustments]}, Perneger {[perneger1998_bonferroni]}), the case for modelling instead (Gelman et al. {[gelman2012_multiple]}), and the case that the whole framing is wrong (Gelman & Loken {[gelman2014_statistical]}, the ASA statement {[wasserstein2016_asa]}) are all missing. A paper that cites any of them is doing something no paper in these seven venues has done.

==== Methodology and limitations of these figures ====

  * **Two independent sources, kept apart.** The schema field and a regex sweep over full text disagree in both directions and the report prints the disagreement rather than a union. Of the 279 papers whose full text names a procedure, **246 also carry a correction tuple (88.2% schema recall)**; 33 are named in text and missed by the schema, and 19 carry a tuple whose procedure is not in the named list.
  * **The free-text fold is a rule list with a printed residue of 0.** ''mcc_fold.mjs'' has a self-test asserting the rule precedence that matters (''Holm-Bonferroni'' folds to Holm, ''Dunn-Bonferroni'' to Bonferroni), and a string matching no rule is printed rather than silently bucketed.
  * **Two buckets in the fold are not corrections** and are excluded from every "corrected" count: the three papers that declare they did not correct, and the two Greenhouse–Geisser sphericity corrections.
  * **Sentinels are never answers.** ''statistics'' has no ''not-stated'' value — a paper that corrected and did not say so is simply absent, which is why every correction figure here is a **lower bound on the practice and an upper bound on the reporting**.
  * **The family-size figure is measured twice and hand-checked.** The tuple pass sees one sentence per correction; the full-text pass sees the whole paper but needs guards against χ² notation, exponents and page furniture, and then a reading of all 20 surviving hits. 4.9% and 5.7% are the two answers, five rejected hits are named with reasons, and all 20 are printed with their sentences on the provenance page. **Neither pass is a bound**: a count in a distant table caption is missed by both.
  * **A paper counts once**, never once per mention. 292 tuples across 269 papers.
  * **''statistics.kind'' is a mid-band field**: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. That caveat applies to every share on this page and does not apply to the folded procedure ranking, which is a ranking.
  * **Quotes were checked.** All 292 correction quotes were run against ''paper.cols.txt'': 167 exact, 63 partial (≥60% of five-word windows), 62 below that threshold — mostly extractor ellipses and dropped citation markers. Every quote used on this page was then located by hand in the source; **all 18 checked out**, one of them (Pu & Grossklags) only after the extractor's ''...'' was expanded. See [[provenance:statistics:pvalue_corrections]].
  * **Venue coverage.** Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and **SOUPS** are absent, and SOUPS matters most: it is where the usable-privacy work that corrects most reliably would appear, so the correction rate here is if anything flattered by its absence.
  * **2025 and 2026 are provisional.** See the box above.
  * **Every external claim was fetched on 2026-08-13**, not recalled: nine DOIs through ''doi.org'' content negotiation, and the SciPy and statsmodels documentation from their own sites. The full log is on [[provenance:statistics:pvalue_corrections]]; corpus-level caveats are on [[Literature:Corpus]].

===== What to Report =====

  - **//k//, the size of the family, and what the family //is//.** "Per outcome variable", "per pair of countries", "across the 16 personas". This is the single most-omitted item — 4.9% of correction tuples state it.
  - **The procedure by name, with a citation.** "Corrected for multiple comparisons" is not a method.
  - **Which error rate you chose to control, and why.** One sentence: is a single false row fatal (FWER) or costly (FDR)?
  - **Adjusted //p//-values, not just a count of survivors.** A reader cannot re-derive the adjustment from "12 comparisons were significant".
  - **Which comparisons are confirmatory and which are exploratory**, with the correction applied inside the confirmatory family only, and no significance language on the exploratory part.
  - **Your dependence assumption**, if you used Benjamini–Hochberg. If you cannot argue positive dependence between site-level tests, say so and use Benjamini–Yekutieli, or resample.
  - **An effect size with an interval, for every corrected comparison.** At crawl scale this matters more than the //p//-value, and 178 papers here report a correction without one.
  - **The correction inside your power analysis**, if you are sizing a study rather than analysing one {[ho2025_efficacy]}.
  - **If you decline to correct, say so and say why** — and then drop the word "significant" {[pu2016_model]} {[goetzen2022_ctrl]} {[naji2025_responsibility]}. CONSORT 2025 {[hopewell2025_consort]} makes this an explicit requirement in clinical trials: reporting that //no// method was used is itself a reporting obligation.
  - **For a test, write "Holm", not "Bonferroni".** Same assumption, same code path, never fewer findings. Reach for Bonferroni deliberately, not by default: for **simultaneous confidence intervals**, or for a **threshold pre-declared** in a protocol or registration where you do not yet have the //p//-values Holm needs.

===== Open Questions =====

<WRAP todo>
  * **No measurement paper in this corpus corrects a per-site family.** The design that most obviously creates thousands of hypotheses — one test per site — is never followed by a correction, so it is unknown whether the field considers that setting out of scope for testing, or simply does not test it. Both readings have consequences for how a reviewer should treat a per-site claim.
  * **65 papers fit a multilevel model and none frames it as a multiplicity strategy** — at least, none surfaced while reading correction passages, and no one has checked all 65. Whether partial pooling is already doing the work of a correction in this literature without anyone saying so is answerable and unanswered.
  * **Benjamini–Yekutieli is used by four papers**, and it is the procedure whose assumptions actually match site-level crawl data. Whether the BH results in this literature would survive BY is checkable on any paper that released its //p//-values — see [[:Artifacts]] — and nobody has done it.
  * **The debate is absent.** Zero papers cite the ASA statement, Rothman, Perneger, or the garden of forking paths. A short SoK on inference practice in web measurement would be citing an empty shelf, which is unusual and useful.
  * **Nobody has asked these venues to require //k//.** Reporting the family size is a one-line checklist item that would make every correction in the literature checkable, and **94.3% of corrected papers do not do it**. The nearest thing in any field is CONSORT's requirement that trials //describe the method// used — which is a weaker ask than stating //k//, and nobody has proposed even that much to a security or measurement venue. Their review forms are not public, so the current state can only be read off the papers.
  * **The pipeline-level multiplicity is unmeasured.** Nobody has taken a published crawl, enumerated the defensible alternatives at each pipeline decision (seed list, rank cut, exclusion rule, classifier threshold), re-run all of them, and reported the spread of the headline figure. That is a multiverse analysis, it is straightforwardly fundable, and it would say more about this literature's error rate than any correction.
</WRAP>

===== Related Pages =====

  * [[Statistics:Hypothesis testing]] — choosing the test whose //p//-values you are about to correct.
  * [[Statistics:Study preregistration]] — the only mechanism that addresses the multiplicity in the pipeline rather than in the results table.
  * [[Statistics:Regression]] — where the family is the set of coefficients, and the correction is applied per model rather than per paper.
  * [[Statistics:Biases]] — what a fixed population and a fixed exclusion rule protect you from before any //p//-value exists.
  * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]] — the strata that become the family.
  * [[Design:Crawling location]] — per-vantage comparisons, and how many pairs they generate.
  * [[Design:User studies]] — where **two-thirds of the papers that correct live** (167 of 247), and where the rate, 38.6%, is three times the crawl rate. Still a minority practice even there.
  * [[:Artifacts]] — releasing the //p//-values is what makes a correction re-checkable.
  * [[provenance:statistics:pvalue_corrections|Provenance of this page's figures]].

====== References ======

/*
To insert citations, you have to follow 3 steps:

  - Check whether the BibTex entry is in https://measuretheweb.org/literature/bibliography, if not, add it to the end. For example, ''LePochat2019_tranco'' is already present.
  - In the text, where you want to use the citation, use {[LePochat2019_tranco]} - it will be rendered as [1] or number given by order in the page.
  - Keep this section as it is to print the bibliography.

If any of these steps fails, you will see a purple warning in the appropriate page.
*/

<bibtex bibliography></bibtex>


~~DISCUSSION~~