Choosing a statistical test is taught as a decision about your data — is it normal, is it ordinal, how many groups. In a web measurement the binding constraints are somewhere else entirely: your sample is a ranking list, your unit of analysis is a site that shares a tag manager with two hundred other sites, your n is 100,000 so everything is significant (which is somebody else's page, and the answer there is effect sizes), and your “before and after” is the same site crawled twice. None of that is in a statistics textbook, and all of it changes the answer.

This page assumes you know what a p-value, a null hypothesis and a type-I error are. It is about four things a textbook will not tell you: which tests this field actually uses and which of them are current, why the default reflex (“data is skewed, so Mann-Whitney”) answers the wrong question, why the unit of analysis is the error that actually invalidates results here, and how to analyse a before/after crawl.

• Tang, Bauer & Christin [1Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)] (SOUPS 2025) — read this one first. A systematic review of 121 SOUPS papers over 20 years, coding 479 NHST-based assertions. It is the only work found that audits statistical practice in a directly adjacent community, and its two headline findings are the two this page's corpus figures corroborate: “tests in 23% of papers inadequately account for non-independence between samples, leading to potentially invalid claims” and “58% of papers lack information to verify whether an assertion is supported, such as imprecisely specifying the statistical test conducted.” Its worked example of imprecision is the same one measured above: “a 'Wilcoxon test' can indicate either the Mann-Whitney-Wilcoxon test for independent samples or the Wilcoxon Signed-Rank Test for paired data.” SOUPS is not one of this page's seven venues, which is why it is the external anchor rather than a corpus figure.
• Divine et al. 2018 [2Divine, George W.; Norton, H. James; Barón, Anna E.; Juarez-Colunga, Elizabeth (2018): "The Wilcoxon-Mann-Whitney Procedure Fails as a Test of Medians", The American Statistician 72(3):278-286. (DOI)] — “The Wilcoxon–Mann–Whitney Procedure Fails as a Test of Medians”. Four pages of consequence: if you have ever written “we used Mann-Whitney to compare medians”, this says why that sentence is wrong. See What Mann-Whitney U Actually Tests.
• Fagerland & Sandvik 2009 [3Fagerland, Morten W.; Sandvik, Leiv (2009): "The Wilcoxon-Mann-Whitney test under scrutiny", Statistics in Medicine 28(10):1487-1497. (DOI)] and Fagerland 2012 [4Fagerland, Morten W (2012): "t-tests, non-parametric tests, and large studies—a paradox of statistical practice?", BMC Medical Research Methodology 12(1):78. (DOI)] — the Wilcoxon–Mann–Whitney test under unequal variances or skewness, and the paradox that non-parametric tests get less defensible as n grows, not more. The second is the direct rebuttal of the reflex this field runs on.
• Lumley et al. 2002 [5Lumley, Thomas; Diehr, Paula; Emerson, Scott; Chen, Lu (2002): "The Importance of the Normality Assumption in Large Public Health Data Sets", Annual Review of Public Health 23(1):151-169. (DOI)] — why the normality assumption stops mattering in large data sets. The single most useful citation for a reviewer who asks you to justify a t-test on 50,000 sites.
• Cameron & Miller 2015 [6Colin Cameron, A.; Miller, Douglas L. (2015): "A Practitioner’s Guide to Cluster-Robust Inference", Journal of Human Resources 50(2):317-372. (DOI)] — cluster-robust inference, written for practitioners. The fix for the problem in The Unit of Analysis Is the Error That Actually Invalidates Results.
• Lydersen, Fagerland & Laake 2009 [7Lydersen, Stian; Fagerland, Morten W.; Laake, Petter (2009): "Recommended tests for association in 2×2 tables", Statistics in Medicine 28(7):1159-1175. (DOI)] and Campbell 2007 [8Campbell, Ian (2007): "Chi‐squared and Fisher-Irwin tests of two‐by‐two tables with small sample recommendations", Statistics in Medicine 26(19):3661-3675. (DOI)] — recommended tests for 2×2 tables. Between them they retire both Fisher's exact as a default and Yates' continuity correction.
• AdFisher [9Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)] (PoPETs 2015) — the model of a permutation test built into an automated web experiment rather than bolted onto its results. Read it for the apparatus, not the ad-targeting result.

Almost every non-parametric choice in this literature is justified the same way: the data is skewed, therefore a rank test. OmniCrawl [10Cassel, Darion; Lin, Su-Chin; Buraggina, Alessio; Wang, William; Zhang, Andrew; Bauer, Lujo; Hsiao, Hsu-Chun; Jia, Limin; Libert, Timothy (2022): "OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers", in: Proceedings on Privacy Enhancing Technologies. (DOI)] states it plainly — “The distributions we examined were sufficiently skewed such that non-parametric tests were applicable” — and that sentence is representative of the field.

The reflex reaches a defensible test for an indefensible reason, and the reason matters because it stops working exactly when your crawl gets big.

It assumes the sampling distribution of the mean is approximately normal. By the central limit theorem that holds for skewed data at moderate n and holds very comfortably at web-measurement n. Lumley et al. [5Lumley, Thomas; Diehr, Paula; Emerson, Scott; Chen, Lu (2002): "The Importance of the Normality Assumption in Large Public Health Data Sets", Annual Review of Public Health 23(1):151-169. (DOI)] make the point for public-health data sets and the arithmetic is the same here: past a few hundred observations per group, non-normality of the raw values is not what threatens a t-test.

What does threaten it, and what a rank test does not fix:

• Heavy tails make the mean a bad summary, not an invalid one. If third-party counts per site are power-law-ish, the mean is real but is driven by the tail. That is an argument about which estimand you want — mean, median, or a quantile — and it should be settled before any test is chosen. It is not an argument about normality.
• Unequal variances between groups. This is a real problem and the fix is Welch's t-test, used by 28 papers here, not a rank test.
• Dependence between observations. No rank test fixes this either. See below.

38 of the 1,025 papers (3.7%) report a normality check — Shapiro–Wilk in most of them. Using one to choose between a t-test and a rank test is a documented mistake: the two-stage procedure distorts the type-I error rate of whatever runs second, and the pre-test's own power depends on n in the wrong direction, so it waves through non-normality in small samples and rejects trivial non-normality in large ones. Rochon, Gondan & Kieser [11Rochon, Justine; Gondan, Matthias; Kieser, Meinhard (2012): "To test or not to test: Preliminary assessment of normality when comparing two independent samples", BMC Medical Research Methodology 12(1). (DOI)] and Rasch, Kubinger & Moder [12Rasch, Dieter; Kubinger, Klaus D.; Moder, Karl (2011): "The two-sample t test: pre-testing its assumptions does not pay off", Statistical Papers 52(1):219-231. (DOI)] both conclude that pre-testing does not pay off.

Decide from the design and the estimand, before the data arrives, and say so. A defensible sentence looks like: “Because our outcome is a count with a long right tail and we care about the typical site rather than the mean, we pre-specified rank-based tests.” An indefensible one is “Shapiro–Wilk was significant, so we used Mann-Whitney.” The best example of doing this well in the corpus splits the choice per outcome variable and states the reason for each. Mai et al. [13Mai, Cat; Coelho, Bruno; Kieserman, Julia; Matsumoto, Lexie; Spinelli, Kyle; Yang, Eric; Andreou, Athanasios; Greenstadt, Rachel; Lauinger, Tobias; McCoy, Damon (2025): "More and Scammier Ads: The Perils of YouTube's Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)] use “one-way repeated measures ANOVA as the omnibus test since the ad load data follow a normal distribution” for one outcome and, “for the hypothesis on predatory ad rates, since the rates do not approximately follow a normal distribution, we use Friedman test as the omnibus test” for another, in the same paper.¹⁾

Not medians. Mann-Whitney U [14Mann, H. B.; Whitney, D. R. (1947): "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other", The Annals of Mathematical Statistics 18(1):50-60. (DOI)] — identical to Wilcoxon's rank-sum test [15Wilcoxon, Frank (1945): "Individual Comparisons by Ranking Methods", Biometrics Bulletin 1(6):80. (DOI)] — tests stochastic superiority: the probability that a randomly drawn value from one group exceeds one from the other. It equals a test of medians only under an assumption almost never stated and rarely true of web data: that the two distributions have the same shape and differ only by a shift. Divine et al. [2Divine, George W.; Norton, H. James; Barón, Anna E.; Juarez-Colunga, Elizabeth (2018): "The Wilcoxon-Mann-Whitney Procedure Fails as a Test of Medians", The American Statistician 72(3):278-286. (DOI)] give examples where the medians are identical and Mann-Whitney rejects, and where the medians differ and it does not.

Two consequences for a crawl:

• A significant Mann-Whitney result on differently-shaped distributions may be about shape, not location. Comparing third-party counts on news sites against shopping sites, the distributions differ in spread and skew as well as centre. “Trackers are significantly higher on news sites” is then an over-claim about what was tested. Fagerland & Sandvik [3Fagerland, Morten W.; Sandvik, Leiv (2009): "The Wilcoxon-Mann-Whitney test under scrutiny", Statistics in Medicine 28(10):1487-1497. (DOI)] show the type-I error rate departs badly from nominal when variances or skewness differ.
• If you cannot assume equal shapes, the current test is Brunner–Munzel [16Brunner, Edgar; Munzel, Ullrich (2000): "The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation", Biometrical Journal 42(1):17-25. (DOI)], which tests the same stochastic-superiority quantity without needing equal variances. One paper in 5,859 uses it — Schiele & Gadyatskaya [17Schiele, Nathan Daniel; Gadyatskaya, Olga (2025): "A limited technical background is sufficient for attack-defense tree acceptability", in: Proceedings of the USENIX Security Symposium. (Link)], who “opt for the non-parametric Brunner-Munzel (BM) test that is robust in the unequal variance case”. It is in SciPy as scipy.stats.brunnermunzel and has been since SciPy 1.2.0 (2018).²⁾

Report the effect size, and for a rank test that means a rank-based one: the common-language effect size, Vargha–Delaney's Â12, or Cliff's δ — all of which are the stochastic-superiority quantity, expressed as a number a reader can interpret. Only 15 of the 213 Mann-Whitney papers here (7.0%) report one; 198 (93.0%) do not. Where the practice does exist it is largely imported from the fuzzing literature, where reporting Â12 beside Mann-Whitney U is close to standard [18Klees, George; Ruef, Andrew; Cooper, Benji; Wei, Shiyi; Hicks, Michael (2018): "Evaluating Fuzz Testing", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] [19Schloegel, Moritz; Bars, Nils; Schiller, Nico; Bernhard, Lukas; Scharnowski, Tobias; Crump, Addison; Ebrahim, Arash Ale; Bissantz, Nicolai; Muench, Marius; Holz, Thorsten (2024): "SoK: Prudent Evaluation Practices for Fuzzing", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)]. Two 2026 papers do it in a measurement context: Monzani et al. [20Monzani, Andrea; Parata, Antonio; Oliveri, Andrea; Aonzo, Simone; Balzarotti, Davide; Lanzi, Andrea (2026): "Unveiling BYOVD Threats: Malware’s Use and Abuse of Kernel Drivers", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] report “Cliff's delta δ = −0.375” with its direction, and Razuvayevskaya & Bontcheva [21Razuvayevskaya, Olesya; Bontcheva, Kalina (2026): "Truth with a Twist: The Rhetoric of Persuasion in Professional vs. Community-Authored Fact-Checks", in: Proceedings of the ACM Web Conference. (DOI)] pair Mann-Whitney with Cliff's δ as “a non-parametric effect size metric”.

Everything above changes a p-value by a factor of two. This changes whether the test means anything.

A crawl of 100,000 sites is not 100,000 independent observations. Sites share:

• third-party infrastructure — one Google Tag Manager container, one consent platform, one ad exchange moves hundreds or thousands of sites together
• a CMS and its default templates — a WordPress-plus-Cookiebot cohort behaves as one unit, and you did not choose how big that cohort is
• hosting, CDN and IP space — a single provider's configuration correlates rows that look unrelated (see IP classification)
• the same site, twice — every longitudinal or before/after panel is correlated by construction (see Archives)
• an operator — one company's fifty domains are one decision

The test does not know any of this. It takes your n at face value, and its false-positive rate rises with the strength of the clustering. This is not a multiplicity problem and no correction on P-value corrections repairs it: correcting p-values that are individually computed from the wrong n does not make them right.

The script below measures it under a true null — two groups drawn from the identical distribution, so every rejection is a false positive — for a range of within-cluster similarities. ICC is the intra-class correlation, the share of outcome variance explained by which cluster a site is in.

paired_and_clustered.py

#!/usr/bin/env python3
"""Two things that change the answer in a web-measurement hypothesis test.
 
Both are demonstrated on SYNTHETIC data with a fixed seed, because the point is
a property of the test rather than a fact about the web. Nothing here is a
measurement claim; the numbers are reproducible by running the file.
 
  A. A before/after crawl of the SAME sites is paired. Analysing it with a
     chi-square test on the two periods' marginal counts throws the pairing
     away, and because most sites do not change, pooling them dilutes the
     signal you are looking for. McNemar's test looks only at the sites that
     changed.
 
  B. Sites are not independent. They share tag managers, CMS templates, CMPs
     and hosting, so a "sample of n sites" is really a sample of far fewer
     independent units. A test that assumes n independent sites has a
     false-positive rate far above its nominal alpha. This part measures that
     rate under a true null, and shows that testing CLUSTER MEANS fixes it.
     (Aggregating to cluster means is the crudest of the remedies -- it is what
     fits in twenty lines. Permuting whole clusters, cluster-robust standard
     errors and a mixed-effects model are all better, and none is demonstrated
     here.)
 
Run:  python3 paired_and_clustered.py
"""
 
import numpy as np
from scipy.stats import chi2_contingency, mannwhitneyu, binomtest
from statsmodels.stats.contingency_tables import mcnemar
 
SEED = 20260813
 
 
# ---------------------------------------------------------------------------
# A. Paired before/after: McNemar vs chi-square
# ---------------------------------------------------------------------------
def paired_before_after():
    print("=" * 72)
    print("A. A BEFORE/AFTER CRAWL IS PAIRED")
    print("=" * 72)
 
    # 2,000 sites, crawled before and after some intervention. Most sites do
    # not change their behaviour at all; the effect lives entirely in the few
    # that do. These four counts fully determine both tests.
    both_yes = 1300   # set a tracking cookie in both crawls
    both_no = 600     # set one in neither crawl
    yes_to_no = 80    # stopped after the intervention
    no_to_yes = 20    # started after the intervention
 
    n = both_yes + both_no + yes_to_no + no_to_yes
    before = both_yes + yes_to_no
    after = both_yes + no_to_yes
    print(f"{n} sites crawled twice.")
    print(f"  before : {before}/{n} = {100 * before / n:.1f}% set a tracking cookie")
    print(f"  after  : {after}/{n} = {100 * after / n:.1f}%")
    print(f"  unchanged: {both_yes + both_no} sites "
          f"({100 * (both_yes + both_no) / n:.1f}%)")
    print(f"  discordant: {yes_to_no} stopped, {no_to_yes} started")
 
    # WRONG: treat the two crawls as two independent samples of sites.
    # correction=False is passed EXPLICITLY. SciPy's default for a 2x2 is
    # correction=True, which applies Yates' continuity correction -- and Yates
    # is not recommended (Campbell 2007). Leaving the default in would mean the
    # code published alongside "skip Yates' correction" quietly applied it. The
    # difference here is small (p 0.0428 vs 0.0464), which is roughly all Yates
    # ever does.
    unpaired_table = [[before, n - before], [after, n - after]]
    chi2, p_chi2, _, _ = chi2_contingency(unpaired_table, correction=False)
    print(f"\n  chi-square on the marginal 2x2 (IGNORES the pairing; Yates off)")
    print(f"    table = {unpaired_table}")
    print(f"    chi2 = {chi2:.3f}   p = {p_chi2:.4g}")
 
    # RIGHT: McNemar, on the discordant pairs only.
    paired_table = [[both_yes, yes_to_no], [no_to_yes, both_no]]
    mcn = mcnemar(paired_table, exact=True)
    print(f"\n  McNemar exact on the paired 2x2 (USES the pairing)")
    print(f"    table = {paired_table}")
    print(f"    statistic = {mcn.statistic:.0f}   p = {mcn.pvalue:.4g}")
    # McNemar exact is a binomial test on the discordant pairs; show the identity
    b = binomtest(no_to_yes, yes_to_no + no_to_yes, 0.5)
    print(f"    identical to binomtest({no_to_yes}, "
          f"{yes_to_no + no_to_yes}, 0.5) = {b.pvalue:.4g}")
 
    ratio = p_chi2 / mcn.pvalue
    print(f"\n  => the paired test's p-value is {ratio:.3g}x smaller.")
    print("     Same data. The unpaired test spends its power on the 1,900")
    print("     sites that never changed; only 100 sites carry information")
    print("     about change, and McNemar is the test that knows that.")
    print("     Note the direction: here BOTH are significant, but the")
    print("     chi-square is borderline where the truth is overwhelming.")
    return p_chi2, mcn.pvalue
 
 
# ---------------------------------------------------------------------------
# B. Clustered sites: the false-positive rate of assuming independence
# ---------------------------------------------------------------------------
def clustered_null(n_sim=2000, n_clusters=60, per_cluster=10,
                   cluster_sd=1.0, alpha=0.05):
    """Measure the true type-I error rate of a test that assumes n independent
    sites, when the sites actually come in correlated clusters.
 
    The null is TRUE by construction: both groups are drawn from exactly the
    same distribution. Every rejection is a false positive.
    """
    print()
    print("=" * 72)
    print("B. SITES ARE NOT INDEPENDENT, AND THE TEST DOES NOT KNOW")
    print("=" * 72)
    rng = np.random.default_rng(SEED)
    n = n_clusters * per_cluster
    print(f"{n} sites in {n_clusters} clusters of {per_cluster}. A cluster is a")
    print("shared tag manager, CMS template or CMP: sites in one behave alike.")
    print(f"Two groups of {n_clusters // 2} clusters each. THE NULL IS TRUE --")
    print("both groups are drawn from the same distribution, so every")
    print(f"rejection at alpha = {alpha} below is a false positive.")
    print(f"Simulations: {n_sim}. Cluster effect SD = {cluster_sd}, "
          f"site noise SD = 1.0.")
 
    half = n_clusters // 2
    naive_rej = 0
    cluster_rej = 0
    for _ in range(n_sim):
        # cluster-level effect, then site-level noise within cluster
        cl = rng.normal(0.0, cluster_sd, n_clusters)
        y = np.repeat(cl, per_cluster) + rng.normal(0.0, 1.0, n)
        gid = np.repeat(np.arange(n_clusters) < half, per_cluster)
 
        # NAIVE: Mann-Whitney U over sites, pretending n independent units.
        if mannwhitneyu(y[gid], y[~gid], alternative="two-sided").pvalue < alpha:
            naive_rej += 1
 
        # CLUSTER-AWARE: aggregate to one value per cluster, then test the
        # clusters -- the units that actually are exchangeable.
        cm = y.reshape(n_clusters, per_cluster).mean(axis=1)
        if mannwhitneyu(cm[:half], cm[half:], alternative="two-sided").pvalue < alpha:
            cluster_rej += 1
 
    print(f"\n  Mann-Whitney over {n} SITES  (assumes independence)")
    print(f"    false positives: {naive_rej}/{n_sim} = "
          f"{100 * naive_rej / n_sim:.1f}%   (nominal {100 * alpha:.0f}%)")
    print(f"  Mann-Whitney over {n_clusters} CLUSTER means (respects clustering)")
    print(f"    false positives: {cluster_rej}/{n_sim} = "
          f"{100 * cluster_rej / n_sim:.1f}%   (nominal {100 * alpha:.0f}%)")
    print(f"\n  => assuming independence inflates the false-positive rate from")
    print(f"     about {100 * alpha:.0f}% to {100 * naive_rej / n_sim:.1f}%. No "
          "multiplicity correction repairs")
    print("     this: the error is in the unit of analysis, not the number of")
    print("     tests. Correcting p-values that are individually wrong does")
    print("     not make them right.")
    return naive_rej / n_sim, cluster_rej / n_sim
 
 
def clustered_sweep(n_sim=1000, n_clusters=60, per_cluster=10, alpha=0.05):
    """How the inflation grows with the strength of the clustering."""
    print()
    print("-" * 72)
    print("How bad it gets, as a function of how alike sites in a cluster are")
    print("-" * 72)
    print(f"{'cluster SD':>11}  {'ICC':>6}  {'naive FP rate':>14}  "
          f"{'cluster FP rate':>16}")
    half = n_clusters // 2
    n = n_clusters * per_cluster
    for cluster_sd in (0.0, 0.25, 0.5, 1.0, 2.0):
        rng = np.random.default_rng(SEED)
        icc = cluster_sd ** 2 / (cluster_sd ** 2 + 1.0)
        nrej = crej = 0
        for _ in range(n_sim):
            cl = rng.normal(0.0, cluster_sd, n_clusters)
            y = np.repeat(cl, per_cluster) + rng.normal(0.0, 1.0, n)
            gid = np.repeat(np.arange(n_clusters) < half, per_cluster)
            if mannwhitneyu(y[gid], y[~gid], alternative="two-sided").pvalue < alpha:
                nrej += 1
            cm = y.reshape(n_clusters, per_cluster).mean(axis=1)
            if mannwhitneyu(cm[:half], cm[half:], alternative="two-sided").pvalue < alpha:
                crej += 1
        print(f"{cluster_sd:>11.2f}  {icc:>6.2f}  {100 * nrej / n_sim:>13.1f}%  "
              f"{100 * crej / n_sim:>15.1f}%")
    print("\nICC 0 is genuine independence, where the naive test is correct.")
    print("An ICC of 0.20 means a fifth of the variance in the outcome is")
    print("explained by which cluster a site is in -- an entirely ordinary")
    print("figure when a cluster is 'runs Google Tag Manager'.")
 
 
if __name__ == "__main__":
    paired_before_after()
    clustered_null()
    clustered_sweep()

Its real output, part B (SciPy 1.17.1, NumPy 2.4.6, statsmodels 0.14.6, run 2026-08-13):

========================================================================
B. SITES ARE NOT INDEPENDENT, AND THE TEST DOES NOT KNOW
========================================================================
600 sites in 60 clusters of 10. A cluster is a
shared tag manager, CMS template or CMP: sites in one behave alike.
Two groups of 30 clusters each. THE NULL IS TRUE --
both groups are drawn from the same distribution, so every
rejection at alpha = 0.05 below is a false positive.
Simulations: 2000. Cluster effect SD = 1.0, site noise SD = 1.0.

  Mann-Whitney over 600 SITES  (assumes independence)
    false positives: 839/2000 = 42.0%   (nominal 5%)
  Mann-Whitney over 60 CLUSTER means (respects clustering)
    false positives: 128/2000 = 6.4%   (nominal 5%)

  => assuming independence inflates the false-positive rate from
     about 5% to 42.0%. No multiplicity correction repairs
     this: the error is in the unit of analysis, not the number of
     tests. Correcting p-values that are individually wrong does
     not make them right.

------------------------------------------------------------------------
How bad it gets, as a function of how alike sites in a cluster are
------------------------------------------------------------------------
 cluster SD     ICC   naive FP rate   cluster FP rate
       0.00    0.00            5.1%              6.4%
       0.25    0.06           11.9%              5.8%
       0.50    0.20           24.5%              6.1%
       1.00    0.50           41.0%              6.0%
       2.00    0.80           48.1%              5.5%

ICC 0 is genuine independence, where the naive test is correct.
An ICC of 0.20 means a fifth of the variance in the outcome is
explained by which cluster a site is in -- an entirely ordinary
figure when a cluster is 'runs Google Tag Manager'.

Read the ICC = 0.00 row first: with genuine independence the naive test is correct (5.1% against a nominal 5%), which is the check that the simulation is not rigged. At ICC 0.20 — a fifth of the variance explained by which cluster a site is in — the naive false-positive rate is 24.5%. Aggregating to clusters costs a little accuracy when there is no clustering to correct for (6.4% instead of 5.1%, because 30-versus-30 is a small and discrete test) and saves you from a five-fold error rate when there is.

Ordered by how much they cost you:

1. Say what your independent unit is. One sentence. It is the cheapest item on this page and the phrase “unit of analysis” appears in at most 7 of the 1,025 papers.
2. Aggregate to the cluster and test the clusters, as the simulation does. Crude, always valid, and it costs power. Good when the cluster count is comfortable.
3. Cluster-robust standard errors, if you are fitting a model anyway [6Colin Cameron, A.; Miller, Douglas L. (2015): "A Practitioner’s Guide to Cluster-Robust Inference", Journal of Human Resources 50(2):317-372. (DOI)]. This is what the three corpus papers that address it do, and the one whose clustering variable matches a web measurement exactly is Nenadic et al. [22Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)], who “cluster standard errors at the website level” for repeated observations of the same policy. Bobek & Pröllochs [23Bobek, Michelle; Pröllochs, Nicolas (2026): "Community Fact-Checks Do Not Break Follower Loyalty", in: Proceedings of the ACM Web Conference. (DOI)] cluster “at the user level” and add “simultaneous confidence bands based on 5,000 bootstrap replications”; Becerril-Arreola [24Becerril-Arreola, Rafael (2023): "A Method to Assess and Explain Disparate Impact in Online Retailing", in: Proceedings of the ACM Web Conference. (DOI)] uses “errors clustered at the product level”. See Regression.
4. Permute or bootstrap whole clusters, which estimates the dependence instead of assuming it and lets you test any statistic you like. 24 papers here run a permutation test and 3 a bootstrap test. AdFisher [9Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)] uses a blocked permutation test — blocking is the same idea. Fouad et al. [25Fouad, Imane; Santos, Cristiana; Legout, Arnaud; Bielova, Nataliia (2022): "My Cookie is a phoenix: detection, measurement, and lawfulness of cookie respawning with browser fingerprinting", in: Proceedings on Privacy Enhancing Technologies. (DOI)] “perform a permutation test with the 11 observations from the Control crawls using 10,000 permutations”; Zimmeck et al. [26Zimmeck, Sebastian; Li, Jie S.; Kim, Hyungtae; Bellovin, Steven M.; Jebara, Tony (2017): "A Privacy Analysis of Cross-device Tracking", in: Proceedings of the USENIX Security Symposium. (Link)] use an “exact one-tailed permutation test” on ad distributions; Li & Paxson [27Li, Frank; Paxson, Vern (2017): "A Large-Scale Empirical Study of Security Patches", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] use “permutation tests with 1,000 rounds”.
5. Model the hierarchy — a mixed-effects model with a random effect per cluster. See Regression.

This is a design web measurement runs constantly and reports poorly. You crawl the same sites twice — before and after a consent banner, before and after a legal deadline, with and without an extension, desktop and mobile, one vantage point and another. The two measurements are paired: they come from the same site.

Pairing is an asset. It removes every stable difference between sites — popularity, sector, CMS, size — from the comparison, because each site is compared with itself. Throwing it away is a common and entirely avoidable way to lose power.

Corpus usage, out of 1,025: Wilcoxon signed-rank 85, paired t-test 58, Friedman 24, McNemar 19, sign test 6, Cochran's Q 4 — a union of 165 papers (16.1%), against 592 (57.8%) naming at least one unpaired test.

A crawl paper that runs a hypothesis test is a third as likely to use a paired test as a user study that does — in the setting where the same unit is measured twice as a matter of course. The same asymmetry appears for multiplicity correction on Pvalue corrections, and the likeliest explanation is the same one that page gives: these methodological habits arrived with human-subjects methodology and have not crossed over.³⁾

Part A of the script above is this case. 2,000 sites crawled twice; 1,900 do not change; 80 stop setting a tracking cookie and 20 start. Real output:

========================================================================
A. A BEFORE/AFTER CRAWL IS PAIRED
========================================================================
2000 sites crawled twice.
  before : 1380/2000 = 69.0% set a tracking cookie
  after  : 1320/2000 = 66.0%
  unchanged: 1900 sites (95.0%)
  discordant: 80 stopped, 20 started

  chi-square on the marginal 2x2 (IGNORES the pairing; Yates off)
    table = [[1380, 620], [1320, 680]]
    chi2 = 4.103   p = 0.04282

  McNemar exact on the paired 2x2 (USES the pairing)
    table = [[1300, 80], [20, 600]]
    statistic = 20   p = 1.116e-09
    identical to binomtest(20, 100, 0.5) = 1.116e-09

  => the paired test's p-value is 3.84e+07x smaller.
     Same data. The unpaired test spends its power on the 1,900
     sites that never changed; only 100 sites carry information
     about change, and McNemar is the test that knows that.
     Note the direction: here BOTH are significant, but the
     chi-square is borderline where the truth is overwhelming.

Same 2,000 sites, same 4,000 observations. The marginal chi-square scrapes past 0.05; McNemar returns 10⁻⁹. The 1,900 sites that never changed contain no information about change, and pooling them into a 2×2 of marginal counts dilutes the 100 that do. McNemar's exact test is a binomial test on the discordant pairs and nothing else, which is why it is statsmodels.stats.contingency_tables.mcnemar — it is not in SciPy.⁵⁾

Note the direction carefully: here the wrong test is too conservative. It will also be anti-conservative in other configurations. The point is not “chi-square is conservative”, it is that a test which ignores the design answers a different question.

The first thing that actually happens in a before/after crawl is that some sites are not there the second time. They 404, they time out, they start blocking your crawler, they move behind Cloudflare, the domain lapses. A paired test has no cell for a missing second observation — McNemar's 2×2 is built from four counts of pairs, and a site seen once is not a pair.

That leaves three decisions, none of which a test makes for you:

1. Pair only sites observed in both crawls, and say how many that left. This is the defensible default and it is a one-line report: “of the 10,000 sites crawled in March, 9,214 were reachable again in September; the paired tests are over those 9,214.” Without that line a reader cannot tell your denominator from your seed list.
2. Check the attrition is not differential. If sites that dropped out are systematically the ones that had been setting the most cookies, the survivors are a biased panel and the paired comparison is measuring survivorship as much as change. The cheap version is to compare the crawl-1 outcome between the dropped and retained sites — which is itself an unpaired test on the very variable you are studying, so run it and report it.
3. Do not silently switch to an unpaired test to keep the dropouts. Comparing all 10,000 March sites against the 9,214 September ones is a different comparison with a different question, and it reintroduces every between-site difference that pairing removed.

None of this is exotic — it is the same complete-case-analysis problem clinical trials have, and the same one Biases is about — but the crawl version has a twist worth stating: your attrition is not random with respect to your outcome, because the things that make a site unreachable and the things that make it track are both correlated with how the site is run. The demo below crawls 2,000 sites and gets 2,000 pairs, which is exactly the part of a real before/after crawl that synthetic data cannot show you. See Archives for longitudinal panels, where the problem compounds every wave.

Two well-regarded web-measurement papers state the signed-rank-versus-Mann-Whitney choice in terms of something other than pairing:

• OmniCrawl [10Cassel, Darion; Lin, Su-Chin; Buraggina, Alessio; Wang, William; Zhang, Andrew; Bauer, Lujo; Hsiao, Hsu-Chun; Jia, Limin; Libert, Timothy (2022): "OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: “We used the Wilcoxon signed-rank and Mann-Whitney U tests for comparisons of two groups of equal and unequal size, respectively.”
• Demir et al. [30Demir, Nurullah; Hörnemann, Jan; Große-Kampmann, Matteo; Urban, Tobias; Pohlmann, Norbert; Holz, Thorsten; Wressnegger, Christian (2023): "On the Similarity of Web Measurements Under Different Experimental Setups", in: Proceedings of the ACM Internet Measurement Conference, pp. 356-369. ACM DOI 10.1145/3618257.3624795 is listed by DBLP but was not registered with the DOI resolver as of 2026-08-12; the DOI above resolves to the authors' institutional record of the same paper (DOI)]: “(1) the Wilcoxon signed-rank test to assess differences between two continuous variables, (2) the Mann-Whitney U test to determine differences between two independent variables.”

The criterion is neither sample size nor variable type: it is whether the observations are paired. Two independent groups that happen to be the same size still need Mann-Whitney; two paired measurements of different things are still paired. Both designs are ones where the stated rule probably reaches the right test anyway — in OmniCrawl equal-size groups are generally the same sites under two browser configurations, so “equal size” is standing in for “paired” — and neither paper's results are in question here. The stated rule is what a reader copies, and this is exactly the imprecision Tang et al. [1Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)] code as insufficient reporting: their recommended minimum is a “Precise Test Name” giving “sufficient information to distinguish between non-paired and paired versions of a test, whether the test accounts for repeated measures, and what type of data the test is structured to evaluate.”

Papers that state it unambiguously read like Sarabi et al. [31Sarabi, Armin; Huang, Ziyuan; Wang, Chenlan; Karir, Tai; Liu, Mingyan (2025): "The Ransomware Decade: The Creation of a Fine-Grained Dataset and a Longitudinal Study", in: Proceedings of the USENIX Security Symposium. (Link)], whose null hypothesis names the design: “we conduct a Wilcoxon signed-rank test under the null hypothesis that pre- and post-event medians are equal”. Umayya et al. [32Umayya, Zeya; Malik, Dhruv; Gosain, Devashish; Sharma, Piyush Kumar (2023): "PTPerf: On the Performance Evaluation of Tor Pluggable Transports", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] pair the test with everything a reader needs — “we performed paired t-tests [43] for each pair of PTs”, reporting “the corresponding P-value, t-value, 95% confidence interval (CI), and the mean difference”.

Dated, with what it assumes and what it is for. “Papers” is out of the 1,025 in this corpus that ran a hypothesis test, folded by test_fold.mjs.

Every figure below comes from the extraction over 5,859 papers from CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, plus regex probes over 5,855 paper.cols.txt full texts. The script is report_hypothesis_testing.mjs, the free-text fold is test_fold.mjs, and both live with the full query log, the fold residue, the probes that were rejected and the sources rejected on hypothesis_testing.

1,806 hypothesis-test tuples across those 1,025 papers, carrying 798 distinct free-text method strings. By venue, the share of each venue's papers that run a test: PoPETs 29.2%, TheWebConf 28.0%, USENIX Security 17.2%, IEEE S&P 16.2%, NDSS 13.3%, CCS 11.1% and IMC 11.1%. CCS and IMC are the two lowest and are indistinguishable from each other (110/990 = 11.111% against 71/638 = 11.129%), so no ranking between them should be read. The substantive point is that IMC — the most empirical venue in the corpus, 98.0% of its papers measure something — is among the least inferential. It reports descriptive measurements.

798 strings describe about forty procedures. A paper naming three tests appears in three rows, so the shares exceed 100%; the union of papers naming at least one substantive test is 916 of 1,025 (89.4%) where the family counts sum to 1,444. The fold's unmapped residue is 0 strings and 0 papers.

Three groups of tuples are excluded from every figure above, because they are not tests of the paper's own hypothesis. They are reported because their size is itself a finding: assumption checks — normality 38 papers, variance/sphericity 23, model diagnostics 25, outliers 7 — things that are not tests — confidence intervals only 16, estimators and inequalities 25, factor analysis 4 — and 73 papers whose method string names no test. A query that counts kind == “hypothesis-test” and stops there reports Shapiro–Wilk and Clopper–Pearson as the paper's statistical test.

statistics[].method agrees run-to-run on about 20% of exact strings, so it must never be aggregated by exact string. “Undercount” is what ranking raw strings gives you against the folded paper count.

The fold's hardest case is Wilcoxon, and it is not a spelling problem. “Wilcoxon rank-sum” is Mann-Whitney U and belongs in the unpaired family; “Wilcoxon signed-rank” is the paired test. A fold keyed on /wilcoxon/ merges an unpaired test with a paired one, which is the exact error the page is about. A “paired Wilcoxon test” or a “one-sample Wilcoxon test” resolves to signed-rank because that is what those words mean. What is left — a bare “Wilcoxon test” — goes in its own bucket rather than being guessed at.

Each period's counts are of papers that ran a hypothesis test in that period. Both ends of every bucket are bound; 2025–2026 is provisional.

Rank-based use rises monotonically across all four buckets and Mann-Whitney triples its share. Two of those movements survive testing and the rest do not. Fisher's exact, two-sided, on the paper counts — and Holm-corrected across all 15 comparisons the report runs, because a page that tells you to correct cannot report fifteen raw p-values:

What that licenses, and what it does not:

• The move to rank-based testing is established, and the Mann-Whitney part of it comfortably so. This is the page's central currency claim and it survives correction.
• “Papers name their test more often now” does not survive. The share naming no test fell from 12.4% to 5.1%, which is a real change in the descriptive numbers, but at Holm p = 0.372 it is not an established trend and this page does not claim one. Read it as encouraging and unproven.
• Permutation testing being flat is not something this table shows. p = 1.00 on 2/97 against 6/214 is absence of evidence, not evidence of absence — those counts have almost no power, and a large p is not a measure of stability. What can be said is descriptive and is enough: the share sat between 1.8% and 3.1% for sixteen years and never rose, so there is no sign of adoption of the method best suited to dependent, heavy-tailed, arbitrary-statistic web data.
• The most recent step is not significant on its own (2020–2024 against 2025–2026, raw p = 0.121). Read the 69.6% as consistent with the established trend, not as a fresh rise — and remember 2025–2026 is the provisional slice.

The unnamed ones read like “statistical significance test (unspecified)”, “significance tests”, “hypothesis testing”, “p-value threshold”. Because the schema has no not-stated sentinel for statistics, every figure here is a lower bound on the practice and an upper bound on the reporting: a paper that ran a test and never mentioned it is simply absent.

The rest of the inference-hygiene stack, over the same 1,025:

Two tests are reached for more often by crawl papers than by user studies: Kolmogorov–Smirnov (10.7% against 4.6%) and the proportion / z-test (7.1% against 5.5%). Both are what you use when you have two big empirical distributions or two big proportions and no experimental design. Everything else in the table, and every paired test, goes the other way.

Probes over 5,855 full texts. A probe count is not a finding; the third column is what survived reading.

• The fold is a rule list with a printed residue of 0, and a 146-case self-test asserting the precedence that matters: signed-rank before rank-sum, McNemar before chi-square, resampling before the statistic it resamples, “unpaired” before “paired”. Two ordering bugs and two regex bugs were caught by that self-test rather than by inspection; they are listed on the provenance page.
• Non-substantive tuples are excluded, not silently dropped. Assumption checks, confidence intervals and unnamed methods are counted and reported separately.
• A paper counts once, never once per mention. 1,806 tuples across 1,025 papers.
• statistics.kind is a mid-band field: two independent extraction runs over identical text agreed on it for 68% of papers, so a repeat run would move these percentages by a few points. The caveat applies to every share here and not to the folded rankings, which are rankings. That 68% figure was measured on the previous corpus run and has not been re-measured.
• Quotes were checked in bulk and by hand. All 1,806 hypothesis-test quotes against the text the extractor read: 931 exact after whitespace normalisation (51.6%), 540 partial at ≥60% of five-word windows (29.9%), 335 below that (18.5%). Below-threshold is not “unsupported” — five of the 335 were chased by hand and all five were present, damaged by a dropped citation marker, a reflowed table caption or a two-column splice. The other 330 were not individually checked. Every quote published on this page was located by hand; one (Mai et al.) is in the paper's PDF but in none of the corpus's stored text renderings of it. Details on hypothesis_testing.
• Sentinels are never answers, and statistics has none, which is why the reporting-gap figures are bounds rather than rates.
• 138 of the 5,859 records are posters and 251 are ≤4 pages. A poster has its methodology compressed out, so it is systematically likelier to be scored “no test named” — the 5.7% is inflated by an unmeasured amount.
• Venue coverage. Seven venues. EuroS&P, ACSAC, RAID, AsiaCCS, WPES, CHI and SOUPS are absent, and SOUPS matters most here: it is where the most careful statistical practice in this community appears, and it is precisely the venue Tang et al. [1Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)] audit. Every claim on this page is a claim about those seven venues.
• 2025 and 2026 are provisional. See the currency box.
• Every external claim was fetched on 2026-08-13, not recalled: 15 DOIs through the Crossref API, the SciPy and statsmodels documentation and release notes, and the IMC and PoPETs 2026 calls for papers. The full log, including what could not be fetched, is on hypothesis_testing; corpus-level caveats are on Corpus.

1. The precise test name, distinguishing paired from unpaired [1Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)]. “A Wilcoxon test” is not a test name. “t-test” is not either — 43.2% of t-test papers here stop there.
2. What your independent unit is, and what you did about sites that are not independent. One sentence, and it is the highest-value sentence on this list.
3. Why this test, decided from the design and the estimand — not from a normality pre-test.
4. An effect size with an interval, appropriate to the test. For a rank test that means Â12, Cliff's δ or the common-language effect size, not Cohen's d. Only 159 of the 1,025 papers here (15.5%) report an effect size at all.
5. n per group, and the number of discordant pairs if the test is McNemar or a sign test — that is the test's real sample size.
6. The test statistic, its degrees of freedom and an exact p, not a bare “p < 0.05”. Umayya et al. [32Umayya, Zeya; Malik, Dhruv; Gosain, Devashish; Sharma, Piyush Kumar (2023): "PTPerf: On the Performance Evaluation of Tor Pluggable Transports", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] are the model: P-value, t-value, 95% CI and the mean difference.
7. Whether the comparison is confirmatory or exploratory, and the multiplicity treatment if confirmatory — see Pvalue corrections.
8. For a permutation or bootstrap test, the number of resamples and what was permuted. Permuting sites and permuting clusters are different tests; only the second is valid under clustering.
9. For Mann-Whitney, whether you are claiming a location shift, and if so on what grounds you assume equal shapes [2Divine, George W.; Norton, H. James; Barón, Anna E.; Juarez-Colunga, Elizabeth (2018): "The Wilcoxon-Mann-Whitney Procedure Fails as a Test of Medians", The American Statistician 72(3):278-286. (DOI)]. If you cannot, use Brunner–Munzel [16Brunner, Edgar; Munzel, Ullrich (2000): "The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation", Biometrical Journal 42(1):17-25. (DOI)].

* Nobody has measured the real intra-class correlation of web-measurement outcomes. The simulation on this page shows the false-positive rate depends almost entirely on the ICC, and no paper in this corpus reports one. Estimating the ICC of “sets a tracking cookie before consent” by tag manager, by CMS and by hosting provider is a small, self-contained, immediately useful study, and it would tell every crawl paper how badly its p-values are wrong. * How many published crawl findings survive a cluster-aware re-analysis? Answerable on any paper that released per-site data (see Artifacts), and nobody has done it. The three corpus papers that cluster are all 2023+, so essentially the whole literature is un-re-analysed. * Permutation testing has not grown in sixteen years — 2.1% to 2.8% — despite being the best fit for dependent, heavy-tailed, arbitrary-statistic web data, and despite scipy.stats.permutation_test existing since SciPy 1.8.0 (2022). Whether that is unfamiliarity, compute cost, or reviewers not asking is unknown. * Brunner–Munzel is used by one paper in 5,859. Whether the Mann-Whitney results in this literature would survive dropping the equal-shape assumption is checkable on released data and unchecked. * No paper states the Mann-Whitney estimand. 213 papers use the test; the probe for “stochastic superiority” or “stochastic dominance” in that sense returns zero. Whether authors know and do not write it, or write “median” because they believe it, is not answerable from text. * None of the seven venues asks for a precise test name. Tang et al. [1Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)] propose a minimum-reporting list for SOUPS; nobody has proposed it to IMC, PoPETs or a security venue, and their reviewer forms are not public, so the current state can only be read off the papers.

• P-value corrections — what to do once you have many of the p-values this page produces. The two problems are independent: a correction does not fix a wrong unit of analysis.
• Regression — where cluster-robust standard errors and mixed-effects models live, and the right home for a per-site analysis with covariates.
• Study preregistration — the mechanism that makes “we chose the test from the design, in advance” checkable rather than assertable.
• Biases — what a fixed population and exclusion rule protect you from before any test is run.
• Representative sampling methods and Website selection — where your n, and the dependence inside it, come from.
• Stateful stateless and Automated measurements — a paired before/after crawl is a crawl-configuration decision before it is a statistical one.
• Archives — longitudinal panels, where every observation is paired with itself.
• User studies — where most of the paired testing in this corpus actually happens (23.8% against the crawl rate of 8.6%).
• Artifacts — releasing per-site outcomes is what makes a cluster-aware re-analysis possible.
• Provenance of this page's figures.

[1]

Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)

[2]

Divine, George W.; Norton, H. James; Barón, Anna E.; Juarez-Colunga, Elizabeth (2018): "The Wilcoxon-Mann-Whitney Procedure Fails as a Test of Medians", The American Statistician 72(3):278-286. (DOI)

[3]

Fagerland, Morten W.; Sandvik, Leiv (2009): "The Wilcoxon-Mann-Whitney test under scrutiny", Statistics in Medicine 28(10):1487-1497. (DOI)

[4]

Fagerland, Morten W (2012): "t-tests, non-parametric tests, and large studies—a paradox of statistical practice?", BMC Medical Research Methodology 12(1):78. (DOI)

[5]

Lumley, Thomas; Diehr, Paula; Emerson, Scott; Chen, Lu (2002): "The Importance of the Normality Assumption in Large Public Health Data Sets", Annual Review of Public Health 23(1):151-169. (DOI)

[6]

Colin Cameron, A.; Miller, Douglas L. (2015): "A Practitioner’s Guide to Cluster-Robust Inference", Journal of Human Resources 50(2):317-372. (DOI)

[7]

Lydersen, Stian; Fagerland, Morten W.; Laake, Petter (2009): "Recommended tests for association in 2×2 tables", Statistics in Medicine 28(7):1159-1175. (DOI)

[8]

Campbell, Ian (2007): "Chi‐squared and Fisher-Irwin tests of two‐by‐two tables with small sample recommendations", Statistics in Medicine 26(19):3661-3675. (DOI)

[9]

Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[10]

Cassel, Darion; Lin, Su-Chin; Buraggina, Alessio; Wang, William; Zhang, Andrew; Bauer, Lujo; Hsiao, Hsu-Chun; Jia, Limin; Libert, Timothy (2022): "OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[11]

Rochon, Justine; Gondan, Matthias; Kieser, Meinhard (2012): "To test or not to test: Preliminary assessment of normality when comparing two independent samples", BMC Medical Research Methodology 12(1). (DOI)

[12]

Rasch, Dieter; Kubinger, Klaus D.; Moder, Karl (2011): "The two-sample t test: pre-testing its assumptions does not pay off", Statistical Papers 52(1):219-231. (DOI)

[13]

Mai, Cat; Coelho, Bruno; Kieserman, Julia; Matsumoto, Lexie; Spinelli, Kyle; Yang, Eric; Andreou, Athanasios; Greenstadt, Rachel; Lauinger, Tobias; McCoy, Damon (2025): "More and Scammier Ads: The Perils of YouTube's Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[14]

Mann, H. B.; Whitney, D. R. (1947): "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other", The Annals of Mathematical Statistics 18(1):50-60. (DOI)

[15]

Wilcoxon, Frank (1945): "Individual Comparisons by Ranking Methods", Biometrics Bulletin 1(6):80. (DOI)

[16]

Brunner, Edgar; Munzel, Ullrich (2000): "The Nonparametric Behrens-Fisher Problem: Asymptotic Theory and a Small-Sample Approximation", Biometrical Journal 42(1):17-25. (DOI)

[17]

Schiele, Nathan Daniel; Gadyatskaya, Olga (2025): "A limited technical background is sufficient for attack-defense tree acceptability", in: Proceedings of the USENIX Security Symposium. (Link)

[18]

Klees, George; Ruef, Andrew; Cooper, Benji; Wei, Shiyi; Hicks, Michael (2018): "Evaluating Fuzz Testing", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[19]

Schloegel, Moritz; Bars, Nils; Schiller, Nico; Bernhard, Lukas; Scharnowski, Tobias; Crump, Addison; Ebrahim, Arash Ale; Bissantz, Nicolai; Muench, Marius; Holz, Thorsten (2024): "SoK: Prudent Evaluation Practices for Fuzzing", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[20]

Monzani, Andrea; Parata, Antonio; Oliveri, Andrea; Aonzo, Simone; Balzarotti, Davide; Lanzi, Andrea (2026): "Unveiling BYOVD Threats: Malware’s Use and Abuse of Kernel Drivers", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[21]

Razuvayevskaya, Olesya; Bontcheva, Kalina (2026): "Truth with a Twist: The Rhetoric of Persuasion in Professional vs. Community-Authored Fact-Checks", in: Proceedings of the ACM Web Conference. (DOI)

[22]

Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)

[23]

Bobek, Michelle; Pröllochs, Nicolas (2026): "Community Fact-Checks Do Not Break Follower Loyalty", in: Proceedings of the ACM Web Conference. (DOI)

[24]

Becerril-Arreola, Rafael (2023): "A Method to Assess and Explain Disparate Impact in Online Retailing", in: Proceedings of the ACM Web Conference. (DOI)

[25]

Fouad, Imane; Santos, Cristiana; Legout, Arnaud; Bielova, Nataliia (2022): "My Cookie is a phoenix: detection, measurement, and lawfulness of cookie respawning with browser fingerprinting", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[26]

Zimmeck, Sebastian; Li, Jie S.; Kim, Hyungtae; Bellovin, Steven M.; Jebara, Tony (2017): "A Privacy Analysis of Cross-device Tracking", in: Proceedings of the USENIX Security Symposium. (Link)

[27]

Li, Frank; Paxson, Vern (2017): "A Large-Scale Empirical Study of Security Patches", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[28]

McNemar, Quinn (1947): "Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages", Psychometrika 12(2):153-157. (DOI)

[29]

Cochran, W. G. (1950): "The Comparison of Percentages in Matched Samples", Biometrika 37(3-4):256-266. (DOI)

[30]

Demir, Nurullah; Hörnemann, Jan; Große-Kampmann, Matteo; Urban, Tobias; Pohlmann, Norbert; Holz, Thorsten; Wressnegger, Christian (2023): "On the Similarity of Web Measurements Under Different Experimental Setups", in: Proceedings of the ACM Internet Measurement Conference, pp. 356-369. ACM DOI 10.1145/3618257.3624795 is listed by DBLP but was not registered with the DOI resolver as of 2026-08-12; the DOI above resolves to the authors' institutional record of the same paper (DOI)

[31]

Sarabi, Armin; Huang, Ziyuan; Wang, Chenlan; Karir, Tai; Liu, Mingyan (2025): "The Ransomware Decade: The Creation of a Fine-Grained Dataset and a Longitudinal Study", in: Proceedings of the USENIX Security Symposium. (Link)

[32]

Umayya, Zeya; Malik, Dhruv; Gosain, Devashish; Sharma, Piyush Kumar (2023): "PTPerf: On the Performance Evaluation of Tor Pluggable Transports", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[33]

Lilliefors, Hubert W. (1967): "On the Kolmogorov-Smirnov Test for Normality with Mean and Variance Unknown", Journal of the American Statistical Association 62(318):399-402. (DOI)

[34]

Dechand, Sergej; Schürmann, Dominik; Busse, Karoline; Acar, Yasemin; Fahl, Sascha; Smith, Matthew (2016): "An Empirical Study of Textual Key-Fingerprint Representations", in: Proceedings of the USENIX Security Symposium. (Link)

[35]

Wasserstein, Ronald L.; Lazar, Nicole A. (2016): "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2):129-133. (DOI)