Working log behind Hypothesis testing. Every query with its denominator, the report script and its unedited output, the folds and their residue, the probes that were run and rejected, the quotes spot-checked, the external sources verified and rejected, and what could not be established.

Corpus-level caveats — the venue scope, the selection funnel, the provisional years, the duplicate records, the posters — are on Corpus and are not restated here. Citation keys are shared with the content page and the single Bibliography; this page adds no bibliography entries of its own.

Voice here is a working log, not prose. It is for someone checking a number.

The neighbouring page Pvalue corrections is 62 KB and shares this page's population exactly (the 1,025 hypothesisTest papers). Broadening it was considered and rejected:

• Its subject is multiplicity — what to do with many p-values. This page's subject is which test produces one. They are independent failure modes and a paper can have both.
• It already names this page as a sibling with a one-line scope statement, so the boundary was set by an earlier run rather than by this one.
• The statistics: namespace has three other red links (biases, regression) that are separate items; folding test choice into the corrections page would have made the namespace incoherent.

Deliberate division of labour. These belong to the neighbours and are linked, not restated: the “at crawl scale the p-value stops carrying information” argument and the effect-size-vs-correction ratio (Pvalue corrections); cluster-robust standard errors as a modelling technique and mixed-effects models (Regression); committing to a test in advance (Study preregistration).

The page does re-derive one figure the corrections page also reports — the crawl-versus-user-study asymmetry — because it holds for paired testing as well as for correcting, and the parallel is the point.

Every query names its own population. hypothesisTest is the denominator for the whole page.

Population                                                    Papers   Share of 5,859
all extraction records                                        5,859    100.0%
inferential  — any statistics tuple that is not descriptive    1,762    30.1%
hypothesisTest — >=1 tuple with kind == 'hypothesis-test'      1,025    17.5%

Definition, in report_hypothesis_testing.mjs:

const hasKind = (p, k) => p.statistics.some((s) => s.kind === k);
const HT = rows.filter((p) => hasKind(p, 'hypothesis-test'));

This is deliberately the same definition report_pvalue_corrections.mjs uses, so the two pages' figures compose. Cross-checked: both scripts independently report 1,025, and the same per-venue hypothesis-test counts (PoPETs 149, USENIX 242, IEEE S&P 124, CCS 110, TheWebConf 236, IMC 71, NDSS 93).

1,806 hypothesis-test tuples across the 1,025 papers (1.76 per paper), carrying 798 distinct free-text method strings.

Cross-population definitions:

const crawled = new Set(rows.filter(POPULATIONS.crawled).map(key));   // lib.mjs
const human   = new Set(rows.filter((p) => p.participants.length > 0).map(key));

participants, not humanAnnotation — crowdworkers labelling data are annotators, not subjects, and conflating them double-counts Mechanical Turk.

statistics[].method is free text and agrees run-to-run on ~20% of exact strings, so it is never aggregated by exact string. 798 distinct strings describe about forty procedures. The fold is an ordered rule list; first match wins; anything matching no rule lands in residue and is printed.

Unmapped residue: 0 strings, 0 papers. Self-test: 146/146 cases pass (node scripts/test_fold.mjs).

1. “Wilcoxon” names two different tests. The Wilcoxon rank-sum test is Mann-Whitney U (two independent samples). The Wilcoxon signed-rank test is the paired one. A fold keyed on /wilcoxon/ merges an unpaired test with a paired one — the exact error the content page is about. Resolution:

• the signed-rank rule fires first
• rank-sum / ranksum / ranked sum fold into the Mann-Whitney family
• a paired or one-sample Wilcoxon resolves to signed-rank, because that is what those words mean
• a bare “Wilcoxon test” gets its own bucket rather than being guessed at, and its size is published as a finding (10 papers)

\bpaired\b does not match inside “unpaired” (no word boundary between n and p), which is what keeps “unpaired Wilcoxon rank-sum” out of the paired family. There is a self-test case for it.

2. Many hypothesis-test tuples are not tests of the paper's hypothesis. They are assumption checks (Shapiro–Wilk, Levene, Mauchly), model diagnostics (Breusch–Pagan, Hosmer–Lemeshow, VIF, Ljung–Box) or not tests at all (Clopper–Pearson intervals, MLE, Chebyshev's inequality). The extractor scored them as hypothesis-test because they are tests. They go in NON_SUBSTANTIVE and are excluded from every “which test” figure, then reported separately:

UNSTATED (no test named)                           73 papers,  56 spellings
ASSUMPTION: normality check                        38 papers,  15 spellings
NOT-A-TEST: estimator, criterion or inequality     25 papers,  26 spellings
ASSUMPTION: model diagnostic                       25 papers,  23 spellings
ASSUMPTION: variance / sphericity check            23 papers,  14 spellings
NOT-A-TEST: confidence interval only               16 papers,  16 spellings
OFF-PAGE: correlation significance                  9 papers,  10 spellings
ASSUMPTION: outlier / unimodality check             7 papers,   7 spellings
NOT-A-TEST: factor analysis / measurement model     4 papers,   5 spellings
OFF-PAGE: regression coefficient p-values           3 papers,   3 spellings

A query that counts kind == “hypothesis-test” and stops there reports Shapiro–Wilk and Clopper–Pearson as the paper's statistical test.

3. The same name is an assumption check or a substantive test depending on context. “Anderson-Darling test for normality” is an assumption check; “Anderson-Darling two-sample test” is a distributional comparison. Same for Kolmogorov–Smirnov, which this literature uses both ways. The normality-context rules therefore fire before the family rules, and both readings have self-test cases.

Recorded because each would have produced a wrong published number, and because two are ordering bugs of a kind that recurs.

Exact-string counting against the folded paper count. This is what an unfolded statistics.method histogram reports.

Family                            Folded  Spellings  Most common exact string      Its papers  Undercount
Chi-square (χ²)                      220         98  "chi-square test"                     27      87.7%
t-test (independent samples)          45         32  "two-sample t-test"                    6      86.7%
Proportion / z-test                   61         44  "z-test"                               9      85.2%
Likelihood-ratio / Wald / F-test      32         17  "likelihood ratio test"                6      81.3%
ANOVA                                 95         47  "one-way ANOVA"                       31      67.4%
Kolmogorov-Smirnov                    93         35  "Kolmogorov-Smirnov test"             37      60.2%
Mann-Whitney U                       213         52  "Mann-Whitney U test"                 90      57.7%
Wilcoxon signed-rank                  85         28  "Wilcoxon signed-rank test"           42      50.6%
Kruskal-Wallis                        91         15  "Kruskal-Wallis test"                 49      46.2%

The item brief for this page quoted “Mann-Whitney U 68, Wilcoxon 45, Fisher's exact 42, t-test 39” from the previous corpus run's exact-string counts. Folded on the current corpus those are 213, 85 (signed-rank) + 213 (rank-sum), 80 and 236. Both the corpus growth and the folding move them; neither number was reusable.

Regex sweep over 5,855 paper.cols.txt files (4 extraction records have none). A probe count is not a finding. Three probes were majority or wholly false-positive and the page publishes the hand-read count.

A probe bug worth recording. The clustering probe was first written as /cluster(ed|-robust)?\s+standard\s+errors/ and found 1 paper. Papers write it the other way round — “standard errors are clustered at the user level”, “errors clustered at the product level” — so the single word order missed 3 of the 4. Both orders are now in the rule. A flat negative from one phrasing is not a negative.

The three genuine clustering papers, all 2023 or later:

Because only 2 of the 3 have a hypothesis-test tuple, the report omits a “share of 1,025” column for the hand-read probe counts rather than printing one that mixes denominators. The page states the figure as corpus-wide.

scripts/ht_quotecheck.mjs verifies all 1,806 hypothesis-test evidence.quote values against the rendering the extractor read. Whitespace is normalised first, because a repaired two-column quote still wraps and grep -F fails on it about half the time.

quotes checked: 1806
  exact (verbatim after whitespace normalisation) : 931 (51.6%)
  partial (>=60% of 5-word windows present)       : 540 (29.9%)
  below threshold                                 : 335 (18.5%)
  no full text on disk                            : 0

Below threshold is not “unsupported”. Five below-threshold quotes were chased by hand and all five are present:

Every quote published on the content page was located by hand. All checked out except one class, below.

Mai et al., PoPETs 2025, more-and-scammier-ads-…. The extraction carries four detailed methodology quotes (repeated-measures ANOVA, Friedman, Tukey HSD, Conover). None of paper.cols.txt, paper.norm.txt or paper.txt contains the string “Friedman”, “Conover”, “omnibus” or “172.47”. Re-extracting the stored paper.pdf with pypdf finds Friedman 7×, Conover 6×, and every quoted sentence verbatim:

FOUND  since the rates do not approximately follow a normal distribution, we use
       Friedman test as the omnibus test, which is a non-parametric alternative to ANOVA
FOUND  we use one-way repeated measures ANOVA as the omnibus test since the ad load
       data follow a normal distribution
FOUND  we use the Conover post-hoc test with Holm-Bonferroni correction
FOUND  Fleiss kappa score of κ = 0.74

paper.cols.txt also drops five of the ten authors from its header (Kieserman, Matsumoto, Andreou, Greenstadt, McCoy), which the PDF has.

Consequences, which generalise beyond this page:

• The extraction was faithful; the stored rendering is deficient. This is the opposite of a hallucination and would have been scored as one by any check keyed on paper.cols.txt.
• quote_check.mjs and ht_quotecheck.mjs can therefore produce false “unlocatable” verdicts. The dataset's own ~0.9% unlocatable rate is partly an artefact of text extraction, not of fabrication.
• The content page footnotes this where it quotes Mai et al., so a reader who greps paper.cols.txt and finds nothing is not misled.
• Not generalised further. Whether other papers have the same defect was not measured — it would need a corpus-wide pypdf-vs-cols diff, which is a separate job. Logged as a TODO below.

All fetched 2026-08-13. Nothing here is from recall.

pages/paired_and_clustered.py, embedded on the page as <file python> and reproduced there in full.

• Synthetic data, fixed seed (SEED = 20260813). Labelled as such on the page and in the docstring. It demonstrates a property of the tests, not a fact about the web. No measurement claim is made from it.
• Embedded block verified byte-identical to the file by exact line-list equality, not by substring containment — 187 lines each as published. This claim went stale twice during the run: the Yates fix (A7) changed the file to 183 lines and the docstring fix (R4-2) to 187, and each time the embedded block had to be regenerated and re-verified. The generic reviewer caught the first of those staleness gaps. Re-embed and re-verify after every edit to the script, and re-state the line count from the check, not from memory.
• Both <code> blocks on the page are the complete output of their part of the script — 25 lines for part A and 35 for part B — verified by exact line-list equality in both directions, page_lines == run_lines.
• The first version of that check was wrong, and a reviewer caught it. It tested only all(line in run for line in page) — that page ⊆ run — which passes on a truncated or spliced block. Both blocks were in fact silently cut: part A dropped its closing four-line interpretation, and part B dropped an entire mid-block paragraph, with no ellipsis marking either. This is precisely the failure the workdir README records as “'Real output' must be the whole output, or say where you cut”, and a subset test cannot see it. A containment check is not an equality check.
• Runtime 3.2 s.

• Part A's counts are chosen to make the point, and the point is not “chi-square is conservative”. With 1,900 concordant and 100 discordant pairs, McNemar returns 10⁻⁹ where the marginal chi-square returns 0.046. The page says explicitly that the direction reverses in other configurations and that the real claim is “a test that ignores the design answers a different question”.
• Part B's ICC = 0.00 row is the rig-check. The naive test's false-positive rate there is 5.1% against a nominal 5%, which is what a correct test should do. Without that row the simulation would be unfalsifiable.
• The cluster-aware test is not exactly nominal either: 5.5%–6.4% across the sweep. That is aggregation to 30-versus-30 units making Mann-Whitney discrete and slightly anti-conservative, not a bug. The page states the range rather than claiming the fix is exact — at ICC 0 the cluster test is in fact worse (6.4% vs 5.1%), and the page says so.
• Weakness, stated: the cluster-mean aggregation is the crudest of the five remedies the page lists, chosen because it is the one that can be demonstrated in twenty lines. It is not a recommendation that it is the best remedy, and the page ranks cluster-robust errors and cluster permutation above it.

node scripts/test_fold.mjs                                  # 146/146 self-tests
node scripts/report_hypothesis_testing.mjs                  # every figure, with denominators
node scripts/report_hypothesis_testing.mjs --wiki           # DokuWiki tables
node scripts/report_hypothesis_testing.mjs --list           # the 1,025 papers
node scripts/report_hypothesis_testing.mjs --quotes 'McNemar'   # evidence behind a family
node scripts/ht_quotecheck.mjs 'Mann-Whitney' --show        # quote verification
python3 pages/paired_and_clustered.py                       # the published demo

# the guard, run BOTH windowed and whole-page
node scripts/check_page_numbers.mjs pages/statistics_hypothesis_testing.txt \
  out/ht_report_all.txt --code
node scripts/check_page_numbers.mjs pages/statistics_hypothesis_testing.txt \
  out/ht_report_all.txt '===== Use in Publications =====' '===== What to Report ====='

out/ht_report_all.txt is the concatenation of the report, the demo output and the quote-check output — the guard needs all three, because the page quotes figures from each.

Section Z of the report lists every number on the page that is not a corpus count — test dates, the four Tang et al. figures, per-paper figures quoted from individual papers, software versions, the demo's constants, and the corpus facts documented elsewhere (138 posters, the 68% and 20% run-to-run agreement figures). That block exists so the guard can account for every digit without adding entries to its shared ALLOW map. Both guard runs report OK.

Three focused reviewers in parallel on a frozen snapshot (out/review_ht/, frozen 2026-08-13T12:04Z), then fixes, then a generic pass. All were told explicitly that the author's context may not be exhaustive, and all were handed the page text, the report script, its output and these notes. Freezing first is deliberate: the workdir README records that applying fixes mid-review makes reviewers report already-fixed items and, once, report the fix itself as the error.

Run first because the README records that reviewers do not check rhetorical sentences. It found four things, three of them wrong, in prose rather than in tables.

Re-ran test_fold.mjs (146/146), report_hypothesis_testing.mjs, ht_quotecheck.mjs and the demo, confirmed the scripts were byte-identical to the frozen copies, and checked every table cell.

Run on the post-fix snapshot out/review_ht/page_r2.txt. It found the single most consequential item of the whole review, and its closing observation is the one to keep.

The reviewer's closing observation, which is the most useful sentence in the whole review log: across findings R4-1, R4-3, R4-4 and R4-6, every one is a case where the page's careful version of a claim exists somewhere — in the provenance, in a later section, in a table — and the prominent version dropped the qualifier. The opening box and the section leads are where hedges go to die. That is the lens to apply to the next page on this site before publishing it, and it is a different failure mode from the stale-figure one the guard catches.

Six of the eight author findings were in prose, not tables — the framing sentences, a superlative, an internal contradiction and a scope word. The three focused reviewers between them found four items: two the author had already found independently (IMC/CCS, Yates), one genuinely new and the most valuable single find of the review (the code blocks were spliced and the check that “verified” them was a subset test), and one factual error in an author name that the bibliography got right and the prose got wrong. The figures themselves — hundreds of them — survived a full independent re-run without a single discrepancy, which is what the check_page_numbers.mjs guard is for and is the first time on this site that pass has come back empty.

* Whether the paper.cols.txt deficiency found in Mai et al. is widespread. One paper is confirmed to have content in its PDF and in none of its three stored text renderings. A corpus-wide pypdf-vs-cols diff would settle it and was not run. It bears on every quote-check figure on this site, so it is the highest-value follow-up here. * Whether the 68% run-to-run agreement on statistics.kind still holds. It was measured on the previous corpus run and has not been re-measured; the page says so where it uses it. * How much the 138 posters inflate the “no test named” rate of 5.7%. Not measured. The figure was not re-run excluding short records. * Whether the 20 “unit of analysis” and 11 “normality assumption” matches are genuine uses. Not read one by one; published as upper bounds and marked as such in the report. * Whether authors who write “median” about a Mann-Whitney result believe it or are being loose. Not answerable from text, and the page says so. * The real intra-class correlation of web-measurement outcomes. The simulation shows everything depends on it and no paper reports one. This is the open question the page leads with, and it is genuinely open. * Whether the two papers whose stated signed-rank/Mann-Whitney rule is keyed on sample size or variable type actually analysed their data correctly. Their designs make the stated rule and the correct rule coincide in the common case, and the page says the results are not in question. Establishing it properly would need their data.

• Hypothesis testing — the page these notes are for.
• Corpus — corpus-wide provenance: venues, funnel, provisional years, posters, duplicates.
• pvalue_corrections — the sibling log; shares this page's 1,025-paper population.