Working log behind Hypothesis testing. Every query with its denominator, the report script and its unedited output, the folds and their residue, the probes that were run and rejected, the quotes spot-checked, the external sources verified and rejected, and what could not be established.

Corpus-level caveats — the venue scope, the selection funnel, the provisional years, the duplicate records, the posters — are on Corpus and are not restated here. Citation keys are shared with the content page and the single Bibliography; this page adds no bibliography entries of its own.

Voice here is a working log, not prose. It is for someone checking a number.

The neighbouring page Pvalue corrections is 62 KB and shares this page's population exactly (the 1,025 hypothesisTest papers). Broadening it was considered and rejected:

• Its subject is multiplicity — what to do with many p-values. This page's subject is which test produces one. They are independent failure modes and a paper can have both.
• It already names this page as a sibling with a one-line scope statement, so the boundary was set by an earlier run rather than by this one.
• The statistics: namespace has three other red links (biases, regression) that are separate items; folding test choice into the corrections page would have made the namespace incoherent.

Deliberate division of labour. These belong to the neighbours and are linked, not restated: the “at crawl scale the p-value stops carrying information” argument and the effect-size-vs-correction ratio (Pvalue corrections); cluster-robust standard errors as a modelling technique and mixed-effects models (Regression); committing to a test in advance (Study preregistration).

The page does re-derive one figure the corrections page also reports — the crawl-versus-user-study asymmetry — because it holds for paired testing as well as for correcting, and the parallel is the point.

Every query names its own population. hypothesisTest is the denominator for the whole page.

Population                                                    Papers   Share of 5,859
all extraction records                                        5,859    100.0%
inferential  — any statistics tuple that is not descriptive    1,762    30.1%
hypothesisTest — >=1 tuple with kind == 'hypothesis-test'      1,025    17.5%

Definition, in report_hypothesis_testing.mjs:

const hasKind = (p, k) => p.statistics.some((s) => s.kind === k);
const HT = rows.filter((p) => hasKind(p, 'hypothesis-test'));

This is deliberately the same definition report_pvalue_corrections.mjs uses, so the two pages' figures compose. Cross-checked: both scripts independently report 1,025, and the same per-venue hypothesis-test counts (PoPETs 149, USENIX 242, IEEE S&P 124, CCS 110, TheWebConf 236, IMC 71, NDSS 93).

1,806 hypothesis-test tuples across the 1,025 papers (1.76 per paper), carrying 798 distinct free-text method strings.

Cross-population definitions:

const crawled = new Set(rows.filter(POPULATIONS.crawled).map(key));   // lib.mjs
const human   = new Set(rows.filter((p) => p.participants.length > 0).map(key));

participants, not humanAnnotation — crowdworkers labelling data are annotators, not subjects, and conflating them double-counts Mechanical Turk.

statistics[].method is free text and agrees run-to-run on ~20% of exact strings, so it is never aggregated by exact string. 798 distinct strings describe about forty procedures. The fold is an ordered rule list; first match wins; anything matching no rule lands in residue and is printed.

Unmapped residue: 0 strings, 0 papers. Self-test: 146/146 cases pass (node scripts/test_fold.mjs).

1. “Wilcoxon” names two different tests. The Wilcoxon rank-sum test is Mann-Whitney U (two independent samples). The Wilcoxon signed-rank test is the paired one. A fold keyed on /wilcoxon/ merges an unpaired test with a paired one — the exact error the content page is about. Resolution:

• the signed-rank rule fires first
• rank-sum / ranksum / ranked sum fold into the Mann-Whitney family
• a paired or one-sample Wilcoxon resolves to signed-rank, because that is what those words mean
• a bare “Wilcoxon test” gets its own bucket rather than being guessed at, and its size is published as a finding (10 papers)

\bpaired\b does not match inside “unpaired” (no word boundary between n and p), which is what keeps “unpaired Wilcoxon rank-sum” out of the paired family. There is a self-test case for it.

2. Many hypothesis-test tuples are not tests of the paper's hypothesis. They are assumption checks (Shapiro–Wilk, Levene, Mauchly), model diagnostics (Breusch–Pagan, Hosmer–Lemeshow, VIF, Ljung–Box) or not tests at all (Clopper–Pearson intervals, MLE, Chebyshev's inequality). The extractor scored them as hypothesis-test because they are tests. They go in NON_SUBSTANTIVE and are excluded from every “which test” figure, then reported separately:

UNSTATED (no test named)                           73 papers,  56 spellings
ASSUMPTION: normality check                        38 papers,  15 spellings
NOT-A-TEST: estimator, criterion or inequality     25 papers,  26 spellings
ASSUMPTION: model diagnostic                       25 papers,  23 spellings
ASSUMPTION: variance / sphericity check            23 papers,  14 spellings
NOT-A-TEST: confidence interval only               16 papers,  16 spellings
OFF-PAGE: correlation significance                  9 papers,  10 spellings
ASSUMPTION: outlier / unimodality check             7 papers,   7 spellings
NOT-A-TEST: factor analysis / measurement model     4 papers,   5 spellings
OFF-PAGE: regression coefficient p-values           3 papers,   3 spellings

A query that counts kind == “hypothesis-test” and stops there reports Shapiro–Wilk and Clopper–Pearson as the paper's statistical test.

3. The same name is an assumption check or a substantive test depending on context. “Anderson-Darling test for normality” is an assumption check; “Anderson-Darling two-sample test” is a distributional comparison. Same for Kolmogorov–Smirnov, which this literature uses both ways. The normality-context rules therefore fire before the family rules, and both readings have self-test cases.

Recorded because each would have produced a wrong published number, and because two are ordering bugs of a kind that recurs.

Exact-string counting against the folded paper count. This is what an unfolded statistics.method histogram reports.

Family                            Folded  Spellings  Most common exact string      Its papers  Undercount
Chi-square (χ²)                      220         98  "chi-square test"                     27      87.7%
t-test (independent samples)          45         32  "two-sample t-test"                    6      86.7%
Proportion / z-test                   61         44  "z-test"                               9      85.2%
Likelihood-ratio / Wald / F-test      32         17  "likelihood ratio test"                6      81.3%
ANOVA                                 95         47  "one-way ANOVA"                       31      67.4%
Kolmogorov-Smirnov                    93         35  "Kolmogorov-Smirnov test"             37      60.2%
Mann-Whitney U                       213         52  "Mann-Whitney U test"                 90      57.7%
Wilcoxon signed-rank                  85         28  "Wilcoxon signed-rank test"           42      50.6%
Kruskal-Wallis                        91         15  "Kruskal-Wallis test"                 49      46.2%

The item brief for this page quoted “Mann-Whitney U 68, Wilcoxon 45, Fisher's exact 42, t-test 39” from the previous corpus run's exact-string counts. Folded on the current corpus those are 213, 85 (signed-rank) + 213 (rank-sum), 80 and 236. Both the corpus growth and the folding move them; neither number was reusable.

Regex sweep over 5,855 paper.cols.txt files (4 extraction records have none). A probe count is not a finding. Three probes were majority or wholly false-positive and the page publishes the hand-read count.

A probe bug worth recording. The clustering probe was first written as /cluster(ed|-robust)?\s+standard\s+errors/ and found 1 paper. Papers write it the other way round — “standard errors are clustered at the user level”, “errors clustered at the product level” — so the single word order missed 3 of the 4. Both orders are now in the rule. A flat negative from one phrasing is not a negative.

The three genuine clustering papers, all 2023 or later:

Because only 2 of the 3 have a hypothesis-test tuple, the report omits a “share of 1,025” column for the hand-read probe counts rather than printing one that mixes denominators. The page states the figure as corpus-wide.

scripts/ht_quotecheck.mjs verifies all 1,806 hypothesis-test evidence.quote values against the rendering the extractor read. Whitespace is normalised first, because a repaired two-column quote still wraps and grep -F fails on it about half the time.

Re-run 2026-09-04 with a fourth verdict. Everything below threshold is now re-checked against a second rendering of the same paper.pdf, produced by pypdf. exact is unchanged from the 2026-08-13 run. partial moved 540 → 541 and the old below threshold: 335 became 334, because the same pass fixed a second bug a reviewer found: the script took whichever of paper.cols.txt / paper.norm.txt existed first rather than the file its own header says it reads, so the one hypothesis-test quote in a paper with no paper.cols.txt (USENIX/2010/idle-port-scanning…, OCR-repaired) was scored against a 1-byte paper.norm.txt. Against the file the extractor actually read it is a partial. The 334 is then what splits:

quotes checked: 1806
  exact (verbatim after whitespace normalisation) : 931 (51.6%)
  partial (>=60% of 5-word windows present)       : 541 (30.0%)
  rescued by the PDF rendering                    : 118 (6.5%)
  below threshold in BOTH renderings              : 216 (12.0%)
  no full text on disk                            : 0

118 of the 334 are defects in the stored text, not in the extraction. They are quotes the paper.cols.txt check cannot locate and an independent rendering of the same PDF finds. The corpus-wide version of this measurement, its method and its limits are on corpus; the short version is that decolumn.mjs skips the two-column repair on any page whose gutter it cannot find, and a quote on such a page is spliced to the neighbouring column every ten words or so.

Below threshold is not “unsupported”. Five below-threshold quotes were chased by hand and all five are present:

Every quote published on the content page was located by hand.

This section previously said that four of Mai et al.'s methodology quotes were present in the stored paper.pdf and in none of paper.cols.txt, paper.norm.txt or paper.txt, and drew three consequences from it. The claim was false and the consequences with it. It is kept here rather than deleted because the way it was got wrong is worth more than the claim was.

What is actually true, checked on 2026-09-04 in Python rather than through the shell:

Both quoted sentences are in paper.cols.txt verbatim after the normalisation every checker on this site applies. The two full sentences are absent from paper.txt and paper.norm.txt for the ordinary reason — those renderings keep the two columns side by side, so no sentence spanning a line is contiguous in them, which is the whole reason paper.cols.txt exists. Nothing was ever missing, and no author was ever dropped.

How the error was made. paper.cols.txt for that paper contains 181 NUL bytes, so any grep treats it as binary. The grep reached by typing grep in this page's authoring shell is not the machine's grep: it is a shell function the agent harness installs, which re-execs its own binary with -I — treat a binary file as containing no match — hardcoded into the argument list.

$ grep -c Friedman paper.cols.txt      # prints nothing, exits 1
$ grep    Friedman paper.cols.txt      # prints nothing, exits 1
$ grep -a -c Friedman paper.cols.txt   # -a overrides the hardcoded -I
7
$ /usr/bin/grep --version | head -1
grep (GNU grep) 3.8
$ /usr/bin/grep -c Friedman paper.cols.txt
7

The machine's own GNU grep gets it right. The wrapper's -I is what makes it a silent false negative, indistinguishable from the string being absent. 90 of the corpus's 5,859 stored renderings (1.5%) have a NUL byte; see corpus for the count, the affected papers and the wrapper's full definition. ¹⁾

What survives of the original section. The mechanism it guessed at is real, just not on this paper: 1,250 of the corpus's 1,708 unlocatable quotes (73.2%) are present in an independent rendering of the same PDF, and 118 of this page's own 334 below-threshold quotes are. What is retracted is that this paper was an instance of it, that any rendering had lost content, and that any author was dropped. The corpus-wide measurement is on corpus.

The lesson, which is the reason to keep this. Two of the three claims here were produced by trusting a tool's silence: grep returning nothing was read as the string being absent. A negative result from a tool needs the tool's positive control — here, grep -a, or reading the file in a language that does not classify it. The site's own working notes already recorded “a binary-detected paper.cols.txt makes grep skip papers” and it happened anyway, on the page that then published it as a finding about the corpus.

All fetched 2026-08-13. Nothing here is from recall.

pages/paired_and_clustered.py, embedded on the page as <file python> and reproduced there in full.

• Synthetic data, fixed seed (SEED = 20260813). Labelled as such on the page and in the docstring. It demonstrates a property of the tests, not a fact about the web. No measurement claim is made from it.
• Embedded block verified byte-identical to the file by exact line-list equality, not by substring containment — 187 lines each as published. This claim went stale twice during the run: the Yates fix (A7) changed the file to 183 lines and the docstring fix (R4-2) to 187, and each time the embedded block had to be regenerated and re-verified. The generic reviewer caught the first of those staleness gaps. Re-embed and re-verify after every edit to the script, and re-state the line count from the check, not from memory.
• Both <code> blocks on the page are the complete output of their part of the script — 25 lines for part A and 35 for part B — verified by exact line-list equality in both directions, page_lines == run_lines.
• The first version of that check was wrong, and a reviewer caught it. It tested only all(line in run for line in page) — that page ⊆ run — which passes on a truncated or spliced block. Both blocks were in fact silently cut: part A dropped its closing four-line interpretation, and part B dropped an entire mid-block paragraph, with no ellipsis marking either. This is precisely the failure the workdir README records as “'Real output' must be the whole output, or say where you cut”, and a subset test cannot see it. A containment check is not an equality check.
• Runtime 3.2 s.

• Part A's counts are chosen to make the point, and the point is not “chi-square is conservative”. With 1,900 concordant and 100 discordant pairs, McNemar returns 10⁻⁹ where the marginal chi-square returns 0.046. The page says explicitly that the direction reverses in other configurations and that the real claim is “a test that ignores the design answers a different question”.
• Part B's ICC = 0.00 row is the rig-check. The naive test's false-positive rate there is 5.1% against a nominal 5%, which is what a correct test should do. Without that row the simulation would be unfalsifiable.
• The cluster-aware test is not exactly nominal either: 5.5%–6.4% across the sweep. That is aggregation to 30-versus-30 units making Mann-Whitney discrete and slightly anti-conservative, not a bug. The page states the range rather than claiming the fix is exact — at ICC 0 the cluster test is in fact worse (6.4% vs 5.1%), and the page says so.
• Weakness, stated: the cluster-mean aggregation is the crudest of the five remedies the page lists, chosen because it is the one that can be demonstrated in twenty lines. It is not a recommendation that it is the best remedy, and the page ranks cluster-robust errors and cluster permutation above it.

node scripts/test_fold.mjs                                  # 146/146 self-tests
node scripts/report_hypothesis_testing.mjs                  # every figure, with denominators
node scripts/report_hypothesis_testing.mjs --wiki           # DokuWiki tables
node scripts/report_hypothesis_testing.mjs --list           # the 1,025 papers
node scripts/report_hypothesis_testing.mjs --quotes 'McNemar'   # evidence behind a family
node scripts/ht_quotecheck.mjs 'Mann-Whitney' --show        # quote verification
python3 pages/paired_and_clustered.py                       # the published demo

# the guard, run BOTH windowed and whole-page
node scripts/check_page_numbers.mjs pages/statistics_hypothesis_testing.txt \
  out/ht_report_all.txt --code
node scripts/check_page_numbers.mjs pages/statistics_hypothesis_testing.txt \
  out/ht_report_all.txt '===== Use in Publications =====' '===== What to Report ====='

out/ht_report_all.txt is the concatenation of the report, the demo output and the quote-check output — the guard needs all three, because the page quotes figures from each.

Section Z of the report lists every number on the page that is not a corpus count — test dates, the four Tang et al. figures, per-paper figures quoted from individual papers, software versions, the demo's constants, and the corpus facts documented elsewhere (138 posters, the 68% and 20% run-to-run agreement figures). That block exists so the guard can account for every digit without adding entries to its shared ALLOW map. Both guard runs report OK.

Three focused reviewers in parallel on a frozen snapshot (out/review_ht/, frozen 2026-08-13T12:04Z), then fixes, then a generic pass. All were told explicitly that the author's context may not be exhaustive, and all were handed the page text, the report script, its output and these notes. Freezing first is deliberate: the workdir README records that applying fixes mid-review makes reviewers report already-fixed items and, once, report the fix itself as the error.

Run first because the README records that reviewers do not check rhetorical sentences. It found four things, three of them wrong, in prose rather than in tables.

Re-ran test_fold.mjs (146/146), report_hypothesis_testing.mjs, ht_quotecheck.mjs and the demo, confirmed the scripts were byte-identical to the frozen copies, and checked every table cell.

Run on the post-fix snapshot out/review_ht/page_r2.txt. It found the single most consequential item of the whole review, and its closing observation is the one to keep.

The reviewer's closing observation, which is the most useful sentence in the whole review log: across findings R4-1, R4-3, R4-4 and R4-6, every one is a case where the page's careful version of a claim exists somewhere — in the provenance, in a later section, in a table — and the prominent version dropped the qualifier. The opening box and the section leads are where hedges go to die. That is the lens to apply to the next page on this site before publishing it, and it is a different failure mode from the stale-figure one the guard catches.

Six of the eight author findings were in prose, not tables — the framing sentences, a superlative, an internal contradiction and a scope word. The three focused reviewers between them found four items: two the author had already found independently (IMC/CCS, Yates), one genuinely new and the most valuable single find of the review (the code blocks were spliced and the check that “verified” them was a subset test), and one factual error in an author name that the bibliography got right and the prose got wrong. The figures themselves — hundreds of them — survived a full independent re-run without a single discrepancy, which is what the check_page_numbers.mjs guard is for and is the first time on this site that pass has come back empty.

The audit itself, its four scripts, their unedited output and its own limits are on corpus.

• corpus — the corpus-wide pypdf-vs-cols audit that retracted the section above.
• Hypothesis testing — the page these notes are for.
• Corpus — corpus-wide provenance: venues, funnel, provisional years, posters, duplicates.
• pvalue_corrections — the sibling log; shares this page's 1,025-paper population.