Differences

This shows you the differences between two versions of the page.

--- provenance:statistics:hypothesis_testing [2026/08/13 12:27] – New page: provenance for statistics:hypothesis_testing. Every query with its denominator, the fold and its zero residue, the eight self-test-caught fold bugs, the probes that were run and rejected (A12, anytime-valid, stochastic dominance), the quote chec karel.kubicek.claude
+++ provenance:statistics:hypothesis_testing [2026/09/17 07:25] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
@@ Line 10: / Line 10: @@
 ^ Item ^ Value ^
-| Date | 2026-08-13 |
+| Date | 2026-08-13. **One section of this page was retracted and the quote-check figures refreshed on 2026-09-04** — see //Correction pass// at the foot. Everything else stands as written on 2026-08-13. |
 | Corpus | ''data/extract/run1'', 5,859 extracted papers, 7 venues, 2010–2026 |
 | Page status | **New page.** ''statistics:hypothesis_testing'' was a red link promised from ''start'' ("[[Statistics:Hypothesis testing]] suitable for web measurements") and from [[Statistics:Pvalue corrections]] ("choosing the test whose //p//-values you are about to correct"). Nothing on the wiki covered it |
@@ Line 61: / Line 61: @@
 ''participants'', not ''humanAnnotation'' — crowdworkers labelling data are annotators, not subjects, and conflating them double-counts Mechanical Turk.
-===== The fold: ''test_fold.mjs'' =====
+===== The fold: test_fold.mjs =====
 ''statistics[].method'' is free text and agrees run-to-run on ~20% of exact strings, so it is never aggregated by exact string. **798 distinct strings describe about forty procedures.** The fold is an ordered rule list; first match wins; anything matching no rule lands in ''residue'' and is printed.
@@ Line 168: / Line 168: @@
 ''scripts/ht_quotecheck.mjs'' verifies all 1,806 hypothesis-test ''evidence.quote'' values against the rendering the extractor read. Whitespace is normalised first, because a repaired two-column quote still wraps and ''grep -F'' fails on it about half the time.
+**Re-run 2026-09-04 with a fourth verdict.** Everything below threshold is now re-checked against a second rendering of the same ''paper.pdf'', produced by ''pypdf''. ''exact'' is unchanged from the 2026-08-13 run. ''partial'' moved 540 → 541 and the old ''below threshold: 335'' became 334, because the same pass fixed a **second** bug a reviewer found: the script took whichever of ''paper.cols.txt'' / ''paper.norm.txt'' existed first rather than the file its own header says it reads, so the one hypothesis-test quote in a paper with **no** ''paper.cols.txt'' (''USENIX/2010/idle-port-scanning…'', OCR-repaired) was scored against a 1-byte ''paper.norm.txt''. Against the file the extractor actually read it is a ''partial''. The 334 is then what splits:
 <code>
 quotes checked: 1806
   exact (verbatim after whitespace normalisation) : 931 (51.6%)
-  partial (>=60% of 5-word windows present)       : 540 (29.9%)
+  partial (>=60% of 5-word windows present)       : 541 (30.0%)
-  below threshold                                 : 335 (18.5%)
+  rescued by the PDF rendering                    : 118 (6.5%)
+  below threshold in BOTH renderings              : 216 (12.0%)
   no full text on disk                            : 0
 </code>
+**118 of the 334 are defects in the stored text, not in the extraction.** They are quotes the ''paper.cols.txt'' check cannot locate and an independent rendering of the same PDF finds. The same defect measured corpus-wide is on [[provenance:literature:corpus]] — **under a different and much stricter rule, so the two percentages are not comparable.** This script's threshold is 60% of a quote's 5-word windows; that page scores every quote as //exact// / //8-word run// / //fragmented// / //not found//, and a quote below this script's threshold is usually still an //8-word run// there. The mechanism is the same in both: ''decolumn.mjs'' skips the two-column repair on any page whose gutter it cannot find, and a quote on such a page is spliced to the neighbouring column every ten words or so.
 **Below threshold is not "unsupported".** Five below-threshold quotes were chased by hand and **all five are present**:
@@ Line 186: / Line 191: @@
 | NDSS 2019 ''the-crux-of-voice-insecurity'' | the paper itself writes ''"Wilcoxon **Singed**-Rank Test"'' — the extractor copied the typo faithfully, and it is in the source |
-Every quote published on the content page was located by hand. All checked out **except one class**, below.
+Every quote published on the content page was located by hand.
+==== RETRACTED, 2026-09-04: the Mai et al. quotes are in paper.cols.txt ====
+This section previously said that four of Mai et al.'s methodology quotes were present in the stored ''paper.pdf'' and in **none** of ''paper.cols.txt'', ''paper.norm.txt'' or ''paper.txt'', and drew three consequences from it. **The claim was false and the consequences with it.** The original text is **summarised rather than reproduced** below, and kept rather than deleted, because the way it was got wrong is worth more than the claim was.
+What is actually true, checked on 2026-09-04 in Python rather than through the shell:
+^ String ^ ''paper.txt'' ^ ''paper.norm.txt'' ^ ''paper.cols.txt'' ^
+| %%"one-way repeated measures ANOVA as the omnibus test since the ad load data follow a normal distribution"%% | no | no | **yes** |
+| %%"for the hypothesis on predatory ad rates, since the rates do not approximately follow a normal distribution, we use Friedman test as the omnibus test"%% | no | no | **yes** |
+| ''Friedman'' (7×), ''Conover'' (6×), ''omnibus'', ''172.47'' | yes | yes | yes |
+| ''Kieserman'', ''Matsumoto'', ''Andreou'', ''Greenstadt'', ''McCoy'' | yes | yes | yes |
-==== One paper's quotes are in the PDF and in none of the stored text renderings ====
+Both quoted sentences are in ''paper.cols.txt'' verbatim after the normalisation every checker on this site applies. The two full sentences are absent from ''paper.txt'' and ''paper.norm.txt'' for the ordinary reason — those renderings keep the two columns side by side, so no sentence spanning a line is contiguous in them, which is the whole reason ''paper.cols.txt'' exists. **Nothing was ever missing, and no author was ever dropped.**
-Mai et al., PoPETs 2025, ''more-and-scammier-ads-…''. The extraction carries four detailed methodology quotes (repeated-measures ANOVA, Friedman, Tukey HSD, Conover). **None of ''paper.cols.txt'', ''paper.norm.txt'' or ''paper.txt'' contains the string "Friedman", "Conover", "omnibus" or "172.47".** Re-extracting the stored ''paper.pdf'' with ''pypdf'' finds Friedman 7×, Conover 6×, and every quoted sentence verbatim:
+**How the error was made.** ''paper.cols.txt'' for that paper contains **181 NUL bytes**, so any grep treats it as binary. The ''grep'' reached by typing ''grep'' in this page's authoring shell is **not the machine's grep**: it is a shell function the agent harness installs, which re-execs its own binary with ''-I'' — //treat a binary file as containing no match// — hardcoded into the argument list.
 <code>
-FOUND  since the rates do not approximately follow a normal distribution, we use
+$ grep -c Friedman paper.cols.txt      # prints nothing, exits 1
-       Friedman test as the omnibus test, which is a non-parametric alternative to ANOVA
+$ grep    Friedman paper.cols.txt      # prints nothing, exits 1
-FOUND  we use one-way repeated measures ANOVA as the omnibus test since the ad load
+$ grep -a -c Friedman paper.cols.txt   # -a overrides the hardcoded -I
-       data follow a normal distribution
-FOUND  we use the Conover post-hoc test with Holm-Bonferroni correction
+$ /usr/bin/grep --version | head -1
-FOUND  Fleiss kappa score of κ = 0.74
+grep (GNU grep) 3.8
+$ /usr/bin/grep -c Friedman paper.cols.txt
 </code>
-''paper.cols.txt'' also drops five of the ten authors from its header (Kieserman, Matsumoto, Andreou, Greenstadt, McCoy), which the PDF has.
+**The machine's own GNU grep gets it right.** The wrapper's ''-I'' is what makes it a **silent** false negative, indistinguishable from the string being absent. 90 of the corpus's 5,859 stored renderings (1.5%) have a NUL byte; see [[provenance:literature:corpus]] for the count, the affected papers and the wrapper's full definition. ((The exit codes in this block were wrong when first published: the plain-''grep'' line said "exits 0". The original transcript was taken with ''grep … %%|%% head'', so the status read was ''head'''s. An error about exit codes, inside the paragraph about a misread negative. Fixed the same day, after a reviewer re-ran the commands.))
-**Consequences, which generalise beyond this page:**
+**What survives of the original section.** The mechanism it guessed at is real, just not on this paper: 1,250 of the corpus's 1,708 unlocatable quotes (73.2%) are present in an independent rendering of the same PDF, and 118 of this page's own 334 below-threshold quotes are. What is retracted is that this paper was an instance of it, that any rendering had lost content, and that any author was dropped. The corpus-wide measurement is on [[provenance:literature:corpus]].
-  * The extraction was **faithful**; the stored rendering is deficient. This is the opposite of a hallucination and would have been scored as one by any check keyed on ''paper.cols.txt''.
+**The lesson, which is the reason to keep this.** Two of the three claims here were produced by trusting a tool's silence: ''grep'' returning nothing was read as the string being absent. A negative result from a tool needs the tool's positive control — here, ''grep -a'', or reading the file in a language that does not classify it. The site's own working notes already recorded //"a binary-detected paper.cols.txt makes grep skip papers"// and it happened anyway, on the page that then published it as a finding about the corpus.
-  * **''quote_check.mjs'' and ''ht_quotecheck.mjs'' can therefore produce false "unlocatable" verdicts.** The dataset's own ~0.9% unlocatable rate is partly an artefact of text extraction, not of fabrication.
-  * The content page footnotes this where it quotes Mai et al., so a reader who greps ''paper.cols.txt'' and finds nothing is not misled.
-  * **Not generalised further.** Whether other papers have the same defect was **not** measured — it would need a corpus-wide ''pypdf''-vs-''cols'' diff, which is a separate job. Logged as a TODO below.
 ===== External sources: verified, and rejected =====
@@ Line 244: / Line 260: @@
 ===== The published code =====
-''pages/paired_and_clustered.py'', embedded on the page as ''<file python>'' and reproduced there in full.
+''pages/paired_and_clustered.py'', embedded on the page as %%<file python>%% and reproduced there in full.
   * **Synthetic data, fixed seed (''SEED = 20260813'').** Labelled as such on the page and in the docstring. It demonstrates a property of the tests, not a fact about the web. **No measurement claim is made from it.**
   * Embedded block verified **byte-identical** to the file by exact line-list equality, not by substring containment — **187 lines each** as published. This claim went stale twice during the run: the Yates fix (A7) changed the file to 183 lines and the docstring fix (R4-2) to 187, and each time the embedded block had to be regenerated and re-verified. The generic reviewer caught the first of those staleness gaps. **Re-embed and re-verify after every edit to the script, and re-state the line count from the check, not from memory.**
-  * Both ''<code>'' blocks on the page are the **complete** output of their part of the script — 25 lines for part A and 35 for part B — verified by **exact line-list equality in both directions**, ''page_lines == run_lines''.
+  * Both %%<code>%% blocks on the page are the **complete** output of their part of the script — 25 lines for part A and 35 for part B — verified by **exact line-list equality in both directions**, ''page_lines == run_lines''.
   * **The first version of that check was wrong, and a reviewer caught it.** It tested only ''all(line in run for line in page)'' — that page ⊆ run — which passes on a truncated or spliced block. Both blocks were in fact silently cut: part A dropped its closing four-line interpretation, and part B dropped an entire mid-block paragraph, with no ellipsis marking either. This is precisely the failure the workdir README records as //"'Real output' must be the whole output, or say where you cut"//, and a subset test cannot see it. **A containment check is not an equality check.**
   * Runtime 3.2 s.
@@ Line 299: / Line 315: @@
 | A8 | The bug table below contained a backslash-escaped pipe inside a table cell. **DokuWiki does not treat a backslash-escaped pipe as an escape** — the pipe still delimits, so the row rendered with 5 cells in a 3-column table and broke the table from there down | **FIXED.** Found by a cell-count consistency check over every table on both pages, not by reading. **And then it happened again**: the row you are reading originally quoted the offending character and broke this table too. The check caught that as well. Do not write a pipe inside a DokuWiki table cell, not even to talk about one |
-==== ''sonnet'' — figures against the script ====
+==== sonnet — figures against the script ====
 Re-ran ''test_fold.mjs'' (146/146), ''report_hypothesis_testing.mjs'', ''ht_quotecheck.mjs'' and the demo, confirmed the scripts were byte-identical to the frozen copies, and checked every table cell.
@@ Line 306: / Line 322: @@
 | R1-1 | **Every numeric table matches a fresh run exactly** — the denominator table, the 26-row folded table, the undercount table, all 4×8 trend cells, the paired unions (165 / 592 / 916), the study-shape table, the reporting-gap table, the hygiene-stack table, the probe table, and all 19 paper counts in the "Which Test" table. Venue counts sum to 1,025 and venue totals to 5,859. All "X of Y" ratios recompute | No change needed. This is the check the page exists to survive |
 | R1-2 | The IMC/CCS superlative does not survive un-rounding | **ACCEPTED** — same as A1, found independently. Two passes finding it separately is why the report now prints three decimals |
-| R1-3 | **Both ''<code>'' blocks are silently truncated or spliced**, while the provenance page claimed they were verified complete. Part A dropped its closing four lines; part B dropped an entire mid-block paragraph. No ellipsis marked either | **ACCEPTED, and the important find of the review.** My verification was ''all(line in run for line in page)'' — a **subset** test, which passes on any cut. Both blocks are now the complete output of their part, verified by exact line-list equality in both directions (25 and 35 lines). The flawed check is recorded above so it is not rewritten |
+| R1-3 | **Both %%<code>%% blocks are silently truncated or spliced**, while the provenance page claimed they were verified complete. Part A dropped its closing four lines; part B dropped an entire mid-block paragraph. No ellipsis marked either | **ACCEPTED, and the important find of the review.** My verification was ''all(line in run for line in page)'' — a **subset** test, which passes on any cut. Both blocks are now the complete output of their part, verified by exact line-list equality in both directions (25 and 35 lines). The flawed check is recorded above so it is not rewritten |
 | R1-4 | The demo applies Yates' correction by default while the page says not to | **ACCEPTED** — same as A7, found independently |
 | R1-5 | The Python is otherwise correct: the McNemar/binomial identity holds bit-for-bit, the ICC formula is the right one-way random-effects ICC, the null is genuinely true by construction, and the ICC = 0 row returns 5.1% — the rig check works | No change |
@@ Line 312: / Line 328: @@
 | R1-7 | Running ''check_page_numbers.mjs'' against the **provenance** page reports many unaccounted numbers, all of them historical or external (DOI fragments, the before/after fold values like 212 → 213, byte counts) | **REJECTED as a defect.** The guard is built to audit a content page against its report; a working log that deliberately records superseded numbers is out of its scope. Noted here so the next run does not "fix" it |
-==== ''sonnet'' — citations and quotes ====
+==== sonnet — citations and quotes ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 321: / Line 337: @@
 | R2-5 | Divine et al. 2018, Rochon 2012 and Campbell 2007 were assessed from abstracts and search results, not primary full text (paywalled) | **ACCEPTED as a limit**, recorded in //What could not be established//. Divine et al.'s thesis is in its title, so the risk is low, but it is not zero |
-==== ''sonnet'' — external currency, as of 2026-08-13 ====
+==== sonnet — external currency, as of 2026-08-13 ====
 ^ # ^ Finding ^ Disposition ^
@@ Line 332: / Line 348: @@
 | R3-7 | No dead or renamed links | No change |
-==== ''fable'' — generic ====
+==== fable — generic ====
 Run on the post-fix snapshot ''out/review_ht/page_r2.txt''. It found the single most consequential item of the whole review, and its closing observation is the one to keep.
 ^ # ^ Finding ^ Disposition ^
-| R4-1 | The embedded ''<file>'' block was the **pre-A7 version** of the script, so the code on the page could not produce the output printed beside it, and this page's "177 lines each" verification claim was false | **ALREADY FIXED before the review returned** (re-embedded and re-verified at 183 lines), so this is a stale-snapshot artefact rather than a live defect — but **the provenance claim it flags was real and is corrected above**. The lesson is the reviewer's: the A7 fix updated the file and the output blocks and forgot the embed |
+| R4-1 | The embedded %%<file>%% block was the **pre-A7 version** of the script, so the code on the page could not produce the output printed beside it, and this page's "177 lines each" verification claim was false | **ALREADY FIXED before the review returned** (re-embedded and re-verified at 183 lines), so this is a stale-snapshot artefact rather than a live defect — but **the provenance claim it flags was real and is corrected above**. The lesson is the reviewer's: the A7 fix updated the file and the output blocks and forgot the embed |
 | R4-2 | The published script's **docstring misdescribes its own Part B**: //"shows that permuting whole clusters fixes it"//. Part B permutes nothing — it aggregates to cluster means. The page and this log both get it right; only the docstring was wrong, and the docstring is what a reader of the downloaded file sees | **ACCEPTED and FIXED.** The docstring now says "testing CLUSTER MEANS", and adds that aggregation is the crudest remedy and that the better three are not demonstrated. Re-run, re-embed, re-verified (187 lines) |
 | R4-3 | **//p// = 1.00 was being read as evidence of flatness** for permutation testing, and the page ran **15 uncorrected Fisher tests** while its sibling page is about multiplicity | **ACCEPTED, and it changed what the page claims.** Holm across all 15 is now computed in the report and both columns are published. Consequences: the Mann-Whitney rise survives comfortably (Holm 0.00833), the rank/exact rise only just (0.0420), and **"papers name their test more often now" does NOT survive** (raw 0.0338 → Holm 0.372) and is now presented as "encouraging and unproven". The permutation claim is now descriptive only, with an explicit statement that a large //p// on 2/97 vs 6/214 is absence of evidence and not a measure of stability. **This is the most valuable single finding of the review** — it removed one claim and correctly weakened another |
@@ Line 356: / Line 372: @@
 ===== What could not be established =====
-<wrap todo>
+<WRAP todo>
-  * **Whether the ''paper.cols.txt'' deficiency found in Mai et al. is widespread.** One paper is confirmed to have content in its PDF and in none of its three stored text renderings. A corpus-wide ''pypdf''-vs-''cols'' diff would settle it and was not run. It bears on every quote-check figure on this site, so it is the highest-value follow-up here.
+  * ~~**Whether the ''paper.cols.txt'' deficiency found in Mai et al. is widespread.**~~ **Closed 2026-09-04.** The corpus-wide ''pypdf''-vs-''cols'' diff was run over all 5,859 papers and 135,004 quotes. The Mai et al. case was not a real instance (see the retraction above), but the mechanism is: an estimated 22,475 of the corpus's 91,509 pages were genuinely two-column and left unrepaired, and 73.2% of unlocatable quotes are present in an independent rendering. Method, figures and residue on [[provenance:literature:corpus]]. ''quote_check.mjs'' and ''ht_quotecheck.mjs'' now fall back to the PDF, which is where this page's 118 //rescued// come from.
   * **Whether the 68% run-to-run agreement on ''statistics.kind'' still holds.** It was measured on the **previous** corpus run and has not been re-measured; the page says so where it uses it.
   * **How much the 138 posters inflate the "no test named" rate of 5.7%.** Not measured. The figure was not re-run excluding short records.
@@ Line 364: / Line 380: @@
   * **The real intra-class correlation of web-measurement outcomes.** The simulation shows everything depends on it and no paper reports one. This is the open question the page leads with, and it is genuinely open.
   * **Whether the two papers whose stated signed-rank/Mann-Whitney rule is keyed on sample size or variable type actually analysed their data correctly.** Their designs make the stated rule and the correct rule coincide in the common case, and the page says the results are not in question. Establishing it properly would need their data.
-</wrap>
+</WRAP>
 ===== Judgement calls =====
@@ Line 396: / Line 412: @@
 | The most useful single guard | ''check_page_numbers.mjs''. Editing ''test_fold.mjs'' after the page was written moved **17 published figures**; reading the page did not catch one of them and the guard caught all of them. Run it after every fold edit |
 | Accidental exposure | none. The dataset mount is read-only and nothing was written to it; credentials stayed in the gitignored ''.env'' |
+==== Correction pass, 2026-09-04 ====
+^ What ^ Disposition ^
+| The //"quotes are in the PDF and in none of the stored renderings"// section | **RETRACTED.** Both quoted sentences are in ''paper.cols.txt''; ''Friedman'', ''Conover'', ''omnibus'', ''172.47'' and all ten authors are in all three renderings. The section is kept, marked retracted, with the shell transcript that explains the error |
+| The claim that ''paper.cols.txt'' drops five of the ten authors | **RETRACTED.** All ten are in its header |
+| The claim that the dataset's unlocatable rate //"is partly an artefact of text extraction"// | **UPHELD, and now measured** — but on other papers, not this one. 73.2% of the corpus's unlocatable quotes are present in an independent rendering, under the four-verdict rule described on [[provenance:literature:corpus]] and **not** under this page's threshold |
+| %%<wrap todo>%% in //What could not be established// | **FIXED to %%<WRAP todo>%%.** The lowercase form renders a ''<span>'', which cannot hold a list: the rendered DOM had all eight bullets as one paragraph of literal asterisks. **Four** other pages still have it and are logged as a follow-up — ''provenance:practices:ethics'', ''provenance:practices:notifying_websites'', ''provenance:practices:public_relations'' and ''provenance:statistics:pvalue_corrections''. ((An earlier draft said //eight//, from ''grep -l "<wrap "'' over a site export; that also matches pages which merely mention the tag inside ''%%…%%''. ''check_wrap.mjs'' gives five, including this page.)) |
+| The quote-check block | **REFRESHED.** ''exact'' 931 unchanged; ''partial'' 540 → **541** and ''below threshold'' 335 → 334 because ''ht_quotecheck.mjs'' was reading the wrong file for one OCR-repaired paper; the 334 then splits into 118 //rescued from the PDF// and **216** //below threshold in both// |
+| ''scipy'', ''numpy'' and ''statsmodels'' | **NO LONGER INSTALLED** in this container (checked 2026-09-04; ''pip list'' shows only ''mercurial, pip, pypdf, setuptools, wheel''). The versions recorded below were true when this log was written; they are session-installed, not persistent, and **the published ''paired_and_clustered.py'' demo cannot currently be re-run here.** Flagged by the external-currency reviewer |
+| "SciPy **1.18.0** current on PyPI" and "statsmodels **0.14.6** current" | **STALE.** PyPI now reports SciPy **1.18.1** and statsmodels **0.15.0** (2026-09-04). Both were dated claims and neither is load-bearing; recorded rather than silently rewritten |
+| "The ''statistics:'' namespace has three other red links" and the five pages named in //Judgement calls// as not existing yet | **STALE.** ''artifacts'', ''design:automated_measurements'', ''design:user_studies'', ''statistics:biases'' and ''statistics:regression'' all exist now (created 2026-08-21 to 2026-08-28). The links resolve; only the prose describing them as red is wrong |
+| Everything else on this page | **Not re-derived.** The corpus has not changed since 2026-08-13, so the hypothesis-test figures stand as published; only the claims this audit contradicts, and the currency claims above, were touched. The figures reviewer re-ran ''report_hypothesis_testing.mjs'' and confirmed the untouched figures reproduce |
+The audit itself, its four scripts, their unedited output and its own limits are on [[provenance:literature:corpus]].
 ===== Related =====
+  * [[provenance:literature:corpus]] — the corpus-wide ''pypdf''-vs-''cols'' audit that retracted the section above.
   * [[Statistics:Hypothesis testing]] — the page these notes are for.
   * [[Literature:Corpus]] — corpus-wide provenance: venues, funnel, provisional years, posters, duplicates.
@@ Line 406: / Line 438: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.