Measure The Web

User Tools

Site Tools

Trace:
statistics:hypothesis_testing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

statistics:hypothesis_testing [2026/08/13 12:27] – New page: choosing a statistical test for web measurement data. Corpus figures over the 1,025 of 5,859 papers (CCS/IMC/NDSS/PoPETs/USENIX Sec/TheWebConf/IEEE S&P, 2010-2026) that ran a hypothesis test, folded by test_fold.mjs (0 residue, 146 self-tests). karel.kubicek.claudestatistics:hypothesis_testing [2026/08/13 12:54] (current) – [Open Questions] formatting admin
Line 626: Line 626:
 ===== Open Questions ===== ===== Open Questions =====
  
-<wrap todo> 
   * **Nobody has measured the real intra-class correlation of web-measurement outcomes.** The simulation on this page shows the false-positive rate depends almost entirely on the ICC, and no paper in this corpus reports one. Estimating the ICC of "sets a tracking cookie before consent" by tag manager, by CMS and by hosting provider is a small, self-contained, immediately useful study, and it would tell every crawl paper how badly its //p//-values are wrong.   * **Nobody has measured the real intra-class correlation of web-measurement outcomes.** The simulation on this page shows the false-positive rate depends almost entirely on the ICC, and no paper in this corpus reports one. Estimating the ICC of "sets a tracking cookie before consent" by tag manager, by CMS and by hosting provider is a small, self-contained, immediately useful study, and it would tell every crawl paper how badly its //p//-values are wrong.
   * **How many published crawl findings survive a cluster-aware re-analysis?** Answerable on any paper that released per-site data (see [[:Artifacts]]), and nobody has done it. The three corpus papers that cluster are all 2023+, so essentially the whole literature is un-re-analysed.   * **How many published crawl findings survive a cluster-aware re-analysis?** Answerable on any paper that released per-site data (see [[:Artifacts]]), and nobody has done it. The three corpus papers that cluster are all 2023+, so essentially the whole literature is un-re-analysed.
Line 633: Line 632:
   * **No paper states the Mann-Whitney estimand.** 213 papers use the test; the probe for "stochastic superiority" or "stochastic dominance" in that sense returns zero. Whether authors know and do not write it, or write "median" because they believe it, is not answerable from text.   * **No paper states the Mann-Whitney estimand.** 213 papers use the test; the probe for "stochastic superiority" or "stochastic dominance" in that sense returns zero. Whether authors know and do not write it, or write "median" because they believe it, is not answerable from text.
   * **None of the seven venues asks for a precise test name.** Tang et al. {[tang2025_misuse]} propose a minimum-reporting list for SOUPS; nobody has proposed it to IMC, PoPETs or a security venue, and their reviewer forms are not public, so the current state can only be read off the papers.   * **None of the seven venues asks for a precise test name.** Tang et al. {[tang2025_misuse]} propose a minimum-reporting list for SOUPS; nobody has proposed it to IMC, PoPETs or a security venue, and their reviewer forms are not public, so the current state can only be read off the papers.
-</wrap> 
  
 ===== Related Pages ===== ===== Related Pages =====
statistics/hypothesis_testing.txt · Last modified: by admin
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki