| Both sides previous revisionPrevious revision | |
| design:algorithm_audits [2026/09/11 11:02] – Re-base the crawl-config reporting rows on a shared denominator (interaction depth flips from 65.6/75.1 to 87.5/77.9); name the 8 audits that never drive a browser. Authored by Claude karel.kubicek.claude | design:algorithm_audits [2026/09/11 11:21] (current) – Generic review: population 32->34 (Cart-ology, IMC 2024 poster); the null is now hand-adjudicated per paper (10 of 34) instead of read off a term count; tighten rule (O) to exclude reachability; soften the growth claim to papers-per-year. Authored by Clau karel.kubicek.claude |
|---|
| //"Even queries launched simultaneously from two identically configured clients on the same subnet can produce wildly different ads over multiple timescales."// | //"Even queries launched simultaneously from two identically configured clients on the same subnet can produce wildly different ads over multiple timescales."// |
| |
| If you have not measured how far apart two //identical// arms land, you cannot say what a gap between two //different// arms means. That measurement — an **A/A test**, a control–control run, a null experiment, whatever you call it — is the single most-skipped step in this literature. The exact phrase "A/A test" appears in **2 of 5,859** papers in this corpus, and in **none** of the 32 audit papers; under **eight** different names, only **11 of 32** establish a same-treatment baseline at all. | If you have not measured how far apart two //identical// arms land, you cannot say what a gap between two //different// arms means. That measurement — an **A/A test**, a control–control run, a null experiment, whatever you call it — is the most-skipped step in this literature. Reading all 34 audit papers in this corpus for it: **10 establish a null and 24 do not.** |
| </WRAP> | </WRAP> |
| |
| - **A permutation test over the arm labels.** {[datta2015_automated]} does not assume a noise model: it trains a classifier to tell the two arms apart and asks how often a classifier does that well when the arm labels are shuffled. The null is generated from your own data. {[lecuyer2015_sunlight]} and {[breuer2026_ad]} use the same machinery. | - **A permutation test over the arm labels.** {[datta2015_automated]} does not assume a noise model: it trains a classifier to tell the two arms apart and asks how often a classifier does that well when the arm labels are shuffled. The null is generated from your own data. {[lecuyer2015_sunlight]} and {[breuer2026_ad]} use the same machinery. |
| |
| The industry name for step 1 is an **A/A test**, and in this corpus that phrase belongs to the //platform// side, not the audit side: both occurrences are TheWebConf papers about running experiments on a platform you own — //"We simulate A/A tests by re-randomizing the treatment assignments on the observed exposure logs from our experiment corpus"// {[chandar2022_using]}. Nobody auditing from outside uses the term. That is a vocabulary gap, not a knowledge gap — but the practice is missing too, in about two thirds of the audits here. | The industry name for route 1 is an **A/A test**, and in this corpus that phrase belongs to the //platform// side, not the audit side: both occurrences are TheWebConf papers about running experiments on a platform you own — //"We simulate A/A tests by re-randomizing the treatment assignments on the observed exposure logs from our experiment corpus"// {[chandar2022_using]}. Nobody auditing from outside uses the term. **That vocabulary gap is not the same as the practice gap**, and the two have to be counted separately: see [[#The null, and how it was counted]]. |
| |
| <WRAP tip> | <WRAP tip> |
| ===== How many arms, how many repeats ===== | ===== How many arms, how many repeats ===== |
| |
| The unit of analysis is **not** the individual result, ad impression or video. It is the **arm-run**: one identity, trained once, measured once. Ten thousand impressions collected from two profiles give you //two// independent observations of the thing you are claiming, not twenty thousand. This is the same unit-of-analysis error that [[Statistics:Hypothesis testing#The Unit of Analysis Is the Error That Actually Invalidates Results]] measures across the corpus, and audits are its most natural habitat, because the impression counts are so large that a //p//-value computed over them is guaranteed to be tiny. | The unit of analysis is **not** the individual result, ad impression or video. It is the **arm-run**: one identity, trained once, measured once. Ten thousand impressions collected from two profiles give you //two// independent observations of the thing you are claiming, not twenty thousand. This is the same unit-of-analysis error that [[Statistics:Hypothesis testing#The Unit of Analysis Is the Error That Actually Invalidates Results]] measures across the corpus. An audit is an easy place to make it, because the impression counts are so large that a //p//-value computed over them is guaranteed to be tiny — that page measures how often the field makes it; this one only says where the temptation is. |
| |
| Two consequences: | Two consequences: |
| |
| * **Replicate the identity, not the query.** Twenty profiles per condition measured ten times beats two profiles measured a hundred times, for the same total cost. | * **Replicate the identity, not the query.** Twenty profiles per condition measured ten times beats two profiles measured a hundred times: the same 200 observations, but //n// = 20 instead of //n// = 2 in the test. It is not free — you pay for twenty training runs instead of two, and training is the expensive part — which is the real reason the corpus is full of two-profile designs. Budget for it at design time, not after. |
| * **Multiple comparisons arrive by default.** One audit with 16 interest personas against a control is 16 tests, not one. {[liu2024_opted]} is explicit — //"we also conduct Bonferroni correction on the statistical test"// — and a multiple-comparison correction is named by **7 of the 32** — Holm–Bonferroni in 5 of them, plain Bonferroni in 3. See [[Statistics:Pvalue corrections]]. | * **Multiple comparisons arrive by default.** One audit with 16 interest personas against a control is 16 tests, not one. {[liu2024_opted]} is explicit — //"we also conduct Bonferroni correction on the statistical test"// — and a multiple-comparison correction is named by **7 of the 34** — Holm–Bonferroni in 5 of them, plain Bonferroni in 3. See [[Statistics:Pvalue corrections]]. |
| |
| ===== Which methods are current ===== | ===== Which methods are current ===== |
| |
| ^ Method ^ Status ^ Evidence ^ | ^ Method ^ Status ^ Evidence ^ |
| | **Trained browser-profile personas** driven at a live ad or search system | **current, and the workhorse** | 14 of the 32 audit papers use the word "persona"; used continuously 2010 → 2026 {[bashir2016tracing]}, {[musa2022_atom]}, {[liu2024_opted]}, {[breuer2026_ad]} | | | **Trained browser-profile personas** driven at a live ad or search system | **current, and the workhorse** | 14 of the 34 audit papers use the word "persona", in every year from 2020 on and first in 2016 — {[guha2010_challenges]} does the same thing in 2010 and calls them //personae// {[bashir2016tracing]}, {[musa2022_atom]}, {[liu2024_opted]}, {[breuer2026_ad]} | |
| | **Sock-puppet accounts** (a logged-in identity you created) | **current, and unavoidable for feeds and recommenders** | 5 of the 32; 26 papers corpus-wide use the term. All since 2019 {[le2019_political]}, {[zhang2023_utility]}, {[mai2025_more]}, {[karnam2026_setting]} | | | **Sock-puppet accounts** (a logged-in identity you created) | **current, and unavoidable for feeds and recommenders** | 5 of the 34; 26 papers corpus-wide use the term. All since 2019 {[le2019_political]}, {[zhang2023_utility]}, {[mai2025_more]}, {[karnam2026_setting]} | |
| | **Paired-campaign advertiser-side audits** (you buy the ads) | **current** | {[kaplan2022_implied]} ran //"200 versions of this ad at the same time, all from the same account and with the same budget"//; {[imana2021_auditing]} is the canonical design and is outside this corpus | | | **Paired-campaign advertiser-side audits** (you buy the ads) | **current** | {[kaplan2022_implied]} ran //"200 versions of this ad at the same time, all from the same account and with the same budget"//; {[imana2021_auditing]} is the canonical design and is outside this corpus | |
| | **Randomised field experiment with real users** (browser extension, random assignment) | **current, growing, and the strongest design available** | {[mccrosky2026_does]}, 22,722 participants, //"depending on randomized assignment"// | | | **Randomised field experiment with real users** (browser extension, random assignment) | **current, growing, and the strongest design available** | {[mccrosky2026_does]}, 22,722 participants, //"depending on randomized assignment"// | |
| | **Data donation instead of arms** | **current, and the answer to a different question** | {[silva2020_facebook]}, {[ali2023_problematic]}, {[gkiouzepi2023_collaborative]} — the last one names the trade-off: //"One method that does not use fake personas"// is to model over donated exposures. You get ecological validity and lose the treatment. | | | **Data donation instead of arms** | **current, and the answer to a different question** | {[silva2020_facebook]}, {[ali2023_problematic]}, {[gkiouzepi2023_collaborative]} — the last one names the trade-off: //"One method that does not use fake personas"// is to model over donated exposures. You get ecological validity and lose the treatment. | |
| | **Right-of-access requests as the measuring instrument** | **new (2026), and complementary** | {[karnam2026_setting]} drives sock puppets to generate ground truth, then compares the platform's own GDPR Art. 15 export against it | | | **Right-of-access requests as the measuring instrument** | **new (2026), and complementary** | {[karnam2026_setting]} drives sock puppets to generate ground truth, then compares the platform's own GDPR Art. 15 export against it | |
| | **LLM-driven agents as the audit instrument** | **new (2026), evaluated rather than assumed** | {[sun2026_suitability]}: //"We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows"// — and the contribution is characterising where it fails. See [[Programming:Crawler:LLM agents]] | | | **LLM-driven agents as the audit instrument** | **new (2026), and outside this page's population** | {[sun2026_suitability]} audits dark patterns in rights-request workflows — not a differential audit, so it is not one of the 34 — but it is the first paper here to ask whether an agent can //be// the instrument, and its answer is a failure analysis. [[Programming:Crawler:LLM agents]] | |
| | **Price discrimination / price steering as the outcome** | **rare in these seven venues, not dormant** | 9 of the 32 mention the terms; only **2** measure a price — {[chen2015_peeking]} (Uber surge pricing, IMC 2015) and {[becerrilarreola2023_method]} (retail prices, recommendations and delivery fees matched to census demographics, TheWebConf 2023). The founding papers {[hannak2014_price]} and Vissers et al. (PETS 2014) are **outside this corpus**, so the scarcity here is partly the screen and not the field | | | **Price discrimination / price steering as the outcome** | **rare in these seven venues, not dormant** | 9 of the 34 mention the terms; only **2** measure a price — {[chen2015_peeking]} (Uber surge pricing, IMC 2015) and {[becerrilarreola2023_method]} (retail prices, recommendations and delivery fees matched to census demographics, TheWebConf 2023). The founding papers {[hannak2014_price]} and Vissers et al. (PETS 2014) are **outside this corpus**, so the scarcity here is partly the screen and not the field | |
| | **Single blank profile vs single aged profile, one run each, no null** | **superseded** | the design {[guha2010_challenges]} argued against in 2010; it survives in this corpus only as a component of attack papers {[meng2014_pollution]}, {[kim2018_adbudgetkiller]} | | | **Single blank profile vs single aged profile, one run each, no null** | **superseded** | the design {[guha2010_challenges]} argued against in 2010; it survives in this corpus only as a component of attack papers {[meng2014_pollution]}, {[kim2018_adbudgetkiller]} | |
| |
| **What is not superseded, despite looking old.** The 2015 instrument papers are still the state of the art for the statistics. {[datta2015_automated]}'s permutation-test-over-arm-labels design has not been replaced by anything in this corpus; the 2025–2026 audits that do build a null ({[le2025_voice]}, {[breuer2026_ad]}) use the same idea. If a reviewer asks how you established significance, AdFisher is still the citation. | **What is not superseded, despite looking old.** The 2015 instrument papers are still the state of the art for the statistics. {[datta2015_automated]}'s permutation-test-over-arm-labels design has not been replaced by anything in this corpus; the 2025–2026 audits that do build a null ({[le2025_voice]}, {[breuer2026_ad]}) use the same idea. If a reviewer asks how you established significance, AdFisher is still the citation. |
| |
| **What is genuinely newer.** Three things arrived after 2023 and none of them existed when the method was defined: **randomised assignment over real users' own sessions** {[mccrosky2026_does]}, **the platform's own data export as ground truth** {[karnam2026_setting]}, and **agentic LLM crawlers** as the thing doing the auditing {[sun2026_suitability]}. The first is the strongest design on this page and the one most constrained by ethics review and recruitment; the third is the least validated. | **What is genuinely newer.** Two things arrived after 2023 and neither existed when the method was defined: **randomised assignment over real users' own sessions** {[mccrosky2026_does]}, and **the platform's own data export as ground truth** {[karnam2026_setting]}. The first is the strongest design on this page and the one most constrained by ethics review and recruitment. A third, agentic LLM crawlers as the thing doing the auditing, has exactly one paper and it is a failure analysis {[sun2026_suitability]}. |
| |
| ===== What this corpus cannot tell you ===== | ===== What this corpus cannot tell you ===== |
| and among the 3 venue-year gaps, Vissers et al., PETS 2014, //Crying Wolf? On the Price Discrimination of Online Airline Tickets//, and Khattak et al., NDSS 2016, //Do You See What I See? Differential Treatment of Anonymous Users// {[khattak2016_differential]}. | and among the 3 venue-year gaps, Vissers et al., PETS 2014, //Crying Wolf? On the Price Discrimination of Online Airline Tickets//, and Khattak et al., NDSS 2016, //Do You See What I See? Differential Treatment of Anonymous Users// {[khattak2016_differential]}. |
| |
| **So every count on this page is a lower bound, and it is a biased one** — it under-counts precisely the audits whose framing is fairness or economics rather than privacy. Treat the 32 as a reading list and a shape, not as a census. The corresponding claim about the wider literature — how many algorithm audits exist — this corpus cannot make at all. | **So every count on this page is a lower bound, and it is a biased one** — it under-counts precisely the audits whose framing is fairness or economics rather than privacy. Treat the 34 as a reading list and a shape, not as a census. The corresponding claim about the wider literature — how many algorithm audits exist — this corpus cannot make at all. |
| </WRAP> | </WRAP> |
| |
| Two more things it cannot tell you: | Two more things it cannot tell you: |
| |
| * **Whether an audit's arms were actually independent.** ''crawlConfig'' carries one evidence quote for the whole configuration object, so the extraction cannot distinguish "two profiles" from "one profile reset twice". Every verdict behind the 32-paper set was made by reading the paper; the sentences are on [[provenance:design:algorithm_audits]]. | * **Whether an audit's arms were actually independent.** ''crawlConfig'' carries one evidence quote for the whole configuration object, so the extraction cannot distinguish "two profiles" from "one profile reset twice". Every verdict behind the 34-paper set was made by reading the paper; the sentences are on [[provenance:design:algorithm_audits]]. |
| * **Effect sizes across audits.** Each paper's effect is measured on its own platform, outcome and metric. There is no comparable quantity to pool, and this page deliberately publishes none. | * **Effect sizes across audits.** Each paper's effect is measured on its own platform, outcome and metric. There is no comparable quantity to pool, and this page deliberately publishes none. |
| |
| - **(C)** the result is a difference (or a bounded absence of difference) **between arms**, not a prevalence over a crawl of many sites. | - **(C)** the result is a difference (or a bounded absence of difference) **between arms**, not a prevalence over a crawl of many sites. |
| |
| **32** papers of 5,859 qualify. **12** further candidates were read and rejected; the commonest reason is (C) — an "audit" that is observational. | **34** papers of 5,859 qualify. **36** further candidates were read and rejected; the commonest reason is (C) — an "audit" that is observational. Clause (O) excludes studies whose outcome is **reachability** rather than a discriminating response: geoblocking, censorship and Tor-exit refusal vary a vantage point rather than an identity, and belong to [[Design:Blocking and geodifference]]. |
| </WRAP> | </WRAP> |
| |
| ==== The shape of the literature ==== | ==== The shape of the literature ==== |
| |
| ^ Window ^ Audit papers ^ Share of the 32 ^ | ^ Window ^ Audit papers ^ Share of the 34 ^ Papers per year ^ |
| | 2010–2015 | 6 | 18.8% | | | 2010–2015 (6 yr) | 6 | 17.6% | 1.00 | |
| | 2016–2019 | 5 | 15.6% | | | 2016–2019 (4 yr) | 5 | 14.7% | 1.25 | |
| | 2020–2023 | 11 | 34.4% | | | 2020–2023 (4 yr) | 12 | 35.3% | 3.00 | |
| | 2024–2026* | 10 | 31.3% | | | 2024–2026* (3 yr) | 11 | 32.4% | 3.67 | |
| |
| Per year: 2010 **1**, 2014 **1**, 2015 **4**, 2016 **1**, 2018 **2**, 2019 **2**, 2020 **2**, 2022 **5**, 2023 **4**, 2024 **2**, 2025 **5**, 2026 **3**. **2011, 2012, 2013, 2017 and 2021 have none.** The 2025–2026 cells are provisional venue-years (CCS and IMC 2026 have not been held; see [[literature:corpus]]) and are nevertheless the joint-largest two-year block in the series. **This method is not historical. It is growing.** | Per year: 2010 **1**, 2014 **1**, 2015 **4**, 2016 **1**, 2018 **2**, 2019 **2**, 2020 **2**, 2022 **6**, 2023 **4**, 2024 **3**, 2025 **5**, 2026 **3**. **2011, 2012, 2013, 2017 and 2021 have none.** |
| | |
| | The windows are unequal (6, 4, 4, 3 years), so the comparable figure is papers per year: **1.00**, **1.25**, **3.00**, **3.67**. Read that carefully. The second window is barely above the first; the rise is between 2016–2019 and 2020–2023 and it holds after. The last column is provisional (CCS and IMC 2026 have not been held; see [[literature:corpus]]), and //n// = 34 from a screen this page calls biased will not carry a trend claim. What the table supports is narrow and still useful: **more of this work appeared in the last six years than in the first ten, and none of it has stopped.** If you were told this method belongs to 2015, that is wrong. |
| |
| ==== Where audits publish ==== | ==== Where audits publish ==== |
| ^ Venue ^ Venue papers ^ Audits ^ Share of venue ^ | ^ Venue ^ Venue papers ^ Audits ^ Share of venue ^ |
| | PETS | 510 | **10** | **2.0%** | | | PETS | 510 | **10** | **2.0%** | |
| | IMC | 638 | 7 | 1.1% | | | IMC | 638 | 8 | 1.3% | |
| | TheWebConf | 843 | 8 | 0.9% | | | TheWebConf | 843 | 8 | 0.9% | |
| | CCS | 990 | 3 | 0.3% | | | CCS | 990 | 4 | 0.4% | |
| | NDSS | 701 | 1 | 0.1% | | | NDSS | 701 | 1 | 0.1% | |
| | IEEE S&P | 767 | 1 | 0.1% | | | IEEE S&P | 767 | 1 | 0.1% | |
| PoPETs is where this work lands, and USENIX Security — the largest venue in the corpus — has published two audits by this definition in seventeen years. Read that with [[#What this corpus cannot tell you]]: TheWebConf's row is the one most depressed by the screen, because TheWebConf is where the fairness-framed audits are. | PoPETs is where this work lands, and USENIX Security — the largest venue in the corpus — has published two audits by this definition in seventeen years. Read that with [[#What this corpus cannot tell you]]: TheWebConf's row is the one most depressed by the screen, because TheWebConf is where the fairness-framed audits are. |
| |
| Platform measured (multi-valued): web **25** of 32 (78.1%), other-online-service **18** (56.3%), mobile **4** (12.5%), IoT **3** (9.4%), offline **0**. | Platform measured (multi-valued): web **27** of 34 (79.4%), other-online-service **19** (55.9%), mobile **4** (11.8%), IoT **3** (8.8%), offline **0**. |
| |
| ==== Audits report more than the corpus does — except where it counts ==== | ==== Audits report more than the corpus does — except where it counts ==== |
| |
| ^ Indicator ^ Population ^ Audit set ^ Comparable baseline ^ | ^ Indicator ^ Population ^ Audit set ^ Comparable baseline ^ |
| | runs a non-descriptive statistic | all audits vs ''empirical'' | **21 of 32 (65.6%)** | 1,637 of 5,118 (32.0%) | | | runs a non-descriptive statistic | all audits vs ''empirical'' | **21 of 34 (61.8%)** | 1,637 of 5,118 (32.0%) | |
| | states an ethics-review outcome | all audits vs ''empirical'' | **20 of 32 (62.5%)** | 1,728 of 5,118 (33.8%) | | | states an ethics-review outcome | all audits vs ''empirical'' | **20 of 34 (58.8%)** | 1,728 of 5,118 (33.8%) | |
| | states artifact availability | all audits vs ''empirical'' | 22 of 32 (68.8%) | 2,890 of 5,118 (56.5%) | | | states artifact availability | all audits vs ''empirical'' | 22 of 34 (64.7%) | 2,890 of 5,118 (56.5%) | |
| | states a vantage location | all audits vs ''measuredFrom'' | 14 of 32 (43.8%) | 1,228 of 3,908 (31.4%) | | | states a vantage location | all audits vs ''measuredFrom'' | 15 of 34 (44.1%) | 1,228 of 3,908 (31.4%) | |
| | states crawl statefulness | **papers with a crawl config** | **22 of 24 (91.7%)** | 219 of 1,080 (20.3%) | | | states crawl statefulness | **papers with a crawl config** | **24 of 26 (92.3%)** | 219 of 1,080 (20.3%) | |
| | states interaction depth | **papers with a crawl config** | 21 of 24 (87.5%) | 841 of 1,080 (77.9%) | | | states interaction depth | **papers with a crawl config** | 22 of 26 (84.6%) | 841 of 1,080 (77.9%) | |
| | states a consent action | **papers with a crawl config** | 11 of 24 (45.8%) | 349 of 1,080 (32.3%) | | | states a consent action | **papers with a crawl config** | 12 of 26 (46.2%) | 349 of 1,080 (32.3%) | |
| | states whether it ran headless | **papers with a crawl config** | 5 of 24 (20.8%) | 140 of 1,080 (13.0%) | | | states whether it ran headless | **papers with a crawl config** | 5 of 26 (19.2%) | 140 of 1,080 (13.0%) | |
| |
| The bottom four rows are counted over papers that **have** a crawl configuration on both sides, because a field that can only be stated on such a paper cannot fairly be divided by a population that includes papers without one. **Eight of the 32 audits have no crawl configuration**, and they are not oversights — they are audits that never drive a browser: an app ({[chen2015_peeking]}), ad campaigns bought on the platform ({[venkatadri2019_investigating]}, {[kaplan2022_implied]}), a field study in real participants' own browsers ({[zeng2022_factors]}), voice assistants ({[le2025_voice]}, {[khezresmaeilzadeh2025_echoes]}), app stores ({[breuer2026_ad]}), and a browser-extension experiment in participants' own sessions ({[mccrosky2026_does]}). | The bottom four rows are counted over papers that **have** a crawl configuration on both sides, because a field that can only be stated on such a paper cannot fairly be divided by a population that includes papers without one. **Eight of the 34 audits have no crawl configuration**, and they are not oversights — they are audits the authors did not drive a browser //for//: an app ({[chen2015_peeking]}), ad campaigns bought on the platform ({[venkatadri2019_investigating]}, {[kaplan2022_implied]}), voice assistants ({[le2025_voice]}, {[khezresmaeilzadeh2025_echoes]}), app stores ({[breuer2026_ad]}), and two studies that run in real participants' own browsers rather than a crawler ({[zeng2022_factors]}, {[mccrosky2026_does]}). |
| |
| On the fair denominator the statefulness row is the striking one: **audits are the sub-population that actually reports profile state**, 91.7% against 20.3%, because in an audit the profile //is// the result and you cannot describe the result without it. Audits are above the crawling baseline on every one of these four — which is what you would hope, and is not true of the vantage-location row, where 43.8% is better than the 31.4% baseline but still means **more than half of these audits do not say where they measured from**, on a design where the vantage point must be identical across arms or it is part of the treatment. | On the fair denominator the statefulness row is the striking one: **audits are the sub-population that actually reports profile state**, 92.3% against 20.3%, because in an audit the profile //is// the result and you cannot describe the result without it. Audits are above the crawling baseline on every one of these four — which is what you would hope, and is not true of the vantage-location row, where 44.1% is better than the 31.4% baseline but still means **more than half of these audits do not say where they measured from**, on a design where the vantage point must be identical across arms or it is part of the treatment. |
| |
| ==== The null is missing from two thirds of them ==== | ==== The null, and how it was counted ==== |
| |
| A full-text sweep for **eight** ways of naming a same-treatment baseline, over the 5,855 papers with text: | This is the page's central claim, so it is worth being exact about what supports it. **Two different things are counted here and they are not the same number.** |
| | |
| | **The vocabulary.** A full-text sweep for eight ways of naming a same-treatment baseline, over the 5,855 papers with text: |
| |
| ^ Term ^ Papers, corpus-wide ^ In the audit set ^ | ^ Term ^ Papers, corpus-wide ^ In the audit set ^ |
| | ''A/A test'' | 2 | **0** of 32 | | | ''A/A test'' | 2 | **0** of 34 | |
| | ''control-control'' / ''null experiment'' | 31 | 1 | | | ''control-control'' / ''null experiment'' | 31 | 1 | |
| | ''noise floor'' | 35 | 2 | | | ''noise floor'' | 35 | 2 | |
| | baseline / measurement / inherent / background ''noise'' | 219 | 2 | | | baseline / measurement / inherent / background ''noise'' | 219 | 2 | |
| | ''same treatment'' / ''no-treatment'' arm | 13 | 1 | | | ''same treatment'' / ''no-treatment'' arm | 13 | 1 | |
| | **at least one of the eight** | — | **11 of 32 (34.4%)** | | |
| |
| The eleven are {[guha2010_challenges]}, {[lecuyer2015_sunlight]}, {[klimansilver2015_location]}, {[datta2015_automated]}, {[robertson2018_auditing]}, {[le2019_political]}, {[iqbal2022_left]}, {[vombatkere2024_tiktok]}, {[mai2025_more]}, {[le2025_voice]} and {[breuer2026_ad]}. A term probe is a lower bound — a paper can build a null without using any of these eight phrasings, and this is a **regex over prose**, not a reading of every methods section. But the direction is not in doubt: the two papers most cited for //how to do this// ({[guha2010_challenges]}, {[datta2015_automated]}) are both in the eleven, and the majority of the set states a difference without stating what a non-difference looks like. | Both ''A/A test'' occurrences are platform-side experimentation papers, not audits. **The audit side of this literature has no word for the thing.** |
| | |
| | **The practice.** A term count cannot answer whether a paper //did// it, and this one is wrong in both directions. It let in {[robertson2018_auditing]}, whose only ''noise floor'' hit describes //Hannak et al.'s// result and not its own design, and {[iqbal2022_left]}, whose ''same treatment'' is the phrasing of its research question. It missed {[oh2022_cartology]}, which runs four unused baseline profiles //"created and mechanistically measured in the same way"// and matches none of the eight phrasings. So all 34 were read for the question instead, and each verdict carries the sentence that settled it: |
| | |
| | ^ Route ^ Papers ^ What it means ^ |
| | | **A/A** | 4 | two or more arms that differ in nothing | |
| | | **generated** | 3 | a null built from the paper's own observations — permutation over arm labels, or a randomised baseline | |
| | | **repeats** | 1 | the same condition run several times, with the spread reported | |
| | | **A/A + generated** | 2 | both | |
| | | **any null** | **10 of 34 (29.4%)** | | |
| | | **none** | **24 of 34 (70.6%)** | states a difference without measuring what a non-difference looks like | |
| | |
| | The ten: {[guha2010_challenges]} and {[klimansilver2015_location]} and {[le2019_political]} and {[oh2022_cartology]} (A/A), {[lecuyer2015_sunlight]} and {[datta2015_automated]} and {[vombatkere2024_tiktok]} (generated), {[mai2025_more]} (repeats), {[le2025_voice]} and {[breuer2026_ad]} (both). |
| | |
| | Two things this does **not** say. It is a judgement per paper, not a mechanical count, so a second reader would move one or two of them — the definition and every evidence sentence are on [[provenance:design:algorithm_audits]] so you can disagree with a specific verdict rather than with the total. And a paper with no null is not thereby wrong: several of the 24 report effects far larger than any plausible noise floor. What they cannot do is tell the reader that. |
| |
| ==== And a third of them run no inferential test ==== | ==== And a third of them run no inferential test ==== |
| |
| **11 of the 32** report no statistic beyond descriptives, by the extraction's ''statistics.kind'' field: {[guha2010_challenges]}, {[meng2014_pollution]}, {[klimansilver2015_location]}, {[bashir2016tracing]}, {[kim2018_adbudgetkiller]}, {[venkatadri2019_investigating]}, {[zhang2022_harpo]}, {[medjkoune2023_marketing]}, {[zhang2023_utility]}, {[khezresmaeilzadeh2025_echoes]} and {[karnam2026_setting]}. Several of those are attack or system papers where the audit is the evaluation rather than the claim, and {[klimansilver2015_location]} builds a noise floor without a named test — so this is a count of //reported// tests, not a verdict on any paper. The methods that are named, paper-counted over the set: a Bonferroni-family correction **7** (Holm–Bonferroni **5**, plain Bonferroni **3**; one paper names both), linear regression **2**, Mann–Whitney U **2**, then a long tail of one each (blocked permutation test, Benjamini–Yekutieli, Clopper–Pearson interval, exact test on Pearson's correlation, cross-correlation). The **7** is a hand-folded family count printed by the report script; the automatic skeleton fold leaves //Holm-Bonferroni correction// (3), //Holm-Bonferroni// (1) and //Holm-Bonferroni method// (1) as three separate rows, which is why an unfolded reading of this field would have said 3. Free-text method names are ~20% stable run-to-run, so read the tail as a ranking. See [[Statistics:Hypothesis testing]]. | **11 of the 34** report no statistic beyond descriptives, by the extraction's ''statistics.kind'' field: {[guha2010_challenges]}, {[meng2014_pollution]}, {[klimansilver2015_location]}, {[bashir2016tracing]}, {[kim2018_adbudgetkiller]}, {[venkatadri2019_investigating]}, {[zhang2022_harpo]}, {[medjkoune2023_marketing]}, {[zhang2023_utility]}, {[khezresmaeilzadeh2025_echoes]} and {[karnam2026_setting]}. Several of those are attack or system papers where the audit is the evaluation rather than the claim, and {[klimansilver2015_location]} builds a noise floor without a named test — so this is a count of //reported// tests, not a verdict on any paper. The methods that are named, paper-counted over the set: a Bonferroni-family correction **7** (Holm–Bonferroni **5**, plain Bonferroni **3**; one paper names both), linear regression **2**, Mann–Whitney U **2**, then a long tail of one each (blocked permutation test, Benjamini–Yekutieli, Clopper–Pearson interval, exact test on Pearson's correlation, cross-correlation). The **7** is a hand-folded family count printed by the report script; the automatic skeleton fold leaves //Holm-Bonferroni correction// (3), //Holm-Bonferroni// (1) and //Holm-Bonferroni method// (1) as three separate rows, which is why an unfolded reading of this field would have said 3. Free-text method names are ~20% stable run-to-run, so read the tail as a ranking. See [[Statistics:Hypothesis testing]]. |
| |
| ==== The vocabulary, and why it matters for your related-work search ==== | ==== The vocabulary, and why it matters for your related-work search ==== |
| |
| ^ Term in full text ^ Papers, corpus-wide ^ In the audit set ^ | ^ Term in full text ^ Papers, corpus-wide ^ In the audit set ^ |
| | "persona" (any use) | 190 | 14 | | | "persona" (any use) | 190 | 14 of 34 | |
| | "sock puppet" / "sockpuppet" | 26 | 5 | | | "sock puppet" / "sockpuppet" | 26 | 5 of 34 | |
| | "control profile/account/persona/browser" | 33 | 11 | | | "control profile/account/persona/browser" | 33 | 11 of 34 | |
| | "treatment group/profile/persona/condition/arm" | 66 | 5 | | | "treatment group/profile/persona/condition/arm" | 66 | 5 of 34 | |
| | "price discrimination" / "price steering" | 33 | 9 | | | "price discrimination" / "price steering" | 33 | 9 of 34 | |
| |
| These five vocabularies barely overlap, and none of them is a reliable handle on its own: the "sock puppet" 26 include an observational study **of** sockpuppets in discussion forums, a study of app-store fraud workers, and interviews with influence-operation participants, none of which is an audit. ((Kumar et al., TheWebConf 2017, //An Army of Me: Sockpuppets in Online Discussion Communities//; Rahman et al., CCS 2019, //The Art and Craft of Fraudulent App Promotion in Google Play//; Recabarren and Carbunar, USENIX Security 2023, //Strategies and Vulnerabilities of Participants in Venezuelan Influence Operations//.)) If you search for one term you will miss most of the field — which is exactly what happened here, and the probes that recovered the rest are listed on the provenance page. [[Design:Platforms]] publishes the same 26 against its own, smaller platform-study population, where it is **14**; the two numbers are the same query over different denominators. | These five vocabularies barely overlap, and none of them is a reliable handle on its own: the "sock puppet" 26 include an observational study **of** sockpuppets in discussion forums, a study of app-store fraud workers, and interviews with influence-operation participants, none of which is an audit. ((Kumar et al., TheWebConf 2017, //An Army of Me: Sockpuppets in Online Discussion Communities//; Rahman et al., CCS 2019, //The Art and Craft of Fraudulent App Promotion in Google Play//; Recabarren and Carbunar, USENIX Security 2023, //Strategies and Vulnerabilities of Participants in Venezuelan Influence Operations//.)) If you search for one term you will miss most of the field — which is exactly what happened here, and the probes that recovered the rest are listed on the provenance page. [[Design:Platforms]] publishes the same 26 against its own, smaller platform-study population, where it is **14**; the two numbers are the same query over different denominators. |
| ==== Methodology and limitations of these figures ==== | ==== Methodology and limitations of these figures ==== |
| |
| * **Populations.** ''empirical'' = 5,118; ''crawled'' = a crawl-configuration record exists **or** ''studyTypes'' includes ''automated-web-crawl'' = 1,120; ''measuredFrom'' = has a vantage tuple = 3,908. The **audit** population of 32 is hand-adjudicated against a written rule and is not an extraction field. | * **Populations.** ''empirical'' = 5,118; ''crawled'' = a crawl-configuration record exists **or** ''studyTypes'' includes ''automated-web-crawl'' = 1,120; ''measuredFrom'' = has a vantage tuple = 3,908. The **audit** population of 34 is hand-adjudicated against a written rule and is not an extraction field. |
| * **It is a lower bound, and a biased one.** See [[#What this corpus cannot tell you]]: 48 audit-topical papers in the index are outside the extraction, 45 of them because the selection screen looks for security or privacy and an audit is often neither. | * **It is a lower bound, and a biased one.** See [[#What this corpus cannot tell you]]: 48 audit-topical papers in the index are outside the extraction, 45 of them because the selection screen looks for security or privacy and an audit is often neither. |
| * **Counts are of papers, never tuples.** Sentinels (''not-stated'', ''none-mentioned'') are never counted as answers. | * **Counts are of papers, never tuples.** Sentinels (''not-stated'', ''none-mentioned'') are never counted as answers. |
| * **Seven venues.** EuroS&P, ACSAC, RAID, AsiaCCS, CHI, SOUPS, FAccT and WWW's fairness track siblings are absent. FAccT's absence matters more here than on any other page on this site. | * **Seven venues.** EuroS&P, ACSAC, RAID, AsiaCCS, CHI, SOUPS, FAccT and WWW's fairness track siblings are absent. FAccT's absence matters more here than on any other page on this site. |
| * **2025–2026 are provisional.** Do not read the last bucket as a completed period. [[literature:corpus]]. | * **2025–2026 are provisional.** Do not read the last bucket as a completed period. [[literature:corpus]]. |
| * **Full query log, the 32 verdicts and the 12 rejections, the probes and their residue, quote checks and reviewer findings:** [[provenance:design:algorithm_audits]]. | * **Full query log, the 34 verdicts and the 36 rejections, the probes and their residue, quote checks and reviewer findings:** [[provenance:design:algorithm_audits]]. |
| |
| ===== What to report ===== | ===== What to report ===== |
| |
| A reviewer who knows this literature will look for these, in roughly this order. Most audit papers in this corpus supply the first three and skip the fourth. | A reviewer who knows this literature will look for these, in roughly this order. The corpus says the field is good at the first item and bad at the rest: 24 of the 26 audits that drive a browser state their profile state, but only 14 of 34 say where they measured from and only 10 of 34 establish a null. |
| |
| - **The arms**, as a table: how many identities, what each one's treatment was, and what the control arm's treatment was. "We created personas" is not a design. | - **The arms**, as a table: how many identities, what each one's treatment was, and what the control arm's treatment was. "We created personas" is not a design. |
| - **Attrition**: identities that got banned, rate-limited or challenged, and whether losing them is correlated with the treatment. A treatment arm that gets banned more often is a finding, not a nuisance. | - **Attrition**: identities that got banned, rate-limited or challenged, and whether losing them is correlated with the treatment. A treatment arm that gets banned more often is a finding, not a nuisance. |
| |
| One sentence that does most of it: //"Twelve Chrome 141 profiles (six treatment, six control) ran from one residential vantage in Germany, each trained on 200 page loads over four hours in randomised order, then measured on the same 50 publisher pages; six further control–control pairs established a same-treatment baseline, and all tests are over profile-runs (n = 6 per condition) with Holm–Bonferroni across the four outcomes."// | One sentence that does most of it: //"Twelve Chrome 154 profiles (six treatment, six control) ran from one residential vantage in Germany, each trained on 200 page loads over four hours in randomised order, then measured on the same 50 publisher pages; six further control–control pairs established a same-treatment baseline, and all tests are over profile-runs (n = 6 per condition) with Holm–Bonferroni across the four outcomes."// |
| |
| ===== Open Questions ===== | ===== Open Questions ===== |
