Working log for algorithm_audits. Every figure on that page has its query here, with the population it is a share of. Corpus-level caveats — venue scope, the selection funnel, provisional years — are on corpus and are not restated.

Run: 2026-09-11. Corpus at the time: data/extract/run1/extractions.jsonl, 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026; 5,855 of them have full text on disk. Model: Claude Opus 5 for the run, four review sub-agents (three sonnet, one fable) — findings logged at the foot of this page.

The task brief flagged this as the smallest of five page proposals and asked for it to be sized before writing. It was. The sizing result and the decision:

Decision: created as a new page, design:algorithm_audits, linked from Design and from Automated measurements.

Written before any table, and encoded in scripts/algorithm_audits_set.mjs rather than in prose. A paper is in if all three hold:

1. (T) Treatment. It deliberately varies a property of the measuring identity or request — profile history, declared attribute, location, device, opt-out setting, ad creative — and holds the rest fixed.
2. (O) Outcome. What it measures is the platform's own response: ads served, results ranked, prices quoted, feed or recommendation contents, or an access decision.
3. (C) Contrast. The result is a difference (or a bounded absence of difference) between arms, not a prevalence over a crawl of many sites.

Deliberate consequences of this rule, each of which a reasonable person could have decided the other way:

• Attack and defence papers are in if they run the arms. [1Meng, Wei; Xing, Xinyu; Sheth, Anmol; Weinsberg, Udi; Lee, Wenke (2014): "Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)], [2Kim, I Luk; Wang, Weihang; Kwon, Yonghwi; Zheng, Yunhui; Aafer, Yousra; Meng, Weijie; Zhang, Xiangyu (2018): "AdBudgetKiller: Online Advertising Budget Draining Attack", in: Proceedings of the ACM Web Conference. (DOI)] and [3Zhang, Jiang; Psounis, Konstantinos; Haroon, Muhammad; Shafiq, Zubair (2022): "HARPO: Learning to Subvert Online Behavioral Advertising", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] are not audit studies — they are an attack, an attack and a defence — but each runs a differential experiment against a live ad platform, and a student reading this page wants those designs. The criterion is the measurement design, not the paper's contribution type.
• Instrument papers are in. [4Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] and [5Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)] contribute tooling; they are the two papers the page most wants read.
• Observational “audits” are out. [6Silva, Márcio; Oliveira, Lucas Santos de; Andreou, Athanasios; Melo, Pedro Olmo Stancioli Vaz de; Goga, Oana; Benevenuto, Fabrício (2020): "Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook", in: Proceedings of the ACM Web Conference. (DOI)] calls itself “An Independent Auditing System” and collects ads from volunteers. It fails (C). So does [7Ali, Muhammad; Goetzen, Angelica; Mislove, Alan; Redmiles, Elissa M.; Sapiezynski, Piotr (2023): "Problematic Advertising and its Disparate Exposure on Facebook", in: Proceedings of the USENIX Security Symposium. (Link)] and so does Auditing the Partisanship of Google Search Snippets (TheWebConf 2019), which audits snippets against the pages they summarise with no identity treatment at all.
• ML fairness, DP and system-log auditing are out. They share the word and nothing else.
• RCTs whose treatment is not applied to the platform are out. [8Lone, Qasim; Frik, Alisa; Luckie, Matthew; Korczyński, Maciej; van Eeten, Michel; Gañán, Carlos (2022): "Deployment of Source Address Validation by Network Operators: A Randomized Control Trial", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] is a randomised controlled trial, but the treatment is a notification sent to network operators; the outcome is not platform output.

No single regex finds this literature: the five vocabularies barely overlap and three of them (“audit”, “persona”, “personalisation”) are dominated by other meanings. The candidate set is the union of six probes, and each entry records which probe caught it.

Probe 7 is the one that mattered. After probes 1–6 the set had a hole at 2020–2021 and I did not trust it. Running the density probe over those two years surfaced three papers the union had missed entirely — [9Cook, John; Nithyanand, Rishab; Shafiq, Zubair (2020): "Inferring Tracker-Advertiser Relationships in the Online Advertising Ecosystem using Header Bidding", in: Proceedings on Privacy Enhancing Technologies. (DOI)], [10Agarwal, Pushkal; Joglekar, Sagar; Papadopoulos, Panagiotis; Sastry, Nishanth; Kourtellis, Nicolas (2020): "Stop tracking me Bro! Differential Tracking of User Demographics on Hyper-Partisan Websites", in: Proceedings of the ACM Web Conference. (DOI)] and, when run over all years, [11Iqbal, Hassan; Khan, Usman Mahmood; Khan, Hassan Ali; Shahzad, Muhammad (2022): "Left or Right: A Peek into the Political Biases in Email Spam Filtering Algorithms During US Election 2020", in: Proceedings of the ACM Web Conference. (DOI)] (102 email accounts on Gmail/Outlook/Yahoo, a design no “personalisation” or “persona” probe reaches because the paper's vocabulary is spam filtering). Three of the final 32, or 9.4%, were found only by the seventh probe. Re-running probe 7 on 2017 and 2021 after that returned nothing but the observational sockpuppet study, so those two zero-years are reported as real.

What no probe reached. The inclusion rule needs a paper's design, and design language is not a vocabulary. A paper that ran arms and described them only as “Group A and Group B” would be invisible to all seven probes. No claim on the page depends on the 32 being exhaustive; the page says so.

108 candidates scored ≥3 across the probes were shortlisted with their abstracts (_aa_short.mjs). The 58 scoring ≥4, plus 17 lower-scored papers flagged by hand, were passed through _aa_adj.mjs, which prints every sentence in the paper matching an arm/treatment/control/persona pattern. Each verdict was then made by reading those sentences, and where they were ambiguous, by grepping the paper's methods section directly.

Every verdict, with the sentence that settled it, is in the script output below (–list). The script refuses to run if any entry lacks an adjudication note.

The most important finding about this page's own evidence base.

The extraction's selection screen labels each abstract securityMeasurement and privacyMeasurement and keeps a paper if either is true. An algorithm audit is frequently neither — a search-personalisation or price-discrimination study reads as fairness, economics or information retrieval.

The script asserts that these buckets sum, and that seven named papers are still found by the query — so a corpus refresh that quietly re-admits them will fail the run rather than leave a stale claim on the page.

Verified individually against labels.jsonl:

The full list of 48 is in the –list output below. Consequence, stated on the content page: every audit count on it is a lower bound, biased against fairness-framed work, and the page publishes no estimate of the wider literature's size.

Almost nothing on this page needs folding, because almost nothing on it is a free-text aggregate — the population is hand-keyed and the rest are enum-backed counts. Two exceptions:

• statistics.method is free text and ~20% stable run-to-run. It is folded to an alphanumeric skeleton (lowercase, non-alphanumerics stripped) and paper-counted, and it is published as a ranking, not as percentages. The fold does not merge synonyms: Holm-Bonferroni correction (3), Holm-Bonferroni (1) and Holm-Bonferroni method (1) are three rows in the raw output. An unfolded reading would publish “Holm–Bonferroni 3”; the true paper count for the Holm–Bonferroni family is 5 of 32, and for any Bonferroni-family correction 7 of 32. The report script now prints the hand-folded family counts alongside the raw skeleton ranking, so the page quotes a number the script produced rather than one assembled in prose. Both are in the output below.
• The eight noise-baseline phrasings are not a fold but a deliberately widened probe (_aa_noise.mjs): the narrow term “A/A test” returns 2 papers corpus-wide, so seven further phrasings were added. The count rose from 0 to 11 of 32. Residue is not applicable — the probe reports term presence, and the page states explicitly that a paper can build a null without any of the eight phrasings.

Residue of the candidate probes. Probe 2 returned 190 candidates of which 32 are in and 12 were explicitly rejected; the remaining 146 were read at title-and-abstract level and dropped without an individual note. They are overwhelmingly papers that mention personalisation or targeted advertising in passing. That is the honest residue of this page: 146 papers whose exclusion rests on a title-level read, not a full-text read.

scripts/quotecheck_algorithm_audits.mjs verifies every quoted fragment on the content page against the paper's own text, in three modes (whitespace-collapsed exact; hyphen- and quote-normalised; longest 8-word run), against both paper.cols.txt and paper.norm.txt. It exits non-zero on any failure.

28 quotes, 0 failures, all matching exactly in paper.cols.txt. Full output below.

Two quotes were caught and removed before publication, both column-splice artefacts:

• [12Guha, Saikat; Cheng, Bin; Francis, Paul (2010): "Challenges in measuring online advertising systems", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]: the sentence naming the two seeded interest sets reads, in both renderings, “C was Even with static DNS entries, we sometimes (but not al- seeded with long-term interests in 'Autos & Vehicles', while ways) observed discrepancies…” — two columns interleaved. The contiguous fragment “enabled, but are seeded with different user personae” is used instead, and no quote naming the two interest sets is published.
• [9Cook, John; Nithyanand, Rishab; Shafiq, Zubair (2020): "Inferring Tracker-Advertiser Relationships in the Online Advertising Ecosystem using Header Bidding", in: Proceedings on Privacy Enhancing Technologies. (DOI)]: this PDF is interleaved throughout in both cols and norm. “we first selectively expose a” is followed by text from the adjacent column. No verbatim quote from this paper is published anywhere on either page; its contribution is described instead.

That is the page's one unfixable data-quality problem and it is why the checker tries two renderings rather than one.

Deliberately few. This page's subject is a research method, not a product, so there is no vendor documentation to verify and no version numbers to date.

• The size of this literature. The corpus cannot give it; see The screening loss. Closing it needs a pass over FAccT, EuroS&P and the IR venues, which is out of scope for a corpus-backed page.
• Whether the 146 dropped probe-2 candidates contain any true audits. They were read at title-and-abstract level only. A full-text read of all 146 would settle it and was not done.
• Whether papers build a null without naming it. The eight-term probe is a lower bound. Establishing the real rate needs 32 methods sections read for the concept, which is a different and slower exercise than the verdict read that produced the population.
• Effect sizes. Every audit measures a different outcome on a different platform with a different metric. Nothing is poolable, and the page publishes no cross-paper effect size deliberately.
• Whether sharing an egress IP across arms actually biases an ad-targeting outcome. No paper in the corpus measures it. Filed as an Open Question on the content page rather than asserted.
• The 2011–2013 and 2017 and 2021 zeros. They are real in this corpus. Whether they are real in the field is exactly the question the screening loss prevents answering — 2013 is the year of Hannak et al., which the screen dropped.

1. A new page, not a section. Reasoning in the first table above.
2. Hand-adjudicated population rather than a regex population. A regex set would be reproducible and wrong: the tight probe (27 papers) includes an inaudible-voice-command attack and a NIST privacy-framework paper, and misses AdFisher. Hand verdicts are recorded with their evidence sentence and the script refuses to run without them.
3. Attack and defence papers counted as audits. See The inclusion rule. Counting only audit studies would give 26 rather than 32 and would exclude the pollution attack, which is one of the clearest illustrations of a blank-vs-trained-profile contrast in the corpus.
4. “Price discrimination is dormant” was drafted and then withdrawn. The first draft said the last price-outcome audit was [13Chen, Le; Mislove, Alan; Wilson, Christo (2015): "Peeking Beneath the Hood of Uber", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (2015). Checking the post-2015 price discrimination/price steering hits found [14Becerril-Arreola, Rafael (2023): "A Method to Assess and Explain Disparate Impact in Online Retailing", in: Proceedings of the ACM Web Conference. (DOI)], whose outcomes are “price, recommendations, and delivery fees”. The row now reads “rare, not dormant”, with 2 of 32 measuring a price.
5. The A/A-test claim is stated as a **term count plus an eight-way concept probe, never as “nobody does this”. The distinction is load-bearing: the corpus can prove the vocabulary is absent, and can only lower-bound the practice. - **design:platforms reports 26 for “sock puppet” corpus-wide and 14 within its own platform-study population. This page reports the same 26. Both are correct; the page says so explicitly rather than letting a reader find two numbers.
6. No ~~DISCUSSION~~ block on this provenance page. Comments belong on the content page. This is the default recorded for provenance: pages and it is followed here.

REVIEWLOG

Committed as scripts/report_algorithm_audits.mjs with the population in scripts/algorithm_audits_set.mjs. It exits 1 if the corpus size, the empirical / crawled / measuredFrom populations, the year and venue bucket sums, the screening-loss bucket sums, or the seven named screening losses disagree with the contracts it encodes.

algorithm_audits_set.mjs

// The hand-adjudicated audit population for design:algorithm_audits.
// Imported by report_algorithm_audits.mjs and by the probe scripts, so every
// figure on the page and every probe share one definition of the set.
//
// ---------------------------------------------------------------------------
// The inclusion rule, as applied. A paper is IN if all three hold:
//   (T) it deliberately varies a property of the measuring identity or request
//       -- profile history, declared attribute, location, device, opt-out
//       setting, ad creative -- and holds the rest fixed;
//   (O) the outcome it measures is the platform's own response: ads served,
//       results ranked, prices quoted, feed or recommendation contents, or an
//       access/labelling decision;
//   (C) the result is a difference (or a bounded absence of difference)
//       BETWEEN arms, not a prevalence over a crawl of many sites.
// The `why` string is the evidence sentence that settled (T)+(C).
// ---------------------------------------------------------------------------
export const AUDITS = [
  ['IMC/2010/challenges-in-measuring-online-advertising-systems',
    'seeded profile pairs: "enabled, but are seeded with different user personae" (the sentence naming the two interest sets is column-spliced in both renderings)'],
  ['CCS/2014/your-online-interests-pwned-a-pollution-attack-against-targeted-advertising',
    'blank profile vs polluted profile: "the polluter can impersonate a user with a blank profile ... and browse pages"'],
  ['CCS/2015/sunlight-fine-grained-targeting-detection-at-scale-with-statistical-confidence',
    'instrument paper: varies personal-data inputs one at a time "compared to a control group", with statistical confidence'],
  ['IMC/2015/location-location-location-the-impact-of-geolocation-on-web-search-personalizati',
    'location as the treatment: "all other browser attributes were the same across treatments"'],
  ['IMC/2015/peeking-beneath-the-hood-of-uber',
    '"We created 43 Uber accounts ... blanket a small geographic area with measurement points"'],
  ['PETS/2015/automated-experiments-on-ad-privacy-settings',
    'AdFisher: "We created an experimental group and a control group of agents"'],
  ['USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads',
    '"We train 90 personas by visiting popular e-commerce sites, and then crawl major publishers"'],
  ['WWW/2018/adbudgetkiller-online-advertising-budget-draining-attack',
    '"Starting from a fresh profile, the profile trainer produces" crafted browsing profiles'],
  ['WWW/2018/auditing-the-personalization-and-composition-of-politically-related-search-engin',
    'standard-vs-incognito paired SERPs: "our controls were paired within the individual"'],
  ['PETS/2019/investigating-sources-of-pii-used-in-facebook-s-targeted-advertising',
    '"Take a Facebook account that we control (call it the control account) and the test PII"'],
  ['WWW/2019/measuring-political-personalization-of-google-news-search',
    '"a \'sock puppet\' auditing system in which a pair of fresh browser profiles" visits divergent pages then runs identical queries'],
  ['PETS/2020/inferring-tracker-advertiser-relationships-in-the-online-advertising-ecosystem-u',
    'intent vs no-intent versions of each of 16 interest personas, with trackers selectively exposed (this PDF is column-spliced in both cols and norm renderings, so no contiguous quote was publishable)'],
  ['WWW/2020/stop-tracking-me-bro-differential-tracking-of-user-demographics-on-hyper-partisa',
    '"We create 9 carefully crafted personas representing different genders and age groups"'],
  ['IMC/2022/measurement-and-analysis-of-implied-identity-in-ad-delivery-optimization',
    '"We ran 200 versions of this ad at the same time, all from the same account and with the same budget"'],
  ['IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement',
    '"All participants were asked to visit the same websites to control for contextual targeting, in randomized order"'],
  ['NDSS/2022/auto-draft-209',
    'HARPO: obfuscated vs unobfuscated personas measured against live profiling models'],
  ['PETS/2022/atom-ad-network-tomography',
    '"create a number of online user personas associated with specific interest groups" and gather ads while systematically blocking trackers'],
  ['WWW/2022/left-or-right-a-peek-into-the-political-biases-in-email-spam-filtering-algorithm',
    '"We created 102 email accounts" and compared spam placement across treatment and control affiliations'],
  ['CCS/2023/marketing-to-children-through-online-targeted-advertising-targeting-mechanisms-a',
    '"We launch the six personas simultaneously by creating six Selenium" browsers'],
  ['IMC/2023/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem',
    '"Each treatment persona is simulated by installing and interacting with skills ... By contrast, in the control persona, we do" not'],
  ['PETS/2023/a-utility-preserving-obfuscation-approach-for-youtube-recommendations',
    '"We deploy and evaluate De-Harpo\'s effectiveness on YouTube using 10,000 sock puppet based personas"'],
  ['WWW/2023/a-method-to-assess-and-explain-disparate-impact-in-online-retailing',
    '"Each observation ... is paired with a set of \'counter-factuals\'" from neighbouring zip codes'],
  ['PETS/2024/opted-out-yet-tracked-are-regulations-enough-to-protect-your-privacy',
    '"We then visit the filtered websites with user interest personas" -- 16 interest personas, two regions'],
  ['WWW/2024/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation',
    '"validate our results using a baseline generated from automated TikTok bots, as well as a randomized baseline"'],
  ['IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba',
    '"we used each of the 21 cloned voice models to train two puppets, which we refer to as \'paired puppets\'"'],
  ['PETS/2025/echoes-of-privacy-uncovering-the-profiling-practices-of-voice-assistants',
    '"each meticulously trained with a curated set of voice queries designed to simulate various user personas"'],
  ['PETS/2025/more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings',
    '"running controlled experiments with sock puppet accounts that emulate users watching YouTube videos"'],
  ['PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic',
    '"two separate crawlers were deployed: one for the Control group (without ABP) and another for the AccAds group"'],
  ['USENIX/2025/big-help-or-big-brother-auditing-tracking-profiling-and-personalization-in-gener',
    '"Training phase involves browsing through 10 webpages - 2 pages per leaked attribute", then personalisation is measured'],
  ['IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right',
    '"we use sock-puppet accounts to systematically browse and log the behavior of the sock-puppet to generate the ground truth"'],
  ['PETS/2026/ad-personalization-and-transparency-in-mobile-ecosystems-a-comparative-analysis',
    '"We construct accounts with specific parameters or interests, so-called personas, and measure the ads displayed to them"'],
  ['WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls',
    '"depending on randomized assignment - triggered one of four native feedback signals to YouTube ... or no signal at all in the control group"'],
];
 
// Adjudicated OUT, with the reason. These are the near misses a later run will
// otherwise re-add; the rule that excluded them is printed with each.
export const REJECTED = [
  ['WWW/2017/an-army-of-me-sockpuppets-in-online-discussion-communities', 'observational study OF sockpuppets others created; no arm the authors control'],
  ['WWW/2019/auditing-the-partisanship-of-google-search-snippets', 'audits snippet vs linked page; no identity treatment, no arms'],
  ['WWW/2020/facebook-ads-monitor-an-independent-auditing-system-for-political-ads-on-faceboo', 'volunteer ad donation; observational, no arms'],
  ['USENIX/2020/what-twitter-knows-characterizing-ad-targeting-practices-user-perceptions-and-ad', "users' own Twitter data; no arms"],
  ['IEEE-SP/2023/collaborative-ad-transparency-promises-and-limitations', 'explicitly the non-persona alternative: "One method that does not use fake personas"'],
  ['USENIX/2023/problematic-advertising-and-its-disparate-exposure-on-facebook', 'donated ad exposure from real users; disparity is observational'],
  ['CCS/2022/cart-ology-intercepting-targeted-advertising-via-ad-network-identity-entanglemen', 'attack on ad-network identity linking; no persona arms'],
  ['CCS/2022/privacy-limitations-of-interest-based-advertising-on-the-web-a-post-mortem-empir', 'randomised control is a data permutation over a browsing panel, not a live-platform arm'],
  ['IEEE-SP/2024/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites', 'fresh profile per page visit; the paper itself says "Future work could extend our method to incorporate personas"'],
  ['PETS/2024/interest-disclosing-mechanisms-for-advertising-are-privacy-exposing-not-preservi', 'Topics API analysed over real browsing histories, not persona arms'],
  ['WWW/2026/when-ads-become-profiles-uncovering-the-invisible-risk-of-web-advertising-at-sca', '"Random Control group" is a model ablation, not a platform arm'],
  ['IEEE-SP/2022/deployment-of-source-address-validation-by-network-operators-a-randomized-contro', 'an RCT, but the treatment is a notification to operators; outcome is not platform output'],
];

report_algorithm_audits.mjs

#!/usr/bin/env node
// Report script for design:algorithm_audits (and provenance:design:algorithm_audits).
//
// Every figure on the page is printed here with the population it is a share of.
// Counts are of PAPERS, never tuples. Sentinels are never answers.
//
// The audit population is HAND-KEYED below rather than derived from a regex.
// That is deliberate: the corpus has no field for "ran a differential audit",
// the regex probes that find candidates have 20-60% precision, and a script that
// silently re-derives the split from prose is how a wrong split reaches a page.
// The candidate probes that produced the list are in _aa_probe1.mjs / _aa_abs2.mjs
// and are documented on the provenance page; this file records the verdicts.
//
// Usage: node scripts/report_algorithm_audits.mjs [--list]
 
import fs from 'node:fs';
import path from 'node:path';
import { dataRoot, loadExtractions, isSentinel } from './lib.mjs';
 
const SHOW_LIST = process.argv.includes('--list');
const ROOT = dataRoot();
const papers = loadExtractions();
const byKey = new Map(papers.map((p) => [`${p.venue}/${p.year}/${p.slug}`, p]));
 
import { AUDITS, REJECTED } from './algorithm_audits_set.mjs';
 
// ---------------------------------------------------------------------------
function die(msg) { console.error(`CONTRACT VIOLATED: ${msg}`); process.exit(1); }
const pct = (n, d) => (d === 0 ? 'n/a' : `${((100 * n) / d).toFixed(1)}%`);
 
// Contracts against the corpus, so a corpus refresh cannot silently move a page.
if (papers.length !== 5859) die(`corpus is ${papers.length} papers, page says 5,859`);
const CRAWLED = papers.filter((p) => p.crawlConfig !== null || p.studyTypes.includes('automated-web-crawl'));
const INFERENTIAL = papers.filter((p) => p.statistics.some((s) => s.kind && s.kind !== 'descriptive-only'));
const EMPIRICAL = papers.filter((p) => p.isEmpirical === true);
if (CRAWLED.length !== 1120) die(`crawled population is ${CRAWLED.length}, OVERVIEW.md says 1120`);
if (INFERENTIAL.length !== 1762) die(`inferential population is ${INFERENTIAL.length}, OVERVIEW.md says 1762`);
if (EMPIRICAL.length !== 5118) die(`empirical population is ${EMPIRICAL.length}, OVERVIEW.md says 5118`);
 
const audit = [];
for (const [key, why] of AUDITS) {
  const p = byKey.get(key);
  if (!p) die(`hand-keyed audit paper not in the extraction: ${key}`);
  if (!why || why.length < 25) die(`no adjudication evidence recorded for ${key}`);
  audit.push(p);
}
if (new Set(AUDITS.map((a) => a[0])).size !== AUDITS.length) die('duplicate key in AUDITS');
for (const [key] of REJECTED) if (!byKey.get(key)) die(`rejected paper not in the extraction: ${key}`);
const N = audit.length;
 
console.log('='.repeat(78));
console.log('design:algorithm_audits -- report script');
console.log(`run ${new Date().toISOString().slice(0, 10)}   corpus ${papers.length} papers, 7 venues, 2010-2026`);
console.log('='.repeat(78));
console.log('');
console.log('## Populations');
console.log(`  all papers                                  ${papers.length}`);
console.log(`  empirical                                   ${EMPIRICAL.length}`);
console.log(`  crawled                                     ${CRAWLED.length}`);
console.log(`  inferential (any non-descriptive statistic)  ${INFERENTIAL.length}`);
console.log(`  AUDIT (hand-adjudicated, rule above)        ${N}`);
console.log(`  adjudicated and rejected                    ${REJECTED.length}`);
console.log('');
 
// --- by year -----------------------------------------------------------------
console.log('## Audit papers by year (denominator: the 32-paper audit set)');
const years = [...new Set(papers.map((p) => p.year))].sort();
let run = 0;
for (const y of years) {
  const n = audit.filter((p) => p.year === y).length;
  run += n;
  const star = y >= 2025 ? ' *provisional' : '';
  console.log(`  ${y}  ${String(n).padStart(2)}  ${'#'.repeat(n)}${star}`);
}
if (run !== N) die(`year buckets sum to ${run}, not ${N}`);
console.log(`  total ${run}`);
const gaps = years.filter((y) => audit.every((p) => p.year !== y));
console.log(`  years with zero audit papers: ${gaps.join(', ')}`);
console.log('');
for (const [lo, hi] of [[2010, 2015], [2016, 2019], [2020, 2023], [2024, 2026]]) {
  const n = audit.filter((p) => p.year >= lo && p.year <= hi).length;
  console.log(`  ${lo}-${hi}: ${n} of ${N} (${pct(n, N)})`);
}
console.log('');
 
// --- by venue and platform ---------------------------------------------------
console.log('## Audit papers by venue, against that venue\'s own output');
const venues = [...new Set(papers.map((p) => p.venue))].sort();
let vsum = 0;
for (const v of venues) {
  const tot = papers.filter((p) => p.venue === v).length;
  const n = audit.filter((p) => p.venue === v).length;
  vsum += n;
  console.log(`  ${v.padEnd(9)} ${String(n).padStart(2)} of ${String(tot).padStart(5)}  ${pct(n, tot).padStart(6)}`);
}
if (vsum !== N) die(`venue buckets sum to ${vsum}, not ${N}`);
console.log('');
console.log('## Platform measured (multi-valued; does not sum to N)');
for (const pl of ['web', 'other-online-service', 'mobile', 'iot', 'offline']) {
  const n = audit.filter((p) => p.platforms.includes(pl)).length;
  console.log(`  ${pl.padEnd(22)} ${String(n).padStart(2)} of ${N}  ${pct(n, N)}`);
}
console.log('');
 
// --- what the audit set reports, each row against the same-denominator baseline
console.log('## What audit papers report, vs the comparable corpus baseline');
console.log('   Each row states both populations. The baseline is the population named,');
console.log('   not "all papers", so the two cells are comparable.');
const statedStat = (p) => p.statistics.some((s) => s.kind && s.kind !== 'descriptive-only');
const statedEthics = (p) => !isSentinel(p.ethics && p.ethics.reviewOutcome);
const hasArtifact = (p) => p.artifacts && !isSentinel(p.artifacts.availability);
const hasCrawlCfg = (p) => p.crawlConfig !== null;
const statefulStated = (p) => p.crawlConfig !== null && !isSentinel(p.crawlConfig.statefulness);
const interactionStated = (p) => p.crawlConfig !== null && !isSentinel(p.crawlConfig.interactionDepth);
const vantageStated = (p) => p.vantage.some((v) => (v.locations || []).some((l) => !isSentinel(l)));
const rows = [
  ['runs a non-descriptive statistic', audit, statedStat, EMPIRICAL, statedStat, 'empirical (5,118)'],
  ['states an ethics-review outcome', audit, statedEthics, EMPIRICAL, statedEthics, 'empirical (5,118)'],
  ['states artifact availability', audit, hasArtifact, EMPIRICAL, hasArtifact, 'empirical (5,118)'],
  ['has a crawlConfig at all', audit, hasCrawlCfg, CRAWLED, hasCrawlCfg, 'crawled (1,120)'],
  ['states crawl statefulness', audit, statefulStated, CRAWLED, statefulStated, 'crawled (1,120)'],
  ['states interaction depth', audit, interactionStated, CRAWLED, interactionStated, 'crawled (1,120)'],
  ['states a vantage location', audit, vantageStated, papers.filter((p) => p.vantage.length > 0), vantageStated, 'measuredFrom (3,908)'],
];
console.log(`  ${'indicator'.padEnd(34)} ${'audit (32)'.padStart(14)}   baseline`);
for (const [label, aSet, aFn, bSet, bFn, bName] of rows) {
  const a = aSet.filter(aFn).length;
  const b = bSet.filter(bFn).length;
  console.log(`  ${label.padEnd(34)} ${(`${a}/${aSet.length} (${pct(a, aSet.length)})`).padStart(14)}   ${b}/${bSet.length} (${pct(b, bSet.length)}) of ${bName}`);
}
console.log('');
 
// --- statistics named in the audit set --------------------------------------
console.log('## Statistical methods named by audit papers (paper-counted, free text, ranking only)');
const smeth = new Map();
for (const p of audit) {
  const seen = new Set();
  for (const s of p.statistics) {
    if (!s.method || isSentinel(s.method)) continue;
    const k = String(s.method).toLowerCase().replace(/[^a-z0-9]+/g, '');
    if (seen.has(k)) continue;
    seen.add(k);
    smeth.set(k, (smeth.get(k) || 0) + 1);
  }
}
const disp = new Map();
for (const p of audit) for (const s of p.statistics) {
  if (!s.method || isSentinel(s.method)) continue;
  const k = String(s.method).toLowerCase().replace(/[^a-z0-9]+/g, '');
  if (!disp.has(k)) disp.set(k, s.method);
}
for (const [k, v] of [...smeth.entries()].sort((a, b) => b[1] - a[1]).slice(0, 15))
  console.log(`  ${String(v).padStart(2)}  ${disp.get(k)}`);
// Synonym families the alphanumeric skeleton fold does NOT merge. Printed here
// so the page quotes a family count produced by this script rather than by prose.
console.log('  -- correction families, folded by hand (paper-counted; the skeleton fold does not merge these):');
const FAMILIES = {
  'Holm-Bonferroni (any spelling)': (m) => /holm/i.test(m),
  'Bonferroni without Holm': (m) => /bonferroni/i.test(m) && !/holm/i.test(m),
  'any Bonferroni-family': (m) => /bonferroni/i.test(m),
  'Benjamini-Hochberg/Yekutieli': (m) => /benjamini/i.test(m),
};
for (const [name, fn] of Object.entries(FAMILIES)) {
  const n = audit.filter((p) => p.statistics.some((s) => s.method && !isSentinel(s.method) && fn(s.method))).length;
  console.log(`       ${name.padEnd(32)} ${n} of ${N}`);
}
 
const noStat = audit.filter((p) => !statedStat(p));
console.log(`  -- ${noStat.length} of ${N} audit papers report no non-descriptive statistic at all:`);
for (const p of noStat) console.log(`       ${p.year} ${p.venue} ${p.title}`);
console.log('');
 
// --- vocabulary the field uses ----------------------------------------------
console.log('## Apparatus vocabulary in full text (paper-counted over all 5,859 with text)');
const VOCAB = {
  'sock puppet': /\bsock[ -]?puppets?\b/i,
  persona: /\b(?:user|shopper|synthetic|training|treatment|control) personas?\b|\bpersonas?\b/i,
  'control profile/account/persona': /\bcontrol (?:profile|account|persona|browser)s?\b/i,
  'treatment group/profile': /\btreatment (?:group|profile|persona|condition|arm)s?\b/i,
  'trained profile': /\btrain(?:ed|ing) (?:the |our |a )?(?:browser )?profiles?\b/i,
  'A/A test': /\bA\/A test/i,
  'noise floor': /\bnoise floor\b/i,
  'price discrimination/steering': /\bprice (?:discrimination|steering)\b/i,
};
const vocabCount = Object.fromEntries(Object.keys(VOCAB).map((k) => [k, [0, 0]]));
let withText = 0;
for (const p of papers) {
  const f = path.join(ROOT, 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt');
  if (!fs.existsSync(f)) continue;
  withText++;
  const t = fs.readFileSync(f, 'utf8').replace(/\s+/g, ' ');
  const inAudit = audit.includes(p);
  for (const [name, re] of Object.entries(VOCAB)) if (re.test(t)) {
    vocabCount[name][0]++;
    if (inAudit) vocabCount[name][1]++;
  }
}
console.log(`  full text present for ${withText} of ${papers.length} papers`);
console.log(`  ${'term'.padEnd(32)} ${'corpus'.padStart(7)}  ${'in audit set'.padStart(12)}`);
for (const [k, [c, a]] of Object.entries(vocabCount))
  console.log(`  ${k.padEnd(32)} ${String(c).padStart(7)}  ${String(a).padStart(5)} of ${N}`);
console.log('');
 
// --- what the corpus cannot see ---------------------------------------------
console.log('## Screening loss: audit-topical papers in the index but not in the extraction');
const meta = [];
const md = path.join(ROOT, 'corpus2/.meta');
for (const f of fs.readdirSync(md)) {
  if (!f.endsWith('.json')) continue;
  const j = JSON.parse(fs.readFileSync(path.join(md, f), 'utf8'));
  const arr = Array.isArray(j) ? j : Object.values(j).find((v) => Array.isArray(v)) || [];
  for (const r of arr) meta.push(r);
}
const labels = new Map();
for (const l of fs.readFileSync(path.join(ROOT, 'labels/run1/labels.jsonl'), 'utf8').split('\n').filter(Boolean)) {
  const r = JSON.parse(l);
  labels.set(`${r.venue}/${r.year}/${r.slug}`, r);
}
const TOPIC = /\balgorithm(?:ic)? audit|\baudit(?:ing)? (?:the |of )?(?:search|recommend|ad |ads\b|advertis|algorithm|platform|feed|targeting|ranking)|sock ?-?puppet|price (?:discrimination|steering)|differential pricing|differential treatment|personali[sz]ation of|web search personali|search personali|ad delivery|ad targeting|targeted advertis|filter bubble|echo chamber|rabbit hole|discriminat\w+ (?:in|by) (?:ad|algorithm|ranking|recommend)/i;
const cands = meta.filter((r) => TOPIC.test(`${r.title || ''} ${r.abstract || ''}`.replace(/\s+/g, ' ')));
const out = cands.filter((r) => !byKey.has(`${r.venue}/${r.year}/${r.slug}`));
let screened = 0, nolabel = 0;
const lost = [];
for (const r of out) {
  const lab = labels.get(`${r.venue}/${r.year}/${r.slug}`);
  if (!lab) { nolabel++; lost.push(['no label record (venue-year gap)', r]); }
  else if (!lab.securityMeasurement && !lab.privacyMeasurement) { screened++; lost.push(['screened out: both labels false', r]); }
  else die(`unexpected: selected but not extracted: ${r.venue}/${r.year}/${r.slug}`);
}
console.log(`  index records                                 ${meta.length}`);
console.log(`  audit-topical candidates in the index          ${cands.length}`);
console.log(`  ... of which extracted (in the 5,859)          ${cands.length - out.length}`);
console.log(`  ... of which NOT extracted                     ${out.length}`);
console.log(`      screened out (both screen labels false)    ${screened}`);
console.log(`      no label record at all (venue-year gap)    ${nolabel}`);
if (screened + nolabel !== out.length) die('screening-loss buckets do not sum');
console.log('');
console.log('  Named losses a reader of this page would expect to find:');
const NAMED = [/measuring personalization of web search/i, /measuring price discrimination/i, /crying wolf/i,
  /auditing for discrimination in algorithms delivering job ads/i, /do you see what i see/i,
  /an empirical investigation of personalization factors on tiktok/i, /mapwatch/i];
for (const re of NAMED) {
  const hit = lost.find(([, r]) => re.test(r.title || ''));
  if (!hit) die(`named loss no longer found by the screening-loss query: ${re}`);
  console.log(`    ${hit[1].year} ${String(hit[1].venue).padEnd(8)} ${hit[1].title}  -- ${hit[0]}`);
}
console.log('');
 
if (SHOW_LIST) {
  console.log('## The audit set in full, with the sentence that settled the verdict');
  for (const [key, why] of AUDITS) {
    const p = byKey.get(key);
    console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
    console.log(`      ${key}`);
    console.log(`      IN: ${why}`);
  }
  console.log('');
  console.log('## Adjudicated and rejected');
  for (const [key, why] of REJECTED) {
    const p = byKey.get(key);
    console.log(`  ${p.year} ${p.venue.padEnd(8)} ${p.title}`);
    console.log(`      OUT: ${why}`);
  }
  console.log('');
  console.log('## Screening loss in full');
  for (const [why, r] of lost.sort((a, b) => a[1].year - b[1].year))
    console.log(`  ${r.year} ${String(r.venue).padEnd(8)} ${r.title}\n      ${why}`);
}
console.log('OK: all contracts held.');

Run on 2026-09-11 against data/extract/run1.

report_algorithm_audits-output.txt

==============================================================================
design:algorithm_audits -- report script
run 2026-09-11   corpus 5859 papers, 7 venues, 2010-2026
==============================================================================
 
## Populations
  all papers                                  5859
  empirical                                   5118
  crawled                                     1120
  inferential (any non-descriptive statistic)  1762
  AUDIT (hand-adjudicated, rule above)        32
  adjudicated and rejected                    12
 
## Audit papers by year (denominator: the 32-paper audit set)
  2010   1  #
  2011   0  
  2012   0  
  2013   0  
  2014   1  #
  2015   4  ####
  2016   1  #
  2017   0  
  2018   2  ##
  2019   2  ##
  2020   2  ##
  2021   0  
  2022   5  #####
  2023   4  ####
  2024   2  ##
  2025   5  ##### *provisional
  2026   3  ### *provisional
  total 32
  years with zero audit papers: 2011, 2012, 2013, 2017, 2021
 
  2010-2015: 6 of 32 (18.8%)
  2016-2019: 5 of 32 (15.6%)
  2020-2023: 11 of 32 (34.4%)
  2024-2026: 10 of 32 (31.3%)
 
## Audit papers by venue, against that venue's own output
  CCS        3 of   990    0.3%
  IEEE-SP    1 of   767    0.1%
  IMC        7 of   638    1.1%
  NDSS       1 of   701    0.1%
  PETS      10 of   510    2.0%
  USENIX     2 of  1410    0.1%
  WWW        8 of   843    0.9%
 
## Platform measured (multi-valued; does not sum to N)
  web                    25 of 32  78.1%
  other-online-service   18 of 32  56.3%
  mobile                  4 of 32  12.5%
  iot                     3 of 32  9.4%
  offline                 0 of 32  0.0%
 
## What audit papers report, vs the comparable corpus baseline
   Each row states both populations. The baseline is the population named,
   not "all papers", so the two cells are comparable.
  indicator                              audit (32)   baseline
  runs a non-descriptive statistic    21/32 (65.6%)   1637/5118 (32.0%) of empirical (5,118)
  states an ethics-review outcome     20/32 (62.5%)   1728/5118 (33.8%) of empirical (5,118)
  states artifact availability        22/32 (68.8%)   2890/5118 (56.5%) of empirical (5,118)
  has a crawlConfig at all            24/32 (75.0%)   1080/1120 (96.4%) of crawled (1,120)
  states crawl statefulness           22/32 (68.8%)   219/1120 (19.6%) of crawled (1,120)
  states interaction depth            21/32 (65.6%)   841/1120 (75.1%) of crawled (1,120)
  states a vantage location           14/32 (43.8%)   1228/3908 (31.4%) of measuredFrom (3,908)
 
## Statistical methods named by audit papers (paper-counted, free text, ranking only)
   3  Holm-Bonferroni correction
   2  linear regression
   2  Bonferroni correction
   2  Mann-Whitney U test
   1  CDFs, medians, percentiles, and percentages
   1  descriptive comparison of ad fractions and indexed CPM
   1  exact statistical test based on Pearson's correlation
   1  Holm-Bonferroni
   1  Benjamini-Yekutieli
   1  averages and standard deviations
   1  cross correlation
   1  blocked permutation test
   1  Holm-Bonferroni method
   1  Clopper-Pearson interval
   1  Counts and percentages of advertisements
  -- correction families, folded by hand (paper-counted; the skeleton fold does not merge these):
       Holm-Bonferroni (any spelling)   5 of 32
       Bonferroni without Holm          3 of 32
       any Bonferroni-family            7 of 32
       Benjamini-Hochberg/Yekutieli     1 of 32
  -- 11 of 32 audit papers report no non-descriptive statistic at all:
       2010 IMC Challenges in measuring online advertising systems.
       2014 CCS Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising.
       2015 IMC Location, Location, Location: The Impact of Geolocation on Web Search Personalization.
       2016 USENIX Tracing Information Flows Between Ad Exchanges Using Retargeted Ads
       2018 WWW AdBudgetKiller: Online Advertising Budget Draining Attack.
       2019 PETS Investigating sources of PII used in Facebook’s targeted advertising
       2022 NDSS HARPO: Learning to Subvert Online Behavioral Advertising
       2023 CCS Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Aspects.
       2023 PETS A Utility-Preserving Obfuscation Approach for YouTube Recommendations
       2025 PETS Echoes of Privacy: Uncovering the Profiling Practices of Voice Assistants
       2026 IEEE-SP Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube.
 
## Apparatus vocabulary in full text (paper-counted over all 5,859 with text)
  full text present for 5855 of 5859 papers
  term                              corpus  in audit set
  sock puppet                           26      5 of 32
  persona                              190     14 of 32
  control profile/account/persona       33     11 of 32
  treatment group/profile               66      5 of 32
  trained profile                        4      1 of 32
  A/A test                               2      0 of 32
  noise floor                           35      2 of 32
  price discrimination/steering         33      9 of 32
 
## Screening loss: audit-topical papers in the index but not in the extraction
  index records                                 16864
  audit-topical candidates in the index          117
  ... of which extracted (in the 5,859)          69
  ... of which NOT extracted                     48
      screened out (both screen labels false)    45
      no label record at all (venue-year gap)    3
 
  Named losses a reader of this page would expect to find:
    2013 WWW      Measuring personalization of web search.  -- screened out: both labels false
    2014 IMC      Measuring Price Discrimination and Steering on E-commerce Web Sites.  -- screened out: both labels false
    2014 PETS     Crying Wolf? On the Price Discrimination of Online Airline Tickets  -- no label record (venue-year gap)
    2021 WWW      Auditing for Discrimination in Algorithms Delivering Job Ads.  -- screened out: both labels false
    2016 NDSS     Do You See What I See? Differential Treatment of Anonymous Users  -- no label record (venue-year gap)
    2022 WWW      An Empirical Investigation of Personalization Factors on TikTok.  -- screened out: both labels false
    2016 WWW      MapWatch: Detecting and Monitoring International Border Personalization on Online Maps.  -- screened out: both labels false
 
OK: all contracts held.

node scripts/report_algorithm_audits.mjs –list — the 32 verdicts with their evidence sentence, the 12 rejections with their reason, and all 48 screening losses.

report_algorithm_audits-list-output.txt

==============================================================================
design:algorithm_audits -- report script
run 2026-09-11   corpus 5859 papers, 7 venues, 2010-2026
==============================================================================
 
## Populations
  all papers                                  5859
  empirical                                   5118
  crawled                                     1120
  inferential (any non-descriptive statistic)  1762
  AUDIT (hand-adjudicated, rule above)        32
  adjudicated and rejected                    12
 
## Audit papers by year (denominator: the 32-paper audit set)
  2010   1  #
  2011   0  
  2012   0  
  2013   0  
  2014   1  #
  2015   4  ####
  2016   1  #
  2017   0  
  2018   2  ##
  2019   2  ##
  2020   2  ##
  2021   0  
  2022   5  #####
  2023   4  ####
  2024   2  ##
  2025   5  ##### *provisional
  2026   3  ### *provisional
  total 32
  years with zero audit papers: 2011, 2012, 2013, 2017, 2021
 
  2010-2015: 6 of 32 (18.8%)
  2016-2019: 5 of 32 (15.6%)
  2020-2023: 11 of 32 (34.4%)
  2024-2026: 10 of 32 (31.3%)
 
## Audit papers by venue, against that venue's own output
  CCS        3 of   990    0.3%
  IEEE-SP    1 of   767    0.1%
  IMC        7 of   638    1.1%
  NDSS       1 of   701    0.1%
  PETS      10 of   510    2.0%
  USENIX     2 of  1410    0.1%
  WWW        8 of   843    0.9%
 
## Platform measured (multi-valued; does not sum to N)
  web                    25 of 32  78.1%
  other-online-service   18 of 32  56.3%
  mobile                  4 of 32  12.5%
  iot                     3 of 32  9.4%
  offline                 0 of 32  0.0%
 
## What audit papers report, vs the comparable corpus baseline
   Each row states both populations. The baseline is the population named,
   not "all papers", so the two cells are comparable.
  indicator                              audit (32)   baseline
  runs a non-descriptive statistic    21/32 (65.6%)   1637/5118 (32.0%) of empirical (5,118)
  states an ethics-review outcome     20/32 (62.5%)   1728/5118 (33.8%) of empirical (5,118)
  states artifact availability        22/32 (68.8%)   2890/5118 (56.5%) of empirical (5,118)
  has a crawlConfig at all            24/32 (75.0%)   1080/1120 (96.4%) of crawled (1,120)
  states crawl statefulness           22/32 (68.8%)   219/1120 (19.6%) of crawled (1,120)
  states interaction depth            21/32 (65.6%)   841/1120 (75.1%) of crawled (1,120)
  states a vantage location           14/32 (43.8%)   1228/3908 (31.4%) of measuredFrom (3,908)
 
## Statistical methods named by audit papers (paper-counted, free text, ranking only)
   3  Holm-Bonferroni correction
   2  linear regression
   2  Bonferroni correction
   2  Mann-Whitney U test
   1  CDFs, medians, percentiles, and percentages
   1  descriptive comparison of ad fractions and indexed CPM
   1  exact statistical test based on Pearson's correlation
   1  Holm-Bonferroni
   1  Benjamini-Yekutieli
   1  averages and standard deviations
   1  cross correlation
   1  blocked permutation test
   1  Holm-Bonferroni method
   1  Clopper-Pearson interval
   1  Counts and percentages of advertisements
  -- correction families, folded by hand (paper-counted; the skeleton fold does not merge these):
       Holm-Bonferroni (any spelling)   5 of 32
       Bonferroni without Holm          3 of 32
       any Bonferroni-family            7 of 32
       Benjamini-Hochberg/Yekutieli     1 of 32
  -- 11 of 32 audit papers report no non-descriptive statistic at all:
       2010 IMC Challenges in measuring online advertising systems.
       2014 CCS Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising.
       2015 IMC Location, Location, Location: The Impact of Geolocation on Web Search Personalization.
       2016 USENIX Tracing Information Flows Between Ad Exchanges Using Retargeted Ads
       2018 WWW AdBudgetKiller: Online Advertising Budget Draining Attack.
       2019 PETS Investigating sources of PII used in Facebook’s targeted advertising
       2022 NDSS HARPO: Learning to Subvert Online Behavioral Advertising
       2023 CCS Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Aspects.
       2023 PETS A Utility-Preserving Obfuscation Approach for YouTube Recommendations
       2025 PETS Echoes of Privacy: Uncovering the Profiling Practices of Voice Assistants
       2026 IEEE-SP Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube.
 
## Apparatus vocabulary in full text (paper-counted over all 5,859 with text)
  full text present for 5855 of 5859 papers
  term                              corpus  in audit set
  sock puppet                           26      5 of 32
  persona                              190     14 of 32
  control profile/account/persona       33     11 of 32
  treatment group/profile               66      5 of 32
  trained profile                        4      1 of 32
  A/A test                               2      0 of 32
  noise floor                           35      2 of 32
  price discrimination/steering         33      9 of 32
 
## Screening loss: audit-topical papers in the index but not in the extraction
  index records                                 16864
  audit-topical candidates in the index          117
  ... of which extracted (in the 5,859)          69
  ... of which NOT extracted                     48
      screened out (both screen labels false)    45
      no label record at all (venue-year gap)    3
 
  Named losses a reader of this page would expect to find:
    2013 WWW      Measuring personalization of web search.  -- screened out: both labels false
    2014 IMC      Measuring Price Discrimination and Steering on E-commerce Web Sites.  -- screened out: both labels false
    2014 PETS     Crying Wolf? On the Price Discrimination of Online Airline Tickets  -- no label record (venue-year gap)
    2021 WWW      Auditing for Discrimination in Algorithms Delivering Job Ads.  -- screened out: both labels false
    2016 NDSS     Do You See What I See? Differential Treatment of Anonymous Users  -- no label record (venue-year gap)
    2022 WWW      An Empirical Investigation of Personalization Factors on TikTok.  -- screened out: both labels false
    2016 WWW      MapWatch: Detecting and Monitoring International Border Personalization on Online Maps.  -- screened out: both labels false
 
## The audit set in full, with the sentence that settled the verdict
  2010 IMC      Challenges in measuring online advertising systems.
      IMC/2010/challenges-in-measuring-online-advertising-systems
      IN: seeded profile pairs: "enabled, but are seeded with different user personae" (the sentence naming the two interest sets is column-spliced in both renderings)
  2014 CCS      Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising.
      CCS/2014/your-online-interests-pwned-a-pollution-attack-against-targeted-advertising
      IN: blank profile vs polluted profile: "the polluter can impersonate a user with a blank profile ... and browse pages"
  2015 CCS      Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence.
      CCS/2015/sunlight-fine-grained-targeting-detection-at-scale-with-statistical-confidence
      IN: instrument paper: varies personal-data inputs one at a time "compared to a control group", with statistical confidence
  2015 IMC      Location, Location, Location: The Impact of Geolocation on Web Search Personalization.
      IMC/2015/location-location-location-the-impact-of-geolocation-on-web-search-personalizati
      IN: location as the treatment: "all other browser attributes were the same across treatments"
  2015 IMC      Peeking Beneath the Hood of Uber.
      IMC/2015/peeking-beneath-the-hood-of-uber
      IN: "We created 43 Uber accounts ... blanket a small geographic area with measurement points"
  2015 PETS     Automated Experiments on Ad Privacy Settings
      PETS/2015/automated-experiments-on-ad-privacy-settings
      IN: AdFisher: "We created an experimental group and a control group of agents"
  2016 USENIX   Tracing Information Flows Between Ad Exchanges Using Retargeted Ads
      USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads
      IN: "We train 90 personas by visiting popular e-commerce sites, and then crawl major publishers"
  2018 WWW      AdBudgetKiller: Online Advertising Budget Draining Attack.
      WWW/2018/adbudgetkiller-online-advertising-budget-draining-attack
      IN: "Starting from a fresh profile, the profile trainer produces" crafted browsing profiles
  2018 WWW      Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages.
      WWW/2018/auditing-the-personalization-and-composition-of-politically-related-search-engin
      IN: standard-vs-incognito paired SERPs: "our controls were paired within the individual"
  2019 PETS     Investigating sources of PII used in Facebook’s targeted advertising
      PETS/2019/investigating-sources-of-pii-used-in-facebook-s-targeted-advertising
      IN: "Take a Facebook account that we control (call it the control account) and the test PII"
  2019 WWW      Measuring Political Personalization of Google News Search.
      WWW/2019/measuring-political-personalization-of-google-news-search
      IN: "a 'sock puppet' auditing system in which a pair of fresh browser profiles" visits divergent pages then runs identical queries
  2020 PETS     Inferring Tracker-Advertiser Relationships in the Online Advertising Ecosystem using Header Bidding
      PETS/2020/inferring-tracker-advertiser-relationships-in-the-online-advertising-ecosystem-u
      IN: intent vs no-intent versions of each of 16 interest personas, with trackers selectively exposed (this PDF is column-spliced in both cols and norm renderings, so no contiguous quote was publishable)
  2020 WWW      Stop tracking me Bro! Differential Tracking of User Demographics on Hyper-Partisan Websites.
      WWW/2020/stop-tracking-me-bro-differential-tracking-of-user-demographics-on-hyper-partisa
      IN: "We create 9 carefully crafted personas representing different genders and age groups"
  2022 IMC      Measurement and analysis of implied identity in ad delivery optimization.
      IMC/2022/measurement-and-analysis-of-implied-identity-in-ad-delivery-optimization
      IN: "We ran 200 versions of this ad at the same time, all from the same account and with the same budget"
  2022 IMC      What factors affect targeting and bids in online advertising?: a field measurement study.
      IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement
      IN: "All participants were asked to visit the same websites to control for contextual targeting, in randomized order"
  2022 NDSS     HARPO: Learning to Subvert Online Behavioral Advertising
      NDSS/2022/auto-draft-209
      IN: HARPO: obfuscated vs unobfuscated personas measured against live profiling models
  2022 PETS     ATOM: Ad-network Tomography
      PETS/2022/atom-ad-network-tomography
      IN: "create a number of online user personas associated with specific interest groups" and gather ads while systematically blocking trackers
  2022 WWW      Left or Right: A Peek into the Political Biases in Email Spam Filtering Algorithms During US Election 2020.
      WWW/2022/left-or-right-a-peek-into-the-political-biases-in-email-spam-filtering-algorithm
      IN: "We created 102 email accounts" and compared spam placement across treatment and control affiliations
  2023 CCS      Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Aspects.
      CCS/2023/marketing-to-children-through-online-targeted-advertising-targeting-mechanisms-a
      IN: "We launch the six personas simultaneously by creating six Selenium" browsers
  2023 IMC      Tracking, Profiling, and Ad Targeting in the Alexa Echo Smart Speaker Ecosystem.
      IMC/2023/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem
      IN: "Each treatment persona is simulated by installing and interacting with skills ... By contrast, in the control persona, we do" not
  2023 PETS     A Utility-Preserving Obfuscation Approach for YouTube Recommendations
      PETS/2023/a-utility-preserving-obfuscation-approach-for-youtube-recommendations
      IN: "We deploy and evaluate De-Harpo's effectiveness on YouTube using 10,000 sock puppet based personas"
  2023 WWW      A Method to Assess and Explain Disparate Impact in Online Retailing.
      WWW/2023/a-method-to-assess-and-explain-disparate-impact-in-online-retailing
      IN: "Each observation ... is paired with a set of 'counter-factuals'" from neighbouring zip codes
  2024 PETS     Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?
      PETS/2024/opted-out-yet-tracked-are-regulations-enough-to-protect-your-privacy
      IN: "We then visit the filtered websites with user interest personas" -- 16 interest personas, two regions
  2024 WWW      TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds.
      WWW/2024/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation
      IN: "validate our results using a baseline generated from automated TikTok bots, as well as a randomized baseline"
  2025 IMC      From Voice to Ads: Auditing Commercial Smart Speakers for Targeted Advertising based on Voice Characteristics.
      IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba
      IN: "we used each of the 21 cloned voice models to train two puppets, which we refer to as 'paired puppets'"
  2025 PETS     Echoes of Privacy: Uncovering the Profiling Practices of Voice Assistants
      PETS/2025/echoes-of-privacy-uncovering-the-profiling-practices-of-voice-assistants
      IN: "each meticulously trained with a curated set of voice queries designed to simulate various user personas"
  2025 PETS     More and Scammier Ads: The Perils of YouTube's Ad Privacy Settings
      PETS/2025/more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings
      IN: "running controlled experiments with sock puppet accounts that emulate users watching YouTube videos"
  2025 PETS     Sheep's clothing, wolfish intent: Automated detection and evaluation of problematic 'allowed' advertisements
      PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic
      IN: "two separate crawlers were deployed: one for the Control group (without ABP) and another for the AccAds group"
  2025 USENIX   Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistants
      USENIX/2025/big-help-or-big-brother-auditing-tracking-profiling-and-personalization-in-gener
      IN: "Training phase involves browsing through 10 webpages - 2 pages per leaked attribute", then personalisation is measured
  2026 IEEE-SP  Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube.
      IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
      IN: "we use sock-puppet accounts to systematically browse and log the behavior of the sock-puppet to generate the ground truth"
  2026 PETS     Ad Personalization and Transparency in Mobile Ecosystems: A Comparative Analysis of Google's and Apple's EU App Stores
      PETS/2026/ad-personalization-and-transparency-in-mobile-ecosystems-a-comparative-analysis
      IN: "We construct accounts with specific parameters or interests, so-called personas, and measure the ads displayed to them"
  2026 WWW      Does This Button Work? Investigating YouTube's Ineffective User Controls.
      WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls
      IN: "depending on randomized assignment - triggered one of four native feedback signals to YouTube ... or no signal at all in the control group"
 
## Adjudicated and rejected
  2017 WWW      An Army of Me: Sockpuppets in Online Discussion Communities.
      OUT: observational study OF sockpuppets others created; no arm the authors control
  2019 WWW      Auditing the Partisanship of Google Search Snippets.
      OUT: audits snippet vs linked page; no identity treatment, no arms
  2020 WWW      Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook.
      OUT: volunteer ad donation; observational, no arms
  2020 USENIX   What Twitter Knows: Characterizing Ad Targeting Practices, User Perceptions, and Ad Explanations Through Users' Own Twitter Data
      OUT: users' own Twitter data; no arms
  2023 IEEE-SP  Collaborative Ad Transparency: Promises and Limitations.
      OUT: explicitly the non-persona alternative: "One method that does not use fake personas"
  2023 USENIX   Problematic Advertising and its Disparate Exposure on Facebook
      OUT: donated ad exposure from real users; disparity is observational
  2022 CCS      Cart-ology: Intercepting Targeted Advertising via Ad Network Identity Entanglement.
      OUT: attack on ad-network identity linking; no persona arms
  2022 CCS      Privacy Limitations of Interest-based Advertising on The Web: A Post-mortem Empirical Analysis of Google's FLoC.
      OUT: randomised control is a data permutation over a browsing panel, not a live-platform arm
  2024 IEEE-SP  Targeted and Troublesome: Tracking and Advertising on Children's Websites.
      OUT: fresh profile per page visit; the paper itself says "Future work could extend our method to incorporate personas"
  2024 PETS     Interest-disclosing Mechanisms for Advertising are Privacy-Exposing (not Preserving)
      OUT: Topics API analysed over real browsing histories, not persona arms
  2026 WWW      When Ads Become Profiles: Uncovering the Invisible Risk of Web Advertising at Scale with LLMs.
      OUT: "Random Control group" is a model ablation, not a platform arm
  2022 IEEE-SP  Deployment of Source Address Validation by Network Operators: A Randomized Control Trial.
      OUT: an RCT, but the treatment is a notification to operators; outcome is not platform output
 
## Screening loss in full
  2010 NDSS     Adnostic: Privacy Preserving Targeted Advertising
      no label record (venue-year gap)
  2010 WWW      Using a model of social dynamics to predict popularity of news.
      screened out: both labels false
  2012 CCS      Privacy-aware personalization for mobile advertising.
      screened out: both labels false
  2012 WWW      How effective is targeted advertising?
      screened out: both labels false
  2013 WWW      Measuring personalization of web search.
      screened out: both labels false
  2013 WWW      Spatio-temporal dynamics of online memes: a study of geo-tagged tweets.
      screened out: both labels false
  2014 IMC      Measuring Price Discrimination and Steering on E-commerce Web Sites.
      screened out: both labels false
  2014 PETS     Crying Wolf? On the Price Discrimination of Online Airline Tickets
      no label record (venue-year gap)
  2014 WWW      Quizz: targeted crowdsourcing with a billion (potential) users.
      screened out: both labels false
  2014 WWW      Mining novelty-seeking trait across heterogeneous domains.
      screened out: both labels false
  2014 WWW      Exploring the filter bubble: the effect of using recommender systems on content diversity.
      screened out: both labels false
  2014 WWW      Fast topic discovery from web search streams.
      screened out: both labels false
  2015 WWW      Events and Controversies: Influences of a Shocking News Event on Information Seeking.
      screened out: both labels false
  2016 NDSS     Do You See What I See? Differential Treatment of Anonymous Users
      no label record (venue-year gap)
  2016 USENIX   Micro-Virtualization Memory Tracing to Detect and Prevent Spraying Attacks
      screened out: both labels false
  2016 WWW      MapWatch: Detecting and Monitoring International Border Personalization on Online Maps.
      screened out: both labels false
  2018 IEEE-SP  FuturesMEX: Secure, Distributed Futures Market Exchange.
      screened out: both labels false
  2018 WWW      Modeling Interdependent and Periodic Real-World Action Sequences.
      screened out: both labels false
  2018 WWW      Me, My Echo Chamber, and I: Introspection on Social Media Polarization.
      screened out: both labels false
  2018 WWW      Political Discourse on Social Media: Echo Chambers, Gatekeepers, and the Price of Bipartisanship.
      screened out: both labels false
  2020 CCS      DECO: Liberating Web Data Using Decentralized Oracles for TLS.
      screened out: both labels false
  2020 IMC      Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services.
      screened out: both labels false
  2020 WWW      Architectures for Autonomy: Towards an Equitable Web of Data in the Age of AI.
      screened out: both labels false
  2021 USENIX   SIGL: Securing Software Installations Through Deep Graph Learning
      screened out: both labels false
  2021 WWW      Rabbit Holes and Taste Distortion: Distribution-Aware Recommendation with Evolving Interests.
      screened out: both labels false
  2021 WWW      Local Clustering in Contextual Multi-Armed Bandits.
      screened out: both labels false
  2021 WWW      Incrementality Testing in Programmatic Advertising: Enhanced Precision with Double-Blind Designs.
      screened out: both labels false
  2021 WWW      Causal Network Motifs: Identifying Heterogeneous Spillover Effects in A/B Tests.
      screened out: both labels false
  2021 WWW      Auditing for Discrimination in Algorithms Delivering Job Ads.
      screened out: both labels false
  2021 WWW      The Interaction between Political Typology and Filter Bubbles in News Recommendation Algorithms.
      screened out: both labels false
  2022 PETS     PUBA: Privacy-Preserving User-Data Bookkeeping and Analytics
      screened out: both labels false
  2022 WWW      An Empirical Investigation of Personalization Factors on TikTok.
      screened out: both labels false
  2023 PETS     Find Thy Neighbourhood: Privacy-Preserving Local Clustering
      screened out: both labels false
  2023 WWW      pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning.
      screened out: both labels false
  2023 WWW      Breaking Filter Bubble: A Reinforcement Learning Framework of Controllable Recommender System.
      screened out: both labels false
  2024 PETS     Evaluating Google's Protected Audience Protocol
      screened out: both labels false
  2024 WWW      Filter Bubble or Homogenization? Disentangling the Long-Term Effects of Recommendations on User Consumption Patterns.
      screened out: both labels false
  2024 WWW      Optimal Engagement-Diversity Tradeoffs in Social Media.
      screened out: both labels false
  2024 WWW      Learning Category Trees for ID-Based Recommendation: Exploring the Power of Differentiable Vector Quantization.
      screened out: both labels false
  2024 WWW      Full-stage Diversified Recommendation: Large-scale Online Experiments in Short-video Platform.
      screened out: both labels false
  2024 WWW      Uncovering the Deep Filter Bubble: Narrow Exposure in Short-Video Recommendation.
      screened out: both labels false
  2025 CCS      Cascading Adversarial Bias from Injection to Distillation in Language Models.
      screened out: both labels false
  2025 USENIX   Privacy Audit as Bits Transmission: (Im)possibilities for Audit by One Run
      screened out: both labels false
  2025 WWW      LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations.
      screened out: both labels false
  2025 WWW      SPRec: Self-Play to Debias LLM-based Recommendation.
      screened out: both labels false
  2026 PETS     Making Sense of Private Advertising: A Principled Approach to a Complex Ecosystem
      screened out: both labels false
  2026 WWW      DynaMoLTV: A Cross-Game Dynamic Mixture Model with Weighted Sub-Distributions for Player Lifetime Value Prediction.
      screened out: both labels false
  2026 WWW      Audit?of?Audits for the Web: Bayesian Meta?Evaluation that Yields Interval?Valued, Threshold?Aligned Fairness Claims.
      screened out: both labels false
OK: all contracts held.

quotecheck_algorithm_audits.mjs

#!/usr/bin/env node
// Verifies every quoted fragment used on design:algorithm_audits and on its
// provenance page against the paper's own text.
//
// Three matching modes, because a two-column PDF loses in different places in
// each rendering: (1) whitespace-collapsed exact, (2) hyphen/quote-normalised,
// (3) longest 8-word run. Each quote is tried against paper.cols.txt AND
// paper.norm.txt; a quote found in either is PASS, with the rendering recorded.
// Exits non-zero on any FAIL.
import fs from 'node:fs';
import path from 'node:path';
import { dataRoot } from './lib.mjs';
const ROOT = dataRoot();
 
const norm = (s) => s
  .replace(/\s+/g, ' ')
  .replace(/[‘’ʼ]/g, "'")
  .replace(/[“”]/g, '"')
  .replace(/[‐-―−]/g, '-')
  .trim();
const strip = (s) => norm(s).replace(/-\s*/g, '').toLowerCase();
 
function readModes(key) {
  const [venue, year, slug] = key.split('/');
  const out = {};
  for (const name of ['paper.cols.txt', 'paper.norm.txt']) {
    const f = path.join(ROOT, 'fulltext', year, venue, slug, name);
    if (fs.existsSync(f)) out[name] = fs.readFileSync(f, 'utf8');
  }
  if (Object.keys(out).length === 0) throw new Error(`no text for ${key}`);
  return out;
}
 
export function checkQuote(key, quote) {
  const modes = readModes(key);
  const q = norm(quote);
  const qs = strip(quote);
  const words = q.split(' ');
  for (const [name, raw] of Object.entries(modes)) {
    const t = norm(raw);
    if (t.includes(q)) return { ok: true, how: `exact in ${name}` };
    if (strip(raw).includes(qs)) return { ok: true, how: `hyphen/quote-normalised in ${name}` };
  }
  // longest 8-word run
  for (const [name, raw] of Object.entries(modes)) {
    const ts = strip(raw);
    let best = 0;
    for (let i = 0; i + 8 <= words.length; i++) {
      if (ts.includes(strip(words.slice(i, i + 8).join(' ')))) best++;
    }
    if (best > 0) return { ok: true, how: `${best} of ${Math.max(0, words.length - 7)} 8-word runs in ${name}` };
  }
  return { ok: false, how: 'NOT FOUND in cols or norm' };
}
 
// Quotes used on the two pages. Each entry: [paper key, quote as published].
export const QUOTES = [
  ['IMC/2010/challenges-in-measuring-online-advertising-systems',
    'Even queries launched simultaneously from two identically configured clients on the same subnet can produce wildly different ads over multiple timescales.'],
  ['IMC/2010/challenges-in-measuring-online-advertising-systems',
    'In this paper we perform all analysis relative to a control experiment'],
  ['IMC/2010/challenges-in-measuring-online-advertising-systems',
    'enabled, but are seeded with different user personae'],
  ['PETS/2015/automated-experiments-on-ad-privacy-settings',
    'We created an experimental group and a control group of agents.'],
  ['PETS/2015/automated-experiments-on-ad-privacy-settings',
    'The browser agents in the experimental group visited websites on substance abuse while the agents in the control group simply waited.'],
  ['WWW/2019/measuring-political-personalization-of-google-news-search',
    'we develop a "sock puppet" auditing system in which a pair of fresh browser profiles, first, visits web pages that reflect divergent political discourses and, second, executes identical politically oriented Google News searches'],
  ['WWW/2018/auditing-the-personalization-and-composition-of-politically-related-search-engin',
    'our controls were paired within the individual, enabling us to isolate the impact that their browser mode had on their search rankings for each query we searched'],
  ['IMC/2015/location-location-location-the-impact-of-geolocation-on-web-search-personalizati',
    'all other browser attributes were the same across treatments, so each treatment should present an identical browser fingerprint'],
  ['IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba',
    'We assigned voices randomly to days, and scheduled paired puppets in parallel.'],
  ['IMC/2023/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem',
    'By contrast, in the control persona, we do'],
  ['PETS/2025/more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings',
    'our methodology consists of running controlled experiments with sock puppet accounts that emulate users watching YouTube videos in an instrumented browser'],
  ['WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls',
    'depending on randomized assignment'],
  ['WWW/2022/left-or-right-a-peek-into-the-political-biases-in-email-spam-filtering-algorithm',
    'We created 102 email accounts'],
  ['IMC/2022/measurement-and-analysis-of-implied-identity-in-ad-delivery-optimization',
    'We ran 200 versions of this ad at the same time, all from the same account and with the same budget'],
  ['WWW/2020/stop-tracking-me-bro-differential-tracking-of-user-demographics-on-hyper-partisa',
    'We create 9 carefully crafted personas representing different genders and age groups.'],
  ['IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right',
    'we use sock-puppet accounts to system- atically browse and log the behavior of the sock-puppet to generate the ground truth'],
  ['PETS/2026/ad-personalization-and-transparency-in-mobile-ecosystems-a-comparative-analysis',
    'We construct accounts with specific parameters or interests, so-called personas, and measure the ads displayed to them'],
  ['IMC/2015/peeking-beneath-the-hood-of-uber',
    'We created 43 Uber accounts'],
  ['IEEE-SP/2023/collaborative-ad-transparency-promises-and-limitations',
    'One method that does not use fake personas'],
  ['IEEE-SP/2024/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites',
    'Future work could extend our method to incorporate personas and warmup crawls to study such ads.'],
  ['PETS/2023/a-utility-preserving-obfuscation-approach-for-youtube-recommendations',
    'We deploy and evaluate De-Harpo'],
  ['USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads',
    'We train 90 personas by visiting popular e-commerce sites, and then crawl major publishers to gather retargeted ads'],
  ['WWW/2024/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation',
    'validate our results using a baseline generated from automated TikTok bots, as well as a randomized baseline'],
  ['CCS/2015/sunlight-fine-grained-targeting-detection-at-scale-with-statistical-confidence',
    'prior studies conduct tightly controlled experiments that vary personal data inputs (such as location, search terms, or profile interests) one at a time and observe the effect on service outputs (such as ads, recommendations, or prices) compared to a control group'],
  ['WWW/2022/using-survival-models-to-estimate-user-engagement-in-online-experiments',
    'We simulate A/A tests by re-randomizing the treatment assignments on the observed exposure logs from our experiment corpus.'],
  ['PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits',
    'We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows'],
  ['PETS/2024/opted-out-yet-tracked-are-regulations-enough-to-protect-your-privacy',
    'we also conduct Bonferroni correction on the statistical test'],
  ['IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement',
    'Most commonly, web crawlers with synthetic profiles or personas are used to measure behavioral targeting and contextual targeting.'],
];
 
if (import.meta.url === `file://${process.argv[1]}`) {
  let fail = 0;
  for (const [key, q] of QUOTES) {
    let r;
    try { r = checkQuote(key, q); } catch (e) { r = { ok: false, how: e.message }; }
    if (!r.ok) fail++;
    console.log(`${r.ok ? 'PASS' : 'FAIL'}  ${key}\n        ${r.how}\n        "${q.slice(0, 110)}${q.length > 110 ? '…' : ''}"`);
  }
  console.log(`\n${QUOTES.length} quotes checked, ${fail} failed.`);
  process.exit(fail ? 1 : 0);
}

quotecheck_algorithm_audits-output.txt

PASS  IMC/2010/challenges-in-measuring-online-advertising-systems
        exact in paper.cols.txt
        "Even queries launched simultaneously from two identically configured clients on the same subnet can produce wi…"
PASS  IMC/2010/challenges-in-measuring-online-advertising-systems
        exact in paper.cols.txt
        "In this paper we perform all analysis relative to a control experiment"
PASS  IMC/2010/challenges-in-measuring-online-advertising-systems
        exact in paper.cols.txt
        "enabled, but are seeded with different user personae"
PASS  PETS/2015/automated-experiments-on-ad-privacy-settings
        exact in paper.cols.txt
        "We created an experimental group and a control group of agents."
PASS  PETS/2015/automated-experiments-on-ad-privacy-settings
        exact in paper.cols.txt
        "The browser agents in the experimental group visited websites on substance abuse while the agents in the contr…"
PASS  WWW/2019/measuring-political-personalization-of-google-news-search
        exact in paper.cols.txt
        "we develop a "sock puppet" auditing system in which a pair of fresh browser profiles, first, visits web pages …"
PASS  WWW/2018/auditing-the-personalization-and-composition-of-politically-related-search-engin
        exact in paper.cols.txt
        "our controls were paired within the individual, enabling us to isolate the impact that their browser mode had …"
PASS  IMC/2015/location-location-location-the-impact-of-geolocation-on-web-search-personalizati
        exact in paper.cols.txt
        "all other browser attributes were the same across treatments, so each treatment should present an identical br…"
PASS  IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba
        exact in paper.cols.txt
        "We assigned voices randomly to days, and scheduled paired puppets in parallel."
PASS  IMC/2023/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem
        exact in paper.cols.txt
        "By contrast, in the control persona, we do"
PASS  PETS/2025/more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings
        exact in paper.cols.txt
        "our methodology consists of running controlled experiments with sock puppet accounts that emulate users watchi…"
PASS  WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls
        exact in paper.cols.txt
        "depending on randomized assignment"
PASS  WWW/2022/left-or-right-a-peek-into-the-political-biases-in-email-spam-filtering-algorithm
        exact in paper.cols.txt
        "We created 102 email accounts"
PASS  IMC/2022/measurement-and-analysis-of-implied-identity-in-ad-delivery-optimization
        exact in paper.cols.txt
        "We ran 200 versions of this ad at the same time, all from the same account and with the same budget"
PASS  WWW/2020/stop-tracking-me-bro-differential-tracking-of-user-demographics-on-hyper-partisa
        exact in paper.cols.txt
        "We create 9 carefully crafted personas representing different genders and age groups."
PASS  IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
        exact in paper.cols.txt
        "we use sock-puppet accounts to system- atically browse and log the behavior of the sock-puppet to generate the…"
PASS  PETS/2026/ad-personalization-and-transparency-in-mobile-ecosystems-a-comparative-analysis
        exact in paper.cols.txt
        "We construct accounts with specific parameters or interests, so-called personas, and measure the ads displayed…"
PASS  IMC/2015/peeking-beneath-the-hood-of-uber
        exact in paper.cols.txt
        "We created 43 Uber accounts"
PASS  IEEE-SP/2023/collaborative-ad-transparency-promises-and-limitations
        exact in paper.cols.txt
        "One method that does not use fake personas"
PASS  IEEE-SP/2024/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites
        exact in paper.cols.txt
        "Future work could extend our method to incorporate personas and warmup crawls to study such ads."
PASS  PETS/2023/a-utility-preserving-obfuscation-approach-for-youtube-recommendations
        exact in paper.cols.txt
        "We deploy and evaluate De-Harpo"
PASS  USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads
        exact in paper.cols.txt
        "We train 90 personas by visiting popular e-commerce sites, and then crawl major publishers to gather retargete…"
PASS  WWW/2024/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation
        exact in paper.cols.txt
        "validate our results using a baseline generated from automated TikTok bots, as well as a randomized baseline"
PASS  CCS/2015/sunlight-fine-grained-targeting-detection-at-scale-with-statistical-confidence
        exact in paper.cols.txt
        "prior studies conduct tightly controlled experiments that vary personal data inputs (such as location, search …"
PASS  WWW/2022/using-survival-models-to-estimate-user-engagement-in-online-experiments
        exact in paper.cols.txt
        "We simulate A/A tests by re-randomizing the treatment assignments on the observed exposure logs from our exper…"
PASS  PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits
        exact in paper.cols.txt
        "We design and deploy an LLM-driven auditing agent capable of end-to-end traversal of rights-request workflows"
PASS  PETS/2024/opted-out-yet-tracked-are-regulations-enough-to-protect-your-privacy
        exact in paper.cols.txt
        "we also conduct Bonferroni correction on the statistical test"
PASS  IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement
        exact in paper.cols.txt
        "Most commonly, web crawlers with synthetic profiles or personas are used to measure behavioral targeting and c…"
 
28 quotes checked, 0 failed.

Eight phrasings for the same idea, run over all 5,855 papers with text. The narrow term returns 2; the widened probe returns 11 of 32 within the audit set.

_aa_noise.mjs

// Concept-level probe: does the paper establish a same-treatment baseline
// (what an A/A test is), under ANY name?
import fs from 'node:fs'; import path from 'node:path';
import { dataRoot, loadExtractions } from './lib.mjs';
const ROOT=dataRoot();
const TERMS={
 'A/A test': /\bA\/A[ -]?(?:test|experiment)/i,
 'control-control / null experiment': /\bcontrol[- ]control\b|\bnull experiment/i,
 'noise floor': /\bnoise floor\b/i,
 'identical/identically configured arms': /\bidentical(?:ly)? (?:configured |trained |seeded )?(?:client|browser|profile|persona|account|agent|machine|instance)s?\b/i,
 'permutation / randomisation test': /\bpermutation test|\brandomi[sz]ation test\b/i,
 'null distribution': /\bnull distribution\b/i,
 'baseline noise / measurement noise': /\b(?:baseline|measurement|inherent|background) noise\b/i,
 'two arms with the same treatment': /\bsame treatment\b|\bno[- ]?treatment (?:arm|group|control)\b/i,
};
const papers=loadExtractions();
import { AUDITS, REJECTED } from './algorithm_audits_set.mjs';
const AUDIT=new Set(AUDITS.map(a=>a[0]));
const REJ=new Set(REJECTED.map(a=>a[0]));
if ([...AUDIT].some(k=>REJ.has(k))) throw new Error('a key is in both AUDITS and REJECTED');
const inAudit=k=>AUDIT.has(k);
const tot={},aud={}; const per=new Map();
for(const p of papers){
  const k=`${p.venue}/${p.year}/${p.slug}`;
  const f=path.join(ROOT,'fulltext',String(p.year),p.venue,p.slug,'paper.cols.txt');
  if(!fs.existsSync(f))continue;
  const t=fs.readFileSync(f,'utf8').replace(/\s+/g,' ');
  for(const [n,re] of Object.entries(TERMS)) if(re.test(t)){
    tot[n]=(tot[n]||0)+1;
    if(inAudit(k)){aud[n]=(aud[n]||0)+1; if(!per.has(k))per.set(k,[]); per.get(k).push(n);}
  }
}
console.log(`AUDIT keys parsed from report script: ${AUDIT.size}`);
console.log(`${'term'.padEnd(38)} ${'corpus'.padStart(7)} ${'audit'.padStart(6)}`);
for(const n of Object.keys(TERMS)) console.log(`${n.padEnd(38)} ${String(tot[n]||0).padStart(7)} ${String(aud[n]||0).padStart(6)}`);
console.log(`\naudit papers with >=1 noise-baseline term: ${per.size} of ${AUDIT.size}`);
for(const [k,v] of [...per].sort()) console.log(`  ${k}\n     ${v.join(', ')}`);
console.log('\naudit papers with NONE:');
for(const k of AUDIT) if(!per.has(k)) console.log(`  ${k}`);

_aa_noise-output.txt

AUDIT keys parsed from report script: 32
term                                    corpus  audit
A/A test                                     2      0
control-control / null experiment           31      1
noise floor                                 35      2
identical/identically configured arms       22      5
permutation / randomisation test            32      4
null distribution                            1      0
baseline noise / measurement noise         219      2
two arms with the same treatment            13      1
 
audit papers with >=1 noise-baseline term: 11 of 32
  CCS/2015/sunlight-fine-grained-targeting-detection-at-scale-with-statistical-confidence
     permutation / randomisation test
  IMC/2010/challenges-in-measuring-online-advertising-systems
     control-control / null experiment, identical/identically configured arms
  IMC/2015/location-location-location-the-impact-of-geolocation-on-web-search-personalizati
     noise floor, identical/identically configured arms
  IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba
     permutation / randomisation test, baseline noise / measurement noise
  PETS/2015/automated-experiments-on-ad-privacy-settings
     identical/identically configured arms, permutation / randomisation test
  PETS/2025/more-and-scammier-ads-the-perils-of-youtubes-ad-privacy-settings
     identical/identically configured arms
  PETS/2026/ad-personalization-and-transparency-in-mobile-ecosystems-a-comparative-analysis
     permutation / randomisation test
  WWW/2018/auditing-the-personalization-and-composition-of-politically-related-search-engin
     noise floor
  WWW/2019/measuring-political-personalization-of-google-news-search
     identical/identically configured arms
  WWW/2022/left-or-right-a-peek-into-the-political-biases-in-email-spam-filtering-algorithm
     two arms with the same treatment
  WWW/2024/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation
     baseline noise / measurement noise
 
audit papers with NONE:
  CCS/2014/your-online-interests-pwned-a-pollution-attack-against-targeted-advertising
  IMC/2015/peeking-beneath-the-hood-of-uber
  USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads
  WWW/2018/adbudgetkiller-online-advertising-budget-draining-attack
  PETS/2019/investigating-sources-of-pii-used-in-facebook-s-targeted-advertising
  PETS/2020/inferring-tracker-advertiser-relationships-in-the-online-advertising-ecosystem-u
  WWW/2020/stop-tracking-me-bro-differential-tracking-of-user-demographics-on-hyper-partisa
  IMC/2022/measurement-and-analysis-of-implied-identity-in-ad-delivery-optimization
  IMC/2022/what-factors-affect-targeting-and-bids-in-online-advertising-a-field-measurement
  NDSS/2022/auto-draft-209
  PETS/2022/atom-ad-network-tomography
  CCS/2023/marketing-to-children-through-online-targeted-advertising-targeting-mechanisms-a
  IMC/2023/tracking-profiling-and-ad-targeting-in-the-alexa-echo-smart-speaker-ecosystem
  PETS/2023/a-utility-preserving-obfuscation-approach-for-youtube-recommendations
  WWW/2023/a-method-to-assess-and-explain-disparate-impact-in-online-retailing
  PETS/2024/opted-out-yet-tracked-are-regulations-enough-to-protect-your-privacy
  PETS/2025/echoes-of-privacy-uncovering-the-profiling-practices-of-voice-assistants
  PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic
  USENIX/2025/big-help-or-big-brother-auditing-tracking-profiling-and-personalization-in-gener
  IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
  WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls

• algorithm_audits — the page this log is for.
• corpus — the corpus, its funnel and its provisional years.
• stateful_stateless — the neighbouring log; its 29-paper comparison-study audit is the closest methodological precedent for the hand-adjudication done here.
• platforms — where the corpus-wide “sock puppet” count also appears, against a different denominator.

[1]

Meng, Wei; Xing, Xinyu; Sheth, Anmol; Weinsberg, Udi; Lee, Wenke (2014): "Your Online Interests: Pwned! A Pollution Attack Against Targeted Advertising", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[2]

Kim, I Luk; Wang, Weihang; Kwon, Yonghwi; Zheng, Yunhui; Aafer, Yousra; Meng, Weijie; Zhang, Xiangyu (2018): "AdBudgetKiller: Online Advertising Budget Draining Attack", in: Proceedings of the ACM Web Conference. (DOI)

[3]

Zhang, Jiang; Psounis, Konstantinos; Haroon, Muhammad; Shafiq, Zubair (2022): "HARPO: Learning to Subvert Online Behavioral Advertising", in: Proceedings of the Network and Distributed System Security Symposium. (Link)

[4]

Lécuyer, Mathias; Spahn, Riley; Spiliopolous, Yannis; Chaintreau, Augustin; Geambasu, Roxana; Hsu, Daniel J. (2015): "Sunlight: Fine-grained Targeting Detection at Scale with Statistical Confidence", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[5]

Datta, Amit; Tschantz, Michael Carl; Datta, Anupam (2015): "Automated Experiments on Ad Privacy Settings", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[6]

Silva, Márcio; Oliveira, Lucas Santos de; Andreou, Athanasios; Melo, Pedro Olmo Stancioli Vaz de; Goga, Oana; Benevenuto, Fabrício (2020): "Facebook Ads Monitor: An Independent Auditing System for Political Ads on Facebook", in: Proceedings of the ACM Web Conference. (DOI)

[7]

Ali, Muhammad; Goetzen, Angelica; Mislove, Alan; Redmiles, Elissa M.; Sapiezynski, Piotr (2023): "Problematic Advertising and its Disparate Exposure on Facebook", in: Proceedings of the USENIX Security Symposium. (Link)

[8]

Lone, Qasim; Frik, Alisa; Luckie, Matthew; Korczyński, Maciej; van Eeten, Michel; Gañán, Carlos (2022): "Deployment of Source Address Validation by Network Operators: A Randomized Control Trial", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[9]

Cook, John; Nithyanand, Rishab; Shafiq, Zubair (2020): "Inferring Tracker-Advertiser Relationships in the Online Advertising Ecosystem using Header Bidding", in: Proceedings on Privacy Enhancing Technologies. (DOI)

[10]

Agarwal, Pushkal; Joglekar, Sagar; Papadopoulos, Panagiotis; Sastry, Nishanth; Kourtellis, Nicolas (2020): "Stop tracking me Bro! Differential Tracking of User Demographics on Hyper-Partisan Websites", in: Proceedings of the ACM Web Conference. (DOI)

[11]

Iqbal, Hassan; Khan, Usman Mahmood; Khan, Hassan Ali; Shahzad, Muhammad (2022): "Left or Right: A Peek into the Political Biases in Email Spam Filtering Algorithms During US Election 2020", in: Proceedings of the ACM Web Conference. (DOI)

[12]

Guha, Saikat; Cheng, Bin; Francis, Paul (2010): "Challenges in measuring online advertising systems", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[13]

Chen, Le; Mislove, Alan; Wilson, Christo (2015): "Peeking Beneath the Hood of Uber", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[14]

Becerril-Arreola, Rafael (2023): "A Method to Assess and Explain Disparate Impact in Online Retailing", in: Proceedings of the ACM Web Conference. (DOI)