Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: privacy:data_subject_rights ======

Working notes behind **[[privacy:data_subject_rights]]** — every query, the scripts and their unedited output, the hand audit of all 471 candidates, the quote checks, the external sources including the ones that were rejected, and what could not be established. Corpus-level caveats (venue scope, selection funnel, the provisional 2025–2026 years, extraction stability) are on [[Literature:Corpus]] and are not repeated here.

Written 2026-09-16 against ''data/extract/run1'' of the ''pdf_fetcher'' corpus: **5,859** extracted papers, **5,855** with full text on disk, seven venues, 2010–2026.

Comments belong on the content page. Per the convention set by the first of these pages, provenance pages carry no ''%%~~DISCUSSION~~%%'' block.

===== 1. The scripts =====

^ File ^ What it does ^ Fails loudly when ^
| ''scripts/dsr_probe.mjs'' | fifteen full-text probes over the 5,855 papers with full text; prints per-probe counts, the union, and a per-year table | a tight probe is not a subset of its loose pair |
| ''scripts/dsr_verdicts_build.py'' | turns the hand audit into ''scripts/dsr_verdicts.json'' | any candidate has no verdict, two verdicts, or is both IN and OUT |
| ''scripts/dsr_report.mjs'' | every figure on the page, each with its denominator | a request-sending paper is missing from the hand-keyed //controllers asked// table; the instrument categories do not sum to the population; a verdict names a paper that is not in the corpus |
| ''scripts/dsr_ctxdump.mjs'' | prints, for every candidate, the sentence each core probe matched — the input to the audit | — |
| ''scripts/dsr_quotecheck.py'' | checks every ''%%//"..."//%%'' span on the page against its source | a quote is not found, or the page carries a quoted span with no source listed |

===== 2. The probes =====

There is no enum in the extraction schema for data-subject rights, so the candidate set is a full-text sweep. Fifteen probes, each with a **tight** pattern (what the count comes from) and a **loose** superset used only to prove the tight one is a narrowing of the same idea; the script throws if the tight set is not contained in the loose one. The guard fired twice while the probes were being written — ''portability'' and ''authorized_agent'' both had tight alternations their loose pattern did not cover — which is exactly what it is for.

Probes are split into **core** (a rights request or an opt-out signal is the thing being named) and **context** (the surrounding law or a predecessor signal, too noisy to drive a candidate set but useful for recall checks). The candidate set is the union of the core probes.

<file text dsr_probe-output.txt>
corpus: 5859 extracted papers; 5855 with full text on disk (4 without)

probe	tier	tight	loose	what
dsar_phrase	core	43	168	a request for a copy of one’s own data
erasure	core	107	1862	a request to have data erased
gpc	core	18	34	the Global Privacy Control signal
do_not_sell	core	56	242	the CCPA/CPRA "Do Not Sell" opt-out surface
dnt	context	69	119	the Do Not Track header (GPC’s predecessor)
portability	context	44	1156	a data-export / portability request
ccpa_law	context	240	262	the California statutes that create the opt-out right
gdpr_article	context	26	145	the GDPR data-subject-rights articles, named
authorized_agent	context	26	1709	the CCPA authorised-agent / verifiable-request machinery
request_sent	core	99	4508	the paper says a request was SENT — the instrument, not the topic
account_deletion	core	62	5223	account deletion as the user-facing form of the erasure right
broker_optout	core	32	1266	data-broker / people-search opt-out and removal services
optout_tool	core	156	423	an industry or site opt-out tool (NAI, DAA/AdChoices, an opt-out page)
self_service	core	117	4552	the in-product self-service surface people actually use (dashboard, export)
usp_string	core	26	515	the IAB US Privacy / Global Privacy Platform opt-out strings

UNION, core probes only: 471 papers (8.0% of 5855 with full text)
UNION, all probes:       623 papers (10.6%)
context-only candidates (no core probe fired): 152

year	candidates	corpus papers	share
2010	3	119	2.5%
2011	6	116	5.2%
2012	5	151	3.3%
2013	4	125	3.2%
2014	1	166	0.6%
2015	8	190	4.2%
2016	18	182	9.9%
2017	21	231	9.1%
2018	16	254	6.3%
2019	35	402	8.7%
2020	26	404	6.4%
2021	46	379	12.1%
2022	66	546	12.1%
2023	74	719	10.3%
2024	107	690	15.5%
2025	120	770	15.6%
2026	67	415	16.1%
</file>

Three probe defects worth recording, because each one produced a wrong number before it was caught:

  * **''\bGPC\b'' is a homonym.** NVIDIA calls a GPU subunit a //Graphics Processing Cluster//, so a case-sensitive word-boundary match returned 27 papers of which four were GPU side-channel and TEE work. Requiring either the spelled-out //Global Privacy Control// or ''GPC'' within 120 characters of a privacy-signal word takes it to 18, all real.
  * **The first "did this paper send requests?" probe matched 3 papers.** It required //sent/submitted/filed// and a request noun in one clause. It missed every DSAR study in the corpus, including //Access Denied!//, //Leave No Data Behind// and //Revisiting Identification Issues//. Six alternations, each added after reading what the previous one missed, take it to 99. A narrow probe here is a recall claim, and the narrow one was false.
  * **Two probes were added after the first audit was finished.** ''optout_tool'' and ''self_service'' were written because //Do Opt-Outs Really Opt Me Out?// (CCS 2022) — a paper the page is partly about — fired no core probe at all: the industry opt-out tools it measures are never called "do not sell". The two probes added **182 candidates** and **10 population papers**, taking the population from 41 to 51. That is a 20% under-count that survived a complete hand audit of 289 candidates, and there is no reason to think a third blind spot of the same kind does not exist.

===== 3. The inclusion rule, and the audit of all 471 candidates =====

The rule is on the content page and is repeated here so the verdict file can be read against it: a paper is in if it reports an empirical result produced by, or about, a data-subject-rights request or a machine-readable opt-out signal, in one of five ways — **R** requests sent to real controllers, **S** an opt-out signal or setting measured across a population, **M** a rights mechanism's availability/correctness/usability measured across a population, **D** research data obtained through an access or export surface, **O** an observational study of requests other people sent.

Each of the 471 candidates was judged from **the sentence its probes matched** (printed by ''dsr_ctxdump.mjs'') **plus the title**. For the 51 that went in, and for roughly 25 borderline cases, the paper's abstract and methodology section were also read. **The other ~395 verdicts rest on one matched sentence plus the title** — that is a screen, not a reading, and the content page's phrase "hand-auditing all 471" should be read that way. **No verdict was defaulted** — ''dsr_verdicts_build.py'' throws if any candidate is unassigned, precisely so that a canned reason cannot pass itself off as per-paper evidence.

^ Verdict ^ N ^ Share of 471 ^
| **IN** | 51 | 10.8% |
| OUT — ''homonym'': the probe matched an unrelated sense (an HTTP request, NVIDIA's ''GPC'', "do not sell" of goods or domains, DNS/cache/stream deletion) | 129 | 27.4% |
| OUT — ''background'': names a right in motivation, related work or a reference list; the paper's object is something else | 108 | 22.9% |
| OUT — ''research-ethics'': the paper //received// or honoured requests about its own study, almost always an internet-scanning paper offering an "opt-out request" for its own probes | 53 | 11.3% |
| OUT — ''attitudes'': survey or interview on awareness or perception, no request exercised and no mechanism observed | 44 | 9.3% |
| OUT — ''policy-text'': measures what a policy or store declaration //says// about rights → [[Privacy:Policies]] | 34 | 7.2% |
| OUT — ''consent'': consent banner or consent withdrawal → [[Privacy:Consent]] | 11 | 2.3% |
| OUT — ''unlearning'': right to be forgotten as a machine-learning problem | 10 | 2.1% |
| OUT — ''system'': proposes or builds a rights mechanism, measures no deployment | 10 | 2.1% |
| OUT — ''developer-side'': studies the controller's implementation side, not the subject's request | 8 | 1.7% |
| OUT — ''takedown'': content removal, delisting, blocklist or abuse removal | 7 | 1.5% |
| OUT — ''sok'': systematisation; worth reading, measures nothing itself | 3 | 0.6% |
| OUT — ''enforcement'': regulator decisions and fines → [[Practices:Legal enforcement]] | 2 | 0.4% |
| OUT — ''agent-benchmark'': the object measured is an LLM agent's capability and a rights task is one item in a wider benchmark | 1 | 0.2% |

The complete verdict map. One row per candidate, with the probes that fired, so any rejection can be checked against the reason:

<file text dsr_verdicts.tsv>
verdict	category_or_reason	year	venue	probes_fired	title
OUT	homonym	2010	IMC	request_sent	Netalyzr: illuminating the edge network.
OUT	homonym	2010	IMC	self_service	What happened in my network: mining network events from router syslogs.
OUT	homonym	2010	USENIX	self_service	SEPIA: Privacy-Preserving Aggregation of Multi-Domain Network Events and Statistics
OUT	background	2011	IEEE-SP	optout_tool	RePriv: Re-imagining Content Personalization and In-browser Privacy.
OUT	homonym	2011	IMC	request_sent,self_service	Measurement and analysis of a large scale commercial mobile internet TV system.
OUT	homonym	2011	IMC	self_service	Over the top video: the gorilla in cellular networks.
OUT	homonym	2011	IMC	self_service	YouTube everywhere: impact of device and infrastructure synergies on user experience.
OUT	background	2012	CCS	dnt,optout_tool	Non-tracking web analytics.
OUT	sok	2012	IEEE-SP	dnt,optout_tool	Third-Party Web Tracking: Policy and Technology.
OUT	homonym	2012	USENIX	do_not_sell	PharmaLeaks: Understanding the Business of Online Pharmaceutical Affiliate Programs
OUT	homonym	2012	USENIX	self_service	Throttling Tor Bandwidth Parasites
OUT	background	2013	CCS	optout_tool	A footprint of third-party tracking on mobile web.
OUT	takedown	2013	WWW	request_sent,account_deletion	The role of web hosting providers in detecting compromised websites.
IN	S	2014	CCS	dnt,optout_tool	The Web Never Forgets: Persistent Tracking Mechanisms in the Wild.
OUT	research-ethics	2015	CCS	optout_tool,self_service	A Search Engine Backed by Internet-Wide Scanning.
OUT	homonym	2015	IMC	do_not_sell	From .academy to .zone: An Analysis of the New TLD Land Rush.
OUT	background	2015	IMC	erasure	Location, Location, Location: The Impact of Geolocation on Web Search Personalization.
IN	S	2015	PETS	self_service	Automated Experiments on Ad Privacy Settings
OUT	homonym	2015	USENIX	request_sent	Cookies Lack Integrity: Real-World Implications
OUT	background	2015	WWW	dnt,optout_tool	PriVaricator: Deceiving Fingerprinters with Little White Lies.
OUT	homonym	2016	CCS	account_deletion	POSTER: Mapping the Landscape of Large-Scale Vulnerability Notifications.
OUT	homonym	2016	IEEE-SP	account_deletion	The Cracked Cookie Jar: HTTP Cookie Hijacking and the Exposure of Private Information.
OUT	background	2016	IMC	optout_tool	"Recommended For You": A First Look at Content Recommendation Networks.
OUT	research-ethics	2016	IMC	optout_tool	Yarrp'ing the Internet: Randomized High-Speed Active Topology Discovery.
OUT	research-ethics	2016	IMC	optout_tool	Zone Poisoning: The How and Where of Non-Secure DNS Dynamic Updates.
OUT	attitudes	2016	PETS	dsar_phrase,erasure,request_sent	Access Denied! Contrasting Data Access in the United States and Ireland
OUT	background	2016	PETS	optout_tool	Are You Sure You Want to Contact Us? Quantifying the Leakage of PII via Website Contact Forms
OUT	attitudes	2016	PETS	dnt,optout_tool	(Do Not) Track Me Sometimes: Users’ Contextual Preferences for Web Tracking
OUT	homonym	2016	PETS	usp_string	Flying Eyes and Hidden Controllers: A Qualitative Study of People’s Privacy Perceptions of Civilian Drones in The US
OUT	policy-text	2016	PETS	dsar_phrase	Privacy Challenges in the Quantified Self Movement – An EU Perspective
OUT	background	2016	PETS	optout_tool	Privacy vs. Reward in Indoor Location-Based Services
OUT	background	2016	PETS	account_deletion	Tales from the Dark Side: Privacy Dark Strategies and Privacy Dark Patterns
OUT	homonym	2016	PETS	self_service	The Curious Case of the PDF Converter that Likes Mozart: Dissecting and Mitigating the Privacy Risk of Personal Cloud Apps
IN	O	2016	PETS	erasure,authorized_agent,request_sent,broker_optout	The Right to be Forgotten in the Media: A Data-Driven Study
OUT	research-ethics	2016	USENIX	optout_tool	You've Got Vulnerability: Exploring Effective Vulnerability Notifications
OUT	homonym	2017	CCS	request_sent	Quantifying the Pressure of Legal Risks on Third-party Vulnerability Research.
OUT	research-ethics	2017	IEEE-SP	optout_tool	Augur: Internet-Wide Detection of Connectivity Disruptions.
OUT	homonym	2017	IEEE-SP	account_deletion	Obstacles to the Adoption of Secure Communication Tools.
OUT	homonym	2017	IMC	self_service	Email typosquatting.
OUT	homonym	2017	IMC	account_deletion	Fifteen minutes of unwanted fame: detecting and characterizing doxing.
OUT	background	2017	IMC	optout_tool	If you are not paying for it, you are the product: how much do advertisers pay to reach you?
OUT	research-ethics	2017	IMC	optout_tool	Large-scale scanning of TCP's initial window.
OUT	research-ethics	2017	IMC	optout_tool	Target generation for internet-wide IPv6 scanning.
OUT	background	2017	IMC	dnt,optout_tool	The ad wars: retrospective measurement and analysis of anti-adblock filter lists.
OUT	homonym	2017	NDSS	do_not_sell	Are We There Yet? On RPKI’s Deployment and Security
OUT	homonym	2017	NDSS	self_service	Catching Worms, Trojan Horses and PUPs: Unsupervised Detection of Silent Delivery Campaigns
OUT	homonym	2017	NDSS	self_service	Dissecting Tor Bridges: a Security Evaluation of their Private and Public Infrastructures
OUT	homonym	2017	PETS	usp_string	Analyzing Remote Server Locations for Personal Data Transfers in Mobile Apps
OUT	background	2017	PETS	dnt,broker_optout,optout_tool	Cross-Device Tracking: Measurement and Disclosures
OUT	background	2017	PETS	optout_tool	On the Privacy and Security of the Ultrasound Ecosystem
OUT	homonym	2017	USENIX	request_sent	A Longitudinal, End-to-End View of the DNSSEC Ecosystem
OUT	background	2017	USENIX	dnt,optout_tool	A Privacy Analysis of Cross-device Tracking
OUT	homonym	2017	USENIX	do_not_sell,request_sent	Game of Registrars: An Empirical Analysis of Post-Expiration Domain Name Takeovers
OUT	homonym	2018	CCS	self_service	Medical Devices are at Risk: Information Security on Diagnostic Imaging System.
OUT	homonym	2018	CCS	request_sent	Rendered Insecure: GPU Side Channel Attacks are Practical.
OUT	background	2018	IMC	optout_tool	How Tracking Companies Circumvented Ad Blockers Using WebSockets.
OUT	research-ethics	2018	IMC	optout_tool	In the IP of the Beholder: Strategies for Active IPv6 Topology Discovery.
OUT	homonym	2018	IMC	request_sent	Where The Light Gets In: Analyzing Web Censorship Mechanisms in India.
OUT	homonym	2018	PETS	self_service	Feature Selection for Website Fingerprinting
OUT	homonym	2018	PETS	usp_string	Panoptispy: Characterizing Audio and Video Exfiltration from Android Applications
OUT	background	2018	PETS	optout_tool	Privacy-preserving Wi-Fi Analytics
OUT	background	2018	PETS	optout_tool	Turtles, Locks, and Bathrooms: Understanding Mental Models of Privacy Through Illustration
OUT	homonym	2018	USENIX	request_sent	Freezing the Web: A Study of ReDoS Vulnerabilities in JavaScript-based Web Servers
OUT	policy-text	2018	USENIX	dnt,optout_tool	Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning
OUT	policy-text	2018	WWW	dnt,optout_tool	An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies.
OUT	background	2018	WWW	optout_tool	The Cost of Digital Advertisement: Comparing User and Advertiser Views.
OUT	background	2019	CCS	optout_tool	AdVersarial: Perceptual Ad Blocking meets Adversarial Machine Learning.
IN	O	2019	CCS	erasure,broker_optout	Five Years of the Right to be Forgotten.
OUT	attitudes	2019	CCS	dnt,optout_tool,self_service	Oh, the Places You've Been! User Reactions to Longitudinal Transparency About Third-Party Web Tracking and Inferencing.
OUT	consent	2019	CCS	dnt,ccpa_law,request_sent,optout_tool	(Un)informed Consent: Studying GDPR Consent Notices in the Field.
OUT	background	2019	IEEE-SP	erasure	Exploiting Unintended Feature Leakage in Collaborative Learning.
OUT	background	2019	IMC	self_service	A Longitudinal Analysis of the ads.txt Standard.
OUT	research-ethics	2019	IMC	optout_tool	An End-to-End, Large-Scale Measurement of DNS-over-Encryption: How Far Have We Come?
OUT	background	2019	NDSS	self_service	measuring-the-facebook-advertising-ecosystem
IN	D	2019	NDSS	dnt,self_service	quantity-vs-quality-evaluating-user-interest-profiles-using-ad-preference-managers
OUT	homonym	2019	NDSS	self_service	statistical-privacy-for-streaming-traffic
OUT	consent	2019	NDSS	dnt,portability,optout_tool	we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy
OUT	consent	2019	PETS	optout_tool	4 Years of EU Cookie Law: Results and Lessons Learned
OUT	research-ethics	2019	PETS	optout_tool	Detecting TCP/IP Connections via IPID Hash Collisions
OUT	background	2019	PETS	self_service	Investigating sources of PII used in Facebook’s targeted advertising
OUT	system	2019	PETS	erasure,request_sent	Lethe: Conceal Content Deletion from Persistent Observers
OUT	homonym	2019	PETS	self_service	MAPS: Scaling Privacy Compliance Analysis to a Million Apps
OUT	background	2019	PETS	erasure	Privacy Attitudes of Smart Speaker Users
OUT	attitudes	2019	PETS	optout_tool	Skip, Skip, Skip, Accept!!!: A Study on the Usability of Smartphone Manufacturer Provided Default Features and User Privacy
OUT	background	2019	USENIX	ccpa_law,optout_tool	50 Ways to Leak Your Data: An Exploration of Apps' Circumvention of the Android Permissions System
OUT	background	2019	USENIX	optout_tool	All Things Considered: An Analysis of IoT Devices on Home Networks
OUT	homonym	2019	USENIX	request_sent	Leaky Images: Targeted Privacy Attacks in the Web
OUT	homonym	2019	USENIX	self_service	Point Break: A Study of Bandwidth Denial-of-Service Attacks against Tor
OUT	policy-text	2019	USENIX	do_not_sell,optout_tool	PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play
OUT	homonym	2019	USENIX	self_service	Protecting accounts from credential stuffing with password breach alerting
OUT	research-ethics	2019	USENIX	optout_tool	The Betrayal At Cloud City: An Empirical Analysis Of Cloud-Based Mobile Backends
OUT	background	2019	WWW	self_service	Auditing Offline Data Brokers via Facebook's Advertising Platform.
IN	D	2019	WWW	portability,self_service	Detecting Low Self-Esteem in Youths from Web Search Data.
OUT	homonym	2019	WWW	request_sent	Google Dataset Search: Building a search engine for datasets in an open Web ecosystem.
OUT	takedown	2019	WWW	erasure	Who Watches the Watchmen: Exploring Complaints on the Web.
OUT	homonym	2020	CCS	broker_optout	The Boon and Bane of Cross-Signing: Shedding Light on a Common Practice in Public Key Infrastructures.
OUT	homonym	2020	CCS	request_sent	The Cookie Hunter: Automated Black-box Auditing for Web Authentication and Authorization Flaws.
OUT	homonym	2020	IMC	self_service	FlashRoute: Efficient Traceroute on a Massive Scale.
OUT	homonym	2020	IMC	self_service	Lumos5G: Mapping and Predicting Commercial mmWave 5G Throughput.
OUT	homonym	2020	IMC	account_deletion	Turning Up the Dial: the Evolution of a Cybercrime Market Through Set-up, Stable, and Covid-19 Eras.
OUT	homonym	2020	IMC	request_sent	When Push Comes to Ads: Measuring the Rise of (Malicious) Push Advertising.
OUT	policy-text	2020	PETS	ccpa_law,account_deletion	An Analysis of the Current State of the Consumer Credit Reporting System in China
IN	R	2020	PETS	erasure	CanaryTrap: Detecting Data Misuse by Third-Party Apps on Online Social Networks
OUT	policy-text	2020	PETS	request_sent,optout_tool	How private is your period?: A systematic analysis of menstrual app privacy policies
OUT	policy-text	2020	PETS	dsar_phrase,portability,optout_tool	The Privacy Policy Landscape After the GDPR
OUT	research-ethics	2020	USENIX	optout_tool	A Longitudinal and Comprehensive Study of the DANE Ecosystem in Email
OUT	policy-text	2020	USENIX	do_not_sell,ccpa_law	Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck
OUT	system	2020	USENIX	erasure,ccpa_law,account_deletion	DELF: Safeguarding deletion correctness in Online Social Networks
OUT	homonym	2020	USENIX	usp_string	From Needs to Actions to Secure Apps? The Effect of Requirements and Developer Practices on App Security
IN	D	2020	USENIX	dsar_phrase,ccpa_law,optout_tool,self_service	What Twitter Knows: Characterizing Ad Targeting Practices, User Perceptions, and Ad Explanations Through Users' Own Twitter Data
OUT	policy-text	2020	WWW	do_not_sell,ccpa_law,optout_tool	Finding a Choice in a Haystack: Automatic Extraction of Opt-Out Statements from Privacy Policy Text.
OUT	homonym	2020	WWW	broker_optout	I've Got Your Packages: Harvesting Customers' Delivery Order Information using Package Tracking Number Enumeration Attacks.
OUT	homonym	2020	WWW	ccpa_law,self_service	Understanding the Performance Costs and Benefits of Privacy-focused Browser Extensions.
OUT	policy-text	2021	CCS	do_not_sell	Consistency Analysis of Data-Usage Purposes in Mobile Apps.
OUT	research-ethics	2021	CCS	optout_tool	The Effect of Google Search on Software Security: Unobtrusive Security Interventions via Content Re-ranking.
OUT	background	2021	IEEE-SP	optout_tool	Detecting Filter List Evasion with Event-Loop-Turn Granularity JavaScript Signatures.
OUT	research-ethics	2021	IMC	optout_tool	Follow the scent: defeating IPv6 prefix rotation privacy.
OUT	homonym	2021	IMC	erasure	Risky BIZness: risks derived from registrar name management.
OUT	research-ethics	2021	IMC	optout_tool	Third time's not a charm: exploiting SNMPv3 for router fingerprinting.
OUT	takedown	2021	IMC	broker_optout	Throttling Twitter: an emerging censorship technique in Russia.
OUT	homonym	2021	IMC	broker_optout	Tracing your roots: exploring the TLS trust anchor ecosystem.
OUT	background	2021	NDSS	self_service	Awakening the Web’s Sleeper Agents: Misusing Service Workers for Privacy Leakage
OUT	system	2021	NDSS	erasure	Deceptive Deletions for Protecting Withdrawn Posts on Social Media Platforms
OUT	background	2021	NDSS	ccpa_law,optout_tool	From WHOIS to WHOWAS: A Large-Scale Measurement Study of Domain Registration Privacy under the GDPR
OUT	system	2021	NDSS	erasure,do_not_sell,dnt,portability,ccpa_law,authorized_agent	PrivacyFlash Pro: Automating Privacy Policy Generation for Mobile Apps
OUT	background	2021	NDSS	usp_string	Understanding Worldwide Private Information Collection on Android
OUT	policy-text	2021	PETS	optout_tool	Automated Extraction and Presentation of Data Practices in Privacy Policies
IN	R	2021	PETS	dsar_phrase,erasure,portability,ccpa_law,self_service	Data Portability between Online Services: An Empirical Analysis on the Effectiveness of GDPR Art. 20
OUT	attitudes	2021	PETS	dsar_phrase,erasure,do_not_sell,dnt,ccpa_law	Defining Privacy: How Users Interpret Technical Terms in Privacy Policies
OUT	attitudes	2021	PETS	ccpa_law,optout_tool	“Did you know this camera tracks your mood?”: Understanding Privacy Expectations and Preferences in the Age of Video Analytics
OUT	attitudes	2021	PETS	request_sent	Exploring mental models of the right to informational self-determination of office workers in Germany
OUT	attitudes	2021	PETS	dnt,ccpa_law,optout_tool,self_service	Managing Potentially Intrusive Practices in the Browser: A User-Centered Perspective
OUT	background	2021	PETS	self_service	ML-CB: Machine Learning Canvas Block
IN	S	2021	PETS	gpc,do_not_sell,dnt,ccpa_law,optout_tool	Privacy Preference Signals: Past, Present and Future
OUT	background	2021	PETS	erasure,ccpa_law	The Audio Auditor: User-Level Membership Inference in Internet of Things Voice Services
OUT	attitudes	2021	USENIX	dsar_phrase,optout_tool,self_service	Are Privacy Dashboards Good for End Users? Evaluating User Perceptions and Reactions to Google's My Activity
OUT	homonym	2021	USENIX	erasure	Collective Information Security in Large-Scale Urban Protests: the Case of Hong Kong
OUT	research-ethics	2021	USENIX	optout_tool	How Great is the Great Firewall? Measuring China's DNS Censorship
OUT	attitudes	2021	USENIX	account_deletion	"Now I'm a bit angry:" Individuals' Awareness, Perception, and Responses to Data Breaches that Affected Them
OUT	system	2021	USENIX	ccpa_law,optout_tool	PriSEC: A Privacy Settings Enforcement Controller
OUT	consent	2021	USENIX	ccpa_law,request_sent,optout_tool	Share First, Ask Later (or Never?) Studying Violations of GDPR's Explicit Consent in Android Apps
IN	D	2021	USENIX	self_service	SocialHEISTing: Understanding Stolen Facebook Accounts
OUT	research-ethics	2021	USENIX	broker_optout	Weaponizing Middleboxes for TCP Reflected Amplification
OUT	homonym	2021	WWW	account_deletion	An Investigation of Identity-Account Inconsistency in Single Sign-On.
OUT	attitudes	2021	WWW	optout_tool	From Personal Data to Digital Legacy: Exploring Conflicts in the Sharing, Security and Privacy of Post-mortem Data.
OUT	policy-text	2021	WWW	portability,ccpa_law,request_sent,optout_tool	Have You been Properly Notified? Automatic Compliance Analysis of Privacy Policy Text with GDPR Article 13.
OUT	policy-text	2021	WWW	ccpa_law,optout_tool	Privacy Policies over Time: Curation and Analysis of a Million-Document Dataset.
OUT	background	2022	CCS	erasure,ccpa_law,request_sent	Collect Responsibly But Deliver Arbitrarily?: A Study on Cross-User Privacy Leakage in Mobile Apps.
IN	S	2022	CCS	dnt,optout_tool	Do Opt-Outs Really Opt Me Out?
OUT	unlearning	2022	CCS	erasure	Feature Inference Attack on Shapley Values.
OUT	consent	2022	CCS	ccpa_law,request_sent,optout_tool	Freely Given Consent?: Studying Consent Notice of Third-Party Tracking and Its Violations of GDPR in Android Apps.
OUT	homonym	2022	CCS	self_service	TickTock: Detecting Microphone Status in Laptops Leveraging Electromagnetic Leakage of Clock Signals.
OUT	homonym	2022	IEEE-SP	self_service	DEEPCASE: Semi-Supervised Contextual Analysis of Security Events.
IN	R	2022	IEEE-SP	dsar_phrase,account_deletion	Scraping Sticky Leftovers: App User Information Left on Servers After Account Deletion.
OUT	background	2022	IEEE-SP	erasure	Towards Automated Auditing for Account and Session Management Flaws in Single Sign-On Deployments.
OUT	homonym	2022	IEEE-SP	account_deletion	TrollMagnifier: Detecting State-Sponsored Troll Accounts on Reddit.
OUT	homonym	2022	IMC	self_service	Are we ready for metaverse?: a measurement study of social virtual reality platforms.
OUT	homonym	2022	IMC	self_service	Causal impact of Android go on mobile web performance.
OUT	research-ethics	2022	IMC	optout_tool	Deep dive into the IoT backend ecosystem.
OUT	homonym	2022	IMC	optout_tool	HTML violations and where to find them: a longitudinal analysis of specification violations in HTML.
OUT	homonym	2022	IMC	request_sent	PhishInPatterns: measuring elicited user interactions at scale on phishing websites.
OUT	homonym	2022	IMC	request_sent	Toppling top lists: evaluating the accuracy of popular website lists.
OUT	research-ethics	2022	IMC	optout_tool	Towards a tectonic traffic shift?: investigating Apple's new relay network.
OUT	takedown	2022	IMC	broker_optout	TSPU: Russia's decentralized censorship system.
OUT	homonym	2022	NDSS	self_service	A Lightweight IoT Cryptojacking Detection Mechanism in Heterogeneous Smart Home Networks
OUT	background	2022	NDSS	erasure,ccpa_law,self_service	HARPO: Learning to Subvert Online Behavioral Advertising
OUT	unlearning	2022	PETS	erasure,ccpa_law,request_sent	Athena: Probabilistic Verification of Machine Unlearning
OUT	background	2022	PETS	ccpa_law,optout_tool	ATOM: Ad-network Tomography
OUT	developer-side	2022	PETS	gpc,do_not_sell,ccpa_law,optout_tool	Charting App Developers’ Journey Through Privacy Regulation Features in Ad Networks
OUT	homonym	2022	PETS	request_sent	Collection, usage and privacy of mobility data in the enterprise and public administrations
OUT	developer-side	2022	PETS	do_not_sell,ccpa_law,optout_tool	Developers Say the Darnedest Things: Privacy Compliance Processes Followed by Developers of Child-Directed Apps
OUT	attitudes	2022	PETS	request_sent,self_service	Employees’ privacy perceptions: exploring the dimensionality and antecedents of personal data sensitivity and willingness to disclose
OUT	attitudes	2022	PETS	dnt,ccpa_law,optout_tool,self_service	How Can and Would People Protect From Online Tracking?
OUT	background	2022	PETS	dsar_phrase,erasure,portability,gdpr_article,optout_tool	Investigating GDPR Fines in the Light of Data Flows
IN	M	2022	PETS	erasure,ccpa_law,broker_optout,optout_tool	“It Feels Like Whack-a-mole”: User Experiences of Data Removal from People Search Websites
OUT	background	2022	PETS	dsar_phrase	Keeping Privacy Labels Honest
IN	R	2022	PETS	dsar_phrase,erasure,portability,gdpr_article,request_sent,account_deletion,broker_optout,optout_tool,self_service	Leave No Data Behind – Empirical Insights into Data Erasure from Online Services
OUT	attitudes	2022	PETS	self_service	On the Challenges of Developing a Concise Questionnaire to Identify Privacy Personas
OUT	background	2022	PETS	dsar_phrase,ccpa_law	Personal information inference from voice recordings: User awareness and privacy concerns
IN	R	2022	PETS	dsar_phrase,erasure,ccpa_law,gdpr_article,request_sent	Revisiting Identification Issues in GDPR ‘Right Of Access’ Policies: A Technical and Longitudinal Analysis
IN	M	2022	PETS	erasure,do_not_sell,ccpa_law,request_sent,optout_tool,usp_string	Setting the Bar Low: Are Websites Complying With the Minimum Requirements of the CCPA?
OUT	policy-text	2022	PETS	dsar_phrase	“We may share the number of diaper changes”: A Privacy and Security Analysis of Mobile Child Care Applications
OUT	background	2022	PETS	ccpa_law,optout_tool	Who Knows I Like Jelly Beans? An Investigation Into Search Privacy
OUT	takedown	2022	USENIX	ccpa_law,broker_optout	A Large-scale Investigation into Geodifferences in Mobile Apps
OUT	attitudes	2022	USENIX	do_not_sell,ccpa_law	Caring about Sharing: User Perceptions of Multiparty Data Sharing
OUT	policy-text	2022	USENIX	erasure,do_not_sell,ccpa_law	Electronic Monitoring Smartphone Apps: An Analysis of Risks from Technical, Human-Centered, and Legal Perspectives
OUT	attitudes	2022	USENIX	erasure,gdpr_article	Empirical Understanding of Deletion Privacy: Experiences, Expectations, and Measures
IN	M	2022	USENIX	dsar_phrase,erasure,ccpa_law,request_sent,account_deletion,optout_tool,self_service	How Are Your Zombie Accounts? Understanding Users' Practices and Expectations on Mobile App Account Deletion
OUT	homonym	2022	USENIX	do_not_sell	Mistrust Plugins You Must: A Large-Scale Study Of Malicious Plugins In WordPress Marketplaces
OUT	research-ethics	2022	USENIX	request_sent	Pre-hijacked accounts: An Empirical Study of Security Failures in User Account Creation on the Web
OUT	system	2022	USENIX	ccpa_law,request_sent	PrivGuard: Privacy Regulation Compliance Made Easier
OUT	attitudes	2022	USENIX	self_service	Security and Privacy Perceptions of Third-Party Application Access for Google Accounts
OUT	policy-text	2022	USENIX	do_not_sell,ccpa_law,optout_tool	Smart Home Privacy Policies Demystified: A Study of Availability, Content, and Coverage
OUT	homonym	2022	USENIX	broker_optout	The Security Lottery: Measuring Client-Side Web Security Inconsistencies
OUT	background	2022	USENIX	optout_tool	Towards More Robust Keyword Spotting for Voice Assistants
IN	D	2022	USENIX	portability,self_service,usp_string	Understanding and Improving Usability of Data Dashboards for Simplified Privacy Control of Voice Assistant Data
OUT	homonym	2022	WWW	request_sent	Am I a Real or Fake Celebrity? Evaluating Face Recognition and Verification APIs under Deepfake Impersonation Attack.
OUT	background	2023	CCS	do_not_sell,ccpa_law	Comprehension from Chaos: Towards Informed Consent for Private Computation.
OUT	system	2023	CCS	dsar_phrase,erasure,portability,ccpa_law,gdpr_article	General Data Protection Runtime: Enforcing Transparent GDPR Compliance for Existing Applications.
OUT	homonym	2023	CCS	self_service	"I just stopped using one and started using the other": Motivations, Techniques, and Challenges When Switching Password Managers.
OUT	research-ethics	2023	CCS	request_sent	Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Aspects.
OUT	research-ethics	2023	CCS	optout_tool	Passive SSH Key Compromise via Lattices.
OUT	policy-text	2023	CCS	erasure,portability,ccpa_law,request_sent	PolicyChecker: Analyzing the GDPR Completeness of Mobile Apps' Privacy Policies.
OUT	homonym	2023	CCS	optout_tool	Poster: Towards a Dataset for the Discrimination between Warranted and Unwarranted Emails.
OUT	background	2023	CCS	erasure	Understanding and Detecting Abused Image Hosting Modules as Malicious Services.
OUT	homonym	2023	CCS	account_deletion	"We've Disabled MFA for You": An Evaluation of the Security and Usability of Multi-Factor Authentication Recovery Deployments.
IN	D	2023	IEEE-SP	self_service	Collaborative Ad Transparency: Promises and Limitations.
OUT	policy-text	2023	IEEE-SP	do_not_sell,request_sent	Detection of Inconsistencies in Privacy Practices of Browser Extensions.
OUT	homonym	2023	IEEE-SP	account_deletion	"In Eighty Percent of the Cases, I Select the Password for Them": Security and Privacy Challenges, Advice, and Opportunities at Cybercafes in Kenya.
OUT	homonym	2023	IEEE-SP	account_deletion	Investigating Package Related Security Threats in Software Registries.
OUT	research-ethics	2023	IEEE-SP	broker_optout	"It's up to the Consumer to be Smart": Understanding the Security and Privacy Attitudes of Smart Home Users on Reddit.
OUT	background	2023	IEEE-SP	erasure	No Privacy in the Electronics Repair Industry.
OUT	policy-text	2023	IEEE-SP	do_not_sell	Protected or Porous: A Comparative Analysis of Threat Detection Capability of IoT Safeguards.
OUT	homonym	2023	IMC	account_deletion	Evolving Bots: The New Generation of Comment Bots and their Underlying Scam Campaigns in YouTube.
OUT	research-ethics	2023	IMC	optout_tool	Illuminating Router Vendor Diversity Within Providers and Along Network Paths.
OUT	research-ethics	2023	IMC	optout_tool	Inferring Changes in Daily Human Activity from Internet Response.
OUT	research-ethics	2023	IMC	optout_tool	Pushing Alias Resolution to the Limit.
OUT	consent	2023	IMC	do_not_sell,ccpa_law	Thou Shalt Not Reject: Analyzing Accept-Or-Pay Cookie Banners on the Web.
OUT	system	2023	NDSS	dsar_phrase,erasure,do_not_sell,ccpa_law,authorized_agent,request_sent	VICEROY: GDPR-/CCPA-compliant Enforcement of Verifiable Accountless Consumer Requests
IN	M	2023	NDSS	erasure,ccpa_law,gdpr_article,self_service	CHKPLUG: Checking GDPR Compliance of WordPress Plugins via Cross-language Code Property Graph
IN	M	2023	PETS	do_not_sell,dnt,ccpa_law,request_sent,optout_tool	Comparing Large-Scale Privacy and Security Notifications
OUT	attitudes	2023	PETS	erasure	Creative beyond TikToks: Investigating Adolescents' Social Privacy Management on TikTok
OUT	background	2023	PETS	request_sent	Everybody's Looking for SSOmething: A large-scale evaluation on the privacy of OAuth authentication on the web
OUT	policy-text	2023	PETS	dsar_phrase,erasure,dnt,optout_tool	Evolution of Composition, Readability, and Structure of Privacy Policies over Two Decades
OUT	enforcement	2023	PETS	erasure,ccpa_law,request_sent,broker_optout,optout_tool	GDPRxiv: Establishing the State of the Art in GDPR Enforcement
OUT	background	2023	PETS	optout_tool	Heads in the Clouds? Measuring Universities’ Migration to Public Clouds: Implications for Privacy & Academic Freedom
OUT	homonym	2023	PETS	broker_optout	I-GWAS: Privacy-Preserving Interdependent Genome-Wide Association Studies
OUT	attitudes	2023	PETS	self_service	Is There a Reverse Privacy Paradox? An Exploratory Analysis of Gaps Between Privacy Perspectives and Privacy-Seeking Behaviors
IN	R	2023	PETS	dsar_phrase,erasure,portability,ccpa_law,gdpr_article,authorized_agent,request_sent,self_service	Lessons in VCR Repair: Compliance of Android App Developers with the California Consumer Privacy Act (CCPA)
OUT	background	2023	PETS	account_deletion,self_service	No Privacy Among Spies: Assessing the Functionality and Insecurity of Consumer Android Spyware Apps
OUT	homonym	2023	PETS	request_sent	Not Your Average App: A Large-scale Privacy Analysis of Android Browsers
OUT	background	2023	PETS	do_not_sell,ccpa_law	Privacy Rarely Considered: Exploring Considerations in the Adoption of Third-Party Services by Websites
OUT	policy-text	2023	PETS	optout_tool	Researchers’ Experiences in Analyzing Privacy Policies: Challenges and Opportunities
OUT	attitudes	2023	PETS	erasure,ccpa_law	Speculative Privacy Concerns about AR Glasses Data Collection
OUT	attitudes	2023	PETS	erasure,portability,ccpa_law,request_sent,optout_tool,self_service	'Surprised, Shocked, Worried': User Reactions to Facebook Data Collection from Third Parties
OUT	background	2023	PETS	self_service	Trust TEE?: Exploring the Impact of Trusted Execution Environments on Smart Home Privacy Norms
IN	S	2023	PETS	gpc,do_not_sell,dnt,ccpa_law,authorized_agent,optout_tool,usp_string	Usability and Enforceability of Global Privacy Control
OUT	homonym	2023	USENIX	request_sent	A Large-Scale Measurement of Website Login Policies
OUT	research-ethics	2023	USENIX	erasure,request_sent	Anatomy of a High-Profile Data Breach: Dissecting the Aftermath of a Crypto-Wallet Case
OUT	consent	2023	USENIX	do_not_sell,ccpa_law,authorized_agent,optout_tool	Automated Cookie Notice Analysis and Enforcement
OUT	background	2023	USENIX	ccpa_law,self_service	Defining "Broken": User Experiences and Remediation Tactics When Ad-Blocking or Tracking-Protection Tools Break a Website’s User Experience
OUT	research-ethics	2023	USENIX	broker_optout	How the Great Firewall of China Detects and Blocks Fully Encrypted Traffic
OUT	attitudes	2023	USENIX	account_deletion	"I'm going to trust this until it burns me" Parents' Privacy Concerns and Delegation of Trust in K-8 Educational Technology
OUT	policy-text	2023	USENIX	do_not_sell,ccpa_law,request_sent,self_service	PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs
OUT	homonym	2023	USENIX	usp_string	Pushed by Accident: A Mixed-Methods Study on Strategies of Handling Secret Information in Source Code Repositories
OUT	homonym	2023	USENIX	account_deletion	Squint Hard Enough: Attacking Perceptual Hashing with Adversarial Machine Learning
OUT	homonym	2023	USENIX	account_deletion	Strategies and Vulnerabilities of Participants in Venezuelan Influence Operations
OUT	homonym	2023	USENIX	account_deletion	The Digital-Safety Risks of Financial Technologies for Survivors of Intimate Partner Violence
OUT	homonym	2023	WWW	request_sent	Do NFTs' Owners Really Possess their Assets? A First Look at the NFT-to-Asset Connection Fragility.
OUT	research-ethics	2023	WWW	optout_tool	Measuring and Evading Turkmenistan's Internet Censorship: A Case Study in Large-Scale Measurements of a Low-Penetration Country.
OUT	unlearning	2023	WWW	erasure	Membership Inference Attacks Against Sequential Recommender Systems.
OUT	homonym	2023	WWW	account_deletion	Misbehavior and Account Suspension in an Online Financial Communication Platform.
OUT	homonym	2024	CCS	request_sent	Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power.
OUT	homonym	2024	CCS	account_deletion	End-to-End Encrypted Cloud Storage in the Wild: A Broken Ecosystem.
OUT	unlearning	2024	CCS	erasure,ccpa_law,request_sent	ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach.
OUT	homonym	2024	CCS	request_sent	Stealing Trust: Unraveling Blind Message Attacks in Web3 Authentication.
OUT	homonym	2024	CCS	account_deletion	Toward Understanding the Security of Plugins in Continuous Integration Services.
OUT	background	2024	CCS	do_not_sell	Trust, Because You Can't Verify: Privacy and Security Hurdles in Education Technology Acquisition Practices.
OUT	sok	2024	CCS	dsar_phrase,do_not_sell,ccpa_law,gdpr_article,request_sent,self_service	Understanding and Addressing Online Tracking: Online Privacy's Regulatory Turn.
OUT	background	2024	IEEE-SP	self_service	C-Frame: Characterizing and measuring in-the-wild CAPTCHA attacks.
OUT	research-ethics	2024	IEEE-SP	optout_tool	DNSBomb: A New Practical-and-Powerful Pulsing DoS Attack Exploiting DNS Queries-and-Responses.
OUT	homonym	2024	IEEE-SP	broker_optout	Practical Attacks Against DNS Reputation Systems.
OUT	background	2024	IEEE-SP	erasure	SoK: Safer Digital-Safety Research Involving At-Risk Users.
OUT	sok	2024	IEEE-SP	dsar_phrase,erasure,do_not_sell,portability,ccpa_law,authorized_agent,account_deletion,broker_optout,optout_tool,self_service	SoK: Technical Implementation and Human Impact of Internet Privacy Regulations.
OUT	background	2024	IEEE-SP	ccpa_law,optout_tool	Targeted and Troublesome: Tracking and Advertising on Children's Websites.
OUT	policy-text	2024	IEEE-SP	ccpa_law,optout_tool	Understanding the Privacy Practices of Political Campaigns: A Perspective from the 2020 US Election Websites.
IN	M	2024	IEEE-SP	erasure,do_not_sell,ccpa_law,account_deletion,optout_tool	Withdrawing is believing? Detecting Inconsistencies between Withdrawal Choices and Third-party Data Collections in Mobile Apps.
OUT	homonym	2024	IMC	request_sent	A First Look at Related Website Sets.
OUT	policy-text	2024	IMC	do_not_sell,account_deletion,optout_tool	Analyzing Corporate Privacy Policies using AI Chatbots.
OUT	background	2024	IMC	optout_tool	Analyzing the (In)Accessibility of Online Advertisements.
OUT	background	2024	IMC	account_deletion	Bounce in the Wild: A Deep Dive into Email Delivery Failures from a Large Email Service Provider.
OUT	background	2024	IMC	ccpa_law,optout_tool	Browsing without Third-Party Cookies: What Do You See?
OUT	background	2024	IMC	ccpa_law,usp_string	DiffAudit: Auditing Privacy Practices of Online Services for Children and Adolescents.
OUT	research-ethics	2024	IMC	optout_tool	Ten Years of ZMap.
OUT	background	2024	IMC	do_not_sell,ccpa_law,optout_tool	Watching TV with the Second-Party: A First Look at Automatic Content Recognition Tracking in Smart TVs.
OUT	homonym	2024	IMC	self_service	Whatcha Lookin' At: Investigating Third-Party Web Content in Popular Android Apps.
OUT	unlearning	2024	NDSS	request_sent	GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networks
OUT	homonym	2024	NDSS	portability,account_deletion	Maginot Line: Assessing a New Cross-app Threat to PII-as-Factor Authentication in Chinese Mobile Apps
OUT	homonym	2024	NDSS	request_sent	On Precisely Detecting Censorship Circumvention in Real-World Networks
OUT	homonym	2024	NDSS	account_deletion	The Dark Side of E-Commerce: Dropshipping Abuse as a Business Model
OUT	policy-text	2024	NDSS	erasure,do_not_sell,dnt,portability,ccpa_law,authorized_agent,optout_tool,usp_string	Towards Automated Regulation Analysis for Effective Privacy Compliance
OUT	policy-text	2024	PETS	dsar_phrase,erasure,gpc,do_not_sell,portability,ccpa_law,gdpr_article,authorized_agent,request_sent,account_deletion,optout_tool,usp_string	A Bilingual Longitudinal Analysis of Privacy Policies Measuring the Impacts of the GDPR and the CCPA/CPRA
OUT	background	2024	PETS	erasure,gpc,dnt,ccpa_law	Block Cookies, Not Websites: Analysing Mental Models and Usability of the Privacy-Preserving Browser Extension CookieBlock
OUT	homonym	2024	PETS	dnt,request_sent	Client-side and Server-side Tracking on Meta: Effectiveness and Accuracy
OUT	homonym	2024	PETS	ccpa_law,request_sent,usp_string	Connecting the Dots: Tracing Data Endpoints in IoT Devices
OUT	attitudes	2024	PETS	erasure,ccpa_law	Data Safety vs. App Privacy: Comparing the Usability of Android and iOS Privacy Labels
OUT	attitudes	2024	PETS	erasure	Decision-based Data Distribution (D³): Enabling Users to Minimize Data Propagation in Privacy-sensitive Scenarios
OUT	background	2024	PETS	do_not_sell	Exploring the Privacy Experiences of Closeted Users of Online Dating Services in the US
OUT	background	2024	PETS	erasure,ccpa_law,request_sent	GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
IN	M	2024	PETS	gpc,do_not_sell,dnt,ccpa_law,request_sent,optout_tool	Generalizable Active Privacy Choice: Designing a Graphical User Interface for Global Privacy Control
OUT	attitudes	2024	PETS	erasure,do_not_sell,ccpa_law	Generational Differences in Understandings of Privacy Terminology
OUT	attitudes	2024	PETS	self_service	How Does Connecting Online Activities to Advertising Inferences Impact Privacy Perceptions?
IN	S	2024	PETS	gpc,do_not_sell,dnt,ccpa_law,optout_tool,usp_string	Johnny Still Can't Opt-out: Assessing the IAB CCPA Compliance Framework
OUT	background	2024	PETS	dsar_phrase,erasure,ccpa_law	Mitigating Inference Risks with the NIST Privacy Framework
OUT	homonym	2024	PETS	self_service	"My Best Friend's Husband Sees and Knows Everything": A Cross-Contextual and Cross-Country Approach to Understanding Smart Home Privacy
IN	S	2024	PETS	dsar_phrase,do_not_sell,dnt,ccpa_law,request_sent,optout_tool	Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?
OUT	attitudes	2024	PETS	erasure,gdpr_article,broker_optout,self_service	Our Data, Our Solutions: A Participatory Approach for Enhancing Privacy in Wearable Activity Tracker Third-Party Apps
IN	D	2024	PETS	ccpa_law,self_service	Overprofiling Analysis on Major Internet Players
OUT	homonym	2024	PETS	ccpa_law,broker_optout	PrivacyLens: On-Device PII Removal from RGB Images using Thermally-Enhanced Sensing
OUT	background	2024	PETS	self_service	Support Personas: A Concept for Tailored Support of Users of Privacy-Enhancing Technologies
OUT	background	2024	PETS	do_not_sell,ccpa_law,gdpr_article,request_sent	The Devil is in the Details: Detection, Measurement and Lawfulness of Server-Side Tracking on the Web
OUT	developer-side	2024	PETS	dsar_phrase,erasure,ccpa_law,gdpr_article	“Those things are written by lawyers, and programmers are reading that.” Mapping the Communication Gap Between Software Developers and Privacy Experts
IN	S	2024	PETS	erasure,gpc,do_not_sell,dnt,ccpa_law,request_sent,optout_tool,usp_string	Two Steps Forward and One Step Back: The Right to Opt-out of Sale under CPRA
OUT	system	2024	PETS	gpc,self_service	Website Data Transparency in the Browser
OUT	background	2024	PETS	self_service	What Do Privacy Advertisements Communicate to Consumers?
OUT	attitudes	2024	PETS	dnt,ccpa_law,optout_tool,self_service	What Does It Mean to Be Creepy? Responses to Visualizations of Personal Browsing Activity, Online Tracking, and Targeted Ads
IN	R	2024	PETS	dsar_phrase,erasure,do_not_sell,ccpa_law,authorized_agent,request_sent,account_deletion,broker_optout	What to Expect When You’re Accessing: An Exploration of User Privacy Rights in People Search Websites
OUT	background	2024	USENIX	erasure	A Decade of Privacy-Relevant Android App Reviews: Large Scale Trends
OUT	homonym	2024	USENIX	account_deletion	A Mixed-Methods Study on User Experiences and Challenges of Recovery Codes for an End-to-End Encrypted Service
OUT	attitudes	2024	USENIX	self_service	ATTention Please! An Investigation of the App Tracking Transparency Permission
IN	R	2024	USENIX	dsar_phrase,erasure,portability,ccpa_law,gdpr_article,request_sent,optout_tool,self_service	Data Subjects' Reactions to Exercising Their Right of Access
OUT	attitudes	2024	USENIX	erasure	"Did They F***ing Consent to That?": Safer Digital Intimacy via Proactive Protection Against Image-Based Sexual Abuse
OUT	consent	2024	USENIX	ccpa_law,optout_tool,usp_string	Dissecting Privacy Perspectives of Websites Around the World: "Aceptar Todo, Alle Akzeptieren, Accept All..."
OUT	attitudes	2024	USENIX	erasure,account_deletion	Exploring digital security and privacy in relative poverty in Germany through qualitative interviews
OUT	takedown	2024	USENIX	broker_optout	Guardians of the Galaxy: Content Moderation in the InterPlanetary File System
OUT	homonym	2024	USENIX	account_deletion	"I chose to fight, be brave, and to deal with it": Threat Experiences and Security Practices of Pakistani Content Creators
OUT	homonym	2024	USENIX	portability,account_deletion	"I feel physically safe but not politically safe": Understanding the Digital Threats and Safety Practices of OnlyFans Creators
OUT	background	2024	USENIX	erasure,ccpa_law	iHunter: Hunting Privacy Violations at Scale in the Software Supply Chain on iOS
OUT	policy-text	2024	USENIX	erasure,do_not_sell,ccpa_law,authorized_agent	Is It a Trap? A Large-scale Empirical Study And Comprehensive Assessment of Online Automated Privacy Policy Generators for Mobile Apps
OUT	research-ethics	2024	USENIX	optout_tool	LanDscAPe: Exploring LDAP Weaknesses and Data Leaks at Internet Scale
OUT	developer-side	2024	USENIX	dsar_phrase,do_not_sell,ccpa_law	Navigating the Privacy Compliance Maze: Understanding Risks with Privacy-Configurable Mobile SDKs
OUT	research-ethics	2024	USENIX	optout_tool	ResolverFuzz: Automated Discovery of DNS Resolver Vulnerabilities with Query-Response Fuzzing
OUT	homonym	2024	USENIX	self_service	SIMurai: Slicing Through the Complexity of SIM Card Security Research
OUT	homonym	2024	USENIX	erasure	Sync+Sync: A Covert Channel Built on fsync with Storage
OUT	consent	2024	USENIX	gpc,do_not_sell,ccpa_law	The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions
OUT	homonym	2024	USENIX	self_service	Tickets or Privacy? Understand the Ecosystem of Chinese Ticket Grabbing Apps
OUT	background	2024	USENIX	optout_tool	Understanding How to Inform Blind and Low-Vision Users about Data Privacy through Privacy Question Answering Assistants
OUT	consent	2024	WWW	gpc,ccpa_law,usp_string	A Study of GDPR Compliance under the Transparency and Consent Framework.
OUT	research-ethics	2024	WWW	optout_tool	A Worldwide View on the Reachability of Encrypted DNS Services.
OUT	homonym	2024	WWW	account_deletion	Identifying Risky Vendors in Cryptocurrency P2P Marketplaces.
OUT	unlearning	2024	WWW	erasure	Incentive and Dynamic Client Selection for Federated Unlearning.
IN	D	2024	WWW	dsar_phrase,request_sent	TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds.
OUT	background	2025	CCS	ccpa_law,optout_tool	Automatically Detecting Online Deceptive Patterns.
OUT	background	2025	CCS	dnt,ccpa_law,self_service	Exploiting the Shared Storage API.
IN	R	2025	CCS	dsar_phrase,portability,ccpa_law,gdpr_article,request_sent,self_service	Hidden in Plain Bytes: Investigating Interpersonal Account Compromise with Data Exports.
OUT	homonym	2025	CCS	account_deletion	How to Design Secure Honey Vault Schemes.
OUT	research-ethics	2025	CCS	optout_tool	In the DOM We Trust: Exploring the Hidden Dangers of Reading from the DOM on the Web.
IN	M	2025	CCS	gpc,do_not_sell,ccpa_law,request_sent,optout_tool,self_service	Layered, Overlapping, and Inconsistent: A Large-Scale Analysis of the Multiple Privacy Policies and Controls of U.S. Banks.
OUT	homonym	2025	CCS	account_deletion	Phishing Susceptibility and the (In-)Effectiveness of Common Anti-Phishing Interventions in a Large University Hospital.
OUT	research-ethics	2025	CCS	optout_tool	RebirthDay Attack: Reviving DNS Cache Poisoning with the Birthday Paradox.
OUT	background	2025	CCS	optout_tool	The Odyssey of robots.txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services.
OUT	background	2025	IEEE-SP	ccpa_law,self_service	A Big Step Forward? A User-Centric Examination of iOS App Privacy Report and Enhancements.
OUT	research-ethics	2025	IEEE-SP	optout_tool	Beyond the Horizon: Uncovering Hosts and Services Behind Misconfigured Firewalls.
OUT	homonym	2025	IEEE-SP	account_deletion	(Blind) Users Really Do Heed Aural Telephone Scam Warnings.
OUT	background	2025	IEEE-SP	ccpa_law,self_service	Characterizing the Usability and Usefulness of U.S. Ad Transparency Systems.
OUT	homonym	2025	IEEE-SP	self_service	Differentially Private Release of Israel's National Registry of Live Births.
OUT	research-ethics	2025	IEEE-SP	optout_tool	Is Nobody There? Good! Globally Measuring Connection Tampering Without Responsive Endhosts.
OUT	background	2025	IEEE-SP	request_sent	"It's been Lovely Watching you": Institutional Decision-Making on Online Proctoring Software.
OUT	homonym	2025	IEEE-SP	request_sent	Mind the Location Leakage in LEO Direct-to-Cell Satellite Networks.
OUT	background	2025	IEEE-SP	broker_optout	SoK: A Framework and Guide for Human-Centered Threat Modeling in Security and Privacy Research.
OUT	research-ethics	2025	IEEE-SP	erasure,request_sent	SoK: A Privacy Framework for Security Research Using Social Media Data.
OUT	background	2025	IEEE-SP	erasure	SoK: Self-Generated Nudes over Private Chats: How can Technology Contribute to a Safer Sexting?
OUT	developer-side	2025	IEEE-SP	dsar_phrase,erasure,ccpa_law,gdpr_article	"Sorry for Bugging you so much." Exploring Developers' Behavior Towards Privacy-Compliant Implementation.
OUT	background	2025	IEEE-SP	optout_tool	Teaching Data Science Students to Sketch Privacy Designs Through Heuristics.
OUT	attitudes	2025	IEEE-SP	erasure,request_sent,account_deletion	Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms.
OUT	background	2025	IMC	ccpa_law,request_sent	An In-Depth Investigation of Data Collection in LLM App Ecosystems.
OUT	homonym	2025	IMC	self_service	Automated, Cross-Layer Root Cause Analysis of 5G Video-Conferencing Quality Degradation.
OUT	background	2025	IMC	ccpa_law,usp_string	$CookieGuard: $ Characterizing and Isolating the First-Party Cookie Jar.
IN	R	2025	IMC	erasure,request_sent	From Voice to Ads: Auditing Commercial Smart Speakers for Targeted Advertising based on Voice Characteristics.
OUT	homonym	2025	IMC	erasure,optout_tool	Measuring the deployment of DNSSEC Bootstrapping Using Authenticated Signals.
OUT	homonym	2025	IMC	self_service	Replication: Performance of Cellular Networks on the Wheels.
OUT	background	2025	IMC	self_service	Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study.
OUT	background	2025	IMC	gpc,ccpa_law,usp_string	Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers.
OUT	homonym	2025	IMC	self_service	The Decentralization Dilemma: Performance Trade-Offs in IPFS and Breakpoints.
OUT	research-ethics	2025	IMC	optout_tool	Time To Scan: Digging into NTP-based IPv6 Scanning.
OUT	research-ethics	2025	IMC	optout_tool	Tracking Internet Disruptions in Ukraine: Insights from Three Years of Active Full Block Scans.
OUT	research-ethics	2025	NDSS	optout_tool	A Large-Scale Measurement Study of the PROXY Protocol and its Security Implications
OUT	attitudes	2025	NDSS	ccpa_law,authorized_agent,self_service	Balancing Privacy and Data Utilization: A Comparative Vignette Study on User Acceptance of Data Trustees in Germany and the US
OUT	homonym	2025	NDSS	broker_optout	HADES Attack: Understanding and Evaluating Manipulation Risks of Email Blocklists
OUT	policy-text	2025	NDSS	dnt,optout_tool	PolicyPulse: Precision Semantic Role Extraction for Enhanced Privacy Policy Comprehension
OUT	unlearning	2025	NDSS	erasure,broker_optout	Reinforcement Unlearning
OUT	attitudes	2025	NDSS	self_service	Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report
OUT	research-ethics	2025	NDSS	optout_tool	Wallbleed: A Memory Disclosure Vulnerability in the Great Firewall of China
OUT	homonym	2025	NDSS	ccpa_law,account_deletion,self_service	”Who is Trying to Access My Account?” Exploring User Perceptions and Reactions to Risk-based Authentication Notifications
OUT	background	2025	PETS	erasure,account_deletion	Can Social Media Privacy and Safety Features Protect Targets of Interpersonal Attacks? A Systematic Analysis
IN	R	2025	PETS	dsar_phrase,ccpa_law,optout_tool,self_service	Echoes of Privacy: Uncovering the Profiling Practices of Voice Assistants
IN	M	2025	PETS	erasure,account_deletion	``Erasing the Echo'': The Usability of Data Deletion in Smart Personal Assistants
OUT	attitudes	2025	PETS	erasure,do_not_sell,ccpa_law,optout_tool	Evaluating a Data Fiduciary Standard for Privacy: Developer and End-user Perspectives
OUT	attitudes	2025	PETS	erasure,do_not_sell,ccpa_law	"Free WiFi is not ultimately free": Privacy Perceptions of Users in the US regarding City-wide WiFi Services
OUT	attitudes	2025	PETS	erasure,portability,ccpa_law,self_service	Gig Work at What Cost? Exploring Privacy Risks of Gig Work Platform Participation in the U.S.
OUT	attitudes	2025	PETS	erasure,gdpr_article,account_deletion,optout_tool,self_service	Hoovered up as a data point: Exploring Privacy Behaviours, Awareness, and Concerns Among UK Users of LLM-based Conversational Agents
IN	S	2025	PETS	gpc,ccpa_law,request_sent,optout_tool	Intractable Cookie Crumbs: Unveiling the Nexus of Stateful Banner Interaction and Tracking Cookies
OUT	consent	2025	PETS	erasure,ccpa_law,request_sent,account_deletion,optout_tool	Johnny Can’t Revoke Consent Either: Measuring Compliance of Consent Revocation on the Web
OUT	system	2025	PETS	dsar_phrase,erasure,portability,gdpr_article,request_sent,self_service	Making Web Applications GDPR Compliant: A Comparative Evaluation of GDPR-Enforcement Frameworks
IN	R	2025	PETS	ccpa_law,request_sent,broker_optout,optout_tool	Measuring the Accuracy and Effectiveness of PII Removal Services
OUT	attitudes	2025	PETS	self_service	Misalignments and Demographic Differences in Expected and Actual Privacy Settings on Facebook
IN	S	2025	PETS	self_service	More and Scammier Ads: The Perils of YouTube's Ad Privacy Settings
OUT	attitudes	2025	PETS	optout_tool,self_service	Privacy Perceptions and Behaviors Towards Targeted Advertising on Social Media: A Cross-Country Study on the Effect of Culture and Religion
OUT	background	2025	PETS	erasure	Real-World Deniability in Messaging
OUT	background	2025	PETS	optout_tool	Sheep's clothing, wolfish intent: Automated detection and evaluation of problematic 'allowed' advertisements
OUT	policy-text	2025	PETS	erasure,request_sent	Surveillance Disguised as Protection: A Comparative Analysis of Sideloaded and In-Store Parental Control Apps
OUT	unlearning	2025	PETS	erasure,ccpa_law	Unlearning Clients, Features and Samples in Vertical Federated Learning
OUT	attitudes	2025	PETS	erasure,ccpa_law	"What are they gonna do with my data?": Privacy Expectations, Concerns, and Behaviors in Virtual Reality
OUT	background	2025	PETS	do_not_sell	Your Signal, Their Data: An Empirical Privacy Analysis of Wireless-scanning SDKs in Android
OUT	background	2025	USENIX	erasure,ccpa_law,request_sent,account_deletion	A Stakeholder-Based Framework to Highlight Tensions when Implementing Privacy Features
OUT	background	2025	USENIX	self_service	Abusability of Automation Apps in Intimate Partner Violence
IN	R	2025	USENIX	dsar_phrase,erasure,portability,ccpa_law,gdpr_article,request_sent,self_service	Addressing the Address Books' (Interdependent) Privacy Issues
OUT	background	2025	USENIX	account_deletion	Analyzing the AI Nudification Application Ecosystem
IN	D	2025	USENIX	portability,self_service	As Advertised? Understanding the Impact of Influencer VPN Ads
OUT	takedown	2025	USENIX	broker_optout	Assessing the Aftermath: the Effects of a Global Takedown against DDoS-for-hire Services
OUT	research-ethics	2025	USENIX	optout_tool	Catch-22: Uncovering Compromised Hosts using SSH Public Keys
OUT	homonym	2025	USENIX	erasure	eSIMplicity or eSIMplification? Privacy and Security Risks in the eSIM Ecosystem
OUT	policy-text	2025	USENIX	erasure,gpc,dnt,portability,ccpa_law,authorized_agent,request_sent,optout_tool,usp_string	Evaluating Privacy Policies under Modern Privacy Laws At Scale: An LLM-Based Automated Approach
OUT	homonym	2025	USENIX	request_sent	"Helps me Take the Post With a Grain of Salt:" Soft Moderation Effects on Accuracy Perceptions and Sharing Intentions of Inauthentic Political Content on X
OUT	homonym	2025	USENIX	self_service	Hercules Droidot and the murder on the JNI Express
OUT	research-ethics	2025	USENIX	optout_tool	IRBlock: A Large-Scale Measurement Study of the Great Firewall of Iran
OUT	homonym	2025	USENIX	erasure	Misty Registry: An Empirical Study of Flawed Domain Registry Operation
IN	R	2025	USENIX	dsar_phrase,erasure,portability,ccpa_law,gdpr_article,request_sent,account_deletion	No Way to Sign Out? Unpacking Non-Compliance with Google Play's App Account Deletion Requirements
OUT	homonym	2025	USENIX	account_deletion	"Please don't send that bot anything": A Mixed-methods Study of Personal Impersonation Attacks Targeting Digital Payments on Social Media
OUT	enforcement	2025	USENIX	ccpa_law,usp_string	Privacy Law Enforcement Under Centralized Governance: A Qualitative Analysis of Four Years' Special Privacy Rectification Campaigns
OUT	unlearning	2025	USENIX	erasure,gdpr_article,broker_optout	Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective
OUT	background	2025	USENIX	dsar_phrase,request_sent	Regulating Smart Device Support Periods: User Expectations and the European Cyber Resilience Act
OUT	research-ethics	2025	USENIX	optout_tool	S/MINE: Collecting and Analyzing S/MIME Certificates at Scale
OUT	background	2025	USENIX	self_service	TapTrap: Animation-Driven Tapjacking on Android
IN	S	2025	USENIX	gpc,do_not_sell,dnt,ccpa_law,authorized_agent,request_sent,optout_tool,usp_string	Websites' Global Privacy Control Compliance at Scale and over Time
OUT	homonym	2025	WWW	request_sent	50 Shades of Deceptive Patterns: A Unified Taxonomy, Multimodal Detection, and Security Implications.
OUT	background	2025	WWW	ccpa_law,optout_tool	Assessing Compliance in Digital Advertising: A Deep Dive into Acceptable Ads Standards.
OUT	unlearning	2025	WWW	erasure	Dynamic Graph Unlearning: A General and Efficient Post-Processing Method via Gradient Transformation.
OUT	policy-text	2025	WWW	account_deletion	Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale.
OUT	background	2025	WWW	ccpa_law,optout_tool	The First Early Evidence of the Use of Browser Fingerprinting for Online Tracking.
OUT	homonym	2025	WWW	dnt,self_service	Unveiling Network Performance in the Wild: An Ad-Driven Analysis of Mobile Download Speeds.
IN	R	2026	IEEE-SP	dsar_phrase,erasure,portability,ccpa_law,authorized_agent,request_sent,broker_optout,self_service	Consumer Beware! Exploring Data Brokers' CCPA Compliance.
OUT	developer-side	2026	IEEE-SP	portability,ccpa_law,account_deletion	Privacy Perspectives and Practices of Chinese Smart Home Product Teams.
IN	R	2026	IEEE-SP	dsar_phrase,portability,gdpr_article,request_sent,account_deletion,self_service	Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube.
OUT	research-ethics	2026	NDSS	optout_tool	Actively Understanding the Dynamics and Risks of the Threat Intelligence Ecosystem
OUT	homonym	2026	NDSS	self_service	Crack in the Armor: Underlying Infrastructure Threats to RPKI Publication Point Reachability
OUT	research-ethics	2026	NDSS	erasure	FLIPPYRAM: A Large-Scale Study of Rowhammer Prevalence
OUT	homonym	2026	NDSS	account_deletion	Hey there! You are using WhatsApp: Enumerating Three Billion Accounts for Security and Privacy
OUT	background	2026	NDSS	gpc	MVPNalyzer: An Investigative Framework for Auditing the Security & Privacy of Mobile VPNs
OUT	homonym	2026	NDSS	self_service	Pruning the Tree: Rethinking RPKI Architecture from the Ground up
OUT	background	2026	NDSS	erasure	SoK: Cryptographic Authenticated Dictionaries
OUT	homonym	2026	NDSS	self_service	The Heat is On: Understanding and Mitigating Vulnerabilities of Thermal Image Perception in Autonomous Systems
OUT	homonym	2026	NDSS	self_service	Understanding the Stealthy BGP Hijacking Risk in the ROV Era
OUT	background	2026	PETS	ccpa_law,self_service	A Year Under the DSA: Ad Transparency's Uneven Landscape
OUT	background	2026	PETS	self_service	Ad Personalization and Transparency in Mobile Ecosystems: A Comparative Analysis of Google's and Apple's EU App Stores
OUT	background	2026	PETS	dnt,self_service	Analyzing Societal Awareness and Perception of Digital Fingerprinting and Fingerprinting Countermeasures
OUT	attitudes	2026	PETS	dnt,ccpa_law,self_service	Are Bite-Size Data Safety Details a Healthy Diet for Android Telehealth App Users? Impacts of Privacy Nutrition Labels on Users’ Privacy Perceptions
OUT	policy-text	2026	PETS	ccpa_law,account_deletion	AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents
OUT	attitudes	2026	PETS	erasure	Bot Among Us: Exploring User Awareness and Privacy Concerns About Chatbots in Group Chats
OUT	research-ethics	2026	PETS	erasure,optout_tool	Designing Reflective Thinking-Based Contextual Privacy Policy for Mobile Applications
IN	S	2026	PETS	gpc,do_not_sell,dnt,ccpa_law,request_sent,optout_tool,usp_string	Exercising the CCPA Opt-out Right on Android: Legally Mandated but Practically Challenging
OUT	homonym	2026	PETS	request_sent	From Syntactic Matching to Taint Tracking and Back: A Comparative Study of Web Tracking Detection Techniques
OUT	homonym	2026	PETS	usp_string	Gaze3P: Gaze-Based Prediction of User-Perceived Privacy
OUT	background	2026	PETS	dsar_phrase	How Experts Personalize Privacy & Security Advice for At-Risk Users
OUT	attitudes	2026	PETS	erasure,ccpa_law,optout_tool	“I don’t think it needs to be political”: Privacy Experiences and Concerns of FemHealth App Users in the United States
OUT	homonym	2026	PETS	self_service	Location-Enhanced Information Flow for Home Automations
OUT	background	2026	PETS	optout_tool	More Modalities, More Problems: Examining User Understanding of The Meta Quest Permissions Framework
OUT	background	2026	PETS	erasure,do_not_sell,ccpa_law	No Privacy for Privates: How Military Communities Experience and Perceive the Privacy Risks of Military-Marketed Mobile Apps
IN	M	2026	PETS	dsar_phrase,do_not_sell,ccpa_law,gdpr_article,authorized_agent,request_sent,broker_optout,optout_tool	On the Suitability of LLM-Driven Agents for Dark Pattern Audits
OUT	policy-text	2026	PETS	erasure,portability,ccpa_law,gdpr_article,optout_tool	Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law’s Impact
OUT	background	2026	PETS	self_service	Personal Data Flows and Privacy Policy Traceability in Third-party LLM Apps in the GPT Ecosystem
OUT	background	2026	PETS	dsar_phrase,erasure	The PET Paradox: How Amazon Instrumentalises PETs in Sidewalk to Entrench Its Infrastructural Power
OUT	developer-side	2026	PETS	dsar_phrase,erasure,ccpa_law,request_sent,usp_string	The Role of Online Forums in Developer Understanding of Privacy Law - A Reddit Case Study
OUT	background	2026	PETS	request_sent	The TCF doesn’t really A(A)ID – Automatic Privacy Analysis and Legal Compliance of TCF-based Android Applications
OUT	developer-side	2026	PETS	ccpa_law,account_deletion,self_service	``Users are worried, but we are confused'': Exploring the Privacy, Security, and Safety Perspectives and Practices of FemHealth App Product Team Members
OUT	homonym	2026	PETS	request_sent	Waterfall: A Capsule-Based Framework for Evaluating Traffic Watermarking in Anonymity Systems
OUT	agent-benchmark	2026	PETS	do_not_sell,ccpa_law,account_deletion,optout_tool	WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks
OUT	policy-text	2026	PETS	erasure,portability	Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models
OUT	research-ethics	2026	USENIX	optout_tool	A Large-Scale Study of Personalized Phishing using Large Language Models
OUT	homonym	2026	USENIX	self_service	Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse
OUT	research-ethics	2026	USENIX	optout_tool	DaLens: Charting DNS Self-Amplification Threats at Large
OUT	homonym	2026	USENIX	account_deletion	Digital Risks and Coping Practices among Roblox Game Creators
OUT	homonym	2026	USENIX	usp_string	End-to-End Encrypted Collaborative Documents
OUT	background	2026	USENIX	self_service	Inconsistent, Incomplete, and Insecure: A Survey of Account Security Interfaces
OUT	homonym	2026	USENIX	erasure	Invariant-Guided Logical Testing of Open RAN Controllers
OUT	background	2026	USENIX	account_deletion	Security and Privacy Analysis of Tile's Location Tracking Protocol
IN	D	2026	WWW	dsar_phrase,gdpr_article,request_sent	Bowling with ChatGPT: On the Evolving User Interactions with Conversational AI Systems.
IN	R	2026	WWW	gdpr_article,request_sent,self_service	Does Ad-Free Mean Less Data Collection? An Empirical Study of Platform Data Practices and User Expectations.
OUT	background	2026	WWW	self_service	Does This Button Work? Investigating YouTube's Ineffective User Controls.
OUT	research-ethics	2026	WWW	optout_tool	Tracking the Stray Sheep: Understanding DNS Response Manipulation in the Wild.
OUT	research-ethics	2026	WWW	broker_optout	Unveiling the Resilience of LLM-Enhanced Search Engines against Black-Hat SEO Manipulation.
</file>

===== 4. Every figure, with its query =====

''scripts/dsr_report.mjs'', run against ''data/extract/run1'' on 2026-09-16. Unedited:

<file text dsr_report-output.txt>
corpus: 5859 extracted papers (7 venues, 2010-2026)
candidates from the full-text sweep: 471
population after hand audit: 51 papers
  precision of the candidate sweep: 10.8%
  share of the corpus:              0.9%

1. The population by category of instrument (N = 51)
----------------------------------------------------
R  sent requests to real controllers	17	33.3%
S  measured an opt-out signal or setting	12	23.5%
M  measured a rights mechanism across a population	10	19.6%
D  research dataset obtained via access / export	10	19.6%
O  observational study of requests others sent	2	3.9%

2. Why the candidate sweep is mostly noise (N = 420 rejected)
-------------------------------------------------------------
homonym	129	30.7%
background	108	25.7%
research-ethics	53	12.6%
attitudes	44	10.5%
policy-text	34	8.1%
consent	11	2.6%
unlearning	10	2.4%
system	10	2.4%
developer-side	8	1.9%
takedown	7	1.7%
sok	3	0.7%
enforcement	2	0.5%
agent-benchmark	1	0.2%

3. Per year. 2025-2026 are PROVISIONAL venue-years (see literature:corpus)
--------------------------------------------------------------------------
year	population	candidates	corpus	share of corpus
2010	0	3	119	0.0%
2011	0	4	116	0.0%
2012	0	4	151	0.0%
2013	0	2	125	0.0%
2014	1	1	166	0.6%
2015	1	6	190	0.5%
2016	1	15	182	0.5%
2017	0	18	231	0.0%
2018	0	13	254	0.0%
2019	3	29	402	0.7%
2020	2	18	404	0.5%
2021	3	34	379	0.8%
2022	8	50	546	1.5%
2023	5	55	719	0.7%
2024	9	80	690	1.3%
2025	12	89	770	1.6%
2026	6	50	415	1.4%

2010-2020: 8 papers    2021-2026: 43 papers

4. Per venue, against each venue’s own size
-------------------------------------------
venue	population	venue papers	share of that venue
PETS	25	510	4.9%
USENIX	9	1410	0.6%
CCS	5	990	0.5%
IEEE-SP	5	767	0.7%
WWW	4	843	0.5%
NDSS	2	701	0.3%
IMC	1	638	0.2%

5. What the roadmap’s title-and-summary probe was worth
-------------------------------------------------------
title+summary candidates: 9
  of those, in the population: 9  (precision 100.0%)
  of the population, caught:   9/51  (recall 17.6%)

the title probe MISSES these population papers:
  2014 CCS	[S]	The Web Never Forgets: Persistent Tracking Mechanisms in the Wild.
  2015 PETS	[S]	Automated Experiments on Ad Privacy Settings
  2019 NDSS	[D]	quantity-vs-quality-evaluating-user-interest-profiles-using-ad-preference-managers
  2019 WWW	[D]	Detecting Low Self-Esteem in Youths from Web Search Data.
  2020 PETS	[R]	CanaryTrap: Detecting Data Misuse by Third-Party Apps on Online Social Networks
  2020 USENIX	[D]	What Twitter Knows: Characterizing Ad Targeting Practices, User Perceptions, and Ad Explanations Through Users' Own Twitter Data
  2021 PETS	[S]	Privacy Preference Signals: Past, Present and Future
  2021 PETS	[R]	Data Portability between Online Services: An Empirical Analysis on the Effectiveness of GDPR Art. 20
  2021 USENIX	[D]	SocialHEISTing: Understanding Stolen Facebook Accounts
  2022 CCS	[S]	Do Opt-Outs Really Opt Me Out?
  2022 PETS	[M]	“It Feels Like Whack-a-mole”: User Experiences of Data Removal from People Search Websites
  2022 PETS	[R]	Leave No Data Behind – Empirical Insights into Data Erasure from Online Services
  2022 PETS	[R]	Revisiting Identification Issues in GDPR ‘Right Of Access’ Policies: A Technical and Longitudinal Analysis
  2022 PETS	[M]	Setting the Bar Low: Are Websites Complying With the Minimum Requirements of the CCPA?
  2022 USENIX	[M]	How Are Your Zombie Accounts? Understanding Users' Practices and Expectations on Mobile App Account Deletion
  2022 USENIX	[D]	Understanding and Improving Usability of Data Dashboards for Simplified Privacy Control of Voice Assistant Data
  2022 IEEE-SP	[R]	Scraping Sticky Leftovers: App User Information Left on Servers After Account Deletion.
  2023 PETS	[R]	Lessons in VCR Repair: Compliance of Android App Developers with the California Consumer Privacy Act (CCPA)
  2023 NDSS	[M]	CHKPLUG: Checking GDPR Compliance of WordPress Plugins via Cross-language Code Property Graph
  2023 PETS	[M]	Comparing Large-Scale Privacy and Security Notifications
  2023 IEEE-SP	[D]	Collaborative Ad Transparency: Promises and Limitations.
  2024 PETS	[S]	Johnny Still Can't Opt-out: Assessing the IAB CCPA Compliance Framework
  2024 PETS	[R]	What to Expect When You’re Accessing: An Exploration of User Privacy Rights in People Search Websites
  2024 PETS	[D]	Overprofiling Analysis on Major Internet Players
  2024 USENIX	[R]	Data Subjects' Reactions to Exercising Their Right of Access
  2024 PETS	[S]	Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?
  2024 WWW	[D]	TikTok and the Art of Personalization: Investigating Exploration and Exploitation on Social Media Feeds.
  2024 IEEE-SP	[M]	Withdrawing is believing? Detecting Inconsistencies between Withdrawal Choices and Third-party Data Collections in Mobile Apps.
  2025 CCS	[R]	Hidden in Plain Bytes: Investigating Interpersonal Account Compromise with Data Exports.
  2025 PETS	[R]	Echoes of Privacy: Uncovering the Profiling Practices of Voice Assistants
  2025 PETS	[S]	More and Scammier Ads: The Perils of YouTube's Ad Privacy Settings
  2025 IMC	[R]	From Voice to Ads: Auditing Commercial Smart Speakers for Targeted Advertising based on Voice Characteristics.
  2025 USENIX	[D]	As Advertised? Understanding the Impact of Influencer VPN Ads
  2025 USENIX	[R]	Addressing the Address Books' (Interdependent) Privacy Issues
  2025 PETS	[M]	``Erasing the Echo'': The Usability of Data Deletion in Smart Personal Assistants
  2025 USENIX	[R]	No Way to Sign Out? Unpacking Non-Compliance with Google Play's App Account Deletion Requirements
  2025 PETS	[R]	Measuring the Accuracy and Effectiveness of PII Removal Services
  2026 IEEE-SP	[R]	Consumer Beware! Exploring Data Brokers' CCPA Compliance.
  2026 WWW	[D]	Bowling with ChatGPT: On the Evolving User Interactions with Conversational AI Systems.
  2026 WWW	[R]	Does Ad-Free Mean Less Data Collection? An Empirical Study of Platform Data Practices and User Expectations.
  2026 IEEE-SP	[R]	Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube.
  2026 PETS	[M]	On the Suitability of LLM-Driven Agents for Dark Pattern Audits

the title probe returns these NON-population papers:

6. How big is the sample when a human sends the requests?
---------------------------------------------------------
17 papers sent requests. Controllers asked, hand-keyed from each paper:

year	venue	asked	what
2020	PETS	100	third-party Facebook apps sent a data-deletion request: 87 by email, 13 through a contact form. "Only 52% of the apps (45 out of 87) responded back to our requests" counts only the emailed 87
2021	PETS	182	online services sent an Art. 20 export request ("135 of 182 services (74.2%)")
2022	PETS	90	online services ("we investigated the data deletion practices of a sample of 90 online services")
2022	PETS	40	organisations re-evaluated in 2021, by registered letter, under 5 assumed subject identities
2022	IEEE-SP	678	apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand
2023	PETS	109	of 160 selected Android apps -- the subset whose privacy policies carried CCPA disclosures -- sent a verifiable consumer request: "We then submitted VCRs to these 109 companies". 160 is the number selected, not the number asked
2024	PETS	20	people-search websites, access and removal attempted by four researchers
2024	USENIX	-	33 participants requested their own exports; 801 files. Controllers are not enumerated
2025	CCS	6	services (Apple iCloud, Google, Facebook, Instagram, Snapchat, Discord), 12 researcher accounts
2025	PETS	2	Amazon and Apple, DSARs used to discover the interest labels to target; 200 fresh accounts
2025	IMC	1	Amazon, asked for the puppet accounts’ advertising data as a cross-check on the audit
2025	USENIX	5	address-book service providers sent a data-access request ("No provider supplied the requested contact-card data")
2025	USENIX	494	apps whose deletion method was classified, out of 863 with deletion-link data
2025	PETS	4	paid PII-removal services subscribed to by 71 participants, which then send the opt-outs on their behalf. The paper surveys 10 services, which "cover a total of 1,759 unique data brokers"; its abstract says 2,024, and the body sentence is the one used
2026	IEEE-SP	454	of the 543 registered California data brokers (84%; the rest were excluded), each sent a verifiable consumer request by hand: "In total, 9 hours and 57 minutes were spent submitting VCRs to all 454 DBRs"
2026	WWW	6	author-owned platform accounts; Art. 15(3) requests before and after taking an ad-free subscription
2026	IEEE-SP	3	platforms (Instagram, TikTok, YouTube), via sock-puppet accounts and 80 recruited participants

  a number is stated for 16 of 17
  median 30   min 1   max 678
  at or below 100: 11 of 16

  BASE RATE: of the 1120 corpus papers that ran a crawl, 603 state a site-or-domain population[].n; their median is 10000.

7. Is it an annotation problem? (share with humanAnnotation, vs the corpus)
---------------------------------------------------------------------------
population       : 43/51	84.3%
corpus base rate : 3318/5859	56.6%

of those that annotated, share stating an agreement metric:
population       : 16/43	37.2%
corpus base rate : 512/3318	15.4%

8. Ethics review, against the corpus base rate for the same field
-----------------------------------------------------------------
population, empirical : 32/51	62.7% state a review outcome
corpus base rate      : 1728/5118	33.8%

reviewOutcome values in the population (sentinels shown, never counted as answers):
  approved	22
  none-mentioned	19	(sentinel)
  explicitly-discussed-no-review	4
  not-required	4
  exempt	2

9. Which laws the population names (legal[], paper-counted)
-----------------------------------------------------------
31 of 51 papers assess a law. Folded law names, counted by paper:
  GDPR	19
  CCPA	15
  CPRA	5
  ePrivacy Directive	2
  Right to be Forgotten (RTBF) ruling	1
  Right to be Forgotten	1
  Payment Accounts Directive 2014/92/EU	1
  Gramm-Leach-Bliley Act (GLBA)	1
  California Financial Information Privacy Act (CalFIPA)	1

BASE RATE: 402 of 5859 corpus papers assess any law (6.9%);
the population's share is 60.8%.

10. foundViolations, over the tuples in the population that assessed one
------------------------------------------------------------------------
48 legal tuples across 31 papers (TUPLES, not papers -- a paper can assess several laws):
  yes	20	41.7%
  not-assessed	17	35.4%
  partial	11	22.9%

11. The page’s own population table, checked against the verdicts both ways
---------------------------------------------------------------------------
page table: 51 rows, all match the verdicts, none missing
page 'By instrument' sentence: {"R":17,"S":12,"M":10,"D":10,"O":2} — agrees
S papers from 2024 onward: 7 of 12
</file>

Two things in that output need a note.

**The //controllers asked// table is hand-keyed, and that is deliberate.** The first version derived it from ''population[].n'' by taking the largest value whose unit could plausibly be a request target. It gave CanaryTrap **43,332** — a website list the paper never contacted — against a real request sample of 87 apps, and a spurious median of 190. ''population[].n'' answers "how big was the biggest thing this paper drew", not "how many organisations did it write to", and no filter on ''unit'' fixes that. The table is now keyed from each paper's own sentence and the script throws if an R paper is missing from it. Four corrections came out of hand-keying: CanaryTrap 43,332 → 87, //Consumer Beware!// 543 → **454** (543 are registered; 89 were excluded and 454 actually received a request), //Scraping Sticky Leftovers// 1,435 → **678** (the apps with their own sign-up, which is the set whose accounts were created and deleted), and — found by the generic reviewer after publication — //Lessons in VCR Repair// 160 → **109**, because 160 is the number of apps selected and 109 is the subset whose policies carried CCPA disclosures and which actually received a request. The guards check that every R paper //has// a value, not that the value is right; only reading the paper does that.

**Every subgroup share is printed next to the corpus base rate for the same field.** The population's 84.3% hand-annotation rate only means something against the corpus's 56.6%; its 60.8% law-assessment rate only means something against 6.9%. Sections 7, 8 and 9 of the output print both sides.

===== 5. Folding =====

Only one field on this page is aggregated from free text, and it is small enough to fold by hand and print in full.

''legal[].law'' over the 51 population papers, 48 tuples across 31 papers. Four rules, applied in order, each a case-insensitive match on the law string:

| ''/ccpa|california consumer/'' → **CCPA** | ''/cpra|california privacy rights/'' → **CPRA** |
| ''/gdpr|general data protection/'' → **GDPR** | ''/eprivacy/'' → **ePrivacy Directive** |

**Residue — every law string the four rules did not touch, printed in full**, one paper each: //Right to be Forgotten (RTBF) ruling//, //Right to be Forgotten//, //Payment Accounts Directive 2014/92/EU//, //Gramm-Leach-Bliley Act (GLBA)//, //California Financial Information Privacy Act (CalFIPA)//. The two "Right to be Forgotten" strings are the same object under two spellings and are **not** merged, because merging them would create a fifth folded family of size 2 that the page does not use; they are counted as they are and the page quotes only GDPR 19 / CCPA 15 / CPRA 5 / ePrivacy 2. Folding them would not change any published figure.

Nothing else on the page aggregates a free-text field. ''detection[].phenomenon'' and ''detection[].metric'' are used only as pointers to a paper's own sentence, never counted; ''tools[]'', ''classification.resourceName'' and ''vantage.serviceName'' are not used at all.

===== 6. Quotes checked =====

''scripts/dsr_quotecheck.py'' extracts every ''%%//"..."//%%'' span from the page source and checks it appears verbatim in its source, after collapsing whitespace, joining hyphens split at a line break, folding curly quotes and dashes, and unwrapping Markdown link syntax. Corpus quotes are checked against ''paper.cols.txt'', then ''paper.norm.txt'', then a fresh ''pypdf'' extraction of ''paper.pdf''. External quotes are checked against a **byte snapshot taken at fetch time** and stored under ''dsr/external/'', not re-fetched at check time: a live re-fetch turns a transient rate-limit or 403 into "quote not found", and would silently pass a sentence that has since been removed. The script exits non-zero if any quote fails **or** if the page carries a quoted span with no source listed, so the list cannot drift away from the page.

<file text dsr_quotecheck-output.txt>
OK  [cols]  Above 40% failed to respond at all, in an apparent violation of the CC...
OK  [snapshot]  one month of receipt of the request...
OK  [snapshot]  by a maximum of two months...
OK  [snapshot]  once by an additional 45 days when reasonably necessary, provided the ...
OK  [cols]  In total, 9 hours and 57 minutes were spent submitting VCRs to all 454...
OK  [cols]  varied both across and within platforms...
OK  [pdf]  the failure to disclose processing purposes, retention periods, and ot...
OK  [cols]  requested personal information as part of their identity verification ...
OK  [cols]  exercising one's privacy rights under CCPA introduces new privacy risk...
OK  [snapshot]  closed on 17 January 2019...
OK  [snapshot]  the right to object to the processing of personal data can also be exp...
OK  [snapshot]  Disclose how the operator responds to Web browser “do not track” signa...
OK  [snapshot]  The US Privacy signal has been deprecated as of January 31, 2024. We s...
OK  [snapshot]  Currently, GPC is the only UOOM considered valid by The Department...
OK  [snapshot]  Beginning July 1, 2024...
OK  [snapshot]  Tell websites not to sell or share my data...
OK  [snapshot]  beginning January 1, 2027...
OK  [snapshot]  from developing or maintaining a browser, as defined, that does not in...
OK  [snapshot]  This year California signed a bill under the CCPA/CPRA that obligates ...
OK  [cols]  Do Not Sell My Personal Information...
OK  [cols]  about a third of sites that have evidence of selling or sharing person...
OK  [cols]  only 48 out of 100 apps implement a respective setting...
OK  [cols]  largely ineffective...
OK  [snapshot]  A total of 1,185 controllers responded to the questionnaire...
OK  [snapshot]  a total of 764 controllers...
OK  [snapshot]  surprised about the large number of controllers responding having rece...
OK  [snapshot]  As of January 1, 2026, California residents may use DROP...
OK  [snapshot]  Data brokers are required to begin processing these requests on August...
OK  [cols]  opt-out request...

29 quoted spans, 0 failed, 0 unlisted
</file>

Two of those results are worth recording:

  * **''[pdf]'' on the //Setting the Course// quote is not a formatting detail.** The sentence //"the failure to disclose processing purposes, retention periods, and other third-party data recipients serves as a further indicator of non-compliance"// is absent from both ''.cols'' and ''.norm'': the two-column repair splices the adjacent column into the middle of it. Checking only ''.cols'' would have published a false FAIL against a correct citation, and — worse — a run that trusted the FAIL would have deleted a true quote.
  * **The W3C snapshot had to be taken twice.** The first fetch, with a browser-like User-Agent, returned **HTTP 403** and a 5,667-byte error page, which the snapshot writer happily stored; the quote then failed against it. The second fetch, with a plain ''Mozilla/5.0'', returned **HTTP 200** and 40,527 bytes. The snapshot writer now asserts a 200 before writing. A cached transport failure is not evidence of anything.

Separately, six ''evidence.quote'' values from the extraction — the ones behind the largest published figures — were checked against their papers by the same normalisation. Four matched a ''.txt'' rendering directly:

^ Paper ^ Extraction quote ^ Result ^
| {[syrmoudis2021_data]} | //"Only 135 of the 182 services (74.2%) managed to execute some sort of data export in the legally valid time horizon"// | found in ''.cols'' and in the PDF |
| {[nortwick2022_setting]} | //"In total, 9,838 websites (2%) in our first crawl contain a DNSMPI link."// | found in ''.cols'' and in the PDF |
| {[aziz2024_johnny]} | //"In Crawl 1 we detected the presence of the USP API on 821 (8.2%) publishers."// | found in ''.cols'' and in the PDF |
| {[hausladen2025_websites]} | //"In December 2023, 44% (1,411/3,226) of such sites opted users out via all implemented privacy strings."// | found in ''.cols'' and in the PDF |

The other two are the interesting ones. Both are **ellipsis-joined** extraction quotes — the extractor stitched two non-adjacent sentences with ''...'' — so neither matches as a single span, and a naive check reports NOT FOUND on a correct figure. Split at the ellipsis and checked fragment by fragment:

  * {[farooqi2020_canarytrap]}: //"Only 52% of the apps (45 out of 87) responded back to our requests"// is in ''.cols'' verbatim. The second fragment is a paraphrase of Table 2, whose cells the column repair has flattened into running text (''Requested data deletion successfully 87 apps Not responded 42 apps Responded 45 apps''); the numbers are the paper's. **This check also corrected the page**: the 87 is the number of apps contacted //by email//, and 13 more were contacted through a contact form, so the request sample is **100**, not 87. The page and the report script now say 100 and name the split.
  * {[rupp2022_leave]}: //"we find that 24 (27%) of the 90 services did not complete the data erasure request in a way which ensures that all personal data is deleted or anonymized"// is in the paper, interrupted in ''.cols'' by a table caption spliced in from the adjacent column. Confirmed by reading the sentence around it.

No extraction quote used on this page turned out to be wrong about its paper. One was wrong about its **denominator**, and only reading the surrounding paragraph caught it.

===== 7. External sources =====

Every external claim on the content page, with the primary source it was read from and the HTTP status at fetch time. Byte snapshots of the ones carrying a published quote are under ''dsr/external/'' and are what ''dsr_quotecheck.py'' checks against.

^ Claim on the page ^ Primary source, read 2026-09-16 ^ Status ^
| GPC is a W3C Working Draft of 11 June 2026, Privacy Working Group; editors Zimmeck, Snyder, Brookman, Zucker-Scharff | https://www.w3.org/TR/gpc/ | 200, 99,238 bytes |
| GPC browser support: Firefox 120 desktop, Firefox 122 Android, opt-in; not Chrome, not Safari | ''mdn/browser-compat-data'', ''api/Navigator.json'', field ''api.Navigator.globalPrivacyControl'' | 200, 196,919 bytes |
| Brave, DuckDuckGo and Mozilla are among the organisations implementing GPC | https://globalprivacycontrol.org/orgs | 200, 24,928 bytes |
| The US Privacy String was deprecated on 31 January 2024 | https://github.com/InteractiveAdvertisingBureau/USPrivacy README | 200, 5,163 bytes |
| GPP renamed //Global Privacy Platform// → //Global Privacy Protocol//; spec page last updated 12 August 2026; August 2026 MSPA update in public comment until 11 September 2026 | https://iabtechlab.com/gpp/ | 200, 121,863 bytes |
| The W3C Tracking Protection Working Group closed on 17 January 2019; DNT is a retired specification | https://www.w3.org/2011/tracking-protection/ | 200, 40,527 bytes (see §6 — the first attempt returned 403) |
| GPC is the only universal opt-out mechanism Colorado recognises, and is mandatory from 1 July 2024 | https://coag.gov/uoom/ | 200, 81,392 bytes |
| DROP: consumers from 1 January 2026, brokers must process from 1 August 2026 | https://cppa.ca.gov/data_broker_registry/ | 200, 95,726 bytes |
| EDPB CEF 2024, right of access: 30 SAs, 1,185 controllers, adopted 16 January 2025 | ''edpb_cef-report-2024_20250116_rightofaccess_en.pdf'' | 200, 659,756 bytes, 31 pages |
| EDPB CEF 2025, right to erasure: 32 SAs, 764 controllers, adopted 10 February 2026 | ''edpb_cef-report_2025_right-to-erasure_en.pdf'' | 200, 441,986 bytes, 29 pages |

**Rejected, and why.** These were found and deliberately not used:

  * **Every vendor and consultancy "GPC in 2026" guide** returned by search — weblegal.ai, consenteo.com, vucense.com, ketch.com, getadmiral.com, cookiebeam.com, sourcepoint, cookiehub, onetrust's knowledge base. Several were accurate. None is a primary source, all are marketing for a consent product, and at least two disagreed with the W3C and IAB documents on the specification's status. The page cites the W3C and the IAB instead.
  * **An AI search summary claiming the California data broker registry currently lists 0 brokers.** It was a mis-parse of a dynamically loaded table; the raw page contains no such statement. The page publishes no current broker count at all — it quotes {[kempen2026_consumer]}'s **543 registered / 454 asked** with the study's own date, which is checkable. The registry's live table is behind a viewer this sandbox could not read, and guessing was not worth it.
  * **Law-firm and IAPP commentary on the EDPB coordinated actions.** Useful for orientation, and the source of the pointer to the 2025 erasure action, but every figure on the page comes from the EDPB's own PDFs, read page by page.
  * **The Colorado AG's //Universal Opt-Out Mechanisms: Global Privacy Control Comments// page** (https://coag.gov/resources/universal-opt-out-mechanisms/) — a navigation stub with no substantive content at fetch time. The substantive statement is on ''coag.gov/uoom/'' and that is what is cited.

===== 8. What could not be established =====

  * **A current count of registered California data brokers.** See above. The Delete Act's 1 August 2026 processing deadline has now passed and there is presumably compliance data somewhere; none was found in a form this sandbox could read.
  * ~~Whether the EDPB has chosen a 2026 CEF topic.~~ **Closed during the run** by the external-currency reviewer: CEF 2026 launched 19 March 2026 with 25 DPAs, on transparency and information obligations under Arts. 12–14 — not a rights action. It is now on the content page. The EDPB's own search endpoint returns 404; the news page had to be reached by its slug.
  * **The false-negative rate of the hand audit.** Two probes added mid-run moved the population from 41 to 51 — a 20% under-count that survived a complete audit — which bounds the error at "at least this large, once". There is no way to estimate it from inside the process. A second, independently written probe set, ideally by someone else, is the only thing that would.
  * **Whether any DSAR study has been replicated.** {[kempen2026_consumer]}'s Table 1 lines up **13 prior studies'** response rates (**55%–100%**, over populations of 20 to 454) but they are thirteen populations, not thirteen attempts at one. An earlier draft of the content page said "seven … 57% to 93%" in one place and "four … none of which is in this corpus" in another; both were invented counts, both were wrong, and three of the thirteen are in this page's own population. Neither the quote checker nor the figure checker could see it — a paraphrased count with no quoted span and no script behind it is outside every guard this page has. Nothing in the corpus is a replication.
  * **Inter-requester variance.** No paper in the population sends the same request twice, or has two people send it. The content page states this as a gap, and **scopes it to the 51 papers' abstracts and methodology sections** — which is as far as the reading went. The ''request_sent'' probe's 99 hits were screened at the matched-sentence level like every other candidate, so an appendix could refute either claim.
  * **What share of the population's crawls used Chrome without injecting ''Sec-GPC''.** ''crawlConfig.browsers'' is stated by only 47.2% of the 1,120 crawling papers corpus-wide, and the population is too small for that field to say anything. The page makes the point as advice, not as a measured claim.

===== 9. Judgement calls =====

Each of these could reasonably have gone the other way.

  * **A new page rather than a section on [[Privacy:Consent]] or [[Privacy:Policies]].** The queued reasoning held. The decisive argument is the one the content page opens with: the instrument is a person, so the sample-size, ethics and non-response problems are structurally different from anything else in the ''privacy:'' namespace, and they would have been a footnote inside either neighbour.
  * **Requests and opt-out signals on one page.** They are two instruments with different scales — 30 controllers against 10,000 sites. The alternative was two pages. They are together because they answer the same question ("does the right work in practice"), a student who needs one usually needs the other, and splitting them would have put the CCPA on both pages and the GDPR on neither. The cost is a long page.
  * **//Johnny Can't Revoke Consent Either// (PoPETs 2025) is OUT.** Consent withdrawal under Art. 7(3) is a data-subject right, and the paper measures whether websites let you exercise it. It is excluded because the interface it measures is the cookie banner, which [[Privacy:Consent]] owns. A reasonable person would include it. The boundary is stated on both the content page and here so the decision is visible rather than silent.
  * **//On the Suitability of LLM-Driven Agents for Dark Pattern Audits// (PoPETs 2026) is IN, //WebSP-Eval// (PoPETs 2026) is OUT.** Both measure an LLM agent rather than a website. The first is included because its entire substrate is CCPA rights-request portals and it reports on those portals across a 456-site registry census; the second is excluded because an opt-out task is one item in a general web-security benchmark. This is the least comfortable call in the audit and is flagged on the content page.
  * **//Access Denied!// (PoPETs 2016) is OUT.** It is the earliest paper in the corpus about the right of access and it is a survey of 873 people's attitudes, with no request sent and no mechanism observed. Excluding it makes the population's start date 2014 rather than 2016 and removes the one paper a reader might expect to find. It is cited nowhere on the content page, which is arguably a loss.
  * **Category **D** requires the paper's //dataset// to come from an access or export surface, regardless of the research question.** That is why //Detecting Low Self-Esteem in Youths from Web Search Data// (TheWebConf 2019) — a mental-health prediction paper — is in the population: its data is Google Takeout histories donated by participants. Under a research-question rule it would be out. The dataset rule was chosen because "you can get data this way" is the methodologically useful fact for the reader, and because a research-question rule made the boundary unstable across four other papers.
  * **//Are Privacy Dashboards Good for End Users?// (USENIX 2021) is OUT and //Understanding and Improving Usability of Data Dashboards// (USENIX 2022) is IN**, although both are surveys about a Google transparency surface. The 2022 paper's participants donated their actual My Activity data, which the paper then analyses; the 2021 paper's dataset is survey responses about having looked. That is the D rule applied consistently, and it separates two papers that look alike.
  * **Machine unlearning is out, categorically.** Ten candidates. The word "right to be forgotten" is the same; nothing else is.

===== 10. The run =====

^ ^ ^
| Date | 2026-09-16 |
| Corpus | ''data/extract/run1'', 5,859 extracted papers, 5,855 with full text, seven venues, 2010–2026 |
| Pages written | [[privacy:data_subject_rights]] (new), this page (new) |
| Pages edited | [[:roadmap]] (queued row → assessed, with corrected counts; queue now empty), [[:privacy|Privacy]] (18th child, count 17→18), [[Privacy:Requests]] (one-line disambiguator), [[literature:bibliography]] (+20 entries) |
| Scripts added | ''dsr_probe.mjs'', ''dsr_ctxdump.mjs'', ''dsr_verdicts_build.py'', ''dsr_report.mjs'', ''dsr_quotecheck.py'', and ''dsr_verdicts.json'' / ''dsr_quote_sources.json'' |
| Scripts changed | ''sitemap.mjs'' — the roadmap gate threw on an empty Queued table, which is now a legitimate state; it accepts one only if the section says //the queue is empty// in words |
| Bibliography | 20 new entries, 0 citekey collisions, 0 definite duplicates under ''bib_dedup_scan.py'' (96 same-surname candidates, all judged noise) |
| Models | Opus 5 wrote the page and ran the audit; three Sonnet reviewers (figures-vs-script, citations-and-quotes, external currency) and one Fable generic reviewer, findings in §11 |

**Two things worth recording about the run itself.**

**Seven PoPETs author lists had to be filled by hand**, because the corpus index carries no authors for any PoPETs or USENIX record. ''fetch_authors.py'' resolved the USENIX ones and failed on all seven PoPETs landing pages. The first-page text of ''paper.cols.txt'' was **not** sufficient: for three of the seven it silently dropped authors — //Erasing the Echo// lost Kopo M. Ramokapane, //Measuring the Accuracy and Effectiveness of PII Removal Services// lost Peter Snyder, Hamed Haddadi and Gareth Tyson, and //Overprofiling Analysis// lost José González-Cabañas and Rubén Cuevas — because the two-column repair reads only one column of the author block. All seven were taken from the PoPETs landing page instead, which lists authors and affiliations in one line.

**The bibliography needed a cache purge.** After 20 entries were appended and the page saved, the content page rendered **32 of 52** references, with gaps in the numbering — exactly the 32 that were in the bibliography before. Requesting ''literature:bibliography'' with ''?purge=true'' and then the content page with ''?purge=true'' fixed it: 52 references, no gaps. Counting rendered ''%%<dt>%%'' entries against distinct citekey markers is the check; the citation links themselves were present and blue in both states, so nothing but the count would have caught it.

===== 11. Review =====

Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the scripts and their output, and these notes. The three focused passes ran in parallel first; the generic pass ran last, after their findings were applied, and its own findings are in §11.4. Findings are logged below whether accepted or rejected — a rejection is the only record of whether a reviewer slot is earning its place.

==== 11.1 External currency (model: sonnet) ====

The most productive of the four. It re-fetched all ten external claims and checked each against a primary source.

^ Finding ^ Disposition ^
| The ten claims are all correct as written: GPC's Working Draft date and editors, the Firefox/Chrome/Safari support matrix, the US Privacy String deprecation, the GPP rename and dates, the W3C group's closure, Colorado's UOOM list, both DROP dates, both EDPB reports, and the two statutory deadlines. | **Confirmed, no change.** It also noted that several secondary sources date the EDPB erasure report to 18 February 2026 where the PDF itself says 10 February; the page already had the primary-source date. |
| **Chrome is no longer static on GPC.** | **Accepted, with the reviewer's own evidence rejected.** The reviewer gave chromestatus id ''6249068542164992'', which is a TC39 iterator proposal, and asserted that Chrome Canary began sending ''Sec-GPC'' behind a flag in mid-2026. Neither checks out. The real entry is **5137324344213504**, at stage //Proposed//, last touched 2026-01-02, no milestone, filed from a non-Google address. What //is// real and load-bearing is the thing behind it: **California AB 566**, Chapter 465, approved 8 October 2025, which from 1 January 2027 forbids maintaining a browser without a configurable opt-out preference signal. That is now on the page with the statute as its source, and the chromestatus entry is cited for what it actually is. |
| The Colorado-only framing undersells a multi-state cluster (CT, DE, MT, OR, TX confirmed; five more reported but not verified against state sources). | **Accepted in substance, not in form.** A state count is now explicitly //not// published: the circulating lists are vendor trackers, effective dates differ from enforcement dates, and no authoritative register exists. The page names the states it can name, dates the two claims it can source, and tells the reader to derive the count from each AG's own page. |
| DROP has an early volume figure (345,000+ requests by 7 August 2026). | **Rejected as a citation, accepted as a prompt.** Every source the reviewer offered was a law firm or a vendor. The agency's own announcements page carries no such figure. The page instead says the 1 August 2026 deadline has passed, that no agency figure could be found, and that this is the most obviously available study on the topic. |
| The EDPB's 2024→2025 series looks cut off; the 2026 topic has been announced. | **Accepted.** Verified at the EDPB's own news page: CEF 2026 launched 19 March 2026, 25 DPAs, on transparency and information obligations under Arts. 12–14 — not a rights action. One sentence added so the series does not read as unfinished. |
| The GPP public-comment window closed on 11 September 2026 with no outcome posted. | **Accepted**; the row is now in the past tense and says the outcome is still pending. |

==== 11.2 Citations and quotes (model: sonnet) ====

Checked all 52 citekeys, the 20 new bibliography entries against both the PDFs and the publisher landing pages, all quoted phrases, and the external footnotes.

^ Finding ^ Disposition ^
| All 20 new bibliography entries are correct — authors, author order, title, venue and year — including the two 2026 TheWebConf entries, which it confirmed are registered ACM publications rather than preprints. All footnotes resolve and match. All 34 quotes and figures it checked are verbatim or numerically exact with the right denominator. | **Confirmed, no change.** |
| **The lead box said "43% of California's registered data brokers never replied at all".** The paper's 43% is **195 of the 454** it wrote to; against the 543-entry registry it would be 36%. | **Accepted, fixed.** This is the exact defect the page spends a section warning about, in the page's most-read paragraph, on a figure the author had already keyed correctly two screens further down. |
| **{[zimmeck2024_generalizable]} was categorised S but is a 410-participant usability study of a GPC choice interface**, not a measurement of site-side deployment; it fits **M**. | **Accepted.** Recategorised, and every dependent figure moved with it: the by-instrument line is now **R 17, S 12, M 10, D 10, O 2**, and the //methods are current// table now says seven of the twelve S papers are 2024 or later. The report script gained a check that reads the published table and compares it to the verdicts **in both directions** — a row the page invented, and a paper the page dropped — plus the by-instrument sentence and the category letters. Both mutations (delete a row; flip one letter) make it throw. |
| **''UNMATCHED: []'' was published inside the population table.** | **Accepted, fixed.** A diagnostic ''print'' from the throwaway script that generated the table's rows leaked into the file that was pasted into the page, and it survived the wrap check, the quote check, the citekey check and a rendered-DOM inspection, because none of them looks for text that is merely //wrong to be there//. Same failure as a generator's stdout reaching a live bibliography page on this wiki before. |
| **"Brave and DuckDuckGo ship it; Chrome does not" sat immediately after the MDN footnote but is not covered by it** — neither vendor is in MDN's compatibility table. | **Accepted, rewritten.** Brave's own support site returns 403 to this sandbox and DuckDuckGo's help page renders client-side, so neither vendor's documentation could be read. The page now cites the GPC project's implementers list for the weaker claim it actually supports, says explicitly that the list does not state default-on behaviour, and turns the paragraph into advice that is better anyway: **verify the signal on the wire against a server you control** rather than trusting any support table. |
| The two red links to ''provenance:privacy:data_subject_rights'' look like a stale cache, because the page "returns HTTP 200". | **Rejected.** DokuWiki serves the themed error page with HTTP 200 for a page that does not exist; ''export_raw'' on a missing page returns HTML rather than an error. The provenance page genuinely did not exist at review time — it is this one. |
| ''abramova2023_anatomy'' is missing from ''dsr/citekeys.json''. | **Accepted as accurate, not fixed.** That map is built from the population only, and the entry is an out-of-population aside. The report script's new page-vs-verdicts check reads the map in the direction that matters (citekey → population paper) and throws on a page row it cannot resolve, so an incomplete map cannot hide a bad row. |

**Re-review after the fixes.** The citations reviewer was re-run on the changed page and confirmed all four of its own findings fixed, and checked the material added afterwards: the Landgericht Berlin and CalOPPA quotes, AB 566's text and chapter number, the EDPB CEF 2026 figures, both statutory-deadline quotes, and the two rescoped "no paper in this population" claims against {[martino2022_revisiting]} and {[take2024_expect]}. All matched. It also independently re-derived **R 17 / S 12 / M 10 / D 10 / O 2** and the seven-of-twelve claim from the verdict file. Two notes from it, one accepted and one rejected:

  * **Accepted:** the EDPB CEF 2026 URL 301-redirects to a slightly different slug. It resolves 200 with ''-L''; a future check that omits ''-L'' will see the redirect and should follow it rather than record a failure.
  * **Rejected as a finding, recorded as a limitation:** it could not verify the ''chromestatus'' feature entry or the Colorado ''coag.gov/uoom/'' page, because ''api.chromestatus.com'' refused its TLS connection and both pages render client-side for ''WebFetch''. Both **were** fetched successfully during the run — chromestatus through its JSON API (HTTP 200, 11,059 bytes) and ''coag.gov/uoom/'' with ''curl'' and a browser User-Agent (HTTP 200, 81,392 bytes) — and both are stored as dated byte snapshots under ''dsr/external/'', which is what ''dsr_quotecheck.py'' checks the published quotes against. This is the case the snapshot design exists for: a source that one tool can read and another cannot is not a source that has gone away. **Closed after publication:** the reviewer's own delegated check later reached the chromestatus JSON and confirmed every claim independently — the feature name, the motivation string as an exact substring, the //Proposed// stage, the absence of a milestone on all four stages, the 2026-01-02 update timestamp, and the non-Google creator address. The same pass re-verified all 8 citekeys on this page and all six rows of the extraction-quote table in §6, including the two awkward ones: {[farooqi2020_canarytrap]}'s table paraphrase and {[rupp2022_leave]}'s column splice. ''coag.gov/uoom/'' remains unreadable to that toolchain and readable to ''curl''; the snapshot stands.

==== 11.3 Figures against the script (model: sonnet) ====

Re-ran every script, reproduced ''dsr_report-output.txt'' byte for byte, checked all 51 population-table rows on three fields each, all 28 quoted spans, and about 80 individual paper-level numbers across 20 papers — well over 150 figures in total. It also mutation-tested every guard.

^ Finding ^ Disposition ^
| Every figure quoted from the report script matches its output, and every paper-level number it spot-checked is exact with the right denominator — including the ''821'' versus ''825'' publishers in {[aziz2024_johnny]}, which are two different crawls and not an error. | **Confirmed, no change.** |
| **The //methods are current// table said hand-sent requests run "2016 → 2026".** The earliest **R** paper in the population is {[farooqi2020_canarytrap]}, 2020. | **Accepted, fixed** — and turned into a finding rather than a correction: **no paper in this corpus sent a rights request before 2020**, two years after the GDPR applied. That is now stated in the row. The error was a date, on a page whose own argument is that dating matters. |
| **The category-sum guard in ''dsr_report.mjs'' was vacuous.** It summed ''byCat''’s own values, so an invalid category still summed to the population while the printed table — which only prints R/S/M/D/O — was silently one paper short, and the script exited 0. The reviewer demonstrated this by assigning a paper the category ''X''. The page's //Methodology// section claimed the script throws in exactly that case. | **Accepted, fixed, and re-mutation-tested.** The guard now rejects any category outside the five and sums only over the five; the reviewer's mutation now throws. ''dsr_verdicts_build.py'' did already enforce the five-letter set, so no published number was ever wrong — but the page made a claim about the script that the script did not honour, which is the more interesting half. |
| All other guards fired correctly under mutation: the ASKED-map gap and stale-category checks, the corpus-membership check, the tight ⊆ loose narrowing check, and both asserts in ''dsr_verdicts_build.py''. | **Confirmed.** The page-versus-verdicts check added after §11.2 was mutation-tested separately by the author: deleting a table row and flipping a category letter both throw. |

==== 11.4 Generic, no checklist (model: fable) ====

The most useful pass of the four, because it found the class of error the other three cannot reach: **paraphrased counts with no quoted span and no script behind them.**

^ Finding ^ Disposition ^
| **{[kempen2026_consumer]}'s Table 1 was described twice, differently, and both descriptions were invented.** "Seven earlier studies … 57% to 93%" in one place, "four earlier response-rate studies, none of which is in this corpus" in another. The table has **13 prior rows**, the rates run **55%–100%**, and **three of the studies are in this page's own population**. | **Accepted, fixed in both places and in §8 above.** The strongest finding of the review layer. Nothing guards a number that is neither quoted nor computed. |
| **{[samarin2023_lessons]} was keyed at 160 controllers asked; the paper submitted VCRs to 109.** 160 is the number of apps selected. | **Accepted, fixed** in the page and in the ASKED map. No downstream figure moved — the median stays 30 and "eleven of sixteen at or below 100" holds — but it is precisely the denominator error the page spends a section warning about. |
| **The short version said "Do Not Track is dead as a legal signal" while the table two screens down said "legal status not zero".** And "three generations of signal … only one alive" sat above a four-row table calling GPP "live and the successor". | **Accepted, both fixed.** Two drafts of the same paragraph written a day apart, neither reconciled against the other. |
| Five arithmetic and wording slips: "40 times … twice over" (764/30 ≈ 25), "two papers in eleven years", "three in five" against "two in three" for the same 62.8%, "uses four of them" for four of four, and "Chrome and Chromium cannot send GPC" when Brave is Chromium and does. | **All accepted and fixed.** |
| **The lede promises the data-donation use case and the page does not deliver it.** Ten papers — a fifth of the population — were represented by one bullet inside the opt-out-signal section. | **Accepted.** A new section, //The export as a dataset//, now covers it: the six papers that do it, and the four things it costs (product export ≠ Art. 15 response; participants and IRB; format drift as a longitudinal confound; reusing somebody else's donation corpus). |
| **The DNT row in //Which methods are current// dated a set nobody audited.** No DNT-compliance paper is in the population and the ''dnt'' probe never drove a candidate. | **Accepted.** The row's period column now reads "none in this population" and says why. |
| **Provenance §3 pointed at a list of paper reads in §7 that does not exist**, and the page's "hand-auditing all 471" overstates what ~395 of the verdicts rest on. | **Accepted, both fixed here.** §3 now says plainly that ~395 verdicts are a screen on one matched sentence plus the title. |
| Provenance §8 still said the 2026 EDPB topic could not be established, and that the ''request_sent'' probe's 99 hits "were all read". | **Accepted, both fixed.** |
| The GPC boundary with [[Privacy:Consent]] is stated on one side only, and that page's advice ("do not inject the header; set the browser preference") reads as the opposite of this page's. | **Accepted in part.** A pointer has been added to [[Privacy:Consent]]. The deeper merge — deciding which page owns the GPC section outright — is a two-page edit that belongs to a sitting that can re-review both, and is recorded here rather than half-done. |
| Structure: "Read first" is last; the provisional-years caveat appears three times; the two results sections are split by instrument without saying so; tooling is the thinnest part of the page. | **Partly accepted.** The two results sections are now named //What the signal-side papers measured// and //What the request-side papers measured//. The rest — section order, the duplicate caveat, and a tools section the page genuinely lacks — is **not fixed**, and is the clearest thing for a later sitting to pick up. |
| ''utz2023_comparing'' is IN as M with no explanation on either page. | **Accepted as accurate, not fixed.** Its DNSMPI-link check is one of four privacy issues it measures, which is why it is in; the page never cites it in prose. A borderline inclusion that the verdict file records and the prose does not justify. |

===== 12. The scripts, as committed =====

These are the files themselves, not a sample. Each is pasted from the committed copy under ''scripts/''.

<file javascript dsr_probe.mjs>
// Full-text candidate probes for privacy:data_subject_rights.
//
// The roadmap row that queued this page was a TITLE+SUMMARY probe (9 papers,
// scripts/gap_probe_roadmap.mjs, family `dsar_rights`). A title probe is a
// floor: a paper that sends 100 access requests in one section of a study about
// something else is invisible to it. So the candidate set here is a full-text
// sweep over data/fulltext/<year>/<venue>/<slug>/paper.cols.txt, and every
// candidate is then hand-judged (see dsr_verdicts.json).
//
// Three things this script does on purpose, each because of a past defect:
//  * whitespace is COLLAPSED before matching, because a PDF line break inside
//    "data subject access request" otherwise silently drops the paper;
//  * acronym probes are CASE-SENSITIVE (\bGPC\b/i also matches "gpcr"), and
//    every acronym probe prints its residue so the homonyms stay visible;
//  * each tight probe is paired with a LOOSE superset and the script asserts
//    tight ⊆ loose, so a "narrowing" regex that is really a different regex
//    fails the run instead of returning a confident wrong number.
//
//   node scripts/dsr_probe.mjs                 # per-probe counts + union
//   node scripts/dsr_probe.mjs --list <probe>  # papers matching one probe
//   node scripts/dsr_probe.mjs --ctx <probe>   # + the matching sentence
//   node scripts/dsr_probe.mjs --union         # every candidate, one per line
//
import fs from 'node:fs';
import path from 'node:path';
import { loadExtractions, dataRoot } from './lib.mjs';

const ROOT = dataRoot();
const rows = loadExtractions();
const byKey = new Map(rows.map((p) => [`${p.venue}/${p.year}/${p.slug}`, p]));

export const key = (p) => `${p.venue}/${p.year}/${p.slug}`;

export function fullText(p) {
  const f = path.join(ROOT, 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt');
  if (!fs.existsSync(f)) return null;
  // Collapse every run of whitespace (including the newlines the two-column
  // repair leaves mid-phrase) to a single space. Also join a hyphen that a PDF
  // line break split: "dele-\ntion" -> "deletion".
  return fs.readFileSync(f, 'utf8').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' ');
}

// --- the probes -------------------------------------------------------------
// TIGHT is what the count is reported from; LOOSE is a strict superset used
// only to prove the tight pattern is a narrowing of the same idea.
export const PROBES = {
  dsar_phrase: {
    tier: 'core',
    tight: /\b(data )?subject access request|\bDSAR\b|subject rights request|\bright of access\b|\bright to access (their|his|her|one'?s|the) (own )?(personal )?data/,
    loose: /subject access|\bDSAR|access request|right of access|right to access/i,
    what: 'a request for a copy of one’s own data',
  },
  erasure: {
    tier: 'core',
    tight: /right to be forgotten|\bRTBF\b|right to erasure|right to deletion|right to delete\b|deletion request|erasure request|data deletion request|request(ed|ing)? (the )?(deletion|erasure) of/i,
    loose: /forgotten|erasure|deletion|delete/i,
    what: 'a request to have data erased',
  },
  gpc: {
    tier: 'core',
    tight: /Global Privacy Control|\bGPC\b[^.]{0,120}(privacy|opt-?out|signal|header|sell)|(privacy|opt-?out|signal|header|sell)[^.]{0,120}\bGPC\b/,
    loose: /global privacy control|\bgpc/i,
    what: 'the Global Privacy Control signal',
  },
  do_not_sell: {
    tier: 'core',
    tight: /[Dd]o [Nn]ot [Ss]ell|right to opt.?out of (the )?sale|opt.?out of sale|limit the use of (my )?sensitive personal/i,
    loose: /do not sell|opt.?out of|sale of (personal|my)/i,
    what: 'the CCPA/CPRA "Do Not Sell" opt-out surface',
  },
  dnt: {
    tier: 'context',
    tight: /Do Not Track|\bDNT\b/,
    loose: /do not track|\bdnt/i,
    what: 'the Do Not Track header (GPC’s predecessor)',
  },
  portability: {
    tier: 'context',
    tight: /right to (data )?portability|data portability|download (your|my|their) (own )?data|takeout/i,
    loose: /portab|takeout|(download|export) (your|my|their|the)/i,
    what: 'a data-export / portability request',
  },
  ccpa_law: {
    tier: 'context',
    tight: /\bCCPA\b|\bCPRA\b|California Consumer Privacy Act|California Privacy Rights Act/,
    loose: /ccpa|cpra|california consumer privacy|california privacy rights/i,
    what: 'the California statutes that create the opt-out right',
  },
  gdpr_article: {
    tier: 'context',
    tight: /Article ?1[5-8] (of )?(the )?GDPR|GDPR Article ?1[5-8]|Art\.? ?1[5-8] GDPR|data subject rights?\b/i,
    loose: /article ?1[5-8]|art\.? ?1[5-8]|data subject right/i,
    what: 'the GDPR data-subject-rights articles, named',
  },
  authorized_agent: {
    tier: 'context',
    tight: /authoriz(ed|ing) agent|privacy request|consumer request|verifiable (consumer )?request/i,
    loose: /agent|privacy request|consumer request|verifiab/i,
    what: 'the CCPA authorised-agent / verifiable-request machinery',
  },
  request_sent: {
    tier: 'core',
    // The instrument: the paper says a request was SENT. A first attempt at this
    // ("we sent \u2026 an access request", one clause) matched 3 papers and missed
    // every DSAR study in the corpus, including //Access Denied!// and //Leave No
    // Data Behind//. Six alternations, each added after reading what the previous
    // one missed, take it to the number below. A narrow probe here is a recall
    // claim, and the narrow one was wrong.
    tight: new RegExp(
      [
        /\b(sent|submitted|filed|issued|exercised|dispatched)\b[^.]{0,120}\b(subject access request|access request|DSAR|deletion request|erasure request|privacy request|opt-?out request|data request)s?\b/,
        /\b(subject access request|access request|DSAR|deletion request|erasure request|privacy request|opt-?out request)s?\b[^.]{0,120}\b(sent|submitted|filed|issued|exercised|were made|we made)\b/,
        /\bwe\b[^.]{0,150}\brequest(ed)?\b[^.]{0,80}\b(our|the) (personal )?data\b/,
        /exercis(e|ed|ing) (our|their|the|his|her|my) (right|rights|data subject)/,
        /\b(sent|submitted|filed|issued)\b[^.]{0,60}\brequests?\b[^.]{0,80}\b(controller|company|companies|website|websites|vendor|data broker|app developer)s?\b/,
        /\bright to be forgotten\b[^.]{0,120}\brequest/,
      ].map((r) => r.source).join('|'),
      'i',
    ),
    loose: /request|exercis/i,
    what: 'the paper says a request was SENT \u2014 the instrument, not the topic',
  },
  account_deletion: {
    tier: 'core',
    tight: /account deletion|delete (your|my|their|the) account|account (closure|termination)|deactivat(e|ion) (your|my|the) account/i,
    loose: /account|deletion|delete|closure|terminat|deactivat/i,
    what: 'account deletion as the user-facing form of the erasure right',
  },
  broker_optout: {
    tier: 'core',
    tight: /people.?search (site|website|engine|service)|data (broker|removal) (opt.?out|service|request)|PII removal|data removal service|removal request/i,
    loose: /people search|data broker|removal|broker/i,
    what: 'data-broker / people-search opt-out and removal services',
  },
  optout_tool: {
    tier: 'core',
    // RECALL PASS. Added after the first audit: //Do Opt-Outs Really Opt Me
    // Out?// (CCS 2022) is squarely in the population and fired only the
    // context-tier `dnt` probe, because the industry opt-out tools it measures
    // are never called "do not sell". A probe that cannot see the paper the
    // page is partly about is a recall claim that was wrong.
    tight: /opt-?out (tool|mechanism|link|page|choice|setting|button|interface|process|request|form|portal)s?|Network Advertising Initiative|Digital Advertising Alliance|YourAdChoices|\bAdChoices\b|aboutads\.info|optout\.networkadvertising/i,
    loose: /opt-?out|advertising initiative|advertising alliance|adchoices|aboutads/i,
    what: 'an industry or site opt-out tool (NAI, DAA/AdChoices, an opt-out page)',
  },
  self_service: {
    tier: 'core',
    tight: /privacy dashboard|data download|Google Takeout|Download Your Information|ad(vertising)? (preference|settings) (manager|page|centre|center)|Off-?Facebook Activity|My ?Activity|data export|export (your|my|their) data/i,
    loose: /dashboard|takeout|download|preference|activity|export/i,
    what: 'the in-product self-service surface people actually use (dashboard, export)',
  },
  usp_string: {
    tier: 'core',
    tight: /\bUS ?Privacy\b|\busprivacy\b|\bUSP ?String\b|\bGPP\b|Global Privacy Platform/i,
    loose: /usprivacy|us privacy|gpp|global privacy platform/i,
    what: 'the IAB US Privacy / Global Privacy Platform opt-out strings',
  },
};

// Everything below is the CLI. It is skipped when this module is imported
// (dsr_ctxdump.mjs imports PROBES/fullText/key), so importing does not re-stream
// all 5,855 full texts and does not print the summary table into someone else's
// stdout.
const IS_CLI = process.argv[1] && process.argv[1].endsWith('dsr_probe.mjs');
if (IS_CLI) {
// Stream: read each paper once, keep only the hit sets and, for --ctx, one
// snippet per matching paper. Holding 5,859 full texts resident OOMs node.
const hits = {};
for (const n of Object.keys(PROBES)) hits[n] = { tight: new Set(), loose: new Set() };
const ctx = new Map(); // `${probe}|${key}` -> snippet, only for the --ctx probe
const wantCtx = process.argv[2] === '--ctx' ? process.argv[3] : null;
let noText = 0;
let withText = 0;
for (const p of rows) {
  const t = fullText(p);
  if (t === null) { noText++; continue; }
  withText++;
  const k = key(p);
  for (const [name, def] of Object.entries(PROBES)) {
    const m = def.tight.exec(t);
    if (m) {
      hits[name].tight.add(k);
      if (name === wantCtx) {
        const i = t.indexOf(m[0]);
        ctx.set(k, t.slice(Math.max(0, i - 200), i + 260));
      }
    }
    if (def.loose.test(t)) hits[name].loose.add(k);
  }
}
for (const [name, def] of Object.entries(PROBES)) {
  const escaped = [...hits[name].tight].filter((k) => !hits[name].loose.has(k));
  if (escaped.length) {
    throw new Error(
      `probe ${name}: ${escaped.length} paper(s) match the TIGHT pattern but not the LOOSE one, ` +
        `so LOOSE is not a superset and the "narrowing" claim is false. e.g. ${escaped[0]}`,
    );
  }
  if (hits[name].tight.size > hits[name].loose.size) {
    throw new Error(`probe ${name}: tight ${hits[name].tight.size} > loose ${hits[name].loose.size}`);
  }
  void def;
}
const texts = { size: withText };

const arg = process.argv[2];
const which = process.argv[3];

if (arg === '--list' || arg === '--ctx') {
  const def = PROBES[which];
  if (!def) throw new Error(`unknown probe ${which}; one of ${Object.keys(PROBES).join(', ')}`);
  void def;
  const ks = [...hits[which].tight].sort();
  for (const k of ks) {
    const p = byKey.get(k);
    console.log(`${p.year}\t${p.venue}\t${p.slug}\t${p.title}`);
    if (arg === '--ctx') {
      const t = texts.get(k);
      const m = t.match(def.tight);
      const i = t.indexOf(m[0]);
      console.log(`\t…${t.slice(Math.max(0, i - 160), i + 200)}…`);
    }
  }
  console.log(`\n${ks.length} papers`);
  process.exit(0);
}

const unionOf = (names) => {
  const u = new Set();
  for (const n of names) for (const k of hits[n].tight) u.add(k);
  return u;
};
const CORE = Object.keys(PROBES).filter((n) => PROBES[n].tier === 'core');
const CONTEXT = Object.keys(PROBES).filter((n) => PROBES[n].tier === 'context');
const coreUnion = unionOf(CORE);
const union = unionOf(Object.keys(PROBES));

if (arg === '--union') {
  const set = which === 'core' ? coreUnion : which === 'context' ? unionOf(CONTEXT) : union;
  for (const k of [...set].sort()) {
    const p = byKey.get(k);
    const fired = Object.keys(PROBES).filter((n) => hits[n].tight.has(k));
    console.log(`${k}\t${p.year}\t${p.venue}\t${fired.join(',')}\t${p.title}`);
  }
  console.log(`\n${set.size} papers`);
  process.exit(0);
}

console.log(`corpus: ${rows.length} extracted papers; ${texts.size} with full text on disk (${noText} without)\n`);
console.log('probe\ttier\ttight\tloose\twhat');
for (const [n, def] of Object.entries(PROBES)) {
  console.log(`${n}\t${def.tier}\t${hits[n].tight.size}\t${hits[n].loose.size}\t${def.what}`);
}
console.log(`\nUNION, core probes only: ${coreUnion.size} papers (${((100 * coreUnion.size) / texts.size).toFixed(1)}% of ${texts.size} with full text)`);
console.log(`UNION, all probes:       ${union.size} papers (${((100 * union.size) / texts.size).toFixed(1)}%)`);
console.log(`context-only candidates (no core probe fired): ${union.size - coreUnion.size}`);

const yrs = new Map();
for (const k of union) {
  const y = byKey.get(k).year;
  yrs.set(y, (yrs.get(y) ?? 0) + 1);
}
console.log('\nyear\tcandidates\tcorpus papers\tshare');
const corpusYr = new Map();
for (const p of rows) corpusYr.set(p.year, (corpusYr.get(p.year) ?? 0) + 1);
for (const y of [...corpusYr.keys()].sort()) {
  const c = yrs.get(y) ?? 0;
  console.log(`${y}\t${c}\t${corpusYr.get(y)}\t${((100 * c) / corpusYr.get(y)).toFixed(1)}%`);
}

}
</file>

<file javascript dsr_report.mjs>
// Every figure on privacy:data_subject_rights, with its denominator.
//
//   node --max-old-space-size=3000 scripts/dsr_report.mjs
//
// The population is the hand audit in scripts/dsr_verdicts.json (built by
// scripts/dsr_verdicts_build.py from the full-text candidate sweep in
// scripts/dsr_probe.mjs). Nothing here is a title probe: the roadmap row that
// queued this page was one, and its precision and recall are reported below as
// a finding about the row rather than used as a population.
//
// Rules this file follows, each because breaking it has published a wrong
// number on this wiki before:
//   * every table prints its own denominator in the heading;
//   * every subgroup share is printed next to the corpus base rate for the
//     same field, so "DSAR papers do X more" is checkable;
//   * sentinels are counted separately and never as an answer;
//   * a shelled-out denominator is asserted, not assumed.
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { loadExtractions, isSentinel } from './lib.mjs';

const HERE = path.dirname(fileURLToPath(import.meta.url));
const verdicts = JSON.parse(fs.readFileSync(path.join(HERE, 'dsr_verdicts.json'), 'utf8'));
const rows = loadExtractions();
const key = (p) => `${p.venue}/${p.year}/${p.slug}`;

const CAND = new Set(Object.keys(verdicts));
const IN = new Set(Object.entries(verdicts).filter(([, v]) => v.verdict === 'IN').map(([k]) => k));
const CAT = new Map(
  Object.entries(verdicts).filter(([, v]) => v.verdict === 'IN').map(([k, v]) => [k, v.category]),
);

const pop = rows.filter((p) => IN.has(key(p)));
if (pop.length !== IN.size) {
  throw new Error(
    `${IN.size} papers are marked IN but only ${pop.length} of them are in extractions.jsonl. ` +
      `A verdict key that matches no paper is a typo, not a finding.`,
  );
}
const corpus = rows.length;
const pct = (a, b) => `${((100 * a) / b).toFixed(1)}%`;
const rule = (s) => console.log(`\n${s}\n${'-'.repeat(s.length)}`);

console.log(`corpus: ${corpus} extracted papers (7 venues, 2010-2026)`);
console.log(`candidates from the full-text sweep: ${CAND.size}`);
console.log(`population after hand audit: ${pop.length} papers`);
console.log(`  precision of the candidate sweep: ${pct(IN.size, CAND.size)}`);
console.log(`  share of the corpus:              ${pct(pop.length, corpus)}`);

rule('1. The population by category of instrument (N = ' + pop.length + ')');
const CATNAME = {
  R: 'R  sent requests to real controllers',
  S: 'S  measured an opt-out signal or setting',
  M: 'M  measured a rights mechanism across a population',
  D: 'D  research dataset obtained via access / export',
  O: 'O  observational study of requests others sent',
};
const byCat = new Map();
for (const p of pop) {
  const c = CAT.get(key(p));
  byCat.set(c, (byCat.get(c) ?? 0) + 1);
}
for (const c of ['R', 'S', 'M', 'D', 'O']) {
  console.log(`${CATNAME[c]}\t${byCat.get(c) ?? 0}\t${pct(byCat.get(c) ?? 0, pop.length)}`);
}
// Sum over the five KNOWN categories, not over whatever keys happen to be in
// the map. Summing the map's own values is vacuous: an invalid category still
// lands in byCat, the total still equals the population, and the printed table
// -- which only prints R/S/M/D/O -- is silently one paper short. Caught by a
// reviewer's mutation test on 2026-09-16.
const CATS = ['R', 'S', 'M', 'D', 'O'];
const bad = [...new Set(pop.map((p) => CAT.get(key(p))))].filter((c) => !CATS.includes(c));
if (bad.length) {
  throw new Error(`population papers carry categories outside R/S/M/D/O: ${bad.join(', ')}`);
}
const catTotal = CATS.reduce((a, c) => a + (byCat.get(c) ?? 0), 0);
if (catTotal !== pop.length) {
  throw new Error(
    `the five instrument categories sum to ${catTotal}, but the population is ${pop.length}`,
  );
}

rule('2. Why the candidate sweep is mostly noise (N = ' + (CAND.size - IN.size) + ' rejected)');
const reasons = new Map();
for (const v of Object.values(verdicts)) {
  if (v.verdict === 'OUT') reasons.set(v.reason, (reasons.get(v.reason) ?? 0) + 1);
}
for (const [r, n] of [...reasons].sort((a, b) => b[1] - a[1])) {
  console.log(`${r}\t${n}\t${pct(n, CAND.size - IN.size)}`);
}

rule('3. Per year. 2025-2026 are PROVISIONAL venue-years (see literature:corpus)');
const yrCorpus = new Map();
const yrPop = new Map();
const yrCand = new Map();
for (const p of rows) {
  yrCorpus.set(p.year, (yrCorpus.get(p.year) ?? 0) + 1);
  if (IN.has(key(p))) yrPop.set(p.year, (yrPop.get(p.year) ?? 0) + 1);
  if (CAND.has(key(p))) yrCand.set(p.year, (yrCand.get(p.year) ?? 0) + 1);
}
console.log('year\tpopulation\tcandidates\tcorpus\tshare of corpus');
for (const y of [...yrCorpus.keys()].sort()) {
  console.log(
    `${y}\t${yrPop.get(y) ?? 0}\t${yrCand.get(y) ?? 0}\t${yrCorpus.get(y)}\t${pct(yrPop.get(y) ?? 0, yrCorpus.get(y))}`,
  );
}
const before = [...yrPop].filter(([y]) => y <= 2020).reduce((a, [, n]) => a + n, 0);
const after = [...yrPop].filter(([y]) => y >= 2021).reduce((a, [, n]) => a + n, 0);
console.log(`\n2010-2020: ${before} papers    2021-2026: ${after} papers`);

rule('4. Per venue, against each venue’s own size');
const vCorpus = new Map();
const vPop = new Map();
for (const p of rows) {
  vCorpus.set(p.venue, (vCorpus.get(p.venue) ?? 0) + 1);
  if (IN.has(key(p))) vPop.set(p.venue, (vPop.get(p.venue) ?? 0) + 1);
}
console.log('venue\tpopulation\tvenue papers\tshare of that venue');
for (const [v, n] of [...vPop].sort((a, b) => b[1] - a[1])) {
  console.log(`${v}\t${n}\t${vCorpus.get(v)}\t${pct(n, vCorpus.get(v))}`);
}
for (const v of [...vCorpus.keys()].filter((v) => !vPop.has(v))) {
  console.log(`${v}\t0\t${vCorpus.get(v)}\t0.0%`);
}

rule('5. What the roadmap’s title-and-summary probe was worth');
// scripts/gap_probe_roadmap.mjs, family dsar_rights, re-run here verbatim.
const TITLE_RE =
  /data subject access|\bDSAR\b|right to (be forgotten|erasure|access|delete)|deletion request|opt-?out request|\bGPC\b|Global Privacy Control|do not sell/i;
const titleHits = rows.filter((p) => TITLE_RE.test(`${p.title ?? ''} • ${p.summary ?? ''}`));
const titleIn = titleHits.filter((p) => IN.has(key(p)));
console.log(`title+summary candidates: ${titleHits.length}`);
console.log(`  of those, in the population: ${titleIn.length}  (precision ${pct(titleIn.length, titleHits.length)})`);
console.log(`  of the population, caught:   ${titleIn.length}/${pop.length}  (recall ${pct(titleIn.length, pop.length)})`);
console.log('\nthe title probe MISSES these population papers:');
for (const p of pop.filter((q) => !TITLE_RE.test(`${q.title ?? ''} • ${q.summary ?? ''}`)).sort((a, b) => a.year - b.year)) {
  console.log(`  ${p.year} ${p.venue}\t[${CAT.get(key(p))}]\t${p.title}`);
}
console.log('\nthe title probe returns these NON-population papers:');
for (const p of titleHits.filter((q) => !IN.has(key(q)))) {
  console.log(`  ${p.year} ${p.venue}\t${verdicts[key(p)] ? verdicts[key(p)].reason : 'not even a full-text candidate'}\t${p.title}`);
}

rule('6. How big is the sample when a human sends the requests?');
// HAND-KEYED, on purpose. `population[].n` is the largest number the paper
// drew, not the number of controllers it wrote to: CanaryTrap's biggest n is a
// 43,332-website list it never contacted, and its request sample is 87 apps.
// Deriving this from the extraction produced exactly that error, so the count
// is keyed here from each paper's own sentence, and the script throws if an R
// paper is missing from the map.
const ASKED = {
  'PETS/2020/canarytrap-detecting-data-misuse-by-third-party-apps-on-online-social-networks':
    [100, 'third-party Facebook apps sent a data-deletion request: 87 by email, 13 through a contact form. "Only 52% of the apps (45 out of 87) responded back to our requests" counts only the emailed 87'],
  'PETS/2021/data-portability-between-online-services-an-empirical-analysis-on-the-effectiven':
    [182, 'online services sent an Art. 20 export request ("135 of 182 services (74.2%)")'],
  'PETS/2022/leave-no-data-behind-empirical-insights-into-data-erasure-from-online-services':
    [90, 'online services ("we investigated the data deletion practices of a sample of 90 online services")'],
  'PETS/2022/revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an':
    [40, 'organisations re-evaluated in 2021, by registered letter, under 5 assumed subject identities'],
  'IEEE-SP/2022/scraping-sticky-leftovers-app-user-information-left-on-servers-after-account-del':
    [678, 'apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand'],
  'PETS/2023/lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c':
    [109, 'of 160 selected Android apps -- the subset whose privacy policies carried CCPA disclosures -- sent a verifiable consumer request: "We then submitted VCRs to these 109 companies". 160 is the number selected, not the number asked'],
  'USENIX/2024/data-subjects-reactions-to-exercising-their-right-of-access':
    [null, '33 participants requested their own exports; 801 files. Controllers are not enumerated'],
  'PETS/2024/what-to-expect-when-you-re-accessing-an-exploration-of-user-privacy-rights-in-pe':
    [20, 'people-search websites, access and removal attempted by four researchers'],
  'CCS/2025/hidden-in-plain-bytes-investigating-interpersonal-account-compromise-with-data-e':
    [6, 'services (Apple iCloud, Google, Facebook, Instagram, Snapchat, Discord), 12 researcher accounts'],
  'IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba':
    [1, 'Amazon, asked for the puppet accounts\u2019 advertising data as a cross-check on the audit'],
  'PETS/2025/echoes-of-privacy-uncovering-the-profiling-practices-of-voice-assistants':
    [2, 'Amazon and Apple, DSARs used to discover the interest labels to target; 200 fresh accounts'],
  'PETS/2025/measuring-the-accuracy-and-effectiveness-of-pii-removal-services':
    [4, 'paid PII-removal services subscribed to by 71 participants, which then send the opt-outs on their behalf. The paper surveys 10 services, which "cover a total of 1,759 unique data brokers"; its abstract says 2,024, and the body sentence is the one used'],
  'USENIX/2025/addressing-the-address-books-interdependent-privacy-issues':
    [5, 'address-book service providers sent a data-access request ("No provider supplied the requested contact-card data")'],
  'USENIX/2025/no-way-to-sign-out-unpacking-non-compliance-with-google-plays-app-account-deleti':
    [494, 'apps whose deletion method was classified, out of 863 with deletion-link data'],
  'IEEE-SP/2026/consumer-beware-exploring-data-brokers-ccpa-compliance':
    [454, 'of the 543 registered California data brokers (84%; the rest were excluded), each sent a verifiable consumer request by hand: "In total, 9 hours and 57 minutes were spent submitting VCRs to all 454 DBRs"'],
  'IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right':
    [3, 'platforms (Instagram, TikTok, YouTube), via sock-puppet accounts and 80 recruited participants'],
  'WWW/2026/does-ad-free-mean-less-data-collection-an-empirical-study-of-platform-data-pract':
    [6, 'author-owned platform accounts; Art. 15(3) requests before and after taking an ad-free subscription'],
};
const rPapers = pop.filter((p) => CAT.get(key(p)) === 'R').sort((a, b) => a.year - b.year);
const gaps = rPapers.filter((p) => !(key(p) in ASKED));
if (gaps.length) {
  throw new Error(
    `${gaps.length} R paper(s) have no hand-keyed "controllers asked": ${gaps.map(key).join(', ')}. ` +
      `Read the paper and add the row; do not let the table silently omit it.`,
  );
}
const stale = Object.keys(ASKED).filter((k) => !IN.has(k) || CAT.get(k) !== 'R');
if (stale.length) throw new Error(`ASKED has rows that are no longer R papers: ${stale.join(', ')}`);
console.log(`${rPapers.length} papers sent requests. Controllers asked, hand-keyed from each paper:\n`);
console.log('year\tvenue\tasked\twhat');
const ns = [];
for (const p of rPapers) {
  const [n, what] = ASKED[key(p)];
  if (n !== null) ns.push(n);
  console.log(`${p.year}\t${p.venue}\t${n === null ? '-' : n}\t${what}`);
}
ns.sort((a, b) => a - b);
const median = ns.length % 2 ? ns[(ns.length - 1) / 2] : (ns[ns.length / 2 - 1] + ns[ns.length / 2]) / 2;
console.log(`\n  a number is stated for ${ns.length} of ${rPapers.length}`);
console.log(`  median ${median}   min ${ns[0]}   max ${ns[ns.length - 1]}`);
console.log(`  at or below 100: ${ns.filter((n) => n <= 100).length} of ${ns.length}`);
// The comparable crawl figure, same schema field, so the contrast has a base rate.
const crawled = rows.filter(
  (p) => p.crawlConfig !== null || p.studyTypes.includes('automated-web-crawl'),
);
const crawlNs = [];
for (const p of crawled) {
  const c = p.population.filter(
    (x) => typeof x.n === 'number' && (x.unit === 'websites' || x.unit === 'domains'),
  );
  if (c.length) crawlNs.push(Math.max(...c.map((x) => x.n)));
}
crawlNs.sort((a, b) => a - b);
console.log(
  `\n  BASE RATE: of the ${crawled.length} corpus papers that ran a crawl, ${crawlNs.length} state a ` +
    `site-or-domain population[].n; their median is ${crawlNs[Math.floor(crawlNs.length / 2)]}.`,
);

rule('7. Is it an annotation problem? (share with humanAnnotation, vs the corpus)');
const withAnn = (set) => set.filter((p) => p.humanAnnotation.length > 0).length;
console.log(`population       : ${withAnn(pop)}/${pop.length}\t${pct(withAnn(pop), pop.length)}`);
console.log(`corpus base rate : ${withAnn(rows)}/${corpus}\t${pct(withAnn(rows), corpus)}`);
const annStated = pop.filter((p) =>
  p.humanAnnotation.some((a) => !isSentinel(a.agreementMetric)),
).length;
const annAll = pop.filter((p) => p.humanAnnotation.length > 0).length;
const corpusAnn = rows.filter((p) => p.humanAnnotation.length > 0);
const corpusAnnStated = corpusAnn.filter((p) =>
  p.humanAnnotation.some((a) => !isSentinel(a.agreementMetric)),
).length;
console.log(
  `\nof those that annotated, share stating an agreement metric:\n` +
    `population       : ${annStated}/${annAll}\t${pct(annStated, annAll)}\n` +
    `corpus base rate : ${corpusAnnStated}/${corpusAnn.length}\t${pct(corpusAnnStated, corpusAnn.length)}`,
);

rule('8. Ethics review, against the corpus base rate for the same field');
const EMP = rows.filter((p) => p.isEmpirical === true);
const revStated = (set) => set.filter((p) => p.ethics && !isSentinel(p.ethics.reviewOutcome)).length;
const popEmp = pop.filter((p) => p.isEmpirical === true);
console.log(`population, empirical : ${revStated(popEmp)}/${popEmp.length}\t${pct(revStated(popEmp), popEmp.length)} state a review outcome`);
console.log(`corpus base rate      : ${revStated(EMP)}/${EMP.length}\t${pct(revStated(EMP), EMP.length)}`);
const outcomes = new Map();
for (const p of popEmp) {
  const v = p.ethics.reviewOutcome;
  outcomes.set(v, (outcomes.get(v) ?? 0) + 1);
}
console.log('\nreviewOutcome values in the population (sentinels shown, never counted as answers):');
for (const [v, n] of [...outcomes].sort((a, b) => b[1] - a[1])) {
  console.log(`  ${v}\t${n}${isSentinel(v) ? '\t(sentinel)' : ''}`);
}

rule('9. Which laws the population names (legal[], paper-counted)');
const laws = new Map();
let withLegal = 0;
for (const p of pop) {
  if (!p.legal.length) continue;
  withLegal++;
  const seen = new Set();
  for (const l of p.legal) {
    const n = /ccpa|california consumer/i.test(l.law)
      ? 'CCPA'
      : /cpra|california privacy rights/i.test(l.law)
        ? 'CPRA'
        : /gdpr|general data protection/i.test(l.law)
          ? 'GDPR'
          : /eprivacy/i.test(l.law)
            ? 'ePrivacy Directive'
            : l.law;
    if (seen.has(n)) continue;
    seen.add(n);
    laws.set(n, (laws.get(n) ?? 0) + 1);
  }
}
console.log(`${withLegal} of ${pop.length} papers assess a law. Folded law names, counted by paper:`);
for (const [l, n] of [...laws].sort((a, b) => b[1] - a[1])) console.log(`  ${l}\t${n}`);
const corpusLegal = rows.filter((p) => p.legal.length > 0).length;
console.log(`\nBASE RATE: ${corpusLegal} of ${corpus} corpus papers assess any law (${pct(corpusLegal, corpus)});`);
console.log(`the population's share is ${pct(withLegal, pop.length)}.`);

rule('10. foundViolations, over the tuples in the population that assessed one');
const fv = new Map();
for (const p of pop) for (const l of p.legal) fv.set(l.foundViolations, (fv.get(l.foundViolations) ?? 0) + 1);
const fvTot = [...fv.values()].reduce((a, b) => a + b, 0);
console.log(`${fvTot} legal tuples across ${withLegal} papers (TUPLES, not papers -- a paper can assess several laws):`);
for (const [v, n] of [...fv].sort((a, b) => b[1] - a[1])) console.log(`  ${v}\t${n}\t${pct(n, fvTot)}`);

rule('11. The page’s own population table, checked against the verdicts both ways');
// "Assert the table against the data, and the data against the table." Only the
// first direction catches a row the page invented; only the second catches a
// paper the page silently dropped. Both have shipped on this wiki before.
const PAGE = path.join(HERE, '..', 'dsr', 'page.txt');
if (fs.existsSync(PAGE)) {
  const src = fs.readFileSync(PAGE, 'utf8');
  const keyed = JSON.parse(
    fs.readFileSync(path.join(HERE, '..', 'dsr', 'citekeys.json'), 'utf8'),
  ); // slug -> citekey
  const byCitekey = new Map(Object.entries(keyed).map(([k, c]) => [c, k]));
  const rowsOnPage = [...src.matchAll(/^\| (\d{4}) \| ([\w &]+) \| ([RSMDO]) \| \{\[([^\]]+)\]\}/gm)].map(
    (m) => ({ year: +m[1], cat: m[3], citekey: m[4] }),
  );
  const seen = new Set();
  for (const r of rowsOnPage) {
    const slug = byCitekey.get(r.citekey);
    if (!slug) throw new Error(`page table row cites ${r.citekey}, which is in no population paper`);
    if (!IN.has(slug)) throw new Error(`page table row ${r.citekey} is not in the population`);
    if (CAT.get(slug) !== r.cat) {
      throw new Error(
        `page table says ${r.citekey} is category ${r.cat}; the verdicts say ${CAT.get(slug)}`,
      );
    }
    if (seen.has(slug)) throw new Error(`page table lists ${r.citekey} twice`);
    seen.add(slug);
  }
  const dropped = [...IN].filter((k) => !seen.has(k));
  if (dropped.length) {
    throw new Error(
      `${dropped.length} population paper(s) are in the verdicts but NOT on the page's table: ` +
        dropped.join(', '),
    );
  }
  console.log(`page table: ${rowsOnPage.length} rows, all match the verdicts, none missing`);
  // The counts sentence has to agree with the table it sits under.
  const stated = src.match(/By instrument: \*\*R (\d+)\*\*, \*\*S (\d+)\*\*, \*\*M (\d+)\*\*, \*\*D (\d+)\*\*, \*\*O (\d+)\*\*/);
  if (!stated) throw new Error("the page's 'By instrument' sentence did not parse");
  const want = { R: 17, S: 0, M: 0, D: 0, O: 0 };
  for (const c of Object.keys(want)) want[c] = pop.filter((p) => CAT.get(key(p)) === c).length;
  const got = { R: +stated[1], S: +stated[2], M: +stated[3], D: +stated[4], O: +stated[5] };
  for (const c of ['R', 'S', 'M', 'D', 'O']) {
    if (want[c] !== got[c]) throw new Error(`page says ${c} ${got[c]}; the verdicts say ${want[c]}`);
  }
  console.log(`page 'By instrument' sentence: ${JSON.stringify(got)} — agrees`);
  // And the "N of the M S papers are 2024 or later" claim.
  const sLate = pop.filter((p) => CAT.get(key(p)) === 'S' && p.year >= 2024).length;
  console.log(`S papers from 2024 onward: ${sLate} of ${want.S}`);
} else {
  console.log(`(${PAGE} not present; page-vs-verdicts check skipped)`);
}
</file>

<file python dsr_verdicts_build.py>
# Build scripts/dsr_verdicts.json: the hand audit of the 289 full-text candidates
# for privacy:data_subject_rights. Every candidate must appear exactly once, and
# the script throws rather than defaulting a verdict -- a canned default reason
# would read as per-paper evidence when it is not.
#
#   python3 scripts/dsr_verdicts_build.py
#
# IN categories (see the page's inclusion rule):
#   R  the authors or their participants SENT requests to real controllers
#   S  measured a machine-readable opt-out SIGNAL's deployment or honouring
#   M  measured a rights-exercise MECHANISM across a population
#   D  the research DATASET was obtained through the right of access
#   O  OBSERVATIONAL study of requests other people sent
#
# OUT reasons:
#   homonym         the probe matched an unrelated sense (HTTP request, NVIDIA
#                   "GPC", "do not sell" of goods, DNS/cache/stream deletion)
#   background      names a right in motivation, related work or references; the
#                   paper's object is something else
#   policy-text     measures what a policy or store declaration SAYS about rights
#                   -> privacy:policies
#   unlearning      right to be forgotten as a machine-learning problem
#   takedown        content removal, delisting, blocklist or abuse removal
#   consent         consent banner or consent withdrawal -> privacy:consent
#   system          proposes or builds a rights mechanism, measures no deployment
#   attitudes       survey/interview on awareness or perception, with no request
#                   exercised and no mechanism observed
#   research-ethics the paper RECEIVED or honoured requests about its own study
#                   -> practices:ethics
#   developer-side  studies the controller's implementation side, not the
#                   subject's request
#   enforcement     regulator decisions and fines -> practices:legal_enforcement
#   sok             systematization; read it, but it measures nothing itself
#   agent-benchmark the object measured is an LLM agent's capability, and a
#                   rights task is one item in a wider benchmark
import json, os
from collections import Counter

HERE = os.path.dirname(os.path.abspath(__file__))
DSR = os.path.join(HERE, '..', 'dsr')

OUT_BY_IDX = {
 'homonym': [1,2,3,5,6,16,17,18,20,21,24,25,26,27,30,32,33,37,40,42,49,50,51,53,54,55,56,57,59,60,61,63,65,72,74,82,83,84,86,88,90,94,97,99,111,122,124,130,131,139,158,159,168,171,172,173,174,175,176,180,182,190,194,195,201,202,203,204,206,211,212,216,221,223,224,225,229,230,231,234,237,238,240,241,243,245,247],
 'background': [8,11,15,22,28,29,35,38,41,43,45,52,67,68,69,70,71,73,77,78,91,92,96,98,101,108,114,115,116,119,123,125,129,134,135,137,140,143,150,154,160,162,164,166,205,213,218,219,228,232],
 'policy-text': [7,14,31,36,66,85,95,102,103,104,117,120,128,151,155,163,170,177,178,188,193,200,214,222,236,239,249],
 'unlearning': [9,19,81,87,109,152,227,242,246,248],
 'takedown': [58,62,186,210,220,233,235],
 'consent': [4,10,64,148,184,197,208,217,244],
 'system': [12,75,76,79,100,142,149,179,192],
 'attitudes': [47,93,106,107,113,118,126,127,132,133,136,138,144,145,146,147,153,156,161,181,183,187,189,199,207,209],
 'research-ethics': [13,34,44,89,157,185,191,196,198,250],
 'developer-side': [46,48,110,112,141,165,167,215],
 'enforcement': [121,226],
 'sok': [23,39],
 'agent-benchmark': [169],
}

NEW_BY_IDX = {
 'homonym': [1,2,3,4,6,18,20,25,26,29,32,44,49,53,54,58,59,61,76,78,79,80,82,84,92,94,110,112,116,118,131,133,142,146,148,149,156,163,169,172,174,176],
 'research-ethics': [9,13,14,19,22,23,31,41,47,55,60,62,63,64,71,81,83,91,93,97,98,104,105,114,115,119,121,122,124,125,128,130,135,140,141,147,151,153,158,161,164,167,179],
 'background': [5,7,11,12,15,17,21,24,27,28,30,33,34,37,38,40,42,48,51,52,56,65,66,67,85,88,90,95,100,101,102,103,107,108,117,123,126,127,132,139,144,145,150,152,154,155,159,160,162,166,168,170,171,173,175,177,181,182],
 'attitudes': [16,39,50,69,70,74,86,87,89,96,106,109,113,129,134,137,138,157],
 'policy-text': [35,36,68,75,99,136,180],
 'consent': [45,46],
 'system': [5.5, 72],
 'sok': [120],
}
NEW_BY_IDX['system'] = [72]

rows = [l.rstrip('\n').split('\t') for l in open(os.path.join(DSR, 'out_list.txt'))]
idx2key = {i + 1: r[0] for i, r in enumerate(rows)}
# Second wave: the 182 candidates the two recall probes (optout_tool, self_service)
# added after the first audit. Numbered in the order dsr/ctx_new.txt prints them.
new_ordered = [l.strip() for l in open(os.path.join(DSR, 'new_ordered.txt')) if l.strip()]
OFF = 1000
for i, k in enumerate(new_ordered):
    idx2key[OFF + i + 1] = k
for reason, idxs in NEW_BY_IDX.items():
    OUT_BY_IDX.setdefault(reason, [])
    OUT_BY_IDX[reason].extend(OFF + i for i in idxs)

ins = {}
for l in open(os.path.join(DSR, 'in_keys.txt')):
    if not l.strip():
        continue
    k, c = l.rstrip('\n').split('\t')
    assert c in 'RSMDO' and len(c) == 1, (k, c)
    ins[k] = c

seen = {}
for reason, idxs in OUT_BY_IDX.items():
    for i in idxs:
        assert i in idx2key, i
        assert i not in seen, f'index {i} assigned twice: {seen[i]} and {reason}'
        seen[i] = reason

unassigned = [(i, k) for i, k in idx2key.items() if i not in seen and k not in ins]
assert not unassigned, f'unassigned candidates: {unassigned}'
crossed = [k for i, k in idx2key.items() if i in seen and k in ins]
assert not crossed, f'both IN and OUT: {crossed}'

verdicts = {k: {'verdict': 'IN', 'category': c} for k, c in ins.items()}
for i, k in idx2key.items():
    if k in verdicts:
        continue
    verdicts[k] = {'verdict': 'OUT', 'reason': seen[i]}

cands = set(open(os.path.join(DSR, 'keys.txt')).read().split())
assert set(verdicts) == cands, (len(verdicts), len(cands), list(set(verdicts) ^ cands)[:5])

json.dump(verdicts, open(os.path.join(HERE, 'dsr_verdicts.json'), 'w'), indent=0, sort_keys=True)
n_in = sum(1 for v in verdicts.values() if v['verdict'] == 'IN')
print(f'{len(verdicts)} candidates, {n_in} IN ({100 * n_in / len(verdicts):.1f}% precision), {len(verdicts) - n_in} OUT')
print('IN by category :', dict(sorted(Counter(v['category'] for v in verdicts.values() if v['verdict'] == 'IN').items())))
print('OUT by reason  :', dict(Counter(v['reason'] for v in verdicts.values() if v['verdict'] == 'OUT').most_common()))
</file>

<file python dsr_quotecheck.py>
#!/usr/bin/env python3
"""Verify every quoted span on privacy:data_subject_rights against its source.

Extracts each //"..."// span from the page source and checks it appears verbatim
in the cited paper's full text. Three normalisations, each because a PDF
extraction defect otherwise produces a false FAIL:
  * whitespace collapsed (a line break inside a phrase),
  * soft hyphen at a line break removed,
  * curly quotes and dashes folded to ASCII.
Both paper.cols.txt and paper.norm.txt are searched: the two-column repair and
the raw extraction fail on different sentences.

    python3 scripts/dsr_quotecheck.py dsr/page.txt

Exits non-zero if any quote is not found, or if the page contains a quoted span
this script was not given a source for -- a curated quote list drifts from its
page otherwise.
"""
import json
import os
import re
import sys
import unicodedata

ROOT = '/workspace/publications_dataset/data/fulltext'
HERE = os.path.dirname(os.path.abspath(__file__))

# citekey -> corpus slug, for the quotes that come from corpus papers.
SRC = json.load(open(os.path.join(HERE, 'dsr_quote_sources.json')))


def fold(s):
    s = unicodedata.normalize('NFKD', s)
    s = s.replace('', '')
    for a, b in [('‘', "'"), ('’', "'"), ('“', '"'), ('”', '"'),
                 ('–', '-'), ('—', '-'), ('−', '-'), (' ', ' ')]:
        s = s.replace(a, b)
    s = re.sub(r'-\s*\n\s*', '', s)
    # Markdown link syntax: the IAB snapshot is a README, so a sentence that
    # reads "transition to the Global Privacy Protocol" on the page is
    # "transition to the [Global Privacy Protocol](https://...)" in the source.
    s = re.sub(r'\[([^\]]+)\]\((?:[^()]|\([^()]*\))*\)', r'\1', s)
    s = re.sub(r'\s+', ' ', s)
    return s.strip()


_pdf_cache = {}


def text(slug, which):
    # 'pdf' re-extracts with pypdf. The two-column repair in paper.cols.txt
    # splices the other column into the middle of a sentence on roughly a
    # quarter of pages, so a quote that is genuinely in the paper can be absent
    # from every .txt rendering. Checking only .cols publishes a false FAIL.
    if which == 'pdf':
        if slug in _pdf_cache:
            return _pdf_cache[slug]
        f = os.path.join(ROOT, slug, 'paper.pdf')
        if not os.path.exists(f):
            return None
        try:
            from pypdf import PdfReader
        except ImportError:
            from PyPDF2 import PdfReader
        t = fold(' '.join(pg.extract_text() or '' for pg in PdfReader(f).pages))
        _pdf_cache[slug] = t
        return t
    f = os.path.join(ROOT, slug, f'paper.{which}.txt')
    if not os.path.exists(f):
        return None
    return fold(open(f, encoding='utf8', errors='replace').read())


def main():
    page = open(sys.argv[1], encoding='utf8').read()
    spans = re.findall(r'//"(.+?)"//', page, re.S)
    if not spans:
        print('no //"..."// spans found -- did the page markup change?')
        return 1
    unlisted = [q for q in spans if fold(q) not in {fold(k) for k in SRC}]
    fails = []
    for q in spans:
        key = next((k for k in SRC if fold(k) == fold(q)), None)
        if key is None:
            continue
        src = SRC[key]
        if src.startswith('EXTERNAL:'):
            # Checked against the byte snapshot taken at fetch time, not
            # re-fetched: a live re-fetch turns a rate-limit stub into
            # "quote not found" and a removed sentence into a silent pass.
            snap = os.path.join(HERE, '..', 'dsr', 'external',
                                re.sub(r'[^A-Za-z0-9]+', '_', src.split(':', 1)[1].strip())[:90] + '.txt')
            if not os.path.exists(snap):
                fails.append((q, src))
                print(f'FAIL (no snapshot)  {q[:70]}...  ({snap})')
                continue
            body = fold(open(snap, encoding='utf8', errors='replace').read())
            if fold(q) in body:
                print(f'OK  [snapshot]  {q[:70]}...')
            else:
                fails.append((q, src))
                print(f'FAIL (snapshot)  {q[:70]}...  ({src})')
            continue
        hit = None
        for which in ('cols', 'norm', 'pdf'):
            t = text(src, which)
            if t and fold(q) in t:
                hit = which
                break
        if hit:
            print(f'OK  [{hit}]  {q[:70]}...')
        else:
            fails.append((q, src))
            print(f'FAIL        {q[:70]}...  ({src})')
    if unlisted:
        print(f'\n{len(unlisted)} quoted span(s) on the page have no source listed:')
        for q in unlisted:
            print('  ' + q[:100])
    print(f'\n{len(spans)} quoted spans, {len(fails)} failed, {len(unlisted)} unlisted')
    return 1 if (fails or unlisted) else 0


if __name__ == '__main__':
    sys.exit(main())
</file>

====== References ======

<bibtex bibliography></bibtex>