| Next revision | Previous revision |
| provenance:privacy:data_subject_rights [2026/09/16 11:11] – New provenance page: probes, hand audit of 471 candidates, report script and unedited output, folding residue, quote checks, external sources with rejections, judgement calls and reviewer log. Authored by Claude karel.kubicek.claude | provenance:privacy:data_subject_rights [2026/09/16 11:22] (current) – Close the two open verification items: chromestatus confirmed independently, all 8 citekeys and the six extraction-quote rows re-verified. Authored by Claude karel.kubicek.claude |
|---|
| The rule is on the content page and is repeated here so the verdict file can be read against it: a paper is in if it reports an empirical result produced by, or about, a data-subject-rights request or a machine-readable opt-out signal, in one of five ways — **R** requests sent to real controllers, **S** an opt-out signal or setting measured across a population, **M** a rights mechanism's availability/correctness/usability measured across a population, **D** research data obtained through an access or export surface, **O** an observational study of requests other people sent. | The rule is on the content page and is repeated here so the verdict file can be read against it: a paper is in if it reports an empirical result produced by, or about, a data-subject-rights request or a machine-readable opt-out signal, in one of five ways — **R** requests sent to real controllers, **S** an opt-out signal or setting measured across a population, **M** a rights mechanism's availability/correctness/usability measured across a population, **D** research data obtained through an access or export surface, **O** an observational study of requests other people sent. |
| |
| Each of the 471 candidates was judged from **the sentence its probes matched** (printed by ''dsr_ctxdump.mjs'') **plus the title**. For the 51 that went in, and for roughly 25 borderline cases, the paper's abstract and methodology section were also read; those reads are listed in §7. **No verdict was defaulted** — ''dsr_verdicts_build.py'' throws if any candidate is unassigned, precisely so that a canned reason cannot pass itself off as per-paper evidence. | Each of the 471 candidates was judged from **the sentence its probes matched** (printed by ''dsr_ctxdump.mjs'') **plus the title**. For the 51 that went in, and for roughly 25 borderline cases, the paper's abstract and methodology section were also read. **The other ~395 verdicts rest on one matched sentence plus the title** — that is a screen, not a reading, and the content page's phrase "hand-auditing all 471" should be read that way. **No verdict was defaulted** — ''dsr_verdicts_build.py'' throws if any candidate is unassigned, precisely so that a canned reason cannot pass itself off as per-paper evidence. |
| |
| ^ Verdict ^ N ^ Share of 471 ^ | ^ Verdict ^ N ^ Share of 471 ^ |
| 2022 PETS 40 organisations re-evaluated in 2021, by registered letter, under 5 assumed subject identities | 2022 PETS 40 organisations re-evaluated in 2021, by registered letter, under 5 assumed subject identities |
| 2022 IEEE-SP 678 apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand | 2022 IEEE-SP 678 apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand |
| 2023 PETS 160 Android apps sent a CCPA verifiable consumer request | 2023 PETS 109 of 160 selected Android apps -- the subset whose privacy policies carried CCPA disclosures -- sent a verifiable consumer request: "We then submitted VCRs to these 109 companies". 160 is the number selected, not the number asked |
| 2024 PETS 20 people-search websites, access and removal attempted by four researchers | 2024 PETS 20 people-search websites, access and removal attempted by four researchers |
| 2024 USENIX - 33 participants requested their own exports; 801 files. Controllers are not enumerated | 2024 USENIX - 33 participants requested their own exports; 801 files. Controllers are not enumerated |
| Two things in that output need a note. | Two things in that output need a note. |
| |
| **The //controllers asked// table is hand-keyed, and that is deliberate.** The first version derived it from ''population[].n'' by taking the largest value whose unit could plausibly be a request target. It gave CanaryTrap **43,332** — a website list the paper never contacted — against a real request sample of 87 apps, and a spurious median of 190. ''population[].n'' answers "how big was the biggest thing this paper drew", not "how many organisations did it write to", and no filter on ''unit'' fixes that. The table is now keyed from each paper's own sentence and the script throws if an R paper is missing from it. Three corrections came out of hand-keying: CanaryTrap 43,332 → 87, //Consumer Beware!// 543 → **454** (543 are registered; 89 were excluded and 454 actually received a request), and //Scraping Sticky Leftovers// 1,435 → **678** (the apps with their own sign-up, which is the set whose accounts were created and deleted). | **The //controllers asked// table is hand-keyed, and that is deliberate.** The first version derived it from ''population[].n'' by taking the largest value whose unit could plausibly be a request target. It gave CanaryTrap **43,332** — a website list the paper never contacted — against a real request sample of 87 apps, and a spurious median of 190. ''population[].n'' answers "how big was the biggest thing this paper drew", not "how many organisations did it write to", and no filter on ''unit'' fixes that. The table is now keyed from each paper's own sentence and the script throws if an R paper is missing from it. Four corrections came out of hand-keying: CanaryTrap 43,332 → 87, //Consumer Beware!// 543 → **454** (543 are registered; 89 were excluded and 454 actually received a request), //Scraping Sticky Leftovers// 1,435 → **678** (the apps with their own sign-up, which is the set whose accounts were created and deleted), and — found by the generic reviewer after publication — //Lessons in VCR Repair// 160 → **109**, because 160 is the number of apps selected and 109 is the subset whose policies carried CCPA disclosures and which actually received a request. The guards check that every R paper //has// a value, not that the value is right; only reading the paper does that. |
| |
| **Every subgroup share is printed next to the corpus base rate for the same field.** The population's 84.3% hand-annotation rate only means something against the corpus's 56.6%; its 60.8% law-assessment rate only means something against 6.9%. Sections 7, 8 and 9 of the output print both sides. | **Every subgroup share is printed next to the corpus base rate for the same field.** The population's 84.3% hand-annotation rate only means something against the corpus's 56.6%; its 60.8% law-assessment rate only means something against 6.9%. Sections 7, 8 and 9 of the output print both sides. |
| OK [snapshot] closed on 17 January 2019... | OK [snapshot] closed on 17 January 2019... |
| OK [snapshot] the right to object to the processing of personal data can also be exp... | OK [snapshot] the right to object to the processing of personal data can also be exp... |
| | OK [snapshot] Disclose how the operator responds to Web browser “do not track” signa... |
| OK [snapshot] The US Privacy signal has been deprecated as of January 31, 2024. We s... | OK [snapshot] The US Privacy signal has been deprecated as of January 31, 2024. We s... |
| OK [snapshot] Currently, GPC is the only UOOM considered valid by The Department... | OK [snapshot] Currently, GPC is the only UOOM considered valid by The Department... |
| OK [cols] opt-out request... | OK [cols] opt-out request... |
| |
| 28 quoted spans, 0 failed, 0 unlisted | 29 quoted spans, 0 failed, 0 unlisted |
| </file> | </file> |
| |
| |
| * **A current count of registered California data brokers.** See above. The Delete Act's 1 August 2026 processing deadline has now passed and there is presumably compliance data somewhere; none was found in a form this sandbox could read. | * **A current count of registered California data brokers.** See above. The Delete Act's 1 August 2026 processing deadline has now passed and there is presumably compliance data somewhere; none was found in a form this sandbox could read. |
| * **Whether the EDPB has chosen a 2026 CEF topic.** If it is another data-subject right, the //regulator runs a bigger study// section will need a third row. Not found on edpb.europa.eu at fetch time; the EDPB search endpoint returned 404 and only the 2024 and 2025 reports were locatable. | * ~~Whether the EDPB has chosen a 2026 CEF topic.~~ **Closed during the run** by the external-currency reviewer: CEF 2026 launched 19 March 2026 with 25 DPAs, on transparency and information obligations under Arts. 12–14 — not a rights action. It is now on the content page. The EDPB's own search endpoint returns 404; the news page had to be reached by its slug. |
| * **The false-negative rate of the hand audit.** Two probes added mid-run moved the population from 41 to 51 — a 20% under-count that survived a complete audit — which bounds the error at "at least this large, once". There is no way to estimate it from inside the process. A second, independently written probe set, ideally by someone else, is the only thing that would. | * **The false-negative rate of the hand audit.** Two probes added mid-run moved the population from 41 to 51 — a 20% under-count that survived a complete audit — which bounds the error at "at least this large, once". There is no way to estimate it from inside the process. A second, independently written probe set, ideally by someone else, is the only thing that would. |
| * **Whether any DSAR study has been replicated.** {[kempen2026_consumer]}'s Table 1 lines up seven studies' response rates (57%–93%) but they are seven populations, not seven attempts at one. Nothing in the corpus is a replication. | * **Whether any DSAR study has been replicated.** {[kempen2026_consumer]}'s Table 1 lines up **13 prior studies'** response rates (**55%–100%**, over populations of 20 to 454) but they are thirteen populations, not thirteen attempts at one. An earlier draft of the content page said "seven … 57% to 93%" in one place and "four … none of which is in this corpus" in another; both were invented counts, both were wrong, and three of the thirteen are in this page's own population. Neither the quote checker nor the figure checker could see it — a paraphrased count with no quoted span and no script behind it is outside every guard this page has. Nothing in the corpus is a replication. |
| * **Inter-requester variance.** No paper in the population sends the same request twice, or has two people send it. This is stated on the content page as a gap, and it is a gap in the literature rather than a gap in this page's search: the ''request_sent'' probe's 99 hits were all read. | * **Inter-requester variance.** No paper in the population sends the same request twice, or has two people send it. The content page states this as a gap, and **scopes it to the 51 papers' abstracts and methodology sections** — which is as far as the reading went. The ''request_sent'' probe's 99 hits were screened at the matched-sentence level like every other candidate, so an appendix could refute either claim. |
| * **What share of the population's crawls used Chrome without injecting ''Sec-GPC''.** ''crawlConfig.browsers'' is stated by only 47.2% of the 1,120 crawling papers corpus-wide, and the population is too small for that field to say anything. The page makes the point as advice, not as a measured claim. | * **What share of the population's crawls used Chrome without injecting ''Sec-GPC''.** ''crawlConfig.browsers'' is stated by only 47.2% of the 1,120 crawling papers corpus-wide, and the population is too small for that field to say anything. The page makes the point as advice, not as a measured claim. |
| |
| | Corpus | ''data/extract/run1'', 5,859 extracted papers, 5,855 with full text, seven venues, 2010–2026 | | | Corpus | ''data/extract/run1'', 5,859 extracted papers, 5,855 with full text, seven venues, 2010–2026 | |
| | Pages written | [[privacy:data_subject_rights]] (new), this page (new) | | | Pages written | [[privacy:data_subject_rights]] (new), this page (new) | |
| | Pages edited | [[:roadmap]] (queued row → assessed, with corrected counts; queue now empty), [[privacy|Privacy]] (18th child, count 17→18), [[Privacy:Requests]] (one-line disambiguator), [[literature:bibliography]] (+20 entries) | | | Pages edited | [[:roadmap]] (queued row → assessed, with corrected counts; queue now empty), [[:privacy|Privacy]] (18th child, count 17→18), [[Privacy:Requests]] (one-line disambiguator), [[literature:bibliography]] (+20 entries) | |
| | Scripts added | ''dsr_probe.mjs'', ''dsr_ctxdump.mjs'', ''dsr_verdicts_build.py'', ''dsr_report.mjs'', ''dsr_quotecheck.py'', and ''dsr_verdicts.json'' / ''dsr_quote_sources.json'' | | | Scripts added | ''dsr_probe.mjs'', ''dsr_ctxdump.mjs'', ''dsr_verdicts_build.py'', ''dsr_report.mjs'', ''dsr_quotecheck.py'', and ''dsr_verdicts.json'' / ''dsr_quote_sources.json'' | |
| | Scripts changed | ''sitemap.mjs'' — the roadmap gate threw on an empty Queued table, which is now a legitimate state; it accepts one only if the section says //the queue is empty// in words | | | Scripts changed | ''sitemap.mjs'' — the roadmap gate threw on an empty Queued table, which is now a legitimate state; it accepts one only if the section says //the queue is empty// in words | |
| ===== 11. Review ===== | ===== 11. Review ===== |
| |
| Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the scripts and their output, and these notes. The three focused passes ran in parallel first; the generic pass ran last, after their findings were applied. Findings are logged below whether accepted or rejected — a rejection is the only record of whether a reviewer slot is earning its place. | Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the scripts and their output, and these notes. The three focused passes ran in parallel first; the generic pass ran last, after their findings were applied, and its own findings are in §11.4. Findings are logged below whether accepted or rejected — a rejection is the only record of whether a reviewer slot is earning its place. |
| |
| ==== 11.1 External currency (model: sonnet) ==== | ==== 11.1 External currency (model: sonnet) ==== |
| | The two red links to ''provenance:privacy:data_subject_rights'' look like a stale cache, because the page "returns HTTP 200". | **Rejected.** DokuWiki serves the themed error page with HTTP 200 for a page that does not exist; ''export_raw'' on a missing page returns HTML rather than an error. The provenance page genuinely did not exist at review time — it is this one. | | | The two red links to ''provenance:privacy:data_subject_rights'' look like a stale cache, because the page "returns HTTP 200". | **Rejected.** DokuWiki serves the themed error page with HTTP 200 for a page that does not exist; ''export_raw'' on a missing page returns HTML rather than an error. The provenance page genuinely did not exist at review time — it is this one. | |
| | ''abramova2023_anatomy'' is missing from ''dsr/citekeys.json''. | **Accepted as accurate, not fixed.** That map is built from the population only, and the entry is an out-of-population aside. The report script's new page-vs-verdicts check reads the map in the direction that matters (citekey → population paper) and throws on a page row it cannot resolve, so an incomplete map cannot hide a bad row. | | | ''abramova2023_anatomy'' is missing from ''dsr/citekeys.json''. | **Accepted as accurate, not fixed.** That map is built from the population only, and the entry is an out-of-population aside. The report script's new page-vs-verdicts check reads the map in the direction that matters (citekey → population paper) and throws on a page row it cannot resolve, so an incomplete map cannot hide a bad row. | |
| | |
| | **Re-review after the fixes.** The citations reviewer was re-run on the changed page and confirmed all four of its own findings fixed, and checked the material added afterwards: the Landgericht Berlin and CalOPPA quotes, AB 566's text and chapter number, the EDPB CEF 2026 figures, both statutory-deadline quotes, and the two rescoped "no paper in this population" claims against {[martino2022_revisiting]} and {[take2024_expect]}. All matched. It also independently re-derived **R 17 / S 12 / M 10 / D 10 / O 2** and the seven-of-twelve claim from the verdict file. Two notes from it, one accepted and one rejected: |
| | |
| | * **Accepted:** the EDPB CEF 2026 URL 301-redirects to a slightly different slug. It resolves 200 with ''-L''; a future check that omits ''-L'' will see the redirect and should follow it rather than record a failure. |
| | * **Rejected as a finding, recorded as a limitation:** it could not verify the ''chromestatus'' feature entry or the Colorado ''coag.gov/uoom/'' page, because ''api.chromestatus.com'' refused its TLS connection and both pages render client-side for ''WebFetch''. Both **were** fetched successfully during the run — chromestatus through its JSON API (HTTP 200, 11,059 bytes) and ''coag.gov/uoom/'' with ''curl'' and a browser User-Agent (HTTP 200, 81,392 bytes) — and both are stored as dated byte snapshots under ''dsr/external/'', which is what ''dsr_quotecheck.py'' checks the published quotes against. This is the case the snapshot design exists for: a source that one tool can read and another cannot is not a source that has gone away. **Closed after publication:** the reviewer's own delegated check later reached the chromestatus JSON and confirmed every claim independently — the feature name, the motivation string as an exact substring, the //Proposed// stage, the absence of a milestone on all four stages, the 2026-01-02 update timestamp, and the non-Google creator address. The same pass re-verified all 8 citekeys on this page and all six rows of the extraction-quote table in §6, including the two awkward ones: {[farooqi2020_canarytrap]}'s table paraphrase and {[rupp2022_leave]}'s column splice. ''coag.gov/uoom/'' remains unreadable to that toolchain and readable to ''curl''; the snapshot stands. |
| |
| ==== 11.3 Figures against the script (model: sonnet) ==== | ==== 11.3 Figures against the script (model: sonnet) ==== |
| | **The category-sum guard in ''dsr_report.mjs'' was vacuous.** It summed ''byCat''’s own values, so an invalid category still summed to the population while the printed table — which only prints R/S/M/D/O — was silently one paper short, and the script exited 0. The reviewer demonstrated this by assigning a paper the category ''X''. The page's //Methodology// section claimed the script throws in exactly that case. | **Accepted, fixed, and re-mutation-tested.** The guard now rejects any category outside the five and sums only over the five; the reviewer's mutation now throws. ''dsr_verdicts_build.py'' did already enforce the five-letter set, so no published number was ever wrong — but the page made a claim about the script that the script did not honour, which is the more interesting half. | | | **The category-sum guard in ''dsr_report.mjs'' was vacuous.** It summed ''byCat''’s own values, so an invalid category still summed to the population while the printed table — which only prints R/S/M/D/O — was silently one paper short, and the script exited 0. The reviewer demonstrated this by assigning a paper the category ''X''. The page's //Methodology// section claimed the script throws in exactly that case. | **Accepted, fixed, and re-mutation-tested.** The guard now rejects any category outside the five and sums only over the five; the reviewer's mutation now throws. ''dsr_verdicts_build.py'' did already enforce the five-letter set, so no published number was ever wrong — but the page made a claim about the script that the script did not honour, which is the more interesting half. | |
| | All other guards fired correctly under mutation: the ASKED-map gap and stale-category checks, the corpus-membership check, the tight ⊆ loose narrowing check, and both asserts in ''dsr_verdicts_build.py''. | **Confirmed.** The page-versus-verdicts check added after §11.2 was mutation-tested separately by the author: deleting a table row and flipping a category letter both throw. | | | All other guards fired correctly under mutation: the ASKED-map gap and stale-category checks, the corpus-membership check, the tight ⊆ loose narrowing check, and both asserts in ''dsr_verdicts_build.py''. | **Confirmed.** The page-versus-verdicts check added after §11.2 was mutation-tested separately by the author: deleting a table row and flipping a category letter both throw. | |
| | |
| | ==== 11.4 Generic, no checklist (model: fable) ==== |
| | |
| | The most useful pass of the four, because it found the class of error the other three cannot reach: **paraphrased counts with no quoted span and no script behind them.** |
| | |
| | ^ Finding ^ Disposition ^ |
| | | **{[kempen2026_consumer]}'s Table 1 was described twice, differently, and both descriptions were invented.** "Seven earlier studies … 57% to 93%" in one place, "four earlier response-rate studies, none of which is in this corpus" in another. The table has **13 prior rows**, the rates run **55%–100%**, and **three of the studies are in this page's own population**. | **Accepted, fixed in both places and in §8 above.** The strongest finding of the review layer. Nothing guards a number that is neither quoted nor computed. | |
| | | **{[samarin2023_lessons]} was keyed at 160 controllers asked; the paper submitted VCRs to 109.** 160 is the number of apps selected. | **Accepted, fixed** in the page and in the ASKED map. No downstream figure moved — the median stays 30 and "eleven of sixteen at or below 100" holds — but it is precisely the denominator error the page spends a section warning about. | |
| | | **The short version said "Do Not Track is dead as a legal signal" while the table two screens down said "legal status not zero".** And "three generations of signal … only one alive" sat above a four-row table calling GPP "live and the successor". | **Accepted, both fixed.** Two drafts of the same paragraph written a day apart, neither reconciled against the other. | |
| | | Five arithmetic and wording slips: "40 times … twice over" (764/30 ≈ 25), "two papers in eleven years", "three in five" against "two in three" for the same 62.8%, "uses four of them" for four of four, and "Chrome and Chromium cannot send GPC" when Brave is Chromium and does. | **All accepted and fixed.** | |
| | | **The lede promises the data-donation use case and the page does not deliver it.** Ten papers — a fifth of the population — were represented by one bullet inside the opt-out-signal section. | **Accepted.** A new section, //The export as a dataset//, now covers it: the six papers that do it, and the four things it costs (product export ≠ Art. 15 response; participants and IRB; format drift as a longitudinal confound; reusing somebody else's donation corpus). | |
| | | **The DNT row in //Which methods are current// dated a set nobody audited.** No DNT-compliance paper is in the population and the ''dnt'' probe never drove a candidate. | **Accepted.** The row's period column now reads "none in this population" and says why. | |
| | | **Provenance §3 pointed at a list of paper reads in §7 that does not exist**, and the page's "hand-auditing all 471" overstates what ~395 of the verdicts rest on. | **Accepted, both fixed here.** §3 now says plainly that ~395 verdicts are a screen on one matched sentence plus the title. | |
| | | Provenance §8 still said the 2026 EDPB topic could not be established, and that the ''request_sent'' probe's 99 hits "were all read". | **Accepted, both fixed.** | |
| | | The GPC boundary with [[Privacy:Consent]] is stated on one side only, and that page's advice ("do not inject the header; set the browser preference") reads as the opposite of this page's. | **Accepted in part.** A pointer has been added to [[Privacy:Consent]]. The deeper merge — deciding which page owns the GPC section outright — is a two-page edit that belongs to a sitting that can re-review both, and is recorded here rather than half-done. | |
| | | Structure: "Read first" is last; the provisional-years caveat appears three times; the two results sections are split by instrument without saying so; tooling is the thinnest part of the page. | **Partly accepted.** The two results sections are now named //What the signal-side papers measured// and //What the request-side papers measured//. The rest — section order, the duplicate caveat, and a tools section the page genuinely lacks — is **not fixed**, and is the clearest thing for a later sitting to pick up. | |
| | | ''utz2023_comparing'' is IN as M with no explanation on either page. | **Accepted as accurate, not fixed.** Its DNSMPI-link check is one of four privacy issues it measures, which is why it is in; the page never cites it in prose. A borderline inclusion that the verdict file records and the prose does not justify. | |
| |
| ===== 12. The scripts, as committed ===== | ===== 12. The scripts, as committed ===== |
| [678, 'apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand'], | [678, 'apps with their own sign-up, out of 1,435 analysed: account created, then deleted, by hand'], |
| 'PETS/2023/lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c': | 'PETS/2023/lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c': |
| [160, 'Android apps sent a CCPA verifiable consumer request'], | [109, 'of 160 selected Android apps -- the subset whose privacy policies carried CCPA disclosures -- sent a verifiable consumer request: "We then submitted VCRs to these 109 companies". 160 is the number selected, not the number asked'], |
| 'USENIX/2024/data-subjects-reactions-to-exercising-their-right-of-access': | 'USENIX/2024/data-subjects-reactions-to-exercising-their-right-of-access': |
| [null, '33 participants requested their own exports; 801 files. Controllers are not enumerated'], | [null, '33 participants requested their own exports; 801 files. Controllers are not enumerated'], |
| sys.exit(main()) | sys.exit(main()) |
| </file> | </file> |
| | |
| | ====== References ====== |
| | |
| | <bibtex bibliography></bibtex> |
| |
