| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:practices:notifying_websites [2026/08/13 20:07] – Record the channel-fold rescoping; regenerate byte-exact script output; fix a literal citation placeholder and nested markup. Authored by Claude karel.kubicek.claude | provenance:practices:notifying_websites [2026/09/17 07:23] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| Not prose. Read it if you are checking a number. | Not prose. Read it if you are checking a number. |
| |
| <wrap todo>House convention on ''provenance:'' pages is mixed and this page picked the more traceable end of it. ''provenance:privacy:fingerprinting'' and ''provenance:design:ip_classification'' use no inline citation markers at all and name papers by slug; ''provenance:privacy:requests'' uses citekeys and a bibliography block, as this page does. Two of the four carry a footer back-link, so this one does too. **No ''~~DISCUSSION~~'' block**, matching all four: comments belong on the content page.</wrap> | <WRAP todo>House convention on ''provenance:'' pages is mixed and this page picked the more traceable end of it. ''provenance:privacy:fingerprinting'' and ''provenance:design:ip_classification'' use no inline citation markers at all and name papers by slug; ''provenance:privacy:requests'' uses citekeys and a bibliography block, as this page does. Two of the four carry a footer back-link, so this one does too. **No ''~~DISCUSSION~~'' block**, matching all four: comments belong on the content page.</WRAP> |
| |
| ===== The run ===== | ===== The run ===== |
| |
| ^ Field ^ Value ^ | ^ Field ^ Value ^ |
| | Date | 2026-08-13 | | | Date | 2026-08-13 (page created); **2026-09-04 (this revision: the 144 unreviewed campaign candidates screened, 18 of them read in full, 10 campaigns added, security.txt questions resolved as far as the sources allow)** | |
| | Page status | **new page**. ''practices:notifying_websites'' was a red link promised from [[start]] since the site's outline was written; nothing existed at that id. | | | Page status | **new page**. ''practices:notifying_websites'' was a red link promised from [[:start]] since the site's outline was written; nothing existed at that id. | |
| | Corpus at the time | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026. 5,869 ''paper.cols.txt'' files. | | | Corpus at the time | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P), 2010–2026. 5,869 ''paper.cols.txt'' files. | |
| | Scripts written | ''scripts/report_notifying_websites.mjs'', ''scripts/notify_fold.mjs'', ''notes/find_contacts.py'' | | | Scripts written | ''scripts/report_notifying_websites.mjs'', ''scripts/notify_fold.mjs'', ''notes/find_contacts.py''; 2026-09-04: ''scripts/notify_residue_context.mjs'' (per-candidate notification-sentence dump), ''scripts/_notify_recall_probe.mjs'' (the wider rule, since folded into ''notify_fold.mjs''), ''scripts/notify_quotecheck2.mjs'' + ''out/notify_quotes_20260904.json'' (67 quotes) | |
| | Models | Orchestration, all corpus queries, the fold classification, all quote spot-checks and the page text: Opus 5. Four Sonnet sub-agents: three paper-extraction passes (Li 2016 ×2; Maass 2021 + PETS 2023 ×2; security.txt + 2022–2026 papers) and one external-currency pass. Four Sonnet/Fable reviewers, logged below. | | | Models | 2026-08-13: orchestration, all corpus queries, the fold classification, all quote spot-checks and the page text: Opus 5. Four Sonnet sub-agents: three paper-extraction passes (Li 2016 ×2; Maass 2021 + PETS 2023 ×2; security.txt + 2022–2026 papers) and one external-currency pass. Four Sonnet/Fable reviewers, logged below. 2026-09-04: orchestration, the sentence-level screen of all 222 candidates, the fold edits, every quote re-check and all page text: Fable 5.1. Four Sonnet sub-agents read nine papers in full (batches 1–4 below). Reviewers logged below. | |
| | Sub-agent outputs kept | ''notes/notify_paper_li2016.md'', ''notes/notify_papers_campaigns.md'' | | | Sub-agent outputs kept | ''notes/notify_paper_li2016.md'', ''notes/notify_papers_campaigns.md''; 2026-09-04: ''notes/notify_read_batch1.md'' … ''batch4.md'' | |
| | Corrections caught in review | listed under //Reviewer findings// | | | Corrections caught in review | listed under //Reviewer findings// | |
| |
| ===== Scope decision ===== | ===== Scope decision ===== |
| |
| The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Two judgement calls: | The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Three judgement calls: |
| |
| * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. | * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. |
| * **Scope includes network-scanning campaigns, not only web ones.** The page is titled "Notifying Websites" but 10 of the 22 hand-classified campaigns notified network operators rather than site owners, and the methodological literature (control groups, channel comparison, decay curves, the one RCT) is disproportionately on that side. Excluding it would have produced a page with three campaigns on it. The page states which population each figure comes from. **A reasonable person would have narrowed this to web only and produced a thinner, more on-title page.** | * **Scope includes network-scanning campaigns, not only web ones.** The page is titled "Notifying Websites" but 13 of the 32 hand-classified campaigns notified network operators and 6 notified app or skill developers rather than site owners (10 of 22 and 2 of 22 on 2026-08-13), and the methodological literature (control groups, channel comparison, decay curves, the one RCT) is disproportionately on that side. Excluding it would have produced a page with three campaigns on it. The page states which population each figure comes from. **A reasonable person would have narrowed this to web only and produced a thinner, more on-title page.** |
| * **Not broadening a neighbour.** [[Practices:Ethics]] exists but is 2.6 kB of bullet-point notes covering crawling harm, robots.txt and ethics approval. Notification is one line in its outline. Broadening it would have buried a 30 kB page inside a stub on a different topic. The two pages cross-link instead. | * **Not broadening a neighbour.** [[Practices:Ethics]] exists but is 2.6 kB of bullet-point notes covering crawling harm, robots.txt and ethics approval. Notification is one line in its outline. Broadening it would have buried a 30 kB page inside a stub on a different topic. The two pages cross-link instead. |
| |
| ===== Report script and its unedited output ===== | ===== Report script and its unedited output ===== |
| |
| Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. | Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''%%--wiki%%'' for DokuWiki tables, ''%%--list%%'' for the hand-classified campaigns and every rejection with its reason, ''%%--quotes%%'' for ethics evidence quotes, ''%%--residue%%'' for every unmapped fold entry and every unreviewed candidate (none, as of 2026-09-04). The block below is the unedited output of a fresh run on 2026-09-04 (''out/notifying_websites-output-20260904.txt''); sections A–D are byte-identical to the 2026-08-13 run, section E changed. |
| |
| <wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run (lines 1–143) followed by the ''--list'' run (lines 144–448), which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows ~305 spurious extra lines. This misled one reviewer; ''head -143'' and ''tail -n +144'' each match a fresh run byte-for-byte. The script itself is deterministic — verified.</wrap> | <WRAP todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run followed by the ''%%--list%%'' run, which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows a few hundred spurious extra lines, and this misled one reviewer. Do not hard-code the split point — it moves whenever a table row is added. Derive it: ''BASE=$(node scripts/report_notifying_websites.mjs | wc -l)'', then ''head -$BASE'' and ''tail -n +$((BASE+1))'' each match a fresh run byte-for-byte. As of the 2026-09-04 revision ''BASE'' is 168 (''out/notifying_websites-output-20260904.txt'' is a single plain run; ''out/notifying_websites-output-list-20260904.txt'' is a single ''%%--list%%'' run). The script itself is deterministic — re-verified 2026-09-04.</WRAP> |
| |
| Sections of the script, so a figure on the content page can be traced to the block that produced it: | Sections of the script, so a figure on the content page can be traced to the block that produced it: |
| * **C** — ''ethics.regulatorContact''; ''ethics.harmMitigation'' folded into families with residue | * **C** — ''ethics.regulatorContact''; ''ethics.harmMitigation'' folded into families with residue |
| * **D** — ''ethics.disclosureDetail'' folded into channel and outcome families with residue, plus every entry carrying a countable ratio | * **D** — ''ethics.disclosureDetail'' folded into channel and outcome families with residue, plus every entry carrying a countable ratio |
| * **E** — full-text candidate scan, hand classification, and the unreviewed residue | * **E** — full-text candidate scan (narrow rule and, since 2026-09-04, the wide recall probe), hand classification, and the unreviewed residue |
| * **F** — (''--quotes'') ethics evidence quotes for web papers with ''notifiedAffectedParties == yes'' | * **F** — (''%%--quotes%%'') ethics evidence quotes for web papers with ''notifiedAffectedParties == yes'' |
| |
| <code> | <code> |
| |
| of the 376 that assessed a law: yes 27 (7.2%) | of the 376 that assessed a law: yes 27 (7.2%) |
| | |
| | of the 2160 that notified an affected party, 135 also contacted a regulator or CERT (6.3%) |
| | of the 147 that contacted a regulator or CERT, 135 ALSO notified the affected party directly (91.8%) — the intermediary is an addition, not a substitute |
| |
| -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- | -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- |
| E. NOTIFICATION CAMPAIGNS IN THE FULL TEXT (hand-classified) | E. NOTIFICATION CAMPAIGNS IN THE FULL TEXT (hand-classified) |
| ========================================================================== | ========================================================================== |
| Stage Papers | Stage Papers |
| ------------------------------------------------------ ------ | ---------------------------------------------------------------------------------------- ------ |
| paper.cols.txt files read 5869 | paper.cols.txt files read 5869 |
| candidates (notify ∧ outcome ∧ number in one sentence) 179 | NARROW rule: notify ∧ outcome ∧ number in one sentence 179 |
| hand-classified as a campaign 22 | → hand-classified as a campaign 28 |
| hand-classified and rejected, with a reason 16 | → hand-classified and rejected, with a reason 151 |
| UNREVIEWED residue — matched, nobody read the paper 144 | → UNREVIEWED residue — matched, nobody read the paper 0 |
| | WIDE probe only: wider outcome/number vocabulary ∧ we/our, not caught by the narrow rule 78 |
| | → hand-classified as a campaign 3 |
| | → hand-classified and rejected, with a reason 75 |
| | → UNREVIEWED residue 0 |
| | campaigns in total (CAMPAIGNS) 32 |
| |
| sanity: campaigns not caught by the regex (added from a wider first-pass scan): 0 | sanity: campaigns caught by neither rule (found by a topical full-text grep, see notify_fold.mjs caught:'grep'): 1 — the-security-lottery-measuring-client-side-web-security-inconsistencies |
| rejected slugs not caught by the regex: 3 — how-website-owners-face-privacy-issues-thematic-analysis-of-responses-from-a-cov, from-whois-to-whowas-a-large-scale-measurement-study-of-domain-registration-privacy-under-the-gdpr, the-cve-wayback-machine-measuring-coordinated-disclosure-from-exploits-against-t | campaigns caught only by the wide probe: 3 — playing-without-paying-detecting-vulnerable-payment-verification-in-native-binar, lazy-gatekeepers-a-large-scale-study-on-spf-configuration-in-the-wild, demystifying-rpki-invalid-prefixes-hidden-causes-and-security-risks |
| | REJECTED slugs caught by neither rule: 3 — how-website-owners-face-privacy-issues-thematic-analysis-of-responses-from-a-cov, from-whois-to-whowas-a-large-scale-measurement-study-of-domain-registration-privacy-under-the-gdpr, the-cve-wayback-machine-measuring-coordinated-disclosure-from-exploits-against-t |
| | REJECTED_PROBE slugs caught by neither rule: 0 |
| |
| -- campaigns by decade-ish bucket, of 22 hand-classified -- | -- campaigns by four-year bucket, of 32 hand-classified -- |
| Bucket Campaigns | Bucket Campaigns |
| ---------- --------- | ---------- --------- |
| 2010–2013 1 | 2010–2013 1 |
| 2014–2017 4 | 2014–2017 6 |
| 2018–2021 4 | 2018–2021 5 |
| 2022–2024 8 | 2022–2024 14 |
| 2025–2026* 5 | 2025–2026* 6 |
| What was notified about Campaigns | Who was notified (network operators / web site operators / app or skill developers) Campaigns |
| ----------------------- --------- | ----------------------------------------------------------------------------------- --------- |
| network 10 | network 13 |
| web 10 | web 13 |
| mobile 2 | apps 6 |
| </code> | </code> |
| |
| Two of the four figures in ''ethics.disclosureDetail'' ratios above are quoted on the content page and were re-verified against the JSONL by exact substring match: ''an-extensive-study-of-residential-proxies-in-china'' (CCS 2022) and ''fv8-a-forced-execution-javascript-engine-for-detecting-evasive-techniques'' (USENIX Sec 2024), one match each. | Two of the seven ''ethics.disclosureDetail'' ratios above are quoted on the content page and were re-verified against the JSONL by exact substring match: ''an-extensive-study-of-residential-proxies-in-china'' (CCS 2022) and ''fv8-a-forced-execution-javascript-engine-for-detecting-evasive-techniques'' (USENIX Sec 2024), one match each. |
| |
| ===== Folding: rules and residue ===== | ===== Folding: rules and residue ===== |
| Free-text fields agree run-to-run on roughly 20% of exact strings, so nothing here is aggregated by exact string. Three folds were written, in ''scripts/report_notifying_websites.mjs'' and ''scripts/notify_fold.mjs''. | Free-text fields agree run-to-run on roughly 20% of exact strings, so nothing here is aggregated by exact string. Three folds were written, in ''scripts/report_notifying_websites.mjs'' and ''scripts/notify_fold.mjs''. |
| |
| ==== 1. ''ethics.disclosureDetail'' → channel families ==== | ==== 1. ethics.disclosureDetail → channel families ==== |
| |
| Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates. | Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates. |
| * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. | * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. |
| * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. | * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. |
| | * **Two claims on the content page about this fold were checked directly against the JSONL rather than taken from the script.** (a) "all 2,160 that said they notified are among them": of the 2,160 ''yes''/''partial'' papers, **0** lack a ''disclosureDetail''. (b) The scoping footnote's 38.5%: of the 710 papers that gave a detail without saying they notified, 273 match a participant/consent/debrief/interview/survey/recruit/volunteer term. That second figure exists because a reviewer challenged an earlier, stronger claim that the excluded group //was// participant-debriefing notes — it is not, and the page now says only that the channel question is ill-posed for that group. |
| * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** | * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** |
| * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. | * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. |
| |
| ==== 2. ''ethics.disclosureDetail'' → outcome families ==== | ==== 2. ethics.disclosureDetail → outcome families ==== |
| |
| Five families, non-exclusive. Residue 2,155 of 2,870 (75.1%), published above. Not used on the content page except as the 75.1% figure. | Five families, non-exclusive. Residue 2,155 of 2,870 (75.1%), published above. Not used on the content page except as the 75.1% figure. |
| |
| ==== 3. ''ethics.harmMitigation'' → families ==== | ==== 3. ethics.harmMitigation → families ==== |
| |
| Eleven families, non-exclusive. **Residue 1,069 of 3,186 (33.6%).** This fold is **diagnostic only — no folded harm-mitigation figure appears on the content page**; the page uses only the non-null count (71.2%) and its per-bucket trend, neither of which depends on the fold. The residue is therefore recorded here by shape rather than in full, and prints with ''node scripts/report_notifying_websites.mjs --residue''. | Eleven families, non-exclusive. **Residue 1,069 of 3,186 (33.6%).** This fold is **diagnostic only — no folded harm-mitigation figure appears on the content page**; the page uses only the non-null count (71.2%) and its per-bucket trend, neither of which depends on the fold. The residue is therefore recorded here by shape rather than in full, and prints with ''node scripts/report_notifying_websites.mjs %%--residue%%''. |
| |
| Shape of the 1,069, read from the first ~60 entries and a shuffled sample of 45: almost entirely **study-specific harm limitation with no generalisable family** — "Used a wired testbed and 900-MHz transmitter to minimize GPS interference", "Purchased unlimited daily plans so test traffic was not counted", "Suppressed uploads from BitTorrent clients to avoid distributing illegal software", "Captured only four bytes of application payload to minimise privacy impact". A first attempt with mutually-exclusive first-match-wins rules and narrower regexes gave 46.5% residue and a nonsensical zero in one family; broadening the regexes and making families non-exclusive took it to 33.6%. **It was not driven lower, because the remainder is genuinely one-off.** Anyone wanting a publishable harm-mitigation taxonomy should treat that residue as the work, not as noise. | Shape of the 1,069, read from the first ~60 entries and a shuffled sample of 45: almost entirely **study-specific harm limitation with no generalisable family** — "Used a wired testbed and 900-MHz transmitter to minimize GPS interference", "Purchased unlimited daily plans so test traffic was not counted", "Suppressed uploads from BitTorrent clients to avoid distributing illegal software", "Captured only four bytes of application payload to minimise privacy impact". A first attempt with mutually-exclusive first-match-wins rules and narrower regexes gave 46.5% residue and a nonsensical zero in one family; broadening the regexes and making families non-exclusive took it to 33.6%. **It was not driven lower, because the remainder is genuinely one-off.** Anyone wanting a publishable harm-mitigation taxonomy should treat that residue as the work, not as noise. |
| |
| ==== 4. The campaign classification (''scripts/notify_fold.mjs'') ==== | ==== 4. The campaign classification (scripts/notify_fold.mjs) ==== |
| |
| The one fold that is a hand list rather than a regex family list, and the reason is in the file: the corpus signal for "the authors ran a campaign" is a sentence, not a name. ''ethics.notifiedAffectedParties == yes'' does not distinguish a bulk campaign to thousands of strangers from one email to Google's security team, and those are different methods with different rates. | The one fold that is a hand list rather than a regex family list, and the reason is in the file: the corpus signal for "the authors ran a campaign" is a sentence, not a name. ''ethics.notifiedAffectedParties == yes'' does not distinguish a bulk campaign to thousands of strangers from one email to Google's security team, and those are different methods with different rates. |
| * ''NUM_RE'' = a percentage, a comma-grouped integer, or an ''N of M'' ratio | * ''NUM_RE'' = a percentage, a comma-grouped integer, or an ''N of M'' ratio |
| |
| Result over 5,869 files: **179 candidates → 22 campaigns, 16 rejected with a reason, 144 unreviewed.** | Result over 5,869 files on 2026-08-13: **179 candidates → 22 campaigns, 16 rejected with a reason, 144 unreviewed.** On 2026-09-04 the 144 were read: **6 promoted, 138 rejected with a reason** (see the listing below). A **wider recall probe** was added the same day after a full-text search for ''security.txt'' turned up two campaigns the narrow rule had missed — IMC 2023 ''lazy-gatekeepers'' ("we have contacted 111 951 operators": thin-space thousands, and "success rate" is not an outcome word) and USENIX Sec 2022 ''the-security-lottery''. The probe (''isWideCandidate'': same notification verbs; outcome vocabulary + ''success|deliver|opened|unopened|acted|took action|clean-up|mitigat|feedback''; numbers may be space-grouped; a first-person ''we''/''our'' is required so that "hosts contacted by malware" does not flood it) catches **78 further papers: 3 promoted, 75 rejected.** ''the-security-lottery'' is caught by neither rule and is in ''CAMPAIGNS'' with ''caught: 'grep'''. The script prints the two rules as separate stages and asserts that every slug either catches is classified; both UNREVIEWED counts are 0. |
| |
| Inclusion threshold, applied by hand: the authors notified parties they did not already know, at a scale where the contact had to be looked up, //and// reported a delivery, response or remediation outcome. Rough scale floor ≥100 parties, which is why PETS 2022's 40-organisation SAR re-test is in the rejected list and quoted only in prose. | Inclusion threshold, applied by hand: the authors notified parties they did not already know, at a scale where the contact had to be looked up, //and// reported a delivery, response or remediation outcome. Rough scale floor ≥100 parties, which is why PETS 2022's 40-organisation SAR re-test is in the rejected list and quoted only in prose. |
| |
| An earlier, wider first-pass scan (a looser regex, 197 candidates) was used to seed the hand list; the script reports that **0 campaigns and 3 rejections fall outside the committed narrow regex**, so the committed rule reproduces the campaign set. The three rejections it misses are ''how-website-owners-face-privacy-issues…'', ''from-whois-to-whowas…'' and ''the-cve-wayback-machine…'' — kept in ''REJECTED'' anyway because they are papers a future run will want the reason for. | An earlier, wider first-pass scan (a looser regex, 197 candidates) was used to seed the 2026-08-13 hand list; the script reports that 3 rejections fall outside both committed rules — ''how-website-owners-face-privacy-issues…'', ''from-whois-to-whowas…'' and ''the-cve-wayback-machine…'' — kept in ''REJECTED'' because they are papers a future run will want the reason for. **The recall lesson of 2026-09-04:** the narrow rule had a precision of 28/179 and caught 28 of the 32 campaigns now known; **true recall is unmeasured**, because the 32 are the union of what the rules and two ad-hoc greps found, not an independent gold set, so the miss count is a lower bound. The two committed rules together still miss one of the 32. A third rule — say, any sentence with a notification verb and a five-digit or larger count — would find more. That is the residual uncertainty behind "32 campaigns". |
| |
| **Rejections, with the deciding reason** (this is the part that stops the next run re-adding them): | **Rejections, with the deciding reason** (this is the part that stops the next run re-adding them): |
| | ''revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an'' | PETS 2022 — 40 organisations re-tested after disclosure. Under the ≥100-party floor; quoted in prose only. | | | ''revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an'' | PETS 2022 — 40 organisations re-tested after disclosure. Under the ≥100-party floor; quoted in prose only. | |
| |
| **The six campaigns not on the content page's rates table**, because they report a volume sent but no response or remediation figure: ''freely-given-consent…'' (CCS 2022, 1,127 developers), ''extended-hell-o…'' (USENIX Sec 2023), ''dancer-in-the-dark…'' (USENIX Sec 2024, 19% response — this one //is// quoted in prose), ''beyond-exploit-scanning…'' (USENIX Sec 2025, 4,983 emails), ''hey-your-secrets-leaked…'' (IEEE S&P 2025, 3,906 emails), and ''beyond-the-horizon…'' (IEEE S&P 2025, 15,837 ASes). | **The six campaigns not on the content page's rates table**, because they report a volume sent but no response or remediation figure: ''freely-given-consent…'' (CCS 2022, 1,127 developers), ''dancer-in-the-dark…'' (USENIX Sec 2024, 19% response — this one //is// quoted in prose), ''beyond-exploit-scanning…'' (USENIX Sec 2025, 4,983 emails), ''hey-your-secrets-leaked…'' (IEEE S&P 2025, 3,906 emails), ''beyond-the-horizon…'' (IEEE S&P 2025, 15,837 ASes), and — added 2026-09-04 — {[zuo2022_playing]} (USENIX Sec 2022, 5,494 game developers emailed via their Play Store address; "Many of the developers have acknowledged our findings", no count). |
| |
| So the rates table has **17 rows: 16 of the 22 hand-classified campaigns, plus {[stock2018_didnt]}**, which is not in ''CAMPAIGNS'' because its full text is not in the corpus (see below) and the fold only classifies papers the scan could read. ''catch-22…'' is on the table for its channel, not for a rate. | So the rates table has **27 rows: 26 of the 32 hand-classified campaigns, plus {[stock2018_didnt]}**, which is not in ''CAMPAIGNS'' because its full text is not in the corpus (see below) and the fold only classifies papers the scan could read. ''catch-22…'' and ''seeking-nonsense…'' are on the table for their channel, not for a rate. 26 + 6 = 32. (''extended-hell-o…'', USENIX Sec 2023, was listed among the not-shown on 2026-08-13 although ''CAMPAIGNS'' records its 39.7% remediation figure; the 2026-09-04 figures reviewer caught the carry-over and it is now row {[blechschmidt2023_hello]} on the table.) |
| |
| ==== The 144 unreviewed candidates, in full ==== | ==== The 144 candidates that were unreviewed on 2026-08-13, and the 78 from the recall probe — all classified 2026-09-04 ==== |
| |
| Matched the candidate rule; **nobody read the paper.** Any of these may be a campaign the content page should have listed. The rule has known false positives — a candidate sentence can be a paper describing //someone else's// campaign in related work, or a survey response rate, or a sentence about "disclosure" in an unrelated sense (self-disclosure in social media, information disclosure as a vulnerability class), all of which are visible in the list below. | How they were read. ''scripts/notify_residue_context.mjs'' dumped, for each candidate, **every sentence in ''paper.cols.txt'' that carries a notification-act verb** (notify/disclose/report to/contact/reach out/inform, minus the noise senses: information disclosure, self-disclosure, push notification, disclosure date), with its line number — 5,584 lines for the 144 papers, tightened to 1,701 by requiring a number or an outcome word in the same sentence. Those were read in full by the orchestrator. **The 78 probe candidates were screened differently and more thinly: on the one to three probe-matching sentences the probe itself printed (''out/notify_recall_probe-output.txt''), not on every notification-verb sentence.** Anything that looked like a campaign, or was cited on the page for some other reason, was then read in full: nine papers by four Sonnet sub-agents (''notes/notify_read_batch1.md''–''batch4.md''), eight more by the orchestrator from the relevant passages. The tag on each rejection records which: ''[sentences]'' or ''[read]''. **201 of the 229 rejections are sentence-level screens (126 on all notification-verb sentences, 75 on the probe sentences only); 28 rest on a full read** — a paper that describes its campaign without a notification verb in the same sentence as a number would have slipped past. |
| | |
| | Inclusion rule, unchanged from 2026-08-13: the authors notified parties they did not already know, at a scale where the contact had to be looked up (roughly ≥100 parties), and reported a delivery, response or remediation outcome. Two judgement calls at the edge, both recorded in the fold file: ''seeking-nonsense…'' (IEEE S&P 2016) is in because a national CERT notified the organisations on the authors' behalf and 27 fixes are reported, even though the denominator is not — the same pattern as ''catch-22…''; ''demystifying-rpki-invalid-prefixes…'' (NDSS 2026) is in although its mail asked for confirmation rather than a fix, because it is the only 2025–2026 delivery figure for RIR WHOIS contacts in the corpus and the page says what it is. |
| | |
| | Rejections that a looser rule would have counted: ''weak-keys…'' (61 vendors, tabulated responses — manufacturers, not operators); ''mind-your-manrs…'' (six operators); ''inconsistent-incomplete…'' (58 disclosure briefings to a defined set of services); ''behind-closed-doors…'' (43 administrators); ''an-evaluation-of-the-google-chrome-extension…'' (32 developers, 19 fixed); ''the-unusual-suspects…'' (482 webshop owners cold-called — to recruit interviewees, not to notify). All quoted or noted on the page where they carry a point; none in the table. |
| |
| <code> | <code> |
| IMC 2010 internet-background-radiation-revisited | # Classification of the 138 candidates unreviewed on 2026-08-13 (narrow rule), 2026-09-04 |
| IMC 2010 selecting-representative-ip-addresses-for-internet-topology-studies | # PROMOTED (6) |
| CCS 2012 before-we-knew-it-an-empirical-study-of-zero-day-attacks-in-the-real-world | seeking-nonsense-looking-for-trouble-efficient-promotional-infection-detection-t :: IEEE-SP 2016 — "27 responded and fixed their problems" among the organisations CCERT notified; the denominator is not stated, so no rate is computable |
| NDSS 2013 when-firmware-modifications-attack-a-case-study-of-embedded-exploitation | are-we-there-yet-on-rpki-s-deployment-and-security :: NDSS 2017 — only 168 of 552 emails did not bounce (69.6% bounced); "Over 42%" of bad-ROA alerts fixed within a month against "about 15%" for operators R |
| USENIX 2013 an-empirical-study-of-vulnerability-rewards-programs | complex-security-policy-a-longitudinal-analysis-of-deployed-content-security-policies :: NDSS 2020 — "most emails bounced"; 117 non-automated responses; sites deploying frame-ancestors rose from 511 to 554 (+43) by 12 June 2019; 62 responder |
| WWW 2013 two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure | measuring-alexa-skill-privacy-practices-across-three-years :: WWW 2022 — of the 246: 111 (45.12%) "no longer pose a threat" about a year later (45 removed, 24 permission removed, 41 fixed — the parts sum to 110); |
| CCS 2013 rethinking-ssl-development-in-an-appified-world (also in REJECTED) | are-you-spying-on-me-large-scale-analysis-on-iot-data-exposure-through-companion :: USENIX 2023 — after one month: opened by 381, unopened for 850, bounced from 150 invalid addresses; 21 of the 381 openers acknowledged and changed their p |
| USENIX 2014 a-look-at-targeted-attacks-through-the-lense-of-an-ngo | are-we-getting-well-informed-an-in-depth-study-of-runtime-privacy-notice-practic :: CCS 2024 — 4,169 delivered; 821 unique replies, 60 of them not automatic, 13 confirming the report with substantive feedback; no re-scan, so no remedia |
| USENIX 2014 brahmastra-driving-apps-to-test-the-security-of-third-party-components | |
| CCS 2015 an-empirical-study-of-web-vulnerability-discovery-ecosystems (also in REJECTED) | # REJECTED (138), tag = screening level |
| IEEE-SP 2015 the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p | internet-background-radiation-revisited :: [sentences] IMC 2010 — contacted the software vendors responsible for a misconfiguration; a few vendors, no counts. |
| NDSS 2015 checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo | selecting-representative-ip-addresses-for-internet-topology-studies :: [sentences] IMC 2010 — 'responsive' and 'informed' are probing vocabulary (hitlist responsiveness); nothing was notified. |
| PETS 2015 portrait-of-a-privacy-invasion | before-we-knew-it-an-empirical-study-of-zero-day-attacks-in-the-real-world :: [sentences] CCS 2012 — 'disclosure' is the public disclosure date of a vulnerability; no notification sent. |
| IEEE-SP 2016 seeking-nonsense-looking-for-trouble-efficient-promotional-infection-detection-t | when-firmware-modifications-attack-a-case-study-of-embedded-exploitation :: [sentences] NDSS 2013 — one vendor (HP) disclosure; the 1.08% is end-user patching of 90,000 printers after the vendor fix, not a campaign outcome. |
| IEEE-SP 2016 talos-neutralizing-vulnerabilities-with-security-workarounds-for-rapid-response | an-empirical-study-of-vulnerability-rewards-programs :: [sentences] USENIX Sec 2013 — Chrome/Firefox reward programmes analysed from public data. |
| IMC 2016 weak-keys-remain-widespread-in-network-devices | two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure :: [sentences] WWW 2013 — users contacted the authors about a browser add-on; no notification. |
| PETS 2016 towards-mining-latent-client-identifiers-from-network-traffic | a-look-at-targeted-attacks-through-the-lense-of-an-ngo :: [sentences] USENIX Sec 2014 — CVE public disclosure dates. |
| CCS 2017 a-large-scale-empirical-study-of-security-patches (also in REJECTED) | brahmastra-driving-apps-to-test-the-security-of-third-party-components :: [sentences] USENIX Sec 2014 — list of vulnerable apps shared with the Facebook security team, which contacted developers itself; one platform. |
| IMC 2017 if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to | the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p :: [sentences] IEEE S&P 2015 — patch deployment against public disclosure dates; no notification. |
| NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security | checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo :: [sentences] NDSS 2015 — 'privacy disclosures' are app data flows. |
| PETS 2017 wiretapping-end-to-end-encrypted-voip-calls-real-world-attacks-on-zrtp | portrait-of-a-privacy-invasion :: [sentences] PETS 2015 — users disclosing relationship status. |
| USENIX 2018 from-patching-delays-to-infection-symptoms-using-risk-profiles-for-an-early-disc | talos-neutralizing-vulnerabilities-with-security-workarounds-for-rapid-response :: [sentences] IEEE S&P 2016 — patch timing after public disclosure from CVE data. |
| USENIX 2018 wpse-fortifying-web-protocols-via-browser-side-security-monitoring | weak-keys-remain-widespread-in-network-devices :: [read] IMC 2016 — vendor disclosure: 61 vendors in 2012 (37 with RSA issues: 5 public advisory, 11 private response, 3 auto-response, 18 no response — Table 2 is a flattened four-column layout; the 12/9 reading of 2026-09-04 was corrected to 3/18 by a reviewer working from the PDF's column positions, and the caption's 'about half acknowledged receipt' (19 of 37) confirms it) and 5 more in May 2016 (2 acknowledged, 1 advisory). Manufacturers rather than operators, and under the 100-party floor; quoted on the page for 'little correlation with end-user security'. |
| CCS 2019 a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right | towards-mining-latent-client-identifiers-from-network-traffic :: [sentences] PETS 2016 — Apple Push Notification traffic. |
| CCS 2019 network-hygiene-incentives-and-regulation-deployment-of-source-address-validatio (also in REJECTED) | if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to :: [sentences] IMC 2017 — RTB winning-price notifications. |
| NDSS 2019 automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries | wiretapping-end-to-end-encrypted-voip-calls-real-world-attacks-on-zrtp :: [sentences] PETS 2017 — one vendor, one CVE. |
| NDSS 2019 digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system | from-patching-delays-to-infection-symptoms-using-risk-profiles-for-an-early-disc :: [sentences] USENIX Sec 2018 — public disclosure and patch data. |
| NDSS 2019 we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy | wpse-fortifying-web-protocols-via-browser-side-security-monitoring :: [sentences] USENIX Sec 2018 — disclosed to the owners of 55 websites and to library vendors; no counts, under the floor. |
| PETS 2019 accessileaks-investigating-privacy-leaks-exposed-by-the-android-accessibility-se | a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right :: [sentences] CCS 2019 — the matched sentence is related work about Heartbleed; the paper is a lab study. |
| USENIX 2019 50-ways-to-leak-your-data-an-exploration-of-apps-circumvention-of-the-android-pe | automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries :: [sentences] NDSS 2019 — describes Google's App Security Improvement Program; no notification by the authors. |
| USENIX 2019 evaluating-the-contextual-integrity-of-privacy-regulation-parents-iot-toy-privac | digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system :: [sentences] NDSS 2019 — one hospital system's administrators. |
| USENIX 2019 the-betrayal-at-cloud-city-an-empirical-analysis-of-cloud-based-mobile-backends | we-value-your-privacy-now-take-some-cookies-measuring-the-gdprs-impact-on-web-privacy :: [sentences] NDSS 2019 — 'cookie consent notifications'. |
| WWW 2019 studying-preferences-and-concerns-about-information-disclosure-in-email-notifica | accessileaks-investigating-privacy-leaks-exposed-by-the-android-accessibility-se :: [sentences] PETS 2019 — a user-facing warning notification and pull requests to app developers; no campaign. |
| CCS 2020 dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat | 50-ways-to-leak-your-data-an-exploration-of-apps-circumvention-of-the-android-pe :: [sentences] USENIX Sec 2019 — disclosed to Google and the FTC. |
| CCS 2020 the-cookie-hunter-automated-black-box-auditing-for-web-authentication-and-author (also in REJECTED) | evaluating-the-contextual-integrity-of-privacy-regulation-parents-iot-toy-privac :: [sentences] USENIX Sec 2019 — COPPA notification-and-consent principles in a survey. |
| IEEE-SP 2020 a-security-analysis-of-the-facebook-ad-library | the-betrayal-at-cloud-city-an-empirical-analysis-of-cloud-based-mobile-backends :: [read] USENIX Sec 2019 — tiered notification (Play Store contact, then WHOIS, then Google's tracker) to developers of vulnerable backends, but 'awaiting remediation' with no counts of parties, deliveries or replies. |
| IEEE-SP 2020 the-many-kinds-of-creepware-used-for-interpersonal-attacks | studying-preferences-and-concerns-about-information-disclosure-in-email-notifica :: [sentences] WWW 2019 — privacy of email notifications shown on screens; a user study. |
| IMC 2020 behind-closed-doors-a-network-tale-of-spoofing-intrusion-and-false-dns-security | dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat :: [sentences] CCS 2020 — responsible disclosure to Amazon and Google; platforms. |
| NDSS 2020 broken-metre-attacking-resource-metering-in-evm | a-security-analysis-of-the-facebook-ad-library :: [sentences] IEEE S&P 2020 — 'undisclosed' political ads. |
| NDSS 2020 complex-security-policy-a-longitudinal-analysis-of-deployed-content-security-policies | the-many-kinds-of-creepware-used-for-interpersonal-attacks :: [read] IEEE S&P 2020 — 1,095 apps reported to Google, 813 removed. A bulk report to one platform rather than to operators; quoted on the page as a platform take-down figure. |
| PETS 2020 the-price-is-not-right-comparing-privacy-in-free-and-paid-apps | behind-closed-doors-a-network-tale-of-spoofing-intrusion-and-false-dns-security :: [read] IMC 2020 — 43 resolver administrators reached through the SOA RNAME field, 5 replies (3 from people the authors knew), plus two operators of loopback resolvers; under the floor. The wider outreach is described as planned. Quoted for the SOA-RNAME channel. |
| USENIX 2020 actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an | broken-metre-attacking-resource-metering-in-evm :: [sentences] NDSS 2020 — Ethereum Foundation bounty. |
| USENIX 2020 cached-and-confused-web-cache-deception-in-the-wild | the-price-is-not-right-comparing-privacy-in-free-and-paid-apps :: [sentences] PETS 2020 — privacy-policy disclosure. |
| USENIX 2020 plug-n-pwned-comprehensive-vulnerability-analysis-of-obd-ii-dongles-as-a-new-ove | actions-speak-louder-than-words-entity-sensitive-privacy-policy-and-data-flow-an :: [sentences] USENIX Sec 2020 — policy 'disclosures'. |
| WWW 2020 an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre | cached-and-confused-web-cache-deception-in-the-wild :: [sentences] USENIX Sec 2020 — notification letters to the publicly listed security contacts of 16 sites (93 domains); no delivery or response figures; under the floor. |
| CCS 2021 i-need-a-better-description-an-investigation-into-user-expectations-for-differen | plug-n-pwned-comprehensive-vulnerability-analysis-of-obd-ii-dongles-as-a-new-ove :: [sentences] USENIX Sec 2020 — 29 dongle vendors emailed; vendors, under the floor. |
| CCS 2021 out-of-sight-out-of-mind-detecting-orphaned-web-pages-at-internet-scale (also in REJECTED) | an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre :: [sentences] WWW 2020 — survey respondents. |
| IEEE-SP 2021 sok-quantifying-cyber-risk | i-need-a-better-description-an-investigation-into-user-expectations-for-differen :: [sentences] CCS 2021 — 'information disclosure' in a differential-privacy user study. |
| IMC 2021 who-you-gonna-call-an-empirical-evaluation-of-website-security-txt-deployment (also in REJECTED) | sok-quantifying-cyber-risk :: [sentences] IEEE S&P 2021 — an SoK citing the notification studies already on the page. |
| NDSS 2021 all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers | all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers :: [sentences] NDSS 2021 — WhatsApp, Signal and Telegram. |
| NDSS 2021 shadow-attacks-hiding-and-replacing-content-in-signed-pdfs | shadow-attacks-hiding-and-replacing-content-in-signed-pdfs :: [sentences] NDSS 2021 — 15 PDF vendors via CERT-Bund. |
| PETS 2021 defining-privacy-how-users-interpret-technical-terms-in-privacy-policies | defining-privacy-how-users-interpret-technical-terms-in-privacy-policies :: [sentences] PETS 2021 — survey. |
| PETS 2021 digital-inequality-through-the-lens-of-self-disclosure | digital-inequality-through-the-lens-of-self-disclosure :: [sentences] PETS 2021 — self-disclosure on social media. |
| USENIX 2021 a-large-scale-interview-study-on-information-security-in-and-attacks-against-sma | a-large-scale-interview-study-on-information-security-in-and-attacks-against-sma :: [sentences] USENIX Sec 2021 — CATI recruitment response rate (43,219 companies called). |
| USENIX 2021 injection-attacks-reloaded-tunnelling-malicious-payloads-over-dns | injection-attacks-reloaded-tunnelling-malicious-payloads-over-dns :: [sentences] USENIX Sec 2021 — 'disclose contents of /some/file'. |
| USENIX 2021 now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach | now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach :: [sentences] USENIX Sec 2021 — breach notifications to consumers; a user study. |
| CCS 2022 clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam | clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam :: [sentences] CCS 2022 — 'several attempts' to disclose to the developers of a few text-messaging apps. |
| IEEE-SP 2022 the-state-of-the-samesite-studying-the-usage-effectiveness-and-adequacy-of-sames (also in REJECTED) | timing-based-browsing-privacy-vulnerabilities-via-site-isolation :: [sentences] IEEE S&P 2022 — Chrome and Edge teams. |
| IEEE-SP 2022 timing-based-browsing-privacy-vulnerabilities-via-site-isolation | characterizing-permanently-dead-links-on-wikipedia :: [sentences] IMC 2022 — the Internet Archive informed the authors. |
| IMC 2022 characterizing-permanently-dead-links-on-wikipedia | mind-your-manrs-measuring-the-manrs-ecosystem :: [read] IMC 2022 — six large network operators told about non-conformant announcements; five replied, two fixes verified. Direct disclosure to already-identified parties, under the floor. |
| IMC 2022 mind-your-manrs-measuring-the-manrs-ecosystem | auto-draft-219 :: [sentences] NDSS 2022 (SpiralSpy) — 'system notification' of a fan-status change. |
| NDSS 2022 auto-draft-219 | developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev :: [sentences] PETS 2022 — developers report receiving app-store notifications; a survey. |
| PETS 2022 developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev | exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective :: [sentences] PETS 2022 — notifying bystanders about data practices; a user study. |
| PETS 2022 exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective | we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob :: [sentences] PETS 2022 — vendors of child-care apps. |
| PETS 2022 revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an (also in REJECTED) | movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi :: [sentences] USENIX Sec 2022 — 'disclosed vulnerable functions' from CVE data. |
| PETS 2022 we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob | understanding-the-practice-of-security-patch-management-across-multiple-branches :: [sentences] WWW 2022 — CVE disclosure dates. |
| USENIX 2022 movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi | dont-leak-your-keys-understanding-measuring-and-exploiting-the-appsecret-leaks-i :: [sentences] CCS 2023 — Tencent and Baidu bounties. |
| WWW 2022 measuring-alexa-skill-privacy-practices-across-three-years | passive-ssh-key-compromise-via-lattices :: [sentences] CCS 2023 — four manufacturers and CERT/CC; the authors 'considered notifying operators of affected devices ... but we determined this would be infeasible'. Quoted on the page as a reasoned decision not to run a campaign. |
| WWW 2022 understanding-the-practice-of-security-patch-management-across-multiple-branches | no-one-drinks-from-the-firehose-how-organizations-filter-and-prioritize-vulnerab :: [sentences] IEEE S&P 2023 — interviews on how organisations consume vulnerability notifications; receiving end, no campaign. |
| CCS 2023 dont-leak-your-keys-understanding-measuring-and-exploiting-the-appsecret-leaks-i | no-privacy-in-the-electronics-repair-industry :: [sentences] IEEE S&P 2023 — survey respondents. |
| CCS 2023 passive-ssh-key-compromise-via-lattices | three-birds-with-one-stone-efficient-partitioning-attacks-on-interdependent-cryp :: [sentences] IEEE S&P 2023 — nodes that 'publicly disclose' overlay addresses. |
| IEEE-SP 2023 no-one-drinks-from-the-firehose-how-organizations-filter-and-prioritize-vulnerab | utopia-automatic-generation-of-fuzz-driver-using-unit-tests :: [sentences] IEEE S&P 2023 — 123 bug reports to open-source projects, 70 confirmed; bug reporting to maintainers, not operator notification. |
| IEEE-SP 2023 no-privacy-in-the-electronics-repair-industry | a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda :: [sentences] IMC 2023 — disclosure to Google's CDN. |
| IEEE-SP 2023 three-birds-with-one-stone-efficient-partitioning-attacks-on-interdependent-cryp | rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki :: [sentences] IMC 2023 — reached out to network operators, one response (BIT), plus AT&T and APNIC; under the floor. |
| IEEE-SP 2023 utopia-automatic-generation-of-fuzz-driver-using-unit-tests | wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d :: [sentences] IMC 2023 — DNS hosting providers named in the paper. |
| IMC 2023 a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda | investigating-how-users-imagine-their-personal-privacy-assistant :: [sentences] PETS 2023 — UI notifications. |
| IMC 2023 rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki | lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c :: [sentences] PETS 2023 — CCPA verifiable consumer requests to app developers (69 substantive responses); a rights-request study, not a notification. See practices:legal_enforcement. |
| IMC 2023 the-cve-wayback-machine-measuring-coordinated-disclosure-from-exploits-against-t (also in REJECTED) | on-the-role-and-form-of-personal-information-disclosure-in-cyberbullying-inciden :: [sentences] PETS 2023 — self-disclosure. |
| IMC 2023 wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d | privacy-rarely-considered-exploring-considerations-in-the-adoption-of-third-part :: [sentences] PETS 2023 — 37,000 survey invitations; recruitment, not notification. |
| PETS 2023 investigating-how-users-imagine-their-personal-privacy-assistant | revoked-just-now-users-behaviors-toward-fitness-data-sharing-with-third-party-ap :: [sentences] PETS 2023 — survey. |
| PETS 2023 lessons-in-vcr-repair-compliance-of-android-app-developers-with-the-california-c | a-bugs-life-analyzing-the-lifecycle-and-mitigation-process-of-content-security-p :: [sentences] USENIX Sec 2023 — CSP browser-bug lifecycle from bug trackers. |
| PETS 2023 on-the-role-and-form-of-personal-information-disclosure-in-cyberbullying-inciden | all-of-them-claim-to-be-the-best-multi-perspective-study-of-vpn-users-and-vpn-pr :: [sentences] USENIX Sec 2023 — 15 VPN providers invited to interviews. |
| PETS 2023 privacy-rarely-considered-exploring-considerations-in-the-adoption-of-third-part | arana-discovering-and-characterizing-password-guessing-attacks-in-practice :: [sentences] USENIX Sec 2023 — compromised usernames disclosed to two universities' security teams. |
| PETS 2023 revoked-just-now-users-behaviors-toward-fitness-data-sharing-with-third-party-ap | downgrading-dnssec-how-to-exploit-crypto-agility-for-hijacking-signed-zones :: [sentences] USENIX Sec 2023 — five named DNS vendors and operators. |
| USENIX 2023 a-bugs-life-analyzing-the-lifecycle-and-mitigation-process-of-content-security-p | internet-service-providers-and-individuals-attitudes-barriers-and-incentives-to :: [sentences] USENIX Sec 2023 — ISP survey on barriers to notifying IoT owners under Japan's NOTICE programme; receiving end, no campaign. |
| USENIX 2023 all-of-them-claim-to-be-the-best-multi-perspective-study-of-vpn-users-and-vpn-pr | my-privacy-for-their-security-employees-privacy-perspectives-and-expectations-wh :: [sentences] USENIX Sec 2023 — employee survey. |
| USENIX 2023 arana-discovering-and-characterizing-password-guessing-attacks-in-practice | panda-security-analysis-of-algorand-smart-contracts :: [sentences] USENIX Sec 2023 — one DeFi developer, a bounty. |
| USENIX 2023 are-you-spying-on-me-large-scale-analysis-on-iot-data-exposure-through-companion | pushed-by-accident-a-mixed-methods-study-on-strategies-of-handling-secret-inform :: [sentences] USENIX Sec 2023 — GitHub secret-leak notifications recalled by survey participants. |
| USENIX 2023 downgrading-dnssec-how-to-exploit-crypto-agility-for-hijacking-signed-zones | whip-improving-static-vulnerability-detection-in-web-application-by-forcing-tool :: [sentences] USENIX Sec 2023 — 35 zero-days disclosed to the developers of 14 applications. |
| USENIX 2023 internet-service-providers-and-individuals-attitudes-barriers-and-incentives-to | airgapagent-protecting-privacy-conscious-conversational-agents :: [sentences] CCS 2024 — 'disclosure of extraneous user information' by an LLM agent. |
| USENIX 2023 my-privacy-for-their-security-employees-privacy-perspectives-and-expectations-wh | defying-the-odds-solanas-unexpected-resilience-in-spite-of-the-security-challeng :: [sentences] CCS 2024 — participant recruitment. |
| USENIX 2023 panda-security-analysis-of-algorand-smart-contracts | breach-extraction-attacks-exposing-and-addressing-the-leakage-in-second-generati :: [sentences] IEEE S&P 2024 — Cloudflare. |
| USENIX 2023 pushed-by-accident-a-mixed-methods-study-on-strategies-of-handling-secret-inform | from-virtual-touch-to-tesla-command-unlocking-unauthenticated-control-chains-fro :: [sentences] IEEE S&P 2024 — named stakeholders. |
| USENIX 2023 whip-improving-static-vulnerability-detection-in-web-application-by-forcing-tool | shedding-light-on-cvss-scoring-inconsistencies-a-user-centric-study-on-evaluatin :: [sentences] IEEE S&P 2024 — study participants re-contacted. |
| CCS 2024 airgapagent-protecting-privacy-conscious-conversational-agents | sok-sgx-fail-how-stuff-gets-exposed :: [sentences] IEEE S&P 2024 — SoK; disclosure timing from prior work. |
| CCS 2024 are-we-getting-well-informed-an-in-depth-study-of-runtime-privacy-notice-practic | tabbed-out-subverting-the-android-custom-tab-security-model :: [sentences] IEEE S&P 2024 — Google's Chrome VRP. |
| CCS 2024 defying-the-odds-solanas-unexpected-resilience-in-spite-of-the-security-challeng | dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome :: [sentences] NDSS 2024 — browser notification-permission prompts. |
| IEEE-SP 2024 breach-extraction-attacks-exposing-and-addressing-the-leakage-in-second-generati | understanding-the-implementation-and-security-implications-of-protective-dns-services :: [sentences] NDSS 2024 — DNS vendors. |
| IEEE-SP 2024 from-virtual-touch-to-tesla-command-unlocking-unauthenticated-control-chains-fro | exploring-the-privacy-experiences-of-closeted-users-of-online-dating-services-in :: [sentences] PETS 2024 — survey. |
| IEEE-SP 2024 shedding-light-on-cvss-scoring-inconsistencies-a-user-centric-study-on-evaluatin | security-and-privacy-with-second-hand-storage-devices-a-user-centric-perspective :: [sentences] PETS 2024 — whether buyers would notify anyone about found data; a survey. |
| IEEE-SP 2024 sok-sgx-fail-how-stuff-gets-exposed | the-medium-is-the-message-how-secure-messaging-apps-leak-sensitive-data-to-push :: [sentences] PETS 2024 — 11 app developers, 6 had not replied; under the floor. |
| IEEE-SP 2024 tabbed-out-subverting-the-android-custom-tab-security-model | diffie-hellman-picture-show-key-exchange-stories-from-commercial-vowifi-deployme :: [sentences] USENIX Sec 2024 — MediaTek, the GSMA CVD programme, Apple and Google. |
| IEEE-SP 2024 where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac (also in REJECTED) | digital-discrimination-of-users-in-sanctioned-states-the-case-of-the-cuba-embarg :: [sentences] USENIX Sec 2024 — geoblocking 'disclosures' shown to users. |
| NDSS 2024 dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome | globalconfusion-trustzone-trusted-application-0-days-by-design :: [sentences] USENIX Sec 2024 — TrustZone vendors, four CVEs. |
| NDSS 2024 understanding-the-implementation-and-security-implications-of-protective-dns-services | malla-demystifying-real-world-large-language-model-integrated-malicious-services :: [sentences] USENIX Sec 2024 — OpenAI, Anthropic, Meta, Poe and FlowGPT; platforms. |
| PETS 2024 exploring-the-privacy-experiences-of-closeted-users-of-online-dating-services-in | rethinking-the-security-threats-of-stale-dns-glue-records :: [read] USENIX Sec 2024 — TLD registries, resolver vendors and ICANN; the 6 million affected domains were handed to the national CERT ('await their response'), no outcome reported. |
| PETS 2024 security-and-privacy-with-second-hand-storage-devices-a-user-centric-perspective | spf-beyond-the-standard-management-and-operational-challenges-in-practice-and-pr :: [sentences] USENIX Sec 2024 — one vendor report and an operator survey. |
| PETS 2024 the-medium-is-the-message-how-secure-messaging-apps-leak-sensitive-data-to-push | towards-more-practical-threat-models-in-artificial-intelligence-security :: [sentences] USENIX Sec 2024 — survey respondents. |
| USENIX 2024 diffie-hellman-picture-show-key-exchange-stories-from-commercial-vowifi-deployme | unpacking-privacy-labels-a-measurement-and-developer-perspective-on-googles-data :: [read] USENIX Sec 2024 — the top 10,000 developers per label pattern were emailed questions about their Data Safety labels (3,500 responses, 889 usable): recruitment for a developer study, not notification of a defect. |
| USENIX 2024 digital-discrimination-of-users-in-sanctioned-states-the-case-of-the-cuba-embarg | unfiltered-measuring-cloud-based-email-filtering-bypasses :: [read] WWW 2024 — chose NOT to notify the affected domains, citing Bennett et al.'s finding that 'over 80% of the domains contacted were unresponsive', and worked with seven filtering vendors instead. Quoted on the page as a reasoned decision against a campaign; the 80% is not this paper's measurement. |
| USENIX 2024 globalconfusion-trustzone-trusted-application-0-days-by-design | a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and :: [sentences] CCS 2025 — CVE disclosure dates. |
| USENIX 2024 malla-demystifying-real-world-large-language-model-integrated-malicious-services | is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams :: [sentences] CCS 2025 — Reddit posts. |
| USENIX 2024 rethinking-the-security-threats-of-stale-dns-glue-records | gptracker-a-large-scale-measurement-of-misused-gpts :: [read] IEEE S&P 2025 — two disclosures to OpenAI; 1,316 of 1,804 GPTs reported on 11 September 2024 removed by 25 September. A bulk report to one platform; quoted on the page as a platform take-down figure. |
| USENIX 2024 spf-beyond-the-standard-management-and-operational-challenges-in-practice-and-pr | sniffing-location-privacy-of-video-conference-users-using-free-audio-channels :: [sentences] IEEE S&P 2025 — notification tones. |
| USENIX 2024 towards-more-practical-threat-models-in-artificial-intelligence-security | hitchhiking-vaccine-enhancing-botnet-remediation-with-remote-code-deployment-reuse :: [sentences] NDSS 2025 — remediation payloads that notify infected users; not a campaign by the authors. |
| USENIX 2024 unpacking-privacy-labels-a-measurement-and-developer-perspective-on-googles-data | was-this-you-investigating-the-design-considerations-for-suspicious-login-notifications :: [sentences] NDSS 2025 — login-notification design, interviews. |
| WWW 2024 unfiltered-measuring-cloud-based-email-filtering-bypasses | who-is-trying-to-access-my-account-exploring-user-perceptions-and-reactions-to-risk-based-authentication-notifications :: [sentences] NDSS 2025 — risk-based-authentication notifications, survey. |
| CCS 2025 a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and | ai-is-from-the-devil-behaviors-and-concerns-toward-personal-data-sharing-with-ll :: [sentences] PETS 2025 — survey. |
| CCS 2025 is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams | how-unique-is-whose-web-browser-the-role-of-demographics-in-browser-fingerprinti :: [sentences] PETS 2025 — informed consent of participants. |
| IEEE-SP 2025 beyond-the-horizon-uncovering-hosts-and-services-behind-misconfigured-firewalls (in CAMPAIGNS) | privacy-perceptions-and-behaviors-towards-targeted-advertising-on-social-media-a :: [sentences] PETS 2025 — survey. |
| IEEE-SP 2025 gptracker-a-large-scale-measurement-of-misused-gpts | the-effect-of-platform-policies-on-app-privacy-compliance-a-study-of-child-direc :: [sentences] PETS 2025 — Google Play's own notifications to developers. |
| IEEE-SP 2025 sniffing-location-privacy-of-video-conference-users-using-free-audio-channels | a-mixed-methods-study-of-open-source-software-maintainers-on-vulnerability-manag :: [sentences] USENIX Sec 2025 — recruitment. |
| NDSS 2025 hitchhiking-vaccine-enhancing-botnet-remediation-with-remote-code-deployment-reuse | an-industry-interview-study-of-software-signing-for-supply-chain-security :: [sentences] USENIX Sec 2025 — recruitment (30 candidates). |
| NDSS 2025 was-this-you-investigating-the-design-considerations-for-suspicious-login-notifications | collisionrepair-first-aid-and-automated-patching-for-storage-collision-vulnerabi :: [sentences] USENIX Sec 2025 — a prior tool 'did not disclose runtime overhead'. |
| NDSS 2025 who-is-trying-to-access-my-account-exploring-user-perceptions-and-reactions-to-risk-based-authentication-notifications | exploring-user-security-and-privacy-attitudes-and-concerns-toward-the-use-of-gen :: [sentences] USENIX Sec 2025 — interviews. |
| PETS 2025 ai-is-from-the-devil-behaviors-and-concerns-toward-personal-data-sharing-with-ll | gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical :: [sentences] USENIX Sec 2025 — NVIDIA. |
| PETS 2025 how-unique-is-whose-web-browser-the-role-of-demographics-in-browser-fingerprinti | malicious-llm-based-conversational-ai-makes-users-reveal-personal-information :: [sentences] USENIX Sec 2025 — UI pop-up notifications. |
| PETS 2025 privacy-perceptions-and-behaviors-towards-targeted-advertising-on-social-media-a | misty-registry-an-empirical-study-of-flawed-domain-registry-operation :: [sentences] USENIX Sec 2025 — registries and their backend providers, named. |
| PETS 2025 the-effect-of-platform-policies-on-app-privacy-compliance-a-study-of-child-direc | privacy-law-enforcement-under-centralized-governance-a-qualitative-analysis-of-f :: [sentences] USENIX Sec 2025 — government notifications to app providers, interviews. |
| USENIX 2025 a-mixed-methods-study-of-open-source-software-maintainers-on-vulnerability-manag | regulating-smart-device-support-periods-user-expectations-and-the-european-cyber :: [sentences] USENIX Sec 2025 — survey. |
| USENIX 2025 an-industry-interview-study-of-software-signing-for-supply-chain-security | your-shield-is-my-sword-a-persistent-denial-of-service-attack-via-the-reuse-of-u :: [sentences] USENIX Sec 2025 — DNS vendors. |
| USENIX 2025 collisionrepair-first-aid-and-automated-patching-for-storage-collision-vulnerabi | consumer-beware-exploring-data-brokers-ccpa-compliance :: [sentences] IEEE S&P 2026 — CCPA requests to data brokers (41.2% answered emails where the form was broken); a rights-request study. See practices:legal_enforcement. |
| USENIX 2025 exploring-user-security-and-privacy-attitudes-and-concerns-toward-the-use-of-gen | chameleoscan-demystifying-and-detecting-ios-chameleon-apps-via-llm-powered-ui-exploration :: [sentences] NDSS 2026 — Apple. |
| USENIX 2025 gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical | cross-boundary-mobile-tracking-exploring-java-to-javascript-information-diffusion-in-webviews :: [sentences] NDSS 2026 — 'zip code disclosures' are data flows. |
| USENIX 2025 malicious-llm-based-conversational-ai-makes-users-reveal-personal-information | demystifying-the-access-control-mechanism-of-esxi-vmkernel :: [sentences] NDSS 2026 — VMware bounty. |
| | tickets-to-hide-an-inside-look-into-the-anti-abuse-ecosystem-through-internal-abuse-data :: [read] NDSS 2026 — the receiving end: 1.3 million abuse reports at one hosting provider, 2.6% linked to a client notification. Not a campaign; cited on the page for what happens to a report after it arrives. |
| | time-and-time-again-leveraging-tcp-timestamps-to-improve-remote-timing-attacks :: [sentences] NDSS 2026 — 'reached out to the homepages' is an HTTP request. |
| | unveiling-byovd-threats-malwares-use-and-abuse-of-kernel-drivers :: [sentences] NDSS 2026 — Microsoft and driver vendors. |
| | more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr :: [sentences] PETS 2026 — no notification. |
| | personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t :: [sentences] PETS 2026 — privacy-policy disclosure. |
| | precision-leads-recalling-you-improved-location-privacy-for-shared-mobility-serv :: [sentences] PETS 2026 — dataset maintainers. |
| | the-role-of-online-forums-in-developer-understanding-of-privacy-law-a-reddit-cas :: [sentences] PETS 2026 — GDPR breach notification as a discussion topic. |
| | abuse-risks-are-often-inherent-to-product-features-exploring-ai-vendors-bug-boun :: [read] USENIX Sec 2026 — catalogues AI vendors' disclosure policies (96 vendors have no channel at all); nothing was notified. security.txt was one of its three discovery sources. |
| | inconsistent-incomplete-and-insecure-a-survey-of-account-security-interfaces :: [read] USENIX Sec 2026 — 58 disclosure briefings to a defined set of top-100 services (38 acknowledged, 25 substantive, most 'Not applicable'/'Out of scope'), no fix rate; under the floor. Quoted on the page for its security.txt count (34 of 100) and HackerOne rate limiting. |
| | the-art-of-hide-and-seek-making-pickle-based-model-supply-chain-poisoning-stealt :: [sentences] USENIX Sec 2026 — NVIDIA, Keras, Protect AI; bounty. |
| | bowling-with-chatgpt-on-the-evolving-user-interactions-with-conversational-ai-sy :: [sentences] WWW 2026 — survey. |
| | falconscope-effective-and-efficient-detection-of-hidden-web-interfaces-in-iot-de :: [sentences] WWW 2026 — coordinated disclosure to IoT vendors, 50 CVEs. |
| | unveiling-the-resilience-of-llm-enhanced-search-engines-against-black-hat-seo-ma :: [sentences] WWW 2026 — 1,602 sites reported to Google; a platform report. |
| | |
| | # WIDE PROBE: 78 candidates outside the narrow rule |
| | # PROMOTED (3) |
| | playing-without-paying-detecting-vulnerable-payment-verification-in-native-binar :: USENIX 2022 — "Many of the developers have acknowledged our findings"; no counts, so a volume-only row |
| | lazy-gatekeepers-a-large-scale-study-on-spf-configuration-in-the-wild :: IMC 2023 — 6,931 of 211,018 errors fixed two weeks later — "a success rate of 3.3 %"; 300 thank-you mails, 3 spam complaints put on an opt-out list; bo |
| | demystifying-rpki-invalid-prefixes-hidden-causes-and-security-risks :: NDSS 2026 — 302 of 1,385 undeliverable (21.8%); 174 organisations responded (16.1% of the 1,083 delivered), confirming 349 prefixes as misconfigurations |
| | # FOUND OUTSIDE BOTH RULES (1) |
| | the-security-lottery-measuring-client-side-web-security-inconsistencies :: USENIX 2022 — delivery-failed for 197 of 256 domains; 21 non-automatic answers, 7 confirmed and fixed or explained; second round May 2022: 184 still affec |
| | # REJECTED (75), all screened on the probe-matching sentences only |
| | botmagnifier-locating-spambots-on-the-internet :: bots contacted C&C servers |
| | peerpress-utilizing-enemies-p2p-strength-against-them :: probing accuracy |
| | an-evaluation-of-the-google-chrome-extension-security-architecture :: 32 extension developers contacted, 19 fixed — real but under the floor and to a known set |
| | juice-a-longitudinal-study-of-an-seo-botnet :: IPs contacted a proxy |
| | an-internet-wide-view-of-internet-wide-scanning :: how organisations contacted the scanning team to opt out; receiving end of scanning, see practices:ethics |
| | on-the-practical-exploitability-of-dual-ec-in-tls-implementations :: servers contacted by a scan |
| | security-by-any-other-name-on-the-effectiveness-of-provider-based-email-security :: MTAs contacted by a scan |
| | ad-injection-at-scale-assessing-deceptive-advertisement-modifications :: domains contacted by extensions |
| | copperdroid-automatic-reconstruction-of-android-malware-behaviors :: 'disclosed' malware behaviours |
| | domain-z-28-registrations-later-measuring-the-exploitation-of-residual-trust-in :: domains contacted by malware |
| | recommended-for-you-a-first-look-at-content-recommendation-networks :: websites contacting a CRN |
| | internet-jones-and-the-raiders-of-the-lost-trackers-an-archaeological-study-of-w :: third parties contacted by sites |
| | investigating-commercial-pay-per-install-and-the-distribution-of-unwanted-softwa :: domains contacted by binaries |
| | privacy-in-epigenetics-temporal-linkability-of-microrna-expression-profiles :: 'disclosing' miRNAs |
| | no-honor-among-thieves-a-large-scale-analysis-of-malicious-web-shells :: domains contacted by shells |
| | a-lustrum-of-malware-network-communication-evolution-and-insights :: domains contacted by malware |
| | vuddy-a-scalable-approach-for-vulnerable-code-clone-discovery :: one Red Hat security officer |
| | internet-scale-probing-of-cps-inference-characterization-and-orchestration-analy :: darkspace contacted by a scanning campaign |
| | a-longitudinal-end-to-end-view-of-the-dnssec-ecosystem :: four operators asked for a reason |
| | how-tracking-companies-circumvented-ad-blockers-using-websockets :: receivers contacted by parties |
| | o-peer-where-art-thou-uncovering-remote-peering-interconnections-at-ixps :: IXP operators asked to validate |
| | understanding-tor-usage-with-privacy-preserving-measurement :: Amazon employees; guards contacted by clients |
| | adversarial-perceptual-ad-blocking-meets-adversarial-machine-learning :: ad-disclosure logos |
| | just-the-tip-of-the-iceberg-internet-scale-exploitation-of-routers-for-cryptojac :: asked Censys about its crawl |
| | watching-you-watch-the-tracking-ecosystem-of-over-the-top-tv-streaming-devices :: tracker domains contacted |
| | scanning-the-scanners-sensing-the-internet-from-a-massively-distributed-network :: IPs contacted by scanners |
| | hack-for-hire-exploring-the-emerging-market-for-account-hijacking :: 27 hack-for-hire services contacted as customers |
| | a-systematic-study-of-elastic-objects-in-kernel-exploitation :: 'disclose data from kernel heap' |
| | a-haystack-full-of-needles-scalable-detection-of-iot-devices-in-the-wild :: service IPs contacted by devices |
| | when-push-comes-to-ads-measuring-the-rise-of-malicious-push-advertising :: notification permission requests |
| | angel-or-devil-a-privacy-study-of-mobile-parental-control-apps :: service contacted by apps |
| | nomoats-towards-automatic-detection-of-mobile-tracking :: hostnames contacted |
| | ethbmc-a-bounded-model-checker-for-smart-contracts :: authors of a prior tool contacted for data |
| | dark-matter-uncovering-the-darkcomet-rat-ecosystem :: IRB approval to report 57,805 victims to their ISPs — 'engaging ISPs'; no outcome reported |
| | the-chameleon-attack-manipulating-content-display-in-online-social-media :: Facebook groups contacted as a test |
| | crawlphish-large-scale-analysis-of-client-side-cloaking-techniques-in-phishing :: users allowing notifications |
| | follow-the-scent-defeating-ipv6-prefix-rotation-privacy :: one CPE manufacturer |
| | privacyflash-pro-automating-privacy-policy-generation-for-mobile-apps :: a few policy-generator operators |
| | reining-in-the-webs-inconsistencies-with-site-policy :: disclosure to a few named sites, qualitative feedback |
| | privacy-preference-signals-past-present-and-future :: second-level domains contacted per site |
| | shhh-be-quiet-reducing-the-unwanted-interruptions-of-notification-permission-pro :: notification permission prompts |
| | websocket-adoption-and-the-landscape-of-the-real-time-web :: WebSocket push notifications |
| | measuring-and-mitigating-the-risk-of-ip-reuse-on-public-clouds :: IPs contacted by traffic |
| | deep-dive-into-the-iot-backend-ecosystem :: backend servers contacted |
| | malnet-a-binary-centric-network-level-profiling-of-iot-malware :: C2 servers contacted |
| | where-to-recruit-for-security-development-studies-comparing-six-software-develop :: 76,978 developers contacted for study recruitment |
| | when-free-tier-becomes-free-to-enter-a-non-intrusive-way-to-identify-security-ca :: camera motion notifications |
| | detection-of-inconsistencies-in-privacy-practices-of-browser-extensions :: dashboard disclosures |
| | hope-of-delivery-extracting-user-locations-from-mobile-instant-messengers :: delivery-notification timing |
| | usability-and-enforceability-of-global-privacy-control :: BuiltWith operators contacted with a suggestion |
| | dscope-a-cloud-native-internet-telescope :: scanners contacting telescopes |
| | locin-inferring-semantic-location-from-spatial-maps-in-mixed-reality :: IRB contacted |
| | the-ok-is-not-enough-a-large-scale-study-of-consent-dialogs-in-smartphone-applic :: tracking endpoints contacted by apps |
| | timeless-timing-attacks-and-preload-defenses-in-tors-dns-cache :: the Tor Project |
| | user-awareness-and-behaviors-concerning-encrypted-dns-settings-in-web-browsers :: survey |
| | on-how-zero-knowledge-proof-blockchain-mixers-improve-and-worsen-user-privacy :: authors of a prior paper contacted for data |
| | characterizing-and-mitigating-phishing-attacks-at-cctld-scale :: registry-side suspension time after a Netcraft notification; not the authors' campaign |
| | content-nudges-and-incentives-a-study-on-the-effectiveness-and-perception-of-emb :: recruitment |
| | targeted-and-troublesome-tracking-and-advertising-on-childrens-websites :: ad disclosures |
| | diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc :: domains contacted |
| | the-roots-go-deep-measuring-under-change :: anycast site contacted |
| | content-censorship-in-the-interplanetary-file-system :: peers contacted |
| | like-comment-get-scammed-characterizing-comment-scams-on-media-platforms :: scammers contacted as bait |
| | masterkey-automated-jailbreaking-of-large-language-model-chatbots :: jailbreak disclosure to vendors |
| | why-privacy-preserving-protocols-are-sometimes-not-enough-a-case-study-of-the-br :: 'disclosure attack' |
| | can-virtual-reality-protect-users-from-keystroke-inference-attacks :: 'disclosure of confidential' data |
| | ghostrace-exploiting-and-mitigating-speculative-race-conditions :: NFC driver maintainers, no answer |
| | the-unusual-suspects-studying-reasons-for-lacking-updates-in-wordpress :: 482 webshop owners cold-called to recruit interviewees (287 answered), deliberately not framed as a vulnerability notification |
| | transparency-or-information-overload-evaluating-users-comprehension-and-perceptions-of-the-ios-app-privacy-report :: 'Most Contacted Domains' |
| | empirically-measuring-data-localization-in-the-eu :: servers contacted |
| | lost-in-the-mists-of-time-expirations-in-dns-footprints-of-mobile-apps :: app hostnames |
| | because-i-didnt-touch-these-and-even-dont-know-why-i-should-to-change-these-why :: 5,000 developers contacted for a survey about privacy labels |
| | dead-domains-living-data-a-privacy-risk-analysis-of-domain-lifecycle-in-android :: apps contacting a domain |
| | the-tcf-doesn-t-really-a-a-id-automatic-privacy-analysis-and-legal-compliance-of :: domains contacted with AAID |
| | missing-present-and-conflicting-a-large-scale-analysis-of-iot-update-information :: customer service contacted as a buyer |
| </code> | </code> |
| |
| <wrap todo>The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap> | The kind label ''mobile'' was renamed ''apps'' in ''CAMPAIGNS'' because the Alexa-skill campaign is not mobile; the three 2021–2022 Play Store campaigns moved with it. The printed kinds table changed accordingly. |
| |
| ===== Quotes spot-checked ===== | ===== Quotes spot-checked ===== |
| |
| ==== ''ethics.evidence.quote'' against the source text ==== | ==== ethics.evidence.quote against the source text ==== |
| |
| Seven papers with ''notifiedAffectedParties == yes ∧ platforms ∋ web'', picked deterministically one per year across 2013–2026, checked after de-hyphenating and whitespace-collapsing both the quote and ''paper.cols.txt'': | Seven papers with ''notifiedAffectedParties == yes ∧ platforms ∋ web'', picked deterministically one per year across 2013–2026, checked after de-hyphenating and whitespace-collapsing both the quote and ''paper.cols.txt'': |
| | {[stock2016_hey]} 5.8% received / 74.5% still exploitable / Table 1 fix rates / Abusix / RFC 2142 aliases / legal threat on day two / 5 opt-outs covering 187 domains | read directly by the orchestrator from ''paper.cols.txt''; no sub-agent involved | | | {[stock2016_hey]} 5.8% received / 74.5% still exploitable / Table 1 fix rates / Abusix / RFC 2142 aliases / legal threat on day two / 5 opt-outs covering 187 domains | read directly by the orchestrator from ''paper.cols.txt''; no sub-agent involved | |
| | {[lone2022_sav]} PeeringDB-before-WHOIS contact cascade | read directly by the orchestrator | | | {[lone2022_sav]} PeeringDB-before-WHOIS contact cascade | read directly by the orchestrator | |
| | |
| | ==== Quotes added on 2026-09-04 ==== |
| | |
| | Every quote written onto the content page in this revision, plus the figures behind the new table rows, is in ''out/notify_quotes_20260904.json'' — **69 quotes across 22 papers** — and ''scripts/notify_quotecheck2.mjs'' re-checks them against ''paper.cols.txt'' after normalising both sides (de-hyphenation, straight quotes, non-ASCII dropped, whitespace collapsed). Result on 2026-09-04: **69 EXACT, 0 MISSING** (67 before the figures review added two) (''out/notify_quotecheck2-output.txt''). Five of the 69 are short fragments because the full sentence is column-spliced in ''paper.cols.txt'' and only exists interleaved with another column: the Alexa paper's "111 (45.12%) no longer pose a threat" and "whenever we have their contact details"; the SEISE paper's whole disclosure paragraph (lines 1668–1678 interleave with a Related-Work paragraph about Evilseed); and the ASI paper's "acknowledgement for 38 disclosures". Each was read in place by the orchestrator. Three normaliser bugs were found and fixed before the check passed — the first version dropped newlines with the non-ASCII characters and reported 54 of 67 missing; the ''sed'' repair then deleted the whitespace collapse. Recorded because a quote check that fails for the wrong reason is easy to "fix" by weakening it. The repaired checker was mutation-tested: a fabricated sentence appended to the JSON is reported MISSING and the exit code is non-zero. |
| | |
| | Sub-agent findings **not** carried onto the page, because the orchestrator re-read the passage and judged them inferences rather than the paper's statement: the ROAlert paper's "52 administrators" (assembled across a 58-line column gap; the page says "52 administrators engaged", which is what the two fragments say, but the denominator for the 52 is not stated and is not implied on the page); the ROAlert "~15% of uncontactable operators" as a "control" (the page calls it a comparison group the paper reports, not a randomised control). Internal inconsistencies the sub-agents flagged and the page carries as the paper states them: the Alexa paper's 45 + 24 + 41 = 110 against its own 111 (stated on the page); the CSP paper's discussion-section denominators of 30 against Section IX-B's 31 (the page uses none of the survey percentages); "Tickets to Hide" printing 11.1% (table) and 11.7% (prose) for the same botnet-C&C rate (neither is on the page). |
| |
| **Two errors in the first published version, caught by the orchestrator's own re-check after publication and fixed in revision 2:** | **Two errors in the first published version, caught by the orchestrator's own re-check after publication and fixed in revision 2:** |
| * **CERT/CC Vultron.** Reported as explicitly "not yet ready for production use". Cut, because a non-production protocol is not a channel a PhD student can use this year. | * **CERT/CC Vultron.** Reported as explicitly "not yet ready for production use". Cut, because a non-production protocol is not a channel a PhD student can use this year. |
| * **Bugcrowd / Intigriti / YesWeHack third-party report acceptance.** No primary policy page found either way. The page therefore claims the 2016 "reward programmes only serve their customers" premise no longer holds **for HackerOne specifically**, and says nothing about the others. | * **Bugcrowd / Intigriti / YesWeHack third-party report acceptance.** No primary policy page found either way. The page therefore claims the 2016 "reward programmes only serve their customers" premise no longer holds **for HackerOne specifically**, and says nothing about the others. |
| | |
| | ==== Fetched by the orchestrator, 2026-09-04 (the security.txt questions) ==== |
| | |
| | ^ Claim on the page ^ Source ^ How verified ^ |
| | | 0.49% of the Tranco top 1M and 1.6% of the top 100K had a valid ''security.txt'' in October–November 2021; only 11.5% of files matched the then-current draft; 84.3% offered an email contact; no notification was sent | Findlay & Abdou, MADweb 2022 (NDSS workshop), DOI ''10.14722/madweb.2022.23014'' | PDF fetched from ''ndss-symposium.org'', text extracted with ''pypdf'' (''out/findlay2022_madweb.txt''), figures read in place. ''WebFetch'' could not read the PDF; the author-copy URL at Carleton 404s. The paper's own WHOIS cross-check — 2,477 WHOIS abuse contacts, "not a single WHOIS abuse contact was found to match a Security.txt contact" — is noted here, not on the page | |
| | | 42 successful weekly scans of the Tranco top 1M over 55 weeks, 26 December 2021 to 15 January 2023: top 100 32.0→34.0%, top 1K 16.1→18.8%, top 10K 7.9→9.8%, top 100K 2.4→3.2%, top 1M 0.7→1.0%; Contact field in 89% of files; testing the contacts named as future work | Hilbig, Geras, Kupris, Schreck, //Digital Threats: Research and Practice// 4(3):36, October 2023, DOI ''10.1145/3609234'' | PDF fetched from the authors' lab site (''seclab.cs.hm.edu''), extracted with ''pypdf'' (''out/sectxt_revisited_2023.txt''); Table 2 and Section 6.2 read in place. The list was generated 6 February 2021, a year before the first scan — the paper says so and gives the intersection sizes | |
| | | A gTLD zone-file scan finds 573,123 of 241,285,150 domains (0.238%) with a ''security.txt'' in 2026, from 82,343 of 151,402,656 (0.054%) in 2021 | [[https://blog.iotdef.com/the-state-of-security-txt-adoption-an-analysis-of-240-million-domains-in-2026/|blog.iotdef.com]], 1 February 2026 | Fetched 2026-09-04. Methodology is stated (gTLD zone files only, no ccTLDs, annual scans, DuckDB over a 68 GB JSONL); **data and code are not published**, so the figure cannot be checked. **This reverses the 2026-08-13 decision to reject iotdef as an adoption-statistics blog**, for one reason: it is the only 2025–2026 figure of any kind, and the page carries it in a footnote labelled as an unverifiable vendor scan and an order of magnitude, never as a citation. A reader who wants a citeable current figure is told there is none | |
| | | Six corpus papers that notified used ''security.txt'' as a channel and none separates its delivery | ''drakonakis2020_cookie'' (only 23 domains had one), ''ghasemisharif2022_saat'', ''roth2022_security'' (25 of 256), ''czybik2023_lazy'', ''innocenti2025_only'' (first of six contact-point priorities for 55 brokers), ''bhattacharya2026_asi'' (34 of 100) | all six passages read in ''paper.cols.txt''; the six were found by ''grep -a -l -i 'security\.txt''' over every ''paper.cols.txt'' — 13 files mention it at all, listed in ''out/securitytxt_papers.txt'' | |
| | |
| | ==== Rejected or not used, 2026-09-04 ==== |
| | |
| | * **uriports.com, "Follow-Up on Security.txt Adoption: Progress and Pitfalls in 2025"** (24 January 2025): 0.7% of "the top one million domains" in April 2024, 1.25% in 2025, 44% RFC-compliant. Rejected again: the ranking list is not named, no method, a vendor's validator-marketing blog. Recorded because its numbers will come up in a search. |
| | * **hexiosec.com "1 Million Websites — How Many Use Security.txt?"** and **scotthelme.co.uk Top 1 Million Analysis, June 2026**: surfaced by search; not fetched, because neither was needed once two peer-reviewed measurements existed. The Helme crawl is a general web-security crawl, not a ''security.txt'' study. |
| | * **Vekaria, Nithyanand, Shafiq, "Towards Multi-Stakeholder Vulnerability Notifications in the Ad-Tech Supply Chain"**, arXiv 2406.06958 (v. April 2026): a nine-month notification study using ''ads.txt'' and ''sellers.json'' contacts, with a reported 79.5% delivery-and-opening rate. **Not cited**: the venue could not be established (the arXiv page names none; the corpus index has three other Vekaria papers but not this one), and its figures were only seen in a search-engine summary, which this page does not quote. Worth adding if it appears in a corpus venue — it would be the highest delivery rate on the page and the first campaign through ''ads.txt''. |
| | * **Brandt et al., "Snail Mail Beats Email Any Day" (ARES 2021)**: surfaced by search as a postal-letter notification study on network operators. Outside the seven venues and not fetched; the postal result the page already carries is {[maass2021_effective]}. |
| |
| ==== Rejected sources ==== | ==== Rejected sources ==== |
| ===== What could not be established ===== | ===== What could not be established ===== |
| |
| * **Whether the 22 hand-classified campaigns are the whole set.** They are not; 144 candidates were never read. See the TODO above. | * **Whether the 32 hand-classified campaigns are the whole set.** Nothing is unreviewed, but two of the ten campaigns added on 2026-09-04 were found by a topical grep rather than by the narrow rule; the probe written afterwards catches one of them, so one campaign (''the-security-lottery'') remains outside both committed rules, and 201 of 229 rejections were screened at sentence level. A third rule would find more. See //The recall lesson// above. |
| * **Any response rate from the corpus schema.** ''ethics.disclosureDetail'' carries a countable ratio in **7 of 2,870** cases. Every rate on the content page came from a paper's results section, read by hand. A schema field for "notification outcome, with denominator" would make this page regenerable; there isn't one. | * **Any response rate from the corpus schema.** ''ethics.disclosureDetail'' carries a countable ratio in **7 of 2,870** cases. Every rate on the content page came from a paper's results section, read by hand. A schema field for "notification outcome, with denominator" would make this page regenerable; there isn't one. |
| * **Whether ''security.txt'' contacts actually work.** {[poteat2021_securitytxt]} measured adoption and field validity but never sent anything, and explicitly leaves that to future work. No paper in this corpus tests delivery through ''security.txt''. This is the most obvious open question on the page and it is five years old. | * **Whether ''security.txt'' contacts actually work — narrowed on 2026-09-04, not closed.** Six corpus papers that notified sent to ''security.txt'' contacts (list above; two of them are campaigns in ''CAMPAIGNS'', four are rejected as campaigns), so the channel has been used; none reports delivery or response for it separately, so the per-channel question stands. What the corpus does answer is coverage: 23 domains ({[drakonakis2020_cookie]}), 25 of 256 ({[roth2022_security]}), 34 of the top-100 services ({[bhattacharya2026_asi]}). {[hilbig2023_securitytxt]} states the delivery test as future work in so many words. |
| * **Current ''security.txt'' adoption.** The only corpus figure is from a 2021 paper on a list (Alexa) that was discontinued in 2022. Nothing in the 2022–2026 slice re-measures it. The page therefore gives a 2021 figure and dates it; a reader in 2026 should assume it has risen and has no published number to cite. | * **Current ''security.txt'' adoption — closed to the extent the literature allows.** Still no re-measurement in the seven corpus venues after 2021. Two peer-reviewed measurements outside them ({[findlay2022_securitytxt]}, {[hilbig2023_securitytxt]}) run to January 2023 on Tranco, and the page now carries them; the only 2025–2026 figure is a vendor zone scan (0.24%) that the page footnotes as unverifiable. A reader in September 2026 still has no citeable figure newer than January 2023. |
| * **Whether the framing effect is real.** {[maass2021_effective]} found a large, significant fine-framing effect; {[utz2023_comparing]} found none. Different countries, issues, media and populations. The page presents the contradiction rather than picking a side. Resolving it needs a study that varies framing while holding medium and jurisdiction fixed. | * **Whether the framing effect is real.** {[maass2021_effective]} found a large, significant fine-framing effect; {[utz2023_comparing]} found none. Different countries, issues, media and populations. The page presents the contradiction rather than picking a side. Resolving it needs a study that varies framing while holding medium and jurisdiction fixed. |
| * **Whether the RCT result generalises.** {[lone2022_sav]} found no effect on **network operators deploying SAV** — an expensive, low-visibility configuration change. It does not follow that notification fails for a web vulnerability an operator can fix in ten minutes, and {[maass2021_effective]}'s 56.6%-vs-9.2% with a proper control says it does not. The page says the RCT undermines control-free studies, not that notification does not work. **A reasonable person could read the page as more pessimistic than the evidence supports; this was the hardest calibration call on it.** | * **Whether the RCT result generalises.** {[lone2022_sav]} found no effect on **network operators deploying SAV** — an expensive, low-visibility configuration change. It does not follow that notification fails for a web vulnerability an operator can fix in ten minutes, and {[maass2021_effective]}'s 56.6%-vs-9.2% with a proper control says it does not. The page says the RCT undermines control-free studies, not that notification does not work. **A reasonable person could read the page as more pessimistic than the evidence supports; this was the hardest calibration call on it.** |
| |
| ===== Reviewer findings ===== | ===== Reviewer findings ===== |
| | |
| | <WRAP todo>**2026-09-04 revision: published before its review layer completed, then corrected as the reviews arrived.** Three Sonnet reviewers (figures vs script, citations and quotes, external currency) were launched on the drafts and had not reported when the harness forced publication; all three, and the Fable generic pass, reported afterwards and every finding is logged below with what was done. Every corpus figure in this revision comes from a fresh run of ''scripts/report_notifying_websites.mjs'' (deterministic, re-verified), all 67 added quotes pass ''scripts/notify_quotecheck2.mjs'' (mutation-tested), both pages pass ''check_wrap.mjs'' and ''check_tables.mjs'', and every citekey resolves — but no second pair of eyes has read this revision. The next run should re-run the three focused reviewers and the generic one on the live pages and log their findings here, accepted or rejected. The 2026-08-13 reviewer log below is unchanged. **Update, same day:** the external-currency reviewer reported after publication; its findings are logged directly below. The figures reviewer and a verification sub-agent of the citations reviewer reported next (both logged below); the citations reviewer's own summary had not arrived when this was written.</WRAP> |
| | |
| | ==== Fable — generic, 2026-09-04 revision ==== |
| | |
| | Run after the three focused passes and their fixes. Seventeen findings; fourteen accepted, three partly. |
| | |
| | ^ Finding ^ Action ^ |
| | | The lead box said "both randomised, control-arm experiments on network operators … found no significant effect", contradicted by two rows of the page's own table: {[li2016_youve]} and {[durumeric2014_heartbleed]} randomised network-operator contacts and found effects. "The one sentence they will remember." | **ACCEPTED.** Rewritten to the two most recent RCTs on SAV and ROV, with the distinction (expensive, low-visibility configuration changes) and the earlier positive RCTs named. | |
| | | "The one campaign in the corpus at the 100,000 scale" — Utz 2023 (159,035), Durumeric 2014 (150,000) and Li 2016 (760,935) are larger. | **ACCEPTED**, now "the largest fully automated email campaign since {[utz2023_comparing]}". | |
| | | The new {[blechschmidt2023_hello]} row omitted the paper's own funnel (4,484 sent, ≥2,700 bounced, 26 human replies) and its comparison group (bounced domains fell 20.7% vs 39.7%), and marked the control column "—". | **ACCEPTED**, figures verified in ''paper.cols.txt'' lines 943–1001 and filled in, with the paper's own "cannot conclude a causal relation". | |
| | | The rates table's Outcome column mixes remediation, response, delivery and platform removal, and the intro named only the two channel-only exceptions. | **ACCEPTED in part.** One sentence now names the four kinds with example rows. Per-cell tags were not added — each cell already states what it measures. | |
| | | The ''security.txt'' channel row is a section in a table cell and says the corpus counts "agree" although Poteat 2021 and Hilbig 2023 differ by about 2× at the same ranks. | **ACCEPTED in part.** The contradiction is now stated (Alexa vs Tranco; Kaplan–Meier estimate vs snapshots; corpus counts sit with the Tranco series). The cell was **not** moved into its own subsection — a structural change deferred to the next revision, recorded here. | |
| | | "Six campaigns in the corpus used it": four of the six are rejected as campaigns. "23 of the vulnerable domains" has no denominator. | **ACCEPTED**, both pages now say "six corpus papers that notified", and "23 domains (no denominator given)". | |
| | | The script-output caption compared five giants' 80% with Poteat's 3–4% at the top 10K as "an order of magnitude", while the page now carries 32–34% for the top 100. | **ACCEPTED**, compares with both tiers. | |
| | | "Three papers look at what happens after a report arrives" cites four, two of them about disclosure programmes, and omits {[stivala2026_behind]}. | **ACCEPTED**, count fixed and Stivala named; the three-point restructuring was not done. | |
| | | "seven vendors instead of thousands of domains" (Rao's sample is 1,577 domains, 1,262 misconfigured); ">80% unresponsive" in quotation marks is not verbatim; Ryan "recovered SSH host keys from millions of devices" (189 keys from a scan of millions). No cross-link from the earlier "deciding not to notify is defensible" paragraph. | **ACCEPTED**, all four fixed; figures verified in the papers (Rao line 473; Ryan lines 108–109, 944–945). | |
| | | Fix-difficulty row: "the message was identical" overstates — the paper says "fixed template", and its explanation is "We assume that these are often non-trivial to fix"; no control, so churn may differ by error type. | **ACCEPTED**, reworded with the paper's own hedge. | |
| | | Methodology bullet bolded the reassuring half ("Nothing is unreviewed") and described one screening method for two populations that were screened differently. | **ACCEPTED**: the caveat is bolded and the count split 126 / 75 / 28; the provenance paragraph "How they were read" now says the 78 probe candidates were screened on the probe sentences only. | |
| | | The provenance classification code block still said 12 / 9 for weak-keys after the fold was corrected to 3 / 18 — it had been edited, not regenerated. | **ACCEPTED**, regenerated from ''notify_fold.mjs''. The listing's header for the probe rejections now also says "screened on the probe-matching sentences only". | |
| | | "a recall of 28/32 against the union of everything found" is not recall; no gold set exists. | **ACCEPTED**, reworded: true recall unmeasured, the miss count a lower bound. | |
| | | The "What could not be established" bullet contradicted itself about how many campaigns lie outside the committed rules. | **ACCEPTED**, now: two found by grep, the probe catches one of them, one remains outside both rules. | |
| | | The script's sanity line labelled ''the-security-lottery'' as "added from a wider first-pass scan" although the fold records ''caught: 'grep'''. | **ACCEPTED**, string fixed in ''report_notifying_websites.mjs'', outputs regenerated and re-pasted above. | |
| | | "Two of the four figures in disclosureDetail ratios" — seven are listed (pre-existing). The review WRAP note was stale. | **ACCEPTED**, both fixed. | |
| | | Verified and not a problem: the Roth 2020, Nan 2023, Li 2024, Gilad 2017, Liao 2016 and Edu 2022 rows; Ryan's quote; all "Tickets to Hide" figures; Czybik's 5.73% / 1.60% and 111,951; the live render (25 headings, 6 tables, 0 raw citekeys). Verdict: the page answers its question for its reader once the lead box is fixed. | No action. | |
| | |
| | ==== Sonnet — figures against the script, 2026-09-04 revision ==== |
| | |
| | Re-ran both script invocations (byte-identical to the committed outputs), checked every number in sections A–D, the trend table, the count reconciliation (32 = 26 + 6; 179 = 28 + 151; 78 = 3 + 75; 229 = 154 + 75; 201 sentence-level; 13/13/6), all rates-table rows and the new prose against ''paper.cols.txt'', ran the quote checker and ''find_contacts.py''. Roughly 150 figures; five findings. |
| | |
| | ^ Finding ^ Action ^ |
| | | ''extended-hell-o…'' (USENIX Sec 2023) was listed among the campaigns "not on the table because they report no remediation figure", yet ''CAMPAIGNS'' itself records its "decrease of 39.7%". A carry-over from 2026-08-13. | **ACCEPTED.** Added as a table row (quote re-checked, bibliography entry generated from the index with authors from usenix.org); the not-shown count is now six and the table has 27 rows. | |
| | | "61 device vendors — 5 issued a public advisory, 11 answered privately, …": the split is Table 2's 37 RSA-key vendors, not all 61. The fold entry had it right; the page prose dropped the qualifier. | **ACCEPTED**, prose now says "of the 37 with weak RSA keys". | |
| | | The quote JSON carried a quote from ''towards-automated-auditing…'' (gifyu.com) that supports no sentence on the page. | **ACCEPTED in part.** The paper //is// cited (''ghasemisharif2022_saat'', in the security.txt row's list of campaigns that used the file); the quote that supports that claim ("This included collecting websites' security.txt files") was missing from the JSON and is now in it. The gifyu quote stays as the record of why the paper was rejected as a campaign. | |
| | | ''find_contacts.py'' on the page differs from ''notes/find_contacts.py'' in nine lines: ASCII ''--'' on the page against em-dashes in the file. | **REJECTED as a defect** — deliberate and already recorded in the 2026-08-13 log ("a transcribed copy with em-dashes substituted for ASCII"); the reviewer re-ran the downloaded code and it works. | |
| | | Only the NDSS 2026 row carried the provisional asterisk although the legend says "2025–2026". | **ACCEPTED**, the two 2025 rows are now starred too. | |
| | |
| | ==== Sonnet — citations and quotes, 2026-09-04 revision (verification sub-agent) ==== |
| | |
| | Checked six load-bearing attributions against the source text. Five matched: the "Tickets to Hide" figures (2.6%; 72% Netcraft; Spamhaus 60%, 83% and 84% across three categories, which the page brackets as 60–84%; 126,085 → 411 → 4; 30,467 → none); the ROAlert "about 15%" as an observational comparison group, not a randomised arm; every ''bhattacharya2026_asi'' figure and quote; the Hilbig et al. table and its future-work sentence; the Findlay & Abdou 0.49% / 1.6% and dates. Two findings: |
| | |
| | ^ Finding ^ Action ^ |
| | | **Weak-keys Table 2 was mis-reconstructed.** The flattened four-column layout reads 5 public advisory / 11 private response / **3 auto-response / 18 no response**, not 12 / 9. The reviewer re-extracted the PDF in layout mode and cross-checked the caption's "About half of the vendors acknowledged receipt" (5 + 11 + 3 = 19 of 37). | **ACCEPTED.** The batch-4 sub-agent's reconstruction (which it had itself flagged as a reconstruction) was wrong on the second block; the page, the fold entry and this log now say 3 / 18. The 2012 population is 61 vendors notified, 37 of them about RSA keys; the page says both. | |
| | | "55 weekly scans": the paper says 55 weeks with **42 successful scans**. | **ACCEPTED**, reworded on both pages. | |
| | |
| | ==== Sonnet — external currency, 2026-09-04 revision ==== |
| | |
| | ^ Finding ^ Action ^ |
| | | iotdef blog: URL live, every number the provenance table quotes (573,123 / 241,285,150; 82,343 / 151,402,656; DuckDB, zone files, 68 GB, dated 2026-02-01) present in the page body; no data or code published. | No action — confirms the footnote's labelling as an unverifiable vendor scan. | |
| | | DOI ''10.1145/3609234'' resolves (Crossref: DTRAP vol. 4 issue 3, published 2023-09-30); DOI ''10.14722/madweb.2022.23014'' resolves to the MADweb PDF, authors confirmed. | No action. | |
| | | RFC 9116 unchanged, three errata; erratum 7264 (Reported) concerns the case of the ISO 8601 example in the ''Expires'' field, so the page's "none touching the field list" is loose but not wrong. | **NOTED, no change**: the field names the script parses are unaffected. | |
| | | RFC 2142, ICANN RDAP announcement, IANA bootstrap (publication 2026-07-23, 590 services), RIPEstat (build v0.11.13-2026.09.03), HackerOne Disclosure Assistance (11 June 2024), Shadowserver (201 CSIRTs, 175 countries), CERT/CC 45 days, Project Zero 90+30 and the blogspot redirect, all five venue CfP quotes (usenix.org fetched with a browser User-Agent, no 403): verbatim as the page states. | No action. | |
| | | EUR-Lex returned HTTP 202 with an empty body to every automated fetch on 2026-09-04; the NIS2 Article 12 sentence was confirmed only against three secondary legal mirrors. | **NOTED**: the 2026-08-13 primary fetch stands; the page's quote is unchanged. Re-fetch the primary when EUR-Lex answers. | |
| | | Abusix, PeeringDB API, Tranco, Search Console security-issue messages, 360 Store and APKPure: none renamed or discontinued. | No action. | |
| | | A newer peer-reviewed ''security.txt'' data point exists: "Do (Not) Tell Me About My Insecurities: Assessing the Status Quo of Coordinated Vulnerability Disclosure in Germany Amid New EU Cybersecurity Regulations", EuroUSEC 2025, DOI ''10.1109/EuroUSEC69254.2025.00020'' — 10 of the 40 DAX companies had a ''security.txt'' in 2023, with 25 files added by January 2025. | **NOT ADDED to the content page** in this revision: the orchestrator did not open the paper, and a 40-company national blue-chip sample does not answer the page's question about the top million. Recorded here as the first thing the next run should read; it would be the only post-January-2023 peer-reviewed figure and fits the page's NIS2 framing. The reviewer found no 2024–2026 web-scale re-measurement anywhere, so the page's "no citeable figure newer than January 2023" stands. | |
| |
| Four reviewers, all told the context might not be exhaustive, all given the page text, the report script, its output, and these notes. Three focused passes ran in parallel first; the generic pass ran after their findings were applied. | Four reviewers, all told the context might not be exhaustive, all given the page text, the report script, its output, and these notes. Three focused passes ran in parallel first; the generic pass ran after their findings were applied. |
| |
| * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. | * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. |
| * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the page's ''<file>'' block is a hand-transcribed copy with em-dashes substituted, and a substitution inside a string literal would have broken it silently. | * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the code block on the page is a transcribed copy with em-dashes substituted for ASCII, and a substitution inside a string literal would have broken it silently. |
| | * **A stale page cache made most citations render as raw keys, and it survived four reviewers and every verification pass above.** After the bibliography was extended and the pages saved, ''https://measuretheweb.org/practices/notifying_websites'' rendered ''[lone2022_sav]'', ''[utz2023_comparing]'', ''[stock2018_didnt]'' and most others as literal bracketed keys, while the two pre-existing keys resolved normally. Nothing was wrong with the source, the bibliography or the keys — DokuWiki was serving a cached render made before the bibliography grew. Fixed by requesting each page once with ''&purge=true''; both then rendered 218 and 98 resolved citations with zero raw keys left. **Add a purge to the end of any run that touches ''literature:bibliography'', and re-check the rendered page after purging rather than before.** The earlier checks in this section counted ''bibtex_citekey'' markers and found 38 on a page with ~60 citations, which should have been the tell and was not. |
| * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected. | * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected. |
| |
| |
| The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. | The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. |
| | |
| | ==== Fable — generic, no checklist, run after the three focused passes ==== |
| | |
| | The most productive of the four. Eleven findings, **all accepted**; five were wrong or self-contradicting claims on the content page. |
| | |
| | ^ Finding ^ Action ^ |
| | | The lead box said "both of the **properly randomised**, control-arm experiments … found no significant effect", which implies {[maass2021_effective]} was not properly randomised. It is: "a covert randomized controlled notification experiment", "full factorial design", with a control group — and it is the strongest positive result on the page. The real distinction is population, not rigour. | **ACCEPTED**, verified against the paper, rewritten to "both randomised, control-arm experiments **on network operators**", and the Maass sentence now names it as a randomised controlled experiment rather than "which also had a control group". This was the single most prominent sentence on the page. | |
| | | The rescoping justification claimed the excluded group "is participant-debriefing notes". Reviewer checked: only 273 of 710. | **ACCEPTED**, verified, and the reason replaced with the correct one — the channel question is ill-posed for a paper that did not notify — with the 38.5% in a footnote. | |
| | | "scanning papers … are **less than half as likely** to leave the question unanswered" is contradicted by the page's own table: 24.0% vs 31.1%. The column that //is// less than half is ''no'' (4.2% vs 9.4%). | **ACCEPTED**, both comparisons now stated with their figures. | |
| | | The opening said "you **have to tell** those 8,000 operators, and **every venue** … expects you to say so" — contradicting the page's own PoPETs bullet, and its own paragraph saying that deciding not to notify is defensible. The footnote also covered only four of the seven corpus venues. | **ACCEPTED**: "decide what to do about", "the major venues", and the footnote now names PoPETs' absence of a requirement and says NDSS and TheWebConf are not addressed. | |
| | | {[stock2018_didnt]}'s 74.4% / 33.3% was labelled "among those who **opened** the report" in two places. It is among those who **viewed** it — and the page's own funnel insists opened and viewed are distinct stages. | **ACCEPTED**, both corrected to //viewed//. | |
| | | 22 + 16 + 144 = 182 ≠ 179 on the content page, with the explanation only here. | **ACCEPTED**: the page now says 13 reasoned rejections inside the regex plus three carried over from the wider first-pass scan, totalling 16. The 5,859-records / 5,869-files gap is also now explained in the same sentence. | |
| | | The rates table's intro says it shows campaigns "that reported a quotable outcome", but the {[munteanu2025_catch22]} row has no figure — the stated exclusion criterion. | **ACCEPTED**, the exception is now stated on the content page, not only here. | |
| | | "**Three** of those channels are machine-queryable" is wrong twice: RDAP is not one of the channels in the table, and PeeringDB is machine-queryable but is not in the script. | **ACCEPTED**, rewritten to two per-domain channels plus RDAP, with PeeringDB noted as per-AS. | |
| | | The provenance itself conceded that neither the script docstring nor the page says why RIPEstat was chosen over Abusix, and left it unfixed. | **ACCEPTED**, the docstring now says so — Abusix is better at scale, RIPEstat is what runs with only the standard library and is therefore testable here. | |
| | | This provenance page said "Four reviewers … logged below" with three logged, and "**Two** judgement calls" followed by three bullets. The ''head -143'' / ''tail -n +144'' instruction had gone stale when the script grew. | **ACCEPTED**, all three fixed; the split point is now derived rather than hard-coded, since it moves every time a table row is added. | |
| | | Voice: the only first-person singular on the site ("no cadence **I** could verify"); //survival// used without saying lower is better; "That is not negligence" asserts a motive the data does not show; "The one flat line" for a series that rose tenfold then declined. | **ACCEPTED**, all four reworded. | |
| | |
| | Also confirmed by this pass and not changed: the page answers its own question and the top-to-bottom ordering works; no dead-weight sections; all DokuWiki constructs well-formed in the current revision; the rates table's 17 rows reconcile with this page's accounting; every corpus figure matches a fresh script run; the embedded script runs as documented; and the voice, while denser in bolding and aphorism than the older hand-written pages, sits inside the range of the site's newer corpus-backed pages. |
| |
| ==== Nothing was rejected outright ==== | ==== Nothing was rejected outright ==== |
| |
| Every substantive finding across the three passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page: | Every substantive finding across all four passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page: |
| |
| * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading. | * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading. |
| |
| [[practices:notifying_websites|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[practices:notifying_websites|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| |
