| Next revision | Previous revision |
| provenance:practices:notifying_websites [2026/08/13 19:47] – New provenance page for practices:notifying_websites: queries with denominators, report script output, folds and residue, quote spot-checks, external verification and rejections. Authored by Claude karel.kubicek.claude | provenance:practices:notifying_websites [2026/08/13 20:16] (current) – Record the stale-page-cache bug that made most citations render as raw keys, and the purge that fixed it. Authored by Claude karel.kubicek.claude |
|---|
| |
| Not prose. Read it if you are checking a number. | Not prose. Read it if you are checking a number. |
| | |
| | <wrap todo>House convention on ''provenance:'' pages is mixed and this page picked the more traceable end of it. ''provenance:privacy:fingerprinting'' and ''provenance:design:ip_classification'' use no inline citation markers at all and name papers by slug; ''provenance:privacy:requests'' uses citekeys and a bibliography block, as this page does. Two of the four carry a footer back-link, so this one does too. **No ''~~DISCUSSION~~'' block**, matching all four: comments belong on the content page.</wrap> |
| |
| ===== The run ===== | ===== The run ===== |
| ===== Scope decision ===== | ===== Scope decision ===== |
| |
| The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Two judgement calls: | The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Three judgement calls: |
| |
| * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. | * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. |
| | network scan ∧ ethics | ''studyTypes ∋ network-scan-or-probe ∧ ethics != null'' | 858 | | | network scan ∧ ethics | ''studyTypes ∋ network-scan-or-probe ∧ ethics != null'' | 858 | |
| | legal ∧ ethics | ''empirical ∧ legal.length > 0 ∧ ethics != null'' | 376 | | | legal ∧ ethics | ''empirical ∧ legal.length > 0 ∧ ethics != null'' | 376 | |
| | | **crawled ∧ legal ∧ ethics** | ''crawled ∧ legal.length > 0 ∧ ethics != null'' | **123** | |
| | human subjects ∧ ethics | ''empirical ∧ participants.length > 0 ∧ ethics != null'' | 1,283 | | | human subjects ∧ ethics | ''empirical ∧ participants.length > 0 ∧ ethics != null'' | 1,283 | |
| |
| |
| Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. | Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. |
| | |
| | <wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run followed by the ''--list'' run, which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows a few hundred spurious extra lines, and this misled one reviewer. Do not hard-code the split point — it moves whenever a table row is added. Derive it: ''BASE=$(node scripts/report_notifying_websites.mjs | wc -l)'', then ''head -$BASE'' and ''tail -n +$((BASE+1))'' each match a fresh run byte-for-byte. As of this revision ''BASE'' is 161. The script itself is deterministic — verified.</wrap> |
| |
| Sections of the script, so a figure on the content page can be traced to the block that produced it: | Sections of the script, so a figure on the content page can be traced to the block that produced it: |
| |
| -- the same field, per population -- | -- the same field, per population -- |
| Population (all ∧ has an ethics record) N yes partial no not-applicable not-stated | Population (all ∧ has an ethics record) N yes partial no not-applicable not-stated |
| --------------------------------------- ---- ----- ------- ---- -------------- ---------- | --------------------------------------- ---- ----- ------- ----- -------------- ---------- |
| empirical 4472 36.6% 11.7% 4.3% 22.4% 25.0% | empirical 4472 36.6% 11.7% 4.3% 22.4% 25.0% |
| empirical ∧ measured the web platform 1362 33.2% 11.7% 7.0% 19.5% 28.6% | empirical ∧ measured the web platform 1362 33.2% 11.7% 7.0% 19.5% 28.6% |
| ran an automated web crawl 992 29.8% 13.7% 9.4% 15.9% 31.1% | ran an automated web crawl 992 29.8% 13.7% 9.4% 15.9% 31.1% |
| ran a network scan or probe 858 37.6% 23.4% 4.2% 10.7% 24.0% | ran a network scan or probe 858 37.6% 23.4% 4.2% 10.7% 24.0% |
| assessed compliance with a law 376 42.0% 14.6% 9.8% 21.5% 12.0% | assessed compliance with a law 376 42.0% 14.6% 9.8% 21.5% 12.0% |
| recruited human participants 1283 39.7% 7.6% 1.9% 35.3% 15.4% | crawled ∧ assessed a law 123 35.0% 14.6% 14.6% 15.4% 20.3% |
| | recruited human participants 1283 39.7% 7.6% 1.9% 35.3% 15.4% |
| |
| ========================================================================== | ========================================================================== |
| |
| of the 376 that assessed a law: yes 27 (7.2%) | of the 376 that assessed a law: yes 27 (7.2%) |
| | |
| | of the 2160 that notified an affected party, 135 also contacted a regulator or CERT (6.3%) |
| | of the 147 that contacted a regulator or CERT, 135 ALSO notified the affected party directly (91.8%) — the intermediary is an addition, not a substitute |
| |
| -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- | -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- |
| ethics board or legal review 25 0.8% | ethics board or legal review 25 0.8% |
| unmapped residue: 1069 of 3186 (33.6%) | unmapped residue: 1069 of 3186 (33.6%) |
| | (run with --residue to print all of it) |
| |
| ========================================================================== | ========================================================================== |
| the notable thing. These are rankings of an under-reported field, not adoption rates. | the notable thing. These are rankings of an under-reported field, not adoption rates. |
| |
| -- channel named in disclosureDetail, of 2870 papers that gave any -- | -- channel named in disclosureDetail, of 2870 papers that gave any disclosure detail -- |
| Channel family Papers Share of 2870 | Channel family Papers Share of 2870 |
| ----------------------------------------------------------------------------- ------ ------------- | ----------------------------------------------------------------------------- ------ ------------- |
| a named large platform or app store (Google, Apple, Meta, Microsoft, Amazon…) 335 11.7% | a named large platform or app store (Google, Apple, Meta, Microsoft, Amazon…) 335 11.7% |
| direct email to the operator or developer 113 3.9% | direct email to the operator or developer 113 3.9% |
| CERT / CSIRT / national coordinator 39 1.4% | CERT / CSIRT / national coordinator 39 1.4% |
| hosting provider / ISP / upstream 38 1.3% | hosting provider / ISP / upstream 38 1.3% |
| bug bounty / vulnerability reward programme 36 1.3% | bug bounty / vulnerability reward programme 36 1.3% |
| WHOIS / registry contact 18 0.6% | WHOIS / registry contact 18 0.6% |
| data protection authority / regulator 17 0.6% | data protection authority / regulator 17 0.6% |
| standards body / working group 13 0.5% | standards body / working group 13 0.5% |
| generic contact form or phone 10 0.3% | generic contact form or phone 10 0.3% |
| no channel named at all: 2303 of 2870 (80.2%) — the single largest group, and the finding | no channel named at all: 2303 of 2870 (80.2%) |
| | |
| | -- channel named in disclosureDetail, of 2160 papers that notified (yes|partial) AND gave a detail — the population where "which channel?" is a fair question -- |
| | Channel family Papers Share of 2160 |
| | ----------------------------------------------------------------------------- ------ ------------- |
| | a named large platform or app store (Google, Apple, Meta, Microsoft, Amazon…) 318 14.7% |
| | direct email to the operator or developer 99 4.6% |
| | CERT / CSIRT / national coordinator 39 1.8% |
| | bug bounty / vulnerability reward programme 35 1.6% |
| | hosting provider / ISP / upstream 35 1.6% |
| | WHOIS / registry contact 15 0.7% |
| | data protection authority / regulator 14 0.6% |
| | standards body / working group 13 0.6% |
| | generic contact form or phone 9 0.4% |
| | no channel named at all: 1633 of 2160 (75.6%) |
| |
| -- outcome named in disclosureDetail, of 2870 -- | -- outcome named in disclosureDetail, of 2870 -- |
| 2022–2024 8 | 2022–2024 8 |
| 2025–2026* 5 | 2025–2026* 5 |
| |
| What was notified about Campaigns | What was notified about Campaigns |
| ----------------------- --------- | ----------------------- --------- |
| Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates. | Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates. |
| |
| * The residue is the headline: **2,303 of 2,870 (80.2%) match no channel family.** That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. Published on the content page as the finding. | * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. |
| | * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. |
| | * **Two claims on the content page about this fold were checked directly against the JSONL rather than taken from the script.** (a) "all 2,160 that said they notified are among them": of the 2,160 ''yes''/''partial'' papers, **0** lack a ''disclosureDetail''. (b) The scoping footnote's 38.5%: of the 710 papers that gave a detail without saying they notified, 273 match a participant/consent/debrief/interview/survey/recruit/volunteer term. That second figure exists because a reviewer challenged an earlier, stronger claim that the excluded group //was// participant-debriefing notes — it is not, and the page now says only that the channel question is ill-posed for that group. |
| * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** | * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** |
| * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. | * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. |
| | ''revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an'' | PETS 2022 — 40 organisations re-tested after disclosure. Under the ≥100-party floor; quoted in prose only. | | | ''revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an'' | PETS 2022 — 40 organisations re-tested after disclosure. Under the ≥100-party floor; quoted in prose only. | |
| |
| **The seven campaigns not on the content page's rates table**, because they report a volume sent but no response or remediation figure: ''freely-given-consent…'' (CCS 2022, 1,127 developers), ''extended-hell-o…'' (USENIX Sec 2023), ''dancer-in-the-dark…'' (USENIX Sec 2024, 19% response — this one //is// quoted in prose), ''beyond-exploit-scanning…'' (USENIX Sec 2025, 4,983 emails), ''hey-your-secrets-leaked…'' (IEEE S&P 2025, 3,906 emails), ''beyond-the-horizon…'' (IEEE S&P 2025, 15,837 ASes), and ''catch-22…'' (USENIX Sec 2025, on the table for its channel not its rate). | **The six campaigns not on the content page's rates table**, because they report a volume sent but no response or remediation figure: ''freely-given-consent…'' (CCS 2022, 1,127 developers), ''extended-hell-o…'' (USENIX Sec 2023), ''dancer-in-the-dark…'' (USENIX Sec 2024, 19% response — this one //is// quoted in prose), ''beyond-exploit-scanning…'' (USENIX Sec 2025, 4,983 emails), ''hey-your-secrets-leaked…'' (IEEE S&P 2025, 3,906 emails), and ''beyond-the-horizon…'' (IEEE S&P 2025, 15,837 ASes). |
| | |
| | So the rates table has **17 rows: 16 of the 22 hand-classified campaigns, plus {[stock2018_didnt]}**, which is not in ''CAMPAIGNS'' because its full text is not in the corpus (see below) and the fold only classifies papers the scan could read. ''catch-22…'' is on the table for its channel, not for a rate. |
| |
| ==== The 144 unreviewed candidates, in full ==== | ==== The 144 unreviewed candidates, in full ==== |
| USENIX 2013 an-empirical-study-of-vulnerability-rewards-programs | USENIX 2013 an-empirical-study-of-vulnerability-rewards-programs |
| WWW 2013 two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure | WWW 2013 two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure |
| CCS 2013 rethinking-ssl-development-in-an-appified-world (also in REJECTED) | |
| USENIX 2014 a-look-at-targeted-attacks-through-the-lense-of-an-ngo | USENIX 2014 a-look-at-targeted-attacks-through-the-lense-of-an-ngo |
| USENIX 2014 brahmastra-driving-apps-to-test-the-security-of-third-party-components | USENIX 2014 brahmastra-driving-apps-to-test-the-security-of-third-party-components |
| CCS 2015 an-empirical-study-of-web-vulnerability-discovery-ecosystems (also in REJECTED) | |
| IEEE-SP 2015 the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p | IEEE-SP 2015 the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p |
| NDSS 2015 checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo | NDSS 2015 checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo |
| IMC 2016 weak-keys-remain-widespread-in-network-devices | IMC 2016 weak-keys-remain-widespread-in-network-devices |
| PETS 2016 towards-mining-latent-client-identifiers-from-network-traffic | PETS 2016 towards-mining-latent-client-identifiers-from-network-traffic |
| CCS 2017 a-large-scale-empirical-study-of-security-patches (also in REJECTED) | |
| IMC 2017 if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to | IMC 2017 if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to |
| NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security | NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security |
| USENIX 2018 wpse-fortifying-web-protocols-via-browser-side-security-monitoring | USENIX 2018 wpse-fortifying-web-protocols-via-browser-side-security-monitoring |
| CCS 2019 a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right | CCS 2019 a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right |
| CCS 2019 network-hygiene-incentives-and-regulation-deployment-of-source-address-validatio (also in REJECTED) | |
| NDSS 2019 automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries | NDSS 2019 automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries |
| NDSS 2019 digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system | NDSS 2019 digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system |
| WWW 2019 studying-preferences-and-concerns-about-information-disclosure-in-email-notifica | WWW 2019 studying-preferences-and-concerns-about-information-disclosure-in-email-notifica |
| CCS 2020 dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat | CCS 2020 dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat |
| CCS 2020 the-cookie-hunter-automated-black-box-auditing-for-web-authentication-and-author (also in REJECTED) | |
| IEEE-SP 2020 a-security-analysis-of-the-facebook-ad-library | IEEE-SP 2020 a-security-analysis-of-the-facebook-ad-library |
| IEEE-SP 2020 the-many-kinds-of-creepware-used-for-interpersonal-attacks | IEEE-SP 2020 the-many-kinds-of-creepware-used-for-interpersonal-attacks |
| WWW 2020 an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre | WWW 2020 an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre |
| CCS 2021 i-need-a-better-description-an-investigation-into-user-expectations-for-differen | CCS 2021 i-need-a-better-description-an-investigation-into-user-expectations-for-differen |
| CCS 2021 out-of-sight-out-of-mind-detecting-orphaned-web-pages-at-internet-scale (also in REJECTED) | |
| IEEE-SP 2021 sok-quantifying-cyber-risk | IEEE-SP 2021 sok-quantifying-cyber-risk |
| IMC 2021 who-you-gonna-call-an-empirical-evaluation-of-website-security-txt-deployment (also in REJECTED) | |
| NDSS 2021 all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers | NDSS 2021 all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers |
| NDSS 2021 shadow-attacks-hiding-and-replacing-content-in-signed-pdfs | NDSS 2021 shadow-attacks-hiding-and-replacing-content-in-signed-pdfs |
| USENIX 2021 now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach | USENIX 2021 now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach |
| CCS 2022 clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam | CCS 2022 clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam |
| IEEE-SP 2022 the-state-of-the-samesite-studying-the-usage-effectiveness-and-adequacy-of-sames (also in REJECTED) | |
| IEEE-SP 2022 timing-based-browsing-privacy-vulnerabilities-via-site-isolation | IEEE-SP 2022 timing-based-browsing-privacy-vulnerabilities-via-site-isolation |
| IMC 2022 characterizing-permanently-dead-links-on-wikipedia | IMC 2022 characterizing-permanently-dead-links-on-wikipedia |
| PETS 2022 developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev | PETS 2022 developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev |
| PETS 2022 exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective | PETS 2022 exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective |
| PETS 2022 revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an (also in REJECTED) | |
| PETS 2022 we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob | PETS 2022 we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob |
| USENIX 2022 movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi | USENIX 2022 movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi |
| IMC 2023 a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda | IMC 2023 a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda |
| IMC 2023 rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki | IMC 2023 rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki |
| IMC 2023 the-cve-wayback-machine-measuring-coordinated-disclosure-from-exploits-against-t (also in REJECTED) | |
| IMC 2023 wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d | IMC 2023 wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d |
| PETS 2023 investigating-how-users-imagine-their-personal-privacy-assistant | PETS 2023 investigating-how-users-imagine-their-personal-privacy-assistant |
| IEEE-SP 2024 sok-sgx-fail-how-stuff-gets-exposed | IEEE-SP 2024 sok-sgx-fail-how-stuff-gets-exposed |
| IEEE-SP 2024 tabbed-out-subverting-the-android-custom-tab-security-model | IEEE-SP 2024 tabbed-out-subverting-the-android-custom-tab-security-model |
| IEEE-SP 2024 where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac (also in REJECTED) | |
| NDSS 2024 dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome | NDSS 2024 dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome |
| NDSS 2024 understanding-the-implementation-and-security-implications-of-protective-dns-services | NDSS 2024 understanding-the-implementation-and-security-implications-of-protective-dns-services |
| CCS 2025 a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and | CCS 2025 a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and |
| CCS 2025 is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams | CCS 2025 is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams |
| IEEE-SP 2025 beyond-the-horizon-uncovering-hosts-and-services-behind-misconfigured-firewalls (in CAMPAIGNS) | |
| IEEE-SP 2025 gptracker-a-large-scale-measurement-of-misused-gpts | IEEE-SP 2025 gptracker-a-large-scale-measurement-of-misused-gpts |
| IEEE-SP 2025 sniffing-location-privacy-of-video-conference-users-using-free-audio-channels | IEEE-SP 2025 sniffing-location-privacy-of-video-conference-users-using-free-audio-channels |
| USENIX 2025 gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical | USENIX 2025 gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical |
| USENIX 2025 malicious-llm-based-conversational-ai-makes-users-reveal-personal-information | USENIX 2025 malicious-llm-based-conversational-ai-makes-users-reveal-personal-information |
| | USENIX 2025 misty-registry-an-empirical-study-of-flawed-domain-registry-operation |
| | USENIX 2025 privacy-law-enforcement-under-centralized-governance-a-qualitative-analysis-of-f |
| | USENIX 2025 regulating-smart-device-support-periods-user-expectations-and-the-european-cyber |
| | USENIX 2025 your-shield-is-my-sword-a-persistent-denial-of-service-attack-via-the-reuse-of-u |
| | IEEE-SP 2026 consumer-beware-exploring-data-brokers-ccpa-compliance |
| | NDSS 2026 chameleoscan-demystifying-and-detecting-ios-chameleon-apps-via-llm-powered-ui-exploration |
| | NDSS 2026 cross-boundary-mobile-tracking-exploring-java-to-javascript-information-diffusion-in-webviews |
| | NDSS 2026 demystifying-the-access-control-mechanism-of-esxi-vmkernel |
| | NDSS 2026 tickets-to-hide-an-inside-look-into-the-anti-abuse-ecosystem-through-internal-abuse-data |
| | NDSS 2026 time-and-time-again-leveraging-tcp-timestamps-to-improve-remote-timing-attacks |
| | NDSS 2026 unveiling-byovd-threats-malwares-use-and-abuse-of-kernel-drivers |
| | PETS 2026 more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr |
| | PETS 2026 personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t |
| | PETS 2026 precision-leads-recalling-you-improved-location-privacy-for-shared-mobility-serv |
| | PETS 2026 the-role-of-online-forums-in-developer-understanding-of-privacy-law-a-reddit-cas |
| | USENIX 2026 abuse-risks-are-often-inherent-to-product-features-exploring-ai-vendors-bug-boun |
| | USENIX 2026 inconsistent-incomplete-and-insecure-a-survey-of-account-security-interfaces |
| | USENIX 2026 the-art-of-hide-and-seek-making-pickle-based-model-supply-chain-poisoning-stealt |
| | WWW 2026 bowling-with-chatgpt-on-the-evolving-user-interactions-with-conversational-ai-sy |
| | WWW 2026 falconscope-effective-and-efficient-detection-of-hidden-web-interfaces-in-iot-de |
| | WWW 2026 unveiling-the-resilience-of-llm-enhanced-search-engines-against-black-hat-seo-ma |
| </code> | </code> |
| |
| <wrap todo>The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap> | <wrap todo>**Filed as deferred work**, not left as a TODO on a wiki page: ''DRAIN_DB=/workspace/drain/work.db python3 /workspace/drain/sync.py add-item --task wiki-measuretheweb --key 'notifying_websites: read the 144 unreviewed campaign candidates''', with the shortlist above and the two open ''security.txt'' questions attached. |
| | |
| | **The first published version of that list had 135 of the 144 entries**, because it was built through a shell pipeline that dropped nine rows rather than copied from the script. It is now generated directly from ''report_notifying_websites.mjs --residue'' and asserted to be exactly 144 long. A residue list that is itself truncated is worse than no residue list, and this one was, for three revisions. |
| | |
| | The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap> |
| |
| ===== Quotes spot-checked ===== | ===== Quotes spot-checked ===== |
| | {[stock2016_hey]} 5.8% received / 74.5% still exploitable / Table 1 fix rates / Abusix / RFC 2142 aliases / legal threat on day two / 5 opt-outs covering 187 domains | read directly by the orchestrator from ''paper.cols.txt''; no sub-agent involved | | | {[stock2016_hey]} 5.8% received / 74.5% still exploitable / Table 1 fix rates / Abusix / RFC 2142 aliases / legal threat on day two / 5 opt-outs covering 187 domains | read directly by the orchestrator from ''paper.cols.txt''; no sub-agent involved | |
| | {[lone2022_sav]} PeeringDB-before-WHOIS contact cascade | read directly by the orchestrator | | | {[lone2022_sav]} PeeringDB-before-WHOIS contact cascade | read directly by the orchestrator | |
| | |
| | **Two errors in the first published version, caught by the orchestrator's own re-check after publication and fixed in revision 2:** |
| | |
| | * The {[sasaki2022_ics]} row said "93 (58%) responded — 'higher than most previous notification experiments'". Wrong on three counts. The channel is **manual telephone calls**, not email — the paper explicitly contrasts itself with prior studies that used IP-WHOIS abuse contacts and claims to be "the first study to directly contact the organization operating the device". The ''93 (58%)'' is a spliced Table VI cell, not a response rate. And "higher than most previous notification experiments" is about the paper's **remediation** rate, not its response rate. The corrected row uses the paper's own funnel: 160 operators notified about 317 devices, the person in charge reached for 212, "50% of the persons in charge … stated that they mitigated or will mitigate", devices "reduced by 58% when we were able to contact the persons in charge" against 13% for un-notified devices, χ² p<0.0001. **This came from a sub-agent report that the orchestrator did not re-verify before first publication — the one place on the page where the verification discipline recorded above was not actually followed.** |
| | * The {[cetin2019_cleaning]} control column said "74% natural remediation in the control". The paper says "surprisingly high natural remediation rates of 58[–]74% for this control group **and for two reference networks**" — a range across three groups, not a single control figure. Also added: "Email-only notifications have no observable impact compared to a control group", which is the more important half of that paper for this page. |
| | |
| | Discovering the first of these prompted a new subsection on the content page: **phone calls and hand-read imprints are the channels with the highest measured rates in this literature**, and the page's earlier flat statement that telephone "does not scale" was true only at {[stock2016_hey]}'s N of 44,790. That claim is now qualified rather than repeated. |
| |
| **One internal inconsistency, not resolved:** {[stock2016_hey]} states a 2.4% control fix rate for client-side XSS in prose and 2.2% (4/185) in Table 1. The page quotes the table. | **One internal inconsistency, not resolved:** {[stock2016_hey]} states a 2.4% control fix rate for client-side XSS in prose and 2.2% (4/185) in Table 1. The page quotes the table. |
| ===== The embedded code ===== | ===== The embedded code ===== |
| |
| ''notes/find_contacts.py'' on the page is the real file, and the ''<code>'' block under it is its real, unedited output from a run on 2026-08-13 against ''google.com ethz.ch cispa.de bbc.co.uk wikipedia.org''. | ''notes/find_contacts.py'' on the page is the real file, and the code block under it is its real, unedited output from a run on 2026-08-13 against ''google.com ethz.ch cispa.de bbc.co.uk wikipedia.org''. |
| |
| Two things about that output are worth recording rather than tidying away: | Three things about that output are worth recording rather than tidying away: |
| |
| * The ''RIPEstat: TimeoutError'' line for ''google.com'' is genuine. An earlier run in the same session returned ''network-abuse@google.com'' for it, so RIR-abuse coverage read 100% then and 80% in the saved run. **The output on the page is the saved run, timeout included**, because that transient is exactly the failure a reader needs to see distinguished from an absent contact. | * **The same script on the same five domains gave 80% RIR-abuse coverage in one run and 100% in another**, purely because ''stat.ripe.net'' timed out once for ''google.com''. The published block is the 100% run (it matches the current code); the 80% run is recorded here, and the content page states the discrepancy explicitly rather than presenting either number as a coverage measurement. Five domains and a transient — this is why the page says the sample is hand-picked and not representative. |
| * ''.ch'' and ''.de'' having no RDAP service in the IANA bootstrap is a real finding, not a script bug — confirmed by reading ''dns.json''. | * ''.ch'' and ''.de'' having no RDAP service in the IANA bootstrap is a real finding, not a script bug — confirmed by reading ''dns.json'' directly. |
| | * **A reviewer found a real bug and it was fixed.** ''_rdap_base()'' fetched the IANA bootstrap registry with no error handling, so a single transient failure there raised an uncaught ''URLError'' and killed the whole batch — contradicting the script's own promise that failures are printed rather than swallowed. It now raises a ''RuntimeError'' naming the URL and the reason, and refuses to continue, on the grounds that the alternative (a column of dashes) would silently understate RDAP coverage for every domain at once. Verified both paths: the happy path prints the table above, and pointing ''BOOTSTRAP'' at an unreachable host raises with full context. **This is the second defect on this page that only surfaced because someone re-ran the code rather than reading it.** |
| |
| Design decisions in that script, all of them arguable: | Design decisions in that script, all of them arguable: |
| ===== Bibliography ===== | ===== Bibliography ===== |
| |
| 21 new entries added to [[Literature:Bibliography]] before its closing ''</bibtex>''. Generated with ''node scripts/bibgen.mjs'', i.e. from ''data/corpus2/.meta'' publisher metadata rather than from recall. Checked before saving: 22 distinct citation keys used on the content page, **0 unresolved, 0 colliding with the 239 keys already in the file**. | 21 new entries added to [[Literature:Bibliography]] before its closing ''</bibtex>''. Generated with ''node scripts/bibgen.mjs'', i.e. from ''data/corpus2/.meta'' publisher metadata rather than from recall. Checked before saving: 23 distinct citation keys used on the content page, **0 unresolved, 0 colliding with the 239 keys already in the file**. |
| |
| Two entries already existed and were reused rather than duplicated: ''maass2021_effective'' and ''hantke2024_redlines''. | Two entries already existed and were reused rather than duplicated: ''maass2021_effective'' and ''hantke2024_redlines''. |
| ===== Reviewer findings ===== | ===== Reviewer findings ===== |
| |
| Four reviewers, all told the context might not be exhaustive, all given the page text, the report script, its output, and these notes. | Four reviewers, all told the context might not be exhaustive, all given the page text, the report script, its output, and these notes. Three focused passes ran in parallel first; the generic pass ran after their findings were applied. |
| |
| <wrap todo>To be filled in as each reviewer reports; rejections recorded alongside fixes.</wrap> | ==== Sonnet — figures against the script ==== |
| | |
| | ^ Finding ^ Action ^ |
| | | ''_rdap_base()'' in ''find_contacts.py'' fetches the IANA bootstrap registry with no error handling, so one transient failure raises an uncaught ''URLError'' and kills the whole batch — contradicting both the script's docstring and the page's claim that failures are printed. Reviewer reproduced it. | **ACCEPTED and fixed.** Now raises a ''RuntimeError'' naming the URL and the reason, and refuses to continue. Both paths re-tested. | |
| | | ''out/notifying_websites-output.txt'' is two concatenated runs, so a naive ''diff'' against one fresh invocation shows ~305 spurious lines. | **ACCEPTED as a documentation gap**, not a figure error: ''head -143'' and ''tail -n +144'' each match a fresh run byte-for-byte. Now stated in the //Report script// section above so the next reviewer is not misled the same way. | |
| | | Provenance said "the seven campaigns not on the table" and listed ''catch-22…'', which **is** on the table. | Already fixed before the reviewer read the file; the count is six. Recorded here because the reviewer independently derived the same arithmetic. | |
| | | Every number, percentage and denominator named in the brief — 4,472; 48.3%; 61.0% vs 43.5%; 3.3%; 7.2%; 2,303/2,870; 7/2,870; 2,155/2,870; all nine channel shares; 22/16/144/179 — matches the live script output, cross-checked against ''extractions.jsonl''. | No action. | |
| | |
| | ==== Sonnet — citations and quotes ==== |
| | |
| | ^ Finding ^ Action ^ |
| | | {[sasaki2022_ics]}: the page conflated **two different 58% figures** — 93 of 160 operators reached (58%) and 58% of devices remediated once reached — and attached the "higher than most previous notification experiments" quote to the wrong one. | Already caught and fixed by the orchestrator before the reviewer reported (see the correction note above); the reviewer confirms the diagnosis independently, which is the strongest evidence the fix is right. | |
| | | {[stivala2026_behind]}: "reports the same thing a decade later" reads as an independent 2026 re-measurement, but the sentence is a related-work summary carrying citations to four earlier papers. | **ACCEPTED.** The content page now says so in the sentence itself, not only here. | |
| | | {[lone2022_sav]}: "found PeeringDB better maintained" is that paper relaying prior work — "we preferred peeringDB because it has been used in previous studies and **they** found the database up-to-date". | **ACCEPTED**, reworded with the paper's own sentence. | |
| | | This provenance page said "22 distinct citation keys"; it is 23. | **ACCEPTED**, corrected. | |
| | | All 23 keys resolve uniquely against the live bibliography (262 entries), no duplicates, no collisions. All 21 new entries have correct author/citekey match, correct titles against ''data/corpus2/.meta'', and DOIs that resolve at doi.org — including ''stock2018_didnt''. Every load-bearing quote across 18 papers confirmed verbatim or as a faithful cross-column splice, including the whole of {[stock2018_didnt]} re-fetched fresh from ndss-symposium.org. | No action. | |
| | | {[cetin2019_cleaning]}: 74% is specifically the control group's own cleanup rate, not merely the top of the abstract's 58–74% range. | **ACCEPTED — this reverted an over-correction.** An earlier fix had replaced "74% in the control" with the range across three groups; the reviewer's check of the body text shows "The control group achieved the lowest cleanup rate (74%)". The page now quotes that, and keeps the "email-only had no observable impact" finding that the same pass surfaced. | |
| | |
| | ==== Sonnet — external currency ==== |
| | |
| | ^ Finding ^ Action ^ |
| | | The content page quotes 2021 ''security.txt'' adoption by **Alexa** rank with no signal that Alexa is defunct; the caveat existed only here, in //What could not be established//. | **ACCEPTED.** The caveat is now inline on the page that makes the claim: 2021 measurement, defunct ranking frame, predates RFC 9116, and nothing in the corpus re-measures it. | |
| | | "annually validated by RIPE and ARIN" — ARIN's annual POC verification is in NRPM §3.6 and was fetched; **no RIPE primary source states a cadence**. | **ACCEPTED.** Split: ARIN's annual verification is quoted with its source, RIPE's is reduced to "keeps abuse contacts valid" with an explicit note that the cadence could not be verified. | |
| | | The footnote saying ''usenix.org'' "rejects automated fetches" is too broad — ''curl'' with a browser User-Agent returns 200. The reviewer fetched the CFP and confirmed both quoted sentences verbatim. | **ACCEPTED, and it upgrades the page**: the USENIX claim is no longer cache-based. Re-fetched independently by the orchestrator, quotes confirmed, footnote rewritten to cite the page directly and to note that only ''WebFetch'' gets 403. | |
| | | The footnote's "Registration Data Policy took effect … in August 2025" could not be pinned to a primary source. | **ACCEPTED.** Reduced to what the ICANN site does say ("Now In Effect for Contracted Parties") with the date removed. | |
| | | ''g.co/vulnz'' — the address in Google's own ''security.txt'', visible in the script output — now 302-redirects to ''bughunters.google.com''. | **NOTED, no change.** The published block is a dated run and is accurate as of that run; changing it would falsify the audit trail. Recorded here for anyone re-running the script. | |
| | | RFC 9910 (RDAP RIR Search, January 2026) exists, plus several active REGEXT drafts, but **none obsoletes or updates RFC 9082/9083/9224**. | **NOTED, no change** — the page's "current, not superseded" claim holds, and 9910 is a search extension the page has no reason to cite. | |
| | | RFC 9116, RFC 2142, RFC 9082/9083/9224, the ICANN sunset quote, CERT/CC's 45 days, Project Zero's 90+30/7-day policy and the Reporting Transparency trial, NIS2 Article 12(1), HackerOne Disclosure Assistance, Shadowserver, the FIRST directory, the IANA bootstrap, the RIPEstat API, and all four fetchable venue CFP quotes: verified verbatim against primary sources. No dead links on either page. | No action. | |
| | | ISO/IEC 29147 / 30111: ''iso.org'' 403s for the reviewer too. Because the page makes no edition or revision-status claim, this is a safe non-claim rather than an error. | No action — confirms the earlier decision to cut those claims. | |
| | |
| | ==== Caught after the review passes, by the orchestrator ==== |
| | |
| | * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. |
| | * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the code block on the page is a transcribed copy with em-dashes substituted for ASCII, and a substitution inside a string literal would have broken it silently. |
| | * **A stale page cache made most citations render as raw keys, and it survived four reviewers and every verification pass above.** After the bibliography was extended and the pages saved, ''https://measuretheweb.org/practices/notifying_websites'' rendered ''[lone2022_sav]'', ''[utz2023_comparing]'', ''[stock2018_didnt]'' and most others as literal bracketed keys, while the two pre-existing keys resolved normally. Nothing was wrong with the source, the bibliography or the keys — DokuWiki was serving a cached render made before the bibliography grew. Fixed by requesting each page once with ''&purge=true''; both then rendered 218 and 98 resolved citations with zero raw keys left. **Add a purge to the end of any run that touches ''literature:bibliography'', and re-check the rendered page after purging rather than before.** The earlier checks in this section counted ''bibtex_citekey'' markers and found 38 on a page with ~60 citations, which should have been the tell and was not. |
| | * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected. |
| | |
| | ==== Added after the review passes ==== |
| | |
| | The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. |
| | |
| | ==== Fable — generic, no checklist, run after the three focused passes ==== |
| | |
| | The most productive of the four. Eleven findings, **all accepted**; five were wrong or self-contradicting claims on the content page. |
| | |
| | ^ Finding ^ Action ^ |
| | | The lead box said "both of the **properly randomised**, control-arm experiments … found no significant effect", which implies {[maass2021_effective]} was not properly randomised. It is: "a covert randomized controlled notification experiment", "full factorial design", with a control group — and it is the strongest positive result on the page. The real distinction is population, not rigour. | **ACCEPTED**, verified against the paper, rewritten to "both randomised, control-arm experiments **on network operators**", and the Maass sentence now names it as a randomised controlled experiment rather than "which also had a control group". This was the single most prominent sentence on the page. | |
| | | The rescoping justification claimed the excluded group "is participant-debriefing notes". Reviewer checked: only 273 of 710. | **ACCEPTED**, verified, and the reason replaced with the correct one — the channel question is ill-posed for a paper that did not notify — with the 38.5% in a footnote. | |
| | | "scanning papers … are **less than half as likely** to leave the question unanswered" is contradicted by the page's own table: 24.0% vs 31.1%. The column that //is// less than half is ''no'' (4.2% vs 9.4%). | **ACCEPTED**, both comparisons now stated with their figures. | |
| | | The opening said "you **have to tell** those 8,000 operators, and **every venue** … expects you to say so" — contradicting the page's own PoPETs bullet, and its own paragraph saying that deciding not to notify is defensible. The footnote also covered only four of the seven corpus venues. | **ACCEPTED**: "decide what to do about", "the major venues", and the footnote now names PoPETs' absence of a requirement and says NDSS and TheWebConf are not addressed. | |
| | | {[stock2018_didnt]}'s 74.4% / 33.3% was labelled "among those who **opened** the report" in two places. It is among those who **viewed** it — and the page's own funnel insists opened and viewed are distinct stages. | **ACCEPTED**, both corrected to //viewed//. | |
| | | 22 + 16 + 144 = 182 ≠ 179 on the content page, with the explanation only here. | **ACCEPTED**: the page now says 13 reasoned rejections inside the regex plus three carried over from the wider first-pass scan, totalling 16. The 5,859-records / 5,869-files gap is also now explained in the same sentence. | |
| | | The rates table's intro says it shows campaigns "that reported a quotable outcome", but the {[munteanu2025_catch22]} row has no figure — the stated exclusion criterion. | **ACCEPTED**, the exception is now stated on the content page, not only here. | |
| | | "**Three** of those channels are machine-queryable" is wrong twice: RDAP is not one of the channels in the table, and PeeringDB is machine-queryable but is not in the script. | **ACCEPTED**, rewritten to two per-domain channels plus RDAP, with PeeringDB noted as per-AS. | |
| | | The provenance itself conceded that neither the script docstring nor the page says why RIPEstat was chosen over Abusix, and left it unfixed. | **ACCEPTED**, the docstring now says so — Abusix is better at scale, RIPEstat is what runs with only the standard library and is therefore testable here. | |
| | | This provenance page said "Four reviewers … logged below" with three logged, and "**Two** judgement calls" followed by three bullets. The ''head -143'' / ''tail -n +144'' instruction had gone stale when the script grew. | **ACCEPTED**, all three fixed; the split point is now derived rather than hard-coded, since it moves every time a table row is added. | |
| | | Voice: the only first-person singular on the site ("no cadence **I** could verify"); //survival// used without saying lower is better; "That is not negligence" asserts a motive the data does not show; "The one flat line" for a series that rose tenfold then declined. | **ACCEPTED**, all four reworded. | |
| | |
| | Also confirmed by this pass and not changed: the page answers its own question and the top-to-bottom ordering works; no dead-weight sections; all DokuWiki constructs well-formed in the current revision; the rates table's 17 rows reconcile with this page's accounting; every corpus figure matches a fresh script run; the embedded script runs as documented; and the voice, while denser in bolding and aphorism than the older hand-written pages, sits inside the range of the site's newer corpus-backed pages. |
| | |
| | ==== Nothing was rejected outright ==== |
| | |
| | Every substantive finding across all four passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page: |
| | |
| | * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading. |
| | * The **quotes** pass independently reproduced the {[sasaki2022_ics]} diagnosis the orchestrator had already made, and **caught an over-correction** on {[cetin2019_cleaning]} that the orchestrator's own fix had introduced. A correction is as capable of being wrong as the thing it corrects. |
| |
| ====== References ====== | ====== References ====== |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| | |
| | [[practices:notifying_websites|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| |
