Measure The Web

User Tools

Site Tools

Trace:
provenance:practices:notifying_websites

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
provenance:practices:notifying_websites [2026/08/13 20:03] – Byte-exact script output; record the table-rendering bug the figures reviewer missed, the end-to-end check of the downloadable code, and the post-review cross-tab. Authored by Claude karel.kubicek.claudeprovenance:practices:notifying_websites [2026/08/13 20:16] (current) – Record the stale-page-cache bug that made most citations render as raw keys, and the purge that fixed it. Authored by Claude karel.kubicek.claude
Line 4: Line 4:
  
 Not prose. Read it if you are checking a number. Not prose. Read it if you are checking a number.
 +
 +<wrap todo>House convention on ''provenance:'' pages is mixed and this page picked the more traceable end of it. ''provenance:privacy:fingerprinting'' and ''provenance:design:ip_classification'' use no inline citation markers at all and name papers by slug; ''provenance:privacy:requests'' uses citekeys and a bibliography block, as this page does. Two of the four carry a footer back-link, so this one does too. **No ''~~DISCUSSION~~'' block**, matching all four: comments belong on the content page.</wrap>
  
 ===== The run ===== ===== The run =====
Line 18: Line 20:
 ===== Scope decision ===== ===== Scope decision =====
  
-The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Two judgement calls:+The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Three judgement calls:
  
   * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them.   * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them.
Line 48: Line 50:
 Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate.
  
-<wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run (lines 1–143) followed by the ''--list'' run (lines 144–448), which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows ~305 spurious extra lines. This misled one reviewer; ''head -143'' and ''tail -n +144'' each match a fresh run byte-for-byte. The script itself is deterministic — verified.</wrap>+<wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run followed by the ''--list'' run, which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows a few hundred spurious extra lines, and this misled one reviewer. Do not hard-code the split point — it moves whenever a table row is added. Derive it: ''BASE=$(node scripts/report_notifying_websites.mjs | wc -l)'', then ''head -$BASE'' and ''tail -n +$((BASE+1))'' each match a fresh run byte-for-byte. As of this revision ''BASE'' is 161. The script itself is deterministic — verified.</wrap>
  
 Sections of the script, so a figure on the content page can be traced to the block that produced it: Sections of the script, so a figure on the content page can be traced to the block that produced it:
Line 113: Line 115:
  
 of the 376 that assessed a law: yes 27 (7.2%) of the 376 that assessed a law: yes 27 (7.2%)
 +
 +of the 2160 that notified an affected party, 135 also contacted a regulator or CERT (6.3%)
 +of the 147 that contacted a regulator or CERT, 135 ALSO notified the affected party directly (91.8%) — the intermediary is an addition, not a substitute
  
 -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step --
Line 137: Line 142:
 the notable thing. These are rankings of an under-reported field, not adoption rates. the notable thing. These are rankings of an under-reported field, not adoption rates.
  
--- channel named in disclosureDetail, of 2870 papers that gave any --+-- channel named in disclosureDetail, of 2870 papers that gave any disclosure detail --
 Channel family                                                                 Papers  Share of 2870 Channel family                                                                 Papers  Share of 2870
 -----------------------------------------------------------------------------  ------  ------------- -----------------------------------------------------------------------------  ------  -------------
Line 149: Line 154:
 standards body / working group                                                 13      0.5% standards body / working group                                                 13      0.5%
 generic contact form or phone                                                  10      0.3% generic contact form or phone                                                  10      0.3%
-no channel named at all: 2303 of 2870 (80.2%) — the single largest group, and the finding+no channel named at all: 2303 of 2870 (80.2%) 
 + 
 +-- channel named in disclosureDetail, of 2160 papers that notified (yes|partial) AND gave a detail — the population where "which channel?" is a fair question -- 
 +Channel family                                                                 Papers  Share of 2160 
 +-----------------------------------------------------------------------------  ------  ------------- 
 +a named large platform or app store (Google, Apple, Meta, Microsoft, Amazon…)  318     14.7% 
 +direct email to the operator or developer                                      99      4.6% 
 +CERT / CSIRT / national coordinator                                            39      1.8% 
 +bug bounty / vulnerability reward programme                                    35      1.6% 
 +hosting provider / ISP / upstream                                              35      1.6% 
 +WHOIS / registry contact                                                       15      0.7% 
 +data protection authority / regulator                                          14      0.6% 
 +standards body / working group                                                 13      0.6% 
 +generic contact form or phone                                                  9       0.4% 
 +no channel named at all: 1633 of 2160 (75.6%)
  
 -- outcome named in disclosureDetail, of 2870 -- -- outcome named in disclosureDetail, of 2870 --
Line 216: Line 235:
 Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates. Nine regex families, **not mutually exclusive** (a paper that told a CERT and the vendor belongs in both; papers counted once per family). Published on the content page as a ranking, never as adoption rates.
  
-  * The residue is the headline: **2,303 of 2,870 (80.2%) match no channel family.** That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. Published on the content page as the finding.+  * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. 
 +  * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. 
 +  * **Two claims on the content page about this fold were checked directly against the JSONL rather than taken from the script.** (a) "all 2,160 that said they notified are among them": of the 2,160 ''yes''/''partial'' papers, **0** lack a ''disclosureDetail''. (b) The scoping footnote's 38.5%: of the 710 papers that gave a detail without saying they notified, 273 match a participant/consent/debrief/interview/survey/recruit/volunteer term. That second figure exists because a reviewer challenged an earlier, stronger claim that the excluded group //was// participant-debriefing notes — it is not, and the page now says only that the channel question is ill-posed for that group.
   * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".**   * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".**
   * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved.   * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved.
Line 281: Line 302:
 USENIX 2013 an-empirical-study-of-vulnerability-rewards-programs USENIX 2013 an-empirical-study-of-vulnerability-rewards-programs
 WWW 2013 two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure WWW 2013 two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure
-CCS 2013 rethinking-ssl-development-in-an-appified-world (also in REJECTED) 
 USENIX 2014 a-look-at-targeted-attacks-through-the-lense-of-an-ngo USENIX 2014 a-look-at-targeted-attacks-through-the-lense-of-an-ngo
 USENIX 2014 brahmastra-driving-apps-to-test-the-security-of-third-party-components USENIX 2014 brahmastra-driving-apps-to-test-the-security-of-third-party-components
-CCS 2015 an-empirical-study-of-web-vulnerability-discovery-ecosystems (also in REJECTED) 
 IEEE-SP 2015 the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p IEEE-SP 2015 the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p
 NDSS 2015 checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo NDSS 2015 checking-more-and-alerting-less-detecting-privacy-leakages-via-enhanced-data-flo
Line 292: Line 311:
 IMC 2016 weak-keys-remain-widespread-in-network-devices IMC 2016 weak-keys-remain-widespread-in-network-devices
 PETS 2016 towards-mining-latent-client-identifiers-from-network-traffic PETS 2016 towards-mining-latent-client-identifiers-from-network-traffic
-CCS 2017 a-large-scale-empirical-study-of-security-patches (also in REJECTED) 
 IMC 2017 if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to IMC 2017 if-you-are-not-paying-for-it-you-are-the-product-how-much-do-advertisers-pay-to
 NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security
Line 299: Line 317:
 USENIX 2018 wpse-fortifying-web-protocols-via-browser-side-security-monitoring USENIX 2018 wpse-fortifying-web-protocols-via-browser-side-security-monitoring
 CCS 2019 a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right CCS 2019 a-usability-evaluation-of-lets-encrypt-and-certbot-usable-security-done-right
-CCS 2019 network-hygiene-incentives-and-regulation-deployment-of-source-address-validatio (also in REJECTED) 
 NDSS 2019 automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries NDSS 2019 automating-patching-of-vulnerable-open-source-software-versions-in-application-binaries
 NDSS 2019 digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system NDSS 2019 digital-healthcare-associated-infection-a-case-study-on-the-security-of-a-major-multi-campus-hospital-system
Line 309: Line 326:
 WWW 2019 studying-preferences-and-concerns-about-information-disclosure-in-email-notifica WWW 2019 studying-preferences-and-concerns-about-information-disclosure-in-email-notifica
 CCS 2020 dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat CCS 2020 dangerous-skills-got-certified-measuring-the-trustworthiness-of-skill-certificat
-CCS 2020 the-cookie-hunter-automated-black-box-auditing-for-web-authentication-and-author (also in REJECTED) 
 IEEE-SP 2020 a-security-analysis-of-the-facebook-ad-library IEEE-SP 2020 a-security-analysis-of-the-facebook-ad-library
 IEEE-SP 2020 the-many-kinds-of-creepware-used-for-interpersonal-attacks IEEE-SP 2020 the-many-kinds-of-creepware-used-for-interpersonal-attacks
Line 321: Line 337:
 WWW 2020 an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre WWW 2020 an-empirical-study-of-the-use-of-integrity-verification-mechanisms-for-web-subre
 CCS 2021 i-need-a-better-description-an-investigation-into-user-expectations-for-differen CCS 2021 i-need-a-better-description-an-investigation-into-user-expectations-for-differen
-CCS 2021 out-of-sight-out-of-mind-detecting-orphaned-web-pages-at-internet-scale (also in REJECTED) 
 IEEE-SP 2021 sok-quantifying-cyber-risk IEEE-SP 2021 sok-quantifying-cyber-risk
-IMC 2021 who-you-gonna-call-an-empirical-evaluation-of-website-security-txt-deployment (also in REJECTED) 
 NDSS 2021 all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers NDSS 2021 all-the-numbers-are-us-large-scale-abuse-of-contact-discovery-in-mobile-messengers
 NDSS 2021 shadow-attacks-hiding-and-replacing-content-in-signed-pdfs NDSS 2021 shadow-attacks-hiding-and-replacing-content-in-signed-pdfs
Line 332: Line 346:
 USENIX 2021 now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach USENIX 2021 now-im-a-bit-angry-individuals-awareness-perception-and-responses-to-data-breach
 CCS 2022 clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam CCS 2022 clues-in-tweets-twitter-guided-discovery-and-analysis-of-sms-spam
-IEEE-SP 2022 the-state-of-the-samesite-studying-the-usage-effectiveness-and-adequacy-of-sames (also in REJECTED) 
 IEEE-SP 2022 timing-based-browsing-privacy-vulnerabilities-via-site-isolation IEEE-SP 2022 timing-based-browsing-privacy-vulnerabilities-via-site-isolation
 IMC 2022 characterizing-permanently-dead-links-on-wikipedia IMC 2022 characterizing-permanently-dead-links-on-wikipedia
Line 339: Line 352:
 PETS 2022 developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev PETS 2022 developers-say-the-darnedest-things-privacy-compliance-processes-followed-by-dev
 PETS 2022 exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective PETS 2022 exploring-the-privacy-concerns-of-bystanders-in-smart-homes-from-the-perspective
-PETS 2022 revisiting-identification-issues-in-gdpr-right-of-access-policies-a-technical-an (also in REJECTED) 
 PETS 2022 we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob PETS 2022 we-may-share-the-number-of-diaper-changes-a-privacy-and-security-analysis-of-mob
 USENIX 2022 movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi USENIX 2022 movery-a-precise-approach-for-modified-vulnerable-code-clone-discovery-from-modi
Line 352: Line 364:
 IMC 2023 a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda IMC 2023 a-longitudinal-study-of-vulnerable-client-side-resources-and-web-developers-upda
 IMC 2023 rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki IMC 2023 rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki
-IMC 2023 the-cve-wayback-machine-measuring-coordinated-disclosure-from-exploits-against-t (also in REJECTED) 
 IMC 2023 wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d IMC 2023 wolf-in-sheeps-clothing-evaluating-security-risks-of-the-undelegated-record-on-d
 PETS 2023 investigating-how-users-imagine-their-personal-privacy-assistant PETS 2023 investigating-how-users-imagine-their-personal-privacy-assistant
Line 377: Line 388:
 IEEE-SP 2024 sok-sgx-fail-how-stuff-gets-exposed IEEE-SP 2024 sok-sgx-fail-how-stuff-gets-exposed
 IEEE-SP 2024 tabbed-out-subverting-the-android-custom-tab-security-model IEEE-SP 2024 tabbed-out-subverting-the-android-custom-tab-security-model
-IEEE-SP 2024 where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac (also in REJECTED) 
 NDSS 2024 dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome NDSS 2024 dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome
 NDSS 2024 understanding-the-implementation-and-security-implications-of-protective-dns-services NDSS 2024 understanding-the-implementation-and-security-implications-of-protective-dns-services
Line 394: Line 404:
 CCS 2025 a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and CCS 2025 a-decade-long-landscape-of-advanced-persistent-threats-longitudinal-analysis-and
 CCS 2025 is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams CCS 2025 is-this-a-scam-the-nature-and-quality-of-reddit-discussion-about-scams
-IEEE-SP 2025 beyond-the-horizon-uncovering-hosts-and-services-behind-misconfigured-firewalls (in CAMPAIGNS) 
 IEEE-SP 2025 gptracker-a-large-scale-measurement-of-misused-gpts IEEE-SP 2025 gptracker-a-large-scale-measurement-of-misused-gpts
 IEEE-SP 2025 sniffing-location-privacy-of-video-conference-users-using-free-audio-channels IEEE-SP 2025 sniffing-location-privacy-of-video-conference-users-using-free-audio-channels
Line 410: Line 419:
 USENIX 2025 gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical USENIX 2025 gpuhammer-rowhammer-attacks-on-gpu-memories-are-practical
 USENIX 2025 malicious-llm-based-conversational-ai-makes-users-reveal-personal-information USENIX 2025 malicious-llm-based-conversational-ai-makes-users-reveal-personal-information
 +USENIX 2025 misty-registry-an-empirical-study-of-flawed-domain-registry-operation
 +USENIX 2025 privacy-law-enforcement-under-centralized-governance-a-qualitative-analysis-of-f
 +USENIX 2025 regulating-smart-device-support-periods-user-expectations-and-the-european-cyber
 +USENIX 2025 your-shield-is-my-sword-a-persistent-denial-of-service-attack-via-the-reuse-of-u
 +IEEE-SP 2026 consumer-beware-exploring-data-brokers-ccpa-compliance
 +NDSS 2026 chameleoscan-demystifying-and-detecting-ios-chameleon-apps-via-llm-powered-ui-exploration
 +NDSS 2026 cross-boundary-mobile-tracking-exploring-java-to-javascript-information-diffusion-in-webviews
 +NDSS 2026 demystifying-the-access-control-mechanism-of-esxi-vmkernel
 +NDSS 2026 tickets-to-hide-an-inside-look-into-the-anti-abuse-ecosystem-through-internal-abuse-data
 +NDSS 2026 time-and-time-again-leveraging-tcp-timestamps-to-improve-remote-timing-attacks
 +NDSS 2026 unveiling-byovd-threats-malwares-use-and-abuse-of-kernel-drivers
 +PETS 2026 more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
 +PETS 2026 personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t
 +PETS 2026 precision-leads-recalling-you-improved-location-privacy-for-shared-mobility-serv
 +PETS 2026 the-role-of-online-forums-in-developer-understanding-of-privacy-law-a-reddit-cas
 +USENIX 2026 abuse-risks-are-often-inherent-to-product-features-exploring-ai-vendors-bug-boun
 +USENIX 2026 inconsistent-incomplete-and-insecure-a-survey-of-account-security-interfaces
 +USENIX 2026 the-art-of-hide-and-seek-making-pickle-based-model-supply-chain-poisoning-stealt
 +WWW 2026 bowling-with-chatgpt-on-the-evolving-user-interactions-with-conversational-ai-sy
 +WWW 2026 falconscope-effective-and-efficient-detection-of-hidden-web-interfaces-in-iot-de
 +WWW 2026 unveiling-the-resilience-of-llm-enhanced-search-engines-against-black-hat-seo-ma
 </code> </code>
  
-<wrap todo>The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap>+<wrap todo>**Filed as deferred work**, not left as a TODO on a wiki page: ''DRAIN_DB=/workspace/drain/work.db python3 /workspace/drain/sync.py add-item --task wiki-measuretheweb --key 'notifying_websites: read the 144 unreviewed campaign candidates''', with the shortlist above and the two open ''security.txt'' questions attached. 
 + 
 +**The first published version of that list had 135 of the 144 entries**, because it was built through a shell pipeline that dropped nine rows rather than copied from the script. It is now generated directly from ''report_notifying_websites.mjs --residue'' and asserted to be exactly 144 long. A residue list that is itself truncated is worse than no residue list, and this one was, for three revisions. 
 + 
 +The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap>
  
 ===== Quotes spot-checked ===== ===== Quotes spot-checked =====
Line 587: Line 621:
 ==== Caught after the review passes, by the orchestrator ==== ==== Caught after the review passes, by the orchestrator ====
  
-  * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row read ''| Notified (''yes''\|''partial'') | …'' — but ''\|'' is **not** a DokuWiki escape for a literal pipe. The renderer split the cell there, shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket was empty. Verified by parsing the live page's HTML: the row rendered as ''['Notified (yes\\', 'partial)', 'empirical ∧ ethics', '21.3%', …]'' against the header's eight cells. Fixed by rewriting the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; the numbers were right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source. +  * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. 
-  * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the page's ''<file>'' block is a hand-transcribed copy with em-dashes substituted, and a substitution inside a string literal would have broken it silently.+  * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the code block on the page is a transcribed copy with em-dashes substituted for ASCII, and a substitution inside a string literal would have broken it silently. 
 +  * **A stale page cache made most citations render as raw keys, and it survived four reviewers and every verification pass above.** After the bibliography was extended and the pages saved, ''https://measuretheweb.org/practices/notifying_websites'' rendered ''[lone2022_sav]'', ''[utz2023_comparing]'', ''[stock2018_didnt]'' and most others as literal bracketed keys, while the two pre-existing keys resolved normally. Nothing was wrong with the source, the bibliography or the keys — DokuWiki was serving a cached render made before the bibliography grew. Fixed by requesting each page once with ''&purge=true''; both then rendered 218 and 98 resolved citations with zero raw keys left. **Add a purge to the end of any run that touches ''literature:bibliography'', and re-check the rendered page after purging rather than before.** The earlier checks in this section counted ''bibtex_citekey'' markers and found 38 on a page with ~60 citations, which should have been the tell and was not.
   * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected.   * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected.
  
Line 594: Line 629:
  
 The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it.
 +
 +==== Fable — generic, no checklist, run after the three focused passes ====
 +
 +The most productive of the four. Eleven findings, **all accepted**; five were wrong or self-contradicting claims on the content page.
 +
 +^ Finding ^ Action ^
 +| The lead box said "both of the **properly randomised**, control-arm experiments … found no significant effect", which implies {[maass2021_effective]} was not properly randomised. It is: "a covert randomized controlled notification experiment", "full factorial design", with a control group — and it is the strongest positive result on the page. The real distinction is population, not rigour. | **ACCEPTED**, verified against the paper, rewritten to "both randomised, control-arm experiments **on network operators**", and the Maass sentence now names it as a randomised controlled experiment rather than "which also had a control group". This was the single most prominent sentence on the page. |
 +| The rescoping justification claimed the excluded group "is participant-debriefing notes". Reviewer checked: only 273 of 710. | **ACCEPTED**, verified, and the reason replaced with the correct one — the channel question is ill-posed for a paper that did not notify — with the 38.5% in a footnote. |
 +| "scanning papers … are **less than half as likely** to leave the question unanswered" is contradicted by the page's own table: 24.0% vs 31.1%. The column that //is// less than half is ''no'' (4.2% vs 9.4%). | **ACCEPTED**, both comparisons now stated with their figures. |
 +| The opening said "you **have to tell** those 8,000 operators, and **every venue** … expects you to say so" — contradicting the page's own PoPETs bullet, and its own paragraph saying that deciding not to notify is defensible. The footnote also covered only four of the seven corpus venues. | **ACCEPTED**: "decide what to do about", "the major venues", and the footnote now names PoPETs' absence of a requirement and says NDSS and TheWebConf are not addressed. |
 +| {[stock2018_didnt]}'s 74.4% / 33.3% was labelled "among those who **opened** the report" in two places. It is among those who **viewed** it — and the page's own funnel insists opened and viewed are distinct stages. | **ACCEPTED**, both corrected to //viewed//. |
 +| 22 + 16 + 144 = 182 ≠ 179 on the content page, with the explanation only here. | **ACCEPTED**: the page now says 13 reasoned rejections inside the regex plus three carried over from the wider first-pass scan, totalling 16. The 5,859-records / 5,869-files gap is also now explained in the same sentence. |
 +| The rates table's intro says it shows campaigns "that reported a quotable outcome", but the {[munteanu2025_catch22]} row has no figure — the stated exclusion criterion. | **ACCEPTED**, the exception is now stated on the content page, not only here. |
 +| "**Three** of those channels are machine-queryable" is wrong twice: RDAP is not one of the channels in the table, and PeeringDB is machine-queryable but is not in the script. | **ACCEPTED**, rewritten to two per-domain channels plus RDAP, with PeeringDB noted as per-AS. |
 +| The provenance itself conceded that neither the script docstring nor the page says why RIPEstat was chosen over Abusix, and left it unfixed. | **ACCEPTED**, the docstring now says so — Abusix is better at scale, RIPEstat is what runs with only the standard library and is therefore testable here. |
 +| This provenance page said "Four reviewers … logged below" with three logged, and "**Two** judgement calls" followed by three bullets. The ''head -143'' / ''tail -n +144'' instruction had gone stale when the script grew. | **ACCEPTED**, all three fixed; the split point is now derived rather than hard-coded, since it moves every time a table row is added. |
 +| Voice: the only first-person singular on the site ("no cadence **I** could verify"); //survival// used without saying lower is better; "That is not negligence" asserts a motive the data does not show; "The one flat line" for a series that rose tenfold then declined. | **ACCEPTED**, all four reworded. |
 +
 +Also confirmed by this pass and not changed: the page answers its own question and the top-to-bottom ordering works; no dead-weight sections; all DokuWiki constructs well-formed in the current revision; the rates table's 17 rows reconcile with this page's accounting; every corpus figure matches a fresh script run; the embedded script runs as documented; and the voice, while denser in bolding and aphorism than the older hand-written pages, sits inside the range of the site's newer corpus-backed pages.
  
 ==== Nothing was rejected outright ==== ==== Nothing was rejected outright ====
  
-Every substantive finding across the three passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page:+Every substantive finding across all four passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page:
  
   * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading.   * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading.
Line 604: Line 658:
 ====== References ====== ====== References ======
 <bibtex bibliography></bibtex> <bibtex bibliography></bibtex>
 +
 +[[practices:notifying_websites|← back to the content page]] · [[literature:corpus|corpus-level provenance]]
  
provenance/practices/notifying_websites.1786651405.txt.gz · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki