| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:practices:notifying_websites [2026/08/13 20:08] – Publish the full 144-entry unreviewed residue (the earlier list was silently truncated to 135 by a shell pipeline) and record that. Authored by Claude karel.kubicek.claude | provenance:practices:notifying_websites [2026/08/13 20:16] (current) – Record the stale-page-cache bug that made most citations render as raw keys, and the purge that fixed it. Authored by Claude karel.kubicek.claude |
|---|
| ===== Scope decision ===== | ===== Scope decision ===== |
| |
| The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Two judgement calls: | The item as specified pointed at ''ethics.notifiedAffectedParties'' and ''ethics.harmMitigation'' and asked for "how to reach an operator at scale, response rates reported in the literature, and the disclosure-timeline conventions". Three judgement calls: |
| |
| * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. | * **Scope excludes one-off coordinated disclosure to a named vendor.** That is a queue with an SLA and nothing on this page would help. The page says so in its second paragraph. The consequence is that a large share of the 1,636 ''notifiedAffectedParties == yes'' papers are out of scope for the //how-to// part while remaining in scope for the //how often// part; the page keeps those two questions in separate sections rather than pooling them. |
| Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. | Run as ''node scripts/report_notifying_websites.mjs''. Flags: ''--wiki'' for DokuWiki tables, ''--list'' for the hand-classified campaigns, ''--quotes'' for ethics evidence quotes, ''--residue'' for every unmapped fold entry and every unreviewed candidate. |
| |
| <wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run (lines 1–143) followed by the ''--list'' run (lines 144–448), which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows ~305 spurious extra lines. This misled one reviewer; ''head -143'' and ''tail -n +144'' each match a fresh run byte-for-byte. The script itself is deterministic — verified.</wrap> | <wrap todo>''out/notifying_websites-output.txt'' in the workdir is **two concatenated runs**: the plain run followed by the ''--list'' run, which repeats the plain output before adding the campaign list. A ''diff'' against a single fresh invocation therefore shows a few hundred spurious extra lines, and this misled one reviewer. Do not hard-code the split point — it moves whenever a table row is added. Derive it: ''BASE=$(node scripts/report_notifying_websites.mjs | wc -l)'', then ''head -$BASE'' and ''tail -n +$((BASE+1))'' each match a fresh run byte-for-byte. As of this revision ''BASE'' is 161. The script itself is deterministic — verified.</wrap> |
| |
| Sections of the script, so a figure on the content page can be traced to the block that produced it: | Sections of the script, so a figure on the content page can be traced to the block that produced it: |
| |
| of the 376 that assessed a law: yes 27 (7.2%) | of the 376 that assessed a law: yes 27 (7.2%) |
| | |
| | of the 2160 that notified an affected party, 135 also contacted a regulator or CERT (6.3%) |
| | of the 147 that contacted a regulator or CERT, 135 ALSO notified the affected party directly (91.8%) — the intermediary is an addition, not a substitute |
| |
| -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- | -- ethics.harmMitigation: 3186 of 4472 (71.2%) name at least one step -- |
| * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. | * **Scoped to the right population, after a post-review check.** The first version reported the channel fold over all 2,870 papers with a ''disclosureDetail'', giving 80.2% with no channel named. Reading a sample of that residue showed the problem: ''disclosureDetail'' serves two purposes in the schema, notifying an operator //and// debriefing a participant, so the residue included entries like "Participants provided written or verbal consent; screen recordings were deleted after data collection" — papers that were never notifying an operator, for which "which channel?" is not a question. The script now prints both populations, and the page quotes the scoped one: **1,633 of the 2,160 papers that said they notified (75.6%) name no channel.** All 2,160 ''yes''/''partial'' papers turn out to carry a detail, so that subset needs no separate silence figure. |
| * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. | * The residue is still the headline: three quarters of the papers that say they notified do not say through what. That is not a fold failure, it is the field's content — a 20-word summary of "we disclosed responsibly" names no channel. |
| | * **Two claims on the content page about this fold were checked directly against the JSONL rather than taken from the script.** (a) "all 2,160 that said they notified are among them": of the 2,160 ''yes''/''partial'' papers, **0** lack a ''disclosureDetail''. (b) The scoping footnote's 38.5%: of the 710 papers that gave a detail without saying they notified, 273 match a participant/consent/debrief/interview/survey/recruit/volunteer term. That second figure exists because a reviewer challenged an earlier, stronger claim that the excluded group //was// participant-debriefing notes — it is not, and the page now says only that the channel question is ill-posed for that group. |
| * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** | * The largest matched family, "a named large platform or app store", is a **deliberately loose** rule matching the strings ''google play, app store, play store, apple, google, amazon, facebook, meta, microsoft, marketplace, mozilla, chrome, android''. It over-matches: a paper that used Google Play as a //population source// and disclosed elsewhere lands here. The label on the content page says "a named large platform", not "notified a platform", for that reason. **Do not read 11.7% as "11.7% notified a platform".** |
| * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. | * ''hosting provider / ISP / upstream'' also matches the bare token ''registrar'', which overlaps the WHOIS family. Both are under 1.5% and reported as a ranking, so the overlap is stated rather than resolved. |
| </code> | </code> |
| |
| <wrap todo>**The first published version of that list had 135 of the 144 entries**, because it was built through a shell pipeline that dropped nine rows rather than copied from the script. It is now generated directly from ''report_notifying_websites.mjs --residue'' and asserted to be exactly 144 long. A residue list that is itself truncated is worse than no residue list, and this one was, for three revisions. | <wrap todo>**Filed as deferred work**, not left as a TODO on a wiki page: ''DRAIN_DB=/workspace/drain/work.db python3 /workspace/drain/sync.py add-item --task wiki-measuretheweb --key 'notifying_websites: read the 144 unreviewed campaign candidates''', with the shortlist above and the two open ''security.txt'' questions attached. |
| | |
| | **The first published version of that list had 135 of the 144 entries**, because it was built through a shell pipeline that dropped nine rows rather than copied from the script. It is now generated directly from ''report_notifying_websites.mjs --residue'' and asserted to be exactly 144 long. A residue list that is itself truncated is worse than no residue list, and this one was, for three revisions. |
| |
| The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap> | The list above is the honest coverage boundary of this page. The highest-value follow-up is reading the ~15 of these that look like real campaigns — ''NDSS 2017 are-we-there-yet-on-rpki-s-deployment-and-security'' (a "19% success rate" for ROAlert notifications), ''IMC 2016 weak-keys-remain-widespread-in-network-devices'', ''IMC 2020 behind-closed-doors…'', ''NDSS 2020 complex-security-policy…'', ''WWW 2024 unfiltered…'' ("over 80% of the domains contacted were unresponsive"), ''IEEE-SP 2025 gptracker…'' — and adding them to ''CAMPAIGNS''.</wrap> |
| |
| * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. | * **A table rendered with every figure in the wrong column, and no reviewer caught it.** The trend table's first row labelled its indicator with a backslash-escaped pipe — %%| Notified (''yes''\|''partial'') |%% — but a backslash is **not** a DokuWiki escape for a literal pipe. The renderer treated it as a cell separator, splitting the label in two and shifting all six percentages one column left, so the 2010–2013 figure appeared under //Denominator// and the last bucket rendered empty. Caught by parsing the live page's HTML, where that row came back as %%['Notified (yes\', 'partial)', 'empirical ∧ ethics', '21.3%', …]%% against a seven-column header. Fixed by writing the label as "''yes'' or ''partial''" and avoiding the pipe entirely. **The figures reviewer checked every number against the script and passed the page; every number was right and the table was still wrong.** Checking a wiki page means parsing the rendered HTML, not only the source — the table-cell counts of both pages are now asserted from the live HTML. |
| * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the page's ''<file>'' block is a hand-transcribed copy with em-dashes substituted, and a substitution inside a string literal would have broken it silently. | * **The downloadable code was verified end to end.** ''curl'' on ''/_export/code/practices/notifying_websites?codeblock=0'' → ''python3 -m py_compile'' → an actual run against ''ethz.ch''. What a reader downloads from the page compiles and works. Worth doing because the code block on the page is a transcribed copy with em-dashes substituted for ASCII, and a substitution inside a string literal would have broken it silently. |
| | * **A stale page cache made most citations render as raw keys, and it survived four reviewers and every verification pass above.** After the bibliography was extended and the pages saved, ''https://measuretheweb.org/practices/notifying_websites'' rendered ''[lone2022_sav]'', ''[utz2023_comparing]'', ''[stock2018_didnt]'' and most others as literal bracketed keys, while the two pre-existing keys resolved normally. Nothing was wrong with the source, the bibliography or the keys — DokuWiki was serving a cached render made before the bibliography grew. Fixed by requesting each page once with ''&purge=true''; both then rendered 218 and 98 resolved citations with zero raw keys left. **Add a purge to the end of any run that touches ''literature:bibliography'', and re-check the rendered page after purging rather than before.** The earlier checks in this section counted ''bibtex_citekey'' markers and found 38 on a page with ~60 citations, which should have been the tell and was not. |
| * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected. | * All eight internal links (''Artifacts'', ''Design:Website selection'', ''Literature:Corpus'', ''Practices:Ethics'', ''Practices:Legal enforcement'', ''Practices:Public relations'', ''Statistics:Hypothesis testing'', ''Statistics:Pvalue corrections'') fetched 200 with real content — no red links. ''?do=sitemap'' is cached and still listed the new page as missing, as expected. |
| |
| |
| The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. | The ''crawled ∧ assessed a law'' cross-tab (N=123) was **not** a reviewer finding — it was added afterwards because none of the existing rows described the reader the page is written for, someone who crawled the web and found a legal-compliance violation. It is the one cell where ''no'' (14.6%) is as common as ''partial'', more than three times the corpus-wide 4.3%. The script row and the page row were added together and the pasted output above was regenerated. **This means the reviewers did not see this figure**; it is the newest and least-checked number on the page, though it comes from the same enum and the same script as the rows above it. |
| | |
| | ==== Fable — generic, no checklist, run after the three focused passes ==== |
| | |
| | The most productive of the four. Eleven findings, **all accepted**; five were wrong or self-contradicting claims on the content page. |
| | |
| | ^ Finding ^ Action ^ |
| | | The lead box said "both of the **properly randomised**, control-arm experiments … found no significant effect", which implies {[maass2021_effective]} was not properly randomised. It is: "a covert randomized controlled notification experiment", "full factorial design", with a control group — and it is the strongest positive result on the page. The real distinction is population, not rigour. | **ACCEPTED**, verified against the paper, rewritten to "both randomised, control-arm experiments **on network operators**", and the Maass sentence now names it as a randomised controlled experiment rather than "which also had a control group". This was the single most prominent sentence on the page. | |
| | | The rescoping justification claimed the excluded group "is participant-debriefing notes". Reviewer checked: only 273 of 710. | **ACCEPTED**, verified, and the reason replaced with the correct one — the channel question is ill-posed for a paper that did not notify — with the 38.5% in a footnote. | |
| | | "scanning papers … are **less than half as likely** to leave the question unanswered" is contradicted by the page's own table: 24.0% vs 31.1%. The column that //is// less than half is ''no'' (4.2% vs 9.4%). | **ACCEPTED**, both comparisons now stated with their figures. | |
| | | The opening said "you **have to tell** those 8,000 operators, and **every venue** … expects you to say so" — contradicting the page's own PoPETs bullet, and its own paragraph saying that deciding not to notify is defensible. The footnote also covered only four of the seven corpus venues. | **ACCEPTED**: "decide what to do about", "the major venues", and the footnote now names PoPETs' absence of a requirement and says NDSS and TheWebConf are not addressed. | |
| | | {[stock2018_didnt]}'s 74.4% / 33.3% was labelled "among those who **opened** the report" in two places. It is among those who **viewed** it — and the page's own funnel insists opened and viewed are distinct stages. | **ACCEPTED**, both corrected to //viewed//. | |
| | | 22 + 16 + 144 = 182 ≠ 179 on the content page, with the explanation only here. | **ACCEPTED**: the page now says 13 reasoned rejections inside the regex plus three carried over from the wider first-pass scan, totalling 16. The 5,859-records / 5,869-files gap is also now explained in the same sentence. | |
| | | The rates table's intro says it shows campaigns "that reported a quotable outcome", but the {[munteanu2025_catch22]} row has no figure — the stated exclusion criterion. | **ACCEPTED**, the exception is now stated on the content page, not only here. | |
| | | "**Three** of those channels are machine-queryable" is wrong twice: RDAP is not one of the channels in the table, and PeeringDB is machine-queryable but is not in the script. | **ACCEPTED**, rewritten to two per-domain channels plus RDAP, with PeeringDB noted as per-AS. | |
| | | The provenance itself conceded that neither the script docstring nor the page says why RIPEstat was chosen over Abusix, and left it unfixed. | **ACCEPTED**, the docstring now says so — Abusix is better at scale, RIPEstat is what runs with only the standard library and is therefore testable here. | |
| | | This provenance page said "Four reviewers … logged below" with three logged, and "**Two** judgement calls" followed by three bullets. The ''head -143'' / ''tail -n +144'' instruction had gone stale when the script grew. | **ACCEPTED**, all three fixed; the split point is now derived rather than hard-coded, since it moves every time a table row is added. | |
| | | Voice: the only first-person singular on the site ("no cadence **I** could verify"); //survival// used without saying lower is better; "That is not negligence" asserts a motive the data does not show; "The one flat line" for a series that rose tenfold then declined. | **ACCEPTED**, all four reworded. | |
| | |
| | Also confirmed by this pass and not changed: the page answers its own question and the top-to-bottom ordering works; no dead-weight sections; all DokuWiki constructs well-formed in the current revision; the rates table's 17 rows reconcile with this page's accounting; every corpus figure matches a fresh script run; the embedded script runs as documented; and the voice, while denser in bolding and aphorism than the older hand-written pages, sits inside the range of the site's newer corpus-backed pages. |
| |
| ==== Nothing was rejected outright ==== | ==== Nothing was rejected outright ==== |
| |
| Every substantive finding across the three passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page: | Every substantive finding across all four passes was accepted. Two are worth separating from the rest because they say something about the process rather than the page: |
| |
| * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading. | * The **figures** pass found no figure error at all — every corpus number checked out — but found the one real **code** defect. Re-running beats re-reading. |
