Working log for Ethics. Every query with its population and denominator, the report script with its unedited output, both folds with their residue printed in full, every quote with its verification verdict, the external sources with how each was checked, and what could not be established. Corpus-wide caveats — the venue scope, the selection funnel, the provisional 2025–2026 slice — are on Corpus and are not repeated here.

Voice here is a working log, not prose. It is read by somebody checking a number.

No ~~DISCUSSION~~ block: comments belong on the content page, following the convention set by the earlier provenance pages.

Every figure on the page names one of these. None of them is “of 5,859 papers”.

crawlOnly is this page's own construction and is not one of the corpus's named populations. It exists because the page's central claim is a contrast between two study kinds, and crawled alone conflates them: 149 of the 1,120 crawling papers also recruited participants, and those 149 report at nearly the human-subjects rate. Without splitting them the crawl figure reads 27.7% instead of 21.3%, and the contrast the page is about is diluted by a third.

ethics is an object, not an array, and it can be null. 4,472 of the 5,118 empirical papers carry one; 646 do not. This matters more than it sounds:

• scripts/site_queries.mjs –page ethics reports reviewOutcome against 5,118 but reports notifiedAffectedParties, regulatorContact and robotsTxt against 4,472. Mixing the two moves every share by about 13%.
• report_ethics.mjs therefore always prints no-ethics-record as its own visible row and never merges it into none-mentioned. Sections A, B and F use 5,118 or 1,120 (the full population, with the absent record shown); section H uses 4,472 and says so in the column header.

const OUTCOMES = ['approved', 'exempt', 'not-required', 'sought-outcome-unstated',
                  'explicitly-discussed-no-review'];
const statesReview = (p) => p.ethics !== null && OUTCOMES.includes(p.ethics.reviewOutcome);
const approved     = (p) => p.ethics !== null && ['approved','exempt'].includes(p.ethics.reviewOutcome);

explicitly-discussed-no-review is counted as stated. A paper that says “we considered the ethics and did not seek review” has said something checkable, which is what this page measures. It is 318 of 5,118 empirical papers, and every table that uses statesReview also prints the approved/exempt column so a reader who disagrees can subtract. The page says this in its limitations section rather than burying it here.

scripts/report_ethics.mjs — every figure on the page with its denominator, plus both folds' residue, the full-text sweeps, and a block listing every non-corpus number so check_page_numbers.mjs can run whole-page.

node scripts/report_ethics.mjs                  # plain text (below)
node scripts/report_ethics.mjs --wiki           # DokuWiki tables
node scripts/report_ethics.mjs --list <key>     # the papers behind a figure
node scripts/report_ethics.mjs --quotes <re>    # ethics evidence quotes matching a regex

–list keys: crawlonly-approved, crawlonly-silent, robots-stated, no-review-discussed, menlo, nhs, residential-proxy, aup, no-review-body, tos.

Unedited output, 2026-08-18:

report_ethics.txt

# practices/ethics — dataset queries
Corpus: 7 venues, 2010–2026, 5859 papers with full text.
2025 is thin at the edges and 2026 is provisional (CCS/IMC 2026 not yet held; IEEE S&P and WWW 2026 abstracts absent from OpenAlex).
 
Population       Definition                                                   Papers
---------------  -----------------------------------------------------------  ------
`empirical`      measured the world as it is                                  5118
`crawled`        `crawlConfig` fired or `studyTypes` has automated-web-crawl  1120
`humanSubjects`  `participants[]` fired                                       1357
`crawlOnly`      crawled **and** recruited nobody                             971
 
## A. Ethics review outcome, of the 5,118 empirical papers
 
`ethics.reviewOutcome`            Papers  Share of 5118  Sentinel?
--------------------------------  ------  -------------  ---------
`approved`                        994     19.4%
`exempt`                          157     3.1%
`not-required`                    179     3.5%
`sought-outcome-unstated`         80      1.6%
`explicitly-discussed-no-review`  318     6.2%
`none-mentioned`                  2744    53.6%          yes
`no-ethics-record`                646     12.6%          yes
 
**Stated something about review: 1728 of 5118 (33.8%).**
Of those, 1151 (66.6% of the 1728) report an approval or an exemption; the rest state a review was not required, was sought with no outcome given, or was explicitly discussed and not obtained.
 
## B. Who states a review outcome: participants vs no participants
 
Population                                      Papers  States a review outcome  Approved or exempt  Silent
----------------------------------------------  ------  -----------------------  ------------------  ------------
`humanSubjects` — recruited participants        1357    1022 (75.3%)             896 (66.0%)         335 (24.7%)
`crawled` — ran a crawl (any)                   1120    310 (27.7%)              188 (16.8%)         810 (72.3%)
  …of which also recruited participants         149     103 (69.1%)              95 (63.8%)          46 (30.9%)
  …**`crawlOnly` — crawled, recruited nobody**  971     207 (21.3%)              93 (9.6%)           764 (78.7%)
`empirical` — all empirical papers              5118    1728 (33.8%)             1151 (22.5%)        3390 (66.2%)
 
"Silent" = `none-mentioned` plus papers where no `ethics` record was extracted at all.
 
## C. Reporting a review outcome, over time
 
Window      `empirical` N  states  %      `humanSubjects` N  states  %      `crawlOnly` N  states  %
----------  -------------  ------  -----  -----------------  ------  -----  -------------  ------  -----
2010–2013   460            47      10.2%  84                 24      28.6%  89             6       6.7%
2014–2017   718            155     21.6%  155                91      58.7%  155            23      14.8%
2018–2021   1272           378     29.7%  329                230     69.9%  262            42      16.0%
2022–2024   1649           681     41.3%  486                414     85.2%  299            86      28.8%
2025–2026*  1019           467     45.8%  303                263     86.8%  166            50      30.1%
 
\* 2025–2026 is provisional; see the corpus caveat above.
Gap between the two study kinds, latest complete window (2022–2024): 85.2% vs 28.8%.
 
## D. Reporting a review outcome, by venue (empirical papers)
 
Venue            Empirical papers  States a review outcome  Share  Cites the Menlo Report  Share
---------------  ----------------  -----------------------  -----  ----------------------  -----
PoPETs           477               259                      54.3%  14                      2.9%
USENIX Security  1226              506                      41.3%  76                      6.2%
IEEE S&P         646               246                      38.1%  26                      4.0%
IMC              625               208                      33.3%  46                      7.4%
NDSS             586               189                      32.3%  29                      4.9%
CCS              818               197                      24.1%  24                      2.9%
TheWebConf       740               123                      16.6%  10                      1.4%
 
## E. What kind of body reviewed it (of the papers that name one)
 
1347 of 5118 empirical papers name a review body, in **595 distinct strings**.
The fold is single-label and ordered (most specific family first); 2 strings matched nothing, and 2 are a sentinel written into the free-text field ("not stated") and belong to no family.
 
Kind of body                                                   Papers  Share of 1347 naming one
-------------------------------------------------------------  ------  ------------------------
A US-style Institutional Review Board (IRB)                    986     73.2%
An ethics committee or ethical review board (non-IRB wording)  241     17.9%
The institution, named only as such                            47      3.5%
Data protection officer or data protection office              28      2.1%
Legal department, general counsel or compliance office         25      1.9%
A regulator or government body                                 7       0.5%
An internal company or partner review                          5       0.4%
An external or community safety board                          4       0.3%
 
Unmapped residue, in full:
  - internal review process
  - ethics and data protection review
 
Papers naming a network operations / services / management department among the reviewing bodies: 7.
  IMC/2014/forced-perspectives-evaluating-an-ssl-trust-enhancement-at-scale  —  university network services department
  USENIX/2016/website-targeted-false-content-injection-by-network-operators  —  networks' administration teams
  NDSS/2019/the-use-of-tls-in-censorship-circumvention  —  IRB and institutional networking and IT staff
  IMC/2024/bounce-in-the-wild-a-deep-dive-into-email-delivery-failures-from-a-large-email-s  —  network management department and regulatory authority of Coremail
  NDSS/2024/on-precisely-detecting-censorship-circumvention-in-real-world-networks  —  campus institutional review board (IRB) and network operations staff
  NDSS/2024/understanding-the-implementation-and-security-implications-of-protective-dns-services  —  network management department
  NDSS/2025/hades-attack-understanding-and-evaluating-manipulation-risks-of-email-blocklists  —  network management department
 
## F. robots.txt (population: the 1,120 papers that ran a crawl)
 
`ethics.robotsTxt`  Papers  Share of 1120
------------------  ------  -------------
`respected`         15      1.3%
`ignored`           9       0.8%
`discussed`         29      2.6%
`not-stated`        939     83.8%
`no-ethics-record`  128     11.4%
 
**Says anything at all about robots.txt: 53 of 1120 (4.7%).**
 
How the robots.txt reporting rate compares with the other low-reporting enums (each against its own population,
and counting `not-applicable` as a sentinel, which is how OVERVIEW.md counts them):
Field                              Population N  Stated  Share
---------------------------------  ------------  ------  -----
`ethics.robotsTxt`                 1120          53      4.7%
`crawlConfig.headless`             1120          140     12.5%
`humanAnnotation.agreementMetric`  3318          512     15.4%
`crawlConfig.statefulness`         1120          219     19.6%
`crawlConfig.consentAction`        1120          349     31.2%
`ethics.reviewOutcome`             5118          1728    33.8%
Full-text cross-check: the literal string `robots.txt` appears anywhere in 55 of 1120 crawling papers (4.9%); 0 full-text files were missing.
The sweep is the upper bound (it fires on a reference-list title too); the schema figure is the lower one. Both say the same thing.
 
## G. What crawling papers say they did to limit harm
 
Like-for-like on the crawlOnly population: 585 of 971 (60.2%) state a mitigation, against 207 of 971 (21.3%) that state a review outcome.
697 of 1120 crawling papers (62.2%) give a free-text harm-mitigation step. The other 423 say nothing — which is not the same as doing nothing, and is not counted as such anywhere below.
 
The fold is **multi-label** — one sentence routinely names three steps — so shares do not sum to 100%.
 
Mitigation family                                                           Papers  Share of 697 that state one
--------------------------------------------------------------------------  ------  ---------------------------
Data minimisation — collecting less than you could                          180     25.8%
Rate limiting, pacing and load control                                      164     23.5%
Disclosure and notification of affected parties                             123     17.6%
Anonymisation, pseudonymisation and hashing                                 108     15.5%
Staying non-intrusive: no exploitation, no tampering, no logins             104     14.9%
Test accounts, synthetic identities and the researchers’ own assets         92      13.2%
Isolation: sandboxes, VMs, containers, lab-only deployment                  59      8.5%
Access control, encryption and secure storage                               53      7.6%
Deletion and retention limits                                               50      7.2%
Avoiding financial harm to operators and advertisers                        33      4.7%
Consent, opt-in and opt-out mechanisms                                      28      4.0%
Screening and handling of harmful or illegal content                        25      3.6%
Respecting robots.txt, terms of service and acceptable-use policies         17      2.4%
Formal review, legal advice and named legal regimes                         17      2.4%
Identifying the crawler and offering an operator an opt-out (hand-decided)  10      1.4%
 
Unmapped residue: 28 of 697 (4.0%). In full:
  - Stripped JavaScript and rewrote links to prevent malicious traffic reaching other sites.
  - Closed experimental Twitter accounts after 48 hours.
  - Advertised legitimate Chinese e-commerce sites and filtered search engines and bots.
  - Limited orders to three per day per campaign to reduce detection.
  - WHOIS queries were limited to a small percentage of domains.
  - Collected URLs without automated visits and avoided sending automated traffic to monetization services.
  - Stored only raced sessions and at most their last 30 packets.
  - They state that the attacks cannot automatically compromise devices at scale.
  - They decided not to contact affected app publishers about findings.
  - Used bucketing, daily writes, and rounding counters to protect anonymity.
  - Sent one email per provider and did not follow up.
  - Used static screening to focus later dynamic analysis on selected apps.
  - Limited clicks to fewer than 15 per app and advertiser visits to fewer than 25 on average.
  - Only scraped publicly accessible pages; landmarks could be removed and webcams were never accessed or controlled.
  - One visit per site per OS, with visits at different times and no concurrent visits.
  - Collected alerts were transferred to an EDR client for manual verification.
  - Registered unintended domains to prevent attackers from exploiting them.
  - Used anonymous expressions and did not focus on particular forum users.
  - Avoided residential IPs and used university addresses instead.
  - Avoided explicitly clicking ads; discussed ecosystem harms.
  - Only downloaded firmware images released to the general public.
  - Tested each API once, used 10 kB labeled beacons, uploaded 447 images totaling 4.4 MB.
  - They discuss potential sensitive-information leakage from sharing frontend URLs.
  - Contacted each address once, offered opt-out, explained recruitment, and stopped GitHub recruitment.
  - Avoided highlighting individual government strategies or security weaknesses.
  - Artifact warns that aggressive eviction may destabilize or crash systems.
  - Did not report sites to preserve observation of scam operations.
  - Crawler visits used a fresh browser and a dynamic pool of IPs.
 
Crawler identification and opt-out — a hand-decided family, because a bare
"opt-out" fires on participant and notification-recipient opt-out too.
Sweep and hand map agree on all 21 candidates.
 
Verdict      Papers  Share of 21 candidates
-----------  ------  ----------------------
operator     10      47.6%
participant  4       19.0%
telemetry    4       19.0%
recipient    2       9.5%
evasion      1       4.8%
 
**Directed at the party being measured: 10 of 697 papers that state a mitigation (1.4%).** This is the figure the page uses.
 
Every candidate with its verdict and deciding text:
  [participant] CCS/2014/private-by-design-advertising-meets-the-real-world
                 Opt-in participation and an easy opt-out mechanism — the opt-out is for the users of the advertising system, not for site operators.
  [telemetry  ] IEEE-SP/2015/the-attack-of-the-clones-a-study-of-the-impact-of-shared-code-on-vulnerability-p
                 Used opt-in telemetry and excluded user-identifiable information.
  [operator   ] CCS/2020/deploying-android-security-updates-an-extensive-study-involving-manufacturers-ca
                 did not obfuscate user agents or IP addresses.
  [operator   ] CCS/2021/out-of-sight-out-of-mind-detecting-orphaned-web-pages-at-internet-scale
                 customized opt-out User-Agent.
  [recipient  ] USENIX/2021/effective-notification-campaigns-on-the-web-a-matter-of-trust-framing-and-suppor
                 honored opt-outs — from the notification campaign, not from the crawl.
  [evasion    ] PETS/2022/setting-the-bar-low-are-websites-complying-with-the-minimum-requirements-of-the
                 Used legitimate User-Agent strings ... avoided detectable automation frameworks. This is the opposite of identifying yourself.
  [telemetry  ] USENIX/2022/online-website-fingerprinting-evaluating-website-fingerprinting-attacks-on-tor-i
                 opt-in entry circuits — consent from relay operators contributing data, not from a measured site.
  [operator   ] IMC/2023/thou-shalt-not-reject-analyzing-accept-or-pay-cookie-banners-on-the-web
                 Dedicated machines, informative rDNS, measurement website, and blocklisting option — the full set.
  [recipient  ] PETS/2023/comparing-large-scale-privacy-and-security-notifications
                 opt-out from the notification campaign.
  [participant] PETS/2023/privacy-rarely-considered-exploring-considerations-in-the-adoption-of-third-part
                 offered opt-out, explained recruitment — study recruitment.
  [telemetry  ] USENIX/2023/rods-with-laser-beams-understanding-browser-fingerprinting-on-phishing-pages
                 participation was explicitly opt-in — the telemetry contributors.
  [operator   ] PETS/2024/a-black-box-privacy-analysis-of-messaging-service-providers-chat-message-process
                 warnings, opt-out — directed at the messaging providers under measurement.
  [participant] PETS/2024/what-does-it-mean-to-be-creepy-responses-to-visualizations-of-personal-browsing
                 participants could opt out or uninstall.
  [operator   ] CCS/2025/in-the-dom-we-trust-exploring-the-hidden-dangers-of-reading-from-the-dom-on-the
                 identifying header, opt-out link.
  [operator   ] IEEE-SP/2025/gptracker-a-large-scale-measurement-of-misused-gpts
                 opt-out offered to the measured GPT authors.
  [operator   ] PETS/2025/intractable-cookie-crumbs-unveiling-the-nexus-of-stateful-banner-interaction-and
                 Dedicated identifiable measurement machines; stakeholders could opt out.
  [participant] IMC/2025/where-in-the-world-are-my-trackers-mapping-web-tracking-flow-across-diverse-geog
                 opt-outs offered to the volunteers, per-website.
  [operator   ] NDSS/2025/hades-attack-understanding-and-evaluating-manipulation-risks-of-email-blocklists
                 opt-out information published for the measured mail operators.
  [operator   ] USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale
                 opt-out contact.
  [operator   ] NDSS/2026/one-email-many-faces-a-deep-dive-into-identity-confusion-in-email-aliases
                 opt-out notices.
  [telemetry  ] IEEE-SP/2021/trouble-over-the-air-an-analysis-of-fota-apps-in-the-android-ecosystem
                 Used anonymized, opt-in telemetry.
 
## H. Disclosure and regulator contact (context only — see the two sibling pages)
 
`ethics.notifiedAffectedParties`  Papers  Share of 4472 empirical papers with an ethics record
--------------------------------  ------  ----------------------------------------------------
`yes`                             1636    36.6%
`partial`                         524     11.7%
`no`                              194     4.3%
`not-applicable`                  1001    22.4%
`not-stated`                      1117    25.0%
 
`ethics.regulatorContact`  Papers  Share of 4472
-------------------------  ------  -------------
`yes`                      147     3.3%
`no`                       1982    44.3%
`not-stated`               2343    52.4%
 
## I. Full-text sweeps (upper bounds — a sweep fires on a reference-list title too)
 
Probe                                            Papers  Population        Share
-----------------------------------------------  ------  ----------------  -----
`Menlo Report`                                   225     empirical (5118)  4.4%
`Belmont Report`                                 36      empirical (5118)  0.7%
`IRB` anywhere                                   987     empirical (5118)  19.3%
a "not human subjects" determination             79      empirical (5118)  1.5%
`ACM Code of Ethics`                             6       empirical (5118)  0.1%
`robots.txt`                                     55      crawled (1120)    4.9%
terms of service / terms of use                  129     crawled (1120)    11.5%
`acceptable use policy`                          5       crawled (1120)    0.4%
`residential prox*`                              25      empirical (5118)  0.5%
data donation / donated data                     24      empirical (5118)  0.5%
opt-out signalling (rDNS, info page, blocklist)  63      crawled (1120)    5.6%
`Common Rule` or `45 CFR 46`                     32      empirical (5118)  0.6%
`Tor Research Safety Board`                      7       empirical (5118)  0.1%
a named ethics/REC committee of the venue        39      empirical (5118)  0.8%
**"our institution has no review body"**         20      empirical (5118)  0.4%
 
Of the 20 papers stating their institution has no review body: 16 recruited participants, 2 ran a crawl, and 10 name the Menlo Report.
 
Menlo Report citations over time (population: empirical papers in the window):
Window      Empirical papers  Cite the Menlo Report  Share
----------  ----------------  ---------------------  -----
2010–2013   460               0                      0.0%
2014–2017   718               7                      1.0%
2018–2021   1272              32                     2.5%
2022–2024   1649              84                     5.1%
2025–2026*  1019              102                    10.0%
 
## J. Sensitivity check: does the reporting gap survive dropping posters and short records
 
Population       N (all)  States review (all)  N (trimmed)  States review (trimmed)  Dropped
---------------  -------  -------------------  -----------  -----------------------  -------
`empirical`      5118     33.8%                4920         34.8%                    198
`crawlOnly`      971      21.3%                948          21.6%                    23
`humanSubjects`  1357     75.3%                1322         76.4%                    35
 
Trimmed = posters (slug `poster-*` or title `Poster:`) and records of 4 pages or fewer removed.
 
## K. Figures on the page that do NOT come from this corpus
 
Every number below is external and is cited on the page with its source and the date it was read.
They are listed here so a whole-page number audit does not flag them as unsourced corpus figures.
 
  Ramulu et al., CCS 2025 (in the corpus as a paper, but these are ITS figures, not ours):
    1154 papers reviewed; 839 with no ethics discussion; 315 with one; 270 with a dedicated
    section; 45 in the body only; 36 naming the Menlo Report; 251 mapped to Menlo principles;
    159 beneficence; 110 confidentiality; 89 informed consent; 79 compensation; 56 law;
    18 justice; 11 vulnerable persons; 257 mentioning vulnerability disclosure; 116 human-subject
    papers, of which 102 discuss ethics and 90 have a dedicated section; 17 of those 116 do not
    mention a review board; 52 non-human-subject papers that do; 24 interviewees.
 
  Load and cost figures quoted from individual papers, each verified against the paper by
  scripts/eth_quotecheck.mjs (all 34 quotes exact):
    2.7 URLs per second (Zhang et al., NDSS 2014); a 1 minute delay between page loads
    (Bashir et al., USENIX Security 2016); no more than ten queries per second (Almansoori
    et al., PoPETs 2022); no more than 4 visits per day (Zeng et al., IMC 2021); 256 parallel
    workers, pausing on HTTP 429 and HTTP 503 (Amos et al., TheWebConf 2021); about $400 in
    impressions (Datta et al., PoPETs 2015); USD 1.12 maximum and USD 0.04 average per landing
    domain (Subramani et al., IMC 2020).
 
  Hantke et al., IEEE S&P 2024 (its own study design, not ours): 23 interviews, 119 survey
    respondents, 5 scenarios; and the figure it reports for IEEE S&P 2022 — a Reject
    recommendation for 2 of 67 flagged papers out of 1,006.
 
  Venue policies, standards and legal instruments, each cited on the page with its URL and the
  date it was read (2026-08-18). The bare numbers they contribute, so a whole-page number audit
  can account for them:
    45 CFR 46.102, 46.104 (and the subsection numbers 1, 4, 6 inside them); the year 2012 for
    the Menlo Report and 2013 for its companion; GDPR Article 89; EU AI Act Article 2, and its
    paragraphs 6 and 8; DSA Article 40, its 49 applications received by 19 May 2026, and the
    delegated act of 2 July 2025; Van Buren v. United States, 593 U.S. 374 (2021); hiQ Labs v.
    LinkedIn, No. 17-16783 (9th Cir., 18 April 2022); Jisc MF-POL-007 Issue 5.1 and its 1
    working day of notice; ACM policy approved 15 August 2021; HTTP statuses 200 and 403
    encountered while fetching those sources; Mozilla Rally archived 29 July 2024; Citizen
    Browser last post October 2022; DataSkop 2022-2023; Tracking Exposed ended 2023;
    Bright Data residential KYC for zones created after 7 July 2026; arXiv:2511.10408 and its
    hiQ Labs v. LinkedIn district-court docket 3:17-cv-03301-EMC, entry 406, the $500,000
    consent judgment of 8 December 2022; Regulation (EU) 2016/679 Art. 89(1) and 89(2);
    Regulation (EU) 2024/1689 Art. 2(6), 2(8) and Recital 25; Swiss HRA SR 810.30 Art. 2;
    the European Commission ethics self-assessment guidance version 2.0 of 13 July 2021;
    HTTP 301 redirects on the DHS and Jisc pages; Hantke et al. interviewee composition
    (9 legal experts, 10 operators, 5 REC members); Ramulu et al. "all but 17" read as 99 of
    116 or 85 of 102; the Menlo Report companion of 2013; Azure dropping pre-approval in
    June 2017; Singh et al. 22 volunteers across 23 countries in 2025; Chehade et al. 100 live
    web applications; and the conference-cycle shorthands 26 and 27 (as in "USENIX Security
    '27"), which are years, not counts;
    16 interviewees; the rhetorical 100,000 sites in the introduction; Wu et al. NDSS 2025 own
    figures (1,407 scanner IPs, 28 honeypots, one year); Ramulu et al. 257 disclosure papers;
    Hantke et al. 100 web applications tested by Chehade et al.; and the derived ~1,038
    (1,154 - 116) non-human-subject papers in Ramulu et al.

scripts/harm_fold.mjs. ethics.harmMitigation is a free-text, at-most-20-word summary of the steps a paper says it took. Two design decisions, both load-bearing:

• It is not a name field. There is no canonical string to converge on, so a spelling fold (the name_fold.mjs approach used for tools and locations) is meaningless here. What is stable across two readings of the same sentence is the *kind* of step, so the fold maps a sentence to a set of mitigation families.
• It is multi-label. “Rate-limited requests, avoided off-limit information, and anonymized collected data” is three steps in one string. Single-labelling it would throw two away. So a string can match several families, each family is counted in papers, and the shares on the page do not sum to 100% — which the page states above the table.

A paper with harmMitigation === null is never counted as “took no steps”. It is counted as “did not say”, which is a different and larger set (423 of the 1,120 crawling papers).

14 regex families, ordered but not exclusive, plus one hand-decided family (below) that has no regex at all. 14 self-tests, each a real string from the corpus:

14 fold self-tests pass

Residue: 28 of 697 papers (4.0%), printed in full in section G of the report output above. Reading it: most of the 27 are genuinely idiosyncratic single-instance steps (“Registered unintended domains to prevent attackers from exploiting them”, “Advertised legitimate Chinese e-commerce sites and filtered search engines and bots”) rather than a family the fold is missing. Two near-misses were left unmapped on purpose rather than widened into: “Avoided residential IPs and used university addresses instead” (a vantage-point choice framed as harm mitigation) and “Crawler visits used a fresh browser and a dynamic pool of IPs” (the opposite choice, also framed as harm mitigation). A family for “vantage-point hygiene” would have two members pointing in opposite directions.

The first version of the anonymisation regex ended …|anonymised|/i — a trailing empty alternative, which matches the empty string, so every non-null harmMitigation string scored as anonymisation. The self-test caught it because two test cases asserted families that string did not contain and got anonymisation back instead. Recorded here because the failure is silent in every other respect: the family counts looked plausible (106 papers) and the residue went down, which is the direction that normally reads as an improvement.

The first complete run had a residue of 162 of 697 (23.2%). Reading the residue showed almost all of it belonged to families that already existed — the regexes were written from the first 60 strings and did not generalise (“Visited each page once”, “one request per URL per day”, “Time-gapped measurements” are all rate limiting; “Did not retain IP addresses” is data minimisation). Broadening the fourteen families cut it to 4.0% with no new family except one: crawler-identification, split out of consent-and-optout because identifying your crawler and offering an operator an opt-out is a different act from obtaining a participant's consent, and it is the one the page argues for. That family then turned out to need a hand map rather than a regex — see below.

The page publishes a count for identifying the crawler and offering an operator an opt-out and builds an argument on it (“the cheapest credible thing on the list”). The first version was a regex whose broadest alternative was a bare opt-?out, and it reported 17 of 697.

Reviewer 1 pulled the 17 and found that 7 of them recruited human participants, quoting four whose text is unambiguously about a study participant's right to withdraw. That is not the same act as telling a site operator who is crawling them.

The fix was to remove the family from the regex list entirely. What replaced it:

• CRAWLER_ID_CANDIDATES — a loose sweep (opt-out, opt-in, rDNS, measurement website, identifying header, legitimate User-Agent, …), deliberately over-broad.
• CRAWLER_ID — a hand verdict for every candidate, with the deciding text, in one of five categories: operator (directed at the party being measured), participant, telemetry (opt-in consent from people whose client data is reused), recipient (opt-out from a notification campaign — that is disclosure hygiene and belongs to Notifying websites), and evasion (a “legitimate” User-Agent chosen so the crawler is not identifiable, which is the opposite act).
• report_ethics.mjs fails loudly if the sweep returns a paper the map does not cover, or the reverse.

That guard fired on the first run, surfacing four opt-in-telemetry papers that neither the tightened regex nor the reviewer had found. Final split of the 21 candidates: 10 operator, 4 participant, 4 telemetry, 2 recipient, 1 evasion.

The published figure moved 17 → 10, and foldHarm() now takes an optional paper key so the table row comes from the hand map rather than from a string match. Without the key it simply does not emit that family — it is never guessed.

scripts/body_fold.mjs. ethics.reviewBody is free text naming who reviewed the study: 1,347 papers, 595 distinct strings. Single-label and ordered, most specific family first, because the question the page asks is “what kind of body signs a web measurement off” — a student in a department with no IRB needs to know what the alternatives are called.

Order matters and was set by the failures:

• dpo and regulator before legal and irb, because “data protection officer” and “CNIL” are more specific than the words around them.
• irb before ethics-committee, because “institutional ethics review board (IRB)” should count as an IRB.
• institution-unspecified last, because it is a catch-all containing institution|universit|department|board|office and would otherwise swallow everything.

Two keyed exceptions, checked before the families. “Data Protection Inspectorate of Estonia” is a regulator, not a DPO, and the string for the first contains the string for the second. “Privacy and Security Board of Politecnico di Torino” is a university body, not a company one — the company family exists to catch corporate “privacy review board” language and fired on it first; caught in review, and it moved two published rows by one paper each. One sentinel rule: the extractor occasionally writes “not stated” into this free-text field; that is a non-answer, reported apart from both the families and the residue (2 papers).

17 self-tests:

16 body-fold self-tests pass

Residue: 2 of 1,347 strings — “internal review process” and “ethics and data protection review”. Both are real review bodies that no family names; neither was worth a rule.

The page says 4.7% is “the lowest reporting rate of any field in the extraction behind this site”. That is a claim about fields this page does not otherwise touch, so section F of the report computes them rather than quoting them, each against its own population and counting not-applicable as a sentinel:

All six reproduce data/extract/OVERVIEW.md's Where the field goes quiet table exactly, which is a useful independent check that this script's sentinel handling matches the corpus's own.

A first version of this table did not reproduce it, and the reason is the one this site keeps re-learning: it tested !== 'not-stated' instead of isSentinel(…), so not-applicable counted as an answer. That put consentAction at 52.2% instead of 31.2% and statefulness at 21.1% instead of 19.6% — plausible-looking numbers that would have been published with no way to notice. The fix was to use lib.mjs's isSentinel everywhere, which is why it exists.

Section I of the report. Every sweep is an upper bound: it matches the reference list and the related-work section as well as the methods, so a hit is not a claim that the paper did the thing.

Three of them are used on the page as evidence, and each is paired with a schema figure so the two can be read against each other:

Sweeps run but not published on the page, with why:

The page says “Seven papers in the corpus name a network operations, network services or network management department among the bodies that reviewed the study”. That is not a fold family — it came from reading the reviewBody residue and noticing a recurring institution kind that the family list treats as institution-unspecified. Section E of the report lists all seven with their exact strings. Two of the seven name an IRB as well, so this is “among the bodies”, not “instead of a board”, and the page says it that way.

An earlier draft said “four”, from an eyeball count of the reviewBody frequency table. The report now computes it.

scripts/eth_quotecheck.mjs checks every sentence the page quotes from a corpus paper against the rendering the extractor read, with the ligature / line-break-hyphen / smart-quote normalisation from le_quotecheck.mjs, and falls back to a pypdf re-extraction when paper.cols.txt fails — the two renderings defeat different sentences on two-column ACM PDFs.

exact    100%  [cols] IMC/2017/fifteen-minutes-of-unwanted-fame-detecting-and-characterizing-doxing
exact    100%  [cols] CCS/2020/censored-planet-an-internet-wide-longitudinal-censorship-observatory
exact    100%  [cols] IMC/2022/spfail-discovering-measuring-and-remediating-vulnerabilities-in-email-sender-val
exact    100%  [cols] IMC/2022/spfail-discovering-measuring-and-remediating-vulnerabilities-in-email-sender-val
exact    100%  [cols] USENIX/2017/i-have-no-idea-what-im-doing-on-the-usability-of-deploying-https
exact    100%  [cols] NDSS/2025/hades-attack-understanding-and-evaluating-manipulation-risks-of-email-blocklists
exact    100%  [cols] USENIX/2013/enabling-fine-grained-permissions-for-augmented-reality-applications-with-recogn
exact    100%  [cols] NDSS/2026/should-i-trust-you-rethinking-the-principle-of-zone-based-isolation-dns-bailiwick-checking
exact    100%  [cols] IEEE-SP/2024/digital-security-a-question-of-perspective-a-large-scale-telephone-survey-with-f
exact    100%  [cols] CCS/2017/why-do-developers-get-password-storage-wrong-a-qualitative-usability-study
exact    100%  [cols] PETS/2024/internet-users-willingness-to-disclose-biometric-data-for-continuous-online-acco
exact    100%  [cols] NDSS/2014/dspin-detecting-automatically-spun-content-on-the-web
exact    100%  [cols] USENIX/2016/tracing-information-flows-between-ad-exchanges-using-retargeted-ads
exact    100%  [pdf] PETS/2022/a-global-survey-of-android-dual-use-applications-used-in-intimate-partner-survei
exact    100%  [pdf] IMC/2021/polls-clickbait-and-commemorative-2-bills-problematic-political-advertising-on-n
exact    100%  [cols] WWW/2021/privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset
exact    100%  [cols] WWW/2021/privacy-policies-over-time-curation-and-analysis-of-a-million-document-dataset
exact    100%  [cols] PETS/2015/automated-experiments-on-ad-privacy-settings
exact    100%  [cols] IMC/2020/when-push-comes-to-ads-measuring-the-rise-of-malicious-push-advertising
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [cols] IEEE-SP/2024/where-are-the-red-lines-towards-ethical-server-side-scans-in-security-and-privac
exact    100%  [pdf] IMC/2025/where-in-the-world-are-my-trackers-mapping-web-tracking-flow-across-diverse-geog
exact    100%  [pdf] IMC/2025/where-in-the-world-are-my-trackers-mapping-web-tracking-flow-across-diverse-geog
exact    100%  [cols] NDSS/2025/revealing-the-black-box-of-device-search-engine-scanning-assets-strategies-and-ethical-consideration
exact    100%  [cols] CCS/2025/ethics-in-computer-security-research-a-data-driven-assessment-of-the-past-the-pr

34 quotes: 34 exact, 0 partial (>=60% of 5-word windows), 0 below threshold.

34 of 34 exact. Four quotes across three papers needed the pypdf fallback (Almansoori et al. PoPETs 2022; Zeng et al. IMC 2021; Singh et al. IMC 2025, two quotes), which is the expected rate and the reason the fallback exists.

External quotes — venue calls for papers, statutes, court opinions, vendor documentation, network AUPs — are not in this checker. They are cited on the page with the source URL and the date they were read, and the load-bearing ones were re-fetched by hand; see External sources below.

Both were headed for the page's rate-limiting list and were cut. Recorded because the failure mode — a plausible harmMitigation summary that the source does not support — is invisible unless you open the paper.

Five papers are attributed by author name on the page (in the rate-limiting table). All five were checked with scripts/check_attributions.mjs against the bibliography: 5/5 match.

One was wrong in the draft and is recorded here because it is the exact failure that guard exists for: the PoPETs 2022 dual-use-apps paper was first attributed to “Chatterjee et al.” — Rahul Chatterjee is its last author. First author is Majed Almansoori, confirmed from Crossref for DOI 10.56553/popets-2022-0102. PoPETs index records in this corpus carry no authors, so the name had to be fetched rather than read.

Authors fetched from a landing page rather than the index, because PETS and USENIX records have neither authors nor DOIs:

The page leans on [1Ramulu, Harshini Sri; Schmitt, Helen; Rerich, Bogdan; Rodriguez, Rachel Gonzalez; Kohno, Tadayoshi; Acar, Yasemin (2025): "Ethics in Computer Security Research: A Data-Driven Assessment of the Past, the Present, and the Possible Future", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] for an independent count. Its figures are its own, not this corpus's, and they are listed in section K of the report so a whole-page number audit does not treat them as ours: 1,154 papers hand-read across CCS, IEEE S&P, NDSS and USENIX Security 2024; 839 with no ethics discussion; 315 with one; 270 with a dedicated section; 45 in the body only; 36 naming the Menlo Report; 116 human-subject papers of which 102 discuss ethics and 90 have a dedicated section; 17 of those 116 not mentioning a review board; 52 non-human-subject papers that do.

The two counts are not directly comparable and the page does not present them as such. Theirs is “does the paper discuss ethics at all”, read by hand over one year and four venues. Ours is “does the paper state a review outcome”, extracted over seventeen years and seven venues. They agree on shape (the apparatus tracks participants, not harm) and on nothing numerically.

The ~1,038 figure the page quotes (“52 of the ~1,038 papers without human subjects”) is derived by arithmetic: 1,154 − 116. Their paper gives 52 and 116 but not the complement. The tilde is doing real work and is deliberate.

Required by the corpus notes: 138 posters and 251 records of ≤4 pages have their methodology compressed out, so they bias every silence figure upward. Section J of the report re-runs the headline with them dropped.

The gap the page is about — 75.3% against 21.3% — is 76.4% against 21.6% after trimming. It survives. The page reports the untrimmed figures and states the check in its limitations section.

Every external claim on the page was fetched on 2026-08-18. A sonnet sub-agent did the first pass into notes/ethics_external.md under instructions to give a primary-source URL, a fetch date and a verbatim quote for every claim and to mark anything else UNVERIFIED. The load-bearing claims were then re-fetched by hand before they reached the page.

Used as cited, each with the source URL and read-date on the page: the IEEE S&P '27, NDSS '27, CCS '26, PoPETs '27, IMC '26 and TheWebConf '26 CFP text; ACM's 2021 human-participants policy; 45 CFR 46.102(e) and 46.104(d)(4)(i) from the eCFR renderer API; GDPR Art. 89; EU AI Act Art. 2(6) and 2(8); the Swiss HRA and Dutch WMO scope clauses; the Jisc, SURF and Internet2 policies; Verisign's WHOIS and RDAP terms; the AWS and Azure penetration-testing policies; the Menlo Report and companion, including their DHS “Archived Content” banner; Mozilla Rally's GitHub archive date; the Citizen Browser, DataSkop and Tracking Exposed / AI Forensics status; the DSA Art. 40 delegated act and the 49 applications; Bright Data's and Oxylabs' consent claims; and EWDCI's membership and audit mechanics.

Three of those carry a caveat the page reproduces rather than hides:

• dhs.gov, acm.org and jisc.ac.uk all return HTTP 403 to scripted clients. The Menlo pages resolve in a browser; the ACM policy and the Jisc security policy were read from Wayback captures (2025-12-02 and 2026-08-14). The page says so in the footnote in each case, and gives script-friendly CAIDA mirrors for the Menlo PDFs.
• The EWDCI claim was tightened. The sub-agent's own note flagged that the accrediting body and the auditor are the same trade association and that the audited companies are its paying members. The page says that in those words rather than calling it a third-party audit.
• arXiv:2511.10408 is a preprint and the page labels it as one.

• Whether any of this changes behaviour. The page measures what papers report. A silent paper may have done everything right, and the corpus cannot tell the difference. Every framing on the page is about disclosure, deliberately.
• The recall of the harm-mitigation fold. Reviewer 1's spot check found real misses — “respected the 40-interaction limit” does not match the rate-limiting family because the regex expects limit(ed) <object> word order, and “controlled rates” matches nothing. These were not fixed: each is another phrasing-specific alternative rather than a rule, and the residue is already published in full. The consequence is directional and should be stated wherever these figures are reused: every family count on the page is a lower bound.
• What the 423 crawling papers with no stated mitigation actually did. Nothing in the schema distinguishes “did nothing” from “did not write it down”, and the page never counts them as the former.
• How often measurement papers disclose LLM use in the pipeline. There is no schema field for it, and the venue requirements are newer than most of the corpus. The page states the requirements and marks the measurement as not done.
• Whether copyright or database rights have ever been asserted against an academic crawl. Not searched. The page's licensing bullet is about practice, not about case law, and says so.
• The precision of the terms of service sweep. 129 papers, unread. A hand pass separating “we complied” from “the service's ToS says” would make that a real figure.
• Whether venue policy causes the reporting rate or tracks it. PoPETs states an outcome at 54.3% and mandates an ethics section from the 2027 cycle — the mandate is later than the behaviour, so the page describes the correlation and claims no direction.
• Non-US, non-EU ethics regimes. Only the Swiss and Dutch statutes were checked, and both turned out to be biomedical-only. A German (DFG) counterpart was not checked against a primary source. The page's claim is scoped to “most of the world has no equivalent standing body”, supported by the 20 papers that say so, not by a survey of national law.
• Whether the 20 “no review body” papers are representative. They are the papers that said so. Institutions without a board that simply wrote nothing are invisible to this sweep and are certainly more numerous.

1. Extend the stub, not write a sibling. practices:ethics was a skeleton whose own headings matched the material almost exactly. Broadening it was the obvious call; the only real question was whether the pre-measurement material deserved a separate page from Notifying websites, and it does — that page is 59 KB about a channel, this one is about a decision made before any measurement runs.
2. Lead with the participants-versus-crawls split rather than the headline reporting rate. 33.8% of empirical papers is a fact about a mixed population and reads as an indictment of the whole field. 75.3% against 21.3% is the actionable version, and it is the one that tells the reader which column they are in.
3. Count explicitly-discussed-no-review as stated. Argued above. A reasonable person would exclude it; the page prints the approved/exempt column beside every such figure so they can.
4. Publish the robots.txt non-answer as a non-answer. The literature has no consensus and inventing one would be dishonest. The page says decide, be consistent, and write the sentence — and specifically asks for the share of the sample an honoured exclusion removes, because that is a denominator change and nobody reports it.
5. Give the venue table a read-date and a warning rather than presenting it as stable. Two of its seven rows moved within twelve months, in opposite directions.
6. Do not write a proxy-vendor consent checklist. The sub-agent established that no such checklist exists anywhere and explicitly declined to fabricate one. The page reports the absence and gives three rules of thumb clearly framed as such, rather than presenting original synthesis as an established standard.
7. Name the volunteer platforms that are dead. A student reading a 2021 paper will find Mozilla Rally and plan around it. Reporting its archive date is more useful than listing only what is alive.
8. Publish “the contract claim is what the final judgment rests on” rather than “what ended the company”. The draft said the latter; the docket supports the former and nothing cited supports a causal claim about hiQ's business. Flagged by the generic reviewer.
9. Put “What the venues require” third, not seventh. It was drafted after the harm and vantage-point sections, on the logic that practice comes before paperwork. It was moved ahead of them before publication because it is the section a reader arrives looking for, and because the <WRAP important> box at the top forward-references it.
10. Keep the corpus's “Legal aspects of crawling” heading from the stub, but move the substance to Legal enforcement. The stub listed fraud, trespass and breach of contract. This page carries only what a crawler's ethics section needs — the CFAA narrowing and the surviving contract exposure — and points at the sibling for enforcement.

Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the report script and its output, the folds, and these notes. The three focused passes ran in parallel first; the generic pass ran after their findings were applied.

Re-ran report_ethics.mjs and confirmed byte-identical output to the committed file, then checked every table cell and percentage, audited both folds, and spot-checked 15 random harmMitigation strings by hand.

Reviewer 1 also confirmed as correct, having recomputed them: sections A–J in full, the 149 + 971 = 1,120 arithmetic, the 4,472 = 5,118 − 646 arithmetic, the year-window and by-venue tables, the Menlo-by-window series, and the sensitivity figures.

Reviewer 2 confirmed as correct: all 23 citekeys resolve with no duplicates and no unused additions; all 15 added BibTeX entries against Crossref and venue landing pages; the rate-limiting table's five attributions; and the Singh, Chehade, Datta and Subramani figures.

Fetched every external URL on the page. Full inventory in the reviewer's report; the findings that changed the page:

Reviewer 3 confirmed verbatim: the Menlo four-principles quote, GDPR Art. 89, AI Act Art. 2(6)/2(8), 45 CFR 46.102(e)(1) and 46.104(d)(4)(i), the “publicly available” undefined-term claim, the DSA 49-applications figure as still current, Prolific's and MTurk's quoted wording, Bright Data's KYC date, the SURF, Verisign and Azure quotes, the Tor board, and the Rally / Citizen Browser / DataSkop / AI Forensics statuses. It also confirmed that User studies and Artifacts are genuine red links already promised by several other pages — a pre-existing gap, not one this page introduces.

No checklist; handed the page, the provenance page, the scripts and the stub, and told to look for whatever the focused three were not. It re-derived several figures independently and re-fetched the neighbouring pages.

Rejected: none. Every finding was accepted, which is itself worth recording — a generic pass with no checklist found the page's largest substantive gap (O) after three focused passes had gone over it.

Three of the four reviewers found something that changed a published number or a published claim. Reviewer 1's finding 1 was the expensive one: a figure the page built an argument on (“the cheapest credible thing on the list”) was inflated by 70%, and the defect was in code the page never shows. Reviewer 3's finding H was the one no corpus query could have produced: the page had a legally accurate but incomplete account of a case whose ending is the page's own argument. Reviewer 2's findings E and F are the ordinary kind — citations dropped in editing — and are the reason a citations pass is worth its slot even when nothing is wrong with the sources.

• Ethics — the page this log is for.
• Corpus — corpus-wide provenance: the venue scope, the selection funnel, the provisional 2025–2026 slice.
• notifying_websites and legal_enforcement — the two sibling logs; the disclosure figures on this page are recomputed here rather than copied from either.
• crawling_location — the vantage-point log, which carries the residential-proxy market research this page does not repeat.

[1]

Ramulu, Harshini Sri; Schmitt, Helen; Rerich, Bogdan; Rodriguez, Rachel Gonzalez; Kohno, Tadayoshi; Acar, Yasemin (2025): "Ethics in Computer Security Research: A Data-Driven Assessment of the Past, the Present, and the Possible Future", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)