Back to Internet scanning. Corpus-wide selection and extraction notes are on corpus. This is the page-specific query log: every query with its population, the fold and its residue, the probes and the hand verdicts behind them, the external sources and the ones rejected, and what could not be established.

No ~~DISCUSSION~~ block here. Comments belong on the content page; this is a working log.

• Run date: 2026-09-10 (UTC). Single sitting.
• Authoring agent: Claude Opus 5 (Claude Code), working the wiki-measuretheweb item “scanning has no instrument page”, section A2 of the 2026-09-02 brainstorm pass.
• Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under /workspace/publications_dataset/data/. No write to the publication mount.
• Creating, not extending. Before writing: node scripts/sitemap.mjs (168 existing pages, 7 promised-but-missing, all 7 declared on roadmap), then exported and read automated_measurements, design, programming, ip_classification, dns, ethics, email_authentication, crawler and roadmap.
• Wiki saves through scripts/dw.mjs (JSON-RPC) with –if-rev on every existing page.
• Pages saved this sitting: literature:bibliography (rev 1789063332 → 1789074254, 8 entries appended), programming:internet_scanning (new), roadmap (1789062297 →), programming (1788006004 →), design:automated_measurements (1787943845 →), security:email_authentication (1788954595 →), and this page.

Counts are papers, never tuples. Sentinels are never answers. An absent ethics or artifacts object is counted as neither a stated value nor a sentinel and is printed as its own row.

Poster sensitivity. Dropping the 6 short records moves “vantage location stated” from 55.5% to 56.1% — 0.6 points. The corpus-wide poster caveat (posters have methods compressed out and bias every silence figure upward) applies, but ACTIVE is 2.4% posters against SCAN's 3.7%, so it bites less here than on most pages.

scripts/scan_fold.mjs. The written boundary is at the top of the file and was fixed before the first count.

• Coverage is a contract, not an aspiration. Every category == “network-scanner” used-or-produced string must resolve to a family or to a HAND verdict. 746 tuples, 0 unmapped, 0 stale HAND entries. The report exits 1 in either direction — an unmapped string and a hand verdict that nothing matches, because a shadowed verdict is a lie about what was read.
• Kinds, not one ranking. Twenty kinds. Five are membership (active-scan, grabber, reachability, censorship-probe, ipv6-targeting); the rest are reported beside ACTIVE and routed elsewhere (dataset, topology, dns, vantage-platform, routing, vuln-scanner, passive, local-link, perf, simulation, p2p-crawler, name-enumeration, packet-crafting, registry, lab-protocol-test, host-tool).
• One negative guard. /z-?map/i matches WizMap, an embedding-visualisation tool that appears in one paper under category: other. Without the guard it would have entered ACTIVE as a ZMap use. This is the \b-does-not-protect-an-acronym trap in a different costume, and it was found only by printing the strings that matched outside network-scanner.
• Generic families. custom scanner and ping are flagged generic and only fire inside category == “network-scanner”. Measured cost of not doing this: ACTIVE went from 245 to 708, with custom scanner alone at 464 papers.
• ZMap appears under 9 spellings in ACTIVE: TCP Options module for ZMap, ZMap, ZMap-based script, ZMap6, ZMapv6, Zmap, zmap, zmap2, zmap6. nmap has 10, ZGrab 7, TLS banner grabbers 8, and the generic “custom” strings 21.
• Self-tests: 74 cases plus 6 structural assertions, including both directions of the generic/category rule and the WizMap guard. The module throws at import if any fails, so every script that imports it inherits the check.

The full residue — which is empty — and the complete 174-entry hand map with its kinds are in the fold source at the foot of this page.

All probes read paper.cols.txt with whitespace collapsed (a PDF line break inside a phrase otherwise silently undercounts). Three of the 930 SCAN papers have no paper.cols.txt; the report prints the missing count for every probe.

Each practice gets two widths. The loose regex is the recall claim; the first-person regex requires “we”/“our” near the term. Neither is a population, and the width decides the claim — publishing either alone would have been wrong in a different direction.

Samples are every k-th paper of the sorted hit list, so they are reproducible and the report exits 1 if a sampled paper has no verdict or if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read.

own = the matched sentence is about the authors' own scan. other = the term is in the paper for a different reason.

scripts/verify_scanning_figures.mjs looks up every quoted sentence and per-paper figure on the content page in the cited paper's paper.cols.txt with whitespace collapsed, falling back to a pypdf re-extraction of paper.pdf. 30 needles, 0 not located, 0 weak needles under 20 characters. 26 are in-corpus and all 26 resolved in .cols, so the PDF fallback never had to fire for those. The other 4 are from [1Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)], which is not in the extraction: they are checked against out/scan/zmap2013_paper.txt (72,192 bytes), a pypdf extraction of the authors' own copy at zmap.io/paper.pdf. That check needs its own de-hyphenation step — pypdf keeps the printed hyphen at a line break, so accu- racy and theoreti- cal are two words in the extraction and one in the PDF, and two of the four quotes failed until it was added.

Two things that went wrong and were fixed:

• A misread digit. A first pass read the Durumeric et al. USENIX Security 2014 sentence off a 180-character context window as “908 organizations have requested that we exclude their networks”. The paper says 208, in both paper.cols.txt and an independent pypdf extraction of paper.pdf. The wrong number never reached the page, but it did reach a hand-verdict comment in the report script and had to be corrected there. A narrow context window is not a source.
• Two quotes failed on capitalisation. “Consider whether existing open source datasets…” and “In early 2024, ZMap changed…” are sentence-initial in the paper, and the draft had lower-cased them to fit mid-sentence. The page now quotes them with the source's capitalisation. A quote check that had been case-insensitive would have passed both silently.

pages/scan_budget.py is embedded in the page as a <file python> block. Verified after saving:

• The embedded block is byte-identical to pages/scan_budget.py.
• Downloading it back off the rendered page (/_export/code/programming:internet_scanning?codeblock=3) and diffing gives one difference, a stripped trailing newline. The downloaded copy runs its own –self-test: 9 checks, OK.
• The <code> block beside it is the whole output of a real run, compared line-by-line for exact equality — not a containment check, which passes on a truncated block.
• The self-test has a mutation guard: replacing collapse_addresses() with a bare sum makes the overlapping-CIDR case return 33,554,432 instead of 16,777,216 and check 3 fails. Check 4 exists so that “collapse does nothing at all” cannot also pass — a disjoint pair must still add up.
• The collapse is doing real work on the real file: 255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum reports 592,708,865 removed against the correct 592,708,864.

scripts/external_checks_internet_scanning.sh re-fetches every non-corpus fact and prints FAILED explicitly per check. Its unedited output is at the foot of this page. Honours GH_TOKEN via ${GH_TOKEN:+x} — see the exposure note above for the form that leaks.

• How many scan papers actually rate-limited, honoured an exclusion list, or offered an opt-out. Only estimates, from a regex probe with a hand-read precision sample of 10–12. Closing this needs the same treatment harm_fold.mjs got on ethics: a loose candidate sweep plus a per-paper verdict over the whole hit list, ~90 papers per practice. That is a day of reading and it would replace four “about a fifth”s with four numbers.
• Whether the 534 SCAN papers that name no network-scanner tool scanned without saying with what, or queried a dataset. The schema cannot separate those two, and neither can a name fold. A full-text pass over a sample would.
• The recall of ACTIVE outside category == “network-scanner”. One-off scanner names filed under other are not hand-read — that would be the whole 16,737-name tool vocabulary. ACTIVE is a floor and the page says so.
• Venue coverage. PAM, TMA, ANRW, CoNEXT and ACSAC are not in this corpus, and for network scanning that is a larger gap than for the web-measurement pages: Yarrp, MIDAR, scamper and several IPv6 target generators have their tool papers at those venues. Any “the literature does X” sentence on this page should be read as “these seven venues do X”. Nothing here can fix that; it needs the corpus extended.
• Whether ZMap's 33% traffic-share figure still holds. It is [2Durumeric, Zakir; Adrian, David; Stephens, Phillip; Wustrow, Eric; Halderman, J. Alex (2024): "Ten Years of ZMap", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]'s Q1 2024 measurement of scan traffic seen at one telescope, not a corpus figure and not a 2026 figure. No newer public measurement was found.
• Whether the roadmap's original “143” was ever right. It re-derives to 141 in SCAN / 145 corpus-wide under this fold and 136/137 under am_fold.mjs. The difference is entirely in how loosely the nmap regex is anchored. No reconstruction of the original query exists.

Four passes, all told explicitly that the author's context might not be exhaustive, and all handed the page text, the report script and its unedited output, the fold, the published code and the provenance notes. The three focused passes were spawned in parallel against the first published revision (programming:internet_scanning rev 1789074265); the generic pass was spawned afterwards, against the corrected page, so it saw a different snapshot. That matters for reading their findings and is recorded here rather than left to be inferred.

Two of the three focused passes had to be prompted to return partial findings — they were still fetching after a long wall-clock stretch. All three returned in full in the end.

Re-ran all four scripts, confirmed they reproduce the committed outputs byte-for-byte and that the provenance page's embedded copies are byte-identical to fresh runs. Mutation-tested excluded_addresses, line_rate_pps, scan_plan, both directions of the section-8 sample check and the section-0 fold-coverage check, and confirmed each fires. Found no figure that mixes the SCAN/ACTIVE denominators. Four findings:

All 15 citekeys resolve exactly once in a fresh export; bib_dedup_scan.py finds 0 definite duplicates; check_attributions.mjs passes 10/10; every quoted sentence checked matched verbatim; both hand-fetched USENIX author lists match the PDF header in order, including the last author and the umlauts; the published code contains no names at all. One finding:

Independently re-fetched and confirmed: both ZMap man pages (tag and main, byte-identical, 332 lines each), every published flag and default, the absence of --blocklist-file-extra, the three helper binaries, conf/blocklist.conf (16 entries, 2013 header, no opt-out language) with the CIDR arithmetic recomputed by hand including the 255.255.255.255/32-inside-240.0.0.0/4 overlap, all ZGrab2 README quotes, every repository state and release date, nmap 7.991 against nmap.org's own changelog, all three IPv6 Hitlist line counts downloaded and counted independently, the Rapid7 Sonar redirect and the 8 / 47,457 / 60.7 TB figures (summed by hand from the eight per-dataset counts), and both USENIX author lists. Two findings:

It returned after the “did not return” note above had already been written and published, in response to a second prompt for partial findings. Fifteen findings on the already-corrected page, and it earned its slot: five of them are things the three focused passes structurally could not see, and two are wrong numbers the number guard passed because the digits occur elsewhere in the report. This is the fourth page in a row where the generic pass found the worst item.

It also read the provenance probe table and found the IRB row filed under the “first-person” column although the report calls it loose. That is correct and the row is now labelled. Its own NOT READ list is long — the embedded fold source, the external-checks output, `practices:ethics` and `design_dns.txt` for duplication, and a possible disagreement between this page's ZMap 95 + XMap 16 and dns' “ZMap and XMap (112 papers)” that it could not resolve. That last one is unresolved and is the first thing a follow-up should check.

Each pass named its own gaps and they are worth recording, because they are the parts of this page nothing checked:

• The 112-vs-111 question above. dns reports “ZMap and XMap (112 papers)” against this page's ZMap 95 + XMap 16 = 111. That may be an overlap artefact (a paper naming both) or a real disagreement between two folds. Not resolved.
• Nobody hand-audited the 174 HAND verdicts in scan_fold.mjs for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire.
• Nobody re-read the 44 probe papers to confirm the own/other verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates.
• The figures pass did not re-fetch the external facts; the currency pass did not re-derive the corpus figures. That split is deliberate, but it means no single pass read an external figure and a corpus figure in the same sentence.
• The citations pass flagged that nobody had verified [1Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)]'s claimed absence from the corpus. Closed by the author on 2026-09-10: 0 extraction records, 0 data/fulltext/2013/USENIX directories and 0 of the 28 corpus2/.meta/USENIX-2013.json records match /zmap|internet-wide-scanning/i. Twenty-eight records is itself thin for that venue-year — a corpus-level observation, recorded on corpus' territory rather than here.

report_internet_scanning-output.txt

==============================================================================
0. FOLD COVERAGE — every category=network-scanner name must be decided
==============================================================================
category='network-scanner' used/produced tuples: 746
unmapped strings: 0
HAND entries: 174, matched: 174, stale: 0
 
==============================================================================
0b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)
==============================================================================
extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: 0
data/fulltext/2013/USENIX directories matching /zmap|scann/i:            0
corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 0 of 28
PUBLISHED_ZMAP2013_ABSENT yes
PUBLISHED_USENIX2013_INDEX 28
 
==============================================================================
1. POPULATIONS
==============================================================================
Population                                                  Papers  Share
----------------------------------------------------------  ------  --------------
corpus (7 venues, 2010–2026)                                5859    100%
SCAN — studyTypes 'network-scan-or-probe'                   930     15.9%
ACTIVE — names an active-scan instrument (page population)  245     4.2% of corpus
  … of those, inside SCAN                                   235     25.3% of SCAN
  … of those, outside SCAN                                  10      4.1% of ACTIVE
SCAN papers naming ANY category='network-scanner' tool      396     42.6%
SCAN papers naming NO category='network-scanner' tool       534     57.4%
PUBLISHED_CORPUS 5859
PUBLISHED_SCAN 930
PUBLISHED_ACTIVE 245
PUBLISHED_ACTIVE_IN_SCAN 235
PUBLISHED_ACTIVE_IN_SCAN_PCT 25.3%
PUBLISHED_ACTIVE_OUTSIDE 10
PUBLISHED_NAMES_NS 396
PUBLISHED_NO_NS 534 57.4%
 
category-restricted membership would give 231; the any-category rule adds 14
PUBLISHED_NS_ONLY 231
  +CCS/2015/iris-vetting-private-api-abuse-in-ios-applications  Iris[program-analysis]
  +IMC/2016/entropy-ip-uncovering-structure-in-ipv6-addresses  Entropy/IP[other]
  +IMC/2018/digging-into-browser-based-crypto-mining  ZGrab[http-client]
  +IMC/2018/where-the-light-gets-in-analyzing-web-censorship-mechanisms-in-india  OONI[other]
  +CCS/2019/geneva-evolving-censorship-evasion-strategies  Geneva[other]
  +IMC/2020/flashroute-efficient-traceroute-on-a-massive-scale  IPv6 Hitlist Service[other]
  +USENIX/2022/get-out-automated-discovery-of-application-layer-censorship-evasion-strategies  Geneva[other]
  +IMC/2023/replication-towards-a-publicly-available-internet-scale-ip-geolocation-dataset  IPv6 Hitlist Service[other]
  +PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates  Censored Planet[infrastructure]
  +IMC/2023/inferring-changes-in-daily-human-activity-from-internet-response  Trinocular[infrastructure]
  +USENIX/2023/deresistor-toward-detection-resistant-probing-for-evasion-of-internet-censorship  Geneva[other]
  +NDSS/2024/modeling-and-detecting-internet-censorship-events  Censored Planet[other], OONI[other]
  +IMC/2024/poster-traffic-engineering-security-implications  IPv6 Hitlist Service[other]
  +USENIX/2024/with-great-power-come-great-side-channels-statistical-timing-side-channel-analys  TLS banner grabbers[other]
 
==============================================================================
2. KIND UNION over SCAN (a paper can name several kinds)
==============================================================================
kind               membership?  Papers  Share of 930  routed to
-----------------  -----------  ------  ------------  ----------------------------
active-scan        YES          180     19.4%
dataset                         97      10.4%         design:existing_datasets
topology                        95      10.2%         design:ip_classification
passive                         85      9.1%
vantage-platform                74      8.0%          design:crawling_location
routing                         72      7.7%          design:ip_classification
dns                             72      7.7%          design:dns
grabber            YES          53      5.7%
censorship-probe   YES          23      2.5%
reachability       YES          19      2.0%
vuln-scanner                    15      1.6%          security:web_vulnerabilities
ipv6-targeting     YES          11      1.2%
local-link                      9       1.0%
perf                            7       0.8%
p2p-crawler                     6       0.6%
name-enumeration                3       0.3%
packet-crafting                 3       0.3%
simulation                      3       0.3%
lab-protocol-test               2       0.2%
registry                        1       0.1%
 
SCAN papers whose tools[] name no instrument this fold recognises: 416 (44.7%)
PUBLISHED_SCAN_NO_INSTRUMENT 416 44.7%
 
==============================================================================
3. THE INSTRUMENTS — membership families over ACTIVE
==============================================================================
Family                kind              Papers  Share of ACTIVE  distinct spellings
--------------------  ----------------  ------  ---------------  ------------------
ZMap                  active-scan       95      38.8%            9
nmap                  active-scan       46      18.8%            10
ZGrab                 grabber           32      13.1%            7
custom scanner        active-scan       25      10.2%            21
ping                  reachability      18      7.3%             7
XMap                  active-scan       16      6.5%             3
TLS banner grabbers   grabber           11      4.5%             8
OONI                  censorship-probe  7       2.9%             3
Geneva                censorship-probe  6       2.4%             1
Masscan               active-scan       5       2.0%             1
Censored Planet       censorship-probe  4       1.6%             1
IPv6 Hitlist Service  ipv6-targeting    4       1.6%             5
Quack                 censorship-probe  4       1.6%             1
6Gen                  ipv6-targeting    3       1.2%             1
Hyperquack            censorship-probe  3       1.2%             1
Trinocular            reachability      3       1.2%             1
addr6                 ipv6-targeting    2       0.8%             1
Augur                 censorship-probe  2       0.8%             1
Entropy/IP            ipv6-targeting    2       0.8%             1
GeoInspector          censorship-probe  2       0.8%             1
Goscanner             grabber           2       0.8%             1
Iris                  censorship-probe  2       0.8%             2
Karma                 active-scan       2       0.8%             1
LZR                   grabber           2       0.8%             1
QScanner              grabber           2       0.8%             1
Satellite             censorship-probe  2       0.8%             1
 
families named by exactly one paper: 40 — 6Graph, 6Sense, 6Tree, AmpMap, BreadCrumb, CnCHunter, dMAP (DPI Mapper), DTLS-MitM-Scanner (DMS), EAST, ERIS, exitmap, Exitmap, forwarders-based scanner, gateway checker, gQUIC-module, ICMP_Sonar, ike-scan, IRBlock, IRLscanner, JARM, LanDscAPe, Lumscan, Mint, modified ntpdedup, Monocle, nc, Netcat, node probing program, onesixtyone, onion-grab, OSAVRoute, PLCScan, Python script, quicgrabber, RAT-Scan, smbmap, SNMPv3, TMC, Unicorn, WhatWeb
PUBLISHED_ONEOFF_FAMILIES 40
PUBLISHED_FAM_ZMAP 95
PUBLISHED_FAM_NMAP 46
PUBLISHED_FAM_ZGRAB 32
PUBLISHED_FAM_CUSTOM_SCANNER 25
PUBLISHED_FAM_PING 18
PUBLISHED_FAM_XMAP 16
PUBLISHED_FAM_TLS_BANNER_GRABBERS 11
PUBLISHED_FAM_OONI 7
 
 
ZMap|ZGrab|Masscan|nmap, used or produced: 145 corpus-wide, 141 inside SCAN (the roadmap row said 143)
PUBLISHED_CORE_FOUR_ALL 145
PUBLISHED_CORE_FOUR_SCAN 141
ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6
PUBLISHED_ZMAP_SPELLINGS 9
ACTIVE papers whose scanner is home-grown: 45 (18.4%)
  = generic 'custom scanner' string:                    25
  + a one-paper name the paper itself PRODUCED:         20
  - overlap:                                            0
  (the other 20 one-paper families are third-party tools only this corpus's one paper used)
PUBLISHED_HOMEGROWN 45 18.4%
PUBLISHED_HOMEGROWN_CUSTOM 25
PUBLISHED_HOMEGROWN_PRODUCED 20
PUBLISHED_ONEOFF_THIRDPARTY 20
 
==============================================================================
4. CURRENCY — membership families by year bucket, share of that bucket's ACTIVE papers
==============================================================================
Window      ACTIVE papers  ZMap        ZGrab       nmap        XMap        Masscan   custom scanner  ping
----------  -------------  ----------  ----------  ----------  ----------  --------  --------------  ---------
2010–2013   12             1 (8.3%)    0 (0.0%)    5 (41.7%)   0 (0.0%)    0 (0.0%)  3 (25.0%)       3 (25.0%)
2014–2017   45             26 (57.8%)  4 (8.9%)    8 (17.8%)   0 (0.0%)    1 (2.2%)  7 (15.6%)       2 (4.4%)
2018–2021   56             21 (37.5%)  9 (16.1%)   15 (26.8%)  0 (0.0%)    0 (0.0%)  5 (8.9%)        3 (5.4%)
2022–2024   86             26 (30.2%)  9 (10.5%)   14 (16.3%)  10 (11.6%)  4 (4.7%)  6 (7.0%)        9 (10.5%)
2025–2026*  46             21 (45.7%)  10 (21.7%)  4 (8.7%)    6 (13.0%)   0 (0.0%)  4 (8.7%)        1 (2.2%)
 
Per year, ZMap and the whole ACTIVE population (2026 is provisional):
Year  corpus  SCAN  ACTIVE  ZMap
----  ------  ----  ------  ----
2010  119     32    6       0
2011  116     23    5       0
2012  151     19    0       0
2013  125     15    1       1
2014  166     33    9       6
2015  190     40    13      6
2016  182     46    12      5
2017  231     54    11      9
2018  254     54    18      6
2019  402     65    12      3
2020  404     71    12      7
2021  379     65    14      5
2022  546     70    25      10
2023  719     79    29      7
2024  690     96    32      9
2025  770     122   34      17
2026  415     46    12      4
first ZMap year in ACTIVE: 2013
PUBLISHED_ZMAP_FIRST_YEAR 2013
 
==============================================================================
5. WHERE SCANNING PUBLISHES — venue
==============================================================================
Venue    Papers  SCAN  SCAN share of venue  ACTIVE  ACTIVE share of venue
-------  ------  ----  -------------------  ------  ---------------------
CCS      990     132   13.3%                23      2.3%
IEEE-SP  767     87    11.3%                18      2.3%
IMC      638     296   46.4%                80      12.5%
NDSS     701     117   16.7%                34      4.9%
PETS     510     49    9.6%                 8       1.6%
USENIX   1410    199   14.1%                70      5.0%
WWW      843     50    5.9%                 12      1.4%
 
==============================================================================
6. WHAT A SCAN PAPER REPORTS — schema fields, denominator ACTIVE and SCAN
==============================================================================
Field                           ACTIVE (245)  share  SCAN (930)  share
------------------------------  ------------  -----  ----------  -----
vantage location                136           55.5%  487         52.4%
vantage infrastructure          168           68.6%  612         65.8%
any used/produced tool version  118           48.2%  401         43.1%
version of the SCANNER itself   20            8.2%   20          2.2%
own artifact URL                146           59.6%  506         54.4%
ethics review outcome           93            38.0%  290         31.2%
notified affected parties       186           75.9%  560         60.2%
harm mitigation described       212           86.5%  700         75.3%
robots.txt                      11            4.5%   22          2.4%
temporal span start             196           80.0%  656         70.5%
 
ACTIVE papers with NO ethics record at all: 7 (2.9%) — not counted as stated, not counted as a sentinel
ACTIVE papers with NO artifacts record at all: 3 (1.2%)
PUBLISHED_NO_ETHICS_RECORD 7 2.9%
PUBLISHED_NO_ARTIFACT_RECORD 3 1.2%
PUBLISHED_REPORT_VANTAGE_LOCATION 136 55.5% | SCAN 487 52.4%
PUBLISHED_REPORT_VANTAGE_INFRASTRUCTURE 168 68.6% | SCAN 612 65.8%
PUBLISHED_REPORT_ANY_USED_PRODUCED_TOOL_VERSION 118 48.2% | SCAN 401 43.1%
PUBLISHED_REPORT_VERSION_OF_THE_SCANNER_ITSELF 20 8.2% | SCAN 20 2.2%
PUBLISHED_REPORT_OWN_ARTIFACT_URL 146 59.6% | SCAN 506 54.4%
PUBLISHED_REPORT_ETHICS_REVIEW_OUTCOME 93 38.0% | SCAN 290 31.2%
PUBLISHED_REPORT_NOTIFIED_AFFECTED_PARTIES 186 75.9% | SCAN 560 60.2%
PUBLISHED_REPORT_HARM_MITIGATION_DESCRIBED 212 86.5% | SCAN 700 75.3%
PUBLISHED_REPORT_ROBOTS_TXT 11 4.5% | SCAN 22 2.4%
PUBLISHED_REPORT_TEMPORAL_SPAN_START 196 80.0% | SCAN 656 70.5%
 
==============================================================================
7. FULL-TEXT PROBES over ACTIVE — a loose upper bound and a tight estimate
==============================================================================
PUBLISHED_PROBE_STATES_ITS_OWN_PROBE_RATE_OR_RATE_LIMIT_LOOSE 127 51.8%
PUBLISHED_PROBE_STATES_ITS_OWN_PROBE_RATE_OR_RATE_LIMIT_TIGHT 84 34.3%
PUBLISHED_PROBE_HONOURS_A_BLOCKLIST_EXCLUSION_LIST_LOOSE 106 43.3%
PUBLISHED_PROBE_HONOURS_A_BLOCKLIST_EXCLUSION_LIST_TIGHT 90 36.7%
PUBLISHED_PROBE_OFFERS_AN_OPT_OUT_LOOSE 73 29.8%
PUBLISHED_PROBE_OFFERS_AN_OPT_OUT_TIGHT 69 28.2%
PUBLISHED_PROBE_SOURCE_ADDRESS_HYGIENE_DEDICATED_IPS_RDNS_PTR_WHOIS_WEB_PAGE__LOOSE 170 69.4%
PUBLISHED_PROBE_SOURCE_ADDRESS_HYGIENE_DEDICATED_IPS_RDNS_PTR_WHOIS_WEB_PAGE__TIGHT 114 46.5%
PUBLISHED_PROBE_ABUSE_COMPLAINTS_RECEIVED_OR_HANDLED_LOOSE 58 23.7%
PUBLISHED_PROBE_ABUSE_COMPLAINTS_RECEIVED_OR_HANDLED_TIGHT 46 18.8%
Practice                                                           loose probe (upper bound)  first-person probe
-----------------------------------------------------------------  -------------------------  ------------------
states its own probe rate or rate limit                            127 (51.8%)                84 (34.3%)
honours a blocklist / exclusion list                               106 (43.3%)                90 (36.7%)
offers an opt-out                                                  73 (29.8%)                 69 (28.2%)
source-address hygiene (dedicated IPs, rDNS/PTR, WHOIS, web page)  170 (69.4%)                114 (46.5%)
abuse complaints received or handled                               58 (23.7%)                 46 (18.8%)
 
Neither column is a population. The loose regex fires on any sentence in
the paper, including one about somebody else's rate limit or somebody
else's blocklist; the first-person regex requires "we"/"our" near the term
and still admits sentences about the authors' own *subject*. Section 8
hand-reads a deterministic sample of each first-person hit list and gives
the measured precision.
 
ACTIVE papers naming an IRB / ethics board: 56 (22.9%) [loose, no precision measured]
ACTIVE papers citing Durumeric / Partridge-Allman / Menlo / "good Internet citizen": 174 (71.0%) [loose]
PUBLISHED_IRB 56 22.9%
PUBLISHED_GOODPRACTICE 174 71.0%
 
==============================================================================
8. HAND-READ PRECISION — deterministic every-k-th sample of each first-person hit list
==============================================================================
 
--- own probe rate: 84 first-person hits of 245 ACTIVE; sample of 12
    other  CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other  CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels
    other  IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet
    other  IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks
    own    IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns
    own    IMC/2025/chaos-in-the-chain-evaluate-deployment-and-construction-compliance-of-web-pki-ce
    own    NDSS/2020/measuring-the-deployment-of-network-censorship-filters-at-global-scale
    other  NDSS/2024/proof-of-backhaul-trustfree-measurement-of-broadband-bandwidth
    other  USENIX/2011/toward-secure-embedded-web-interfaces
    other  USENIX/2021/accurately-measuring-global-risk-of-amplification-attacks-using-ampmap
    own    USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
    own    USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch
    precision 5/12; 84 x 5/12 = about 35 papers (14.3% of ACTIVE)
PUBLISHED_PRECISION_OWN_PROBE_RATE 5/12 -> ~35 14.3%
 
--- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12
    own    CCS/2018/practical-state-recovery-attacks-against-legacy-rng-implementations
    other  IEEE-SP/2017/augur-internet-wide-detection-of-connectivity-disruptions
    other  IMC/2011/detecting-validating-and-characterizing-computer-infections-in-the-wild
    own    IMC/2016/measuring-the-security-harm-of-tls-crypto-shortcuts
    own    IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists
    other  IMC/2022/deep-dive-into-the-iot-backend-ecosystem
    own    IMC/2023/ecn-with-quic-challenges-in-the-wild
    other  NDSS/2020/decentralized-control-a-case-study-of-russia
    other  NDSS/2025/revealing-the-black-box-of-device-search-engine-scanning-assets-strategies-and-ethical-consideration
    own    USENIX/2014/an-internet-wide-view-of-internet-wide-scanning
    own    USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship
    other  USENIX/2022/spoki-unveiling-a-new-wave-of-scanners-through-a-reactive-network-telescope
    precision 6/12; 90 x 6/12 = about 45 papers (18.4% of ACTIVE)
PUBLISHED_PRECISION_OWN_BLOCKLIST_EXCLUSION 6/12 -> ~45 18.4%
 
--- own opt-out offer: 69 first-person hits of 245 ACTIVE; sample of 10
    own    CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other  IEEE-SP/2020/meddling-middlemen-empirical-analysis-of-the-risks-of-data-saving-mobile-browser
    own    IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner
    own    IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
    own    IMC/2022/deep-dive-into-the-iot-backend-ecosystem
    own    IMC/2024/destination-reachable-what-icmpv6-error-messages-reveal-about-their-sources
    own    NDSS/2024/understanding-the-implementation-and-security-implications-of-protective-dns-services
    own    PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates
    own    USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship
    own    USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale
    precision 9/10; 69 x 9/10 = about 62 papers (25.3% of ACTIVE)
PUBLISHED_PRECISION_OWN_OPT_OUT_OFFER 9/10 -> ~62 25.3%
 
--- own source-address hygiene: 114 first-person hits of 245 ACTIVE; sample of 10
    own    CCS/2015/a-search-engine-backed-by-internet-wide-scanning
    other  IEEE-SP/2010/investigation-of-triangular-spamming-a-stealthy-and-efficient-spamming-technique
    own    IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner
    other  IMC/2016/identifying-and-aggregating-homogeneous-ipv4-24-blocks-with-hobbit
    own    IMC/2019/an-end-to-end-large-scale-measurement-of-dns-over-encryption-how-far-have-we-com
    own    IMC/2023/does-it-spin-on-the-adoption-and-use-of-quics-spin-bit
    own    NDSS/2020/on-using-application-layer-middlebox-protocols-for-peeking-behind-nat-gateways
    other  NDSS/2026/osavroute-advancing-outbound-source-address-validation-deployment-detection-with-non-cooperative-measurement
    other  USENIX/2021/lzr-identifying-unexpected-internet-services
    own    USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
    precision 6/10; 114 x 6/10 = about 68 papers (27.8% of ACTIVE)
PUBLISHED_PRECISION_OWN_SOURCE_ADDRESS_HYGIENE 6/10 -> ~68 27.8%
 
Practice                    first-person probe  hand-read precision  point estimate
--------------------------  ------------------  -------------------  --------------
own probe rate              84 (34.3%)          5/12                 ~35 (14.3%)
own blocklist / exclusion   90 (36.7%)          6/12                 ~45 (18.4%)
own opt-out offer           69 (28.2%)          9/10                 ~62 (25.3%)
own source-address hygiene  114 (46.5%)         6/10                 ~68 (27.8%)
 
The point estimate is probe count x sample precision. It is an estimate on a
sample of 10-12, so read it as "about a fifth" rather than as a percentage.
 
==============================================================================
9. IPv6 — the address space is not enumerable
==============================================================================
Query                                          Population    Papers  Share
---------------------------------------------  ------------  ------  -----
mentions IPv6 anywhere                         SCAN 930      281     30.2%
mentions IPv6 anywhere                         ACTIVE 245    108     44.1%
names an IPv6-targeting instrument in tools[]  SCAN 930      11      1.2%
names an IPv6-targeting instrument in tools[]  corpus 5,859  11      0.2%
full text says "hitlist"/"hit list" (loose)    SCAN 930      62      6.7%
"hitlist" within 80 chars of "IPv6"            SCAN 930      39      4.2%
names a target-generation algorithm            SCAN 930      20      2.2%
discusses aliased prefixes / dealiasing        SCAN 930      10      1.1%
PUBLISHED_V6_SCAN 281 30.2%
PUBLISHED_V6_ACTIVE 108 44.1%
PUBLISHED_V6_INSTR_SCAN 11
PUBLISHED_V6_INSTR_ALL 11
PUBLISHED_HITLIST_LOOSE 62 6.7%
PUBLISHED_HITLIST_V6 39 4.2%
REJECTED_PROBE /Gasser/ over SCAN: 93 (10.0%) — surname is not unique, hits are reference-list entries; not published
PUBLISHED_TARGETGEN 20 2.2%
PUBLISHED_ALIASED 10 1.1%
 
IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): 42 papers
  of those, discuss aliased prefixes / dealiasing: 8 (19.0%)
  the same 10 papers against all of SCAN would be 1.1% — the wrong denominator, most of SCAN never touches IPv6
PUBLISHED_V6_SUBSET 42
PUBLISHED_ALIAS_IN_SUBSET 8 19.0%
 
IPv6-targeting instruments named, corpus-wide:
    4 IPv6 Hitlist Service  (IMC/2020/flashroute-efficient-traceroute-on-a-massive-scale, IMC/2023/replication-towards-a-publicly-available-internet-scale-ip-geolocation-dataset, IMC/2024/poster-traffic-engineering-security-implications, PETS/2025/path-to-encrypted-dns-with-ddr-adoption-configuration-patterns-and-privacy-impli)
    3 6Gen  (IMC/2017/target-generation-for-internet-wide-ipv6-scanning, IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists, IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery)
    2 Entropy/IP  (IMC/2016/entropy-ip-uncovering-structure-in-ipv6-addresses, IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists)
    2 addr6  (IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery, NDSS/2026/on-borrowed-time-measurement-informed-understanding-of-the-ntp-pools-robustness-to-monopoly-attacks)
    1 6Graph  (IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation)
    1 6Tree  (IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation)
    1 6Sense  (USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications)
 
Per-year IPv6 mention rate inside SCAN (2026 provisional):
Year  SCAN  mentions IPv6  share
----  ----  -------------  -----
2010  32    4              12.5%
2011  23    2              8.7%
2012  19    1              5.3%
2013  15    0              0.0%
2014  33    6              18.2%
2015  40    11             27.5%
2016  46    12             26.1%
2017  54    13             24.1%
2018  54    19             35.2%
2019  65    21             32.3%
2020  71    21             29.6%
2021  65    22             33.8%
2022  70    24             34.3%
2023  79    31             39.2%
2024  96    28             29.2%
2025  122   54             44.3%
2026  46    12             26.1%
 
==============================================================================
10. THE JOIN TO design:ip_classification
==============================================================================
Population  N     classification[].target == 'ip-address'  share
----------  ----  ---------------------------------------  -----
ACTIVE      245   77                                       31.4%
SCAN        930   213                                      22.9%
corpus      5859  295                                      5.0%
PUBLISHED_IPCLS_ACTIVE 77 31.4%
PUBLISHED_IPCLS_SCAN 213 22.9%
PUBLISHED_IPCLS_CORPUS 295 5.0%
 
distinct raw resourceName strings on ACTIVE ip-address classifications: 116 (free text; folded on design:ip_classification, not here)
    5 MaxMind
    4 MaxMind GeoLite2
    3 IPinfo
    3 VirusTotal
    2 MaxMind GeoIP
    2 regular expressions (custom)
    2 ipinfo.io
    2 Censys
    2 ASdb
    2 IPinfo Lite
    1 custom response categorization
    1 Snacktime (custom implementation)
    1 five public and one commercial blacklist
    1 12 IP reputation blacklists
    1 MaxMind GeoLiteCity
 
==============================================================================
11. POSTERS AND SHORT RECORDS
==============================================================================
ACTIVE posters: 4 (1.6%); SCAN posters: 34 (3.7%)
ACTIVE records of <=4 pages: 6 (2.4%)
vantage location stated: all ACTIVE 55.5%, excluding <=4-page records 56.1%
PUBLISHED_ACTIVE_POSTERS 4
PUBLISHED_VANTAGE_FULLONLY 56.1%
 
==============================================================================
12. MEASURED RESULTS available for the page (detection[].prevalence on ACTIVE)
==============================================================================
ACTIVE papers with at least one detection[].prevalence: 244 (99.6%)
PUBLISHED_ACTIVE_PREVALENCE 244 99.6%
 
==============================================================================
Z. NON-CORPUS FIGURES ON THE PAGE (checked by external_checks_internet_scanning.sh)
==============================================================================
ZMap v4.4.0, released 2026-05-29; repo zmap/zmap not archived, pushed 2026-08-28
ZGrab2 v1.0.0, released 2025-12-04; repo zmap/zgrab2 pushed 2026-09-07
ZDNS v2.1.1, released 2026-05-28; repo zmap/zdns pushed 2026-09-09
masscan latest GitHub Release 1.3.2, 2021-01-31; repo pushed 2026-04-23, not archived
XMap 2.0.6, 2026-03-10; repo idealeer/xmap pushed 2026-03-10
LZR repo stanford-esrg/lzr pushed 2026-01-06, not archived
nmap 7.991 is the newest tarball on nmap.org/dist
ZMap v4.4.0 has ONE --blocklist-file flag (-b); there is no --blocklist-file-extra. The port flag is -p/--target-ports. ziterate, zblocklist and ztee ship in src/ alongside zmap
ZGrab2 input is CSV "IP, DOMAIN, TAG, PORT"; the multiple module takes a .ini
ZMap changed to random per-probe IP IDs in early 2024 (durumeric2024_years section 3)
ZMap defaults (v4.4.0 and main agree): --rate 10000 pps, --probes 1, --cooldown-time 8, --retries 10, --batch 64
ZMap conf/blocklist.conf ships 16 CIDR entries, all IANA special-purpose and multicast ranges, header dated 2013
IPv6 Hitlist Service open files, snapshot 2026-09-05: responsive-addresses 29,050,771 lines; aliased-prefixes 24,476,266; non-aliased-prefixes 222,836,153
Rapid7 Project Sonar: opendata.rapid7.com now redirects to sonardata.rapid7.com, which is commercial access only (8 datasets, 47,457 files, 60.7 TB)
search.censys.io returns HTTP 403 to automated clients; censys.com returns 200
2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)
pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe
 
==============================================================================
RESULT
==============================================================================
OK — all contracts hold

verify_scanning_figures-output.txt

==============================================================================
A. QUOTES AND PER-PAPER FIGURES
==============================================================================
  OK   [.cols] IMC/2024/ten-years-of-zmap
         over 33% of all Internet-wide IPv4 scan traffic can be fingerprinted as coming from ZMap
  OK   [.cols] IMC/2024/ten-years-of-zmap
         35.4% of all Internet-
  OK   [.cols] IMC/2024/ten-years-of-zmap
         TCP scan packets originated from ZMap
  OK   [.cols] IMC/2024/ten-years-of-zmap
         the best way to mitigate transient drop is to scan from 2-3 geographically and topologically div…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Provide a simple mechanism for operators to request exclusion from future scans. Indicate the IP…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Consider whether existing open source datasets provide the data you need
  OK   [.cols] IMC/2024/ten-years-of-zmap
         In early 2024, ZMap changed its default behavior to use random per-probe IP IDs
  OK   [.cols] IMC/2024/ten-years-of-zmap
         is typically driven by a handful of small service and cloud providers
  OK   [.cols] IMC/2024/ten-years-of-zmap
         Masscan [48] finds notably fewer hosts than ZMap, likely due to biases in its randomization algo…
  OK   [.cols] IMC/2024/ten-years-of-zmap
         None of the top 100 ASes that emit the most ZMap
  OK   [.cols] USENIX/2014/an-internet-wide-view-of-internet-wide-scanning
         only 208 organizations have requested that we exclude their networks from our scans, reducing th…
  OK   [.cols] IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
         we received two opt-out requests with
  OK   [.cols] IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery
         which we immediately complied
  OK   [.cols] USENIX/2021/lzr-identifying-unexpected-internet-services
         only 3% of HTTP and 6% of TLS services run on ports 80 and 443, respectively
  OK   [.cols] USENIX/2021/lzr-identifying-unexpected-internet-services
         Services on non-standard ports are more likely to be insecure
  OK   [.cols] USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
         a low scanning rate of 10,000 packets per second, total, across all networks
  OK   [.cols] IMC/2025/tracking-internet-disruptions-in-ukraine-insights-from-three-years-of-active-ful
         a low probing rate of 8,000 packets per second, i.e., around 500KB/s from a single vantage point
  OK   [.cols] CCS/2015/a-search-engine-backed-by-internet-wide-scanning
         All of the scanning hosts have WHOIS records and reverse DNS entries that describe the intent of…
  OK   [.cols] CCS/2015/a-search-engine-backed-by-internet-wide-scanning
         each scanning host runs a simple website on port 80 that describes the goals of the research
  OK   [.cols] USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale
         On the scanning IPs, a website gives project details and contact information for opt-out request…
  OK   [.cols] IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists
         identifies 1.5 % of our prefixes as aliased, pertaining to about half of our target addresses
  OK   [.cols] IMC/2022/rusty-clusters-dusting-an-ipv6-research-foundation
         More than 98 % of all IPv6 addresses announced by Fastly were labeled as aliased and Cloudflare …
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         Internet scanning has increased 30-fold over the last ten years
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         the ecosystem is extremely volatile, where targeted ports and geographical scanner locations dra…
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         this number has increased to 345 million in 2024
  OK   [.cols] IMC/2024/have-you-syn-me-characterizing-ten-years-of-internet-scanning
         in 2015 we observed 11 million scanning
 
A2. OFF-CORPUS PAPER (fetched copy, not in the extraction)
  OK   [out/scan/zmap2013_paper.txt] capable of scanning the IPv4 public address space over 1300 times faster than the most agg…
  OK   [out/scan/zmap2013_paper.txt] surveying the entire IPv4 address space in under 45 minutes from user space
  OK   [out/scan/zmap2013_paper.txt] over 97% of the theoretical maximum speed of gigabit Ethernet
  OK   [out/scan/zmap2013_paper.txt] an estimated 98% coverage of publicly available hosts
 
30 needles, 0 not located
 
==============================================================================
B. EXTERNAL FIGURES (re-fetched by external_checks_internet_scanning.sh)
==============================================================================
  ZMap v4.4.0 / 2026-05-29 / pushed 2026-08-28 / 6,374 stars
      GitHub REST API repos/zmap/zmap and /releases/latest, 2026-09-10
  ZGrab2 v1.0.0 / 2025-12-04 / pushed 2026-09-07
      GitHub REST API repos/zmap/zgrab2, 2026-09-10
  masscan 1.3.2 / 2021-01-31 / pushed 2026-04-23
      GitHub REST API repos/robertdavidgraham/masscan, 2026-09-10
  XMap 2.0.6 / 2026-03-10
      GitHub REST API repos/idealeer/xmap, 2026-09-10
  nmap 7.991
      newest nmap-*.tar.bz2 on https://nmap.org/dist/, 2026-09-10
  ZMap defaults: --rate 10000 pps, --probes 1, --cooldown-time 8
      src/zmap.1.ronn at tag v4.4.0 (byte-identical defaults to main), 2026-09-10
  1300x faster than the most aggressive Nmap default settings; under 45 minutes; 97% of theoretical maximum; 98% coverage
      Durumeric, Wustrow and Halderman, USENIX Security 2013 — the paper is NOT in this extraction; read from the authors' copy at zmap.io/paper.pdf (Last-Modified 2025-06-22) on 2026-09-10, extracted with pypdf. Needles A27-A30.
  41.5 / 0.97 = 42.8
      the page's own arithmetic reconciling scan_budget.py's theoretical 1 GbE figure with the ZMap paper's measured 45-minute ceiling
  174 / 56 = 3.1
      the page's own arithmetic: cites-the-norm over named-an-IRB, both from the report
  16 CIDR entries; 592,708,864 removed (13.8%); 3,702,258,432 in scope
      conf/blocklist.conf at tag v4.4.0, computed by pages/scan_budget.py
  4.29 days / 10.3 h / 41.5 min / 4.1 min; 7, 67, 1,000 and 10,000 Mb/s; 1,488,095 and 14,880,952 pps; 84 bytes
      pages/scan_budget.py run 2026-09-10, whole output quoted verbatim on the page
  17.14 days (--ports 2 --probes 2) and 34.28 days (--ports 8)
      pages/scan_budget.py with those flags, 2026-09-10
  0.15% of the IPv4 address space
      inside the durumeric2014_view quote, checked in needle A11
  IPv6 Hitlist open snapshot 2026-09-05: 29,050,771 / 24,476,266 / 222,836,153 lines
      alcatraz.net.in.tum.de/ipv6-hitlist-service/open/*.txt.xz, decompressed line counts, 2026-09-10
  Rapid7 Sonar: 8 datasets, 47,457 files, 60.7 TB, commercial access only
      sonardata.rapid7.com and /about with a browser User-Agent, 2026-09-10
  search.censys.io HTTP 403; censys.com HTTP 200
      curl with a browser User-Agent, 2026-09-10
 
==============================================================================
C. CODE CONSTANTS in pages/scan_budget.py and the example invocations
==============================================================================
  84
      WIRE_OVERHEAD_BYTES in scan_budget.py: 60-byte minimum frame + 4 FCS + 8 preamble/SFD + 12 IFG
  32 / 128
      IPv4 and IPv6 address widths in bits
  16777216 / 65536 / 33554432
      hand-computed /8, /16 and the mutation-guard double count in scan_budget.py self-test
  120 / 120.0 / 7199.0 / 7200 / 7200.0 / 171999.0 / 172800 / 172800.0 / 3600 / 3600.0 / 86400 / 86400.0 / 60 / 60.0 / 119.9 / 24.0 / 2.00
      second thresholds, divisors and the human_time() boundary cases in scan_budget.py --self-test check 9
  1e9 / 1e6
      bits per gigabit and per megabit in line_rate_pps() and scan_plan()
  8
      bits per byte in scan_plan(); also ZMap --cooldown-time default
  443 / 80 / 22 / 8080 / 53
      ports in the example invocations and in the ZGrab2 .ini example
  20260910
      the example --seed value: today's date, chosen so it is obviously arbitrary
  198.51.100.10 / 198.51.100.13 / 10.0.0.0 / 10.1.0.0 / 192.168.0.0 / 0.0.0.0
      RFC 5737 and RFC 1918 literals in the example command and the self-test
  4
      example --sender-threads; also the "four times" arithmetic for 2 ports x 2 probes
  1
      example --probes; ZMap default
  10000
      example --rate; ZMap default
  9 checks
      pages/scan_budget.py --self-test count
 
==============================================================================
D. NOT FIGURES
==============================================================================
  1918 / 5737 / 1122 / 6598 / 3927
      RFC numbers named in blocklist.conf and in the page text
  2013
      the year in the blocklist.conf header comment, and ZMap's release year
  1.0.0 / 4.4.0 / 2.0.6 / 1.3.2 / 7.991 / 2.1.1
      version strings
  30-fold / 2-3
      ratios inside quoted sentences
 
RESULT: OK

external_checks_internet_scanning-output.txt

=== GitHub repository state ===
  zmap/zmap                        archived=False pushed=2026-08-28T04:15:23Z stars=6374
  zmap/zgrab2                      archived=False pushed=2026-09-07T09:29:47Z stars=2173
  zmap/zdns                        archived=False pushed=2026-09-09T17:12:25Z stars=1135
  robertdavidgraham/masscan        archived=False pushed=2026-04-23T20:34:38Z stars=25996
  idealeer/xmap                    archived=False pushed=2026-03-10T14:15:08Z stars=491
  stanford-esrg/lzr                archived=False pushed=2026-01-06T17:27:02Z stars=196
 
=== Latest release AND newest tag (a tag-only repo 404s on /releases/latest; /tags is unsorted) ===
  zmap/zmap                        release=v4.4.0     published=2026-05-29T06:37:41Z   tags[0:6]=v4.4.0,v4.3.4,v4.3.3,v4.3.2,v4.3.1,v4.3.0
  zmap/zgrab2                      release=v1.0.0     published=2025-12-04T23:04:38Z   tags[0:6]=v1.0.0,v1.0.0-RC3,v1.0.0-RC2,v1.0.0-RC1,v0.2.0,v0.1.8
  zmap/zdns                        release=v2.1.1     published=2026-05-28T23:11:26Z   tags[0:6]=v2.1.1,v2.1.0,v2.0.5,v2.0.4,v2.0.3,v2.0.2
  robertdavidgraham/masscan        release=1.3.2      published=2021-01-31T09:19:32Z   tags[0:6]=v2,v1,kali_1_0,1.3.2,1.3.1,1.3.0
  idealeer/xmap                    release=2.0.6      published=2026-03-10T14:15:08Z   tags[0:6]=2.0.6,2.0.5,2.0.4,2.0.3,2.0.2,2.0.1
 
=== ZMap v4.4.0 man page: the flags and defaults the page publishes ===
     * `-p`, `--target-ports=port(s)`:
     * `-b`, `--blocklist-file=path`:
     * `-w`, `--allowlist-file=path`:
     * `-r`, `--rate=pps`:
     * `-B`, `--bandwidth=bps`:
     * `-c`, `--cooldown-time=secs`:
     * `-e`, `--seed=n`:
     * `-P`, `--probes=n`:
       sequentially for a given IP/Port, so if --probes=4, we'll send 4 probe packets
     * `--shards=N`:
     * `-S`, `--source-ip=ip|range`:
     * `-O`, `--output-module=name`:
     * `-f`, `--output-fields=fields`:
     * `-m`, `--metadata-file=filename`:
     * `-T`, `--sender-threads=n`:
  DEFAULT      at full line rate. Default: 10000 pps.
  DEFAULT      How long to continue receiving after sending has completed (default=8)
  DEFAULT      Number of probes to send to each IP/Port pair (default=1). Probes are sent
  DEFAULT      zmap (default=1). When sharding, **--seed** is required.
  OK: no --blocklist-file-extra (the page's claim)
 
=== ZMap ships ziterate / zblocklist / ztee ===
  ziterate.c       present
  zblocklist.c     present
  ztee.c           present
  zmap.c           present
 
=== conf/blocklist.conf at v4.4.0 and the arithmetic the page publishes ===
  lines mentioning "Updated 2013": 2
  OK: no opt-out entries in the shipped file (the page's claim)
  blocklist            /tmp/bl.conf
    CIDR entries       16
    addresses removed  592,708,864  (13.8% of IPv4)
    addresses in scope 3,702,258,432
 
  plan: 1 port(s) x 1 probe(s) per target = 3,702,258,432 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000     4.29 days          7 Mb/s
         100,000        10.3 h         67 Mb/s
       1,488,095      41.5 min      1,000 Mb/s
      14,880,952       4.1 min     10,000 Mb/s
 
  Reference rates: ZMap's default --rate is 10,000 pps; 1 GbE line rate is
  1,488,095 pps and 10 GbE is 14,880,952 pps at 84 bytes of wire time per probe.
 
  What this file does NOT tell you: whether anyone asked to be excluded.
  The shipped blocklist is IANA special-purpose space. Opt-out entries are
  yours to add, and to keep across runs.
  self-test OK (9 checks)
  plan: 2 port(s) x 2 probe(s) per target = 14,809,033,728 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000    17.14 days          7 Mb/s
  plan: 8 port(s) x 1 probe(s) per target = 29,618,067,456 packets
 
      rate (pps)    wall clock     wire rate
  --------------  ------------  ------------
          10,000    34.28 days          7 Mb/s
 
=== ZGrab2 README at v1.0.0: the input format the page quotes ===
  115:IP, DOMAIN, TAG, PORT
  118:Each line must specify `IP`, `DOMAIN`, or both.  If only `DOMAIN` is provided, scanners perform a DNS hostname lookup to determine the IP address.  If both `IP` and `DOMAIN` are provided, scanners connect to `IP` but use `DOMAIN` in protocol-specific contexts, such as the HTTP HOST header and TLS SNI extension.
  4:ZGrab is a fast, modular application-layer network scanner designed for completing large Internet-wide surveys. ZGrab is built to work with ZMap (ZMap identifies L4 responsive hosts, ZGrab performs in-depth, follow-up L7 handshakes). Unlike many other network scanners, ZGrab outputs detailed transcripts of network handshakes (e.g., all messages exchanged in a TLS handshake) for offline analysis.  
 
=== nmap: newest tarball on nmap.org/dist ===
  nmap-7.991.tar.bz2
 
=== IPv6 Hitlist Service (TUM) — open tier ===
  responsive-addresses   lines=29050771     Last-Modified: Sat, 05 Sep 2026 11:37:21 GMT
  aliased-prefixes       lines=24476266     Last-Modified: Sat, 05 Sep 2026 11:43:49 GMT
  non-aliased-prefixes   lines=222836153    Last-Modified: Sat, 05 Sep 2026 11:49:08 GMT
  ipv6hitlist.github.io http=200
 
=== Scan datasets: access terms ===
  opendata.rapid7.com   http=200 final=https://sonardata.rapid7.com/
  sonardata about: provides commercial access to data from Project Sonar
  Sign In (existing accounts only) | Datasets: 8 | Files: 47,457 | Total size: 60.7 TB
  search.censys.io      http=403
  censys.com            http=200
  shodan.io             http=200
 
=== The two USENIX author lists in the bibliography (venue records carry no authors) ===
  usenixsecurity24/presentation/williams
    Grant Williams
    Mert Erdemir
    Amanda Hsu
    Shraddha Bhat
    Abhishek Bhaskar
    Frank Li
    Paul Pearce
  usenixsecurity24/presentation/kaspereit
    Jonas Kaspereit
    Gurur Öndarö
    Gustavo Luvizotto Cesar
    Simon Ebbers
    Fabian Ising
    Christoph Saatjohann
    Mattijs Jonker
    Ralph Holz
    Sebastian Schinzel
 
DONE

scan_fold.mjs

// Instrument fold and written population boundary for programming:internet_scanning.
//
// WRITTEN BOUNDARY (fixed 2026-09-10, before the first figure was counted).
// The page is about ACTIVE INTERNET SCANNING: sending probe packets to an
// address set the researcher chose, over the public Internet, to learn which
// hosts answer and what they say.
//
//   SCAN    = the 930 papers tagged studyTypes 'network-scan-or-probe'. This is
//             the branch population design:automated_measurements routes with.
//             Every reporting-rate denominator on the page is a subset of it.
//   ACTIVE  = papers naming, as a tool they USED or PRODUCED, an instrument
//             whose folded kind is in MEMBERSHIP_KINDS below. This is the
//             page's own population.
//
// Membership is decided by the INSTRUMENT, not by tools[].category: a scanner
// filed under 'other' or 'infrastructure' is still a scanner (see
// tool-category-is-a-denominator). Concretely:
//   (a) any tools[] entry in ANY category whose name folds to a membership
//       family, plus
//   (b) any tools[] entry with category === 'network-scanner' whose name is
//       unmapped by (a) and whose HAND verdict below is a membership kind.
// Residue outside category 'network-scanner' is not hand-read — it would be
// the whole 16,737-name tool vocabulary — so ACTIVE is a floor and the report
// prints how many papers (a) adds beyond the category restriction.
//
// Reported beside ACTIVE but deliberately NOT in it, each with the page it is
// routed to:
//   dataset          someone else's scan (Censys, Shodan, FOFA, Rapid7 ...)
//   topology         traceroute / alias resolution / anycast census
//   dns              DNS-specific instruments            -> design:dns
//   vantage-platform RIPE Atlas, looking glasses         -> design:crawling_location
//   routing          BGP and AS-relationship inference
//   vuln-scanner     one host's application layer        -> security:web_vulnerabilities
//   passive          telescopes, sniffers, IDS
//   local-link       Wi-Fi / Bluetooth / cellular / LAN probes
//   perf, simulation, p2p-crawler, name-enumeration, packet-crafting, registry
//
// Every unmapped category==='network-scanner' string is a FAILURE, not a drop.
 
const F = (family, kind, ...patterns) => ({ family, kind, patterns, generic: false });
// A GENERIC family's regex is a *descriptor* ("a custom scanner", "ping"), not a
// coined product name, so it is only trusted inside tools[].category ===
// 'network-scanner'. Without that restriction /^custom\b/i alone puts 464
// papers in the population, almost all of them custom crawlers and scripts.
const G = (family, kind, ...patterns) => ({ family, kind, patterns, generic: true });
 
// Checked before the families. A name here is never folded, with its reason.
export const NEGATIVE = [
  [/^wizmap$/i, 'WizMap is an embedding-visualisation tool; /zmap/i matches it'],
];
 
// Ordered: first match wins. Specific products before the generic term they
// contain (ZGrab / ZDNS / ZMapv6 before ZMap; Paris traceroute before
// traceroute; Yarrp6 before Yarrp).
export const FAMILIES = [
  // --- address-space scanners and banner grabbers: the page's subject --------
  F('ZGrab', 'grabber', /zgrab/i),
  F('ZDNS', 'dns', /^zdns$/i),
  F('ZMap', 'active-scan', /z-?map/i),
  F('XMap', 'active-scan', /^x-?map\b/i),
  F('Masscan', 'active-scan', /masscan/i),
  F('nmap', 'active-scan', /n-?map/i),
  F('LZR', 'grabber', /^lzr$/i),
  F('Goscanner', 'grabber', /^goscanner$/i),
  F('TLS banner grabbers', 'grabber', /sslscan|sslyze|testssl|cipherscan|tls-?scanner|tls scanner|tls-crawler/i),
  F('JARM', 'grabber', /^jarm$/i),
  G('custom scanner', 'active-scan', /^custom\b|^(efficient|systematic) scanning (tool|software)$|^stateless scanner|^scanner application$|^scanning tool$/i),
 
  // --- IPv6 targeting: the only way a v6 "scan" has a target list ------------
  F('IPv6 Hitlist Service', 'ipv6-targeting', /hitlist|hit-list/i),
  F('6Gen', 'ipv6-targeting', /^6gen$/i),
  F('6Sense', 'ipv6-targeting', /^6sense$/i),
  F('6Graph', 'ipv6-targeting', /^6graph$/i),
  F('6Tree', 'ipv6-targeting', /^6tree$/i),
  F('Entropy/IP', 'ipv6-targeting', /entropy\/ip/i),
  F('addr6', 'ipv6-targeting', /^addr6$/i),
 
  // --- reachability sweeps ---------------------------------------------------
  G('ping', 'reachability', /^[hf]?ping\d?$/i, /^icmp( ping)?$/i, /^tcp pings?$/i, /^pingtools$/i),
  F('Trinocular', 'reachability', /^trinocular$/i),
 
  // --- remote censorship / reachability probing ------------------------------
  F('Censored Planet', 'censorship-probe', /censored planet/i),
  F('OONI', 'censorship-probe', /^ooni/i),
  F('Quack', 'censorship-probe', /^quack$/i),
  F('Hyperquack', 'censorship-probe', /^hyperquack$/i),
  F('Satellite', 'censorship-probe', /^satellite$/i),
  F('Augur', 'censorship-probe', /^augur$/i),
  F('Iris', 'censorship-probe', /^iris$/i),
  F('GeoInspector', 'censorship-probe', /^geoinspector$/i),
  F('Geneva', 'censorship-probe', /^geneva$/i),
 
  // --- scan datasets: someone else's scan ------------------------------------
  F('Censys', 'dataset', /censys/i),
  F('Shodan', 'dataset', /shodan/i),
  F('FOFA', 'dataset', /^fofa$/i),
  F('ZoomEye', 'dataset', /^zoomeye$/i),
  F('Rapid7 Open Data', 'dataset', /rapid7 (labs open data|sonar)/i),
  F('GreyNoise', 'dataset', /^greynoise$/i),
  F('LeakIX', 'dataset', /^leakix$/i),
  F('Open Resolver Project', 'dataset', /open resolver project/i),
 
  // --- topology --------------------------------------------------------------
  F('Yarrp', 'topology', /^(extended )?yarrp(6|box)?$/i),
  F('scamper', 'topology', /^scamper( python module)?$/i),
  F('Paris traceroute', 'topology', /paris traceroute/i),
  F('traIXroute', 'topology', /^traixroute$/i),
  F('mtr', 'topology', /^mtr$/i),
  F('MIDAR', 'topology', /^midar$/i),
  F('bdrmap', 'topology', /bdrmap/i),
  F('TNT', 'topology', /^p?y?tnt$/i),
  F('traceroute', 'topology', /traceroute/i),
 
  // --- routing ---------------------------------------------------------------
  F('RIPE RIS', 'routing', /ripe ris/i),
  F('RouteViews', 'routing', /^route ?views\b/i),
  F('BGPStream', 'routing', /bgpstream/i),
 
  // --- distributed vantage platforms ----------------------------------------
  F('RIPE Atlas', 'vantage-platform', /ripe atlas/i),
  F('perfSONAR', 'vantage-platform', /^perfsonar$/i),
  F('looking glass', 'vantage-platform', /looking glass|looking glasses|^periscope$/i),
 
  // --- DNS instruments -------------------------------------------------------
  F('massdns', 'dns', /^massdns$/i),
  F('fpdns', 'dns', /^fpdns$/i),
  F('OpenINTEL', 'dns', /openintel/i),
  F('dig/drill', 'dns', /^(dig|drill|nsupdate|unbound|dnspython|dnsviz)$/i),
 
  // --- passive ---------------------------------------------------------------
  F('Zeek/Bro', 'passive', /^(zeek|bro|bro ?-?ids?)$/i),
  F('Suricata', 'passive', /^suricata$/i),
  F('Snort', 'passive', /^snort$/i),
  F('p0f', 'passive', /^p0f\d?$/i),
  F('tshark/tcpdump', 'passive', /^(tshark|tcpdump|argus)$/i),
 
  // --- one host's application layer -----------------------------------------
  F('Nessus', 'vuln-scanner', /^nessus$/i),
  F('Acunetix', 'vuln-scanner', /acunetix/i),
  F('OpenVAS', 'vuln-scanner', /^openvas$/i),
  F('OWASP ZAP', 'vuln-scanner', /zed attack proxy|owasp zap|^zap$/i),
  F('Nikto', 'vuln-scanner', /^nikto$/i),
  F('sqlmap', 'vuln-scanner', /^sqlmap$/i),
];
 
export const KIND_OF = new Map(FAMILIES.map((f) => [f.family, f.kind]));
export const GENERIC_FAMILIES = new Set(FAMILIES.filter((f) => f.generic).map((f) => f.family));
 
// Kinds that put a paper in ACTIVE.
export const MEMBERSHIP_KINDS = new Set([
  'active-scan',
  'grabber',
  'reachability',
  'censorship-probe',
  'ipv6-targeting',
]);
 
export function foldScannerName(raw) {
  const v = String(raw).trim();
  for (const [re] of NEGATIVE) if (re.test(v)) return null;
  for (const { family, patterns } of FAMILIES) {
    for (const re of patterns) {
      if (re instanceof RegExp && re.test(v)) return family;
    }
  }
  return null;
}
 
// ---------------------------------------------------------------------------
// HAND verdicts for every category==='network-scanner' name the families above
// do not fold. One entry per distinct string, read off the tool's own
// `purpose` field and, where the name alone was ambiguous, the paper's title.
// The report throws on an unmapped string and on an entry nothing matches.
export const HAND = {
  '0trace': 'topology',
  '5GSniffer': 'local-link',
  'active DNS crawler': 'dns',
  'active scanning tool': 'dns',                 // "Query DKIM, SPF, DMARC records"
  'active-geolocator': 'topology',
  'AdaptOver': 'local-link',
  'Airmon-ng': 'local-link',
  'Ally': 'topology',
  'amass': 'name-enumeration',
  'AmpMap': 'active-scan',
  'Anaximander': 'topology',
  'Aquantone': 'name-enumeration',
  'Arachni': 'vuln-scanner',
  'arpscan': 'local-link',
  'Bitnodes': 'p2p-crawler',
  'BitTorrent crawler': 'p2p-crawler',
  'Black Widow': 'vuln-scanner',
  'Bluesniff': 'local-link',
  'BreadCrumb': 'censorship-probe',
  'Btsniffer': 'local-link',
  'BurpSuite Scanner': 'vuln-scanner',
  'CAIDA AS Relationships Dataset': 'routing',
  'Cenzic HailStorm Pro': 'vuln-scanner',
  'Chaos': 'simulation',
  'CheckHost': 'vantage-platform',
  'cloud_enum': 'name-enumeration',
  'CnCHunter': 'active-scan',
  'Commix': 'vuln-scanner',
  'COVScan': 'vuln-scanner',
  'Critical.IO': 'dataset',
  'ct-sans': 'dataset',
  'danglingcname': 'dns',
  'Diamond Miner': 'topology',
  'DMiner': 'dns',
  'dMAP (DPI Mapper)': 'active-scan',
  'tcp_probe': 'passive',
  'TMC': 'censorship-probe',
  'ThunderPing': 'vantage-platform',
  'DNS crawler': 'dns',
  'DNS Scanner': 'dns',
  'DNS X-Ray Tool': 'dns',
  'DoE-Hunter': 'dns',
  'DTLS-Fuzzer': 'lab-protocol-test',
  'DTLS-MitM-Scanner (DMS)': 'active-scan',
  'EAST': 'active-scan',                          // STARTTLS test cases against live MTAs
  'Enemy of the State': 'vuln-scanner',
  'ERIS': 'active-scan',
  'exitmap': 'active-scan',
  'Exitmap': 'active-scan',
  'FastMDA': 'topology',
  'FingerprinTLS': 'passive',
  'FlashRoute': 'topology',
  'forwarders-based scanner': 'active-scan',
  'gateway checker': 'active-scan',
  'Google Tsunami': 'vuln-scanner',
  'gQUIC-module': 'grabber',
  'h2spec': 'lab-protocol-test',
  'HP WebInspect': 'vuln-scanner',
  'HTTP/OneProbe': 'topology',
  'IBM Rational AppScan': 'vuln-scanner',
  'ibportstate': 'local-link',
  'ICMP Paris': 'topology',
  'ICMP_Sonar': 'active-scan',
  'iffinder': 'topology',
  'iGreedy': 'topology',
  'ike-scan': 'grabber',
  'Internet Census': 'dataset',
  'iperf': 'perf',
  'iPerf3': 'perf',
  'IPFS Crawler': 'p2p-crawler',
  'IRBlock': 'censorship-probe',
  'IRLscanner': 'active-scan',
  'IRTT': 'perf',
  'Iterative Network Tracing (INT)': 'topology',
  'James': 'topology',
  'kapar': 'topology',
  'Karma': 'active-scan',
  'l2ping': 'local-link',
  'LACeS': 'topology',
  'LanDscAPe': 'active-scan',
  'LFP': 'topology',
  'LTE PROBE': 'local-link',
  'Lumscan': 'censorship-probe',
  'MacStealer': 'local-link',
  'MAnycast anycast census': 'topology',
  'MAnycast2': 'topology',
  'Mao et al. AS-path inference algorithm': 'routing',
  'MAP-IT': 'routing',
  'MarcoPolo': 'routing',
  'McAfee SECURE': 'vuln-scanner',
  'Mercator': 'topology',
  'Mint': 'censorship-probe',
  'modified ntpdedup': 'grabber',
  'Monocle': 'censorship-probe',
  'mrinfo': 'topology',
  'mrinfo-rec': 'topology',
  'N-Stalker QA Edition': 'vuln-scanner',
  'Namehelp': 'vantage-platform',
  'nc': 'active-scan',
  'Netcat': 'active-scan',
  'ndisc6': 'local-link',
  'NetfilterQueue': 'packet-crafting',
  'netperf': 'perf',
  'netstat': 'host-tool',
  'Netstat': 'host-tool',
  'Network Analyzer': 'topology',
  'NextTrace': 'topology',
  'node probing program': 'active-scan',
  'NodeFinder': 'p2p-crawler',
  'NR-Scope': 'local-link',
  'ns-3': 'simulation',
  'Nuclei': 'vuln-scanner',
  'nuttcp': 'perf',
  'OneProbe': 'topology',
  'onesixtyone': 'grabber',
  'onion-grab': 'grabber',
  'OSAVRoute': 'active-scan',
  'PciCheckerLite': 'vuln-scanner',
  'Perf': 'host-tool',
  'PLCScan': 'grabber',
  'Prefixscan': 'topology',
  'Python script': 'active-scan',                 // "Scan IPv4 space with DNS queries"
  'QScanner': 'grabber',
  'QualysGuard PCI': 'vuln-scanner',
  'quicgrabber': 'grabber',
  'RAINBOW': 'p2p-crawler',
  'Rapid7 NeXpose': 'vuln-scanner',
  'RAT-Scan': 'active-scan',
  'RDScan': 'dns',
  'revtr 2.0': 'topology',
  'RMap': 'dns',
  'SAD DNS': 'dns',
  'SadDNS': 'dns',
  'Scapy': 'packet-crafting',
  'SecSpider': 'dataset',
  'Shadow': 'simulation',
  'ShaperProbe': 'perf',
  'Skipfish': 'vuln-scanner',
  'smbmap': 'grabber',
  'SNMPv3': 'grabber',
  'Speedtrap': 'topology',
  'Spoki': 'passive',
  'Spoofer project': 'vantage-platform',
  'SSFNet': 'simulation',
  'Swordqiu': 'routing',
  'Time Series Latency Probing (TSLP)': 'topology',
  'TLS-Attacker': 'lab-protocol-test',
  'Tor DNSEL': 'dataset',
  'TorFlow': 'p2p-crawler',
  'tracebox': 'topology',
  'tracefilter': 'topology',
  'tracenet': 'topology',
  'Trufflehunter': 'dns',
  'Tsunami': 'vuln-scanner',
  'Ubertooth One': 'local-link',
  'Unicorn': 'active-scan',
  'University of Michigan Daily Full IPv4 HTTPS Handshakes': 'dataset',
  'URHunter': 'dns',
  'v6disc.sh': 'local-link',
  'Verfploeter': 'topology',
  'Verloc': 'topology',
  'VerLoc': 'topology',
  'W3af': 'vuln-scanner',
  'Wapiti': 'vuln-scanner',
  'WeHeY': 'perf',
  'WhatWeb': 'grabber',
  'whois': 'registry',
  'WiFi Inspector': 'local-link',
  'WMap': 'vuln-scanner',
  'WPScan': 'vuln-scanner',
  'wrk': 'perf',
  'Xray': 'vuln-scanner',
  'YoDNS': 'dns',
  'ZreFinder': 'dns',
};
 
// A hand verdict is a family of one: its display name is the string itself.
//
// `category` is the tools[] category the name was found under. Hand verdicts
// and GENERIC families are only trusted under 'network-scanner', because those
// strings ("Python script", "Unicorn", "custom scanner", "nc") are ordinary
// words elsewhere in the tool vocabulary. Pass category === null to fold a bare
// string, which is what the self-test does.
export function kindOfName(raw, category = null) {
  const fam = foldScannerName(raw);
  if (fam) {
    if (GENERIC_FAMILIES.has(fam) && category !== null && category !== 'network-scanner') return null;
    return { family: fam, kind: KIND_OF.get(fam), via: 'family' };
  }
  const v = String(raw).trim();
  if (Object.prototype.hasOwnProperty.call(HAND, v)) {
    if (category !== null && category !== 'network-scanner') return null;
    return { family: v, kind: HAND[v], via: 'hand' };
  }
  return null;
}
 
export const ROUTED_TO = {
  'dns': 'design:dns',
  'vantage-platform': 'design:crawling_location',
  'vuln-scanner': 'security:web_vulnerabilities',
  'dataset': 'design:existing_datasets',
  'topology': 'design:ip_classification',
  'routing': 'design:ip_classification',
};
 
function selfTest() {
  const cases = [
    ['ZMap', 'ZMap', 'active-scan'],
    ['zmap', 'ZMap', 'active-scan'],
    ['ZMapv6', 'ZMap', 'active-scan'],
    ['zmap6', 'ZMap', 'active-scan'],
    ['ZMap-based script', 'ZMap', 'active-scan'],
    ['TCP Options module for ZMap', 'ZMap', 'active-scan'],
    ['ZGrab/ZMap', 'ZGrab', 'grabber'],
    ['zgrab2', 'ZGrab', 'grabber'],
    ['ZGrab 2.0', 'ZGrab', 'grabber'],
    ['XMap', 'XMap', 'active-scan'],
    ['Xmap (extended)', 'XMap', 'active-scan'],
    ['Masscan', 'Masscan', 'active-scan'],
    ['Nmap Security Scanner', 'nmap', 'active-scan'],
    ['nmap plugin for Special DROWN', 'nmap', 'active-scan'],
    ['LZR', 'LZR', 'grabber'],
    ['Goscanner', 'Goscanner', 'grabber'],
    ['TLS-Crawler', 'TLS banner grabbers', 'grabber'],
    ['custom scanner', 'custom scanner', 'active-scan'],
    ['efficient scanning software', 'custom scanner', 'active-scan'],
    ['stateless scanner (custom)', 'custom scanner', 'active-scan'],
    ['systematic scanning tool', 'custom scanner', 'active-scan'],
    ['scanner application', 'custom scanner', 'active-scan'],
    ['ARGUS', 'tshark/tcpdump', 'passive'],
    ['Routeviews BGP table', 'RouteViews', 'routing'],
    ['Tnt', 'TNT', 'topology'],
    ['active scanning tool', 'active scanning tool', 'dns'],
    ['IPv6 Hitlist Service', 'IPv6 Hitlist Service', 'ipv6-targeting'],
    ['Hitlist-Downloader', 'IPv6 Hitlist Service', 'ipv6-targeting'],
    ['6Gen', '6Gen', 'ipv6-targeting'],
    ['6Sense', '6Sense', 'ipv6-targeting'],
    ['addr6', 'addr6', 'ipv6-targeting'],
    ['ping', 'ping', 'reachability'],
    ['hping3', 'ping', 'reachability'],
    ['fping', 'ping', 'reachability'],
    ['ICMP ping', 'ping', 'reachability'],
    ['TCP pings', 'ping', 'reachability'],
    ['Censys', 'Censys', 'dataset'],
    ['Censys Search API', 'Censys', 'dataset'],
    ['ZoomEye', 'ZoomEye', 'dataset'],
    ['Zoomeye', 'ZoomEye', 'dataset'],
    ['FOFA', 'FOFA', 'dataset'],
    ['Fofa', 'FOFA', 'dataset'],
    ['Rapid7 Sonar', 'Rapid7 Open Data', 'dataset'],
    ['Yarrp6', 'Yarrp', 'topology'],
    ['Yarrpbox', 'Yarrp', 'topology'],
    ['extended Yarrpbox', 'Yarrp', 'topology'],
    ['scamper Python module', 'scamper', 'topology'],
    ['Paris traceroute', 'Paris traceroute', 'topology'],
    ['tcptraceroute', 'traceroute', 'topology'],
    ['nmap traceroute', 'nmap', 'active-scan'],  // nmap wins: it is the instrument
    ['PyTNT', 'TNT', 'topology'],
    ['TNT', 'TNT', 'topology'],
    ['Route Views', 'RouteViews', 'routing'],
    ['RIPE Atlas', 'RIPE Atlas', 'vantage-platform'],
    ['public looking glasses', 'looking glass', 'vantage-platform'],
    ['Periscope', 'looking glass', 'vantage-platform'],
    ['dig', 'dig/drill', 'dns'],
    ['drill', 'dig/drill', 'dns'],
    ['DNSViz', 'dig/drill', 'dns'],
    ['Bro IDS', 'Zeek/Bro', 'passive'],
    ['p0f3', 'p0f', 'passive'],
    ['Tshark', 'tshark/tcpdump', 'passive'],
    ['Zed Attack Proxy (ZAP)', 'OWASP ZAP', 'vuln-scanner'],
    ['OWASP ZAP', 'OWASP ZAP', 'vuln-scanner'],
    ['Nikto', 'Nikto', 'vuln-scanner'],
    ['Quack', 'Quack', 'censorship-probe'],
    ['Satellite', 'Satellite', 'censorship-probe'],
    ['OONI Probe', 'OONI', 'censorship-probe'],
    // hand verdicts
    ['LanDscAPe', 'LanDscAPe', 'active-scan'],
    ['Scapy', 'Scapy', 'packet-crafting'],
    ['iperf', 'iperf', 'perf'],
    ['Ubertooth One', 'Ubertooth One', 'local-link'],
    ['whois', 'whois', 'registry'],
  ];
  for (const [raw, wantFam, wantKind] of cases) {
    const got = kindOfName(raw);
    if (!got || got.family !== wantFam || got.kind !== wantKind) {
      throw new Error(
        `scan_fold self-test: ${JSON.stringify(raw)} -> ${JSON.stringify(got)}, want ${wantFam}/${wantKind}`
      );
    }
  }
  // NEGATIVE guard: /z-?map/i matches an unrelated visualisation tool.
  if (foldScannerName('WizMap') !== null) throw new Error('scan_fold self-test: WizMap must not fold');
  // GENERIC and HAND names are category-restricted; distinctive names are not.
  if (kindOfName('custom scanner', 'other') !== null) throw new Error('scan_fold self-test: generic family must not fire outside network-scanner');
  if (kindOfName('custom scanner', 'network-scanner') === null) throw new Error('scan_fold self-test: generic family must fire inside network-scanner');
  if (kindOfName('Python script', 'other') !== null) throw new Error('scan_fold self-test: hand verdict must not fire outside network-scanner');
  if (kindOfName('ZMap', 'infrastructure') === null) throw new Error('scan_fold self-test: distinctive name must fire in any category');
  if (kindOfName('LEGO Mindstorms NXT') !== null) throw new Error('scan_fold self-test: unknown name must stay residue');
  for (const k of Object.values(HAND)) {
    if (typeof k !== 'string' || !k.length) throw new Error('scan_fold self-test: empty HAND kind');
  }
  if (MEMBERSHIP_KINDS.has('dataset')) throw new Error('scan_fold self-test: dataset must not be membership');
}
 
selfTest();

report_internet_scanning.mjs

#!/usr/bin/env node
// Every figure on programming:internet_scanning, with its denominator.
//
// Population boundary is written down in scripts/scan_fold.mjs and was fixed
// before the first count. Two denominators, never mixed:
//   SCAN   930 papers tagged studyTypes 'network-scan-or-probe'
//   ACTIVE 245 papers naming an active-scan / grabber / reachability /
//          censorship-probe / IPv6-targeting instrument they used or produced
//
// Flags: --wiki (DokuWiki tables)  --list <kind>  --quotes <regex>  --hits <regex>
//
// Exits 1 if a corpus contract, the fold coverage check, or a hand-map
// consistency check fails.
 
import fs from 'node:fs';
import path from 'node:path';
import { loadExtractions, pct, table, wikiTable, isSentinel, dataRoot, YEAR_BUCKETS } from './lib.mjs';
import { kindOfName, MEMBERSHIP_KINDS, HAND, KIND_OF, ROUTED_TO, GENERIC_FAMILIES } from './scan_fold.mjs';
 
const argv = process.argv.slice(2);
const WIKI = argv.includes('--wiki');
const opt = (n) => { const i = argv.indexOf(n); return i >= 0 ? argv[i + 1] : null; };
const LIST = opt('--list');
const QUOTES = opt('--quotes');
const HITS = opt('--hits');
 
const out = [];
const say = (s = '') => out.push(s);
const h = (s) => { say(''); say('='.repeat(78)); say(s); say('='.repeat(78)); };
const t = (headers, rows) => say(WIKI ? wikiTable(headers, rows) : table(headers, rows));
const fails = [];
const fail = (m) => { fails.push(m); say(`FAILURE: ${m}`); };
 
// ---------------------------------------------------------------- corpus ----
const rows = loadExtractions();
const DROVE = new Set(['used', 'produced']);
const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
 
if (rows.length !== 5859) fail(`corpus ${rows.length} != 5859`);
 
const SCAN = rows.filter((p) => p.studyTypes.includes('network-scan-or-probe'));
if (SCAN.length !== 930) fail(`network-scan-or-probe ${SCAN.length} != 930 (OVERVIEW.md)`);
 
// Every instrument a paper names, folded. Category is passed through so the
// generic families and the hand map stay inside category 'network-scanner'.
function instruments(p) {
  const seen = new Map(); // family -> {family, kind, via, names:Set, versions:Set}
  for (const tool of p.tools) {
    if (!DROVE.has(tool.usedOrMentioned)) continue;
    const v = kindOfName(tool.name, tool.category);
    if (!v) continue;
    if (!seen.has(v.family)) seen.set(v.family, { ...v, names: new Set(), versions: new Set(), cats: new Set() });
    const e = seen.get(v.family);
    e.names.add(String(tool.name).trim());
    e.cats.add(tool.category);
    if (!isSentinel(tool.version)) e.versions.add(String(tool.version));
  }
  return [...seen.values()];
}
const memberOf = (p) => instruments(p).filter((v) => MEMBERSHIP_KINDS.has(v.kind));
const ACTIVE = rows.filter((p) => memberOf(p).length > 0);
const ACTIVE_KEYS = new Set(ACTIVE.map(key));
const SCAN_KEYS = new Set(SCAN.map(key));
 
// --------------------------------------------------------------- probes ----
// Two widths per practice. The loose one is the recall claim; the tight one
// requires the authors to be the subject of the sentence. Both are upper
// bounds, and section 8 measures the precision of the tight one by hand.
const PROBE = {
  rateLoose: /(rate[- ]?limit|packets? per second|\bpps\b|probing rate|scan(ning)? rate|bandwidth (of|cap|limit|is set)|--bandwidth|-B \d)/i,
  rateTight: /((we|our (scan|scanner|probe|measurement|experiment)s?)[^.]{0,120}(rate[- ]?limit|limit(ed)? (the |our )?(scan|probe|probing|sending|bandwidth|rate)|throttl)|(we|our)[^.]{0,80}(scan|prob|sent|send)[^.]{0,80}at (a )?(rate of |about |approximately )?\d[\d,.]*\s?(k|m|g)?(bps|b\/s|pps|packets)|(probing|scan(ning)?|sending) rate of \d|limit(ed)? (the |our )?(scan|probe|probing|bandwidth|rate)[^.]{0,60}\d)/i,
  blockLoose: /(black ?list|block ?list|exclusion (list|file)|excluded? (address|IP|prefix|network|range)|--blacklist-file|--blocklist-file)/i,
  blockTight: /((we|our)[^.]{0,140}(black ?list|block ?list|exclusion (list|file))|(black ?list|block ?list|exclusion list)[^.]{0,100}(we |our |scan)|--black(list)?-file|--block(list)?-file|honou?r(ed|ing)?[^.]{0,40}(black ?list|block ?list|exclusion|opt-?out|removal)|exclud(e|ed|ing)[^.]{0,80}(IP|address|prefix|network|range|subnet)[^.]{0,80}(scan|prob|request|our))/i,
  optLoose: /(opt(ed)?[- ]out|removal request|asked (us )?to be excluded)/i,
  optTight: /((we|our)[^.]{0,140}opt(ed)?[- ]out|opt[- ]out (request|list|form|page|mechanism)|(request(ed|s)?|asked)[^.]{0,60}(be )?(excluded|removed) (from|our)|removal request)/i,
  srcLoose: /(source (IP|address)|dedicated (IP|machine|server|host)|scanning (host|machine|server|IP)|reverse DNS|rDNS|PTR record|WHOIS)/i,
  srcTight: /((we|our)[^.]{0,120}(dedicated (IP|machine|server|host|address)|scanning (host|machine|server|IP|address)|(source|scanner) (IP|address)(es)?|our (scanner|scanning) (IP|host|machine))|(reverse DNS|rDNS|PTR record|WHOIS (record|entry|entries))[^.]{0,120}(our |we |explain|scan))/i,
  abuseLoose: /(abuse (complaint|report|email|e-mail|contact|team)|abuse@|complaints?)/i,
  abuseTight: /((we|our)[^.]{0,140}(abuse (complaint|report|email|e-mail|contact)|complaint)|abuse@|(received|got) [^.]{0,40}(complaint|abuse))/i,
  irb: /(\bIRB\b|institutional review board|ethics (board|committee|review|approval)|REB approval)/i,
  goodpractice: /(Durumeric|Partridge and Allman|Menlo Report|scanning best practice|good (Internet|internet) citizen)/i,
};
 
// Hand verdicts. 'own' = the matched sentence is about the authors' own scan.
// 'other' = the term is in the paper for a different reason (a target's rate
// limiting, a censor's blocklist, a defence recommendation, an analysis-time
// exclusion). Read on 2026-09-10 from paper.cols.txt with whitespace collapsed,
// first two matches per paper. Every entry names the deciding reading.
const RATE_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other',   // rate-limits its own *web interface*, not its scan
  'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study
  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'other', // "sending ARP requests at 100 Mbps" is an attack rate on a TTEthernet lab testbed, not a scan rate. Read as 'own' on the first pass and flipped after the generic review pass disputed it on 2026-09-10
  'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled
  'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers"
  'IMC/2025/chaos-in-the-chain-evaluate-deployment-and-construction-compliance-of-web-pki-ce': 'own', // "we limited the scanning rate to below 500KB/s"
  'NDSS/2020/measuring-the-deployment-of-network-censorship-filters-at-global-scale': 'own', // "we abide by best practices and limit the rate of sending requests"
  'NDSS/2024/proof-of-backhaul-trustfree-measurement-of-broadband-bandwidth': 'other', // tc-shaped test rig, not a scan rate
  'USENIX/2011/toward-secure-embedded-web-interfaces': 'other', // the embedded device throttles requests
  'USENIX/2021/accurately-measuring-global-risk-of-amplification-attacks-using-ampmap': 'other', // "block or rate-limit specific queries" is the defence it recommends
  'USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications': 'own', // "a low scanning rate of 10,000 packets per second"
  'USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch': 'own', // "at a rate of 100 packets per second"
};
const BLOCK_VERDICT = {
  'CCS/2018/practical-state-recovery-attacks-against-legacy-rng-implementations': 'own', // "respecting hosts who wished to be blacklisted"
  'IEEE-SP/2017/augur-internet-wide-detection-of-connectivity-disruptions': 'other', // block lists of *censored sites*, used as validation data
  'IMC/2011/detecting-validating-and-characterizing-computer-infections-in-the-wild': 'other', // blacklists are an input dataset
  'IMC/2016/measuring-the-security-harm-of-tls-crypto-shortcuts': 'own', // "followed the institutional blacklist"
  'IMC/2018/clusters-in-the-expanse-understanding-and-unbiasing-ipv6-hitlists': 'own', // "maintaining a blacklist and using dedicated servers"
  'IMC/2022/deep-dive-into-the-iot-backend-ecosystem': 'other', // FireHOL blocklist is the object of measurement
  'IMC/2023/ecn-with-quic-challenges-in-the-wild': 'own', // "how to be added to our blocklist to be excluded from further measurements"
  'NDSS/2020/decentralized-control-a-case-study-of-russia': 'other', // Roskomnadzor blocklists are the subject
  'NDSS/2025/revealing-the-black-box-of-device-search-engine-scanning-assets-strategies-and-ethical-consideration': 'other', // operators blocklisting the scanners it studies
  'USENIX/2014/an-internet-wide-view-of-internet-wide-scanning': 'own', // "only 208 organizations have requested that we exclude their networks from our scans" — 208, checked against paper.pdf with pypdf; a first read of a narrow context window misread it as 908
  'USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship': 'own', // "respecting an IP blacklist shared with other scanning research"
  'USENIX/2022/spoki-unveiling-a-new-wave-of-scanners-through-a-reactive-network-telescope': 'other', // analysis-time exclusion of known scan projects
};
const OPT_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'own', // "we treat opt-out requests for scanning as a request to be removed from the search index"
  'IEEE-SP/2020/meddling-middlemen-empirical-analysis-of-the-risks-of-data-saving-mobile-browser': 'other', // browser users opting out of data saving
  'IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner': 'own', // TXT records pointing at a page with opt-out instructions
  'IMC/2018/in-the-ip-of-the-beholder-strategies-for-active-ipv6-topology-discovery': 'own', // "we received two opt-out requests with which we immediately complied"
  'IMC/2022/deep-dive-into-the-iot-backend-ecosystem': 'own', // "We run a Web server with experiment and opt-out information"
  'IMC/2024/destination-reachable-what-icmpv6-error-messages-reveal-about-their-sources': 'own', // "We received a single request for opt-out and acted accordingly"
  'NDSS/2024/understanding-the-implementation-and-security-implications-of-protective-dns-services': 'own', // PTR records giving contacts for opt-out
  'PETS/2023/certainty-detecting-dns-manipulation-at-scale-using-tls-certificates': 'own', // "offer administrators the option to opt out of our scanning"
  'USENIX/2018/quack-scalable-remote-measurement-of-application-layer-censorship': 'own', // "simple ways for packet recipients to opt out of future probes"
  'USENIX/2024/landscape-exploring-ldap-weaknesses-and-data-leaks-at-internet-scale': 'own', // "a website gives project details and contact information for opt-out requests"
};
const SRC_VERDICT = {
  'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'own', // "All of the scanning hosts have WHOIS records and reverse DNS entries"
  'IEEE-SP/2010/investigation-of-triangular-spamming-a-stealthy-and-efficient-spamming-technique': 'other', // source-IP rewriting is the attack under study
  'IMC/2010/demystifying-service-discovery-implementing-an-internet-wide-scanner': 'own', // describes its own source IPs and how many it used
  'IMC/2016/identifying-and-aggregating-homogeneous-ipv4-24-blocks-with-hobbit': 'other', // rDNS patterns of the *targets*
  'IMC/2019/an-end-to-end-large-scale-measurement-of-dns-over-encryption-how-far-have-we-com': 'own', // "setting a reverse DNS record for our scanning system and building a website"
  'IMC/2023/does-it-spin-on-the-adoption-and-use-of-quics-spin-bit': 'own', // rDNS entries, project name in every request, dedicated website
  'NDSS/2020/on-using-application-layer-middlebox-protocols-for-peeking-behind-nat-gateways': 'own', // "The reverse DNS record of the scanning host was set to indicate its use for research purposes"
  'NDSS/2026/osavroute-advancing-outbound-source-address-validation-deployment-detection-with-non-cooperative-measurement': 'other', // source-address validation is the subject
  'USENIX/2021/lzr-identifying-unexpected-internet-services': 'other', // discusses scanners' source IPs as a defence property
  'USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications': 'own', // "The webpage hosted on each scanning IP/domain includes a contact email"
};
 
// Every non-corpus number that appears on the page, so check_page_numbers.mjs
// can run whole-page. Re-checked by scripts/external_checks_internet_scanning.sh.
const EXTERNAL = [
  'ZMap v4.4.0, released 2026-05-29; repo zmap/zmap not archived, pushed 2026-08-28',
  'ZGrab2 v1.0.0, released 2025-12-04; repo zmap/zgrab2 pushed 2026-09-07',
  'ZDNS v2.1.1, released 2026-05-28; repo zmap/zdns pushed 2026-09-09',
  'masscan latest GitHub Release 1.3.2, 2021-01-31; repo pushed 2026-04-23, not archived',
  'XMap 2.0.6, 2026-03-10; repo idealeer/xmap pushed 2026-03-10',
  'LZR repo stanford-esrg/lzr pushed 2026-01-06, not archived',
  'nmap 7.991 is the newest tarball on nmap.org/dist',
  'ZMap v4.4.0 has ONE --blocklist-file flag (-b); there is no --blocklist-file-extra. The port flag is -p/--target-ports. ziterate, zblocklist and ztee ship in src/ alongside zmap',
  'ZGrab2 input is CSV "IP, DOMAIN, TAG, PORT"; the multiple module takes a .ini',
  'ZMap changed to random per-probe IP IDs in early 2024 (durumeric2024_years section 3)',
  'ZMap defaults (v4.4.0 and main agree): --rate 10000 pps, --probes 1, --cooldown-time 8, --retries 10, --batch 64',
  'ZMap conf/blocklist.conf ships 16 CIDR entries, all IANA special-purpose and multicast ranges, header dated 2013',
  'IPv6 Hitlist Service open files, snapshot 2026-09-05: responsive-addresses 29,050,771 lines; aliased-prefixes 24,476,266; non-aliased-prefixes 222,836,153',
  'Rapid7 Project Sonar: opendata.rapid7.com now redirects to sonardata.rapid7.com, which is commercial access only (8 datasets, 47,457 files, 60.7 TB)',
  'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200',
  '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)',
  'pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',
];
 
h('0. FOLD COVERAGE — every category=network-scanner name must be decided');
{
  const un = new Map();
  const usedHand = new Set();
  const usedFam = new Set();
  let tuples = 0;
  for (const p of rows) {
    for (const tool of p.tools) {
      if (!DROVE.has(tool.usedOrMentioned) || tool.category !== 'network-scanner') continue;
      tuples += 1;
      const v = kindOfName(tool.name, tool.category);
      if (!v) { un.set(String(tool.name).trim(), (un.get(String(tool.name).trim()) || 0) + 1); continue; }
      if (v.via === 'hand') usedHand.add(String(tool.name).trim()); else usedFam.add(v.family);
    }
  }
  say(`category='network-scanner' used/produced tuples: ${tuples}`);
  say(`unmapped strings: ${un.size}`);
  for (const [k, v] of [...un].sort()) say(`  UNMAPPED ${v}  ${JSON.stringify(k)}`);
  if (un.size > 0) fail(`${un.size} unmapped network-scanner strings — every one must get a family or a HAND verdict`);
  const stale = Object.keys(HAND).filter((k) => !usedHand.has(k));
  say(`HAND entries: ${Object.keys(HAND).length}, matched: ${usedHand.size}, stale: ${stale.length}`);
  for (const s of stale) say(`  STALE HAND ${JSON.stringify(s)}`);
  if (stale.length > 0) fail(`${stale.length} HAND entries match nothing — a shadowed verdict is a lie about what was read`);
}
 
h('0b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)');
{
  const inExtract = rows.filter((p) => p.year === 2013 && p.venue === 'USENIX' && /zmap|internet-wide-scanning/i.test(p.slug));
  const ftDir = path.join(dataRoot(), 'fulltext/2013/USENIX');
  const ftHits = fs.existsSync(ftDir) ? fs.readdirSync(ftDir).filter((x) => /zmap|scann/i.test(x)) : [];
  const metaFile = path.join(dataRoot(), 'corpus2/.meta/USENIX-2013.json');
  const meta = JSON.parse(fs.readFileSync(metaFile, 'utf8'));
  const recs = Array.isArray(meta) ? meta : (meta.papers ?? meta.records ?? []);
  const metaHits = recs.filter((r) => /zmap/i.test(JSON.stringify(r)));
  say(`extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: ${inExtract.length}`);
  say(`data/fulltext/2013/USENIX directories matching /zmap|scann/i:            ${ftHits.length}`);
  say(`corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 ${metaHits.length} of ${recs.length}`);
  say(`PUBLISHED_ZMAP2013_ABSENT ${inExtract.length === 0 && ftHits.length === 0 && metaHits.length === 0 ? 'yes' : 'NO — the page says it is absent'}`);
  say(`PUBLISHED_USENIX2013_INDEX ${recs.length}`);
  if (inExtract.length || ftHits.length || metaHits.length) fail('the ZMap tool paper is in the corpus after all; the page says it is not');
}
 
h('1. POPULATIONS');
{
  const activeInScan = ACTIVE.filter((p) => SCAN_KEYS.has(key(p)));
  const activeOutside = ACTIVE.filter((p) => !SCAN_KEYS.has(key(p)));
  // What the tools[].category restriction alone would give.
  const nsOnly = rows.filter((p) =>
    p.tools.some((tool) => {
      if (!DROVE.has(tool.usedOrMentioned) || tool.category !== 'network-scanner') return false;
      const v = kindOfName(tool.name, tool.category);
      return v && MEMBERSHIP_KINDS.has(v.kind);
    })
  );
  const namesNS = SCAN.filter((p) =>
    p.tools.some((tool) => DROVE.has(tool.usedOrMentioned) && tool.category === 'network-scanner')
  );
  t(['Population', 'Papers', 'Share'], [
    ['corpus (7 venues, 2010–2026)', rows.length, '100%'],
    ["SCAN — studyTypes 'network-scan-or-probe'", SCAN.length, pct(SCAN.length, rows.length)],
    ['ACTIVE — names an active-scan instrument (page population)', ACTIVE.length, `${pct(ACTIVE.length, rows.length)} of corpus`],
    ['  … of those, inside SCAN', activeInScan.length, pct(activeInScan.length, SCAN.length) + ' of SCAN'],
    ['  … of those, outside SCAN', activeOutside.length, pct(activeOutside.length, ACTIVE.length) + ' of ACTIVE'],
    ["SCAN papers naming ANY category='network-scanner' tool", namesNS.length, pct(namesNS.length, SCAN.length)],
    ["SCAN papers naming NO category='network-scanner' tool", SCAN.length - namesNS.length, pct(SCAN.length - namesNS.length, SCAN.length)],
  ]);
  say(`PUBLISHED_CORPUS ${rows.length}`);
  say(`PUBLISHED_SCAN ${SCAN.length}`);
  say(`PUBLISHED_ACTIVE ${ACTIVE.length}`);
  say(`PUBLISHED_ACTIVE_IN_SCAN ${activeInScan.length}`);
  say(`PUBLISHED_ACTIVE_IN_SCAN_PCT ${pct(activeInScan.length, SCAN.length)}`);
  say(`PUBLISHED_ACTIVE_OUTSIDE ${activeOutside.length}`);
  say(`PUBLISHED_NAMES_NS ${namesNS.length}`);
  say(`PUBLISHED_NO_NS ${SCAN.length - namesNS.length} ${pct(SCAN.length - namesNS.length, SCAN.length)}`);
  say('');
  say(`category-restricted membership would give ${nsOnly.length}; the any-category rule adds ${ACTIVE.length - nsOnly.length}`);
  say(`PUBLISHED_NS_ONLY ${nsOnly.length}`);
  const added = ACTIVE.filter((p) => !nsOnly.some((q) => key(q) === key(p)));
  for (const p of added) {
    const fam = memberOf(p).map((v) => `${v.family}[${[...v.cats].join('/')}]`).join(', ');
    say(`  +${key(p)}  ${fam}`);
  }
}
 
h('2. KIND UNION over SCAN (a paper can name several kinds)');
{
  const kinds = new Map();
  for (const p of SCAN) for (const k of new Set(instruments(p).map((v) => v.kind))) kinds.set(k, (kinds.get(k) || 0) + 1);
  t(['kind', 'membership?', 'Papers', 'Share of 930', 'routed to'], [...kinds.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([k, v]) => [k, MEMBERSHIP_KINDS.has(k) ? 'YES' : '', v, pct(v, SCAN.length), ROUTED_TO[k] ?? '']));
  const none = SCAN.filter((p) => instruments(p).length === 0);
  say('');
  say(`SCAN papers whose tools[] name no instrument this fold recognises: ${none.length} (${pct(none.length, SCAN.length)})`);
  say(`PUBLISHED_SCAN_NO_INSTRUMENT ${none.length} ${pct(none.length, SCAN.length)}`);
}
 
h('3. THE INSTRUMENTS — membership families over ACTIVE');
{
  const fam = new Map();
  const spell = new Map();
  for (const p of ACTIVE) {
    for (const v of memberOf(p)) {
      if (!fam.has(v.family)) { fam.set(v.family, new Set()); spell.set(v.family, new Set()); }
      fam.get(v.family).add(key(p));
      for (const n of v.names) spell.get(v.family).add(n);
    }
  }
  const ordered = [...fam.entries()].sort((a, b) => b[1].size - a[1].size || a[0].localeCompare(b[0]));
  t(['Family', 'kind', 'Papers', 'Share of ACTIVE', 'distinct spellings'], ordered
    .filter(([, s]) => s.size >= 2)
    .map(([f, s]) => [f, KIND_OF.get(f) ?? HAND[f], s.size, pct(s.size, ACTIVE.length), spell.get(f).size]));
  const ones = ordered.filter(([, s]) => s.size === 1);
  say('');
  say(`families named by exactly one paper: ${ones.length} — ${ones.map(([f]) => f).join(', ')}`);
  say(`PUBLISHED_ONEOFF_FAMILIES ${ones.length}`);
  for (const [f, s] of ordered.slice(0, 8)) say(`PUBLISHED_FAM_${f.replace(/\W+/g, '_').toUpperCase()} ${s.size}`);
  say('');
  // The roadmap queued this page on "143 name ZMap/ZGrab/masscan/nmap". Re-derive it.
  {
    const core = new Set(['ZMap', 'ZGrab', 'Masscan', 'nmap']);
    const hasCore = (p) => instruments(p).some((v) => core.has(v.family));
    const all = rows.filter(hasCore);
    const inScan = all.filter((p) => SCAN_KEYS.has(key(p)));
    say('');
    say(`ZMap|ZGrab|Masscan|nmap, used or produced: ${all.length} corpus-wide, ${inScan.length} inside SCAN (the roadmap row said 143)`);
    say(`PUBLISHED_CORE_FOUR_ALL ${all.length}`);
    say(`PUBLISHED_CORE_FOUR_SCAN ${inScan.length}`);
  }
  say('ZMap spellings: ' + [...spell.get('ZMap')].sort().join(' | '));
  say(`PUBLISHED_ZMAP_SPELLINGS ${spell.get('ZMap').size}`);
  // Home-grown: a membership instrument whose family is 'custom scanner' or a
  // one-paper coined name found only in this paper.
  const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))));
  const customOnly = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner'));
  const oneoffProduced = ACTIVE.filter((p) => memberOf(p).some((v) => v.via === 'hand' && fam.get(v.family).size === 1 && p.tools.some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family)));
  say(`ACTIVE papers whose scanner is home-grown: ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`);
  say(`  = generic 'custom scanner' string:                    ${customOnly.length}`);
  say(`  + a one-paper name the paper itself PRODUCED:         ${oneoffProduced.length}`);
  say(`  - overlap:                                            ${customOnly.length + oneoffProduced.length - homegrown.length}`);
  say(`  (the other ${ones.length - oneoffProduced.length} one-paper families are third-party tools only this corpus's one paper used)`);
  say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`);
  say(`PUBLISHED_HOMEGROWN_CUSTOM ${customOnly.length}`);
  say(`PUBLISHED_HOMEGROWN_PRODUCED ${oneoffProduced.length}`);
  say(`PUBLISHED_ONEOFF_THIRDPARTY ${ones.length - oneoffProduced.length}`);
  if (LIST === 'homegrown') for (const p of homegrown) say(`  ${key(p)}`);
}
 
h('4. CURRENCY — membership families by year bucket, share of that bucket\'s ACTIVE papers');
{
  const FAMS = ['ZMap', 'ZGrab', 'nmap', 'XMap', 'Masscan', 'custom scanner', 'ping'];
  const hdr = ['Window', 'ACTIVE papers', ...FAMS];
  const body = YEAR_BUCKETS.map(([label, inb]) => {
    const b = ACTIVE.filter((p) => inb(p.year));
    return [label, b.length, ...FAMS.map((f) => {
      const n = b.filter((p) => memberOf(p).some((v) => v.family === f)).length;
      return `${n} (${pct(n, b.length)})`;
    })];
  });
  t(hdr, body);
  say('');
  say('Per year, ZMap and the whole ACTIVE population (2026 is provisional):');
  const years = [...new Set(rows.map((p) => p.year))].sort();
  t(['Year', 'corpus', 'SCAN', 'ACTIVE', 'ZMap'], years.map((y) => [
    y,
    rows.filter((p) => p.year === y).length,
    SCAN.filter((p) => p.year === y).length,
    ACTIVE.filter((p) => p.year === y).length,
    ACTIVE.filter((p) => p.year === y && memberOf(p).some((v) => v.family === 'ZMap')).length,
  ]));
  const zFirst = Math.min(...ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'ZMap')).map((p) => p.year));
  say(`first ZMap year in ACTIVE: ${zFirst}`);
  say(`PUBLISHED_ZMAP_FIRST_YEAR ${zFirst}`);
}
 
h('5. WHERE SCANNING PUBLISHES — venue');
{
  const venues = [...new Set(rows.map((p) => p.venue))].sort();
  t(['Venue', 'Papers', 'SCAN', 'SCAN share of venue', 'ACTIVE', 'ACTIVE share of venue'], venues.map((v) => {
    const all = rows.filter((p) => p.venue === v);
    const s = all.filter((p) => SCAN_KEYS.has(key(p)));
    const a = all.filter((p) => ACTIVE_KEYS.has(key(p)));
    return [v, all.length, s.length, pct(s.length, all.length), a.length, pct(a.length, all.length)];
  }));
}
 
// ---------------------------------------------------- full-text machinery ---
const ftCache = new Map();
function fulltextFlat(p) {
  const k = key(p);
  if (!ftCache.has(k)) {
    const f = path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt');
    ftCache.set(k, fs.existsSync(f) ? fs.readFileSync(f, 'latin1').replace(/\s+/g, ' ') : null);
  }
  return ftCache.get(k);
}
function probe(pop, re) {
  const hits = [];
  let missing = 0;
  for (const p of pop) {
    const txt = fulltextFlat(p);
    if (txt === null) { missing += 1; continue; }
    if (re.test(txt)) hits.push(p);
  }
  return { hits, missing };
}
function ctx(p, re, width = 300) {
  const txt = fulltextFlat(p);
  if (txt === null) return [];
  const g = new RegExp(re.source, re.flags.includes('g') ? re.flags : re.flags + 'g');
  const o = [];
  let m;
  while ((m = g.exec(txt)) !== null) {
    o.push(txt.slice(Math.max(0, m.index - width / 2), m.index + width / 2));
    if (o.length >= 6) break;
  }
  return o;
}
 
h('6. WHAT A SCAN PAPER REPORTS — schema fields, denominator ACTIVE and SCAN');
{
  const stated = {
    'vantage location': (p) => p.vantage.some((v) => (v.locations ?? []).some((l) => !isSentinel(l))),
    'vantage infrastructure': (p) => p.vantage.some((v) => !isSentinel(v.infrastructure)),
    'any used/produced tool version': (p) => p.tools.some((tool) => DROVE.has(tool.usedOrMentioned) && !isSentinel(tool.version)),
    'version of the SCANNER itself': (p) => memberOf(p).some((v) => v.versions.size > 0),
    // `artifacts` and `ethics` are objects that can legitimately be absent
    // (320 and 894 papers corpus-wide). An absent record is NOT a stated value
    // and NOT a sentinel: it is its own row, printed below the table.
    'own artifact URL': (p) => p.artifacts !== null && !isSentinel(p.artifacts.availability) && (p.artifacts.codeUrl !== null || p.artifacts.dataUrl !== null || p.artifacts.links.length > 0),
    'ethics review outcome': (p) => p.ethics !== null && !isSentinel(p.ethics.reviewOutcome),
    'notified affected parties': (p) => p.ethics !== null && !isSentinel(p.ethics.notifiedAffectedParties),
    'harm mitigation described': (p) => p.ethics !== null && !isSentinel(p.ethics.harmMitigation),
    "robots.txt": (p) => p.ethics !== null && !isSentinel(p.ethics.robotsTxt),
    'temporal span start': (p) => p.temporal.some((x) => !isSentinel(x.spanStart)),
  };
  t(['Field', 'ACTIVE (245)', 'share', 'SCAN (930)', 'share'], Object.entries(stated).map(([label, f]) => {
    const a = ACTIVE.filter(f).length;
    const s = SCAN.filter(f).length;
    return [label, a, pct(a, ACTIVE.length), s, pct(s, SCAN.length)];
  }));
  const noEthics = ACTIVE.filter((p) => p.ethics === null).length;
  const noArt = ACTIVE.filter((p) => p.artifacts === null).length;
  say('');
  say(`ACTIVE papers with NO ethics record at all: ${noEthics} (${pct(noEthics, ACTIVE.length)}) — not counted as stated, not counted as a sentinel`);
  say(`ACTIVE papers with NO artifacts record at all: ${noArt} (${pct(noArt, ACTIVE.length)})`);
  say(`PUBLISHED_NO_ETHICS_RECORD ${noEthics} ${pct(noEthics, ACTIVE.length)}`);
  say(`PUBLISHED_NO_ARTIFACT_RECORD ${noArt} ${pct(noArt, ACTIVE.length)}`);
  for (const [label, f] of Object.entries(stated)) {
    say(`PUBLISHED_REPORT_${label.replace(/\W+/g, '_').toUpperCase()} ${ACTIVE.filter(f).length} ${pct(ACTIVE.filter(f).length, ACTIVE.length)} | SCAN ${SCAN.filter(f).length} ${pct(SCAN.filter(f).length, SCAN.length)}`);
  }
}
 
h('7. FULL-TEXT PROBES over ACTIVE — a loose upper bound and a tight estimate');
{
  const PAIRS = [
    ['states its own probe rate or rate limit', PROBE.rateLoose, PROBE.rateTight],
    ['honours a blocklist / exclusion list', PROBE.blockLoose, PROBE.blockTight],
    ['offers an opt-out', PROBE.optLoose, PROBE.optTight],
    ['source-address hygiene (dedicated IPs, rDNS/PTR, WHOIS, web page)', PROBE.srcLoose, PROBE.srcTight],
    ['abuse complaints received or handled', PROBE.abuseLoose, PROBE.abuseTight],
  ];
  const body = [];
  for (const [label, loose, tight] of PAIRS) {
    const l = probe(ACTIVE, loose);
    const tt = probe(ACTIVE, tight);
    body.push([label, `${l.hits.length} (${pct(l.hits.length, ACTIVE.length)})`, `${tt.hits.length} (${pct(tt.hits.length, ACTIVE.length)})`]);
    say(`PUBLISHED_PROBE_${label.replace(/\W+/g, '_').toUpperCase()}_LOOSE ${l.hits.length} ${pct(l.hits.length, ACTIVE.length)}`);
    say(`PUBLISHED_PROBE_${label.replace(/\W+/g, '_').toUpperCase()}_TIGHT ${tt.hits.length} ${pct(tt.hits.length, ACTIVE.length)}`);
  }
  t(['Practice', 'loose probe (upper bound)', 'first-person probe'], body);
  say('');
  say('Neither column is a population. The loose regex fires on any sentence in');
  say('the paper, including one about somebody else\'s rate limit or somebody');
  say('else\'s blocklist; the first-person regex requires "we"/"our" near the term');
  say('and still admits sentences about the authors\' own *subject*. Section 8');
  say('hand-reads a deterministic sample of each first-person hit list and gives');
  say('the measured precision.');
  const also = probe(ACTIVE, PROBE.irb).hits;
  const cite = probe(ACTIVE, PROBE.goodpractice).hits;
  say('');
  say(`ACTIVE papers naming an IRB / ethics board: ${also.length} (${pct(also.length, ACTIVE.length)}) [loose, no precision measured]`);
  say(`ACTIVE papers citing Durumeric / Partridge-Allman / Menlo / "good Internet citizen": ${cite.length} (${pct(cite.length, ACTIVE.length)}) [loose]`);
  say(`PUBLISHED_IRB ${also.length} ${pct(also.length, ACTIVE.length)}`);
  say(`PUBLISHED_GOODPRACTICE ${cite.length} ${pct(cite.length, ACTIVE.length)}`);
}
 
h('8. HAND-READ PRECISION — deterministic every-k-th sample of each first-person hit list');
{
  const sample = (re, n) => {
    const hits = probe(ACTIVE, re).hits.slice().sort((a, b) => key(a).localeCompare(key(b)));
    const step = Math.max(1, Math.floor(hits.length / n));
    return { hits, sample: hits.filter((_, i) => i % step === 0).slice(0, n) };
  };
  const CHECKS = [
    ['own probe rate', PROBE.rateTight, RATE_VERDICT, 12],
    ['own blocklist / exclusion', PROBE.blockTight, BLOCK_VERDICT, 12],
    ['own opt-out offer', PROBE.optTight, OPT_VERDICT, 10],
    ['own source-address hygiene', PROBE.srcTight, SRC_VERDICT, 10],
  ];
  const summary = [];
  for (const [label, re, verdicts, n] of CHECKS) {
    const { hits, sample: sm } = sample(re, n);
    say('');
    say(`--- ${label}: ${hits.length} first-person hits of ${ACTIVE.length} ACTIVE; sample of ${sm.length}`);
    // Both directions: every sampled paper must have a verdict, and every
    // verdict must belong to a paper the sample still returns.
    const sampleKeys = new Set(sm.map(key));
    for (const p of sm) {
      const v = verdicts[key(p)];
      say(`    ${(v ?? 'UNREAD').padEnd(6)} ${key(p)}`);
      if (!v) fail(`${label}: sampled paper has no hand verdict: ${key(p)}`);
    }
    for (const k of Object.keys(verdicts)) {
      if (!sampleKeys.has(k)) fail(`${label}: hand verdict for a paper the sample no longer returns: ${k} — the sample moved, re-read it`);
    }
    const own = Object.values(verdicts).filter((v) => v === 'own').length;
    const tot = Object.keys(verdicts).length;
    const est = Math.round((hits.length * own) / tot);
    say(`    precision ${own}/${tot}; ${hits.length} x ${own}/${tot} = about ${est} papers (${pct(est, ACTIVE.length)} of ACTIVE)`);
    summary.push([label, `${hits.length} (${pct(hits.length, ACTIVE.length)})`, `${own}/${tot}`, `~${est} (${pct(est, ACTIVE.length)})`]);
    say(`PUBLISHED_PRECISION_${label.replace(/\W+/g, '_').toUpperCase()} ${own}/${tot} -> ~${est} ${pct(est, ACTIVE.length)}`);
  }
  say('');
  t(['Practice', 'first-person probe', 'hand-read precision', 'point estimate'], summary);
  say('');
  say('The point estimate is probe count x sample precision. It is an estimate on a');
  say('sample of 10-12, so read it as "about a fifth" rather than as a percentage.');
}
 
h('9. IPv6 — the address space is not enumerable');
{
  const v6 = probe(ACTIVE, /IPv6/).hits;
  const v6scan = probe(SCAN, /IPv6/).hits;
  const hitlistLoose = probe(SCAN, /hit ?list/i).hits;
  const hitlistV6 = probe(SCAN, /(IPv6[^.]{0,80}hit ?list|hit ?list[^.]{0,80}IPv6|IPv6 Hitlist)/i).hits;
  // A /Gasser/ probe returned 93 SCAN papers (10.0%) and was REJECTED: the
  // surname is not unique and the hits are reference-list entries, so it
  // counts citations, not use. Kept here as a rejected probe, not published.
  const gasser = probe(SCAN, /Gasser/).hits;
  const targetGen = probe(SCAN, /(target generation|address generation algorithm|\b6Gen\b|Entropy\/IP|\b6Graph\b|\b6Sense\b|\b6Tree\b|\b6Hit\b|\b6Forest\b)/i).hits;
  const aliased = probe(SCAN, /(aliased (prefix|region)|alias(ed)? detection|dealias)/i).hits;
  const v6instr = SCAN.filter((p) => instruments(p).some((v) => v.kind === 'ipv6-targeting'));
  const v6instrAll = rows.filter((p) => instruments(p).some((v) => v.kind === 'ipv6-targeting'));
  t(['Query', 'Population', 'Papers', 'Share'], [
    ['mentions IPv6 anywhere', 'SCAN 930', v6scan.length, pct(v6scan.length, SCAN.length)],
    ['mentions IPv6 anywhere', 'ACTIVE 245', v6.length, pct(v6.length, ACTIVE.length)],
    ["names an IPv6-targeting instrument in tools[]", 'SCAN 930', v6instr.length, pct(v6instr.length, SCAN.length)],
    ["names an IPv6-targeting instrument in tools[]", 'corpus 5,859', v6instrAll.length, pct(v6instrAll.length, rows.length)],
    ['full text says "hitlist"/"hit list" (loose)', 'SCAN 930', hitlistLoose.length, pct(hitlistLoose.length, SCAN.length)],
    ['"hitlist" within 80 chars of "IPv6"', 'SCAN 930', hitlistV6.length, pct(hitlistV6.length, SCAN.length)],
    ['names a target-generation algorithm', 'SCAN 930', targetGen.length, pct(targetGen.length, SCAN.length)],
    ['discusses aliased prefixes / dealiasing', 'SCAN 930', aliased.length, pct(aliased.length, SCAN.length)],
  ]);
  say(`PUBLISHED_V6_SCAN ${v6scan.length} ${pct(v6scan.length, SCAN.length)}`);
  say(`PUBLISHED_V6_ACTIVE ${v6.length} ${pct(v6.length, ACTIVE.length)}`);
  say(`PUBLISHED_V6_INSTR_SCAN ${v6instr.length}`);
  say(`PUBLISHED_V6_INSTR_ALL ${v6instrAll.length}`);
  say(`PUBLISHED_HITLIST_LOOSE ${hitlistLoose.length} ${pct(hitlistLoose.length, SCAN.length)}`);
  say(`PUBLISHED_HITLIST_V6 ${hitlistV6.length} ${pct(hitlistV6.length, SCAN.length)}`);
  say(`REJECTED_PROBE /Gasser/ over SCAN: ${gasser.length} (${pct(gasser.length, SCAN.length)}) — surname is not unique, hits are reference-list entries; not published`);
  say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`);
  say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`);
  const v6subset = new Set([...hitlistV6, ...v6instr].map(key));
  const aliasIn = aliased.filter((p) => v6subset.has(key(p)));
  say('');
  say(`IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): ${v6subset.size} papers`);
  say(`  of those, discuss aliased prefixes / dealiasing: ${aliasIn.length} (${pct(aliasIn.length, v6subset.size)})`);
  say(`  the same 10 papers against all of SCAN would be ${pct(aliased.length, SCAN.length)} — the wrong denominator, most of SCAN never touches IPv6`);
  say(`PUBLISHED_V6_SUBSET ${v6subset.size}`);
  say(`PUBLISHED_ALIAS_IN_SUBSET ${aliasIn.length} ${pct(aliasIn.length, v6subset.size)}`);
  say('');
  say('IPv6-targeting instruments named, corpus-wide:');
  const m = new Map();
  for (const p of v6instrAll) for (const v of instruments(p).filter((x) => x.kind === 'ipv6-targeting')) {
    if (!m.has(v.family)) m.set(v.family, new Set());
    m.get(v.family).add(key(p));
  }
  for (const [f, s] of [...m].sort((a, b) => b[1].size - a[1].size)) say(`  ${String(s.size).padStart(3)} ${f}  (${[...s].join(', ')})`);
  say('');
  say('Per-year IPv6 mention rate inside SCAN (2026 provisional):');
  const years = [...new Set(SCAN.map((p) => p.year))].sort();
  t(['Year', 'SCAN', 'mentions IPv6', 'share'], years.map((y) => {
    const b = SCAN.filter((p) => p.year === y);
    const n = probe(b, /IPv6/).hits.length;
    return [y, b.length, n, pct(n, b.length)];
  }));
}
 
h('10. THE JOIN TO design:ip_classification');
{
  const ipCls = (p) => p.classification.some((c) => c.target === 'ip-address');
  const a = ACTIVE.filter(ipCls);
  const s = SCAN.filter(ipCls);
  const c = rows.filter(ipCls);
  t(['Population', 'N', "classification[].target == 'ip-address'", 'share'], [
    ['ACTIVE', ACTIVE.length, a.length, pct(a.length, ACTIVE.length)],
    ['SCAN', SCAN.length, s.length, pct(s.length, SCAN.length)],
    ['corpus', rows.length, c.length, pct(c.length, rows.length)],
  ]);
  say(`PUBLISHED_IPCLS_ACTIVE ${a.length} ${pct(a.length, ACTIVE.length)}`);
  say(`PUBLISHED_IPCLS_SCAN ${s.length} ${pct(s.length, SCAN.length)}`);
  say(`PUBLISHED_IPCLS_CORPUS ${c.length} ${pct(c.length, rows.length)}`);
  // Which resources those papers reach for.
  const res = new Map();
  for (const p of a) for (const cl of p.classification) {
    if (cl.target !== 'ip-address') continue;
    for (const rn of cl.resourceName ? [cl.resourceName] : (cl.resources ?? [])) {
      if (isSentinel(rn)) continue;
      const kk = String(rn).trim();
      if (!res.has(kk)) res.set(kk, new Set());
      res.get(kk).add(key(p));
    }
  }
  say('');
  say(`distinct raw resourceName strings on ACTIVE ip-address classifications: ${res.size} (free text; folded on design:ip_classification, not here)`);
  for (const [k, v] of [...res].sort((x, y) => y[1].size - x[1].size).slice(0, 15)) say(`  ${String(v.size).padStart(3)} ${k}`);
}
 
h('11. POSTERS AND SHORT RECORDS');
{
  const poster = (p) => /^poster-/.test(p.slug) || /^Poster:/.test(p.title);
  const ap = ACTIVE.filter(poster);
  const sp = SCAN.filter(poster);
  const short = ACTIVE.filter((p) => p.pages !== null && p.pages <= 4);
  say(`ACTIVE posters: ${ap.length} (${pct(ap.length, ACTIVE.length)}); SCAN posters: ${sp.length} (${pct(sp.length, SCAN.length)})`);
  say(`ACTIVE records of <=4 pages: ${short.length} (${pct(short.length, ACTIVE.length)})`);
  const full = ACTIVE.filter((p) => !(p.pages !== null && p.pages <= 4));
  const vantageStated = (p) => p.vantage.some((v) => (v.locations ?? []).some((l) => !isSentinel(l)));
  say(`vantage location stated: all ACTIVE ${pct(ACTIVE.filter(vantageStated).length, ACTIVE.length)}, excluding <=4-page records ${pct(full.filter(vantageStated).length, full.length)}`);
  say(`PUBLISHED_ACTIVE_POSTERS ${ap.length}`);
  say(`PUBLISHED_VANTAGE_FULLONLY ${pct(full.filter(vantageStated).length, full.length)}`);
}
 
h('12. MEASURED RESULTS available for the page (detection[].prevalence on ACTIVE)');
{
  let n = 0;
  const withPrev = ACTIVE.filter((p) => p.detection.some((d) => !isSentinel(d.prevalence)));
  say(`ACTIVE papers with at least one detection[].prevalence: ${withPrev.length} (${pct(withPrev.length, ACTIVE.length)})`);
  say(`PUBLISHED_ACTIVE_PREVALENCE ${withPrev.length} ${pct(withPrev.length, ACTIVE.length)}`);
  if (QUOTES) {
    const re = new RegExp(QUOTES, 'i');
    for (const p of ACTIVE) for (const d of p.detection) {
      if (isSentinel(d.prevalence)) continue;
      const blob = `${d.phenomenon} ${d.technique} ${d.metric} ${d.prevalence}`;
      if (!re.test(blob)) continue;
      say(`--- ${key(p)}`);
      say(`    phenomenon: ${d.phenomenon}`);
      say(`    technique : ${d.technique}`);
      say(`    metric    : ${d.metric}`);
      say(`    prevalence: ${d.prevalence}`);
      say(`    quote     : ${(d.evidence?.quote ?? '').replace(/\s+/g, ' ')}`);
      if (++n > 40) break;
    }
  }
}
 
if (HITS) {
  h(`SENTENCES matching ${HITS} in ACTIVE`);
  const re = new RegExp(HITS, 'i');
  let n = 0;
  for (const p of ACTIVE) {
    const cs = ctx(p, re);
    if (!cs.length) continue;
    say(`--- ${key(p)}`);
    for (const c of cs.slice(0, 2)) say(`    …${c}…`);
    if (++n > 40) break;
  }
}
 
if (LIST === 'active') { h('ACTIVE paper list'); for (const p of ACTIVE) say(`${key(p)}  [${memberOf(p).map((v) => v.family).join(', ')}]`); }
if (LIST === 'scan') { h('SCAN paper list'); for (const p of SCAN) say(key(p)); }
 
h('Z. NON-CORPUS FIGURES ON THE PAGE (checked by external_checks_internet_scanning.sh)');
for (const line of EXTERNAL) say(line);
 
h('RESULT');
say(fails.length === 0 ? 'OK — all contracts hold' : `${fails.length} FAILURE(S)`);
console.log(out.join('\n'));
if (fails.length) process.exit(1);