Differences

This shows you the differences between two versions of the page.

--- provenance:programming:internet_scanning [2026/09/10 21:12] – Anchor two namespace links that resolved relative to provenance:programming. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:internet_scanning [2026/09/10 21:35] (current) – Point the review log at the corrected follow-up item. Authored by Claude. karel.kubicek.claude
@@ Line 25: / Line 25: @@
 | **id = ''programming:internet_scanning''**, not ''design:scanning'' | Already decided on [[:roadmap]] on 2026-09-07 (see ''provenance:roadmap'' section 3) and already a red link that ''scripts/sitemap.mjs'' gates on. The decision holds on its merits too: this page is about the invocation, the flags, the rate and the output format — the same register as [[programming:crawler]]. The //whether to scan// decision stays on [[design:automated_measurements]]. No change to the roadmap row was needed beyond retiring it. |
 | **Creating a page rather than broadening [[design:automated_measurements]]** | That page is a three-branch //router// with a 20 KB budget; it already carries the crawl branch's routing without absorbing [[programming:crawler]]. Absorbing the scan instrument would have made it the only branch page that is also a tool page. |
-| **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. Where the two pages disagree on a count, the content page says so. |
+| **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. The two folds therefore give different counts for the same family — ZMap 95 here against 93 there, nmap 46 against 40, ZGrab 32 against 31, RIPE Atlas 25 there and excluded here. **The content page does not currently enumerate those differences**; the difference is the category restriction plus the looser ''nmap'' anchor, and it is set out in this row rather than there. |
-| **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. Every //share of 930// on the page is therefore labelled as ranking-grade. |
+| **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. The page's //Methodology// section says to treat every //share of 930// as ranking-grade; the individual shares (10.4%, 10.2%, 57.4%, 30.2% …) are **not** individually labelled where they appear, which a review pass fairly called out. |
 | **ACTIVE membership is decided by the instrument, not by ''tools[].category''** | ''category'' is a denominator: restricting to ''network-scanner'' gives 231 and loses 14 papers whose ZMap/OONI/Geneva/Censored Planet/hitlist entry is filed under ''other'', ''infrastructure'', ''http-client'' or ''program-analysis''. All 14 are listed in section 1 of the report. |
 | **…but generic strings and hand verdicts stay inside ''category == "network-scanner"''** | Without that restriction ''%%/^custom\b/i%%'' alone put **464** papers in the population, almost all custom //crawlers// and custom //scripts//, and the hand entries ''Python script'', ''Unicorn'', ''nc'' and ''whois'' fired on unrelated papers. Families whose regex is a descriptor rather than a coined name are flagged ''generic'' in the fold and only trusted inside that category. |
@@ Line 83: / Line 83: @@
 | source-address hygiene | 170 (69.4%) | 114 (46.5%) | **6/10** | ~68, "a little over a quarter" |
 | abuse complaints received or handled | 58 (23.7%) | 46 (18.8%) | not sampled | published as an upper bound only |
-| names an IRB or ethics board | — | 56 (22.9%) | not sampled | upper bound |
+| names an IRB or ethics board | 56 (22.9%) //(loose; there is no first-person version)// | — | not sampled | upper bound, and **not** to be read against ''ethics.reviewOutcome''  (93, 38.0%) |
-| cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | — | 174 (71.0%) | not sampled | upper bound |
+| cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | 174 (71.0%) //(loose)// | — | not sampled | upper bound |
 Samples are **every //k//-th paper of the sorted hit list**, so they are reproducible and the report **exits 1** if a sampled paper has no verdict //or// if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read.
@@ Line 149: / Line 149: @@
 ===== Quotes and per-paper figures =====
-''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **26 needles, 0 not located, 0 weak needles under 20 characters.** All 26 resolved in ''.cols''; the PDF route did not have to fire.
+''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **30 needles, 0 not located, 0 weak needles under 20 characters.** **26** are in-corpus and all 26 resolved in ''.cols'', so the PDF fallback never had to fire for those. The other **4** are from {[durumeric2013_zmap]}, which is **not** in the extraction: they are checked against ''out/scan/zmap2013_paper.txt'' (72,192 bytes), a ''pypdf'' extraction of the authors' own copy at ''zmap.io/paper.pdf''. That check needs its own **de-hyphenation** step — ''pypdf'' keeps the printed hyphen at a line break, so //accu- racy// and //theoreti- cal// are two words in the extraction and one in the PDF, and two of the four quotes failed until it was added.
 Two things that went wrong and were fixed:
@@ Line 161: / Line 161: @@
   * The embedded block is **byte-identical** to ''pages/scan_budget.py''.
-  * Downloading it back off the rendered page (''/_export/code/programming:internet_scanning?codeblock=3'') and diffing gives **one** difference, a stripped trailing newline. The downloaded copy runs its own ''--self-test'': **8 checks, OK**.
+  * Downloading it back off the rendered page (''/_export/code/programming:internet_scanning?codeblock=3'') and diffing gives **one** difference, a stripped trailing newline. The downloaded copy runs its own ''--self-test'': **9 checks, OK**.
   * The ''%%<code>%%'' block beside it is the **whole** output of a real run, compared line-by-line for exact equality — not a containment check, which passes on a truncated block.
   * The self-test has a **mutation guard**: replacing ''collapse_addresses()'' with a bare ''sum'' makes the overlapping-CIDR case return 33,554,432 instead of 16,777,216 and check 3 fails. Check 4 exists so that "collapse does nothing at all" cannot also pass — a disjoint pair must still add up.
@@ Line 205: / Line 205: @@
 | ''node scripts/check_tables.mjs'' on the same | "every table has one width" |
 | ''python3 scripts/bib_dedup_scan.py --bib <fresh export + 8 new>'' | 0 definite duplicate pairs; the one candidate touching a new key is ''durumeric2013_zmap''/''williams2024_6sense'', different papers |
-| rendered DOM after save | 35 source ''%%{[key]}%%'' markers → 70 ''bibtex_citekey'' spans → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings |
+| rendered DOM after save | **37** source ''%%{[key]}%%'' markers → **74** ''bibtex_citekey'' spans (the plugin emits two per marker) → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings; **0** red links |
 | red links in the rendered DOM | 3, all of them to this page, before it existed |
 | published ''%%<file>%%'' round-trip | byte-identical apart from a stripped trailing newline; runs its self-test |
@@ Line 211: / Line 211: @@
 ===== Review log =====
-**Review in flight.** The three focused passes and the generic pass were spawned
+Four passes, all told explicitly that the author's context might not be
-after this page was first published; this section is replaced when they return.
+exhaustive, and all handed the page text, the report script and its unedited
+output, the fold, the published code and the provenance notes. The three focused
+passes were spawned in parallel against the **first published revision**
+(''programming:internet_scanning'' rev 1789074265); the generic pass was spawned
+afterwards, against the corrected page, so it saw a different snapshot. That
+matters for reading their findings and is recorded here rather than left to be
+inferred.
+Two of the three focused passes had to be prompted to return partial findings —
+they were still fetching after a long wall-clock stretch. All three returned in
+full in the end.
+==== Applied before the passes returned (author-found) ====
+^ # ^ Finding ^ Disposition ^
+| A1 | The Masscan comparison was quoted as //"Masscan finds notably fewer hosts than ZMap…"//; the paper reads //"Masscan [48] finds…"//, and the result is **Adrian et al.'s**, relayed by {[durumeric2024_years]}. | **Fixed.** Ellipsis added and Adrian et al. named. The citations pass independently reached the same place and called the original phrasing "borderline but not dishonest". |
+| A2 | Family table wrote "Entropy-IP"; the tool and the report say ''Entropy/IP''. | **Fixed.** |
+| A3 | //"orders of magnitude slower"// for nmap was an unsourced assertion. | **Fixed** with the ZMap paper's own measurement — //"over 1300 times faster than the most aggressive Nmap default settings, with equivalent accuracy"// — and the framing changed to "a different instrument", not an obsolete one. That paper is not in the extraction; the authors' copy at ''zmap.io/paper.pdf'' was fetched, extracted with ''pypdf'' and added to the verifier as four off-corpus needles. |
+| A4 | The rate table had no independent check. | **Added**: the ZMap paper measured //"under 45 minutes"// at //"over 97% of the theoretical maximum speed of gigabit Ethernet"//; the script's theoretical 41.5 min ÷ 0.97 = 42.8. Two independent routes to the same number is worth more than either alone. |
+==== Pass 1 — figures vs script (''sonnet'') ====
+Re-ran all four scripts, confirmed they reproduce the committed outputs
+byte-for-byte and that the provenance page's embedded copies are byte-identical
+to fresh runs. **Mutation-tested** ''excluded_addresses'', ''line_rate_pps'',
+''scan_plan'', both directions of the section-8 sample check and the section-0
+fold-coverage check, and confirmed each fires. Found no figure that mixes the
+SCAN/ACTIVE denominators. Four findings:
+^ # ^ Finding ^ Disposition ^
+| P1.1 | //"''--ports 2 --probes 2'' is four times every number in that table"// is **false**: ''scan_plan'' computes ''wire_bps'' from ''rate_pps'' alone, so the wire-rate column does not move. Verified by calling the function both ways — seconds ratio 4.0, ''wire_mbps'' ratio 1.0. | **Accepted and fixed.** The sentence now says the wall clock scales and the wire rate does not, and gives the unchanged 7 Mb/s. This is exactly the class of error a figures-vs-script check normally cannot see: the number came from prose, not from the script. |
+| P1.2 | //"citing the norm is four times more common than being reviewed against it"// — 174 / 56 = **3.1**. | **Accepted and fixed** to "about three times", with the division shown. |
+| P1.3 | The opening box said "roughly a fifth to a quarter"; the four estimates are 17.1 / 18.4 / 25.3 / 27.8%, so the band is wrong at both ends. | **Accepted and fixed** to "between a sixth and a little over a quarter (17%, 18%, 25%, 28%)". This is the probe-scoped-summary-in-the-prominent-place failure, caught in its usual place. |
+| P1.4 | ''pages/scan_budget.py --self-test'' **never called ''human_time()''**: mutating its bucket boundaries from 120/7200/172800 to 60/3600/86400 still printed "OK (8 checks)". The pass then swept the boundaries by hand and found them correct, so no published number was wrong. | **Accepted and fixed.** Check 9 added: six ±0.01 s boundary cases plus three exact conversions. The published block, the quoted output and the "8 checks" figure everywhere else were regenerated. A self-test that claims coverage it does not have is worse than none. |
+==== Pass 2 — citations and quotes (''sonnet'') ====
+All 15 citekeys resolve exactly once in a fresh export;
+''bib_dedup_scan.py'' finds 0 definite duplicates; ''check_attributions.mjs''
+passes 10/10; every quoted sentence checked matched verbatim; both hand-fetched
+USENIX author lists match the PDF header in order, including the last author and
+the umlauts; the published code contains no names at all. One finding:
+^ # ^ Finding ^ Disposition ^
+| P2.1 | //"Beverly et al. …, on a **six-month** IPv6 topology campaign"//. The word "month" does not appear anywhere in that paper; its campaigns run 2018-02-01 to 2018-05-18. The **six months belonged to Gasser et al.**, a different paper quoted three paragraphs earlier, and had been carried across. The quote itself was verbatim, so the quote checker could not see it — the duration was an unchecked descriptive claim wrapped around a checked quote. | **Accepted and fixed** to "between February and May 2018". |
+==== Pass 3 — external currency (''sonnet'') ====
+Independently re-fetched and confirmed: both ZMap man pages (tag and ''main'',
+byte-identical, 332 lines each), every published flag and default, the absence of
+''%%--blocklist-file-extra%%'', the three helper binaries, ''conf/blocklist.conf''
+(16 entries, 2013 header, no opt-out language) with the CIDR arithmetic
+**recomputed by hand** including the ''255.255.255.255/32''-inside-''240.0.0.0/4''
+overlap, all ZGrab2 README quotes, every repository state and release date, nmap
+.991 against nmap.org's own changelog, all three IPv6 Hitlist line counts
+downloaded and counted independently, the Rapid7 Sonar redirect and the 8 / 47,457
+/ 60.7 TB figures (summed by hand from the eight per-dataset counts), and both
+USENIX author lists. Two findings:
+^ # ^ Finding ^ Disposition ^
+| P3.1 | The ''ziterate'' man page says //"psuedorandom"// — a transposition in the source. The page quoted it as "pseudorandom", silently correcting it inside quotation marks. | **Accepted and fixed** to the source's own spelling with ''[sic]''. Small, and exactly the bar the page sets for every other man-page quote. |
+| P3.2 | ''external_checks_internet_scanning.sh'' printed release **tag names** but not release **dates**, so the three dates the page publishes were not actually verified by the script — the pass had to hit ''/releases/tags/<tag>'' itself. | **Accepted and fixed**: the script now fetches ''published_at'' per release and prints it beside the tag. A date is precisely the fact that goes stale without anyone noticing. |
+| P3.3 | ''www.zoomeye.org'' answers HTTP 521 with an ''X-Via-JSL'' header (Knownsec's own bot challenge), while ''zoomeye.ai'' answers 200. | **Noted, no change.** The page names ZoomEye in a list of query targets and makes no uptime or URL claim about it. Recorded here so a later tightening of the "still operating" check does not read the 521 as a shutdown. |
+==== Pass 4 — generic (''fable'') ====
+**It returned after the "did not return" note above had already been written and
+published**, in response to a second prompt for partial findings. Fifteen
+findings on the already-corrected page, and it earned its slot: five of them are
+things the three focused passes structurally could not see, and two are wrong
+numbers the number guard passed because the digits occur elsewhere in the report.
+This is the fourth page in a row where the generic pass found the worst item.
+^ # ^ Finding ^ Disposition ^
+| P4.1 | **A contradiction at a seam.** //"56 (22.9%) name an IRB or ethics board **at all**"// — with a ratio built on it — sits two paragraphs above the extraction's own ''ethics.reviewOutcome'', **stated on 93 (38.0%)** of the same 245 papers. The 56 is a loose regex with no measured precision, and "at all" is the probe-scoped qualifier dropped in the prominent place. | **Accepted and fixed.** Both numbers now appear together, the ratio is gone, and the sentence says the 93 is the number to quote. Neither a figures pass (both numbers match the script) nor a citations pass (no citation involved) could see this. |
+| P4.2 | **The opening box republished the precision the page's own limitations forbid** — it listed "(17%, 18%, 25%, 28%)" while §//Methodology// says "read 'about a fifth', not '18.4%'". That box had been rewritten **by finding P1.3 of the figures pass** two hours earlier: fixing a wrong band introduced a false precision. | **Accepted and fixed** to "somewhere between a seventh and a quarter … read them as 'a minority, and a small one', not as percentages". A fix is a new edit and needs the same scrutiny as the thing it replaced. |
+| P4.3 | **"Forty more families are named by exactly one paper each … 45 of 245 rolled their own" does not add up**, and the reviewer worked out why from the report source: only **20** of the 40 one-paper families are a tool the paper //produced//; the rest are third-party tools (''nc'', ''ike-scan'', ''smbmap'', ''JARM'', ''WhatWeb'') that appear once. Re-derived: 20 produced + 25 "custom scanner" − 5 overlap = 45. | **Accepted and fixed**, and the report now prints the decomposition. The follow-on flourish //"the scanner is a hundred lines around a raw socket"// was cut as unbacked and, for an Internet-wide scanner, untrue. |
+| P4.4 | //"several instruments above (Yarrp, MIDAR, scamper) have their tool papers at those venues [PAM, TMA, ANRW, CoNEXT, ACSAC]"// — flagged as probably wrong. **Checked: Yarrp's tool paper is ''IMC 2016'' and is in this corpus.** | **Accepted and fixed.** The venue-gap caveat stands; the examples were removed and replaced with the Yarrp counter-example. A confident parenthetical inside a correct caveat is a good hiding place. |
+| P4.5 | This page claimed //"All 26 [needles] resolved in ''.cols''; the PDF route did not have to fire"//. By then it was 30 needles, four of them off-corpus against a ''pypdf'' extraction that **needs a de-hyphenation step** two quotes failed without. | **Accepted and fixed** above. The claim had gone stale inside the section whose purpose is auditability. |
+| P4.6 | The ''scan_budget.py'' **docstring** still said "eight checks" after check 9 was added. | **Accepted and fixed.** A byte-identity guard proves the page matches the file, not that the file is right. |
+| P4.7 | //"The convention this literature has converged on"// is contradicted eight lines later by the page's own ~28% estimate. | **Accepted and fixed** to "the convention the ZMap-era methods papers set — and about a quarter of scanning papers follow". |
+| P4.8 | **The aliasing figure had the wrong denominator.** 10 of 930 (1.1%) counts papers that never touch IPv6. Against the **42** papers that name an IPv6-targeting instrument or say "hitlist" near "IPv6" it is **8 (19.0%)**. //"Almost nobody writes it"// was a "nobody" from a probe. | **Accepted and fixed**, both denominators now published side by side, and the report computes the subset. The rewritten sentence is also a stronger finding: four in five of the papers doing IPv6 targeting do not discuss de-aliasing. |
+| P4.9 | Two overclaims **on this page**: that the content page says where the two folds disagree (it does not), and that every //share of 930// is labelled ranking-grade (only the Methodology section says it). | **Accepted and fixed**, by writing down what is actually true — including the four family counts where this fold and ''am_fold.mjs'' disagree. |
+| P4.10 | The "Review in flight" placeholder was still on the published page. | **Already fixed** before the pass returned; it saw an older snapshot. |
+| P4.11 | **OONI (7) and Censored Planet (4) contradict the ACTIVE definition**: a paper re-analysing OONI data did not send probes to addresses it chose. And an IPv6 target generator "sends nothing". | **Accepted, definition rewritten rather than the population changed.** Both edges are now stated on the content page as judgement calls: a censorship platform counts either way because the platform is the instrument and the corpus rarely says which; a target generator counts because in IPv6 it is the step that decides what gets probed. |
+| P4.12 | Four technical errors in the instrument table and one in the published docstring: ZGrab does not "complete the handshake ZMap only started" (ZMap RSTs; ZGrab opens its own connection); nmap is not "connection-oriented"; ZMap was called "IPv4/IPv6" while a later section says its design does not extend to 128 bits; masscan has had IPv6 since 1.3; and ZMap's default ''synscan'' sends **no** TCP options, so the docstring's "SYN with an MSS option" was wrong (the 84-byte constant is unchanged — the frame is padded to the 60-byte minimum either way). | **All accepted and fixed.** None of these has a number in it, which is why nothing caught them. |
+| P4.13 | Four unbacked framing sentences, including //"the fourth party most papers forget"// and a per-year IPv6 series quoted only at its two most favourable endpoints (2010 12.5% → 2025 44.3%, skipping 2013's 0% and 2024's 29.2%). | **Accepted and fixed**, the series now given at five points including the non-monotone ones. |
+| P4.14 | **A hand verdict disputed.** IEEE S&P 2023 //PCspoof// was marked ''own'' on "sending ARP requests at 100 Mbps" — an attack rate on a TTEthernet lab testbed, not a scan rate. | **Accepted and flipped**, which moved the rate precision from 6/12 to **5/12** and the estimate from ~42 to **~35 (one in seven)**. Two figures on the page were stale afterwards and ''check_page_numbers.mjs'' **passed anyway**, because 42, 6 and 17 all occur elsewhere in the report — the documented presence-not-binding weakness, hit for real. Found by grep, not by the guard. |
+| P4.15 | **A structure gap**: the invocation only ever scans the whole space on TCP/443, when most first scans are a target list. | **Accepted**, and ''%%-w/--allowlist-file%%'', ''%%-I/--list-of-ips-file%%'' and ''%%--probe-module%%'' added with the note that the blocklist still applies on top. |
+It also read the provenance probe table and found the IRB row filed under the
+"first-person" column although the report calls it loose. That is correct and
+the row is now labelled. Its own **NOT READ** list is long — the embedded fold
+source, the external-checks output, `practices:ethics` and `design_dns.txt` for
+duplication, and a possible disagreement between this page's ZMap 95 + XMap 16
+and [[design:dns]]' "ZMap and XMap (112 papers)" that it could not resolve.
+**That last one is unresolved.** A follow-up item
+(''internet_scanning: three follow-ups the review left open'') is filed for it,
+together with the two gaps below.
+==== What the passes did not cover ====
+Each pass named its own gaps and they are worth recording, because they are the
+parts of this page nothing checked:
+  * **The 112-vs-111 question above.** [[design:dns]] reports "ZMap and XMap (112 papers)" against this page's ZMap 95 + XMap 16 = 111. That may be an overlap artefact (a paper naming both) or a real disagreement between two folds. Not resolved.
+  * **Nobody hand-audited the 174 ''HAND'' verdicts** in ''scan_fold.mjs'' for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire.
+  * **Nobody re-read the 44 probe papers** to confirm the ''own''/''other'' verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates.
+  * The figures pass did not re-fetch the external facts; the currency pass did not re-derive the corpus figures. That split is deliberate, but it means **no single pass read an external figure and a corpus figure in the same sentence**.
+  * The citations pass flagged that nobody had verified {[durumeric2013_zmap]}'s claimed absence from the corpus. **Closed by the author on 2026-09-10**: 0 extraction records, 0 ''data/fulltext/2013/USENIX'' directories and 0 of the **28** ''corpus2/.meta/USENIX-2013.json'' records match ''/zmap|internet-wide-scanning/i''. Twenty-eight records is itself thin for that venue-year — a corpus-level observation, recorded on [[literature:corpus]]' territory rather than here.
 ===== The report script and its unedited output =====
@@ Line 224: / Line 333: @@
 unmapped strings: 0
 HAND entries: 174, matched: 174, stale: 0
+==============================================================================
+b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)
+==============================================================================
+extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: 0
+data/fulltext/2013/USENIX directories matching /zmap|scann/i:            0
+corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 0 of 28
+PUBLISHED_ZMAP2013_ABSENT yes
+PUBLISHED_USENIX2013_INDEX 28
 ==============================================================================
@@ Line 341: / Line 459: @@
 ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6
 PUBLISHED_ZMAP_SPELLINGS 9
-ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced): 45 (18.4%)
+ACTIVE papers whose scanner is home-grown: 45 (18.4%)
+  = generic 'custom scanner' string:                    25
+  + a one-paper name the paper itself PRODUCED:         20
+  - overlap:                                            0
+  (the other 20 one-paper families are third-party tools only this corpus's one paper used)
 PUBLISHED_HOMEGROWN 45 18.4%
+PUBLISHED_HOMEGROWN_CUSTOM 25
+PUBLISHED_HOMEGROWN_PRODUCED 20
+PUBLISHED_ONEOFF_THIRDPARTY 20
 ==============================================================================
@@ Line 462: / Line 587: @@
     other  CCS/2015/a-search-engine-backed-by-internet-wide-scanning
     other  CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels
-    own    IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet
+    other  IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet
     other  IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks
     own    IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns
@@ Line 472: / Line 597: @@
     own    USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
     own    USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch
-    precision 6/12; 84 x 6/12 = about 42 papers (17.1% of ACTIVE)
+    precision 5/12; 84 x 5/12 = about 35 papers (14.3% of ACTIVE)
-PUBLISHED_PRECISION_OWN_PROBE_RATE 6/12 -> ~42 17.1%
+PUBLISHED_PRECISION_OWN_PROBE_RATE 5/12 -> ~35 14.3%
 --- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12
@@ Line 521: / Line 646: @@
 Practice                    first-person probe  hand-read precision  point estimate
 --------------------------  ------------------  -------------------  --------------
-own probe rate              84 (34.3%)          6/12                 ~42 (17.1%)
+own probe rate              84 (34.3%)          5/12                 ~35 (14.3%)
 own blocklist / exclusion   90 (36.7%)          6/12                 ~45 (18.4%)
 own opt-out offer           69 (28.2%)          9/10                 ~62 (25.3%)
@@ Line 551: / Line 676: @@
 PUBLISHED_TARGETGEN 20 2.2%
 PUBLISHED_ALIASED 10 1.1%
+IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): 42 papers
+  of those, discuss aliased prefixes / dealiasing: 8 (19.0%)
+  the same 10 papers against all of SCAN would be 1.1% — the wrong denominator, most of SCAN never touches IPv6
+PUBLISHED_V6_SUBSET 42
+PUBLISHED_ALIAS_IN_SUBSET 8 19.0%
 IPv6-targeting instruments named, corpus-wide:
@@ Line 645: / Line 776: @@
 search.censys.io returns HTTP 403 to automated clients; censys.com returns 200
 ^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)
-pages/scan_budget.py --self-test: 8 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe
+pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe
 ==============================================================================
@@ Line 710: / Line 841: @@
          in 2015 we observed 11 million scanning
- needles, 0 not located in either rendering
+A2. OFF-CORPUS PAPER (fetched copy, not in the extraction)
+  OK   [out/scan/zmap2013_paper.txt] capable of scanning the IPv4 public address space over 1300 times faster than the most agg…
+  OK   [out/scan/zmap2013_paper.txt] surveying the entire IPv4 address space in under 45 minutes from user space
+  OK   [out/scan/zmap2013_paper.txt] over 97% of the theoretical maximum speed of gigabit Ethernet
+  OK   [out/scan/zmap2013_paper.txt] an estimated 98% coverage of publicly available hosts
+ needles, 0 not located
 ==============================================================================
@@ Line 727: / Line 864: @@
   ZMap defaults: --rate 10000 pps, --probes 1, --cooldown-time 8
       src/zmap.1.ronn at tag v4.4.0 (byte-identical defaults to main), 2026-09-10
+x faster than the most aggressive Nmap default settings; under 45 minutes; 97% of theoretical maximum; 98% coverage
+      Durumeric, Wustrow and Halderman, USENIX Security 2013 — the paper is NOT in this extraction; read from the authors' copy at zmap.io/paper.pdf (Last-Modified 2025-06-22) on 2026-09-10, extracted with pypdf. Needles A27-A30.
+.5 / 0.97 = 42.8
+      the page's own arithmetic reconciling scan_budget.py's theoretical 1 GbE figure with the ZMap paper's measured 45-minute ceiling
+/ 56 = 3.1
+      the page's own arithmetic: cites-the-norm over named-an-IRB, both from the report
 CIDR entries; 592,708,864 removed (13.8%); 3,702,258,432 in scope
       conf/blocklist.conf at tag v4.4.0, computed by pages/scan_budget.py
@@ Line 751: / Line 894: @@
   16777216 / 65536 / 33554432
       hand-computed /8, /16 and the mutation-guard double count in scan_budget.py self-test
-/ 7200 / 172800 / 3600 / 86400 / 60
+ / 120.0 / 7199.0 / 7200 / 7200.0 / 171999.0 / 172800 / 172800.0 / 3600 / 3600.0 / 86400 / 86400.0 / 60 / 60.0 / 119.9 / 24.0 / 2.00
-      second thresholds and divisors in human_time() and scan_plan()
+      second thresholds, divisors and the human_time() boundary cases in scan_budget.py --self-test check 9
 e9 / 1e6
       bits per gigabit and per megabit in line_rate_pps() and scan_plan()
@@ Line 769: / Line 912: @@
       example --rate; ZMap default
  checks
       pages/scan_budget.py --self-test count
@@ Line 798: / Line 941: @@
 === Latest release AND newest tag (a tag-only repo 404s on /releases/latest; /tags is unsorted) ===
-  zmap/zmap                        release=v4.4.0     tags[0:6]=v4.4.0,v4.3.4,v4.3.3,v4.3.2,v4.3.1,v4.3.0
+  zmap/zmap                        release=v4.4.0     published=2026-05-29T06:37:41Z   tags[0:6]=v4.4.0,v4.3.4,v4.3.3,v4.3.2,v4.3.1,v4.3.0
-  zmap/zgrab2                      release=v1.0.0     tags[0:6]=v1.0.0,v1.0.0-RC3,v1.0.0-RC2,v1.0.0-RC1,v0.2.0,v0.1.8
+  zmap/zgrab2                      release=v1.0.0     published=2025-12-04T23:04:38Z   tags[0:6]=v1.0.0,v1.0.0-RC3,v1.0.0-RC2,v1.0.0-RC1,v0.2.0,v0.1.8
-  zmap/zdns                        release=v2.1.1     tags[0:6]=v2.1.1,v2.1.0,v2.0.5,v2.0.4,v2.0.3,v2.0.2
+  zmap/zdns                        release=v2.1.1     published=2026-05-28T23:11:26Z   tags[0:6]=v2.1.1,v2.1.0,v2.0.5,v2.0.4,v2.0.3,v2.0.2
-  robertdavidgraham/masscan        release=1.3.2      tags[0:6]=v2,v1,kali_1_0,1.3.2,1.3.1,1.3.0
+  robertdavidgraham/masscan        release=1.3.2      published=2021-01-31T09:19:32Z   tags[0:6]=v2,v1,kali_1_0,1.3.2,1.3.1,1.3.0
-  idealeer/xmap                    release=2.0.6      tags[0:6]=2.0.6,2.0.5,2.0.4,2.0.3,2.0.2,2.0.1
+  idealeer/xmap                    release=2.0.6      published=2026-03-10T14:15:08Z   tags[0:6]=2.0.6,2.0.5,2.0.4,2.0.3,2.0.2,2.0.1
 === ZMap v4.4.0 man page: the flags and defaults the page publishes ===
@@ Line 855: / Line 998: @@
   The shipped blocklist is IANA special-purpose space. Opt-out entries are
   yours to add, and to keep across runs.
-  self-test OK (8 checks)
+  self-test OK (9 checks)
   plan: 2 port(s) x 2 probe(s) per target = 14,809,033,728 packets
@@ Line 1487: / Line 1630: @@
   'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other',   // rate-limits its own *web interface*, not its scan
   'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study
-  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'own', // "sending ARP requests at 100 Mbps"
+  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'other', // "sending ARP requests at 100 Mbps" is an attack rate on a TTEthernet lab testbed, not a scan rate. Read as 'own' on the first pass and flipped after the generic review pass disputed it on 2026-09-10
   'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled
   'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers"
@@ Line 1556: / Line 1699: @@
   'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200',
   '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)',
-  'pages/scan_budget.py --self-test: 8 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',
+  'pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',
 ];
@@ Line 1582: / Line 1725: @@
   for (const s of stale) say(`  STALE HAND ${JSON.stringify(s)}`);
   if (stale.length > 0) fail(`${stale.length} HAND entries match nothing — a shadowed verdict is a lie about what was read`);
+}
+h('0b. IS THE ZMAP TOOL PAPER IN THE CORPUS? (the page says no, twice)');
+{
+  const inExtract = rows.filter((p) => p.year === 2013 && p.venue === 'USENIX' && /zmap|internet-wide-scanning/i.test(p.slug));
+  const ftDir = path.join(dataRoot(), 'fulltext/2013/USENIX');
+  const ftHits = fs.existsSync(ftDir) ? fs.readdirSync(ftDir).filter((x) => /zmap|scann/i.test(x)) : [];
+  const metaFile = path.join(dataRoot(), 'corpus2/.meta/USENIX-2013.json');
+  const meta = JSON.parse(fs.readFileSync(metaFile, 'utf8'));
+  const recs = Array.isArray(meta) ? meta : (meta.papers ?? meta.records ?? []);
+  const metaHits = recs.filter((r) => /zmap/i.test(JSON.stringify(r)));
+  say(`extraction records matching /zmap|internet-wide-scanning/i in USENIX 2013: ${inExtract.length}`);
+  say(`data/fulltext/2013/USENIX directories matching /zmap|scann/i:            ${ftHits.length}`);
+  say(`corpus2/.meta/USENIX-2013.json records matching /zmap/i:                 ${metaHits.length} of ${recs.length}`);
+  say(`PUBLISHED_ZMAP2013_ABSENT ${inExtract.length === 0 && ftHits.length === 0 && metaHits.length === 0 ? 'yes' : 'NO — the page says it is absent'}`);
+  say(`PUBLISHED_USENIX2013_INDEX ${recs.length}`);
+  if (inExtract.length || ftHits.length || metaHits.length) fail('the ZMap tool paper is in the corpus after all; the page says it is not');
 }
@@ Line 1676: / Line 1836: @@
   // one-paper coined name found only in this paper.
   const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))));
-  say(`ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced): ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`);
+  const customOnly = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner'));
+  const oneoffProduced = ACTIVE.filter((p) => memberOf(p).some((v) => v.via === 'hand' && fam.get(v.family).size === 1 && p.tools.some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family)));
+  say(`ACTIVE papers whose scanner is home-grown: ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`);
+  say(`  = generic 'custom scanner' string:                    ${customOnly.length}`);
+  say(`  + a one-paper name the paper itself PRODUCED:         ${oneoffProduced.length}`);
+  say(`  - overlap:                                            ${customOnly.length + oneoffProduced.length - homegrown.length}`);
+  say(`  (the other ${ones.length - oneoffProduced.length} one-paper families are third-party tools only this corpus's one paper used)`);
   say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`);
+  say(`PUBLISHED_HOMEGROWN_CUSTOM ${customOnly.length}`);
+  say(`PUBLISHED_HOMEGROWN_PRODUCED ${oneoffProduced.length}`);
+  say(`PUBLISHED_ONEOFF_THIRDPARTY ${ones.length - oneoffProduced.length}`);
   if (LIST === 'homegrown') for (const p of homegrown) say(`  ${key(p)}`);
 }
@@ Line 1896: / Line 2065: @@
   say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`);
   say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`);
+  const v6subset = new Set([...hitlistV6, ...v6instr].map(key));
+  const aliasIn = aliased.filter((p) => v6subset.has(key(p)));
+  say('');
+  say(`IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): ${v6subset.size} papers`);
+  say(`  of those, discuss aliased prefixes / dealiasing: ${aliasIn.length} (${pct(aliasIn.length, v6subset.size)})`);
+  say(`  the same 10 papers against all of SCAN would be ${pct(aliased.length, SCAN.length)} — the wrong denominator, most of SCAN never touches IPv6`);
+  say(`PUBLISHED_V6_SUBSET ${v6subset.size}`);
+  say(`PUBLISHED_ALIAS_IN_SUBSET ${aliasIn.length} ${pct(aliasIn.length, v6subset.size)}`);
   say('');
   say('IPv6-targeting instruments named, corpus-wide:');