| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:programming:internet_scanning [2026/09/10 21:28] – Add the review log: three focused passes' findings and dispositions, the ZMap-2013-absence check that closed a reviewer gap, and an explicit note that the generic pass did not return. Refresh the embedded script outputs. Authored by Claude. karel.kubicek.claude | provenance:programming:internet_scanning [2026/09/10 21:35] (current) – Point the review log at the corrected follow-up item. Authored by Claude. karel.kubicek.claude |
|---|
| | **id = ''programming:internet_scanning''**, not ''design:scanning'' | Already decided on [[:roadmap]] on 2026-09-07 (see ''provenance:roadmap'' section 3) and already a red link that ''scripts/sitemap.mjs'' gates on. The decision holds on its merits too: this page is about the invocation, the flags, the rate and the output format — the same register as [[programming:crawler]]. The //whether to scan// decision stays on [[design:automated_measurements]]. No change to the roadmap row was needed beyond retiring it. | | | **id = ''programming:internet_scanning''**, not ''design:scanning'' | Already decided on [[:roadmap]] on 2026-09-07 (see ''provenance:roadmap'' section 3) and already a red link that ''scripts/sitemap.mjs'' gates on. The decision holds on its merits too: this page is about the invocation, the flags, the rate and the output format — the same register as [[programming:crawler]]. The //whether to scan// decision stays on [[design:automated_measurements]]. No change to the roadmap row was needed beyond retiring it. | |
| | **Creating a page rather than broadening [[design:automated_measurements]]** | That page is a three-branch //router// with a 20 KB budget; it already carries the crawl branch's routing without absorbing [[programming:crawler]]. Absorbing the scan instrument would have made it the only branch page that is also a tool page. | | | **Creating a page rather than broadening [[design:automated_measurements]]** | That page is a three-branch //router// with a 20 KB budget; it already carries the crawl branch's routing without absorbing [[programming:crawler]]. Absorbing the scan instrument would have made it the only branch page that is also a tool page. | |
| | **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. Where the two pages disagree on a count, the content page says so. | | | **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. The two folds therefore give different counts for the same family — ZMap 95 here against 93 there, nmap 46 against 40, ZGrab 32 against 31, RIPE Atlas 25 there and excluded here. **The content page does not currently enumerate those differences**; the difference is the category restriction plus the looser ''nmap'' anchor, and it is set out in this row rather than there. | |
| | **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. Every //share of 930// on the page is therefore labelled as ranking-grade. | | | **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. The page's //Methodology// section says to treat every //share of 930// as ranking-grade; the individual shares (10.4%, 10.2%, 57.4%, 30.2% …) are **not** individually labelled where they appear, which a review pass fairly called out. | |
| | **ACTIVE membership is decided by the instrument, not by ''tools[].category''** | ''category'' is a denominator: restricting to ''network-scanner'' gives 231 and loses 14 papers whose ZMap/OONI/Geneva/Censored Planet/hitlist entry is filed under ''other'', ''infrastructure'', ''http-client'' or ''program-analysis''. All 14 are listed in section 1 of the report. | | | **ACTIVE membership is decided by the instrument, not by ''tools[].category''** | ''category'' is a denominator: restricting to ''network-scanner'' gives 231 and loses 14 papers whose ZMap/OONI/Geneva/Censored Planet/hitlist entry is filed under ''other'', ''infrastructure'', ''http-client'' or ''program-analysis''. All 14 are listed in section 1 of the report. | |
| | **…but generic strings and hand verdicts stay inside ''category == "network-scanner"''** | Without that restriction ''%%/^custom\b/i%%'' alone put **464** papers in the population, almost all custom //crawlers// and custom //scripts//, and the hand entries ''Python script'', ''Unicorn'', ''nc'' and ''whois'' fired on unrelated papers. Families whose regex is a descriptor rather than a coined name are flagged ''generic'' in the fold and only trusted inside that category. | | | **…but generic strings and hand verdicts stay inside ''category == "network-scanner"''** | Without that restriction ''%%/^custom\b/i%%'' alone put **464** papers in the population, almost all custom //crawlers// and custom //scripts//, and the hand entries ''Python script'', ''Unicorn'', ''nc'' and ''whois'' fired on unrelated papers. Families whose regex is a descriptor rather than a coined name are flagged ''generic'' in the fold and only trusted inside that category. | |
| | source-address hygiene | 170 (69.4%) | 114 (46.5%) | **6/10** | ~68, "a little over a quarter" | | | source-address hygiene | 170 (69.4%) | 114 (46.5%) | **6/10** | ~68, "a little over a quarter" | |
| | abuse complaints received or handled | 58 (23.7%) | 46 (18.8%) | not sampled | published as an upper bound only | | | abuse complaints received or handled | 58 (23.7%) | 46 (18.8%) | not sampled | published as an upper bound only | |
| | names an IRB or ethics board | — | 56 (22.9%) | not sampled | upper bound | | | names an IRB or ethics board | 56 (22.9%) //(loose; there is no first-person version)// | — | not sampled | upper bound, and **not** to be read against ''ethics.reviewOutcome'' (93, 38.0%) | |
| | cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | — | 174 (71.0%) | not sampled | upper bound | | | cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | 174 (71.0%) //(loose)// | — | not sampled | upper bound | |
| |
| Samples are **every //k//-th paper of the sorted hit list**, so they are reproducible and the report **exits 1** if a sampled paper has no verdict //or// if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read. | Samples are **every //k//-th paper of the sorted hit list**, so they are reproducible and the report **exits 1** if a sampled paper has no verdict //or// if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read. |
| ===== Quotes and per-paper figures ===== | ===== Quotes and per-paper figures ===== |
| |
| ''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **26 needles, 0 not located, 0 weak needles under 20 characters.** All 26 resolved in ''.cols''; the PDF route did not have to fire. | ''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **30 needles, 0 not located, 0 weak needles under 20 characters.** **26** are in-corpus and all 26 resolved in ''.cols'', so the PDF fallback never had to fire for those. The other **4** are from {[durumeric2013_zmap]}, which is **not** in the extraction: they are checked against ''out/scan/zmap2013_paper.txt'' (72,192 bytes), a ''pypdf'' extraction of the authors' own copy at ''zmap.io/paper.pdf''. That check needs its own **de-hyphenation** step — ''pypdf'' keeps the printed hyphen at a line break, so //accu- racy// and //theoreti- cal// are two words in the extraction and one in the PDF, and two of the four quotes failed until it was added. |
| |
| Two things that went wrong and were fixed: | Two things that went wrong and were fixed: |
| | ''node scripts/check_tables.mjs'' on the same | "every table has one width" | | | ''node scripts/check_tables.mjs'' on the same | "every table has one width" | |
| | ''python3 scripts/bib_dedup_scan.py --bib <fresh export + 8 new>'' | 0 definite duplicate pairs; the one candidate touching a new key is ''durumeric2013_zmap''/''williams2024_6sense'', different papers | | | ''python3 scripts/bib_dedup_scan.py --bib <fresh export + 8 new>'' | 0 definite duplicate pairs; the one candidate touching a new key is ''durumeric2013_zmap''/''williams2024_6sense'', different papers | |
| | rendered DOM after save | 35 source ''%%{[key]}%%'' markers → 70 ''bibtex_citekey'' spans → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings | | | rendered DOM after save | **37** source ''%%{[key]}%%'' markers → **74** ''bibtex_citekey'' spans (the plugin emits two per marker) → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings; **0** red links | |
| | red links in the rendered DOM | 3, all of them to this page, before it existed | | | red links in the rendered DOM | 3, all of them to this page, before it existed | |
| | published ''%%<file>%%'' round-trip | byte-identical apart from a stripped trailing newline; runs its self-test | | | published ''%%<file>%%'' round-trip | byte-identical apart from a stripped trailing newline; runs its self-test | |
| | P3.3 | ''www.zoomeye.org'' answers HTTP 521 with an ''X-Via-JSL'' header (Knownsec's own bot challenge), while ''zoomeye.ai'' answers 200. | **Noted, no change.** The page names ZoomEye in a list of query targets and makes no uptime or URL claim about it. Recorded here so a later tightening of the "still operating" check does not read the 521 as a shutdown. | | | P3.3 | ''www.zoomeye.org'' answers HTTP 521 with an ''X-Via-JSL'' header (Knownsec's own bot challenge), while ''zoomeye.ai'' answers 200. | **Noted, no change.** The page names ZoomEye in a list of query targets and makes no uptime or URL claim about it. Recorded here so a later tightening of the "still operating" check does not read the 521 as a shutdown. | |
| |
| ==== Pass 4 — generic (''fable'') — SPAWNED, DID NOT RETURN ==== | ==== Pass 4 — generic (''fable'') ==== |
| |
| <WRAP important> | **It returned after the "did not return" note above had already been written and |
| **The generic pass did not return within this sitting, and the page has not had | published**, in response to a second prompt for partial findings. Fifteen |
| one.** It was spawned against the corrected page after the three focused passes | findings on the already-corrected page, and it earned its slot: five of them are |
| were applied, given the content page, this page, the report, the fold, the | things the three focused passes structurally could not see, and two are wrong |
| published code and four neighbouring pages, and asked for whatever the focused | numbers the number guard passed because the digits occur elsewhere in the report. |
| three were not looking for. It was nudged once to return partial findings and | This is the fourth page in a row where the generic pass found the worst item. |
| did not. It was still running when the sitting ended. | |
| |
| This matters more than it sounds. On ''programming:traffic_files'' the three | ^ # ^ Finding ^ Disposition ^ |
| focused passes returned ten findings and looked comprehensive; the generic pass | | P4.1 | **A contradiction at a seam.** //"56 (22.9%) name an IRB or ethics board **at all**"// — with a ratio built on it — sits two paragraphs above the extraction's own ''ethics.reviewOutcome'', **stated on 93 (38.0%)** of the same 245 papers. The 56 is a loose regex with no measured precision, and "at all" is the probe-scoped qualifier dropped in the prominent place. | **Accepted and fixed.** Both numbers now appear together, the ratio is gone, and the sentence says the 93 is the number to quote. Neither a figures pass (both numbers match the script) nor a citations pass (no citation involved) could see this. | |
| then found fifteen more on the already-corrected page, **including the two | | P4.2 | **The opening box republished the precision the page's own limitations forbid** — it listed "(17%, 18%, 25%, 28%)" while §//Methodology// says "read 'about a fifth', not '18.4%'". That box had been rewritten **by finding P1.3 of the figures pass** two hours earlier: fixing a wrong band introduced a false precision. | **Accepted and fixed** to "somewhere between a seventh and a quarter … read them as 'a minority, and a small one', not as percentages". A fix is a new edit and needs the same scrutiny as the thing it replaced. | |
| worst** — both claims that were true everywhere a focused pass would check and | | P4.3 | **"Forty more families are named by exactly one paper each … 45 of 245 rolled their own" does not add up**, and the reviewer worked out why from the report source: only **20** of the 40 one-paper families are a tool the paper //produced//; the rest are third-party tools (''nc'', ''ike-scan'', ''smbmap'', ''JARM'', ''WhatWeb'') that appear once. Re-derived: 20 produced + 25 "custom scanner" − 5 overlap = 45. | **Accepted and fixed**, and the report now prints the decomposition. The follow-on flourish //"the scanner is a hundred lines around a raw socket"// was cut as unbacked and, for an Internet-wide scanner, untrue. | |
| false where two sections met. On ''statistics:hypothesis_testing'' the generic | | P4.4 | //"several instruments above (Yarrp, MIDAR, scamper) have their tool papers at those venues [PAM, TMA, ANRW, CoNEXT, ACSAC]"// — flagged as probably wrong. **Checked: Yarrp's tool paper is ''IMC 2016'' and is in this corpus.** | **Accepted and fixed.** The venue-gap caveat stands; the examples were removed and replaced with the Yarrp counter-example. A confident parenthetical inside a correct caveat is a good hiding place. | |
| pass was the only one that noticed the page was publishing fifteen uncorrected | | P4.5 | This page claimed //"All 26 [needles] resolved in ''.cols''; the PDF route did not have to fire"//. By then it was 30 needles, four of them off-corpus against a ''pypdf'' extraction that **needs a de-hyphenation step** two quotes failed without. | **Accepted and fixed** above. The claim had gone stale inside the section whose purpose is auditability. | |
| //p//-values on a site whose sibling page is about multiplicity. | | P4.6 | The ''scan_budget.py'' **docstring** still said "eight checks" after check 9 was added. | **Accepted and fixed.** A byte-identity guard proves the page matches the file, not that the file is right. | |
| | | P4.7 | //"The convention this literature has converged on"// is contradicted eight lines later by the page's own ~28% estimate. | **Accepted and fixed** to "the convention the ZMap-era methods papers set — and about a quarter of scanning papers follow". | |
| So: **treat the opening ''%%<WRAP important>%%'' box, the section leads and the | | P4.8 | **The aliasing figure had the wrong denominator.** 10 of 930 (1.1%) counts papers that never touch IPv6. Against the **42** papers that name an IPv6-targeting instrument or say "hitlist" near "IPv6" it is **8 (19.0%)**. //"Almost nobody writes it"// was a "nobody" from a probe. | **Accepted and fixed**, both denominators now published side by side, and the report computes the subset. The rewritten sentence is also a stronger finding: four in five of the papers doing IPv6 targeting do not discuss de-aliasing. | |
| seams between sections on the content page as unreviewed**, and the same for | | P4.9 | Two overclaims **on this page**: that the content page says where the two folds disagree (it does not), and that every //share of 930// is labelled ranking-grade (only the Methodology section says it). | **Accepted and fixed**, by writing down what is actually true — including the four family counts where this fold and ''am_fold.mjs'' disagree. | |
| this page's account of its own rigour. A follow-up work item | | P4.10 | The "Review in flight" placeholder was still on the published page. | **Already fixed** before the pass returned; it saw an older snapshot. | |
| (''run-generic-review-internet-scanning'') has been filed to run it. | | P4.11 | **OONI (7) and Censored Planet (4) contradict the ACTIVE definition**: a paper re-analysing OONI data did not send probes to addresses it chose. And an IPv6 target generator "sends nothing". | **Accepted, definition rewritten rather than the population changed.** Both edges are now stated on the content page as judgement calls: a censorship platform counts either way because the platform is the instrument and the corpus rarely says which; a target generator counts because in IPv6 it is the step that decides what gets probed. | |
| </WRAP> | | P4.12 | Four technical errors in the instrument table and one in the published docstring: ZGrab does not "complete the handshake ZMap only started" (ZMap RSTs; ZGrab opens its own connection); nmap is not "connection-oriented"; ZMap was called "IPv4/IPv6" while a later section says its design does not extend to 128 bits; masscan has had IPv6 since 1.3; and ZMap's default ''synscan'' sends **no** TCP options, so the docstring's "SYN with an MSS option" was wrong (the 84-byte constant is unchanged — the frame is padded to the 60-byte minimum either way). | **All accepted and fixed.** None of these has a number in it, which is why nothing caught them. | |
| | | P4.13 | Four unbacked framing sentences, including //"the fourth party most papers forget"// and a per-year IPv6 series quoted only at its two most favourable endpoints (2010 12.5% → 2025 44.3%, skipping 2013's 0% and 2024's 29.2%). | **Accepted and fixed**, the series now given at five points including the non-monotone ones. | |
| | | P4.14 | **A hand verdict disputed.** IEEE S&P 2023 //PCspoof// was marked ''own'' on "sending ARP requests at 100 Mbps" — an attack rate on a TTEthernet lab testbed, not a scan rate. | **Accepted and flipped**, which moved the rate precision from 6/12 to **5/12** and the estimate from ~42 to **~35 (one in seven)**. Two figures on the page were stale afterwards and ''check_page_numbers.mjs'' **passed anyway**, because 42, 6 and 17 all occur elsewhere in the report — the documented presence-not-binding weakness, hit for real. Found by grep, not by the guard. | |
| | | P4.15 | **A structure gap**: the invocation only ever scans the whole space on TCP/443, when most first scans are a target list. | **Accepted**, and ''%%-w/--allowlist-file%%'', ''%%-I/--list-of-ips-file%%'' and ''%%--probe-module%%'' added with the note that the blocklist still applies on top. | |
| |
| The three focused passes were also slow — two had to be prompted to return | It also read the provenance probe table and found the IRB row filed under the |
| partial findings after a long wall-clock stretch. If that recurs, prompt for | "first-person" column although the report calls it loose. That is correct and |
| partials early rather than waiting. | the row is now labelled. Its own **NOT READ** list is long — the embedded fold |
| | source, the external-checks output, `practices:ethics` and `design_dns.txt` for |
| | duplication, and a possible disagreement between this page's ZMap 95 + XMap 16 |
| | and [[design:dns]]' "ZMap and XMap (112 papers)" that it could not resolve. |
| | **That last one is unresolved.** A follow-up item |
| | (''internet_scanning: three follow-ups the review left open'') is filed for it, |
| | together with the two gaps below. |
| |
| ==== What the passes did not cover ==== | ==== What the passes did not cover ==== |
| parts of this page nothing checked: | parts of this page nothing checked: |
| |
| | * **The 112-vs-111 question above.** [[design:dns]] reports "ZMap and XMap (112 papers)" against this page's ZMap 95 + XMap 16 = 111. That may be an overlap artefact (a paper naming both) or a real disagreement between two folds. Not resolved. |
| * **Nobody hand-audited the 174 ''HAND'' verdicts** in ''scan_fold.mjs'' for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire. | * **Nobody hand-audited the 174 ''HAND'' verdicts** in ''scan_fold.mjs'' for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire. |
| * **Nobody re-read the 44 probe papers** to confirm the ''own''/''other'' verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates. | * **Nobody re-read the 44 probe papers** to confirm the ''own''/''other'' verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates. |
| ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6 | ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6 |
| PUBLISHED_ZMAP_SPELLINGS 9 | PUBLISHED_ZMAP_SPELLINGS 9 |
| ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced): 45 (18.4%) | ACTIVE papers whose scanner is home-grown: 45 (18.4%) |
| | = generic 'custom scanner' string: 25 |
| | + a one-paper name the paper itself PRODUCED: 20 |
| | - overlap: 0 |
| | (the other 20 one-paper families are third-party tools only this corpus's one paper used) |
| PUBLISHED_HOMEGROWN 45 18.4% | PUBLISHED_HOMEGROWN 45 18.4% |
| | PUBLISHED_HOMEGROWN_CUSTOM 25 |
| | PUBLISHED_HOMEGROWN_PRODUCED 20 |
| | PUBLISHED_ONEOFF_THIRDPARTY 20 |
| |
| ============================================================================== | ============================================================================== |
| other CCS/2015/a-search-engine-backed-by-internet-wide-scanning | other CCS/2015/a-search-engine-backed-by-internet-wide-scanning |
| other CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels | other CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels |
| own IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet | other IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet |
| other IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks | other IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks |
| own IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns | own IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns |
| own USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications | own USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications |
| own USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch | own USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch |
| precision 6/12; 84 x 6/12 = about 42 papers (17.1% of ACTIVE) | precision 5/12; 84 x 5/12 = about 35 papers (14.3% of ACTIVE) |
| PUBLISHED_PRECISION_OWN_PROBE_RATE 6/12 -> ~42 17.1% | PUBLISHED_PRECISION_OWN_PROBE_RATE 5/12 -> ~35 14.3% |
| |
| --- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12 | --- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12 |
| Practice first-person probe hand-read precision point estimate | Practice first-person probe hand-read precision point estimate |
| -------------------------- ------------------ ------------------- -------------- | -------------------------- ------------------ ------------------- -------------- |
| own probe rate 84 (34.3%) 6/12 ~42 (17.1%) | own probe rate 84 (34.3%) 5/12 ~35 (14.3%) |
| own blocklist / exclusion 90 (36.7%) 6/12 ~45 (18.4%) | own blocklist / exclusion 90 (36.7%) 6/12 ~45 (18.4%) |
| own opt-out offer 69 (28.2%) 9/10 ~62 (25.3%) | own opt-out offer 69 (28.2%) 9/10 ~62 (25.3%) |
| PUBLISHED_TARGETGEN 20 2.2% | PUBLISHED_TARGETGEN 20 2.2% |
| PUBLISHED_ALIASED 10 1.1% | PUBLISHED_ALIASED 10 1.1% |
| | |
| | IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): 42 papers |
| | of those, discuss aliased prefixes / dealiasing: 8 (19.0%) |
| | the same 10 papers against all of SCAN would be 1.1% — the wrong denominator, most of SCAN never touches IPv6 |
| | PUBLISHED_V6_SUBSET 42 |
| | PUBLISHED_ALIAS_IN_SUBSET 8 19.0% |
| |
| IPv6-targeting instruments named, corpus-wide: | IPv6-targeting instruments named, corpus-wide: |
| search.censys.io returns HTTP 403 to automated clients; censys.com returns 200 | search.censys.io returns HTTP 403 to automated clients; censys.com returns 200 |
| 2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more) | 2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more) |
| pages/scan_budget.py --self-test: 9 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe | pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe |
| |
| ============================================================================== | ============================================================================== |
| 'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other', // rate-limits its own *web interface*, not its scan | 'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other', // rate-limits its own *web interface*, not its scan |
| 'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study | 'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study |
| 'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'own', // "sending ARP requests at 100 Mbps" | 'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'other', // "sending ARP requests at 100 Mbps" is an attack rate on a TTEthernet lab testbed, not a scan rate. Read as 'own' on the first pass and flipped after the generic review pass disputed it on 2026-09-10 |
| 'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled | 'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled |
| 'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers" | 'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers" |
| 'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200', | 'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200', |
| '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)', | '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)', |
| 'pages/scan_budget.py --self-test: 9 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe', | 'pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe', |
| ]; | ]; |
| |
| // one-paper coined name found only in this paper. | // one-paper coined name found only in this paper. |
| const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family)))); | const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family)))); |
| say(`ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced): ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`); | const customOnly = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner')); |
| | const oneoffProduced = ACTIVE.filter((p) => memberOf(p).some((v) => v.via === 'hand' && fam.get(v.family).size === 1 && p.tools.some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))); |
| | say(`ACTIVE papers whose scanner is home-grown: ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`); |
| | say(` = generic 'custom scanner' string: ${customOnly.length}`); |
| | say(` + a one-paper name the paper itself PRODUCED: ${oneoffProduced.length}`); |
| | say(` - overlap: ${customOnly.length + oneoffProduced.length - homegrown.length}`); |
| | say(` (the other ${ones.length - oneoffProduced.length} one-paper families are third-party tools only this corpus's one paper used)`); |
| say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`); | say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`); |
| | say(`PUBLISHED_HOMEGROWN_CUSTOM ${customOnly.length}`); |
| | say(`PUBLISHED_HOMEGROWN_PRODUCED ${oneoffProduced.length}`); |
| | say(`PUBLISHED_ONEOFF_THIRDPARTY ${ones.length - oneoffProduced.length}`); |
| if (LIST === 'homegrown') for (const p of homegrown) say(` ${key(p)}`); | if (LIST === 'homegrown') for (const p of homegrown) say(` ${key(p)}`); |
| } | } |
| say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`); | say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`); |
| say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`); | say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`); |
| | const v6subset = new Set([...hitlistV6, ...v6instr].map(key)); |
| | const aliasIn = aliased.filter((p) => v6subset.has(key(p))); |
| | say(''); |
| | say(`IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): ${v6subset.size} papers`); |
| | say(` of those, discuss aliased prefixes / dealiasing: ${aliasIn.length} (${pct(aliasIn.length, v6subset.size)})`); |
| | say(` the same 10 papers against all of SCAN would be ${pct(aliased.length, SCAN.length)} — the wrong denominator, most of SCAN never touches IPv6`); |
| | say(`PUBLISHED_V6_SUBSET ${v6subset.size}`); |
| | say(`PUBLISHED_ALIAS_IN_SUBSET ${aliasIn.length} ${pct(aliasIn.length, v6subset.size)}`); |
| say(''); | say(''); |
| say('IPv6-targeting instruments named, corpus-wide:'); | say('IPv6-targeting instruments named, corpus-wide:'); |