User Tools

Site Tools


provenance:programming:internet_scanning

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
provenance:programming:internet_scanning [2026/09/10 21:28] – Add the review log: three focused passes' findings and dispositions, the ZMap-2013-absence check that closed a reviewer gap, and an explicit note that the generic pass did not return. Refresh the embedded script outputs. Authored by Claude. karel.kubicek.claudeprovenance:programming:internet_scanning [2026/09/10 21:35] (current) – Point the review log at the corrected follow-up item. Authored by Claude. karel.kubicek.claude
Line 25: Line 25:
 | **id = ''programming:internet_scanning''**, not ''design:scanning'' | Already decided on [[:roadmap]] on 2026-09-07 (see ''provenance:roadmap'' section 3) and already a red link that ''scripts/sitemap.mjs'' gates on. The decision holds on its merits too: this page is about the invocation, the flags, the rate and the output format — the same register as [[programming:crawler]]. The //whether to scan// decision stays on [[design:automated_measurements]]. No change to the roadmap row was needed beyond retiring it. | | **id = ''programming:internet_scanning''**, not ''design:scanning'' | Already decided on [[:roadmap]] on 2026-09-07 (see ''provenance:roadmap'' section 3) and already a red link that ''scripts/sitemap.mjs'' gates on. The decision holds on its merits too: this page is about the invocation, the flags, the rate and the output format — the same register as [[programming:crawler]]. The //whether to scan// decision stays on [[design:automated_measurements]]. No change to the roadmap row was needed beyond retiring it. |
 | **Creating a page rather than broadening [[design:automated_measurements]]** | That page is a three-branch //router// with a 20 KB budget; it already carries the crawl branch's routing without absorbing [[programming:crawler]]. Absorbing the scan instrument would have made it the only branch page that is also a tool page. | | **Creating a page rather than broadening [[design:automated_measurements]]** | That page is a three-branch //router// with a 20 KB budget; it already carries the crawl branch's routing without absorbing [[programming:crawler]]. Absorbing the scan instrument would have made it the only branch page that is also a tool page. |
-| **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. Where the two pages disagree on a count, the content page says so. | +| **A separate fold (''scan_fold.mjs'') rather than extending ''am_fold.mjs''** | ''am_fold.mjs'' backs [[design:automated_measurements]]. Editing it would silently move that page's published 93/40/38/31/25 and its 205-string residue. ''scan_fold.mjs'' is a superset with different //kinds//, a negative-guard list, a generic/proper-name distinction and a 174-entry hand map. The two folds therefore give different counts for the same family — ZMap 95 here against 93 there, nmap 46 against 40, ZGrab 32 against 31, RIPE Atlas 25 there and excluded here. **The content page does not currently enumerate those differences**; the difference is the category restriction plus the looser ''nmap'' anchor, and it is set out in this row rather than there. | 
-| **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. Every //share of 930// on the page is therefore labelled as ranking-grade. |+| **ACTIVE = 245, not SCAN = 930, as the page's population** | 930 is a ''studyTypes'' tag and ''studyTypes'' is the least stable field in the schema (57% run-to-run). An instrument page should be denominated by the instrument. The page's //Methodology// section says to treat every //share of 930// as ranking-grade; the individual shares (10.4%, 10.2%, 57.4%, 30.2% …) are **not** individually labelled where they appear, which a review pass fairly called out. |
 | **ACTIVE membership is decided by the instrument, not by ''tools[].category''** | ''category'' is a denominator: restricting to ''network-scanner'' gives 231 and loses 14 papers whose ZMap/OONI/Geneva/Censored Planet/hitlist entry is filed under ''other'', ''infrastructure'', ''http-client'' or ''program-analysis''. All 14 are listed in section 1 of the report. | | **ACTIVE membership is decided by the instrument, not by ''tools[].category''** | ''category'' is a denominator: restricting to ''network-scanner'' gives 231 and loses 14 papers whose ZMap/OONI/Geneva/Censored Planet/hitlist entry is filed under ''other'', ''infrastructure'', ''http-client'' or ''program-analysis''. All 14 are listed in section 1 of the report. |
 | **…but generic strings and hand verdicts stay inside ''category == "network-scanner"''** | Without that restriction ''%%/^custom\b/i%%'' alone put **464** papers in the population, almost all custom //crawlers// and custom //scripts//, and the hand entries ''Python script'', ''Unicorn'', ''nc'' and ''whois'' fired on unrelated papers. Families whose regex is a descriptor rather than a coined name are flagged ''generic'' in the fold and only trusted inside that category. | | **…but generic strings and hand verdicts stay inside ''category == "network-scanner"''** | Without that restriction ''%%/^custom\b/i%%'' alone put **464** papers in the population, almost all custom //crawlers// and custom //scripts//, and the hand entries ''Python script'', ''Unicorn'', ''nc'' and ''whois'' fired on unrelated papers. Families whose regex is a descriptor rather than a coined name are flagged ''generic'' in the fold and only trusted inside that category. |
Line 83: Line 83:
 | source-address hygiene | 170 (69.4%) | 114 (46.5%) | **6/10** | ~68, "a little over a quarter" | | source-address hygiene | 170 (69.4%) | 114 (46.5%) | **6/10** | ~68, "a little over a quarter" |
 | abuse complaints received or handled | 58 (23.7%) | 46 (18.8%) | not sampled | published as an upper bound only | | abuse complaints received or handled | 58 (23.7%) | 46 (18.8%) | not sampled | published as an upper bound only |
-| names an IRB or ethics board | — | 56 (22.9%) | not sampled | upper bound | +| names an IRB or ethics board | 56 (22.9%) //(loose; there is no first-person version)// | — | not sampled | upper bound, and **not** to be read against ''ethics.reviewOutcome''  (93, 38.0%) 
-| cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | — | 174 (71.0%) | not sampled | upper bound |+| cites Durumeric / Partridge-Allman / Menlo / "good Internet citizen" | 174 (71.0%) //(loose)// | — | not sampled | upper bound |
  
 Samples are **every //k//-th paper of the sorted hit list**, so they are reproducible and the report **exits 1** if a sampled paper has no verdict //or// if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read. Samples are **every //k//-th paper of the sorted hit list**, so they are reproducible and the report **exits 1** if a sampled paper has no verdict //or// if a verdict exists for a paper the sample no longer returns. That second direction matters: if the fold or the regex moves, the sample moves, and a stale verdict would otherwise sit there looking read.
Line 149: Line 149:
 ===== Quotes and per-paper figures ===== ===== Quotes and per-paper figures =====
  
-''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **26 needles, 0 not located, 0 weak needles under 20 characters.** All 26 resolved in ''.cols''the PDF route did not have to fire.+''scripts/verify_scanning_figures.mjs'' looks up every quoted sentence and per-paper figure on the content page in the cited paper's ''paper.cols.txt'' with whitespace collapsed, falling back to a ''pypdf'' re-extraction of ''paper.pdf''. **30 needles, 0 not located, 0 weak needles under 20 characters.** **26** are in-corpus and all 26 resolved in ''.cols'', so the PDF fallback never had to fire for those. The other **4** are from {[durumeric2013_zmap]}, which is **not** in the extraction: they are checked against ''out/scan/zmap2013_paper.txt'' (72,192 bytes), a ''pypdf'' extraction of the authors' own copy at ''zmap.io/paper.pdf''. That check needs its own **de-hyphenation** step — ''pypdf'' keeps the printed hyphen at a line break, so //accu- racy// and //theoreti- cal// are two words in the extraction and one in the PDF, and two of the four quotes failed until it was added.
  
 Two things that went wrong and were fixed: Two things that went wrong and were fixed:
Line 205: Line 205:
 | ''node scripts/check_tables.mjs'' on the same | "every table has one width" | | ''node scripts/check_tables.mjs'' on the same | "every table has one width" |
 | ''python3 scripts/bib_dedup_scan.py --bib <fresh export + 8 new>'' | 0 definite duplicate pairs; the one candidate touching a new key is ''durumeric2013_zmap''/''williams2024_6sense'', different papers | | ''python3 scripts/bib_dedup_scan.py --bib <fresh export + 8 new>'' | 0 definite duplicate pairs; the one candidate touching a new key is ''durumeric2013_zmap''/''williams2024_6sense'', different papers |
-| rendered DOM after save | 35 source ''%%{[key]}%%'' markers → 70 ''bibtex_citekey'' spans → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings |+| rendered DOM after save | **37** source ''%%{[key]}%%'' markers → **74** ''bibtex_citekey'' spans (the plugin emits two per marker) → **15** ''%%<dt>%%'' references for 15 distinct keys; **0** "could not be found" warnings; **0** red links |
 | red links in the rendered DOM | 3, all of them to this page, before it existed | | red links in the rendered DOM | 3, all of them to this page, before it existed |
 | published ''%%<file>%%'' round-trip | byte-identical apart from a stripped trailing newline; runs its self-test | | published ''%%<file>%%'' round-trip | byte-identical apart from a stripped trailing newline; runs its self-test |
Line 276: Line 276:
 | P3.3 | ''www.zoomeye.org'' answers HTTP 521 with an ''X-Via-JSL'' header (Knownsec's own bot challenge), while ''zoomeye.ai'' answers 200. | **Noted, no change.** The page names ZoomEye in a list of query targets and makes no uptime or URL claim about it. Recorded here so a later tightening of the "still operating" check does not read the 521 as a shutdown. | | P3.3 | ''www.zoomeye.org'' answers HTTP 521 with an ''X-Via-JSL'' header (Knownsec's own bot challenge), while ''zoomeye.ai'' answers 200. | **Noted, no change.** The page names ZoomEye in a list of query targets and makes no uptime or URL claim about it. Recorded here so a later tightening of the "still operating" check does not read the 521 as a shutdown. |
  
-==== Pass 4 — generic (''fable''— SPAWNED, DID NOT RETURN ====+==== Pass 4 — generic (''fable'') ====
  
-<WRAP important> +**It returned after the "did not return" note above had already been written and 
-**The generic pass did not return within this sitting, and the page has not had +published**, in response to a second prompt for partial findings. Fifteen 
-one.** It was spawned against the corrected page after the three focused passes +findings on the already-corrected page, and it earned its slot: five of them are 
-were applied, given the content page, this page, the reportthe fold, the +things the three focused passes structurally could not see, and two are wrong 
-published code and four neighbouring pages, and asked for whatever the focused +numbers the number guard passed because the digits occur elsewhere in the report
-three were not looking for. It was nudged once to return partial findings and +This is the fourth page in a row where the generic pass found the worst item.
-did not. It was still running when the sitting ended.+
  
-This matters more than it soundsOn ''programming:traffic_files'' the three +^ # ^ Finding ^ Disposition ^ 
-focused passes returned ten findings and looked comprehensive; the generic pass +| P4.1 | **A contradiction at a seam.** //"56 (22.9%) name an IRB or ethics board **at all**"// — with a ratio built on it — sits two paragraphs above the extraction's own ''ethics.reviewOutcome'', **stated on 93 (38.0%)** of the same 245 papers. The 56 is a loose regex with no measured precision, and "at all" is the probe-scoped qualifier dropped in the prominent place. | **Accepted and fixed.** Both numbers now appear together, the ratio is gone, and the sentence says the 93 is the number to quote. Neither a figures pass (both numbers match the script) nor a citations pass (no citation involved) could see this. | 
-then found fifteen more on the already-corrected page, **including the two +| P4.2 | **The opening box republished the precision the page's own limitations forbid** — it listed "(17%18%, 25%, 28%)" while §//Methodology// says "read 'about a fifth', not '18.4%'". That box had been rewritten **by finding P1.3 of the figures pass** two hours earlier: fixing a wrong band introduced a false precision. | **Accepted and fixed** to "somewhere between a seventh and a quarter … read them as 'a minority, and a small one', not as percentages". A fix is a new edit and needs the same scrutiny as the thing it replaced. | 
-worst** — both claims that were true everywhere focused pass would check and +| P4.3 | **"Forty more families are named by exactly one paper each … 45 of 245 rolled their own" does not add up**, and the reviewer worked out why from the report source: only **20** of the 40 one-paper families are a tool the paper //produced//; the rest are third-party tools (''nc'', ''ike-scan'', ''smbmap'', ''JARM'', ''WhatWeb''that appear once. Re-derived: 20 produced + 25 "custom scanner" − 5 overlap = 45. | **Accepted and fixed**, and the report now prints the decomposition. The follow-on flourish //"the scanner is hundred lines around a raw socket"// was cut as unbacked and, for an Internet-wide scanner, untrue. | 
-false where two sections metOn ''statistics:hypothesis_testing'' the generic +| P4.4 | //"several instruments above (Yarrp, MIDAR, scamper) have their tool papers at those venues [PAM, TMA, ANRW, CoNEXT, ACSAC]"// — flagged as probably wrong. **Checked: Yarrp's tool paper is ''IMC 2016'' and is in this corpus.** | **Accepted and fixed.** The venue-gap caveat stands; the examples were removed and replaced with the Yarrp counter-example. A confident parenthetical inside a correct caveat is a good hiding place. | 
-pass was the only one that noticed the page was publishing fifteen uncorrected +| P4.5 | This page claimed //"All 26 [needles] resolved in ''.cols''; the PDF route did not have to fire"//. By then it was 30 needles, four of them off-corpus against a ''pypdf'' extraction that **needs a de-hyphenation step** two quotes failed without. | **Accepted and fixed** above. The claim had gone stale inside the section whose purpose is auditability. | 
-//p//-values on a site whose sibling page is about multiplicity+| P4.6 | The ''scan_budget.py'' **docstring** still said "eight checks" after check 9 was added. | **Accepted and fixed.** A byte-identity guard proves the page matches the file, not that the file is right. | 
- +| P4.7 | //"The convention this literature has converged on"// is contradicted eight lines later by the page's own ~28% estimate. | **Accepted and fixed** to "the convention the ZMap-era methods papers set — and about a quarter of scanning papers follow". | 
-So: **treat the opening ''%%<WRAP important>%%'' box, the section leads and the +| P4.8 | **The aliasing figure had the wrong denominator.** 10 of 930 (1.1%) counts papers that never touch IPv6. Against the **42** papers that name an IPv6-targeting instrument or say "hitlist" near "IPv6" it is **8 (19.0%)**. //"Almost nobody writes it"// was a "nobody" from a probe. | **Accepted and fixed**, both denominators now published side by side, and the report computes the subset. The rewritten sentence is also a stronger finding: four in five of the papers doing IPv6 targeting do not discuss de-aliasing. | 
-seams between sections on the content page as unreviewed**, and the same for +| P4.9 | Two overclaims **on this page**: that the content page says where the two folds disagree (it does not), and that every //share of 930// is labelled ranking-grade (only the Methodology section says it). | **Accepted and fixed**, by writing down what is actually true — including the four family counts where this fold and ''am_fold.mjs'' disagree| 
-this page's account of its own rigour. A follow-up work item +| P4.10 | The "Review in flight" placeholder was still on the published page. | **Already fixed** before the pass returned; it saw an older snapshot. | 
-(''run-generic-review-internet-scanning'') has been filed to run it. +| P4.11 | **OONI (7) and Censored Planet (4) contradict the ACTIVE definition**a paper re-analysing OONI data did not send probes to addresses it chose. And an IPv6 target generator "sends nothing". | **Accepted, definition rewritten rather than the population changed.** Both edges are now stated on the content page as judgement calls: a censorship platform counts either way because the platform is the instrument and the corpus rarely says which; a target generator counts because in IPv6 it is the step that decides what gets probed. | 
-</WRAP>+| P4.12 | Four technical errors in the instrument table and one in the published docstring: ZGrab does not "complete the handshake ZMap only started" (ZMap RSTs; ZGrab opens its own connection); nmap is not "connection-oriented"; ZMap was called "IPv4/IPv6" while a later section says its design does not extend to 128 bits; masscan has had IPv6 since 1.3; and ZMap's default ''synscan'' sends **no** TCP options, so the docstring's "SYN with an MSS option" was wrong (the 84-byte constant is unchanged — the frame is padded to the 60-byte minimum either way). | **All accepted and fixed.** None of these has a number in it, which is why nothing caught them. | 
 +| P4.13 | Four unbacked framing sentences, including //"the fourth party most papers forget"// and a per-year IPv6 series quoted only at its two most favourable endpoints (2010 12.5→ 2025 44.3%, skipping 2013's 0and 2024's 29.2%). | **Accepted and fixed**, the series now given at five points including the non-monotone ones. | 
 +| P4.14 | **A hand verdict disputed.** IEEE S&P 2023 //PCspoof// was marked ''own'' on "sending ARP requests at 100 Mbps" — an attack rate on a TTEthernet lab testbednot a scan rate. | **Accepted and flipped**, which moved the rate precision from 6/12 to **5/12** and the estimate from ~42 to **~35 (one in seven)**. Two figures on the page were stale afterwards and ''check_page_numbers.mjs'' **passed anyway**, because 42and 17 all occur elsewhere in the report — the documented presence-not-binding weakness, hit for real. Found by grep, not by the guard. | 
 +| P4.15 | **structure gap**: the invocation only ever scans the whole space on TCP/443, when most first scans are a target list. | **Accepted**, and ''%%-w/--allowlist-file%%'', ''%%-I/--list-of-ips-file%%'' and ''%%--probe-module%%'' added with the note that the blocklist still applies on top|
  
-The three focused passes were also slow — two had to be prompted to return +It also read the provenance probe table and found the IRB row filed under the 
-partial findings after a long wall-clock stretchIf that recursprompt for +"first-person" column although the report calls it loose. That is correct and 
-partials early rather than waiting.+the row is now labelled. Its own **NOT READ** list is long — the embedded fold 
 +source, the external-checks output, `practices:ethics` and `design_dns.txt` for 
 +duplicationand a possible disagreement between this page's ZMap 95 + XMap 16 
 +and [[design:dns]]' "ZMap and XMap (112 papers)" that it could not resolve. 
 +**That last one is unresolved.** A follow-up item 
 +(''internet_scanning: three follow-ups the review left open'') is filed for it, 
 +together with the two gaps below.
  
 ==== What the passes did not cover ==== ==== What the passes did not cover ====
Line 309: Line 317:
 parts of this page nothing checked: parts of this page nothing checked:
  
 +  * **The 112-vs-111 question above.** [[design:dns]] reports "ZMap and XMap (112 papers)" against this page's ZMap 95 + XMap 16 = 111. That may be an overlap artefact (a paper naming both) or a real disagreement between two folds. Not resolved.
   * **Nobody hand-audited the 174 ''HAND'' verdicts** in ''scan_fold.mjs'' for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire.   * **Nobody hand-audited the 174 ''HAND'' verdicts** in ''scan_fold.mjs'' for judgement (only the coverage and staleness guards were mutation-tested). A wrong kind on a one-off scanner name moves ACTIVE by one paper and nothing would fire.
   * **Nobody re-read the 44 probe papers** to confirm the ''own''/''other'' verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates.   * **Nobody re-read the 44 probe papers** to confirm the ''own''/''other'' verdicts in section 8 independently of the author's reading. Those verdicts carry the four practice estimates.
Line 450: Line 459:
 ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6 ZMap spellings: TCP Options module for ZMap | ZMap | ZMap-based script | ZMap6 | ZMapv6 | Zmap | zmap | zmap2 | zmap6
 PUBLISHED_ZMAP_SPELLINGS 9 PUBLISHED_ZMAP_SPELLINGS 9
-ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced)45 (18.4%)+ACTIVE papers whose scanner is home-grown: 45 (18.4%) 
 +  = generic 'custom scanner' string:                    25 
 +  + a one-paper name the paper itself PRODUCED        20 
 +  - overlap:                                            0 
 +  (the other 20 one-paper families are third-party tools only this corpus's one paper used)
 PUBLISHED_HOMEGROWN 45 18.4% PUBLISHED_HOMEGROWN 45 18.4%
 +PUBLISHED_HOMEGROWN_CUSTOM 25
 +PUBLISHED_HOMEGROWN_PRODUCED 20
 +PUBLISHED_ONEOFF_THIRDPARTY 20
  
 ============================================================================== ==============================================================================
Line 571: Line 587:
     other  CCS/2015/a-search-engine-backed-by-internet-wide-scanning     other  CCS/2015/a-search-engine-backed-by-internet-wide-scanning
     other  CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels     other  CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels
-    own    IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet+    other  IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet
     other  IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks     other  IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks
     own    IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns     own    IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns
Line 581: Line 597:
     own    USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications     own    USENIX/2024/6sense-internet-wide-ipv6-scanning-and-its-security-applications
     own    USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch     own    USENIX/2025/exposing-and-circumventing-sni-based-quic-censorship-of-the-great-firewall-of-ch
-    precision 6/12; 84 x 6/12 = about 42 papers (17.1% of ACTIVE) +    precision 5/12; 84 x 5/12 = about 35 papers (14.3% of ACTIVE) 
-PUBLISHED_PRECISION_OWN_PROBE_RATE 6/12 -> ~42 17.1%+PUBLISHED_PRECISION_OWN_PROBE_RATE 5/12 -> ~35 14.3%
  
 --- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12 --- own blocklist / exclusion: 90 first-person hits of 245 ACTIVE; sample of 12
Line 630: Line 646:
 Practice                    first-person probe  hand-read precision  point estimate Practice                    first-person probe  hand-read precision  point estimate
 --------------------------  ------------------  -------------------  -------------- --------------------------  ------------------  -------------------  --------------
-own probe rate              84 (34.3%)          6/12                 ~42 (17.1%)+own probe rate              84 (34.3%)          5/12                 ~35 (14.3%)
 own blocklist / exclusion   90 (36.7%)          6/12                 ~45 (18.4%) own blocklist / exclusion   90 (36.7%)          6/12                 ~45 (18.4%)
 own opt-out offer           69 (28.2%)          9/10                 ~62 (25.3%) own opt-out offer           69 (28.2%)          9/10                 ~62 (25.3%)
Line 660: Line 676:
 PUBLISHED_TARGETGEN 20 2.2% PUBLISHED_TARGETGEN 20 2.2%
 PUBLISHED_ALIASED 10 1.1% PUBLISHED_ALIASED 10 1.1%
 +
 +IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): 42 papers
 +  of those, discuss aliased prefixes / dealiasing: 8 (19.0%)
 +  the same 10 papers against all of SCAN would be 1.1% — the wrong denominator, most of SCAN never touches IPv6
 +PUBLISHED_V6_SUBSET 42
 +PUBLISHED_ALIAS_IN_SUBSET 8 19.0%
  
 IPv6-targeting instruments named, corpus-wide: IPv6-targeting instruments named, corpus-wide:
Line 754: Line 776:
 search.censys.io returns HTTP 403 to automated clients; censys.com returns 200 search.censys.io returns HTTP 403 to automated clients; censys.com returns 200
 2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more) 2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)
-pages/scan_budget.py --self-test: 9 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe+pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe
  
 ============================================================================== ==============================================================================
Line 1608: Line 1630:
   'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other',   // rate-limits its own *web interface*, not its scan   'CCS/2015/a-search-engine-backed-by-internet-wide-scanning': 'other',   // rate-limits its own *web interface*, not its scan
   'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study   'CCS/2025/poster-eris-evaluating-rov-via-icmpv6-rate-limiting-side-channels': 'other', // ICMPv6 rate limiting is the side channel under study
-  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'own', // "sending ARP requests at 100 Mbps"+  'IEEE-SP/2023/pcspoof-compromising-the-safety-of-time-triggered-ethernet': 'other', // "sending ARP requests at 100 Mbps" is an attack rate on a TTEthernet lab testbed, not a scan rate. Read as 'own' on the first pass and flipped after the generic review pass disputed it on 2026-09-10
   'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled   'IMC/2016/on-the-free-bridge-across-the-digital-divide-assessing-the-quality-of-facebooks': 'other', // Free Basics traffic is throttled
   'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers"   'IMC/2022/saving-brians-privacy-the-perils-of-privacy-exposure-through-reverse-dns': 'own', // "We rate-limit requests to authoritative name servers"
Line 1677: Line 1699:
   'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200',   'search.censys.io returns HTTP 403 to automated clients; censys.com returns 200',
   '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)',   '2^32 = 4,294,967,296 IPv4 addresses; the shipped blocklist removes 592,708,864 (13.8%), leaving 3,702,258,432 — computed by pages/scan_budget.py from the live file, collapsing overlaps (255.255.255.255/32 is inside 240.0.0.0/4, so a naive sum gives one address more)',
-  'pages/scan_budget.py --self-test: 9 checks. Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',+  'pages/scan_budget.py --self-test: 9 checks (check 9 covers human_time() boundaries). Default run: 4.29 days at 10,000 pps, 10.3 h at 100,000 pps, 41.5 min at 1 GbE line rate (1,488,095 pps), 4.1 min at 10 GbE (14,880,952 pps), on 84 bytes of wire time per probe',
 ]; ];
  
Line 1814: Line 1836:
   // one-paper coined name found only in this paper.   // one-paper coined name found only in this paper.
   const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))));   const homegrown = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner' || (v.via === 'hand' && fam.get(v.family).size === 1 && [...p.tools].some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))));
-  say(`ACTIVE papers whose scanner is home-grown (generic 'custom' string, or a one-off name the paper itself produced): ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`);+  const customOnly = ACTIVE.filter((p) => memberOf(p).some((v) => v.family === 'custom scanner')); 
 +  const oneoffProduced = ACTIVE.filter((p) => memberOf(p).some((v) => v.via === 'hand' && fam.get(v.family).size === 1 && p.tools.some((tool) => tool.usedOrMentioned === 'produced' && String(tool.name).trim() === v.family))); 
 +  say(`ACTIVE papers whose scanner is home-grown: ${homegrown.length} (${pct(homegrown.length, ACTIVE.length)})`); 
 +  say(`  = generic 'custom scanner' string:                    ${customOnly.length}`); 
 +  say(`  + a one-paper name the paper itself PRODUCED:         ${oneoffProduced.length}`)
 +  say(`  - overlap                                           ${customOnly.length + oneoffProduced.length - homegrown.length}`); 
 +  say(`  (the other ${ones.length - oneoffProduced.length} one-paper families are third-party tools only this corpus's one paper used)`);
   say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`);   say(`PUBLISHED_HOMEGROWN ${homegrown.length} ${pct(homegrown.length, ACTIVE.length)}`);
 +  say(`PUBLISHED_HOMEGROWN_CUSTOM ${customOnly.length}`);
 +  say(`PUBLISHED_HOMEGROWN_PRODUCED ${oneoffProduced.length}`);
 +  say(`PUBLISHED_ONEOFF_THIRDPARTY ${ones.length - oneoffProduced.length}`);
   if (LIST === 'homegrown') for (const p of homegrown) say(`  ${key(p)}`);   if (LIST === 'homegrown') for (const p of homegrown) say(`  ${key(p)}`);
 } }
Line 2034: Line 2065:
   say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`);   say(`PUBLISHED_TARGETGEN ${targetGen.length} ${pct(targetGen.length, SCAN.length)}`);
   say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`);   say(`PUBLISHED_ALIASED ${aliased.length} ${pct(aliased.length, SCAN.length)}`);
 +  const v6subset = new Set([...hitlistV6, ...v6instr].map(key));
 +  const aliasIn = aliased.filter((p) => v6subset.has(key(p)));
 +  say('');
 +  say(`IPv6-SCANNING SUBSET (hitlist-near-IPv6 UNION names an IPv6-targeting instrument): ${v6subset.size} papers`);
 +  say(`  of those, discuss aliased prefixes / dealiasing: ${aliasIn.length} (${pct(aliasIn.length, v6subset.size)})`);
 +  say(`  the same 10 papers against all of SCAN would be ${pct(aliased.length, SCAN.length)} — the wrong denominator, most of SCAN never touches IPv6`);
 +  say(`PUBLISHED_V6_SUBSET ${v6subset.size}`);
 +  say(`PUBLISHED_ALIAS_IN_SUBSET ${aliasIn.length} ${pct(aliasIn.length, v6subset.size)}`);
   say('');   say('');
   say('IPv6-targeting instruments named, corpus-wide:');   say('IPv6-targeting instruments named, corpus-wide:');
provenance/programming/internet_scanning.1789075719.txt.gz · Last modified: by karel.kubicek.claude