| Next revision | Previous revision |
| design:automated_measurements [2026/08/27 14:18] – New page: orient between crawling, scanning, and app analysis — three instruments, not three topics. Corpus-backed (967/930/529 of 5,859). Authored by Claude karel.kubicek.claude | design:automated_measurements [2026/09/11 10:50] (current) – Route differential/sock-puppet audits to the new Design:Algorithm audits page; note why its population is not in this page's studyTypes counts. Authored by Claude karel.kubicek.claude |
|---|
| ^ If you need to observe… ^ The instrument is ^ Then read ^ | ^ If you need to observe… ^ The instrument is ^ Then read ^ |
| | a rendered page, cookies, JavaScript, banners, third-party requests | a **crawl** | [[Programming:Crawler]], [[Programming:Stateful stateless]], [[Programming:Interaction]], [[Design:Crawling location]], [[Privacy:Consent]] | | | a rendered page, cookies, JavaScript, banners, third-party requests | a **crawl** | [[Programming:Crawler]], [[Programming:Stateful stateless]], [[Programming:Interaction]], [[Design:Crawling location]], [[Privacy:Consent]] | |
| | open ports, certificates, protocol banners, DNS, IPv4/IPv6 hosts | a **scan** (or a search engine over someone else's scans) | [[Security:TLS certificates]], [[Design:IP classification]], [[Practices:Ethics]] (scanning checklist), [[Practices:Notifying websites]] | | | open ports, certificates, protocol banners, IPv4/IPv6 hosts | a **scan** (or a search engine over someone else's scans) | [[Programming:Internet scanning]], [[Security:TLS certificates]], [[Design:IP classification]], [[Practices:Ethics]] (scanning checklist), [[Practices:Notifying websites]] | |
| | | which resolver answered a name, whether the answer was manipulated, encrypted DNS | a **DNS measurement**, which is a scan with its own failure modes | [[Design:DNS]] | |
| | an APK or IPA, a store listing, an app's runtime | **app analysis** | [[Design:Mobile and app measurement]] | | | an APK or IPA, a store listing, an app's runtime | **app analysis** | [[Design:Mobile and app measurement]] | |
| | | whether a platform treats **two constructed identities** differently — personalised results, targeted ads, a feed, a quoted price | a **differential audit**, which is a crawl whose population is the arms rather than the sites | [[Design:Algorithm audits]], [[Programming:Stateful stateless]], [[Statistics:Hypothesis testing]] | |
| |
| <WRAP important> | <WRAP important> |
| * **Censys is often a dataset, not a scan you ran.** Durumeric et al. {[durumeric2015_search]} introduced it as //"a public search engine and data processing facility backed by data collected from ongoing Internet-wide scans"//. **72** papers in the corpus name Censys as a used or produced tool; **63** of those are in the scan branch; **43** of those 63 are also tagged ''existing-dataset-analysis''. That tag does not prove they only queried Censys — a paper can scan and re-analyse — but it is why you must not count the name as "they ran ZMap". Of scan papers, Censys is named by about 10% in 2018–2024 (26 of 255, then 24 of 245). The 2025–2026* cell is 5 of 168 (3.0%) and is too thin to read as a decline. ''censys.com'' served HTTP 200 on 2026-08-27; ''search.censys.io'' returns **403** to automated clients, including with a browser User-Agent — a bot wall, not an outage. | * **Censys is often a dataset, not a scan you ran.** Durumeric et al. {[durumeric2015_search]} introduced it as //"a public search engine and data processing facility backed by data collected from ongoing Internet-wide scans"//. **72** papers in the corpus name Censys as a used or produced tool; **63** of those are in the scan branch; **43** of those 63 are also tagged ''existing-dataset-analysis''. That tag does not prove they only queried Censys — a paper can scan and re-analyse — but it is why you must not count the name as "they ran ZMap". Of scan papers, Censys is named by about 10% in 2018–2024 (26 of 255, then 24 of 245). The 2025–2026* cell is 5 of 168 (3.0%) and is too thin to read as a decline. ''censys.com'' served HTTP 200 on 2026-08-27; ''search.censys.io'' returns **403** to automated clients, including with a browser User-Agent — a bot wall, not an outage. |
| * **nmap, traceroute, RIPE Atlas** are the other named families (40, 31, 25 papers in the scan branch). They answer different questions than ZMap does (host discovery vs path vs volunteer vantage points). **167** scan papers name a ''network-scanner'' the fold did not map — one-off research scanners, listed on the provenance page. That residue is the scanning analogue of the home-grown crawler row on [[Programming:Crawler]]. | * **nmap, traceroute, RIPE Atlas** are the other named families (40, 31, 25 papers in the scan branch). They answer different questions than ZMap does (host discovery vs path vs volunteer vantage points). **167** scan papers name a ''network-scanner'' the fold did not map — one-off research scanners, listed on the provenance page. That residue is the scanning analogue of the home-grown crawler row on [[Programming:Crawler]]. |
| | |
| | **The instrument page for this branch is [[Programming:Internet scanning]]**, added on 2026-09-10 — the ZMap/ZGrab invocation, the probe rate and its wall-clock cost, the exclusion file, source-address hygiene, and the IPv6 hitlist problem. It defines its own population — the papers that name an active-scan instrument, a smaller and differently derived set — rather than reusing the 930, and its scanner fold is a separate, stricter one; where the two pages disagree on a count, that page carries the number and says why. |
| |
| Scanning ethics is not this page. Durumeric, Bailey and Halderman {[durumeric2014_view]} measured who was already scanning the Internet; the operational checklist (identify yourself, publish an opt-out, rate-limit) lives on [[Practices:Ethics]], and telling the operator lives on [[Practices:Notifying websites]]. A scan from cloud address space is also a [[Design:Crawling location|vantage-point]] decision. | Scanning ethics is not this page. Durumeric, Bailey and Halderman {[durumeric2014_view]} measured who was already scanning the Internet; the operational checklist (identify yourself, publish an opt-out, rate-limit) lives on [[Practices:Ethics]], and telling the operator lives on [[Practices:Notifying websites]]. A scan from cloud address space is also a [[Design:Crawling location|vantage-point]] decision. |
| |
| Certificate and TLS measurement is [[Security:TLS certificates]]. Turning the addresses you collected into a claim is [[Design:IP classification]]. | Certificate and TLS measurement is [[Security:TLS certificates]]. Turning the addresses you collected into a claim is [[Design:IP classification]]. **DNS is its own page.** It sits in the scan branch — **186** of the 244 papers in [[Design:DNS|that page's population]] are tagged ''network-scan-or-probe'' — but it has failure modes no port scan has (the answer depends on which resolver you asked and on your source port, and a cache you cannot see sits in the middle), so this page routes it to [[Design:DNS]] rather than to the TLS and IP pages, which answer different questions. |
| |
| ===== App analysis ===== | ===== App analysis ===== |
| |
| * **A user study.** [[Design:User studies]] is the other top-level Design branch (red link until written). 1,149 papers are tagged ''user-study''. Do not put Mechanical Turk in a crawlConfig. | * **A user study.** [[Design:User studies]] is the other top-level Design branch (red link until written). 1,149 papers are tagged ''user-study''. Do not put Mechanical Turk in a crawlConfig. |
| * **An existing-dataset analysis by itself.** Re-analysing Censys, a CT log, AndroZoo, or last year's crawl is empirical work and it is most of the corpus (2,615 papers). It is not running a measurement. The design questions that remain — which snapshot, which query, which join key — belong on the child page for that artefact ([[Programming:Tranco]], [[Design:Archives]], [[Security:TLS certificates]], [[Design:Mobile and app measurement]]). | * **An existing-dataset analysis by itself.** Re-analysing Censys, a CT log, AndroZoo, or last year's crawl is empirical work and it is most of the corpus (2,615 papers). It is not running a measurement. The design questions that remain — which snapshot, which query, which join key — are [[Design:Existing datasets]], and the artefact-specific advice is on the child page for that artefact ([[Programming:Tranco]], [[Design:Archives]], [[Security:TLS certificates]], [[Design:Mobile and app measurement]]). |
| * **A manual audit, a code analysis, or a system paper.** Those are the three larger ''studyTypes'' (1,826 / 1,484 / 3,967). A system paper that also crawled is in the 153-paper gap above. | * **A manual audit, a code analysis, or a system paper.** Those are the three larger ''studyTypes'' (1,826 / 1,484 / 3,967). A system paper that also crawled is in the 153-paper gap above. |
| * **A tutorial on HTTP, DNS, or TLS.** Those are specs. This page is which instrument the field uses to observe them. | * **A tutorial on HTTP, DNS, or TLS.** Those are specs. This page is which instrument the field uses to observe them. |
| | * **A differential audit.** Driving two deliberately different profiles at one platform and comparing what comes back is a crawl by machinery and an experiment by design; the arms, not the site list, are the population. [[Design:Algorithm audits]] owns it, and its 32-paper population is derived by hand rather than from ''studyTypes'', so it does not appear in any count on this page. |
| |
| ===== What to report ===== | ===== What to report ===== |
| * [[Design:Longitudinal]] — doing it twice. | * [[Design:Longitudinal]] — doing it twice. |
| * [[Design:Mobile and app measurement]] — the app branch in full. | * [[Design:Mobile and app measurement]] — the app branch in full. |
| | * [[Programming:Internet scanning]] — the scan branch's instrument page: invocation, rate, exclusion file, IPv6. |
| * [[Security:TLS certificates]] — the scan branch when the question is the web PKI. | * [[Security:TLS certificates]] — the scan branch when the question is the web PKI. |
| | * [[Design:DNS]] — the scan branch when the question is names, not certificates. |
| * [[Design:IP classification]] — the addresses a crawl or a scan left you with. | * [[Design:IP classification]] — the addresses a crawl or a scan left you with. |
| * [[Practices:Ethics]] / [[Practices:Notifying websites]] — scanning and crawling other people's machines. | * [[Practices:Ethics]] / [[Practices:Notifying websites]] — scanning and crawling other people's machines. |
