You are about to measure something without recruiting humans. That is one of two top-level design branches on this site — the other is User studies — and it is not one method. It is three instruments that see different things, fail in different ways, and already have specialised pages. This page exists to pick the instrument. It does not replace those pages.

Everything below about “the literature” is a claim about seven venues — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026, 5,859 papers with extracted full text. The extractor field this page is built on, studyTypes, agrees with itself run-to-run on 57% of papers in a 100-paper sample that has not been re-measured on the 5,859-paper run.¹⁾ Treat every studyTypes share on this page as a ranking, not as a precise percentage. Venue counts, year buckets and reporting rates are paper counts over other fields; they still have the usual extraction error, but they are not the 57% field. See Corpus.

Of 5,859 papers, 2,198 (37.5%) are tagged with at least one of the three automated-measurement types. Almost all of those papers are only one type. The overlap is small and it is a design, not a rounding error:

The three branches are close in size — 967 crawl, 930 scan, 529 app — and they are not a partition of empirical work. 2,615 papers analyse an existing dataset; 1,149 are tagged user-study. 234 of the 2,198 (10.6%) are also tagged user-study: automated measurement and a user study are not exclusive, they are two instruments that co-occur when a paper both crawls and then asks people about what it found. Crowdworkers who only label data are annotators, not participants — see User studies when that page exists.

Two traps sit under those headlines.

1. The site's “crawled” population is 1,120, not 967. Crawler and the other crawl pages count every paper with a crawlConfig or the automated-web-crawl tag. All 967 tagged crawls are inside that 1,120. The extra 153 have a crawl configuration and were tagged as something else — 125 of them as a system-or-defence-proposal, 44 as mobile-app-analysis, 30 as a scan. They ran a crawler; the extractor did not treat the crawl as the study type. When this page says “crawl papers” it means the 967. When a child page says “crawling papers” it means the 1,120. Both numbers are real; they answer different questions.
2. Almost half of “scans” also analyse an existing dataset (429 of 930, 46.1%). Naming Censys or Shodan is often not running ZMap. The specialised TLS page already has to separate “we scanned” from “we queried a scan search engine”. Do that split before you count.

A crawl drives a browser (or an HTTP client) at URLs and records what the site does in response. It is the instrument for cookies, scripts, banners, third-party requests, and anything else that only exists after a page is rendered. It is a bad instrument for a question about hosts, ports, or binaries.

Read the child pages, do not re-derive them here. Crawler is the library comparison (browser vs control channel). Stateful stateless is the profile. Interaction is what you do on the page. Crawling location is where from. Consent is the banner. Website selection is which URLs. Longitudinal is how to do it twice.

What this page adds is the routing evidence:

1. Of the 967, 640 (66.2%) name a crawler-framework or browser-automation tool they used or produced. Selenium is the most-named family (214, 22.1%), then Puppeteer (71), OpenWPM (57), Playwright (31), PhantomJS and other legacy scriptable browsers (27). Those are papers, not a recommendation.
2. Currency, dated 2026-08-27. Playwright is the library that arrived in this corpus in 2022–2024 (15 of 294 crawl papers, 5.1%) and is 16 of 169 (9.5%) in the provisional 2025–2026* window. Selenium is still the most-named family in that same window (44 of 169, 26.0%). PhantomJS is gone: 0 of 294 in 2022–2024. npm playwright is 1.62.1 today. “Use Playwright” is current advice for a new crawl; “the literature used Selenium” is a statement about the literature. They are not the same sentence. The comparison itself is Crawler.
3. A crawl is not a user. Zeber et al. [1Zeber, David; Bird, Sarah; Oliveira, Camila; Rudametkin, Walter; Segall, Ilana; Wolls´en, Fredrik; Lopatka, Martin (2020): "The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing", in: Proceedings of The Web Conference 2020, pp. 167–178. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] compared crawls to Mozilla telemetry and found that for a majority of the top domains they inspected, “the number of third parties reported by the crawler is well into the right tail” of the human distribution. Ahmad et al. [2Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)] drove several crawlers at the same task and report “a variation of over 16% in the number of successful page loads”. If your claim is about people, a crawl is a surrogate with a known direction of bias; if your claim is about what a site serves a browser, pick the browser on purpose and report it.

A scan probes addresses, ports, or names. It is the instrument for “what does this host speak”, “which certificates are live”, “who is scanning the Internet”. It does not render a page. Mixing a crawl and a scan in one paper is a legitimate design (97 papers here do crawl-and-scan without also doing apps); mixing them in one count is a mistake.

Current instruments, dated 2026-08-27.

• ZMap is still an actively maintained IPv4 scanner. Durumeric, Wustrow and Halderman introduced it at USENIX Security 2013 [3Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)] — that paper is not in this extraction (it is missing from both the USENIX 2013 index and data/fulltext/2013/USENIX/), which is why the paper to read in-corpus is the ten-year retrospective [4Durumeric, Zakir; Adrian, David; Stephens, Phillip; Wustrow, Eric; Halderman, J. Alex (2024): "Ten Years of ZMap", in: Proceedings of the ACM Internet Measurement Conference. (DOI)]: “more than 300 research papers have used ZMap”, and “over 33% of all Internet-wide IPv4 scan traffic can be fingerprinted as coming from ZMap”. The 300 is that paper's count of the literature, not a count in these 5,859. In this corpus, 93 of 930 scan papers name ZMap as a used or produced network-scanner. The GitHub repo zmap/zmap is not archived and was pushed on 2026-08-26; zmap/zgrab2 was pushed on 2026-07-27. “The” scanner would overclaim what a repo date can prove; the 33% traffic figure is the usage evidence, and it is one paper's measurement of scan traffic, not of this corpus.
• Censys is often a dataset, not a scan you ran. Durumeric et al. [5Durumeric, Zakir; Adrian, David; Mirian, Ariana; Bailey, Michael D.; Halderman, J. Alex (2015): "A Search Engine Backed by Internet-Wide Scanning", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] introduced it as “a public search engine and data processing facility backed by data collected from ongoing Internet-wide scans”. 72 papers in the corpus name Censys as a used or produced tool; 63 of those are in the scan branch; 43 of those 63 are also tagged existing-dataset-analysis. That tag does not prove they only queried Censys — a paper can scan and re-analyse — but it is why you must not count the name as “they ran ZMap”. Of scan papers, Censys is named by about 10% in 2018–2024 (26 of 255, then 24 of 245). The 2025–2026* cell is 5 of 168 (3.0%) and is too thin to read as a decline. censys.com served HTTP 200 on 2026-08-27; search.censys.io returns 403 to automated clients, including with a browser User-Agent — a bot wall, not an outage.
• nmap, traceroute, RIPE Atlas are the other named families (40, 31, 25 papers in the scan branch). They answer different questions than ZMap does (host discovery vs path vs volunteer vantage points). 167 scan papers name a network-scanner the fold did not map — one-off research scanners, listed on the provenance page. That residue is the scanning analogue of the home-grown crawler row on Crawler.

Scanning ethics is not this page. Durumeric, Bailey and Halderman [6Durumeric, Zakir; Bailey, Michael; Halderman, J. Alex (2014): "An Internet-Wide View of Internet-Wide Scanning", in: Proceedings of the USENIX Security Symposium. (Link)] measured who was already scanning the Internet; the operational checklist (identify yourself, publish an opt-out, rate-limit) lives on Ethics, and telling the operator lives on Notifying websites. A scan from cloud address space is also a vantage-point decision.

Certificate and TLS measurement is TLS certificates. Turning the addresses you collected into a claim is IP classification.

An app is not a URL. Getting the binary, driving it, and knowing whether you saw its traffic are the three problems Mobile and app measurement is for. This page only records that the branch exists and how large it is: 529 papers, 402 of them only this type, 99.2% on the mobile platform. PETS (12.9% of venue output) and NDSS (11.7%) are where this work is densest as a share; USENIX Security (143 papers) and CCS (106) are where it is largest as a count. IMC is scan-heavy; app analysis is less common there (32 papers, 5.0% of venue output) but it is not absent.

Certificate pinning, Frida, AndroZoo, and the coverage number you owe the reader are on that page. Start with Pradeep et al. [7Pradeep, Amogh; Paracha, Muhammad Talha; Bhowmick, Protick; Davanian, Ali; Razaghpanah, Abbas; Chung, Taejoong; Lindorfer, Martina; Vallina-Rodriguez, Narseo; Levin, Dave; Choffnes, David (2022): "A Comparative Analysis of Certificate Pinning in Android & iOS", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] if you are about to intercept app traffic.

• A user study. User studies is the other top-level Design branch (red link until written). 1,149 papers are tagged user-study. Do not put Mechanical Turk in a crawlConfig.
• An existing-dataset analysis by itself. Re-analysing Censys, a CT log, AndroZoo, or last year's crawl is empirical work and it is most of the corpus (2,615 papers). It is not running a measurement. The design questions that remain — which snapshot, which query, which join key — belong on the child page for that artefact (Tranco, Archives, TLS certificates, Mobile and app measurement).
• A manual audit, a code analysis, or a system paper. Those are the three larger studyTypes (1,826 / 1,484 / 3,967). A system paper that also crawled is in the 153-paper gap above.
• A tutorial on HTTP, DNS, or TLS. Those are specs. This page is which instrument the field uses to observe them.

The same four columns, asked of each branch. Location, tool version and an artifact URL are comparable. robots.txt is a crawl/scan-ethics question — the app row is 3 of 529 (0.6%) because the question barely applies, not because app papers forgot a robots file. The methods-section checklist after the table names fields this table does not measure; those live on the child pages.

A crawl methods section that does not name the browser, the library and its version, the list pin, the vantage point, and the consent action is incomplete — those pages exist because the field does not report them. A scan methods section that does not name the scanner and version, the port set, the source addresses, and the opt-out story is incomplete for the same reason. An app methods section that does not report the share of the app set whose traffic you actually decrypted is incomplete — that sentence is the one Mobile and app measurement is built on.

Englehardt and Narayanan [8Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] is the canonical million-site crawl (“based on a crawl of the top 1 million websites”) and lives on OpenWPM. Jueckstock et al. [9Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)] is the vantage-point experiment and lives on Crawling location. Demir et al. [10Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)] is the repeat-run noise floor and lives on Longitudinal. Durumeric, Bailey and Halderman [6Durumeric, Zakir; Bailey, Michael; Halderman, J. Alex (2014): "An Internet-Wide View of Internet-Wide Scanning", in: Proceedings of the USENIX Security Symposium. (Link)] is who was already scanning you, and lives next to Ethics.

Figures below are from scripts/report_automated_measurements.mjs over the 5,859-paper extraction. studyTypes is multi-valued: a paper can be in more than one row, so the type table does not sum to 5,859. 2025–2026 venue-years are provisional — CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are under-selected by construction. Do not read the last year-bucket as a completed year.

14 papers have an empty studyTypes. They are listed on the provenance page; they are not forced into a branch.

Share of that venue's own output, not share of the branch. IMC is the scanning venue. TheWebConf is the crawling venue. IMC is not the crawling venue — that claim was false last time someone made it from memory, and the numbers here are why.

Crawl and scan shares are below their 2014–2017 / 2010–2013 peaks; app analysis is below its 2014–2017 peak (14.7%) but above its earliest share (5.9% → 7.6%). Absolute counts are up because the venues are bigger. The 2025–2026* column is starred.

138 of 5,859 papers are posters (poster- slug or Poster: title); 251 records are 4 pages or fewer. Scan papers are the most poster-heavy of the three (34 of 930, 3.7%). Dropping posters does not move the ranking of the three branches. Posters omit methods more often, which raises silence rates and lowers the reporting rates in the table above — those rates may understate how often full papers state the same things.

Every number above is a count of papers, from the 5,859-paper extraction, with the denominator in the same sentence or table header. Sentinels (not-stated, none-mentioned) are never answers. Free-text tool names are folded before counting; the scanner fold and its 205 unmapped strings / 167 residue papers are on automated_measurements. studyTypes is the least stable field in the schema (57% on a 100-paper sample, not re-measured on this run) — rankings only. 2025–2026 are provisional; see corpus. The report script is scripts/report_automated_measurements.mjs. It exits 1 if the corpus size, the three OVERVIEW.md type counts, the crawled population, or the Venn arithmetic disagree with the contracts it encodes.

• User studies — the other top-level Design branch (not yet written).
• Crawler — which library, which browser, which version.
• Crawling location — the address you crawl from.
• Website selection / Sampling — which URLs.
• Longitudinal — doing it twice.
• Mobile and app measurement — the app branch in full.
• TLS certificates — the scan branch when the question is the web PKI.
• IP classification — the addresses a crawl or a scan left you with.
• Ethics / Notifying websites — scanning and crawling other people's machines.
• Corpus — venue scope, funnel, provisional years.

[1]

Zeber, David; Bird, Sarah; Oliveira, Camila; Rudametkin, Walter; Segall, Ilana; Wolls´en, Fredrik; Lopatka, Martin (2020): "The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing", in: Proceedings of The Web Conference 2020, pp. 167–178. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[2]

Ahmad, Syed Suleman; Dar, Muhammad Daniyal; Zaffar, Muhammad Fareed; Vallina-Rodriguez, Narseo; Nithyanand, Rishab (2020): "Apophanies or Epiphanies? How Crawlers Impact Our Understanding of the Web", in: Proceedings of The Web Conference, pp. 271-280. (DOI)

[3]

Durumeric, Zakir; Wustrow, Eric; Halderman, J. Alex (2013): "ZMap: Fast Internet-wide Scanning and Its Security Applications", in: Proceedings of the USENIX Security Symposium. (Link)

[4]

Durumeric, Zakir; Adrian, David; Stephens, Phillip; Wustrow, Eric; Halderman, J. Alex (2024): "Ten Years of ZMap", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[5]

Durumeric, Zakir; Adrian, David; Mirian, Ariana; Bailey, Michael D.; Halderman, J. Alex (2015): "A Search Engine Backed by Internet-Wide Scanning", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[6]

Durumeric, Zakir; Bailey, Michael; Halderman, J. Alex (2014): "An Internet-Wide View of Internet-Wide Scanning", in: Proceedings of the USENIX Security Symposium. (Link)

[7]

Pradeep, Amogh; Paracha, Muhammad Talha; Bhowmick, Protick; Davanian, Ali; Razaghpanah, Abbas; Chung, Taejoong; Lindorfer, Martina; Vallina-Rodriguez, Narseo; Levin, Dave; Choffnes, David (2022): "A Comparative Analysis of Certificate Pinning in Android & iOS", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[8]

Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[9]

Jueckstock, Jordan; Sarker, Shaown; Snyder, Peter; Beggs, Aidan; Papadopoulos, Panagiotis; Varvello, Matteo; Livshits, Benjamin; Kapravelos, Alexandros (2021): "Towards Realistic and Reproducible Web Crawl Measurements", in: Proceedings of the ACM Web Conference. (DOI)

[10]

Demir, Nurullah; Große-Kampmann, Matteo; Urban, Tobias; Wressnegger, Christian; Holz, Thorsten; Pohlmann, Norbert (2022): "Reproducibility and Replicability of Web Measurement Studies", in: Proceedings of the ACM Web Conference. (DOI)