Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Sampling ======

There is no register of the web. No authority enumerates websites, so no study can draw a random sample of them — every measurement substitutes some **list** for the population it wants to talk about, and inherits whatever that list covers, over-covers and misses. Sampling on the web is therefore not the textbook problem of drawing from a frame; it is the problem of choosing a frame that does not exist, and then being honest about the gap.

This page is about **how you draw**: which method, at what size, at what unit, and what you have to record for the draw to be repeatable. [[Design:Website selection]] is about **which list** — the ranking services, their provenance and their biases — and the two pages are meant to be read together. Where a claim here is about a particular list, it links there rather than restating it.

The short version, from 1,153 papers in our corpus of seven security and privacy venues that drew a population of websites, domains or web pages (see [[#Use in Publications]]):

  * **57.6% take a top-//n//.** It has not been superseded, it has consolidated — the share has sat between 58% and 60% since 2018, on the year-buckets that are complete. Only **6.0%** stratify at all, by rank or by anything else.
  * **99.1% name the list, 97.2% give a size, and 51.7% say which version** — so for half of them a reader cannot tell which draw was made. About a third of the rest publish the drawn list instead, which closes the gap; the remainder is genuinely unrecoverable.
  * The list itself changed under the field: papers naming Alexa fell from **47.1% of the population in 2021 to 8.3% in 2025**, while Tranco rose from 31.0% to 56.4% over the same years. **66 papers published in 2023–2026 still sample from "the Alexa top //n//"**, a list that has not been published since 2022, and **51.5% of them give no version at all**.

<WRAP important>
**The one thing to take away.** A top-//n// sample is not a sample of the web. It is a **census of the head** of one list. Every statistic you compute on it is a statistic about that head, with no sampling error inside it and unquantified coverage error outside it. That is often a perfectly good study — but the population you can generalise to is "the top //n// of list //L// on date //D//", and that is what the paper has to say.
</WRAP>

===== What You Are Actually Sampling =====

==== Four populations, and the gaps between them ====

Almost every methodological argument about a web measurement is really an argument about one of three gaps:

^ Layer ^ What it is ^ The gap below it ^
| **Target population** | What the paper's claim is about: "the web", "European news sites", "sites using a CMP" | //Coverage error.// No list enumerates this. |
| **Sampling frame** | The list you actually had: Tranco 1M of 2026-08-12, the ''.nl'' zone file, a CrUX country slice | //Sampling error// — but only if you sampled at random from it. |
| **Drawn sample** | The rows you selected from the frame | //Attrition.// Not everything you draw can be measured. |
| **Analysed sample** | What survived loading, parsing, and every filter in your pipeline | This is the denominator of your results. |

The first gap is the one nobody can close and the one to be explicit about. The last gap is the one that is routinely closed silently, and it is where percentages go wrong.

==== Attrition is a denominator, and it is usually invisible ====

You draw 10,000 sites; some do not resolve, some return 5xx, some serve a bot wall, some time out, some are parked. What you report a percentage //of// is now a choice, and readers cannot check it unless you say which one you made.

The reference behaviour is Englehardt and Narayanan's million-site measurement {[englehardt2016online]}, which states the attrition and the rule in one sentence:

> In presenting our results we only consider sites that loaded successfully. For example, for the 1 Million site measurement, we present statistics for 917,261 sites.

That is **8.3% of the drawn sample gone before analysis**, and the paper's percentages mean something precise because it says so. A 2025 example at the other scale: {[annamalai2025_beyond]}((Cited here for its reporting of attrition; see the paper for its actual subject.)) reports that "the automated crawler failed to visit 15 out of the 3K websites (≈ 1%)", and attributes most of them to bot detection — hedged as "we believe", on the evidence that 86.7% of those sites returned 4XX errors. Even at 1%, that attrition is itself a finding, because it is not random with respect to what is being measured.

Attrition is not random. Sites that block datacenter IPs, sites behind bot management, and sites in the tail are all more likely to drop out, and all three plausibly correlate with the things privacy and security papers measure.((Plausibly, not measurably: we found no study that characterises which sites drop out of a crawl and how that biases the result. Jueckstock et al. {[jueckstock2021_realistic]} show that the vantage point changes what a crawl sees, and {[annamalai2025_beyond]} attributes its own failures to bot detection, but neither measures the composition of the lost set. It is an open question — see [[#Open Questions]].)) Report both denominators.

===== The Methods, and What Each Licenses =====

The extraction behind the figures on this page assigns one of nine sampling methods (plus a ''not-stated'' sentinel), and they are not interchangeable. What matters is not the label but **what population the result generalises to**:

^ Method ^ What it does ^ Generalises to ^ Papers ^
| **top-//n//** | Take the first //n// rows of a ranking | The top //n// of that list, on that date. Nothing below rank //n//. | 664 (57.6%) |
| **stratified** | Draw from each rank band (or country, category, TLD) | Each stratum separately; the whole frame with weights | 69 (6.0%) |
| **random** | Uniform draw from a frame | The frame, with a real confidence interval | 190 (16.5%) |
| **exhaustive** | Measure the whole frame | The frame, exactly. No sampling error at all | 165 (14.3%) |
| **seed-and-crawl** | Follow links out from seeds | Whatever the link graph reaches from those seeds | 100 (8.7%) |
| **purposive** | Pick for a property ("sites with a cookie banner") | Sites with that property — and only if the selection rule is reproducible | 319 (27.7%) |
| **convenience** | Whatever was to hand | Itself | 29 (2.5%) |
| **pre-existing-dataset** | Reuse someone else's sample | Whatever //they// sampled, as of //their// date | 278 (24.1%) |

//Counts are papers with at least one web population drawn that way, of 1,153; a paper can appear in several rows. The ninth method, snowball, fires on 2 papers and is omitted.//

==== Top-n is a census of the head ====

The dominant design in the field is "we crawled the Tranco top 10k". It has real advantages — cheap, comparable across papers, weighted towards sites that matter to users — and one property that is constantly forgotten: **it involves no sampling at all**. There is no randomness, so a confidence interval computed on it is answering a question nobody asked. If 6.4% of your top-10k sites do //X//, that 6.4% is an exact census figure for those 10,000 sites, and the uncertainty that matters is entirely about how unlike the rest of the web those 10,000 sites are.

They are, measurably, very unlike it. Scheitle et al. {[scheitle2018_long]} compared list domains against the general ''com''/''net''/''org'' population and found the head is a technological outlier. Their 2018 figures: HTTP/2 adoption **26.6% for the Alexa Top 1M against 7.84% for com/net/org overall**, IPv6 enablement 11–13% against about 4%, and CDN use on the Top 1M lists exceeding the general population "by at least a factor of two". The absolute numbers are eight years old and every one of them has moved since; the //gap// is the durable finding. Any measurement of adoption, modernity or professionalism run on a top list is biased upward, and on these three indicators the bias was a factor of two to three — large enough to swamp the effect a typical paper reports, though we know of no study that compares the two systematically.

The head is also a small part of the web but most of the //browsing//. Ruth et al.'s study of Chrome telemetry {[ruth2022_world]} finds that "the top million sites capture over 95% of all page loads and time spent online, but they do so extremely unequally": six sites account for 25% of page loads, and one site alone for 17% of desktop page loads globally. The consequence they draw is the one that should change your design — a study "calculated directly from a simple set of the top million sites place[s] disproportionate focus on the long tail of the web", because treating a million ranks as a million equal observations weights the rank-900,000 site exactly like Google.

So a top-//n// sample and a uniform sample of a top-1M list are both wrong in a knowable direction, in opposite directions, and the fix is the same in both cases: say which stratum you are talking about.

==== Rank-stratified sampling is the standard fix, and it is rare ====

Only **69 of 1,153 papers (6.0%)** stratify. The designs that do are worth copying verbatim, because they make the head/tail trade-off explicit and cost nothing extra:

  * "We collect data for 21,965 sites in total, including 11,965 of the Tranco top 15K, 5,000 websites uniformly sampled between the ranks of 15K to 100K, and 5,000 websites between ranks 100K and 1M." {[ukani2025_local]}
  * "We selected 20K sites based on their ranking in the Tranco dataset: 2K, 3K, 5K, and 10K sites, respectively, from the rank ranges [1, 10K], [10K, 100K], [100K, 500K], and [500K, 1M]." {[zhu2025_toward]}
  * "For Switzerland, we selected all top 50k websites as well as 20k randomly sampled ones from the 100k-500k and 500k-1M CrUX popularity buckets, respectively." {[nenadic2026_swiss]} — the same design carried across two jurisdictions, with the list version stated (CrUX, August 2023).

Note what each does: an **exhaustive head** plus a **random draw per decade of rank**. That lets you report head figures as a census, tail figures with an interval, and the whole-frame figure as a weighted estimate, with the weights being the stratum sizes you already know. It also survives the reviewer question "would this hold outside the top 10k?" — which a top-//n// design cannot answer at all.

The script below implements exactly this against a live Tranco or CrUX list and writes the manifest a methods section needs; see [[#versioning_making_the_draw_redrawable|versioning]].

==== Random, exhaustive, and seed-and-crawl ====

**Random** from a frame (190 papers) is the only design where the classical machinery applies, and it applies //to the frame//, not to the web. It is the right choice when the frame is genuinely the target population — all ''.nl'' domains, all extensions in a store, all certificates in a CT log — and a poor one when the frame is a top list, because a uniform draw from a top-1M list is a sample of the tail with a rounding error of head sites.

**Exhaustive** (165 papers, median stated size 708,000) means the frame //is// the study: measure every registered domain under a TLD, every host in a zone file, every page in a crawl corpus. It is the strongest design available on the web because it removes sampling error entirely, and it is only available where somebody maintains a register — which is why it clusters on DNS, certificates and app stores rather than on websites.

**Seed-and-crawl** (100 papers) follows links out from a seed set. It is a snowball sample on the link graph, with the biases of one: well-linked pages are over-represented, islands are unreachable, and the sample depends on crawl order and depth. It is the only way to reach pages no list enumerates, and it should never be described as random.

==== Purposive samples, and the claim they do not license ====

Purposive sampling — picking sites because they have a property — is the field's **second most common design (319 papers, 27.7%)** and the one most often described in a way a reader cannot act on. Two things have to be on the page.

**The selection rule, precisely enough to re-run.** "Sites with a cookie banner", "European news sites", "e-commerce sites" are not rules; they are the output of one. What did the rule take as input — a categoriser, a filter list, a manual pass, a search? Over what frame? With what threshold? If a categoriser drew the boundary, **its error rate is now your sample's composition error**, and it needs reporting as such: at 90% precision, one site in ten of your "news sites" is not one, and any difference you measure between categories is attenuated toward zero by that mixing. See [[Design:Website classification]] for what those services actually achieve.

**And no prevalence claim about anything larger.** A purposive sample of 200 sites that use a given CMP supports "of the 200 sites we selected, //x//% did //y//" and supports nothing at all about how common //y// is on the web, or on the top 10k, or among CMP users generally — because the selection was not a draw. This is the same denominator discipline as everywhere else on this page, and it is easiest to lose here, because a purposive sample //feels// like a sample. In the corpus, purposive populations are small by nature (median 102, against 10,000 for top-//n//), which is a good reason to treat them as case studies and describe them as such.

==== Reusing someone else's sample ====

278 papers (24.1%) take a pre-existing dataset. This inherits two things people forget: **the original sampling design**, which becomes yours and must be described as such, and **the original date**. A 2026 paper that reuses a 2019 crawl of the Alexa top 10k is measuring the web of 2019 through a list that no longer exists. That can be exactly right — longitudinal and replication work depends on it — but it is a different claim from a fresh measurement, and the year has to be in the sentence.

===== How Big, and Why That Number =====

<WRAP tip>
**[[Statistics:How many sites]] is the page for this question in full** — what precision, rare-event capture and rank-tail effects each buy you, the arithmetic for each, the nineteen papers that report the same measurement at more than one rank depth, and the six in 1,121 that derive their //n// from anything. This section is the draw-side summary. Its figures are **tuple-level** (2,379 populations); that page's are **paper-level** (one number per paper, so a median of 20,000 rather than 6,755). They differ by construction, not because either is stale.
</WRAP>

Sample sizes in this literature are chosen by round number, not by design. Of 2,379 web populations with a stated size, **43.5% are 1, 2 or 5 times a power of ten**, and the single most common values are 1,000,000 (198 populations), 100 (156) and 10,000 (148). The median stated size is **6,755** — far smaller than the "million-site measurement" framing of the field suggests — and a quarter of populations are under 283.

That is not necessarily wrong. What size buys you depends on which question you are asking, and the two questions have opposite scaling:

  * **Prevalence of a common thing** ("what share of sites set a third-party cookie before consent?"). Precision improves with the square root of //n//, so the returns die quickly: going from 1,000 to 10,000 sites narrows a proportion's interval by about a factor of three, and 10,000 to 1,000,000 by another factor of ten at a hundred times the crawl cost. Beyond a few thousand sites, **coverage error typically dominates sampling error**((This is a judgement, not a measurement: it follows from the frame-bias magnitudes Scheitle et al. report above — tens of percentage points between list domains and the general population — being far larger than a binomial interval at //n// = 5,000, which is under ±1.4 points. We know of no paper that quantifies the two side by side; see [[#Open Questions]].)), and adding sites from the same list does nothing about coverage. Spend the budget on a second frame instead.
  * **Discovery of a rare thing** ("which sites run this specific fingerprinting script?"). Here //n// is the whole game, because your expected count is //n//×//p// and //p// may be 10^-4. This is what genuinely justifies a million-site crawl, and it is worth saying so explicitly when it is your reason — the reader cannot tell the two motivations apart from the number alone.

The corpus median size by method reflects this: top-//n// designs sit at 10,000, stratified at 4,084, random at 2,000, and exhaustive at 708,000. Purposive samples are small by nature (median 102): they are chosen, not drawn.

==== Weight by rank instead of buying more sites ====

There is a third option that the size argument usually misses, and the canonical example is in {[englehardt2016online]}. Rather than count how many of //n// sites a third party appears on — a figure that moves whenever you change //n//, because the sites you add are always less popular than the ones you had — they define **prominence**: sum 1/rank(//s//) over the sites //s// where the third party appears. That is a rank-weighted prevalence, and their motivation for it is a sampling one: a plain prevalence rank is sensitive to how many websites the measurement visited, and so "different studies may rank third parties differently" for no reason but their //n//. Prominence instead "de-emphasizes obscure sites, and hence can be adequately … approximated by relatively small-scale measurements".

The general form is the useful part. If your frame is a ranking, you already have a weight for every row, and a weighted estimate is both more comparable across studies and cheaper to compute at a given //n// than an unweighted one is to stabilise. The weights need a stated model — prominence assumes the power-law visit distribution that {[ruth2022_world]} later measured directly — and, like every weighting scheme, it has to be reported as one, because a weighted and an unweighted percentage over the same crawl are different numbers.

<WRAP important>
**Do not report a confidence interval on a top-//n// crawl without saying what it is an interval over.** Almost all of these designs are censuses of their frame. The number with no uncertainty in it is the census figure; the number with uncertainty in it is the extrapolation to the target population, and that uncertainty is not the binomial one. If you want an interval you can defend, stratify and report per stratum.
</WRAP>

===== Site, Page, or Visit =====

The unit is a sampling decision, and it is the one most often made by default. Of the 1,153 papers on this page, 723 sample **websites**, 393 **domains** and 251 **web pages**. Of the 680 that also crawled, just **32.4% visit more than one page per site**; and of the 492 whose unit is explicitly ''websites'' and which crawled, **24.8% visit only the landing page**.

==== A list row is not yet a URL ====

Before any of that, there is a step the lists do not do for you and most methods sections skip. **Tranco rows are registrable domains** (''example.com'' — no scheme, no host); **CrUX rows are origins** (''https://www.example.com''). Getting from a row to something a browser visits means deciding:

  * **Scheme.** ''http://'' or ''https://'' first, and what to do when only one answers. A crawl that tries HTTPS only measures a different population from one that falls back.
  * **''www'' or apex.** They can serve different content, different cookies and different consent configurations, and a domain-based list does not tell you which the site prefers.
  * **Where the redirect lands.** If ''example.com'' redirects to ''example.co.uk'', you have measured a different eTLD+1 from the one you sampled. Decide in advance whether that counts as the sampled unit or as attrition, and say which.
  * **Duplicates.** The same site under several rows — country domains, redirect chains, ''www'' and apex both listed — silently weights it more heavily than the design says. De-duplicating by eTLD+1 is the common fix and it changes your //n//, so report the number before and after.

None of these is exotic and all of them change the denominator. They are the first source of attrition, they happen before the crawler is ever blocked by anything, and a reader cannot reconstruct your population without them.

==== A landing page is not a sample of a site ====

The strongest result here is Aqeel et al. {[aqeel2020_landing]}, who characterised landing versus internal pages of 1,000 sites and then went back through the literature to see how much it mattered. Reviewing web-performance papers from five networking venues 2015–2019, they scored the 119 that used a top list: **41 (34.5%) needed no revision, 48 (40.3%) a minor revision, and 30 (25.2%) a major revision** for their claims to apply to internal pages. All 119 were claims about one page per website; for nearly two-thirds of them, that turned out to matter.

Their incidental figure is worth repeating too: of those 119 papers, **only 10 (8.4%) used a top list other than Alexa** — a monoculture that the corpus data below shows has since broken up, but that shaped a decade of results.

==== Sampling internal pages, and the tooling gap ====

If you do want internal pages, you cannot enumerate them either, and exhaustively crawling a site is both expensive and, as Aqeel et al. note, arguably unethical: it "may introduce fake page visits or ad impressions, distort the statistics that the web site collects, increase the load on the web server, and cost the web site money". Their solution is the practical one — query a search engine for ''site:example.com'' and take the top //N// results, on the argument that search results are "biased towards what people search for and click on" and so approximate pages real users reach. Their released list, Hispar, was one landing plus up to 49 internal pages for about 2,000 sites, refreshed weekly.

<WRAP todo>**Hispar is gone.** ''hispar.cs.duke.edu'' does not resolve as of 2026-08-12 (NXDOMAIN). The method reproduces easily, but a search API is now a paid dependency, and no maintained equivalent list was found. If you know of one, please add it.</WRAP>

Only **15.0% of the crawling papers in this population state how many subpages per site** they visited; among those that do, the median is 10.

==== A crawl is not a sample of what people load ====

Even a perfect sample of sites is not a sample of //visits//. Zeber et al. {[zeber2020representativeness]} ran a crawler and a panel of real users over the same sites and found the crawler sees a systematically busier web: a median of **11.6 third-party domains per site visit for the crawler against 4.5 for humans**, and **6.1 tracking domains against 1.9**. Users' browsers connected to at most 8 trackers in 99% of visits to list sites; "the crawler may reach 26".

Ahmad et al. {[ahmad2020_apophanies]} come at it from the other side, comparing crawlers to each other: over 16% variation in the fraction of pages that load successfully, 2–22% differences in website-fingerprinting classifier accuracy, and third-party prevalence //rankings// that change with the crawler. Demir et al. {[demir2023_similarity]} quantify the same instability within one crawler across configurations: comparing two profiles, "48% of the underlying data varies", and 42% of observed cookies appear in only one of five profiles.

None of this makes crawling invalid. It means the sample is a sample of //crawler visits to sites//, and a claim about user exposure needs either a panel, telemetry, or an explicit argument about the direction of the gap.

===== Versioning: Making the Draw Redrawable =====

A sample is reproducible when a reader can reconstruct the frame. That takes a list identity, not a list name — and this is the field's clearest single failure: **51.7% of papers state a version or access date**, and among the 764 papers drawing from a popularity ranking, only **48.0%** version the ranking.

The mechanics are cheap, and differ per source:

^ Frame ^ How to pin it ^
| **Tranco** | Every generated list has a permanent id and a citable page: ''https://tranco-list.eu/top-1m-id'' returns today's id (e.g. ''645KX'' on 2026-08-12), and ''/list/<id>'' is a permalink. Cite the id, not the date. |
| **CrUX** | Released monthly, typically the second Tuesday. Ranks are **magnitude buckets** (1000 / 10K / 100K / 1M), randomly ordered within a bucket — the file's line order is not a rank. Cite the month; the [[https://github.com/zakird/crux-top-lists|crux-top-lists]] mirror keeps every monthly snapshot. |
| **Cisco Umbrella, Majestic** | Daily files at a fixed URL with no version in the path. Archive the file yourself and publish its hash. |
| **Common Crawl, HTTP Archive** | Cite the crawl or dataset identifier (''CC-MAIN-YYYY-WW''), which is permanent. |
| **Zone files, CT logs** | Cite the snapshot date and the source; for CT, the log and tree size. |
| **Anything else** | Publish the drawn list as an artifact. |

The last row is the general escape hatch, and it is under-used: of the 557 papers in this population that version nothing, only **29.4% release a dataset of their own** that could stand in for the version.

Tranco's permanent id is the best case available, and the corpus shows it being used — by a minority. Of the **262 papers that name Tranco, 157 (59.9%) give a version or date and at least 31 (11.8%) cite the list id itself**((A lower bound: the count matches only labelled forms — ''ID: W9L9'', ''list 4NKX/1000000'', ''tranco-list.eu/list/64WX''. Papers that print a bare id next to a date are missed, and reading the ''listVersion'' strings by hand suggests roughly five more.)). What that looks like in a methods section, from papers in the corpus:

  * "We used the list generated on 03/26/2019 (ID: W9L9)."
  * "Tranco list ''https://tranco-list.eu/list/4NKX/1000000'', generated on 2019.09.20" — in a table alongside the Disconnect list commit, the crawler commit and the crawl date, which is the whole configuration pinned at once.
  * "the research-oriented Tranco list … as of November 5, 2019 (ID: GVWK)", then "the list from January 31, 2021 (ID: WQW9)", then "the Tranco list from December 22, 2022 (ID: 82V9V)" — a longitudinal study with one id per wave.

That last one is the pattern to copy for repeated measurements: without a per-wave id, a longitudinal comparison cannot separate a change in the web from a change in the list. That paper goes further and publishes a table of how the top domains moved between its three lists, which is the honest way to show the frame shifting under a trend.

<WRAP important>
**"The Alexa top //n//" is no longer a frame.** Amazon retired ''alexa.com'' on **1 May 2022** and the Alexa Top Sites and Web Information Service APIs on **15 December 2022**.((Alexa Support, "We retired Alexa.com on May 1, 2022", archived at [[https://web.archive.org/web/2023/https://support.alexa.com/hc/en-us/articles/4410503838999-We-retired-Alexa-com-on-May-1-2022|web.archive.org]]: "we made the difficult decision to retire Alexa.com on May 1, 2022 … The APIs will be retired on December 15, 2022.")) A 2025 paper that says it crawled "the Alexa top 10,000" with no date is describing a copy of unknown provenance — a mirror, a cached CSV, or a list inherited from an earlier paper. **66 papers published 2023–2026 in this corpus still draw from Alexa, and 34 of them (51.5%) give no version or date at all**; those 34 are the ones a reader cannot pin to any snapshot. The other 32 do date it, several explicitly as an archive ("March 2018 snapshot", "Alexa archived 2022-01-01"), which is a defensible longitudinal choice — say so and say which snapshot. Citing it as though it were live is not.

Note also that **Tranco is not what it was in 2019.** Its published list of 2026-08-01 was built from CrUX, Farsight, Majestic, Cloudflare Radar and Umbrella — Alexa is long gone from the inputs — over a 30-day window with the Dowdall rule.((Read from Tranco's own API for the daily list, ''https://tranco-list.eu/api/lists/date/2026-08-01'', on 2026-08-12: ''providers: [crux, farsight, majestic, radar, umbrella]'', ''combinationMethod: dowdall'', ''startDate: 2026-07-03'', ''endDate: 2026-08-01''. Tranco's prose methodology page still says "all four providers".)) Describing Tranco's composition from the 2019 paper is a small error that a reviewer who works on lists will catch.
</WRAP>

==== A sampler that records its own provenance ====

This draws a rank-stratified sample from a live Tranco or CrUX list and writes a manifest containing the list identity, the download URL, the SHA-256 of the frame bytes, the strata, the seed, and a ready-made sentence for your methods section. For CrUX it resolves the newest **dated** monthly snapshot (''202607'' when run on 2026-08-12) rather than ''current.csv.gz'', because an unversioned URL is the thing this page is arguing against. Tested against both sources on 2026-08-12.

<file python stratified_sample.py>
#!/usr/bin/env python3
"""Draw a rank-stratified sample of websites, and record enough to redraw it.

Two things every sampling section needs and half of them lack: the exact list
that was drawn from, and the rule that turned it into a sample. This script
emits both as a manifest next to the sample.

    # current Tranco list, 200 sites per rank stratum, seed 20260812
    python3 stratified_sample.py --source tranco --per-stratum 200 --seed 20260812

    # a specific, permanently addressable Tranco list
    python3 stratified_sample.py --source tranco --list-id V3JQN --per-stratum 200

    # CrUX, whose ranks are magnitude buckets rather than positions
    python3 stratified_sample.py --source crux --crux-month 202607 --per-stratum 200

Writes sample.csv (rank,name,stratum) and sample.manifest.json. Paste the
manifest's `cite` line into your methods section.

Note what the rows are, because it decides your unit: Tranco rows are
registrable domains (no scheme, no host), CrUX rows are origins (scheme + host).
Turning either into a URL a browser visits is a further sampling decision -- see
the page this script is published on.
"""
import argparse, csv, gzip, hashlib, io, json, random, sys, urllib.request, zipfile
from datetime import datetime, timezone

UA = "measuretheweb-stratified-sampler/1.0"
TRANCO_DAILY = "https://tranco-list.eu/top-1m-id"
TRANCO_DOWNLOAD = "https://tranco-list.eu/download/{list_id}/1000000"
TRANCO_PERMALINK = "https://tranco-list.eu/list/{list_id}"
CRUX_MONTHLY = "https://raw.githubusercontent.com/zakird/crux-top-lists/main/data/global/{month}.csv.gz"
CRUX_INDEX = "https://api.github.com/repos/zakird/crux-top-lists/contents/data/global"

# Decades of rank. The head is where the popular-site literature lives and the
# tail is where most of the web is, so a sample that is uniform over ranks is
# almost entirely tail and a top-n sample is entirely head. Strata make the
# trade-off explicit instead of implicit.
STRATA = [(1, 1_000), (1_001, 10_000), (10_001, 100_000), (100_001, 1_000_000)]


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=180) as resp:
        return resp.read()


def load_tranco(list_id):
    if list_id is None:
        list_id = fetch(TRANCO_DAILY).decode().strip()
    raw = fetch(TRANCO_DOWNLOAD.format(list_id=list_id))
    # /download/<id>/<n> serves bare CSV; /top-1m.csv.zip serves a zip. Sniff,
    # so the script keeps working whichever endpoint you point it at.
    if raw[:2] == b"PK":
        with zipfile.ZipFile(io.BytesIO(raw)) as z:
            text = z.read(z.namelist()[0]).decode()
    else:
        text = raw.decode()
    ranked = [(int(r), d) for r, d in csv.reader(io.StringIO(text)) if r]
    return ranked, {
        "source": "tranco",
        "list_id": list_id,
        "download": TRANCO_DOWNLOAD.format(list_id=list_id),
        "permalink": TRANCO_PERMALINK.format(list_id=list_id),
        "bytes_sha256": hashlib.sha256(raw).hexdigest(),
        "rank_semantics": "position, 1..1000000",
    }


def latest_crux_month():
    """Newest YYYYMM snapshot in the mirror. Never `current.csv.gz`: that URL is
    unversioned, which is the anti-pattern this script exists to avoid."""
    listing = json.loads(fetch(CRUX_INDEX))
    months = sorted(f["name"][:6] for f in listing if f["name"][:6].isdigit())
    return months[-1]


def load_crux(month):
    if month is None:
        month = latest_crux_month()
    raw = fetch(CRUX_MONTHLY.format(month=month))
    text = gzip.decompress(raw).decode()
    rows = list(csv.DictReader(io.StringIO(text)))
    # CrUX publishes a rank-magnitude bucket, not a position, and shuffles
    # within the bucket. Treating the file's line order as a rank is the single
    # most common way to misuse this list; we keep the bucket instead.
    ranked = [(int(r["rank"]), r["origin"]) for r in rows]
    return ranked, {
        "source": "crux",
        "list_id": month,
        "download": CRUX_MONTHLY.format(month=month),
        "permalink": "https://github.com/zakird/crux-top-lists",
        "bytes_sha256": hashlib.sha256(raw).hexdigest(),
        "rank_semantics": "magnitude bucket (1000/10000/100000/1000000), random order within bucket; strata below are bucket membership, not positions",
    }


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--source", choices=["tranco", "crux"], default="tranco")
    ap.add_argument("--list-id", help="Tranco list id; default is today's list")
    ap.add_argument("--crux-month", help="CrUX snapshot as YYYYMM; default is the newest published")
    ap.add_argument("--per-stratum", type=int, default=200)
    ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--out", default="sample")
    args = ap.parse_args()

    ranked, prov = load_tranco(args.list_id) if args.source == "tranco" else load_crux(args.crux_month)
    by_rank = {}
    for rank, name in ranked:
        by_rank.setdefault(rank, []).append(name)

    rng = random.Random(args.seed)
    sample, strata_report = [], []
    for lo, hi in STRATA:
        pool = [(r, n) for r, ns in by_rank.items() if lo <= r <= hi for n in ns]
        pool.sort()
        take = min(args.per_stratum, len(pool))
        drawn = rng.sample(pool, take) if take < len(pool) else pool
        label = f"{lo}-{hi}"
        sample += [(r, n, label) for r, n in sorted(drawn)]
        strata_report.append({"stratum": label, "in_frame": len(pool), "drawn": take})
        if take < args.per_stratum:
            print(f"WARNING: stratum {label} has only {len(pool)} entries", file=sys.stderr)

    with open(f"{args.out}.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["rank", "name", "stratum"])
        w.writerows(sample)

    manifest = {
        "drawn_at_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "frame": prov,
        "frame_entries": len(ranked),
        "design": "rank-stratified, simple random sample without replacement within each stratum",
        "strata": strata_report,
        "seed": args.seed,
        "n": len(sample),
        "sample_sha256": hashlib.sha256(open(f"{args.out}.csv", "rb").read()).hexdigest(),
        "cite": (
            f"We drew {len(sample)} sites from the {prov['source']} list "
            f"{prov['list_id'] or '(dated snapshot, see manifest)'} "
            f"({prov['permalink']}), {args.per_stratum} per rank stratum over "
            f"{', '.join(s['stratum'] for s in strata_report)}, "
            f"simple random sampling without replacement within each stratum, seed {args.seed}."
        ),
    }
    with open(f"{args.out}.manifest.json", "w") as fh:
        json.dump(manifest, fh, indent=2)

    print(json.dumps(manifest, indent=2))


if __name__ == "__main__":
    main()
</file>

Real output, ''%%--source%% tranco %%--per-stratum%% 200 %%--seed%% 20260812'', run on 2026-08-12 (the four ''strata'' objects are reflowed to one line each here; everything else is verbatim):

<code>
{
  "drawn_at_utc": "2026-08-12T21:36:44+00:00",
  "frame": {
    "source": "tranco",
    "list_id": "645KX",
    "download": "https://tranco-list.eu/download/645KX/1000000",
    "permalink": "https://tranco-list.eu/list/645KX",
    "bytes_sha256": "feab56e5b4f48e3c1393916d74226d1d6234416c621390be29bb72199caf18d7",
    "rank_semantics": "position, 1..1000000"
  },
  "frame_entries": 1000000,
  "design": "rank-stratified, simple random sample without replacement within each stratum",
  "strata": [
    { "stratum": "1-1000",        "in_frame": 1000,   "drawn": 200 },
    { "stratum": "1001-10000",    "in_frame": 9000,   "drawn": 200 },
    { "stratum": "10001-100000",  "in_frame": 90000,  "drawn": 200 },
    { "stratum": "100001-1000000", "in_frame": 900000, "drawn": 200 }
  ],
  "seed": 20260812,
  "n": 800,
  "sample_sha256": "e3c38a539111c9872b62682f8d289f5d573248d95fafe6ce3bf49ea02bf92249",
  "cite": "We drew 800 sites from the tranco list 645KX (https://tranco-list.eu/list/645KX), 200 per rank stratum over 1-1000, 1001-10000, 10001-100000, 100001-1000000, simple random sampling without replacement within each stratum, seed 20260812."
}
</code>

The ''in_frame'' column is the point: the four strata contain 1,000, 9,000, 90,000 and 900,000 sites, so drawing 200 from each over-samples the head by a factor of 900 relative to a uniform draw. That is deliberate and it is recoverable — reweight by ''in_frame'' when you want a frame-wide estimate, and report per stratum when you do not.

===== Use in Publications =====

The figures below come from a structured extraction over **5,859 full-text papers** from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. The population for this page is the **1,153 papers that drew at least one study population whose unit is websites, domains or web pages** — 19.7% of the corpus. Only those web-unit populations are counted, so a paper that sampled websites top-//n// and participants by convenience contributes only the first. Sentinel values (''not-stated'') are counted as silence, never as answers, and papers are counted, never tuples, except where a table says otherwise. 2025 and 2026 are provisional venue-years: CCS and IMC 2026 have not been held and two more 2026 venue-years are incompletely indexed, so they are under-represented by construction.

==== The population ====

^ ^ Papers ^ ^
| Corpus | 5,859 | |
| Drew a web population (this page) | 1,153 | 19.7% of corpus |
| …that also ran a crawl | 680 | 59.0% of 1,153 |
| …that sampled the web without crawling((DNS, certificate-log, archive and reanalysis studies. They draw a domain or page population and never open a browser.)) | 473 | 41.0% of 1,153 |
| Crawling papers with no web-unit population((Crawls whose population is apps, social-media accounts, IoT devices or "other".)) | 440 | of 1,120 crawling papers |

Web-unit populations in scope: 2,525 across the 1,153 papers. By unit: **websites 723 (62.7%)**, domains 393 (34.1%), web pages 251 (21.8%). By venue: USENIX 230, IMC 193, WWW 186, CCS 179, PETS 132, IEEE S&P 117, NDSS 116 — sampling the web is not concentrated in one venue.

==== Which methods, and how that moved ====

^ Method ^ Papers ^ Share of 1,153 ^
| top-//n// | 664 | 57.6% |
| purposive | 319 | 27.7% |
| pre-existing-dataset | 278 | 24.1% |
| random | 190 | 16.5% |
| exhaustive | 165 | 14.3% |
| seed-and-crawl | 100 | 8.7% |
| **stratified** | **69** | **6.0%** |
| not-stated | 44 | 3.8% |
| convenience | 29 | 2.5% |
| snowball | 2 | 0.2% |

Shares exceed 100% because a paper may draw several populations different ways. **98.4% state a sampling method for at least one web population** — this is the field's best-reported design decision, and the contrast with the 51.7% that state a list version is the finding.

^ Method ^ 2010–2013 ^ 2014–2017 ^ 2018–2021 ^ 2022–2024 ^ 2025–2026 //(provisional)// ^
| //papers in bucket// | //104// | //187// | //322// | //350// | //190// |
| top-//n// | 41.3% | 55.1% | 59.6% | 58.0% | 64.7% |
| purposive | 38.5% | 23.5% | 27.0% | 27.4% | 27.4% |
| random | 15.4% | 20.3% | 13.4% | 18.0% | 15.8% |
| exhaustive | 5.8% | 19.8% | 13.0% | 13.7% | 16.8% |
| stratified | 1.0% | 4.8% | 5.0% | 8.9% | 6.3% |
| seed-and-crawl | 8.7% | 8.0% | 10.2% | 9.4% | 5.3% |

**Top-//n// has not been superseded; it has consolidated.** Stratified sampling grew from 1.0% to a peak of 8.9% in 2022–2024 and did not continue rising. Nothing in the 2025–2026 slice suggests a new sampling method entering the field: searching the page population from 2024 onward for a frame whose name mentions a language model returns **three papers**, and none of them uses a model to draw a sample of the web — two sample the GPT Store as a platform and one builds a synthetic corpus. Sampling practice has been stable for a decade; what changed underneath it is the list.

==== Which frames ====

Free-text source names were folded into frame families by ''scripts/sample_fold.mjs''; **1,143 papers (99.1%) name a source**.

^ Frame kind ^ What it enumerates ^ Papers ^ Share of 1,143 ^ Spellings folded ^
| Popularity ranking list | the head of the web, ordered by a popularity proxy | 764 | 66.8% | 496 |
| Custom or ad-hoc seed list | a list the authors assembled | 257 | 22.5% | 137 |
| Curated topical or abuse list | a set assembled around a property (phishing, censorship, category) | 96 | 8.4% | 89 |
| Registry, zone file or DNS | names registered under a TLD, or seen by a resolver | 87 | 7.6% | 90 |
| Dataset from earlier research | someone else's sample, inherited whole | 79 | 6.9% | 92 |
| Crawl corpus or web archive | pages another crawler already fetched | 38 | 3.3% | 14 |
| Certificate log or scan | hosts that presented a certificate or answered a probe | 25 | 2.2% | 17 |
| Search engine result pages | what a search engine returns for a query set | 19 | 1.7% | 14 |
| Observed traffic or telemetry | pages people were seen to visit | 8 | 0.7% | 8 |

Two thirds of web sampling is done from a popularity ranking, and the frames that would answer "what does the //whole// web do" — registries, zone files, certificate logs — carry under a tenth of the field between them. 472 distinct source strings across 519 populations did not fold into any family; 115 papers are left with no frame kind at all. The full residue is on the [[provenance:design:sampling|provenance page]].

==== Counting list names by exact string undercounts them by up to 91% ====

This is a warning about reusing this field, and about every published table of "which list do papers use":

^ List ^ Papers, exact string only ^ Papers, any spelling ^ Distinct spellings ^ Undercount ^
| Alexa | 50 | 463 | 355 | **89.2%** |
| Cisco Umbrella | 2 | 23 | 22 | 91.3% |
| Majestic | 4 | 26 | 17 | 84.6% |
| CrUX | 9 | 36 | 23 | 75.0% |
| Tranco | 178 | 262 | 91 | 32.1% |
| Common Crawl | 13 | 18 | 6 | 27.8% |

"Alexa" appears as //Alexa top 1M//, //Alexa Top 1 Million Global Sites//, //1M-top Alexa//, //Alexa's list//, //Alex Top 1500 websites// and 350 other ways. Tranco is the least mangled because its name carries no size, which is a small argument for naming a list without a number.

==== The list changed under the field ====

Papers naming each list, as a share of the page population in each year:

^ Year ^ Papers ^ Alexa ^ Tranco ^ CrUX ^
| 2021 | 87 | 47.1% | 31.0% | 0.0% |
| 2022 | 106 | 37.7% | 26.4% | 1.9% |
| 2023 | 122 | 24.6% | 38.5% | 4.1% |
| 2024 | 122 | 17.2% | 45.9% | 9.8% |
| 2025 //(provisional)// | 133 | 8.3% | 56.4% | 9.0% |
| 2026 //(provisional)// | 57 | 7.0% | 38.6% | 7.0% |

**Tranco is current practice; Alexa is residue.** The transition tracks the retirement (May 2022) with the two-to-three-year lag you would expect from submission cycles. CrUX is the outlier: Ruth et al. {[ruth2022_toppling]} measured every public list against Cloudflare's own traffic data and found **CrUX the most accurate across all metrics**. What "inaccurate" costs, in their numbers, is a property of the //other// lists — of the 1,790 Alexa top-10K domains they could measure, 70% were placed by Cloudflare in a lower rank-magnitude bucket and 27.2% two or more orders of magnitude lower. Four years on, **CrUX is still named by under 10% of papers: the most accurate list is not the most used one.** Read the 2026 row with care: it rests on 57 papers from incomplete venue-years.

==== Sample sizes are round numbers ====

Tuple-level, because a paper may draw several populations: 2,379 of 2,525 web populations (94.2%) state a size, and 1,121 papers (97.2%) state one for at least one population.

^ Population size ^ Populations ^ Share of 2,379 ^
| 1–100 | 473 | 19.9% |
| 101–1,000 | 411 | 17.3% |
| 1,001–10,000 | 511 | 21.5% |
| 10,001–100,000 | 387 | 16.3% |
| 100,001–1,000,000 | 354 | 14.9% |
| more than 1,000,000 | 243 | 10.2% |

Median 6,755; quartiles 283 and 103,541; the 90th percentile is 1,067,968. **43.5% of stated sizes are 1, 2 or 5 times a power of ten**, and 198 populations are exactly 1,000,000. Median size by method: top-//n// 10,000 (879 populations), exhaustive 708,000 (235), seed-and-crawl 47,847 (116), pre-existing-dataset 24,000 (346), stratified 4,084 (76), random 2,000 (237), purposive 102 (406).

==== Half the field never says which version it drew ====

^ Reported for the same web population ^ Papers ^ Share of 1,153 ^
| source list named | 1,143 | 99.1% |
| sampling method | 1,134 | 98.4% |
| size | 1,121 | 97.2% |
| **version or access date** | **596** | **51.7%** |
| source + size + version | 579 | 50.2% |
| source + size + version + method | 573 | 49.7% |

Of the 764 papers drawing from a popularity ranking — the case where the version matters most, because the list changes daily — **367 (48.0%) version it and 397 (52.0%) do not.**

^ Indicator ^ 2010–2013 ^ 2014–2017 ^ 2018–2021 ^ 2022–2024 ^ 2025–2026 //(provisional)// ^
| states a version or date | 26.9% | 44.4% | 48.1% | 60.9% | 61.6% |
| ranking users who version the ranking | 14.9% | 37.3% | 45.7% | 55.5% | 60.8% |

This is genuine improvement — versioning has more than doubled — and it has stalled around 60%. Releasing the drawn list would close the gap for the rest, and mostly does not happen: **58.0% of this population release an artifact of their own and 33.8% release a dataset**, but of the 557 papers that version nothing, only **164 (29.4%)** release a dataset that could stand in for the missing version.

==== One page per site is the norm ====

Of the 680 papers in this population that also crawled:

^ Interaction depth ^ Papers ^ Share of 680 ^
| one designated page per unit((Not necessarily an //inner// page: this value covers any design that visits one nominated URL per item — a login form, a privacy policy, a specific search-result URL — rather than a site root and its links. Where the population unit is already //web pages// it is the natural value and says nothing about depth. Reading the crawl-configuration evidence behind eight of these papers found search-result URLs, CAPTCHA pages and ad-audit pages among them, not only site-specific inner pages.)) | 222 | 32.6% |
| landing page only | 155 | 22.8% |
| landing plus subpages | 128 | 18.8% |
| deep crawl | 92 | 13.5% |
| not stated | 69 | 10.1% |
| no crawl configuration recorded | 14 | 2.1% |

**Only 32.4% visit more than one page per site**, and only **15.0% state how many subpages per site** (median 10, where stated). Among the 492 papers whose unit is explicitly ''websites'' and which crawled, **24.8% visited the landing page only** — which is precisely the population Aqeel et al. {[aqeel2020_landing]} showed would need its claims revised.

==== Methodology and limitations of these figures ====

  * **How they were produced.** One structured record per paper was extracted from full text; each tuple carries a verbatim evidence quote and its section, so any figure here traces to a sentence. The population is defined by the extraction's ''population[].unit'' enum, which is a stable field; the free-text ''sourceList'' is not, and is folded before counting (see below).
  * **Folding.** ''sourceList'' agrees run-to-run on roughly a fifth of exact strings, so it is reported as folded families and as rankings, never as precise percentages. The fold, its rules and its full unmapped residue are on the [[provenance:design:sampling|provenance page]]. Family shares are of papers naming any source and exceed 100% because a paper may draw from several frames.
  * **Silence is not absence.** "Does not state a version" means the paper did not say, not that the authors did not know. These are reporting figures.
  * **''samplingMethod'' is an enum assigned by the extractor**, not a phrase copied from the paper. A design that a paper calls "a representative sample" may be scored ''top-n'', correctly. Read the enum as the extractor's reading of what was done.
  * **Venue coverage.** Seven venues only. 2025 and 2026 are incomplete for reasons of calendar and indexing, so per-year trends should be read as ending in 2024; the two provisional rows are labelled everywhere they appear.
  * **Every query behind this section, its report script and its unedited output** are on [[provenance:design:sampling]]; corpus-level caveats are on [[literature:corpus]].

===== What to Report =====

A sampling paragraph a reader can act on names, in this order:

  - **Target population** — what the claim is about, in words, not in list names.
  - **Frame** — the list, with its **identity**: a Tranco list id, a CrUX month, a crawl id, a zone-file date, or a hash of the file you archived.
  - **Design** — top-//n//, random, stratified (with the strata), exhaustive, purposive (with the selection rule), or reuse (with whose sample and from when).
  - **Size** — drawn, and **analysed after attrition**, with the reason for the difference.
  - **Unit** — site, registrable domain, origin, page; and if pages, how they were chosen and how many per site.
  - **Seed**, if any randomness was involved, so the draw is repeatable.
  - **The drawn list itself**, as an artifact, whenever licensing allows it. It makes every other item above checkable.

===== Open Questions =====

<WRAP todo>
  * **How much do results actually move between rank strata?** Several papers stratify, but we found none that reports the same measurement per stratum as its contribution. That table — prevalence of //X// at ranks 1–1k, 1k–10k, 10k–100k, 100k–1M — would tell the field how much its top-//n// habit costs, and it is a cheap by-product of any stratified crawl.
  * **No maintained internal-page list.** Hispar is offline (see above), and a search API is now a paid dependency. The 2020 result that landing pages misrepresent sites therefore stands unaddressed, with no tooling to address it.
  * **Attrition is not in any structured record.** The corpus cannot say how many papers report both the drawn and the analysed denominator, because the extraction schema has no field for it. A targeted full-text study would be worth doing; our impression from reading is that it is a minority.
  * **Is rank weighting used outside third-party analysis?** Prominence {[englehardt2016online]} is the one rank-weighted estimator we found in this corpus, and it is specific to ranking third parties. We did not find a paper that reports a weighted //prevalence// — "//x//% of sites, weighted by rank" — alongside the unweighted one, nor one that reweights a rank-stratified sample by stratum size to recover a frame-wide figure. Both are routine in survey statistics. If you know of an example, please add it.
</WRAP>

===== Related Pages =====

  * [[Design:Website selection]] — which list, and what each ranking actually measures. This page assumes you have chosen one.
  * [[Design:Website classification]] — sampling by category needs a categoriser, and its error rate becomes your stratum error.
  * [[Design:Crawling location]] — the vantage point is a second sampling decision; geo-targeted sites are a different population from each country.
  * [[Design:Archives]] — sampling from an archive samples what the archive captured, which is its own frame.
  * [[Programming:Crawler]] — attrition, timeouts and bot walls are properties of the tool as much as of the sample.
  * [[Statistics:Biases]] — coverage and selection bias, downstream of everything on this page.
  * [[Statistics:Hypothesis testing]] — what a p-value means when the sample is a census of a list.
  * [[:Artifacts]] — publishing the drawn list is the cheapest reproducibility win available here.

====== References ======

<bibtex bibliography></bibtex>


/* This enables discussion under this article. */
~~DISCUSSION~~