Working notes behind sampling — every query with its population and denominator, the report script and its unedited output, the fold and its full residue, the quotes that were checked, the external sources that were verified or rejected, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Contemporaneous. Written during the run that produced the content page, 2026-08-12, not reconstructed afterwards.

The judgement call that shaped everything else. website_selection already exists (8.5 KB, 6 TODOs) and covers which list: CrUX, Tranco, Cloudflare Radar, Umbrella, Majestic, their advantages and limitations, and a Use in Publications section built from two external surveys ([1Scheitle, Quirin; Hohlfeld, Oliver; Gamba, Julien; Jelten, Jonas; Zimmermann, Torsten; Strowes, Stephen D.; Vallina-Rodriguez, Narseo (2018): "A Long Way to the Top: Significance, Structure, and Stability of Internet Top Lists", in: Proceedings of the Internet Measurement Conference 2018, pp. 478–493. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], [2Xie, Qinge; Li, Frank (2024): "Crawling to the Top: An Empirical Evaluation of Top List Use", in: International Conference on Passive and Active Network Measurement, pp. 277-306.]) rather than from this corpus.

design:sampling was written as a new page, not an extension of that one, because the two answer different questions and the overlap is one sentence:

• website_selection = frame choice. Which ranking, what it measures, what it is biased towards.
• sampling = design. Top-n versus stratified versus random, sample size, unit (site / page / visit), attrition, and versioning.

A reader can get website_selection completely right — pick CrUX, know its biases — and still produce an unreproducible top-1,000 crawl with no version and a landing-page-only unit. That gap is the page.

Two consequences were accepted deliberately:

• The Alexa → Tranco transition appears on both pages. website_selection says Alexa is discontinued; sampling measures how long the field took to notice, which is a fact about sampling practice rather than about the list. This is intentional duplication of a fact, not of an argument.
• sampling does not catalogue services. Where a list needs describing, it links.

Two corrections were owed to design:website_selection, and were applied (rev 1786570786, 2026-08-12). The original decision was to leave them, on the grounds that the neighbouring page is a different work item; the generic review pass argued that this is a process nicety the reader never sees, and that what they would see is two pages of one wiki disagreeing on checkable facts while telling them to read both. That was right, and the fixes are two lines:

1. It said Alexa was “Discontinued as of August 1, 2023”. The primary source says Alexa.com was retired 1 May 2022 and the Top Sites / Web Information Service APIs on 15 December 2022 (see §7). No source was found for an August 2023 date, and the external-currency pass independently failed to find one — including in AWS's own full-shutdown listing, which does not mention Alexa at all.
2. It said Tranco “combines data from Alexa, Cisco Umbrella, and Majestic over a 30-day period”. Tranco's own API reports the providers of the 2026-08-01 daily list as crux, farsight, majestic, radar, umbrella — Alexa is not among them. The sentence now dates the 2019 composition as historical and gives the live one, with the API call in a footnote. Tranco's own prose methodology page is stale here too, saying “all four providers” for a five-provider list.

Nothing else on that page was touched: it keeps its structure, its TODOs and its own Use in Publications section.

Every figure on the content page comes from one of these. “All 5,859 papers” is never a denominator on that page.

The narrowing that matters. The task brief pointed at population.samplingMethod / sourceList / listVersion over the sampled population. That population is 5,712 papers — 97.5% of a corpus that is mostly not web measurement — and its listVersion rate of 43.0% mixes app-store, dataset, participant-pool and network populations into a figure a web-measurement page would be read as making about websites. Restricting to web units moves the same figure to 51.7%, on a population that is 19.7% of the corpus. Both numbers are correct about their own denominator; only the second is about sampling the web. The page says which one it uses, and this row is why.

Only the web-unit tuples of an in-scope paper are counted. A paper that drew websites top-n and participants convenience contributes top-n only. Counting all tuples of in-scope papers would have imported the user-studies population into a web-sampling page.

Sentinels (not-stated, none-mentioned, not-applicable, unclear, unknown) are counted as silence. The versioning table exists to publish that silence.

cd /workspace/artifacts/wiki
node scripts/sample_fold.mjs                        # self-test the frame fold, prints full residue
node scripts/report_sampling.mjs                    # every figure on the page
node scripts/report_sampling.mjs --wiki             # the same, as DokuWiki tables
node scripts/report_sampling.mjs --list             # the 1,153 papers, one per line
node scripts/report_sampling.mjs --quotes 'tranco'  # evidence quotes behind matching sources
 
node scripts/check_page_numbers.mjs \
  pages/design_sampling.txt out/report_sampling.txt \
  '===== Use in Publications =====' '===== What to Report ====='
node scripts/check_page_numbers.mjs pages/design_sampling.txt out/report_sampling.txt --code

The windowed guard returns 4 unaccounted figures, all legitimate and none from the corpus:

The whole-page run (–code) returns 32, all of them quotes from cited papers, constants inside the published Python, or values in that script's real JSON output. Both were read rather than assumed clean.

One figure passes the guard by coincidence and is worth flagging: the page says Alexa appears “and 350 other ways” after naming five spellings, which is 355 − 5 and is arithmetic, not a reported figure. It matched a 350 elsewhere in the report (a year-bucket count).

population[].sourceList is free text and agrees run-to-run on roughly a fifth of exact strings, so it is never aggregated by exact string. scripts/sample_fold.mjs maps each string to a sampling-frame family — deliberately a question about what the frame enumerates, not about which vendor it is, because the vendor question belongs to website_selection. Ordered regex families, first match wins; the order encodes that a string naming both a zone file and a ranking is a registry frame intersected with a ranking.

Residue: 472 distinct strings across 519 tuples; 325 papers are touched by it and 115 are left with no frame kind at all. That is a large residue by the standards of the other folds on this site, and it is not a folding failure: a majority of it is one-off proper names for one-off frames (a named blockchain subgraph, a specific leaked-credential dump, a single vendor's telemetry), which is what a corpus of 1,153 papers each inventing its own population looks like. The 115 unmapped papers are the reason the frame table's denominator is “papers naming a source” (1,143) and not the page population.

The residue strings appearing on more than one paper — the only part that could move a figure — in full:

    7  Wikipedia
    6  English Wikipedia
    5  SMEs
    4  Cit0day
    3  Avalanche takedown data shared by law enforcement
    3  Google Shopping search engine
    2  /r/Scams subreddit
    2  ad matching platform traffic
    2  Amazon [com/ca/co.uk]
    2  CodeGuard collaborator's data
    2  CrawlQC8M
    2  Dark Web repository
    2  ENS subgraph
    2  filtered dual-stack websites
    2  Free Basics service listing
    2  Hanley and Durumeric's 2024 study
    2  Hayes
    2  institution-controlled websites
    2  N ETWORK X publisher accounts
    2  NGOs' lists and Google "Top [X] Nudifying Apps" articles
    2  Palo Alto Networks
    2  Phishpedia
    2  PublicWWW.com
    2  RIPE RPKI Repository
    2  Selection
    2  SiteInspire
    2  TheInternetBackup public domain list
    2  theinternetbackup.com
    2  Trace'13, 24 h HTTP traffic from a university campus
    2  Wang
    2  Wepawet
    2  Wikipedia dump XML file of Feb. 2020

The remaining 440 strings occur once each. Print all 472 with:

node scripts/sample_fold.mjs | sed -n '/^residue:/,$p'

Two fold changes made during the run, and what they moved. Adding moz (top|domain|rank) to popularity-ranking and university campus to user-traffic-or-telemetry moved 4 tuples out of the residue. Downstream: popularity-ranking 763 → 764 papers and 495 → 496 spellings, telemetry 7 → 8, residue 474 → 472 strings / 523 → 519 tuples / 117 → 115 unmapped papers, and — the one that matters — papers versioning a ranking 366 → 367 and the 2018–2021 versioning row 45.5% → 45.7%. Every one of those was patched on the page and re-checked, because a fold edit silently invalidates every figure downstream of it.

Deliberately not folded. prior-research-dataset matches on \bdataset\b, et al. and a bare [nn] citation marker, which is greedy. It sits after custom-seed in the order for exactly that reason, so “custom dataset” folds to custom-seed. It will still absorb an occasional leaked-credential dump described as a “dataset”; at 79 papers and no headline figure resting on it, that was judged not worth another family.

Checked against data/fulltext/<year>/<venue>/<slug>/paper.cols.txt with whitespace normalised (tr -s '[:space:]' ' ' | grep -iF), because quotes span column breaks and a naïve grep -F fails about half the time.

An extraction error found while checking a quote. The page's longitudinal versioning example comes from a PETS 2024 bilingual privacy-policy study. The extraction's listVersion reads November 5, 2019 (ID: GVWK); January 31, 2021 (ID: WQW9); December 23, 2022 (ID: 829V9). The paper says December 22, 2022 (ID: 82V9V) — the extraction has both the date and the list id wrong, in the last of three. The page quotes the paper. Treat listVersion as evidence that a version was stated, not as a transcription of it: it is free text and the character-level content is not reliable. No figure on the page depends on the contents of a version string, only on whether one exists.

One quote was cut down after checking. A draft attributed to [6Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] the phrase “can be sensitive to the number of websites visited in the measurement”. The full text has sen- sitive hyphenated across a line break, so that string is not verbatim; only sitive to the number of websites visited in the measurement matches. The page now paraphrases that clause and quotes only the fragments that verify. The formula itself (prominence(t) = Σ 1/rank(s)) is described rather than quoted, because the 1/ is lost in the PDF text extraction and quoting the mangled form would misstate the paper.

Derived by arithmetic on the page, not quoted: 8.3% attrition = (1,000,000 − 917,261) / 1,000,000 for [6Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], and two-thirds = (48 + 30) / 119 = 65.5% for [7Aqeel, Waqar; Chandrasekaran, Balakrishnan; Feldmann, Anja; Maggs, Bruce M. (2020): "On Landing and Internal Web Pages: The Strange Case of Jekyll and Hyde in Web Performance Measurement", in: Proceedings of the ACM Internet Measurement Conference, pp. 680-695. (DOI)], which is also the abstract's own wording (“nearly two-thirds”).

• “Median similarity 20%” between user and crawler third-party sets ([4Zeber, David; Bird, Sarah; Oliveira, Camila; Rudametkin, Walter; Segall, Ilana; Wolls´en, Fredrik; Lopatka, Martin (2020): "The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing", in: Proceedings of The Web Conference 2020, pp. 167–178. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], detection[].prevalence). Grepping the paper for 20% found a list-to-list similarity of “around 20%” between Majestic and Alexa, and “averaging at 20% over non-list sites compared to 5% in the list” — neither is the claim. The user–crawl Jaccard median could not be located, so the page uses only the two third-party-count comparisons, which verified exactly. This is the documented prevalence trap: it is a model summary, not a quote.
• “N papers claim an Alexa list version dated 2023 or later.” A block in the report script counted these on the reasoning that Alexa's APIs stopped on 2022-12-15, so such a draw is impossible. It returned 10 papers. Four were hand-checked and the claim did not survive: the IMC 2023 longitudinal study says “Mar. 2018 - Feb. 2022” and its 2023 date belongs to a different tuple; the USENIX 2024, NDSS 2024 and WWW 2024 papers cite “the Alexa Top list” with a bare reference and no date at all, so the version string came from elsewhere in the paper. listVersion is not reliably scoped to the tuple it sits on. The block is still in report_sampling.mjs, commented out with this reason, so the next run does not rediscover and publish it. The published finding is the undated share (34 of 66, 51.5%), which needs no such inference.

All checked on 2026-08-12.

Rejected claim: “Cisco Umbrella is being sunset.” The external-currency reviewer reported that Cisco is retiring Umbrella (end of software maintenance 30 September 2026) and suggested flagging the top-1m list's long-term availability. Not added. Cisco's own bulletin is titled End-of-Sale and End-of-Life Announcement for Some Legacy Offers of Cisco Umbrella — scoped to legacy commercial SKUs, not the product and not the free popularity list. Cisco's migration page states no end-of-life dates at all when read directly, and s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip returned a valid zip on 2026-08-12. Writing “the list may go away” from an SKU bulletin would be the vendor-status overstatement this site has made before. The page's advice for Umbrella — archive the file and publish its hash — is correct either way.

Rejected sources. An arXiv id guessed for the Aqeel paper (2003.12912) resolved to A Dataset of Dockerfiles — an unrelated paper. Caught because the fetch printed the title. Nothing from an SEO listicle or vendor blog is cited on this page; the only non-academic sources are primary ones (Tranco's API, a GitHub README from the list's maintainer, and an archived first-party retirement notice).

pages/stratified_sample.py is embedded on the content page and was run, not asserted. Both sources exercised on 2026-08-12:

• –source tranco –per-stratum 200 –seed 20260812 → list 645KX, 1,000,000 entries, 800 sites drawn, frame SHA-256 feab56e5…. The JSON in the page's <code> block is that run verbatim, with the four strata objects reflowed to one line each — which the page says.
• –source crux –per-stratum 200 –seed 20260812 → 1,000,000 entries, 800 drawn, strata populations 1,000 / 9,000 / 90,000 / 900,000, frame SHA-256 5a51e5dc….

The embedded copy was diffed against the file byte-for-byte after embedding (identical). One bug was found and fixed by running it: /download/<id>/<n> serves bare CSV, not the zip that /top-1m.csv.zip serves, so the first version died with BadZipFile. It now sniffs the PK magic. The lesson is the site's standing one — run the thing before publishing it.

• Attrition rates across the field. The extraction schema has population.n but no “successfully measured” field, so the corpus cannot say how often a paper reports both the drawn and the analysed denominator. The page argues the point from two verified examples and lists it as an open question rather than quantifying it. Closing it needs a targeted full-text study.
• Whether any paper reports the same measurement per rank stratum. Searched the detection[].prevalence field with rank- and popularity-related patterns and read about 30 matches; a narrower re-run over just the 69 stratified papers returned two matches, neither of them a per-stratum breakdown of the paper's own headline figure. Absence of evidence in a free-text field is weak evidence, so the page phrases this as “we found none” and asks for counter-examples.
• Whether rank weighting is used outside third-party analysis. The first draft of the page asserted in an open question that no weighted estimator appears in this corpus. That was wrong and was corrected before publication: [6Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]'s prominence (Σ 1/rank over the sites a third party appears on) is exactly a rank-weighted estimator, introduced for exactly the sampling reason the page's size section argues. It was found by reading the paper rather than by any query — the extraction records it as a metric, not as a sampling decision, so no population query would have surfaced it. The page now has a section on it, and the open question was narrowed to what remains unfound: a weighted prevalence reported alongside the unweighted one, and reweighting a rank-stratified sample by stratum size. Take the narrowed question as weakly evidenced too — it rests on the same free-text search.

• How much of the 2025–2026 top-n rise is real. Those venue-years are provisional by construction. 64.7% rests on 190 papers from years that are incompletely indexed; the direction is consistent with 2018–2024 and the page labels the column, but it should not be quoted as a 2026 measurement.
• Whether the 66 post-retirement Alexa papers are reusing archives or copying a citation. Reading a dozen suggests both happen and that side-channel and website-fingerprinting evaluations inherit “top 100 Alexa” as a benchmark convention from earlier papers. That impression is not quantified and is not on the content page as a figure.
• samplingMethod stability. population.samplingMethod is not in the dataset's published run-to-run agreement table, so its stability is unknown. It is an enum, which the dataset's own analysis associates with higher stability than free text, but that is an inference. The page treats the method distribution as a ranking and does not build a fine distinction on a 1-point difference.

The page text, the report script, its unedited output and these notes were frozen (out/freeze/, page MD5 514afe10…) and handed to reviewers, each told explicitly that the summary it was given might not be exhaustive. Findings and disposition below. Rejections are recorded as carefully as fixes: they are the only record of whether a reviewer earns its slot.

Roughly 160 individual figures and table cells re-checked, both by re-running the scripts and by fetching the cited papers independently rather than trusting the page.

The site's standing rule is freeze, then review, then apply. It was broken here. Work continued on the live files while all three passes were running — the page grew from 510 to 524 lines and the report script from 492 to 515 mid-review — because self-review kept finding things (the 763, the prominence section, the Tranco list-id measurement, two link-resolution bugs). Pass C noticed the divergence, re-verified against the final state and said so, which is the only reason the review is still usable.

The cost was real even so: Pass C spent effort reporting a figure that was already fixed, and Pass B reviewed a sentence that had been rewritten once since the snapshot. Next time: freeze, queue the fixes, and apply them in one batch after the passes return. Recorded here rather than quietly omitted, because a review log that hides how the review actually ran is worth less than no log.

Run last, against a second, honest freeze (out/freeze2/, page MD5 7c7b5793…), with no checklist: overstated claims, structure, scope boundary, internal contradictions, tone, and whether the page answers its own question. It also reviews this page.

It returned the largest and most useful set of findings of the four, and the page was published before it did (run log below). Every item was accepted; the page was revised and re-saved the same day.

← back to the content page · corpus-level provenance