Table of Contents
Welcome to Measure The Web
Empirical studies on the web require researchers to navigate a complex landscape of experimental design choices, ranging from selecting a representative sample of websites to choosing the appropriate crawling technology. Similarly, analyzing results involves critical decisions, such as website categorization and statistical methodology. Too often, these decisions are made based on limited guidance, informal advice, or trial and error, despite their profound impact on research outcomes and their applicability.
Measure The Web aims to bridge this gap, by providing evidence-based guidance for empirical web measurement studies. The platform evaluates design choices and their implications by referencing relevant academic publications or conducting original studies where necessary. It is written for someone starting a PhD in web privacy or security measurement, and for experienced researchers looking for current practice or for a citation that justifies a design they have already chosen. If you are in a mentoring position, consider sharing it with your mentees — and consider contributing: even a small edit that backs a subjective claim with a citation makes the site stronger.
Start here
| If you… | Start at |
|---|---|
| have a study idea and no instrument yet | Automated measurements — crawl, scan, or unpack an app, and what each can answer |
| have data and need to turn it into labels | Website classification for the protocol; Privacy or Security for the axis |
| have results and need to defend them | Biases — what your frame, vantage and denominator already did to the number |
| want to know what order to do this in | Research journey — the decisions that are expensive to revisit later |
The seven namespaces
Each namespace page carries its own annotated list of the pages inside it, what a student needs each one for, and where the namespace stops. That list lives there rather than here, so it stays current.
| Namespace | What it is for |
|---|---|
| Design | Choices made before the crawler runs: which list stands in for the web, how you draw from it, where you appear to be, live or archived, and what you hold fixed between waves. |
| Programming | The instrument: which browser, which control protocol, what happens after page.goto, whether the profile survives, and what file you write. |
| Privacy | Classifying what a crawl observed on the privacy axis — requests, cookies, scripts, fingerprints, syncing, consent — and the two ways the instrument lies to you. |
| Security | Web security as it is deployed, not how to break it: TLS, headers, label sources for “malicious”, phishing in the wild. |
| Statistics | What these methods do to web-measurement data: the unit-of-analysis error, the hypothesis family a crawl invents, hand-coded ground truth. Not a textbook. |
| Practices | Around the study rather than inside it: ethics review, notifying operators, regulators, and the sentence that gets quoted once the paper is public. |
| Writing | Where the paper goes, and how the related-work section is allowed to count. |
Outside the seven: Artifacts — a GitHub URL is not an archival artifact — and the bibliography every citation on this site resolves against.
How to read a page
Pages whose figures come from the publication corpus carry a working log under provenance:, mirroring the page id — fingerprinting for Fingerprinting. Each log holds every query with its denominator, the report script and its unedited output, the name-folds and their unmapped residue, the review passes, and what could not be established.
The corpus is seven venues — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P — 2010–2026, 5,859 extracted papers. Every figure is about the population its query defines: only 1,120 of those papers ran a crawl, 1,762 ran statistical inference, 402 assess a law. Two pages quoting different populations correctly are not in conflict. 2025 is thin and 2026 is provisional; the corpus-wide caveats are on Corpus.
What is coming, and what was ruled out
Roadmap lists the pages that are scoped and queued but not yet written, with the evidence behind each one — and the proposals that were assessed and deliberately not written, with their paper counts, so the next gap pass does not re-propose them.
Contributing
We welcome contributions. Editing is limited to registered users; Contributing covers the syntax, the namespace structure, and the rule that every published figure names its query. External reading we point at rather than duplicate: PhD and research advice and khstats on statistics.