| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| start [2026/08/29 06:37] – Link the new programming:crawler:llm_agents page. Authored by Claude karel.kubicek.claude | start [2026/09/07 07:03] (current) – Rewrite: index stops being a catalogue — namespace pages own their own page lists. Adds Start here entry points, provenance/corpus reading note, and links the new roadmap. Authored by Claude karel.kubicek.claude |
|---|
| Empirical studies on the web require researchers to navigate a complex landscape of experimental design choices, ranging from selecting a representative sample of websites to choosing the appropriate crawling technology. Similarly, analyzing results involves critical decisions, such as website categorization and statistical methodology. Too often, these decisions are made based on limited guidance, informal advice, or trial and error, despite their profound impact on research outcomes and their applicability. | Empirical studies on the web require researchers to navigate a complex landscape of experimental design choices, ranging from selecting a representative sample of websites to choosing the appropriate crawling technology. Similarly, analyzing results involves critical decisions, such as website categorization and statistical methodology. Too often, these decisions are made based on limited guidance, informal advice, or trial and error, despite their profound impact on research outcomes and their applicability. |
| |
| Measure The Web aims to bridge this gap, by providing evidence-based guidance for empirical web measurement studies. The platform evaluates design choices and their implications by referencing relevant academic publications or conducting original studies where necessary. | Measure The Web aims to bridge this gap, by providing evidence-based guidance for empirical web measurement studies. The platform evaluates design choices and their implications by referencing relevant academic publications or conducting original studies where necessary. It is written for someone starting a PhD in web privacy or security measurement, and for experienced researchers looking for current practice or for a citation that justifies a design they have already chosen. If you are in a mentoring position, consider sharing it with your mentees — and consider [[contributing]]: even a small edit that backs a subjective claim with a citation makes the site stronger. |
| |
| For novice researchers in the web measurement field, Measure The Web should provide a complete knowledge base to conduct their study. But it should be helpful also to experienced researchers, who can find here newest practices or proper arguments (or citations) to justify their design. If you are in a mentoring position, consider sharing the website (or directly [[Research journey]]) to your mentees and also consider reviewing and contributing to the page. Even small edits, such as supporting some subjective claim, can help make this website stronger. | ===== Start here ===== |
| |
| | ^ If you… ^ Start at ^ |
| | | have a study idea and no instrument yet | [[Design:Automated measurements]] — crawl, scan, or unpack an app, and what each can answer | |
| | | have data and need to turn it into labels | [[Design:Website classification]] for the protocol; [[Privacy]] or [[Security]] for the axis | |
| | | have results and need to defend them | [[Statistics:Biases]] — what your frame, vantage and denominator already did to the number | |
| | | want to know what order to do this in | [[Research journey]] — the decisions that are expensive to revisit later | |
| |
| ===== Outline ===== | ===== The seven namespaces ===== |
| |
| The website is organized as follows. Note that it is ordered by website structure, for order by research design, navigate to [[Research journey]]. | Each namespace page carries its own annotated list of the pages inside it, what a student needs each one for, and where the namespace stops. That list lives there rather than here, so it stays current. |
| |
| * [[Design|Research design]] clarifies choices needed to conduct the measurements. Example pages: | ^ Namespace ^ What it is for ^ |
| * [[Design:Automated measurements]] — crawl, scan, or unpack an app; the instrument is the design decision — or [[Design:User studies]] (Mechanical Turk shuts 30 September 2026) | | [[Design]] | Choices made **before the crawler runs**: which list stands in for the web, how you draw from it, where you appear to be, live or archived, and what you hold fixed between waves. | |
| * [[Design:Website selection]] and [[Design:Sampling|Representative sampling methods]], [[Design:Website classification]], and [[Design:IP classification]] | | [[Programming]] | **The instrument**: which browser, which control protocol, what happens after ''page.goto'', whether the profile survives, and what file you write. | |
| * [[Design:Crawling location]] and [[Design:Archives|Crawling live or using archives]] ([[Programming:Traffic files|using traffic files, e.g., HAR]]) | | [[Privacy]] | **Classifying what a crawl observed** on the privacy axis — requests, cookies, scripts, fingerprints, syncing, consent — and the two ways the instrument lies to you. | |
| * [[Design:DNS|Measuring DNS]] — which resolver answered, whether the answer was manipulated, and what encrypted DNS changed. Three public resolvers gave completely disjoint answers for 28 of the Tranco top 100 on 2026-08-27. | | [[Security]] | **Web security as it is deployed**, not how to break it: TLS, headers, label sources for "malicious", phishing in the wild. | |
| * [[Design:Existing datasets|Measuring from an existing dataset]] — when the measurement is a query (Censys, a CT log, AndroZoo, last year's crawl) rather than a crawl you ran: which snapshot, which query, which join key, and what a producer's port set and vantage point do to your denominator | | [[Statistics]] | **What these methods do to web-measurement data**: the unit-of-analysis error, the hypothesis family a crawl invents, hand-coded ground truth. Not a textbook. | |
| * [[Design:Longitudinal|Repeating a measurement over time]] — pinning the list, the browser, the vantage point and the classifier so wave two is comparable to wave one | | [[Practices]] | **Around the study rather than inside it**: ethics review, notifying operators, regulators, and the sentence that gets quoted once the paper is public. | |
| * [[Design:Mobile and app measurement|Measuring mobile apps]] — store scraping, static and dynamic analysis, and the certificate-pinning problem that decides whether you saw the traffic at all | | [[Writing]] | **Where the paper goes**, and how the related-work section is allowed to count. | |
| * [[Design:Platforms|Research of specific large platforms]] — when the population is one platform rather than a sample of the web, the **access route is the design decision** and it fixes your denominator. Which routes still exist in 2026, what they cost, and why there are no per-company sub-pages. | |
| * [[Programming]] | Outside the seven: [[Artifacts]] — a GitHub URL is not an archival artifact — and [[literature:bibliography|the bibliography]] every citation on this site resolves against. |
| * [[Programming:Crawler|Comparison of crawling libraries]] such as [[Programming:Crawler:OpenWPM]], [[Programming:Crawler:Tracker Radar Collector]], [[Programming:Crawler:PageGraph]], etc. | |
| * [[Programming:Crawler Detection|When the website notices your crawler]] — bot management, headless and CDP leaks, CAPTCHAs and challenge pages, and what a blocked crawl does to the headline number | ===== How to read a page ===== |
| * [[Programming:Crawler:LLM Agents|Crawling with an LLM agent]] — Browser Use, BrowserGym/AgentLab and the Computer-Use APIs as a measurement instrument: what five 2026 papers measured about completion rates and per-site cost, what nobody has measured about run-to-run variance, and why this is not yet the default | |
| * [[Programming:Multilingual support]] | Pages whose figures come from the publication corpus carry a working log under ''provenance:'', mirroring the page id — [[provenance:privacy:fingerprinting]] for [[Privacy:Fingerprinting]]. Each log holds every query with its denominator, the report script and its unedited output, the name-folds and their unmapped residue, the review passes, and what could not be established. |
| * [[Programming:Interaction|Interaction with websites]] | |
| * [[Programming:Registration|Automating login and registration]] | The corpus is seven venues — CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P — 2010–2026, **5,859** extracted papers. **Every figure is about the population its query defines**: only **1,120** of those papers ran a crawl, **1,762** ran statistical inference, **402** assess a law. Two pages quoting different populations correctly are not in conflict. 2025 is thin and 2026 is provisional; the corpus-wide caveats are on [[Literature:Corpus]]. |
| * [[Programming:Deployment|Running a measurement for weeks]] — the crawl as a process that must survive: checkpoints, retries, monitoring, cost, and what to report when the machine dies on day four | |
| * Multitude of pages linked from elsewhere documenting specific technologies, e.g., [[Programming:Tranco]], [[Programming:Cloudflare Radar]], [[Programming:Docker]], [[Programming:CrUX]], [[Programming:Similarweb]]. | ===== What is coming, and what was ruled out ===== |
| * [[Privacy]] | |
| * Classifying [[Privacy:Requests|Web requests]], [[Privacy:Cookies]], [[Privacy:Fingerprinting]], or [[Privacy:JavaScript]] | [[Roadmap]] lists the pages that are scoped and queued but not yet written, with the evidence behind each one — and the proposals that were assessed and **deliberately** not written, with their paper counts, so the next gap pass does not re-propose them. |
| * [[Privacy:Server side tracking|Measuring server-side tracking]] — when the tracking request never reaches the tracker, and every request-level method quietly stops working | |
| * [[Privacy:Cookie syncing|Measuring cookie and ID syncing]] — how third parties learn that their two identifiers are the same person, and the crawl configuration that decides whether you see it at all | |
| * [[Privacy:consent|Granting consent to websites]] | |
| * [[Privacy:TCF consent strings|Decoding TCF consent strings]] — the IAB TC string and Google's Additional Consent string as measurement artefacts, and what a decoded string does and does not prove | |
| * [[Privacy:darkpatterns|Dark patterns]] in the interfaces you measure — and in the way they defeat your crawler | |
| * [[Security]] — measuring web security as it is deployed, not how to break it: | |
| * [[Security:TLS certificates|TLS and the web PKI]] | |
| * [[Security:Headers|CSP, HSTS and the other security headers]] | |
| * [[Security:VirusTotal|VirusTotal as a label source]] | |
| * [[Security:Phishing]] | |
| * [[Security:Web vulnerabilities|Measuring web vulnerabilities in the wild]] | |
| * [[Statistics]]: [[Statistics:Study preregistration]], [[Statistics:Hypothesis testing]] suitable for web measurements, [[Statistics:Regression]], [[Statistics:Pvalue corrections|P-value corrections]], [[Statistics:Biases]], [[Statistics:Interrater agreement|Inter-rater agreement]] for hand-coded data | |
| * [[Writing]] | |
| * [[Writing:Conferences]] — where to send a web-measurement paper, and the current CFP dates. | |
| * [[Writing:Literature review]] — the keyword-derived denominator trap, measured on the SoKs in this corpus. | |
| * [[Artifacts]] — a GitHub URL is not an archival artifact. What to put in a crawl deposit, where 2026 Available badges will accept it, and what to withhold when the data has people in it. | |
| * Pages whose figures come from the publication corpus carry a working log under ''provenance:'' — every query with its denominator, the report script and its unedited output, the folds and their residue, and what could not be established. The corpus-wide caveats are on [[Literature:Corpus]]; the per-page logs mirror the page id, e.g. [[provenance:privacy:fingerprinting]] for [[Privacy:Fingerprinting]]. | |
| * After publication steps to improve the impact of your work: | |
| * [[Practices:Notifying websites]] to remediate the security vulnerabilities and privacy threats. | |
| * [[Practices:Legal enforcement]] with information about local data protection authorities. | |
| * [[Practices:Public relations]] regarding how to present your results to public and journalists. | |
| * [[Practices|Other research practices]] explains various aspects affiliated to the research, but not directly the study design. | |
| * [[Practices:Ethics]] | |
| * [[https://github.com/pliang279/awesome-phd-advice|PhD and research advice]] | |
| * [[https://www.khstats.com/blog|Statistics]] | |
| |
| ===== Contributing ===== | ===== Contributing ===== |
| |
| We welcome contributions! Editing functionality is limited to registered users. You might want to check [[Contributing]] page, which helps with the syntax. | We welcome contributions. Editing is limited to registered users; [[Contributing]] covers the syntax, the namespace structure, and the rule that every published figure names its query. External reading we point at rather than duplicate: [[https://github.com/pliang279/awesome-phd-advice|PhD and research advice]] and [[https://www.khstats.com/blog|khstats]] on statistics. |
| |
| /* | /* |
