The other pages on this site are organised by what a thing is. This one is organised by when you have to decide it, and it only lists decisions that are expensive or impossible to revisit later. A choice you can change on a Tuesday afternoon is not here; a choice that costs you a re-crawl, a re-annotation, or a paragraph in Limitations is.

This page is about automated measurements. If your study recruits people rather than crawling sites, User studies is the entry point, and the ordering below mostly does not apply — but the first three items still do.

Everything here is either impossible to add retroactively, or only credible if it predates the data.

1. Ethics approval — the review that has to happen before the crawl, not before the submission. Some venues now ask what your board said.
2. Preregistration — the only way a confirmatory claim stays confirmatory. A crawl generates hypotheses as fast as it generates rows; a plan deposited afterwards cannot tell the two apart.
3. Literature review — done now it tells you what to measure; done at writing time it tells you what you should have measured. Watch the keyword-derived denominator: the set of papers your search returns is not the set of papers that exist.
4. What you will be able to release — decide before collection, because consent, withholding and the shape of the deposit all constrain what you may record. Retrofitting a releasable dataset onto a finished crawl is the common way a badge is lost.

These four decisions define the population every later number is about. Changing any of them invalidates the data you already have.

1. Which list stands in for the web — and which version of it. Record the list id on the day you drew it; Tranco ids are the reason this is possible at all.
2. How you draw from that list, and how many. Top-n and stratified answer different questions, and the size you need depends on the effect, not on what fits in the week.
3. Where you appear to be — country, and datacenter versus residential. A single-vantage result is a result about one path, and the wrong vantage silently hands you the wrong legal regime.
4. Crawl, scan or unpack — the instrument is the design decision. Then which crawler, remembering that the instrument's own defaults are choices you are making.

Each of these is cheap to set and expensive to have set wrongly, because the answer is baked into every row you collect. Reviewers ask about all four.

1. Stateful or stateless — whether the profile survives between page loads. No paper in this corpus runs a controlled comparison of the two, so you cannot cite your way out of stating what you did.
2. What the crawler did with the consent banner — accepted, rejected, ignored, or never saw one. On the audited corpus only 4.9% of crawling papers actually state this (55 of 1,120; see consent section 7.3, where the unaudited figure was 31.2%); it changes what the crawl observed more than most instrument choices do.
3. How deep you go, and what you touch — landing page only, n subpages, or scripted interaction. Depth is a population decision disguised as an implementation detail.
4. What you record — requests, cookies, storage, script calls, a HAR. You can always analyse less than you recorded and never more; Browser storage is the usual thing people wish they had captured.

1. Pinning for wave two — list, browser, vantage and classifier. All four have to be pinned and recorded in wave one, or wave two measures your toolchain instead of the web.
2. Surviving the run — checkpoints, retries, monitoring, and what you will report when the machine dies on day four. Decide the failure accounting before the failures, so a partial crawl stays analysable.

Still not fully reversible: re-annotating is a month, and a validation set has to be drawn honestly rather than chosen.

1. Labelling, then validating the labels — with humans, a classifier, or an LLM. Corpus-wide, 42.4% of the 4,439 papers that classified something report no validation at all (see website_classification); inter-rater agreement is what makes hand-coded ground truth a measurement.
2. Your unit of analysis — sites, requests, or third parties. Getting it wrong is the most common statistical error in this literature, and it is not fixable at revision time.
3. What the frame already did to the number — top-n, one vantage, a missing denominator. Better named in your own Limitations than in a review.

1. Disclosure — to site operators or vendors, and on what timeline.
2. Enforcement — taking a finding to a data protection authority, which is a different document from a paper.
3. Public relations — the sentence a journalist will quote is one you can choose in advance.
4. Where to submit, and the current CFP dates. External advice on rebuttals: Ankit Singla (computer systems, closest to the security and privacy style), Michael Hicks (programming languages), Matt Might (bioinformatics, long rebuttals); on attending the conference, Michael Ernst.

It is not a table of contents — start and the seven namespace pages carry those, and a step here links one page rather than every page that touches the step. Several of these decisions recur (you will choose a vantage point again in wave two); they are listed at the point where getting them wrong costs the most.