Differences

This shows you the differences between two versions of the page.

--- design:longitudinal [2026/08/27 02:40] – Fix relative-namespace link: [[Artifacts]] resolved to design:artifacts. Authored by Claude karel.kubicek.claude
+++ design:longitudinal [2026/09/11 11:21] (current) – Point the noise-floor section at Design:Algorithm audits for the within-wave, across-arms version of the same question. Authored by Claude karel.kubicek.claude
@@ Line 9: / Line 9: @@
 </WRAP>
-Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Of those 250, **23 (9.2%)** state all four of the list version, the browser, the vantage point and the statefulness of the profile — and the papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
+Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Reading all 250, only **124** turn out to be the authors' own repeated crawl of the web — so the true number of repeated crawls is unknown in both directions, since 250 also misses every paper that repeated without saying so. Of the 124, **15 (12.1%)** plausibly held all four of the list, the browser, the vantage point and the statefulness fixed across their waves. //That reading was done by language-model agents, not people; see the caveats below before quoting the figure.// The papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
 ===== What to Read First =====
@@ Line 38: / Line 38: @@
 > studies that analyze the ecosystem will find similar results, while studies that aim to analyze the extent of a tracking phenomenon might see different results based on the measurement day
+The same question asked //within// one wave, across arms rather than across days, is [[Design:Algorithm audits]]: if two profiles that differ in nothing land this far apart, a difference between two profiles that //do// differ has to clear that gap first.
 Read as design advice: **in that experiment the set-valued outcome survived repetition and the count-valued one did not.** The entity set — which tracking domains are present — moved 3.5% across twelve identical days. The event count — how many tracking requests were made — moved 27%. That is one study on one pair of outcomes, so it does not license a general law; what it does license is a design question you should answer before you crawl. If your outcome is a count and a 27%-scale wobble would swamp the effect you are looking for, you need either many more waves than you planned, or a paired within-site design that differences the day out, or a different outcome.
@@ Line 66: / Line 68: @@
 | 1 Jan 2022 | ''XVWN'' | Alexa, Umbrella, Majestic |
 | 1 Jan 2023 | ''3VYXL'' | Alexa, Umbrella, Majestic, Farsight |
+| **31 Jul 2023** | ''W9ZN9'' | Alexa, Umbrella, Majestic, Farsight |
+| **1 Aug 2023** | ''25299'' | CrUX, Farsight, Majestic, Radar, Umbrella |
 | 1 Jan 2024 | ''V929N'' | CrUX, Farsight, Majestic, Radar, Umbrella |
 | 26 Aug 2026 | ''46W9X'' | CrUX, Farsight, Majestic, Radar, Umbrella |
-Quantcast dropped out, Farsight came in, and then in 2023–24 Alexa was replaced by CrUX and Cloudflare Radar.((All five rows read from the Tranco API on 2026-08-27: ''%%https://tranco-list.eu/api/lists/date/YYYYMMDD%%'' returns the list id and its ''configuration.providers'' for that day. Alexa's disappearance is forced — Amazon retired the Alexa Top Sites API on 15 December 2022, see [[Design:Sampling]].)) So "we used the Tranco top 1M in both waves" does **not** give you a fixed frame across 2022–2024: the default list is a different instrument on either side. Tranco lets you fix the provider set when you generate a custom list, but it cannot resurrect a retired provider, so there is no configuration that makes a 2019 list and a 2026 list commensurable.
+Quantcast dropped out between 14 and 15 August 2020, Farsight came in on 1 May 2022, and Alexa was replaced by CrUX and Cloudflare Radar on **1 August 2023**. That last change is the sharpest form of the problem. ''W9ZN9'' and ''25299'' are **consecutive daily lists, one day apart, built from different providers** — and the default prefix changed from ''full'' to one million on the same day.((Rows read from the Tranco API, ''%%https://tranco-list.eu/api/lists/date/YYYYMMDD%%'', which returns a day's list id and its ''configuration'': the other five rows on 2026-08-27, the 31 July / 1 August 2023 pair on 2026-09-03. ''date/20230731'' gives ''W9ZN9'', ''providers: [alexa, umbrella, majestic, farsight]'', ''listPrefix: full''; ''date/20230801'' gives ''25299'', ''providers: [crux, farsight, majestic, radar, umbrella]'', ''listPrefix: 1000000''. The Quantcast date was narrowed by bisecting the same endpoint on 2026-09-07: ''date/20200814'' gives ''GK9K'', ''providers: [alexa, umbrella, majestic, quantcast]''; ''date/20200815'' gives ''Z7NG'', ''providers: [alexa, umbrella, majestic]''. An earlier version of this page said //"after 1 April 2020"//, which was only as precise as the sampled rows in the table above. The date is also Tranco's own, on its front page: //"The Chrome User Experience Report and Cloudflare Radar rankings have been integrated into the default Tranco list, starting from the daily updated list of August 1, 2023"// (fetched 2026-09-03). It is **Tranco's** date, not Amazon's — ''alexa.com'' was retired on 1 May 2022 and the Alexa Top Sites and Web Information Service APIs on 15 December 2022 — so Alexa's removal from the list was forced eight months earlier than it happened. Retirement dates and sources: [[Design:Sampling]]. [[Programming:Tranco]] has the id table and the API; the pre-swap id ''W9ZN9'' is only here.)) So "we used the Tranco top 1M in both waves" does **not** give you a fixed frame across 2022–2024: the default list is a different instrument on either side. Tranco lets you fix the provider set when you generate a custom list, but it cannot resurrect a retired provider, so there is no configuration that makes a 2019 list and a 2026 list commensurable.
 That leaves two honest options, and you should say which you took:
@@ Line 106: / Line 110: @@
   * **Chrome.** [[https://developer.chrome.com/docs/automation-and-testing/chrome-for-testing|Chrome for Testing]] publishes immutable, non-auto-updating builds of Chrome and the matching ChromeDriver, addressable by exact version, with a JSON endpoint listing every one. As of 2026-08-27 it lists **2,484 versions from 113.0.5672.0 to 154.0.8026.0**, so any Chrome from the 113 line (spring 2023) onward can be re-instantiated exactly; anything older is not available through this channel.((Read from ''%%https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json%%'' on 2026-08-27; the ''last-known-good-versions.json'' endpoint on the same day reported Stable 152.0.7977.64.)) It is the correct default for any new repeated crawl, and it costs one line of setup.
-  * **Firefox.** Every release is retained at ''archive.mozilla.org/pub/firefox/releases/''. If you drive Firefox through [[Programming:Crawler:OpenWPM|OpenWPM]] you get the pin for free and should cite it: OpenWPM installs a specific **unbranded** Firefox build identified by a Mozilla release tag, and each OpenWPM release states which. v0.36.0 (24 August 2026) pins Firefox 154.0.((''%%https://github.com/openwpm/OpenWPM/blob/master/scripts/install-firefox.sh%%'' pins ''%%TAG='9ce1ee6baeb9a3c326dbd180bdece65d8fc2eadc' # FIREFOX_154_0_RELEASE%%''; release notes for v0.36.0 read "Firefox 152 → 154.0, with the Extension manifest's ''strict_min_version'' pinned to match". Both checked 2026-08-27.)) So "OpenWPM v0.36.0" **is** a browser version statement, and "OpenWPM" alone is not. Demir et al. write it the right way: //"We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)"// {[demir2022_reproducibility]}.
+  * **Firefox.** Every release is retained at ''archive.mozilla.org/pub/firefox/releases/''. If you drive Firefox through [[Programming:Crawler:OpenWPM|OpenWPM]] you get the pin for free and should cite it: OpenWPM installs a specific **unbranded** Firefox build identified by a Mozilla release tag, and each OpenWPM release states which. v0.37.0 (6 September 2026) pins Firefox 155.0.1.((''%%https://github.com/openwpm/OpenWPM/blob/master/scripts/install-firefox.sh%%'' pins ''%%TAG='5fdfd0092780e85643e2cddc0e1b590c8b9ef860' # FIREFOX_155_0_1_RELEASE%%''; the v0.37.0 release notes read "**Firefox 154 → 155.0.1**, with the Extension manifest's ''strict_min_version'' pinned to match". Both checked 2026-09-07 via ''%%api.github.com/repos/openwpm/OpenWPM/releases/latest%%''. This sentence named v0.36.0 / Firefox 154.0 when the page was written on 2026-08-27; v0.37.0 shipped ten days later, which is the rate of drift the paragraph is about.)) So "OpenWPM v0.37.0" **is** a browser version statement, and "OpenWPM" alone is not. Demir et al. write it the right way: //"We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)"// {[demir2022_reproducibility]}.
-  * **Playwright already does this for you, if you pin Playwright.** Its ''browsers.json'' pins an exact browser build per Playwright release — the Chromium entry is literally titled "Chrome for Testing" and carries a ''browserVersion'', and Firefox and WebKit carry an internal ''revision''.((''%%https://raw.githubusercontent.com/microsoft/playwright/main/packages/playwright-core/browsers.json%%'', checked 2026-08-27: ''%%{"name": "chromium", "revision": "1241", "browserVersion": "152.0.7977.54", "title": "Chrome for Testing"}%%''.)) So a pinned Playwright version is a browser pin, in the same way an OpenWPM release tag is — but only if you pin Playwright rather than installing whatever npm resolves.
+  * **Playwright already does this for you, if you pin Playwright.** Its ''browsers.json'' pins an exact browser build per Playwright release — the Chromium entry is literally titled "Chrome for Testing" and carries a ''browserVersion'', and Firefox and WebKit carry an internal ''revision''.((Read from the **tagged release**, not from ''main'': ''%%https://raw.githubusercontent.com/microsoft/playwright/refs/tags/v1.63.0/packages/playwright-core/browsers.json%%'', fetched 2026-09-07, gives ''%%{"name": "chromium", "revision": "1243", "browserVersion": "153.0.8010.12", "title": "Chrome for Testing"}%%''. An earlier version of this footnote quoted ''main'' — ''revision 1241'', ''browserVersion 152.0.7977.54'' — which is a development snapshot that shipped in no release, and so was the wrong evidence for a claim about what a **pinned Playwright version** gives you.)) So a pinned Playwright version is a browser pin, in the same way an OpenWPM release tag is — but only if you pin Playwright rather than installing whatever npm resolves.
   * **The driver too.** A geckodriver or ChromeDriver upgrade can change what the browser accepts; OpenWPM's own v0.36.0 notes a geckodriver 0.37.1 change that made every browser fail to launch. Record the driver version alongside the browser.
@@ Line 159: / Line 163: @@
   * **[[Programming:CrUX|CrUX]]** — monthly, origin-level, real-user sourced. It is the frame under both HTTP Archive and modern Tranco, so it is not an independent check on either.
   * **[[Design:Archives|The Wayback Machine and Common Crawl]]** — the only way to measure before you started. Read that page before using either: the replay problems are severe and specific.
-  * **Censorship observatories** — Censored Planet {[raman2020_censored]} and ICLab {[niaki2020_iclab]} are continuous, longitudinal and public, and they are the model for what a maintained observatory looks like in this field.
+  * **Censorship observatories** — Censored Planet {[raman2020_censored]} is continuous, longitudinal and public, and is the model for what a maintained observatory looks like in this field. ICLab {[niaki2020_iclab]} is the other standard reference and its **paper** remains worth reading, but the observatory itself looks dormant: as of 2026-09-07 ''iclab.org'' serves an expired TLS certificate (''curl'' fails without ''-k''), its data page returns 404, and its news stops in 2019. Cite the paper; do not plan on the feed.
 <WRAP important>
@@ Line 169: / Line 173: @@
 The figures below come from a structured extraction over **5,859 full-text papers** from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Every block names its own population; sentinel values (''not-stated'', ''not-applicable'') are counted as silence and never as an answer. The 2025 and 2026 venue-years are provisional — CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source — so they are under-represented by construction. Methodology and limitations are at the end of this section; the full query log is at [[provenance:design:longitudinal]].
-Throughout, a paper is **repeated** when it ran a crawl and any stated measurement period has more than one snapshot. That is the same rule [[Statistics:Biases]] uses, so the two pages agree on the 250.
+Throughout, a paper is **repeated** when it ran a crawl and any stated measurement period has more than one snapshot. That is the same rule [[Statistics:Biases]] uses, so the two pages agree on the 250. **That rule is right about half the time** — hand-reading all 250 finds 124 that are the authors' own repeated crawl of the web (see //Were the pins actually held fixed?// below). Figures below are shares of 250 unless they say otherwise, and the 124 is the population to use when the question is specifically about comparability between waves.
 ==== Most crawls happen once, and most do not say ====
@@ Line 231: / Line 235: @@
 The second row is a finding, not a bug in the fold. **58 papers answer "how often?" with "for three weeks"** — a duration, which does not tell a reader whether that was one crawl or twenty. Counting papers rather than values, so that a paper stating both a duration and a real cadence is not penalised, **88 of 389 (22.6%) fill the cadence slot without ever stating a cadence**: a duration, a bare round count, "periodically", or the sentinel itself.
-==== The four pins, measured ====
+==== The four pins: how often they are stated ====
-Population: the **250 repeated** crawling papers against the **870 that crawled once or did not say**. Each row asks only whether the paper stated the thing at all — the extraction cannot tell whether the value was the //same// in every wave, so every figure here is an upper bound on comparability.
+Population: the **250 repeated** crawling papers against the **870 that crawled once or did not say**. Each row asks only whether the paper stated the thing at all. Whether the value was the //same// in every wave is a different question, and it is answered by hand in the next section.
 ^ Pin ^ Repeated (of 250) ^ Other crawls (of 870) ^
@@ Line 254: / Line 258: @@
   * **6 of 250 (2.4%)** do it with a browser //version// rather than a browser name.
   * **28 of 250 (11.2%)** state none of the four.
+==== Were the pins actually held fixed? ====
+The table above counts papers that //stated// a value. Stating it once says nothing about wave two, so on its own it is only a ceiling. To turn the ceiling into an estimate, **all 250 repeated-crawl papers were read by hand** against a published codebook: for each of the four pins, what does the paper say the value was in //each// wave? Every pin is coded as one of ''same-explicit'' (the paper says in words that it was unchanged), ''same-implied'' (one configuration is described covering the whole repeated measurement and never said to change), ''varied-deliberate'', ''changed'', ''not-stated'' or ''unclear'', each with a verbatim quote from the paper. A second coder blind-re-read a random 30 of the 250 from the full text. **Both coders and the quote adjudicator were language-model agents (Claude Sonnet), reading text that a language-model extraction had already been run over; no person read any of the 250 papers.** The agreement figures below measure consistency between two such agents, which is not the same thing as human inter-rater reliability, and should be discounted accordingly. The codebook, both coders' output, the agreement figures and the quote check are on [[provenance:design:longitudinal]].
+**The first finding is about the population, not the pins.** The 250 were selected mechanically — a crawling paper whose extraction records more than one snapshot. Reading all of them:
+^ What the "more than one snapshot" really is ^ Papers ^ Share of 250 ^
+| The authors re-crawled the web over time | 124 | 49.6% |
+| Repeated, but not a web crawl (DNS probes, app stores, API pulls) | 44 | 17.6% |
+| Parallel conditions at one time, not waves | 30 | 12.0% |
+| Time points come from an archive or somebody else's dataset | 21 | 8.4% |
+| One crawl — the snapshot count is a mis-extraction | 31 | 12.4% |
+| Undecidable after reading | 0 | 0.0% |
+**Half of the papers a "snapshots > 1" filter returns are not repeated web crawls.** They repeat something the four pins do not apply to, they run parallel conditions at one moment and call them snapshots, they take their time points from an archive somebody else collected, or they crawled once and had a page count land in the snapshot field. If you are building your own related-work set from a structured field, budget for this: **a mechanical repeat-crawl predicate was right about half the time here**, and the same caveat applies to every figure on this page and on [[Statistics:Biases]] that is a share of 250.
+Over the **124 papers that really are the authors' own repeated web crawl**:
+^ Pin ^ Held fixed, stated in words ^ Held fixed, estimate ^ Stated at all ^ Moved between waves ^ Never stated ^
+| Population list | 48 (38.7%) | **95 (76.6%)** | 86 (69.4%) | 28 (22.6%) | 1 (0.8%) |
+| Browser | 14 (11.3%) | **69 (55.6%)** | 83 (66.9%) | 10 (8.1%) | 44 (35.5%) |
+| Vantage point | 26 (21.0%) | **45 (36.3%)** | 50 (40.3%) | 9 (7.3%) | 69 (55.6%) |
+| Statefulness | 16 (12.9%) | **38 (30.6%)** | 45 (36.3%) | 2 (1.6%) | 81 (65.3%) |
+| **All four** | 2 (1.6%) | **15 (12.1%)** | 15 (12.1%) | 42 (33.9%) | 0 (0.0%) |
+//"Stated at all" is the structured extraction's figure, restricted to these 124. On the All-four row the last two columns read differently from the others: "moved" is at least one pin moving, "never stated" is none of the four appearing at all. The same table over all 250 is in the [[provenance:design:longitudinal]] report output.//
+The numbers to carry away:
+  * **15 of 124 (12.1%)** plausibly held all four fixed. Two are coded ''same-explicit'' on every pin — but a review pass found that ''same-explicit'' was applied more loosely than the codebook defines it, often to a permanent list ID or a one-off construction rather than to a paper saying //unchanged//, so read the left-hand column as "pinned by a stated identifier //or// an explicit statement", not as "said so in words". The strict count is smaller than two and was not re-derived.
+  * **42 of 124 (33.9%)** have at least one pin that demonstrably moved — refreshed, updated, or varied on purpose.
+  * The **estimate** column is the middle case and it is generous: ''same-implied'' means a paper described one setup and never mentioned changing it, which is what most methods sections look like whether or not anything moved. Treat it as an upper-middle bound, not a measurement.
+  * **The list is the pin the field does best and the browser the one it does worst relative to how fast it moves.** 95 of 124 plausibly held the population fixed; only 14 carry explicit browser evidence at all, against a browser release cycle of six weeks.
+**Where "stated" and "held fixed" diverge, and in which direction.** For the browser, vantage and statefulness the hand-coded estimate is //below// the stated figure — a paper names a browser and then compares two of them, or names a vantage point for wave one only. For the population list it is //above// it: 62 of the 250 describe a fixed frame in prose without ever giving the version string the extraction looks for ("the same 200 sites", "one Tranco list ID"), so the structured field understates the list pin. The two columns measure different things and the gap is not an error in either.
+**Does the longer panel still pin less?** The direction is consistent but the counts are too small to call. Over the 124, **1 of 33 (3.0%)** studies spanning twelve months or more plausibly held all four fixed, against **12 of 77 (15.6%)** under twelve months — Fisher's exact //p// = 0.10. The same split over the 250 using stated values gives //p// = 0.08 and the live-crawl-only split //p// = 0.69. Read it as a description that points one way in three cuts, not as a difference. Fourteen have no parsable end date and drop out.
+**By period,** over the same 124: 0 of 2 (2010–2011), 0 of 14 (2012–2015), 3 of 32 (2016–2019), 8 of 41 (2020–2023), 4 of 35 (2024–2026, provisional). The practice of pinning all four appears somewhere around 2016, peaks in the 2020–2023 bucket at roughly one repeated crawl in five, and the last bucket is both lower and incomplete. Read it as "no established norm", not as a decline.
+**How reliable is the coding?** On the 30 papers both coders read, they agreed on 71.7% of the 120 individual pin values exactly (Cohen's //κ// = 0.60) and on 81.7% of the fixed-versus-not collapse (//κ// = 0.64); on whether a paper was really the authors' own repeat crawl, 90.0% (//κ// = 0.80). The disagreements are not symmetric: the coder working from an excerpt called the //browser// fixed seven times where the full-text coder found nothing stated, and only once the other way. So the browser row above is more likely too high than too low. Every disagreement is printed in full on [[provenance:design:longitudinal]].
 ==== The longer the panel, the less it pins ====
@@ Line 315: / Line 361: @@
 By year the ID rate is 1/7 (2020), 2/28, 2/29, 5/48, 4/56, 8/76, 1/22 (2026) — noisy, small, and with no trend. Permanent IDs have been available since Tranco's 2019 paper {[LePochat2019_tranco]}. The 23 were found by two wide probes and confirmed by reading all 37 candidates by hand; the list is in [[provenance:design:longitudinal]].
-==== Papers that pin all four ====
+==== Papers that stated all four, and what they actually did ====
+The 23 repeated-crawl papers whose extraction records all four of list version, browser, vantage location and statefulness. This table used to be presented as "the methods sections to copy from". Hand-reading each one, **6 of the 23 plausibly held all four fixed across their waves.** The other 17 are not badly reported — most state more than the field average — they simply are not the thing the row claimed: **seven** turn out not to be repeated crawls at all (they run parallel conditions at one moment), and the remaining **ten** are real repeat crawls that deliberately vary a pin, change one between waves, or describe it for one wave only.
-The 23 repeated-crawl papers that state list version, browser, vantage location and statefulness. These are the methods sections to copy from. Eight of the 23 are PETS papers from 2024 onward, which is where most of the recent repeated web-privacy measurement is appearing.
+^ Year ^ Venue ^ Paper ^ Really a repeat crawl? ^ List ^ Browser ^ Vantage ^ State ^ All four fixed? ^
+| 2026 | PETS | Google click identifier in YouTube {[dao2026_gclid]} | parallel, not waves | fixed, implied | varied by design | varied by design | varied by design | no |
+| 2026 | PETS | IP-based website fingerprinting in IPv6 {[ahmad2026_more]} | own repeat crawl | fixed, in words | fixed, implied | fixed, in words | fixed, in words | **yes** |
+| 2026 | PETS | Manifest V3 and ad-blocker effectiveness {[lukic2026_mv3]} | own repeat crawl | fixed, implied | varied by design | fixed, in words | fixed, implied | no |
+| 2025 | PETS | HTTP response headers for cross-browser tracker detection {[rieder2025_beyond]} | own repeat crawl | **changed** | fixed, implied | unclear | unclear | no |
+| 2025 | PETS | Intractable cookie crumbs {[rasaii2025_crumbs]} | own repeat crawl | fixed, in words | varied by design | fixed, in words | varied by design | no |
+| 2025 | PETS | More and scammier ads {[mai2025_more]} | own repeat crawl | varied by design | fixed, implied | fixed, in words | fixed, implied | no |
+| 2025 | USENIX | Navigating cookie consent violations across the globe {[tang2025_navigating]} | own repeat crawl | fixed, in words | fixed, implied | fixed, in words | fixed, implied | **yes** |
+| 2025 | WWW | Before & After: the EU Code of Practice on Disinformation {[papadogiannakis2025_before]} | own repeat crawl | fixed, implied | not stated | not stated | not stated | no |
+| 2024 | IEEE S&P | Targeted and troublesome {[moti2024_targeted]} | parallel, not waves | fixed, implied | varied by design | varied by design | fixed, implied | no |
+| 2024 | PETS | Cookie banner interaction tools {[demir2024_bannertools]} | parallel, not waves | fixed, in words | fixed, in words | varied by design | fixed, in words | no |
+| 2024 | PETS | The devil is in the details: server-side tracking {[fouad2024_devil]} | own repeat crawl | fixed, implied | **changed** | fixed, in words | fixed, in words | no |
+| 2023 | USENIX | Multi-factor and risk-based authentication availability {[gavazzi2023_multi]} | own repeat crawl | fixed, in words | varied by design | varied by design | fixed, in words | no |
+| 2022 | IMC | Respect the ORIGIN! {[singanamalla2022_respect]} | own repeat crawl | fixed, in words | **changed** | fixed, in words | fixed, in words | no |
+| 2022 | USENIX | Geodifferences in mobile apps {[kumar2022_investigation]} | parallel, not waves | fixed, implied | fixed, implied | varied by design | fixed, in words | no |
+| 2022 | WWW | Reproducibility and replicability {[demir2022_reproducibility]} | own repeat crawl | fixed, implied | not stated | not stated | unclear | no |
+| 2020 | NDSS | Encrypted DNS and traffic analysis {[siby2020_encrypted]} | own repeat crawl | fixed, implied | fixed, implied | fixed, implied | fixed, implied | **yes** |
+| 2020 | PETS | Missed by filter lists {[fouad2020_missed]} | parallel, not waves | fixed, implied | fixed, implied | fixed, implied | fixed, implied | **yes** |
+| 2020 | WWW | Beyond the front page {[urban2020beyond]} | parallel, not waves | fixed, in words | fixed, in words | varied by design | fixed, implied | no |
+| 2020 | WWW | Representativeness of automated web crawls {[zeber2020representativeness]} | own repeat crawl | fixed, in words | fixed, in words | fixed, in words | fixed, implied | **yes** |
+| 2019 | WWW | Political personalization of Google News search {[le2019_political]} | own repeat crawl | fixed, implied | fixed, implied | fixed, in words | fixed, implied | **yes** |
+| 2017 | NDSS | Thou shalt not depend on me {[lauinger2017_thou]} | parallel, not waves | varied by design | fixed, implied | fixed, in words | fixed, in words | no |
+| 2016 | IEEE S&P | Cloak of visibility {[invernizzi2016_cloak]} | parallel, not waves | fixed, in words | varied by design | varied by design | fixed, in words | no |
+| 2016 | USENIX | Tracing information flows between ad exchanges {[bashir2016tracing]} | own repeat crawl | fixed, in words | fixed, implied | fixed, in words | not stated | no |
-^ Year ^ Venue ^ Paper ^ Snapshots ^ Distinct stated vantage locations ^
+//"Fixed, in words" is the paper stating the value was unchanged; "fixed, implied" is one configuration described for the whole study and never said to change. "Varied by design" is not a defect — comparing browsers or vantage points across conditions is a legitimate design; it just means the pin is not held. Regenerate this table with %%``node scripts/report_pins_audit.mjs --wiki``%%; it is emitted from the coding files, not typed.//
-| 2026 | PETS | Google click identifier in YouTube {[dao2026_gclid]} | 5 | 6 |
-| 2026 | PETS | Manifest V3 and ad-blocker effectiveness {[lukic2026_mv3]} | 5 | 1 |
-| 2026 | PETS | IP-based website fingerprinting in IPv6 {[ahmad2026_ipfp]} | 5 | 1 |
-| 2025 | PETS | HTTP response headers for cross-browser tracker detection {[rieder2025_beyond]} | 18 | 1 |
-| 2025 | PETS | Intractable cookie crumbs {[rasaii2025_crumbs]} | 4 | 1 |
-| 2025 | PETS | More and scammier ads {[mai2025_more]} | 6 | 5 |
-| 2025 | USENIX | Navigating cookie consent violations across the globe {[tang2025_navigating]} | 10 | 8 |
-| 2025 | WWW | Before & After: the EU Code of Practice on Disinformation {[papadogiannakis2025_before]} | 2 | 3 |
-| 2024 | IEEE S&P | Targeted and troublesome {[moti2024_targeted]} | 7 | 5 |
-| 2024 | PETS | The devil is in the details: server-side tracking {[fouad2024_devil]} | 3 | 2 |
-| 2024 | PETS | Cookie banner interaction tools {[demir2024_bannertools]} | 3 | 5 |
-| 2023 | USENIX | Multi-factor and risk-based authentication availability {[gavazzi2023_multi]} | 2 | 5 |
-| 2022 | IMC | Respect the ORIGIN! {[singanamalla2022_respect]} | 2 | 1 |
-| 2022 | USENIX | Geodifferences in mobile apps {[kumar2022_investigation]} | 31 | 26 |
-| 2022 | WWW | Reproducibility and replicability {[demir2022_reproducibility]} | 12 | 3 |
-| 2020 | NDSS | Encrypted DNS and traffic analysis {[siby2020_encrypted]} | 5 | 3 |
-| 2020 | PETS | Missed by filter lists {[fouad2020_missed]} | 6 | 1 |
-| 2020 | WWW | Beyond the front page {[urban2020beyond]} | 3 | 3 |
-| 2020 | WWW | Representativeness of automated web crawls {[zeber2020representativeness]} | 2 | 2 |
-| 2019 | WWW | Political personalization of Google News search {[le2019_political]} | 7 | 2 |
-| 2017 | NDSS | Thou shalt not depend on me {[lauinger2017_thou]} | 2 | 1 |
-| 2016 | IEEE S&P | Cloak of visibility {[invernizzi2016_cloak]} | 3 | 1 |
-| 2016 | USENIX | Tracing information flows between ad exchanges {[bashir2016tracing]} | 9 | 1 |
-//Snapshot counts are the paper's own stated figure and are not directly comparable — one paper's "snapshot" is a daily crawl, another's is a yearly wave. Vantage locations are distinct stated places, sentinels excluded.//
+The six that survive are {[ahmad2026_more]}, {[tang2025_navigating]}, {[siby2020_encrypted]}, {[fouad2020_missed]}, {[zeber2020representativeness]} and {[le2019_political]}. One of the six, {[fouad2020_missed]}, is a parallel-condition study rather than a temporal one, so as //longitudinal// exemplars the list is five. Twelve further papers outside this 23 are estimated to hold all four fixed while never stating one of them in the form the structured extraction records — the full list of 18 is in the report output on [[provenance:design:longitudinal]].
 ==== Methodology and limitations of these figures ====
   * **Population.** ''crawled'' is the 1,120 papers with a crawl configuration or an ''automated-web-crawl'' study type — the same definition the corpus overview uses. ''repeated'' is those with a stated snapshot count above one.
-  * **"Stated" is not "held fixed".** Every pin figure counts papers that //mentioned// the thing. A paper can name a browser in wave one and silently upgrade it in wave two, and the extraction cannot see that. Treat the pin figures as ceilings.
+  * **"Stated" is not "held fixed".** The structured pin figures count papers that //mentioned// the thing; a paper can name a browser in wave one and silently upgrade it in wave two, and the extraction cannot see that. That gap is why all 250 were re-read by hand — see //Were the pins actually held fixed?// above. Where the two disagree, the hand-coded figure is the one about comparability and the structured figure is the one about reporting.
+  * **The hand-coding is one coder per paper, from an excerpt, with a blind second coder on 30.** Agreement was //κ// = 0.60 on exact pin values and 0.80 on whether a paper was really a repeat crawl; the excerpt coder over-calls a fixed browser. The estimate column is an estimate with a measured error rate, not a count.
+  * **The 15 were not individually re-read as a set.** A review pass sampling them found one, a 2020 WWW 5G-performance paper, whose own coding note says its repeated measurement is a cellular walk test rather than a web crawl — which makes it ''own-repeat-nonweb'' and would move the figure to 14 of 123. It was not re-coded before publication. Treat 15 as ±1.
+  * **The 250 is half false positives for "repeated web crawl".** 124 of 250 are the authors' own repeated crawl of the web. Every other figure on this page that is a share of 250 — cadence, span, probe hits — inherits that, and none of them has been recomputed on the 124.
   * **Snapshot counts are noisy at the top.** Nine papers report more than 1,000 snapshots and hand-checking finds mis-extracted page and record counts. Bands are used; raw maxima are not.
   * **Spans are the longest //stated// period on any measurement, which for a paper that also reuses an old dataset is not the span of its own repeated crawl.** This is why the multi-year group is contaminated with retrospective work, and it is reported alongside the mode breakdown rather than hidden.