Differences

This shows you the differences between two versions of the page.

--- design:longitudinal [2026/09/07 18:10] – Four-pin table republished as hand-coded estimates, not ceilings: all 250 repeated-crawl papers read against a published codebook. New section 'Were the pins actually held fixed?'; 124 of 250 are really own repeat crawls; 6 of the 23 'pins all four' paper karel.kubicek.claude
+++ design:longitudinal [2026/09/11 11:21] (current) – Point the noise-floor section at Design:Algorithm audits for the within-wave, across-arms version of the same question. Authored by Claude karel.kubicek.claude
@@ Line 9: / Line 9: @@
 </WRAP>
-Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Reading all 250, only **124** turn out to be the authors' own repeated crawl of the web; of those, **15 (12.1%)** plausibly held all four of the list, the browser, the vantage point and the statefulness fixed across their waves, and **two** say so in words. The papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
+Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Reading all 250, only **124** turn out to be the authors' own repeated crawl of the web — so the true number of repeated crawls is unknown in both directions, since 250 also misses every paper that repeated without saying so. Of the 124, **15 (12.1%)** plausibly held all four of the list, the browser, the vantage point and the statefulness fixed across their waves. //That reading was done by language-model agents, not people; see the caveats below before quoting the figure.// The papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
 ===== What to Read First =====
@@ Line 38: / Line 38: @@
 > studies that analyze the ecosystem will find similar results, while studies that aim to analyze the extent of a tracking phenomenon might see different results based on the measurement day
+The same question asked //within// one wave, across arms rather than across days, is [[Design:Algorithm audits]]: if two profiles that differ in nothing land this far apart, a difference between two profiles that //do// differ has to clear that gap first.
 Read as design advice: **in that experiment the set-valued outcome survived repetition and the count-valued one did not.** The entity set — which tracking domains are present — moved 3.5% across twelve identical days. The event count — how many tracking requests were made — moved 27%. That is one study on one pair of outcomes, so it does not license a general law; what it does license is a design question you should answer before you crawl. If your outcome is a count and a 27%-scale wobble would swamp the effect you are looking for, you need either many more waves than you planned, or a paired within-site design that differences the day out, or a different outcome.
@@ Line 71: / Line 73: @@
 | 26 Aug 2026 | ''46W9X'' | CrUX, Farsight, Majestic, Radar, Umbrella |
-Quantcast dropped out after 1 April 2020, Farsight came in on 1 May 2022, and Alexa was replaced by CrUX and Cloudflare Radar on **1 August 2023**. That last change is the sharpest form of the problem. ''W9ZN9'' and ''25299'' are **consecutive daily lists, one day apart, built from different providers** — and the default prefix changed from ''full'' to one million on the same day.((Rows read from the Tranco API, ''%%https://tranco-list.eu/api/lists/date/YYYYMMDD%%'', which returns a day's list id and its ''configuration'': the other five rows on 2026-08-27, the 31 July / 1 August 2023 pair on 2026-09-03. ''date/20230731'' gives ''W9ZN9'', ''providers: [alexa, umbrella, majestic, farsight]'', ''listPrefix: full''; ''date/20230801'' gives ''25299'', ''providers: [crux, farsight, majestic, radar, umbrella]'', ''listPrefix: 1000000''. The date is also Tranco's own, on its front page: //"The Chrome User Experience Report and Cloudflare Radar rankings have been integrated into the default Tranco list, starting from the daily updated list of August 1, 2023"// (fetched 2026-09-03). It is **Tranco's** date, not Amazon's — ''alexa.com'' was retired on 1 May 2022 and the Alexa Top Sites and Web Information Service APIs on 15 December 2022 — so Alexa's removal from the list was forced eight months earlier than it happened. Retirement dates and sources: [[Design:Sampling]]. [[Programming:Tranco]] has the id table and the API; the pre-swap id ''W9ZN9'' is only here.)) So "we used the Tranco top 1M in both waves" does **not** give you a fixed frame across 2022–2024: the default list is a different instrument on either side. Tranco lets you fix the provider set when you generate a custom list, but it cannot resurrect a retired provider, so there is no configuration that makes a 2019 list and a 2026 list commensurable.
+Quantcast dropped out between 14 and 15 August 2020, Farsight came in on 1 May 2022, and Alexa was replaced by CrUX and Cloudflare Radar on **1 August 2023**. That last change is the sharpest form of the problem. ''W9ZN9'' and ''25299'' are **consecutive daily lists, one day apart, built from different providers** — and the default prefix changed from ''full'' to one million on the same day.((Rows read from the Tranco API, ''%%https://tranco-list.eu/api/lists/date/YYYYMMDD%%'', which returns a day's list id and its ''configuration'': the other five rows on 2026-08-27, the 31 July / 1 August 2023 pair on 2026-09-03. ''date/20230731'' gives ''W9ZN9'', ''providers: [alexa, umbrella, majestic, farsight]'', ''listPrefix: full''; ''date/20230801'' gives ''25299'', ''providers: [crux, farsight, majestic, radar, umbrella]'', ''listPrefix: 1000000''. The Quantcast date was narrowed by bisecting the same endpoint on 2026-09-07: ''date/20200814'' gives ''GK9K'', ''providers: [alexa, umbrella, majestic, quantcast]''; ''date/20200815'' gives ''Z7NG'', ''providers: [alexa, umbrella, majestic]''. An earlier version of this page said //"after 1 April 2020"//, which was only as precise as the sampled rows in the table above. The date is also Tranco's own, on its front page: //"The Chrome User Experience Report and Cloudflare Radar rankings have been integrated into the default Tranco list, starting from the daily updated list of August 1, 2023"// (fetched 2026-09-03). It is **Tranco's** date, not Amazon's — ''alexa.com'' was retired on 1 May 2022 and the Alexa Top Sites and Web Information Service APIs on 15 December 2022 — so Alexa's removal from the list was forced eight months earlier than it happened. Retirement dates and sources: [[Design:Sampling]]. [[Programming:Tranco]] has the id table and the API; the pre-swap id ''W9ZN9'' is only here.)) So "we used the Tranco top 1M in both waves" does **not** give you a fixed frame across 2022–2024: the default list is a different instrument on either side. Tranco lets you fix the provider set when you generate a custom list, but it cannot resurrect a retired provider, so there is no configuration that makes a 2019 list and a 2026 list commensurable.
 That leaves two honest options, and you should say which you took:
@@ Line 108: / Line 110: @@
   * **Chrome.** [[https://developer.chrome.com/docs/automation-and-testing/chrome-for-testing|Chrome for Testing]] publishes immutable, non-auto-updating builds of Chrome and the matching ChromeDriver, addressable by exact version, with a JSON endpoint listing every one. As of 2026-08-27 it lists **2,484 versions from 113.0.5672.0 to 154.0.8026.0**, so any Chrome from the 113 line (spring 2023) onward can be re-instantiated exactly; anything older is not available through this channel.((Read from ''%%https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json%%'' on 2026-08-27; the ''last-known-good-versions.json'' endpoint on the same day reported Stable 152.0.7977.64.)) It is the correct default for any new repeated crawl, and it costs one line of setup.
-  * **Firefox.** Every release is retained at ''archive.mozilla.org/pub/firefox/releases/''. If you drive Firefox through [[Programming:Crawler:OpenWPM|OpenWPM]] you get the pin for free and should cite it: OpenWPM installs a specific **unbranded** Firefox build identified by a Mozilla release tag, and each OpenWPM release states which. v0.36.0 (24 August 2026) pins Firefox 154.0.((''%%https://github.com/openwpm/OpenWPM/blob/master/scripts/install-firefox.sh%%'' pins ''%%TAG='9ce1ee6baeb9a3c326dbd180bdece65d8fc2eadc' # FIREFOX_154_0_RELEASE%%''; release notes for v0.36.0 read "Firefox 152 → 154.0, with the Extension manifest's ''strict_min_version'' pinned to match". Both checked 2026-08-27.)) So "OpenWPM v0.36.0" **is** a browser version statement, and "OpenWPM" alone is not. Demir et al. write it the right way: //"We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)"// {[demir2022_reproducibility]}.
+  * **Firefox.** Every release is retained at ''archive.mozilla.org/pub/firefox/releases/''. If you drive Firefox through [[Programming:Crawler:OpenWPM|OpenWPM]] you get the pin for free and should cite it: OpenWPM installs a specific **unbranded** Firefox build identified by a Mozilla release tag, and each OpenWPM release states which. v0.37.0 (6 September 2026) pins Firefox 155.0.1.((''%%https://github.com/openwpm/OpenWPM/blob/master/scripts/install-firefox.sh%%'' pins ''%%TAG='5fdfd0092780e85643e2cddc0e1b590c8b9ef860' # FIREFOX_155_0_1_RELEASE%%''; the v0.37.0 release notes read "**Firefox 154 → 155.0.1**, with the Extension manifest's ''strict_min_version'' pinned to match". Both checked 2026-09-07 via ''%%api.github.com/repos/openwpm/OpenWPM/releases/latest%%''. This sentence named v0.36.0 / Firefox 154.0 when the page was written on 2026-08-27; v0.37.0 shipped ten days later, which is the rate of drift the paragraph is about.)) So "OpenWPM v0.37.0" **is** a browser version statement, and "OpenWPM" alone is not. Demir et al. write it the right way: //"We use the popular Open-WPM Framework [21] (v0.15.0 - Firefox version 88)"// {[demir2022_reproducibility]}.
-  * **Playwright already does this for you, if you pin Playwright.** Its ''browsers.json'' pins an exact browser build per Playwright release — the Chromium entry is literally titled "Chrome for Testing" and carries a ''browserVersion'', and Firefox and WebKit carry an internal ''revision''.((''%%https://raw.githubusercontent.com/microsoft/playwright/main/packages/playwright-core/browsers.json%%'', checked 2026-08-27: ''%%{"name": "chromium", "revision": "1241", "browserVersion": "152.0.7977.54", "title": "Chrome for Testing"}%%''.)) So a pinned Playwright version is a browser pin, in the same way an OpenWPM release tag is — but only if you pin Playwright rather than installing whatever npm resolves.
+  * **Playwright already does this for you, if you pin Playwright.** Its ''browsers.json'' pins an exact browser build per Playwright release — the Chromium entry is literally titled "Chrome for Testing" and carries a ''browserVersion'', and Firefox and WebKit carry an internal ''revision''.((Read from the **tagged release**, not from ''main'': ''%%https://raw.githubusercontent.com/microsoft/playwright/refs/tags/v1.63.0/packages/playwright-core/browsers.json%%'', fetched 2026-09-07, gives ''%%{"name": "chromium", "revision": "1243", "browserVersion": "153.0.8010.12", "title": "Chrome for Testing"}%%''. An earlier version of this footnote quoted ''main'' — ''revision 1241'', ''browserVersion 152.0.7977.54'' — which is a development snapshot that shipped in no release, and so was the wrong evidence for a claim about what a **pinned Playwright version** gives you.)) So a pinned Playwright version is a browser pin, in the same way an OpenWPM release tag is — but only if you pin Playwright rather than installing whatever npm resolves.
   * **The driver too.** A geckodriver or ChromeDriver upgrade can change what the browser accepts; OpenWPM's own v0.36.0 notes a geckodriver 0.37.1 change that made every browser fail to launch. Record the driver version alongside the browser.
@@ Line 161: / Line 163: @@
   * **[[Programming:CrUX|CrUX]]** — monthly, origin-level, real-user sourced. It is the frame under both HTTP Archive and modern Tranco, so it is not an independent check on either.
   * **[[Design:Archives|The Wayback Machine and Common Crawl]]** — the only way to measure before you started. Read that page before using either: the replay problems are severe and specific.
-  * **Censorship observatories** — Censored Planet {[raman2020_censored]} and ICLab {[niaki2020_iclab]} are continuous, longitudinal and public, and they are the model for what a maintained observatory looks like in this field.
+  * **Censorship observatories** — Censored Planet {[raman2020_censored]} is continuous, longitudinal and public, and is the model for what a maintained observatory looks like in this field. ICLab {[niaki2020_iclab]} is the other standard reference and its **paper** remains worth reading, but the observatory itself looks dormant: as of 2026-09-07 ''iclab.org'' serves an expired TLS certificate (''curl'' fails without ''-k''), its data page returns 404, and its news stops in 2019. Cite the paper; do not plan on the feed.
 <WRAP important>
@@ Line 259: / Line 261: @@
 ==== Were the pins actually held fixed? ====
-The table above counts papers that //stated// a value. Stating it once says nothing about wave two, so on its own it is only a ceiling. To turn the ceiling into an estimate, **all 250 repeated-crawl papers were read by hand** against a published codebook: for each of the four pins, what does the paper say the value was in //each// wave? Every pin is coded as one of ''same-explicit'' (the paper says in words that it was unchanged), ''same-implied'' (one configuration is described covering the whole repeated measurement and never said to change), ''varied-deliberate'', ''changed'', ''not-stated'' or ''unclear'', each with a verbatim quote from the paper. A second coder blind-re-read a random 30 of the 250 from the full text. The codebook, both coders' output, the agreement figures and the quote check are on [[provenance:design:longitudinal]].
+The table above counts papers that //stated// a value. Stating it once says nothing about wave two, so on its own it is only a ceiling. To turn the ceiling into an estimate, **all 250 repeated-crawl papers were read by hand** against a published codebook: for each of the four pins, what does the paper say the value was in //each// wave? Every pin is coded as one of ''same-explicit'' (the paper says in words that it was unchanged), ''same-implied'' (one configuration is described covering the whole repeated measurement and never said to change), ''varied-deliberate'', ''changed'', ''not-stated'' or ''unclear'', each with a verbatim quote from the paper. A second coder blind-re-read a random 30 of the 250 from the full text. **Both coders and the quote adjudicator were language-model agents (Claude Sonnet), reading text that a language-model extraction had already been run over; no person read any of the 250 papers.** The agreement figures below measure consistency between two such agents, which is not the same thing as human inter-rater reliability, and should be discounted accordingly. The codebook, both coders' output, the agreement figures and the quote check are on [[provenance:design:longitudinal]].
 **The first finding is about the population, not the pins.** The 250 were selected mechanically — a crawling paper whose extraction records more than one snapshot. Reading all of them:
@@ Line 286: / Line 288: @@
 The numbers to carry away:
-  * **15 of 124 (12.1%)** plausibly held all four fixed. **Two** say so in words.
+  * **15 of 124 (12.1%)** plausibly held all four fixed. Two are coded ''same-explicit'' on every pin — but a review pass found that ''same-explicit'' was applied more loosely than the codebook defines it, often to a permanent list ID or a one-off construction rather than to a paper saying //unchanged//, so read the left-hand column as "pinned by a stated identifier //or// an explicit statement", not as "said so in words". The strict count is smaller than two and was not re-derived.
   * **42 of 124 (33.9%)** have at least one pin that demonstrably moved — refreshed, updated, or varied on purpose.
   * The **estimate** column is the middle case and it is generous: ''same-implied'' means a paper described one setup and never mentioned changing it, which is what most methods sections look like whether or not anything moved. Treat it as an upper-middle bound, not a measurement.
-  * **The list is the pin the field does best and the browser the one it does worst relative to how fast it moves.** 95 of 124 plausibly held the population fixed; only 14 say in words that the browser was unchanged, against a browser release cycle of six weeks.
+  * **The list is the pin the field does best and the browser the one it does worst relative to how fast it moves.** 95 of 124 plausibly held the population fixed; only 14 carry explicit browser evidence at all, against a browser release cycle of six weeks.
 **Where "stated" and "held fixed" diverge, and in which direction.** For the browser, vantage and statefulness the hand-coded estimate is //below// the stated figure — a paper names a browser and then compares two of them, or names a vantage point for wave one only. For the population list it is //above// it: 62 of the 250 describe a fixed frame in prose without ever giving the version string the extraction looks for ("the same 200 sites", "one Tranco list ID"), so the structured field understates the list pin. The two columns measure different things and the gap is not an error in either.
-**Does the longer panel still pin less?** Over the 124, yes, and more sharply than the stated-value figures suggested: **1 of 33 (3.0%)** studies spanning twelve months or more plausibly held all four fixed, against **12 of 77 (15.6%)** under twelve months. Fourteen have no parsable end date and drop out; both denominators are small.
+**Does the longer panel still pin less?** The direction is consistent but the counts are too small to call. Over the 124, **1 of 33 (3.0%)** studies spanning twelve months or more plausibly held all four fixed, against **12 of 77 (15.6%)** under twelve months — Fisher's exact //p// = 0.10. The same split over the 250 using stated values gives //p// = 0.08 and the live-crawl-only split //p// = 0.69. Read it as a description that points one way in three cuts, not as a difference. Fourteen have no parsable end date and drop out.
 **By period,** over the same 124: 0 of 2 (2010–2011), 0 of 14 (2012–2015), 3 of 32 (2016–2019), 8 of 41 (2020–2023), 4 of 35 (2024–2026, provisional). The practice of pinning all four appears somewhere around 2016, peaks in the 2020–2023 bucket at roughly one repeated crawl in five, and the last bucket is both lower and incomplete. Read it as "no established norm", not as a decline.
@@ Line 361: / Line 363: @@
 ==== Papers that stated all four, and what they actually did ====
-The 23 repeated-crawl papers whose extraction records all four of list version, browser, vantage location and statefulness. This table used to be presented as "the methods sections to copy from". Hand-reading each one, **6 of the 23 plausibly held all four fixed across their waves.** The other 17 are not badly reported — most state more than the field average — they simply are not the thing the row claimed: nine turn out not to be repeated crawls at all, and the rest deliberately vary a pin, change one between waves, or describe it for one wave only.
+The 23 repeated-crawl papers whose extraction records all four of list version, browser, vantage location and statefulness. This table used to be presented as "the methods sections to copy from". Hand-reading each one, **6 of the 23 plausibly held all four fixed across their waves.** The other 17 are not badly reported — most state more than the field average — they simply are not the thing the row claimed: **seven** turn out not to be repeated crawls at all (they run parallel conditions at one moment), and the remaining **ten** are real repeat crawls that deliberately vary a pin, change one between waves, or describe it for one wave only.
 ^ Year ^ Venue ^ Paper ^ Really a repeat crawl? ^ List ^ Browser ^ Vantage ^ State ^ All four fixed? ^
@@ Line 376: / Line 378: @@
 | 2024 | PETS | The devil is in the details: server-side tracking {[fouad2024_devil]} | own repeat crawl | fixed, implied | **changed** | fixed, in words | fixed, in words | no |
 | 2023 | USENIX | Multi-factor and risk-based authentication availability {[gavazzi2023_multi]} | own repeat crawl | fixed, in words | varied by design | varied by design | fixed, in words | no |
-| 2022 | IMC | Respect the ORIGIN! {[singanamalla2022_respect]} | own repeat crawl | fixed, in words | **changed** | **changed** | fixed, in words | no |
+| 2022 | IMC | Respect the ORIGIN! {[singanamalla2022_respect]} | own repeat crawl | fixed, in words | **changed** | fixed, in words | fixed, in words | no |
 | 2022 | USENIX | Geodifferences in mobile apps {[kumar2022_investigation]} | parallel, not waves | fixed, implied | fixed, implied | varied by design | fixed, in words | no |
 | 2022 | WWW | Reproducibility and replicability {[demir2022_reproducibility]} | own repeat crawl | fixed, implied | not stated | not stated | unclear | no |
@@ Line 397: / Line 399: @@
   * **"Stated" is not "held fixed".** The structured pin figures count papers that //mentioned// the thing; a paper can name a browser in wave one and silently upgrade it in wave two, and the extraction cannot see that. That gap is why all 250 were re-read by hand — see //Were the pins actually held fixed?// above. Where the two disagree, the hand-coded figure is the one about comparability and the structured figure is the one about reporting.
   * **The hand-coding is one coder per paper, from an excerpt, with a blind second coder on 30.** Agreement was //κ// = 0.60 on exact pin values and 0.80 on whether a paper was really a repeat crawl; the excerpt coder over-calls a fixed browser. The estimate column is an estimate with a measured error rate, not a count.
+  * **The 15 were not individually re-read as a set.** A review pass sampling them found one, a 2020 WWW 5G-performance paper, whose own coding note says its repeated measurement is a cellular walk test rather than a web crawl — which makes it ''own-repeat-nonweb'' and would move the figure to 14 of 123. It was not re-coded before publication. Treat 15 as ±1.
   * **The 250 is half false positives for "repeated web crawl".** 124 of 250 are the authors' own repeated crawl of the web. Every other figure on this page that is a share of 250 — cadence, span, probe hits — inherits that, and none of them has been recomputed on the 124.
   * **Snapshot counts are noisy at the top.** Nine papers report more than 1,000 snapshots and hand-checking finds mis-extracted page and record counts. Bands are used; raw maxima are not.