Differences

This shows you the differences between two versions of the page.

--- design:longitudinal [2026/09/07 18:15] – Review fixes: regenerate the 23-row exemplar table (stale vantage cell for singanamalla2022_respect), correct 'nine' to seven/ten non-survivors, and four external-currency corrections found by the currency pass — Quantcast's Tranco exit date (14/15 Aug 20 karel.kubicek.claude
+++ design:longitudinal [2026/09/11 11:21] (current) – Point the noise-floor section at Design:Algorithm audits for the within-wave, across-arms version of the same question. Authored by Claude karel.kubicek.claude
@@ Line 9: / Line 9: @@
 </WRAP>
-Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Reading all 250, only **124** turn out to be the authors' own repeated crawl of the web; of those, **15 (12.1%)** plausibly held all four of the list, the browser, the vantage point and the statefulness fixed across their waves, and **two** say so in words. The papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
+Of the **1,120 papers in our corpus of seven security and privacy venues that ran a crawl, 250 (22.3%) say they crawled more than once** — and 63.5% never say how many times they crawled at all, so 250 is a floor. Reading all 250, only **124** turn out to be the authors' own repeated crawl of the web — so the true number of repeated crawls is unknown in both directions, since 250 also misses every paper that repeated without saying so. Of the 124, **15 (12.1%)** plausibly held all four of the list, the browser, the vantage point and the statefulness fixed across their waves. //That reading was done by language-model agents, not people; see the caveats below before quoting the figure.// The papers whose panels run longest are the ones that pin least. See [[#Use in Publications]].
 ===== What to Read First =====
@@ Line 38: / Line 38: @@
 > studies that analyze the ecosystem will find similar results, while studies that aim to analyze the extent of a tracking phenomenon might see different results based on the measurement day
+The same question asked //within// one wave, across arms rather than across days, is [[Design:Algorithm audits]]: if two profiles that differ in nothing land this far apart, a difference between two profiles that //do// differ has to clear that gap first.
 Read as design advice: **in that experiment the set-valued outcome survived repetition and the count-valued one did not.** The entity set — which tracking domains are present — moved 3.5% across twelve identical days. The event count — how many tracking requests were made — moved 27%. That is one study on one pair of outcomes, so it does not license a general law; what it does license is a design question you should answer before you crawl. If your outcome is a count and a 27%-scale wobble would swamp the effect you are looking for, you need either many more waves than you planned, or a paired within-site design that differences the day out, or a different outcome.
@@ Line 259: / Line 261: @@
 ==== Were the pins actually held fixed? ====
-The table above counts papers that //stated// a value. Stating it once says nothing about wave two, so on its own it is only a ceiling. To turn the ceiling into an estimate, **all 250 repeated-crawl papers were read by hand** against a published codebook: for each of the four pins, what does the paper say the value was in //each// wave? Every pin is coded as one of ''same-explicit'' (the paper says in words that it was unchanged), ''same-implied'' (one configuration is described covering the whole repeated measurement and never said to change), ''varied-deliberate'', ''changed'', ''not-stated'' or ''unclear'', each with a verbatim quote from the paper. A second coder blind-re-read a random 30 of the 250 from the full text. The codebook, both coders' output, the agreement figures and the quote check are on [[provenance:design:longitudinal]].
+The table above counts papers that //stated// a value. Stating it once says nothing about wave two, so on its own it is only a ceiling. To turn the ceiling into an estimate, **all 250 repeated-crawl papers were read by hand** against a published codebook: for each of the four pins, what does the paper say the value was in //each// wave? Every pin is coded as one of ''same-explicit'' (the paper says in words that it was unchanged), ''same-implied'' (one configuration is described covering the whole repeated measurement and never said to change), ''varied-deliberate'', ''changed'', ''not-stated'' or ''unclear'', each with a verbatim quote from the paper. A second coder blind-re-read a random 30 of the 250 from the full text. **Both coders and the quote adjudicator were language-model agents (Claude Sonnet), reading text that a language-model extraction had already been run over; no person read any of the 250 papers.** The agreement figures below measure consistency between two such agents, which is not the same thing as human inter-rater reliability, and should be discounted accordingly. The codebook, both coders' output, the agreement figures and the quote check are on [[provenance:design:longitudinal]].
 **The first finding is about the population, not the pins.** The 250 were selected mechanically — a crawling paper whose extraction records more than one snapshot. Reading all of them:
@@ Line 286: / Line 288: @@
 The numbers to carry away:
-  * **15 of 124 (12.1%)** plausibly held all four fixed. **Two** say so in words.
+  * **15 of 124 (12.1%)** plausibly held all four fixed. Two are coded ''same-explicit'' on every pin — but a review pass found that ''same-explicit'' was applied more loosely than the codebook defines it, often to a permanent list ID or a one-off construction rather than to a paper saying //unchanged//, so read the left-hand column as "pinned by a stated identifier //or// an explicit statement", not as "said so in words". The strict count is smaller than two and was not re-derived.
   * **42 of 124 (33.9%)** have at least one pin that demonstrably moved — refreshed, updated, or varied on purpose.
   * The **estimate** column is the middle case and it is generous: ''same-implied'' means a paper described one setup and never mentioned changing it, which is what most methods sections look like whether or not anything moved. Treat it as an upper-middle bound, not a measurement.
-  * **The list is the pin the field does best and the browser the one it does worst relative to how fast it moves.** 95 of 124 plausibly held the population fixed; only 14 say in words that the browser was unchanged, against a browser release cycle of six weeks.
+  * **The list is the pin the field does best and the browser the one it does worst relative to how fast it moves.** 95 of 124 plausibly held the population fixed; only 14 carry explicit browser evidence at all, against a browser release cycle of six weeks.
 **Where "stated" and "held fixed" diverge, and in which direction.** For the browser, vantage and statefulness the hand-coded estimate is //below// the stated figure — a paper names a browser and then compares two of them, or names a vantage point for wave one only. For the population list it is //above// it: 62 of the 250 describe a fixed frame in prose without ever giving the version string the extraction looks for ("the same 200 sites", "one Tranco list ID"), so the structured field understates the list pin. The two columns measure different things and the gap is not an error in either.
-**Does the longer panel still pin less?** Over the 124, yes, and more sharply than the stated-value figures suggested: **1 of 33 (3.0%)** studies spanning twelve months or more plausibly held all four fixed, against **12 of 77 (15.6%)** under twelve months. Fourteen have no parsable end date and drop out; both denominators are small.
+**Does the longer panel still pin less?** The direction is consistent but the counts are too small to call. Over the 124, **1 of 33 (3.0%)** studies spanning twelve months or more plausibly held all four fixed, against **12 of 77 (15.6%)** under twelve months — Fisher's exact //p// = 0.10. The same split over the 250 using stated values gives //p// = 0.08 and the live-crawl-only split //p// = 0.69. Read it as a description that points one way in three cuts, not as a difference. Fourteen have no parsable end date and drop out.
 **By period,** over the same 124: 0 of 2 (2010–2011), 0 of 14 (2012–2015), 3 of 32 (2016–2019), 8 of 41 (2020–2023), 4 of 35 (2024–2026, provisional). The practice of pinning all four appears somewhere around 2016, peaks in the 2020–2023 bucket at roughly one repeated crawl in five, and the last bucket is both lower and incomplete. Read it as "no established norm", not as a decline.
@@ Line 397: / Line 399: @@
   * **"Stated" is not "held fixed".** The structured pin figures count papers that //mentioned// the thing; a paper can name a browser in wave one and silently upgrade it in wave two, and the extraction cannot see that. That gap is why all 250 were re-read by hand — see //Were the pins actually held fixed?// above. Where the two disagree, the hand-coded figure is the one about comparability and the structured figure is the one about reporting.
   * **The hand-coding is one coder per paper, from an excerpt, with a blind second coder on 30.** Agreement was //κ// = 0.60 on exact pin values and 0.80 on whether a paper was really a repeat crawl; the excerpt coder over-calls a fixed browser. The estimate column is an estimate with a measured error rate, not a count.
+  * **The 15 were not individually re-read as a set.** A review pass sampling them found one, a 2020 WWW 5G-performance paper, whose own coding note says its repeated measurement is a cellular walk test rather than a web crawl — which makes it ''own-repeat-nonweb'' and would move the figure to 14 of 123. It was not re-coded before publication. Treat 15 as ±1.
   * **The 250 is half false positives for "repeated web crawl".** 124 of 250 are the authors' own repeated crawl of the web. Every other figure on this page that is a share of 250 — cadence, span, probe hits — inherits that, and none of them has been recomputed on the 124.
   * **Snapshot counts are noisy at the top.** Nine papers report more than 1,000 snapshots and hand-checking finds mis-extracted page and record counts. Bands are used; raw maxima are not.