| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:stateful_stateless [2026/08/31 19:48] – Correct the storageState() sentence: measured on Playwright 1.62.1, it does not carry IndexedDB unless {indexedDB:true} is passed, and never carries sessionStorage, Cache Storage or service-worker registrations. Link Privacy:Browser storage for the stores karel.kubicek.claude | programming:stateful_stateless [2026/09/17 10:45] (current) – ConsentAction audit propagation and bucket series; Authored by Claude karel.kubicek.claude |
|---|
| - **Almost nobody reports it.** Of the **1,120 papers in our corpus that ran an automated web crawl, 219 (19.6%) say whether the crawl was stateful** — the second-least-reported crawl-configuration field after headless mode, and the **flattest**: its reporting rate moved 2.5 percentage points between the first and last four-year bucket, against +18.2 for naming a browser and +10.1 for headless (see [[#Reporting has not improved in sixteen years]]). | - **Almost nobody reports it.** Of the **1,120 papers in our corpus that ran an automated web crawl, 219 (19.6%) say whether the crawl was stateful** — the second-least-reported crawl-configuration field after headless mode, and the **flattest**: its reporting rate moved 2.5 percentage points between the first and last four-year bucket, against +18.2 for naming a browser and +10.1 for headless (see [[#Reporting has not improved in sixteen years]]). |
| - **"We cleared cookies" is not a reset.** Measured here on Playwright 1.62.1 / Chromium 151: ''context.clearCookies()'' leaves ''localStorage'' intact //and// leaves the HTTP cache warm, so the next visit serves subresources from disk and never touches the origin. See [[#What a reset actually resets]]. | - **"We cleared cookies" is not a reset.** Measured here on Playwright 1.62.1 / Chromium 151: ''context.clearCookies()'' leaves ''localStorage'' intact //and// leaves the HTTP cache warm, so the next visit serves subresources from disk and never touches the origin. See [[#What a reset actually resets]]. |
| - **Since 2022 the browser decides how stateful your stateful crawl is, not you.** Firefox partitions third-party cookies per top-level site by default; Chrome partitions third-party //storage// for every user from Chrome 115 on. Playwright launches Chromium with ''--disable-features=ThirdPartyStoragePartitioning'', Puppeteer does not. Same engine, two different webs. See [[#Since 2022 the engine decides, not you]]. | - **Since 2022 the browser decides how stateful your stateful crawl is, not you.** Firefox partitions third-party cookies per top-level site by default; Chrome partitions third-party //storage// for every user from Chrome 115 on. Playwright launches Chromium with ''%%--disable-features%%=ThirdPartyStoragePartitioning'', Puppeteer does not. Same engine, two different webs. See [[#Since 2022 the engine decides, not you]]. |
| </WRAP> | </WRAP> |
| |
| | Third-party presence, request counts, filter-list hit rates | either, but say which | a fresh profile draws //more// third-party traffic than an aged one, so the two are not interchangeable | {[jueckstock2021_realistic]}, {[zeber2020representativeness]} | | | Third-party presence, request counts, filter-list hit rates | either, but say which | a fresh profile draws //more// third-party traffic than an aged one, so the two are not interchangeable | {[jueckstock2021_realistic]}, {[zeber2020representativeness]} | |
| | Cookie syncing / ID sharing | **either, but they measure different things** | a fresh profile sees //first-contact// syncing and over-triggers it {[zeber2020representativeness]}; reconstructing the sync graph of an aged identity, or how much history a partner can merge, needs accumulation | {[englehardt2016online]} runs its sync analysis on the //stateful// 100k crawl; {[agarwal2020_stop]}, {[acar2014_never]} | | | Cookie syncing / ID sharing | **either, but they measure different things** | a fresh profile sees //first-contact// syncing and over-triggers it {[zeber2020representativeness]}; reconstructing the sync graph of an aged identity, or how much history a partner can merge, needs accumulation | {[englehardt2016online]} runs its sync analysis on the //stateful// 100k crawl; {[agarwal2020_stop]}, {[acar2014_never]} | |
| | Cookie respawning, evercookies | **stateful** across a clear | the phenomenon //is// state surviving a reset | {[acar2014_never]}; detectable from a stateless harness by comparing paired visits {[fouad2022my]} | | | Cookie respawning, evercookies | **stateful** across a clear | the phenomenon //is// state surviving a reset | {[acar2014_never]}; detectable from a stateless harness by comparing paired visits {[fouad2022_cookie]} | |
| | Ad retargeting, personalisation, differential pricing | **stateful** training, then usually seeded-stateless measurement | the profile is the independent variable | {[bashir2016tracing]}, {[agarwal2020_stop]}, {[liu2024_opted]}, {[meng2014_pollution]}, {[robertson2018_auditing]} | | | Ad retargeting, personalisation, differential pricing | **stateful** training, then usually seeded-stateless measurement | the profile is the independent variable — the experimental design is [[Design:Algorithm audits]] | {[bashir2016tracing]}, {[agarwal2020_stop]}, {[liu2024_opted]}, {[meng2014_pollution]}, {[robertson2018_auditing]} | |
| | Effect of a consent choice on //other// sites | **stateful** | the consent decision only travels via stored state | {[rasaii2025_crumbs]} | | | Effect of a consent choice on //other// sites | **stateful** | the consent decision only travels via stored state | {[rasaii2025_crumbs]} | |
| | Consent revocation, opt-out persistence | **stateful** within a session at minimum | you must be in the consented state before you can revoke it | {[kancherla2025_johnny]}, {[liu2024_opted]} | | | Consent revocation, opt-out persistence | **stateful** within a session at minimum | you must be in the consented state before you can revoke it | {[kancherla2025_johnny]}, {[liu2024_opted]} | |
| |
| * **Consent given on one site follows you to the next.** Rasaii et al. accepted banners across the first half of Tranco's top 20,000 and then measured the second half with that profile loaded — the denominator for the headline figure is the second-half domains where a banner was successfully rejected, not all 20,000. "Our findings reveal that around 50% of websites send at least one intractable cookie" — a tracking cookie transmitted before any consent on the site sending it. Sites with a CMP banner sent **6.91×** more of them than sites with a native banner; enabling Global Privacy Control cut them by about **30%**, with a further **32%** on later visits after rejecting; and about **25%** stop being sent only after the page is reloaded {[rasaii2025_crumbs]}. Partitioning does not yet blunt this: "only 1.3% of all unique tracking cookies are partitioned, with more than half accompanied by nonpartitioned cookies from the same tracker domain". | * **Consent given on one site follows you to the next.** Rasaii et al. accepted banners across the first half of Tranco's top 20,000 and then measured the second half with that profile loaded — the denominator for the headline figure is the second-half domains where a banner was successfully rejected, not all 20,000. "Our findings reveal that around 50% of websites send at least one intractable cookie" — a tracking cookie transmitted before any consent on the site sending it. Sites with a CMP banner sent **6.91×** more of them than sites with a native banner; enabling Global Privacy Control cut them by about **30%**, with a further **32%** on later visits after rejecting; and about **25%** stop being sent only after the page is reloaded {[rasaii2025_crumbs]}. Partitioning does not yet blunt this: "only 1.3% of all unique tracking cookies are partitioned, with more than half accompanied by nonpartitioned cookies from the same tracker domain". |
| * **Respawning plus syncing survives a state clear.** The 2014 mechanism is historical — Flash reached end of life in December 2020 — but the finding is the reason a state clear cannot be assumed to work, and the technique moved to fingerprint-keyed respawning rather than disappearing ({[fouad2022my]} in 2022, server-side in {[fouad2024_devil]} in 2024). Acar et al. found "33 different Flash cookies from 30 different domains respawned a total of 355 cookies on 107 first party domains", and concluded that through one ad exchange present on ~11% of first parties, "This scenario enables at least 11% of a user's history to be tracked over time" {[acar2014_never]}. | * **Respawning plus syncing survives a state clear.** The 2014 mechanism is historical — Flash reached end of life in December 2020 — but the finding is the reason a state clear cannot be assumed to work, and the technique moved to fingerprint-keyed respawning rather than disappearing ({[fouad2022_cookie]} in 2022, server-side in {[fouad2024_devil]} in 2024). Acar et al. found "33 different Flash cookies from 30 different domains respawned a total of 355 cookies on 107 first party domains", and concluded that through one ad exchange present on ~11% of first parties, "This scenario enables at least 11% of a user's history to be tracked over time" {[acar2014_never]}. |
| * **A trained profile is treated differently.** Agarwal et al.: "having an established persona from a particular demographic … results in up to 15% more cookies stored than for a baseline with no set persona" {[agarwal2020_stop]}. | * **A trained profile is treated differently.** Agarwal et al.: "having an established persona from a particular demographic … results in up to 15% more cookies stored than for a baseline with no set persona" {[agarwal2020_stop]}. |
| * **State accumulates within a site, not only across sites.** Urban et al., under a seeded-stateless design with state kept across a site's own subpages: "subsites set considerably more (36 %) cookies than the respective landing pages. On average, 55 cookies were set when loading a landing page while 78 were set when a subsite was accessed" {[urban2020beyond]}. See [[Programming:Interaction]]. | * **State accumulates within a site, not only across sites.** Urban et al., under a seeded-stateless design with state kept across a site's own subpages: "subsites set considerably more (36 %) cookies than the respective landing pages. On average, 55 cookies were set when loading a landing page while 78 were set when a subsite was accessed" {[urban2020beyond]}. See [[Programming:Interaction]]. |
| |
| And one that cuts the other way: **cookie respawning with browser fingerprinting** was measured on 30,000 Alexa sites with a stateless harness, by comparing paired visits rather than by accumulating a profile — "1, 150 (3.83%) of the Alexa top 30, 000 websites use cookie respawning with browser fingerprinting" {[fouad2022my]}. Note the qualifier: that figure counts respawning //combined with// fingerprinting, not respawning in general. A stateful //phenomenon// does not always require a stateful //crawl//; sometimes it requires two controlled visits. The same is true of cookie syncing at first contact. What accumulation buys is the //aged// identity, not the mechanism. | And one that cuts the other way: **cookie respawning with browser fingerprinting** was measured on 30,000 Alexa sites with a stateless harness, by comparing paired visits rather than by accumulating a profile — "1, 150 (3.83%) of the Alexa top 30, 000 websites use cookie respawning with browser fingerprinting" {[fouad2022_cookie]}. Note the qualifier: that figure counts respawning //combined with// fingerprinting, not respawning in general. A stateful //phenomenon// does not always require a stateful //crawl//; sometimes it requires two controlled visits. The same is true of cookie syncing at first contact. What accumulation buys is the //aged// identity, not the mechanism. |
| |
| ===== Since 2022 the engine decides, not you ===== | ===== Since 2022 the engine decides, not you ===== |
| * **Chrome partitions third-party //storage// for every user.** "The feature has been enabled for all users on Chrome 115 and later."((Google, //Privacy Sandbox: Storage Partitioning//, ''developers.google.com/privacy-sandbox/cookies/storage-partitioning''. Checked 2026-08-19.)) Cookies are the exception, not the rule, here. | * **Chrome partitions third-party //storage// for every user.** "The feature has been enabled for all users on Chrome 115 and later."((Google, //Privacy Sandbox: Storage Partitioning//, ''developers.google.com/privacy-sandbox/cookies/storage-partitioning''. Checked 2026-08-19.)) Cookies are the exception, not the rule, here. |
| * **Third-party cookies were //not// deprecated.** On 22 April 2025 Google announced it would "maintain our current approach to offering users third-party cookie choice in Chrome, and will not be rolling out a new standalone prompt for third-party cookies", and in October 2025 confirmed CHIPS and FedCM continue while other Privacy Sandbox APIs are phased out.((Privacy Sandbox, "Next steps for Privacy Sandbox and tracking protections in Chrome", 22 April 2025, and "Update on Plans for Privacy Sandbox Technologies", 17 October 2025. Checked 2026-08-19.)) A 2023-vintage paper that frames its design around imminent third-party cookie removal is describing a future that did not arrive. Chrome's **Incognito mode** does block third-party cookies by default, which is a separate trap: "we ran in incognito for a clean profile" silently also changes the blocking policy. | * **Third-party cookies were //not// deprecated.** On 22 April 2025 Google announced it would "maintain our current approach to offering users third-party cookie choice in Chrome, and will not be rolling out a new standalone prompt for third-party cookies", and in October 2025 confirmed CHIPS and FedCM continue while other Privacy Sandbox APIs are phased out.((Privacy Sandbox, "Next steps for Privacy Sandbox and tracking protections in Chrome", 22 April 2025, and "Update on Plans for Privacy Sandbox Technologies", 17 October 2025. Checked 2026-08-19.)) A 2023-vintage paper that frames its design around imminent third-party cookie removal is describing a future that did not arrive. Chrome's **Incognito mode** does block third-party cookies by default, which is a separate trap: "we ran in incognito for a clean profile" silently also changes the blocking policy. |
| * **Your crawler probably turns the partitioning off.** Playwright launches Chromium with ''--disable-features=…,ThirdPartyStoragePartitioning,…''; Puppeteer does not. Measured here: | * **Your crawler probably turns the partitioning off.** Playwright launches Chromium with ''%%--disable-features%%=…,ThirdPartyStoragePartitioning,…''; Puppeteer does not. Measured here: |
| |
| ^ Chromium feature ^ Playwright 1.62.1 ^ Puppeteer 25.5.0((The same list in Puppeteer 25.8.0, the latest release as of 2026-08-19, is byte-identical, so the comparison is not an artefact of the pinned version.)) ^ | ^ Chromium feature ^ Playwright 1.62.1 ^ Puppeteer 25.5.0((The same list in Puppeteer 25.8.0, the latest release as of 2026-08-19, is byte-identical, so the comparison is not an artefact of the pinned version.)) ^ |
| | Authentication | 779 | 69.6% | | | Authentication | 779 | 69.6% | |
| | At least one browser named | 529 | 47.2% | | | At least one browser named | 529 | 47.2% | |
| | Consent action | 349 | 31.2% | | | Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% | |
| | **Stateful or stateless** | **219** | **19.6%** | | | **Stateful or stateless** | **219** | **19.6%** | |
| | Headless or headful | 140 | 12.5% | | | Headless or headful | 140 | 12.5% | |
| | ''authentication'' | 59.8% | 70.1% | 69.2% | 72.8% | 69.2% | 12.9 pp | +9.4 pp | | | ''authentication'' | 59.8% | 70.1% | 69.2% | 72.8% | 69.2% | 12.9 pp | +9.4 pp | |
| | ''browsers'' (≥1 named) | 32.4% | 48.5% | 49.7% | 47.0% | 50.5% | 18.2 pp | **+18.2 pp** | | | ''browsers'' (≥1 named) | 32.4% | 48.5% | 49.7% | 47.0% | 50.5% | 18.2 pp | **+18.2 pp** | |
| | ''consentAction'' | 24.5% | 29.9% | 32.1% | 33.6% | 29.8% | 9.1 pp | +5.3 pp | | | ''consentAction'' (schema field populated; not a paper-claim series) | 24.5% | 29.9% | 32.1% | 33.6% | 29.8% | 9.1 pp | +5.3 pp | |
| | **''statefulness''** | 15.7% | 21.0% | 19.8% | 20.6% | 18.2% | **5.3 pp** | **+2.5 pp** | | | **''statefulness''** | 15.7% | 21.0% | 19.8% | 20.6% | 18.2% | **5.3 pp** | **+2.5 pp** | |
| | ''headless'' | 1.0% | 15.6% | 13.3% | 14.5% | 11.1% | 14.6 pp | +10.1 pp | | | ''headless'' | 1.0% | 15.6% | 13.3% | 14.5% | 11.1% | 14.6 pp | +10.1 pp | |
| | |
| | The raw ''consentAction'' row above is a schema-population baseline, not a paper-claim series. The 2026-09-05 audit produces this separate paper-claim series: |
| | |
| | ^ Bucket ^ Crawling papers ^ Audited papers stating a consent action ^ Share ^ Of those, interacted ^ |
| | | 2010–2013 | 102 | 0 | 0.0% | 0 | |
| | | 2014–2017 | 167 | 0 | 0.0% | 0 | |
| | | 2018–2021 | 308 | 11 | 3.6% | 3 | |
| | | 2022–2024 | 345 | 25 | 7.2% | 16 | |
| | | 2025–2026* | 198 | 19 | 9.6% | 13 | |
| | |
| | The old raw-series adjacent tests are all non-significant (p = 0.4008, 0.679, 0.7387 and 0.3911). On the audited series, 2010–2013 versus 2014–2017 is p = 1, 2014–2017 versus 2018–2021 is p = 0.009883, 2018–2021 versus 2022–2024 is p = 0.05751, and 2022–2024 versus provisional 2025–2026 is p = 0.3324. |
| |
| Denominators are the crawling papers in each bucket, from the table above (102 / 167 / 308 / 345 / 198). Naming the browser gained 18 points and headless mode gained 10 from a near-zero base; statefulness gained 2.5 and has the narrowest **range** of the six (5.3 pp). Interaction depth is the only field whose range is nearly as narrow (6.2 pp), and it got that way by //declining// from 78.4% to 72.2% rather than by standing still. It is not that the field decided the axis does not matter — Demir et al. made it a named criterion in 2022, and Zeber et al. and Jueckstock et al. had made it a measured concern in 2020 and 2021. It is that nothing turned the concern into a reporting norm: no venue asks for it on a checklist, and no widely used tool writes it into a config file that ends up in an artifact. | Denominators are the crawling papers in each bucket, from the table above (102 / 167 / 308 / 345 / 198). Naming the browser gained 18 points and headless mode gained 10 from a near-zero base; statefulness gained 2.5 and has the narrowest **range** of the six (5.3 pp). Interaction depth is the only field whose range is nearly as narrow (6.2 pp), and it got that way by //declining// from 78.4% to 72.2% rather than by standing still. It is not that the field decided the axis does not matter — Demir et al. made it a named criterion in 2022, and Zeber et al. and Jueckstock et al. had made it a measured concern in 2020 and 2021. It is that nothing turned the concern into a reporting norm: no venue asks for it on a checklist, and no widely used tool writes it into a config file that ends up in an artifact. |
| |
| ^ Subset of crawling papers ^ N ^ State statefulness ^ Share stating ^ stateless ^ stateful ^ both ^ | ^ Subset of crawling papers ^ N ^ State statefulness ^ Share stating ^ stateless ^ stateful ^ both ^ |
| | Crawls that acted on a consent notice | 36 | 22 | **61.1%** | 11 | 4 | 7 | | | Crawls that an audit found acted on a consent notice | 32 | 20 | **62.5%** | 10 | 4 | 6 | |
| | …the 28 of those whose interaction was verified by hand | 28 | 16 | **57.1%** | 8 | 3 | 5 | | | Crawls with an audited consent action (interaction or explicit no-action) | 55 | 32 | **58.2%** | 18 | 8 | 6 | |
| | Crawls that logged in | 90 | 54 | **60.0%** | 7 | 39 | 8 | | | Crawls that logged in | 90 | 54 | **60.0%** | 7 | 39 | 8 | |
| | Repeat-visit designs (2 or more visits per target) | 199 | 85 | 42.7% | 49 | 21 | 15 | | | Repeat-visit designs (2 or more visits per target) | 199 | 85 | 42.7% | 49 | 21 | 15 | |
| | //all crawling papers// | 1,120 | 219 | 19.6% | 113 | 77 | 29 | | | //all crawling papers// | 1,120 | 219 | 19.6% | 113 | 77 | 29 | |
| |
| The consent row needs a caveat that [[privacy:consent]] supplies: the same 36 papers were hand-audited there and **7 (19.4%) turned out to be extraction false positives** — they never touched a banner. The second row recomputes the rate on the 28 the audit **fully** supported (a 29th is supported but with an overstated enum value), and it barely moves, so the finding is robust to the error. | The consent rows use the 2026-09-05 full-text audit. Of the **313** ''no-interaction'' labels, **279 (89.1%)** were unsupported defaults; on the interacting side, **7 of 36 (19.4%)** labels were false positives. Of the **32** audited-interacting papers, **20** state statefulness (62.5%); of the **55** audited-action papers, **32** state statefulness (58.2%). The old 36/28 rows are not used because they were based on only one side of the audit. |
| |
| The good news first: where the design makes the axis unavoidable, reporting roughly triples. A login is state, and 39 of the 54 login crawls that say anything say stateful. The bad news is the repeat-visit row: **199 papers visit the same target two or more times and 114 of them (57.3%) never say whether state carried between the visits** — which is the one thing that determines whether the repeat visit is a replication or a second step in a sequence. If you take one reporting rule from this page, take that one. | The good news first: where the design makes the axis unavoidable, reporting roughly triples. A login is state, and 39 of the 54 login crawls that say anything say stateful. The bad news is the repeat-visit row: **199 papers visit the same target two or more times and 114 of them (57.3%) never say whether state carried between the visits** — which is the one thing that determines whether the repeat visit is a replication or a second step in a sequence. If you take one reporting rule from this page, take that one. |
| - **If you need a rolling profile, run one browser** or report how many you ran and how the site list was partitioned across them. | - **If you need a rolling profile, run one browser** or report how many you ran and how the site list was partitioned across them. |
| - **Never claim a reset you did not measure.** Run [[#The code|the probe]] against your own harness once; it takes a minute and it is the cheapest methodological insurance on this page. | - **Never claim a reset you did not measure.** Run [[#The code|the probe]] against your own harness once; it takes a minute and it is the cheapest methodological insurance on this page. |
| | - **If the profile //is// your independent variable, this page is not enough.** Two arms need more than a statefulness decision: a control arm that is identical in everything else, no carry-over between them, and a measured noise floor before you call a gap a finding. [[Design:Algorithm audits]]. |
| - **Do not compare your numbers to a paper on the other side of this axis** without saying so. Third-party counts from a fresh-profile crawl and from an aged profile are different quantities. | - **Do not compare your numbers to a paper on the other side of this axis** without saying so. Third-party counts from a fresh-profile crawl and from an aged profile are different quantities. |
| - **State the engine's partitioning posture**, and if you disable partitioning to get cross-site accumulation, say that you did and why. | - **State the engine's partitioning posture**, and if you disable partitioning to get cross-site accumulation, say that you did and why. |
