Differences

This shows you the differences between two versions of the page.

--- programming:crawler:openwpm [2026/08/14 08:41] – Add a 'Papers to read first' section (six papers, in the order useful before running a crawl). Authored by Claude. karel.kubicek.claude
+++ programming:crawler:openwpm [2026/09/17 07:41] (current) – Markup rendering sweep correction: escape residual CLI flag. Authored by Claude. karel.kubicek.claude
@@ Line 3: / Line 3: @@
 OpenWPM {[englehardt2016online]} is the closest thing web privacy measurement has to a standard instrument: a Python platform that drives an **unbranded Firefox** through **Selenium**, and records what the browser did through a **privileged WebExtension** rather than through the automation protocol. It is the most widely shared specialised crawler in our corpus: **60 papers used or extended it**, against 21 each for the next two, on the mention-matching count over all 5,859 papers in [[Programming:Crawler#Which specialised crawlers actually get used|the comparison page]]. 59 of those 60 are inside the 1,120 papers that ran a crawl; the exception re-analysed someone else's OpenWPM data.
-This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]].
+This page is about running it and reading papers that ran it: which instrumentation surfaces exist and which are silent by default, what stateful and stateless mean //in OpenWPM's implementation//, what the version number commits you to, and what its maintenance looks like today. For the choice between OpenWPM and Playwright, Tracker Radar Collector or a patched browser, see [[Programming:Crawler]]. For the design question of whether to keep browser state at all, see [[Programming:Stateful stateless]], which also covers what a "reset" actually resets and why the browser's own partitioning defaults now decide part of the answer.
 <WRAP important>
 Two findings from our corpus that should change how you write your methodology section:
-  - **OpenWPM's own README asks researchers to publish the version number. 15 of the 60 papers do (25.0%).** The version is not cosmetic: each release pins one Firefox build, so ''OpenWPM v0.17.0'' ships Firefox 90 — and a 2024 paper crawled with it. But the pin is a default, not a guarantee, and only three of the fifteen papers say which browser they actually ran (see [[#An OpenWPM version is a Firefox version, by default]]).
+  - **OpenWPM's own README asks researchers to publish the version number. 15 of the 60 papers do (25.0%).** The version is not cosmetic: each release pins one Firefox build, so ''OpenWPM v0.17.0'' ships Firefox 90 — and a 2024 paper still crawled with that 2021 release. The pin is only a default, though: that paper is one of the nine that name a browser too, and it ran Firefox 102, not 90 (see [[#An OpenWPM version is a Firefox version, by default]]).
   - **Naming the version is still not enough.** Only **3 of the 60 papers name a single OpenWPM configuration key or output table** in their full text. Which instruments were switched on is the difference between a crawl that recorded JavaScript API calls and one that did not, and it is almost never stated. ''cookie_instrument'' is the **only** instrument enabled by default; everything else is off until you set it.
 </WRAP>
@@ Line 18: / Line 18: @@
   * **A privileged WebExtension** using ''webRequest'', ''webNavigation'', ''cookies'' and ''dns'', plus three experimental APIs of its own (''sockets'', ''profileDirIO'', ''stackDump''). Because it uses privileged APIs, it can only be loaded by an **unbranded** or custom Firefox build with add-on security disabled — a documented requirement((''docs/Platform-Architecture.md'': "The Extension makes heavy use of privileged APIs and can only be installed on unbranded or custom builds of Firefox with add-on security disabled." Repository read at commit ''b9dd4c3'', 2026-08-14.)) and the reason ''scripts/install-firefox.sh'' downloads a specific build from Mozilla's CI rather than using whatever Firefox you have. The extension is still **Manifest V2**.
   * **A process architecture built for failure**: one ''TaskManager'' supervises N ''BrowserManager'' processes, each owning one Firefox and one geckodriver; a separate storage controller process serialises records. A browser that crashes is restarted and its profile restored, and the visit is recorded as incomplete rather than lost silently — in principle (see [[#Known pitfalls]]).
-  * **Three output stores**: structured records to SQLite or Parquet (local, S3 or GCS), response bodies to LevelDB de-duplicated by MD5 content hash. Every record carries ''visit_id'', ''browser_id'' and ''instance_id'', which is what makes a multi-browser crawl analysable at all.
+  * **Three output stores**: structured records to SQLite or Parquet (local, S3 or GCS), response bodies to LevelDB de-duplicated by content hash. Instrument records carry ''visit_id'' and ''browser_id'', which is what makes a multi-browser crawl analysable at all; the Parquet path adds an ''instance_id'' per run for partitioning, and the SQLite schema has no such column.
   * **Commands as objects**: a ''CommandSequence'' per site, built from ''GetCommand'', ''BrowseCommand'' (follow a random internal link), screenshot, page-source and profile-dump commands, with per-command timeouts and retries — plus your own ''BaseCommand'' subclasses, which is how most papers extend it.
@@ Line 25: / Line 25: @@
 Each instrument is a boolean on ''BrowserParams'' and writes to its own table(s). The defaults matter more than the table: **a ''BrowserParams()'' with nothing set records cookies and nothing else.**
-^ Instrument ^ What it records ^ Tables ^ Default ^ Status ^
+^ Instrument ^ What it records ^ Tables ^ Default ^ Status(("works" here is from the code, the documentation and the papers that used it — **not** from a run on our host, where no browser ever launched. Only "broken" was confirmed by execution, which needs no browser. See [[#Installing it, and what we could and could not run]].)) ^
 | ''http_instrument'' | request and response headers, redirect chains, POST bodies, ''resource_type'', third-party flags, triggering and loading origin | ''http_requests'', ''http_responses'', ''http_redirects'' | off | works; headers for cached content are saved except for images((''docs/Configuration.md'', citing [[https://bugzilla.mozilla.org/show_bug.cgi?id=634073|Bugzilla 634073]]. OCSP POST bodies are also not recorded.)) |
 | ''js_instrument'' | method calls with arguments and property accesses, for the APIs you configure | ''javascript'' | off | works; the default configuration is narrower than it looks — see below |
@@ Line 212: / Line 212: @@
 ===== An OpenWPM version is a Firefox version, by default =====
-Every OpenWPM release pins one unbranded Firefox build, in one line of ''scripts/install-firefox.sh''. Reading that line out of every ''v*'' tag gives the mapping. The table below omits seven point releases for width (0.11.0, 0.14.1, 0.16.0, 0.19.0, 0.21.0, 0.24.0, 0.33.0); the complete map is in ''report_openwpm.mjs''. It is the reason the README asks for the version number:
+Every OpenWPM release pins one unbranded Firefox build, in one line of ''scripts/install-firefox.sh''. Reading that line out of every ''v*'' tag gives the mapping. Note before you read it as an installation recipe: for every release below v0.32.0 the build that line points at **no longer exists** — see [[#Known pitfalls]]. The table below omits seven point releases for width (0.11.0, 0.14.1, 0.16.0, 0.19.0, 0.21.0, 0.24.0, 0.33.0); the complete map is in ''report_openwpm.mjs''. It is the reason the README asks for the version number:
 ^ OpenWPM ^ Tagged ^ Firefox ^ ^ OpenWPM ^ Tagged ^ Firefox ^
@@ Line 226: / Line 226: @@
 | 0.21.1 | 2022-10-13 | 105 | | 0.34.0 | 2026-05-08 | 150 |
 | 0.22.0 | 2023-06-25 | 114 | | 0.35.0 | 2026-06-17 | 152 |
+| | | | | 0.36.0 | 2026-08-24 | 154 |
 The project is explicit about what it wants from researchers, and about why:
@@ Line 234: / Line 235: @@
 Both halves are routinely ignored. 15 of the 60 papers state a version, and the versions they state are old: a 2024 paper crawled with v0.17.0 of mid-2021 {[shaoor2024purl]}. The full table with each paper's lag is on [[#Which version, and how far behind]].
-**The pin is a default, not a guarantee — check before you infer the browser.** OpenWPM launches whatever ''FIREFOX_BINARY'' points at, so a paper can pair an old OpenWPM with a newer browser, and two of them do. CookieGraph {[shaoor2023cookiegraph]} and PURL {[shaoor2024purl]} both write "OpenWPM (v0.17.0) and Firefox (v102)", where v0.17.0 ships Firefox 90. They are also the only two papers of the fifteen that state both numbers for the crawl; a third {[ahmad2020_apophanies]} states "0.8; Firefox 52.9", which is consistent with the pin of that era. So the //Firefox// column in the table below is **nominal** — what that release ships — and for twelve of the fifteen papers it is the best a reader can do, which is the whole argument for reporting both.
+**The pin is a default, not a guarantee — check before you infer the browser.** OpenWPM launches whatever ''FIREFOX_BINARY'' points at, so a paper can pair an OpenWPM release with a different browser, and three of them do. **Nine of the fifteen state both numbers.** Two used a pre-0.10 release that pins nothing, so there is no pin to check them against ({[ahmad2020_apophanies]} "0.8 … Firefox v52.9"; {[fouad2022_cookie]}, whose setup table gives Firefox 68.0 and 45.0.1 for its two machines). Of the **seven that can be checked, three do not match**: CookieGraph {[shaoor2023cookiegraph]} and PURL {[shaoor2024purl]} both write "OpenWPM (v0.17.0) and Firefox (v102)" where v0.17.0 ships Firefox 90, and {[bottger2025_regional]} states "OpenWPM [24] (v0.27.0), which uses the Firefox browser (Version 123.0)" where v0.27.0 ships 122. The other four ran what their release pins — and one of them, {[demir2023_similarity]}, deliberately ran //two// browsers as an experimental variable: "the most recent stable Firefox version available when we started the experiment (v95.0 …) and a version that is roughly one year older (v86.0.1 …)", where v0.18.0 pins 95. So the //Firefox// column in the table below is **nominal** — what that release ships — and for the six papers that name no browser it is the best a reader can do, which is the whole argument for reporting both.((This count has been wrong twice, in the same way each time, and both are worth knowing before you trust a number like it. The hand map behind it was seeded on 2026-08-14 with ''/Firefox v?\d+/'' and found **3**; that regex misses "Firefox version 88" and a "Fire-|fox v121.0" cut by a column splice. A wide re-sweep on 2026-09-05 found **7** — but it was run only over papers a reviewer had named, so it still missed the two whose browser sits in a setup table. Re-swept over all fifteen: **9**. The sweep, the map and the hits the map //rejects// are all printed by ''report_openwpm.mjs''; see [[provenance:programming:crawler:openwpm|the provenance page]].))
 ===== Known pitfalls =====
+  * **It is written to be forked, not imported — and a fork does not survive an upgrade.** There is no ''openwpm'' package on PyPI and no stable public API surface: the documented path clones the repository, builds a conda environment inside it, and expects your crawl script to live in that tree. The obvious thing to do next is to edit the tree, and it is the mistake. Every subsequent upgrade is then a merge against a codebase that has repeatedly rewritten the parts you edited (the whole architecture at v0.10.0, the command and storage interfaces at v0.14.0), so in practice the upgrade never happens: **a crawler built by editing OpenWPM stops being buildable roughly when its pinned Firefox stops being downloadable** (next bullet), which is why a three-year-old artefact so often cannot be run by its own authors. Contrast [[https://github.com/duckduckgo/tracker-radar-collector|DuckDuckGo's Tracker Radar Collector]], which is a library you depend on.\\ **Do this instead: treat it as a library it does not claim to be.** Pin a tagged release as a submodule or a ''git clone %%--branch%% v0.xx.0'' in your Dockerfile, keep every line you write in your own repository, and express changes as ''BaseCommand'' subclasses where the API allows and **monkey patches** where it does not — so an upgrade is a version bump plus whatever patches broke, and the patches are a short, reviewable list of exactly where you diverge from upstream. A worked example is the ALSAcnc crawler {[bouhoula2024_automated]}, whose ''cookie_crawler/utils/monkey_patches/'' is five files and 583 lines rebinding ''BrowserManager'', ''BrowserManagerHandle'', ''CommandSequence'', the storage-controller shutdown and the default screen resolution; on that footing it tracked upstream through v0.23.0 → v0.29.0 → v0.31.0 → v0.34.0, and **four of its five patch files never changed** across the twelve releases v0.23.0 to v0.34.0 and four different pinned Firefoxes (115, 128, 134, 150). The fifth shows what an upgrade actually costs, and it is small: ''browser_manager.py'' stayed 130 lines but had to rebind ''BrowserManager.run_impl'' instead of ''BrowserManager.run'' when upstream renamed the method((''bouhoula/alsacnc'', ''docker/crawler/Dockerfile'': ''235a510'' (2024-06-22, v0.23.0), ''d5c449a'' (2024-09-24, v0.29.0), ''3c6cfd1'' (2025-01-29, v0.31.0) on ''main'', and ''4acb3fd'' (2026-07-10, v0.34.0) on branch ''cnil''. All five files are 23/130/194/199/37 lines on both branches, and four are byte-identical; ''browser_manager.py'' differs by the ''run''→''run_impl'' rebinding and one commented-out extension gate. Read 2026-08-14; the fourth pin, and this diff, were found on 2026-09-05 by review passes that looked past ''main''. Karel reports the project began on v0.21.x before this repository was published, which is not checkable from it.)).\\ The contrast with a fork is about **upgradability, not buildability**: today a fresh ''git clone %%--branch%% v0.31.0'' fails just as a fork of v0.31.0 would, because that release's Firefox is gone (next bullet). The difference is that the library-shaped project could //move//, and did — its current pin, v0.34.0, is one of the five whose build still downloads. A fork stops at the release it was cut from. Upstream agrees this is the right shape and says the project has not made it easy: "//Iirc other people keep the OpenWPM code and their code separated by using a git submodule. Tbh I always assumed that people would just check out one version and build on top of it (as you apparently have done) and never upgrade.\\ We should provide a better story here.//"((Maintainer ''vringar'' in [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], 2021-12-16, which also notes that #743 and #753 — an issue and a pull request respectively — were released in v0.14.0 and made keeping the two codebases separate easier. Read 2026-08-14, re-read 2026-09-05. The parenthetical is addressed to the person who opened the issue, who is the same practitioner whose experience this pitfall comes from; an earlier version of this page cut it silently, which is the one thing a quote must never do.))
+  * **The pinned Firefox build is deleted after about a year — mirror it or your crawl is not reproducible.** ''scripts/install-firefox.sh'' fetches the unbranded build from Mozilla's **CI index**, not from a release archive, and those artefacts expire: the index entry for the build 0.35.0 pins reports ''expires: 2027-06-10''. We resolved the pinned URL for every release from v0.10.0 — 29 of them on 2026-08-14, and the 30th, v0.36.0, on 2026-09-05: **only the five 2026 releases (Firefox 148–154) still download. Everything up to v0.31.0 — including the Firefox 134 pinned as recently as January 2025 — returns HTTP 404.** ''install.sh'' therefore fails on any older release, which is the state most published artefacts are in; the failure has been [[https://github.com/openwpm/OpenWPM/issues/964|open since 2021]], and the current script at least names it in the error message. Expiry is not the only way the build can be missing, either: **some Firefox releases never had an unbranded build under their tag at all.** OpenWPM skipped Firefox 153 for exactly that reason, and since v0.36.0 its ''scripts/firefox_version.py'' checks that a candidate revision actually has unbranded builds on every platform before pinning it, instead of pinning happily and failing with a 404 at download time((OpenWPM ''CHANGELOG.md'', v0.36.0: "''FIREFOX_153_0_3_RELEASE'' is the motivating case: its tagged revision has no unbranded builds on any platform, because 153.0.3's builds were produced from a later, untagged ''mozilla-release'' revision." Read 2026-09-05.)). The branded build of the same version is still on ''archive.mozilla.org'' and is **not** a substitute — the instrumenting extension needs privileged APIs that only an unbranded build will load.\\ **Do this instead:** archive the ''target.tar.*'' you actually crawled with, alongside your data, and give its version and checksum in the paper. If you are reviving someone else's crawl, the practical escape hatch is the container image — ''docker pull openwpm/openwpm:0.17.0'' still works today and the image bakes the Firefox binary in at ''/opt/firefox-bin''. **Tags go back to the first release of the current architecture**, but mind the naming: the four oldest are ''v''-prefixed (''v0.10.0'', ''v0.11.0'', ''v0.12.0'', ''v0.13.0''), everything from ''0.14.1'' on is not, and ''0.14.0'' was never published((Registry manifest lookups on 2026-09-05 against ''registry-1.docker.io/v2/openwpm/openwpm/manifests/<tag>''. An earlier version of this page said "tags go back to 0.15.0" — that came from probing unprefixed names only, so the ''v''-prefixed era read as absent and readers with a 2020–2021 artefact were told there was no image for it. All tags are ''linux/amd64''.)). Failing both, ask around: a handful of groups keep private copies of these tarballs and pass them between each other, which is not a reproducibility story anyone should be content with.((That last sentence is one practitioner's account, not something we could verify — no archive of these builds was found outside Mozilla's CI. It is here because a reader hunting a Firefox 80 tarball should know who to ask, not as evidence of anything.))
   * **An empty database looks exactly like a complete one.** A crawl in which every browser failed to launch still produces a ''crawl-data.sqlite'' containing all 13 of OpenWPM's tables (14 with SQLite's own ''sqlite_sequence''), all of them empty — including ''incomplete_visits'', because a visit that never started is never recorded as incomplete. We produced exactly that file (see [[#Installing it, and what we could and could not run]]). **Print row counts per table and check ''crawl_history.command_status'' before you analyse anything.** A crawl that half-failed is worse than one that failed: it looks like a low-tracking result.
   * **You are detectable, and specifically as OpenWPM.** Krumnow et al. {[krumnow2022_gullible]} analyse how detectable OpenWPM is in the wild, how resilient its recording is against a page that fights back, and how widespread OpenWPM-specific detection is. A tool 60 papers share is worth writing a detector for. ''bot_mitigation = True'' is not an answer to this: it performs up to ten random mouse moves, one scroll, and a random 1–7 second sleep, and its own documentation says these "aren't comprehensive and automated interaction with the site will still appear very bot-like".
-  * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page scores a paper that says "we ran under Xvfb" as not stating headlessness at all.
+  * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page has no slot for it: the two papers of the 60 whose text mentions Xvfb were coerced to //opposite// values, headless for the 1-million-site paper and headful for {[zeber2020representativeness]}. Neither is wrong exactly; the dichotomy is.
   * **It is Firefox, and only Firefox.** ''BrowserParams.browser'' accepts ''firefox'' and nothing else. A result about Chrome's behaviour cannot be obtained here, and a result about "the web" measured only in Firefox inherits Firefox's cookie policy, its tracking protection defaults and its API surface. The instrumenting extension is Manifest V2, which is itself part of why this is a Firefox tool.
-  * **Pin geckodriver, not just OpenWPM.** ''environment.yaml'' pins ''geckodriver=0.37.0''. With the current geckodriver release, **0.37.1** (2026-07-20), browser launch fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument --remote-allow-system-access can't be set via capabilities''. We measured this as a clean A/B: same Firefox, same Selenium 4.45.0, only the driver changed. Install from ''environment.yaml''; if you install geckodriver by hand, pin it.
+  * **Pin geckodriver, not just OpenWPM.** On v0.35.0 and every earlier release ''environment.yaml'' pinned ''geckodriver=0.37.0''. With geckodriver **0.37.1** (2026-07-20), browser launch on those releases fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument %%--remote-allow-system-access%% can't be set via capabilities''. We measured this as a clean A/B on v0.35.0: same Firefox, same Selenium 4.45.0, only the driver changed.\\ **v0.36.0 (2026-08-24) fixes it upstream** and pins ''geckodriver=0.37.1'' — it now asks for the privilege through geckodriver's own ''%%--allow-system-access%%'' flag instead((''openwpm/deploy_browsers/deploy_firefox.py'' at ''61d285f'' passes ''service_args=["%%--allow-system-access%%"]''; the CHANGELOG entry for v0.36.0 describes the capabilities route as a privilege escalation geckodriver now refuses, "which made every browser fail to launch and retry until the crawl gave up". Checked 2026-09-05.)). So: install from ''environment.yaml'' and let it choose the driver. This bullet is the shape of pitfall to expect from a one-maintainer project — real for three weeks, then fixed — which is the argument for dating everything you read about a tool, including this page.
   * **''tracking_protection'' does not work.** It is documented as **NOT SUPPORTED** ([[https://github.com/openwpm/OpenWPM/issues/101|issue #101]]). To measure with protections on you set the underlying ''about:config'' preferences through ''BrowserParams.prefs'' and report exactly which ones.
   * **Long crawls need the watchdogs, and they are off by default.** ''memory_watchdog'' and ''process_watchdog'' (which kills orphaned geckodriver and Xvfb processes) exist for cloud-scale crawls; ''maximum_profile_size'' recycles a browser whose profile has grown too large — relevant precisely in a stateful crawl, where the profile is what you are accumulating. ''failure_limit'' defaults to ''2 × num_browsers + 10'' consecutive command failures before the crawl aborts, which on a large list is a threshold you should choose deliberately.
-  * **Response bodies are de-duplicated by MD5.** ''save_content'' stores each distinct body once in LevelDB, keyed by content hash, and ''http_responses.content_hash'' is the join. Count responses in ''http_responses'', never rows in LevelDB.
+  * **The content hash is SHA-256, whatever the documentation says.** ''save_content'' stores each distinct body once in LevelDB keyed by that hash, and ''http_responses.content_hash'' is the join — so count responses in ''http_responses'', never rows in LevelDB. ''docs/Configuration.md'' calls it an **md5** hash in two places; the extension computes ''crypto.subtle.digest("SHA-256", …)''((''Extension/src/lib/sha256.ts'' and ''response-body-listener.ts'' at commit ''b9dd4c3''; ''docs/Configuration.md'' lines 419–420 still say md5. Hashing a body with MD5 to look it up will simply miss.)). Also: ''save_content'' does nothing unless you pass an unstructured storage provider to ''TaskManager'' — with ''None'' every body is dropped with one logged error and no other trace.
   * **''num_browsers'' is not free parallelism.** Each browser is a full Firefox with its own profile and its own state; see the per-browser cookie jar point above, and remember that concurrency changes what the sites see (N simultaneous connections from one address) and therefore what they serve {[jueckstock2021_realistic]}.
   * **Two repetitions of the same OpenWPM crawl do not agree.** Demir et al. {[demir2022_reproducibility]} measured the variation between repetitions of the same web measurement, and Demir et al. {[demir2023_similarity]} between the same measurement under different setups. Neither is an OpenWPM defect; both mean a single OpenWPM crawl is a point estimate.
@@ Line 251: / Line 254: @@
 ===== Maintenance status =====
-Checked 2026-08-14, against the repository and the GitHub, Docker Hub and conda-forge APIs (the script is ''openwpm_platform_probe.sh'', on [[provenance:programming:crawler:openwpm|the provenance page]]).
+Checked **2026-09-05**, against the repository and the GitHub, Docker Hub and conda-forge APIs (the script is ''openwpm_platform_probe.sh'', on [[provenance:programming:crawler:openwpm|the provenance page]]). Every row is a claim about that date; re-run the script before trusting it.
 ^ Signal ^ State on 2026-08-14 ^
-| Latest release | **0.35.0**, tagged 2026-06-17, bundling Firefox 152 |
+| Latest release | **0.36.0**, tagged 2026-08-24, bundling Firefox 154 |
-| Repository | ''github.com/openwpm/OpenWPM'', not archived; 1,415 stars, 332 forks, 180 open issues; last push 2026-08-06 |
+| Repository | ''github.com/openwpm/OpenWPM'', not archived; 1,419 stars, 334 forks, **141 open issues** and 39 open pull requests; last push 2026-08-25((GitHub's ''open_issues_count'' field, which several dashboards quote directly, is 180 — it counts pull requests as issues. The split here is from two ''search/issues'' queries.)) |
-| Release cadence, 2026 | four releases: 0.32.0 (Mar), 0.33.0 (Mar), 0.34.0 (May), 0.35.0 (Jun) — tracking Firefox 148→152 |
+| Release cadence, 2026 | **five releases**: 0.32.0 (Mar), 0.33.0 (Mar), 0.34.0 (May), 0.35.0 (Jun), 0.36.0 (Aug) — tracking Firefox 148→154 |
-| Release cadence, 2025 | **one release** (0.31.0, 19 January), then a 13-month gap; the newest release spent 2025 pinned to Firefox 134 while Firefox reached 147 |
+| Release cadence, 2025 | **one release** (0.31.0, 19 January), then a 13-month gap; the newest release spent 2025 pinned to Firefox 134 while Firefox reached 146 |
-| Commits per year | 569 (2019), 380 (2020), 85, 28, 23, **11 (2024), 12 (2025)**, 72 (2026 to 11 July) |
+| Commits per year | 569 (2019), 380 (2020), 85, 28, 23, **11 (2024), 12 (2025)**, 73 (2026, to ''61d285f'') — author date throughout((Committer date gives 74 for 2026. Mixing the two clocks is how an earlier version of this table reported 73 commits of which 72 were by one author: the total was on one clock and the split on the other.)) |
-| Contributors | 63 all-time; **72 of the 73 commits in 2026 are by one person** (''vringar'', Stefan Zabka) |
+| Contributors | 63 all-time; **all 73 commits in 2026 are by one person** (''vringar'', Stefan Zabka) |
-| CI | "Tests and linting" runs on a two-day cron and **passed on 2026-08-13**; CodeQL passing |
+| CI | "Tests and linting" runs on a two-day cron and **passed on 2026-09-05**; CodeQL passing |
-| Container image | ''openwpm/openwpm'' on Docker Hub, rebuilt per commit (''latest'' 2026-08-02) — **''linux/amd64'' only** |
+| Container image | ''openwpm/openwpm'' on Docker Hub, rebuilt per commit (''latest'' 2026-08-24) — **''linux/amd64'' only** |
 | Python packaging | none. There is no ''openwpm'' package on PyPI; installation is conda plus two shell scripts, or the container |
 | Platforms | Linux and macOS. **No Windows**((''OpenWPM does not support windows'' — README, and [[https://github.com/openwpm/OpenWPM/issues/503|issue #503]].)); no ARM Linux (see below) |
@@ Line 285: / Line 288: @@
 </code>
-Note the third row: an unbranded **Apple Silicon** build does exist (''macosx64-aarch64-add-on-devel''), but ''install.sh'' asks for ''macosx64'' and sets ''CONDA_SUBDIR=osx-64'', so on an M-series Mac OpenWPM installs the x86-64 build and runs under Rosetta. That works; it is worth knowing that it is what is happening.
+Note the third row: an unbranded **Apple Silicon** build does exist (''macosx64-aarch64-add-on-devel''), but ''install.sh'' asks for ''macosx64'' and sets ''CONDA_SUBDIR=osx-64'', so on an M-series Mac OpenWPM installs the x86-64 build and runs under Rosetta. That is the supported path rather than a bug — but no Mac was available to this run, so unlike everything else in this section it is read from ''install.sh'' and not from an execution.
 **2. Three of the pinned conda packages have no ARM Linux build.** The environment does not solve, and it is not a version-range problem — below, the solver's own output, then the conda-forge build counts it reflects:
@@ Line 317: / Line 320: @@
 </code>
-We did not isolate the cause and do not claim one: installing the same ''.xpi'' into the same Nightly by hand succeeds (''INSTALL OK id= openwpm@mozilla.org''), so the failure is in what the extension is allowed to //do// once loaded, not in loading it. The practical answer on ARM is the amd64 container under emulation, or an x86-64 machine.
+We did not isolate the cause and do not claim one: installing the same ''.xpi'' into the same Nightly by hand succeeds (''INSTALL OK id= openwpm@mozilla.org''), so the failure is in what the extension is allowed to //do// once loaded, not in loading it. The practical answer on ARM is the amd64 container under emulation, or an x86-64 machine. Neither could be tested here — this host has no Docker and cannot execute x86-64 binaries at all — so treat the first as what it is: the project's own recommended path (its README says OpenWPM "is commonly used via the docker container that this repo builds", and is tested on Ubuntu 24.04 in CI), on an image that exists, is rebuilt per commit and is amd64-only.
 What we //could// run on this host, and what it establishes:
@@ Line 352: / Line 355: @@
 from openwpm.commands.browser_commands import GetCommand
 from openwpm.config import BrowserParams, ManagerParams
+from openwpm.storage.leveldb import LevelDbProvider
 from openwpm.storage.sql_provider import SQLiteStorageProvider
 from openwpm.task_manager import TaskManager
@@ Line 371: / Line 375: @@
 bp.save_content = "script"  # response bodies for scripts only, into LevelDB
 browser_params = [bp]
+# save_content needs an UNSTRUCTURED storage provider as well. Pass None and
+# OpenWPM logs one error per body and drops it (storage_controller.py: "Tried to
+# save content while not having provided any unstructured storage provider"),
+# which no per-table row count can see — exactly the silent-instrument failure
+# this script exists to catch.
+CONTENT = DATA / "content.ldb"
 with TaskManager(
@@ Line 376: / Line 387: @@
     browser_params,
     SQLiteStorageProvider(DATA / "crawl-data.sqlite"),
-    None,
+    LevelDbProvider(CONTENT),
 ) as manager:
     for i, site in enumerate(SITES):
@@ Line 414: / Line 425: @@
 for row in db.execute("select hostname, addresses, is_TRR from dns_responses"):
     print("  %-24s %-30s trr=%s" % row)
+# The bodies do not live in SQLite, so a per-table row count cannot see them.
+print("\nresponse bodies in LevelDB:")
+try:
+    import plyvel
+    db2 = plyvel.DB(str(CONTENT))
+    print(f"  {sum(1 for _ in db2.iterator(include_value=False))} distinct bodies"
+          f" (keyed by content_hash)")
+except Exception as e:  # no plyvel, or nothing was written
+    print(f"  could not read {CONTENT}: {e}")
 </file>
@@ Line 435: / Line 457: @@
 | IEEE S&P | 1 | 1.7% | 110 | 0.9% |
-**OpenWPM is a PETS instrument first.** Nearly a fifth of PoPETs crawling papers use it, against 3% or less at CCS, NDSS and USENIX Security and one paper in fifteen years at IEEE S&P. If your reviewers come from the security venues, they will not treat "we used OpenWPM" as self-explanatory; if they come from PETS, they may.
+**OpenWPM is a PETS instrument first.** Nearly a fifth of PoPETs crawling papers use it, against about 3% at CCS, NDSS and USENIX Security, and one paper in the whole corpus at IEEE S&P. If your reviewers come from the security venues, they will not treat "we used OpenWPM" as self-explanatory; if they come from PETS, they may.
 ^ Year ^ OpenWPM papers ^ Crawling papers ^ OpenWPM's share ^
@@ Line 451: / Line 473: @@
 | 2026* | 1 | 69 | 1.4% |
-Use in every year from 2015 on, and in the eleven complete years 2015–2024 no year is above 12% or below 3%: OpenWPM is a steady minority instrument, not a rising or falling one. The 2025 and 2026 rows are lower (4.7% and 1.4%) and should not be read as a decline — those are the provisional venue-years, and 2026 is one paper.
+Use in every year from 2015 on, and in the ten complete years 2015–2024 no year is above 12% or below 3%: OpenWPM is a steady minority instrument, not a rising or falling one. The 2025 and 2026 rows are lower (4.7% and 1.4%) and should not be read as a decline — those are the provisional venue-years, and 2026 is one paper.
 ==== Which version, and how far behind ====
@@ Line 460: / Line 482: @@
 | {[englehardt2015_cookies]} Cookies that give you away (2015) | 0.1.0 | //pre-tag// | ? | 0.2.1 (2014-12-31) | — |
 | {[ahmad2020_apophanies]} Apophanies or Epiphanies (2020) | 0.8; Firefox 52.9 | //pre-0.10 architecture// | ? | 0.8.0 (2017-10-09) | — |
-| {[fouad2022my]} My Cookie is a Phoenix (2022) | 0.9.0 (machine A); 0.7.0 (machine B) | //no such tag// | ? | 0.18.0 (2021-12-12) | — |
+| {[fouad2022_cookie]} My Cookie is a Phoenix (2022) | 0.9.0 (machine A); 0.7.0 (machine B) | //no such tag// | ? | 0.18.0 (2021-12-12) | — |
 | {[bollinger2022automating]} Automating consent and violation detection (2022) | 0.12.0 | 2020-08-26 | 80 | 0.18.0 (2021-12-12) | 7 |
 | {[iqbal2022_khaleesi]} Khaleesi (2022) | 0.10.0 | 2020-06-22 | 77 | 0.18.0 (2021-12-12) | 9 |
@@ Line 474: / Line 496: @@
 | {[bottger2025_regional]} Regional filter lists (2025) | 0.27.0 | 2024-02-08 | 122 | 0.30.0 (2024-10-02) | 3 |
-Three of the fifteen name a version that was never tagged (0.9.0, 0.13, and the 0.1.0 of 2015), so a reader cannot even resolve them to a browser. The best-documented row is arguably {[rasaii2025_crumbs]}, which states the OpenWPM version //and// the Firefox version it implies: "BannerClick is built on top of OpenWPM [24] (version 0.26.0), which uses Firefox v121.0".
+Three of the fifteen name a version that was never tagged (0.9.0, 0.13, and the 0.1.0 of 2015), so a reader cannot resolve them to a browser from the pin — though {[fouad2022_cookie]}, one of the three, states its browsers outright. **Two rows name two versions, and only the first is resolved**: {[fouad2024_devil]} used 0.14.0 //and// 0.19.1, and the "Behind 15" is for 0.14.0 (0.19.1 is 8 behind); {[fouad2022_cookie]} used 0.9.0 on one machine and 0.7.0 on another, and the "no such tag" is true of 0.9.0 only — v0.7.0 was tagged on 2016-11-15. The report's ''normalise()'' deliberately takes the first version it finds, and the verbatim string is printed beside it so the second is never lost; the lag column is not defined for a paper that used two. The best-documented row is arguably {[rasaii2025_crumbs]}, which states the OpenWPM version //and// the Firefox version it implies: "BannerClick is built on top of OpenWPM [24] (version 0.26.0), which uses Firefox v121.0".
 ==== What papers say about the configuration ====
@@ Line 505: / Line 527: @@
   * **As a reader**, you cannot tell from most papers whether the ''javascript'' table was populated. If a paper reports "no fingerprinting on X% of sites" without saying that ''js_instrument'' was on and with what settings, the absence may be the instrument's.
-  * **As an author**, one sentence fixes it: //"OpenWPM 0.35.0 (Firefox 152), stateless, ''http_instrument'', ''js_instrument'' with ''collection_fingerprinting'' plus WebGL, ''cookie_instrument'', ''display_mode='xvfb''', ''num_browsers=8''."// That is fully reproducible and costs a line. One paper in the corpus does close to this and is worth copying: {[agarwal2020_stop]} writes that its "updates include enabling the ''http_instrument'' which logs HTTP responses, requests and redirects, using the selenium headless browser to perform crawling and, setting ''js_instrument'', ''cookie_instrum''[''ent'']…", and names the extensions it turned //off// as well.
+  * **As an author**, one sentence fixes it: //"OpenWPM 0.36.0 (Firefox 154), stateless, ''http_instrument'', ''js_instrument'' with ''collection_fingerprinting'' plus WebGL, ''cookie_instrument'', ''display_mode='xvfb''', ''num_browsers=8''."// That is fully reproducible and costs a line. One paper in the corpus does close to this and is worth copying: {[agarwal2020_stop]} writes that its "updates include enabling the ''http_instrument'' which logs HTTP responses, requests and redirects, using the selenium headless browser to perform crawling and, setting ''js_instrument'', ''cookie_instrument''…", and names the extensions it turned //off// as well.
 ==== What they use it for ====
-Enum counts over ''classification[].target'' tuples in the 60 papers, publishable as a ranking. The catch-all ''other'' bucket (24 tuples) is omitted from the table but would rank fourth:
+Enum counts over ''classification[].target'' tuples in the 60 papers, publishable as a ranking. This is the top nine; the catch-all ''other'' (24 tuples) would rank fifth and is omitted, as are ''malware'' and ''javascript'' (3 each) below the cut:
 ^ What the paper classified ^ Tuples ^
@@ Line 526: / Line 548: @@
 ==== Forks and extensions are the norm, not the exception ====
-Three of the 60 papers //produced// something rather than merely using it, and several more describe local modifications:
+Three of the 60 papers //produced// rather than used: OpenWPM itself {[englehardt2016online]}, OpenWPM-Mobile {[das2018_sixth]}, and one paper's own extended instrumentation {[acar2020_noboundaries]}. Several more describe local modifications:
   * **OpenWPM-Mobile** {[das2018_sixth]} adds mobile-focused instrumentation; OmniCrawl {[cassel2022_omnicrawl]} later ran it alongside real mobile browsers and used the comparison to show what emulation misses.
   * **Extended JS instrumentation.** "No boundaries" {[acar2020_noboundaries]} extended OpenWPM's JavaScript instrumentation to a further set of DOM properties and function calls — the standard move when ''collection_fingerprinting'' is not enough, and the one to copy.
   * Papers also describe "a lightly modified version of OpenWPM" without saying what was modified. If you extend it, publish the diff with your artefact; a modified instrument with no published modification is not a shared instrument.
+  * **How you modify it decides whether your crawler outlives the paper.** A fork you edit in place cannot practically be upgraded; the same changes written as ''BaseCommand'' subclasses and monkey patches over a pinned release can be, and have been. This is the first entry under [[#Known pitfalls]], with a worked example.
 **33 of the 60 (55.0%) release an artefact publicly**, against 51.8% of other crawling papers — no better than the field, for a tool whose whole value is shared instrumentation.
@@ Line 558: / Line 581: @@
 ===== What to report if you use it =====
-  - **The OpenWPM release and the Firefox version it bundles.** ''0.35.0 (Firefox 152)''. Not "the latest version", not "OpenWPM".
+  - **The OpenWPM release and the Firefox version it bundles.** ''0.36.0 (Firefox 154)''. Not "the latest version", not "OpenWPM". And if you pointed ''FIREFOX_BINARY'' at a different build, say so — of the seven papers whose stated browser can be checked against a pin, three ran something else.
   - **Which instruments were enabled**, by their parameter names. If you touched ''js_instrument_settings'', give the API list or point to it in your artefact.
   - **Stateful or stateless, and ''num_browsers''.** Together, not separately: they jointly determine how many cookie jars your "user" had.
@@ Line 564: / Line 587: @@
   - **''tp_cookies'', ''donottrack'', and any ''prefs''** you set. The defaults are a policy choice, and ''tp_cookies'' defaults to accepting everything.
   - **Failure accounting**: how many visits are in ''incomplete_visits'', how many commands have a non-''ok'' ''command_status'', and what you did about them. This is the number nobody publishes and every reviewer should ask for.
-  - **Your diff**, if you modified the instrumentation, with the artefact (see [[:Artifacts]], a page this wiki still owes you).
+  - **The Firefox binary itself**, archived with your artefact and checksummed. Mozilla's CI deletes the unbranded build about a year after it is produced, so a version number alone does not let anyone rebuild your crawler (see [[#Known pitfalls]]).
+  - **Your diff**, if you modified the instrumentation, with the artefact (see [[:Artifacts]]).
+  - **The citation the project asks for**, which is the CCS 2016 paper {[englehardt2016online]} — its README says so — in the same sentence as the version, so a reader gets the method and the build together.
 ===== Open Questions =====
-<wrap todo>
+<WRAP todo>
   * Nobody has published what OpenWPM's default ''collection_fingerprinting'' misses on a modern fingerprinting corpus. The list has not changed since 2020 and the surface has; measuring the gap is a small paper.
   * How much the unbranded Firefox build differs from a stock Firefox of the same version, in fingerprint and in behaviour, is unmeasured — the same question [[Programming:Crawler]] raises for Playwright's patched builds.
-  * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property; no paper we found reports ''num_browsers'' alongside a stateful claim.
+  * Whether the per-browser cookie-jar partition in a stateful multi-browser crawl changes published results, and by how much. Every stateful OpenWPM crawl with ''num_browsers > 1'' has this property, and papers do report the instance count — {[englehardt2018_email]} "configure OpenWPM to run 15 measurement instances in parallel" while loading each email once statelessly and once statefully, and {[englehardt2016online]} runs 20 — but none we found treats the resulting //N// cookie jars as a property of the measurement rather than a throughput setting, or reports what it costs.((An earlier version of this question claimed no paper reports ''num_browsers'' alongside a stateful claim. That was wrong, and it was wrong in the way an Open Question most easily is: an absence asserted from the papers that were read rather than from a query. Corrected 2026-09-05 after a review pass read the two papers.))
   * Where ARM support would come from: Mozilla's ''add-on-devel'' build matrix, or an OpenWPM-side path that does not need an unbranded build.
-</wrap>
+</WRAP>
 ====== References ======