| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler:openwpm [2026/09/05 14:15] – Verification pass on the 2026-09-05 corrections: the corrected Firefox-version count was itself short — 9 of 15 papers state a crawl browser, not 7 (the re-sweep had run over reviewer-named slugs, not the population), split 4 match the pin / 3 differ / 2 karel.kubicek.claude | programming:crawler:openwpm [2026/09/17 07:41] (current) – Markup rendering sweep correction: escape residual CLI flag. Authored by Claude. karel.kubicek.claude |
|---|
| ===== Known pitfalls ===== | ===== Known pitfalls ===== |
| |
| * **It is written to be forked, not imported — and a fork does not survive an upgrade.** There is no ''openwpm'' package on PyPI and no stable public API surface: the documented path clones the repository, builds a conda environment inside it, and expects your crawl script to live in that tree. The obvious thing to do next is to edit the tree, and it is the mistake. Every subsequent upgrade is then a merge against a codebase that has repeatedly rewritten the parts you edited (the whole architecture at v0.10.0, the command and storage interfaces at v0.14.0), so in practice the upgrade never happens: **a crawler built by editing OpenWPM stops being buildable roughly when its pinned Firefox stops being downloadable** (next bullet), which is why a three-year-old artefact so often cannot be run by its own authors. Contrast [[https://github.com/duckduckgo/tracker-radar-collector|DuckDuckGo's Tracker Radar Collector]], which is a library you depend on.\\ **Do this instead: treat it as a library it does not claim to be.** Pin a tagged release as a submodule or a ''git clone --branch v0.xx.0'' in your Dockerfile, keep every line you write in your own repository, and express changes as ''BaseCommand'' subclasses where the API allows and **monkey patches** where it does not — so an upgrade is a version bump plus whatever patches broke, and the patches are a short, reviewable list of exactly where you diverge from upstream. A worked example is the ALSAcnc crawler {[bouhoula2024_automated]}, whose ''cookie_crawler/utils/monkey_patches/'' is five files and 583 lines rebinding ''BrowserManager'', ''BrowserManagerHandle'', ''CommandSequence'', the storage-controller shutdown and the default screen resolution; on that footing it tracked upstream through v0.23.0 → v0.29.0 → v0.31.0 → v0.34.0, and **four of its five patch files never changed** across the twelve releases v0.23.0 to v0.34.0 and four different pinned Firefoxes (115, 128, 134, 150). The fifth shows what an upgrade actually costs, and it is small: ''browser_manager.py'' stayed 130 lines but had to rebind ''BrowserManager.run_impl'' instead of ''BrowserManager.run'' when upstream renamed the method((''bouhoula/alsacnc'', ''docker/crawler/Dockerfile'': ''235a510'' (2024-06-22, v0.23.0), ''d5c449a'' (2024-09-24, v0.29.0), ''3c6cfd1'' (2025-01-29, v0.31.0) on ''main'', and ''4acb3fd'' (2026-07-10, v0.34.0) on branch ''cnil''. All five files are 23/130/194/199/37 lines on both branches, and four are byte-identical; ''browser_manager.py'' differs by the ''run''→''run_impl'' rebinding and one commented-out extension gate. Read 2026-08-14; the fourth pin, and this diff, were found on 2026-09-05 by review passes that looked past ''main''. Karel reports the project began on v0.21.x before this repository was published, which is not checkable from it.)).\\ The contrast with a fork is about **upgradability, not buildability**: today a fresh ''git clone --branch v0.31.0'' fails just as a fork of v0.31.0 would, because that release's Firefox is gone (next bullet). The difference is that the library-shaped project could //move//, and did — its current pin, v0.34.0, is one of the five whose build still downloads. A fork stops at the release it was cut from. Upstream agrees this is the right shape and says the project has not made it easy: "//Iirc other people keep the OpenWPM code and their code separated by using a git submodule. Tbh I always assumed that people would just check out one version and build on top of it (as you apparently have done) and never upgrade.\\ We should provide a better story here.//"((Maintainer ''vringar'' in [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], 2021-12-16, which also notes that #743 and #753 — an issue and a pull request respectively — were released in v0.14.0 and made keeping the two codebases separate easier. Read 2026-08-14, re-read 2026-09-05. The parenthetical is addressed to the person who opened the issue, who is the same practitioner whose experience this pitfall comes from; an earlier version of this page cut it silently, which is the one thing a quote must never do.)) | * **It is written to be forked, not imported — and a fork does not survive an upgrade.** There is no ''openwpm'' package on PyPI and no stable public API surface: the documented path clones the repository, builds a conda environment inside it, and expects your crawl script to live in that tree. The obvious thing to do next is to edit the tree, and it is the mistake. Every subsequent upgrade is then a merge against a codebase that has repeatedly rewritten the parts you edited (the whole architecture at v0.10.0, the command and storage interfaces at v0.14.0), so in practice the upgrade never happens: **a crawler built by editing OpenWPM stops being buildable roughly when its pinned Firefox stops being downloadable** (next bullet), which is why a three-year-old artefact so often cannot be run by its own authors. Contrast [[https://github.com/duckduckgo/tracker-radar-collector|DuckDuckGo's Tracker Radar Collector]], which is a library you depend on.\\ **Do this instead: treat it as a library it does not claim to be.** Pin a tagged release as a submodule or a ''git clone %%--branch%% v0.xx.0'' in your Dockerfile, keep every line you write in your own repository, and express changes as ''BaseCommand'' subclasses where the API allows and **monkey patches** where it does not — so an upgrade is a version bump plus whatever patches broke, and the patches are a short, reviewable list of exactly where you diverge from upstream. A worked example is the ALSAcnc crawler {[bouhoula2024_automated]}, whose ''cookie_crawler/utils/monkey_patches/'' is five files and 583 lines rebinding ''BrowserManager'', ''BrowserManagerHandle'', ''CommandSequence'', the storage-controller shutdown and the default screen resolution; on that footing it tracked upstream through v0.23.0 → v0.29.0 → v0.31.0 → v0.34.0, and **four of its five patch files never changed** across the twelve releases v0.23.0 to v0.34.0 and four different pinned Firefoxes (115, 128, 134, 150). The fifth shows what an upgrade actually costs, and it is small: ''browser_manager.py'' stayed 130 lines but had to rebind ''BrowserManager.run_impl'' instead of ''BrowserManager.run'' when upstream renamed the method((''bouhoula/alsacnc'', ''docker/crawler/Dockerfile'': ''235a510'' (2024-06-22, v0.23.0), ''d5c449a'' (2024-09-24, v0.29.0), ''3c6cfd1'' (2025-01-29, v0.31.0) on ''main'', and ''4acb3fd'' (2026-07-10, v0.34.0) on branch ''cnil''. All five files are 23/130/194/199/37 lines on both branches, and four are byte-identical; ''browser_manager.py'' differs by the ''run''→''run_impl'' rebinding and one commented-out extension gate. Read 2026-08-14; the fourth pin, and this diff, were found on 2026-09-05 by review passes that looked past ''main''. Karel reports the project began on v0.21.x before this repository was published, which is not checkable from it.)).\\ The contrast with a fork is about **upgradability, not buildability**: today a fresh ''git clone %%--branch%% v0.31.0'' fails just as a fork of v0.31.0 would, because that release's Firefox is gone (next bullet). The difference is that the library-shaped project could //move//, and did — its current pin, v0.34.0, is one of the five whose build still downloads. A fork stops at the release it was cut from. Upstream agrees this is the right shape and says the project has not made it easy: "//Iirc other people keep the OpenWPM code and their code separated by using a git submodule. Tbh I always assumed that people would just check out one version and build on top of it (as you apparently have done) and never upgrade.\\ We should provide a better story here.//"((Maintainer ''vringar'' in [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], 2021-12-16, which also notes that #743 and #753 — an issue and a pull request respectively — were released in v0.14.0 and made keeping the two codebases separate easier. Read 2026-08-14, re-read 2026-09-05. The parenthetical is addressed to the person who opened the issue, who is the same practitioner whose experience this pitfall comes from; an earlier version of this page cut it silently, which is the one thing a quote must never do.)) |
| * **The pinned Firefox build is deleted after about a year — mirror it or your crawl is not reproducible.** ''scripts/install-firefox.sh'' fetches the unbranded build from Mozilla's **CI index**, not from a release archive, and those artefacts expire: the index entry for the build 0.35.0 pins reports ''expires: 2027-06-10''. We resolved the pinned URL for every release from v0.10.0 — 29 of them on 2026-08-14, and the 30th, v0.36.0, on 2026-09-05: **only the five 2026 releases (Firefox 148–154) still download. Everything up to v0.31.0 — including the Firefox 134 pinned as recently as January 2025 — returns HTTP 404.** ''install.sh'' therefore fails on any older release, which is the state most published artefacts are in; the failure has been [[https://github.com/openwpm/OpenWPM/issues/964|open since 2021]], and the current script at least names it in the error message. Expiry is not the only way the build can be missing, either: **some Firefox releases never had an unbranded build under their tag at all.** OpenWPM skipped Firefox 153 for exactly that reason, and since v0.36.0 its ''scripts/firefox_version.py'' checks that a candidate revision actually has unbranded builds on every platform before pinning it, instead of pinning happily and failing with a 404 at download time((OpenWPM ''CHANGELOG.md'', v0.36.0: "''FIREFOX_153_0_3_RELEASE'' is the motivating case: its tagged revision has no unbranded builds on any platform, because 153.0.3's builds were produced from a later, untagged ''mozilla-release'' revision." Read 2026-09-05.)). The branded build of the same version is still on ''archive.mozilla.org'' and is **not** a substitute — the instrumenting extension needs privileged APIs that only an unbranded build will load.\\ **Do this instead:** archive the ''target.tar.*'' you actually crawled with, alongside your data, and give its version and checksum in the paper. If you are reviving someone else's crawl, the practical escape hatch is the container image — ''docker pull openwpm/openwpm:0.17.0'' still works today and the image bakes the Firefox binary in at ''/opt/firefox-bin''. **Tags go back to the first release of the current architecture**, but mind the naming: the four oldest are ''v''-prefixed (''v0.10.0'', ''v0.11.0'', ''v0.12.0'', ''v0.13.0''), everything from ''0.14.1'' on is not, and ''0.14.0'' was never published((Registry manifest lookups on 2026-09-05 against ''registry-1.docker.io/v2/openwpm/openwpm/manifests/<tag>''. An earlier version of this page said "tags go back to 0.15.0" — that came from probing unprefixed names only, so the ''v''-prefixed era read as absent and readers with a 2020–2021 artefact were told there was no image for it. All tags are ''linux/amd64''.)). Failing both, ask around: a handful of groups keep private copies of these tarballs and pass them between each other, which is not a reproducibility story anyone should be content with.((That last sentence is one practitioner's account, not something we could verify — no archive of these builds was found outside Mozilla's CI. It is here because a reader hunting a Firefox 80 tarball should know who to ask, not as evidence of anything.)) | * **The pinned Firefox build is deleted after about a year — mirror it or your crawl is not reproducible.** ''scripts/install-firefox.sh'' fetches the unbranded build from Mozilla's **CI index**, not from a release archive, and those artefacts expire: the index entry for the build 0.35.0 pins reports ''expires: 2027-06-10''. We resolved the pinned URL for every release from v0.10.0 — 29 of them on 2026-08-14, and the 30th, v0.36.0, on 2026-09-05: **only the five 2026 releases (Firefox 148–154) still download. Everything up to v0.31.0 — including the Firefox 134 pinned as recently as January 2025 — returns HTTP 404.** ''install.sh'' therefore fails on any older release, which is the state most published artefacts are in; the failure has been [[https://github.com/openwpm/OpenWPM/issues/964|open since 2021]], and the current script at least names it in the error message. Expiry is not the only way the build can be missing, either: **some Firefox releases never had an unbranded build under their tag at all.** OpenWPM skipped Firefox 153 for exactly that reason, and since v0.36.0 its ''scripts/firefox_version.py'' checks that a candidate revision actually has unbranded builds on every platform before pinning it, instead of pinning happily and failing with a 404 at download time((OpenWPM ''CHANGELOG.md'', v0.36.0: "''FIREFOX_153_0_3_RELEASE'' is the motivating case: its tagged revision has no unbranded builds on any platform, because 153.0.3's builds were produced from a later, untagged ''mozilla-release'' revision." Read 2026-09-05.)). The branded build of the same version is still on ''archive.mozilla.org'' and is **not** a substitute — the instrumenting extension needs privileged APIs that only an unbranded build will load.\\ **Do this instead:** archive the ''target.tar.*'' you actually crawled with, alongside your data, and give its version and checksum in the paper. If you are reviving someone else's crawl, the practical escape hatch is the container image — ''docker pull openwpm/openwpm:0.17.0'' still works today and the image bakes the Firefox binary in at ''/opt/firefox-bin''. **Tags go back to the first release of the current architecture**, but mind the naming: the four oldest are ''v''-prefixed (''v0.10.0'', ''v0.11.0'', ''v0.12.0'', ''v0.13.0''), everything from ''0.14.1'' on is not, and ''0.14.0'' was never published((Registry manifest lookups on 2026-09-05 against ''registry-1.docker.io/v2/openwpm/openwpm/manifests/<tag>''. An earlier version of this page said "tags go back to 0.15.0" — that came from probing unprefixed names only, so the ''v''-prefixed era read as absent and readers with a 2020–2021 artefact were told there was no image for it. All tags are ''linux/amd64''.)). Failing both, ask around: a handful of groups keep private copies of these tarballs and pass them between each other, which is not a reproducibility story anyone should be content with.((That last sentence is one practitioner's account, not something we could verify — no archive of these builds was found outside Mozilla's CI. It is here because a reader hunting a Firefox 80 tarball should know who to ask, not as evidence of anything.)) |
| * **An empty database looks exactly like a complete one.** A crawl in which every browser failed to launch still produces a ''crawl-data.sqlite'' containing all 13 of OpenWPM's tables (14 with SQLite's own ''sqlite_sequence''), all of them empty — including ''incomplete_visits'', because a visit that never started is never recorded as incomplete. We produced exactly that file (see [[#Installing it, and what we could and could not run]]). **Print row counts per table and check ''crawl_history.command_status'' before you analyse anything.** A crawl that half-failed is worse than one that failed: it looks like a low-tracking result. | * **An empty database looks exactly like a complete one.** A crawl in which every browser failed to launch still produces a ''crawl-data.sqlite'' containing all 13 of OpenWPM's tables (14 with SQLite's own ''sqlite_sequence''), all of them empty — including ''incomplete_visits'', because a visit that never started is never recorded as incomplete. We produced exactly that file (see [[#Installing it, and what we could and could not run]]). **Print row counts per table and check ''crawl_history.command_status'' before you analyse anything.** A crawl that half-failed is worse than one that failed: it looks like a low-tracking result. |
| * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page has no slot for it: the two papers of the 60 whose text mentions Xvfb were coerced to //opposite// values, headless for the 1-million-site paper and headful for {[zeber2020representativeness]}. Neither is wrong exactly; the dichotomy is. | * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page has no slot for it: the two papers of the 60 whose text mentions Xvfb were coerced to //opposite// values, headless for the 1-million-site paper and headful for {[zeber2020representativeness]}. Neither is wrong exactly; the dichotomy is. |
| * **It is Firefox, and only Firefox.** ''BrowserParams.browser'' accepts ''firefox'' and nothing else. A result about Chrome's behaviour cannot be obtained here, and a result about "the web" measured only in Firefox inherits Firefox's cookie policy, its tracking protection defaults and its API surface. The instrumenting extension is Manifest V2, which is itself part of why this is a Firefox tool. | * **It is Firefox, and only Firefox.** ''BrowserParams.browser'' accepts ''firefox'' and nothing else. A result about Chrome's behaviour cannot be obtained here, and a result about "the web" measured only in Firefox inherits Firefox's cookie policy, its tracking protection defaults and its API surface. The instrumenting extension is Manifest V2, which is itself part of why this is a Firefox tool. |
| * **Pin geckodriver, not just OpenWPM.** On v0.35.0 and every earlier release ''environment.yaml'' pinned ''geckodriver=0.37.0''. With geckodriver **0.37.1** (2026-07-20), browser launch on those releases fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument --remote-allow-system-access can't be set via capabilities''. We measured this as a clean A/B on v0.35.0: same Firefox, same Selenium 4.45.0, only the driver changed.\\ **v0.36.0 (2026-08-24) fixes it upstream** and pins ''geckodriver=0.37.1'' — it now asks for the privilege through geckodriver's own ''--allow-system-access'' flag instead((''openwpm/deploy_browsers/deploy_firefox.py'' at ''61d285f'' passes ''service_args=["--allow-system-access"]''; the CHANGELOG entry for v0.36.0 describes the capabilities route as a privilege escalation geckodriver now refuses, "which made every browser fail to launch and retry until the crawl gave up". Checked 2026-09-05.)). So: install from ''environment.yaml'' and let it choose the driver. This bullet is the shape of pitfall to expect from a one-maintainer project — real for three weeks, then fixed — which is the argument for dating everything you read about a tool, including this page. | * **Pin geckodriver, not just OpenWPM.** On v0.35.0 and every earlier release ''environment.yaml'' pinned ''geckodriver=0.37.0''. With geckodriver **0.37.1** (2026-07-20), browser launch on those releases fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument %%--remote-allow-system-access%% can't be set via capabilities''. We measured this as a clean A/B on v0.35.0: same Firefox, same Selenium 4.45.0, only the driver changed.\\ **v0.36.0 (2026-08-24) fixes it upstream** and pins ''geckodriver=0.37.1'' — it now asks for the privilege through geckodriver's own ''%%--allow-system-access%%'' flag instead((''openwpm/deploy_browsers/deploy_firefox.py'' at ''61d285f'' passes ''service_args=["%%--allow-system-access%%"]''; the CHANGELOG entry for v0.36.0 describes the capabilities route as a privilege escalation geckodriver now refuses, "which made every browser fail to launch and retry until the crawl gave up". Checked 2026-09-05.)). So: install from ''environment.yaml'' and let it choose the driver. This bullet is the shape of pitfall to expect from a one-maintainer project — real for three weeks, then fixed — which is the argument for dating everything you read about a tool, including this page. |
| * **''tracking_protection'' does not work.** It is documented as **NOT SUPPORTED** ([[https://github.com/openwpm/OpenWPM/issues/101|issue #101]]). To measure with protections on you set the underlying ''about:config'' preferences through ''BrowserParams.prefs'' and report exactly which ones. | * **''tracking_protection'' does not work.** It is documented as **NOT SUPPORTED** ([[https://github.com/openwpm/OpenWPM/issues/101|issue #101]]). To measure with protections on you set the underlying ''about:config'' preferences through ''BrowserParams.prefs'' and report exactly which ones. |
| * **Long crawls need the watchdogs, and they are off by default.** ''memory_watchdog'' and ''process_watchdog'' (which kills orphaned geckodriver and Xvfb processes) exist for cloud-scale crawls; ''maximum_profile_size'' recycles a browser whose profile has grown too large — relevant precisely in a stateful crawl, where the profile is what you are accumulating. ''failure_limit'' defaults to ''2 × num_browsers + 10'' consecutive command failures before the crawl aborts, which on a large list is a threshold you should choose deliberately. | * **Long crawls need the watchdogs, and they are off by default.** ''memory_watchdog'' and ''process_watchdog'' (which kills orphaned geckodriver and Xvfb processes) exist for cloud-scale crawls; ''maximum_profile_size'' recycles a browser whose profile has grown too large — relevant precisely in a stateful crawl, where the profile is what you are accumulating. ''failure_limit'' defaults to ''2 × num_browsers + 10'' consecutive command failures before the crawl aborts, which on a large list is a threshold you should choose deliberately. |
