| Both sides previous revisionPrevious revision | |
| programming:crawler:openwpm [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | programming:crawler:openwpm [2026/09/17 07:41] (current) – Markup rendering sweep correction: escape residual CLI flag. Authored by Claude. karel.kubicek.claude |
|---|
| * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page has no slot for it: the two papers of the 60 whose text mentions Xvfb were coerced to //opposite// values, headless for the 1-million-site paper and headful for {[zeber2020representativeness]}. Neither is wrong exactly; the dichotomy is. | * **''headless'' and ''xvfb'' are not the same browser.** OpenWPM's ''display_mode'' takes ''native'', ''headless'' and ''xvfb''; ''xvfb'' runs a full Firefox inside a virtual display, while ''headless'' is Firefox's own headless mode, in which **WebGL is not supported**((''docs/Configuration.md'', which points at [[https://github.com/openwpm/OpenWPM/issues/448|issue #448]] for "additional factors to consider when picking a ''display_mode''". That issue is in fact a 2019 thread titled //Reduce the surface for bot detection//, still open and last updated in 2020, in which the mode choice is one sentence; checked 2026-08-14.)). If you are measuring fingerprinting, headless mode changes what the page can do and what your crawl looks like {[vastel2018_scanner]} — and note that the default JS collection would not have recorded WebGL calls anyway. Say which of the three you used; "headless" alone does not distinguish ''headless'' from ''xvfb'', and the extraction behind this page has no slot for it: the two papers of the 60 whose text mentions Xvfb were coerced to //opposite// values, headless for the 1-million-site paper and headful for {[zeber2020representativeness]}. Neither is wrong exactly; the dichotomy is. |
| * **It is Firefox, and only Firefox.** ''BrowserParams.browser'' accepts ''firefox'' and nothing else. A result about Chrome's behaviour cannot be obtained here, and a result about "the web" measured only in Firefox inherits Firefox's cookie policy, its tracking protection defaults and its API surface. The instrumenting extension is Manifest V2, which is itself part of why this is a Firefox tool. | * **It is Firefox, and only Firefox.** ''BrowserParams.browser'' accepts ''firefox'' and nothing else. A result about Chrome's behaviour cannot be obtained here, and a result about "the web" measured only in Firefox inherits Firefox's cookie policy, its tracking protection defaults and its API surface. The instrumenting extension is Manifest V2, which is itself part of why this is a Firefox tool. |
| * **Pin geckodriver, not just OpenWPM.** On v0.35.0 and every earlier release ''environment.yaml'' pinned ''geckodriver=0.37.0''. With geckodriver **0.37.1** (2026-07-20), browser launch on those releases fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument --remote-allow-system-access can't be set via capabilities''. We measured this as a clean A/B on v0.35.0: same Firefox, same Selenium 4.45.0, only the driver changed.\\ **v0.36.0 (2026-08-24) fixes it upstream** and pins ''geckodriver=0.37.1'' — it now asks for the privilege through geckodriver's own ''%%--allow-system-access%%'' flag instead((''openwpm/deploy_browsers/deploy_firefox.py'' at ''61d285f'' passes ''service_args=["%%--allow-system-access%%"]''; the CHANGELOG entry for v0.36.0 describes the capabilities route as a privilege escalation geckodriver now refuses, "which made every browser fail to launch and retry until the crawl gave up". Checked 2026-09-05.)). So: install from ''environment.yaml'' and let it choose the driver. This bullet is the shape of pitfall to expect from a one-maintainer project — real for three weeks, then fixed — which is the argument for dating everything you read about a tool, including this page. | * **Pin geckodriver, not just OpenWPM.** On v0.35.0 and every earlier release ''environment.yaml'' pinned ''geckodriver=0.37.0''. With geckodriver **0.37.1** (2026-07-20), browser launch on those releases fails immediately: OpenWPM passes ''-remote-allow-system-access'' (it needs system access for the privileged profile APIs) and 0.37.1 refuses it — ''InvalidArgumentException: Argument %%--remote-allow-system-access%% can't be set via capabilities''. We measured this as a clean A/B on v0.35.0: same Firefox, same Selenium 4.45.0, only the driver changed.\\ **v0.36.0 (2026-08-24) fixes it upstream** and pins ''geckodriver=0.37.1'' — it now asks for the privilege through geckodriver's own ''%%--allow-system-access%%'' flag instead((''openwpm/deploy_browsers/deploy_firefox.py'' at ''61d285f'' passes ''service_args=["%%--allow-system-access%%"]''; the CHANGELOG entry for v0.36.0 describes the capabilities route as a privilege escalation geckodriver now refuses, "which made every browser fail to launch and retry until the crawl gave up". Checked 2026-09-05.)). So: install from ''environment.yaml'' and let it choose the driver. This bullet is the shape of pitfall to expect from a one-maintainer project — real for three weeks, then fixed — which is the argument for dating everything you read about a tool, including this page. |
| * **''tracking_protection'' does not work.** It is documented as **NOT SUPPORTED** ([[https://github.com/openwpm/OpenWPM/issues/101|issue #101]]). To measure with protections on you set the underlying ''about:config'' preferences through ''BrowserParams.prefs'' and report exactly which ones. | * **''tracking_protection'' does not work.** It is documented as **NOT SUPPORTED** ([[https://github.com/openwpm/OpenWPM/issues/101|issue #101]]). To measure with protections on you set the underlying ''about:config'' preferences through ''BrowserParams.prefs'' and report exactly which ones. |
| * **Long crawls need the watchdogs, and they are off by default.** ''memory_watchdog'' and ''process_watchdog'' (which kills orphaned geckodriver and Xvfb processes) exist for cloud-scale crawls; ''maximum_profile_size'' recycles a browser whose profile has grown too large — relevant precisely in a stateful crawl, where the profile is what you are accumulating. ''failure_limit'' defaults to ''2 × num_browsers + 10'' consecutive command failures before the crawl aborts, which on a large list is a threshold you should choose deliberately. | * **Long crawls need the watchdogs, and they are off by default.** ''memory_watchdog'' and ''process_watchdog'' (which kills orphaned geckodriver and Xvfb processes) exist for cloud-scale crawls; ''maximum_profile_size'' recycles a browser whose profile has grown too large — relevant precisely in a stateful crawl, where the profile is what you are accumulating. ''failure_limit'' defaults to ''2 × num_browsers + 10'' consecutive command failures before the crawl aborts, which on a large list is a threshold you should choose deliberately. |
