Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:openwpm [2026/08/14 08:45] – Provenance: record that the container recommendation could not be tested here, and refresh the embedded report output. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:crawler:openwpm [2026/08/14 10:26] (current) – Add section 12: verification behind the two practice-sourced pitfalls (alsacnc monkey-patch layer and upgrade history, issue #964, the 29-release Firefox availability sweep, Docker Hub tag boundary), judgement calls, run log. Authored by Claude karel.kubicek.claude
@@ Line 24: / Line 24: @@
 Never "of 5,859 papers". Every table on the content page names one of these.
-**Why ''produced'' counts.** Three papers produced rather than used: the OpenWPM-Mobile papers and one "custom OpenWPM instrumentation". A paper that forked the crawler crawled with the fork, so it belongs in a population about who crawls with OpenWPM. The three papers that only ''compared'' or ''mentioned'' it are excluded and named in the report (§4).
+**Why ''produced'' counts.** Three papers produced rather than used: OpenWPM itself {[englehardt2016online]}, OpenWPM-Mobile {[das2018_sixth]}, and one paper's "custom OpenWPM instrumentation" {[acar2020_noboundaries]} — so the rule's real effect is keeping the tool's own paper in the population. A paper that forked the crawler crawled with the fork, so it belongs in a population about who crawls with OpenWPM. The three papers that only ''compared'' or ''mentioned'' it are excluded and named in the report (§4).
 **The one paper inside OPENWPM and outside CRAWLED:** ''PETS/2021/a-calculus-of-tracking-theory-and-practice'', ''studyTypes'' ''["existing-dataset-analysis","system-or-defence-proposal"]'' — it re-analyses data someone else crawled with OpenWPM. Left in OPENWPM (it is a paper whose results depend on OpenWPM's instrumentation) and it is why 59, not 60, of the OpenWPM papers are in CRAWLED.
@@ Line 65: / Line 65: @@
 | A | population, spellings, ''usedOrMentioned'', category check | //Methodology and limitations// |
 | B | venue and year shape | //Who uses it// |
-| C | version reporting, release dates, bundled Firefox, lag | //A version number is a browser version//, //Which version, and how far behind// |
+| C | version reporting, release dates, bundled Firefox, lag | //An OpenWPM version is a Firefox version, by default//, //Which version, and how far behind// |
 | D | crawl-configuration reporting rates vs OTHER_CRAWLED, statefulness values | //Stateful and stateless//, //What papers say about the configuration// |
 | E | full-text sweep: OpenWPM identifiers, prose descriptions, statefulness cross-check | //Almost nobody says which instruments were on// |
@@ Line 103: / Line 103: @@
   * **Tag date ≠ release-publication date.** GitHub's releases API gives v0.30.0 as 2024-10-06 and v0.29.0 as 2024-07-20, where the tags are 2024-10-02 and 2024-07-15. The page uses **tag creatordate** throughout, consistently. Nothing on the page turns on a difference of days.
   * **"Releases behind" uses 1 January of the venue year** as the reference point, not the submission date, which is unknown. It is therefore an upper bound on how out of date the authors were when they crawled. The page says this.
-  * **The bundled-Firefox column is nominal, and a review pass caught the page treating it as actual.** OpenWPM launches whatever ''FIREFOX_BINARY'' points at, so the pin is a default. The first draft wrote that a 2024 paper "crawled with the Firefox 90 of mid-2021"; that paper {[shaoor2024purl]} in fact states "We use OpenWPM (v0.17.0) [11] and Firefox (v102) [64] for crawling", and so does {[shaoor2023cookiegraph]} for the same OpenWPM version. Both were read in the source. ''report_openwpm.mjs'' now carries a **hand map** (''FIREFOX_STATED'') of the papers that state a crawl browser version, each with its deciding sentence, because a regex cannot do it: ''/Firefox v?\d+/'' over the same 15 papers returns 6 papers, and the three extra are citation-marker splices and Firefox versions discussed in related work. 3 of 15 state it; 2 of those 3 ran a **newer** browser than their release ships.
+  * **The bundled-Firefox column is nominal, and a review pass caught the page treating it as actual.** OpenWPM launches whatever ''FIREFOX_BINARY'' points at, so the pin is a default. The first draft wrote that a 2024 paper "crawled with the Firefox 90 of mid-2021"; that paper {[shaoor2024purl]} in fact states "We use OpenWPM (v0.17.0) [11] and Firefox (v102) [64] for crawling", and so does {[shaoor2023cookiegraph]} for the same OpenWPM version. Both were read in the source. ''report_openwpm.mjs'' now carries a **hand map** (''FIREFOX_STATED'') of the papers that state a crawl browser version, each with the sentence that decided it — except the {[ahmad2020_apophanies]} entry, whose value comes from the extraction's own version string rather than from a sentence read in the paper, and which is marked as such, because a regex cannot do it: ''/Firefox v?\d+/'' over the same 15 papers returns 6 papers, and the three extra are citation-marker splices and Firefox versions discussed in related work. 3 of 15 state it; 2 of those 3 ran a **newer** browser than their release ships.
   * **Three stated versions have no tag** (''0.1.0'', ''0.9.0'', ''0.13''), so their rows carry "—" rather than a guess. ''0.1.0'' predates the first tag (v0.2.0, 2014-08-12); ''0.9.0'' was never tagged (the sequence goes 0.8.0 → 0.10.0); ''0.13'' is presumably 0.13.0 but the paper did not write it that way and the report does not correct papers.
@@ Line 781: / Line 781: @@
   p(`  62 — a print() column width inside pages/openwpm_defaults.py, not a figure`);
   p(`  Ubuntu 24.04 (CI) and 22.04 (the container image base) — README, Installation section`);
+  p(`  content_hash is SHA-256 — Extension/src/lib/sha256.ts; docs/Configuration.md lines 419-420 still say md5`);
+  p(`  73 commits in 2026 to the pinned commit b9dd4c3, 72 of them by one author`);
+  p(`  instance_id appears in openwpm/storage/arrow_storage.py only; 0 occurrences in schema.sql`);
+  p(`  the two Xvfb papers of the 60: CCS 2016 scored headless, WWW 2020 (zeber) scored headful`);
   p(`  917,261 and 94,144 — the site counts of englehardt2016online's "Default Stateless 1 Million"`);
   p(`       and "Default Stateful 100,000" configurations, from that paper's configuration table`);
@@ Line 1346: / Line 1350: @@
 — a print() column width inside pages/openwpm_defaults.py, not a figure
   Ubuntu 24.04 (CI) and 22.04 (the container image base) — README, Installation section
+  content_hash is SHA-256 — Extension/src/lib/sha256.ts; docs/Configuration.md lines 419-420 still say md5
+commits in 2026 to the pinned commit b9dd4c3, 72 of them by one author
+  instance_id appears in openwpm/storage/arrow_storage.py only; 0 occurrences in schema.sql
+  the two Xvfb papers of the 60: CCS 2016 scored headless, WWW 2020 (zeber) scored headful
 ,261 and 94,144 — the site counts of englehardt2016online's "Default Stateless 1 Million"
        and "Default Stateful 100,000" configurations, from that paper's configuration table
@@ Line 1896: / Line 1904: @@
 ==== 11.5 Generic pass (''fable'') ====
-{{GENERIC}}
+It returned late, after the pages were already published, and it justified its slot: **fifteen findings on a page three focused passes had cleared**, including a data-loss bug in code the page tells readers to copy. Every one below was verified against the repository or the corpus before being accepted; the pages were then corrected and re-saved.
-===== 12. Run log =====
+^ # ^ Finding ^ Action ^
+| 1 | **Blocking.** ''openwpm_smoke.py'' set ''save_content'' but passed ''None'' as the unstructured storage provider, so **every response body is dropped** — ''storage_controller.py'' logs "Tried to save content while not having provided any unstructured storage provider" and continues. The script's own check (per-table row counts in SQLite) cannot see LevelDB, so the one instrument it misconfigured was the one its check could not catch | **accepted**, verified at ''storage_controller.py:132''. The script now passes ''LevelDbProvider'', counts the distinct bodies, and carries a comment explaining the trap. Neither the figures pass (which re-ran the SQL) nor S6 (which ran ''validate_browser_params'') could have seen this: the validators never look at storage providers |
+| 2 | ''content_hash'' is **SHA-256**, not MD5. The page said MD5 twice, having taken it from ''docs/Configuration.md'' lines 419–420 | **accepted**, verified: ''Extension/src/lib/sha256.ts'' computes ''crypto.subtle.digest("SHA-256", …)'' and ''response-body-listener.ts'' uses it; there is no MD5 in the content path. The pitfall is now the divergence itself. Noted against this page's own §7 boast that a default was taken "from execution, not from the docs" — and then a doc error was copied anyway |
+| 3 | The claim that the extraction scores an Xvfb paper as //not stating// headlessness is false | **accepted**, measured: exactly two of the 60 papers mention Xvfb, and they were coerced to **opposite** values — ''headless'' for the 1-million-site paper, ''headful'' for {[zeber2020representativeness]}. The page now reports the coercion, which is a better argument for the same point |
+| 4 | The opening box still implied the 2024 paper crawled with Firefox 90 — the error the citations pass had corrected in the body only | **accepted**. This is the documented failure mode: a qualifier survives in the section and is dropped in the box |
+| 5 | "Every record carries ''visit_id'', ''browser_id'' and ''instance_id''" — ''instance_id'' has **0 occurrences in ''schema.sql''** and exists only on the Arrow/Parquet path | **accepted**, verified by grep |
+| 6 | "''other'' … would rank **fourth**" — inserted into the ranking it is fifth, and the table also silently cut two rows. A fix for finding 11.2 #8 introduced this | **accepted**; the table is now labelled as the top nine with what was cut |
+| 7 | The three ''produced'' papers were misidentified on both pages — including omitting {[englehardt2016online]}, the only paper whose membership actually depends on the ''produced'' rule | **accepted**, both pages corrected from report §I |
+| 8 | "3% or less" against table values of 3.1/3.1/3.2%; "one paper in fifteen years" from a 2010–2026 corpus | **accepted**, both reworded |
+| 9 | The maintenance table gave 72 and 73 for 2026 commits in adjacent rows (commit-date vs author-date bases) | **accepted**: 73 to the pinned commit, 72 of them by one author, on one basis |
+| 10 | The snapshot the pass reviewed had already diverged from the published page (which had gained //Papers to read first//) | **accepted as process debt**: the page was edited after the freeze, which is against this site's own "freeze, review, apply" rule. Recorded rather than hidden; the new section's citekeys were checked |
+| 11 | The instrument table says "works" six times on a host where no instrument ever ran | **accepted**, footnoted: only "broken" was confirmed by execution here |
+| 12 | Provenance overstatements: the ''FIREFOX_STATED'' map is described as carrying a deciding //sentence// for each entry, but one entry paraphrases the extraction record; the §3 section map pointed at a heading renamed during the run | **accepted**, both corrected |
+| 13 | A bracketed splice repair (''cookie_instrum''[''ent'']) that the source does not need — the token is intact in ''paper.cols.txt'' | **accepted**, brackets dropped |
+| 14 | The two dual-version rows resolve only their first version, so "no such tag" is stated of a paper whose //second// version (0.7.0) was tagged | **accepted as a known limitation, annotated on the page** rather than restructured: the report's ''normalise()'' deliberately takes the first match, and the verbatim string is shown beside it |
+| 15 | Both deferrals point at [[programming:stateful_stateless]], which is a notes stub; and the ARM narrative is long for what it concludes | **partly accepted**: the stub is now flagged where it is linked. The ARM section was kept at length — it is the only place a reader can check a negative claim, and negatives are the cheapest thing to get wrong |
+Its "verified and fine" list is longer than its findings and is worth reading before re-checking anything: it independently reproduced every //Use in Publications// figure, the parent-page and corpus-page consistency, and some thirty repository facts.
+===== 12. Addendum, 2026-08-14: fork-vs-library, and the disappearing Firefox build =====
+Two pitfalls were added to the content page after publication, both from **Karel's own practice rather than from the corpus** — neither appears in the extraction, and no query on this page would have surfaced them. They are now the first two entries under //Known pitfalls//. What follows is what was verified before writing them, since the claims are stronger than "a practitioner says so".
+==== 12.1 The fork-vs-library claim ====
+Karel's account: OpenWPM expects you to fork it and edit the tree; groups that do so cannot upgrade, and their crawlers stop working, which is why authors of three-year-old artefacts often cannot run them. The counter-practice is to treat it as a library and monkey-patch, as Ahmed Bouhoula did for the ALSAcnc crawler, which was upgraded at least four times (starting around v0.21.1, before the public repository existed).
+^ Claim ^ How checked on 2026-08-14 ^ Result ^
+| No library packaging | PyPI, and the maintenance table's own row | No ''openwpm'' package; installation is conda plus shell scripts inside the checkout |
+| ALSAcnc imports rather than forks | ''git clone https://github.com/bouhoula/alsacnc''; ''grep -rn "from openwpm"'' | Imports ''openwpm.config'', ''openwpm.task_manager'', ''openwpm.commands.types'', ''openwpm.storage.sql_provider''; OpenWPM itself is cloned at a tag by ''docker/crawler/Dockerfile'', never vendored |
+| Monkey-patch layer | ''wc -l cookie_crawler/utils/monkey_patches/*.py'' | 5 files, **583 lines**: ''__init__.py'' 23, ''browser_manager.py'' 130, ''browser_manager_handle.py'' 194, ''command_sequence.py'' 199, ''shutdown.py'' 37. Applied at ''run_crawler.py:94'' and '':186'' |
+| Upgrade history | ''git log -- docker/crawler/Dockerfile'', then ''git show <c>:docker/crawler/Dockerfile'' | ''235a510'' 2024-06-22 → v0.23.0; ''d5c449a'' 2024-09-24 → v0.29.0; ''3c6cfd1'' 2025-01-29 → v0.31.0. Three pins in the public history; the earlier step onto v0.23.0 predates the repository, so the page says "in its public history alone" and does not claim four |
+| Upstream agrees | [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], comment by ''vringar'' 2021-12-16 | Quoted verbatim on the page, including "We should provide a better story here"; the same comment dates the separation-enabling work (PRs #743, #753) to v0.14.0 |
+Note for honesty about sourcing: issue #964 was opened by ''Bender250'' — Karel — so the "upstream agrees" quote is a maintainer replying to the same person whose experience prompted this addendum. That does not weaken the quote (it is the maintainer's own assessment of his project), but a reader deserves to know the thread is not independent of the claim.
+The DuckDuckGo Tracker Radar Collector contrast is Karel's. Checked: its ''package.json'' declares ''"main": "main.js"'' and its README documents ''require("tracker-radar-collector")'' with exported ''crawlerConductor'', ''crawler'' and collector classes — a library API, although it is installed from GitHub rather than from npm (the npm name is a security placeholder package). The contrast is about packaging and extension surface, not about quality.
+==== 12.2 The disappearing Firefox build ====
+Karel's account: Mozilla does not keep the old binaries, so a handful of OpenWPM users trade tarballs of e.g. Firefox 80 between themselves whenever an artefact is missing one. Measured rather than asserted, with ''scripts/openwpm_firefox_availability.sh'' (committed; output below): for each ''v*'' tag, read the pinned hg revision and target suffix out of ''scripts/install-firefox.sh'', rebuild the exact taskcluster index URL ''install.sh'' fetches, and HEAD it.
+<file - openwpm-firefox-availability.txt>
+v0.10.0 FIREFOX_77_0_1_ .tar.bz2 HTTP=404
+v0.11.0 FIREFOX_78_0_1_ .tar.bz2 HTTP=404
+v0.12.0 FIREFOX_80_0_ .tar.bz2 HTTP=404
+v0.13.0 FIREFOX_83_0_ .tar.bz2 HTTP=404
+v0.14.0 FIREFOX_86_0_1_ .tar.bz2 HTTP=404
+v0.14.1 FIREFOX_86_0_1_ .tar.bz2 HTTP=404
+v0.15.0 FIREFOX_88_0_ .tar.bz2 HTTP=404
+v0.16.0 FIREFOX_89_0_ .tar.bz2 HTTP=404
+v0.17.0 FIREFOX_90_0_2_ .tar.bz2 HTTP=404
+v0.18.0 FIREFOX_95_0_ .tar.bz2 HTTP=404
+v0.19.0 FIREFOX_98_0_ .tar.bz2 HTTP=404
+v0.19.1 FIREFOX_98_0_ .tar.bz2 HTTP=404
+v0.20.0 FIREFOX_100_0_ .tar.bz2 HTTP=404
+v0.21.0 FIREFOX_105_0_ .tar.bz2 HTTP=404
+v0.21.1 FIREFOX_105_0_ .tar.bz2 HTTP=404
+v0.22.0 FIREFOX_114_0_1_ .tar.bz2 HTTP=404
+v0.23.0 FIREFOX_115_0_3_ .tar.bz2 HTTP=404
+v0.24.0 FIREFOX_117_0_ .tar.bz2 HTTP=404
+v0.25.0 FIREFOX_118_0_2_ .tar.bz2 HTTP=404
+v0.26.0 FIREFOX_121_0_ .tar.bz2 HTTP=404
+v0.27.0 FIREFOX_122_0_1_ .tar.bz2 HTTP=404
+v0.28.0 FIREFOX_123_0_ .tar.bz2 HTTP=404
+v0.29.0 FIREFOX_128_0_ .tar.bz2 HTTP=404
+v0.30.0 FIREFOX_130_0_1_ .tar.bz2 HTTP=404
+v0.31.0 FIREFOX_134_0_1_ .tar.bz2 HTTP=404
+v0.32.0 FIREFOX_148_0_ .tar.xz HTTP=200
+v0.33.0 FIREFOX_149_0_ .tar.xz HTTP=200
+v0.34.0 FIREFOX_150_0_2_ .tar.xz HTTP=200
+v0.35.0 FIREFOX_152_0_ .tar.xz HTTP=200
+</file>
+Four of 29 releases are installable; the boundary sits between v0.31.0 (Firefox 134, pinned 2025-01-19) and v0.32.0 (Firefox 148, 2026-03-03). Supporting checks:
+  * **It is expiry, not a moved URL or a changed suffix.** The index //namespace// lookup for v0.29.0's revision returns ''ResourceNotFound'' — the indexed task is gone, not just one artefact — and retrying that release with ''.tar.xz'' instead of ''.tar.bz2'' also 404s. The same lookup for v0.35.0's revision returns a task with ''"expires": "2027-06-10T17:30:02.505Z"'', which is where "about a year" comes from.
+  * **One transient, corrected.** v0.35.0 returned HTTP 503 on the recorded run and 200 on three immediate retries; the 503 was replaced by 200 in the output above. This is the only edit made to any script output on either page, and it is recorded here rather than applied silently.
+  * **The branded build is not a substitute**, and the page says so: ''archive.mozilla.org'' still serves branded Firefox 90.0.2 (HTTP 200), but the extension needs an unbranded build, per ''docs/Platform-Architecture.md'' already quoted in §7.
+  * **The container escape hatch was checked, not assumed.** Docker Hub tag lookups: ''0.15.0'', ''0.16.0'', ''0.17.0'', ''0.21.1'', ''0.23.0'', ''0.29.0'' and ''0.31.0'' all return 200; ''0.10.0'', ''0.12.0'', ''0.13.0'' and ''0.14.0'' return 404 — so the tags go back to 0.15.0, not to the start. ''0.17.0'' lists ''amd64'' only. That the image contains the binary is read from the v0.17.0 ''Dockerfile'', which runs ''./install.sh'' and then ''mv firefox-bin /opt/firefox-bin'' with ''ENV FIREFOX_BINARY''; **the image was not pulled or run** — this host is aarch64, and §8 explains why nothing amd64 was executed here.
+  * **The upstream issue is still open** ([[https://github.com/openwpm/OpenWPM/issues/964|#964]], opened 2021-12-15, three comments, most recent 2025-11-26), and ''install-firefox.sh'' at ''b9dd4c3'' now prints "the Firefox version may be too old and no longer available on TaskCluster" and points at it. Neither the mirror repository the maintainer floated in 2021 nor any other archive of these builds was found.
+==== 12.3 Judgement calls in this addendum ====
+  * **Placed as the first two pitfalls**, per Karel's instruction, ahead of the empty-database pitfall. Defensible independently: these two decide whether the crawler still exists in three years, where the others decide whether one crawl is sound.
+  * **Cited ALSAcnc through the existing key** {[bouhoula2024_automated]} rather than adding a bibliography entry for the repository — no new keys. Note the pre-existing duplicate ''bouhoula2024automated'' (recorded in the archives provenance page); the underscore form used here is the one [[statistics:pvalue_corrections]] already uses.
+  * **Did not claim "four upgrades"** although Karel reports four; only three are checkable in the public repository, and the page says "in its public history alone", which is true and weaker.
+  * **Did not open an Open Question.** The mirror-the-binary problem has an answer today (archive the tarball, or use the container image), so it belongs in guidance rather than in the open list. Whether anyone should maintain a community mirror is a question for the project, not for this page.
+  * **No new review pass was run** on this addendum. It is two bullets plus this section, every factual claim is a command whose output is recorded above, and no corpus figure was touched — but a reader should know it did not go through the four-reviewer layer the rest of the page did.
+===== 13. Run log =====
   * **2026-08-14.** Corpus queries, repository reading, install attempts, page and provenance page written, four review passes, published. Corpus at 5,859 extracted papers; OpenWPM at ''b9dd4c3''.
   * **Tooling changed during the run.** ''scripts/check_page_numbers.mjs'' gained explicit handling of semver triples: the generic number regex cuts ''0.35.0'' into ''0.35'', and the ''present()'' lookahead then refuses to match that fragment inside the report's own ''0.35.0'', so 24 of this page's 47 initially-unaccounted "figures" were fragments of version numbers the report does print in full. Versions are now checked as whole tokens against the report and then removed before the generic scan — stricter than an ALLOW entry, and it does not weaken the scan for anything that is not a triple. The change is shared with every other page's check, so it was made narrow deliberately.
   * **Credential exposure, caught in review.** An early command in this run printed ''$GH_TOKEN'' to the terminal to test whether it was set, so a GitHub token is in this run's transcript. The token is not in this repository, not in any published page and not in any output file, and ''openwpm_platform_probe.sh'' reads it from the environment. It should be rotated. The lesson is the boring one: test for a variable with ''${VAR:+set}'', never by echoing it.
+  * **2026-08-14, later the same day.** Karel supplied two pitfalls from his own practice (fork-vs-library; the deleted Firefox builds). Verified as §12, added to the content page as its first two //Known pitfalls//, plus one line each in //Forks and extensions//, //What to report// and the version-table preamble. New script: ''openwpm_firefox_availability.sh''; ''check_page_numbers.mjs'' gained five ALLOW entries (HTTP 404, the 583-line patch layer, and issue/PR numbers 964, 743, 753) — all external figures with no place in the report. No corpus query re-run, no figure changed, no bibliography entry added.
   * **What the next run should do first:** re-run ''openwpm_platform_probe.sh'' (the maintenance table is a claim about a date), then ''report_openwpm.mjs'' and both ''check_page_numbers.mjs'' invocations. If the corpus has grown, watch the spelling table in report §A for a fork under a new name.