| Both sides previous revisionPrevious revision | |
| provenance:programming:crawler:openwpm [2026/08/14 08:54] – Provenance: the generic (fable) review pass returned after publication with 15 findings on a page three focused passes had cleared, including a data-loss bug in published code; log all 15 with verification and disposition, and correct three overstatements karel.kubicek.claude | provenance:programming:crawler:openwpm [2026/08/14 10:26] (current) – Add section 12: verification behind the two practice-sourced pitfalls (alsacnc monkey-patch layer and upgrade history, issue #964, the 29-release Firefox availability sweep, Docker Hub tag boundary), judgement calls, run log. Authored by Claude karel.kubicek.claude |
|---|
| Its "verified and fine" list is longer than its findings and is worth reading before re-checking anything: it independently reproduced every //Use in Publications// figure, the parent-page and corpus-page consistency, and some thirty repository facts. | Its "verified and fine" list is longer than its findings and is worth reading before re-checking anything: it independently reproduced every //Use in Publications// figure, the parent-page and corpus-page consistency, and some thirty repository facts. |
| |
| ===== 12. Run log ===== | ===== 12. Addendum, 2026-08-14: fork-vs-library, and the disappearing Firefox build ===== |
| | |
| | Two pitfalls were added to the content page after publication, both from **Karel's own practice rather than from the corpus** — neither appears in the extraction, and no query on this page would have surfaced them. They are now the first two entries under //Known pitfalls//. What follows is what was verified before writing them, since the claims are stronger than "a practitioner says so". |
| | |
| | ==== 12.1 The fork-vs-library claim ==== |
| | |
| | Karel's account: OpenWPM expects you to fork it and edit the tree; groups that do so cannot upgrade, and their crawlers stop working, which is why authors of three-year-old artefacts often cannot run them. The counter-practice is to treat it as a library and monkey-patch, as Ahmed Bouhoula did for the ALSAcnc crawler, which was upgraded at least four times (starting around v0.21.1, before the public repository existed). |
| | |
| | ^ Claim ^ How checked on 2026-08-14 ^ Result ^ |
| | | No library packaging | PyPI, and the maintenance table's own row | No ''openwpm'' package; installation is conda plus shell scripts inside the checkout | |
| | | ALSAcnc imports rather than forks | ''git clone https://github.com/bouhoula/alsacnc''; ''grep -rn "from openwpm"'' | Imports ''openwpm.config'', ''openwpm.task_manager'', ''openwpm.commands.types'', ''openwpm.storage.sql_provider''; OpenWPM itself is cloned at a tag by ''docker/crawler/Dockerfile'', never vendored | |
| | | Monkey-patch layer | ''wc -l cookie_crawler/utils/monkey_patches/*.py'' | 5 files, **583 lines**: ''__init__.py'' 23, ''browser_manager.py'' 130, ''browser_manager_handle.py'' 194, ''command_sequence.py'' 199, ''shutdown.py'' 37. Applied at ''run_crawler.py:94'' and '':186'' | |
| | | Upgrade history | ''git log -- docker/crawler/Dockerfile'', then ''git show <c>:docker/crawler/Dockerfile'' | ''235a510'' 2024-06-22 → v0.23.0; ''d5c449a'' 2024-09-24 → v0.29.0; ''3c6cfd1'' 2025-01-29 → v0.31.0. Three pins in the public history; the earlier step onto v0.23.0 predates the repository, so the page says "in its public history alone" and does not claim four | |
| | | Upstream agrees | [[https://github.com/openwpm/OpenWPM/issues/964|issue #964]], comment by ''vringar'' 2021-12-16 | Quoted verbatim on the page, including "We should provide a better story here"; the same comment dates the separation-enabling work (PRs #743, #753) to v0.14.0 | |
| | |
| | Note for honesty about sourcing: issue #964 was opened by ''Bender250'' — Karel — so the "upstream agrees" quote is a maintainer replying to the same person whose experience prompted this addendum. That does not weaken the quote (it is the maintainer's own assessment of his project), but a reader deserves to know the thread is not independent of the claim. |
| | |
| | The DuckDuckGo Tracker Radar Collector contrast is Karel's. Checked: its ''package.json'' declares ''"main": "main.js"'' and its README documents ''require("tracker-radar-collector")'' with exported ''crawlerConductor'', ''crawler'' and collector classes — a library API, although it is installed from GitHub rather than from npm (the npm name is a security placeholder package). The contrast is about packaging and extension surface, not about quality. |
| | |
| | ==== 12.2 The disappearing Firefox build ==== |
| | |
| | Karel's account: Mozilla does not keep the old binaries, so a handful of OpenWPM users trade tarballs of e.g. Firefox 80 between themselves whenever an artefact is missing one. Measured rather than asserted, with ''scripts/openwpm_firefox_availability.sh'' (committed; output below): for each ''v*'' tag, read the pinned hg revision and target suffix out of ''scripts/install-firefox.sh'', rebuild the exact taskcluster index URL ''install.sh'' fetches, and HEAD it. |
| | |
| | <file - openwpm-firefox-availability.txt> |
| | v0.10.0 FIREFOX_77_0_1_ .tar.bz2 HTTP=404 |
| | v0.11.0 FIREFOX_78_0_1_ .tar.bz2 HTTP=404 |
| | v0.12.0 FIREFOX_80_0_ .tar.bz2 HTTP=404 |
| | v0.13.0 FIREFOX_83_0_ .tar.bz2 HTTP=404 |
| | v0.14.0 FIREFOX_86_0_1_ .tar.bz2 HTTP=404 |
| | v0.14.1 FIREFOX_86_0_1_ .tar.bz2 HTTP=404 |
| | v0.15.0 FIREFOX_88_0_ .tar.bz2 HTTP=404 |
| | v0.16.0 FIREFOX_89_0_ .tar.bz2 HTTP=404 |
| | v0.17.0 FIREFOX_90_0_2_ .tar.bz2 HTTP=404 |
| | v0.18.0 FIREFOX_95_0_ .tar.bz2 HTTP=404 |
| | v0.19.0 FIREFOX_98_0_ .tar.bz2 HTTP=404 |
| | v0.19.1 FIREFOX_98_0_ .tar.bz2 HTTP=404 |
| | v0.20.0 FIREFOX_100_0_ .tar.bz2 HTTP=404 |
| | v0.21.0 FIREFOX_105_0_ .tar.bz2 HTTP=404 |
| | v0.21.1 FIREFOX_105_0_ .tar.bz2 HTTP=404 |
| | v0.22.0 FIREFOX_114_0_1_ .tar.bz2 HTTP=404 |
| | v0.23.0 FIREFOX_115_0_3_ .tar.bz2 HTTP=404 |
| | v0.24.0 FIREFOX_117_0_ .tar.bz2 HTTP=404 |
| | v0.25.0 FIREFOX_118_0_2_ .tar.bz2 HTTP=404 |
| | v0.26.0 FIREFOX_121_0_ .tar.bz2 HTTP=404 |
| | v0.27.0 FIREFOX_122_0_1_ .tar.bz2 HTTP=404 |
| | v0.28.0 FIREFOX_123_0_ .tar.bz2 HTTP=404 |
| | v0.29.0 FIREFOX_128_0_ .tar.bz2 HTTP=404 |
| | v0.30.0 FIREFOX_130_0_1_ .tar.bz2 HTTP=404 |
| | v0.31.0 FIREFOX_134_0_1_ .tar.bz2 HTTP=404 |
| | v0.32.0 FIREFOX_148_0_ .tar.xz HTTP=200 |
| | v0.33.0 FIREFOX_149_0_ .tar.xz HTTP=200 |
| | v0.34.0 FIREFOX_150_0_2_ .tar.xz HTTP=200 |
| | v0.35.0 FIREFOX_152_0_ .tar.xz HTTP=200 |
| | </file> |
| | |
| | Four of 29 releases are installable; the boundary sits between v0.31.0 (Firefox 134, pinned 2025-01-19) and v0.32.0 (Firefox 148, 2026-03-03). Supporting checks: |
| | |
| | * **It is expiry, not a moved URL or a changed suffix.** The index //namespace// lookup for v0.29.0's revision returns ''ResourceNotFound'' — the indexed task is gone, not just one artefact — and retrying that release with ''.tar.xz'' instead of ''.tar.bz2'' also 404s. The same lookup for v0.35.0's revision returns a task with ''"expires": "2027-06-10T17:30:02.505Z"'', which is where "about a year" comes from. |
| | * **One transient, corrected.** v0.35.0 returned HTTP 503 on the recorded run and 200 on three immediate retries; the 503 was replaced by 200 in the output above. This is the only edit made to any script output on either page, and it is recorded here rather than applied silently. |
| | * **The branded build is not a substitute**, and the page says so: ''archive.mozilla.org'' still serves branded Firefox 90.0.2 (HTTP 200), but the extension needs an unbranded build, per ''docs/Platform-Architecture.md'' already quoted in §7. |
| | * **The container escape hatch was checked, not assumed.** Docker Hub tag lookups: ''0.15.0'', ''0.16.0'', ''0.17.0'', ''0.21.1'', ''0.23.0'', ''0.29.0'' and ''0.31.0'' all return 200; ''0.10.0'', ''0.12.0'', ''0.13.0'' and ''0.14.0'' return 404 — so the tags go back to 0.15.0, not to the start. ''0.17.0'' lists ''amd64'' only. That the image contains the binary is read from the v0.17.0 ''Dockerfile'', which runs ''./install.sh'' and then ''mv firefox-bin /opt/firefox-bin'' with ''ENV FIREFOX_BINARY''; **the image was not pulled or run** — this host is aarch64, and §8 explains why nothing amd64 was executed here. |
| | * **The upstream issue is still open** ([[https://github.com/openwpm/OpenWPM/issues/964|#964]], opened 2021-12-15, three comments, most recent 2025-11-26), and ''install-firefox.sh'' at ''b9dd4c3'' now prints "the Firefox version may be too old and no longer available on TaskCluster" and points at it. Neither the mirror repository the maintainer floated in 2021 nor any other archive of these builds was found. |
| | |
| | ==== 12.3 Judgement calls in this addendum ==== |
| | |
| | * **Placed as the first two pitfalls**, per Karel's instruction, ahead of the empty-database pitfall. Defensible independently: these two decide whether the crawler still exists in three years, where the others decide whether one crawl is sound. |
| | * **Cited ALSAcnc through the existing key** {[bouhoula2024_automated]} rather than adding a bibliography entry for the repository — no new keys. Note the pre-existing duplicate ''bouhoula2024automated'' (recorded in the archives provenance page); the underscore form used here is the one [[statistics:pvalue_corrections]] already uses. |
| | * **Did not claim "four upgrades"** although Karel reports four; only three are checkable in the public repository, and the page says "in its public history alone", which is true and weaker. |
| | * **Did not open an Open Question.** The mirror-the-binary problem has an answer today (archive the tarball, or use the container image), so it belongs in guidance rather than in the open list. Whether anyone should maintain a community mirror is a question for the project, not for this page. |
| | * **No new review pass was run** on this addendum. It is two bullets plus this section, every factual claim is a command whose output is recorded above, and no corpus figure was touched — but a reader should know it did not go through the four-reviewer layer the rest of the page did. |
| | |
| | ===== 13. Run log ===== |
| |
| * **2026-08-14.** Corpus queries, repository reading, install attempts, page and provenance page written, four review passes, published. Corpus at 5,859 extracted papers; OpenWPM at ''b9dd4c3''. | * **2026-08-14.** Corpus queries, repository reading, install attempts, page and provenance page written, four review passes, published. Corpus at 5,859 extracted papers; OpenWPM at ''b9dd4c3''. |
| * **Tooling changed during the run.** ''scripts/check_page_numbers.mjs'' gained explicit handling of semver triples: the generic number regex cuts ''0.35.0'' into ''0.35'', and the ''present()'' lookahead then refuses to match that fragment inside the report's own ''0.35.0'', so 24 of this page's 47 initially-unaccounted "figures" were fragments of version numbers the report does print in full. Versions are now checked as whole tokens against the report and then removed before the generic scan — stricter than an ALLOW entry, and it does not weaken the scan for anything that is not a triple. The change is shared with every other page's check, so it was made narrow deliberately. | * **Tooling changed during the run.** ''scripts/check_page_numbers.mjs'' gained explicit handling of semver triples: the generic number regex cuts ''0.35.0'' into ''0.35'', and the ''present()'' lookahead then refuses to match that fragment inside the report's own ''0.35.0'', so 24 of this page's 47 initially-unaccounted "figures" were fragments of version numbers the report does print in full. Versions are now checked as whole tokens against the report and then removed before the generic scan — stricter than an ALLOW entry, and it does not weaken the scan for anything that is not a triple. The change is shared with every other page's check, so it was made narrow deliberately. |
| * **Credential exposure, caught in review.** An early command in this run printed ''$GH_TOKEN'' to the terminal to test whether it was set, so a GitHub token is in this run's transcript. The token is not in this repository, not in any published page and not in any output file, and ''openwpm_platform_probe.sh'' reads it from the environment. It should be rotated. The lesson is the boring one: test for a variable with ''${VAR:+set}'', never by echoing it. | * **Credential exposure, caught in review.** An early command in this run printed ''$GH_TOKEN'' to the terminal to test whether it was set, so a GitHub token is in this run's transcript. The token is not in this repository, not in any published page and not in any output file, and ''openwpm_platform_probe.sh'' reads it from the environment. It should be rotated. The lesson is the boring one: test for a variable with ''${VAR:+set}'', never by echoing it. |
| | * **2026-08-14, later the same day.** Karel supplied two pitfalls from his own practice (fork-vs-library; the deleted Firefox builds). Verified as §12, added to the content page as its first two //Known pitfalls//, plus one line each in //Forks and extensions//, //What to report// and the version-table preamble. New script: ''openwpm_firefox_availability.sh''; ''check_page_numbers.mjs'' gained five ALLOW entries (HTTP 404, the 583-line patch layer, and issue/PR numbers 964, 743, 753) — all external figures with no place in the report. No corpus query re-run, no figure changed, no bibliography entry added. |
| * **What the next run should do first:** re-run ''openwpm_platform_probe.sh'' (the maintenance table is a claim about a date), then ''report_openwpm.mjs'' and both ''check_page_numbers.mjs'' invocations. If the corpus has grown, watch the spelling table in report §A for a fork under a new name. | * **What the next run should do first:** re-run ''openwpm_platform_probe.sh'' (the maintenance table is a claim about a date), then ''report_openwpm.mjs'' and both ''check_page_numbers.mjs'' invocations. If the corpus has grown, watch the spelling table in report §A for a fork under a new name. |
| |
