Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: programming:crawler ======

Working notes behind [[programming:crawler]] — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on [[literature:corpus]] and are not restated here.

<WRAP important>
**Partly reconstructed.** The content page was written on 2026-08-06, before this site had a provenance convention. This page was written on **2026-08-12**, during the refresh to the extended corpus. Sections marked //recorded// were produced by the 2026-08-12 run. Sections marked //reconstructed// were rebuilt afterwards from the report script, its output and the page text. The browser-automation //sandbox// measurements the content page reports (Selenium/Puppeteer/Playwright/CDP against a local fixture) were made by the original run and are **not** re-derivable from the corpus; they are reconstructed here from the page and the scripts in ''sandbox/'', not re-measured.
</WRAP>

===== 1. What this page is backing =====

^ Item ^ Value ^
| Content page | [[programming:crawler]] |
| Report script | ''scripts/report_crawler.mjs'' |
| Folds it depends on | ''scripts/tool_fold.mjs'' (framework families, browser families) |
| Quote verification | ''scripts/quote_check.mjs'' (new on 2026-08-12) |
| Runnable comparison | ''sandbox/fixture.mjs'', ''sandbox/run_{playwright,puppeteer,cdp,selenium}.mjs'', ''sandbox/compare.sh'' |
| Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
| Refreshed | 2026-08-12 |

===== 2. Populations and denominators =====

//Recorded.// Two populations, and the page never mixes them without saying so.

^ Tag ^ Definition ^ N ^
| ''crawled'' | ''crawlConfig !== null'' OR ''studyTypes'' contains ''automated-web-crawl'' | 1,120 |
| ''all'' | every extracted paper — used **only** for the specialised-crawler mention table, where a tool can be used outside a formal crawl configuration | 5,859 |
| ''crawled'' ∧ names a framework | ≥1 ''tools[]'' tuple with ''category ∈ {crawler-framework, browser-automation}'' and ''usedOrMentioned ∈ {used, produced}'' | 723 |
| ''crawled'' ∧ names a browser | ≥1 non-sentinel ''crawlConfig.browsers'' entry | 529 |
| ''crawled'' ∧ has a crawlConfig record at all | ''crawlConfig !== null'' | 1,080 |

''used'' and ''produced'' both count as driving a crawl: a paper that built its own crawler crawled with it. ''compared'' and ''mentioned'' do not — that is related work, and it is why the specialised-crawler table has a separate "papers only mentioning" column.

===== 3. Running it =====

<code bash>
cd /workspace/artifacts/wiki
node scripts/report_crawler.mjs                       # every figure on the page
node scripts/report_crawler.mjs --wiki                # the same, as DokuWiki tables
node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
node scripts/check_page_numbers.mjs \
  pages/programming_crawler.txt out/new/report_crawler.txt \
  '===== Use in Publications =====' '===== Recommendations ====='
</code>

''check_page_numbers.mjs'' left four figures unaccounted on 2026-08-12. **Three are permanent and legitimate** and are named here so the next run does not re-investigate them: ''100'' ("not against 100%"), ''19.1'' (the crawled share of the corpus, from ''data/extract/OVERVIEW.md'') and ''106'' (the quote-check total, from ''out/new/quote_check_crawler.txt'', not from this report). A fourth, ''53'', is the **old** OpenWPM count quoted on the page as history. The guard also caught a real error: the page briefly carried ''93'' from an intermediate run of the quote check, corrected to 106 before saving. Run it whole-page too (§10).

===== 4. What the refresh changed =====

//Recorded.// Old = 4,322-paper corpus (page as published 2026-08-06). New = 5,859-paper corpus.

^ Figure ^ Old ^ New ^
| ''crawled'' | 859 | 1,120 |
| Names a framework | 544 (63.3%) | 723 (64.6%) |
| Names a browser | 395 (46.0%) | 529 (47.2%) |
| Names neither | 248 (28.9%) | 311 (27.8%) |
| Selenium | 187 (21.8%) | 242 (21.6%) |
| Bespoke, own name | 98 (11.4%) | 184 (16.4%) |
| Bespoke, generic | 130 (15.1%) | 147 (13.1%) |
| Bespoke, union | 221 (25.7%) | 321 (28.7%) |
| …of which named nothing else | 35 | 75 |
| Puppeteer | 54 (6.3%) | 76 (6.8%) |
| OpenWPM | 51 (5.9%) | 58 (5.2%) |
| Playwright | 10 (1.2%) | 34 (3.0%) |
| Direct CDP | 35 (4.1%) | 42 (3.8%) |
| Tracker Radar Collector (folded family) | 5 | 10 |
| Consent-interaction crawlers | 4 | 10 |
| Anti-detection patches | 5 | 9 |
| Chrome / Chromium (of browser-naming) | 225 (57.0%) | 304 (57.5%) |
| Names a library, not a browser | 23 (5.8%) | 49 (9.3%) |
| Safari / WebKit | 10 (2.5%) | 16 (3.0%) |
| Any tool states its version | 63 of 544 (11.6%) | 87 of 723 (12.0%) |
| Distinct tool strings / mentions | 378 / 789 | 501 / 1,075 |
| Unmapped residue | 104 strings / 106 mentions | 204 / 210 |
| Exact string "Selenium" vs folded | 147 vs 187 | 194 vs 242 |
| Artifact share, 2010–2013 → 2022–2024 | 23.6% → 62.1% | 21.6% → 61.2% |
| OpenWPM mentions (all papers) | 53 | 60 |
| Tracker Radar Collector (all papers) | 8 | 21 |
| SAP Project Foxhound (all papers) | 2 | 8 |
| Brave PageGraph (all papers) | 4 | 8 |
| webXray (all papers) | 6 | 7 |
| VisibleV8 (all papers) | 8 | 9 |
| PanoptiChrome (all papers) | 1 | 2 |

**Moved by more than a rounding step, and why it matters:**

  * **Playwright, 10 → 34 papers (1.2% → 3.0%), and in the provisional 2025–2026 bucket it overtakes Puppeteer** (9.6% against 8.1%). This is the one clear reordering the extended corpus produced. The old page's flat statement "Playwright's first appearance in this corpus is 2022" stays true; the sentence after it was rewritten.
  * **Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table.** The old page's closing sentence for that table already warned that "the 2024 boundary cuts them off mid-growth". It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it.
  * **"Names a library, not a browser" 23 → 49**, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write "a Playwright-controlled browser", "Playwright-derived browser", "Puppeteer/TRC desktop browser".
  * **Bespoke, own name 98 → 184.** Corpus growth plus the same effect the README already documents: including ''produced'' in the tool query surfaces one-off names.
  * **Artifact share 23.6% → 21.6% for 2010–2013.** A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets.

**Not changed:** the page's prose about how each library works, the sandbox comparison, the WebDriver BiDi section, the recommendations, and every citation.

===== 5. Folds =====

==== 5.1 Framework families — scripts/tool_fold.mjs ====

//Reconstructed (design), recorded (this run's residue).// An ordered list of regexes, specific tools before the generic libraries they wrap, so ''puppeteer-extra-plugin-stealth'' lands in //Anti-detection patches// and not in //Puppeteer//. Unmatched strings are **not** discarded: they become the //Bespoke crawler, given its own name// row, because almost every one of them is one paper's own tool.

Fold accounting for the current corpus (printed by ''report_crawler.mjs'', section "Fold accounting and the artifact trend", added 2026-08-12 precisely because these three figures were on the page and not in the report):

<code>
1075 tool mentions across 501 distinct strings in the 1120 crawling papers.
Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers
  (exact-string counting would undercount by 19.8%).
Bespoke: 147 generic + 184 own-name, 10 in both, union 321 of 1120 (28.7%).
Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct),
  Scrapy, OpenWPM: 215.
</code>

The 204-string residue is printed in full at the end of ''out/new/report_crawler.txt''. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is **new** in it, below.

==== 5.2 Browser families — the fold that had gone stale ====

//Recorded.// On the 4,322-paper corpus the browser fold left **1** unmapped string (''Ghostery''). On the 5,859-paper corpus it left **15**, and every new one was mappable:

^ Residue string ^ Folded to ^ Rule added ^
| ''Playwright-controlled browser'', ''Playwright browser'', ''Playwright-derived browser'' | Names a library, not a browser | ''^playwright'' (was ''^playwright$'') |
| ''Puppeteer/TRC desktop browser'', ''Puppeteer/TRC emulated mobile browser'' | Names a library, not a browser | ''^puppeteer'' (was ''^puppeteer$'') |
| ''basic HTTP fetcher'' | Names a library, not a browser | ''^basic http fetcher$'' |
| ''Foxhound'' | Firefox | ''^foxhound'' — Foxhound is a Firefox fork |
| ''UC'', ''360'', ''QQ browser'', ''Xunlei'', ''Instabridge'' | Other named browser | five anchored alternatives |
| ''default Android browser'' | Mobile device or WebView | ''android browser'' |
| ''local browser'' | Unnamed or custom browser | ''local '' added to the adjective list |

After the change the residue is back to **1 distinct string, ''Ghostery''** — which is an extension, not a browser, and is deliberately left unfolded so it stays visible.

That the residue grew from 1 to 15 without anything breaking is the general lesson: **a fold silently ages against a growing corpus, and only a printed residue shows it.** Interestingly, all five Chinese-browser strings came from one paper, which is why "Other named browser" stayed at 5 papers while five strings left the residue.

==== 5.3 What is new in the framework residue, and what was deliberately not done with it ====

//Recorded.// The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but **LLM-agent browser automation**: ''Browser-Use'' / ''browser-use'' (two spellings, two papers), ''AgentLab'', ''BrowserGym'', ''MCP Server Crawler''. A sixth paper, ''USENIX/2026/the-state-of-passkeys…'', drives Playwright //through// Browser Use.

**This was deliberately not folded into a family and not written up on the page.** The refresh brief for this run is a mechanical re-derivation, and "does the field now drive browsers with LLM agents" is a new claim needing its own population, its own dating and its own reading of the papers — five or six papers in the thinnest, provisional years of the corpus is not enough to call it current practice. It is logged as a separate work item instead. Anyone picking it up should start from this residue list and from ''programming:crawler''’s //Being Detected// section, since an agent-driven browser is a different detection surface.

A second, smaller cluster in the same residue: consent-automation tools used as instruments — ''autoconsent'', ''DuckDuckGo autoconsent'', ''Super Agent-Automatic Cookie Consent'', ''ConsentChk''. Those belong to [[privacy:consent]] rather than here.

===== 6. Quotes checked =====

//Recorded, 2026-08-12.// ''scripts/quote_check.mjs'' is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure ("66 quotes, 44 exact") cannot be reproduced.

<code>
$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 20
106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows),
16 below threshold, 0 with no full text on disk.
</code>

All **16** below-threshold quotes were then read by hand against ''paper.cols.txt'' with whitespace normalised. **All sixteen are present in the paper. None was unsupported.** Two representative failures:

  * ''PETS/2022/my-cookie-is-a-phoenix…'' — extraction has "All measurements are performed using the Open-WPM platform on the Firefox browser."; the paper has "…using the Open-WPM platform **[52]** on the Firefox browser." The extraction dropped the citation marker.
  * ''CCS/2025/in-the-dom-we-trust…'' — extraction has "we created a Playwright-based crawler to collect snapshots of webpages"; the source reads "we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler to collect snap…". The column repair spliced the adjacent column mid-word.

The lesson for anyone reusing ''evidence.quote'': a failed exact match on ''.cols.txt'' is weak evidence of anything. Check the distinctive noun phrase, not the sentence.

===== 7. External and industry sources =====

//Reconstructed.// The content page's tool claims are sourced to primary material and were verified on 2026-08-06 by the original run. Not re-verified on 2026-08-12 — the refresh touched no external claim — so the "checked 2026-08-06" dates on the page are accurate and should be re-checked before anyone leans on them again.

  * OpenWPM: repository README, ''docs/Configuration.md'', ''scripts/install-firefox.sh'', issues #557 and #503.
  * PageGraph: ''pagegraph-crawl'' README and the ''brave-browser'' wiki's own limitations list.
  * Tracker Radar Collector: repository README (concurrency cap, injected anti-detection script, autoconsent).
  * PhantomJS: ''ariya/phantomjs#15344'', the maintainer's own suspension notice.
  * webXray: **rejected the tool as installable.** ''api.github.com/repos/timlib/webXray'' → 404, ''api.github.com/users/timlib/repos'' → empty, ''pypi.org/pypi/webxray/json'' → not found. The page cites the 2015 paper for the //method// and says plainly that the tool cannot be installed today. This is the clearest example on the site of an industry check changing what the page says.
  * Foxhound: its README's "Cite us!" section, which asks for the EuroS&P paper — a venue this corpus does not cover, which is itself the reason the page warns that a tool's paper count here is a lower bound.

**Rejected:** SEO listicles of "best web scraping tools" were rejected wholesale by the original run; none is cited.

===== 8. What could not be established =====

  * **Whether Playwright has actually overtaken Puppeteer.** It has in the 2025–2026 bucket, and that bucket is provisional. Two complete venue-years would settle it; they do not exist yet.
  * **Whether the anti-detection cohort is 9 or 12 papers.** The folded framework family (crawling population, ''used''/''produced'') gives 9; the all-papers mention table gives 12 using and 2 more only mentioning. Both are on the page, in the two tables whose populations differ. They are not inconsistent, but a reader who reads only one table will get a different number, and there is no single right answer to publish.
  * **Why 40 crawling papers have no ''crawlConfig'' record.** Same unresolved question as on [[provenance:design:crawling_location]].
  * **The original run's spot-check.** "66 quotes, 44 exact" cannot be reproduced; ''quote_check.mjs'' did not exist. Replaced rather than corrected.
  * **The sandbox measurements were not re-run.** The Selenium/Puppeteer/Playwright/CDP comparison, the WebDriver BiDi ''responseCompleted'' finding and the "two network handlers cross-deliver" bug are from 2026-08-06 against ''selenium-webdriver'' 4.46. Library versions have moved since. Nothing in this refresh touched them, and nothing in this refresh verified them.

===== 10. Review pass, 2026-08-12 =====

//Recorded.// The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a **systematic defect, not a scatter of typos**, and it is worth stating because it will recur on the next refresh:

  * ''check_page_numbers.mjs'' was run with a **heading window** — normally ''Use in Publications'' to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's **intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block** was outside the window and stayed at its 4,322-corpus value. Across the six pages **29 such figures** survived the first pass.
  * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason.

Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''<file>'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean.
Fixed on this page's content page as a result: **11.6% → 12.0%** (twice), Selenium **21.8% → 21.6%**, Puppeteer **6.3% → 6.8%**, direct CDP **35 papers (4.1%) → 42 (3.8%)**, headless **12.9% → 12.5%**, and unnamed bespoke crawlers **34 → 75**. Two prose corrections the numbers forced:

  * The Playwright section still said "still rare in the literature — 10 papers", two screens from a corpus table saying 34 and a sentence saying Playwright overtakes Puppeteer. Rewritten.
  * "OpenWPM users are more than twice as good as average … Playwright's cohort is next best" **ranked the two backwards** against the table directly above it (Playwright 26.5%, OpenWPM 25.9%). On the old corpus Playwright led 50.0% to 25.5% on a base of ten papers; with 34 they are effectively tied, and the sentence now says so.
  * The framework table had drifted out of descending order after the refresh moved three rows. Restored.

**''106'' is a permanent legitimate unaccounted figure** on this page: it is the quote-check total, from ''out/new/quote_check_crawler.txt'', not from ''report_crawler.mjs''. Recorded here so the next run does not re-investigate it. §3 above previously said "four … all deliberate" while listing one figure (''93'') that was a real error — corrected.

===== 11. Run log =====

^ ^ ^
| Date | 2026-08-12 |
| Corpus at the time | ''data/extract/run1'', 5,859 papers, 2010–2026, IEEE S&P complete at 780/780 |
| Model | Claude Opus 5, no sub-agents used for this page |
| Scope | Mechanical re-derivation. Prose, structure and method selection were not revisited, with two exceptions, both forced by the numbers: the "Playwright overtakes Puppeteer" sentence, and the specialised-crawler table's closing paragraph, whose old text warned of an effect the new data measures. |
| Script changes | ''lib.mjs'' (''YEAR_BUCKETS'' gained a starred 2025–2026 bucket), ''tool_fold.mjs'' (browser families extended, §5.2), ''report_crawler.mjs'' (new "Fold accounting and the artifact trend" section printing figures the page carried but the report did not), ''quote_check.mjs'' (new) |
| Caveats deleted | "IEEE S&P is only 43% retrieved (paywall)" — 780 of 780 selected papers are now retrieved. "2010–2024". |
| Claim weakened on purpose | The old page said the bespoke rows together were "more than used every off-the-shelf library on this page combined". That was not true of the old numbers either (187+54+35+10 = 286 > 221) and is not true of the new ones. Replaced with the two comparisons the report actually prints: more than Selenium (242), and more than the 215 papers naming any of Puppeteer, Playwright, direct CDP, Scrapy or OpenWPM. |
| Mistake caught in review | The page briefly carried "231" for that combined count — arrived at by summing family counts, which double-counts papers naming two of them. The real union is 215. ''check_page_numbers.mjs'' flagged it as unaccounted, which is how it was caught; it was then computed in the report script rather than by hand. |
| Deferred | LLM-agent browser automation in the 2025–2026 residue (§5.3), logged as a separate work item. |
| Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 7 stale figures plus a mis-ordered table and a backwards ranking sentence on this page. All fixes were applied and re-saved the same day. |

[[programming:crawler|← back to the content page]] · [[literature:corpus|corpus-level provenance]]