Working notes behind crawler — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Recorded. Two populations, and the page never mixes them without saying so.

used and produced both count as driving a crawl: a paper that built its own crawler crawled with it. compared and mentioned do not — that is related work, and it is why the specialised-crawler table has a separate “papers only mentioning” column.

cd /workspace/artifacts/wiki
node scripts/report_crawler.mjs                       # every figure on the page
node scripts/report_crawler.mjs --wiki                # the same, as DokuWiki tables
node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
node scripts/check_page_numbers.mjs \
  pages/programming_crawler.txt out/new/report_crawler.txt \
  '===== Use in Publications =====' '===== Recommendations ====='

check_page_numbers.mjs left four figures unaccounted on 2026-08-12. Three are permanent and legitimate and are named here so the next run does not re-investigate them: 100 (“not against 100%”), 19.1 (the crawled share of the corpus, from data/extract/OVERVIEW.md) and 106 (the quote-check total, from out/new/quote_check_crawler.txt, not from this report). A fourth, 53, is the old OpenWPM count quoted on the page as history. The guard also caught a real error: the page briefly carried 93 from an intermediate run of the quote check, corrected to 106 before saving. Run it whole-page too (§10).

Recorded. Old = 4,322-paper corpus (page as published 2026-08-06). New = 5,859-paper corpus.

Moved by more than a rounding step, and why it matters:

• Playwright, 10 → 34 papers (1.2% → 3.0%), and in the provisional 2025–2026 bucket it overtakes Puppeteer (9.6% against 8.1%). This is the one clear reordering the extended corpus produced. The old page's flat statement “Playwright's first appearance in this corpus is 2022” stays true; the sentence after it was rewritten.
• Tracker Radar Collector 8 → 21 and Foxhound 2 → 8 in the all-papers mention table. The old page's closing sentence for that table already warned that “the 2024 boundary cuts them off mid-growth”. It did, and by a factor of two to four. The sentence was rewritten to say so rather than to warn about it.
• “Names a library, not a browser” 23 → 49, growing about twice as fast as the corpus. The new strings are the reason: 2025–2026 papers write “a Playwright-controlled browser”, “Playwright-derived browser”, “Puppeteer/TRC desktop browser”.
• Bespoke, own name 98 → 184, then 181. Corpus growth plus the same effect the README already documents: including produced in the tool query surfaces one-off names. On 2026-08-29 the LLM-agent family (§5.3) claimed five of those strings, moving the row to 181 and the union to 318.
• Artifact share 23.6% → 21.6% for 2010–2013. A genuine change in the same window, from the newly retrieved IEEE S&P papers landing in the early buckets.

Not changed: the page's prose about how each library works, the sandbox comparison, the WebDriver BiDi section, the recommendations, and every citation.

Reconstructed (design), recorded (this run's residue). An ordered list of regexes, specific tools before the generic libraries they wrap, so puppeteer-extra-plugin-stealth lands in Anti-detection patches and not in Puppeteer. Unmatched strings are not discarded: they become the Bespoke crawler, given its own name row, because almost every one of them is one paper's own tool.

Fold accounting for the current corpus (printed by report_crawler.mjs, section “Fold accounting and the artifact trend”, added 2026-08-12 precisely because these three figures were on the page and not in the report):

1075 tool mentions across 501 distinct strings in the 1120 crawling papers.
Exact string "Selenium": 194 papers. Folded Selenium family: 242 papers
  (exact-string counting would undercount by 19.8%).
Bespoke: 147 generic + 181 own-name, 10 in both, union 318 of 1120 (28.4%).
Papers naming any of Puppeteer, Playwright, Chrome DevTools Protocol (direct),
  Scrapy, OpenWPM: 215.

The residue — 204 strings until 2026-08-29, 199 after the LLM-agent fold — is printed in full at the end of out/new/report_crawler.txt. It is too long to reproduce here and it is genuinely one-name-per-paper; the interesting part is what is new in it, below.

Recorded. On the 4,322-paper corpus the browser fold left 1 unmapped string (Ghostery). On the 5,859-paper corpus it left 15, and every new one was mappable:

After the change the residue is back to 1 distinct string, Ghostery — which is an extension, not a browser, and is deliberately left unfolded so it stays visible.

That the residue grew from 1 to 15 without anything breaking is the general lesson: a fold silently ages against a growing corpus, and only a printed residue shows it. Interestingly, all five Chinese-browser strings came from one paper, which is why “Other named browser” stayed at 5 papers while five strings left the residue.

Recorded 2026-08-12; resolved 2026-08-29. The 2025–2026 slice introduced a cluster of one-off names that are not bespoke crawlers in the old sense but LLM-agent browser automation: Browser-Use / browser-use / Browser Use (three spellings, three papers), AgentLab, BrowserGym. A sixth string, MCP Server Crawler, was listed with them on 2026-08-12 and that was wrong — see below.

The 2026-08-12 run deliberately did not fold it, on the grounds that “does the field now drive browsers with LLM agents” is a new claim needing its own population, its own dating and its own reading of the papers. That work was done on 2026-08-29. The outcome:

• A family LLM browser agents now exists in tool_fold.mjs, ordered before Playwright and Puppeteer, because Browser Use and Skyvern drive Playwright underneath and would otherwise be counted as Playwright.
• It claims 4 of the 1,120 crawling papers (0.4%), all of them 2026. That is the row now on crawler, and the five figures it moved are in §4 above.
• MCP Server Crawler was excluded, and the 2026-08-12 note that grouped it with the agents is corrected here. Reading IEEE-SP/2026 Parasites in the Toolchain shows it is an ordinary crawler over MCP directory listings and GitHub links — “we implemented the MCP Server Crawler to leverage these links as the entry point for data collection”. The name describes what it crawls, not what drives it. It stays in the printed residue. WebAgentDriver (NDSS/2026 ChameleoScan), the Appium XCUITest driver on jailbroken iOS, is excluded for the same kind of reason.
• The dating question was answered rather than deferred: 9 papers in 5,859 name an agent framework at all, 1 in 2025 and 8 in 2026, and six of the nine measure agents rather than crawl with them. It is not current practice and the new page says so.
• The page is llm_agents and its own provenance is llm_agents, which carries the full query log, the fold rule, the residue and the quote adjudication.

A second, smaller cluster in the same residue: consent-automation tools used as instruments — autoconsent, DuckDuckGo autoconsent, Super Agent-Automatic Cookie Consent, ConsentChk. Those belong to consent rather than here.

Recorded, 2026-08-12. scripts/quote_check.mjs is new; before it, this page's spot-check was done by hand and left no artefact, which is why the old figure (“66 quotes, 44 exact”) cannot be reproduced.

Re-run 2026-09-21 with the PDF fallback — see Quote-check refresh, 2026-09-21 at the foot of this page:

$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.

The old figure was 16 below threshold. 13 of the 16 are located in an independent pypdf rendering of the same paper.pdf and are therefore a defect in the stored text, not in the extraction; 3 are below threshold in both renderings. exact (58) and partial (32) did not move, and the below-threshold-in-both rate is 2.8% of 106.

All 16 of the then-below-threshold quotes had already been read by hand against paper.cols.txt with whitespace normalised, before the fallback existed. All sixteen are present in the paper. None was unsupported — a hand result the fallback independently confirms for 13 of them. Two representative failures, one from each bucket:

• PETS/2022/my-cookie-is-a-phoenix… — extraction has “All measurements are performed using the Open-WPM platform on the Firefox browser.”; the paper has “…using the Open-WPM platform [52] on the Firefox browser.” The extraction dropped the citation marker. This one is still below threshold in both renderings (50% of windows in the stored text), because a dropped marker is in the extraction's own string and no rendering of the PDF can repair it. It is one of the 3.
• CCS/2025/in-the-dom-we-trust… — extraction has “we created a Playwright-based crawler to collect snapshots of webpages”; the source reads “we created a Playwrightconstrains the valid child elements, and everything else is moved based [18] crawler to collect snap…”. The column repair spliced the adjacent column mid-word. This one is rescued: 33% of windows in the stored text, 67% in the PDF, which is what a column-repair failure looks like when the second rendering happens not to splice the same sentence.

Those two examples are the distinction the fallback draws. A quote the extraction itself mangled fails in every rendering; a quote the de-columner mangled usually does not. That is why 13 of 16 moved and 3 did not.

The lesson for anyone reusing evidence.quote: a failed exact match on .cols.txt is weak evidence of anything. Check the distinctive noun phrase, not the sentence.

Reconstructed. The content page's tool claims are sourced to primary material and were verified on 2026-08-06 by the original run. Not re-verified on 2026-08-12 — the refresh touched no external claim — so the “checked 2026-08-06” dates on the page are accurate and should be re-checked before anyone leans on them again.

• OpenWPM: repository README, docs/Configuration.md, scripts/install-firefox.sh, issues #557 and #503.
• PageGraph: pagegraph-crawl README and the brave-browser wiki's own limitations list.
• Tracker Radar Collector: repository README (concurrency cap, injected anti-detection script, autoconsent).
• PhantomJS: ariya/phantomjs#15344, the maintainer's own suspension notice.
• webXray: rejected the tool as installable. api.github.com/repos/timlib/webXray → 404, api.github.com/users/timlib/repos → empty, pypi.org/pypi/webxray/json → not found. The page cites the 2015 paper for the method and says plainly that the tool cannot be installed today. This is the clearest example on the site of an industry check changing what the page says.
• Foxhound: its README's “Cite us!” section, which asks for the EuroS&P paper — a venue this corpus does not cover, which is itself the reason the page warns that a tool's paper count here is a lower bound.

Rejected: SEO listicles of “best web scraping tools” were rejected wholesale by the original run; none is cited.

• Whether Playwright has actually overtaken Puppeteer. It has in the 2025–2026 bucket, and that bucket is provisional. Two complete venue-years would settle it; they do not exist yet.
• Whether the anti-detection cohort is 9 or 12 papers. The folded framework family (crawling population, used/produced) gives 9; the all-papers mention table gives 12 using and 2 more only mentioning. Both are on the page, in the two tables whose populations differ. They are not inconsistent, but a reader who reads only one table will get a different number, and there is no single right answer to publish.
• Why 40 crawling papers have no crawlConfig record. Same unresolved question as on crawling_location.
• The original run's spot-check. “66 quotes, 44 exact” cannot be reproduced; quote_check.mjs did not exist. Replaced rather than corrected.
• The sandbox measurements were not re-run. The Selenium/Puppeteer/Playwright/CDP comparison, the WebDriver BiDi responseCompleted finding and the “two network handlers cross-deliver” bug are from 2026-08-06 against selenium-webdriver 4.46. Library versions have moved since. Nothing in this refresh touched them, and nothing in this refresh verified them.

Recorded. The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a systematic defect, not a scatter of typos, and it is worth stating because it will recur on the next refresh:

• check_page_numbers.mjs was run with a heading window — normally Use in Publications to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block was outside the window and stayed at its 4,322-corpus value. Across the six pages 29 such figures survived the first pass.
• The matcher was substring, not word-boundary, so report.includes('59') was satisfied by 11.59 bits. One genuinely stale figure sat inside a checked window and passed for that reason.

Both are fixed in scripts/check_page_numbers.mjs: matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, --code opts into scanning <file> blocks, and omitting the heading markers checks the whole page. Run it windowed and whole-page. The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. Fixed on this page's content page as a result: 11.6% → 12.0% (twice), Selenium 21.8% → 21.6%, Puppeteer 6.3% → 6.8%, direct CDP 35 papers (4.1%) → 42 (3.8%), headless 12.9% → 12.5%, and unnamed bespoke crawlers 34 → 75. Two prose corrections the numbers forced:

• The Playwright section still said “still rare in the literature — 10 papers”, two screens from a corpus table saying 34 and a sentence saying Playwright overtakes Puppeteer. Rewritten.
• “OpenWPM users are more than twice as good as average … Playwright's cohort is next best” ranked the two backwards against the table directly above it (Playwright 26.5%, OpenWPM 25.9%). On the old corpus Playwright led 50.0% to 25.5% on a base of ten papers; with 34 they are effectively tied, and the sentence now says so.
• The framework table had drifted out of descending order after the refresh moved three rows. Restored.

106 is a permanent legitimate unaccounted figure on this page: it is the quote-check total, from out/new/quote_check_crawler.txt, not from report_crawler.mjs. Recorded here so the next run does not re-investigate it. §3 above previously said “four … all deliberate” while listing one figure (93) that was a real error — corrected.

Recorded as the run went, tidied at the end. Work item llm-agent-crawlers asked three things of this page: whether the 2025–2026 agent strings should become a family in tool_fold.mjs, whether crawler should carry it, and what it does to the Being Detected section. The first two were answered on 2026-08-29 (§5.3 above, and §4.3 of llm_agents). This run verified them instead of taking that note at face value, and closed the third.

node scripts/report_crawler.mjs re-run on 2026-09-03 against data/extract/run1 (5,859 papers; population crawled = 1,120). Exit 0. Every figure in the content page's Use in Publications section matches the fresh output, including the rows the 2026-08-29 fold moved:

Grepping the fresh residue listing for every agent-shaped string (browser.?use, agentlab, browsergym, skyvern, stagehand, crawl4ai, firecrawl, computer.use, operator, mcp, webagent) returns exactly one line: MCP Server Crawler. That is the deliberate exclusion recorded in §5.3, so the fold is behaving as documented and nothing new has arrived in the residue since 2026-08-29.

The full output is now written to scripts/report_crawler-output.txt in the wiki workdir, next to the script. §5.1 above still points at out/new/report_crawler.txt, a path in a working directory; that reference was not edited, so both now exist and the committed one is the readable one.

The content page gained an Agent-Driven Crawling section on 2026-08-29, but Being Detected still opened with “Two points from there bear on the choice made on this page” and both points were about scripted automation. Neither neighbour is at fault — crawler_detection says in three places that an agent is a different surface (its intro, the close of What a Blocked Crawl Does to Your Number, and Related Pages), and llm_agents has a full section on it — but a reader who takes the agent option on this page was given no signal on the way past. Added one bullet and changed “Two points from there” to “Three points”. Revisions 1788424922 → 1788468979 → (after review) 1788469575.

The first version of that bullet was wrong in four ways and was rewritten, see §12.5. It is worth recording what it contained, because the rewrite deleted a citation:

jeong2026_network is therefore no longer cited on this content page; it remains cited on llm_agents, where the claim is correctly hedged. No bibliography entry was added, changed or removed at any point in this run.

Rendered checks after each save, with purge=true: after the first save, references 14 → 15 with Jeong present and citekey spans 36 → 38 (the plugin emits two per marker); after the rewrite, back to 14 with Jeong absent. Elements matching <h[123] … id= stayed at 25, <table at 12 and <pre at 5 across all three revisions, and wikilink2 red links stayed at 0. (Those are counts of that regex, not of every heading in the DOM — a reviewer counting all h1/h2/h3 gets 29. The point of the count is that it did not change, not its absolute value.) The anchor #being_detected_and_the_ethics_of_not_being was taken from the child page's rendered heading id, not guessed, and the saved page's href matches it exactly.

Found on 2026-09-03, after the edit above, by running the number guard whole-page. The save summary for revision 1787985453 says the fold moved 75 → 74 unnamed bespoke crawlers, and it did — in the WRAP important box under the framework table. The same figure in the Recommendations section was not moved and shipped as 75 for five days across three subsequent saves. Fixed in revision 1788469206.

This is the exact failure §10 above describes, one refresh later, and it had a specific disguise: the whole-page guard does list 75 as unaccounted, but the context it prints is the first occurrence of the string, which on this page is the sandbox sentence “a silent 75% data loss”. Anyone reading the guard output and recognising that sentence as a legitimate non-corpus figure clears the line and never reaches the second occurrence. The same trap is still there today: after the fix, 75 is still reported, still with the sandbox context, and now legitimately.

Command:

node scripts/check_page_numbers.mjs \
  pages_crawler_live.txt scripts/report_crawler-output.txt

with no heading markers. The whole-page run also reports 23 other unaccounted figures and 3 version identifiers. All were read; none is a corpus figure. They are the sandbox measurements (779 / 877 / 2038 / 1329 / 1838 ms, HTTP 200 and 204, selenium-webdriver 4.46, Node 22.23, Firefox 153, Chromium 151.0.7922.34), figures quoted from other papers (PanoptiChrome 50% and 36.7× against Foxhound's 95% and 1.4×), version constraints (Brave ≥ 1.46, Chrome 116, PolyForm 1.0.0, TRC maxLoadTimeMs || 60000, OpenWPM issue #557) and prose numbers (“Chromium 121 driven by Playwright 1.41”, “nearly twice”, “100%”). The windowed run leaves four, all already recorded as permanent in §3 and §10: 100, 106, 19.1 and 53.

• Did not broaden or re-open llm_agents. The work item says the page belongs to the sibling item and was explicitly out of scope for this one. It is published and reviewed; re-editing it here would have put an unreviewed change on a reviewed page.
• Did not touch crawler_detection. Its own open work item covers the draft-meunier-web-bot-auth-architecture state correction (§8.3 of the child page's provenance). A factual correction to a page this run did not otherwise touch should go through that page's own review, which is the same call the 2026-08-29 run made.
• Did not re-run the sandbox measurements or the quote check. §8's open questions are unchanged, and the sandbox figures are still from 2026-08-06 against selenium-webdriver 4.46.
• check_wrap.mjs false-positives on this page, and did so before this edit. It flags the <file> inside inline code in §10 as a tag that “WILL fire”. It does not: the rendered page escapes it to <file> and §10 renders in full. Left alone rather than nowiki-escaped, because the fix would be an untested change to a line this run has no other reason to touch — but the warning is expected, so the next run does not chase it. ²⁾

Two reviewers, both told explicitly that the brief they were given might not be exhaustive, both handed the page text, the report script and its output, and these notes. A one-bullet addition would not normally get a review layer; it got one because the bullet asserts a framing rather than a number, and that is the kind of claim this wiki gets wrong quietly. The generic pass earned its slot several times over.

Neither reviewer was re-run after the fixes. That is a deliberate call on a change this size and it is the weakest point of this section: the rewritten bullet has been read by nobody but its author.

← back to the content page · corpus-level provenance

Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with check_wrap.mjs and check_typography.mjs. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.

The paired content page's obsolete live-page caveat was removed after the fresh DOM confirmed that the root Artifacts link is live. The post-save DOM was re-checked for red links. No figures or citations changed.

What was wrong. The crawler configuration table and family comparison called the 349/1,120 = 31.2% raw field population States consent action, which could be read as 349 papers making that claim. The schema statistic itself is correct.

Command and real output. node scripts/report_crawler.mjs → Consent action field populated (schema; not an audited paper claim) | 349 | 31.2%; the family table's corresponding column is Consent field populated (schema). node scripts/consent_action_noninteraction_audit.mjs → 55/1,120 = 4.9% audited paper claims and 279/313 = 89.1% unsupported no-interaction labels.

What changed. The page and report output now distinguish field population from paper-claim support; the family percentages remain schema statistics and are labelled accordingly.

Review. One model: 'sonnet' figures-vs-script pass re-ran report_crawler.mjs and checked the edited row, the adjacent statefulness/interaction-depth rows, and the family table. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed.

Rejected finding. The 349 schema statistic was retained rather than rewritten as 55, because this page's table measures extraction-field completeness. The audited claim result remains on consent.

The 2026-09-04 cols-vs-PDF audit on corpus showed that 73.1% of evidence quotes that cannot be located in paper.cols.txt are present in an independent pypdf rendering of the same paper.pdf. scripts/quote_check.mjs was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, RESCUED. §6's figure predates the patch. Re-run, unedited first line:

$ node scripts/quote_check.mjs --tools 'openwpm|playwright' --show 400
106 quotes checked: 58 exact, 32 partial (>=60% of 5-word windows), 13 rescued from the PDF, 3 below threshold in both renderings, 0 with no full text on disk.

This page is the one where the refresh changes nothing about the conclusion. All 16 had already been read by hand and all 16 were present, so the page never claimed a quote failure. What moves is the figure other pages compare themselves against: ip_classification quoted “the --tools checks produce 15%” as its baseline, and that baseline is now 2.8%. That row was corrected on the same day.

Scope of this edit. §6 and the content page's spot-check bullet. report_crawler.mjs and tool_fold.mjs were not re-run in this pass; no fold, framework count, trend or citation was touched, and every other figure on crawler and this page stands as published.

The command is the code block above — it contains a regex alternation, and a DokuWiki table cell cannot hold a |.

What was wrong. The Which specialised crawlers actually get used table read SAP Project Foxhound | 8 | 1, and the comparison table above it read 8. The deeper per-paper audit on foxhound (scripts/report_foxhound.mjs + scripts/fh_fold.mjs, 2026-08-17) finds 9 papers that actually ran the browser, and its own footnote said so and called these two tables queued for an errata edit.

Why the two disagreed — and it is not a denominator. This table's two columns come from tools[].usedOrMentioned: used or produced go left, everything else goes right. Khodayari et al.'s NDSS 2025 open-redirection paper is extracted as compared — “We considered the following state-of-the-art detectors as baselines: dynamic taint-tracking [24, 36, 45] using Foxhound [44, 45]” — and its comparison ran over the paper's 42,288-webpage subset of 50 randomly chosen applications³⁾. Running a tool as a baseline is running it, so compared put a user of the browser in the cites only column.

The fix is in the script, not on the page. report_crawler.mjs §6 now carries a ROLE_AUDITED map, currently one entry, SAP Project Foxhound → the ROLE map exported by fh_fold.mjs. For a row in that map the enum verdict is replaced by the published hand verdict (used / extended / baseline count as ran it; citation does not), and the script throws if any paper the row matches has no verdict in the audit, so the audit and the extraction cannot drift apart silently. Rows with no audit are untouched: this is deliberately not a general rule that compared means “ran it”, because deciding that needs the sentence.

$ node scripts/report_crawler.mjs > scripts/report_crawler-output.txt
$ diff <old> <new>
130c130
< SAP Project Foxhound                         8             1                       2024–2026
---
> SAP Project Foxhound                         9             0                       2024–2026

That is the whole diff of the report. No other row, fold, residue, denominator or year range moved, and the folded-framework table above is untouched — this paper's Foxhound tuple carries category: “program-analysis”, not a crawler-framework category, so it was never in that population.

The 2 in that sentence is the pre-extension corpus figure and is not re-derived here; the paper this errata adds is NDSS 2025 and was not in that corpus, so the comparison is still like-for-like.

A finding rejected. The cites only column now reads 0, while foxhound reports 2 citation-only papers. That is not a new inconsistency and was not “fixed”: those two papers have no tools[] tuple naming Foxhound at all, so this table — which is built from tools[] — cannot see them, and inventing a 2 here would mean this row alone was full-text-swept while every other row was not. The child page's reconciling footnote now says exactly this, and was edited in the same run.