| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler [2026/09/03 08:42] – Fix two Tracker Radar claims found while reviewing the TRC subpage: footnote the 21-paper row with its 10 crawler / 11 dataset split (the tool_fold family is named after the crawler but matches the dataset), and replace the stale README-derived 'caps conc karel.kubicek.claude | programming:crawler [2026/09/17 10:45] (current) – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude |
|---|
| ==== Plain CDP ==== | ==== Plain CDP ==== |
| |
| You can skip the libraries entirely: launch Chromium with ''--remote-debugging-port'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures. | You can skip the libraries entirely: launch Chromium with ''%%--remote-debugging-port%%'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures. |
| |
| Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies. | Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies. |
| |
| <WRAP info> | <WRAP info> |
| **Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: --database is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives. | **Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: %%--database%% is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives. |
| </WRAP> | </WRAP> |
| |
| * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling: ''pagegraph-query'' (Python) under ''brave-experiments'', ''pagegraph-rust'' under ''brave''.)) | * **PageGraph does not track your automation.** Its own documentation warns that "PageGraph currently does not track puppeteer / automation scripts, and so modifying or interacting with the document through devtools/puppeteer while recording a PageGraph file will likely fail".((''pagegraph-crawl'' README, [[https://github.com/brave/pagegraph-crawl|github.com/brave/pagegraph-crawl]], checked 2026-08-06.)) It is a recorder of what the page did, not a driver of interaction. Its own limitations list is unusually honest and worth reading before you rely on it: no WebSocket tracking, no worker tracking, request headers not recorded (responses only), no ''style='' mutation tracking.((PageGraph wiki, [[https://github.com/brave/brave-browser/wiki/PageGraph|"Would Be Nice / Someday / Known Limitations"]], checked 2026-08-06. Query tooling: ''pagegraph-query'' (Python) under ''brave-experiments'', ''pagegraph-rust'' under ''brave''.)) |
| * **Tracker Radar Collector's README is stale, and one of its stale numbers was on this page.** It still claims "a hard limit of 38" concurrent crawlers; ''38'' appears nowhere in the code and the cap was removed in commit ''963f9ec'' (2025-04-16). The default is ''Math.floor(cores * 0.8)'', then capped at the number of URLs. The README's ''maxLoadTimeMs'' default of 30 s is stale the same way — the code fallback is 60 s. TRC does inject an anti-bot script into every frame by default (''-a'' disables it) and can be pointed at a Selenium Hub or a specific Chromium version.((Checked against a clone at commit ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'' on 2026-09-03: ''crawlerConductor.js'' concurrency, ''crawler.js'' ''maxLoadTimeMs || 60000'', and ''git log -S38 -- crawlerConductor.js''. [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]].)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset — and see [[Programming:Crawler:Tracker Radar Collector]] for the rest of the README-versus-code divergences. | * **Tracker Radar Collector's README is stale, and one of its stale numbers was on this page.** It still claims "a hard limit of 38" concurrent crawlers; ''38'' appears nowhere in the code and the cap was removed in commit ''963f9ec'' (2025-04-16). The default is ''Math.floor(cores * 0.8)'', then capped at the number of URLs. The README's ''maxLoadTimeMs'' default of 30 s is stale the same way — the code fallback is 60 s. TRC does inject an anti-bot script into every frame by default (''-a'' disables it) and can be pointed at a Selenium Hub or a specific Chromium version.((Checked against a clone at commit ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'' on 2026-09-03: ''crawlerConductor.js'' concurrency, ''crawler.js'' ''maxLoadTimeMs || 60000'', and ''git log -S38 -- crawlerConductor.js''. [[https://github.com/duckduckgo/tracker-radar-collector|github.com/duckduckgo/tracker-radar-collector]].)) It has no canonical academic paper; cite the repository and the Tracker Radar dataset — and see [[Programming:Crawler:Tracker Radar Collector]] for the rest of the README-versus-code divergences. |
| * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most complete surviving copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS; its licence is PolyForm Strict 1.0.0, which permits noncommercial use but **forbids redistribution**, so there is no lawful route to the code now that upstream is gone. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for what survives of it: the ownership database, and how it compares to Tracker Radar and Disconnect today. | * **webXray cannot be installed, and the reason is now known.** The repository its own documentation points at, ''github.com/timlib/webXray'', **no longer exists**, the ''timlib'' account has no public repositories, and there is no ''webxray'' package on PyPI. webXray is now a **commercial** product, ''webxray.ai'', run by webXray LLC with Libert as founder and CEO; ''webxray.org'' has been reduced to a placeholder.((Checked 2026-08-17: ''api.github.com/repos/timlib/webXray'' → HTTP 404, ''users/timlib'' → HTTP 200 with ''public_repos: 0'' and ''company: webXray.ai''; ''pypi.org/pypi/webxray/json'' → 404; ''https://timlibert.me/'' states "(Dr.) Timothy Libert is founder and CEO of webXray LLC".)) The most widely mirrored copy is ''thezedwards/webXray'', last pushed **2021-03-04** — webXray 3.x, which drives **consumer Chrome over the DevTools protocol**, not PhantomJS. It is **not** the copy to take, and its licence is the reason to be careful rather than the reason to give up: that snapshot is PolyForm Strict 1.0.0, which permits noncommercial use but forbids redistribution — but Libert relicensed the project **back to GPLv3 in 2021 and then to MIT in 2023**, and that history survives in ''peterjoles/webXray'', 36 commits ahead. There **is** a lawful route to the code; it is just not the copy most mirrors carry. Treat {[libert2015_invisible]} and {[libert2018_automated]} as citations for the //method// — third-party request measurement with company attribution — and see [[Programming:Crawler:webXray]] for the tool's remains. The part that outlived it, the **domain-ownership database**, and how it compares to the two live alternatives today, moved on 2026-09-11 to [[Design:Ownership resolution]]: that is the page to read if your question is "whose domain is this?" rather than "which crawler should I run?". |
| * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it. | * **Taint tracking is a different question.** Foxhound and PanoptiChrome answer "did this value reach that sink?", not "what did the page load". If your question is about tracking prevalence, they are the wrong instrument and cost you a browser build; if it is about how data escapes — client-side XSS, DOM-based leaks, fingerprinting inputs — nothing else answers it. **Between the two, start with Foxhound**: the one head-to-head evaluation measured PanoptiChrome at 50% compatibility and 36.7× overhead against Foxhound's 95% and 1.4×, and no paper in this corpus has used PanoptiChrome as an instrument {[calzavara2025_dynamic]} — see [[Programming:Crawler:PanoptiChrome]] for when it is nevertheless the only option. Foxhound tells you what to cite: its README's "Cite us!" section asks for the EuroS&P paper in which the browser is described, Klein et al. {[klein2022_handsanitizers]}, and its wiki separately lists the papers that have used it. |
| |
| ===== Agent-Driven Crawling ===== | ===== Agent-Driven Crawling ===== |
| |
| Since 2025 there is a fifth option that does not fit the two-layer model above: a | Since 2026 there is an option that does not fit the two-layer model above: a |
| **crawler whose next action is chosen by a model** rather than scripted. The open-source | **crawler whose next action is chosen by a model** rather than scripted. The open-source |
| ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or | ones — Browser Use, BrowserGym/AgentLab, Skyvern — still drive Chromium through CDP or |
| gives you away with, why cloaking is the shape that ends up in your abstract, how to | gives you away with, why cloaking is the shape that ends up in your abstract, how to |
| measure the obstruction instead of fighting it, and what the corpus says about how rarely | measure the obstruction instead of fighting it, and what the corpus says about how rarely |
| anyone reports it. Two points from there bear on the choice made //on this page//: | anyone reports it. Three points bear on the choice made //on this page//: |
| |
| * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild. | * **The framework is detectable, not just automation in general.** Vastel et al. {[vastel2018_scanner]} show fingerprint inconsistencies distinguish instrumented browsers from ordinary ones, and Krumnow et al. {[krumnow2022_gullible]} show OpenWPM specifically has detectors deployed against it in the wild. |
| * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference. | * **The anti-detection patches in the tables below are an arms race you lose quietly**, and most of those packages are now unmaintained or have moved to a successor under a different name; [[Programming:Crawler Detection]] dates each of them. Using them is also an ethics decision — see [[Practices:Ethics]] — because you are overriding a site operator's expressed access preference. |
| | * **If you take the agent option above, you are on a different detection surface, not just a more visible one.** Bot management now scores an //agent acting on a user's behalf// separately from both a human and a crawler collecting training data; [[Programming:Crawler Detection]] names the vendor categories and dates their defaults, which were still moving through 2026. The rules you are judged by are those, and they are not the two above: [[Programming:Crawler:LLM Agents#being_detected_and_the_ethics_of_not_being|the child page's detection section]] has them, including three 2026 papers that took three different positions on whether to evade at all. |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
| | Headless or headful | 140 | 12.5% | | | Headless or headful | 140 | 12.5% | |
| | Stateful or stateless | 219 | 19.6% | | | Stateful or stateless | 219 | 19.6% | |
| | Consent action | 349 | 31.2% | | | Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% | |
| | Interaction depth | 841 | 75.1% | | | Interaction depth | 841 | 75.1% | |
| | Authentication | 779 | 69.6% | | | Authentication | 779 | 69.6% | |
| |
| ^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^ | The consent-action count above is field population, not proof that 349 papers stated a choice. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** with an audited consent-action claim and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels. The family percentages below retain the schema-field denominator and should be read the same way. |
| | |
| | ^ Family ^ N ^ States headless ^ States statefulness ^ Consent field populated (schema) ^ Public artifact ^ |
| | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% | | | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% | |
| | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% | | | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% | |
| * Multi-browser or multi-language, or a lab already fluent in it → Selenium. | * Multi-browser or multi-language, or a lab already fluent in it → Selenium. |
| * The lowest-effort modern tracking crawl → Tracker Radar Collector. | * The lowest-effort modern tracking crawl → Tracker Radar Collector. |
| - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 75 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it. | - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it. |
| - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used. | - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used. |
| - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you). | - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]]). |
| - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}. | - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}. |
| - **Run more than once.** A single crawl is a point estimate. Report the spread. | - **Run more than once.** A single crawl is a point estimate. Report the spread. |
