Measure The Web

User Tools

Site Tools

Trace: • crawler
programming:crawler

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
programming:crawler [2026/09/11 17:55] – webXray bullet: repoint the ownership-comparison pointer to the new Design:Ownership resolution, which the material moved to on 2026-09-11. The comparison-row and licence corrections this page owed were already applied (revs 1786953266 and 1788636934); se karel.kubicek.claudeprogramming:crawler [2026/09/17 10:45] (current) – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude
Line 241: Line 241:
 ==== Plain CDP ==== ==== Plain CDP ====
  
-You can skip the libraries entirely: launch Chromium with ''--remote-debugging-port'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures.+You can skip the libraries entirely: launch Chromium with ''%%--remote-debugging-port%%'', open a WebSocket, enable the domains you want, and read the events. **42 papers (3.8%) did**, most of them between 2018 and 2021 — a window that closes as Puppeteer matures.
  
 Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies. Reach for it when your analysis lives in a language with no good automation binding, when you need a CDP domain the wrapper does not expose, or when you want to know exactly what your instrument is doing. The cost, visible in the code below, is that everything a library gives you is now yours to write: waiting for the port, waiting for the load, waiting for the network to settle, and reconnecting when a target dies.
Line 314: Line 314:
  
 <WRAP info> <WRAP info>
-**Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: --database is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives.+**Sandbox gotchas we hit, since they cost half a day.** On this aarch64 container, //every full Chrome/Chromium binary// — Debian's ''/usr/bin/chromium'', Chrome for Testing, and Playwright's own ''chromium-*'' build — died at startup with ''chrome_crashpad_handler: %%--database%% is required'' and never opened the DevTools port. Only Playwright's ''chromium_headless_shell'' build started. Chrome for Testing ships no aarch64 build (the downloaded ''linux_arm'' binary is x86-64 and fails under emulation), and neither does ''chromedriver''; a working aarch64 ''chromedriver'' came from Debian's ''chromium-driver'' package, extracted with ''dpkg -x'' without root. Firefox 153 + geckodriver 0.37.1 never handed over its Marionette port in this container, with or without Xvfb, so the Selenium example is the Chromium one. None of this is a property of the libraries — it is a property of running x86-64-first browser tooling on ARM in a container, which is increasingly where CI lives.
 </WRAP> </WRAP>
  
Line 474: Line 474:
 | Headless or headful | 140 | 12.5% | | Headless or headful | 140 | 12.5% |
 | Stateful or stateless | 219 | 19.6% | | Stateful or stateless | 219 | 19.6% |
-| Consent action | 349 | 31.2% |+| Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% |
 | Interaction depth | 841 | 75.1% | | Interaction depth | 841 | 75.1% |
 | Authentication | 779 | 69.6% | | Authentication | 779 | 69.6% |
  
-^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^+The consent-action count above is field population, not proof that 349 papers stated a choice. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** with an audited consent-action claim and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels. The family percentages below retain the schema-field denominator and should be read the same way. 
 + 
 +^ Family ^ N ^ States headless ^ States statefulness ^ Consent field populated (schema) ^ Public artifact ^
 | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% | | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% |
 | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% | | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% |
Line 531: Line 533:
   - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it.   - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it.
   - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used.   - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used.
-  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you).+  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]]).
   - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}.   - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}.
   - **Run more than once.** A single crawl is a point estimate. Report the spread.   - **Run more than once.** A single crawl is a point estimate. Report the spread.
programming/crawler.1789149309.txt.gz · Last modified: by karel.kubicek.claude