Differences

This shows you the differences between two versions of the page.

--- programming:crawler [2026/09/17 07:22] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
+++ programming:crawler [2026/09/17 10:45] (current) – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude
@@ Line 474: / Line 474: @@
 | Headless or headful | 140 | 12.5% |
 | Stateful or stateless | 219 | 19.6% |
-| Consent action | 349 | 31.2% |
+| Consent action field populated (schema; not an audited paper claim) | 349 | 31.2% |
 | Interaction depth | 841 | 75.1% |
 | Authentication | 779 | 69.6% |
-^ Family ^ N ^ States headless ^ States statefulness ^ States consent action ^ Public artifact ^
+The consent-action count above is field population, not proof that 349 papers stated a choice. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** with an audited consent-action claim and **279 of 313 (89.1%)** unsupported ''no-interaction'' labels. The family percentages below retain the schema-field denominator and should be read the same way.
+^ Family ^ N ^ States headless ^ States statefulness ^ Consent field populated (schema) ^ Public artifact ^
 | Selenium | 242 | 21.1% | 27.7% | 36.0% | 57.0% |
 | Puppeteer | 76 | 35.5% | 35.5% | 57.9% | 67.1% |
@@ Line 531: / Line 533: @@
   - **Do not write a new crawler for a solved problem.** Nearly three in ten crawling papers used a home-grown one, and 74 of them identify it by nothing but a name they invented. If you must build one, say what it is built on, and publish it.
   - **State headless or headful and justify it.** Headless is the most detectable configuration you can choose {[vastel2018_scanner]} and only 12.5% of papers say which they used.
-  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]], a page this wiki still owes you).
+  - **Pin the browser binary, not just the library.** Puppeteer and Playwright do this for you; Selenium does not. Record the exact build and archive it with your research artefact (see [[:Artifacts]]).
   - **Validate against something that is not your crawler.** A manual visit to a sample, a second browser engine, or a second vantage point {[jueckstock2021_realistic]}. Crawls diverge from human browsing in ways your crawl cannot see {[zeber2020representativeness]}, and repeated crawls diverge from each other {[demir2022_reproducibility]}.
   - **Run more than once.** A single crawl is a point estimate. Report the spread.