| Both sides previous revisionPrevious revision | |
| programming:crawler:tracker_radar_collector [2026/09/03 09:02] – Apply the generic review pass: the 10 is nine TRC-driven crawls plus one module reuse (aziz2024_johnny drove its own CDP scripts); per-paper breakdown of how the seven inner-page papers got their pages; new pitfall for open issue #77 (early API calls miss karel.kubicek.claude | programming:crawler:tracker_radar_collector [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| </code> | </code> |
| |
| The real ''--help'' at that commit, unedited. This is the authoritative flag list; the README's prose is not (see [[#Known pitfalls|Known pitfalls]]). | The real ''%%--help%%'' at that commit, unedited. This is the authoritative flag list; the README's prose is not (see [[#Known pitfalls|Known pitfalls]]). |
| |
| <file text trc-crawl-cli-help.txt> | <file text trc-crawl-cli-help.txt> |
| </file> | </file> |
| |
| **The CLI cannot be pointed at a browser you already have.** ''browser/openBrowser.js'' takes ''executablePath || await downloadChrome(log)'', and ''cli/crawl-cli.js'' only ever sets ''executablePath'' from ''--chromium-version''; ''crawlerConductor'' calls ''downloadChrome()'' unconditionally unless ''--selenium-hub'' is given. There is no ''--executable-path'' and ''PUPPETEER_EXECUTABLE_PATH'' is ignored. On a host where Chrome for Testing has no build the CLI therefore cannot start at all — on ''aarch64'' it downloads the x86-64 archive and dies: | **The CLI cannot be pointed at a browser you already have.** ''browser/openBrowser.js'' takes ''executablePath || await downloadChrome(log)'', and ''cli/crawl-cli.js'' only ever sets ''executablePath'' from ''%%--chromium-version%%''; ''crawlerConductor'' calls ''downloadChrome()'' unconditionally unless ''%%--selenium-hub%%'' is given. There is no ''%%--executable-path%%'' and ''PUPPETEER_EXECUTABLE_PATH'' is ignored. On a host where Chrome for Testing has no build the CLI therefore cannot start at all — on ''aarch64'' it downloads the x86-64 archive and dies: |
| |
| <code> | <code> |
| |
| ^ Surface ^ Tracker Radar Collector ^ OpenWPM ^ Measurement consequence ^ | ^ Surface ^ Tracker Radar Collector ^ OpenWPM ^ Measurement consequence ^ |
| | Browser and control | Chromium through Puppeteer and CDP; attaches to page, iframe, worker, shared-worker and service-worker targets. ''--selenium-hub'' can source a remote **Chrome** from a Selenium Grid, but CDP is still the instrument — Selenium here only provisions the browser | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. | | | Browser and control | Chromium through Puppeteer and CDP; attaches to page, iframe, worker, shared-worker and service-worker targets. ''%%--selenium-hub%%'' can source a remote **Chrome** from a Selenium Grid, but CDP is still the instrument — Selenium here only provisions the browser | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. | |
| | Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, a nine-header safelist of response headers, timing, failure and redirect fields, initiators and optionally a response-body hash. WebSocket events go into the same array through ''handleWebSocket'' with fewer fields. The default record contains no raw response bodies, no request headers and no request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and a deduplication hash. If the question is about sent headers, POST contents, or response bytes, extend the collector or use an instrument that records them; do not infer their presence from a URL log. | | | Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, a nine-header safelist of response headers, timing, failure and redirect fields, initiators and optionally a response-body hash. WebSocket events go into the same array through ''handleWebSocket'' with fewer fields. The default record contains no raw response bodies, no request headers and no request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and a deduplication hash. If the question is about sent headers, POST contents, or response bytes, extend the collector or use an instrument that records them; do not infer their presence from a URL log. | |
| | Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalised record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it drops the CDP cookie's ''value'', ''httpOnly'', ''secure'' and ''size''. But ''set-cookie'' **is** in the request collector's default header safelist, so HTTP-set values are still in the crawl | ''cookie_instrument'' records every cookie change — from JavaScript **and** from HTTP responses — in the ''javascript_cookies'' table | TRC's cookie key is a final-state snapshot, not a set-event history. For HTTP-set cookies, read ''requests[].responseHeaders["set-cookie"]'' instead. Only ''document.cookie'' writes need instrumentation you add yourself. | | | Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalised record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it drops the CDP cookie's ''value'', ''httpOnly'', ''secure'' and ''size''. But ''set-cookie'' **is** in the request collector's default header safelist, so HTTP-set values are still in the crawl | ''cookie_instrument'' records every cookie change — from JavaScript **and** from HTTP responses — in the ''javascript_cookies'' table | TRC's cookie key is a final-state snapshot, not a set-event history. For HTTP-set cookies, read ''requests[].responseHeaders["set-cookie"]'' instead. Only ''document.cookie'' writes need instrumentation you add yourself. | |
| | JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither "JavaScript captured" nor "API calls captured" is a sufficient methods description. The one published comparison of these two surfaces found the gap was **configuration, not architecture** — see below. Publish the collector configuration and the API list. | | | JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither "JavaScript captured" nor "API calls captured" is a sufficient methods description. The one published comparison of these two surfaces found the gap was **configuration, not architecture** — see below. Publish the collector configuration and the API list. | |
| | Targets, frames and navigation | TargetCollector records target type and URL, captured at **attach** time | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity — and see the pitfall about what ''targets'' actually contains. | | | Targets, frames and navigation | TargetCollector records target type and URL, captured at **attach** time | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity — and see the pitfall about what ''targets'' actually contains. | |
| | Consent and interaction | CookiePopupsCollector records autoconsent CMP observations; the ''--autoconsent-action'' flag takes ''optIn'' or ''optOut''. The conductor also supports mobile emulation, and scrolling/interaction and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | "Before consent" is not a browser default. Record the action, the autoconsent version and the wait/interaction sequence — **and the per-site success rate**, because TRC records CMPs it did not act on. | | | Consent and interaction | CookiePopupsCollector records autoconsent CMP observations; the ''%%--autoconsent-action%%'' flag takes ''optIn'' or ''optOut''. The conductor also supports mobile emulation, and scrolling/interaction and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | "Before consent" is not a browser default. Record the action, the autoconsent version and the wait/interaction sequence — **and the per-site success rate**, because TRC records CMPs it did not act on. | |
| | State and filtering | The CLI can use mobile emulation, a region code, a proxy and ''--only-3p'', which filters by eTLD+1 during collection. A fresh temporary profile per site is unconditional and cannot be switched off from the CLI | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organisation-level first parties are handled later by Detector, with separate configuration. TRC cannot do a stateful crawl from the CLI at all. | | | State and filtering | The CLI can use mobile emulation, a region code, a proxy and ''%%--only-3p%%'', which filters by eTLD+1 during collection. A fresh temporary profile per site is unconditional and cannot be switched off from the CLI | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organisation-level first parties are handled later by Detector, with separate configuration. TRC cannot do a stateful crawl from the CLI at all. | |
| |
| TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. A final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments, and the corpus shows many such forks. | TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. A final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments, and the corpus shows many such forks. |
| | ''data'' | Object keyed by the selected collectors | | | ''data'' | Object keyed by the selected collectors | |
| |
| The contents of ''data'' depend on ''--data-collectors''. Seven collectors are selectable. ''requests'', ''cookies'', ''apis'', ''targets'' and ''cookiepopups'' below were checked against real output; ''screenshots'' and ''trace'' against source only. | The contents of ''data'' depend on ''%%--data-collectors%%''. Seven collectors are selectable. ''requests'', ''cookies'', ''apis'', ''targets'' and ''cookiepopups'' below were checked against real output; ''screenshots'' and ''trace'' against source only. |
| |
| ^ ''data'' key ^ What to expect ^ What to retain or report ^ | ^ ''data'' key ^ What to expect ^ What to retain or report ^ |
| | ''trace'' | Chrome trace data when the trace collector is selected | Trace configuration and whether it was sampled or retained | | | ''trace'' | Chrome trace data when the trace collector is selected | Trace configuration and whether it was sampled or retained | |
| |
| The ''CollectorData'' typedef lives in ''helpers/collectorsList.js'', and it is not the authority on this list. It declares an ''elements'' key for which no ''ElementsCollector'' exists anywhere in the repository, and it omits ''trace'', which ''TraceCollector.id()'' returns and ''--help'' offers. Take the collector list from ''--help'' and the field list from the collector source. | The ''CollectorData'' typedef lives in ''helpers/collectorsList.js'', and it is not the authority on this list. It declares an ''elements'' key for which no ''ElementsCollector'' exists anywhere in the repository, and it omits ''trace'', which ''TraceCollector.id()'' returns and ''%%--help%%'' offers. Take the collector list from ''%%--help%%'' and the field list from the collector source. |
| |
| The CLI also writes ''metadata.json'', with ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (''urls'', ''skipped'', ''successes'', ''failures''), ''config'' and ''environment'' (''projectVersion'', ''hostname'', ''cpus'', ''username''). Two things about it: | The CLI also writes ''metadata.json'', with ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (''urls'', ''skipped'', ''successes'', ''failures''), ''config'' and ''environment'' (''projectVersion'', ''hostname'', ''cpus'', ''username''). Two things about it: |
| * **There is no ''browserContext'' option.** The README offers one; the code takes ''browserConnection''. Pass a connection if you want to share state, and note that this is a module-only route. | * **There is no ''browserContext'' option.** The README offers one; the code takes ''browserConnection''. Pass a connection if you want to share state, and note that this is a module-only route. |
| * **Headless is not a choice.** ''VISUAL_DEBUG'' is a hard-coded ''const … = false'' in ''constants.js'' with no flag and no environment override. To run headful you must edit that file — and if you do, ''crawler.js'' stops closing browsers. | * **Headless is not a choice.** ''VISUAL_DEBUG'' is a hard-coded ''const … = false'' in ''constants.js'' with no flag and no environment override. To run headful you must edit that file — and if you do, ''crawler.js'' stops closing browsers. |
| * **The default user agent is a constant, not the browser.** ''DEFAULT_USER_AGENT'' is a fixed macOS ''Chrome/135.0.0.0'' string and ''MOBILE_USER_AGENT'' a fixed Android one; emulation is on unless ''--selenium-hub'' is used. A Linux headless crawl presents as macOS Chrome 135 regardless of the Chromium actually running, and the string drifts as the pin moves. Default viewport is 1440×812; mobile is 412×691 at device pixel ratio 2. | * **The default user agent is a constant, not the browser.** ''DEFAULT_USER_AGENT'' is a fixed macOS ''Chrome/135.0.0.0'' string and ''MOBILE_USER_AGENT'' a fixed Android one; emulation is on unless ''%%--selenium-hub%%'' is used. A Linux headless crawl presents as macOS Chrome 135 regardless of the Chromium actually running, and the string drifts as the pin moves. Default viewport is 1440×812; mobile is 412×691 at device pixel ratio 2. |
| * **Detection is not action.** In a real ''optOut'' run against a Sourcepoint site, ''cmps[0]'' came back ''name: "Sourcepoint-frame"'' with ''open'', ''started'' and ''succeeded'' all ''false'' — and three cookies were set anyway. "We opted out with TRC's autoconsent" is not a measurable statement; the per-site ''succeeded'' rate is. | * **Detection is not action.** In a real ''optOut'' run against a Sourcepoint site, ''cmps[0]'' came back ''name: "Sourcepoint-frame"'' with ''open'', ''started'' and ''succeeded'' all ''false'' — and three cookies were set anyway. "We opted out with TRC's autoconsent" is not a measurable statement; the per-site ''succeeded'' rate is. |
| * **''filterListMatched'' is always ''false''.** autoconsent removed EasyList and filter-list support in v16.0.0 (2026-06-23) and TRC pins ''^16.8.1''; in TRC the backing field is initialised ''false'' and never assigned. A reader who filters on it concludes no CMP is ever on a filter list. | * **''filterListMatched'' is always ''false''.** autoconsent removed EasyList and filter-list support in v16.0.0 (2026-06-23) and TRC pins ''^16.8.1''; in TRC the backing field is initialised ''false'' and never assigned. A reader who filters on it concludes no CMP is ever on a filter list. |
| * **''--only-3p'' is an eTLD+1 filter applied while collecting.** It is not an entity or ownership decision, and it is irreversible: if you may later want to study CNAME cloaking or first-party cookies, do not enable it. | * **''%%--only-3p%%'' is an eTLD+1 filter applied while collecting.** It is not an entity or ownership decision, and it is irreversible: if you may later want to study CNAME cloaking or first-party cookies, do not enable it. |
| * **The CLI's success banner lies.** It prints "✅ Finished successfully." after a run with zero successful crawls. Read ''stats'' in ''metadata.json''. | * **The CLI's success banner lies.** It prints "✅ Finished successfully." after a run with zero successful crawls. Read ''stats'' in ''metadata.json''. |
| * **APICallCollector misses calls made early in page load, and the bug is still open.** ''tracker-radar-collector'' issue #77, //"Early browser API accesses and function calls are missed"//, was opened on 2022-09-26 and is **still open**: the maintainers describe "a race condition between API collection script setting things up and scripts on the page already running", note that it makes TRC's own API-collection integration test flaky, and last said "We're exploring different options to fix this" on 2023-03-06.((''github.com/duckduckgo/tracker-radar-collector/issues/77'', fetched through the GitHub API on 2026-09-03: ''"state": "open"'', ''"created_at": "2022-09-26"'', ''"updated_at": "2023-03-06"'', four comments, ''"closed_at": null''.)) Two of the ten corpus papers cite it, and {[senol2024_double]} wrote a replacement collector because of it: //"Although TRC already has the capability to intercept JavaScript API calls, we introduced a separate collector due to a known TRC bug that causes it to miss the initial function calls."// An empty ''callStats'' for a script that runs at the top of the document is not evidence that it touched no API. | * **APICallCollector misses calls made early in page load, and the bug is still open.** ''tracker-radar-collector'' issue #77, //"Early browser API accesses and function calls are missed"//, was opened on 2022-09-26 and is **still open**: the maintainers describe "a race condition between API collection script setting things up and scripts on the page already running", note that it makes TRC's own API-collection integration test flaky, and last said "We're exploring different options to fix this" on 2023-03-06.((''github.com/duckduckgo/tracker-radar-collector/issues/77'', fetched through the GitHub API on 2026-09-03: ''"state": "open"'', ''"created_at": "2022-09-26"'', ''"updated_at": "2023-03-06"'', four comments, ''"closed_at": null''.)) Two of the ten corpus papers cite it, and {[senol2024_double]} wrote a replacement collector because of it: //"Although TRC already has the capability to intercept JavaScript API calls, we introduced a separate collector due to a known TRC bug that causes it to miss the initial function calls."// An empty ''callStats'' for a script that runs at the top of the document is not evidence that it touched no API. |
| Tracker Radar describes itself as a dataset of common third-party domains and their observed behaviour, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking. | Tracker Radar describes itself as a dataset of common third-party domains and their observed behaviour, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking. |
| |
| The two first-party decisions must not be collapsed. TRC's ''--only-3p'' uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]} | The two first-party decisions must not be collapsed. TRC's ''%%--only-3p%%'' uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]} |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
| Before collecting, decide what the output is supposed to answer. Then record at least: | Before collecting, decide what the output is supposed to answer. Then record at least: |
| |
| * the TRC repository **commit** — not ''projectVersion'', which is always ''1.0.0'' — plus the package-lock state, Node.js, Puppeteer and Chromium versions, or the explicit ''--chromium-version''; | * the TRC repository **commit** — not ''projectVersion'', which is always ''1.0.0'' — plus the package-lock state, Node.js, Puppeteer and Chromium versions, or the explicit ''%%--chromium-version%%''; |
| * the site list, its source, version and date, the URL selection, and whether the crawl visits only a homepage or follows inner links, with the sub-page budget; | * the site list, its source, version and date, the URL selection, and whether the crawl visits only a homepage or follows inner links, with the sub-page budget; |
| * crawl dates, region/vantage/IP conditions, proxy, success and timeout policy, concurrency, the real ''maxLoadTimeMs'' you ran with, and the post-load wait; | * crawl dates, region/vantage/IP conditions, proxy, success and timeout policy, concurrency, the real ''maxLoadTimeMs'' you ran with, and the post-load wait; |
| * the denominator for every prevalence number: attempted sites, successfully crawled sites, sites where the relevant interaction worked, or sites retained by Detector. | * the denominator for every prevalence number: attempted sites, successfully crawled sites, sites where the relevant interaction worked, or sites retained by Detector. |
| |
| Anti-bot measures and consent automation are interventions, not harmless crawler options: they change the page, the requests and the cookies. A result from ''--only-3p'' is conditioned on an eTLD+1 filter and is not a result about ownership, entity prevalence or CNAME cloaking. And do not report a "number of trackers" without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records — those are six different units. | Anti-bot measures and consent automation are interventions, not harmless crawler options: they change the page, the requests and the cookies. A result from ''%%--only-3p%%'' is conditioned on an eTLD+1 filter and is not a result about ownership, entity prevalence or CNAME cloaking. And do not report a "number of trackers" without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records — those are six different units. |
| |
| ===== Methodology and limitations of these figures ===== | ===== Methodology and limitations of these figures ===== |
