| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler:tracker_radar_collector [2026/09/03 08:51] – Soften an over-strong inner-page claim: TRC has no link extraction or depth parameter at all (verified by grep over crawler/conductor/CLI/collectors/helpers and the README), so the seven papers either wrote traversal or supplied a URL list — leaky-forms d karel.kubicek.claude | programming:crawler:tracker_radar_collector [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| </code> | </code> |
| |
| The real ''--help'' at that commit, unedited. This is the authoritative flag list; the README's prose is not (see [[#Known pitfalls|Known pitfalls]]). | The real ''%%--help%%'' at that commit, unedited. This is the authoritative flag list; the README's prose is not (see [[#Known pitfalls|Known pitfalls]]). |
| |
| <file text trc-crawl-cli-help.txt> | <file text trc-crawl-cli-help.txt> |
| </file> | </file> |
| |
| **The CLI cannot be pointed at a browser you already have.** ''browser/openBrowser.js'' takes ''executablePath || await downloadChrome(log)'', and ''cli/crawl-cli.js'' only ever sets ''executablePath'' from ''--chromium-version''; ''crawlerConductor'' calls ''downloadChrome()'' unconditionally unless ''--selenium-hub'' is given. There is no ''--executable-path'' and ''PUPPETEER_EXECUTABLE_PATH'' is ignored. On a host where Chrome for Testing has no build the CLI therefore cannot start at all — on ''aarch64'' it downloads the x86-64 archive and dies: | **The CLI cannot be pointed at a browser you already have.** ''browser/openBrowser.js'' takes ''executablePath || await downloadChrome(log)'', and ''cli/crawl-cli.js'' only ever sets ''executablePath'' from ''%%--chromium-version%%''; ''crawlerConductor'' calls ''downloadChrome()'' unconditionally unless ''%%--selenium-hub%%'' is given. There is no ''%%--executable-path%%'' and ''PUPPETEER_EXECUTABLE_PATH'' is ignored. On a host where Chrome for Testing has no build the CLI therefore cannot start at all — on ''aarch64'' it downloads the x86-64 archive and dies: |
| |
| <code> | <code> |
| rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2 | rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2 |
| Max number of retries (2) exceeded for "https://example.com". | Max number of retries (2) exceeded for "https://example.com". |
| ... | |
| ✅ Finished successfully. | ✅ Finished successfully. |
| | ... |
| Sucessful crawls: 0/1 (0.00%) | Sucessful crawls: 0/1 (0.00%) |
| </code> | </code> |
| |
| Note the last three lines: the CLI prints its success banner on a crawl that collected nothing. **The exit banner is not a success signal — read ''stats'' in ''metadata.json''.** | Note the banner and the tally: the CLI prints "✅ Finished successfully." on a crawl that collected nothing. The elided lines are the start, finish and duration that ''CLIReporter'' prints between them. **The exit banner is not a success signal — read ''stats'' in ''metadata.json''.** |
| |
| The single-crawl module entry point does accept ''executablePath'', which is the way round it. ''TRC_UNSAFE_DISABLE_SANDBOX=1'' is TRC's own environment variable, read in ''browser/openBrowser.js'', and is needed in a container without user namespaces. | The single-crawl module entry point does accept ''executablePath'', which is the way round it. ''TRC_UNSAFE_DISABLE_SANDBOX=1'' is TRC's own environment variable, read in ''browser/openBrowser.js'', and is needed in a container without user namespaces. |
| collectorFlags: {}, | collectorFlags: {}, |
| }); | }); |
| | // A timed-out crawl still writes a file, and the file still says so. Crash |
| | // rather than print "wrote" over a partial capture: this page's own lesson is |
| | // that a success message is not a success signal. |
| | if (data.timeout) throw new Error(`${u} hit the crawler's timeout path; capture is partial`); |
| const name = u.replace(/[^a-z0-9]+/gi, '_') + '.json'; | const name = u.replace(/[^a-z0-9]+/gi, '_') + '.json'; |
| fs.writeFileSync(path.join(outDir, name), JSON.stringify(data, null, 2)); | fs.writeFileSync(path.join(outDir, name), JSON.stringify(data, null, 2)); |
| </file> | </file> |
| |
| Its whole output for ''https://example.com'', which is what the [[#Output schema|Output schema]] section below was checked against. Only two fields in it are not reproducible run to run: the timestamps, and ''size'' — a second run of the same script against the same URL gave ''size: 474'' with an identical ''responseBodyHash'', for the reason in the pitfalls below. | Its whole output for ''https://example.com'', which is what the [[#Output schema|Output schema]] section below was checked against. Four fields in it are not reproducible run to run: the two timestamps, ''time'' (a measured duration), ''remoteIPAddress'' (anycast) and ''size'' — a second run of the same script against the same URL gave ''size: 474'' with an identical ''responseBodyHash'', for the reason in the pitfalls below. Everything else is stable. |
| |
| <code javascript> | <code javascript> |
| |
| ^ Surface ^ Tracker Radar Collector ^ OpenWPM ^ Measurement consequence ^ | ^ Surface ^ Tracker Radar Collector ^ OpenWPM ^ Measurement consequence ^ |
| | Browser and control | Chromium through Puppeteer and CDP; attaches to page, iframe, worker, shared-worker and service-worker targets. ''--selenium-hub'' can source a remote **Chrome** from a Selenium Grid, but CDP is still the instrument — Selenium here only provisions the browser | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. | | | Browser and control | Chromium through Puppeteer and CDP; attaches to page, iframe, worker, shared-worker and service-worker targets. ''%%--selenium-hub%%'' can source a remote **Chrome** from a Selenium Grid, but CDP is still the instrument — Selenium here only provisions the browser | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. | |
| | Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, a nine-header safelist of response headers, timing, failure and redirect fields, initiators and optionally a response-body hash. WebSocket events go into the same array through ''handleWebSocket'' with fewer fields. The default record contains no raw response bodies, no request headers and no request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and a deduplication hash. If the question is about sent headers, POST contents, or response bytes, extend the collector or use an instrument that records them; do not infer their presence from a URL log. | | | Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, a nine-header safelist of response headers, timing, failure and redirect fields, initiators and optionally a response-body hash. WebSocket events go into the same array through ''handleWebSocket'' with fewer fields. The default record contains no raw response bodies, no request headers and no request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and a deduplication hash. If the question is about sent headers, POST contents, or response bytes, extend the collector or use an instrument that records them; do not infer their presence from a URL log. | |
| | Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalised record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it drops the CDP cookie's ''value'', ''httpOnly'', ''secure'' and ''size''. But ''set-cookie'' **is** in the request collector's default header safelist, so HTTP-set values are still in the crawl | ''cookie_instrument'' records every cookie change — from JavaScript **and** from HTTP responses — in the ''javascript_cookies'' table | TRC's cookie key is a final-state snapshot, not a set-event history. For HTTP-set cookies, read ''requests[].responseHeaders["set-cookie"]'' instead. Only ''document.cookie'' writes need instrumentation you add yourself. | | | Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalised record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it drops the CDP cookie's ''value'', ''httpOnly'', ''secure'' and ''size''. But ''set-cookie'' **is** in the request collector's default header safelist, so HTTP-set values are still in the crawl | ''cookie_instrument'' records every cookie change — from JavaScript **and** from HTTP responses — in the ''javascript_cookies'' table | TRC's cookie key is a final-state snapshot, not a set-event history. For HTTP-set cookies, read ''requests[].responseHeaders["set-cookie"]'' instead. Only ''document.cookie'' writes need instrumentation you add yourself. | |
| | JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither "JavaScript captured" nor "API calls captured" is a sufficient methods description. The one published comparison of these two surfaces found the gap was **configuration, not architecture** — see below. Publish the collector configuration and the API list. | | | JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither "JavaScript captured" nor "API calls captured" is a sufficient methods description. The one published comparison of these two surfaces found the gap was **configuration, not architecture** — see below. Publish the collector configuration and the API list. | |
| | Targets, frames and navigation | TargetCollector records target type and URL, captured at **attach** time | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity — and see the pitfall about what ''targets'' actually contains. | | | Targets, frames and navigation | TargetCollector records target type and URL, captured at **attach** time | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity — and see the pitfall about what ''targets'' actually contains. | |
| | Consent and interaction | CookiePopupsCollector records autoconsent CMP observations; the ''--autoconsent-action'' flag takes ''optIn'' or ''optOut''. The conductor also supports mobile emulation, and scrolling/interaction and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | "Before consent" is not a browser default. Record the action, the autoconsent version and the wait/interaction sequence — **and the per-site success rate**, because TRC records CMPs it did not act on. | | | Consent and interaction | CookiePopupsCollector records autoconsent CMP observations; the ''%%--autoconsent-action%%'' flag takes ''optIn'' or ''optOut''. The conductor also supports mobile emulation, and scrolling/interaction and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | "Before consent" is not a browser default. Record the action, the autoconsent version and the wait/interaction sequence — **and the per-site success rate**, because TRC records CMPs it did not act on. | |
| | State and filtering | The CLI can use mobile emulation, a region code, a proxy and ''--only-3p'', which filters by eTLD+1 during collection. A fresh temporary profile per site is unconditional and cannot be switched off from the CLI | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organisation-level first parties are handled later by Detector, with separate configuration. TRC cannot do a stateful crawl from the CLI at all. | | | State and filtering | The CLI can use mobile emulation, a region code, a proxy and ''%%--only-3p%%'', which filters by eTLD+1 during collection. A fresh temporary profile per site is unconditional and cannot be switched off from the CLI | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organisation-level first parties are handled later by Detector, with separate configuration. TRC cannot do a stateful crawl from the CLI at all. | |
| |
| TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. A final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments, and the corpus shows many such forks. | TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. A final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments, and the corpus shows many such forks. |
| No paper in these seven venues runs TRC against OpenWPM, or against a plain Playwright crawl, on the same sample — so nothing here ranks the instruments. Two papers do compare crawler architectures empirically on one sample, and both are worth reading before you assume a difference you measure is architectural: | No paper in these seven venues runs TRC against OpenWPM, or against a plain Playwright crawl, on the same sample — so nothing here ranks the instruments. Two papers do compare crawler architectures empirically on one sample, and both are worth reading before you assume a difference you measure is architectural: |
| |
| * {[ahmad2020_apophanies]} ran **eight** crawlers — from ''wget'' and ''curl'' through PhantomJS, Selenium and Puppeteer to OpenWPM and the Tor Browser Crawler — over a list of "932 unique domains" drawn from the Alexa and Umbrella top 500s, and reported that "the choice of crawler has a critical impact on the data generated". It is the closest thing the field has to a crawler-choice benchmark, and it predates TRC. | * {[ahmad2020_apophanies]} ran **eight** crawlers — from ''wget'' and ''curl'' through PhantomJS, Selenium and Puppeteer to OpenWPM and the Tor Browser Crawler — over a list of "932 unique domains" drawn from the Alexa and Umbrella top 500s, and reported that "the choice of crawler has a critical impact on the data generated". It is the closest thing the field has to a crawler-choice benchmark, and its crawls (April 2019) predate TRC's first commit (2020-02-27). |
| * {[sanchezrola2023_rods]} compared its own CDP-breakpoint crawler with OpenWPM on the same random sample of 1,000 phishing sites: //"we found 728 websites calling fingerprint-related APIs. Our crawler was able to detect and log all invocations, while OpenWPM missed calls in 348 (47.8%) of the websites."// The headline number is not the finding. The paper's own explanation is that //"OpenWPM monitors a smaller set of fingerprinting APIs"// — 36 of the 48 API types seen — and after instrumenting the missing ones, //"both our crawler and OpenWPM were able to detect all the 2092 calls performed on the test set"//. The 47.8% gap was a default API list, not a capability of CDP over Firefox instrumentation. | * {[sanchezrola2023_rods]} compared its own CDP-instrumented Chromium crawler with OpenWPM on the same random sample of 1,000 phishing sites: //"we found 728 websites calling fingerprint-related APIs. Our crawler was able to detect and log all invocations, while OpenWPM missed calls in 348 (47.8%) of the websites."// The headline number is not the finding. The paper's own explanation is that //"OpenWPM monitors a smaller set of fingerprinting APIs"// — 36 of the 48 API types seen — and after instrumenting the missing ones, //"both our crawler and OpenWPM were able to detect all the 2092 calls performed on the test set"//. The 47.8% gap was a default API list, not a capability of CDP over Firefox instrumentation. |
| |
| That is the single most useful thing to take from this section: **a difference between two crawlers is a difference between two configurations until you have shown otherwise.** | That is the single most useful thing to take from this section: **a difference between two crawlers is a difference between two configurations until you have shown otherwise.** |
| | ''data'' | Object keyed by the selected collectors | | | ''data'' | Object keyed by the selected collectors | |
| |
| The contents of ''data'' depend on ''--data-collectors''. Seven collectors are selectable; ''requests'', ''cookies'', ''apis'' and ''targets'' below were checked against real output, the rest against source. | The contents of ''data'' depend on ''%%--data-collectors%%''. Seven collectors are selectable. ''requests'', ''cookies'', ''apis'', ''targets'' and ''cookiepopups'' below were checked against real output; ''screenshots'' and ''trace'' against source only. |
| |
| ^ ''data'' key ^ What to expect ^ What to retain or report ^ | ^ ''data'' key ^ What to expect ^ What to retain or report ^ |
| | ''requests'' | An array of request/response observations. Real keys: ''url'', ''method'', ''type'', ''status'', ''size'', ''remoteIPAddress'', ''responseHeaders'', ''responseBodyHash'', ''initiators'', ''time'', plus ''failureReason'', ''redirectedTo'' and ''redirectedFrom'' where they apply. ''size'' is CDP's ''encodedDataLength'' — bytes on the wire, headers included — and is coerced to ''null'' when negative | Whether first-party traffic was filtered; whether hashes, WebSockets and initiators were kept; whether a fork added request headers, POST bodies or response bodies; and that ''size'' is not a body length | | | ''requests'' | An array of request/response observations. Real keys: ''url'', ''method'', ''type'', ''status'', ''size'', ''remoteIPAddress'', ''responseHeaders'', ''responseBodyHash'', ''initiators'', ''time'', plus ''failureReason'', ''redirectedTo'' and ''redirectedFrom'' where they apply. ''size'' is CDP's ''encodedDataLength'' — bytes on the wire, headers included — and is coerced to ''null'' when negative | Whether first-party traffic was filtered; whether hashes, WebSockets and initiators were kept; whether a fork added request headers, POST bodies or response bodies; and that ''size'' is not a body length | |
| | ''cookies'' | An array of final cookie-jar entries: ''name'', ''domain'', ''path'', ''expires'', ''session'', ''sameSite''. ''expires'' is **milliseconds** (''Math.floor(cdpDate * 1000)''), and a session cookie's ''expires'' becomes ''undefined'', so the key is **absent** from the JSON — as is ''sameSite'' when unset | The fact that values and set events are absent from this key; that a missing key is not a null; profile reset and crawl-end timing | | | ''cookies'' | An array of final cookie-jar entries: ''name'', ''domain'', ''path'', ''expires'', ''session'', ''sameSite''. ''expires'' is **milliseconds** (''Math.floor(cdpDate * 1000)''), and a session cookie's ''expires'' becomes ''undefined'', so the key is **absent** from the JSON — as is ''sameSite'' when unset | The fact that values and set events are absent from this key; that a missing key is not a null; profile reset and crawl-end timing | |
| | ''apis'' | ''callStats'' grouped by source URL and API description, plus ''savedCalls'' when configured breakpoints save arguments | The API/breakpoint configuration, because an empty or partial list is a result of configuration rather than proof of no API use | | | ''apis'' | ''callStats'' grouped by source URL and API description, plus ''savedCalls'' when configured breakpoints save arguments | The API/breakpoint configuration //and// how you handled issue #77, because an empty or partial list can be either configuration or the early-call race described in the pitfalls — neither is proof of no API use | |
| | ''targets'' | Target type and URL, captured at attach time. On a plain crawl the top-level entry is ''{"type":"page","url":"about:blank"}'' — the page target is attached before navigation | Whether iframes, workers and service workers were included in the analysis, and that the top-level row carries no URL | | | ''targets'' | Target type and URL, captured at attach time. On a plain crawl the top-level entry is ''{"type":"page","url":"about:blank"}'' — the page target is attached before navigation | Whether iframes, workers and service workers were included in the analysis, and that the top-level row carries no URL | |
| | ''screenshots'' | Screenshot data in the in-process result; the CLI writes JPEG files and replaces the field with a path | Browser viewport/device mode, timing and whether screenshots were used as evidence | | | ''screenshots'' | Screenshot data in the in-process result; the CLI writes JPEG files and replaces the field with a path | Browser viewport/device mode, timing and whether screenshots were used as evidence | |
| | ''cookiepopups'' | ''{cmps, performance, scrapedFrames}''. ''cmps[]'' carries ''name'', ''open'', ''started'', ''succeeded'', ''selfTestFail'', ''errors'', ''patterns'', ''snippets'', ''filterListMatched''. ''scrapedFrames[]'' carries the text and CSS selector of **every** clickable element in the frame, not only the banner | The consent action and autoconsent version; the per-site ''succeeded'' rate; and that this key puts page content in your archive | | | ''cookiepopups'' | ''{cmps, performance, scrapedFrames}''. ''cmps[]'' carries ''final'', ''name'', ''open'', ''started'', ''succeeded'', ''selfTestFail'', ''errors'', ''patterns'', ''snippets'' and ''filterListMatched''. ''scrapedFrames[]'' carries the text and CSS selector of **every** clickable element in the frame, not only the banner | The consent action and autoconsent version; the per-site ''succeeded'' rate; and that this key puts page content in your archive | |
| | ''trace'' | Chrome trace data when the trace collector is selected | Trace configuration and whether it was sampled or retained | | | ''trace'' | Chrome trace data when the trace collector is selected | Trace configuration and whether it was sampled or retained | |
| |
| The ''CollectorData'' typedef lives in ''helpers/collectorsList.js'', and it is not the authority on this list. It declares an ''elements'' key for which no ''ElementsCollector'' exists anywhere in the repository, and it omits ''trace'', which ''TraceCollector.id()'' returns and ''--help'' offers. Take the collector list from ''--help'' and the field list from the collector source. | The ''CollectorData'' typedef lives in ''helpers/collectorsList.js'', and it is not the authority on this list. It declares an ''elements'' key for which no ''ElementsCollector'' exists anywhere in the repository, and it omits ''trace'', which ''TraceCollector.id()'' returns and ''%%--help%%'' offers. Take the collector list from ''%%--help%%'' and the field list from the collector source. |
| |
| The CLI also writes ''metadata.json'', with ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (''urls'', ''skipped'', ''successes'', ''failures''), ''config'' and ''environment'' (''projectVersion'', ''hostname'', ''cpus'', ''username''). Two things about it: | The CLI also writes ''metadata.json'', with ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (''urls'', ''skipped'', ''successes'', ''failures''), ''config'' and ''environment'' (''projectVersion'', ''hostname'', ''cpus'', ''username''). Two things about it: |
| |
| * **''config'' omits every falsy setting.** Each field is written as ''value || undefined'', so ''filterOutFirstParty: false'' produces **no key at all** and the file cannot distinguish "we did not filter first-party traffic" from "this was not recorded". The same applies to ''emulateMobile'' and a zero crawler count. Your negative settings have to be published by you, not by the file. | * **''config'' omits every falsy setting.** Each field is written as ''value || undefined'', so ''filterOutFirstParty: false'' produces **no key at all** and the file cannot distinguish "we did not filter first-party traffic" from "this was not recorded". The same applies to ''emulateMobile'' and a zero crawler count. Your negative settings have to be published by you, not by the file. |
| * **''projectVersion'' carries no information.** It is ''package.json''`s ''version'', which has been ''1.0.0'' since the initial commit and has never changed. That is why pinning the commit is not a preference but the only way to identify the code. | * **''projectVersion'' carries no information.** It is ''package.json'''s ''version'', which has been ''1.0.0'' since the initial commit and has never changed. That is why pinning the commit is not a preference but the only way to identify the code. |
| |
| Treat ''hostname'' and ''username'' as data to redact before sharing a crawl archive. A "TRC JSON" without the commit, the collector list and the metadata is not enough to reproduce a measurement. | Treat ''hostname'' and ''username'' as data to redact before sharing a crawl archive. A "TRC JSON" without the commit, the collector list and the metadata is not enough to reproduce a measurement. |
| * **There is no ''browserContext'' option.** The README offers one; the code takes ''browserConnection''. Pass a connection if you want to share state, and note that this is a module-only route. | * **There is no ''browserContext'' option.** The README offers one; the code takes ''browserConnection''. Pass a connection if you want to share state, and note that this is a module-only route. |
| * **Headless is not a choice.** ''VISUAL_DEBUG'' is a hard-coded ''const … = false'' in ''constants.js'' with no flag and no environment override. To run headful you must edit that file — and if you do, ''crawler.js'' stops closing browsers. | * **Headless is not a choice.** ''VISUAL_DEBUG'' is a hard-coded ''const … = false'' in ''constants.js'' with no flag and no environment override. To run headful you must edit that file — and if you do, ''crawler.js'' stops closing browsers. |
| * **The default user agent is a constant, not the browser.** ''DEFAULT_USER_AGENT'' is a fixed macOS ''Chrome/135.0.0.0'' string and ''MOBILE_USER_AGENT'' a fixed Android one; emulation is on unless ''--selenium-hub'' is used. A Linux headless crawl presents as macOS Chrome 135 regardless of the Chromium actually running, and the string drifts as the pin moves. Default viewport is 1440×812; mobile is 412×691 at device pixel ratio 2. | * **The default user agent is a constant, not the browser.** ''DEFAULT_USER_AGENT'' is a fixed macOS ''Chrome/135.0.0.0'' string and ''MOBILE_USER_AGENT'' a fixed Android one; emulation is on unless ''%%--selenium-hub%%'' is used. A Linux headless crawl presents as macOS Chrome 135 regardless of the Chromium actually running, and the string drifts as the pin moves. Default viewport is 1440×812; mobile is 412×691 at device pixel ratio 2. |
| * **Detection is not action.** In a real ''optOut'' run against a Sourcepoint site, ''cmps[0]'' came back ''name: "Sourcepoint-frame"'' with ''open'', ''started'' and ''succeeded'' all ''false'' — and three cookies were set anyway. "We opted out with TRC's autoconsent" is not a measurable statement; the per-site ''succeeded'' rate is. | * **Detection is not action.** In a real ''optOut'' run against a Sourcepoint site, ''cmps[0]'' came back ''name: "Sourcepoint-frame"'' with ''open'', ''started'' and ''succeeded'' all ''false'' — and three cookies were set anyway. "We opted out with TRC's autoconsent" is not a measurable statement; the per-site ''succeeded'' rate is. |
| * **''filterListMatched'' is always ''false''.** autoconsent removed EasyList and filter-list support in v16.0.0 (2026-06-23) and TRC pins ''^16.8.1''; in TRC the backing field is initialised ''false'' and never assigned. A reader who filters on it concludes no CMP is ever on a filter list. | * **''filterListMatched'' is always ''false''.** autoconsent removed EasyList and filter-list support in v16.0.0 (2026-06-23) and TRC pins ''^16.8.1''; in TRC the backing field is initialised ''false'' and never assigned. A reader who filters on it concludes no CMP is ever on a filter list. |
| * **''--only-3p'' is an eTLD+1 filter applied while collecting.** It is not an entity or ownership decision, and it is irreversible: if you may later want to study CNAME cloaking or first-party cookies, do not enable it. | * **''%%--only-3p%%'' is an eTLD+1 filter applied while collecting.** It is not an entity or ownership decision, and it is irreversible: if you may later want to study CNAME cloaking or first-party cookies, do not enable it. |
| * **The CLI's success banner lies.** It prints "✅ Finished successfully." after a run with zero successful crawls. Read ''stats'' in ''metadata.json''. | * **The CLI's success banner lies.** It prints "✅ Finished successfully." after a run with zero successful crawls. Read ''stats'' in ''metadata.json''. |
| | * **APICallCollector misses calls made early in page load, and the bug is still open.** ''tracker-radar-collector'' issue #77, //"Early browser API accesses and function calls are missed"//, was opened on 2022-09-26 and is **still open**: the maintainers describe "a race condition between API collection script setting things up and scripts on the page already running", note that it makes TRC's own API-collection integration test flaky, and last said "We're exploring different options to fix this" on 2023-03-06.((''github.com/duckduckgo/tracker-radar-collector/issues/77'', fetched through the GitHub API on 2026-09-03: ''"state": "open"'', ''"created_at": "2022-09-26"'', ''"updated_at": "2023-03-06"'', four comments, ''"closed_at": null''.)) Two of the ten corpus papers cite it, and {[senol2024_double]} wrote a replacement collector because of it: //"Although TRC already has the capability to intercept JavaScript API calls, we introduced a separate collector due to a known TRC bug that causes it to miss the initial function calls."// An empty ''callStats'' for a script that runs at the top of the document is not evidence that it touched no API. |
| * **''size'' is not the size of anything you can sum twice.** It is CDP's ''encodedDataLength'' from ''Network.loadingFinished'' — bytes on the wire, including headers — so it moves between runs for a byte-identical body. Two runs of the published script above returned ''473'' and ''474'' for ''example.com'' with the same ''responseBodyHash''. It is also coerced to ''null'' when CDP reports a negative value, with the comment "make sure we can use unsigned int for this field in clickhouse". Use ''responseBodyHash'' for body identity and treat ''size'' as an estimate. | * **''size'' is not the size of anything you can sum twice.** It is CDP's ''encodedDataLength'' from ''Network.loadingFinished'' — bytes on the wire, including headers — so it moves between runs for a byte-identical body. Two runs of the published script above returned ''473'' and ''474'' for ''example.com'' with the same ''responseBodyHash''. It is also coerced to ''null'' when CDP reports a negative value, with the comment "make sure we can use unsigned int for this field in clickhouse". Use ''responseBodyHash'' for body identity and treat ''size'' as an estimate. |
| * **''cookiepopups'' archives page content.** ''scrapedFrames[].buttons'' captures the text of every clickable element, which on a news site means headlines. Include it in your redaction plan alongside ''hostname'' and ''username''. | * **''cookiepopups'' archives page content.** ''scrapedFrames[].buttons'' captures the text of every clickable element, which on a news site means headlines. Include it in your redaction plan alongside ''hostname'' and ''username''. |
| Tracker Radar describes itself as a dataset of common third-party domains and their observed behaviour, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking. | Tracker Radar describes itself as a dataset of common third-party domains and their observed behaviour, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking. |
| |
| The two first-party decisions must not be collapsed. TRC's ''--only-3p'' uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]} | The two first-party decisions must not be collapsed. TRC's ''%%--only-3p%%'' uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]} |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
| |
| The all-paper population in this corpus is **5,859 papers** from seven venues. A broad ''tracker.?radar'' name match hits **21** papers — but 11 of those name the **dataset**, the entity list, the entity map or the wiki, filed by the extraction as a ''classification-service'', a ''cookie-database'' or ''other'', and none of them ran the Collector. The adoption count therefore uses the narrower query: a tool name matching the Collector, an automation category, and a used/produced status. That gives **10 papers in the 1,120-paper CRAWLED population**. | The all-paper population in this corpus is **5,859 papers** from seven venues. A broad ''tracker.?radar'' name match hits **21** papers — but 11 of those name the **dataset**, the entity list, the entity map or the wiki, filed by the extraction as a ''classification-service'', a ''cookie-database'' or ''other'', and none of them ran the Collector. The adoption count therefore uses the narrower query: a tool name matching the Collector, an automation category, and a used/produced status. That gives **10 papers in the 1,120-paper CRAWLED population**. |
| | |
| | One more boundary inside that 10, because this page is about boundaries: **nine drove a TRC fork; the tenth did not.** {[aziz2024_johnny]} built its crawler itself — //"We used custom scripts, written in Python and JavaScript, to drive and instrument an instance of Chrome using the Chrome DevTools Protocol"// — and reused TRC's cookie-method breakpoint code inside it. It is the one ''browser-automation'' tuple in the category audit and it counts under the written rule, but "TRC users" in the table below means nine TRC-driven crawls plus one module reuse. Do not read the ''crawlConfig'' survey's silences as TRC's defaults for that row: it had every choice TRC does not give you. |
| |
| <WRAP important> | <WRAP important> |
| Read the last two rows carefully, because they are the opposite of an under-reporting finding. TRC's CLI gives you **no choice** about either: a fresh temporary profile per site is unconditional, and headless is a hard-coded constant. So "not stated" here mostly means "there was nothing to state" — unlike the same silence on a tool that does offer the option. What a TRC paper does need to state is whether it used the module API to share a browser connection across sites, because that is the only way to make a TRC crawl stateful. | Read the last two rows carefully, because they are the opposite of an under-reporting finding. TRC's CLI gives you **no choice** about either: a fresh temporary profile per site is unconditional, and headless is a hard-coded constant. So "not stated" here mostly means "there was nothing to state" — unlike the same silence on a tool that does offer the option. What a TRC paper does need to state is whether it used the module API to share a browser connection across sites, because that is the only way to make a TRC crawl stateful. |
| |
| The ''interactionDepth'' row is the substantive one: **7 of 10** went beyond the homepage, with sub-page budgets from 3 to 20. TRC has no inner-page traversal at all — no link extraction, no depth parameter, and no mention of either in its README — so it crawls exactly the URLs you hand it, one page each. Each of those seven therefore either wrote link-following on top of TRC or supplied a pre-built inner-page URL list; {[senol2022_leaky]} did both, matching Tranco domains against Chrome User Experience Report URLs //and// clicking up to ten ranked links per site. If your design needs inner pages, budget for building that part. | The ''interactionDepth'' row is the substantive one: **7 of 10** went beyond the homepage, with sub-page budgets from 3 to 20. TRC has no inner-page traversal at all — no link extraction, no depth parameter, and no mention of either in its README — so it crawls exactly the URLs you hand it, one page each. That work is yours, and the seven did it four different ways: |
| | |
| | * **Four added link discovery to their fork.** {[senol2022_leaky]}: "We modify TRC to efficiently discover inner pages", following links matched by a regular expression lifted from Firefox's Password Manager. {[moti2024_targeted]} and {[zagi2025_referrer]} each wrote a ''LinkCollector''. {[senol2024_double]} finds candidate login and sign-up links on the homepage and visits up to fifteen on a second pass. |
| | * **One selected inner pages into its site list** instead of traversing: {[roongta2025_sheep]} added three inner pages for any site with more than ten, reaching "4,710 web pages" from 1,500 sites. |
| | * **One does not say how its pages were found** — {[dimova2021cname]} reports a "preliminary crawl of 20 pages per website" without describing the discovery. |
| | * **One is not a TRC crawl at all**: {[aziz2024_johnny]}'s traversal is in its own crawler. |
| | |
| | If your design needs inner pages, budget for building that part — and say which of these four things you did, because they are not the same measurement. |
| |
| <WRAP important> | <WRAP important> |
| Before collecting, decide what the output is supposed to answer. Then record at least: | Before collecting, decide what the output is supposed to answer. Then record at least: |
| |
| * the TRC repository **commit** — not ''projectVersion'', which is always ''1.0.0'' — plus the package-lock state, Node.js, Puppeteer and Chromium versions, or the explicit ''--chromium-version''; | * the TRC repository **commit** — not ''projectVersion'', which is always ''1.0.0'' — plus the package-lock state, Node.js, Puppeteer and Chromium versions, or the explicit ''%%--chromium-version%%''; |
| * the site list, its source, version and date, the URL selection, and whether the crawl visits only a homepage or follows inner links, with the sub-page budget; | * the site list, its source, version and date, the URL selection, and whether the crawl visits only a homepage or follows inner links, with the sub-page budget; |
| * crawl dates, region/vantage/IP conditions, proxy, success and timeout policy, concurrency, the real ''maxLoadTimeMs'' you ran with, and the post-load wait; | * crawl dates, region/vantage/IP conditions, proxy, success and timeout policy, concurrency, the real ''maxLoadTimeMs'' you ran with, and the post-load wait; |
| * the denominator for every prevalence number: attempted sites, successfully crawled sites, sites where the relevant interaction worked, or sites retained by Detector. | * the denominator for every prevalence number: attempted sites, successfully crawled sites, sites where the relevant interaction worked, or sites retained by Detector. |
| |
| Anti-bot measures and consent automation are interventions, not harmless crawler options: they change the page, the requests and the cookies. A result from ''--only-3p'' is conditioned on an eTLD+1 filter and is not a result about ownership, entity prevalence or CNAME cloaking. And do not report a "number of trackers" without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records — those are six different units. | Anti-bot measures and consent automation are interventions, not harmless crawler options: they change the page, the requests and the cookies. A result from ''%%--only-3p%%'' is conditioned on an eTLD+1 filter and is not a result about ownership, entity prevalence or CNAME cloaking. And do not report a "number of trackers" without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records — those are six different units. |
| |
| ===== Methodology and limitations of these figures ===== | ===== Methodology and limitations of these figures ===== |
| The 10 evidence quotes behind the adoption count were checked against the papers' repaired full text: **1 verbatim, 1 exact except capitalisation, 8 partial** — a partial match being a 5-to-12-word run that contains the phrase "tracker radar collector" itself, split by the column repair. None was unlocatable. Three of the quotes drop a leading clause and capitalise a mid-sentence "we", so they are edited fragments rather than verbatim sentences; the surrounding sentence supports the claim in every case. | The 10 evidence quotes behind the adoption count were checked against the papers' repaired full text: **1 verbatim, 1 exact except capitalisation, 8 partial** — a partial match being a 5-to-12-word run that contains the phrase "tracker radar collector" itself, split by the column repair. None was unlocatable. Three of the quotes drop a leading clause and capitalise a mid-sentence "we", so they are edited fragments rather than verbatim sentences; the surrounding sentence supports the claim in every case. |
| |
| The reproducible reports are ''report_tracker_radar_collector.mjs'', ''trc_review_probe.mjs'', ''trc_index_reconcile.mjs'' and ''trc_smoke.mjs''. The complete query log, the unedited script output, the source-quote checks, the external-source verification, the review log and the judgement calls are on [[:provenance:programming:crawler:tracker_radar_collector|the page provenance record]]. Dataset-wide extraction and retrieval caveats are on [[:literature:corpus|the corpus provenance page]]. | The reproducible reports are ''report_tracker_radar_collector.mjs'', ''trc_review_probe.mjs'', ''trc_index_reconcile.mjs'', ''trc_paper_detail.mjs'' and ''trc_smoke.mjs''. The complete query log, the unedited script output, the source-quote checks, the external-source verification, the review log and the judgement calls are on [[:provenance:programming:crawler:tracker_radar_collector|the page provenance record]]. Dataset-wide extraction and retrieval caveats are on [[:literature:corpus|the corpus provenance page]]. |
| |
| This page does not claim that TRC is faster, more complete or more accurate than OpenWPM, nor that its 10-paper count is a market-share estimate. The repositories were checked for maintenance and schema, and one real crawl was run; the corpus was used to find research use and concrete extensions. A live repository is evidence that TRC remains available, not evidence that every deployed Tracker Radar crawl uses the public defaults. | This page does not claim that TRC is faster, more complete or more accurate than OpenWPM, nor that its 10-paper count is a market-share estimate. The repositories were checked for maintenance and schema, and one real crawl was run; the corpus was used to find research use and concrete extensions. A live repository is evidence that TRC remains available, not evidence that every deployed Tracker Radar crawl uses the public defaults. |
