Differences

This shows you the differences between two versions of the page.

--- programming:crawler:tracker_radar_collector [2026/08/14 14:14] – Clarify Tracker Radar output schema — Authored by Claude karel.kubicek.claude
+++ programming:crawler:tracker_radar_collector [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
@@ Line 4: / Line 4: @@
 <WRAP important>
-Do not use **Tracker Radar**, **Tracker Radar Wiki**, **Tracker Radar Entity Map** or **Tracker Radar Detector** as synonyms for the Collector. The Collector produces per-site crawl records; Detector processes those records into the domain and entity data shipped by Tracker Radar. The corpus has all of these names, and counting them together makes a dataset or a classifier look like a crawler.
+Do not use **Tracker Radar**, **Tracker Radar Wiki**, **Tracker Radar Entity Map** or **Tracker Radar Detector** as synonyms for the Collector. The Collector produces per-site crawl records; Detector processes those records into the domain and entity data shipped by Tracker Radar. The corpus has all of these names, and counting them together makes a dataset or a classifier look like a crawler — in this corpus that mistake takes the count from 10 papers to 21. See [[#Use in Publications|Use in Publications]].
 </WRAP>
 ===== What it is, and what is current =====
-TRC first appears in this corpus in 2021. The public repository is still active: at the 2026-08-14 check, its main branch was at commit ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'', declared Node.js >= 18 and Puppeteer ''^24.7.1''. That makes it a current, available Chromium instrument when those versions and that commit are recorded. It is not a published measurement standard, and the corpus contains no matched-sample study showing that TRC supersedes OpenWPM or a plain Playwright crawl. {[duckduckgo_trc_2026]}
+TRC first appears in this corpus in 2021, and the repository is live. Checked **2026-09-03**: ''main'' is at ''4ad2472b346bf9b544ca4815db29266e3a8369d5'' (committed 2026-08-24), which differs from the commit this page originally pinned, ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'' (2026-07-22), only in a CI workflow file and one deleted test line. Either commit is a fair "current" pin. It declares Node.js ''>= 18'' and Puppeteer ''^24.7.1''. {[duckduckgo_trc_2026]}
-The literature signal is useful for locating examples, not for estimating field share. In the **CRAWLED population of 1,120 papers**, 10 papers have a used/produced automation tuple naming the Collector. The report finds 1 of 75 crawling papers in 2021, 1 of 110 in 2022, 1 of 125 in 2023, 3 of 110 in 2024, 3 of 129 in 2025, and 1 of 69 in 2026. The 2021--2024 series is complete for the selected venue-years; 2025--2026 are provisional. See [[#What the corpus shows|the paper-level examples]] and [[#Methodology and limitations of these figures|the query definition]].
+Two currency caveats the version strings hide:
-Use [[Programming:Crawler:OpenWPM|OpenWPM]] when its Firefox/WebExtension instrumentation is the surface you need, and use this page when you need to reason about a Chromium/CDP crawl or the Tracker Radar data pipeline. [[Programming:Crawler|Comparison of Crawling Libraries]] is the broader choice page.
+  * **''engines.node'' is a dead runtime.** Node 18 reached end of life on 2025-04-30 and Node 20 on 2026-04-30. Node 22 has been in maintenance since 2025-10-21, Node 24 is the active LTS, and Node 26 is Current.((''nodejs/Release'' ''schedule.json'', [[https://raw.githubusercontent.com/nodejs/Release/main/schedule.json|raw.githubusercontent.com/nodejs/Release/main/schedule.json]], and the status table on [[https://nodejs.org/en/about/previous-releases|nodejs.org/en/about/previous-releases]], both fetched 2026-09-03: ''v18'' ''end 2025-04-30'', ''v20'' ''end 2026-04-30'', ''v22'' ''maintenance 2025-10-21'', ''v24'' ''maintenance 2026-10-20'', ''v26'' ''start 2026-05-05''; the page's own status field reads EOL for 18 and 20, LTS for 22 and 24, Current for 26.)) ''>= 18'' is satisfied by a supported Node, so this is not an obstacle — but do not read it as evidence that the project tracks its runtime. Record the Node you actually used.
+  * **The three repositories are not equally alive.** The Collector and the Tracker Radar dataset are actively developed; [[https://github.com/duckduckgo/tracker-radar-detector|tracker-radar-detector]] has had no commit since **2025-03-31**, roughly seventeen months. It is not archived and carries no deprecation notice, and DuckDuckGo has said nothing public about it either way. If your pipeline depends on Detector, that dormancy is your problem to plan around. {[duckduckgo_tracker_radar_detector_2026]}
+TRC is a current, available Chromium instrument when the commit and those versions are recorded. It is not a published measurement standard: it has no academic paper of its own, and no paper in this corpus benchmarks it against another instrument on the same sample.
+The literature signal is useful for locating examples, not for estimating field share. In the **CRAWLED population of 1,120 papers**, 10 papers have a used/produced automation tuple naming the Collector: 1 of 75 crawling papers in 2021, 1 of 110 in 2022, 1 of 125 in 2023, 3 of 110 in 2024, 3 of 129 in 2025 and 1 of 69 in 2026. The 2021--2024 series is complete for the selected venue-years; 2025--2026 are provisional. See [[#Use in Publications|Use in Publications]] for the paper-level examples and [[#Methodology and limitations of these figures|Methodology and limitations of these figures]] for the query definition.
+Use [[:programming:crawler:openwpm|OpenWPM]] when its Firefox/WebExtension instrumentation is the surface you need, and use this page when you need to reason about a Chromium/CDP crawl or the Tracker Radar data pipeline. [[:programming:crawler|Comparison of Crawling Libraries]] is the broader choice page.
+===== Installing it, and what we ran =====
+Installing is the easy part. Getting a browser is not, and the CLI is less configurable than the module.
+<code bash>
+git clone https://github.com/duckduckgo/tracker-radar-collector.git
+cd tracker-radar-collector
+git checkout 8b64006691a1ce3929cfdfeeb425e7cc64be6543
+npm i                       # 445 packages, ~5 s, no build step
+node ./cli/crawl-cli --help
+</code>
+The real ''%%--help%%'' at that commit, unedited. This is the authoritative flag list; the README's prose is not (see [[#Known pitfalls|Known pitfalls]]).
+<file text trc-crawl-cli-help.txt>
+Usage: crawl-cli [options]
+Options:
+  -o, --output <path>                  output folder
+  -u, --url <url>                      single URL
+  -i, --input-list <path>              path to list of URLs
+  -d, --data-collectors <list>         comma separated list of data collectors:
+                                       requests, apis, cookies, targets, trace,
+                                       screenshots, cookiepopups (all by
+                                       default)
+  --reporters <list>                   comma separated list of reporters: cli,
+                                       file, html, clickhouse
+  -l, --log-path <path>                instructs reporters where all logs should
+                                       be written to
+  -v, --verbose                        instructs reporters to log additional
+                                       information (e.g. for "cli" reporter
+                                       progress bar will not be shown when
+                                       verbose logging is enabled)
+  -c, --crawlers <number>              overwrite the default number of concurent
+                                       crawlers
+  -f, --force-overwrite                overwrite existing output files
+  -3, --only-3p                        don't save any first-party data
+  -m, --mobile                         emulate a mobile device
+  -p, --proxy-config <host>            use an optional proxy configuration
+  -r, --region-code <region>           optional 2 letter region code. Used for
+                                       metadata only.
+  -a, --disable-anti-bot               disable anti bot detection protections
+                                       injected to every frame
+  --config <path>                      crawl configuration file
+  --autoconsent-action <action>        dismiss cookie popups. Possible values:
+                                       optOut, optIn. Works only when
+                                       cookiepopups collector is enabled.
+  --chromium-version <version_number>  use custom version of chromium
+  --selenium-hub <url>                 selenium hub endpoint to request browsers
+                                       from
+  -h, --help                           display help for command
+</file>
+**The CLI cannot be pointed at a browser you already have.** ''browser/openBrowser.js'' takes ''executablePath || await downloadChrome(log)'', and ''cli/crawl-cli.js'' only ever sets ''executablePath'' from ''%%--chromium-version%%''; ''crawlerConductor'' calls ''downloadChrome()'' unconditionally unless ''%%--selenium-hub%%'' is given. There is no ''%%--executable-path%%'' and ''PUPPETEER_EXECUTABLE_PATH'' is ignored. On a host where Chrome for Testing has no build the CLI therefore cannot start at all — on ''aarch64'' it downloads the x86-64 archive and dies:
+<code>
+Error: Failed to launch the browser process!
+rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
+Max number of retries (2) exceeded for "https://example.com".
+✅ Finished successfully.
+...
+Sucessful crawls: 0/1 (0.00%)
+</code>
+Note the banner and the tally: the CLI prints "✅ Finished successfully." on a crawl that collected nothing. The elided lines are the start, finish and duration that ''CLIReporter'' prints between them. **The exit banner is not a success signal — read ''stats'' in ''metadata.json''.**
+The single-crawl module entry point does accept ''executablePath'', which is the way round it. ''TRC_UNSAFE_DISABLE_SANDBOX=1'' is TRC's own environment variable, read in ''browser/openBrowser.js'', and is needed in a container without user namespaces.
+<file javascript trc_smoke.mjs>
+// A real Tracker Radar Collector run, so that the output schema published on
+// this page is checked against actual output and not only against the source.
+//
+//   TRC_UNSAFE_DISABLE_SANDBOX=1 \
+//   PUPPETEER_EXECUTABLE_PATH=/path/to/an/aarch64/chrome \
+//   node trc_smoke.mjs /path/to/tracker-radar-collector out/trc_smoke
+//
+// Why not `npm run crawl`:
+//
+//   1. The CLI has no --executable-path. browser/openBrowser.js takes
+//      `executablePath || await downloadChrome(log)`, and cli/crawl-cli.js only
+//      ever sets it from --chromium-version. crawlerConductor() has the same
+//      problem: it calls downloadChrome() unconditionally unless
+//      --selenium-hub is set. The single-crawl `crawler(url, options)` entry
+//      point does accept executablePath, so that is what this uses.
+//   2. Chrome for Testing publishes no linux-arm64 build. On aarch64 the
+//      download resolves to the x86-64 archive and the launch dies with
+//      "rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2".
+import { createRequire } from 'node:module';
+import fs from 'node:fs';
+import path from 'node:path';
+const [checkout, outDir] = process.argv.slice(2);
+if (!checkout || !outDir) throw new Error('usage: trc_smoke.mjs <trc-checkout> <out-dir>');
+const require = createRequire(path.join(path.resolve(checkout), 'package.json'));
+const crawler = require('./crawler');
+const RequestCollector = require('./collectors/RequestCollector');
+const APICallCollector = require('./collectors/APICallCollector');
+const CookieCollector = require('./collectors/CookieCollector');
+const TargetCollector = require('./collectors/TargetCollector');
+const exe = process.env.PUPPETEER_EXECUTABLE_PATH;
+if (!exe) throw new Error('set PUPPETEER_EXECUTABLE_PATH to a Chromium built for this architecture');
+if (!fs.existsSync(exe)) throw new Error(`no such browser: ${exe}`);
+fs.mkdirSync(outDir, { recursive: true });
+// example.com sets no cookies and makes one request; a real site is needed for
+// the cookie and API keys to be non-empty, so both are crawled.
+const urls = ['https://example.com/', 'https://www.wikipedia.org/'];
+for (const u of urls) {
+  const collectors = [
+    new RequestCollector({ saveResponseHash: true }),
+    new APICallCollector(),
+    new CookieCollector(),
+    new TargetCollector(),
+  ];
+  console.error(`[trc] crawling ${u}`);
+  const data = await crawler(new URL(u), {
+    collectors,
+    log: (...a) => console.error('   [log]', ...a),
+    executablePath: exe,
+    filterOutFirstParty: false,
+    emulateMobile: false,
+    maxLoadTimeMs: 30000,
+    extraExecutionTimeMs: 2500,
+    collectorFlags: {},
+  });
+  // A timed-out crawl still writes a file, and the file still says so. Crash
+  // rather than print "wrote" over a partial capture: this page's own lesson is
+  // that a success message is not a success signal.
+  if (data.timeout) throw new Error(`${u} hit the crawler's timeout path; capture is partial`);
+  const name = u.replace(/[^a-z0-9]+/gi, '_') + '.json';
+  fs.writeFileSync(path.join(outDir, name), JSON.stringify(data, null, 2));
+  console.error(`[trc] wrote ${path.join(outDir, name)}`);
+}
+</file>
+Its whole output for ''https://example.com'', which is what the [[#Output schema|Output schema]] section below was checked against. Four fields in it are not reproducible run to run: the two timestamps, ''time'' (a measured duration), ''remoteIPAddress'' (anycast) and ''size'' — a second run of the same script against the same URL gave ''size: 474'' with an identical ''responseBodyHash'', for the reason in the pitfalls below. Everything else is stable.
+<code javascript>
+{
+  "initialUrl": "https://example.com/",
+  "finalUrl": "https://example.com/",
+  "timeout": false,
+  "testStarted": 1788423771768,
+  "testFinished": 1788423775695,
+  "data": {
+    "requests": [
+      {
+        "url": "https://example.com/",
+        "method": "GET",
+        "type": "Document",
+        "status": 200,
+        "size": 473,
+        "remoteIPAddress": "104.20.23.154",
+        "responseHeaders": {},
+        "responseBodyHash": "ff67a9d764d6a2367a187734e697f6a53217db9a21c101d410a113ca871a299d",
+        "initiators": [],
+        "time": 0.0506589999422431
+      }
+    ],
+    "apis": { "callStats": {}, "savedCalls": [] },
+    "cookies": [],
+    "targets": [ { "type": "page", "url": "about:blank" } ]
+  }
+}
+</code>
 ===== What it captures relative to OpenWPM =====
-The useful comparison is the capture surface, not the library name. A TRC crawl is Chromium-centred and CDP-native; OpenWPM is an unbranded Firefox platform whose privileged WebExtension records browser events. The same paper-level result can therefore differ because the browser, instrumentation point, state, or consent action differed.
+The useful comparison is the capture surface, not the library name. A TRC crawl is Chromium-centred and CDP-native; OpenWPM {[englehardt2016online]} is an unbranded Firefox platform whose privileged WebExtension records browser events. The same paper-level result can therefore differ because the browser, instrumentation point, state, or consent action differed.
 ^ Surface ^ Tracker Radar Collector ^ OpenWPM ^ Measurement consequence ^
-| Browser and control | Chromium through Puppeteer and CDP; current TRC attaches to pages, frames and worker/service-worker targets | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. |
+| Browser and control | Chromium through Puppeteer and CDP; attaches to page, iframe, worker, shared-worker and service-worker targets. ''%%--selenium-hub%%'' can source a remote **Chrome** from a Selenium Grid, but CDP is still the instrument — Selenium here only provisions the browser | Firefox driven through Selenium, with a privileged WebExtension | The engine, browser defaults and automation fingerprint are not held constant. A cross-tool comparison is also a cross-browser comparison unless you design it otherwise. |
-| Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, selected response headers, timing, redirect links, initiators and optionally a response-body hash. WebSocket events are represented by the request collector. The current default record does not contain raw response bodies, request headers or request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and deduplication hash. If the question is about sent headers, POST contents, or response bytes, inspect the selected TRC fork/collector or use an instrument that records them; do not infer their presence from a URL log. |
+| Requests and responses | The RequestCollector records URL, method, resource type, status, size, remote IP, a nine-header safelist of response headers, timing, failure and redirect fields, initiators and optionally a response-body hash. WebSocket events go into the same array through ''handleWebSocket'' with fewer fields. The default record contains no raw response bodies, no request headers and no request bodies. | ''http_instrument'' records request/response headers, redirects, POST bodies, resource type, third-party flags and triggering/loading origin; ''save_content'' stores selected response bodies | TRC is good for a compact per-request trace and a deduplication hash. If the question is about sent headers, POST contents, or response bytes, extend the collector or use an instrument that records them; do not infer their presence from a URL log. |
-| Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalized record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it omits the cookie value, ''HttpOnly'', ''Secure'' and size | ''cookie_instrument'' records cookie changes from JavaScript and HTTP responses in ''javascript_cookies'' | TRC's default cookie output is a final-state snapshot, not a set-event history and not a value-based identifier dataset. For creation timing, setter provenance or values, add instrumentation and state exactly what it records. |
+| Cookies | CookieCollector queries the final browser cookie jar through CDP. Its normalised record contains ''name'', ''domain'', ''path'', ''expires'', ''session'' and ''sameSite''; it drops the CDP cookie's ''value'', ''httpOnly'', ''secure'' and ''size''. But ''set-cookie'' **is** in the request collector's default header safelist, so HTTP-set values are still in the crawl | ''cookie_instrument'' records every cookie change — from JavaScript **and** from HTTP responses — in the ''javascript_cookies'' table | TRC's cookie key is a final-state snapshot, not a set-event history. For HTTP-set cookies, read ''requests[].responseHeaders["set-cookie"]'' instead. Only ''document.cookie'' writes need instrumentation you add yourself. |
-| JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints/TrackerTracker and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither “JavaScript captured” nor “API calls captured” is a sufficient methods description. Publish the collector configuration and API list. |
+| JavaScript and browser APIs | APICallCollector produces per-source ''callStats'' and optional ''savedCalls'' with the source, description and arguments. It uses CDP conditional breakpoints and only observes APIs configured for collection | ''js_instrument'' uses Firefox-side instrumentation for configured objects and properties; the default fingerprinting collection is a different surface | Neither "JavaScript captured" nor "API calls captured" is a sufficient methods description. The one published comparison of these two surfaces found the gap was **configuration, not architecture** — see below. Publish the collector configuration and the API list. |
-| Targets, frames and navigation | TargetCollector records target URL/type; the current collector attaches to page, iframe, worker and service-worker targets | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity. |
+| Targets, frames and navigation | TargetCollector records target type and URL, captured at **attach** time | OpenWPM's navigation and tab/window records are WebExtension events | A frame or worker can be the source of a request without being the top-level page. Report whether the analysis retained target/frame identity — and see the pitfall about what ''targets'' actually contains. |
-| Consent and interaction | CookiePopupsCollector can record Consent-O-Matic/autoconsent CMP observations; the CLI has ''optIn''/''optOut'' actions. The conductor also supports mobile emulation, scrolling/interaction in forks, and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | “Before consent” is not a browser default. Record the action, rule/version and wait/interaction sequence. |
+| Consent and interaction | CookiePopupsCollector records autoconsent CMP observations; the ''%%--autoconsent-action%%'' flag takes ''optIn'' or ''optOut''. The conductor also supports mobile emulation, and scrolling/interaction and inner-page traversal in forks | Consent handling is implemented by command sequences and browser/profile configuration | "Before consent" is not a browser default. Record the action, the autoconsent version and the wait/interaction sequence — **and the per-site success rate**, because TRC records CMPs it did not act on. |
-| State and filtering | The CLI can use a fresh context, mobile emulation, a region code, a proxy and ''--only-3p''. The latter filters by eTLD+1 in the crawler | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organization-level first parties are handled later by Detector, with separate configuration. |
+| State and filtering | The CLI can use mobile emulation, a region code, a proxy and ''%%--only-3p%%'', which filters by eTLD+1 during collection. A fresh temporary profile per site is unconditional and cannot be switched off from the CLI | OpenWPM exposes stateful/stateless profiles, third-party-cookie policy and command sequences | TRC's first-party filter is not an entity or ownership judgment. CNAMEs and organisation-level first parties are handled later by Detector, with separate configuration. TRC cannot do a stateful crawl from the CLI at all. |
+TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. A final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments, and the corpus shows many such forks.
+==== What the two comparisons in the corpus actually found ====
+No paper in these seven venues runs TRC against OpenWPM, or against a plain Playwright crawl, on the same sample — so nothing here ranks the instruments. Two papers do compare crawler architectures empirically on one sample, and both are worth reading before you assume a difference you measure is architectural:
+  * {[ahmad2020_apophanies]} ran **eight** crawlers — from ''wget'' and ''curl'' through PhantomJS, Selenium and Puppeteer to OpenWPM and the Tor Browser Crawler — over a list of "932 unique domains" drawn from the Alexa and Umbrella top 500s, and reported that "the choice of crawler has a critical impact on the data generated". It is the closest thing the field has to a crawler-choice benchmark, and its crawls (April 2019) predate TRC's first commit (2020-02-27).
+  * {[sanchezrola2023_rods]} compared its own CDP-instrumented Chromium crawler with OpenWPM on the same random sample of 1,000 phishing sites: //"we found 728 websites calling fingerprint-related APIs. Our crawler was able to detect and log all invocations, while OpenWPM missed calls in 348 (47.8%) of the websites."// The headline number is not the finding. The paper's own explanation is that //"OpenWPM monitors a smaller set of fingerprinting APIs"// — 36 of the 48 API types seen — and after instrumenting the missing ones, //"both our crawler and OpenWPM were able to detect all the 2092 calls performed on the test set"//. The 47.8% gap was a default API list, not a capability of CDP over Firefox instrumentation.
+That is the single most useful thing to take from this section: **a difference between two crawlers is a difference between two configurations until you have shown otherwise.**
-TRC therefore overlaps OpenWPM on request, cookie and JavaScript observations but does not emit the same records. In particular, a final cookie jar cannot answer the same question as OpenWPM's cookie-change table, and a response-body hash cannot substitute for a retained body. Conversely, CDP breakpoints and target attachment make TRC a convenient base for narrowly targeted Chromium experiments. The corpus shows many such forks, but no same-sites, same-browser, same-question head-to-head benchmark. {[englehardt2016online]}
+Playwright is the comparator this table does not have. TRC is a Puppeteer program, and on the [[:programming:crawler|crawler comparison page]] Playwright now overtakes Puppeteer in the provisional 2025--2026 slice. Nothing in this corpus compares TRC with a plain Playwright crawl, so no claim is made either way.
 ===== Output schema =====
-The crawler writes one JSON result per site. At the top level the current result object is:
+The crawler writes one JSON result per site. The top-level result object is exactly six keys, confirmed against the real output above:
 ^ Field ^ Meaning ^
 | ''initialUrl'' | URL supplied to the crawl |
 | ''finalUrl'' | URL reached after navigation/redirects |
 | ''timeout'' | Whether the crawl hit its timeout path |
-| ''testStarted'', ''testFinished'' | Crawl timestamps |
+| ''testStarted'', ''testFinished'' | Crawl timestamps, milliseconds since epoch |
 | ''data'' | Object keyed by the selected collectors |
-The exact contents of ''data'' depend on ''--data-collectors''. The current collector modules expose these principal shapes:
+The contents of ''data'' depend on ''%%--data-collectors%%''. Seven collectors are selectable. ''requests'', ''cookies'', ''apis'', ''targets'' and ''cookiepopups'' below were checked against real output; ''screenshots'' and ''trace'' against source only.
 ^ ''data'' key ^ What to expect ^ What to retain or report ^
-| ''requests'' | An array of request/response observations: URL, method, type, status, size, remote IP, selected response headers, response-body hash when enabled, failure/redirect fields, initiators and time | Whether first-party traffic was filtered; whether hashes, headers, WebSockets and initiators were kept; and whether a fork added request headers, POST bodies or response bodies |
+| ''requests'' | An array of request/response observations. Real keys: ''url'', ''method'', ''type'', ''status'', ''size'', ''remoteIPAddress'', ''responseHeaders'', ''responseBodyHash'', ''initiators'', ''time'', plus ''failureReason'', ''redirectedTo'' and ''redirectedFrom'' where they apply. ''size'' is CDP's ''encodedDataLength'' — bytes on the wire, headers included — and is coerced to ''null'' when negative | Whether first-party traffic was filtered; whether hashes, WebSockets and initiators were kept; whether a fork added request headers, POST bodies or response bodies; and that ''size'' is not a body length |
-| ''cookies'' | An array of final cookie-jar entries with name, domain, path, expiry, session and same-site status | The fact that values and set events are absent by default; profile reset and crawl-end timing |
+| ''cookies'' | An array of final cookie-jar entries: ''name'', ''domain'', ''path'', ''expires'', ''session'', ''sameSite''. ''expires'' is **milliseconds** (''Math.floor(cdpDate * 1000)''), and a session cookie's ''expires'' becomes ''undefined'', so the key is **absent** from the JSON — as is ''sameSite'' when unset | The fact that values and set events are absent from this key; that a missing key is not a null; profile reset and crawl-end timing |
-| ''apis'' | ''callStats'' grouped by source URL and API description, plus ''savedCalls'' when configured breakpoints save arguments | The API/breakpoint configuration, because an empty or partial list is a result of configuration rather than proof of no API use |
+| ''apis'' | ''callStats'' grouped by source URL and API description, plus ''savedCalls'' when configured breakpoints save arguments | The API/breakpoint configuration //and// how you handled issue #77, because an empty or partial list can be either configuration or the early-call race described in the pitfalls — neither is proof of no API use |
-| ''targets'' | Target URL and target type | Whether iframes, workers and service workers were included in the analysis |
+| ''targets'' | Target type and URL, captured at attach time. On a plain crawl the top-level entry is ''{"type":"page","url":"about:blank"}'' — the page target is attached before navigation | Whether iframes, workers and service workers were included in the analysis, and that the top-level row carries no URL |
-| ''elements'' | An optional field in the CollectorData type; the current collector entrypoint checked at the pinned commit does not export an ElementsCollector | Verify the pinned commit or fork before assuming this key is populated |
 | ''screenshots'' | Screenshot data in the in-process result; the CLI writes JPEG files and replaces the field with a path | Browser viewport/device mode, timing and whether screenshots were used as evidence |
-| ''cookiepopups'' | CMP detections, performance information and scraped frames from the consent collector | Consent action and autoconsent/rule version |
+| ''cookiepopups'' | ''{cmps, performance, scrapedFrames}''. ''cmps[]'' carries ''final'', ''name'', ''open'', ''started'', ''succeeded'', ''selfTestFail'', ''errors'', ''patterns'', ''snippets'' and ''filterListMatched''. ''scrapedFrames[]'' carries the text and CSS selector of **every** clickable element in the frame, not only the banner | The consent action and autoconsent version; the per-site ''succeeded'' rate; and that this key puts page content in your archive |
 | ''trace'' | Chrome trace data when the trace collector is selected | Trace configuration and whether it was sampled or retained |
-The CLI also writes a ''metadata.json'' file. The current schema records ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (URLs, skipped, successes and failures), ''config'' (crawler count, collector list, first-party filter, proxy, region and mobile emulation) and ''environment'' (project version, hostname, CPU count and username). Treat ''hostname'' and ''username'' as data to redact before sharing a crawl archive. The current repository README documents a default maximum page-load window of 30 seconds, with an additional short execution window; forks often change both. Record the actual values rather than inheriting the README default. {[duckduckgo_trc_2026]}
+The ''CollectorData'' typedef lives in ''helpers/collectorsList.js'', and it is not the authority on this list. It declares an ''elements'' key for which no ''ElementsCollector'' exists anywhere in the repository, and it omits ''trace'', which ''TraceCollector.id()'' returns and ''%%--help%%'' offers. Take the collector list from ''%%--help%%'' and the field list from the collector source.
-The schema is a versioned implementation detail, not a stable interchange standard. Pin the repository commit and inspect the collector source when a field is load-bearing. A “TRC JSON” without the commit, collector list and metadata is not enough to reproduce a measurement.
+The CLI also writes ''metadata.json'', with ''startTime'', ''endTime'', a ''result'' summary, ''stats'' (''urls'', ''skipped'', ''successes'', ''failures''), ''config'' and ''environment'' (''projectVersion'', ''hostname'', ''cpus'', ''username''). Two things about it:
+  * **''config'' omits every falsy setting.** Each field is written as ''value || undefined'', so ''filterOutFirstParty: false'' produces **no key at all** and the file cannot distinguish "we did not filter first-party traffic" from "this was not recorded". The same applies to ''emulateMobile'' and a zero crawler count. Your negative settings have to be published by you, not by the file.
+  * **''projectVersion'' carries no information.** It is ''package.json'''s ''version'', which has been ''1.0.0'' since the initial commit and has never changed. That is why pinning the commit is not a preference but the only way to identify the code.
+Treat ''hostname'' and ''username'' as data to redact before sharing a crawl archive. A "TRC JSON" without the commit, the collector list and the metadata is not enough to reproduce a measurement.
+===== Known pitfalls =====
+<WRAP todo>
+  * **The README is stale in at least three places, and one of them is a number people quote.** ''maxLoadTimeMs'' is documented as "defaults to 30s" in two example config blocks; the code fallback is ''crawler.js'': ''options.maxLoadTimeMs || 60000'' — **60 seconds**. The code changed in commit ''b7c35b4'' (2025-07-08); the README comments have not been touched since 2022-01-13. No CLI flag sets it, so an unconfigured CLI run gets 60 s. ''extraExecutionTimeMs'' (2.5 s) is documented correctly.
+  * **There is no "hard limit of 38" crawlers.** The README still says so; ''38'' appears nowhere in the code. The cap was removed in commit ''963f9ec'' (2025-04-16). The default is ''Math.floor(cores * 0.8)'', then capped at the number of URLs.
+  * **There is no ''browserContext'' option.** The README offers one; the code takes ''browserConnection''. Pass a connection if you want to share state, and note that this is a module-only route.
+  * **Headless is not a choice.** ''VISUAL_DEBUG'' is a hard-coded ''const … = false'' in ''constants.js'' with no flag and no environment override. To run headful you must edit that file — and if you do, ''crawler.js'' stops closing browsers.
+  * **The default user agent is a constant, not the browser.** ''DEFAULT_USER_AGENT'' is a fixed macOS ''Chrome/135.0.0.0'' string and ''MOBILE_USER_AGENT'' a fixed Android one; emulation is on unless ''%%--selenium-hub%%'' is used. A Linux headless crawl presents as macOS Chrome 135 regardless of the Chromium actually running, and the string drifts as the pin moves. Default viewport is 1440×812; mobile is 412×691 at device pixel ratio 2.
+  * **Detection is not action.** In a real ''optOut'' run against a Sourcepoint site, ''cmps[0]'' came back ''name: "Sourcepoint-frame"'' with ''open'', ''started'' and ''succeeded'' all ''false'' — and three cookies were set anyway. "We opted out with TRC's autoconsent" is not a measurable statement; the per-site ''succeeded'' rate is.
+  * **''filterListMatched'' is always ''false''.** autoconsent removed EasyList and filter-list support in v16.0.0 (2026-06-23) and TRC pins ''^16.8.1''; in TRC the backing field is initialised ''false'' and never assigned. A reader who filters on it concludes no CMP is ever on a filter list.
+  * **''%%--only-3p%%'' is an eTLD+1 filter applied while collecting.** It is not an entity or ownership decision, and it is irreversible: if you may later want to study CNAME cloaking or first-party cookies, do not enable it.
+  * **The CLI's success banner lies.** It prints "✅ Finished successfully." after a run with zero successful crawls. Read ''stats'' in ''metadata.json''.
+  * **APICallCollector misses calls made early in page load, and the bug is still open.** ''tracker-radar-collector'' issue #77, //"Early browser API accesses and function calls are missed"//, was opened on 2022-09-26 and is **still open**: the maintainers describe "a race condition between API collection script setting things up and scripts on the page already running", note that it makes TRC's own API-collection integration test flaky, and last said "We're exploring different options to fix this" on 2023-03-06.((''github.com/duckduckgo/tracker-radar-collector/issues/77'', fetched through the GitHub API on 2026-09-03: ''"state": "open"'', ''"created_at": "2022-09-26"'', ''"updated_at": "2023-03-06"'', four comments, ''"closed_at": null''.)) Two of the ten corpus papers cite it, and {[senol2024_double]} wrote a replacement collector because of it: //"Although TRC already has the capability to intercept JavaScript API calls, we introduced a separate collector due to a known TRC bug that causes it to miss the initial function calls."// An empty ''callStats'' for a script that runs at the top of the document is not evidence that it touched no API.
+  * **''size'' is not the size of anything you can sum twice.** It is CDP's ''encodedDataLength'' from ''Network.loadingFinished'' — bytes on the wire, including headers — so it moves between runs for a byte-identical body. Two runs of the published script above returned ''473'' and ''474'' for ''example.com'' with the same ''responseBodyHash''. It is also coerced to ''null'' when CDP reports a negative value, with the comment "make sure we can use unsigned int for this field in clickhouse". Use ''responseBodyHash'' for body identity and treat ''size'' as an estimate.
+  * **''cookiepopups'' archives page content.** ''scrapedFrames[].buttons'' captures the text of every clickable element, which on a news site means headlines. Include it in your redaction plan alongside ''hostname'' and ''username''.
+</WRAP>
 ===== How Tracker Radar is built from it =====
-The public repositories make the pipeline legible:
+The public repositories make the pipeline legible. Dates and defaults below were checked on 2026-09-03.
   - **Collect.** Run TRC over a declared site list, region and browser configuration. Write one per-site result and the crawl metadata. The raw record is about what this crawl observed; it is not yet a tracker label.
-  - **Process.** Tracker Radar Detector reads those site files (or its configured database), processes ''requests'', ''cookies'' and ''apis'', resolves CNAMEs and groups domains into entities. Its code has a ''minSites'' threshold, defaulting to 2, so a request seen once is not automatically emitted as a common tracker. The threshold and first-party settings belong in your reproduction record. {[duckduckgo_tracker_radar_detector_2026]}
+  - **Process.** Tracker Radar Detector reads those site files (or its configured database), processes ''requests'', ''cookies'' and ''apis'', resolves CNAMEs and groups domains into entities. ''config.json'' sets ''minSites'' to **2**, used in ''src/trackers/classes/crawl.js'', so a request seen on one site is not emitted as a common tracker. That threshold and the ''keepFirstParty'' and ''treatCnameAsFirstParty'' settings belong in your reproduction record. Detector has had no commit since **2025-03-31**. {[duckduckgo_tracker_radar_detector_2026]}
-  - **Aggregate.** Detector counts the sites in which a domain/request is observed, derives cookie prevalence and API-fingerprinting evidence, and combines those observations with ownership/entity data. Its prevalence denominator is the set of sites successfully processed under that run's configuration, not all websites on the Internet and not all domains in the final JSON.
+  - **Aggregate.** Detector counts the sites in which a domain or request is observed, derives cookie prevalence and API-fingerprinting evidence, and combines those observations with ownership/entity data. Its prevalence denominator is the set of sites successfully processed under that run's configuration — not all websites on the Internet, and not all domains in the final JSON.
-  - **Build.** The build step emits region-specific domain files and generated maps such as ''domain_map.json'', ''entity_map.json'', ''domain_summary.json'' and ''data_by_site.json''. Domain records can carry resources and regex rules, owner/source, prevalence, fingerprinting score, cookies, performance, CNAMEs, nameservers and example sites. Resource records can carry cookie/API/fingerprinting observations, response hashes, CNAMEs and first-party-cookie fields. {[duckduckgo_tracker_radar_2026]}
+  - **Build.** The build step emits region-specific domain files and generated maps: ''domain_map.json'', ''entity_map.json'', ''domain_summary.json'' and ''data_by_site.json'', all four present and populated in the live dataset repository. Domain records can carry resources and regex rules, owner/source, prevalence, fingerprinting score, cookies, performance, CNAMEs, nameservers and example sites; resource records can carry cookie/API/fingerprinting observations, response hashes, CNAMEs and first-party-cookie fields. {[duckduckgo_tracker_radar_2026]}
-Tracker Radar describes itself as a dataset of common third-party domains and their observed behavior, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking.
+Tracker Radar describes itself as a dataset of common third-party domains and their observed behaviour, not as a complete blocklist. A domain's presence is conditional on the site list, region, browser, consent action, crawl date, ''minSites'', first-party/CNAME policy and the Detector version. Do not turn a Tracker Radar domain entry into a universal claim that every request from that domain tracks, or that absence means absence of tracking.
-The two first-party decisions must not be collapsed. TRC's ''--only-3p'' option uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. If you want to study CNAME cloaking or first-party cookies, disabling an early filter may be necessary. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]}
+The two first-party decisions must not be collapsed. TRC's ''%%--only-3p%%'' uses eTLD+1 while collecting; Detector's processing has its own first-party, entity and CNAME options. If you enable both, publish both settings. {[duckduckgo_trc_2026]} {[duckduckgo_tracker_radar_detector_2026]}
-===== What the corpus shows =====
+===== Use in Publications =====
+TRC is a sensible choice when the measurement needs Chromium/CDP access, target-level attachment, conditional JavaScript breakpoints, or a modular path into the Tracker Radar pipeline. It is not automatically right for a Firefox-user result, a cookie-set event history, or a raw-body archive.
+==== Who uses it, and the count that is easy to get wrong ====
+The all-paper population in this corpus is **5,859 papers** from seven venues. A broad ''tracker.?radar'' name match hits **21** papers — but 11 of those name the **dataset**, the entity list, the entity map or the wiki, filed by the extraction as a ''classification-service'', a ''cookie-database'' or ''other'', and none of them ran the Collector. The adoption count therefore uses the narrower query: a tool name matching the Collector, an automation category, and a used/produced status. That gives **10 papers in the 1,120-paper CRAWLED population**.
+One more boundary inside that 10, because this page is about boundaries: **nine drove a TRC fork; the tenth did not.** {[aziz2024_johnny]} built its crawler itself — //"We used custom scripts, written in Python and JavaScript, to drive and instrument an instance of Chrome using the Chrome DevTools Protocol"// — and reused TRC's cookie-method breakpoint code inside it. It is the one ''browser-automation'' tuple in the category audit and it counts under the written rule, but "TRC users" in the table below means nine TRC-driven crawls plus one module reuse. Do not read the ''crawlConfig'' survey's silences as TRC's defaults for that row: it had every choice TRC does not give you.
+<WRAP important>
+The [[:programming:crawler|crawler comparison page]] publishes **21** in a row labelled "Tracker Radar Collector", because its shared fold rule matches ''tracker.?radar'' and names the family after the crawler. That 21 is the dataset and the crawler together. Its other row — 10 papers, 0.9% of the folded crawler frameworks within CRAWLED — agrees with this page. When you see a Tracker Radar count anywhere, check whether it counts the crawler, the dataset, or both.
+</WRAP>
-The all-paper population in the refreshed corpus is **5,859 papers** from seven venues. A broad Tracker Radar name match occurs in 21 papers, but 11 of those are names for the dataset, entity map, wiki or a general service rather than a Collector automation tuple. The page's adoption count therefore uses the narrower query: a tool name matching the Collector, an automation category, and a used/produced status. That gives **10 papers in the 1,120-paper CRAWLED population**; the 10 are also the complete TRC-user set returned by this query.
+An independent full-text sweep of all 5,855 papers that have a repaired text rendering, run without reference to the extraction, finds exactly the same 10 papers, with no miss in either direction.
 ^ Year ^ CRAWLED papers in that year ^ TRC users ^ TRC users / CRAWLED papers ^ Status ^
@@ Line 82: / Line 304: @@
 | 2026 | 69 | 1 | 1.4% | provisional |
-The 2021--2024 window contributes 6 users; the provisional 2025--2026 slice contributes 4. The latter is a useful current-use signal, not a complete trend: CCS and IMC 2026 have not been held, while IEEE S&P and WWW 2026 are under-selected by construction.
+The 2021--2024 window contributes 6 users; the provisional 2025--2026 slice contributes 4. The latter is a current-use signal, not a trend: CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 are under-selected by construction.
-The papers use TRC as an extensible base, not as a fixed black-box instrument:
+==== The papers use it as a base, not as a black box ====
 ^ Year ^ Paper and extension ^ What the paper added or used ^
@@ Line 90: / Line 312: @@
 | 2022 | ''Leaky Forms'' | Built an interactive crawler on TRC; added inner-page discovery, email/password filling, Fathom and Consent-O-Matic, browser-API/cookie/request collection, CDP breakpoints with JavaScript stacks, WebSocket capture and POST/input observations. {[senol2022_leaky]} |
 | 2023 | ''Is Your Wallet Snitching on You?'' | Used TRC for popular-site crawls and added wallet-API breakpoints/simulation alongside request and cookie observations. {[torres2023_wallet]} |
-| 2024 | ''Johnny Still Can't Opt-out'' | Reused TRC code for non-invasive breakpoints on DOM cookie methods, JavaScript stacks and parameter values; this is a targeted reuse of the Collector surface, not evidence that the unmodified CLI records those fields by default. {[aziz2024_johnny]} |
+| 2024 | ''Johnny Still Can't Opt-out'' | Reused TRC code for non-invasive breakpoints on DOM cookie methods, JavaScript stacks and parameter values; a targeted reuse of the Collector surface, not evidence that the unmodified CLI records those fields. {[aziz2024_johnny]} |
 | 2024 | ''Targeted and Troublesome'' | Extended TRC with ad and fingerprint collectors while retaining request, target, cookie and CMP observations; also used anti-bot and mobile settings. {[moti2024_targeted]} |
-| 2024 | ''The Double Edged Sword'' | Forked TRC to identify authentication pages and measure their fingerprinting behavior. {[senol2024_double]} |
+| 2024 | ''The Double Edged Sword'' | Forked TRC to identify authentication pages and measure their fingerprinting behaviour. {[senol2024_double]} |
 | 2025 | ''Canvassing the Fingerprinters'' | Modified TRC to save Canvas API arguments, return values, script source and timestamps, with anti-bot handling, scrolling and consent automation. {[luo2025_canvassing]} |
 | 2025 | ''Referrer Policy'' | Extended TRC with inner-page links, Referer/Referrer-Policy and POST-body observations, the page's ''referrerPolicy'' value and API collection. {[zagi2025_referrer]} |
 | 2025 | ''Sheep's Clothing, Wolfish Intent'' | Used the tracker-radar-collector framework to collect advertisements for automated evaluation. {[roongta2025_sheep]} |
-| 2026 | ''Bridges to Self'' | Customized TRC for mobile web-to-app tracking, including anti-bot handling, WebSocket frames, WebRTC APIs and Priv-Accept observations. {[vlummens2026_bridges]} |
+| 2026 | ''Bridges to Self'' | Customised TRC for mobile web-to-app tracking, including anti-bot handling, WebSocket frames, WebRTC APIs and Priv-Accept observations. {[vlummens2026_bridges]} |
-The examples support a practical rule: TRC's value is its CDP/collector architecture, but the paper's measurement surface is the fork and configuration actually run. Cite the upstream Collector for the base and describe every added collector, breakpoint, filter, consent action and interaction.
+Note the 2022 row: Consent-O-Matic was a real part of that crawler in 2022. It is not part of TRC now — autoconsent dropped Consent-O-Matic rule support in v16.0.0 (2026-06-23), before the version TRC pins.
-===== Measurement checklist =====
+==== What they configured, and what they did not say ====
-Before collecting, decide what the output is supposed to answer. Then record at least:
+From ''crawlConfig'', which is an enum-bearing object and so publishable as counts. Sentinels are shown, never subtracted. All 10 papers have a ''crawlConfig'' object.
-  * the TRC repository commit, package-lock state, Node.js, Puppeteer and Chromium versions (or the explicit ''--chromium-version'');
+^ Key ^ Values across the 10 TRC papers ^
-  * the site list, its source/version/date, URL selection and whether the crawl visits only a homepage or follows inner links;
+| ''interactionDepth'' | landing-plus-subpages 7, landing-page-only 3 |
-  * crawl dates, region/vantage/IP conditions, proxy, success/timeout policy, concurrency, maximum load time and post-load wait;
+| ''consentAction'' | accept-all 4, not-stated 3, accept-and-reject 2, no-interaction 1 |
-  * fresh versus reused browser context/profile, one profile per site or shared state, viewport/user agent, mobile emulation and anti-bot injection;
+| ''statefulness'' | not-stated 7, stateless 2, both 1 |
-  * the exact collector list and any custom collector, breakpoint, API list, WebSocket/body/header capture or response-hash setting;
+| ''headless'' | not-stated 8, headless 2 |
-  * consent action (none, opt-in, opt-out), autoconsent/rule version, banner interaction and the time at which collection begins;
+| ''subpagesPerSite'' | null 4, and 3, 5, 9, 10, 15, 20 once each |
-  * first-party filtering at collection time, and the later Detector settings for ''minSites'', ''keepFirstParty'', ''treatCnameAsFirstParty'', entity data and included pages;
-  * raw-archive retention/redaction, especially cookies, response bodies, screenshots, hostname and username in metadata; and
-  * the denominator for every prevalence number: attempted sites, successful sites, sites with a relevant response, or sites retained by Detector.
-Anti-bot measures and consent automation are interventions, not harmless crawler options. They can change the page, requests and cookies. A result from ''--only-3p'' is conditioned on an eTLD+1 filter; it is not a result about ownership, entity prevalence or CNAME cloaking.
+Read the last two rows carefully, because they are the opposite of an under-reporting finding. TRC's CLI gives you **no choice** about either: a fresh temporary profile per site is unconditional, and headless is a hard-coded constant. So "not stated" here mostly means "there was nothing to state" — unlike the same silence on a tool that does offer the option. What a TRC paper does need to state is whether it used the module API to share a browser connection across sites, because that is the only way to make a TRC crawl stateful.
-===== Use in Publications =====
+The ''interactionDepth'' row is the substantive one: **7 of 10** went beyond the homepage, with sub-page budgets from 3 to 20. TRC has no inner-page traversal at all — no link extraction, no depth parameter, and no mention of either in its README — so it crawls exactly the URLs you hand it, one page each. That work is yours, and the seven did it four different ways:
-TRC is a sensible choice when the measurement needs Chromium/CDP access, target-level attachment, conditional JavaScript breakpoints, or a modular path into the Tracker Radar pipeline. It is not automatically the right choice for a Firefox-user result, a cookie-set event history, or a raw-body archive. In all cases, report the capture surface rather than only “we used Tracker Radar Collector”.
+  * **Four added link discovery to their fork.** {[senol2022_leaky]}: "We modify TRC to efficiently discover inner pages", following links matched by a regular expression lifted from Firefox's Password Manager. {[moti2024_targeted]} and {[zagi2025_referrer]} each wrote a ''LinkCollector''. {[senol2024_double]} finds candidate login and sign-up links on the homepage and visits up to fifteen on a second pass.
+  * **One selected inner pages into its site list** instead of traversing: {[roongta2025_sheep]} added three inner pages for any site with more than ten, reaching "4,710 web pages" from 1,500 sites.
+  * **One does not say how its pages were found** — {[dimova2021cname]} reports a "preliminary crawl of 20 pages per website" without describing the discovery.
+  * **One is not a TRC crawl at all**: {[aziz2024_johnny]}'s traversal is in its own crawler.
-At minimum, a methods paragraph should let a reader reconstruct the chain:
+If your design needs inner pages, budget for building that part — and say which of these four things you did, because they are not the same measurement.
-  * **browser:** Chromium build, Puppeteer/TRC commit, viewport/device mode and anti-bot settings;
+<WRAP important>
-  * **crawl:** site-list version, dates, region/proxy, concurrency, navigation depth, waits, retries and failure rule;
+''crawlConfig'', ''ethics'' and ''artifacts'' share **one** evidence quote per paper, so no single key in the table above is independently quote-checked. Treat the row totals as a survey of what the extraction recorded, not as a per-key verified count.
-  * **state/consent:** profile/context reset, consent action and autoconsent version;
+</WRAP>
-  * **capture:** collector list and custom changes, including whether cookie values, event timing, request headers/bodies, response bodies, API arguments, WebSockets and target/frame identity were available; and
-  * **analysis:** first-party filter, domain/entity/CNAME mapping, Detector commit and every denominator used for prevalence.
-Do not report “number of trackers” without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records. Those are different units.
+==== Three denominators worth copying ====
+TRC crawls produce publishable results, and the interesting part is what the results are divided by. These are quoted from the papers, not from the extraction's summary fields, because the extraction's ''metric'' names the unit and often not the population.
+  * {[senol2022_leaky]} reports leaks as ''1,844 / 60,008 (3.0%)'' in the EU desktop crawl and ''2,950 / 60,999 (4.8%)'' in the US, captioned as //"the number of sites leaking emails or passwords to trackers, compared to the number of sites where we could fill an email address"//. The crawl was the Tranco top 100,000. Divide by 100,000 instead and you get 1.8%, a figure that is wrong by a factor of 1.7 and looks perfectly plausible.
+  * {[torres2023_wallet]} states the crawler's own success rate before using it: //"TRC was able to crawl 96,905 out of 100K websites successfully (i.e., 96.91%). We found 1,114 unique scripts on 1,325 websites…"// — so the denominator is successfully-crawled sites, and the 3.09% failure rate is on the page rather than hidden.
+  * {[luo2025_canvassing]} puts the denominator in the sentence: //"Of the 16,276 popular sites crawled successfully, 2,067 (12.7%) of them extracted at least one fingerprintable canvas."//
+==== Papers to read first ====
+  * **For how to fork TRC at all** — {[dimova2021cname]}. The earliest and clearest account of extending the Puppeteer/CDP crawler to capture something it does not capture by default, with the profile, wait and reload spelled out.
+  * **For the most ambitious fork** — {[senol2022_leaky]}. Inner-page discovery, form filling, a classifier, consent automation, CDP stacks and WebSocket capture, all on the TRC base; and the denominator lesson above.
+  * **For the reporting standard to imitate** — {[torres2023_wallet]} and {[luo2025_canvassing]}. Both state the crawl success rate and name the population inside the sentence carrying the number.
+  * **For what a crawler choice costs you** — {[ahmad2020_apophanies]} and {[sanchezrola2023_rods]}. Neither uses TRC; both are about how much of your result is your instrument.
+===== What to report in a paper =====
+Before collecting, decide what the output is supposed to answer. Then record at least:
+  * the TRC repository **commit** — not ''projectVersion'', which is always ''1.0.0'' — plus the package-lock state, Node.js, Puppeteer and Chromium versions, or the explicit ''%%--chromium-version%%'';
+  * the site list, its source, version and date, the URL selection, and whether the crawl visits only a homepage or follows inner links, with the sub-page budget;
+  * crawl dates, region/vantage/IP conditions, proxy, success and timeout policy, concurrency, the real ''maxLoadTimeMs'' you ran with, and the post-load wait;
+  * whether state was shared across sites, which requires the module API — and the viewport and user agent, remembering that the default UA is a fixed constant and not the browser you ran;
+  * the exact collector list and any custom collector, breakpoint, API list, WebSocket/body/header capture or response-hash setting;
+  * the consent action, the autoconsent version, the banner interaction, the time at which collection begins, **and the share of sites where the action actually succeeded**;
+  * first-party filtering at collection time, and the later Detector settings for ''minSites'', ''keepFirstParty'', ''treatCnameAsFirstParty'', entity data and included pages;
+  * raw-archive retention and redaction — cookies, response bodies, screenshots, scraped button text, and ''hostname'' and ''username'' in the metadata; and
+  * the denominator for every prevalence number: attempted sites, successfully crawled sites, sites where the relevant interaction worked, or sites retained by Detector.
+Anti-bot measures and consent automation are interventions, not harmless crawler options: they change the page, the requests and the cookies. A result from ''%%--only-3p%%'' is conditioned on an eTLD+1 filter and is not a result about ownership, entity prevalence or CNAME cloaking. And do not report a "number of trackers" without naming whether that means requests, domains, resources, entities, sites with a request, or Tracker Radar records — those are six different units.
 ===== Methodology and limitations of these figures =====
-The corpus figures above come from the current ''data/extract/run1'' extraction: 5,859 papers, seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P), with 1,120 in the CRAWLED population. The 2025--2026 venue-years are provisional. Names were folded only for the Collector automation query, and the excluded Tracker Radar residue is printed in the report rather than silently discarded. The report counts papers for adoption figures; its raw-name audit counts tuples only to expose the mapping residue.
+The corpus figures come from the current ''data/extract/run1'' extraction: 5,859 papers, seven venues (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P), with 1,120 in the CRAWLED population. Every claim here is a claim about those seven venues. The 2025--2026 venue-years are provisional. Names were folded only for the Collector automation query, and the excluded Tracker Radar residue is printed in the report rather than silently discarded. Adoption figures count papers; the raw-name audit counts tuples only, to expose the mapping residue.
+The 10 evidence quotes behind the adoption count were checked against the papers' repaired full text: **1 verbatim, 1 exact except capitalisation, 8 partial** — a partial match being a 5-to-12-word run that contains the phrase "tracker radar collector" itself, split by the column repair. None was unlocatable. Three of the quotes drop a leading clause and capitalise a mid-sentence "we", so they are edited fragments rather than verbatim sentences; the surrounding sentence supports the claim in every case.
+The reproducible reports are ''report_tracker_radar_collector.mjs'', ''trc_review_probe.mjs'', ''trc_index_reconcile.mjs'', ''trc_paper_detail.mjs'' and ''trc_smoke.mjs''. The complete query log, the unedited script output, the source-quote checks, the external-source verification, the review log and the judgement calls are on [[:provenance:programming:crawler:tracker_radar_collector|the page provenance record]]. Dataset-wide extraction and retrieval caveats are on [[:literature:corpus|the corpus provenance page]].
+This page does not claim that TRC is faster, more complete or more accurate than OpenWPM, nor that its 10-paper count is a market-share estimate. The repositories were checked for maintenance and schema, and one real crawl was run; the corpus was used to find research use and concrete extensions. A live repository is evidence that TRC remains available, not evidence that every deployed Tracker Radar crawl uses the public defaults.
-The reproducible report is ''scripts/report_tracker_radar_collector.mjs'' in the working corpus checkout. The complete query log, real script output, source-quote checks, external-source verification and judgement calls are on [[provenance:programming:crawler:tracker_radar_collector|the page provenance record]]. Dataset-wide extraction and retrieval caveats are on [[literature:corpus|the corpus provenance page]].
+===== Related pages =====
-This page does not claim that TRC is faster, more complete or more accurate than OpenWPM, nor that its 10-paper corpus count is a market-share estimate. The repositories were checked separately for maintenance and schema; the corpus was used to find research use and concrete extensions. The current repository is evidence that TRC remains available, not evidence that every deployed Tracker Radar crawl uses the exact public defaults.
+  * [[:programming:crawler|Comparison of Crawling Libraries]] — the choice page, and the source of the 21-versus-10 discrepancy explained above.
+  * [[:programming:crawler:openwpm|OpenWPM]] — the Firefox alternative, and the other half of this page's comparison.
+  * [[:programming:crawler:pagegraph|Brave PageGraph]] — the other Chromium-based research instrument, with a graph rather than a record stream.
+  * [[:programming:traffic_files|Traffic Files: HAR, pcap, and Proxy Flows]] — what to do when the browser layer is not the layer you need.
+  * [[:privacy:cookies|Cookies]] — for what to do with a cookie jar snapshot once you have one.
 ===== References =====