Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: Tracker Radar Collector ======

This is the working log for [[programming:crawler:tracker_radar_collector|Tracker Radar Collector]]. It records the decisions and checks specific to that page; dataset-wide extraction caveats live on [[literature:corpus|the corpus provenance page]]. It deliberately has no discussion block: comments belong on the content page.

===== Page, scope and run =====

^ Item ^ Record ^
| Content page | [[programming:crawler:tracker_radar_collector|programming:crawler:tracker_radar_collector]] |
| Corpus provenance | [[literature:corpus|literature:corpus]] |
| Report script | ''scripts/report_tracker_radar_collector.mjs'' |
| Run date | 2026-08-14 UTC |
| Corpus at run | 5,859 extracted papers, 2010--2026, seven venues: CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P |
| Relevant population | CRAWLED = 1,120 papers; the report defines this using crawl configuration or the automated-web-crawl study type |
| Models/agents | One primary Codex run. No subagent/Agent tool was available in this environment, so the focused and generic review passes below were performed by the primary run and are recorded as such. |

The existing [[programming:crawler|Comparison of Crawling Libraries]] page already promised a red link to this exact page and acts as the crawler index. I created this new focused subpage rather than broadening the index or duplicating [[programming:crawler:openwpm|OpenWPM]]: the question here is TRC's capture schema and its path into Tracker Radar, while the neighbours compare crawler families and OpenWPM's Firefox instrumentation. No parent-page edit was needed.

===== Queries and populations =====

The following were run from ''/workspace/artifacts/wiki'' unless an absolute path is shown. The first command is an orientation query only; its free-text table was not pasted into the page.

<code bash>
node scripts/sitemap.mjs
node /workspace/publications_dataset/scripts/site_queries.mjs --run /workspace/publications_dataset/data/extract/run1 --page crawlers --format wiki
node scripts/report_tracker_radar_collector.mjs
</code>

The report's exact populations are:

^ Query/result ^ Population and denominator ^ Rule ^
| CRAWLED | 1,120 papers | ''POPULATIONS.crawled'' from the current ''run1'' extraction: a crawl configuration or automated-web-crawl study type |
| TRC crawler users | 10 papers | At any corpus size, a paper has a ''tools[]'' tuple whose name matches ''tracker[- ]radar[- ]collector'', whose category is ''crawler-framework'' or ''browser-automation'', and whose status is ''used'' or ''produced''. The published adoption figure uses the 10-paper intersection with CRAWLED; all 10 are inside it. |
| OpenWPM comparison | 58 papers | Same automation-category and used/produced rule in CRAWLED, with a case/spacing fold for OpenWPM. This is a context comparison, not a claim of relative quality. |
| Broad Tracker Radar boundary | 21 papers | All 5,859 extracted papers whose tool name matches ''tracker[- ]radar'', regardless of whether the tuple names a crawler, dataset, entity map, wiki or service. 11 do not satisfy the Collector automation query. |
| Year table | Per-year CRAWLED denominator | For each year, count CRAWLED papers and TRC papers in that year; percentages are TRC users divided by that year's CRAWLED papers. 2025 and 2026 are marked provisional. |
| Name audit | Tuple counts only | Count every matching raw ''tools[]'' tuple to print the fold and residue. These tuple counts are not paper denominators and are not published as prevalence. |

The page says “10 papers in the 1,120-paper CRAWLED population,” not “10 of 5,859,” because the latter would silently use the wrong population. The all-paper 5,859 and broad 21/11 figures are stated only with the all-paper boundary definition.

===== Folding and residue =====

The report uses the case-insensitive pattern ''tracker[\\s-]?radar[\\s-]?collector''. It permits a DuckDuckGo prefix and a ''(TRC)'' suffix because the match is on the full tool name, then restricts the population to the two automation categories and used/produced statuses. This is deliberately narrower than ''tracker[- ]radar''.

The four raw strings mapped to the Collector are:

  * ''DuckDuckGo Tracker Radar Collector'' — 4 tuples
  * ''Tracker Radar Collector'' — 4 tuples
  * ''DuckDuckGo tracker-radar-collector'' — 1 tuple
  * ''Tracker Radar Collector (TRC)'' — 1 tuple

The complete excluded radar-name residue printed by the script is:

<code>
  8  "DuckDuckGo Tracker Radar"
  1  "DuckDuckGo Tracker Radar Wiki"
  1  "Tracker Radar"
  1  "Tracker Radar Entity List"
  1  "Tracker Radar Entity Map"
</code>

The raw audit therefore has 9 distinct radar strings, 4 mapped Collector strings and 5 excluded strings. The report prints this residue so a later alias change cannot silently turn the dataset or entity map into a crawler count.

===== Verbatim report output =====

The following is the output of ''node scripts/report_tracker_radar_collector.mjs'' from the run above, copied without editing. It is the audit trail for every number on the content page.

<file text>
corpus: 5859 papers, 7 venues, 2010–2026
population CRAWLED: 1120 papers (crawlConfig or automated-web-crawl study type)
population TRC: 10 papers with a used/produced automation-tool tuple naming the Collector
generated by scripts/report_tracker_radar_collector.mjs

=== A. Population and the Collector/dataset boundary ===

Population                        Papers  Definition
--------------------------------  ------  --------------------------------------------------------------------------------
TRC crawler users                 10      tools[].name matches Tracker Radar Collector; automation category; used/produced
TRC crawler users inside CRAWLED  10      same query ∩ CRAWLED (1120)
OpenWPM users inside CRAWLED      58      same automation-category/used-or-produced rule; OpenWPM fold
Any Tracker Radar tool name       21      broad radar name match, including dataset/entity/wiki names
Radar names excluded from TRC     11      any radar name but no Collector automation tuple
The page uses the first row for the Collector adoption count. The broad Tracker Radar rows are a boundary check, not a second crawler count.

=== B. Year-by-year use ===

Year   Crawling papers  TRC users  Share of that year’s crawling papers
-----  ---------------  ---------  ------------------------------------
2021   75               1          1.3%
2022   110              1          0.9%
2023   125              1          0.8%
2024   110              3          2.7%
2025*  129              3          2.3%
2026*  69               1          1.4%
* 2025–2026 are provisional venue-years: CCS/IMC 2026 have not been held, and IEEE S&P/WWW 2026 are incompletely selected.
Across the complete 2021–2024 window: 6 TRC users; in the provisional 2025–2026 window: 4.

=== C. Folded names and unmapped residue ===

Fold rule: case-insensitive `tracker[- ]radar[- ]collector`, after allowing DuckDuckGo as a vendor prefix and `(TRC)` as a suffix; only automation categories and used/produced tuples enter the TRC population. Counts below are tuples for residue auditing, never paper percentages.
Raw tool string folded to Tracker Radar Collector  Tuples
-------------------------------------------------  ------
"DuckDuckGo Tracker Radar Collector"               4
"Tracker Radar Collector"                          4
"DuckDuckGo tracker-radar-collector"               1
"Tracker Radar Collector (TRC)"                    1
Excluded radar-name residue (printed in full):
  8  "DuckDuckGo Tracker Radar"
  1  "DuckDuckGo Tracker Radar Wiki"
  1  "Tracker Radar"
  1  "Tracker Radar Entity List"
  1  "Tracker Radar Entity Map"
Radar-name audit: 9 distinct strings; Collector fold mapped 4; excluded residue 5.

=== D. Papers in the TRC population ===

Year  Paper key                                                                                    Tool string(s)                                           Evidence quote(s)
----  -------------------------------------------------------------------------------------------  -------------------------------------------------------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2021  PETS/2021/the-cname-of-the-game-large-scale-analysis-of-dns-based-tracking-evasion           DuckDuckGo Tracker Radar Collector [crawler-framework]   "We built our crawler by modifying the DuckDuckGo Tracker Radar Collector [21], a Puppeteer-based crawler that uses the Chrome DevTools Protocol (CDP)."
2022  USENIX/2022/leaky-forms-a-study-of-email-and-password-exfiltration-before-form-submission    Tracker Radar Collector [crawler-framework]              "We develop an interactive, instrumented crawler based on DuckDuckGo's Tracker Radar Collector"
2023  USENIX/2023/is-your-wallet-snitching-on-you-an-analysis-on-the-privacy-implications-of-web3  DuckDuckGo Tracker Radar Collector [crawler-framework]   "We use DuckDuckGo's Tracker Radar Collector (TRC) to crawl popular websites and measure their behavior."
2024  IEEE-SP/2024/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites         Tracker Radar Collector [crawler-framework]              "we extended Tracker Radar Collector (TRC) [93]. TRC is a Puppeteer-based [94] web crawler"
2024  PETS/2024/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework              DuckDuckGo Tracker Radar Collector [browser-automation]  "We used code from DuckDuckGo's Tracker Radar Collector (TRC) that allowed us to set non-invasive breakpoints on the DOM cookie methods and record the JavaScript stack traces"
2024  WWW/2024/the-double-edged-sword-identifying-authentication-pages-and-their-fingerprinting    Tracker Radar Collector [crawler-framework]              "Our web crawler is a fork of Tracker Radar Collector (TRC), a crawler created by DuckDuckGo."
2025  IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th    DuckDuckGo Tracker Radar Collector [crawler-framework]   "We modified DuckDuckGo's Tracker Radar Collector [22], a Puppeteer-based Web crawler."
2025  PETS/2025/referrer-policy-implementation-and-circumvention                                   Tracker Radar Collector [crawler-framework]              "We extended the open-source crawler Tracker Radar Collector (TRC) [32]."
2025  PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic   DuckDuckGo tracker-radar-collector [crawler-framework]   "To collect ads, we utilized the DuckDuckGo tracker-radar-collector framework, built on the Puppeteer web crawler."
2026  USENIX/2026/bridges-to-self-silent-web-to-app-tracking-on-mobile-via-localhost               Tracker Radar Collector (TRC) [crawler-framework]        "We perform web crawls using a customized version of DuckDuckGo's Tracker Radar Collector (TRC), an opensource Puppeteer-based crawler equipped with anti-bot measures"

=== E. Evidence quote checks ===

10-word  PETS/2021/the-cname-of-the-game-large-scale-analysis-of-dns-based-tracking-evasion  /workspace/publications_dataset/data/fulltext/2021/PETS/the-cname-of-the-game-large-scale-analysis-of-dns-based-tracking-evasion/paper.cols.txt
  "We built our crawler by modifying the DuckDuckGo Tracker Radar Collector [21], a Puppeteer-based crawler that uses the Chrome DevTools Protocol (CDP)."
7-word-compact  USENIX/2022/leaky-forms-a-study-of-email-and-password-exfiltration-before-form-submission  /workspace/publications_dataset/data/fulltext/2022/USENIX/leaky-forms-a-study-of-email-and-password-exfiltration-before-form-submission/paper.cols.txt
  "We develop an interactive, instrumented crawler based on DuckDuckGo's Tracker Radar Collector"
8-word-compact  USENIX/2023/is-your-wallet-snitching-on-you-an-analysis-on-the-privacy-implications-of-web3  /workspace/publications_dataset/data/fulltext/2023/USENIX/is-your-wallet-snitching-on-you-an-analysis-on-the-privacy-implications-of-web3/paper.cols.txt
  "We use DuckDuckGo's Tracker Radar Collector (TRC) to crawl popular websites and measure their behavior."
exact  IEEE-SP/2024/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites  /workspace/publications_dataset/data/fulltext/2024/IEEE-SP/targeted-and-troublesome-tracking-and-advertising-on-childrens-websites/paper.cols.txt
  "we extended Tracker Radar Collector (TRC) [93]. TRC is a Puppeteer-based [94] web crawler"
10-word-compact  PETS/2024/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework  /workspace/publications_dataset/data/fulltext/2024/PETS/johnny-still-cant-opt-out-assessing-the-iab-ccpa-compliance-framework/paper.cols.txt
  "We used code from DuckDuckGo's Tracker Radar Collector (TRC) that allowed us to set non-invasive breakpoints on the DOM cookie methods and record the JavaScript stack traces"
10-word  WWW/2024/the-double-edged-sword-identifying-authentication-pages-and-their-fingerprinting  /workspace/publications_dataset/data/fulltext/2024/WWW/the-double-edged-sword-identifying-authentication-pages-and-their-fingerprinting/paper.cols.txt
  "Our web crawler is a fork of Tracker Radar Collector (TRC), a crawler created by DuckDuckGo."
exact  IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th  /workspace/publications_dataset/data/fulltext/2025/IMC/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th/paper.cols.txt
  "We modified DuckDuckGo's Tracker Radar Collector [22], a Puppeteer-based Web crawler."
9-word-compact  PETS/2025/referrer-policy-implementation-and-circumvention  /workspace/publications_dataset/data/fulltext/2025/PETS/referrer-policy-implementation-and-circumvention/paper.cols.txt
  "We extended the open-source crawler Tracker Radar Collector (TRC) [32]."
10-word-compact  PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic  /workspace/publications_dataset/data/fulltext/2025/PETS/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic/paper.cols.txt
  "To collect ads, we utilized the DuckDuckGo tracker-radar-collector framework, built on the Puppeteer web crawler."
10-word-compact  USENIX/2026/bridges-to-self-silent-web-to-app-tracking-on-mobile-via-localhost  /workspace/publications_dataset/data/fulltext/2026/USENIX/bridges-to-self-silent-web-to-app-tracking-on-mobile-via-localhost/paper.cols.txt
  "We perform web crawls using a customized version of DuckDuckGo's Tracker Radar Collector (TRC), an opensource Puppeteer-based crawler equipped with anti-bot measures"
Quote checks: 2 exact, 8 partial (longest 5–10 word run), 0 not located, 10 TRC tool tuples checked.

=== F. Current-tool interpretation ===

The corpus identifies TRC users in 2021–2026. The complete 2021–2024 count is historical evidence; the 2025–2026 count is a provisional current-use signal, not a prevalence estimate for the field.
No query here claims that TRC supersedes OpenWPM. The repositories were checked separately for current maintenance and the page compares capture surfaces rather than ranking one as a standard.

</file>

===== Paper evidence and quote checks =====

The script's 10 evidence tuples were checked against the corresponding ''paper.cols.txt'' files under ''/workspace/publications_dataset/data/fulltext/<year>/<venue>/<slug>/''. Two quotes matched after whitespace normalization; eight matched as a 5--10-word run after compacting column/page-break punctuation. The partial status is a repaired-column-layout artifact, not an unsupported citation: the surrounding sentence in each source states the Collector use. No quote was unlocatable. The exact result is 2 exact, 8 partial, 0 not located, 10 tuples checked.

The checked papers and the measurement detail used on the page are:

^ Corpus record ^ Quote-check result ^ Detail read around the evidence quote ^
| PETS 2021, ''The CNAME of the Game'' | 10-word | TRC is Puppeteer/CDP; the fork captured HTTP-request cookies, POST data, ''document.cookie'' assignments and cookie-setting scripts, with fresh profile/wait/reload. {[dimova2021cname]} |
| USENIX 2022, ''Leaky Forms'' | 7-word-compact | TRC was extended with interactive form filling, inner-page discovery, Fathom, Consent-O-Matic, API/cookie/request collectors, CDP stacks, WebSocket and POST/input capture. {[senol2022_leaky]} |
| USENIX 2023, ''Is Your Wallet Snitching on You?'' | 8-word-compact | TRC was used for popular-site measurement with wallet API breakpoints/simulation and network/cookie observations. {[torres2023_wallet]} |
| IEEE S&P 2024, ''Targeted and Troublesome'' | exact | The authors extended TRC and added ad/fingerprint collection, anti-bot and mobile settings. {[moti2024_targeted]} |
| PETS 2024, ''Johnny Still Can't Opt-out'' | 10-word-compact | The paper reused TRC code for DOM cookie-method breakpoints, stack traces and values; it does not say these are default TRC output. {[aziz2024_johnny]} |
| WWW 2024, ''The Double Edged Sword'' | 10-word | The crawler was a TRC fork for authentication-page and fingerprinting analysis. {[senol2024_double]} |
| IMC 2025, ''Canvassing the Fingerprinters'' | exact | Canvas arguments, return values, source and timestamps were added, alongside anti-bot, scrolling and consent automation. {[luo2025_canvassing]} |
| PETS 2025, ''Referrer Policy'' | 9-word-compact | The fork added inner-page links, Referer/Referrer-Policy, POST bodies, the page's ''referrerPolicy'' and API collection. {[zagi2025_referrer]} |
| PETS 2025, ''Sheep's Clothing, Wolfish Intent'' | 10-word-compact | TRC was used to collect advertisements for automated evaluation. {[roongta2025_sheep]} |
| USENIX 2026, ''Bridges to Self'' | 10-word-compact | The custom TRC crawl added anti-bot handling, WebSocket frames, WebRTC APIs and Priv-Accept observations. {[vlummens2026_bridges]} |

The report output above is the primary quote log; no paper claim was added from a free-text tool tuple without a source quote.

===== External and industry sources =====

The external pass used primary DuckDuckGo repositories only. Each was fetched/read on 2026-08-14 and pinned below by the main-branch commit recorded in the bibliography. Claims about the output schema come from source files, not a README summary alone.

^ Source ^ Verification and load-bearing use ^
| [[https://github.com/duckduckgo/tracker-radar-collector|DuckDuckGo tracker-radar-collector]], commit ''8b64006691a1ce3929cfdfeeb425e7cc64be6543'' | Read the README/package metadata and the crawler, collector-list, request, cookie, API-call, cookie-popup, screenshot and metadata-file implementations. Verified Puppeteer/CDP architecture, CLI flags, top-level result shape, collector keys, request fields, cookie omission, API output and metadata fields. {[duckduckgo_trc_2026]} |
| [[https://github.com/duckduckgo/tracker-radar|DuckDuckGo tracker-radar]], commit ''a1d894db2312f3fdeea06d6c784739b97eb727c8'' | Read README, ''docs/DATA_MODEL.md'' and generated-data documentation. Verified that Tracker Radar is a behavior/domain/entity dataset rather than the Collector, and checked region files, prevalence, resources, cookie/fingerprinting/CNAME fields and generated maps. {[duckduckgo_tracker_radar_2026]} |
| [[https://github.com/duckduckgo/tracker-radar-detector|DuckDuckGo tracker-radar-detector]], commit ''0f6eb7db39bbf7cf7357db7279535941e1c0eb4b'' | Read README/configuration and the crawl-processing/build scripts. Verified that Detector reads site JSON/Postgres, processes requests/cookies/APIs, has ''minSites'' and first-party/CNAME options, aggregates prevalence and builds region tracker entries. {[duckduckgo_tracker_radar_detector_2026]} |

Rejected during the external pass:

  * SEO “best scraper” and “top web crawler” listicles: no source-level schema or Tracker Radar provenance, so they were not used for architecture, currency or recommendations.
  * Unverified vendor summaries and search-result snippets: useful for discovery at most, but not primary evidence for a field or a current maintenance claim.
  * A claim that TRC is the industry or research standard: rejected because the repositories show an active project, while the corpus has no matched comparison and only 10 identified research users under the narrow query.

===== What could not be established =====

  * The corpus contains no formal TRC paper and no same-sample, same-browser, same-question comparison of TRC with OpenWPM or plain Playwright. The page does not rank accuracy, speed, coverage or “best” tool.
  * The 10/1,120 figure is adoption in seven selected venues under an extraction query, not field prevalence, commercial usage or the share of all web measurement.
  * The 2025--2026 signal is provisional. It cannot support a complete end-of-year trend, especially for unheld or under-selected venue-years.
  * The public repositories document the pipeline and current source, but do not establish that every production Tracker Radar release used the public defaults or that the current schema is stable across commits.
  * The page does not infer cookie values, cookie set timing or request bodies from TRC's default output. The default CookieCollector is a final jar snapshot and the default request record has no raw bodies.
  * No independent benchmark was run. The content page's comparison is a source/schema comparison and a warning about denominators and capture surfaces.

===== Judgement calls =====

  * I created a new page because the sitemap/parent index already names this exact missing page, and the topic has a distinct implementation/data-pipeline question. I did not broaden or overwrite the OpenWPM page.
  * I counted only ''crawler-framework''/''browser-automation'' tuples marked used/produced. “Tracker Radar Entity Map” and similar names remain printed residue, not silent exclusions.
  * I describe TRC as current/available as of the 2026-08-14 repository check, but describe 2021--2024 use as historical and 2025--2026 use as a provisional signal. I did not call older OpenWPM-style methods superseded: the evidence shows different capture surfaces, not replacement.
  * I used paper-level counts for every published adoption figure. Raw tuple counts appear only in the provenance audit, where they expose name normalization.
  * I retained the broad 21 and excluded 11 only as a boundary check. Publishing 21 as TRC adoption would have conflated a dataset and entity map with a crawler.

===== Review log =====

The requested focused review categories were executed as checks by the primary run because no Agent/subagent capability was exposed in this environment. Findings and disposition:

^ Review pass ^ Finding ^ Disposition ^
| Figures vs script | Re-ran ''node scripts/report_tracker_radar_collector.mjs''; the 10/1,120 count, 21/11 boundary, 58 OpenWPM context count, year cells, percentages, folded names, residue and 2/8/0 quote-check totals match the verbatim output. | Accepted; no stale figure found. |
| Citations and quotes | Checked every content-page key against the shared live bibliography and checked all 10 extraction quotes in ''paper.cols.txt''. Eight are partial because repaired columns split the full sentence; none is unlocated. | Accepted; explained the partial status instead of presenting it as exact. No duplicate bibliography key added. |
| External currency | Re-read the three official repositories, current commits, package metadata, collector code, data model and Detector build scripts on 2026-08-14. | Accepted; version/commit caveats and no-supersession wording retained. SEO/listicle and unverified-vendor claims rejected. |
| Generic/page review | Checked that the page distinguishes Collector, Detector and Tracker Radar; gives a schema, comparison, build path, denominator, reporting checklist and limitations; and links the provenance and corpus pages. | Accepted. A suggestion to call TRC the field standard was rejected as unsupported. |

No credential, private crawl, raw cookie value or response body was exposed. The metadata schema's possible hostname/username fields are called out for redaction; no such values were published.

Post-review adjustment: the schema table gained an explicit ''elements'' row because the current type declares the optional key while the pinned collector entrypoint does not export an ElementsCollector. This changed no corpus figure; the report, quote block, citation-resolution and render checks were rerun.

===== Reproduction =====

<code bash>
cd /workspace/artifacts/wiki
node scripts/report_tracker_radar_collector.mjs
</code>

The report is deterministic against the mounted ''run1'' extraction. If the corpus changes, refresh the output, rerun the quote checks and update this log before changing a figure on the content page.

====== References ======

<bibtex bibliography></bibtex>