User Tools

Site Tools


programming:crawler:webxray

This is an old revision of the document!


webXray

webXray [1Libert, Timothy (2015): "Exposing the Invisible Web: An Analysis of Third-Party HTTP Requests on 1 Million Websites", International Journal of Communication 9. (Link)] is a third-party request and privacy-policy measurement tool by Timothy Libert, and it was one of the three hand-curated lists the pre-2022 literature used to put a company name next to a third-party domain — the other two being Disconnect and WhoTracks.me, which two corpus papers merge with it explicitly [2Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] [3Dambra, Savino; Sanchez-Rola, Iskander; Bilge, Leyla; Balzarotti, Davide (2022): "When Sally Met Trackers: Web Tracking From the Users' Perspective", in: Proceedings of the USENIX Security Symposium. (Link)]. Its distinguishing part was never the crawler: it was the domain-ownership database — 827 corporate owners, arranged in a parent/child tree, each carrying a purpose, a country, trade-body memberships and privacy-policy URLs in dozens of languages.

Two things a new measurement needs to know before citing it:

  • The tool is gone. github.com/timlib/webXray returns HTTP 404, there has never been a PyPI package, the surviving copies are dormant, and the licence forbids redistributing it. The author now runs a commercial product under the same name.
  • The database survives, but it is frozen at 2021 and badly stale wherever it has been checked. Of 28 high-prevalence ownership disagreements settled against primary sources on 2026-08-17, webXray names today's owner for 1 of the 23 domains it covers at all. Those 28 were selected because the lists disagreed on them, so that is not an error rate — but the direction is not in doubt. Tracker Radar and Disconnect are the live alternatives, and they are not equivalent to each other either.

This page is therefore two pages in one: what webXray was and why you cannot install it, and — the part you actually need — how domain-to-company ownership resolution works now, measured. If your question is “which crawler should I run”, go to Crawler; if it is “how do I decide which company a request went to”, you are in the right place, and the short answer is in Choosing a resolution source now with the pipeline around it in Assembling the pipeline. Everything between those and here is the evidence for them.

Do not cite webXray as the tool you used unless you really ran it. Cite [1Libert, Timothy (2015): "Exposing the Invisible Web: An Analysis of Third-Party HTTP Requests on 1 Million Websites", International Journal of Communication 9. (Link)] or [4Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)] for the method — third-party request measurement with corporate attribution, and automated privacy-policy auditing — and cite the ownership list separately from the crawler, because in the literature they came apart years ago: of 15 corpus papers that name webXray, 7 used only the ownership list and exactly one ran the crawler, and that one is Libert's own paper.

The tool: architecture, and why you cannot install it

The last public version (webXray 3.x, “forensic edition”) is not the PhantomJS tool the 2016 OpenWPM paper criticised [5Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]. It drives consumer Chrome over the DevTools protocol directlywebsocket-client, no Selenium and no Puppeteer — and has a client/server split for distributed crawling.1)

Piece What it did Comparable current tool
ChromeDriver.py raw CDP against consumer Chrome, headless by default, configurable dwell time, crawl depth and page-load strategy plain CDP, Puppeteer, Playwright
Collector.py / Client.py / Server.py distributed collection: many browsers, one database OpenWPM's process pool; a job queue of your own
OutputStore.py + SQLite/Postgres schemas requests, cookies, local storage, WebSockets, optionally full response bodies HAR or a CDP log into your own store
policyXray (resources/policyxray/) found and extracted privacy-policy and terms text in several languages, via Mozilla's Readability.js still largely unreplaced; see [4Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)], [6Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)]
Reporter.py / Analyzer.py pre-canned CSV reports, including aggregated_tracking_attribution.csv your own SQL
resources/domain_owners/domain_owners.json the ownership database — the part that outlived the tool Tracker Radar, Disconnect entities.json

Availability, every line checked on 2026-08-17:

Where a paper points you State today
github.com/timlib/webXray HTTP 404. The timlib account itself still exists (HTTP 200) with 0 public repositories.
github.com/timlib/webXray_Domain_Owner_List HTTP 404. This is the URL cited by [7Kashaf, Aqsa; Sekar, Vyas; Agarwal, Yuvraj (2020): "Analyzing Third Party Service Dependencies in Modern Web Services: Have We Learned from the Mirai-Dyn Incident?", in: Proceedings of the ACM Internet Measurement Conference. (DOI)], among others.
webxray.org HTTP 200, but a placeholder: a heading and the line “Public interest projects for the interested public.” No source link, no version, no download.
PyPI webxray / web-xray / policyxray 404 each. webXray was never packaged on PyPI.
github.com/thezedwards/webXray The most complete surviving copy of webXray 3.x, last commit 2021-03-04. Its README still instructs git clone https://github.com/timlib/webXray.git.
the 19 forks of that copy all dormant; the newest activity anywhere in the network is 2023-03-12, three small commits in a personal working copy.2)
github.com/RDBinns/webXray_Domain_Owner_List The ownership list split out as a standalone, GPL-3.0 repository, created and last pushed on 2018-04-05. Dormant since, and an older schema than the in-tool copy (owner_name rather than name, and no uses, platforms or trade_groups).
webxray.ai HTTP 200: a commercial litigation-support product (“Top US class action law firms and Fortune 100 in-house compliance teams use webXray to find actionable privacy violations first”). Libert's own homepage states “(Dr.) Timothy Libert is founder and CEO of webXray LLC.”3)

webXray is not open source, and redistributing it is prohibited. The surviving copy's LICENSE.md is the PolyForm Strict License 1.0.0, which grants use for any noncommercial purpose — explicitly including “public research organization” and “educational institution” — but grants no right to distribute the software or to make “changes or new works based on the software”. PolyForm's own summary is blunter: Strict “removes permission to distribute copies and make changes, leaving only permission to use for noncommercial purposes”. 1.0.0 is still the only version, and note that PolyForm Strict has no SPDX identifier — if your artefact metadata expects one, there is none to give.4) Its README says the same in plainer words: “This software is not open source, it is source available and licensed for non-commercial use only. You may not distribute webXray in whole or in part or sell data generated by webXray without prior written permission.”

The practical consequence is not academic. Upstream is a 404, so the only remaining route to the code and to the ownership file is a copy whose existence the licence does not permit — which is, in plain terms, what every figure on this page was computed from, and what [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] took out of the Internet Archive. So it is obtainable but not redistributable, and not installable: do not plan a study around it, and do not put “we used webXray” in an artefact appendix without deciding what you are asserting about the copy you used. Even if you obtain it, it no longer installs cleanly: requirements.txt pins lxml==4.6.2 and psycopg2-binary==2.8.6, and neither has a PyPI wheel beyond CPython 3.9 — which has been end-of-life since October 2025 — so pip install -r requirements.txt on a current interpreter falls back to source builds that need matching libxml2/libxslt and PostgreSQL headers.5) The 2018 standalone ownership list at RDBinns/webXray_Domain_Owner_List is a separate matter: its own README licenses it under GPLv3, so that snapshot can be used and redistributed — but it is the 2018 schema, not the 2021 file every figure on this page is measured from. Verify the licence of whatever file you actually download rather than assuming one licence covers the project.

The ownership database

domain_owners.json is a flat JSON array of owner records with an explicit tree. It is worth understanding in detail, because it is the most ambitious schema any of the three databases uses, and because most of that ambition is unfilled.

Field What it holds Filled, of 827 owners
id, name slug and display name all 827
parent_id the owner's parent company — the tree 319 (38.6%) have one
domains registrable domains attributed to this owner 3,215 in total; median 1 per owner, max 620 (groupm)
uses purpose vocabulary: marketing (484), hosting (102), audience_measurement (78), video (38), security (27), … 761 (92.0%); 38 distinct values, including two misspellings (content_reccomendation, accesibility)
platforms web (777), mobile (249), tv (176), iot (20), email (5) 803 (97.1%)
country ccTLD-style code: US (487), CN (114), UK (46), DE (36), … 36 distinct codes 826 (99.9%)
trade_groups self-regulatory membership: nai (85), daa (51), iab (36), … 104 (12.6%)
site_privacy_policy_urls etc. policy URLs tagged by ISO 639 language code — 69 languages, eng (554), chi (90), ger (81), fre (71) 591 (71.5%) have at least one
gdpr_statement_urls GDPR-specific statement 132 (16.0%)
ccpa_urls CCPA-specific page 4 (0.5%)
opt_out_urls opt-out mechanism 13 (1.6%)
crunchbase_id external company identifier 34 (4.1%)
health_segment_urls advertiser health-segment lists 52 (6.3%)
notes free-text reason for an attribution 263 (31.8%)
aliases alternate spellings, for matching 275 (33.3%)

Two features of this schema have no equivalent in Tracker Radar or Disconnect, and are the only reasons to still reach for this file: the per-language policy URLs (useful if your study needs a company's German privacy statement, and directly connected to policyXray), and the parent_id tree, which lets you resolve at the level your research question wants instead of the level the list happens to record.

That tree is not cosmetic. It runs up to six levels deep (508 owners at the root, 211 at depth 2, 80 at depth 3, 24 at depth 4, 2 each at depths 5 and 6), and for 2,175 of the 3,215 domains (67.7%) the immediate owner differs from the root of its tree. doubleclick.net is owned by “DoubleClick”, whose parent chain ends at “Alphabet”. Tracker Radar says “Google LLC” and Disconnect says “Google”. None of the three is wrong; they are answers to three different questions, and §“Two lists disagree” below measures what happens when you forget that.

The bundled Public Suffix List is frozen at 2016. webXray ships its own PSL copy plus a hand-written patch file whose header reads: “the public suffix list doesn't have most of the ccSLDs so I had to get them from wikipedia manually :-/ current as of 20160428”. The PSL's own maintainers say not to do this: “If you must use the PSL for this purpose, please do not bake static copies of the PSL into your software without update mechanisms that are frequently checking for updates and incorporating them.”6) The consequence was later measured: [9McQuistin, Stephen; Snyder, Peter; Perkins, Colin; Haddadi, Hamed; Tyson, Gareth (2023): "A First Look at the Privacy Harms of the Public Suffix List", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] lists timlib/webXray among the repositories shipping a fixed PSL version, in its “Production” category. If you reuse any of webXray's URL parsing, replace the suffix list.

How it compares to Tracker Radar and Disconnect

The three lists are not three attempts at the same artefact. They differ in size by more than an order of magnitude, in what a record means, and in what they are licensed for. A fourth live option, Ghostery's trackerdb, is not measured here: its ownership data is spread across per-company .eno files with a separate patterns layer rather than a single domain→owner map, so putting it in the same table would have meant writing a parser whose choices nobody could check against the other three. That is an omission, not a judgement — if you are choosing among the live lists, this page gives you numbers for two of the three.

webXray domain_owners.json DuckDuckGo Tracker Radar entity_map.json Disconnect entities.json
Owners / entities 827 19,148 1,887
Domains covered 3,215 38,368 7,850
Ownership hierarchy yesparent_id, up to 6 levels no — flat, though per-entity files carry an Owner field no — flat
Extra per-owner data purpose, country, trade bodies, per-language policy URLs displayName, aliases; prevalence, categories, fingerprinting and cookie behaviour in sibling files properties vs resources split; category in services.json
How ownership is decided hand curation. Libert's own paper calls the database “the product of years of detective work” [4Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)] “automatically generated” by Tracker Radar Detector; “new development and bug fixes, other than broken sites, are handled internally”7) Disconnect's own process page describes step 4 as “Connect every tracker to its parent entity through DNS, WHOIS, and behavioral evidence”8)
Update cadence none — frozen; last public commit 2021-03-04 monthly regeneration; last commit 2026-08-12, releases tagged 2026.07.27, 2026.06.08, … continuous; last commit 2026-08-07
Licence PolyForm Strict 1.0.0 (no redistribution). The 2018 standalone copy is GPL-3.0 CC BY-NC-SA 4.0 CC BY-NC-SA 4.0 (GPL-3.0 until 2020-06-24)
Documented method the 2018 paper, plus the list's own notes field docs/DATA_MODEL.md, docs/FAQ.md, a vendor blog post; no paper a six-step process page and inclusion criteria on disconnect.me; entities.json's own schema is undocumented
How to get a correction in nowhere — the repository is gone file an issue; the repo says non-breakage development is handled internally not by pull request. The README says “Pull requests are not reviewed and will be closed”; corrections go to evaluations@disconnect.me and must argue from “publicly available materials and technical information”9)

Both live lists are non-commercial. Tracker Radar and Disconnect's list are CC BY-NC-SA 4.0. That is fine for university research and share-alike publication of derived data; it is not fine for a spin-out, a consultancy deliverable, or an industry collaboration without a separate licence, and both vendors offer commercial terms on request. Check this before your data-availability statement, not after. Note also that Disconnect's licence was GPL-3.0 until 2020-06-24 — a paper that quotes the GPL terms is quoting a licence that no longer applies.10)

Coverage: how much of the third-party surface can each one name?

Size is the wrong comparison, because a domain you never meet costs you nothing. The right one is: of the third-party domains a crawl actually encounters, weighted by how often it encounters them, what share can this list attribute to a company?

The measurement below takes Tracker Radar's domain_summary.json as the universe of third-party domains and its prevalence field as the weight, folds each key to a registrable domain with the ICANN section of the current Public Suffix List, and asks each list for an owner. This universe is Tracker Radar's own view of the web, which flatters Tracker Radar and nobody else — read its column as “the list scored on its home ground” and the other two as measured against a denominator they had no part in choosing.

List Domains it can name an owner for Share of 32,369 Weighted by prevalence
webXray 669 2.1% 58.6%
Tracker Radar 5,581 17.2% 84.3%
Disconnect (propertiesresources) 2,268 7.0% 80.5%

The 2.1% and the 58.6% in the same row are the whole story of this file. webXray covers almost none of the web by domain count and well over half of it by weight, because it is a head list: the few hundred domains it knows are the ones on every page. Sliced by rank, the tail falls off a cliff:

Slice of the universe, by prevalence webXray Tracker Radar Disconnect
top 100 domains 71% 98% 94%
top 1,000 domains 26% 69% 68%
top 10,000 domains 5% 29% 17%

Libert said this himself in 2018, and the sentence is worth having to hand when a reviewer asks about coverage: “because webxray's database of domain ownership primarily contains major ad networks rather than small clients, and policyxray only searches for identified parties, variability in the long-tail of trackers may not have an outsized effect on overall findings related to disclosure. Nonetheless, it is important to point out that the number of parties being searched for is fewer than the total number of parties present.” [4Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)]

Three traps here, and the first one will silently ruin any coverage figure you compute:

  • The Public Suffix List has two sections, and using the wrong one manufactures a coverage hole. googleapis.com is a PRIVATE PSL rule (6,941 ICANN rules against 3,290 private ones in the snapshot used here). Fold with the private section and fonts.googleapis.com, ajax.googleapis.com and maps.googleapis.com each survive as their own “registrable domain”; look each up by exact key and every one comes back unowned — even though all three lists name googleapis.com → Google. fonts.googleapis.com alone carries prevalence 0.369, so this single mistake moves webXray's weighted coverage from 58.6% to 54.5% and Tracker Radar's from 84.3% to 79.3%. Fold on the ICANN section, or walk parent labels at lookup time, or both. The four combinations are printed side by side by the script and the spread between them is larger than the difference between the lists.
  • A large fraction of domain_summary.json's 47,836 rows are keyed by hostname, and how large depends on the fold. An ICANN-section fold merges 15,651 of them; the ICANN+PRIVATE fold merges only 2,339, because googleapis.com, s3.amazonaws.com and cloudfront.net are themselves private-section suffixes. (A label-count heuristic — “three or more labels” — gives 16,396 and answers neither question; do not use it.) All three ownership lists key on the registrable domain, so comparing coverage without folding at all charges webXray and Disconnect for subdomains they were never meant to hold.
  • 19 rows are not hostnames at all — 18 bracketed IPv6 literals and the literal string “null”, the last with a prevalence of 0.024 and a full behaviour profile attached. Drop them explicitly and say you did; do not let a null domain become a data point.

With the fold and the lookup done properly, 29,896 of the 32,369 domains (92.4%) have no owner in either webXray or Disconnect, but only 14.4% of the prevalence weight does. The most requested domain no list can name is tiktokw.us at prevalence 0.041 — Tracker Radar attributes it to ByteDance, the other two have nothing. That is what a real coverage hole looks like: recent, mid-tail, and concentrated in domains that appeared after the list was last curated.

Two lists disagree: is that an error, or a different question?

For every domain that two lists both cover, owner_dbs.py compares the owner strings after normalising punctuation and folding away legal-form suffixes (Inc, LLC, GmbH, S.A.S, …) but no synonyms — so the “agree” columns are a lower bound on real agreement and the last column an upper bound on real disagreement. The fold is deliberately timid: it leaves 782 of webXray's 827 owner names untouched, and of the 45 it does change only 16 lose a legal suffix — the other 29 change on punctuation alone (AT&T, JD.com, “Here, There & Everywhere”). If you want the agreement figures to go up, a synonym table is what you would have to add, and there is no principled one.

Pair Domains both cover Same after suffix fold One name contains the other Neither
webXray vs Tracker Radar 612 305 (49.8%) 109 (17.8%) 198 (32.4%)
webXray vs Disconnect 464 213 (45.9%) 34 (7.3%) 217 (46.8%)
Tracker Radar vs Disconnect 1,538 634 (41.2%) 320 (20.8%) 584 (38.0%)

Between a third and a half of jointly covered domains get different company names. Before concluding that some list is broken, note what happens if you resolve webXray up its ownership tree first — the obvious fix, since webXray says “DoubleClick” where the others say “Google”:

Pair Neither, using webXray's immediate owner Neither, using the root of webXray's tree
vs Tracker Radar 198 (32.4%) 239 (39.1%)
vs Disconnect 217 (46.8%) 230 (49.6%)

Resolving to the root makes agreement worse, and the reason is instructive. webXray's roots are holding companies and, worse, historical ones: google.com resolves to “Alphabet” (against “Google LLC” and “Google”), adnxs.com to “AT&T”, yahoo.com to “Verizon”, tapad.com to “Telenor”, turn.com to “Singtel”. Every one of those was true when the file was last curated and none is true now. So the disagreements decompose into two independent axes, and a paper has to state its position on both:

  • Granularity. Brand (DoubleClick, Bing, YouTube) / operating legal entity (Google LLC, Microsoft Corporation) / ultimate parent (Alphabet). webXray can give you any of the three; Tracker Radar gives the legal entity; Disconnect gives a compact grouping name. “Sites contacting Google” is a different number under each, and the difference is not small: bing.com, linkedin.com and adnxs.com all become Microsoft at the entity level.
  • Vintage. Which snapshot, and when was ownership last checked. Ad tech consolidates continuously — the same problem [10Selmo, Carlos; Carisimo, Esteban; Bustamante, Fabián E.; Alvarez-Hamelin, J. Ignacio (2025): "Learning AS-to-Organization Mappings with Borges", in: Proceedings of the 2025 ACM Internet Measurement Conference, pp. 120-133. (DOI)] names at the network layer as “an Internet shaped by constant mergers, rebrandings, and regional variation”.

Reading all three at once

The three files have three different shapes, and the shape is where the granularity choice becomes code. webXray is an array of owners each holding a domains list and a parent_id; Tracker Radar's domain_map.json is already keyed by domain; Disconnect's entities.json is keyed by entity, so it has to be inverted before you can look a domain up at all. Reading all three at once takes about twenty lines and immediately shows what you are choosing between:

owner_lookup.py
import json
 
WX = json.load(open('cache/webxray.json'))
TR = json.load(open('cache/tr_domain_map.json'))
DC = json.load(open('cache/disconnect_entities.json'))['entities']
 
# webXray: an array of owners, each with a `domains` list and a `parent_id`.
wx_by_id = {o['id']: o for o in WX}
wx = {d.lower(): o for o in WX for d in o['domains']}
 
def wx_chain(domain):
    o = wx.get(domain)
    if o is None:
        return None
    chain = [o['name']]
    seen = {o['id']}
    while o['parent_id'] is not None and o['parent_id'] not in seen:
        o = wx_by_id[o['parent_id']]
        seen.add(o['id'])
        chain.append(o['name'])
    return chain                      # brand first, ultimate parent last
 
# Tracker Radar: already keyed by domain.
tr = {d.lower(): v['entityName'] for d, v in TR.items()}
 
# Disconnect: keyed by ENTITY, so it has to be inverted. `properties` is the
# ownership claim; `resources` is what the tracker actually served from.
dc = {}
for name, v in DC.items():
    for d in v.get('resources', []):
        dc.setdefault(d.lower(), name)
for name, v in DC.items():
    for d in v.get('properties', []):
        dc[d.lower()] = name
 
for d in ['doubleclick.net', 'adnxs.com', 'facebook.net', 'fonts.googleapis.com']:
    print(f'{d:24} webXray={wx_chain(d)}  TR={tr.get(d)!r}  Disconnect={dc.get(d)!r}')

Its real output on the 2026-08-17 snapshots:

doubleclick.net          webXray=['DoubleClick', 'Google', 'Alphabet']  TR='Google LLC'  Disconnect='Google'
adnxs.com                webXray=['Xandr', 'AT&T']  TR='Microsoft Corporation'  Disconnect='Microsoft'
facebook.net             webXray=['Facebook']  TR='Facebook, Inc.'  Disconnect='Meta'
fonts.googleapis.com     webXray=None  TR=None  Disconnect=None

Four rows, four different lessons. doubleclick.net is the granularity axis, with nothing wrong anywhere. adnxs.com is the vintage axis, with webXray's whole chain superseded. facebook.net shows the two live lists disagreeing because one carries a five-year-old legal name. And fonts.googleapis.com comes back empty from all three — which is a bug in this snippet, not a coverage hole: every list names googleapis.com → Google, and the snippet looks up the exact key it was given. That is the PSL private-section trap above, reproduced in twenty lines. Fix it by trying each parent label:

def resolve(host, mapping):
    labels = host.split('.')
    for i in range(len(labels) - 1):          # stop at two labels
        candidate = '.'.join(labels[i:])
        if candidate in mapping:
            return mapping[candidate], candidate
    return None, None

with which fonts.googleapis.com resolves to Google in all three. Two further details worth copying: the seen set in wx_chain is not decoration — walk a hand-curated parent chain without a cycle guard and one bad edge hangs your pipeline; and Disconnect's resources are loaded before its properties so the ownership claim wins where the two disagree.

Which list is right, when they disagree?

30 of the highest-prevalence disagreements were adjudicated by hand against primary sources — company newsrooms, SEC filings, or the domain's own legal documents — on 2026-08-17. Every row and source is in scripts/owner_adjudication.py and on the provenance page.

Two of the 30 could not be settled from a primary source and are excluded from the table, leaving 28.

List Names today's owner Stale (a real former owner) Granularity only Outright error No entry
webXray 1 17 4 1 5
Tracker Radar 8 12 6 2 0
Disconnect 21 0 0 0 7

This is not an accuracy ranking, and it cannot be turned into one. The 28 rows were chosen because the lists disagreed on them, ranked by prevalence; a random sample would be dominated by domains all three get right, so no percentage taken from this table means anything about how often a list is correct in general. Two further columns are artefacts of how the sample was drawn rather than results: Tracker Radar's zero in “No entry” is partly because the rows were ranked by Tracker Radar's own prevalence field, and Disconnect's zero in “Stale” is over the 21 rows it covers at all, not over 28.

What the table does establish, and what a random sample would show less sharply, is the shape of the disagreement. It is concentrated in acquisitions and renames rather than spread across the list; it points overwhelmingly one way; and the list regenerated most conservatively is not the one that is most current. If you need an accuracy rate, you have to draw a stratified random sample and adjudicate it — nobody has, and this page is not a substitute.

Named examples, useful as regression tests for your own pipeline: adnxs.com (Xandr, AT&T → Microsoft, closed 2022-06-06); facebook.com (Facebook, Inc. → Meta Platforms, 2021-10-28 — Tracker Radar still says “Facebook, Inc.”); outbrain.com and teads.tv (Outbrain acquired Teads 2025-02-03 and then took its name, inverting the pair); postrelease.com (Nativo → Life360, completed 2026-01-05, which only Disconnect has); crwdcntrl.net (Lotame → Publicis, announced 2025-03-06). Three rows are errors rather than staleness: Tracker Radar attributes jsdelivr.net to “Prospect One”, jsDelivr's infrastructure contractor, where jsDelivr's own data-processing agreement names Volentio JSD Limited; Tracker Radar attributes stackadapt.com to “Collective Roll”, which is StackAdapt's own pre-2014 founding name and not a separate owner; and webXray's root for 1rx.io is “Marimedia”, which no primary source corroborates.

Where these figures come from, and how to redo them

Every figure in the two sections above comes from owner_dbs.py, which is published in full, with its unedited output, on the provenance page — 396 lines, and reading it is an audit task rather than a way to learn the topic. It fetches each list live, prints the SHA-256 prefix of the exact bytes it computed from, and prints the residue of every fold, so a figure quoted from it is pinned to a snapshot rather than to “the list”. Re-run it before citing any number here: two of the three lists move weekly.

Two rules in it are easy to get wrong and both change the answer. Do not use Tracker Radar's own entity_map.json or domain_map.json as the universe — that scores Tracker Radar at 100% by construction, where domain_summary.json is a crawl result and is defensible. And weight a registrable domain by the largest prevalence among its hostnames, never the sum, because one site can request both fonts.googleapis.com and ajax.googleapis.com and summing double-counts it.

Choosing a resolution source now

Dated, because a list of what the literature did is not advice about what to do:

Source Status, 2026-08-17 Use it when
Disconnect entities.json current; last commit 2026-08-07; CC BY-NC-SA 4.0; feeds Firefox's tracking protection via Mozilla's shavar-prod-lists11) and, per Disconnect, Microsoft Edge as well you want the most current owner name, and you can live with a sparser list and no hierarchy. Note that properties (5,843 domains) and resources (4,148) are different claims and 2,007 domains appear only in the second; the schema is not documented, so state which key you read. Do not quote Disconnect's own headline “14,332 verified domains and entity mappings” as the size of entities.json — that file holds 7,850 domains; count the file you read.12)
DuckDuckGo Tracker Radar entity_map.json / domain_map.json current; regenerated monthly, last commit 2026-08-12; CC BY-NC-SA 4.0 you want the broadest coverage, prevalence weights, or per-domain categories and fingerprinting scores in the same dataset. Expect legal-entity names, and expect renames to lag.
Ghostery trackerdb / WhoTracks.me current; trackerdb last commit 2026-08-06, CC BY-NC-SA 4.0 (its package.json says so explicitly; GitHub reports no SPDX id, so do not trust the API field); WhoTracks.me data repo last commit 2026-08-04 (“July update”); the site now redirects to ghostery.com/whotracksme you want an organizations + patterns model where one company can carry several independently categorised behaviours (Google Analytics separate from Google Tag Manager), or the [11Karaj, Arjaldo; Macbeth, Sam; Berson, Rémi; Pujol, Josep M. (2018): "WhoTracks.Me: Shedding light on the opaque world of online tracking". arXiv:1804.08959v2, revised 2019-04-25 (Link)] longitudinal data
webXray domain_owners.json historical; frozen at 2021-03-04, no update path, tool unobtainable reproducing or extending a pre-2022 result that used it, or you specifically need the parent_id tree or the per-language policy URLs and will re-verify each owner you rely on
WHOIS / RDAP current, but see the trap below as a fallback for domains no list covers, and only with privacy-proxy filtering
TLS certificates, DNS/SOA, CNAME chains current first-party CDN and sibling-domain detection, which the ownership lists are worst at: [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] needed it because “those lists frequently miss connections among two hostnames, e.g., twitch.tv and twitchcdn.net
Crunchbase live but paid: the v4 API returns HTTP 401 without a key you have institutional access; used this way by [12Yang, Zhiju; Yue, Chuan (2020): "A Comparative Measurement Study of Web Tracking on Mobile and Desktop Environments", in: Proceedings on Privacy Enhancing Technologies. (DOI)] and [6Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)]
LLM-based entity resolution emerging, and not yet at the domain layer. In this corpus the 2025 work is at the network layer: [10Selmo, Carlos; Carisimo, Esteban; Bustamante, Fabián E.; Alvarez-Hamelin, J. Ignacio (2025): "Learning AS-to-Organization Mappings with Borges", in: Proceedings of the 2025 ACM Internet Measurement Conference, pp. 120-133. (DOI)] maps AS numbers to organisations with an LLM and releases prompts and code; [13Gouda, Deepak; Dainotti, Alberto; Testart, Cecilia (2025): "Prefix2Org: Mapping BGP Prefixes to Organizations", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] maps BGP prefixes. No corpus paper applies this to third-party domain ownership you are willing to build and validate it yourself. The problem statement transfers directly; the validation burden does too

Do not use WHOIS as an ownership database without filtering privacy proxies. [12Yang, Zhiju; Yue, Chuan (2020): "A Comparative Measurement Study of Web Tracking on Mobile and Desktop Environments", in: Proceedings on Privacy Enhancing Technologies. (DOI)] resolved organisations for 411 of 762 mobile-specific trackers using CrunchBase, webXray's list, TLS certificates and WHOIS in that order — and its top-ten table of “organizations” contains Redacted For Privacy (34), Domains By Proxy (25), Whois Guard (14), Global Domain Privacy Services (8) and Whois Privacy (7). Those five registrar-privacy strings account for 88 trackers, more than the largest real company in the table. [2Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] built a deliberately conservative WHOIS-plus-graph pipeline instead, and still had to note that the registration ecosystem “does not aim at providing full transparency on the organizations behind each domain”.

“Tracker Radar” names three different artefacts, and citing the name tells a reader nothing. Of the 32 corpus papers that mention it, 11 used the ownership dataset, 9 used it as a tracker or category database, and 9 used Tracker Radar Collector — a Puppeteer crawler that carries no ownership data at all and has its own page, Tracker Radar Collector. A third artefact, Tracker Radar Detector, is the build pipeline. Name the artefact, the file and the commit.

Use in publications

Figures below are from the seven venues on corpus (CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P, 2010–2026), not from the web-measurement literature as a whole.

15 papers name webXray anywhere in their full text — 0.3% of the 5,859-paper corpus, and 1.3% of the 1,120 papers that ran a crawl. The extraction's tool field fires on only 7 of them, because most of the rest cite it in a related-work sentence or a reference list; both signals are reported rather than merged. Every verdict below was read off a quoted sentence.

Role webXray plays Papers Share of 15
used the ownership list only 7 46.7%
cited in related work or the reference list 4 26.7%
ran the crawler 1 6.7%
compared against, as a baseline 1 6.7%
measured as a subject 1 6.7%
miscitation — a different “WebXRay” 1 6.7%
Paper What it used
Englehardt & Narayanan, CCS 2016, Online Tracking [5Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] compared: “WebXray is a PhantomJS based tool for measuring HTTP traffic… measurements with a stripped-down browser have the potential to miss a large number of resource loads”
Libert, TheWebConf 2018, An Automated Approach to Auditing Disclosure [4Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)] the only corpus paper that ran webXray: crawler, ownership list and policyXray together, over the Alexa top one million in October 2017 — 938,093 pages loaded and 248,029 policy links extracted; 91.27% of loaded pages made a third-party request, 10.89 unique third-party domains per page
Sørensen & Kosta, TheWebConf 2019, Before and After GDPR [14Sørensen, Jannick Kirk; Kosta, Sokol (2019): "Before and After GDPR: The Changes in Third Party Presence at Public and Private European Websites", in: Proceedings of the ACM Web Conference. (DOI)] citation: webxray.org named as the crawler a related study used; this paper uses its own
Matte et al., IEEE S&P 2020, Do Cookie Banners Respect My Choice? [15Matte, Célestin; Bielova, Nataliia; Santos, Cristiana Teixeira (2020): "Do Cookie Banners Respect my Choice? Measuring Legal Compliance of Banners from IAB Europe's Transparency and Consent Framework", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] ownership list, to test whether a tracker's company appears in the IAB Global Vendor List. The one paper that pins the version: “WebXRay commit 04c3c8e8 (2019-06-18)”
Kashaf et al., IMC 2020, Third-Party Service Dependencies [7Kashaf, Aqsa; Sekar, Vyas; Agarwal, Yuvraj (2020): "Analyzing Third Party Service Dependencies in Modern Web Services: Have We Learned from the Mirai-Dyn Incident?", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] citation only (reference dated 2018-06-29); the paper builds its own TLD + certificate-SAN + SOA heuristic
Yang & Yue, PoPETs 2020, Web Tracking on Mobile and Desktop [12Yang, Zhiju; Yue, Chuan (2020): "A Comparative Measurement Study of Web Tracking on Mobile and Desktop Environments", in: Proceedings on Privacy Enhancing Technologies. (DOI)] ownership list (cited as “Tim Libert's library”), second of four sources tried in order: CrunchBase, then webXray, then TLS certificates, then WHOIS
Steffens et al., NDSS 2021, Who's Hosting the Block Party? [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] ownership list, retrieved from the Internet Archive because upstream was already hard to obtain
Sánchez-Rola et al., IEEE S&P 2021, Journey to the Center of the Cookie Ecosystem [2Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] ownership list, ranked third of three merged lists. The corpus files this paper under 2022 and the year table below follows the corpus; Crossref and the paper's own header put it at IEEE S&P 2021.
Musa & Nithyanand, PoPETs 2022, ATOM [16Musa, Maaz Bin; Nithyanand, Rishab (2022): "ATOM: Ad-network Tomography", in: Proceedings on Privacy Enhancing Technologies. (DOI)] ownership list, alongside WHOIS records and TLS certificates
Cassel et al., PoPETs 2022, OmniCrawl [17Cassel, Darion; Lin, Su-Chin; Buraggina, Alessio; Wang, William; Zhang, Andrew; Bauer, Lujo; Hsiao, Hsu-Chun; Jia, Limin; Libert, Timothy (2022): "OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers", in: Proceedings on Privacy Enhancing Technologies. (DOI)] ownership list, “to determine the provenance of the requests”
Kats et al., PoPETs 2022, Who Knows I Like Jelly Beans? [18Kats, Daniel; Silva, David Luz; Roturier, Johann (2022): "Who Knows I Like Jelly Beans? An Investigation Into Search Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)] citation: another study's webXray crawl of 22,484 adult websites
Dambra et al., USENIX Sec 2022, When Sally Met Trackers [3Dambra, Savino; Sanchez-Rola, Iskander; Bilge, Leyla; Balzarotti, Davide (2022): "When Sally Met Trackers: Web Tracking From the Users' Perspective", in: Proceedings of the USENIX Security Symposium. (Link)] ownership list, one of three merged lists
McQuistin et al., IMC 2023, Privacy Harms of the Public Suffix List [9McQuistin, Stephen; Snyder, Peter; Perkins, Colin; Haddadi, Hamed; Tyson, Gareth (2023): "A First Look at the Privacy Harms of the Public Suffix List", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] subject: timlib/webXray appears in its table of repositories shipping a frozen PSL
Jakaria et al., PoPETs 2024, Connecting the Dots [19Jakaria, Md; Huang, Danny Yuxing; Das, Anupam (2024): "Connecting the Dots: Tracing Data Endpoints in IoT Devices", in: Proceedings on Privacy Enhancing Technologies. (DOI)] miscitation — describes Libert's tool but its reference resolves to “Gilbert Held. 1998. Cinco Network's WebXRay”, an unrelated 1998 network-management product
Wu et al., NDSS 2025, Transparency or Information Overload? [20Wu, Xiaoyuan; Hu, Lydia; Zeng, Eric; Habib, Hana; Bauer, Lujo (2025): "Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] citation: “the webXray list includes information on domain ownership”

Three findings from that table matter more than the counts.

The tool and the database came apart, and then the database was superseded. The last corpus paper to use either was published in 2022. Ownership use of Tracker Radar begins in 2021 and rises through the corpus edge — [21Jannett, Louis; Mayer, Andreas; Westers, Maximilian; Mladenov, Vladislav; Mainka, Christian; Schwenk, Jörg (2026): "The State of Passkeys: Studying the Adoption and Security of Passkeys on the Web", in: Proceedings of the USENIX Security Symposium. (Link)] is a 2026 example, using the Entity Map to avoid treating gmail.com and google.com as separate authentication systems:

Year Tracker Radar used for ownership webXray crawler or ownership list
2018 0 1
2019 0 0
2020 0 2
2021 1 1
2022 0 4
2023 2 0
2024 1 0
2025* 4 0
2026* 3 0

The starred years are the provisional corpus edge — CCS 2026 and IMC 2026 have not been held, and IEEE S&P and TheWebConf 2026 are under-selected by construction — so do not read 2026 as a complete year. The counts are small enough that the crossover is a signal about direction, not a market share. Combined with the licence and 404 findings above, the corpus supports “webXray's ownership list is historical and Tracker Radar's is current practice”; it does not support any claim about which is more accurate, which is what §“Two lists disagree” is for.

Nobody says which snapshot they used. Of the 8 papers that used the crawler or the list, 2 say anything at all about which version, and 1 names a commit. That one, [15Matte, Célestin; Bielova, Nataliia; Santos, Cristiana Teixeira (2020): "Do Cookie Banners Respect my Choice? Measuring Legal Compliance of Banners from IAB Europe's Transparency and Consent Framework", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)], pins both lists it uses in a reproducibility table — “Disconnect list commit eb817fb1 (2019-12-10)” and “WebXRay commit 04c3c8e8 (2019-06-18)” — beside the Chromium build, the kernel, the user agent, the vantage point and the Tranco list id. Copy that table. The file has no version field and no release tags, so a commit or an archive URL is the only thing that makes the figure reproducible; owner_dbs.py prints a content hash for the same reason.

No list removes the manual work. [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] is the most honest account of the real cost. They needed same-entity relations for the Tranco top 10,000, found that the curated lists “frequently miss connections among two hostnames, e.g., twitch.tv and twitchcdn.net”, mined their own crawl for candidates, and hand-vetted 2,175 candidate site pairs down to 1,146 confirmed same-entity pairs in about eight person-hours. webXray's list then contributed 133 further relations their own method had missed, and their conclusion was that “it alone does not suffice for our purposes”. Two lessons: budget the person-hours, and treat the lists as additive rather than as alternatives. [2Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] took the same route from the other end — it merged Disconnect, WhoTracks.me and webXray by priority (“Disconnect first, WhoTracks.me second, and webxray third”, explicitly “preferring those that have been updated most recently”), added an automated WHOIS-and-graph pass “to increase the coverage”, found exactly one conflict against the 3,913 domains in any of the three lists, and filed a bug report that fixed an error in the Disconnect list.

How widespread is ownership resolution at all? A full-text sweep for the named resources gives 136 papers (2.3% of the corpus; 12.1% of those that crawled) that mention at least one. Each row is an upper bound — a match, not a verified use — except the two that were hand-verified in full:

Resource named in full text Papers Hand-verified?
Public Suffix List 101 no — upper bound
Disconnect, in a list/entity sense 74 no — upper bound
Tracker Radar (all three artefacts) 32 yes, all 32
WhoTracks.me 23 no — upper bound
Crunchbase 23 no — upper bound
webXray 15 yes, all 15

[22Utz, Christine; Amft, Sabrina; Degeling, Martin; Holz, Thorsten; Fahl, Sascha; Schaub, Florian (2023): "Privacy Rarely Considered: Exploring Considerations in the Adoption of Third-Party Services by Websites", Proceedings on Privacy Enhancing Technologies 2023(1):5-28. (DOI)] is worth reading before you pick one: it compared five third-party categorisations, including WhoTracks.me and Tracker Radar, and reports that “categorizations differ in granularity and focus” while overlapping substantially — the granularity axis again, stated from inside the literature.

Assembling the pipeline

The step most often missing from a paper is not the lookup, it is what surrounds it. To turn a request log into “N% of sites contact Google”:

  1. Extract the request host, and keep the visited site's host beside it.
  2. Fold both to a registrable domain using the ICANN section of a dated PSL. Not the private section — see the trap above.
  3. Resolve the request domain to an owner, walking parent labels rather than looking up the exact key, and record which key matched.
  4. Resolve the visited site to an owner too, and drop the request if they are the same owner. This is the step that is almost always silently skipped, and it is what the ownership list is for: [20Wu, Xiaoyuan; Hu, Lydia; Zeng, Eric; Habib, Hana; Bauer, Lujo (2025): "Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] describes webXray's list as “allowing distinction between first- and third-party domains”. Without it, google.com embedding gstatic.com counts as third-party tracking, and any site whose CDN is on a sibling domain is over-counted. eTLD+1 comparison alone does not do it — that is exactly why [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] needed eight person-hours of hand-vetting.
  5. Aggregate per site, not per request, and state the unit: “sites with at least one request to an owner” is not “requests”, is not “domains”, and is not “owners”.
  6. Report the unattributed remainder twice: as a share of domains and as a share of sites or requests. Those differ by more than an order of magnitude here.

Every one of those six steps is a place where two papers measuring “the same thing” diverge, and only the third is about which list you picked.

What to report in a paper

If a result depends on attributing domains to companies, report all of:

  • which list, which file, and which commit or content hash. “We used Tracker Radar” is not reproducible: name build-data/generated/entity_map.json at a commit, or publish the hash. For Disconnect, say whether you read properties, resources or the union.
  • the date the snapshot was taken, separately from the crawl date. Ownership changes between the two.
  • the granularity you resolved to — brand, operating legal entity, or ultimate parent — and, if you walked a hierarchy, how deep and how you handled cycles and missing parents.
  • the fallback order, if you merged sources, and the conflict rule. [2Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] and [12Yang, Zhiju; Yue, Chuan (2020): "A Comparative Measurement Study of Web Tracking on Mobile and Desktop Environments", in: Proceedings on Privacy Enhancing Technologies. (DOI)] both used strict priority orders and both said so; that is the standard to meet.
  • how many domains you could not attribute, as a count and as a share of requests or sites, not just of domains. The two differ enormously: a list can cover 2.1% of domains and 58.6% of prevalence weight. Say which denominator your unattributed share uses.
  • your privacy-proxy filter, if WHOIS was involved, and the list of proxy strings you removed.
  • the eTLD+1 rule and the PSL version, since every one of these lists keys on the registrable domain and the PSL changes. Do not ship a frozen copy [9McQuistin, Stephen; Snyder, Peter; Perkins, Colin; Haddadi, Hamed; Tyson, Gareth (2023): "A First Look at the Privacy Harms of the Public Suffix List", in: Proceedings of the ACM Internet Measurement Conference. (DOI)].
  • the manual verification you did, its sample size, and the disagreements you found. A prevalence-by-company figure with no manual check is a figure about a list, not about the web.

If the claim is “N% of sites contact Google”, the reader cannot evaluate it without the granularity and the snapshot: it silently includes or excludes YouTube, DoubleClick, Bing-adjacent Microsoft properties and whatever Alphabet has bought or sold since the file was written.

Methodology and limitations of these figures

  • The corpus audit script is scripts/report_webxray.mjs; the live-database comparison is scripts/owner_dbs.py; the hand adjudication with its primary sources is scripts/owner_adjudication.py. Every query, every unedited output, the fold residues, the sources rejected, and the figures deliberately not published are on the provenance page. Corpus-wide selection and extraction caveats are on corpus.
  • The webXray population is a full-text sweep, not the extraction's tool field, because the tool field finds 7 papers where the sweep finds 15 and the difference is exactly the citation-only cases this page needs to separate. All 15 carry a hand verdict with a quoted sentence; the residue between sweep and hand map is zero, and the script fails loudly if that changes.
  • The 26 literal per-paper figures and quotes on this page were checked against each paper's paper.cols.txt rendering: 26 of 26 located. The 16 extraction evidence quotes behind the tool and classification tuples were checked the same way: 4 exact, 11 partial, 1 below threshold; the below-threshold quote is present in the source and mangled by a column splice, not unsupported.
  • One figure was not published: [8Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)] contains the sentence “webXray's list does account for 1,096 of our 1,146 found connections meaning that it alone does not suffice for our purposes”, whose two halves contradict each other. The published PDF reads the same way, so no coverage percentage is taken from it; the unambiguous parts of that paragraph are quoted above instead.
  • The three-way database comparison uses a single snapshot per list, taken on 2026-08-17, with SHA-256 prefixes recorded on the provenance page. Two of the three lists change weekly. Re-run the script rather than quoting these numbers in 2027.
  • The coverage denominator is Tracker Radar's own crawl output. There is no neutral census of third-party domains, so this favours Tracker Radar; the page says so wherever a coverage figure appears rather than pretending otherwise.
  • The 30 adjudicated rows are the highest-prevalence disagreements, deliberately not a random sample, so they characterise the shape of disagreement and not any list's accuracy. Two rows (the acquisition date behind 360yield.com, and fwmrm.net's post-2026-spinoff status) could not be settled from a primary source and are recorded as unresolved rather than guessed.
  • Wayback Machine checks could not be completed: web.archive.org was returning 502/503 throughout 2026-08-17. So the date github.com/timlib/webXray first 404ed, and what webxray.org used to contain, are unknown and are not estimated here.
  • The seven-venue corpus omits EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS, so “15 papers name webXray” is a statement about those seven venues. webXray's own author published most of its results outside those seven venues — his homepage lists The BMJ, JAMA, Communications of the ACM and New Media and Society13) — so the corpus cannot see the work the tool's reputation mostly rests on, and “15 papers” is not a measure of how much webXray was used.
  • Crawler — generic automation libraries and the other specialised crawlers.
  • Tracker Radar Collector — the Puppeteer crawler that shares Tracker Radar's name and carries none of its ownership data.
  • OpenWPM — the instrument webXray was compared against in 2016, and the one still maintained.
  • Requests — filter lists, which answer “is this a tracker?” rather than “whose is it?”, and how the two get conflated.
  • Cookies — where cookie attribution needs an owner name, and what the same granularity choice does to it.
  • IP classification — the network-layer version of this problem, including AS-to-organisation mapping.
  • Traffic files — capturing the requests you are about to attribute.

References

[1]
Libert, Timothy (2015): "Exposing the Invisible Web: An Analysis of Third-Party HTTP Requests on 1 Million Websites", International Journal of Communication 9. (Link)
[2]
Sánchez-Rola, Iskander; Dell'Amico, Matteo; Balzarotti, Davide; Vervier, Pierre-Antoine; Bilge, Leyla (2021): "Journey to the Center of the Cookie Ecosystem: Unraveling Actors' Roles and Relationships", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)
[3]
Dambra, Savino; Sanchez-Rola, Iskander; Bilge, Leyla; Balzarotti, Davide (2022): "When Sally Met Trackers: Web Tracking From the Users' Perspective", in: Proceedings of the USENIX Security Symposium. (Link)
[4]
Libert, Timothy (2018): "An Automated Approach to Auditing Disclosure of Third-Party Data Collection in Website Privacy Policies", in: Proceedings of the ACM Web Conference. (DOI)
[5]
Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)
[6]
Cui, Hao; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2023): "PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs", in: Proceedings of the USENIX Security Symposium. (Link)
[7]
Kashaf, Aqsa; Sekar, Vyas; Agarwal, Yuvraj (2020): "Analyzing Third Party Service Dependencies in Modern Web Services: Have We Learned from the Mirai-Dyn Incident?", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
[8]
Steffens, Marius; Musch, Marius; Johns, Martin; Stock, Ben (2021): "Who’s Hosting the Block Party? Studying Third-Party Blockage of CSP and SRI", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
[9]
McQuistin, Stephen; Snyder, Peter; Perkins, Colin; Haddadi, Hamed; Tyson, Gareth (2023): "A First Look at the Privacy Harms of the Public Suffix List", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
[10]
Selmo, Carlos; Carisimo, Esteban; Bustamante, Fabián E.; Alvarez-Hamelin, J. Ignacio (2025): "Learning AS-to-Organization Mappings with Borges", in: Proceedings of the 2025 ACM Internet Measurement Conference, pp. 120-133. (DOI)
[11]
Karaj, Arjaldo; Macbeth, Sam; Berson, Rémi; Pujol, Josep M. (2018): "WhoTracks.Me: Shedding light on the opaque world of online tracking". arXiv:1804.08959v2, revised 2019-04-25 (Link)
[12]
Yang, Zhiju; Yue, Chuan (2020): "A Comparative Measurement Study of Web Tracking on Mobile and Desktop Environments", in: Proceedings on Privacy Enhancing Technologies. (DOI)
[13]
Gouda, Deepak; Dainotti, Alberto; Testart, Cecilia (2025): "Prefix2Org: Mapping BGP Prefixes to Organizations", in: Proceedings of the ACM Internet Measurement Conference. (DOI)
[14]
Sørensen, Jannick Kirk; Kosta, Sokol (2019): "Before and After GDPR: The Changes in Third Party Presence at Public and Private European Websites", in: Proceedings of the ACM Web Conference. (DOI)
[15]
Matte, Célestin; Bielova, Nataliia; Santos, Cristiana Teixeira (2020): "Do Cookie Banners Respect my Choice? Measuring Legal Compliance of Banners from IAB Europe's Transparency and Consent Framework", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)
[16]
Musa, Maaz Bin; Nithyanand, Rishab (2022): "ATOM: Ad-network Tomography", in: Proceedings on Privacy Enhancing Technologies. (DOI)
[17]
Cassel, Darion; Lin, Su-Chin; Buraggina, Alessio; Wang, William; Zhang, Andrew; Bauer, Lujo; Hsiao, Hsu-Chun; Jia, Limin; Libert, Timothy (2022): "OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers", in: Proceedings on Privacy Enhancing Technologies. (DOI)
[18]
Kats, Daniel; Silva, David Luz; Roturier, Johann (2022): "Who Knows I Like Jelly Beans? An Investigation Into Search Privacy", in: Proceedings on Privacy Enhancing Technologies. (DOI)
[19]
Jakaria, Md; Huang, Danny Yuxing; Das, Anupam (2024): "Connecting the Dots: Tracing Data Endpoints in IoT Devices", in: Proceedings on Privacy Enhancing Technologies. (DOI)
[20]
Wu, Xiaoyuan; Hu, Lydia; Zeng, Eric; Habib, Hana; Bauer, Lujo (2025): "Transparency or Information Overload? Evaluating Users’ Comprehension and Perceptions of the iOS App Privacy Report", in: Proceedings of the Network and Distributed System Security Symposium. (Link)
[21]
Jannett, Louis; Mayer, Andreas; Westers, Maximilian; Mladenov, Vladislav; Mainka, Christian; Schwenk, Jörg (2026): "The State of Passkeys: Studying the Adoption and Security of Passkeys on the Web", in: Proceedings of the USENIX Security Symposium. (Link)
[22]
Utz, Christine; Amft, Sabrina; Degeling, Martin; Holz, Thorsten; Fahl, Sascha; Schaub, Florian (2023): "Privacy Rarely Considered: Exploring Considerations in the Adoption of Third-Party Services by Websites", Proceedings on Privacy Enhancing Technologies 2023(1):5-28. (DOI)
1)
Read out of the surviving copy on 2026-08-17: webxray/ChromeDriver.py imports create_connection from websocket; requirements.txt pins only lxml==4.6.2, psycopg2-binary==2.8.6, textstat==0.7.0 and websocket-client==0.57.0; webxray/Client.py and webxray/Server.py implement the distributed mode.
2)
api.github.com/repos/thezedwards/webXray/forks?per_page=100, checked 2026-08-17. Most recent by pushed_at: peterjoles/webXray 2023-03-12.
3)
https://webxray.ai/ and https://timlibert.me/, both fetched 2026-08-17. The timlib GitHub profile lists company: webXray.ai.
4)
https://polyformproject.org/licenses and …/licenses/strict/1.0.0, checked 2026-08-17: strict/1.0.0 is the only Strict version listed. SPDX's licence list carries PolyForm-Noncommercial-1.0.0 and PolyForm-Small-Business-1.0.0 but no Strict entry, checked against spdx/license-list-data on the same date.
5)
Checked on PyPI, 2026-08-17: the newest wheels for lxml 4.6.2 and psycopg2-binary 2.8.6 are cp39. websocket-client 0.57.0 and textstat 0.7.0 ship universal/py3 wheels and are fine.
6)
https://publicsuffix.org/learn/, checked 2026-08-17.
7)
docs/DATA_MODEL.md in duckduckgo/tracker-radar, checked 2026-08-17.
10)
Verified from the commit history of LICENSE in disconnectme/disconnect-tracking-protection: GPLv3 in the initial commit 89d421e8 (2015-10-13), rewritten to CC BY-NC-SA 4.0 by two commits on 2020-06-24. Checked 2026-08-17.
11)
mozilla-services/shavar-prod-lists README, checked 2026-08-17: “Firefox's Enhanced Tracking Protection features rely on lists of trackers maintained by Disconnect… Mozilla does not maintain these lists”, and disconnect-blacklist.json is “a version controlled copy of Disconnect's list of trackers”.
12)
https://disconnect.me/trackerprotection, checked 2026-08-17, states “14,332 Verified domains and entity mappings” and that the lists “power Microsoft Edge, Mozilla Firefox, other partners”. The 7,850 figure is measured from entities.json by owner_dbs.py.
13)
https://timlibert.me/, checked 2026-08-17.
You could leave a comment if you were logged in.
programming/crawler/webxray.1786954197.txt.gz · Last modified: by karel.kubicek.claude

Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki