| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:programming:crawler:webxray [2026/08/17 07:58] – Add the post-publication reproducibility check (all three scripts byte-identical; owner_dbs.py re-verified against a fresh live fetch) and the bibtex cache-purge gotcha. Authored by Claude. karel.kubicek.claude | provenance:programming:crawler:webxray [2026/08/17 08:13] (current) – Record the relicensing correction and the fifth reviewer's log: how the licence claim came to rest on an unrepresentative snapshot, and the two most serious defects being invisible to every automated guard here. Authored by Claude. karel.kubicek.claude |
|---|
| | B2 | How deep is webXray's tree? | walk ''parent_id'' with a cycle guard | 319 of 827 have a parent; depths 1:508 2:211 3:80 4:24 5:2 6:2; **2,175 of 3,215 domains (67.7%)** have leaf ≠ root | | | B2 | How deep is webXray's tree? | walk ''parent_id'' with a cycle guard | 319 of 827 have a parent; depths 1:508 2:211 3:80 4:24 5:2 6:2; **2,175 of 3,215 domains (67.7%)** have leaf ≠ root | |
| | B3 | How much of webXray's schema is filled? | per-field non-empty count | ''country'' 826 (99.9%), ''platforms'' 803 (97.1%), ''uses'' 761 (92.0%), site policy URLs 591 (71.5%), GDPR 132 (16.0%), ''trade_groups'' 104 (12.6%), ''crunchbase_id'' 34 (4.1%), ''opt_out_urls'' 13 (1.6%), ''ccpa_urls'' **4 (0.5%)** | | | B3 | How much of webXray's schema is filled? | per-field non-empty count | ''country'' 826 (99.9%), ''platforms'' 803 (97.1%), ''uses'' 761 (92.0%), site policy URLs 591 (71.5%), GDPR 132 (16.0%), ''trade_groups'' 104 (12.6%), ''crunchbase_id'' 34 (4.1%), ''opt_out_urls'' 13 (1.6%), ''ccpa_urls'' **4 (0.5%)** | |
| | B4 | What is the universe of third-party domains? | Tracker Radar ''domain_summary.json'', folded to eTLD+1 with the live PSL | 47,836 rows → **45,525** registrable domains; 16,396 rows were hostname-keyed; 19 rows dropped as not hostnames | | | B4 | What is the universe of third-party domains? | Tracker Radar ''domain_summary.json'', folded to eTLD+1 with the **ICANN section** of the live PSL | 47,836 rows → **32,369** registrable domains; the ICANN fold merges **15,651** rows, the ICANN+PRIVATE fold only 2,339; 19 rows dropped as not hostnames | |
| | B5 | Coverage, unweighted | domains in the universe the list can name | webXray 657 (1.4%), Tracker Radar 5,539 (12.2%), Disconnect 2,277 (5.0%) | | | B5 | Coverage, unweighted | domains in the universe the list can name, parent labels walked | webXray 669 (2.1%), Tracker Radar 5,581 (17.2%), Disconnect 2,268 (7.0%) | |
| | B6 | Coverage, prevalence-weighted | Σ prevalence of covered ÷ Σ prevalence of all | webXray **54.5%**, Tracker Radar **79.3%**, Disconnect **75.8%** | | | B6 | Coverage, prevalence-weighted | Σ prevalence of covered ÷ Σ prevalence of all | webXray **58.6%**, Tracker Radar **84.3%**, Disconnect **80.5%** | |
| | B7 | Head versus tail | coverage of the top 100 / 1,000 / 10,000 by prevalence | 70/26/5, 95/67/27, 91/67/16 | | | B6b | How much does the fold/lookup rule move it? | all four combinations of ICANN-vs-full fold and walk-vs-exact lookup | weighted coverage spans 54.5–59.7% (webXray), 79.3–84.8% (TR), 75.8–80.6% (Disconnect). **The rule moves the answer by more than the lists differ.** | |
| | B8 | Pairwise agreement | owner strings after a legal-suffix fold | disagreement 32.8% / 46.9% / 38.1% (see the page table) | | | B7 | Head versus tail | coverage of the top 100 / 1,000 / 10,000 by prevalence | 71/26/5, 98/69/29, 94/68/17 | |
| | B9 | Does resolving webXray's tree help? | repeat B8 with root owners | disagreement **rises** to 39.3% and 49.5% | | | B8 | Pairwise agreement | owner strings after a legal-suffix fold | disagreement 32.4% / 46.8% / 38.0% (see the page table) | |
| | B10 | What does nobody cover? | universe minus webXray minus Disconnect | 43,052 (94.6%) of domains, 19.3% of prevalence weight; the top row is ''fonts.googleapis.com'' at 0.369, which **no** list names | | | B9 | Does resolving webXray's tree help? | repeat B8 with root owners | disagreement **rises** to 39.1% and 49.6% | |
| | | B10 | What does nobody cover? | universe minus webXray minus Disconnect | 29,896 (92.4%) of domains, 14.4% of prevalence weight; the top row is ''tiktokw.us'' at 0.041, which only Tracker Radar names | |
| |
| ==== B.11 Folding: what was folded, the rule, and the residue ==== | ==== B.11 Folding: what was folded, the rule, and the residue ==== |
| Deliberately **not** folded: ''group'', ''media'', ''technologies'', ''digital'', ''networks'', ''solutions''. Each is part of a real company name often enough ("Almondnet Group", "Zeta Global", "Lotame Solutions") that stripping it would manufacture agreement between lists that name different companies. Also **no synonym merging at all**: "Facebook" is not folded to "Meta", "DoubleClick" is not folded to "Google". That is the whole point — those are the disagreements being measured. | Deliberately **not** folded: ''group'', ''media'', ''technologies'', ''digital'', ''networks'', ''solutions''. Each is part of a real company name often enough ("Almondnet Group", "Zeta Global", "Lotame Solutions") that stripping it would manufacture agreement between lists that name different companies. Also **no synonym merging at all**: "Facebook" is not folded to "Meta", "DoubleClick" is not folded to "Google". That is the whole point — those are the disagreements being measured. |
| |
| **Residue, printed in full by the script:** the fold leaves **782 of webXray's 827 owner names untouched (94.6%)**, i.e. it fires on 45. That is the intended behaviour and its consequence is stated on the page: every "agree" figure is a **lower bound** on real agreement and every "disagree" figure an **upper bound** on real disagreement. A synonym-merging fold would move the numbers in a direction the script cannot justify, so it was not written. | **Residue, printed in full by the script:** the fold leaves **782 of webXray's 827 owner names untouched (94.6%)**, i.e. it changes 45 — but only **16** of those 45 lose a legal-form suffix. The other 29 change because ''norm_name()'' strips punctuation before the suffix regex runs (''AT&T'', ''56.com'', ''JD.com'', ''Dun & Bradstreet'', "Here, There & Everywhere"). The first version of this page attributed all 45 to the suffix fold, overstating its reach by nearly 3×; the script now prints the two counts separately and lists the punctuation-only names. That is the intended behaviour and its consequence is stated on the page: every "agree" figure is a **lower bound** on real agreement and every "disagree" figure an **upper bound** on real disagreement. A synonym-merging fold would move the numbers in a direction the script cannot justify, so it was not written. |
| |
| The second normalisation is the **PSL fold** on the universe (B4). Its residue is printed as the 19 dropped rows, listed individually: 18 bracketed IPv6 literals and the literal string ''"null"'' with prevalence 0.024 and a full behaviour profile attached. A row that is not a hostname is not a domain; it is printed rather than silently dropped. | The second normalisation is the **PSL fold** on the universe (B4). Its residue is printed as the 19 dropped rows, listed individually: 18 bracketed IPv6 literals and the literal string ''"null"'' with prevalence 0.024 and a full behaviour profile attached. A row that is not a hostname is not a domain; it is printed rather than silently dropped. |
| ==== B.12 Two denominator choices that change the answer ==== | ==== B.12 Two denominator choices that change the answer ==== |
| |
| | * **The PSL's ICANN/PRIVATE split, and exact-key versus parent-label lookup.** This is the choice that moves the answer most, and the first version of this script got it wrong in both halves at once — see the reviewer log below. ''googleapis.com'' is a PRIVATE rule, so folding with the private section leaves ''fonts.googleapis.com'' (prevalence 0.369) standing as its own registrable domain, and an exact-key lookup then reports it unowned although all three lists name ''googleapis.com''. The script now prints all four combinations; the recommended rule is an ICANN-section fold, and with that fold every key already //is// a registrable domain so the parent walk is a no-op — which is the cleanest possible confirmation that the fold, not the lookup, is the fix. |
| * **Do not use Tracker Radar's ''entity_map.json'' or ''domain_map.json'' as the universe.** That is Tracker Radar's answer key and scores it at 100% by construction. ''domain_summary.json'' is a crawl **result** and is a defensible universe — while still Tracker Radar's own view of the web, which the page states wherever a coverage figure appears. There is no neutral census of third-party domains; if one existed this comparison would use it. | * **Do not use Tracker Radar's ''entity_map.json'' or ''domain_map.json'' as the universe.** That is Tracker Radar's answer key and scores it at 100% by construction. ''domain_summary.json'' is a crawl **result** and is a defensible universe — while still Tracker Radar's own view of the web, which the page states wherever a coverage figure appears. There is no neutral census of third-party domains; if one existed this comparison would use it. |
| * **A registrable domain's weight is the //maximum// prevalence among its hostnames, never the sum.** One site can request ''fonts.googleapis.com'' and ''ajax.googleapis.com'', so summing double-counts sites and can exceed 1. Using the max makes every weighted figure a conservative lower bound, and the script says so in a comment at the line that does it. | * **A registrable domain's weight is the //maximum// prevalence among its hostnames, never the sum.** One site can request ''fonts.googleapis.com'' and ''ajax.googleapis.com'', so summing double-counts sites and can exceed 1. Using the max makes every weighted figure a conservative lower bound, and the script says so in a comment at the line that does it. |
| ''scripts/owner_adjudication.py'' holds 30 rows: domain, the owner a primary source confirms for today, the date ownership changed, the source URL, and one verdict per list from ''{current, stale, granularity, error, absent}''. | ''scripts/owner_adjudication.py'' holds 30 rows: domain, the owner a primary source confirms for today, the date ownership changed, the source URL, and one verdict per list from ''{current, stale, granularity, error, absent}''. |
| |
| **Sourcing bar:** a company newsroom or press release, an SEC filing, or the domain's own legal document (privacy policy, imprint, data-processing agreement). Wikipedia, Crunchbase summaries, PitchBook, Tracxn and "list of ad-tech acquisitions" pages were used **only to locate leads** and never cited. Where only trade press could be found, the row is marked ''UNRESOLVED'' rather than asserted — 2 of 30. | **Sourcing bar:** a company newsroom or press release, an SEC filing, or the domain's own legal document (privacy policy, imprint, data-processing agreement). Wikipedia, Crunchbase summaries, PitchBook, Tracxn and "list of ad-tech acquisitions" pages were used **only to locate leads** and never cited. Where only trade press could be found the row is marked ''UNRESOLVED'' and is **excluded from every tally** — 2 of 30, leaving 28. That exclusion was not in the first version, and it mattered: 2 of webXray's 3 "current" verdicts were exactly those two rows, so the figure most favourable to webXray rested on the evidence the script itself called insufficient. Corrected tallies: webXray current on **1 of 23** entries, Tracker Radar **8 of 28**, Disconnect **21 of 21**. |
| |
| **The selection bias, stated because it is the main limitation:** these are the disagreements with the **highest Tracker Radar prevalence**, not a random sample. They are chosen precisely where the lists differ, so the tallies are **not an error rate** for any list — a random sample would be dominated by domains all three get right. What they do establish is the //shape// of the disagreement: concentrated in acquisitions and renames, pointing overwhelmingly one way, with the most conservatively regenerated list being the least current. The script prints this warning above its own tallies and the page repeats it in a ''%%<WRAP important>%%''. | **The selection bias, stated because it is the main limitation:** these are the disagreements with the **highest Tracker Radar prevalence**, not a random sample. They are chosen precisely where the lists differ, so the tallies are **not an error rate** for any list — a random sample would be dominated by domains all three get right. What they do establish is the //shape// of the disagreement: concentrated in acquisitions and renames, pointing overwhelmingly one way, with the most conservatively regenerated list being the least current. The script prints this warning above its own tallies and the page repeats it in a ''%%<WRAP important>%%''. |
| | ''thezedwards/webXray'' last pushed 2021-03-04, 19 forks, newest fork activity 2023-03-12 | GitHub API repo + ''/forks?per_page=100'' | | | ''thezedwards/webXray'' last pushed 2021-03-04, 19 forks, newest fork activity 2023-03-12 | GitHub API repo + ''/forks?per_page=100'' | |
| | the surviving README still says ''git clone https://github.com/timlib/webXray.git'' | fetched ''raw.githubusercontent.com/thezedwards/webXray/master/README.md'' | | | the surviving README still says ''git clone https://github.com/timlib/webXray.git'' | fetched ''raw.githubusercontent.com/thezedwards/webXray/master/README.md'' | |
| | PolyForm Strict License 1.0.0, no redistribution, noncommercial permitted incl. research organisations | fetched ''LICENSE.md'' from the same repo and read the licence text itself, not a summary | | | PolyForm Strict License 1.0.0 on the ''thezedwards'' snapshot | fetched ''LICENSE.md'' from that repo and read the licence text itself, not a summary. **This is not webXray's final licence** — see the next row | |
| | | webXray was relicensed to GPLv3 (2021-06-14, commit ''245ec5d7'') and then to **MIT** (2023-02-01, commit ''73fe0fc9'', authored by "Tim Libert") | fetched ''api.github.com/repos/peterjoles/webXray'' (''spdx_id: MIT''), its ''LICENSE'' ("Copyright (c) 2023 Tim Libert"), the ''commits?path=LICENSE'' history, and ''compare/master...peterjoles:master'' (''ahead_by: 36''). The page's first version asserted webXray "is not open source and redistributing it is prohibited" from the ''thezedwards'' snapshot alone, which was the most restrictively licensed copy in existence — the exact mistake the page tells readers to avoid | |
| | PolyForm Strict 1.0.0 is still the current version and has no SPDX identifier | ''polyformproject.org/licenses'' lists ''strict/1.0.0'' and no later Strict version; SPDX's own ''license-list-data'' JSON carries only ''PolyForm-Noncommercial-1.0.0'' and ''PolyForm-Small-Business-1.0.0'' | | | PolyForm Strict 1.0.0 is still the current version and has no SPDX identifier | ''polyformproject.org/licenses'' lists ''strict/1.0.0'' and no later Strict version; SPDX's own ''license-list-data'' JSON carries only ''PolyForm-Noncommercial-1.0.0'' and ''PolyForm-Small-Business-1.0.0'' | |
| | raw CDP, no Selenium | ''webxray/ChromeDriver.py'' imports ''create_connection'' from ''websocket''; ''requirements.txt'' pins ''lxml==4.6.2'', ''psycopg2-binary==2.8.6'', ''textstat==0.7.0'', ''websocket-client==0.57.0'' | | | raw CDP, no Selenium | ''webxray/ChromeDriver.py'' imports ''create_connection'' from ''websocket''; ''requirements.txt'' pins ''lxml==4.6.2'', ''psycopg2-binary==2.8.6'', ''textstat==0.7.0'', ''websocket-client==0.57.0'' | |
| | Disconnect last commit 2026-08-07, CC BY-NC-SA 4.0, **GPL-3.0 until 2020-06-24** | GitHub API; then the commit history of ''LICENSE'' — GPLv3 in the initial commit ''89d421e8'' (2015-10-13), rewritten to CC BY-NC-SA by two commits on 2020-06-24 | | | Disconnect last commit 2026-08-07, CC BY-NC-SA 4.0, **GPL-3.0 until 2020-06-24** | GitHub API; then the commit history of ''LICENSE'' — GPLv3 in the initial commit ''89d421e8'' (2015-10-13), rewritten to CC BY-NC-SA by two commits on 2020-06-24 | |
| | Firefox ships Disconnect's ''services.json'' via Mozilla's ''shavar-prod-lists'' | ''mozilla-services/shavar-prod-lists'' README: Firefox's ETP "rely on lists of trackers maintained by Disconnect… Mozilla does not maintain these lists"; ''disconnect-blacklist.json'' is "a version controlled copy of Disconnect's list of trackers" | | | Firefox ships Disconnect's ''services.json'' via Mozilla's ''shavar-prod-lists'' | ''mozilla-services/shavar-prod-lists'' README: Firefox's ETP "rely on lists of trackers maintained by Disconnect… Mozilla does not maintain these lists"; ''disconnect-blacklist.json'' is "a version controlled copy of Disconnect's list of trackers" | |
| | Ghostery ''trackerdb'' 2026-08-06 (MIT); WhoTracks.me data repo 2026-08-04 ("July update"); ''whotracks.me'' now redirects to ''ghostery.com/whotracksme'' | GitHub API on ''ghostery/trackerdb'' and ''whotracksme/whotracks.me'' (the ''ghostery/whotracks.me'' path 301s); followed the site redirect | | | Ghostery ''trackerdb'' 2026-08-06, **CC BY-NC-SA 4.0**; WhoTracks.me data repo 2026-08-04 ("July update"); ''whotracks.me'' now redirects to ''ghostery.com/whotracksme'' | GitHub API on ''ghostery/trackerdb'' and ''whotracksme/whotracks.me'' (the ''ghostery/whotracks.me'' path 301s); followed the site redirect. The licence is from the repo's own ''LICENSE'' and ''package.json'' (''"license": "CC-BY-NC-SA-4.0"''), **not** the GitHub API, which returns no SPDX id — a first pass read "MIT" off the API and was wrong | |
| | WhoTracks.me paper is arXiv 1804.08959, v2 revised 2019-04-25 | ''export.arxiv.org/api/query?id_list=1804.08959'' — title, authors and ''updated'' field read from the Atom response | | | WhoTracks.me paper is arXiv 1804.08959, v2 revised 2019-04-25 | ''export.arxiv.org/api/query?id_list=1804.08959'' — title, authors and ''updated'' field read from the Atom response | |
| | Crunchbase API is paid | ''api.crunchbase.com/api/v4/…'' → HTTP 401 ''"Unauthorized user_key"'' | | | Crunchbase API is paid | ''api.crunchbase.com/api/v4/…'' → HTTP 401 ''"Unauthorized user_key"'' | |
| * **The ''1,096 of 1,146'' sentence in {[steffens2021_blockparty]}** — see A.17. Unresolvable from the text. | * **The ''1,096 of 1,146'' sentence in {[steffens2021_blockparty]}** — see A.17. Unresolvable from the text. |
| * **The list-age and missing-hostname columns for ''timlib/webXray'' in {[mcquistin2023_psl]}'s table.** The ''.cols'' rendering splits that table into separate column runs and the numeric columns cannot be aligned to the repository names reliably. Only the verifiable part is on the page — that webXray appears in the table, in the "Production" category, with 27 stars. **What would close it:** read the published PDF's table directly. | * **The list-age and missing-hostname columns for ''timlib/webXray'' in {[mcquistin2023_psl]}'s table.** The ''.cols'' rendering splits that table into separate column runs and the numeric columns cannot be aligned to the repository names reliably. Only the verifiable part is on the page — that webXray appears in the table, in the "Production" category, with 27 stars. **What would close it:** read the published PDF's table directly. |
| * **A random-sample accuracy rate for any of the three lists.** Would need a hand-adjudicated random sample of a few hundred domains, i.e. tens of hours at the rate section C ran at. Worth doing and not done; the page is explicit that its 30 rows cannot substitute. | * **Whether any //non-numeric// claim on this page has drifted from its script.** ''check_page_numbers.mjs'' compares numerals only, so when a reviewer's fix changed "third of four" to "second of four" on the page, the hand map in the script and its committed output kept saying "third" and every guard still passed. The same happened with "MIT" for Ghostery's licence. Both were caught by a human reviewer reading the two side by side, not by a tool, and no tool here covers that class. ''check_attributions.mjs'' closes one narrow case of it (author names). **What would close the rest:** a guard that extracts the quoted strings a page attributes to a script's hand map and diffs them against the map. |
| | * **A random-sample accuracy rate for any of the three lists.** Would need a hand-adjudicated random sample of a few hundred domains, stratified by prevalence decile so the head does not swamp it, i.e. tens of hours at the rate section C ran at. Worth doing and not done; the page is explicit that its 30 rows cannot substitute. **Recorded as a follow-up item** (''ownership-accuracy-random-sample'' on the ''wiki-measuretheweb'' task) with the sampling frame, the sourcing bar and the scripts to reuse, together with the Wayback questions in the first bullet. |
| * **Whether anyone has built an LLM-based //domain//-owner resolver.** Nothing in this corpus. {[selmo2025_borges]} and {[gouda2025_prefix2org]} do the network-layer equivalent. Named on the page as a gap rather than as a method to use. | * **Whether anyone has built an LLM-based //domain//-owner resolver.** Nothing in this corpus. {[selmo2025_borges]} and {[gouda2025_prefix2org]} do the network-layer equivalent. Named on the page as a gap rather than as a method to use. |
| |
| * **Naming the miscitation.** Recorded as a verifiable fact about a reference list, with the methodological point it illustrates, and with no characterisation of the authors. The alternative — a vague "watch out for homographs" — would have been unfalsifiable. | * **Naming the miscitation.** Recorded as a verifiable fact about a reference list, with the methodological point it illustrates, and with no characterisation of the authors. The alternative — a vague "watch out for homographs" — would have been unfalsifiable. |
| * **Quoting webXray's licence at length.** It is the single most actionable fact on the page: a student who plans a study around webXray has planned a study around software they cannot legally obtain. A one-line "non-commercial licence" would have understated it. | * **Quoting webXray's licence at length.** It is the single most actionable fact on the page: a student who plans a study around webXray has planned a study around software they cannot legally obtain. A one-line "non-commercial licence" would have understated it. |
| | * **Discoverability, and where this material really belongs.** The wiki's only treatment of domain-to-company ownership resolution now sits under the name of a dead tool, where nobody asking "how do I attribute a domain?" will look. The generic reviewer was right that the earlier rejection of a separate ''design:ownership_resolution'' page ("it would leave a red link pointing at a stub") was a weak argument — a real webXray page and a topic page can coexist. Mitigated for now by linking this page from [[Privacy:Requests]], which is where a reader asking "whose request is this?" actually lands. Splitting out a topic page is worth doing and is recorded here rather than done. |
| * **''%%<WRAP important>%%'' boxes rather than ''%%<wrap todo>%%''.** No open TODOs were left on the content page: the unknowns are stated in its methodology section with what would close them, which is where a reader checking a number will look. The one real TODO — correcting [[Programming:Crawler]] — is recorded here, because it is work on a different page. | * **''%%<WRAP important>%%'' boxes rather than ''%%<wrap todo>%%''.** No open TODOs were left on the content page: the unknowns are stated in its methodology section with what would close them, which is where a reader checking a number will look. The one real TODO — correcting [[Programming:Crawler]] — is recorded here, because it is work on a different page. |
| * **No ''~~DISCUSSION~~'' on this provenance page.** Following the convention set by the earlier provenance pages: comments belong on the content page. | * **No ''~~DISCUSSION~~'' on this provenance page.** Following the convention set by the earlier provenance pages: comments belong on the content page. |
| `pushed_at: 2022-11-26T19:43:12Z`. The reviewer's inability to re-fetch under | `pushed_at: 2022-11-26T19:43:12Z`. The reviewer's inability to re-fetch under |
| a rate limit is not evidence against it. | a rate limit is not evidence against it. |
| | |
| | --- Reviewer 4 (generic, Fable): the three findings that mattered most ---------- |
| | |
| | Given no checklist and told to find what the focused three were not looking for. |
| | It found the most serious defect on the page, in the page's own method. |
| | |
| | ACCEPTED (blocking -- this was a real bug in owner_dbs.py): |
| | 1. **googleapis.com is a PRIVATE PSL rule, so the fold + exact-key lookup |
| | manufactured a coverage hole and depressed every weighted coverage figure.** |
| | Verified: the rule is at line 13935 of the snapshot, past BEGIN PRIVATE |
| | DOMAINS at 11276, and all three lists DO name googleapis.com (webXray |
| | "Google APIs", Tracker Radar "Google LLC", Disconnect "Google"). So |
| | fonts.googleapis.com -- which the page showcased three times as "the single |
| | most prevalent third-party name with no owner in any list" -- was an artefact |
| | of my own lookup rule, and it carries prevalence 0.369 on its own. |
| | |
| | Fixed properly rather than patched: fetch_psl() now returns the ICANN and |
| | PRIVATE sections separately, a lookup() helper walks parent labels, and the |
| | script prints all FOUR combinations (ICANN/full fold x walk/exact lookup) so |
| | the reader can see that the choice moves the answer by more than the lists |
| | differ. Every figure in sections C and D of the page was re-derived: |
| | weighted coverage 54.5/79.3/75.8% -> 58.6/84.3/80.5%; domain share |
| | 1.4/12.2/5.0% -> 2.1/17.2/7.0%; top-100 70/95/91 -> 71/98/94; the |
| | no-owner-anywhere figure 94.6% of domains and 19.8% of weight -> 92.4% and |
| | 14.4%. The pairwise disagreement table moved by under a point. |
| | |
| | The replacement example is better than the one it replaces: the most |
| | requested domain no list can name is now tiktokw.us at prevalence 0.041, |
| | which is a genuine, recent, mid-tail coverage hole. The googleapis rows are |
| | re-captioned as the trap they actually are, in the page's trap list and again |
| | in the lookup demo, which now shows the buggy version, says so, and gives the |
| | three-line fix. |
| | |
| | ACCEPTED (blocking): |
| | 2. **The two rows that failed the sourcing bar were still counted in the |
| | adjudication tallies, and two of webXray's three "current" verdicts WERE |
| | those two rows.** Verified from the script: webXray's "current" rows were |
| | casalemedia.com, 360yield.com and fwmrm.net, and the latter two are exactly |
| | the UNRESOLVED pair. So the one figure most favourable to webXray rested on |
| | the evidence the script itself called insufficient -- while both pages |
| | claimed those rows were "recorded as unresolved rather than guessed". |
| | owner_adjudication.py now excludes them from every tally and prints what it |
| | excluded and why: 28 rows, webXray current on 1 of 23 (was 3 of 25), Tracker |
| | Radar 8 of 28, Disconnect 21 of 21. |
| | |
| | Also accepted: the "Read as a share of each list's own entries ... 100%" |
| | sentence invited exactly the accuracy ranking the box below it disclaimed. |
| | Cut. The WRAP box now also says that Tracker Radar's zero in "No entry" and |
| | Disconnect's zero in "Stale" are artefacts of how the sample was drawn, and |
| | the script prints the same caveat above its own tallies. |
| | |
| | ACCEPTED (blocking): |
| | 3. **The provenance page and the committed script output were not brought up to |
| | date after reviewer 2's fixes, while certifying that everything passed.** The |
| | ROLE hand map in report_webxray.mjs still said Yang & Yue tried webXray |
| | "third of four" where the content page had been corrected to "second", and |
| | the section D verification table still said Ghostery trackerdb was "(MIT)". |
| | Both fixed at the source and re-run, so the "unedited output" now agrees with |
| | the page it certifies. The reviewer is right that check_page_numbers.mjs |
| | cannot catch this class -- "third" and "MIT" are not numerals -- and that gap |
| | is now stated in *What could not be established*. |
| | |
| | ACCEPTED (smaller): |
| | 4. "Treat webXray as **unobtainable**" was contradicted by the page's own |
| | practice: every figure on it was computed from the surviving copy. Reworded to |
| | "obtainable but not redistributable, and not installable", with the artefact- |
| | appendix consequence spelled out. |
| | 5. Ghostery trackerdb is named as a live option but is absent from the measured |
| | comparison. The omission and its reason (per-company .eno files plus a |
| | separate patterns layer, not a single domain->owner map) are now stated in the |
| | comparison section, labelled an omission rather than a judgement. |
| | 6. The reader could not assemble a pipeline. A new *Assembling the pipeline* |
| | section gives the six steps, including the one that was entirely missing: |
| | resolve the VISITED site to an owner too and drop same-owner requests. Without |
| | it, google.com embedding gstatic.com counts as third-party tracking. |
| | 7. The actionable answer sat ~120 lines after the lede. Jump links to |
| | *Choosing a resolution source now* and *Assembling the pipeline* added to the |
| | lede, which now says plainly that everything between is the evidence. |
| | |
| | NOTED, not acted on in this sitting: |
| | 8. Discoverability: the wiki's only treatment of domain-to-company resolution |
| | lives under a dead tool's name. The reviewer is right that the earlier |
| | rejection of a separate topic page was weak. Mitigated by adding a link from |
| | [[Privacy:Requests]], which is where a reader asking "whose request is this?" |
| | actually lands. A ''design:ownership_resolution'' page that this one feeds is |
| | worth considering and is recorded as such rather than done. |
| | 9. The 16-row schema table "could halve". Kept: its fill-rate column is the |
| | evidence for "most of that ambition is unfilled", which is the section's |
| | claim, and a reader checking one field wants the row. |
| |
| --- Reviewer 1: figures versus script ------------------------------------------ | --- Reviewer 1: figures versus script ------------------------------------------ |
| |
| Still running when the page was published. Its brief was to re-run all three | Landed after the first publication and after reviewer 4's fixes were applied, and |
| | it re-verified over 150 figures and table cells against freshly re-run script |
| | output, all byte-identical. It found two defects nobody else did, both in the |
| | script's own descriptions of its method rather than in any downstream figure. |
| | |
| | ACCEPTED (blocking): |
| | 1. **"domain_summary.json is keyed by hostname for 16,396 of its 47,836 rows" |
| | was computed by a label-count heuristic (>=3 labels), which answers a |
| | different question and is wrong under either fold.** 14,033 of those 16,396 |
| | rows already ARE their own registrable domain, precisely because |
| | googleapis.com, s3.amazonaws.com and cloudfront.net are private-section PSL |
| | suffixes -- the same root cause as reviewer 4's finding, showing up in a |
| | second place. The honest figure is the number of rows the fold you used |
| | merges: 15,651 under the ICANN fold, 2,339 under ICANN+PRIVATE. The script now |
| | prints both and says explicitly that the label count answers neither |
| | question; the page carries the same correction. The reviewer also |
| | re-implemented the PSL algorithm from scratch and found zero mismatches |
| | against registrable(), so the fold function itself was never wrong -- only its |
| | description. |
| | |
| | ACCEPTED (should fix): |
| | 2. **The suffix-fold residue conflated two operations.** "The fold ... fires on |
| | 45" was measured with the whole norm_name() pipeline, which strips punctuation |
| | before the suffix regex runs. Only 16 of the 45 lose a legal-form suffix; 29 |
| | change on punctuation alone (AT&T, JD.com, "Here, There & Everywhere"). The |
| | causal claim overstated the suffix fold's reach by nearly 3x. The script now |
| | prints both counts and names the punctuation-only cases; both pages say so. |
| | |
| | ACCEPTED (nit, already fixed): |
| | 3. owner_adjudication.py's docstring said "29" where ROWS has 30. Never |
| | published; corrected. |
| | |
| | Its process caveat is fair and worth recording: the repository was being edited |
| | throughout its run, so its findings are pinned to a snapshot. Both defects it |
| | found were still present in the last revision it sampled and are fixed now. Its brief was to re-run all three |
| scripts and diff every figure against the real output. That check was also run | scripts and diff every figure against the real output. That check was also run |
| directly, repeatedly, throughout the session: check_page_numbers.mjs passes | directly, repeatedly, throughout the session: check_page_numbers.mjs passes |
| per-paper figures in the cited sources. Anything reviewer 1 reports after | per-paper figures in the cited sources. Anything reviewer 1 reports after |
| publication goes into the page history, not into this log. | publication goes into the page history, not into this log. |
| | |
| | |
| | --- Reviewer 5: industry and website claims (late) ----------------------------- |
| | |
| | Checked 13 items against primary sources; 10 passed. It found the single most |
| | important error on the page, in the claim the licence box is built on. |
| | |
| | ACCEPTED (blocking -- the page's central licence claim was wrong): |
| | 1. **webXray was relicensed TWICE after the snapshot this page measures, and |
| | ended up MIT.** Verified directly: commit 245ec5d7 (2021-06-14) "Update |
| | LICENSE.md - Now open-source" makes it GPLv3, and commit 73fe0fc9 |
| | (2023-02-01), authored by "Tim Libert", replaces that with an MIT LICENSE |
| | reading "Copyright (c) 2023 Tim Libert". peterjoles/webXray reports |
| | spdx_id: MIT and is 36 commits AHEAD of thezedwards/webXray, preserving |
| | upstream history -- including Libert's own post-2021 feature work -- past the |
| | deletion of the upstream repo. Several other forks report GPL-3.0. |
| | |
| | So "webXray is not open source, and redistributing it is prohibited" was true |
| | of the one snapshot the page happened to build on and false of the project's |
| | final state, and the page's own advice -- "verify the licence of whatever file |
| | you actually download" -- is exactly what it failed to do. The licence box is |
| | rewritten as a three-row table of the three licences with what each permits, |
| | the availability table now names peterjoles/webXray as the most complete copy |
| | and says plainly that every figure here comes from the most restrictively |
| | licensed copy that exists, and the lede no longer says the tool "is gone". |
| | 2. **"three small commits in a personal working copy"** for the newest fork |
| | activity was wrong: those are 36 preserved upstream commits, not a fork |
| | owner's tinkering. Corrected. |
| | 3. **forks_count reports 19 while the forks endpoint returns 20 objects.** The |
| | page cited the endpoint for the number 19. Now states both. |
| | |
| | ACCEPTED (should fix): |
| | 4. The Mozilla shavar footnote is attached to a row about entities.json, but |
| | disconnect-blacklist.json mirrors services.json; the file mirroring |
| | entities.json is disconnect-entitylist.json. The quote is verbatim, the |
| | attachment was to the sibling file. |
| | 5. "whotracks.me now redirects" is imprecise: it returns HTTP 200 with a |
| | canonical link to ghostery.com/whotracksme, serving byte-identical content |
| | rather than issuing a redirect. |
| | |
| | Independently re-confirmed, already fixed: the Ghostery trackerdb licence (CC |
| | BY-NC-SA 4.0, and it is whotracks.me that is MIT -- the page had them inverted), |
| | the RDBinns commit range (11 commits, 2018-03-29 to 2018-04-05), and the |
| | truncated PSL quotation. Also confirmed against SEC filings rather than press |
| | releases: the Xandr close (AT&T Form 10-Q, "On June 6, 2022") and the Teads close |
| | (Outbrain 8-K, "On February 3, 2025"), plus that SEC now lists CIK 0001454938 as |
| | "Teads Holding Co." with formerNames "Outbrain Inc." -- the acquirer took the |
| | target's name, as the page says. |
| | |
| | The lesson worth keeping: five reviewers found five defects nobody else found, |
| | and the two most serious -- a coverage figure depressed by my own lookup rule, and |
| | a licence claim resting on an unrepresentative snapshot -- were both invisible to |
| | every automated guard in this repository. |
| |
| ===== I. Unedited output: scripts/report_webxray.mjs ===== | ===== I. Unedited output: scripts/report_webxray.mjs ===== |
| Per paper: | Per paper: |
| Paper Role What was used | Paper Role What was used |
| --------------------------------------------------------------------------------------------------------------------------- --------------- ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- --------------- -------------------------------------------------------------------------------------------------------------------------------- |
| CCS/2016/online-tracking-a-1-million-site-measurement-and-analysis compared crawler, as a baseline whose browser is criticised | CCS/2016/online-tracking-a-1-million-site-measurement-and-analysis compared crawler, as a baseline whose browser is criticised |
| WWW/2018/an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w instrument crawler + owner list + policy extraction | WWW/2018/an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w instrument crawler + owner list + policy extraction |
| IEEE-SP/2020/do-cookie-banners-respect-my-choice-measuring-legal-compliance-of-banners-from-i owner-list-only owner list | IEEE-SP/2020/do-cookie-banners-respect-my-choice-measuring-legal-compliance-of-banners-from-i owner-list-only owner list |
| IMC/2020/analyzing-third-party-service-dependencies-in-modern-web-services-have-we-learne citation owner list in the reference list only; the paper builds its own TLD + SAN + SOA heuristic | IMC/2020/analyzing-third-party-service-dependencies-in-modern-web-services-have-we-learne citation owner list in the reference list only; the paper builds its own TLD + SAN + SOA heuristic |
| PETS/2020/a-comparative-measurement-study-of-web-tracking-on-mobile-and-desktop-environmen owner-list-only owner list (cited as “Tim Libert's library”), third of four fallback sources | PETS/2020/a-comparative-measurement-study-of-web-tracking-on-mobile-and-desktop-environmen owner-list-only owner list (cited as “Tim Libert's library”), SECOND of four sources tried in order: CrunchBase, webXray, TLS certificate, WHOIS |
| NDSS/2021/whos-hosting-the-block-party-studying-third-party-blockage-of-csp-and-sri owner-list-only owner list, retrieved from the Internet Archive | NDSS/2021/whos-hosting-the-block-party-studying-third-party-blockage-of-csp-and-sri owner-list-only owner list, retrieved from the Internet Archive |
| IEEE-SP/2022/journey-to-the-center-of-the-cookie-ecosystem-unraveling-actors-roles-and-relati owner-list-only owner list, third by priority of three merged lists | IEEE-SP/2022/journey-to-the-center-of-the-cookie-ecosystem-unraveling-actors-roles-and-relati owner-list-only owner list, third by priority of three merged lists |
| f.write(r.read()) | f.write(r.read()) |
| raw = open(path, "rb").read() | raw = open(path, "rb").read() |
| rules, exceptions = set(), set() | # The PSL has TWO sections and the split matters more than any other choice |
| | # in this script. ICANN rules are real TLDs and registry suffixes; PRIVATE |
| | # rules are suffixes companies asked to have treated as boundaries -- |
| | # `googleapis.com` is one of them. Folding with the private section leaves |
| | # `fonts.googleapis.com` standing as its own "registrable domain", and an |
| | # exact-key lookup then fails even though all three lists name |
| | # `googleapis.com` -> Google. So both sections are loaded separately. |
| | icann, private, exceptions = set(), set(), set() |
| | section = "icann" |
| for line in raw.decode("utf8").splitlines(): | for line in raw.decode("utf8").splitlines(): |
| line = line.strip() | stripped = line.strip() |
| if not line or line.startswith("//"): | if "BEGIN PRIVATE DOMAINS" in stripped: |
| | section = "private" |
| continue | continue |
| if line.startswith("!"): | if "BEGIN ICANN DOMAINS" in stripped: |
| exceptions.add(line[1:]) | section = "icann" |
| | continue |
| | if not stripped or stripped.startswith("//"): |
| | continue |
| | if stripped.startswith("!"): |
| | exceptions.add(stripped[1:]) |
| else: | else: |
| rules.add(line) | (icann if section == "icann" else private).add(stripped) |
| return rules, exceptions, hashlib.sha256(raw).hexdigest()[:16] | return icann, private, exceptions, hashlib.sha256(raw).hexdigest()[:16] |
| |
| |
| return None # the host *is* a public suffix | return None # the host *is* a public suffix |
| return ".".join(labels[-(best + 1):]) | return ".".join(labels[-(best + 1):]) |
| | |
| | |
| | def lookup(host, mapping): |
| | """Owner for a hostname, trying the host then each parent label sequence. |
| | |
| | A list keyed on `googleapis.com` must answer for `fonts.googleapis.com`. |
| | Exact-key lookup is the single easiest way to manufacture a coverage hole, |
| | and it is what the first version of this script did. Stops at two labels so |
| | it can never walk up to a bare TLD. |
| | """ |
| | labels = host.split(".") |
| | for i in range(len(labels) - 1): |
| | candidate = ".".join(labels[i:]) |
| | if candidate in mapping: |
| | return mapping[candidate], candidate |
| | return None, None |
| |
| |
| print() | print() |
| summary = data["tr_domain_summary"] | summary = data["tr_domain_summary"] |
| rules, exceptions, psl_hash = fetch_psl(args.cache) | icann, private, exceptions, psl_hash = fetch_psl(args.cache) |
| print(f"Public Suffix List: {psl_hash} {len(rules):,} rules, " | print(f"Public Suffix List: {psl_hash} {len(icann):,} ICANN rules, " |
| f"{len(exceptions):,} exceptions {PSL_URL}") | f"{len(private):,} PRIVATE rules, {len(exceptions):,} exceptions {PSL_URL}") |
| | print("The ICANN/PRIVATE split is the most consequential choice in this script.") |
| | print("`googleapis.com` is a PRIVATE rule, so folding with the private section") |
| | print("leaves fonts.googleapis.com standing as its own 'registrable domain' --") |
| | print("and an exact-key lookup then finds no owner for it, even though all three") |
| | print("lists name googleapis.com -> Google. Both rules are therefore reported.") |
| | print() |
| raw_keys = list(summary) | raw_keys = list(summary) |
| bad = [k for k in raw_keys if "." not in k] | bad = [k for k in raw_keys if "." not in k] |
| prev = {} | |
| folded = 0 | def build(rules_set): |
| for host, v in summary.items(): | prev, folded = {}, 0 |
| host = host.lower() | for host, v in summary.items(): |
| if "." not in host: | host = host.lower() |
| continue # see the 'null' note below | if "." not in host: |
| reg = registrable(host, rules, exceptions) | continue # see the 'null' note below |
| if reg is None: | reg = registrable(host, rules_set, exceptions) |
| continue | if reg is None: |
| if reg != host: | continue |
| folded += 1 | if reg != host: |
| # A registrable domain's weight is the *largest* prevalence among its | folded += 1 |
| # hostnames, not the sum: one site can request fonts.googleapis.com and | # A registrable domain's weight is the *largest* prevalence among its |
| # ajax.googleapis.com, so summing would double-count sites. This makes | # hostnames, not the sum: one site can request fonts.googleapis.com |
| # every weighted coverage figure below a conservative lower bound. | # and ajax.googleapis.com, so summing would double-count sites. This |
| prev[reg] = max(prev.get(reg, 0.0), v["prevalence"]) | # makes every weighted coverage figure a conservative lower bound. |
| total_prev = sum(prev.values()) | prev[reg] = max(prev.get(reg, 0.0), v["prevalence"]) |
| universe = sorted(prev) | return prev, folded |
| print(f"domain_summary.json rows: {len(raw_keys):,}. " | |
| f"{sum(1 for k in raw_keys if k.count('.') >= 2):,} of them are keyed by " | prev_icann, folded_icann = build(icann) |
| f"hostname rather than registrable domain; folding to eTLD+1 merges " | prev_full, folded_full = build(icann | private) |
| f"{folded:,} rows and leaves {len(universe):,} registrable domains.") | # How many rows are "keyed by hostname rather than registrable domain" is not |
| | # a property of the data alone -- it depends on which PSL section you fold |
| | # with, because googleapis.com and s3.amazonaws.com are private-section |
| | # suffixes. A label-count heuristic (>=3 labels) says 16,396 and is wrong |
| | # under either fold; the honest figure is the number of rows the fold you |
| | # actually used merges. Both are printed. Caught by review 2026-08-17. |
| | print(f"domain_summary.json rows: {len(raw_keys):,}, of which " |
| | f"{sum(1 for k in raw_keys if k.count('.') >= 2):,} have three or more " |
| | f"labels -- a count that does NOT answer 'how many are keyed by hostname " |
| | f"rather than registrable domain', because that depends on the fold:") |
| | print(f" ICANN-section fold: merges {folded_icann:,} rows -> " |
| | f"{len(prev_icann):,} registrable domains") |
| | print(f" ICANN+PRIVATE fold: merges {folded_full:,} rows -> " |
| | f"{len(prev_full):,} registrable domains") |
| print(f"Rows dropped as not hostnames at all: {len(bad)} {bad!r} " | print(f"Rows dropped as not hostnames at all: {len(bad)} {bad!r} " |
| "-- a literal 'null' key with a prevalence attached is a defect in the " | "-- a literal 'null' key with a prevalence attached is a defect in the " |
| "published data, not a domain.") | "published data, not a domain.") |
| print() | print() |
| print(f"Denominator: those {len(universe):,} registrable third-party domains, i.e.") | print("Denominator: those registrable third-party domains, i.e. domains Tracker") |
| print("domains Tracker Radar's own crawl of regional top-site lists actually saw.") | print("Radar's own crawl of regional top-site lists actually saw. This is Tracker") |
| print("This is Tracker Radar's view of the surface, not a neutral one.") | print("Radar's view of the surface, not a neutral one.") |
| print() | print() |
| print(f"{'List':30} {'Domains named':>14} {'Share':>7} {'Prevalence-weighted':>20}") | |
| for label, m in (("webXray", wx_owner), ("Tracker Radar", tr_owner), | LISTS = (("webXray", wx_owner), ("Tracker Radar", tr_owner), |
| ("Disconnect (properties+res)", dc_owner)): | ("Disconnect (properties+res)", dc_owner)) |
| hit = [d for d in universe if d in m] | |
| wprev = sum(prev[d] for d in hit) | def coverage(prev, walk): |
| print(f"{label:30} {len(hit):>14,} {100*len(hit)/len(universe):>6.1f}% " | rows = [] |
| f"{100*wprev/total_prev:>19.1f}%") | total = sum(prev.values()) |
| | for label, m in LISTS: |
| | if walk: |
| | hit = [d for d in prev if lookup(d, m)[0] is not None] |
| | else: |
| | hit = [d for d in prev if d in m] |
| | rows.append((label, len(hit), 100 * len(hit) / len(prev), |
| | 100 * sum(prev[d] for d in hit) / total)) |
| | return rows |
| | |
| | VARIANTS = [ |
| | ("ICANN fold + parent-label lookup <- USE THIS", prev_icann, True), |
| | ("ICANN fold + exact-key lookup", prev_icann, False), |
| | ("ICANN+PRIVATE fold + parent-label lookup", prev_full, True), |
| | ("ICANN+PRIVATE fold + exact-key lookup <- THE TRAP", prev_full, False), |
| | ] |
| | for name, prev, walk in VARIANTS: |
| | print(f"{name} (universe {len(prev):,})") |
| | print(f" {'List':30} {'Named':>8} {'Share':>7} {'Prevalence-weighted':>20}") |
| | for label, n, share, wshare in coverage(prev, walk): |
| | print(f" {label:30} {n:>8,} {share:>6.1f}% {wshare:>19.1f}%") |
| | print() |
| | fg = summary.get("fonts.googleapis.com", {}).get("prevalence") |
| | print(f"The single row that drives most of that gap: fonts.googleapis.com, prevalence " |
| | f"{fg:.3f}. googleapis.com is a PRIVATE PSL rule, and all three lists name it: " |
| | f"webXray={lookup('googleapis.com', wx_owner)[0]!r}, " |
| | f"TR={lookup('googleapis.com', tr_owner)[0]!r}, " |
| | f"Disconnect={lookup('googleapis.com', dc_owner)[0]!r}.") |
| | print("The bottom variant is what the first version of this script did, and it") |
| | print("understates every list. The gap between the top and bottom rows is a") |
| | print("measurement artefact, not a property of any list -- report which rule you") |
| | print("used, because it moves the answer by more than the lists differ.") |
| print() | print() |
| | |
| | # Everything below uses the recommended rule. |
| | prev = prev_icann |
| | total_prev = sum(prev.values()) |
| | universe = sorted(prev) |
| | owner_of = {label: {d: lookup(d, m)[0] for d in universe} for label, m in |
| | (("webXray", wx_owner), ("webXray-root", wx_root_owner), |
| | ("Tracker Radar", tr_owner), ("Disconnect", dc_owner))} |
| for n in (100, 1000, 10000): | for n in (100, 1000, 10000): |
| topn = sorted(universe, key=lambda d: -prev[d])[:n] | topn = sorted(universe, key=lambda d: -prev[d])[:n] |
| line = f"top {n:>5} by prevalence: " | line = f"top {n:>5} by prevalence: " |
| line += " ".join( | line += " ".join( |
| f"{label} {100*sum(1 for d in topn if d in m)/len(topn):.0f}%" | f"{label} {100*sum(1 for d in topn if owner_of[label][d] is not None)/len(topn):.0f}%" |
| for label, m in (("webXray", wx_owner), ("TR", tr_owner), ("Disconnect", dc_owner))) | for label in ("webXray", "Tracker Radar", "Disconnect")) |
| print(line) | print(line) |
| print() | print() |
| named_by_none = [d for d in universe if d not in wx_owner and d not in dc_owner] | named_by_none = [d for d in universe if owner_of["webXray"][d] is None |
| | and owner_of["Disconnect"][d] is None] |
| print(f"Domains in the universe that neither webXray nor Disconnect names an owner " | print(f"Domains in the universe that neither webXray nor Disconnect names an owner " |
| f"for: {len(named_by_none):,} ({100*len(named_by_none)/len(universe):.1f}%), " | f"for: {len(named_by_none):,} ({100*len(named_by_none)/len(universe):.1f}%), " |
| print("Top 15 of those by prevalence (Tracker Radar's owner in brackets):") | print("Top 15 of those by prevalence (Tracker Radar's owner in brackets):") |
| for d in sorted(named_by_none, key=lambda x: -prev[x])[:15]: | for d in sorted(named_by_none, key=lambda x: -prev[x])[:15]: |
| print(f" {d:34} prev={prev[d]:.4f} [{tr_owner.get(d, 'TR: none')}]") | print(f" {d:34} prev={prev[d]:.4f} [{owner_of['Tracker Radar'][d] or 'TR: none'}]") |
| top_unowned = max(named_by_none, key=lambda x: prev[x]) | top_unowned = max(named_by_none, key=lambda x: prev[x]) |
| print(f"Most prevalent domain no list can name an owner for: {top_unowned}, " | print(f"Most prevalent domain no list can name an owner for: {top_unowned}, " |
| ("Tracker Radar", tr_owner, "Disconnect", dc_owner)] | ("Tracker Radar", tr_owner, "Disconnect", dc_owner)] |
| disagreement_samples = {} | disagreement_samples = {} |
| for a_label, a, b_label, b in pairs: | for a_label, _a_raw, b_label, _b_raw in pairs: |
| both = [d for d in universe if d in a and d in b] | # Parent-label lookup here too, for the same reason as section C. |
| | a, b = owner_of[a_label], owner_of[b_label] |
| | both = [d for d in universe if a[d] is not None and b[d] is not None] |
| agree_raw = [d for d in both if a[d] == b[d]] | agree_raw = [d for d in both if a[d] == b[d]] |
| agree_norm = [d for d in both if norm_name(a[d]) == norm_name(b[d])] | agree_norm = [d for d in both if norm_name(a[d]) == norm_name(b[d])] |
| print("--- E. Sampled disagreements (highest prevalence first) --------------") | print("--- E. Sampled disagreements (highest prevalence first) --------------") |
| print() | print() |
| maps = {"webXray": wx_owner, "webXray-root": wx_root_owner, | |
| "Tracker Radar": tr_owner, "Disconnect": dc_owner} | |
| for (a_label, b_label), ds in disagreement_samples.items(): | for (a_label, b_label), ds in disagreement_samples.items(): |
| a, b = maps[a_label], maps[b_label] | a, b = owner_of[a_label], owner_of[b_label] |
| print(f"{a_label} vs {b_label}:") | print(f"{a_label} vs {b_label}:") |
| for d in ds[:args.disagreements]: | for d in ds[:args.disagreements]: |
| print() | print() |
| unchanged = [e["name"] for e in wx if norm_name(e["name"]) == e["name"].lower()] | unchanged = [e["name"] for e in wx if norm_name(e["name"]) == e["name"].lower()] |
| | # norm_name() does two things -- strips punctuation, then removes legal forms -- |
| | # so "names the fold changes" is not "names a legal suffix was removed from". |
| | # AT&T, JD.com and "Here, There & Everywhere" change on punctuation alone. |
| | # Reported separately after a review pointed out the conflation, 2026-08-17. |
| | changed = [e["name"] for e in wx if norm_name(e["name"]) != e["name"].lower()] |
| | suffix_hit = [n for n in changed if SUFFIX_RE.search( |
| | re.sub(r"[^a-z0-9]+", " ", re.sub(r"[’']", "", n.lower())).strip())] |
| | punct_only = [n for n in changed if n not in suffix_hit] |
| print(f"webXray owner names the fold leaves untouched: {len(unchanged)} of {len(wx)} " | print(f"webXray owner names the fold leaves untouched: {len(unchanged)} of {len(wx)} " |
| f"({100*len(unchanged)/len(wx):.1f}%). The fold only removes legal forms; it " | f"({100*len(unchanged)/len(wx):.1f}%).") |
| "merges no synonyms, so every figure in section D is a lower bound on real " | print(f"Of the {len(changed)} it does change, only {len(suffix_hit)} lose a legal-form " |
| "agreement and an upper bound on real disagreement.") | f"suffix; the other {len(punct_only)} change on punctuation alone: " |
| | + ", ".join(sorted(punct_only)[:8]) + ", ...") |
| | print("The fold merges no synonyms, so every figure in section D is a lower bound on " |
| | "real agreement and an upper bound on real disagreement.") |
| return 0 | return 0 |
| |
| --- C. Coverage of the third-party surface --------------------------- | --- C. Coverage of the third-party surface --------------------------- |
| |
| Public Suffix List: 155b43d46932e933 10,231 rules, 8 exceptions https://publicsuffix.org/list/public_suffix_list.dat | Public Suffix List: 155b43d46932e933 6,941 ICANN rules, 3,290 PRIVATE rules, 8 exceptions https://publicsuffix.org/list/public_suffix_list.dat |
| domain_summary.json rows: 47,836. 16,396 of them are keyed by hostname rather than registrable domain; folding to eTLD+1 merges 2,339 rows and leaves 45,525 registrable domains. | The ICANN/PRIVATE split is the most consequential choice in this script. |
| | `googleapis.com` is a PRIVATE rule, so folding with the private section |
| | leaves fonts.googleapis.com standing as its own 'registrable domain' -- |
| | and an exact-key lookup then finds no owner for it, even though all three |
| | lists name googleapis.com -> Google. Both rules are therefore reported. |
| | |
| | domain_summary.json rows: 47,836, of which 16,396 have three or more labels -- a count that does NOT answer 'how many are keyed by hostname rather than registrable domain', because that depends on the fold: |
| | ICANN-section fold: merges 15,651 rows -> 32,369 registrable domains |
| | ICANN+PRIVATE fold: merges 2,339 rows -> 45,525 registrable domains |
| Rows dropped as not hostnames at all: 19 ['null', '[2a01:4f9:2a:26e0::2]', '[2604:2dc0:100:5ce5::]', '[2001:41d0:800:4623::]', '[2001:41d0:602:556f::]', '[2604:8380:2e00:5::2]', '[2604:4500:8:2ea::2]', '[2604:8380:3300:1::2]', '[2604:8380:2900:15::2]', '[2001:41d0:403:579b::]', '[2001:41d0:306:44e6::]', '[2604:4500:a:432::2]', '[2402:1f00:8201:4a2::]', '[2604:4500:6:5a0::2]', '[2604:4500:21:8::4]', '[2402:1f00:8001:2518::]', '[2402:1f00:8300:c97::]', '[2604:8380:2f00:16::2]', '[2001:41d0:700:782c::]'] -- a literal 'null' key with a prevalence attached is a defect in the published data, not a domain. | Rows dropped as not hostnames at all: 19 ['null', '[2a01:4f9:2a:26e0::2]', '[2604:2dc0:100:5ce5::]', '[2001:41d0:800:4623::]', '[2001:41d0:602:556f::]', '[2604:8380:2e00:5::2]', '[2604:4500:8:2ea::2]', '[2604:8380:3300:1::2]', '[2604:8380:2900:15::2]', '[2001:41d0:403:579b::]', '[2001:41d0:306:44e6::]', '[2604:4500:a:432::2]', '[2402:1f00:8201:4a2::]', '[2604:4500:6:5a0::2]', '[2604:4500:21:8::4]', '[2402:1f00:8001:2518::]', '[2402:1f00:8300:c97::]', '[2604:8380:2f00:16::2]', '[2001:41d0:700:782c::]'] -- a literal 'null' key with a prevalence attached is a defect in the published data, not a domain. |
| |
| Denominator: those 45,525 registrable third-party domains, i.e. | Denominator: those registrable third-party domains, i.e. domains Tracker |
| domains Tracker Radar's own crawl of regional top-site lists actually saw. | Radar's own crawl of regional top-site lists actually saw. This is Tracker |
| This is Tracker Radar's view of the surface, not a neutral one. | Radar's view of the surface, not a neutral one. |
| | |
| | ICANN fold + parent-label lookup <- USE THIS (universe 32,369) |
| | List Named Share Prevalence-weighted |
| | webXray 669 2.1% 58.6% |
| | Tracker Radar 5,581 17.2% 84.3% |
| | Disconnect (properties+res) 2,268 7.0% 80.5% |
| | |
| | ICANN fold + exact-key lookup (universe 32,369) |
| | List Named Share Prevalence-weighted |
| | webXray 669 2.1% 58.6% |
| | Tracker Radar 5,581 17.2% 84.3% |
| | Disconnect (properties+res) 2,268 7.0% 80.5% |
| | |
| | ICANN+PRIVATE fold + parent-label lookup (universe 45,525) |
| | List Named Share Prevalence-weighted |
| | webXray 11,150 24.5% 59.7% |
| | Tracker Radar 18,099 39.8% 84.8% |
| | Disconnect (properties+res) 10,389 22.8% 80.6% |
| | |
| | ICANN+PRIVATE fold + exact-key lookup <- THE TRAP (universe 45,525) |
| | List Named Share Prevalence-weighted |
| | webXray 657 1.4% 54.5% |
| | Tracker Radar 5,539 12.2% 79.3% |
| | Disconnect (properties+res) 2,277 5.0% 75.8% |
| |
| List Domains named Share Prevalence-weighted | The single row that drives most of that gap: fonts.googleapis.com, prevalence 0.369. googleapis.com is a PRIVATE PSL rule, and all three lists name it: webXray='Google APIs', TR='Google LLC', Disconnect='Google'. |
| webXray 657 1.4% 54.5% | The bottom variant is what the first version of this script did, and it |
| Tracker Radar 5,539 12.2% 79.3% | understates every list. The gap between the top and bottom rows is a |
| Disconnect (properties+res) 2,277 5.0% 75.8% | measurement artefact, not a property of any list -- report which rule you |
| | used, because it moves the answer by more than the lists differ. |
| |
| top 100 by prevalence: webXray 70% TR 95% Disconnect 91% | top 100 by prevalence: webXray 71% Tracker Radar 98% Disconnect 94% |
| top 1000 by prevalence: webXray 26% TR 67% Disconnect 67% | top 1000 by prevalence: webXray 26% Tracker Radar 69% Disconnect 68% |
| top 10000 by prevalence: webXray 5% TR 27% Disconnect 16% | top 10000 by prevalence: webXray 5% Tracker Radar 29% Disconnect 17% |
| |
| Domains in the universe that neither webXray nor Disconnect names an owner for: 43,052 (94.6%), 19.3% of prevalence weight. | Domains in the universe that neither webXray nor Disconnect names an owner for: 29,896 (92.4%), 14.4% of prevalence weight. |
| Top 15 of those by prevalence (Tracker Radar's owner in brackets): | Top 15 of those by prevalence (Tracker Radar's owner in brackets): |
| fonts.googleapis.com prev=0.3692 [TR: none] | |
| ajax.googleapis.com prev=0.0622 [TR: none] | |
| imasdk.googleapis.com prev=0.0435 [TR: none] | |
| tiktokw.us prev=0.0406 [ByteDance Ltd.] | tiktokw.us prev=0.0406 [ByteDance Ltd.] |
| consentmanager.net prev=0.0359 [consentmanager AB] | consentmanager.net prev=0.0359 [consentmanager AB] |
| jnn-pa.googleapis.com prev=0.0270 [TR: none] | |
| maps.googleapis.com prev=0.0261 [TR: none] | |
| rapidedge.io prev=0.0247 [TR: none] | rapidedge.io prev=0.0247 [TR: none] |
| raptivecdn.com prev=0.0231 [TR: none] | raptivecdn.com prev=0.0231 [TR: none] |
| shopifycdn.com prev=0.0181 [Shopify Inc.] | shopifycdn.com prev=0.0181 [Shopify Inc.] |
| digitalaudience.io prev=0.0152 [Social Audience B.V.] | digitalaudience.io prev=0.0152 [Social Audience B.V.] |
| translate.googleapis.com prev=0.0144 [TR: none] | openwebmp.com prev=0.0130 [TR: none] |
| storage.googleapis.com prev=0.0141 [TR: none] | userway.org prev=0.0122 [TR: none] |
| Most prevalent domain no list can name an owner for: fonts.googleapis.com, prevalence 0.369 (rounded to 3 dp for quoting). | copper6.com prev=0.0113 [TR: none] |
| | ahrefs.com prev=0.0112 [Ahrefs Pte Ltd] |
| | anyrtb.com prev=0.0111 [TR: none] |
| | sparteo.com prev=0.0108 [TR: none] |
| | axiom.co prev=0.0102 [TR: none] |
| | Most prevalent domain no list can name an owner for: tiktokw.us, prevalence 0.041 (rounded to 3 dp for quoting). |
| Dropped rows and their prevalence, rounded to 3 dp for quoting: null=0.024, [2a01:4f9:2a:26e0::2]=0.000, [2604:2dc0:100:5ce5::]=0.000 | Dropped rows and their prevalence, rounded to 3 dp for quoting: null=0.024, [2a01:4f9:2a:26e0::2]=0.000, [2604:2dc0:100:5ce5::]=0.000 |
| |
| --- D. Do two lists that both know a domain agree on the owner? ------ | --- D. Do two lists that both know a domain agree on the owner? ------ |
| |
| webXray vs Tracker Radar: 601 domains named by both | webXray vs Tracker Radar: 612 domains named by both |
| identical owner string 29 (4.8%) | identical owner string 29 (4.7%) |
| agree after legal-suffix fold 304 (50.6%) | agree after legal-suffix fold 305 (49.8%) |
| one name contains the other 100 (16.6%) [e.g. Amazon / Amazon Technologies] | one name contains the other 109 (17.8%) [e.g. Amazon / Amazon Technologies] |
| neither 197 (32.8%) <- needs a human | neither 198 (32.4%) <- needs a human |
| ...weighted by prevalence, the 'needs a human' share is 30.2% | ...weighted by prevalence, the 'needs a human' share is 29.0% |
| |
| webXray-root vs Tracker Radar: 601 domains named by both | webXray-root vs Tracker Radar: 612 domains named by both |
| identical owner string 34 (5.7%) | identical owner string 34 (5.6%) |
| agree after legal-suffix fold 275 (45.8%) | agree after legal-suffix fold 276 (45.1%) |
| one name contains the other 90 (15.0%) [e.g. Amazon / Amazon Technologies] | one name contains the other 97 (15.8%) [e.g. Amazon / Amazon Technologies] |
| neither 236 (39.3%) <- needs a human | neither 239 (39.1%) <- needs a human |
| ...weighted by prevalence, the 'needs a human' share is 49.2% | ...weighted by prevalence, the 'needs a human' share is 51.1% |
| |
| webXray vs Disconnect: 461 domains named by both | webXray vs Disconnect: 464 domains named by both |
| identical owner string 170 (36.9%) | identical owner string 171 (36.9%) |
| agree after legal-suffix fold 212 (46.0%) | agree after legal-suffix fold 213 (45.9%) |
| one name contains the other 33 (7.2%) [e.g. Amazon / Amazon Technologies] | one name contains the other 34 (7.3%) [e.g. Amazon / Amazon Technologies] |
| neither 216 (46.9%) <- needs a human | neither 217 (46.8%) <- needs a human |
| ...weighted by prevalence, the 'needs a human' share is 46.5% | ...weighted by prevalence, the 'needs a human' share is 44.5% |
| |
| webXray-root vs Disconnect: 461 domains named by both | webXray-root vs Disconnect: 464 domains named by both |
| identical owner string 163 (35.4%) | identical owner string 164 (35.3%) |
| agree after legal-suffix fold 194 (42.1%) | agree after legal-suffix fold 195 (42.0%) |
| one name contains the other 39 (8.5%) [e.g. Amazon / Amazon Technologies] | one name contains the other 39 (8.4%) [e.g. Amazon / Amazon Technologies] |
| neither 228 (49.5%) <- needs a human | neither 230 (49.6%) <- needs a human |
| ...weighted by prevalence, the 'needs a human' share is 62.6% | ...weighted by prevalence, the 'needs a human' share is 64.1% |
| |
| Tracker Radar vs Disconnect: 1,532 domains named by both | Tracker Radar vs Disconnect: 1,538 domains named by both |
| identical owner string 82 (5.4%) | identical owner string 82 (5.3%) |
| agree after legal-suffix fold 630 (41.1%) | agree after legal-suffix fold 634 (41.2%) |
| one name contains the other 318 (20.8%) [e.g. Amazon / Amazon Technologies] | one name contains the other 320 (20.8%) [e.g. Amazon / Amazon Technologies] |
| neither 584 (38.1%) <- needs a human | neither 584 (38.0%) <- needs a human |
| ...weighted by prevalence, the 'needs a human' share is 30.7% | ...weighted by prevalence, the 'needs a human' share is 29.7% |
| |
| --- E. Sampled disagreements (highest prevalence first) -------------- | --- E. Sampled disagreements (highest prevalence first) -------------- |
| doubleclick.net prev=0.4456 webXray-root='Alphabet' Tracker Radar='Google LLC' | doubleclick.net prev=0.4456 webXray-root='Alphabet' Tracker Radar='Google LLC' |
| gstatic.com prev=0.4013 webXray-root='Alphabet' Tracker Radar='Google LLC' | gstatic.com prev=0.4013 webXray-root='Alphabet' Tracker Radar='Google LLC' |
| | googleapis.com prev=0.3692 webXray-root='Alphabet' Tracker Radar='Google LLC' |
| google-analytics.com prev=0.3489 webXray-root='Alphabet' Tracker Radar='Google LLC' | google-analytics.com prev=0.3489 webXray-root='Alphabet' Tracker Radar='Google LLC' |
| googlesyndication.com prev=0.2115 webXray-root='Alphabet' Tracker Radar='Google LLC' | googlesyndication.com prev=0.2115 webXray-root='Alphabet' Tracker Radar='Google LLC' |
| 360yield.com prev=0.0467 webXray-root='Azerion' Tracker Radar='Improve Digital BV' | 360yield.com prev=0.0467 webXray-root='Azerion' Tracker Radar='Improve Digital BV' |
| fwmrm.net prev=0.0439 webXray-root='Comcast' Tracker Radar='FreeWheel' | fwmrm.net prev=0.0439 webXray-root='Comcast' Tracker Radar='FreeWheel' |
| rfihub.com prev=0.0429 webXray-root='Amazon' Tracker Radar='Zeta Global' | |
| |
| webXray vs Disconnect: | webXray vs Disconnect: |
| doubleclick.net prev=0.4456 webXray-root='Alphabet' Disconnect='Google' | doubleclick.net prev=0.4456 webXray-root='Alphabet' Disconnect='Google' |
| gstatic.com prev=0.4013 webXray-root='Alphabet' Disconnect='Google' | gstatic.com prev=0.4013 webXray-root='Alphabet' Disconnect='Google' |
| | googleapis.com prev=0.3692 webXray-root='Alphabet' Disconnect='Google' |
| google-analytics.com prev=0.3489 webXray-root='Alphabet' Disconnect='Google' | google-analytics.com prev=0.3489 webXray-root='Alphabet' Disconnect='Google' |
| googlesyndication.com prev=0.2115 webXray-root='Alphabet' Disconnect='Google' | googlesyndication.com prev=0.2115 webXray-root='Alphabet' Disconnect='Google' |
| 2mdn.net prev=0.0616 webXray-root='Alphabet' Disconnect='Google' | 2mdn.net prev=0.0616 webXray-root='Alphabet' Disconnect='Google' |
| pippio.com prev=0.0594 webXray-root='Acxiom' Disconnect='LiveRamp' | pippio.com prev=0.0594 webXray-root='Acxiom' Disconnect='LiveRamp' |
| dotomi.com prev=0.0583 webXray-root='Here, There & Everywhere' Disconnect='PublicisGroupe' | |
| |
| Tracker Radar vs Disconnect: | Tracker Radar vs Disconnect: |
| --- F. Named-entity residue of the suffix fold ----------------------- | --- F. Named-entity residue of the suffix fold ----------------------- |
| |
| webXray owner names the fold leaves untouched: 782 of 827 (94.6%). The fold only removes legal forms; it merges no synonyms, so every figure in section D is a lower bound on real agreement and an upper bound on real disagreement. | webXray owner names the fold leaves untouched: 782 of 827 (94.6%). |
| | Of the 45 it does change, only 16 lose a legal-form suffix; the other 29 change on punctuation alone: 56.com, AT&T, Ask.com, Bootstrap_China, Clearstream.TV, Cm_browser, Dictionary.com, Dun & Bradstreet, ... |
| | The fold merges no synonyms, so every figure in section D is a lower bound on real agreement and an upper bound on real disagreement. |
| </code> | </code> |
| |
| |
| <code> | <code> |
| 30 hand-adjudicated disagreements, decided 2026-08-17 against primary sources. | 30 disagreements adjudicated 2026-08-17 against primary sources; 2 failed the sourcing bar and are EXCLUDED from every tally below, leaving 28. |
| | Excluded: 360yield.com (UNRESOLVED -- no dated primary source), fwmrm.net (acquired 2014; post-2026 spinoff UNRESOLVED) |
| Selected as the highest-prevalence disagreements, NOT sampled at random: | Selected as the highest-prevalence disagreements, NOT sampled at random: |
| these tallies describe this set and are not an error rate for any list. | these tallies describe this set and are not an error rate for any list. |
| |
| List current stale granularity error absent | List current stale granularity error absent |
| webXray 3 17 4 1 5 | webXray 1 17 4 1 5 |
| Tracker Radar 9 12 7 2 0 | Tracker Radar 8 12 6 2 0 |
| Disconnect 22 0 0 0 8 | Disconnect 21 0 0 0 7 |
| | |
| | webXray: has an entry for 23 of 28; of those, 1 name today's owner (4% of its own entries). |
| | Tracker Radar: has an entry for 28 of 28; of those, 8 name today's owner (29% of its own entries). |
| | Disconnect: has an entry for 21 of 28; of those, 21 name today's owner (100% of its own entries). |
| |
| webXray: has an entry for 25 of 30; of those, 3 name today's owner (12% of its own entries). | The 'absent' column cannot be read as a coverage result either: the rows were |
| Tracker Radar: has an entry for 30 of 30; of those, 9 name today's owner (30% of its own entries). | ranked by Tracker Radar's own prevalence field, so Tracker Radar covering all |
| Disconnect: has an entry for 22 of 30; of those, 22 name today's owner (100% of its own entries). | 28 of 28 is partly how the sample was drawn. |
| |
| Rows where the primary-source bar was not met: 2 -- 360yield.com, fwmrm.net | |
| Rows scored as an outright error rather than staleness: 3 -- 1rx.io (webXray), jsdelivr.net (Tracker Radar), stackadapt.com (Tracker Radar) | Rows scored as an outright error rather than staleness: 3 -- 1rx.io (webXray), jsdelivr.net (Tracker Radar), stackadapt.com (Tracker Radar) |
| |
| simpli.fi Simplifi Holdings Inc. (GTCR + Blackstone) granularity granularity current https://www.blackstone.com/news/press/simpli-fi-a-leading-programmatic-advertising-platform-announces-completion-of-significant-investment-from-blackstone-at-1-5-billion-valuation/ | simpli.fi Simplifi Holdings Inc. (GTCR + Blackstone) granularity granularity current https://www.blackstone.com/news/press/simpli-fi-a-leading-programmatic-advertising-platform-announces-completion-of-significant-investment-from-blackstone-at-1-5-billion-valuation/ |
| casalemedia.com Index Exchange, Inc. (independent) current current current https://www.indexexchange.com/team/andrew-casale/ | casalemedia.com Index Exchange, Inc. (independent) current current current https://www.indexexchange.com/team/andrew-casale/ |
| 360yield.com Azerion Group N.V. (Improve Digital) current granularity current https://improvedigital.com/about/ | 360yield.com Azerion Group N.V. (Improve Digital) current granularity current https://improvedigital.com/about/ EXCLUDED |
| fwmrm.net Comcast Corporation (FreeWheel) current current absent https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001166691 | fwmrm.net Comcast Corporation (FreeWheel) current current absent https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001166691 EXCLUDED |
| </code> | </code> |
| |
