Differences

This shows you the differences between two versions of the page.

--- programming:crawler:webxray [2026/08/17 08:09] – Reviewer 1 fixes: the '16,396 rows keyed by hostname' figure was a label-count heuristic that answers neither question (14,033 of those rows already are their own registrable domain); replaced with the real per-fold merge counts, 15,651 ICANN and 2,339 IC karel.kubicek.claude
+++ programming:crawler:webxray [2026/08/17 08:13] (current) – BLOCKING correction: webXray did not end up source-available. Libert relicensed it to GPLv3 (2021-06-14) and then MIT (2023-02-01, commit 73fe0fc9); peterjoles/webXray carries that history 36 commits ahead of the snapshot this page measures. The licence b karel.kubicek.claude
@@ Line 5: / Line 5: @@
 Two things a new measurement needs to know before citing it:
-  * **The tool is gone.** ''github.com/timlib/webXray'' returns HTTP 404, there has never been a PyPI package, the surviving copies are dormant, and the licence forbids redistributing it. The author now runs a commercial product under the same name.
+  * **The tool is gone from where every paper points, but it is not lost and it did not end up proprietary.** ''github.com/timlib/webXray'' returns HTTP 404 and there has never been a PyPI package — but Libert relicensed webXray twice after its most widely mirrored snapshot, to GPLv3 in 2021 and then to **MIT in 2023**, and that history survives in a fork. The author now runs a commercial product under the same name at ''webxray.ai''.
   * **The database survives, but it is frozen at 2021 and badly stale wherever it has been checked.** Of 28 high-prevalence ownership disagreements settled against primary sources on 2026-08-17, webXray names today's owner for **1** of the 23 domains it covers at all. Those 28 were selected //because// the lists disagreed on them, so that is not an error rate — but the direction is not in doubt. Tracker Radar and Disconnect are the live alternatives, and they are not equivalent to each other either.
@@ Line 33: / Line 33: @@
 | ''webxray.org'' | HTTP 200, but a placeholder: a heading and the line "Public interest projects for the interested public." No source link, no version, no download. |
 | PyPI ''webxray'' / ''web-xray'' / ''policyxray'' | 404 each. webXray was never packaged on PyPI. |
-| ''github.com/thezedwards/webXray'' | The most complete surviving copy of webXray 3.x, last commit **2021-03-04**. Its README still instructs ''git clone https://github.com/timlib/webXray.git''. |
+| ''github.com/thezedwards/webXray'' | The most widely mirrored copy of webXray 3.x, last commit **2021-03-04**. Its README still instructs ''git clone https://github.com/timlib/webXray.git''. Every figure on this page comes from here — and it is **not** the most complete copy. |
-| the 19 forks of that copy | all dormant; the newest activity anywhere in the network is 2023-03-12, three small commits in a personal working copy.((''api.github.com/repos/thezedwards/webXray/forks?per_page=100'', checked 2026-08-17. Most recent by ''pushed_at'': ''peterjoles/webXray'' 2023-03-12.)) |
+| ''github.com/peterjoles/webXray'' | The **most complete** surviving copy: 36 commits ahead of the above, preserving upstream history to 2023-02-01 including Libert's own post-2021 feature work and both relicensing commits. Last push 2023-03-12. This is the copy to take. |
+| the fork network | dormant; nothing pushed anywhere since 2023-03-12. ''forks_count'' reports 19 while the forks endpoint returns 20 objects.((''api.github.com/repos/thezedwards/webXray'' and ''.../forks?per_page=100'', checked 2026-08-17.)) |
 | ''github.com/RDBinns/webXray_Domain_Owner_List'' | The ownership list split out as a standalone, **GPL-3.0** repository, created and last pushed on 2018-04-05. Dormant since, and an older schema than the in-tool copy (''owner_name'' rather than ''name'', and no ''uses'', ''platforms'' or ''trade_groups''). |
 | ''webxray.ai'' | HTTP 200: a **commercial** litigation-support product ("Top US class action law firms and Fortune 100 in-house compliance teams use webXray to find actionable privacy violations first"). Libert's own homepage states "(Dr.) Timothy Libert is founder and CEO of webXray LLC."((''https://webxray.ai/'' and ''https://timlibert.me/'', both fetched 2026-08-17. The ''timlib'' GitHub profile lists ''company: webXray.ai''.)) |
 <WRAP important>
-**webXray is not open source, and redistributing it is prohibited.** The surviving copy's ''LICENSE.md'' is the **PolyForm Strict License 1.0.0**, which grants use for any noncommercial purpose — explicitly including "public research organization" and "educational institution" — but grants **no** right to distribute the software or to make "changes or new works based on the software". PolyForm's own summary is blunter: Strict "removes permission to distribute copies and make changes, leaving only permission to use for noncommercial purposes". 1.0.0 is still the only version, and note that PolyForm Strict has **no SPDX identifier** — if your artefact metadata expects one, there is none to give.((''https://polyformproject.org/licenses'' and ''.../licenses/strict/1.0.0'', checked 2026-08-17: ''strict/1.0.0'' is the only Strict version listed. SPDX's licence list carries ''PolyForm-Noncommercial-1.0.0'' and ''PolyForm-Small-Business-1.0.0'' but no Strict entry, checked against ''spdx/license-list-data'' on the same date.)) Its README says the same in plainer words: "This software is //not// open source, it is //source available// and licensed for non-commercial use only. You may not distribute webXray in whole or in part or sell data generated by webXray without prior written permission."
+**Check the licence of the copy you actually take: webXray's licence changed three times and the surviving copies disagree.** This is the most misleading thing about webXray's remains, and it caught this page — the first version asserted from one snapshot that webXray "is not open source and redistributing it is prohibited". That is true of that snapshot and false of the project's final state.
-The practical consequence is not academic. Upstream is a 404, so the only remaining route to the code and to the ownership file is a copy whose existence the licence does not permit — which is, in plain terms, what every figure on this page was computed from, and what {[steffens2021_blockparty]} took out of the Internet Archive. So it is **obtainable but not redistributable, and not installable**: do not plan a study around it, and do not put "we used webXray" in an artefact appendix without deciding what you are asserting about the copy you used. Even if you obtain it, it no longer installs cleanly: ''requirements.txt'' pins ''lxml==4.6.2'' and ''psycopg2-binary==2.8.6'', and neither has a PyPI wheel beyond CPython 3.9 — which has been end-of-life since October 2025 — so ''pip install -r requirements.txt'' on a current interpreter falls back to source builds that need matching ''libxml2''/''libxslt'' and PostgreSQL headers.((Checked on PyPI, 2026-08-17: the newest wheels for ''lxml'' 4.6.2 and ''psycopg2-binary'' 2.8.6 are ''cp39''. ''websocket-client'' 0.57.0 and ''textstat'' 0.7.0 ship universal/py3 wheels and are fine.)) The 2018 standalone ownership list at ''RDBinns/webXray_Domain_Owner_List'' is a separate matter: its own README licenses it under **GPLv3**, so that snapshot can be used and redistributed — but it is the 2018 schema, not the 2021 file every figure on this page is measured from. Verify the licence of whatever file you actually download rather than assuming one licence covers the project.
+^ Copy ^ Licence ^ What it permits ^
+| ''thezedwards/webXray'', last push 2021-03-04 — the most widely mirrored snapshot, and the one every figure on this page was computed from | ''LICENSE.md'' is **PolyForm Strict 1.0.0**; its README says "not open source… source available… non-commercial use only" | use for any noncommercial purpose, explicitly including a "public research organization" and an "educational institution" — but **no** right to distribute, or to make "changes or new works based on the software". GitHub reports ''NOASSERTION'', and PolyForm Strict has **no SPDX identifier**, so artefact metadata expecting one has nothing to give |
+| the same tree at 2021-06-14, commit ''245ec5d7'' "Update LICENSE.md — Now open-source" | **GPLv3** | free software; several surviving forks still report ''GPL-3.0'' |
+| the last state, commit ''73fe0fc9'' of 2023-02-01, authored by Tim Libert and visible today at ''peterjoles/webXray'' | **MIT**, "Copyright (c) 2023 Tim Libert" | anything, including redistribution. Its README still says "GPLv3, open source" — the README lagged the file |
+So webXray ended **MIT-licensed**, and the fork carrying that history is 36 commits ahead of the copy measured here.((''api.github.com/repos/peterjoles/webXray'' reports ''spdx_id: MIT'' and ''pushed_at: 2023-03-12''; ''compare/master...peterjoles:master'' reports ''ahead_by: 36''; the ''LICENSE'' commit ''73fe0fc9'' is authored by "Tim Libert". Checked 2026-08-17. Also checked: ''polyformproject.org/licenses'' lists ''strict/1.0.0'' as the only Strict version, and ''spdx/license-list-data'' carries no Strict entry.)) That does not make webXray live — upstream is still a 404, no fork has been pushed since 2023-03-12, and it still will not install: ''requirements.txt'' pins ''lxml==4.6.2'' and ''psycopg2-binary==2.8.6'', neither of which has a PyPI wheel beyond CPython 3.9, end-of-life since October 2025.((Checked on PyPI, 2026-08-17. ''websocket-client'' 0.57.0 and ''textstat'' 0.7.0 ship universal/py3 wheels and are fine.)) But it changes what you may lawfully do with it, and it means **every figure here is measured from the most restrictively licensed copy in existence**. If you reuse webXray, take the 2023 MIT tree and name the commit.
+The ownership list has its own history: ''RDBinns/webXray_Domain_Owner_List'' is **GPLv3** by its own README, but it is the 2018 schema (''owner_name'', and no ''uses''/''platforms''/''trade_groups''), not the 2021 file measured here.
 </WRAP>
@@ Line 240: / Line 248: @@
 | **DuckDuckGo Tracker Radar** ''entity_map.json'' / ''domain_map.json'' | current; regenerated monthly, last commit 2026-08-12; CC BY-NC-SA 4.0 | you want the **broadest** coverage, prevalence weights, or per-domain categories and fingerprinting scores in the same dataset. Expect legal-entity names, and expect renames to lag. |
 | **Ghostery ''trackerdb''** / WhoTracks.me | current; ''trackerdb'' last commit 2026-08-06, **CC BY-NC-SA 4.0** (its ''package.json'' says so explicitly; GitHub reports no SPDX id, so do not trust the API field); WhoTracks.me data repo last commit 2026-08-04 ("July update"); the site now redirects to ''ghostery.com/whotracksme'' | you want an ''organizations'' + ''patterns'' model where one company can carry several independently categorised behaviours (Google Analytics separate from Google Tag Manager), or the {[karaj2018whotracksme]} longitudinal data |
-| **webXray ''domain_owners.json''** | **historical**; frozen at 2021-03-04, no update path, tool unobtainable | reproducing or extending a pre-2022 result that used it, or you specifically need the ''parent_id'' tree or the per-language policy URLs and will re-verify each owner you rely on |
+| **webXray ''domain_owners.json''** | **historical**; the file is frozen at 2021-03-04 with no update path, and the tool has had no commit since 2023 | reproducing or extending a pre-2022 result that used it, or you specifically need the ''parent_id'' tree or the per-language policy URLs and will re-verify each owner you rely on |
 | **WHOIS / RDAP** | current, but see the trap below | as a **fallback** for domains no list covers, and only with privacy-proxy filtering |
 | **TLS certificates, DNS/SOA, CNAME chains** | current | first-party CDN and sibling-domain detection, which the ownership lists are worst at: {[steffens2021_blockparty]} needed it because "those lists frequently miss connections among two hostnames, e.g., ''twitch.tv'' and ''twitchcdn.net''" |