Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray [2026/08/17 08:08] – Review fixes: corrected B4-B10 figures, the PSL ICANN/PRIVATE choice as the largest denominator decision, adjudication tallies over 28 sourced rows, the non-numeric drift the number guard cannot catch, and the full generic-reviewer log with what was accep karel.kubicek.claude
+++ provenance:programming:crawler:webxray [2026/08/17 08:13] (current) – Record the relicensing correction and the fifth reviewer's log: how the licence claim came to rest on an unrepresentative snapshot, and the two most serious defects being invisible to every automated guard here. Authored by Claude. karel.kubicek.claude
@@ Line 116: / Line 116: @@
 | B2 | How deep is webXray's tree? | walk ''parent_id'' with a cycle guard | 319 of 827 have a parent; depths 1:508 2:211 3:80 4:24 5:2 6:2; **2,175 of 3,215 domains (67.7%)** have leaf ≠ root |
 | B3 | How much of webXray's schema is filled? | per-field non-empty count | ''country'' 826 (99.9%), ''platforms'' 803 (97.1%), ''uses'' 761 (92.0%), site policy URLs 591 (71.5%), GDPR 132 (16.0%), ''trade_groups'' 104 (12.6%), ''crunchbase_id'' 34 (4.1%), ''opt_out_urls'' 13 (1.6%), ''ccpa_urls'' **4 (0.5%)** |
-| B4 | What is the universe of third-party domains? | Tracker Radar ''domain_summary.json'', folded to eTLD+1 with the **ICANN section** of the live PSL | 47,836 rows → **32,369** registrable domains (the ICANN+PRIVATE fold gives 45,525); 16,396 rows were hostname-keyed; 19 rows dropped as not hostnames |
+| B4 | What is the universe of third-party domains? | Tracker Radar ''domain_summary.json'', folded to eTLD+1 with the **ICANN section** of the live PSL | 47,836 rows → **32,369** registrable domains; the ICANN fold merges **15,651** rows, the ICANN+PRIVATE fold only 2,339; 19 rows dropped as not hostnames |
 | B5 | Coverage, unweighted | domains in the universe the list can name, parent labels walked | webXray 669 (2.1%), Tracker Radar 5,581 (17.2%), Disconnect 2,268 (7.0%) |
 | B6 | Coverage, prevalence-weighted | Σ prevalence of covered ÷ Σ prevalence of all | webXray **58.6%**, Tracker Radar **84.3%**, Disconnect **80.5%** |
@@ Line 131: / Line 131: @@
 Deliberately **not** folded: ''group'', ''media'', ''technologies'', ''digital'', ''networks'', ''solutions''. Each is part of a real company name often enough ("Almondnet Group", "Zeta Global", "Lotame Solutions") that stripping it would manufacture agreement between lists that name different companies. Also **no synonym merging at all**: "Facebook" is not folded to "Meta", "DoubleClick" is not folded to "Google". That is the whole point — those are the disagreements being measured.
-**Residue, printed in full by the script:** the fold leaves **782 of webXray's 827 owner names untouched (94.6%)**, i.e. it fires on 45. That is the intended behaviour and its consequence is stated on the page: every "agree" figure is a **lower bound** on real agreement and every "disagree" figure an **upper bound** on real disagreement. A synonym-merging fold would move the numbers in a direction the script cannot justify, so it was not written.
+**Residue, printed in full by the script:** the fold leaves **782 of webXray's 827 owner names untouched (94.6%)**, i.e. it changes 45 — but only **16** of those 45 lose a legal-form suffix. The other 29 change because ''norm_name()'' strips punctuation before the suffix regex runs (''AT&T'', ''56.com'', ''JD.com'', ''Dun & Bradstreet'', "Here, There & Everywhere"). The first version of this page attributed all 45 to the suffix fold, overstating its reach by nearly 3×; the script now prints the two counts separately and lists the punctuation-only names. That is the intended behaviour and its consequence is stated on the page: every "agree" figure is a **lower bound** on real agreement and every "disagree" figure an **upper bound** on real disagreement. A synonym-merging fold would move the numbers in a direction the script cannot justify, so it was not written.
 The second normalisation is the **PSL fold** on the universe (B4). Its residue is printed as the 19 dropped rows, listed individually: 18 bracketed IPv6 literals and the literal string ''"null"'' with prevalence 0.024 and a full behaviour profile attached. A row that is not a hostname is not a domain; it is printed rather than silently dropped.
@@ Line 186: / Line 186: @@
 | ''thezedwards/webXray'' last pushed 2021-03-04, 19 forks, newest fork activity 2023-03-12 | GitHub API repo + ''/forks?per_page=100'' |
 | the surviving README still says ''git clone https://github.com/timlib/webXray.git'' | fetched ''raw.githubusercontent.com/thezedwards/webXray/master/README.md'' |
-| PolyForm Strict License 1.0.0, no redistribution, noncommercial permitted incl. research organisations | fetched ''LICENSE.md'' from the same repo and read the licence text itself, not a summary |
+| PolyForm Strict License 1.0.0 on the ''thezedwards'' snapshot | fetched ''LICENSE.md'' from that repo and read the licence text itself, not a summary. **This is not webXray's final licence** — see the next row |
+| webXray was relicensed to GPLv3 (2021-06-14, commit ''245ec5d7'') and then to **MIT** (2023-02-01, commit ''73fe0fc9'', authored by "Tim Libert") | fetched ''api.github.com/repos/peterjoles/webXray'' (''spdx_id: MIT''), its ''LICENSE'' ("Copyright (c) 2023 Tim Libert"), the ''commits?path=LICENSE'' history, and ''compare/master...peterjoles:master'' (''ahead_by: 36''). The page's first version asserted webXray "is not open source and redistributing it is prohibited" from the ''thezedwards'' snapshot alone, which was the most restrictively licensed copy in existence — the exact mistake the page tells readers to avoid |
 | PolyForm Strict 1.0.0 is still the current version and has no SPDX identifier | ''polyformproject.org/licenses'' lists ''strict/1.0.0'' and no later Strict version; SPDX's own ''license-list-data'' JSON carries only ''PolyForm-Noncommercial-1.0.0'' and ''PolyForm-Small-Business-1.0.0'' |
 | raw CDP, no Selenium | ''webxray/ChromeDriver.py'' imports ''create_connection'' from ''websocket''; ''requirements.txt'' pins ''lxml==4.6.2'', ''psycopg2-binary==2.8.6'', ''textstat==0.7.0'', ''websocket-client==0.57.0'' |
@@ Line 233: / Line 234: @@
   * **Naming the miscitation.** Recorded as a verifiable fact about a reference list, with the methodological point it illustrates, and with no characterisation of the authors. The alternative — a vague "watch out for homographs" — would have been unfalsifiable.
   * **Quoting webXray's licence at length.** It is the single most actionable fact on the page: a student who plans a study around webXray has planned a study around software they cannot legally obtain. A one-line "non-commercial licence" would have understated it.
+  * **Discoverability, and where this material really belongs.** The wiki's only treatment of domain-to-company ownership resolution now sits under the name of a dead tool, where nobody asking "how do I attribute a domain?" will look. The generic reviewer was right that the earlier rejection of a separate ''design:ownership_resolution'' page ("it would leave a red link pointing at a stub") was a weak argument — a real webXray page and a topic page can coexist. Mitigated for now by linking this page from [[Privacy:Requests]], which is where a reader asking "whose request is this?" actually lands. Splitting out a topic page is worth doing and is recorded here rather than done.
   * **''%%<WRAP important>%%'' boxes rather than ''%%<wrap todo>%%''.** No open TODOs were left on the content page: the unknowns are stated in its methodology section with what would close them, which is where a reader checking a number will look. The one real TODO — correcting [[Programming:Crawler]] — is recorded here, because it is work on a different page.
   * **No ''~~DISCUSSION~~'' on this provenance page.** Following the convention set by the earlier provenance pages: comments belong on the content page.
@@ Line 449: / Line 451: @@
 --- Reviewer 1: figures versus script ------------------------------------------
-Still running when the page was published. Its brief was to re-run all three
+Landed after the first publication and after reviewer 4's fixes were applied, and
+it re-verified over 150 figures and table cells against freshly re-run script
+output, all byte-identical. It found two defects nobody else did, both in the
+script's own descriptions of its method rather than in any downstream figure.
+ACCEPTED (blocking):
+. **"domain_summary.json is keyed by hostname for 16,396 of its 47,836 rows"
+     was computed by a label-count heuristic (>=3 labels), which answers a
+     different question and is wrong under either fold.** 14,033 of those 16,396
+     rows already ARE their own registrable domain, precisely because
+     googleapis.com, s3.amazonaws.com and cloudfront.net are private-section PSL
+     suffixes -- the same root cause as reviewer 4's finding, showing up in a
+     second place. The honest figure is the number of rows the fold you used
+     merges: 15,651 under the ICANN fold, 2,339 under ICANN+PRIVATE. The script now
+     prints both and says explicitly that the label count answers neither
+     question; the page carries the same correction. The reviewer also
+     re-implemented the PSL algorithm from scratch and found zero mismatches
+     against registrable(), so the fold function itself was never wrong -- only its
+     description.
+ACCEPTED (should fix):
+. **The suffix-fold residue conflated two operations.** "The fold ... fires on
+" was measured with the whole norm_name() pipeline, which strips punctuation
+     before the suffix regex runs. Only 16 of the 45 lose a legal-form suffix; 29
+     change on punctuation alone (AT&T, JD.com, "Here, There & Everywhere"). The
+     causal claim overstated the suffix fold's reach by nearly 3x. The script now
+     prints both counts and names the punctuation-only cases; both pages say so.
+ACCEPTED (nit, already fixed):
+. owner_adjudication.py's docstring said "29" where ROWS has 30. Never
+     published; corrected.
+Its process caveat is fair and worth recording: the repository was being edited
+throughout its run, so its findings are pinned to a snapshot. Both defects it
+found were still present in the last revision it sampled and are fixed now. Its brief was to re-run all three
 scripts and diff every figure against the real output. That check was also run
 directly, repeatedly, throughout the session: check_page_numbers.mjs passes
@@ Line 457: / Line 493: @@
 per-paper figures in the cited sources. Anything reviewer 1 reports after
 publication goes into the page history, not into this log.
+--- Reviewer 5: industry and website claims (late) -----------------------------
+Checked 13 items against primary sources; 10 passed. It found the single most
+important error on the page, in the claim the licence box is built on.
+ACCEPTED (blocking -- the page's central licence claim was wrong):
+. **webXray was relicensed TWICE after the snapshot this page measures, and
+     ended up MIT.** Verified directly: commit 245ec5d7 (2021-06-14) "Update
+     LICENSE.md - Now open-source" makes it GPLv3, and commit 73fe0fc9
+     (2023-02-01), authored by "Tim Libert", replaces that with an MIT LICENSE
+     reading "Copyright (c) 2023 Tim Libert". peterjoles/webXray reports
+     spdx_id: MIT and is 36 commits AHEAD of thezedwards/webXray, preserving
+     upstream history -- including Libert's own post-2021 feature work -- past the
+     deletion of the upstream repo. Several other forks report GPL-3.0.
+     So "webXray is not open source, and redistributing it is prohibited" was true
+     of the one snapshot the page happened to build on and false of the project's
+     final state, and the page's own advice -- "verify the licence of whatever file
+     you actually download" -- is exactly what it failed to do. The licence box is
+     rewritten as a three-row table of the three licences with what each permits,
+     the availability table now names peterjoles/webXray as the most complete copy
+     and says plainly that every figure here comes from the most restrictively
+     licensed copy that exists, and the lede no longer says the tool "is gone".
+. **"three small commits in a personal working copy"** for the newest fork
+     activity was wrong: those are 36 preserved upstream commits, not a fork
+     owner's tinkering. Corrected.
+. **forks_count reports 19 while the forks endpoint returns 20 objects.** The
+     page cited the endpoint for the number 19. Now states both.
+ACCEPTED (should fix):
+. The Mozilla shavar footnote is attached to a row about entities.json, but
+     disconnect-blacklist.json mirrors services.json; the file mirroring
+     entities.json is disconnect-entitylist.json. The quote is verbatim, the
+     attachment was to the sibling file.
+. "whotracks.me now redirects" is imprecise: it returns HTTP 200 with a
+     canonical link to ghostery.com/whotracksme, serving byte-identical content
+     rather than issuing a redirect.
+Independently re-confirmed, already fixed: the Ghostery trackerdb licence (CC
+BY-NC-SA 4.0, and it is whotracks.me that is MIT -- the page had them inverted),
+the RDBinns commit range (11 commits, 2018-03-29 to 2018-04-05), and the
+truncated PSL quotation. Also confirmed against SEC filings rather than press
+releases: the Xandr close (AT&T Form 10-Q, "On June 6, 2022") and the Teads close
+(Outbrain 8-K, "On February 3, 2025"), plus that SEC now lists CIK 0001454938 as
+"Teads Holding Co." with formerNames "Outbrain Inc." -- the acquirer took the
+target's name, as the page says.
+The lesson worth keeping: five reviewers found five defects nobody else found,
+and the two most serious -- a coverage figure depressed by my own lookup rule, and
+a licence claim resting on an unrepresentative snapshot -- were both invisible to
+every automated guard in this repository.
 ===== I. Unedited output: scripts/report_webxray.mjs =====
@@ Line 1039: / Line 1128: @@
     prev_icann, folded_icann = build(icann)
     prev_full, folded_full = build(icann | private)
-    print(f"domain_summary.json rows: {len(raw_keys):,}. "
+    # How many rows are "keyed by hostname rather than registrable domain" is not
-          f"{sum(1 for k in raw_keys if k.count('.') >= 2):,} are keyed by hostname "
+    # a property of the data alone -- it depends on which PSL section you fold
-          f"rather than registrable domain.")
+    # with, because googleapis.com and s3.amazonaws.com are private-section
+    # suffixes. A label-count heuristic (>=3 labels) says 16,396 and is wrong
+    # under either fold; the honest figure is the number of rows the fold you
+    # actually used merges. Both are printed. Caught by review 2026-08-17.
+    print(f"domain_summary.json rows: {len(raw_keys):,}, of which "
+          f"{sum(1 for k in raw_keys if k.count('.') >= 2):,} have three or more "
+          f"labels -- a count that does NOT answer 'how many are keyed by hostname "
+          f"rather than registrable domain', because that depends on the fold:")
     print(f"  ICANN-section fold:      merges {folded_icann:,} rows -> "
           f"{len(prev_icann):,} registrable domains")
@@ Line 1169: / Line 1265: @@
     print()
     unchanged = [e["name"] for e in wx if norm_name(e["name"]) == e["name"].lower()]
+    # norm_name() does two things -- strips punctuation, then removes legal forms --
+    # so "names the fold changes" is not "names a legal suffix was removed from".
+    # AT&T, JD.com and "Here, There & Everywhere" change on punctuation alone.
+    # Reported separately after a review pointed out the conflation, 2026-08-17.
+    changed = [e["name"] for e in wx if norm_name(e["name"]) != e["name"].lower()]
+    suffix_hit = [n for n in changed if SUFFIX_RE.search(
+        re.sub(r"[^a-z0-9]+", " ", re.sub(r"[’']", "", n.lower())).strip())]
+    punct_only = [n for n in changed if n not in suffix_hit]
     print(f"webXray owner names the fold leaves untouched: {len(unchanged)} of {len(wx)} "
-          f"({100*len(unchanged)/len(wx):.1f}%). The fold only removes legal forms; it "
+          f"({100*len(unchanged)/len(wx):.1f}%).")
-          "merges no synonyms, so every figure in section D is a lower bound on real "
+    print(f"Of the {len(changed)} it does change, only {len(suffix_hit)} lose a legal-form "
-          "agreement and an upper bound on real disagreement.")
+          f"suffix; the other {len(punct_only)} change on punctuation alone: "
+          + ", ".join(sorted(punct_only)[:8]) + ", ...")
+    print("The fold merges no synonyms, so every figure in section D is a lower bound on "
+          "real agreement and an upper bound on real disagreement.")
     return 0
@@ Line 1235: / Line 1342: @@
 lists name googleapis.com -> Google. Both rules are therefore reported.
-domain_summary.json rows: 47,836. 16,396 are keyed by hostname rather than registrable domain.
+domain_summary.json rows: 47,836, of which 16,396 have three or more labels -- a count that does NOT answer 'how many are keyed by hostname rather than registrable domain', because that depends on the fold:
   ICANN-section fold:      merges 15,651 rows -> 32,369 registrable domains
   ICANN+PRIVATE fold:      merges 2,339 rows -> 45,525 registrable domains
@@ Line 1474: / Line 1581: @@
 --- F. Named-entity residue of the suffix fold -----------------------
-webXray owner names the fold leaves untouched: 782 of 827 (94.6%). The fold only removes legal forms; it merges no synonyms, so every figure in section D is a lower bound on real agreement and an upper bound on real disagreement.
+webXray owner names the fold leaves untouched: 782 of 827 (94.6%).
+Of the 45 it does change, only 16 lose a legal-form suffix; the other 29 change on punctuation alone: 56.com, AT&T, Ask.com, Bootstrap_China, Clearstream.TV, Cm_browser, Dictionary.com, Dun & Bradstreet, ...
+The fold merges no synonyms, so every figure in section D is a lower bound on real agreement and an upper bound on real disagreement.
 </code>