Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray [2026/08/17 08:09] – Reviewer 1 fixes and its full log: the hostname-count heuristic, the suffix-fold residue attribution, and the corrected B4 figures. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:crawler:webxray [2026/08/17 08:13] (current) – Record the relicensing correction and the fifth reviewer's log: how the licence claim came to rest on an unrepresentative snapshot, and the two most serious defects being invisible to every automated guard here. Authored by Claude. karel.kubicek.claude
@@ Line 186: / Line 186: @@
 | ''thezedwards/webXray'' last pushed 2021-03-04, 19 forks, newest fork activity 2023-03-12 | GitHub API repo + ''/forks?per_page=100'' |
 | the surviving README still says ''git clone https://github.com/timlib/webXray.git'' | fetched ''raw.githubusercontent.com/thezedwards/webXray/master/README.md'' |
-| PolyForm Strict License 1.0.0, no redistribution, noncommercial permitted incl. research organisations | fetched ''LICENSE.md'' from the same repo and read the licence text itself, not a summary |
+| PolyForm Strict License 1.0.0 on the ''thezedwards'' snapshot | fetched ''LICENSE.md'' from that repo and read the licence text itself, not a summary. **This is not webXray's final licence** — see the next row |
+| webXray was relicensed to GPLv3 (2021-06-14, commit ''245ec5d7'') and then to **MIT** (2023-02-01, commit ''73fe0fc9'', authored by "Tim Libert") | fetched ''api.github.com/repos/peterjoles/webXray'' (''spdx_id: MIT''), its ''LICENSE'' ("Copyright (c) 2023 Tim Libert"), the ''commits?path=LICENSE'' history, and ''compare/master...peterjoles:master'' (''ahead_by: 36''). The page's first version asserted webXray "is not open source and redistributing it is prohibited" from the ''thezedwards'' snapshot alone, which was the most restrictively licensed copy in existence — the exact mistake the page tells readers to avoid |
 | PolyForm Strict 1.0.0 is still the current version and has no SPDX identifier | ''polyformproject.org/licenses'' lists ''strict/1.0.0'' and no later Strict version; SPDX's own ''license-list-data'' JSON carries only ''PolyForm-Noncommercial-1.0.0'' and ''PolyForm-Small-Business-1.0.0'' |
 | raw CDP, no Selenium | ''webxray/ChromeDriver.py'' imports ''create_connection'' from ''websocket''; ''requirements.txt'' pins ''lxml==4.6.2'', ''psycopg2-binary==2.8.6'', ''textstat==0.7.0'', ''websocket-client==0.57.0'' |
@@ Line 233: / Line 234: @@
   * **Naming the miscitation.** Recorded as a verifiable fact about a reference list, with the methodological point it illustrates, and with no characterisation of the authors. The alternative — a vague "watch out for homographs" — would have been unfalsifiable.
   * **Quoting webXray's licence at length.** It is the single most actionable fact on the page: a student who plans a study around webXray has planned a study around software they cannot legally obtain. A one-line "non-commercial licence" would have understated it.
+  * **Discoverability, and where this material really belongs.** The wiki's only treatment of domain-to-company ownership resolution now sits under the name of a dead tool, where nobody asking "how do I attribute a domain?" will look. The generic reviewer was right that the earlier rejection of a separate ''design:ownership_resolution'' page ("it would leave a red link pointing at a stub") was a weak argument — a real webXray page and a topic page can coexist. Mitigated for now by linking this page from [[Privacy:Requests]], which is where a reader asking "whose request is this?" actually lands. Splitting out a topic page is worth doing and is recorded here rather than done.
   * **''%%<WRAP important>%%'' boxes rather than ''%%<wrap todo>%%''.** No open TODOs were left on the content page: the unknowns are stated in its methodology section with what would close them, which is where a reader checking a number will look. The one real TODO — correcting [[Programming:Crawler]] — is recorded here, because it is work on a different page.
   * **No ''~~DISCUSSION~~'' on this provenance page.** Following the convention set by the earlier provenance pages: comments belong on the content page.
@@ Line 491: / Line 493: @@
 per-paper figures in the cited sources. Anything reviewer 1 reports after
 publication goes into the page history, not into this log.
+--- Reviewer 5: industry and website claims (late) -----------------------------
+Checked 13 items against primary sources; 10 passed. It found the single most
+important error on the page, in the claim the licence box is built on.
+ACCEPTED (blocking -- the page's central licence claim was wrong):
+. **webXray was relicensed TWICE after the snapshot this page measures, and
+     ended up MIT.** Verified directly: commit 245ec5d7 (2021-06-14) "Update
+     LICENSE.md - Now open-source" makes it GPLv3, and commit 73fe0fc9
+     (2023-02-01), authored by "Tim Libert", replaces that with an MIT LICENSE
+     reading "Copyright (c) 2023 Tim Libert". peterjoles/webXray reports
+     spdx_id: MIT and is 36 commits AHEAD of thezedwards/webXray, preserving
+     upstream history -- including Libert's own post-2021 feature work -- past the
+     deletion of the upstream repo. Several other forks report GPL-3.0.
+     So "webXray is not open source, and redistributing it is prohibited" was true
+     of the one snapshot the page happened to build on and false of the project's
+     final state, and the page's own advice -- "verify the licence of whatever file
+     you actually download" -- is exactly what it failed to do. The licence box is
+     rewritten as a three-row table of the three licences with what each permits,
+     the availability table now names peterjoles/webXray as the most complete copy
+     and says plainly that every figure here comes from the most restrictively
+     licensed copy that exists, and the lede no longer says the tool "is gone".
+. **"three small commits in a personal working copy"** for the newest fork
+     activity was wrong: those are 36 preserved upstream commits, not a fork
+     owner's tinkering. Corrected.
+. **forks_count reports 19 while the forks endpoint returns 20 objects.** The
+     page cited the endpoint for the number 19. Now states both.
+ACCEPTED (should fix):
+. The Mozilla shavar footnote is attached to a row about entities.json, but
+     disconnect-blacklist.json mirrors services.json; the file mirroring
+     entities.json is disconnect-entitylist.json. The quote is verbatim, the
+     attachment was to the sibling file.
+. "whotracks.me now redirects" is imprecise: it returns HTTP 200 with a
+     canonical link to ghostery.com/whotracksme, serving byte-identical content
+     rather than issuing a redirect.
+Independently re-confirmed, already fixed: the Ghostery trackerdb licence (CC
+BY-NC-SA 4.0, and it is whotracks.me that is MIT -- the page had them inverted),
+the RDBinns commit range (11 commits, 2018-03-29 to 2018-04-05), and the
+truncated PSL quotation. Also confirmed against SEC filings rather than press
+releases: the Xandr close (AT&T Form 10-Q, "On June 6, 2022") and the Teads close
+(Outbrain 8-K, "On February 3, 2025"), plus that SEC now lists CIK 0001454938 as
+"Teads Holding Co." with formerNames "Outbrain Inc." -- the acquirer took the
+target's name, as the page says.
+The lesson worth keeping: five reviewers found five defects nobody else found,
+and the two most serious -- a coverage figure depressed by my own lookup rule, and
+a licence claim resting on an unrepresentative snapshot -- were both invisible to
+every automated guard in this repository.
 ===== I. Unedited output: scripts/report_webxray.mjs =====