Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray [2026/09/17 07:24] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude
+++ provenance:programming:crawler:webxray [2026/09/22 04:23] (current) – Escape the literal {[citekey]} in the 2026-09-22 scope note: ''..'' monospace does not escape the bibtex plugin, so it rendered as a live unresolved citation marker. Authored by Claude karel.kubicek.claude
@@ Line 67: / Line 67: @@
 | A8 | Per-year and per-venue shape | year/venue of the 15 | 15 | last use of crawler-or-list: **2022** |
 | A9 | Which ownership resources does the corpus name at all? | six full-text sweeps | 5,859 | PSL 101, Disconnect-as-a-list 74, Tracker Radar 32, WhoTracks.me 23, Crunchbase 23, webXray 15 |
-| A10 | Union of the five owner databases | union of A9 rows minus the PSL row | 5,859 / 1,120 crawled | **136** (2.3% / 12.1%) |
+| A10 | Union of the five owner databases | union of A9 rows minus the PSL row | 5,859 / 1,120 crawled | **136** (2.3% of the corpus); **107** of them are among the 1,120 that crawled (**9.6%** of that population) |
 | A11 | Which Tracker Radar artefact did each of the 32 use? | ''TR_ROLE'' hand map | 32 | ownership 11, tracker/category 9, Collector crawler 9, citation 2, compared 1 |
 | A12 | Ownership use of Tracker Radar over time | year of the 11 | — | 2021:1, 2023:2, 2024:1, 2025:4, 2026:3 |
@@ Line 89: / Line 89: @@
 ==== A.16 Evidence quotes spot-checked ====
-The 16 ''evidence.quote'' values behind the webXray tool and classification tuples, checked by 4-word-window coverage against ''paper.cols.txt'': **4 exact, 11 partial (≥60% of windows), 1 below threshold.** Full list in section H of the report output.
+The 16 ''evidence.quote'' values behind the webXray tool and classification tuples, checked by 4-word-window coverage against ''paper.cols.txt'', with a ''pypdf'' rendering of the same ''paper.pdf'' consulted for anything below threshold: **4 exact, 11 partial (≥60% of windows), 0 rescued from the PDF, 1 below threshold in both renderings.** Full list in section H of the report output.
 The one below threshold is {[musa2022_atom]} at 58%: "We used external data sources including WHOIS records, TLS certificates, and WebXray [61] to identify the parent organizations of each identified tracker." Read by hand in the source: **present and verbatim**; the ''.cols'' rendering splices an adjacent column through it. Below-threshold is not "unsupported".
@@ Line 1332: / Line 1332: @@
 Schema signal: tools[].name matching /webx[\s-]?ray/i with usedOrMentioned in {used, produced} fires on 7 papers.
 Not used/produced (compared/mentioned/unclear): 0 papers.
-Full-text sweep of paper.cols.txt for the same pattern: 15 papers, 0.3% of the 5859-paper corpus and 1.3% of the 1120 that ran a crawl.
+Full-text sweep of paper.cols.txt for the same pattern: 15 papers, 0.3% of the 5859-paper corpus. 13 of them are among the 1120 that ran a crawl (1.2% of that population).
 Schema papers not in the sweep: none (a sweep miss would mean the extractor read a rendering the sweep cannot see).
 Sweep papers the schema misses: 8.
@@ Line 1430: / Line 1430: @@
 Papers matching ANY of webXray / Tracker Radar / WhoTracks.me / Crunchbase / Disconnect-as-a-list:
-papers (2.3% of the corpus; 12.1% of the 1120 that crawled).
+papers (2.3% of the corpus). 107 of them are among the 1120 that crawled -- 9.6% of that population.
 Year   Corpus papers  Naming an ownership resource  Share
 -----  -------------  ----------------------------  -----
@@ Line 1519: / Line 1519: @@
 of 26 literal figures/quotes located in the cited paper's .cols rendering.
-=== H. Evidence quotes behind the schema tuples ===
+=== H. Evidence quotes behind the schema tuples (with PDF fallback) ===
 partial          69%  WWW/2018/an-automated-approach-to-auditing-disclosure-of-third-party-data-collection-in-w | webxray | methodology | "the webxray software platform is used to monitor third-party network traffic generated by loading a given web page and attribute such traffic to the entities which receive the data."
@@ Line 1537: / Line 1537: @@
 partial          67%  IEEE-SP/2022/journey-to-the-center-of-the-cookie-ecosystem-unraveling-actors-roles-and-relati | classification:webxray | methodology | "The first is based on three manually-curated lists (Disconnect [30], WhoTracks.me [31] and webxray [32])"
 exact           100%  IEEE-SP/2022/journey-to-the-center-of-the-cookie-ecosystem-unraveling-actors-roles-and-relati | classification:graph-based label propagation algorithm (custom) | methodology | "We validated our approach by using the manually compiled lists as a “ground truth” reference: we then looked for conflicts"
-Summary: 16 quotes: 4 exact, 11 partial (>=60% of 4-word windows), 1 below threshold.
+Summary: 16 quotes: 4 exact, 11 partial (>=60% of 4-word windows), 0 rescued from the PDF, 1 below threshold in both renderings.
 Below-threshold is not "unsupported": every one checked by hand was present, mangled by a column splice or a dropped citation marker.
@@ Line 1547: / Line 1547: @@
   owner_dbs.py, snapshots of 2026-08-17: webXray 827 owners / 3,215 domains; Tracker Radar 19,148 / 38,368; Disconnect 1,887 / 7,850.
   owner_dbs.py: webXray owners with a parent_id 319 of 827 (38.6%); tree depth up to 6; largest owner groupm with 620 domains.
-  owner_dbs.py: coverage of 45,525 registrable third-party domains -- webXray 657 (1.4%), Tracker Radar 5,539 (12.2%), Disconnect 2,277 (5.0%).
+  owner_dbs.py: coverage of 32,369 registrable third-party domains -- webXray 669 (2.1%), Tracker Radar 5,581 (17.2%), Disconnect 2,268 (7.0%).
-  owner_dbs.py: prevalence-weighted coverage -- webXray 54.5%, Tracker Radar 79.3%, Disconnect 75.8%.
+  owner_dbs.py: prevalence-weighted coverage -- webXray 58.6%, Tracker Radar 84.3%, Disconnect 80.5%.
-  owner_dbs.py: top 100 by prevalence -- webXray 70%, Tracker Radar 95%, Disconnect 91%; top 1,000 -- 26 / 67 / 67%; top 10,000 -- 5 / 27 / 16%.
+  owner_dbs.py: top 100 by prevalence -- webXray 71%, Tracker Radar 98%, Disconnect 94%; top 1,000 -- 26 / 69 / 68%; top 10,000 -- 5 / 29 / 17%.
-  owner_dbs.py: pairwise "needs a human" disagreement -- webXray vs TR 197 of 601 (32.8%), webXray vs Disconnect 216 of 461 (46.9%), TR vs Disconnect 584 of 1,532 (38.1%).
+  owner_dbs.py: pairwise "needs a human" disagreement -- webXray vs TR 198 of 612 (32.4%), webXray vs Disconnect 217 of 464 (46.8%), TR vs Disconnect 584 of 1,538 (38.0%).
-  owner_dbs.py: 16,396 of 47,836 domain_summary.json rows are hostnames not registrable domains; 19 rows are not hostnames at all, one of them the literal string "null".
+  owner_dbs.py: the ICANN-section fold merges 15,651 of domain_summary.json's 47,836 rows into 32,369 registrable domains; 19 rows are not hostnames at all, one of them the literal string "null". (The label-count heuristic that gives 16,396 answers neither question and is not used.)
   webXray licence: PolyForm Strict License 1.0.0 (LICENSE.md in the surviving copy).
   Tracker Radar and Disconnect licences: CC BY-NC-SA 4.0 (LICENSE and entities.json "license" field).
@@ Line 2431: / Line 2431: @@
 Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Embedded-output refresh, 2026-09-22 =====
+**What was wrong.** The counts on this page were right; the //embedded copy of the
+script's output// in [[#I. Unedited output: scripts/report_webxray.mjs|§I]] was a
+pre-2026-09-11 paste. ''report_webxray.mjs'' has been re-run and the whole block
+regenerated rather than line-edited, so what §I shows is again what the script prints.
+Trigger: the shared quote checkers gained a ''pypdf'' fallback on 2026-09-04 and
+''report_webxray.mjs'' was patched to use it, so its Summary line changed from
+"1 below threshold" to "0 rescued from the PDF, 1 below threshold in both renderings".
+The quote **counts** did not move — 16 quotes, 4 exact, 11 partial, 1 below threshold —
+so the wording was the only thing this refresh was filed for. Diffing the rest of the
+block against the fresh output found four more stale lines, listed below.
+<code>
+$ node scripts/report_webxray.mjs --quotes
+Summary: 16 quotes: 4 exact, 11 partial (>=60% of 4-word windows), 0 rescued from the PDF, 1 below threshold in both renderings.
+</code>
+^ Line in §I (or, for A10, in §A) ^ Was ^ Is ^ Why ^
+| §H Summary | ''1 below threshold'' | ''0 rescued from the PDF, 1 below threshold in both renderings'' | the fallback added 2026-09-04; **no count moved** |
+| §H heading | ''Evidence quotes behind the schema tuples'' | ''… (with PDF fallback)'' | same patch |
+| §A sweep line | ''15 papers … and 1.3% of the 1120 that ran a crawl'' | ''15 papers … 13 of them are among the 1120 that ran a crawl (1.2% of that population)'' | the 2026-09-11 denominator fix: a corpus-wide numerator was being divided by the crawled denominator. [[:programming:crawler:webxray]] has carried the corrected 1.2% since then; only this embedded paste was stale |
+| §E union line | ''136 papers (2.3% of the corpus; 12.1% of the 1120 that crawled)'' | ''136 papers (2.3% of the corpus). 107 of them are among the 1120 that crawled -- 9.6% of that population'' | same fix, larger figure |
+| **§A, row A10** | ''**136** (2.3% / 12.1%)'' | ''**136** (2.3% of the corpus); **107** … (**9.6%**)'' | the only **prose** figure this refresh moved. The content page and ''report_webxray.mjs'' both say 107 / 9.6%; this row still said 12.1% |
+| §Z, five ''owner_dbs.py'' lines | 45,525-domain frame; 54.5 / 79.3 / 75.8%; 16,396 rows | 32,369-domain frame; 58.6 / 84.3 / 80.5%; the ICANN fold merges 15,651 of 47,836 rows | the ICANN-section fold, adopted 2026-09-05. [[#C3.1 The frame, and what it excludes|§C3.1]], §B4 and §B6 have carried the 32,369 frame since then; again only the paste was stale |
+**The lesson, and it is the one this site keeps relearning.** Every figure in the
+prose was already correct. What was stale was the block that exists to //prove// the
+prose — so a reader checking the page against its own embedded output would have been
+shown a contradiction, in the direction of the numbers that had already been retracted.
+An embedded output block has to be regenerated whenever its script changes, not when
+its headline figure changes; and ''check_page_numbers.mjs'' cannot see this, because
+every value in the stale block is a real number that really was printed once.
+**Scope of this edit.** The §I block in full, the §A10 row, and the one sentence in §5
+that quotes the quote-check verdicts. ''owner_dbs.py'' and ''owner_sample.py'' were
+**not** re-run — the §Z lines are literals inside ''report_webxray.mjs'' and were taken
+from its output, not recomputed. No other figure, fold or citation on this page was
+touched, and no ''%%{[citekey]}%%'' was added or moved.
+^ Item ^ Value ^
+| Date | 2026-09-22, unsupervised |
+| Command | ''node scripts/report_webxray.mjs %%--quotes%%'' |
+| Artifact | ''out/qc0922/report_webxray_quotes.txt'' (the full 232-line run pasted into §I) |
+| Script changes | none — ''report_webxray.mjs'' already had the fallback |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Pages saved | this page only — [[:programming:crawler:webxray]] already carries 107 / 9.6% / 1.2% and was verified unchanged |