Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: design:website_classification ======

Working notes behind [[design:website_classification]] — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on [[literature:corpus]] and are not restated here.

<WRAP important>
**Partly reconstructed.** The content page was written on 2026-08-07, before this site had a provenance convention. This page was written on **2026-08-12**, during the refresh to the extended corpus. Sections marked //recorded// were produced by the 2026-08-12 run. Sections marked //reconstructed// were rebuilt from the report script, its output, the page text and the traps recorded in the workdir README. The page's large non-corpus half — the service catalogue, the company datasets, the LLM preprint analysis and the ''category_validation.py'' script — was researched and verified by the original run and was **not** re-verified here, with the one exception noted in §7.
</WRAP>

===== 1. What this page is backing =====

^ Item ^ Value ^
| Content page | [[design:website_classification]] |
| Report script | ''scripts/report_website_classification.mjs'' (''--wiki'', ''--list'', ''--quotes <regex>'') |
| Folds | ''scripts/webcat_fold.mjs'' — a **task** fold and a **resource** fold |
| Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
| Refreshed | 2026-08-12 |
| Bibliography change | added ''tsai2025_harmful'' (TheWebConf 2025) |

===== 2. Populations and denominators =====

//Recorded.// The population narrowing is this page's defining methodological move and the report prints every step of it.

^ Step ^ Old ^ New ^
| corpus | 4,322 | 5,859 |
| ≥1 ''classification'' tuple | 3,241 | 4,439 |
| ≥1 tuple with ''target == "website-category"'' | 320 | 424 |
| …of which used or produced it (not merely compared) | 320 | 424 |
| **PAGE POPULATION: task folds to ''topic-or-industry''** | **247** | **330** |
| excluded (fingerprinting attack, abuse verdict, page type, hosting, tech fingerprint, third-party role) | 73 | 94 |

The exclusion matters because ''website-category'' fires on at least seven unrelated tasks. **Website fingerprinting is a traffic-analysis attack** that shares no taxonomy, ground truth or service with topic categorisation; 36 of the 424 are that. Eight of those had generic classifier names ("a 1D CNN", "k-NN") leaving no signal in any extracted field and are excluded by an explicit named list keyed on the paper slug, not by a general rule.

===== 3. Running it =====

<code bash>
cd /workspace/artifacts/wiki
node scripts/webcat_fold.mjs                            # self-test, prints residue
node scripts/report_website_classification.mjs          # every figure
node scripts/report_website_classification.mjs --wiki
node scripts/report_website_classification.mjs --list   # the 330 papers
node scripts/report_website_classification.mjs --quotes 'McAfee'
node scripts/check_page_numbers.mjs \
  pages/design_website_classification.txt out/new/report_website_classification.txt \
  '===== Use in Publications =====' '===== Open Questions ====='
</code>

''check_page_numbers.mjs'' left four figures unaccounted. **Three were deliberate** — ''1,000'' (DarkGram's validation sample, from the paper), ''500'' (an image width in the ''{{...?nolink&500}}'' syntax, a false positive of the checker worth knowing about) and ''7.4'' (the old cross-check share, quoted on the page as the comparison). **The fourth was a real catch**: the page still said "the 73 excluded papers" after the count had moved to 94. Run it whole-page too (§10).

===== 4. What the refresh changed =====

//Recorded.// This page moved more than any other in the refresh, for two independent reasons: IEEE S&P went from 43%-retrieved to complete, and LLM classification arrived.

^ Figure ^ Old ^ New ^
| Page population | 247 | 330 |
| ''website-category'' tuples | 320 | 424 |
| Website fingerprinting (excluded) | 22 | 36 |
| Abuse verdict (excluded) | 47 | 63 |
| **IEEE S&P** | **3 (1.2% of population, 1.1% of venue)** | **25 (7.6%, 3.3%)** |
| TheWebConf | 59 (8.3% of venue) | 71 (8.4%) |
| USENIX Security | 45 (4.0%) | 58 (4.1%) |
| IMC | 45 (8.1%) | 50 (7.8%) |
| PETS | 29 (8.2%) | 43 (8.4%) |
| Third-party service (method) | 105 (42.5%) | 131 (39.7%) |
| Manual labelling | 53 (21.5%) | 68 (20.6%) |
| Heuristic rules | 39 (15.8%) | 59 (17.9%) |
| Curated database | 31 (12.6%) | 46 (13.9%) |
| **''llm'' method** | **0** | **2 (0.6%)** |
| Third-party service by window | 29.2 / 35.8 / 44.4 / 48.8% | 25.0 / 33.9 / 43.5 / 49.4 / **32.3%*** |
| Curated database by window | 16.7 / 18.9 / 11.1 / 8.8% | 14.3 / 21.4 / 10.9 / 9.0 / **18.5%*** |
| **LLM papers corpus-wide** | **29 (2 in 2023, 27 in 2024)** | **177 (2 / 27 / 77 / 71)** |
| …validate the LLM's labels | 24 of 29 (82.8%) | 148 of 177 (83.6%) |
| …use an LLM for a website TOPIC | **0** | **2** |
| McAfee | 19 papers / 15 spellings | 23 / 18 |
| Symantec / Blue Coat | 12 / 10 | 14 / 10 |
| Media bias raters | 5 / 5 | 10 / 9 |
| Cloudflare domain categories | 6 / 3 | 10 / 6 |
| VirusTotal (in topic population) | 3 | 7 |
| Names ≥1 third-party service | 122 of 247 (49.4%) | 155 of 330 (47.0%) |
| Names ≥2 services | 9 (7.4%) | 9 (5.8%) |
| Hand-built rules / keywords family | 61 (24.7%) | 87 (26.4%) |
| Unnamed custom method | 38 | 58 |
| Names the taxonomy | 184 (74.5%) | 248 (75.2%) |
| Names a ground-truth source | 124 (50.2%) | 169 (51.2%) |
| Reports any validation | 95 (38.5%) | 123 (37.3%) |
| Ground truth stated, all classifying papers | 2,324/3,241 = 71.7% | 3,234/4,439 = 72.9% |
| Resource-fold residue | 20 strings | 40 → **33** after extension |

**The three findings that are more than arithmetic:**

  - **IEEE S&P: 3 papers → 25.** The old page said IEEE S&P "essentially never" categorises websites and, in the same breath, that "IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share". **The caveat was right and the fix confirms it.** The page now carries a box saying so, because the general lesson is worth more than the number: a venue that looks uninterested in a topic should be checked for a retrieval gap before it is described as uninterested.
  - **LLM classification: 29 papers → 177**, from 0.3% of 2023 to 17.1% of the partial 2026. This is the fastest-moving quantity anywhere in this corpus. **Two papers now use an LLM for website topic classification**, where the old page could truthfully say zero did. The stronger of the two, ''WWW/2025/harmful-terms-and-where-to-find-them…'' (''tsai2025_harmful''), reports **92% accuracy for GPT-4o prompted with URL + screenshot** on shopping-vs-non-shopping over 500 manually annotated Tranco sites and states that this is "comparable to commercial website classification services". Verified against the source (§6).
  - **The closed-source drift reverses in 2025–2026 — provisionally.** Third-party-service reliance had risen for four windows straight (25.0 → 49.4%) and falls to 32.3%; curated databases had fallen (21.4 → 9.0%) and return to 18.5%. **Two incomplete venue-years cannot establish a turn** and the page says so explicitly, but this is the first movement against a fifteen-year trend the page's central argument rests on, and it is the thing to re-check when 2026 is complete.

===== 5. Folds =====

==== 5.1 The task fold — unchanged, and it held ====

//Recorded.// ''taskFamily()'' was not modified. The task distribution is stable across the corpus growth (topic-or-industry 77.2% → 77.8%), which is reassuring given that the whole page rests on it. The eight hand-excluded website-fingerprinting papers are still excluded by slug.

==== 5.2 The resource fold — 20 → 40 strings, extended back to 33 ====

//Recorded.// Six additions, each defensible, each recorded here because a fold change moves every number downstream of it:

^ Residue string ^ Folded to ^ Why ^
| ''Homepage2Vec'' | **new entry**, ''pretrained-lm'' | The page has a whole section recommending Homepage2Vec. Leaving it in the residue while recommending it was straightforwardly inconsistent. |
| ''BART-large-mnli'' | **new entry** "BART / zero-shot NLI classifier", ''pretrained-lm'' | a pretrained model used zero-shot |
| ''Science Feedback'' | Media bias raters (AllSides, MB/FC) | a media-credibility rater, same family |
| ''IBM Cloud Natural Language Understanding'' | IBM Watson | the same product under its current name |
| ''WhoisXMLAPI'' and ''whoisxmlapi.com'' | **new entry**, ''aggregator'' | **two spellings of one service sitting separately in the residue — a genuine folding failure, not a long tail** |
| ''Netstar URL Categorization…'' | **new entry**, ''security-filtering'' | a URL-categorisation vendor of exactly the catalogued kind |

**Deliberately left in the residue, 33 strings, one paper each.** They are printed in full by the report and on the content page in summary. The judgement is that these are a genuine long tail rather than a fold failure: ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''ORES topic routing'', ''NAICSlite'', ''IAB taxonomy'' (a taxonomy, not a service — it belongs to the //other// fold), ''NudeNet'', ''DAppTotal'', ''DappRadar'', ''Arkham Intel'', ''Googlebot'', ''Censys'', and the two that are the page's own best example of the reporting gap it measures — ''multiple domain classification services'' and ''public classification service from the security vendor'', both papers that used a categoriser and declined to say which.

A reasonable person could fold several of these (''urlblacklist.com'' and ''NextDNS Piracy Blocklist'' are arguably security-filtering; ''ScamAdviser API'' is arguably an aggregator; ''open-source topic classifier'' is arguably generic-ml). They were left out because each is one paper and folding a one-off into a family inflates that family's apparent breadth. The rule applied: **fold when the string names a service the catalogue already knows, or when two spellings of one thing are both in the residue; otherwise leave it visible.**

==== 5.3 Two bugs in the report script ====

  - **The last period bucket was ''(y) => y >= 2022''**, so it reported the 2022–2024 corpus size as **3,140** instead of 1,955 and folded 2025–2026 into it. This also corrupted the //method over time// table's last column, which is the table the page's central "closed-source drift" argument reads from: third-party-service showed 42.2% where the true 2022–2024 figure is 49.4%. Split, with a starred ''2025–2026''. **Any figure copied from those two tables before 2026-08-12 is wrong.**
  - **Two hardcoded cross-references** — ''<- OVERVIEW.md's 320'' next to a value that is now 424, and ''(OVERVIEW.md publishes 71.7%)'' next to 72.9% — printed a stale annotation beside a fresh number. Both removed. The line asserting IEEE S&P is 43% retrieved was rewritten to record that it no longer is, and what changed as a result.

The identical open-ended-bucket bug was in ''report_fingerprinting.mjs'', ''report_ip_classification.mjs'' and ''report_javascript.mjs''. Four of the six report scripts had it. See [[literature:corpus]].

===== 6. Quotes checked =====

//Recorded, 2026-08-12.// The five ''method == "llm"'' + ''target == "website-category"'' tuples are new and load-bearing, so **all five were checked individually** against ''paper.cols.txt'' with whitespace normalised. All five are present.

^ Paper ^ Task fold ^ Model ^ Quote found ^
| ''WWW/2025/harmful-terms-and-where-to-find-them…'' | **topic-or-industry** | GPT-4o (URL + screenshot) | yes |
| ''USENIX/2025/darkgram-a-large-scale-analysis…'' | **topic-or-industry** | GPT-4 API | yes |
| ''USENIX/2024/less-defined-knowledge-and-more-true-alarms…'' | abuse-verdict | GPT-3.5-turbo-16k | yes |
| ''WWW/2025/the-poorest-man-in-babylon…'' | abuse-verdict | Llama3:70b + GPT-4 | yes |
| ''NDSS/2026/ctphishcapture…'' | abuse-verdict | GPT-4o + RAG | yes |

The 92% figure the page now quotes was **not** taken from the extraction. The extraction's quote for that tuple is about the annotation of 500 sites; the accuracy claim was located separately in the full text, which reads: //"GPT-4o, when prompted with URLs and screenshots, achieved an accuracy of 92%, comparable to commercial website classification services [49]"//. This follows the rule the workdir README records for exactly this situation: **grep the full text for any prevalence figure you publish, not just the attached quote.**

**A caveat on the second topic paper.** DarkGram categorises cybercriminal **Telegram channels** into 15 predefined categories (which GPT-4 itself extended to 19), not websites. The task fold calls it ''topic-or-industry'' and that is defensible — it is topic categorisation of an online property — but a reader counting "LLM topic classification of //websites//" should count one paper, not two. The page says "count it as adjacent".

//Reconstructed:// the original run recorded checking six quotes by hand, of which two initially failed a literal grep and were intact but split across a two-column break. Which six is not recoverable.

===== 7. External sources =====

//Reconstructed, except where noted.// This page's non-corpus half is the largest on the site and was verified by the original run on 2026-08-07. It was **not** re-verified on 2026-08-12. Two things are worth carrying forward because they are the reasons the page is trustworthy:

  * **{[vallina2020_misshapes]}, the page's central citation, is not in the population the page measures.** It is in ''data/corpus2/.meta/IMC-2020.json'' with a real DOI but has no ''data/fulltext'' entry, so none of its figures come from the extraction. It was read from the author's own copy at ''lepoch.at/files/domain-classification-imc20.pdf''; the ACM DL copy was not retrievable. Still true on 2026-08-12.
  * **The LLM preprint section rests on one unrefereed arXiv posting whose results moved between versions.** v1 (2025-10-09) benchmarked a cloud model on a 14-category task; v3 (2026-04-30) drops the cloud model, uses 13 categories, and adds both a Homepage2Vec baseline and a citation to Vallina et al. that v1 does not contain. The page reads v3 and says so. **This refresh does not change that section**, but the corpus now supplies a peer-reviewed data point (§4) that the section did not have, and the two should be read together.

Three specific errors the original run made and corrected, recorded because they are the failure modes of this page's subject matter:

  - **"How many services did the paper study" had no single right answer.** Vallina et al. is variously 13, 11, 9, 9 (a different nine) and 11 depending on which table you read. Three drafts got it wrong three different ways. The fix was to name the group each figure belongs to instead of reaching for a headline count.
  - **Two columns per condition.** Vallina et al.'s Table 5 gives //Any// (did the service return a label) and //Related// (was it semantically right). The page first published Dr.Web's and Trend Micro's numbers with the two silently swapped.
  - **A framing sentence outran its own query.** The lead once asserted that papers "routinely reach for a topic categoriser" to answer the other four questions. Measuring it — now a section of the report script — showed 9 of 47 abuse-verdict papers name a categorisation service and 6 of those used VirusTotal, the correct tool. On the new corpus it is 13 of 63, of which 9 use VirusTotal. The claim stayed retracted.

**Rejected:** SEO listicles of "best website categorization APIs" were rejected wholesale by the original run.

===== 8. What could not be established =====

  * **Whether the 2025–2026 reversal of the closed-source drift is real.** See §4. Two provisional venue-years.
  * **Whether LLM topic classification is one paper or two.** See §6, the DarkGram caveat.
  * **Whether the 33 residue strings should be folded.** See §5.2; several are borderline and the rule applied is stated rather than obviously correct.
  * **What the 155 papers that name a service actually got back.** The extraction records the service, not the labels. Only Vallina et al. has ever measured the labels, and that was on a 2020 snapshot.
  * **The 33.0% ''not-applicable'' validation sentinel.** The page argues it should be read as "no validation reported" rather than "validation was unnecessary". That is an interpretation the extraction cannot settle; checking it means reading 109 papers.
  * **Nothing has re-audited the categorisation vendors since 2020.** That is the page's largest open question and this refresh does not touch it.

===== 10. Review pass, 2026-08-12 =====

//Recorded.// The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a **systematic defect, not a scatter of typos**, and it is worth stating because it will recur on the next refresh:

  * ''check_page_numbers.mjs'' was run with a **heading window** — normally ''Use in Publications'' to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's **intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block** was outside the window and stayed at its 4,322-corpus value. Across the six pages **29 such figures** survived the first pass.
  * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason.

Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''<file>'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean.
Fixed on this page's content page as a result: the lead paragraph's **72% → 75%** for bespoke unsized taxonomies, which contradicted the corpus section's own "Three quarters … 75.4%" four screens down.

§3 above previously said ''check_page_numbers.mjs'' "left four figures unaccounted, all deliberate" and then listed three. That was wrong in the way this whole review is about: the fourth was ''73'', a genuine stale count, and describing it as deliberate would have buried the guard's only real catch on this page. Corrected.

===== 11. Run log =====

^ ^ ^
| Date | 2026-08-12 |
| Corpus at the time | ''data/extract/run1'', 5,859 papers, 2010–2026, IEEE S&P complete at 780/780 |
| Model | Claude Opus 5, no sub-agents used for this page |
| Scope | Mechanical re-derivation, plus three substantive rewrites the numbers forced: the IEEE S&P box, the LLM section, and the "closed-source drift" paragraph. Each is a case where the old prose asserted something the new data contradicts, so leaving it would have been the error. |
| Script changes | ''webcat_fold.mjs'' (4 new resource entries, 2 extended, §5.2), ''report_website_classification.mjs'' (bucket bug, two stale hardcoded annotations, §5.3) |
| Bibliography | added ''tsai2025_harmful'' before the closing ''</bibtex>''; checked for a duplicate key first; purged the bibtex cache and confirmed the key renders |
| Caveats deleted | "IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share" — replaced with a box recording that the hypothesis was correct and is now resolved. "anything published after 2024". |
| Mistake caught in review | ''check_page_numbers.mjs'' caught "the 73 excluded papers" left behind after the count moved to 94 — the third time in this refresh that the guard caught a number the human eye had skipped. |
| Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 1 stale figure on this page plus a mis-worded line in §3. All fixes were applied and re-saved the same day. |

[[design:website_classification|← back to the content page]] · [[literature:corpus|corpus-level provenance]]