Differences

This shows you the differences between two versions of the page.

--- design:website_classification [2026/08/07 11:08] – Independent review pass (Fable). Fixes: stale residue count 21 -> 20 (the slug-based fingerprinting fold changed it); Vallina denominators stated per claim (11 services compared, 8 across the full 4.4M, 3 top-10k only, 9 in the case studies) and OpenDNS 8 karel.kubicek.claude
+++ design:website_classification [2026/09/03 22:33] (current) – Re-review fixes: the reproducibility buckets counted a paper on the strength of a model it only COMPARED against, so the population is now the 175 papers that used one and the figures are A 10, B 24, A+B 34 (19.4%), C 131, D 10; widen the target=other pro karel.kubicek.claude
@@ Line 5: / Line 5: @@
 This page is about turning a domain into a category you can defend. Its companion [[Design:Website Selection]] is about which domains to put on the list in the first place; popularity ranking lives there, category and company data live here.
-Before anything else: **"classify a website" is not one measurement.** It is at least five, they use different data, and they are not equally reliable.
+Before anything else: **"classify a website" is not one measurement.** It is at least six, they use different data, and they are not equally reliable.
 ^ Question ^ Where the answer comes from ^ How good it is ^ Free at scale? ^
 | What //topic// is this site about? | Commercial categorisation vendors, Curlie, an open model, an LLM | The subject of this page. Vendor coverage ranges from 94% to 0.5%; vendors agree with each other poorly | Rarely |
+| Which //country// is it aimed at? | TLD, CrUX country lists (via Tranco), site language, host IP, an LLM | None of these is a target-audience label; they are proxies that disagree. See [[#Target country is not topic]] | Yes |
 | What //industry// is the company behind it in? | Company datasets (PeopleDataLabs, Crunchbase, Orbis), not web categorisers | Decent for large firms, noisy in the long tail, and the domain↔company join is the weak link | Partly |
 | Is it //abusive// — phishing, malware, scam? | Reputation feeds: VirusTotal, Google Safe Browsing, PhishTank | A different question with its own literature. Do not read a security verdict as a topic label | Rate-limited |
@@ Line 14: / Line 15: @@
 | What //software// runs it? | Wappalyzer, WhatWeb, response fingerprints | Unrelated to topic. A crawler-instrumentation question, see [[Programming:Crawler]] | Yes |
-These are different measurements with different failure modes, and they are easy to conflate — the structured extraction behind [[#Use in Publications]] files all five under one label, because papers describe all five as "classifying websites". The tracker row is where the conflation costs most, and unlike the others it has been measured: see [[#The trap that has caught the most papers]].
+These are different measurements with different failure modes, and they are easy to conflate — the structured extraction behind [[#Use in Publications]] files them all under one label, because papers describe them all as "classifying websites". The tracker row is where the conflation costs most, and unlike the others it has been measured: see [[#The trap that has caught the most papers]].
 <WRAP important>
 **A categorisation service is a measurement instrument, and you have to characterise it before you use it.** Not "we categorised domains using X" — that is not a method, it is a citation. What a reviewer needs is: which service and which version, how many of your domains it actually labelled, what its taxonomy is, and what precision it achieved //on your sample//, established by hand.
-Of the 247 papers in our corpus that assign topic categories to websites, **38.5% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
+Of the 330 papers in our corpus that assign topic categories to websites, **37.3% report any validation of the resulting labels at all.** That is the single largest methodological gap this page exists to close, and [[#Validating Your Labels]] gives you a script that closes it in an afternoon.
 </WRAP>
@@ Line 28: / Line 29: @@
 ===== The One Study Everyone Cites, and What It Found =====
-Vallina et al. {[vallina2020_misshapes]} studied 13 domain classification services. DMOZ and Curlie were set aside as discovery directories rather than classifiers, leaving 11 in the label comparison; of those, eight were queried across the full **4,424,142 domains** (September 2019, Alexa rankings aggregated Tranco-style) and three — Symantec, Trend Micro queried directly, and Webshrinker — were rate-limited down to the top-10k. The case studies in §6 cover nine services. Six years on it is still the reference, so it is worth having its actual numbers rather than the vibe of them.
+Vallina et al. {[vallina2020_misshapes]} studied 13 domain classification services. DMOZ and Curlie were set aside as discovery directories rather than classifiers, leaving 11 in the label comparison. **Nine label sources** — OpenDNS, McAfee, FortiGuard, and Alexa, Bitdefender, Forcepoint, Dr.Web, Trend Micro and Websense as re-served through VirusTotal — were queried across the full **4,424,142 domains** (September 2019, Alexa rankings aggregated Tranco-style). **Three** — Symantec, Trend Micro queried directly, and Webshrinker — were rate-limited down to the top-10k. The case studies in §6 cover nine services. Trend Micro appears in both groups because it was queried two ways, and Websense appears only via VirusTotal; **counting "how many services" is genuinely ambiguous in this paper, which is why every figure below names its own group.** Six years on it is still the reference, so it is worth having its actual numbers rather than the vibe of them.
 <WRAP right box>
@@ Line 70: / Line 71: @@
 The one modern data point comes from a 2025/2026 arXiv preprint on LLM-assisted web measurement {[bozzolan2026_llmweb]}, whose Appendix B explicitly reuses Vallina et al.'s service list. Its findings:
-  * Of the services they surveyed, **three offer a free API at all** — Google Topics, Cloudflare Radar and VirusTotal — and Topics is deprecated and scheduled for removal, leaving two. The rate-limited ones are hopeless at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days".
+  * Of the services they surveyed, **three offer a free API at all** — Google Topics, Cloudflare Radar and VirusTotal — and Topics is deprecated and scheduled for removal, leaving two. The rate-limited ones are hopeless at scale: Cloudflare's Domain Threat Intelligence API allows 100 requests //per month//, so 100k websites would take "roughly 30,000 days"; VirusTotal's 500/day cap puts the same job at "around 200 days". **For topic labels specifically that leaves one free API, VirusTotal, at 200 days per 100k domains** — Cloudflare's unmetered endpoint is the other one, and it answers a different question (next bullet).
-  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%).
+  * The Cloudflare **Radar** API has no explicit rate limit, so they could benchmark it — **but on country, not on topic**, and the distinction matters because the number below gets quoted as if it were a categorisation result. Against a 7,780-site ground truth for target-audience country, Radar **returned a label for 3,925 sites (50.4%)** and, of those, **agreed with the ground truth in 39.7% of cases**. They hand-checked 50 disagreements: their own label was right in 49 (98%), Cloudflare's in 1 (2%). The reason is mechanical: Radar assigns a domain the country that issues the most DNS queries for it against Cloudflare's resolver, which is a measure of //where the traffic is//, not of who the site is for. **No comparable 2020s coverage-or-accuracy number exists for anyone's topic labels.**
   * Every open-weight LLM they tested reached 100% coverage and at least 76% accuracy on the same data, without visiting the sites.
@@ Line 194: / Line 195: @@
 === VirusTotal ===
-  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]].
+  * **API**: ''GET /api/v3/domains/{domain}'' returns a ''categories'' attribute, //"a dictionary mapping that relates categorisation services with the category it assigns the domain to"//. Documented at [[https://docs.virustotal.com/reference/domains-object|docs.virustotal.com]]. **That is a topic label, not a maliciousness verdict.** The oracle use of the same API — engine disagreement, thresholds, URL versus file lookup, snapshot dating — is [[security:virustotal]].
   * **Which vendors**: Dr.Web, Forcepoint ThreatSeeker, BitDefender, Sophos, Trend Micro, Websense and (legacy) Alexa, among others — so several of the vendors above reach you through VirusTotal without you querying them.
-  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day; there is no self-serve academic application form today, so budget for a direct conversation with VirusTotal rather than assuming access.
+  * **Rate limit**: the free tier is **500 requests per day and 4 requests per minute**.((https://docs.virustotal.com/docs/public-vs-premium-api — fetched 2026-08-07; that page names the "Community" tier but does not mention the October 2025 retiering, so treat the tier names as unconfirmed and the numbers as current.)) The same page adds that the Public API //"must not be used in commercial products or services"//. At 500/day, a 100k-domain study takes 200 days. Vallina et al. had an academic key at 20k/day. As of 2026-08-27 there is still no standalone quota-application form; the public contact form has a subject "I have an academic research request" ([[https://www.virustotal.com/gui/contact-us/legal|virustotal.com/gui/contact-us/legal]]). Budget for that conversation rather than assuming 20k/day. Maliciousness-oracle use of the same API is [[security:virustotal]].
   * **Advantages**: aggregates many providers in a single call; widely used and easy to cite.
   * **Disadvantages**: **the integration is lossy**. In the 2020 audit, several services returned labels when queried directly but not through VirusTotal, and Trend Micro's VirusTotal labels tracked its 2011 taxonomy rather than its 2019 one. You do not control which product version you are reading, and "we used VirusTotal categories" does not identify the underlying source.
@@ Line 208: / Line 209: @@
 Two different products, easily conflated:
-  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). Its measured quality is poor: 50.4% coverage and 39.7% agreement on the one 2026 benchmark that exists {[bozzolan2026_llmweb]}.
+  * **[[https://developers.cloudflare.com/radar/|Radar API]]** — free, no explicit rate limit, data licensed **CC BY-NC 4.0** (non-commercial only, which matters for industry co-authors). **Radar is a popularity-ranking product, not a categoriser**, and this is the single most common misreading of it on this page's topic. It does return a ''categories'' array, but only from ''GET /radar/ranking/domain/{domain}'', only for the ''POPULAR'' ranking, and only for the most recent ranking — so a domain gets categories only if Radar ranks it, and Radar ranks an ordered top 100 (global and per country) plus unordered buckets up to one million.((https://developers.cloudflare.com/api/resources/radar/subresources/ranking/subresources/domain/ and https://developers.cloudflare.com/radar/investigate/domain-ranking-datasets/ — both fetched 2026-08-07. The rate limit is unstated rather than absent; treat it as undocumented, not infinite.)) There is no Radar endpoint that categorises an arbitrary domain. The 50.4% / 39.7% figures quoted above are Radar's **country** output, not its categories {[bozzolan2026_llmweb]}; **nobody has published a coverage or accuracy figure for Radar's topic categories at all.**
   * **Domain Intelligence, now under Security Center Intel** — ''GET /accounts/{account_id}/intel/domain?domain=…'' returns ''content_categories''. This is the one the corpus papers used: {[ruth2022_toppling]} labelled Cloudflare-managed sites with it, and {[ruth2022_world]} built a whole global browsing study on it. It needs an API token with Intel permission scoped to an account; whether it is available on the free plan is not stated in Cloudflare's public docs.((Checked 2026-08-07 against https://developers.cloudflare.com/api/resources/intel/subresources/domains/methods/get/ — the docs describe the permission requirement but not the plan tier.))
@@ Line 219: / Line 220: @@
 === Google Topics API ===
-The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call; the 2026 survey of classification services reports it as deprecated and scheduled for removal {[bozzolan2026_llmweb]}.
+The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call.
+**It is being removed, and Chrome's own two records disagree about whether it already has been.** Checked 2026-09-03: Google's Privacy Sandbox status page lists Topics (Web) under "Deprecate and remove" and every Topics doc page carries the phaseout banner; the Blink //Intent to Deprecate and Remove// estimates deprecate in M144 and remove in M150, which Chrome stable has passed; but the Chrome Platform Status entry still reads "Enabled by default" with no removal milestone, and the removal was made explicitly pausable. If you need a Topics adoption figure for a related-work section, the deprecation thread is the only recent public source and it gives **13% of //page loads//** — not sites and not users, and driven by a few ad-tech scripts on many sites. Quote it with its unit.((https://privacysandbox.google.com/overview/status, https://chromestatus.com/feature/5680923054964736 and the Blink intent thread at https://groups.google.com/a/chromium.org/g/blink-dev/c/_R85yctz4Rs, all fetched 2026-09-03; Chrome stable 153.0.8010.27 from the Chrome Version History API the same day. This page previously reported the status second-hand from {[bozzolan2026_llmweb]}.))
 ==== Discontinued ====
@@ Line 225: / Line 228: @@
 === Alexa ===
-Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. [[Design:Website Selection]] currently gives 1 August 2023; we could not find a primary source for that date, and ''alexa.com'' now redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
+Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. An earlier revision of [[Design:Website Selection]] gave 1 August 2023; that date is when Tranco dropped Alexa from the default list, not when Amazon switched the service off. The selection page now carries both dates. ''alexa.com'' redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together.
-  * **Why it still matters**: it appears in 12 of the 247 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
+  * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate.
   * **What it was**: highly granular (7,557 distinct labels observed through VirusTotal, against 279,716 categories in Alexa's own directory, which was derived from DMOZ), and capped at 500 sites per category — hence the tiny coverage. Leaf labels came back in the language of the branch, so the same concept appeared as //Arts//, //Artes// and //Kultur//.
@@ Line 244: / Line 247: @@
 | SimilarWeb | Alive | Paid API only | — | ~24 categories | you also need traffic and industry data |
 | VirusTotal | Alive | 500/day free | Lower than direct | Per underlying vendor | your sample is small, or you want vendor disagreement |
-| Cloudflare Radar | Alive | Free, CC BY-NC | 50.4% (2026, one task) | Yes | you can validate per-category and drop what fails |
+| Cloudflare Radar | Alive | Free, CC BY-NC | **Unmeasured for topics** — ranked domains only | Yes | your domains are inside Radar's top-1M ranking |
+| Cloudflare Domain Intelligence | Alive | Token, ~100/month free | Unmeasured | 26 super / 114 categories | you can validate per-category and drop what fails |
 | Curlie | Alive | **Free bulk dump** | Small but open | Yes, 15 top-level | you need redistributable ground truth |
 | OpenDNS community | **Dead** | — | 15% | — | never |
@@ Line 281: / Line 285: @@
 ===== LLM Classification =====
+Two questions, with different answers, and conflating them is how this section used to mislead. **Is an LLM a normal thing to classify with in this field?** Yes, since about 2024. **Is there published evidence that one can assign a website a topic category well enough to replace a vendor API?** For a **binary** question, yes, and it is peer-reviewed. For a multi-class taxonomy — which is the shape of most real categorisation tasks — the only systematic evidence is one unrefereed preprint.
+<WRAP tip>**The decision, in three lines, if you do not read the rest of this section.**
+  * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice.
+  * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both.
+  * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|fewer than one paper in five does]]), constrain the output to the taxonomy, and report the rejection count.
+</WRAP>
+**The method itself is now mainstream in these venues.** ''classification.method == "llm"'' fires on **177 of the 5,859 papers** in the 2010–2026 corpus — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026, which is 17.1% of that year's papers and 21.3% of the papers that classified anything that year — spread across all seven venues, with **83.6% validating the model's labels against something**. It is the fastest-moving quantity anywhere in this corpus. The full series, the per-target breakdown and the model-reporting numbers are in [[#Where LLMs actually appear]]; nothing in this section should be read as "unproven method".
+**For website topic classification specifically the peer-reviewed base is two papers, and the stronger one is a binary task.** {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with GPT-4o prompted with the URL and a screenshot, validates against 500 manually annotated sites, and reports **92% accuracy, which the paper itself calls "comparable to commercial website classification services"**. That is the first result in this corpus that makes an LLM a defensible alternative to the vendors catalogued above — and it is a two-class decision. It does not tell you what a model does with 13 or 400 categories, which is the shape of most real categorisation tasks, and nothing peer-reviewed in these venues does. The second paper, DarkGram (USENIX Security 2025), categorises Telegram channels rather than websites; count it as adjacent.
+<WRAP tip>**Why this section still spends most of its length on a preprint.** Because the peer-reviewed evidence stops exactly where the hard part starts. Two papers, one of them binary and one of them not about websites, cannot tell you how an LLM behaves on a multi-label taxonomy against a trained specialist classifier — and that comparison is the decision a reader of this page is actually making. The preprint below is the only source that has run it. It is cited for that, not because it outranks the corpus, and every number taken from it carries the version caveat in the box below.</WRAP>
+The one systematic study is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first of its kind for web-measurement classification.
 <WRAP important>
-**This is where our corpus runs out, and you should know exactly where.** Of the 4,322 papers in the 2010–2024 corpus, 29 use an LLM to classify anything at all — 27 of them in 2024 — and they point it at privacy policies, app reviews, vulnerabilities, jailbreak prompts and content moderation. **Zero of them use an LLM to assign a website a topic category.** Any ranking of methods built from this literature will therefore tell you that third-party services and hand-built keyword rules are the state of the art. As of 2026 that is no longer true, and the corpus cannot see why.
+**It is one unrefereed preprint, and it has moved under its own feet.** Everything in this section rests on a single arXiv posting that has not been through peer review, and whose results are not stable across versions. **v1** (9 October 2025) benchmarked a cloud model — Gemini 2.5 Flash — against Llama 4:109B and Gemma 3:27B, on a **14**-category Curlie task. **v3** (30 April 2026) drops the cloud model entirely, evaluates five open-weight models instead, reduces the topic task to **13** categories, and adds two things that appear nowhere in v1: a Homepage2Vec baseline, and {[vallina2020_misshapes]} — the audit this whole page is built on, which v1 does not cite once.((Version dates from https://arxiv.org/abs/2510.08101 (submission history); v1 text read from https://arxiv.org/html/2510.08101v1, which contains zero occurrences of Vallina et al. 2020 and one passing mention of Homepage2Vec with no comparison. Both fetched 2026-08-07.)) **This page reads v3.** Cite the version you read, and re-derive any number you lean on — this one changed once already, and a preprint that acquires its baseline and its closest related work between postings is a preprint whose next revision may move again.
 </WRAP>
-The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification. It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
+It evaluates five **open-weight, self-hosted** models (gpt-oss:20b, llama3.3:70b, mistral-small3.2:24b, phi4-mini:3.8b, qwen3:32b, all via Ollama, temperature 0) on three tasks with hand-built benchmarks, in two modes: URL string only, and URL plus live browsing through LangChain and Playwright.
 ^ Task ^ N ^ Classes ^ Best URL-only ^ Best URL + live visit ^
@@ Line 294: / Line 314: @@
 The number that matters for this page: on the same 13-category benchmark, **Homepage2Vec scores 0.60 Jaccard and 0.55 macro F1** — the paper's words are that "a general-purpose LLM like gpt-oss can label websites with higher accuracy than a classic machine learning model specifically trained for website categorization." A stock open-weight instruct model, with no fine-tuning, beat the field's reference specialist classifier.
+Two cautions before you repeat that sentence.
+  - **0.55 is not a refutation of the 0.90 in Homepage2Vec's abstract**, and anyone who puts the two side by side has compared different evaluation regimes. 0.90 is the balanced per-class setup; Homepage2Vec's own authors report **0.634** macro F1 on an unbalanced test set, which is the regime a benchmark like this one is in. See [[#Homepage2Vec]].
+  - **0.55 is nonetheless below that 0.634**, on a task that drops one of the 14 classes, and the preprint never states how it ran the baseline. Homepage2Vec emits independent //calibrated probabilities// per class, so turning it into a multi-label prediction requires a decision threshold — its own paper uses 0.5 on calibrated output — and the word "threshold" does not occur anywhere in the preprint.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "threshold".)) A head-to-head in which the winner ran the loser's model, without reporting the one hyperparameter that decides the loser's score, is a result to re-run before you build on it. Both models are free and the benchmark is small.
 Three practical consequences:
   - **URL-only works for coarse questions and fails for topics.** Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly.
-  - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at August 2026 list prices for a nano-tier model and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs on the order of **a few dollars**, roughly halved by a batch API.((Our own arithmetic from published per-token prices, not a figure from any paper. Re-derive it at current prices before quoting it; full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility: an open-weight model you pin by digest is reproducible five years from now, and ''gpt-4o-2024-08-06'' is not.
+  - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at 2026-09-03 list prices and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs **$8.40**, or **$4.20** through a batch API, on the cheapest current-generation hosted model — and $2.30 / $1.15 on the cheapest //legacy// tier, which is on a shutdown list.((Our own arithmetic from the vendor's published per-token prices, not a figure from any paper: 30M input + 2M output tokens, batch at exactly half. ''gpt-5.6-luna'' at $0.20 / $1.20 per 1M gives $8.40 / $4.20; legacy ''gpt-5-nano'' at $0.05 / $0.40 gives $2.30 / $1.15. Re-derived from https://developers.openai.com/api/docs/pricing on 2026-09-03. **Name the tier and the date, not "the cheap one"** — the two current answers differ fourfold and the cheaper one is legacy. And full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility, and the direction of that tradeoff is now demonstrable rather than argued: an open-weight model you pin by digest is reproducible five years from now, and a hosted snapshot is not. **''gpt-4-turbo-2024-04-09'' — the exact string this page holds up as good practice, and the one two corpus papers used — is on OpenAI's published shutdown list for 23 October 2026**, with ''gpt-5.6-sol'' named as its replacement.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' / ''gpt-4-turbo-completions'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, are on the same date.)) Naming the snapshot is still the right thing to do — it makes your paper's result //interpretable// years later — but it does not make it re-runnable, and a paper that needs to be re-runnable needs open weights.
   - **Ensembling helps because single models are unstable.** A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names "inconsistency, hallucination, category inflation, and misclassification" as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.((Kamen and Kamen, //Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization//, arXiv:2511.15714, November 2025.))
@@ Line 307: / Line 332: @@
   * **Label instability across seeds and temperature** for website classification specifically. The one systematic paper set temperature to 0 to sidestep the question rather than quantify it.
   * **Behaviour on parked, dead and holding-page domains** — a large fraction of any long-tail sample, and a case where "visit the site" gives the model a page that is not about anything.
-  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. What does exist, on those same 150 hand-relabelled sites, is a Jaccard similarity of **0.73 between the human labels and gpt-oss's predictions**, against **0.75 between the human labels and the benchmark's own ground truth** — the model is as close to careful humans as the benchmark is. Which is also the reason to read the 0.65 Jaccard above as a floor rather than a ceiling: a multi-label benchmark that under-labels punishes a model for being right. (For scale on how much under-labelling there is: the benchmark's own heuristic label-construction matched the human label set exactly on 86 of 150 sites (57%), where Lugeon et al.'s Curlie snapshot matched on 67 (45%). Both figures are about the ground truth, not about any model.)
+  * **A formal agreement statistic** (Cohen's or Fleiss' kappa) between LLM labels and human annotators for this task. Nobody has published one. What does exist, on those same 150 hand-relabelled sites, is a Jaccard similarity of **0.73 between the human labels and gpt-oss's predictions**, against **0.75 between the human labels and the benchmark's own ground truth** — the model is as close to careful humans as the benchmark is. Which is also the reason to read the 0.65 Jaccard above as a floor rather than a ceiling: a multi-label benchmark that under-labels punishes a model for being right. (For scale on how much under-labelling there is: the benchmark's own heuristic label-construction matched the human label set exactly on 86 of 150 sites (57%), where Lugeon et al.'s Curlie snapshot matched on 67 (45%). Both figures are about the ground truth, not about any model.) **And note what that pair cannot tell you:** those 150 sites were relabelled by the preprint's own authors, with no second annotator and no agreement statistic of any kind — no kappa, no Krippendorff's alpha, nothing.((Verified by full-text search of https://arxiv.org/html/2510.08101v3 on 2026-08-07: zero occurrences of "kappa", "inter-annotator", "Fleiss" or "Krippendorff".)) So 0.73 says the model tracks //that annotator's// labels. It does not say how fuzzy the task is, and a single-annotator relabelling cannot be used to argue that the remaining label noise is small — the disagreement you have not measured is exactly the quantity in question. If you relabel a sample, use two annotators and report the agreement; [[#Validating Your Labels]] is the whole procedure.
+===== Target country is not topic =====
+Studies that check compliance against a national law need to know which country a site is //for//, and this gets treated as a minor preprocessing step. It is not. There is no service that answers it, the available proxies measure different things, and none of them is a target-audience label:
+  * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list.
+  * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX datasets are CC BY 4.0 per Google's methodology, fetched 2026-08-27; Tranco's homepage still labels CrUX CC BY-SA 4.0, while the methodology page states no CrUX licence.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}.
+  * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web.
+  * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]].
+  * **Cloudflare Radar's country field.** DNS-query volume by country against Cloudflare's resolver: 50.4% coverage and 39.7% agreement on the one benchmark that exists (see [[#Has Anything Changed Since 2020?]]).
+**So combine them and report the combination.** The one systematic evaluation on this task got its ground truth by starting from CrUX and then correcting with site language and host IP, and still hand-inspected the result — and an LLM given the URL alone beat the best single proxy on that benchmark. Whatever you do, say which proxies you combined and how you broke ties: "we classified sites by country" is not a method, and a compliance result is only as good as the jurisdiction assignment underneath it.
 ===== Taxonomies =====
-Pick the taxonomy before the service, because it is much harder to change later, and because 72% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
+Pick the taxonomy before the service, because it is much harder to change later, and because 75% of the corpus papers that name a taxonomy at all name a bespoke one and do not even state how many classes it has (see [[#Use in Publications]]) — which makes their results incomparable with everyone else's.
   * **[[https://github.com/InteractiveAdvertisingBureau/Taxonomies|IAB Tech Lab Content Taxonomy]]** — the advertising industry's reference schema, and the right choice if your question is about ad targeting or if you want to be comparable to Webshrinker's output. Machine-readable TSV in the repo under ''Content Taxonomies/'', CC BY 3.0, up to four tiers with a relational ID system. Versions 1.0 through **3.1** sit side by side; 2.x → 3.0 is a **breaking change** and IAB says the two must not be used together. Category rows counted from the TSVs on 2026-08-07, excluding the two header lines: **2.2 has 1,196 rows, 3.0 has 703, 3.1 has 704** — so 3.1 is marginally //larger// than 3.0. Tier-1 rows: 37 in 3.1, but only 31 of those carry a numeric Unique ID; the other six (Communication, Entertainment, Events, Genres, Holidays, Sensitive Topics) are alphanumeric extension rows. **State which convention you counted with** — three independent counts of this one file in preparing this page produced three different numbers. Vendor marketing claiming that 3.x "expands to 1,500+ categories" is wrong in both directions — 3.x is roughly //40% smaller// than 2.2, because it consolidated and restructured rather than adding. Count the rows in the file yourself; it takes ten seconds and the summaries are not reliable.
-  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports.
+  * **Curlie's 15 top-level categories** — the right choice if you want free ground truth, an open model that already predicts into it, and a hierarchy you can prune to whatever depth your sample supports. **Check that your classes are actually in it before you commit.** The 15 are Arts, Business, Computers, Games, Health, Home, News, Recreation, Reference, Regional, Science, Shopping, Society, Sports, and Kids and Teens — a directory taxonomy from the early 2000s. There is no top-level class for social media, finance and banking, education, government, or online services, and none at all for adult content; those questions land inside Business, Society, Reference or Computers, or nowhere. If your paper is about the privacy of banking sites, or trackers on education sites, or anything on social platforms, a Curlie-trained classifier will not give you the class you are measuring, and pruning to top level will hide that rather than fix it.
   * **A vendor's own list** — fine, but pin the version, and expect to discover categories that are not in the documentation. Vallina et al. found at least one undocumented category in eight of the nine services whose documentation they could compare against its output.
   * **A standard industry code** (NAICS, SIC, NACE) — the right choice for a question about firms rather than content, and the natural join key to the company datasets below. Exactly one paper in our corpus did this.
@@ Line 338: / Line 375: @@
     - Based on LinkedIn profiles that are self-reported — prone to adversarial data.
     - Only a subset of PeopleDataLabs' full dataset. The "22M of 70M rows" framing is longstanding on this page; the 22M is confirmed, the 70M total could not be re-confirmed in 2026 and PDL's marketing page now cites 23.8M+ without saying which corpus that is.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Crunchbase ====
@@ Line 353: / Line 390: @@
     - Focuses mostly on variables useful for investments and market competitiveness.
     - Academic access is no longer publicly documented; budget for a sales conversation.
-  <wrap todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</wrap>
+  <WRAP todo>TODO: cite ''Machine Learning Compliance Analysis for Email Regulation'' when it is public.</WRAP>
 ==== Orbis ====
@@ Line 713: / Line 750: @@
 total                                1100      100
+wrote sheet_a.csv  (100 rows, category hidden, order shuffled)
+wrote sheet_b.csv  (100 rows, category hidden, order shuffled)
+wrote sample_key.csv     (the labels under test -- do not show this to annotators)
 note: +/-20% half-width at 90% precision needs 9 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 note: +/-10% half-width at 90% precision needs 35 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 note: +/-5% half-width at 90% precision needs 139 domains per category (normal approximation; the Wilson interval reported by `score` is wider near 100%)
 ==================================================================
 INTER-ANNOTATOR AGREEMENT
@@ Line 723: / Line 762: @@
 raw agreement            : 67/100 = 67.0%
 Cohen's kappa            : 0.564
+Report both. Kappa alone hides how many categories there were, and raw
+agreement alone is inflated when one category dominates.
 ==================================================================
@@ Line 739: / Line 780: @@
 ALL (disputed counted wrong)       63  100  63.0%  [53.2%, 71.8%]
+The true precision is between these two rows. If they are far apart your
+annotators disagreed too much for the number to mean anything -- fix the
+codebook and re-annotate rather than picking the flattering row.
 ==================================================================
@@ Line 744: / Line 788: @@
 ==================================================================
 /1100 domains got a label = 80.0% [77.5%, 82.3%]
+==================================================================
+PASTE THIS INTO YOUR METHODOLOGY SECTION
+==================================================================
+We categorised 1100 domains, of which 880 (80.0%) received a label. To validate the labels we drew a stratified random sample of 100 domains and had two authors independently assign a category while blind to the service's answer (Cohen's kappa = 0.56, raw agreement 67%). On the 67 domains where the annotators agreed, the service's micro-averaged precision was 94.0% (95% Wilson CI [85.6%, 97.7%]); counting every disputed domain as an error gives a lower bound of 63.0%. We discarded the 1 category that fell below 80% precision (Search Engines) and report results over the remaining 3.
 </code>
-Three things to read out of that:
+Four things to read out of that:
   - **It found the bad category.** Search Engines came back at 55.6% and got dropped, exactly as intended — and it did so from 9 usable annotations, which is why per-category stratification matters more than total sample size.
@@ Line 840: / Line 889: @@
 ===== What to Report =====
-A checklist, derived from what the 247 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
+A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply.
-  - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either.
+  - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **19.4% of the 175 corpus papers that actually use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest.
   - **The date you queried.** These are live services whose labels change without notice.
   - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage.
@@ Line 849: / Line 898: @@
   - **Validation**: sample size, how it was stratified, how many annotators, whether they were blind to the service's label, the agreement statistic, and per-category precision — not one global accuracy number. Publish the annotated sample as an [[:Artifacts|artefact]] //(page not yet written)//; it is small, and it is the only part of this a reader can check.
   - **The categories you dropped**, and the rule you dropped them by.
-  - **If you used more than one service, the disagreement between them.** Only 9 of 122 corpus papers that name a service name two, so this is nearly free novelty.
+  - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty.
+  - **If the classifier is an LLM, four more things**, none of which is standard practice yet and all of which a reviewer will now ask for: the **prompt text**, verbatim, as an appendix or artefact; the **temperature and decoding settings** (everyone sets temperature 0 and nobody says whether they checked that it mattered); the **output constraint** — the JSON schema or grammar you forced the answer into; and the **rejection count**, meaning how many responses fell outside your taxonomy and what you did with them. That last one is a result, not housekeeping: it is the closest thing to a measurement of how well the model understood the task. See [[#LLM Classification]].
 <WRAP important>
@@ Line 857: / Line 907: @@
 ===== Use in Publications =====
-Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **4,322 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2024** — so every figure is a claim about those seven venues over that window, not about the field.
+Two sources here. The historical survey is Vallina et al. {[vallina2020_misshapes]}, whose figures on service popularity are reproduced below. The rest is a fresh query over a structured extraction of **5,859 papers from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf and IEEE S&P, 2010–2026** — so every figure is a claim about those seven venues over that window, not about the field. The 2025 and 2026 venue-years are provisional: CCS and IMC 2026 have not been held, and IEEE S&P and WWW 2026 abstracts are not yet in the selection source.
 <WRAP right box>
@@ Line 866: / Line 916: @@
 ==== The population, and what "website category" hides ====
-The extraction marks 320 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
+The extraction marks 424 papers as classifying something with the target ''website-category''. **That number is not the population this page is about**, and the difference is instructive: the same label covers several unrelated tasks.
-^ What is actually being classified ^ Papers ^ Share of the 320 ^
+^ What is actually being classified ^ Papers ^ Share of the 424 ^
-| A topic, industry or sector label on a site or domain | 247 | 77.2% |
+| A topic, industry or sector label on a site or domain | 330 | 77.8% |
-| An abuse verdict — phishing, scam, malicious | 47 | 14.7% |
+| An abuse verdict — phishing, scam, malicious | 63 | 14.9% |
-| Website fingerprinting — which of N monitored sites produced this traffic trace | 22 | 6.9% |
+| Website fingerprinting — which of N monitored sites produced this traffic trace | 36 | 8.5% |
-| Hosting or CDN infrastructure | 7 | 2.2% |
+| Hosting or CDN infrastructure | 8 | 1.9% |
-| Page type — homepage, login page, sign-up | 6 | 1.9% |
+| Page type — homepage, login page, sign-up | 7 | 1.7% |
-| Which software runs the site | 6 | 1.9% |
+| Which software runs the site | 6 | 1.4% |
-| Third-party role — tracker, advertiser, analytics | 5 | 1.6% |
+| Third-party role — tracker, advertiser, analytics | 5 | 1.2% |
-Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 247, which is 5.7% of the corpus.** The other 73 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
+Website fingerprinting is a traffic-analysis attack that shares no method, taxonomy or ground truth with topic categorisation; it is here because "classify a website" is the same English phrase. **Everything below uses the 330, which is 5.6% of the corpus.** The other 94 papers are counted, named, and excluded — see [[#Methodology and limitations of these figures]].
 ==== This is a Web Conference and IMC topic ====
-^ Venue ^ Papers ^ Share of the 247 ^ Share of that venue's papers ^
+^ Venue ^ Papers ^ Share of the 330 ^ Share of that venue's papers ^
-| TheWebConf | 59 | 23.9% | 8.3% |
+| TheWebConf | 71 | 21.5% | 8.4% |
-| IMC | 45 | 18.2% | 8.1% |
+| USENIX Security | 58 | 17.6% | 4.1% |
-| USENIX Security | 45 | 18.2% | 4.0% |
+| IMC | 50 | 15.2% | 7.8% |
-| CCS | 44 | 17.8% | 4.9% |
+| CCS | 48 | 14.5% | 4.8% |
-| PETS | 29 | 11.7% | 8.2% |
+| PETS | 43 | 13.0% | 8.4% |
-| NDSS | 22 | 8.9% | 5.3% |
+| NDSS | 35 | 10.6% | 5.0% |
-| IEEE S&P | 3 | 1.2% | 1.1% |
+| IEEE S&P | 25 | 7.6% | 3.3% |
-TheWebConf, IMC and PETS each devote about 8% of their programme to it; IEEE S&P essentially never does. The overall rate has been flat-to-falling: 5.1% of papers in 2010–2013, peaking at 7.5% in 2014–2017, back to 4.6% in 2022–2024.
+TheWebConf and PETS each devote about 8% of their programme to it, IMC nearly as much. The overall rate has been flat-to-falling: 5.5% of papers in 2010–2013, peaking at 7.3% in 2014–2017, 4.6% in 2022–2024 and 5.5% in the provisional 2025–2026 window.
+<WRAP info>
+**An earlier version of this table said IEEE S&P "essentially never" does this, at 3 papers and 1.2%.** That was an artefact of the corpus: IEEE S&P was then only 43% retrieved. It is now complete (780 of 780 selected papers), and IEEE S&P has 25 such papers, 3.3% of its programme. The old caveat named exactly the right suspect, and the fix confirmed it — which is the useful lesson: a venue that looks absent from a corpus should be checked for a retrieval gap before it is described as uninterested.
+</WRAP>
 ==== How they classify ====
-^ Method ^ Papers ^ Share of 247 ^
+^ Method ^ Papers ^ Share of 330 ^
-| Third-party service | 105 | 42.5% |
+| Third-party service | 131 | 39.7% |
-| Manual labelling | 53 | 21.5% |
+| Manual labelling | 68 | 20.6% |
-| Heuristic rules | 39 | 15.8% |
+| Heuristic rules | 59 | 17.9% |
-| Supervised ML | 35 | 14.2% |
+| Supervised ML | 46 | 13.9% |
-| Curated database | 31 | 12.6% |
+| Curated database | 46 | 13.9% |
-| Unsupervised ML | 11 | 4.5% |
+| Unsupervised ML | 14 | 4.2% |
-| Regex or signature | 9 | 3.6% |
+| Regex or signature | 9 | 2.7% |
+| **LLM** | **2** | **0.6%** |
+Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
+**In the provisional 2025–2026 window third-party-service reliance drops to 32.3%**, with hand-built heuristics up to 23.1% and the first two LLM papers at 3.1%. Two incomplete venue-years cannot establish a turn — but this one survives every test that could have killed it, and it is the first movement against the closed-source drift in fifteen years.
+**The `curated-database` row moves too, from 9.0% to 18.5%. Do not read that as open sources returning, and this page said so wrongly until 2026-09-03.** Two reasons, both checked:
+  - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05.
+  - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve papers are: **three commercial vendors** (Cloudflare Radar, SimilarWeb, Symantec SiteReview); **four media-bias raters** with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); **one model** (Homepage2Vec); **two taxonomies rather than label databases** (IAB, NAICSlite); **one paper's own rule set** ("predefined source rules (custom)"); and **one that genuinely is an open, inspectable database** — DuckDuckGo's Tracker Radar Entity List, which is a public repository((https://github.com/duckduckgo/tracker-radar, checked 2026-09-03: public, not archived, last pushed 2026-09-02.)) — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors.
+**The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024.
+^ Cut ^ Third-party service ^ n ^ vs baseline ^ Fisher //p// ^
+| 2022–2024, all venues //(the baseline)// | 44 (49.4%) | 89 | — | — |
+| 2025 alone | 17 (37.8%) | 45 | −11.7 pp | 0.27 //not supported// |
+| 2026 alone //(provisional)// | 4 (20.0%) | 20 | −29.4 pp | **0.024** |
+| 2025–2026, all venues | 21 (32.3%) | 65 | −17.1 pp | **0.047** |
+| 2022–2024, five venues with a 2026 slice | 34 (54.0%) | 63 | — | — |
+| **2025–2026, those same five venues** | **16 (28.6%)** | 56 | **−25.4 pp** | **0.006** |
+Restricting both windows to the five venues that have any 2026 paper (IEEE S&P, NDSS, PETS, USENIX Security, TheWebConf) makes the drop //larger//, not smaller — 54.0% to 28.6%, p = 0.006. The composition explanation therefore fails in the direction that would have rescued the old trend.
+**But 2025 on its own is not evidence.** At n = 45 and p = 0.27, the 37.8% is a point estimate that happens to sit the right way. An earlier version of this section leaned on it as a control; it is a description, not a test, and the significance comes entirely from the 2026 slice and the five-venue restriction.
+<WRAP todo>**What is still unsettled, and what would settle it.** The direction of the third-party-service drop is supported; its **magnitude** is not — the 2026-alone column rests on 20 papers, and TheWebConf 2025 is itself thin (63 corpus papers, against 284 across 2022–2024). The ''curated-database'' movement should not be quoted at all, in either direction.
+The thing that would settle it is CCS 2026 and IMC 2026, the two venues absent from the 2026 slice — and their absence does **not** cut the way you would guess. Within this population, 2022–2024 third-party-service share by venue runs TheWebConf 66.7%, IMC 64.3%, PETS 62.5%, IEEE S&P 55.6%, USENIX 47.6%, NDSS 20.0% and **CCS 8.3%, the lowest of the seven**. So IMC is the venue whose absence most plausibly inflates the drop, and adding CCS would on this evidence push the recent share //down// and strengthen the reversal. This page asserted the opposite until 2026-09-03; the sentence was the only one in the section not derived from a printed table, which is how it survived.
-Multi-valued, so shares exceed 100%. Reliance on third-party services has grown steadily — 29.2% of categorising papers in 2010–2013, 35.8%, 44.4%, and **48.8% in 2022–2024** — while use of curated databases (DMOZ and friends) halved over the same period, from 16.7% to 8.8%. The field has moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting.
+Re-run ''report_website_classification.mjs'' once those two venue-years are in the corpus and compare the five-venue control against the all-venue row. Until then: report the direction for third-party services, do not report the magnitude, say nothing about curated databases, and do not describe the closed-source drift as over.</WRAP>
 ==== Which services, folded ====
@@ Line 910: / Line 992: @@
 ^ Service ^ Papers ^ Distinct spellings folded ^
-| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 19 | **15** |
+| McAfee (incl. SmartFilter, TrustedSource, SiteAdvisor, URL Ticketing System) | 23 | **18** |
+| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 14 | **10** |
 | Alexa | 12 | 8 |
-| Symantec / Blue Coat (incl. WebPulse, RuleSpace, SiteReview) | 12 | **10** |
 | FortiGuard | 12 | 7 |
+| Media bias raters (AllSides, Media Bias/Fact Check, Science Feedback) | 10 | 9 |
+| Cloudflare domain categories | 10 | 6 |
+| Webshrinker | 9 | 2 |
 | DMOZ / Open Directory Project | 8 | 6 |
-| Webshrinker | 8 | 2 |
+| Google Cloud Natural Language | 8 | 6 |
-| Google Cloud Natural Language | 7 | 6 |
+| SimilarWeb | 7 | 2 |
+| VirusTotal | 7 | 3 |
 | Google Ads keyword and topic tools | 6 | 5 |
-| Cloudflare domain categories | 6 | 3 |
-| Media bias raters (AllSides, Media Bias/Fact Check) | 5 | 5 |
 | Trend Micro | 4 | 3 |
-| SimilarWeb | 4 | 2 |
 | Cisco (OpenDNS, IronPort) | 3 | 3 |
 | Cyren | 3 | 3 |
-| VirusTotal | 3 | 3 |
+| Citizen Lab test list | 3 | 3 |
+| Forcepoint / Websense | 3 | 2 |
+| Google Topics API | 3 | 3 |
-**122 of the 247 papers (49.4%) name at least one third-party service or dataset.** McAfee alone appears under fifteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
+**155 of the 330 papers (47.0%) name at least one third-party service or dataset.** McAfee alone appears under eighteen different names — from bare "McAfee" to "McAfee's URL Ticketing System" to "McAfee SmartFilter Internet Database" — and Symantec under ten. Any count of these services built on exact string matching will be wrong by a factor of several, which is the same failure that has bitten geolocation and crawler-tool tallies on other pages of this site.
-Everything that is not a named service, ranked: unnamed custom methods (38 papers), manual labelling by the authors or hired coders (32), custom keyword and regex rules (25), random forests (10), SVMs (9), clustering (8), then a long tail. Those first and third rows are one family: **61 papers (24.7%) classify websites with rules they wrote themselves, and 38 of them do not name the rules at all.**
+Everything that is not a named service, ranked: unnamed custom methods (58 papers), manual labelling by the authors or hired coders (41), custom keyword and regex rules (32), random forests (13), clustering (10), SVMs (9), then a long tail. Those first and third rows are one family: **87 papers (26.4%) classify websites with rules they wrote themselves, and 58 of them do not name the rules at all.**
 ==== Almost nobody cross-checks ====
 <WRAP important>
-Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 122 papers that name a service name two** — 7.4%. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
+Vallina et al.'s central recommendation is not to rely on a single service. **Nine of the 155 papers that name a service name two** — 5.8%, down from 7.4% on the smaller corpus, because the count of papers naming two has not moved at all while the field has grown. Five of those nine are from 2013–2015. The recommendation has not been taken up, and it remains one of the cheapest ways to strengthen a categorisation-based result.
 </WRAP>
 ==== Most do not validate, and half do not say where their labels came from ====
-^ Question ^ Papers ^ Share of 247 ^
+^ Question ^ Papers ^ Share of 330 ^
-| Names the taxonomy it classified into | 184 | 74.5% |
+| Names the taxonomy it classified into | 248 | 75.2% |
-| Names a ground-truth source | 124 | 50.2% |
+| Names a ground-truth source | 169 | 51.2% |
-| Reports **any** validation of the labels | 95 | **38.5%** |
+| Reports **any** validation of the labels | 123 | **37.3%** |
-| Explicitly reports no validation | 69 | 27.9% |
+| Explicitly reports no validation | 98 | 29.7% |
-| Sentinel: the extraction marked validation ''not-applicable'' | 83 | 33.6% |
+| Sentinel: the extraction marked validation ''not-applicable'' | 109 | 33.0% |
 The last row is a sentinel and is shown because the page's own rule says sentinels are never folded into an answer. It means the extraction could not find a validation step and did not think one was called for — typically because the paper is reporting a distribution of vendor labels with no claim that depends on them being right. Read it as "no validation reported", not as "validation was unnecessary", unless you check the paper.
-Of those that do validate: 70 papers by manual validation, 18 by cross-validation, 9 by a held-out test set, 9 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
+Of those that do validate: 92 papers by manual validation, 21 by cross-validation, 13 by a held-out test set, 10 by comparison to another method. Note the shape — manual spot-checking is the dominant and correct instrument here, because a held-out test set only tells you about the labels you already had.
-The 50.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 71.7% the same extraction measures across all 3,241 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
+The 51.2% figure is worth dwelling on: for **website** classification, the share naming a ground-truth source is well below the 72.9% the same extraction measures across all 4,439 papers that classify anything. Website categorisation is reported worse than classification in general, not better.
 ==== Taxonomies are bespoke, and usually not even sized ====
-Of the 184 papers that name a taxonomy at all:
+Of the 248 papers that name a taxonomy at all:
-^ Taxonomy ^ Papers ^ Share of 184 ^
+^ Taxonomy ^ Papers ^ Share of 248 ^
-| A bespoke scheme, class count not given | 133 | 72.3% |
+| A bespoke scheme, class count not given | 187 | 75.4% |
-| A vendor's own category list | 37 | 20.1% |
+| A vendor's own category list | 43 | 17.3% |
-| DMOZ / ODP / Curlie | 6 | 3.3% |
+| IAB | 9 | 3.6% |
-| IAB | 6 | 3.3% |
+| DMOZ / ODP / Curlie | 7 | 2.8% |
-| A bespoke scheme with a stated class count | 6 | 3.3% |
+| A bespoke scheme with a stated class count | 7 | 2.8% |
-| Google AdWords / Topics | 3 | 1.6% |
+| Google AdWords / Topics | 4 | 1.6% |
-| A standard industry code (NAICS, SIC, NACE) | 1 | 0.5% |
+| A standard industry code (NAICS, SIC, NACE) | 1 | 0.4% |
-Nearly three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
+Three quarters invent a scheme and do not say how many classes it has. That single omission makes a result impossible to compare against any other paper, and it is free to fix.
 ==== Where LLMs actually appear ====
 ^ Year ^ Papers using an LLM to classify anything ^ Corpus that year ^ Share ^
-| 2023 | 2 | 630 | 0.3% |
+| 2023 | 2 | 719 | 0.3% |
-| 2024 | 27 | 628 | 4.3% |
+| 2024 | 27 | 690 | 3.9% |
+| 2025 //(provisional)// | 77 | 770 | 10.0% |
+| 2026 //(provisional)// | 71 | 415 | 17.1% |
+**177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026. Against the arguably fairer denominator — papers that classified //anything// that year, and so had the occasion to reach for a model — the same series reads 0.4%, 5.1%, 12.7% and **21.3%**. It is not concentrated in one venue: USENIX Security 43 papers, TheWebConf 40, NDSS 30, PETS 21, CCS 18, IEEE S&P 14, IMC 11.
+**The 2025 and 2026 //counts// are floors, because those venue-years are incomplete. The 2026 //share// is not** — a share whose numerator and denominator are both drawn from the same incomplete set is not a floor, it is a share of a differently-composed population, and the 2026 slice is 39% NDSS with CCS and IMC absent entirely. So the growth was checked like-for-like, restricting both years to the five venues that have a 2026 slice: **9.8% in 2025 → 17.1% in 2026**, and every one of the five rose on its own — IEEE S&P 8.5→10.7%, NDSS 9.2→11.7%, PETS 6.0→18.2%, USENIX Security 8.0→18.8%, TheWebConf 23.8→29.9%. CCS and IMC together were at 10.6% in 2025, below the five-venue 2026 figure, so their absence does not obviously inflate it. **The growth is robust; the exact 17.1% is not.**
+They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. Two of the 177 only //compared against// an LLM baseline rather than using one, so the adoption count is 175.
+=== Which task, though ===
+**The corpus-wide curve is the wrong number to reason from if you want to know whether there is a baseline for //your// task.** Adoption is extremely uneven by what is being classified, and three pages on this site have to agree about it, so the table is here and they link to it. Papers, not tuples; the right-hand column is the LLM share of the papers that classify that target //at all//, which is the quantity that decides whether "LLMs have arrived" is true for a given page.
+^ ''classification.target'' ^ LLM papers ^ Papers classifying that target at all ^ LLM share ^ Page ^
+| ''privacy-policy'' | 12 | 102 | **11.8%** | — |
+| ''dark-pattern'' | 1 | 13 | 7.7% | [[privacy:darkpatterns]] |
+| ''user-generated-text'' | 32 | 419 | 7.2% | — |
+| ''consent-notice'' | 2 | 39 | 5.1% | [[privacy:consent]] |
+| ''other'' | 116 | 2,592 | 4.4% | — |
+| ''mobile-app'' | 6 | 280 | 2.1% | [[design:mobile_and_app_measurement]] |
+| ''cookie'' | 1 | 53 | 1.9% | [[privacy:cookies]] |
+| ''email-message'' | 1 | 54 | 1.9% | [[privacy:email_tracking]] |
+| ''vulnerability'' | 14 | 880 | 1.6% | — |
+| **''website-category''** | **5** | **424** | **1.2%** | **this page** |
+| ''domain'' | 3 | 351 | 0.9% | — |
+| ''network-traffic'' | 2 | 382 | 0.5% | — |
+| ''web-request'' | 1 | 258 | 0.4% | [[privacy:requests]] |
+| ''ip-address'' | 1 | 295 | 0.3% | [[design:ip_classification]] |
+| ''malware'' | 0 | 159 | 0.0% | — |
+| ''sdk-or-library'' | 0 | 77 | 0.0% | — |
+| **''javascript''** | **0** | **44** | **0.0%** | [[privacy:javascript]] |
+| **''fingerprinting-script''** | **0** | **31** | **0.0%** | [[privacy:fingerprinting]] |
+| ''website-popularity'' | 0 | 15 | 0.0% | [[design:website_selection]] |
+The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets.
+Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, web, script, tracker, cookie, SDK, categor…// returns 20 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zero rows below survive the ''other'' bucket at keyword recall. **The probe's width decided that answer and had to be widened once:** its first version used ''\bpage\b'', which does not match the compound "webpage" and silently dropped two tuples, and it had no bare ''web'' at all. Read the printed list rather than the count — a probe is not a read of 116 papers, and it cannot rule out a ''targetDetail'' phrased in none of these words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage.
+=== And almost nobody names a model you could resolve ===
+The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names:
+The population here is the **175 papers that actually used or produced LLM labels**, not the 177 that mention one — the two papers whose only LLM tuple is a baseline they compared against are excluded, because "did you name the model you used" is not a question about a model you did not use.
+^ What the paper names ^ Papers ^ Share of 175 ^
+| **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 10 | 5.7% |
+| **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 24 | 13.7% |
+| **A or B — resolvable to an actual artefact** | **34** | **19.4%** |
+| **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 131 | 74.9% |
+| **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 10 | 5.7% |
-Twenty-nine papers across the corpus's fifteen years, twenty-seven of them in the final one — the curve is visibly starting, and the corpus ends before it goes anywhere. They point the model at "other" targets (19 papers), user-generated text (6), vulnerabilities (4), privacy policies (1) and domains (1). **Not one of the 29 uses an LLM to assign a website a topic category.**
+**One paper in five names something you could resolve; three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's.
-They do, however, establish a norm worth copying: **24 of 29 (82.8%) validate the model's labels against something**, usually manual annotation of a sample. The single ''website-category'' LLM tuple in the whole corpus is a USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k with a chain-of-thought prompt for a binary credential-page question, validated against three cybersecurity experts' annotations — a security verdict, not a topic.
+**B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so.
-See [[#LLM Classification]] for what has happened since the corpus ends.
+**The concrete instruction, which is one line in your methods section either way:** if the model is hosted, name the dated snapshot //and// the date you ran it, because the snapshot will be withdrawn; if it is open-weight, name family, size, instruct-variant and ideally the weights digest. Every string in all four buckets is listed on [[provenance:design:website_classification]], so this classification is auditable rather than asserted — including the two calls a reasonable person would make differently (''Mistral Large'' counted as B though it is a hosted API model, and ''FLAN-T5-XXL'' counted as B on a word-sized parameter count).
 ==== Two examples worth reading first ====
@@ Line 985: / Line 1120: @@
 ==== Methodology and limitations of these figures ====
-  * **Denominators.** Every figure names its own. The corpus is 4,322 papers; 3,241 classify something; 320 carry a ''website-category'' target; **247 assign a topic, industry or sector label** and are this section's population. Never "of 4,322 papers".
+  * **Denominators.** Every figure names its own. The corpus is 5,859 papers; 4,439 classify something; 424 carry a ''website-category'' target; **330 assign a topic, industry or sector label** and are this section's population. Never "of 5,859 papers".
   * **Papers, not mentions.** A paper naming McAfee three times counts once.
   * **Sentinels are not answers.** ''not-stated'' and ''none-mentioned'' are excluded from every numerator and reported as their own row where they matter — which they do, since "does not validate" is the headline finding.
-  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **20 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures.
+  * **Free-text names are folded** into canonical families by an ordered regex list before counting, because exact strings agree run-to-run only about 20% of the time. The fold left **33 distinct strings unclassified**, each appearing once: among them ''Taobao main-business categories'', ''SNET pillar portals'', ''Rakuten Popshops'', ''NAICSlite'', ''ORES topic routing'', ''multiple domain classification services'' and ''public classification service from the security vendor''. The last two are papers that used a service and declined to say which — a small, exact instance of the reporting gap this section measures. The residue was 40 strings before the fold was extended for this corpus; [[provenance:design:website_classification]] lists what was added and what was deliberately left out.
-  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 73 excluded papers are counted in the table above rather than dropped.
+  * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped.
   * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only.
-  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper.
+  * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about, and **6 of the 30 could not be matched even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers**. All six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched. Its unedited output is on [[provenance:design:website_classification]].
-  * **Coverage.** IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share. EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely, and so is anything published after 2024. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it.
+  * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures.
+  * **The per-target LLM table is shared with two other pages.** [[privacy:javascript]] and [[design:ip_classification]] make currency claims about the same ''classification.method == "llm"'' field for their own targets, so all three read from one script, ''scripts/report_llm_currency.mjs'', rather than from three separate queries. Its cross-page claim-check section restates each page's sentence next to what the corpus says, so a drift between them fails visibly.
+  * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]].
 ===== Open Questions =====
@@ Line 1009: / Line 1147: @@
   * [[Design:Website Selection]] — which domains to study, and popularity rankings.
   * [[Design:IP Classification]] — the same problem for addresses instead of names, with the same "which question are you actually asking" structure.
+  * [[Design:Crawling Location]] — where you crawl from, which is a different question from which country a site is aimed at ([[#Target country is not topic]]).
   * [[Privacy:Requests]] — tracker and advertiser classification, where filter lists beat categorisers.
   * [[Privacy:Cookies]], [[Privacy:Fingerprinting]], [[Privacy:JavaScript]] — classification of the other things a crawl collects.