| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:website_classification [2026/08/27 13:44] – Point VirusTotal API/oracle at security:virustotal; contact-form academic subject. Authored by Claude karel.kubicek.claude | design:website_classification [2026/09/03 22:33] (current) – Re-review fixes: the reproducibility buckets counted a paper on the strength of a model it only COMPARED against, so the population is now the 175 papers that used one and the figures are A 10, B 24, A+B 34 (19.4%), C 131, D 10; widen the target=other pro karel.kubicek.claude |
|---|
| === Google Topics API === | === Google Topics API === |
| |
| The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call; the 2026 survey of classification services reports it as deprecated and scheduled for removal {[bozzolan2026_llmweb]}. | The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call. |
| | |
| | **It is being removed, and Chrome's own two records disagree about whether it already has been.** Checked 2026-09-03: Google's Privacy Sandbox status page lists Topics (Web) under "Deprecate and remove" and every Topics doc page carries the phaseout banner; the Blink //Intent to Deprecate and Remove// estimates deprecate in M144 and remove in M150, which Chrome stable has passed; but the Chrome Platform Status entry still reads "Enabled by default" with no removal milestone, and the removal was made explicitly pausable. If you need a Topics adoption figure for a related-work section, the deprecation thread is the only recent public source and it gives **13% of //page loads//** — not sites and not users, and driven by a few ad-tech scripts on many sites. Quote it with its unit.((https://privacysandbox.google.com/overview/status, https://chromestatus.com/feature/5680923054964736 and the Blink intent thread at https://groups.google.com/a/chromium.org/g/blink-dev/c/_R85yctz4Rs, all fetched 2026-09-03; Chrome stable 153.0.8010.27 from the Chrome Version History API the same day. This page previously reported the status second-hand from {[bozzolan2026_llmweb]}.)) |
| |
| ==== Discontinued ==== | ==== Discontinued ==== |
| === Alexa === | === Alexa === |
| |
| Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. [[Design:Website Selection]] currently gives 1 August 2023; we could not find a primary source for that date, and ''alexa.com'' now redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together. | Amazon retired **Alexa.com on 1 May 2022**, per its own end-of-service notice ("we will be retiring Alexa.com on May 1, 2022").((Captured on the alexa.com login page, https://web.archive.org/web/20220315000000/https://www.alexa.com/ — retrieved 2026-08-07. An earlier revision of [[Design:Website Selection]] gave 1 August 2023; that date is when Tranco dropped Alexa from the default list, not when Amazon switched the service off. The selection page now carries both dates. ''alexa.com'' redirects to the unrelated Amazon Alexa voice assistant.)) Its ranking service and its category service died together. |
| |
| * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate. | * **Why it still matters**: it appears in 12 of the 330 corpus papers that categorise websites, under eight different spellings, and papers published as late as 2024 still use it because their data collection predates the shutdown. If you are reading such a paper, the labels are from a dead service with a documented 0.53% coverage rate. |
| ===== LLM Classification ===== | ===== LLM Classification ===== |
| |
| <WRAP important> | Two questions, with different answers, and conflating them is how this section used to mislead. **Is an LLM a normal thing to classify with in this field?** Yes, since about 2024. **Is there published evidence that one can assign a website a topic category well enough to replace a vendor API?** For a **binary** question, yes, and it is peer-reviewed. For a multi-class taxonomy — which is the shape of most real categorisation tasks — the only systematic evidence is one unrefereed preprint. |
| **The corpus now reaches 2026, and the picture changed.** Of the 5,859 papers in the 2010–2026 corpus, **177 use an LLM to classify something** — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026 — and **two of them use one to assign a website a topic category**. The nearer of the two, {[tsai2025_harmful]}, reports **92% accuracy for GPT-4o on shopping-vs-non-shopping over 500 manually annotated Tranco sites, and says explicitly that this is "comparable to commercial website classification services"**. So a ranking of methods built from this literature no longer tells you that third-party services and keyword rules are the only options — but it still gives you a two-paper base for the topic task specifically, in the corpus's two thinnest and provisional years. Treat the peer-reviewed evidence as thin rather than absent, and read the rest of this section for what is outside the corpus. | |
| | <WRAP tip>**The decision, in three lines, if you do not read the rest of this section.** |
| | * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. |
| | * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. |
| | * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|fewer than one paper in five does]]), constrain the output to the taxonomy, and report the rejection count. |
| </WRAP> | </WRAP> |
| |
| The evidence outside the corpus is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first systematic study of LLMs for web-measurement classification. | **The method itself is now mainstream in these venues.** ''classification.method == "llm"'' fires on **177 of the 5,859 papers** in the 2010–2026 corpus — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026, which is 17.1% of that year's papers and 21.3% of the papers that classified anything that year — spread across all seven venues, with **83.6% validating the model's labels against something**. It is the fastest-moving quantity anywhere in this corpus. The full series, the per-target breakdown and the model-reporting numbers are in [[#Where LLMs actually appear]]; nothing in this section should be read as "unproven method". |
| | |
| | **For website topic classification specifically the peer-reviewed base is two papers, and the stronger one is a binary task.** {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with GPT-4o prompted with the URL and a screenshot, validates against 500 manually annotated sites, and reports **92% accuracy, which the paper itself calls "comparable to commercial website classification services"**. That is the first result in this corpus that makes an LLM a defensible alternative to the vendors catalogued above — and it is a two-class decision. It does not tell you what a model does with 13 or 400 categories, which is the shape of most real categorisation tasks, and nothing peer-reviewed in these venues does. The second paper, DarkGram (USENIX Security 2025), categorises Telegram channels rather than websites; count it as adjacent. |
| | |
| | <WRAP tip>**Why this section still spends most of its length on a preprint.** Because the peer-reviewed evidence stops exactly where the hard part starts. Two papers, one of them binary and one of them not about websites, cannot tell you how an LLM behaves on a multi-label taxonomy against a trained specialist classifier — and that comparison is the decision a reader of this page is actually making. The preprint below is the only source that has run it. It is cited for that, not because it outranks the corpus, and every number taken from it carries the version caveat in the box below.</WRAP> |
| | |
| | The one systematic study is a 2025/2026 preprint, {[bozzolan2026_llmweb]}, which by its authors' account is the first of its kind for web-measurement classification. |
| |
| <WRAP important> | <WRAP important> |
| |
| - **URL-only works for coarse questions and fails for topics.** Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly. | - **URL-only works for coarse questions and fails for topics.** Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly. |
| - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at August 2026 list prices for a nano-tier model and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs on the order of **a few dollars**, roughly halved by a batch API.((Our own arithmetic from published per-token prices, not a figure from any paper. Re-derive it at current prices before quoting it; full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility: an open-weight model you pin by digest is reproducible five years from now, and ''gpt-4o-2024-08-06'' is not. | - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at 2026-09-03 list prices and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs **$8.40**, or **$4.20** through a batch API, on the cheapest current-generation hosted model — and $2.30 / $1.15 on the cheapest //legacy// tier, which is on a shutdown list.((Our own arithmetic from the vendor's published per-token prices, not a figure from any paper: 30M input + 2M output tokens, batch at exactly half. ''gpt-5.6-luna'' at $0.20 / $1.20 per 1M gives $8.40 / $4.20; legacy ''gpt-5-nano'' at $0.05 / $0.40 gives $2.30 / $1.15. Re-derived from https://developers.openai.com/api/docs/pricing on 2026-09-03. **Name the tier and the date, not "the cheap one"** — the two current answers differ fourfold and the cheaper one is legacy. And full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility, and the direction of that tradeoff is now demonstrable rather than argued: an open-weight model you pin by digest is reproducible five years from now, and a hosted snapshot is not. **''gpt-4-turbo-2024-04-09'' — the exact string this page holds up as good practice, and the one two corpus papers used — is on OpenAI's published shutdown list for 23 October 2026**, with ''gpt-5.6-sol'' named as its replacement.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' / ''gpt-4-turbo-completions'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, are on the same date.)) Naming the snapshot is still the right thing to do — it makes your paper's result //interpretable// years later — but it does not make it re-runnable, and a paper that needs to be re-runnable needs open weights. |
| - **Ensembling helps because single models are unstable.** A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names "inconsistency, hallucination, category inflation, and misclassification" as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.((Kamen and Kamen, //Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization//, arXiv:2511.15714, November 2025.)) | - **Ensembling helps because single models are unstable.** A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names "inconsistency, hallucination, category inflation, and misclassification" as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.((Kamen and Kamen, //Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization//, arXiv:2511.15714, November 2025.)) |
| |
| |
| * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list. | * **The TLD.** Precise where it exists and absent where it matters — a ccTLD is strong evidence, but generic TLDs carry no country at all, and that is most of the head of any list. |
| * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX itself is CC BY-SA 4.0.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}. | * **CrUX country lists, via Tranco.** Tranco's list-generation API takes ''filterCRUX'', ''filterCRUXType'' (''global'' / ''country'' / ''region'' / ''subregion''), ''filterCRUXValue'' (e.g. a list of country codes) and ''filterCRUXMonth'', so you can generate a country-restricted list reproducibly, with a permalink, from a free account.((https://tranco-list.eu/api_documentation — fetched 2026-08-07. Basic Auth with your email as username and API token as password; the ''/configure'' page returns 401 without a login. CrUX datasets are CC BY 4.0 per Google's methodology, fetched 2026-08-27; Tranco's homepage still labels CrUX CC BY-SA 4.0, while the methodology page states no CrUX licence.)) This is the cleanest free source and it is under-used. **But it ranks by page loads from Chrome users in a country, which is popularity, not audience** — and the overlap is severe: the union of five country top-10k lists (China, Germany, Italy, Korea, Turkey) is only 18,718 domains rather than 50,000, with 4,017 domains common to all five, and a preliminary labelling built from these lists is incompatible with the site's own TLD in at least 25% of cases for every label {[bozzolan2026_llmweb]}. |
| * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web. | * **Site language.** A good proxy, and cheap, but it splits badly on English, Spanish, Arabic and Portuguese, which is a large share of the web. |
| * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]]. | * **Host IP or CDN location.** Measures where bytes are served from. Behind a CDN — most of the head of any list — it tells you about the CDN. See [[Design:IP Classification]] and [[Design:Crawling Location]]. |
| A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. | A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. |
| |
| - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either. | - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **19.4% of the 175 corpus papers that actually use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. |
| - **The date you queried.** These are live services whose labels change without notice. | - **The date you queried.** These are live services whose labels change without notice. |
| - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. | - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. |
| - **The categories you dropped**, and the rule you dropped them by. | - **The categories you dropped**, and the rule you dropped them by. |
| - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty. | - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty. |
| | - **If the classifier is an LLM, four more things**, none of which is standard practice yet and all of which a reviewer will now ask for: the **prompt text**, verbatim, as an appendix or artefact; the **temperature and decoding settings** (everyone sets temperature 0 and nobody says whether they checked that it mattered); the **output constraint** — the JSON schema or grammar you forced the answer into; and the **rejection count**, meaning how many responses fell outside your taxonomy and what you did with them. That last one is a result, not housekeeping: it is the closest thing to a measurement of how well the model understood the task. See [[#LLM Classification]]. |
| |
| <WRAP important> | <WRAP important> |
| Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. | Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. |
| |
| **In the provisional 2025–2026 window both of those trends reverse**: third-party services drop to 32.3% and curated databases return to 18.5%, with hand-built heuristics up to 23.1% and the first two LLM papers appearing at 3.1%. Two incomplete venue-years cannot establish a turn, and this is exactly the kind of row not to quote as a trend — but it is the first movement against the closed-source drift in fifteen years, and it is worth re-checking when 2026 is complete. | **In the provisional 2025–2026 window third-party-service reliance drops to 32.3%**, with hand-built heuristics up to 23.1% and the first two LLM papers at 3.1%. Two incomplete venue-years cannot establish a turn — but this one survives every test that could have killed it, and it is the first movement against the closed-source drift in fifteen years. |
| | |
| | **The `curated-database` row moves too, from 9.0% to 18.5%. Do not read that as open sources returning, and this page said so wrongly until 2026-09-03.** Two reasons, both checked: |
| | |
| | - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. |
| | - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve papers are: **three commercial vendors** (Cloudflare Radar, SimilarWeb, Symantec SiteReview); **four media-bias raters** with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); **one model** (Homepage2Vec); **two taxonomies rather than label databases** (IAB, NAICSlite); **one paper's own rule set** ("predefined source rules (custom)"); and **one that genuinely is an open, inspectable database** — DuckDuckGo's Tracker Radar Entity List, which is a public repository((https://github.com/duckduckgo/tracker-radar, checked 2026-09-03: public, not archived, last pushed 2026-09-02.)) — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. |
| | |
| | **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. |
| | |
| | ^ Cut ^ Third-party service ^ n ^ vs baseline ^ Fisher //p// ^ |
| | | 2022–2024, all venues //(the baseline)// | 44 (49.4%) | 89 | — | — | |
| | | 2025 alone | 17 (37.8%) | 45 | −11.7 pp | 0.27 //not supported// | |
| | | 2026 alone //(provisional)// | 4 (20.0%) | 20 | −29.4 pp | **0.024** | |
| | | 2025–2026, all venues | 21 (32.3%) | 65 | −17.1 pp | **0.047** | |
| | | 2022–2024, five venues with a 2026 slice | 34 (54.0%) | 63 | — | — | |
| | | **2025–2026, those same five venues** | **16 (28.6%)** | 56 | **−25.4 pp** | **0.006** | |
| | |
| | Restricting both windows to the five venues that have any 2026 paper (IEEE S&P, NDSS, PETS, USENIX Security, TheWebConf) makes the drop //larger//, not smaller — 54.0% to 28.6%, p = 0.006. The composition explanation therefore fails in the direction that would have rescued the old trend. |
| | |
| | **But 2025 on its own is not evidence.** At n = 45 and p = 0.27, the 37.8% is a point estimate that happens to sit the right way. An earlier version of this section leaned on it as a control; it is a description, not a test, and the significance comes entirely from the 2026 slice and the five-venue restriction. |
| | |
| | <WRAP todo>**What is still unsettled, and what would settle it.** The direction of the third-party-service drop is supported; its **magnitude** is not — the 2026-alone column rests on 20 papers, and TheWebConf 2025 is itself thin (63 corpus papers, against 284 across 2022–2024). The ''curated-database'' movement should not be quoted at all, in either direction. |
| | |
| | The thing that would settle it is CCS 2026 and IMC 2026, the two venues absent from the 2026 slice — and their absence does **not** cut the way you would guess. Within this population, 2022–2024 third-party-service share by venue runs TheWebConf 66.7%, IMC 64.3%, PETS 62.5%, IEEE S&P 55.6%, USENIX 47.6%, NDSS 20.0% and **CCS 8.3%, the lowest of the seven**. So IMC is the venue whose absence most plausibly inflates the drop, and adding CCS would on this evidence push the recent share //down// and strengthen the reversal. This page asserted the opposite until 2026-09-03; the sentence was the only one in the section not derived from a printed table, which is how it survived. |
| | |
| | Re-run ''report_website_classification.mjs'' once those two venue-years are in the corpus and compare the five-venue control against the all-venue row. Until then: report the direction for third-party services, do not report the magnitude, say nothing about curated databases, and do not describe the closed-source drift as over.</WRAP> |
| |
| ==== Which services, folded ==== | ==== Which services, folded ==== |
| | 2026 //(provisional)// | 71 | 415 | 17.1% | | | 2026 //(provisional)// | 71 | 415 | 17.1% | |
| |
| **177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026, and both of the last two venue-years are incomplete, so those shares are floors. They point the model at "other" targets (116 papers), user-generated text (32), vulnerabilities (14), privacy policies (12), mobile apps (6), website categories (5), and single-digit counts of domains, network traffic, consent notices, requests, IP addresses, cookies, email and dark patterns. | **177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026. Against the arguably fairer denominator — papers that classified //anything// that year, and so had the occasion to reach for a model — the same series reads 0.4%, 5.1%, 12.7% and **21.3%**. It is not concentrated in one venue: USENIX Security 43 papers, TheWebConf 40, NDSS 30, PETS 21, CCS 18, IEEE S&P 14, IMC 11. |
| | |
| | **The 2025 and 2026 //counts// are floors, because those venue-years are incomplete. The 2026 //share// is not** — a share whose numerator and denominator are both drawn from the same incomplete set is not a floor, it is a share of a differently-composed population, and the 2026 slice is 39% NDSS with CCS and IMC absent entirely. So the growth was checked like-for-like, restricting both years to the five venues that have a 2026 slice: **9.8% in 2025 → 17.1% in 2026**, and every one of the five rose on its own — IEEE S&P 8.5→10.7%, NDSS 9.2→11.7%, PETS 6.0→18.2%, USENIX Security 8.0→18.8%, TheWebConf 23.8→29.9%. CCS and IMC together were at 10.6% in 2025, below the five-venue 2026 figure, so their absence does not obviously inflate it. **The growth is robust; the exact 17.1% is not.** |
| | |
| | They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. Two of the 177 only //compared against// an LLM baseline rather than using one, so the adoption count is 175. |
| | |
| | === Which task, though === |
| | |
| | **The corpus-wide curve is the wrong number to reason from if you want to know whether there is a baseline for //your// task.** Adoption is extremely uneven by what is being classified, and three pages on this site have to agree about it, so the table is here and they link to it. Papers, not tuples; the right-hand column is the LLM share of the papers that classify that target //at all//, which is the quantity that decides whether "LLMs have arrived" is true for a given page. |
| | |
| | ^ ''classification.target'' ^ LLM papers ^ Papers classifying that target at all ^ LLM share ^ Page ^ |
| | | ''privacy-policy'' | 12 | 102 | **11.8%** | — | |
| | | ''dark-pattern'' | 1 | 13 | 7.7% | [[privacy:darkpatterns]] | |
| | | ''user-generated-text'' | 32 | 419 | 7.2% | — | |
| | | ''consent-notice'' | 2 | 39 | 5.1% | [[privacy:consent]] | |
| | | ''other'' | 116 | 2,592 | 4.4% | — | |
| | | ''mobile-app'' | 6 | 280 | 2.1% | [[design:mobile_and_app_measurement]] | |
| | | ''cookie'' | 1 | 53 | 1.9% | [[privacy:cookies]] | |
| | | ''email-message'' | 1 | 54 | 1.9% | [[privacy:email_tracking]] | |
| | | ''vulnerability'' | 14 | 880 | 1.6% | — | |
| | | **''website-category''** | **5** | **424** | **1.2%** | **this page** | |
| | | ''domain'' | 3 | 351 | 0.9% | — | |
| | | ''network-traffic'' | 2 | 382 | 0.5% | — | |
| | | ''web-request'' | 1 | 258 | 0.4% | [[privacy:requests]] | |
| | | ''ip-address'' | 1 | 295 | 0.3% | [[design:ip_classification]] | |
| | | ''malware'' | 0 | 159 | 0.0% | — | |
| | | ''sdk-or-library'' | 0 | 77 | 0.0% | — | |
| | | **''javascript''** | **0** | **44** | **0.0%** | [[privacy:javascript]] | |
| | | **''fingerprinting-script''** | **0** | **31** | **0.0%** | [[privacy:fingerprinting]] | |
| | | ''website-popularity'' | 0 | 15 | 0.0% | [[design:website_selection]] | |
| | |
| | The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. |
| | |
| | Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, web, script, tracker, cookie, SDK, categor…// returns 20 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zero rows below survive the ''other'' bucket at keyword recall. **The probe's width decided that answer and had to be widened once:** its first version used ''\bpage\b'', which does not match the compound "webpage" and silently dropped two tuples, and it had no bare ''web'' at all. Read the printed list rather than the count — a probe is not a read of 116 papers, and it cannot rule out a ''targetDetail'' phrased in none of these words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. |
| | |
| | === And almost nobody names a model you could resolve === |
| | |
| | The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: |
| |
| They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. | The population here is the **175 papers that actually used or produced LLM labels**, not the 177 that mention one — the two papers whose only LLM tuple is a baseline they compared against are excluded, because "did you name the model you used" is not a question about a model you did not use. |
| |
| **Five papers now point an LLM at a website category, and two of those are genuine topic classification** rather than an abuse verdict: | ^ What the paper names ^ Papers ^ Share of 175 ^ |
| | | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 10 | 5.7% | |
| | | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 24 | 13.7% | |
| | | **A or B — resolvable to an actual artefact** | **34** | **19.4%** | |
| | | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 131 | 74.9% | |
| | | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 10 | 5.7% | |
| |
| * {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with **GPT-4o prompted with the URL and a screenshot**, validates against 500 manually annotated sites, and reports **92% accuracy, "comparable to commercial website classification services"**. That sentence is the first thing in this corpus that makes an LLM a real alternative to the vendors catalogued above, and it is worth reading before you buy an API key. | **One paper in five names something you could resolve; three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. |
| * DarkGram (USENIX Security 2025) has GPT-4 label cybercriminal Telegram channels into 15 predefined categories, which the model itself extended to 19, validated by coders on 1,000 sampled posts. Categories of a channel, not of a website — count it as adjacent. | |
| |
| The other three are abuse verdicts: the USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k for a binary credential-page question, a TheWebConf 2025 crypto-scam study using a Llama3:70b + GPT-4 hybrid, and an NDSS 2026 credential-theft phishing detector using GPT-4o with retrieval-augmented prompts. All three validate against manual annotation. | **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. |
| |
| See [[#LLM Classification]] for what has happened outside the corpus. | **The concrete instruction, which is one line in your methods section either way:** if the model is hosted, name the dated snapshot //and// the date you ran it, because the snapshot will be withdrawn; if it is open-weight, name family, size, instruct-variant and ideally the weights digest. Every string in all four buckets is listed on [[provenance:design:website_classification]], so this classification is auditable rather than asserted — including the two calls a reasonable person would make differently (''Mistral Large'' counted as B though it is a hosted API model, and ''FLAN-T5-XXL'' counted as B on a word-sized parameter count). |
| |
| ==== Two examples worth reading first ==== | ==== Two examples worth reading first ==== |
| * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. | * **The task fold is imperfect and its exclusions are visible.** Eight side-channel and website-fingerprinting papers named their classifier only as "a 1D CNN" or "k-NN", leaving no signal in any extracted field; they were caught by matching the paper's own slug and are excluded by an explicit named list rather than by a general rule. The 94 excluded papers are counted in the table above rather than dropped. |
| * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. | * **Enum fields versus free text.** Method and validation are enums, stable enough to publish as rough shares (''classification.method'' agrees 58% run-to-run, so read those as a ranking). Service names and taxonomies are free text and are reported as rankings and folded families only. |
| * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five new LLM website-category tuples were re-checked individually on 2026-08-12; see [[provenance:design:website_classification]]. | * **Quotes were checked.** Every figure above traces to tuples carrying a verbatim evidence quote; a sample of these was re-located in the source PDFs. Of six quotes checked by hand, two initially "failed" a literal grep and turned out to be intact but split across a two-column break — normalise whitespace before concluding that a quote is not in the paper. The five LLM website-category tuples were re-checked individually on 2026-08-12 and again on 2026-09-03, this time by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'' checks all 30 LLM tuples across the targets the three classification pages make claims about, and **6 of the 30 could not be matched even after allowing for elisions, punctuation, the PDF's math-italic glyphs and dropped citation markers**. All six are ''privacy-policy'' or ''consent-notice'' tuples that no page on this site quotes; every tuple behind a figure on //this// page matched. Its unedited output is on [[provenance:design:website_classification]]. |
| | * **The 2025–2026 reversal was stress-tested, not just re-derived.** The two composition controls in [[#How they classify]] are computed by the same report script and printed with their bucket sizes. The direction survives both; the magnitude is not established. See the box there for what would settle it. |
| * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. | * **Coverage.** Seven venues only, with 2025 and 2026 incomplete by construction rather than by relevance, so any row that reaches them is a floor; the scope and the selection funnel are on [[literature:corpus]]. Notably, **{[vallina2020_misshapes]} itself is in the venue index but has no extracted full text** — the reference work for this page is not in the population the page measures. |
| | * **The per-target LLM table is shared with two other pages.** [[privacy:javascript]] and [[design:ip_classification]] make currency claims about the same ''classification.method == "llm"'' field for their own targets, so all three read from one script, ''scripts/report_llm_currency.mjs'', rather than from three separate queries. Its cross-page claim-check section restates each page's sentence next to what the corpus says, so a drift between them fails visibly. |
| * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]]. | * **Every query behind this section, the report script and its unedited output** are on [[provenance:design:website_classification]]; corpus-level caveats are on [[literature:corpus]]. |
| |
