| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:website_classification [2026/09/03 21:41] – LLM currency: lead the LLM Classification section with the peer-reviewed corpus evidence and demote the preprint to 'the one systematic benchmark'; add the shared per-target LLM table (the cross-page anchor for privacy:javascript and design:ip_classificat karel.kubicek.claude | design:website_classification [2026/09/03 22:33] (current) – Re-review fixes: the reproducibility buckets counted a paper on the strength of a model it only COMPARED against, so the population is now the 175 papers that used one and the figures are A 10, B 24, A+B 34 (19.4%), C 131, D 10; widen the target=other pro karel.kubicek.claude |
|---|
| === Google Topics API === | === Google Topics API === |
| |
| The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call; the 2026 survey of classification services reports it as deprecated and scheduled for removal {[bozzolan2026_llmweb]}. | The browser-side interest taxonomy (349 topics, assigned by a model Google trained on a manually curated set of 10,000 domains) is a measurement //target// rather than a tool — {[jha2023_topics]} attacks its re-identification properties. Treat it as an object of study, not a categoriser you can call. |
| | |
| | **It is being removed, and Chrome's own two records disagree about whether it already has been.** Checked 2026-09-03: Google's Privacy Sandbox status page lists Topics (Web) under "Deprecate and remove" and every Topics doc page carries the phaseout banner; the Blink //Intent to Deprecate and Remove// estimates deprecate in M144 and remove in M150, which Chrome stable has passed; but the Chrome Platform Status entry still reads "Enabled by default" with no removal milestone, and the removal was made explicitly pausable. If you need a Topics adoption figure for a related-work section, the deprecation thread is the only recent public source and it gives **13% of //page loads//** — not sites and not users, and driven by a few ad-tech scripts on many sites. Quote it with its unit.((https://privacysandbox.google.com/overview/status, https://chromestatus.com/feature/5680923054964736 and the Blink intent thread at https://groups.google.com/a/chromium.org/g/blink-dev/c/_R85yctz4Rs, all fetched 2026-09-03; Chrome stable 153.0.8010.27 from the Chrome Version History API the same day. This page previously reported the status second-hand from {[bozzolan2026_llmweb]}.)) |
| |
| ==== Discontinued ==== | ==== Discontinued ==== |
| ===== LLM Classification ===== | ===== LLM Classification ===== |
| |
| Two questions, with different answers, and conflating them is how this section used to mislead. **Is an LLM a normal thing to classify with in this field?** Yes, since about 2024. **Is there published evidence that one can assign a website a topic category well enough to replace a vendor API?** For a binary or few-class question, yes and it is peer-reviewed. For a full multi-class taxonomy, the only systematic evidence is one unrefereed preprint. | Two questions, with different answers, and conflating them is how this section used to mislead. **Is an LLM a normal thing to classify with in this field?** Yes, since about 2024. **Is there published evidence that one can assign a website a topic category well enough to replace a vendor API?** For a **binary** question, yes, and it is peer-reviewed. For a multi-class taxonomy — which is the shape of most real categorisation tasks — the only systematic evidence is one unrefereed preprint. |
| | |
| | <WRAP tip>**The decision, in three lines, if you do not read the rest of this section.** |
| | * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. |
| | * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. |
| | * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|fewer than one paper in five does]]), constrain the output to the taxonomy, and report the rejection count. |
| | </WRAP> |
| |
| **The method itself is now mainstream in these venues.** ''classification.method == "llm"'' fires on **177 of the 5,859 papers** in the 2010–2026 corpus — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026, which is 17.1% of that year's papers and 21.3% of the papers that classified anything that year — spread across all seven venues, with **83.6% validating the model's labels against something**. It is the fastest-moving quantity anywhere in this corpus. The full series, the per-target breakdown and the model-reporting numbers are in [[#Where LLMs actually appear]]; nothing in this section should be read as "unproven method". | **The method itself is now mainstream in these venues.** ''classification.method == "llm"'' fires on **177 of the 5,859 papers** in the 2010–2026 corpus — 2 in 2023, 27 in 2024, 77 in 2025 and 71 in the partial 2026, which is 17.1% of that year's papers and 21.3% of the papers that classified anything that year — spread across all seven venues, with **83.6% validating the model's labels against something**. It is the fastest-moving quantity anywhere in this corpus. The full series, the per-target breakdown and the model-reporting numbers are in [[#Where LLMs actually appear]]; nothing in this section should be read as "unproven method". |
| |
| - **URL-only works for coarse questions and fails for topics.** Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly. | - **URL-only works for coarse questions and fails for topics.** Every model reached at least 0.86 accuracy on the binary government task from the URL string alone, and exceeded 0.75 on both accuracy and macro F1 for the 10-way country task. Topic classification is where live access earns its cost: 0.51 → 0.65 Jaccard for the best model. If all you have is a domain list with no crawl, budget accordingly. |
| - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at August 2026 list prices for a nano-tier model and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs on the order of **a few dollars**, roughly halved by a batch API.((Our own arithmetic from published per-token prices, not a figure from any paper. Re-derive it at current prices before quoting it; full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility: an open-weight model you pin by digest is reproducible five years from now, and ''gpt-4o-2024-08-06'' is not. | - **Self-hosting is free in dollars and expensive in wall-clock.** On one A40 GPU, median under 2 s and mean under 4 s per site: ~3 days for 100k sites URL-only, ~4 days 9 hours with live access, and ~30–44 days for the Tranco top 1M. A hosted small model is cheaper in time — at 2026-09-03 list prices and ~300 input / 20 output tokens per domain, URL-only classification of 100k domains costs **$8.40**, or **$4.20** through a batch API, on the cheapest current-generation hosted model — and $2.30 / $1.15 on the cheapest //legacy// tier, which is on a shutdown list.((Our own arithmetic from the vendor's published per-token prices, not a figure from any paper: 30M input + 2M output tokens, batch at exactly half. ''gpt-5.6-luna'' at $0.20 / $1.20 per 1M gives $8.40 / $4.20; legacy ''gpt-5-nano'' at $0.05 / $0.40 gives $2.30 / $1.15. Re-derived from https://developers.openai.com/api/docs/pricing on 2026-09-03. **Name the tier and the date, not "the cheap one"** — the two current answers differ fourfold and the cheaper one is legacy. And full page text instead of a bare URL multiplies the input tokens and the cost.)) The tradeoff is reproducibility, and the direction of that tradeoff is now demonstrable rather than argued: an open-weight model you pin by digest is reproducible five years from now, and a hosted snapshot is not. **''gpt-4-turbo-2024-04-09'' — the exact string this page holds up as good practice, and the one two corpus papers used — is on OpenAI's published shutdown list for 23 October 2026**, with ''gpt-5.6-sol'' named as its replacement.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' / ''gpt-4-turbo-completions'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, are on the same date.)) Naming the snapshot is still the right thing to do — it makes your paper's result //interpretable// years later — but it does not make it re-runnable, and a paper that needs to be re-runnable needs open weights. |
| - **Ensembling helps because single models are unstable.** A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names "inconsistency, hallucination, category inflation, and misclassification" as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.((Kamen and Kamen, //Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization//, arXiv:2511.15714, November 2025.)) | - **Ensembling helps because single models are unstable.** A 2025 evaluation of ten LLMs zero-shot against the IAB taxonomy on 8,660 human-annotated samples names "inconsistency, hallucination, category inflation, and misclassification" as the individual-model failure modes, and reports up to 65% F1 improvement from majority-vote ensembling over the strongest single model.((Kamen and Kamen, //Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization//, arXiv:2511.15714, November 2025.)) |
| |
| A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. | A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. |
| |
| - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either. | - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **19.4% of the 175 corpus papers that actually use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. |
| - **The date you queried.** These are live services whose labels change without notice. | - **The date you queried.** These are live services whose labels change without notice. |
| - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. | - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. |
| - **The categories you dropped**, and the rule you dropped them by. | - **The categories you dropped**, and the rule you dropped them by. |
| - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty. | - **If you used more than one service, the disagreement between them.** Only 9 of 155 corpus papers that name a service name two, so this is nearly free novelty. |
| | - **If the classifier is an LLM, four more things**, none of which is standard practice yet and all of which a reviewer will now ask for: the **prompt text**, verbatim, as an appendix or artefact; the **temperature and decoding settings** (everyone sets temperature 0 and nobody says whether they checked that it mattered); the **output constraint** — the JSON schema or grammar you forced the answer into; and the **rejection count**, meaning how many responses fell outside your taxonomy and what you did with them. That last one is a result, not housekeeping: it is the closest thing to a measurement of how well the model understood the task. See [[#LLM Classification]]. |
| |
| <WRAP important> | <WRAP important> |
| Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. | Multi-valued, so shares exceed 100%. Reliance on third-party services grew steadily for four windows — 25.0% of categorising papers in 2010–2013, 33.9%, 43.5%, **49.4% in 2022–2024** — while use of curated databases (DMOZ and friends) fell from 21.4% to 9.0% across the middle of that run. The field moved from open, inspectable label sources toward closed, unauditable ones, over exactly the period in which the only audit of those sources found them wanting. |
| |
| **In the provisional 2025–2026 window both of those trends reverse**: third-party services drop to 32.3% and curated databases return to 18.5%, with hand-built heuristics up to 23.1% and the first two LLM papers appearing at 3.1%. Two incomplete venue-years cannot establish a turn, and this is exactly the kind of row not to quote as a trend — but it is the first movement against the closed-source drift in fifteen years. | **In the provisional 2025–2026 window third-party-service reliance drops to 32.3%**, with hand-built heuristics up to 23.1% and the first two LLM papers at 3.1%. Two incomplete venue-years cannot establish a turn — but this one survives every test that could have killed it, and it is the first movement against the closed-source drift in fifteen years. |
| |
| **Two controls, run on 2026-09-03, say the reversal is not an artefact of the missing venue-years.** The obvious way for this row to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024, and third-party services might simply be less popular at the venues that //do// have a 2026 slice. They are not. | **The `curated-database` row moves too, from 9.0% to 18.5%. Do not read that as open sources returning, and this page said so wrongly until 2026-09-03.** Two reasons, both checked: |
| |
| ^ Control ^ Third-party service ^ Curated database ^ Population ^ | - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. |
| | 2022–2024, all venues | 44 (49.4%) | 8 (9.0%) | 89 | | - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve papers are: **three commercial vendors** (Cloudflare Radar, SimilarWeb, Symantec SiteReview); **four media-bias raters** with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); **one model** (Homepage2Vec); **two taxonomies rather than label databases** (IAB, NAICSlite); **one paper's own rule set** ("predefined source rules (custom)"); and **one that genuinely is an open, inspectable database** — DuckDuckGo's Tracker Radar Entity List, which is a public repository((https://github.com/duckduckgo/tracker-radar, checked 2026-09-03: public, not archived, last pushed 2026-09-02.)) — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. |
| | **2025 alone** — all seven venues held | **17 (37.8%)** | **8 (17.8%)** | 45 | | |
| | 2026 alone //(provisional)// | 4 (20.0%) | 4 (20.0%) | 20 | | |
| | 2022–2024, only the five venues with a 2026 slice | 34 (54.0%) | 6 (9.5%) | 63 | | |
| | **2025–2026, only those same five venues** | **16 (28.6%)** | **12 (21.4%)** | 56 | | |
| |
| Both movements are already visible in **2025 on its own**, which is a held year at all seven venues, and restricting both windows to the five venues that have any 2026 paper (IEEE S&P, NDSS, PETS, USENIX Security, TheWebConf) makes the third-party-service drop //larger//, not smaller — 54.0% to 28.6%. So the composition explanation fails in the direction that would have rescued the old trend. | **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. |
| |
| <WRAP todo>**What is still unsettled, and what would settle it.** The 2026-alone column rests on 20 papers, and TheWebConf 2025 is itself thin (63 corpus papers against a 2022–2024 average near 95 a year), so the size of the reversal is not established even though its direction survives both controls. The thing that would settle it is CCS 2026 and IMC 2026 — the two venues absent from the 2026 slice, and the two whose 2022–2024 populations lean hardest on third-party services. Re-run ''report_website_classification.mjs'' once those are in the corpus and compare Control 2 against the all-venue row. Until then: report the direction, do not report the magnitude, and do not describe the closed-source drift as over.</WRAP> | ^ Cut ^ Third-party service ^ n ^ vs baseline ^ Fisher //p// ^ |
| | | 2022–2024, all venues //(the baseline)// | 44 (49.4%) | 89 | — | — | |
| | | 2025 alone | 17 (37.8%) | 45 | −11.7 pp | 0.27 //not supported// | |
| | | 2026 alone //(provisional)// | 4 (20.0%) | 20 | −29.4 pp | **0.024** | |
| | | 2025–2026, all venues | 21 (32.3%) | 65 | −17.1 pp | **0.047** | |
| | | 2022–2024, five venues with a 2026 slice | 34 (54.0%) | 63 | — | — | |
| | | **2025–2026, those same five venues** | **16 (28.6%)** | 56 | **−25.4 pp** | **0.006** | |
| | |
| | Restricting both windows to the five venues that have any 2026 paper (IEEE S&P, NDSS, PETS, USENIX Security, TheWebConf) makes the drop //larger//, not smaller — 54.0% to 28.6%, p = 0.006. The composition explanation therefore fails in the direction that would have rescued the old trend. |
| | |
| | **But 2025 on its own is not evidence.** At n = 45 and p = 0.27, the 37.8% is a point estimate that happens to sit the right way. An earlier version of this section leaned on it as a control; it is a description, not a test, and the significance comes entirely from the 2026 slice and the five-venue restriction. |
| | |
| | <WRAP todo>**What is still unsettled, and what would settle it.** The direction of the third-party-service drop is supported; its **magnitude** is not — the 2026-alone column rests on 20 papers, and TheWebConf 2025 is itself thin (63 corpus papers, against 284 across 2022–2024). The ''curated-database'' movement should not be quoted at all, in either direction. |
| | |
| | The thing that would settle it is CCS 2026 and IMC 2026, the two venues absent from the 2026 slice — and their absence does **not** cut the way you would guess. Within this population, 2022–2024 third-party-service share by venue runs TheWebConf 66.7%, IMC 64.3%, PETS 62.5%, IEEE S&P 55.6%, USENIX 47.6%, NDSS 20.0% and **CCS 8.3%, the lowest of the seven**. So IMC is the venue whose absence most plausibly inflates the drop, and adding CCS would on this evidence push the recent share //down// and strengthen the reversal. This page asserted the opposite until 2026-09-03; the sentence was the only one in the section not derived from a printed table, which is how it survived. |
| | |
| | Re-run ''report_website_classification.mjs'' once those two venue-years are in the corpus and compare the five-venue control against the all-venue row. Until then: report the direction for third-party services, do not report the magnitude, say nothing about curated databases, and do not describe the closed-source drift as over.</WRAP> |
| |
| ==== Which services, folded ==== | ==== Which services, folded ==== |
| | 2026 //(provisional)// | 71 | 415 | 17.1% | | | 2026 //(provisional)// | 71 | 415 | 17.1% | |
| |
| **177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026, and both of the last two venue-years are incomplete, so those shares are floors. Against the arguably fairer denominator — papers that classified //anything// that year, and so had the occasion to reach for a model — the same series reads 0.4%, 5.1%, 12.7% and **21.3%**. It is not concentrated in one venue: USENIX Security 43 papers, TheWebConf 40, NDSS 30, PETS 21, CCS 18, IEEE S&P 14, IMC 11. | **177 papers, and the curve is no longer starting — it is the fastest-moving thing in this corpus.** From 2 papers in 2023 to one in six in 2026. Against the arguably fairer denominator — papers that classified //anything// that year, and so had the occasion to reach for a model — the same series reads 0.4%, 5.1%, 12.7% and **21.3%**. It is not concentrated in one venue: USENIX Security 43 papers, TheWebConf 40, NDSS 30, PETS 21, CCS 18, IEEE S&P 14, IMC 11. |
| | |
| | **The 2025 and 2026 //counts// are floors, because those venue-years are incomplete. The 2026 //share// is not** — a share whose numerator and denominator are both drawn from the same incomplete set is not a floor, it is a share of a differently-composed population, and the 2026 slice is 39% NDSS with CCS and IMC absent entirely. So the growth was checked like-for-like, restricting both years to the five venues that have a 2026 slice: **9.8% in 2025 → 17.1% in 2026**, and every one of the five rose on its own — IEEE S&P 8.5→10.7%, NDSS 9.2→11.7%, PETS 6.0→18.2%, USENIX Security 8.0→18.8%, TheWebConf 23.8→29.9%. CCS and IMC together were at 10.6% in 2025, below the five-venue 2026 figure, so their absence does not obviously inflate it. **The growth is robust; the exact 17.1% is not.** |
| |
| They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. Two of the 177 only //compared against// an LLM baseline rather than using one, so the adoption count is 175. | They establish a norm worth copying: **148 of 177 (83.6%) validate the model's labels against something**, usually manual annotation of a sample — the same share as when this was 29 papers, which is the encouraging part. Two of the 177 only //compared against// an LLM baseline rather than using one, so the adoption count is 175. |
| The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. | The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. |
| |
| Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the schema's target enum does not describe what they classified — the ranking is informative about the covered targets and silent about the rest. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. | Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, web, script, tracker, cookie, SDK, categor…// returns 20 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zero rows below survive the ''other'' bucket at keyword recall. **The probe's width decided that answer and had to be widened once:** its first version used ''\bpage\b'', which does not match the compound "webpage" and silently dropped two tuples, and it had no bare ''web'' at all. Read the printed list rather than the count — a probe is not a read of 116 papers, and it cannot rule out a ''targetDetail'' phrased in none of these words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. |
| | |
| === And almost nobody says which model === | |
| |
| The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it: | === And almost nobody names a model you could resolve === |
| |
| ^ What the paper names as its model ^ Papers ^ Share of 177 ^ | The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: |
| | A dated snapshot or a pinned size/tag (''gpt-4-turbo-2024-04-09'', ''llama3.3:70b'') | 13 | **7.3%** | | |
| | A model family with no version (''GPT-4'', ''ChatGPT'', ''Claude'') | 154 | 87.0% | | |
| | No identifiable model at all (''an LLM'', ''foundation LLMs'', ''local LLMs'') | 10 | 5.6% | | |
| |
| **Fewer than one in thirteen pins the model.** And that 7.3% is generous: it counts an Ollama tag such as ''llama3.3:70b'' as pinned, which fixes the parameter count and not the weights. All ten strings in the bottom row are printed in full on [[provenance:design:website_classification]]. If you use a model to classify, you are currently able to be in the top 7% of your field on this axis by writing one extra token into your methods section. | The population here is the **175 papers that actually used or produced LLM labels**, not the 177 that mention one — the two papers whose only LLM tuple is a baseline they compared against are excluded, because "did you name the model you used" is not a question about a model you did not use. |
| |
| **Five papers now point an LLM at a website category, and two of those are genuine topic classification** rather than an abuse verdict: | ^ What the paper names ^ Papers ^ Share of 175 ^ |
| | | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 10 | 5.7% | |
| | | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 24 | 13.7% | |
| | | **A or B — resolvable to an actual artefact** | **34** | **19.4%** | |
| | | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 131 | 74.9% | |
| | | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 10 | 5.7% | |
| |
| * {[tsai2025_harmful]} (TheWebConf 2025) classifies Tranco sites as shopping or non-shopping with **GPT-4o prompted with the URL and a screenshot**, validates against 500 manually annotated sites, and reports **92% accuracy, "comparable to commercial website classification services"**. That sentence is the first thing in this corpus that makes an LLM a real alternative to the vendors catalogued above, and it is worth reading before you buy an API key. | **One paper in five names something you could resolve; three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. |
| * DarkGram (USENIX Security 2025) has GPT-4 label cybercriminal Telegram channels into 15 predefined categories, which the model itself extended to 19, validated by coders on 1,000 sampled posts. Categories of a channel, not of a website — count it as adjacent. | |
| |
| The other three are abuse verdicts: the USENIX Security 2024 phishing detector using GPT-3.5-turbo-16k for a binary credential-page question, a TheWebConf 2025 crypto-scam study using a Llama3:70b + GPT-4 hybrid, and an NDSS 2026 credential-theft phishing detector using GPT-4o with retrieval-augmented prompts. All three validate against manual annotation. | **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. |
| |
| See [[#LLM Classification]] for what has happened outside the corpus, and [[privacy:requests]] and [[privacy:javascript]] for the two neighbouring targets where the same query returns one paper and none. | **The concrete instruction, which is one line in your methods section either way:** if the model is hosted, name the dated snapshot //and// the date you ran it, because the snapshot will be withdrawn; if it is open-weight, name family, size, instruct-variant and ideally the weights digest. Every string in all four buckets is listed on [[provenance:design:website_classification]], so this classification is auditable rather than asserted — including the two calls a reasonable person would make differently (''Mistral Large'' counted as B though it is a hosted API model, and ''FLAN-T5-XXL'' counted as B on a word-sized parameter count). |
| |
| ==== Two examples worth reading first ==== | ==== Two examples worth reading first ==== |
