| Both sides previous revisionPrevious revision | |
| design:website_classification [2026/09/03 22:13] – Generic-review fixes: retract the curated-database 'open sources returning' reading (p>=0.08 on every cut, and the row is mostly mis-filed vendors and media-bias raters - names now printed by the script); add Fisher exact p and n to every drift cut and re karel.kubicek.claude | design:website_classification [2026/09/03 22:33] (current) – Re-review fixes: the reproducibility buckets counted a paper on the strength of a model it only COMPARED against, so the population is now the 175 papers that used one and the figures are A 10, B 24, A+B 34 (19.4%), C 131, D 10; widen the target=other pro karel.kubicek.claude |
|---|
| * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. | * **Binary or two-way** ("is this a shopping site?"): use an LLM. Validate on a manually annotated sample of a few hundred and cite {[tsai2025_harmful]} for the precedent — that is now a defensible, refereed choice. |
| * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. | * **A real multi-class taxonomy** (13 Curlie categories, 400 IAB): Homepage2Vec or a vendor API remains the citable baseline. An LLM may well beat it — the only head-to-head says it does — but that head-to-head is one unrefereed preprint that ran the baseline without reporting its threshold, so if you rely on it, re-run the comparison yourself and report both. |
| * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|only 20% of the field does]]), constrain the output to the taxonomy, and report the rejection count. | * **Either way**: name the model to the artefact ([[#And almost nobody names a model you could resolve|fewer than one paper in five does]]), constrain the output to the taxonomy, and report the rejection count. |
| </WRAP> | </WRAP> |
| |
| A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. | A checklist, derived from what the 330 corpus papers below leave out. Every line is something a reviewer can reasonably ask for and most papers do not supply. |
| |
| - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **20.3% of the 177 corpus papers that use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. | - **Which service or model, and which version.** "McAfee" identified a different company before and after February 2024. "Homepage2Vec" without a commit hash is not reproducible; "GPT-4" without a dated snapshot is not either — and only **19.4% of the 175 corpus papers that actually use an LLM name a model resolvable to an artefact at all** (see [[#And almost nobody names a model you could resolve]]). For a hosted model, the dated snapshot //and// the date you ran it; for an open-weight one, family, size, instruct-variant and ideally the weights digest. |
| - **The date you queried.** These are live services whose labels change without notice. | - **The date you queried.** These are live services whose labels change without notice. |
| - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. | - **Coverage on //your// population**, as a fraction, before any filtering. How many of your domains got no label — and what you did with them. Dropping them silently changes the denominator of every downstream percentage. |
| |
| - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. | - **It is not statistically distinguishable from noise.** Fisher's exact test, two-sided, on every cut: p = 0.09 for the full 2025–2026 window, 0.08 for the five-venue control, 0.16 for 2025 alone, 0.23 for 2026 alone. Nothing under 0.05. |
| - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. The twelve papers in the recent window name //Tracker Radar Entity List, Cloudflare Radar, SimilarWeb, Symantec SiteReview, AllSides, MediaBias/FactCheck (×2), Science Feedback, IAB taxonomy, Homepage2Vec, NAICSlite// and "predefined source rules (custom)". **Three of those are commercial vendors, one is a model, one is a taxonomy and four are media-bias raters with closed editorial processes. Not one is DMOZ, Curlie, or anything like them.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. | - **More importantly, the row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve papers are: **three commercial vendors** (Cloudflare Radar, SimilarWeb, Symantec SiteReview); **four media-bias raters** with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); **one model** (Homepage2Vec); **two taxonomies rather than label databases** (IAB, NAICSlite); **one paper's own rule set** ("predefined source rules (custom)"); and **one that genuinely is an open, inspectable database** — DuckDuckGo's Tracker Radar Entity List, which is a public repository((https://github.com/duckduckgo/tracker-radar, checked 2026-09-03: public, not archived, last pushed 2026-09-02.)) — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** ''classification.method'' agrees run-to-run only 58% of the time (see [[#Methodology and limitations of these figures]]), and a vendor filed under ''curated-database'' rather than ''third-party-service'' is exactly the slip that produces. So the openness half of this page's central argument has **no** counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. |
| |
| **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. | **The third-party-service drop, by contrast, holds up under a composition control and a significance test.** The obvious way for it to be spurious is composition: CCS 2026 and IMC 2026 have not been held, so the last bucket is drawn from a different mix of venues than 2022–2024. |
| The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. | The zero rows are in the table on purpose. **Nothing in these seven venues points an LLM at a JavaScript file or a fingerprinting script** — 0 of the 44 and 0 of the 31 papers that classify those things — which is why [[privacy:javascript]] can still say the method is absent there while this page says it is arriving here. Neither page is out of date; they are different targets. |
| |
| Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, script, tracker, cookie, SDK, categor…// returns 16 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories, image content, decompiler clusters, MCP server categories, GDPR data categories and the like. So the zero rows below survive the ''other'' bucket at keyword recall. That is a probe, not a read of 116 papers, and it cannot rule out a paper whose ''targetDetail'' uses none of those words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. | Two caveats on the table. **''other'' takes 116 of the 177**, so for two thirds of LLM papers the target enum does not describe what they classified. That is less opaque than it sounds: ''other'' carries a free-text ''targetDetail'', it is stated on **all 157** such tuples, and a keyword probe over it for //website, domain, URL, page, web, script, tracker, cookie, SDK, categor…// returns 20 tuples, **none of which is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zero rows below survive the ''other'' bucket at keyword recall. **The probe's width decided that answer and had to be widened once:** its first version used ''\bpage\b'', which does not match the compound "webpage" and silently dropped two tuples, and it had no bare ''web'' at all. Read the printed list rather than the count — a probe is not a read of 116 papers, and it cannot rule out a ''targetDetail'' phrased in none of these words. And a share built on a single paper (''cookie'', ''ip-address'', ''web-request'', ''dark-pattern'') is a count dressed as a rate; read those rows as "one paper exists", not as a percentage. |
| |
| === And almost nobody names a model you could resolve === | === And almost nobody names a model you could resolve === |
| The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: | The reproducibility advice in [[#What to Report]] is checkable against these same 177 papers, and the field is failing it — but //how// it fails depends on whether the model is hosted or open-weight, and an earlier version of this table got that backwards. A paper is counted below in the strongest thing it names: |
| |
| ^ What the paper names ^ Papers ^ Share of 177 ^ | The population here is the **175 papers that actually used or produced LLM labels**, not the 177 that mention one — the two papers whose only LLM tuple is a baseline they compared against are excluded, because "did you name the model you used" is not a question about a model you did not use. |
| | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 11 | 6.2% | | |
| | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 25 | 14.1% | | |
| | **A or B — resolvable to an actual artefact** | **36** | **20.3%** | | |
| | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 130 | 73.4% | | |
| | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 11 | 6.2% | | |
| |
| **One paper in five names something you could resolve; nearly three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. | ^ What the paper names ^ Papers ^ Share of 175 ^ |
| | | **A** a dated **hosted** snapshot — ''gpt-4-turbo-2024-04-09'' | 10 | 5.7% | |
| | | **B** an **open-weight checkpoint** with a size — ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'' | 24 | 13.7% | |
| | | **A or B — resolvable to an actual artefact** | **34** | **19.4%** | |
| | | **C** a family with no version — ''GPT-4'', ''ChatGPT'', ''Mistral'', ''DeepSeek'' | 131 | 74.9% | |
| | | **D** no identifiable model — ''an LLM'', ''foundation LLMs'', ''local LLMs'' | 10 | 5.7% | |
| | |
| | **One paper in five names something you could resolve; three quarters name a family and stop.** ''GPT-4'' is not one model: OpenAI's own deprecations page lists six distinct ''gpt-4'' snapshots — ''-0314'', ''-0613'', ''-1106-preview'', ''-0125-preview'', ''-turbo-preview'' and ''-turbo-2024-04-09''.((Counted from https://developers.openai.com/api/docs/deprecations on 2026-09-03, matching ''gpt-4[-turbo]-<date-or-preview>'' strings. The count is of snapshots named on that page, not of every snapshot that ever existed.)) So a C-bucket paper's result is not reproducible and, more to the point, is not //comparable// to the next C-bucket paper's. |
| |
| **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. | **B is the stronger kind of pin, and the field is already better at it than at A.** An open-weight checkpoint stays resolvable; a hosted snapshot lasts as long as the vendor serves it, and ''gpt-4-turbo-2024-04-09'' — the exact string this page held up as good practice — is on OpenAI's published shutdown list for **23 October 2026**.((https://developers.openai.com/api/docs/deprecations, fetched 2026-09-03: ''gpt-4-turbo'' / ''gpt-4-turbo-2024-04-09'' → ''gpt-5.6-sol'', shutdown 23 October 2026. ''gpt-4o-2024-05-13'' and ''gpt-4-0613'', both also used by corpus papers, share that date.)) So the two buckets are not interchangeable and the table does not add them into a single "pinned" figure without saying so. |
