Working notes behind website_classification — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Recorded. The population narrowing is this page's defining methodological move and the report prints every step of it.

The exclusion matters because website-category fires on at least seven unrelated tasks. Website fingerprinting is a traffic-analysis attack that shares no taxonomy, ground truth or service with topic categorisation; 36 of the 424 are that. Eight of those had generic classifier names (“a 1D CNN”, “k-NN”) leaving no signal in any extracted field and are excluded by an explicit named list keyed on the paper slug, not by a general rule.

cd /workspace/artifacts/wiki
node scripts/webcat_fold.mjs                            # self-test, prints residue
node scripts/report_website_classification.mjs          # every figure
node scripts/report_website_classification.mjs --wiki
node scripts/report_website_classification.mjs --list   # the 330 papers
node scripts/report_website_classification.mjs --quotes 'McAfee'
node scripts/check_page_numbers.mjs \
  pages/design_website_classification.txt out/new/report_website_classification.txt \
  '===== Use in Publications =====' '===== Open Questions ====='

check_page_numbers.mjs left four figures unaccounted. Three were deliberate — 1,000 (DarkGram's validation sample, from the paper), 500 (an image width in the design syntax, a false positive of the checker worth knowing about) and 7.4 (the old cross-check share, quoted on the page as the comparison). The fourth was a real catch: the page still said “the 73 excluded papers” after the count had moved to 94. Run it whole-page too (§10).

Recorded. This page moved more than any other in the refresh, for two independent reasons: IEEE S&P went from 43%-retrieved to complete, and LLM classification arrived.

The three findings that are more than arithmetic:

1. IEEE S&P: 3 papers → 25. The old page said IEEE S&P “essentially never” categorises websites and, in the same breath, that “IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share”. The caveat was right and the fix confirms it. The page now carries a box saying so, because the general lesson is worth more than the number: a venue that looks uninterested in a topic should be checked for a retrieval gap before it is described as uninterested.
2. LLM classification: 29 papers → 177, from 0.3% of 2023 to 17.1% of the partial 2026. This is the fastest-moving quantity anywhere in this corpus. Two papers now use an LLM for website topic classification, where the old page could truthfully say zero did. The stronger of the two, WWW/2025/harmful-terms-and-where-to-find-them… (tsai2025_harmful), reports 92% accuracy for GPT-4o prompted with URL + screenshot on shopping-vs-non-shopping over 500 manually annotated Tranco sites and states that this is “comparable to commercial website classification services”. Verified against the source (§6).
3. The closed-source drift reverses in 2025–2026 — provisionally. Third-party-service reliance had risen for four windows straight (25.0 → 49.4%) and falls to 32.3%; curated databases had fallen (21.4 → 9.0%) and return to 18.5%. Two incomplete venue-years cannot establish a turn and the page says so explicitly, but this is the first movement against a fifteen-year trend the page's central argument rests on, and it is the thing to re-check when 2026 is complete.

Recorded. taskFamily() was not modified. The task distribution is stable across the corpus growth (topic-or-industry 77.2% → 77.8%), which is reassuring given that the whole page rests on it. The eight hand-excluded website-fingerprinting papers are still excluded by slug.

Recorded. Six additions, each defensible, each recorded here because a fold change moves every number downstream of it:

Deliberately left in the residue, 33 strings, one paper each. They are printed in full by the report and on the content page in summary. The judgement is that these are a genuine long tail rather than a fold failure: Taobao main-business categories, SNET pillar portals, Rakuten Popshops, ORES topic routing, NAICSlite, IAB taxonomy (a taxonomy, not a service — it belongs to the other fold), NudeNet, DAppTotal, DappRadar, Arkham Intel, Googlebot, Censys, and the two that are the page's own best example of the reporting gap it measures — multiple domain classification services and public classification service from the security vendor, both papers that used a categoriser and declined to say which.

A reasonable person could fold several of these (urlblacklist.com and NextDNS Piracy Blocklist are arguably security-filtering; ScamAdviser API is arguably an aggregator; open-source topic classifier is arguably generic-ml). They were left out because each is one paper and folding a one-off into a family inflates that family's apparent breadth. The rule applied: fold when the string names a service the catalogue already knows, or when two spellings of one thing are both in the residue; otherwise leave it visible.

1. The last period bucket was (y) ⇒ y >= 2022, so it reported the 2022–2024 corpus size as 3,140 instead of 1,955 and folded 2025–2026 into it. This also corrupted the method over time table's last column, which is the table the page's central “closed-source drift” argument reads from: third-party-service showed 42.2% where the true 2022–2024 figure is 49.4%. Split, with a starred 2025–2026. Any figure copied from those two tables before 2026-08-12 is wrong.
2. Two hardcoded cross-references — ← OVERVIEW.md's 320 next to a value that is now 424, and (OVERVIEW.md publishes 71.7%) next to 72.9% — printed a stale annotation beside a fresh number. Both removed. The line asserting IEEE S&P is 43% retrieved was rewritten to record that it no longer is, and what changed as a result.

The identical open-ended-bucket bug was in report_fingerprinting.mjs, report_ip_classification.mjs and report_javascript.mjs. Four of the six report scripts had it. See corpus.

Recorded, 2026-08-12. The five method == “llm” + target == “website-category” tuples are new and load-bearing, so all five were checked individually against paper.cols.txt with whitespace normalised. All five are present.

The 92% figure the page now quotes was not taken from the extraction. The extraction's quote for that tuple is about the annotation of 500 sites; the accuracy claim was located separately in the full text, which reads: “GPT-4o, when prompted with URLs and screenshots, achieved an accuracy of 92%, comparable to commercial website classification services [49]”. This follows the rule the workdir README records for exactly this situation: grep the full text for any prevalence figure you publish, not just the attached quote.

A caveat on the second topic paper. DarkGram categorises cybercriminal Telegram channels into 15 predefined categories (which GPT-4 itself extended to 19), not websites. The task fold calls it topic-or-industry and that is defensible — it is topic categorisation of an online property — but a reader counting “LLM topic classification of websites” should count one paper, not two. The page says “count it as adjacent”.

Reconstructed: the original run recorded checking six quotes by hand, of which two initially failed a literal grep and were intact but split across a two-column break. Which six is not recoverable.

Reconstructed, except where noted. This page's non-corpus half is the largest on the site and was verified by the original run on 2026-08-07. It was not re-verified on 2026-08-12. Two things are worth carrying forward because they are the reasons the page is trustworthy:

• [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], the page's central citation, is not in the population the page measures. It is in data/corpus2/.meta/IMC-2020.json with a real DOI but has no data/fulltext entry, so none of its figures come from the extraction. It was read from the author's own copy at lepoch.at/files/domain-classification-imc20.pdf; the ACM DL copy was not retrievable. Still true on 2026-08-12.
• The LLM preprint section rests on one unrefereed arXiv posting whose results moved between versions. v1 (2025-10-09) benchmarked a cloud model on a 14-category task; v3 (2026-04-30) drops the cloud model, uses 13 categories, and adds both a Homepage2Vec baseline and a citation to Vallina et al. that v1 does not contain. The page reads v3 and says so. This refresh does not change that section, but the corpus now supplies a peer-reviewed data point (§4) that the section did not have, and the two should be read together.

Three specific errors the original run made and corrected, recorded because they are the failure modes of this page's subject matter:

1. “How many services did the paper study” had no single right answer. Vallina et al. is variously 13, 11, 9, 9 (a different nine) and 11 depending on which table you read. Three drafts got it wrong three different ways. The fix was to name the group each figure belongs to instead of reaching for a headline count.
2. Two columns per condition. Vallina et al.'s Table 5 gives Any (did the service return a label) and Related (was it semantically right). The page first published Dr.Web's and Trend Micro's numbers with the two silently swapped.
3. A framing sentence outran its own query. The lead once asserted that papers “routinely reach for a topic categoriser” to answer the other four questions. Measuring it — now a section of the report script — showed 9 of 47 abuse-verdict papers name a categorisation service and 6 of those used VirusTotal, the correct tool. On the new corpus it is 13 of 63, of which 9 use VirusTotal. The claim stayed retracted.

Rejected: SEO listicles of “best website categorization APIs” were rejected wholesale by the original run.

• Whether the 2025–2026 reversal of the closed-source drift is real. See §4. Two provisional venue-years.
• Whether LLM topic classification is one paper or two. See §6, the DarkGram caveat.
• Whether the 33 residue strings should be folded. See §5.2; several are borderline and the rule applied is stated rather than obviously correct.
• What the 155 papers that name a service actually got back. The extraction records the service, not the labels. Only Vallina et al. has ever measured the labels, and that was on a 2020 snapshot.
• The 33.0% not-applicable validation sentinel. The page argues it should be read as “no validation reported” rather than “validation was unnecessary”. That is an interpretation the extraction cannot settle; checking it means reading 109 papers.
• Nothing has re-audited the categorisation vendors since 2020. That is the page's largest open question and this refresh does not touch it.

Recorded. The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a systematic defect, not a scatter of typos, and it is worth stating because it will recur on the next refresh:

• check_page_numbers.mjs was run with a heading window — normally Use in Publications to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block was outside the window and stayed at its 4,322-corpus value. Across the six pages 29 such figures survived the first pass.
• The matcher was substring, not word-boundary, so report.includes('59') was satisfied by 11.59 bits. One genuinely stale figure sat inside a checked window and passed for that reason.

Both are fixed in scripts/check_page_numbers.mjs: matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, –code opts into scanning <file> blocks, and omitting the heading markers checks the whole page. Run it windowed and whole-page. The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. Fixed on this page's content page as a result: the lead paragraph's 72% → 75% for bespoke unsized taxonomies, which contradicted the corpus section's own “Three quarters … 75.4%” four screens down.

§3 above previously said check_page_numbers.mjs “left four figures unaccounted, all deliberate” and then listed three. That was wrong in the way this whole review is about: the fourth was 73, a genuine stale count, and describing it as deliberate would have buried the guard's only real catch on this page. Corrected.

← back to the content page · corpus-level provenance