Working notes behind website_classification — every query, its population and its denominator, the report script and its unedited output, the folds and their residue, the quotes that were checked, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

Recorded. The population narrowing is this page's defining methodological move and the report prints every step of it.

The exclusion matters because website-category fires on at least seven unrelated tasks. Website fingerprinting is a traffic-analysis attack that shares no taxonomy, ground truth or service with topic categorisation; 36 of the 424 are that. Eight of those had generic classifier names (“a 1D CNN”, “k-NN”) leaving no signal in any extracted field and are excluded by an explicit named list keyed on the paper slug, not by a general rule.

cd /workspace/artifacts/wiki
node scripts/webcat_fold.mjs                            # self-test, prints residue
node scripts/report_website_classification.mjs          # every figure
node scripts/report_website_classification.mjs --wiki
node scripts/report_website_classification.mjs --list   # the 330 papers
node scripts/report_website_classification.mjs --quotes 'McAfee'
node scripts/check_page_numbers.mjs \
  pages/design_website_classification.txt out/new/report_website_classification.txt \
  '===== Use in Publications =====' '===== Open Questions ====='

check_page_numbers.mjs left four figures unaccounted. Three were deliberate — 1,000 (DarkGram's validation sample, from the paper), 500 (an image width in the design syntax, a false positive of the checker worth knowing about) and 7.4 (the old cross-check share, quoted on the page as the comparison). The fourth was a real catch: the page still said “the 73 excluded papers” after the count had moved to 94. Run it whole-page too (§10).

Recorded. This page moved more than any other in the refresh, for two independent reasons: IEEE S&P went from 43%-retrieved to complete, and LLM classification arrived.

The three findings that are more than arithmetic:

1. IEEE S&P: 3 papers → 25. The old page said IEEE S&P “essentially never” categorises websites and, in the same breath, that “IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share”. The caveat was right and the fix confirms it. The page now carries a box saying so, because the general lesson is worth more than the number: a venue that looks uninterested in a topic should be checked for a retrieval gap before it is described as uninterested.
2. LLM classification: 29 papers → 177, from 0.3% of 2023 to 17.1% of the partial 2026. This is the fastest-moving quantity anywhere in this corpus. Two papers now use an LLM for website topic classification, where the old page could truthfully say zero did. The stronger of the two, WWW/2025/harmful-terms-and-where-to-find-them… (tsai2025_harmful), reports 92% accuracy for GPT-4o prompted with URL + screenshot on shopping-vs-non-shopping over 500 manually annotated Tranco sites and states that this is “comparable to commercial website classification services”. Verified against the source (§6).
3. The closed-source drift reverses in 2025–2026 — provisionally. Third-party-service reliance had risen for four windows straight (25.0 → 49.4%) and falls to 32.3%; curated databases had fallen (21.4 → 9.0%) and return to 18.5%. Two incomplete venue-years cannot establish a turn and the page says so explicitly, but this is the first movement against a fifteen-year trend the page's central argument rests on, and it is the thing to re-check when 2026 is complete.

Recorded. taskFamily() was not modified. The task distribution is stable across the corpus growth (topic-or-industry 77.2% → 77.8%), which is reassuring given that the whole page rests on it. The eight hand-excluded website-fingerprinting papers are still excluded by slug.

Recorded. Six additions, each defensible, each recorded here because a fold change moves every number downstream of it:

Deliberately left in the residue, 33 strings, one paper each. They are printed in full by the report and on the content page in summary. The judgement is that these are a genuine long tail rather than a fold failure: Taobao main-business categories, SNET pillar portals, Rakuten Popshops, ORES topic routing, NAICSlite, IAB taxonomy (a taxonomy, not a service — it belongs to the other fold), NudeNet, DAppTotal, DappRadar, Arkham Intel, Googlebot, Censys, and the two that are the page's own best example of the reporting gap it measures — multiple domain classification services and public classification service from the security vendor, both papers that used a categoriser and declined to say which.

A reasonable person could fold several of these (urlblacklist.com and NextDNS Piracy Blocklist are arguably security-filtering; ScamAdviser API is arguably an aggregator; open-source topic classifier is arguably generic-ml). They were left out because each is one paper and folding a one-off into a family inflates that family's apparent breadth. The rule applied: fold when the string names a service the catalogue already knows, or when two spellings of one thing are both in the residue; otherwise leave it visible.

1. The last period bucket was (y) ⇒ y >= 2022, so it reported the 2022–2024 corpus size as 3,140 instead of 1,955 and folded 2025–2026 into it. This also corrupted the method over time table's last column, which is the table the page's central “closed-source drift” argument reads from: third-party-service showed 42.2% where the true 2022–2024 figure is 49.4%. Split, with a starred 2025–2026. Any figure copied from those two tables before 2026-08-12 is wrong.
2. Two hardcoded cross-references — ← OVERVIEW.md's 320 next to a value that is now 424, and (OVERVIEW.md publishes 71.7%) next to 72.9% — printed a stale annotation beside a fresh number. Both removed. The line asserting IEEE S&P is 43% retrieved was rewritten to record that it no longer is, and what changed as a result.

The identical open-ended-bucket bug was in report_fingerprinting.mjs, report_ip_classification.mjs and report_javascript.mjs. Four of the six report scripts had it. See corpus.

Recorded, 2026-08-12. The five method == “llm” + target == “website-category” tuples are new and load-bearing, so all five were checked individually against paper.cols.txt with whitespace normalised. All five are present.

The 92% figure the page now quotes was not taken from the extraction. The extraction's quote for that tuple is about the annotation of 500 sites; the accuracy claim was located separately in the full text, which reads: “GPT-4o, when prompted with URLs and screenshots, achieved an accuracy of 92%, comparable to commercial website classification services [49]”. This follows the rule the workdir README records for exactly this situation: grep the full text for any prevalence figure you publish, not just the attached quote.

A caveat on the second topic paper. DarkGram categorises cybercriminal Telegram channels into 15 predefined categories (which GPT-4 itself extended to 19), not websites. The task fold calls it topic-or-industry and that is defensible — it is topic categorisation of an online property — but a reader counting “LLM topic classification of websites” should count one paper, not two. The page says “count it as adjacent”.

Reconstructed: the original run recorded checking six quotes by hand, of which two initially failed a literal grep and were intact but split across a two-column break. Which six is not recoverable.

Reconstructed, except where noted. This page's non-corpus half is the largest on the site and was verified by the original run on 2026-08-07. It was not re-verified on 2026-08-12. Two things are worth carrying forward because they are the reasons the page is trustworthy:

• [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], the page's central citation, is not in the population the page measures. It is in data/corpus2/.meta/IMC-2020.json with a real DOI but has no data/fulltext entry, so none of its figures come from the extraction. It was read from the author's own copy at lepoch.at/files/domain-classification-imc20.pdf; the ACM DL copy was not retrievable. Still true on 2026-08-12.
• The LLM preprint section rests on one unrefereed arXiv posting whose results moved between versions. v1 (2025-10-09) benchmarked a cloud model on a 14-category task; v3 (2026-04-30) drops the cloud model, uses 13 categories, and adds both a Homepage2Vec baseline and a citation to Vallina et al. that v1 does not contain. The page reads v3 and says so. This refresh does not change that section, but the corpus now supplies a peer-reviewed data point (§4) that the section did not have, and the two should be read together.

Three specific errors the original run made and corrected, recorded because they are the failure modes of this page's subject matter:

1. “How many services did the paper study” had no single right answer. Vallina et al. is variously 13, 11, 9, 9 (a different nine) and 11 depending on which table you read. Three drafts got it wrong three different ways. The fix was to name the group each figure belongs to instead of reaching for a headline count.
2. Two columns per condition. Vallina et al.'s Table 5 gives Any (did the service return a label) and Related (was it semantically right). The page first published Dr.Web's and Trend Micro's numbers with the two silently swapped.
3. A framing sentence outran its own query. The lead once asserted that papers “routinely reach for a topic categoriser” to answer the other four questions. Measuring it — now a section of the report script — showed 9 of 47 abuse-verdict papers name a categorisation service and 6 of those used VirusTotal, the correct tool. On the new corpus it is 13 of 63, of which 9 use VirusTotal. The claim stayed retracted.

Rejected: SEO listicles of “best website categorization APIs” were rejected wholesale by the original run.

• Whether the 2025–2026 reversal of the closed-source drift is real. See §4. Two provisional venue-years.
• Whether LLM topic classification is one paper or two. See §6, the DarkGram caveat.
• Whether the 33 residue strings should be folded. See §5.2; several are borderline and the rule applied is stated rather than obviously correct.
• What the 155 papers that name a service actually got back. The extraction records the service, not the labels. Only Vallina et al. has ever measured the labels, and that was on a 2020 snapshot.
• The 33.0% not-applicable validation sentinel. The page argues it should be read as “no validation reported” rather than “validation was unnecessary”. That is an interpretation the extraction cannot settle; checking it means reading 109 papers.
• Nothing has re-audited the categorisation vendors since 2020. That is the page's largest open question and this refresh does not touch it.

Recorded. The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a systematic defect, not a scatter of typos, and it is worth stating because it will recur on the next refresh:

• check_page_numbers.mjs was run with a heading window — normally Use in Publications to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block was outside the window and stayed at its 4,322-corpus value. Across the six pages 29 such figures survived the first pass.
• The matcher was substring, not word-boundary, so report.includes('59') was satisfied by 11.59 bits. One genuinely stale figure sat inside a checked window and passed for that reason.

Both are fixed in scripts/check_page_numbers.mjs: matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, --code opts into scanning <file> blocks, and omitting the heading markers checks the whole page. Run it windowed and whole-page. The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. Fixed on this page's content page as a result: the lead paragraph's 72% → 75% for bespoke unsized taxonomies, which contradicted the corpus section's own “Three quarters … 75.4%” four screens down.

§3 above previously said check_page_numbers.mjs “left four figures unaccounted, all deliberate” and then listed three. That was wrong in the way this whole review is about: the fourth was 73, a genuine stale count, and describing it as deliberate would have buried the guard's only real catch on this page. Corrected.

The refresh of 2026-08-12 (§4) established that LLM classification had arrived in the corpus. This section is the follow-up it asked for: it settles which of three pages owns the currency claim, retracts two wrong sentences, and stress-tests the closed-source-drift reversal §4 left open — which turned out to be half right. Companion sections were written the same day on javascript and ip_classification; the shared numbers live here and those two link to them.

Sections 12.1–12.9 were kept as a log during the run. §12.10 and §12.12 are the review passes, and several figures in 12.1–12.9 are there in their corrected form because a reviewer changed them — where that happened it is said so, with the wrong value.

Three pages made currency claims about the same schema field, classification.method == “llm”, from three separate queries, and they had drifted:

One script answers all of them: scripts/report_llm_currency.mjs. It exists so the three pages cannot disagree again — its last section restates each page's own sentence next to what the corpus says, so a future drift fails visibly instead of quietly.

cd /workspace/artifacts/wiki
node scripts/report_llm_currency.mjs                         # every figure
node scripts/report_llm_currency.mjs --wiki                  # DokuWiki tables
node scripts/report_llm_currency.mjs --list website-category # the papers behind a row
node scripts/report_llm_currency.mjs --quotes cookie         # the evidence quotes
node scripts/llm_currency_quotecheck.mjs                     # quotes vs paper.cols.txt
node scripts/report_website_classification.mjs               # incl. the drift controls,
                                                             # Fisher tests and compositions

Three denominator traps this section had to avoid, and one it fell into.

1. “177 of 5,859” is not an adoption rate for anything. The per-target column that decides a page's claim is llm papers at target T ÷ papers that classify T at all — 1 of 53 for cookie, 5 of 424 for website-category, 0 of 44 for javascript. Published without that denominator, the same field supports both “LLMs have arrived” and “LLMs are absent”, which is how the three pages drifted apart.
2. validation has two sentinels, not one. The values are manual-validation, held-out-test-set, comparison-to-other-method, cross-validation, none-reported and not-applicable. The first draft of report_llm_currency.mjs used the shared isSentinel() helper, which knows not-applicable but not none-reported, and produced 173 of 177 (97.7%) validated where the truth is 148 (83.6%). Caught before saving by diffing against report_website_classification.mjs, which had the explicit allowlist; both now share it. It never reached a page, but it was the most flattering figure in the section and it was wrong by 14 points.
3. A count from an incomplete venue-year is a floor; a share is not. The page said “both of the last two venue-years are incomplete, so those shares are floors”. That is a logic error — a share whose numerator and denominator come from the same incomplete set is a share of a differently-composed population, not a lower bound. Flagged in the generic review (§12.12, finding 9) and replaced with the like-for-like five-venue check above, which is the thing that actually supports the growth claim.
4. classification.method being 58% stable is not a footnote. §12.4: it is the whole reason one of this page's two headline movements had to be retracted.

Decision: no, and the preprint stays anyway. The section now opens by separating two questions — is an LLM a normal thing to classify with in this field? (yes, 177 papers, since about 2024) and can one assign a website a topic well enough to replace a vendor API? (peer-reviewed for a binary question; only the preprint for a real taxonomy). The corpus evidence moved out of a <WRAP important> box into the body, because a box reads as a caveat bolted onto the surrounding prose and the surrounding prose was preprint-led.

The preprint keeps most of the section's length, and the page now says why: the peer-reviewed evidence stops exactly where the hard part starts. [2Tsai, Elisa; Mangaokar, Neal; Zheng, Boyuan; Zheng, Haizhong; Prakash, Atul (2025): "Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale", in: Proceedings of the ACM Web Conference. (DOI)] is a two-class shopping/non-shopping decision. The comparison a reader of that page is actually making — an LLM against a trained specialist classifier on a multi-label taxonomy — has been run once, in [3Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)], and nowhere else.

Two alternatives were considered. Cutting the preprint to a paragraph was rejected: the alternative to it is not a better source, it is silence, and the page's own instability box is more use to a reader than an omission. Moving the preprint's tables and cautions into Run Your Own Classifier — where Homepage2Vec already lives and where a self-hosted open-weight model belongs — was raised by the generic reviewer (§12.12, finding 11) and not done, because it is a large restructure with anchor breakage of its own and this run had already rewritten the section once. What was done instead is the thing the reviewer said was missing: a three-line decision box at the top of the section, so a reader who does not read the preprint analysis still leaves with an answer. The restructure remains the right call and is deferred, not rejected.

§4 left this as the thing to re-check “when 2026 is complete”. 2026 is not complete and will not be for this corpus, so the run did three other things: split the window by year, control for venue composition, and — after the generic review — run Fisher's exact test and print what the enum row actually contains. The result is that half of what §4 recorded does not survive.

The composition control makes the drop larger, so the missing-venue explanation fails in the direction that would have rescued the old trend. But the earlier draft of this section leaned on “2025 alone, all seven venues held” as a control, and at n = 45 and p = 0.27 that is a description, not a test. The significance comes from the 2026 slice and the five-venue restriction. Corrected on the page and here.

§4 recorded that “curated databases had fallen (21.4 → 9.0%) and return to 18.5%”, and called it the first movement against the closed-source drift. That reading is withdrawn, for two reasons found in the generic review (§12.12, finding 1) and confirmed here:

1. No cut clears p < 0.05. 0.16 for 2025 alone, 0.23 for 2026 alone, 0.095 for the full window, 0.079 for the five-venue control.
2. The row does not contain open directories. report_website_classification.mjs now prints every name behind it. Sorted, the twelve recent-window papers are three commercial vendors (Cloudflare Radar, SimilarWeb, Symantec SiteReview); four media-bias raters with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); one model (Homepage2Vec); two taxonomies rather than label databases (IAB, NAICSlite); one paper's own rule set; and one that genuinely is an open, inspectable repository — DuckDuckGo's Tracker Radar Entity List — but of tracker entities, not website topics. Not one is DMOZ, Curlie or any comparable open topic directory.

The first version of this list, on this page and on the content page, named eleven of the twelve — it silently dropped NDSS/2026/revealing-the-secret-power…, whose resource string is “Media Bias/Fact Check (MBFC)”, while the derived “four media-bias raters” counted it. Caught in re-review (§12.13, finding 2). A hand-written enumeration beside an embedded script output is the one kind of listing on this page that can still go stale, and this is the second time in one run it did. By contrast the eight in 2022–2024 are more genuinely directory-like: DappRadar, a pornhosts blocklist, WebPulse, the Citizen Lab Block List, the AllSides Media Bias Chart, MediaBias/FactCheck, YouTube category labels, and one paper's “external political, government, media and issue-page sources”.

So the openness half of this page's central argument has no counter-evidence in the recent window; the enum row that looked like counter-evidence is mostly mis-filed vendors. This is classification.method's 58% run-to-run stability doing exactly what the methodology bullet warns it does, on the one row where the page had built an argument on top of it.

The <WRAP todo> box said CCS 2026 and IMC 2026 are “the two whose 2022–2024 populations lean hardest on third-party services”. Within this population that share by venue is TheWebConf 66.7%, IMC 64.3%, PETS 62.5%, IEEE S&P 55.6%, USENIX 47.6%, NDSS 20.0% and CCS 8.3% — the lowest of the seven. So IMC's absence is the one that plausibly inflates the drop, and adding CCS would push the recent share down and strengthen the reversal. The page asserted the opposite.

This was the only sentence in the drift section not derived from a printed table, which is why it survived three earlier passes, including a figures-versus-script reviewer who had nothing to diff it against. The by-venue table is now printed by the report script. The lesson generalises: a sentence that tells the reader what a missing input would do is a claim, and needs a query like any other.

classification.resourceName is free text and agrees run-to-run on roughly a fifth of exact strings, so anything counted from it is a fold, reported as a ranking, with the residue printed. Both folds live inside report_llm_currency.mjs rather than in a shared module, because they are specific to this question and a shared version would invite reuse where the families are wrong.

The table above is the 2026-09-21 re-derivation. Until then it read GPT-4 (non-4o) 40 (22.6%), UNMAPPED 23 (13.0%) and OpenAI reasoning 9 (5.1%), because four residue strings — GPT 4.1, GPT-4.0, GPT-4.5 and GPT-o1 — are OpenAI models that the fold missed on a space or a decimal. They were left visible on 2026-09-03 rather than quietly folded, because patching a fold to absorb its own residue after seeing the output stops it being a documented rule. The fold has now been extended as a rule about how OpenAI model strings are spelled, not as a list of those four: see §12.14 for the rule, the two further strings it moves, and the full diff. Grok-3, GLM-4.5, ChatGLM and Kimi stay in the now 20-string residue — they are the genuine long tail, four vendors with no family in the list, and no family was added for them.

The first version of this table had a single “dated or pinned” bucket built from OpenAI snapshot suffixes plus an Ollama size tag, published as “11 papers name a model you could re-run”. The generic reviewer showed that was wrong in a way that inverted the page's own advice (§12.12, finding 3): Llama-3.1-70B-Instruct, Qwen2.5-Coder-32B-Instruct, Mistral-7B-Instruct-v0.2, Gemma 3 27B and T5-11B are fully resolvable artefacts and all landed in the “no version” bucket, while the paragraph two sentences later said open weights are the only re-runnable option. Roughly 25 papers were being counted as unpinned for naming exactly what the page told them to name.

Rebuilt, split by hosting, which is the axis that actually decides reproducibility:

The population is the 175 papers that used or produced LLM labels, not the 177 that mention one, and getting that wrong was the third bug in this table. A re-review pass (§12.13, finding 1) found the bucket loop scanning every llm tuple including compared ones, which promoted two papers on the strength of a baseline they argued against — one of them a paper whose only llm tuple is compared, counted in bucket A for a model it never ran. The script now buckets on llmUsed() and throws if the buckets do not sum to the population, so the two filters cannot diverge again silently.

Every string in all four buckets is printed by the script (§12.9). Two calls a reasonable person would make differently: Mistral Large counts as B though it is a hosted API model, and FLAN-T5-XXL counts as B on a word-sized parameter count (PARAM_SIZE accepts xxl, large, mini). Both are visible in the printed list rather than buried in a share.

The three-bucket rule, verbatim:

const HOSTED_SNAPSHOT = /\d{4}-\d{2}-\d{2}|\b(0125|0613|0314|1106|0409|0806|2024\d{4}|2025\d{4})\b/i;
const OPEN_FAMILY = /llama|qwen|mistral|mixtral|gemma|vicuna|falcon|deepseek|glm|chatglm|\bphi-?\d|gpt-oss|\bt5\b|flan-t5|\bbert\b|roberta|deberta|\bbart\b|olmo|smollm/i;
const PARAM_SIZE = /\b\d+(\.\d+)?\s*b\b|:\d+(\.\d+)?b\b|\b(xxl|xl|large|small|mini|base)\b/i;

1. A bare phi matched “PhishLLM”, a prior detector rather than a model, putting that paper in the “names a family” bucket while the same script's resource fold put the identical string in its residue — two regexes in one script disagreeing about one string. Found by the figures reviewer (§12.10, finding 1).
2. The fix anchored every alternative, including gpt → \bgpt, and “ChatGPT” has no word boundary before “gpt”. Twelve ChatGPT papers dropped into “names no model at all”, taking that bucket from 10 to 24 and the family bucket from 154 to 140. Nothing in the figures looked wrong — 13.6% unnamed is plausible and the table still summed. It was caught only because the same change had added a print of every bucket's strings, and ChatGPT was sitting in the “no identifiable model” list.

The rule adopted from this: a bucket defined by a regex prints its members, or “generous” and “conservative” are words doing work a list should do. Every bucket and every fold in both new scripts now prints in full, and that is what caught findings 1 and 3 as well.

Every LLM tuple across the targets the three pages make a claim about, checked by a committed script rather than by hand: scripts/llm_currency_quotecheck.mjs, 30 tuples, output in §12.9. It reports six tiers rather than pass/fail, because a two-column PDF produces false FAILs and calling those “unlocatable” would overstate the problem while a bare pass/fail would understate it.

Re-run 2026-09-21 with the PDF fallback, and the six FAILs are now six RESCUEDs: nothing in this pass fails in both renderings. See Quote-check refresh, 2026-09-21 at the foot of this page. Four of the six go to a complete n-gram match in the PDF (36/36, 14/14, 21/21, 11/11) and one to 19/19; the sixth, PETS/2026/word-level-annotation…, goes 11/20 → 16/20, which clears the 80% tier. All six are privacy-policy or consent-notice tuples and no page on this site quotes any of them, so no published claim moves; every tuple behind a figure on the three edited pages passed at some tier already.

Two of the soft tiers were added after reading the failures, and that is worth admitting. The first run reported 15 fail; reading them showed the extraction drops inline citation markers, and that paper.cols.txt interleaves table captions into sentences — PETS/2026/disclosure-divergence… failed because the caption “Table 1: LLM backend comparison on 100-app validation.” lands inside the quoted sentence. Adding tiers for those is right, since the alternative is publishing “half the quotes are unlocatable”, which is false. But it is also a checker relaxed until it agreed with a hypothesis. The mitigation is that every soft pass is itemised with its reason and, for PASS-NGRAM, the broken n-grams, so each call is checkable by hand.

A latent bug found in review changed nothing, and is recorded anyway. 90 of the 5,869 paper.cols.txt files (1.5%) contain NUL bytes — 913 in the TGNN paper alone. They are not whitespace to \s, invisible in a terminal, and they make shell grep treat the file as binary and suppress every match silently (use grep -a). The checker did not strip them; it now does. Re-running gave 15 / 9 / 6 before and after, because no NUL happened to land inside one of these 30 quotes. On a different sample it would have been a published false FAIL. ¹⁾

6 of 30 (20%) is far above the corpus-wide 0.9% unlocatable rate on corpus. It is not a re-measurement: 30 tuples, non-random, all 2024–2026, weighted to PETS 2026 whose PDFs are the newest and worst-rendered. Read it as a reason to check quotes from the 2025–2026 slice specifically.

Written during the run, and then overtaken by the review. This section originally said “no new external source was added to the content page in this run”. That was true when written and false by the end: the external-currency pass (§12.10) put six primary sources onto website_classification — Google's Privacy Sandbox status page, Chrome Platform Status, the Blink Intent to Deprecate and Remove thread, the Chrome Version History API, and OpenAI's deprecations and pricing pages. Each replaced or repaired something the page had been carrying second-hand. That sequence is the reason this section is dated: a provenance section written before the review pass will be wrong about the review pass.

Two things rejected:

• Using 2025–2026 LLM papers from outside the seven venues as evidence for the currency claim. The page's method is that a claim is a claim about these seven venues; a hand-picked outside set would be the “we searched and found some papers” the dataset exists to replace. The corpus zeros are stated as corpus zeros and the outside search is reported separately, as a search.
• Fixing five duplicate-paper pairs in bibliography — the same paper under two citekeys, five times. Three were found by a DOI scan: ahmad2026_ipfp/ahmad2026_more (10.56553/popets-2026-0109), fouad2022my/fouad2022_cookie (10.56553/popets-2022-0063), bottger2025_regional/boettger2025_regional (10.56553/popets-2025-0063). Two more were found by a reviewer using normalised titles, and a DOI scan structurally cannot see them because those entries carry only url: lerner2016internet/lerner2016_internet and bouhoula2024automated/bouhoula2024_automated. That is the useful part: PETS and USENIX index records have no DOI at all, 100% of both venues, so a DOI-keyed collision scan is blind to exactly the two venues where bibgen.mjs is weakest, and any guard must key on normalised title too. Deduplicating means finding every page that cites each loser and repointing it — its own piece of work, now a task item. None of the five is cited by the three edited pages.

One entry was added: schwartz2025_llmcloudhunter (TheWebConf 2025, DOI 10.1145/3696410.3714798), from scripts/bibgen.mjs, needed so ip_classification can name the paper behind its single llm firing instead of describing it as “one paper, GPT-4o” next to a sentence about a different paper.

• The magnitude of the third-party-service drop. §12.4. The 2026-alone column is 20 papers. Needs CCS 2026 and IMC 2026.
• Whether the curated-database row means anything at all. §12.4. It is not significant and its contents are mis-filed. A stronger answer would need the twelve papers read and the enum corrected, not re-queried.
• Whether target == “other” hides more LLM website, script or tracker classification. Three earlier drafts of these pages said “reading 116 papers would settle it”. That was lazy and the reviewer was right to say so (§12.12, finding 10): other carries a targetDetail, stated on all 157 such tuples, and a keyword probe over it now runs in the report script. It returns 20 tuples and none is a website-topic, JavaScript, tracker or cookie classification — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zeros survive the other bucket at keyword recall, which is what the pages now say.

The probe's width decided that answer, and the first width was wrong. It used \bpage\b, which does not match the compound “webpage” — there is no word boundary between “web” and “page” — and it had no bare web at all, so it returned 16 rather than 20 and silently dropped “relevant person-specific webpage information” and “IOB presence and trustworthiness in web content”. Found in re-review (§12.13, finding 3). Both were then read and neither changes the conclusion, which is the only reason the published claim survived a probe that was under-recalling by 20%. A probe is not a read of 116 papers, its hits must be read rather than counted, and its regex is a load-bearing part of the claim.

• Whether the four GPT strings in the fold residue change a published share. §12.5a. Settled on 2026-09-21 and no longer open — the fold was extended as a rule and the three GPT rows re-derived; they do change one published share on this page and none anywhere else. See §12.14.
• Whether the six quote failures are extraction paraphrase or .cols rendering. One was read and was rendering. The other five were not, because no page quotes them.

Three scripts, all committed under scripts/. Their output below is unedited: every figure on the three content pages is in it, as is every bucket's membership, every fold's residue, every p value and the composition of the curated-database row. The bucket and residue listings on this page are not transcribed from these files — they are these files, which is why §12.12 finding 6 (a hand-copied list that had gone stale) cannot recur.

report_llm_currency-output.txt

========================================================================
POPULATION
========================================================================
Population                                            Papers  Share of 5,859
----------------------------------------------------  ------  --------------
corpus                                                5859    100%
papers with >=1 classification tuple                  4439    75.8%
>=1 tuple with method == "llm" (any usedOrMentioned)  177     3.0%
  ... of which USED or PRODUCED the LLM labels        175     3.0%
  ... of which only COMPARED against an LLM           2       0.0%
 
>>> The 177 figure published on design:website_classification is the "any"
    row. The used/produced row is the one to quote for adoption.
 
========================================================================
BY YEAR (two denominators)
========================================================================
Year  LLM papers (any)  LLM papers (used)  Corpus that year  Share of corpus  Papers classifying that year  Share of those
----  ----------------  -----------------  ----------------  ---------------  ----------------------------  --------------
2023  2                 2                  719               0.3%             503                           0.4%
2024  27                27                 690               3.9%             529                           5.1%
2025  77                76                 770               10.0%            607                           12.7%
2026  71                70                 415               17.1%            333                           21.3%
 
total: 177 papers (any), 175 (used/produced), of 5859.
2025 and 2026 are provisional venue-years — CCS/IMC 2026 not held, IEEE S&P
and WWW 2026 incompletely selected — so the last two rows are floors.
 
========================================================================
LIKE-FOR-LIKE GROWTH: is the 2026 share a composition artefact?
========================================================================
five-venue 2025: 58/590 = 9.8%
five-venue 2026: 71/415 = 17.1%
Venue    has a 2026 slice?  2025            2026            change
-------  -----------------  --------------  --------------  --------
CCS      NO                 12/101 = 11.9%  —               —
IEEE-SP  yes                10/117 = 8.5%   3/28 = 10.7%    +2.2 pp
IMC      NO                 7/79 = 8.9%     —               —
NDSS     yes                11/119 = 9.2%   19/163 = 11.7%  +2.4 pp
PETS     yes                4/67 = 6.0%     16/88 = 18.2%   +12.2 pp
USENIX   yes                18/224 = 8.0%   13/69 = 18.8%   +10.8 pp
WWW      yes                15/63 = 23.8%   20/67 = 29.9%   +6.0 pp
CCS + IMC 2025 (the two venues absent from 2026): 19/180 = 10.6%
>>> If that figure is BELOW the five-venue 2026 share, the absence of
    those two venues does not obviously inflate 2026.
 
========================================================================
BY VENUE (of the LLM papers, any)
========================================================================
Venue    LLM papers  Share of LLM papers  Papers in venue  Share of that venue
-------  ----------  -------------------  ---------------  -------------------
USENIX   43          24.3%                1410             3.0%
WWW      40          22.6%                843              4.7%
NDSS     30          16.9%                701              4.3%
PETS     21          11.9%                510              4.1%
CCS      18          10.2%                990              1.8%
IEEE-SP  14          7.9%                 767              1.8%
IMC      11          6.2%                 638              1.7%
 
========================================================================
BY TARGET — the quantity the three pages disagree about
========================================================================
Target                 LLM papers (any)  Share of 177  LLM papers (used)  Papers with ANY method at this target  LLM share of that target
---------------------  ----------------  ------------  -----------------  -------------------------------------  ------------------------
other                  116               65.5%         115                2592                                   4.4%
user-generated-text    32                18.1%         30                 419                                    7.2%
vulnerability          14                7.9%          14                 880                                    1.6%
privacy-policy         12                6.8%          12                 102                                    11.8%
mobile-app             6                 3.4%          6                  280                                    2.1%
website-category       5                 2.8%          5                  424                                    1.2%
domain                 3                 1.7%          3                  351                                    0.9%
network-traffic        2                 1.1%          2                  382                                    0.5%
consent-notice         2                 1.1%          2                  39                                     5.1%
ip-address             1                 0.6%          1                  295                                    0.3%
web-request            1                 0.6%          1                  258                                    0.4%
email-message          1                 0.6%          1                  54                                     1.9%
cookie                 1                 0.6%          1                  53                                     1.9%
dark-pattern           1                 0.6%          1                  13                                     7.7%
malware                0                 0.0%          0                  159                                    0.0%
sdk-or-library         0                 0.0%          0                  77                                     0.0%
javascript             0                 0.0%          0                  44                                     0.0%
fingerprinting-script  0                 0.0%          0                  31                                     0.0%
website-popularity     0                 0.0%          0                  15                                     0.0%
 
>>> The last column is the one that decides a currency claim on a
    per-target page. "LLM classification is now routine" is true of the
    corpus as a whole and false of most individual targets.
 
========================================================================
PER-TARGET DETAIL for the targets the three pages are about
========================================================================
 
--- target == "website-category"  (page: design:website_classification) ---
Paper                                                                                                       Year  usedOrMentioned  Validated against  Named model / resource
----------------------------------------------------------------------------------------------------------  ----  ---------------  -----------------  ---------------------------------------
USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w                2024  used             manual-validation  GPT-3.5-turbo-16k
WWW/2025/the-poorest-man-in-babylon-a-longitudinal-study-of-cryptocurrency-investment-sca                   2025  used             manual-validation  Llama3:70b + GPT-4 hybrid
NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets  2026  produced         manual-validation  GPT-4o with retrieval-augmented prompts
WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia                   2025  used             manual-validation  GPT-4o
USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram                  2025  used             manual-validation  GPT-4 API
 
--- target == "javascript"  (page: privacy:javascript) ---
  no paper in the corpus points an LLM at this target.
 
--- target == "fingerprinting-script"  (page: privacy:javascript) ---
  no paper in the corpus points an LLM at this target.
 
--- target == "web-request"  (page: privacy:javascript) ---
Paper                                                                                      Year  usedOrMentioned  Validated against  Named model / resource
-----------------------------------------------------------------------------------------  ----  ---------------  -----------------  ----------------------
WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  2026  used             held-out-test-set  Qwen3-Plus
 
--- target == "ip-address"  (page: design:ip_classification) ---
Paper                                                                                     Year  usedOrMentioned  Validated against  Named model / resource
----------------------------------------------------------------------------------------  ----  ---------------  -----------------  ----------------------
WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from  2025  used             manual-validation  GPT-4o
 
--- target == "domain"  (page: design:ip_classification) ---
Paper                                                                                          Year  usedOrMentioned  Validated against  Named model / resource
---------------------------------------------------------------------------------------------  ----  ---------------  -----------------  ----------------------
USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w   2024  used             manual-validation  GPT-3.5-turbo-16k
IEEE-SP/2025/a-big-step-forward-a-user-centric-examination-of-ios-app-privacy-report-and-enha  2025  used             manual-validation  GPT-4o
USENIX/2025/dns-flare-a-flush-reload-attack-on-dns-forwarders                                  2025  used             none-reported      ChatGPT
 
--- target == "cookie"  (page: privacy:cookies) ---
Paper                                               Year  usedOrMentioned  Validated against                    Named model / resource
--------------------------------------------------  ----  ---------------  -----------------------------------  -------------------------------------------------
WWW/2025/semantics-aware-cookie-purpose-compliance  2025  produced+used    held-out-test-set+manual-validation  GPT-3.5-turbo (custom fine-tuned model) | GPT-3.5
 
--- target == "privacy-policy"  (page: (referenced by design:ip_classification)) ---
Paper                                                                                         Year  usedOrMentioned    Validated against                             Named model / resource
--------------------------------------------------------------------------------------------  ----  -----------------  --------------------------------------------  -------------------------------------------------------------------------------------
IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots                               2024  used               manual-validation                             gpt-4-turbo-2024-04-09
IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems                   2025  produced           manual-validation                             custom LLM-based privacy policy analysis framework
PETS/2025/privacy-settings-of-third-party-libraries-in-android-apps-a-study-of-facebook-sd    2025  used               manual-validation                             ChatGPT
USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto  2025  used               held-out-test-set                             Llama-3.1-70B-Instruct
PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents               2026  used               comparison-to-other-method                    Claude, GPT-4o, Gemini, and DeepSeek (cross-LLM voting)
PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law    2026  used               held-out-test-set                             GPT-5
PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using     2026  used+produced      manual-validation+none-reported               GPT 4.1 | custom LLM passage-level classifier
PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p    2025  produced+compared  held-out-test-set                             GPT-3.5 Turbo (Prompt Engineered, fine-tuned) | GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo
PETS/2025/behavr-user-identification-based-on-vr-sensor-data                                  2025  used               comparison-to-other-method                    GPT-4
PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica    2026  used+compared      manual-validation                             GPT-4o | GPT-5 | Claude-4.5-Sonnet
PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t    2026  used               manual-validation                             GPT-4o-mini
PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s    2026  used+compared      held-out-test-set+comparison-to-other-method  LLaMA 3.1 8B Instruct | Mistral-7B
 
--- target == "consent-notice"  (page: (referenced by design:ip_classification)) ---
Paper                                                                                       Year  usedOrMentioned  Validated against  Named model / resource
------------------------------------------------------------------------------------------  ----  ---------------  -----------------  --------------------------------------------------
CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation                    2025  used             none-reported      Qwen2.5-7B
PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc  2026  used             none-reported      LlamaGuard, LlamaGuard2, LlamaGuard3, and MD-Judge
 
--- target == "dark-pattern"  (page: (none)) ---
Paper                                                                      Year  usedOrMentioned  Validated against           Named model / resource
-------------------------------------------------------------------------  ----  ---------------  --------------------------  ----------------------
PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits  2026  used             comparison-to-other-method  GPT-5
 
========================================================================
VALIDATION
========================================================================
148 of 177 (83.6%) validate the LLM's labels against something.
classification.validation   LLM papers naming it  Share of 177
--------------------------  --------------------  ------------
manual-validation           110                   62.1%
none-reported               35                    19.8%
comparison-to-other-method  31                    17.5%
held-out-test-set           18                    10.2%
not-applicable              4                     2.3%
cross-validation            1                     0.6%
(multi-valued: a paper with several LLM tuples can name several)
 
========================================================================
NAMED MODELS — free text, so a RANKING, not percentages
========================================================================
Model family                           Papers (of 177)  Share
-------------------------------------  ---------------  -----
GPT-4o                                 46               26.0%
GPT-4 (non-4o)                         44               24.9%
GPT-3.5 / GPT-3 / ChatGPT              29               16.4%
UNMAPPED                               18               10.2%
Llama family                           13               7.3%
Gemini / PaLM                          12               6.8%
Unnamed LLM                            11               6.2%
OpenAI reasoning / GPT-5 tier          10               5.6%
Qwen family                            8                4.5%
DeepSeek family                        7                4.0%
Mistral family                         4                2.3%
Other open-weight                      4                2.3%
Claude family                          3                1.7%
Encoder / seq2seq LM (not a chat LLM)  2                1.1%
 
--- UNMAPPED residue: 20 distinct strings, printed in full ---
      2x  Grok-3
      2x  local LLMs (custom prompts)
      1x  BLIP2
      1x  ChatGLM
      1x  custom structured prompts with fine-tuned LLMs
      1x  foundation LLMs
      1x  GLM-4.5
      1x  HtmlLLM-Detector
      1x  Kimi
      1x  LLaVA
      1x  LLaVA-NeXT-Video-7B
      1x  PhishLLM
      1x  Prompt Instruct
      1x  RFCGPT
      1x  text-unicorn (custom prompts)
      1x  tuned defense models
      1x  UGCG-GUARD (InstructBLIP with UGCG-CoT prompts)
      1x  Vertex AI text-bison
      1x  weighted multi-model ensemble (custom)
      1x  YouthSafe
 
========================================================================
DOES THE PAPER NAME A MODEL YOU COULD RESOLVE?
========================================================================
Population: the 175 papers that USED or PRODUCED LLM labels, not the 177 that mention one.
What the strongest thing the paper names is                       Papers (of 175)  Share
----------------------------------------------------------------  ---------------  -----
A a DATED HOSTED SNAPSHOT (gpt-4-turbo-2024-04-09)                10               5.7%
B an OPEN-WEIGHT CHECKPOINT with a size (Llama-3.1-70B-Instruct)  24               13.7%
  A or B — resolvable to an artefact at all                       34               19.4%
C a FAMILY with no version (GPT-4, ChatGPT, Mistral)              131              74.9%
D NO IDENTIFIABLE MODEL ("an LLM", "foundation LLMs")             10               5.7%
 
>>> B is the stronger kind of pin: an open-weight checkpoint stays
    resolvable, and a hosted snapshot only lasts as long as the vendor
    serves it. Reported separately for that reason, and because a single
    "is it pinned" bucket built from hosted snapshot strings alone
    classifies every open-weight paper as unpinned.
>>> PARAM_SIZE accepts a word size (xxl, 7b, mini), so `FLAN-T5-XXL` and
    `Mistral Large` count as B. That is a judgement call: they name a
    released artefact but not a revision. Every string is below.
 
--- bucket A — dated hosted snapshot: 10 distinct strings across 10 papers, printed in full ---
      2x  gpt-4-turbo-2024-04-09   [IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots] [IMC/2024/beyond-the-guidelines-assessing-metas-political-ad-moderation-in-the-eu]
      1x  ChatGPT (gpt-3.5-turbo-0613)   [USENIX/2024/llm-fuzzer-scaling-assessment-of-large-language-model-jailbreaks]
      1x  gpt-3.5-turbo-0125   [USENIX/2025/mind-the-inconspicuous-revealing-the-hidden-weakness-in-aligned-llms-refusal-bou]
      1x  gpt-3.5-turbo-0613   [NDSS/2025/generating-api-parameter-security-rules-with-llm-for-api-misuse-detection]
      1x  GPT-4 (gpt-4-0613)   [USENIX/2024/llm-fuzzer-scaling-assessment-of-large-language-model-jailbreaks]
      1x  GPT-4o (gpt-4o-2024-11-20)   [USENIX/2026/chameleon-channels-measuring-youtube-accounts-repurposed-for-deception-and-profi]
      1x  GPT-4o mini (gpt-4o-mini-2024-07-18)   [IEEE-SP/2025/gptracker-a-large-scale-measurement-of-misused-gpts]
      1x  gpt-4o-mini-2024-07-18   [NDSS/2026/beyond-jailbreak-unveiling-risks-in-llm-applications-arising-from-blurred-capability-boundaries]
      1x  o4-mini-2025-04-16 and gemini-2.5-pro   [USENIX/2026/sok-philter-uncovering-security-and-functional-gaps-in-ai-based-phishing-website]
      1x  OpenAI gpt-4o-2024-05-13   [USENIX/2025/mbfuzzer-a-multi-party-protocol-fuzzer-for-mqtt-brokers]
 
--- bucket B — open-weight checkpoint with a size: 25 distinct strings across 24 papers, printed in full ---
      2x  Llama-3.1-8B-Instruct
      1x  FLAN-T5-XXL
      1x  Gemma 3 27B
      1x  gpt-oss:20b
      1x  Llama 2 70B (custom five-step chain-of-thought)
      1x  Llama 3.1 70B (custom prompts)
      1x  LLaMA 3.1 8B Instruct
      1x  Llama 3.3 70B Instruct
      1x  Llama-3-8B
      1x  Llama-3.1-70B-Instruct
      1x  LLaMA-3.2-11B
      1x  LLaMA-3.2-3B
      1x  LLaMA-7B fine-tuned SmartInv
      1x  Llama3-8B
      1x  Llama3:70b + GPT-4 hybrid
      1x  Mistral-7B-Instruct-v0.2
      1x  Qwen2-72B-Instruct
      1x  Qwen2-7B-Instruct
      1x  Qwen2.5-7B
      1x  Qwen2.5-Coder-32B-Instruct
      1x  Qwen2.5-VL-72B-Instruct
      1x  Qwen3-VL-32B-Instruct
      1x  T5-11B
      1x  Vicuna-33b
      1x  Vicuna-7b
 
--- bucket C — family only, no version: 88 distinct strings across 131 papers, printed in full ---
     25x  GPT-4o
     14x  GPT-4
      6x  GPT-4.1
      6x  GPT-4o-mini
      5x  ChatGPT
      4x  GPT-3.5
      3x  ChatGPT-4
      3x  GPT-3.5-turbo
      3x  GPT-5
      2x  GPT-3.5 Turbo
      2x  GPT-4 Turbo
      1x  BLIP2
      1x  Chat-GPT 3.5 and 4
      1x  ChatGLM
      1x  ChatGPT (OpenAI TextCompletion API)
      1x  ChatGPT Team
      1x  ChatGPT Vision
      1x  ChatGPT-4.0
      1x  Claude Sonnet 3.5
      1x  Claude, GPT-4o, Gemini, and DeepSeek (cross-LLM voting)
      1x  DeepSeek
      1x  DeepSeek R1
      1x  DeepSeek-R1
      1x  DeepSeek-R1 alert-verification module
      1x  DeepSeek-V3
      1x  DeepSeek-V3.2-Exp
      1x  fine-tuned GPT-4o
      1x  fine-tuned GPT-4o with Chain-of-Thought prompting
      1x  Gemini 1.5 Pro and Gemini 2.0 Flash-Thinking
      1x  Gemini 2.0 Flash
      1x  Gemini 2.5 Flash
      1x  Gemini 2.5 pro
      1x  Gemini 2.5 Pro
      1x  Gemini 2.5-Flash
      1x  Gemini Ultra
      1x  Gemini-2.0-Flash
      1x  Gemini-2.5-Flash-Lite and GPT-4o-mini ensemble
      1x  Gemini-2.5-pro-preview-05-06
      1x  Gemini-3.1-Pro evaluator
      1x  GPT 4.1
      1x  GPT-3 curie
      1x  GPT-3.5 and GPT-4o mini
      1x  GPT-3.5 DaVinci
      1x  GPT-3.5 Turbo (Prompt Engineered, fine-tuned)
      1x  GPT-3.5-Turbo
      1x  GPT-3.5-turbo (custom fine-tuned model)
      1x  GPT-3.5-turbo-16k
      1x  GPT-3.5-turbo-instruct
      1x  GPT-4 (custom ontology classifier)
      1x  GPT-4 (fine-tuned)
      1x  GPT-4 API
      1x  GPT-4 Judge and Secondary Judge
      1x  GPT-4-turbo
      1x  GPT-4-Turbo
      1x  GPT-4.0
      1x  GPT-4.1 (custom extraction prompt)
      1x  GPT-4.1 (custom prompt)
      1x  GPT-4.1 with RAG knowledge base
      1x  GPT-4.1-mini
      1x  GPT-4.5
      1x  GPT-4o (Thematic-LM custom multi-agent system)
      1x  GPT-4o and alternative LLM backends
      1x  GPT-4o and DeepSeek-V3
      1x  GPT-4o mini
      1x  GPT-4o with retrieval-augmented prompts
      1x  GPT-4o, LLaMA 3, and Claude 3.5 majority vote
      1x  GPT-5 with chain-of-thought prompting
      1x  GPT-5-mini
      1x  GPT-5-mini with chain-of-thought prompting
      1x  GPT-o1
      1x  GPT, Claude, and Gemini evaluators
      1x  GPT3
      1x  GPT4o
      1x  Grok-3
      1x  Kimi
      1x  LLAMA2
      1x  LlamaGuard, LlamaGuard2, LlamaGuard3, and MD-Judge
      1x  LLaVA
      1x  majority ensemble of Gemini-3.1-Pro, Claude-Opus-4.6, and GPT-5.2
      1x  Mistral
      1x  OpenAI gpt-4
      1x  OpenAI o3-mini
      1x  Qwen-VL
      1x  Qwen3-Plus
      1x  RFCGPT
      1x  text-unicorn (custom prompts)
      1x  UGCG-GUARD (InstructBLIP with UGCG-CoT prompts)
      1x  Vertex AI text-bison
 
--- bucket D — no identifiable model: 10 distinct strings across 10 papers, printed in full ---
      2x  LLM judge (custom)   [WWW/2026/inference-cost-attacks-for-retrieval-augmented-large-language-models] [NDSS/2026/when-cache-poisoning-meets-llm-systems-semantic-cache-poisoning-and-its-countermeasures]
      1x  foundation LLMs   [IEEE-SP/2025/code-speaks-louder-exploring-security-and-privacy-relevant-regional-variations-i]
      1x  general-purpose LLM   [CCS/2025/needle-in-a-haystack-automated-and-scalable-vulnerability-hunting-in-the-windows]
      1x  HtmlLLM-Detector   [USENIX/2025/unsafe-llm-based-search-quantitative-analysis-and-mitigation-of-safety-risks-in]
      1x  large language models (LLMs)   [WWW/2026/opendigger-a-practical-framework-for-assessing-community-health-and-sustainabili]
      1x  LLM (unspecified)   [WWW/2024/experimental-security-analysis-of-sensitive-data-access-by-browser-extensions]
      1x  LLM-as-a-judge (custom)   [WWW/2026/arulecon-agentic-security-rule-conversion]
      1x  local LLMs (custom prompts)   [USENIX/2026/a-large-scale-study-of-personalized-phishing-using-large-language-models]
      1x  PhishLLM   [USENIX/2025/unsafe-llm-based-search-quantitative-analysis-and-mitigation-of-safety-risks-in]
      1x  weighted multi-model ensemble (custom)   [WWW/2026/webgeoinfer-structure-free-multi-stage-framework-for-geolocation-inference-from]
 
========================================================================
IS THE `other` TARGET HIDING ANY OF THIS?
========================================================================
157 llm tuples carry target == "other", across 116 papers.
157 of 157 (100.0%) state a targetDetail.
 
--- targetDetail matching the probe (websit|domain|\burl\b|homepage|page|\bweb\b|script|tracker|cookie|\bsdk\b|first.part|third.part|categor) — 20 tuples, all printed ---
    IMC/2023/in-the-room-where-it-happens-characterizing-local-communication-and-threats-in-s
        IoT device vendors and categories
    PETS/2024/connecting-the-dots-tracing-data-endpoints-in-iot-devices
        IoT device vendor and category
    USENIX/2024/guardians-of-the-galaxy-content-moderation-in-the-interplanetary-file-system
        image content descriptions
    CCS/2025/walking-the-last-mile-studying-decompiler-output-correction-in-practice
        semantic cluster descriptions
    CCS/2025/chekhovs-gun-uncovering-hidden-risks-in-macos-application-sandboxed-pid-domain-s
        Sensitive operations in PID-domain service functions
    IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems
        Action data descriptions assigned to 24 categories and 145 data types
    IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems
        Previously unclassified Action data descriptions
    IEEE-SP/2025/on-the-in-security-of-llm-app-stores
        description-instruction consistency
    IMC/2025/learning-as-to-organization-mappings-with-borges
        favicon and associated final-URL groups
    NDSS/2025/hidden-and-lost-control-on-security-design-risks-in-iot-user-facing-matter-controller
        UMCCI flaws and user-facing Matter control pages
    USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto
        personal information categories, purposes, and third-party recipients
    WWW/2025/50-shades-of-deceptive-patterns-a-unified-taxonomy-multimodal-detection-and-secu
        deceptive-pattern categories in UI images
    IEEE-SP/2026/parasites-in-the-toolchain-a-large-scale-analysis-of-attacks-on-the-mcp-ecosyste
        MCP server functional categories
    WWW/2026/adaptive-location-hierarchy-learning-for-long-tailed-mobility-prediction
        hierarchical mappings between location categories, activities, and needs
    WWW/2026/adaptive-location-hierarchy-learning-for-long-tailed-mobility-prediction
        hierarchical mappings between location categories, activities, and needs
    USENIX/2026/a-large-scale-study-of-personalized-phishing-using-large-language-models
        relevant person-specific webpage information
    NDSS/2026/indicator-of-benignity-an-industry-view-of-false-positive-in-malicious-domain-detection-and-its-mitigation
        IOB presence and trustworthiness in web content
    PETS/2026/operationalizing-the-motivated-intruder-a-codebook-guided-inference-framework-fo
        GDPR personal data, special-category data, and trade-secret sensitivity
    WWW/2026/opendigger-a-practical-framework-for-assessing-community-health-and-sustainabili
        repository technical domain
    WWW/2026/bridging-expert-reasoning-and-llm-detection-a-knowledge-driven-framework-for-mal
        threat-intelligence pages containing actionable malicious-code analysis
 
>>> Read the list, do not trust the count: the question is WHAT these are,
    not whether the probe fired. As of 2026-09-03 none of them is a
    website-topic, JavaScript, tracker or cookie classification, so the
    per-target zeros on privacy:javascript and design:ip_classification
    survive the `other` bucket at keyword recall. A probe is not a read of
    116 papers and cannot be reported as one.
 
========================================================================
CROSS-PAGE CLAIM CHECKS
========================================================================
Page claim                                                                                              What the corpus says
------------------------------------------------------------------------------------------------------  -------------------------------------------------------------------------------------
privacy:javascript: "the llm enum fires twice for a JS-classification task"                             javascript=0 (of 44 papers classifying JS), fingerprinting-script=0 (of 31)
privacy:requests: "exactly one paper classifies web requests with an LLM"                               web-request=1 (of 258 papers classifying requests)
design:ip_classification: "the llm method fires exactly once"                                           ip-address=1 (of 295 papers classifying IPs)
design:ip_classification: "unlike cookie and policy classification, where LLM methods are now routine"  cookie=1 of 53 (1.9%), privacy-policy=12 of 102 (11.8%) — "routine" holds for neither
design:website_classification: "five papers point an LLM at a website category"                         website-category=5 of 424 (1.2%)
 
Denominators in one line, for copying onto a page:
  corpus 2010-2026 = 5859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P)
  papers with >=1 classification tuple = 4439
  method == "llm", any usedOrMentioned = 177
  method == "llm", used or produced    = 175

llm_currency_quotecheck-output.txt

PART A — extraction evidence.quote vs paper.cols.txt
========================================================================
PASS      = present verbatim after collapsing whitespace.
PASS-ELID = the extractor inserted an elision ("..."); every fragment
            either side of it is present, which is the most the check
            can establish — the elided middle is unverified.
PASS-LOOSE= present after also dropping punctuation and folding the
            PDF's math-italic glyphs (F, M, lambda) to ASCII. A .cols
            rendering artefact, not a mismatch with the paper.
PASS-CITE = present once inline citation markers ("[49]") are stripped
            from both sides; the extractor drops them.
PASS-NGRAM= >=80% of the quote's word 5-grams occur in the text. The sentence
            is in the paper, but either the extraction reworded a word or
            two, or paper.cols.txt interleaved a float (table caption,
            running header) into it. The broken n-grams are itemised
            below so every such call is checkable by hand.
FAIL      = not present under any of the above.
 
RESCUED   IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots  [privacy-policy]  26/36 -> 36/36 PDF  "we design a set of task prompts for an AI chatbot to split scraped content into …"
PASS       USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w  [website-category]  "This prompt is fed into a language model using a chain-of-thought approach, enfo…"
PASS-NGRAM IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems  [privacy-policy]  10/10 5-grams  "We develop an LLM-based framework to check the consistency of data collection di…"
PASS       PETS/2025/privacy-settings-of-third-party-libraries-in-android-apps-a-study-of-facebook-sd  [privacy-policy]  "This allows for an automated approach that facilitates analysis of privacy polic…"
PASS       USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto  [privacy-policy]  "A coverage task assesses whether a privacy policy segment covers the required co…"
PASS       WWW/2025/the-poorest-man-in-babylon-a-longitudinal-study-of-cryptocurrency-investment-sca  [website-category]  "We manually categorized a random sample of 300 websites that had passed the cont…"
PASS-LOOSE PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents  [privacy-policy]  "For each privacy policy, we let each formalizer auto-formalize it into a structu…"
PASS-LOOSE NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets  [website-category]  "Based on the extracted evidence, CtPhishCapture assigns each webpage a phishing …"
PASS-CITE  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  [web-request]  "we employ the large language model Qwen3 as the underlying model Mλ for LLM-base…"
PASS-ELID  WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from  [ip-address]  "This component ... parses OSCTIs to identify and extract IoCs, notably IP addres…"
PASS-LOOSE WWW/2025/semantics-aware-cookie-purpose-compliance  [cookie]  "Subsequently, 88% of the labeled cookies are used as training data for the fine-…"
PASS       WWW/2025/semantics-aware-cookie-purpose-compliance  [cookie]  "Coover utilizes the OpenAI client.chat.completions.create() API, passing the spe…"
PASS       PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law  [privacy-policy]  "For each policy, we issue a single inference request to the model and require it…"
PASS       WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia  [website-category]  "To evaluate our classification methods, we manually annotated a sample of 500 we…"
RESCUED   PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using  [privacy-policy]  11/20 -> 16/20 PDF  "On a manually labelled sample of 340 randomly selected documents ... using GPT 4…"
PASS-LOOSE PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using  [privacy-policy]  "This LLM-based classifier predicts the set of labels Lp relevant to each passage"
RESCUED   CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation  [consent-notice]  9/14 -> 14/14 PDF  "we used a more efficient text-only model (Qwen2.5-7B) to detect the presence of …"
RESCUED   PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p  [privacy-policy]  13/21 -> 21/21 PDF  "We randomly reserved 70% of the manual annotations to constitute our training da…"
PASS       PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p  [privacy-policy]  "For the prompted non-fine-tuned LLMs, we used GPT-4, GPT-4 Turbo, and GPT-3.5 Tu…"
PASS       PETS/2025/behavr-user-identification-based-on-vr-sensor-data  [privacy-policy]  "We also use simple string matching to search for relevant content."
PASS       PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica  [privacy-policy]  "We use GPT-4o to fetch and segment the policy text, extracting structured data p…"
PASS       PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica  [privacy-policy]  "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…"
PASS       PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica  [privacy-policy]  "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…"
RESCUED   PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t  [privacy-policy]  15/19 -> 19/19 PDF  "A researcher manually verified whether each LLM classification matched the corre…"
PASS-NGRAM PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s  [privacy-policy]  19/20 5-grams  "the system outputs two sets of data types, C data (collected) and S data (shared…"
RESCUED   PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s  [privacy-policy]  7/11 -> 11/11 PDF  "both models were evaluated using the same preprocessing pipeline, chunking strat…"
PASS       USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram  [website-category]  "GPT-4 added four new categories, bringing the total number of categories to 19. …"
PASS       USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram  [website-category]  "We evaluated 100 randomly selected cases and found GPT-4's predictions to be acc…"
PASS       PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits  [dark-pattern]  "The agent also logs any failures or issues encountered during automation, such a…"
PASS-ELID  PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc  [consent-notice]  "Each model independently evaluated these conversations by predicting moderation …"
 
Soft passes, itemised:
  IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems [privacy-policy] n-gram: 10/10 word 5-grams present, broken at: 
  PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents [privacy-policy] punctuation/glyph only
  NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets [website-category] punctuation/glyph only
  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere [web-request] inline citation marker dropped by the extractor
  WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from [ip-address] elided into 2 fragments
  WWW/2025/semantics-aware-cookie-purpose-compliance [cookie] punctuation/glyph only
  PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] punctuation/glyph only
  PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] n-gram: 19/20 word 5-grams present, broken at: the system outputs two sets
  PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc [consent-notice] elided into 2 fragments
 
PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext.
 
 
PART B — figures the pages publish, located in the full text
========================================================================
PASS  WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia  <<an accuracy of 92%, comparable to commercial website classification services>>
      ctx: … a sample of 500 websites from the Tranco list, categorizing them into "shopping" and "non-shopping." GPT-4o, when prompted with URLs and screenshots, achieved an accuracy of 92%, comparable to commercial website classification services [49] (see Appendix D for details). Therefore, we use this configuration throughout our work. We subsequently crawl the shopping websites to collect terms and c…
 
PASS  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  <<the LLM-driven annotation method attains an>>
      ctx: … of 92.24% in detecting connected requests and 84.49% for isolated requests, outperforming state-ofthe-art graph-based tracking detection methods. Furthermore, the LLM-driven annotation method attains an 𝐹 1 -Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods, underscoring the critical role of LLMs…
 
PASS  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  <<-Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods>>
      ctx: ….49% for isolated requests, outperforming state-ofthe-art graph-based tracking detection methods. Furthermore, the LLM-driven annotation method attains an 𝐹 1 -Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods, underscoring the critical role of LLMs in generating large-scale, high-quality labels. Furthermore, our TGNN framework revealed several concerning findings in…
 
PASS  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  <<we employ the large language model Qwen3>>
      ctx: … to this component is the contextual information C of edge 𝑒 (𝑢,𝑣,𝑖 ) , and the output is its tracking attribute T (𝑒 (𝑢,𝑣,𝑖 ) ) ∈ {0, 1}. To this end, we employ the large language model Qwen3 [49] as the underlying model 𝑀𝜆 for LLM-based labeling. WWW '26, April 13-17, 2026, Dubai, United Arab Emirates. Design Philosophy. To achieve this goal, the…
 
PASS  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  <<at least 16.74% of websites engage in pixel tracking via major third-party infrastructures>>
      ctx: …tially outperforms existing baselines, while the LLM-based annotation achieves accuracy comparable to expert curation. Our large-scale measurement reveals that at least 16.74% of websites engage in pixel tracking via major third-party infrastructures, establishing cross-domain tracking as a pervasive practice in the ∗ Also affiliated with School of Cyber Security, University of Chinese Academy of Sciences, …
 
PASS  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere  <<formulates pixel tracking detection as an edge classification task>>
      ctx: …ften fail to capture the cross-site associations that enable pixel tracking to evade detection. To address this limitation, we introduce TGNN, a framework that formulates pixel tracking detection as an edge classification task on a Tracking Directed Graph (TDG), which models thirdparty associations across websites. TGNN encodes HTTP traffic into structured quadruples and learns both …
 
PASS  IMC/2025/learning-as-to-organization-mappings-with-borges  <<GPT-4o-mini>>
      ctx: … the Information Extraction (IE) pro- We implemented this approach in Borges, utilizing OpenAI's cess by applying a dropout filter to enhance model accuracy by GPT-4o-mini [40] with a temperature set to 0 and a Top P probaonly considering text fields - either notes or aka entries - con- bility mass of 1. This setup ensures the mo…
 
PASS  WWW/2025/semantics-aware-cookie-purpose-compliance  <<fine-tun>>
      ctx: …purpose compared to other features such as expiration time. Coover decomposes the cookie value into primitive segments representing minimal semantic units, and fine-tunes a GPT-3.5 model to automatically interpret their value-inferred semantics. Based on the interpretation, it classifies cookies into four GDPR-defined purposes…

report_website_classification-output.txt

========================================================================
POPULATION
========================================================================
corpus                                              5859
papers with >=1 classification tuple                4439
papers with >=1 target=website-category tuple       424
  ... of which used or produced it                  424
  ... only compared/mentioned it                    0
 
--- What "website-category" actually means, folded by task ---
(papers, of the 424 above; multi-valued, so shares do not sum to 100%)
Task                       Papers  Share
-------------------------  ------  -----
topic-or-industry          330     77.8%
abuse-verdict              63      14.9%
website-fingerprinting     36      8.5%
hosting-infrastructure     8       1.9%
page-type                  7       1.7%
technology-fingerprinting  6       1.4%
third-party-role           5       1.2%
 
>>> PAGE POPULATION: 330 papers that assign a topic/industry/sector label
    to a website or domain, out of 5859 in the corpus (5.6%).
    Excluded: 94 papers whose only website-category tuple is a
    website-fingerprinting attack, an abuse verdict, a technology
    fingerprint, a page-type label, a hosting label or a third-party role.
 
========================================================================
WHERE AND WHEN
========================================================================
--- Venue (of 330) ---
Venue    Papers  Share of population  Share of that venue
-------  ------  -------------------  -------------------
WWW      71      21.5%                8.4%
USENIX   58      17.6%                4.1%
IMC      50      15.2%                7.8%
CCS      48      14.5%                4.8%
PETS     43      13.0%                8.4%
NDSS     35      10.6%                5.0%
IEEE-SP  25      7.6%                 3.3%
 
--- Over time (denominator = all corpus papers in the bucket) ---
Years       Corpus  Categorised websites  Share
----------  ------  --------------------  -----
2010–2013   511     28                    5.5%
2014–2017   769     56                    7.3%
2018–2021   1439    92                    6.4%
2022–2024   1955    89                    4.6%
2025–2026*  1185    65                    5.5%
 
========================================================================
METHOD (enum — publishable as a rough share; run-to-run stability 58%)
========================================================================
Method               Papers  Share
-------------------  ------  -----
third-party-service  131     39.7%
manual-labelling     68      20.6%
heuristic-rules      59      17.9%
supervised-ml        46      13.9%
curated-database     46      13.9%
unsupervised-ml      14      4.2%
regex-or-signature   9       2.7%
other                7       2.1%
graph-analysis       3       0.9%
static-analysis      2       0.6%
llm                  2       0.6%
dynamic-analysis     1       0.3%
blocklist            1       0.3%
(of 330; multi-valued)
 
--- Method over time, as a share of that bucket's categorising papers ---
Method               2010–2013  2014–2017   2018–2021   2022–2024   2025–2026*
-------------------  ---------  ----------  ----------  ----------  ----------
third-party-service  7 (25.0%)  19 (33.9%)  40 (43.5%)  44 (49.4%)  21 (32.3%)
supervised-ml        4 (14.3%)  6 (10.7%)   15 (16.3%)  13 (14.6%)  8 (12.3%)
manual-labelling     5 (17.9%)  15 (26.8%)  19 (20.7%)  18 (20.2%)  11 (16.9%)
heuristic-rules      8 (28.6%)  7 (12.5%)   14 (15.2%)  15 (16.9%)  15 (23.1%)
curated-database     4 (14.3%)  12 (21.4%)  10 (10.9%)  8 (9.0%)    12 (18.5%)
llm                  0 (0.0%)   0 (0.0%)    0 (0.0%)    0 (0.0%)    2 (3.1%)
 
--- Control 1: the 2025–2026 window split by year ---
Method               2022–2024   2025        2026*
-------------------  ----------  ----------  ---------
third-party-service  44 (49.4%)  17 (37.8%)  4 (20.0%)
supervised-ml        13 (14.6%)  3 (6.7%)    5 (25.0%)
manual-labelling     18 (20.2%)  9 (20.0%)   2 (10.0%)
heuristic-rules      15 (16.9%)  11 (24.4%)  4 (20.0%)
curated-database     8 (9.0%)    8 (17.8%)   4 (20.0%)
llm                  0 (0.0%)    2 (4.4%)    0 (0.0%)
bucket sizes: 2022–2024=89  2025=45  2026*=20
 
--- Control 2: both windows restricted to venues that have a 2026 slice ---
venues with any 2026 paper: IEEE-SP, NDSS, PETS, USENIX, WWW
venues with none (2026 not held / not selected): CCS, IMC
Method               2022–2024   2025–2026*
-------------------  ----------  ----------
third-party-service  34 (54.0%)  16 (28.6%)
supervised-ml        11 (17.5%)  7 (12.5%)
manual-labelling     13 (20.6%)  11 (19.6%)
heuristic-rules      8 (12.7%)   11 (19.6%)
curated-database     6 (9.5%)    12 (21.4%)
llm                  0 (0.0%)    2 (3.6%)
bucket sizes: 2022–2024=63  2025–2026*=56
 
--- Fisher's exact test on each control (two-sided) ---
Cut                                         Method               Recent         Baseline       change (pp)  p (two-sided)  Read as
------------------------------------------  -------------------  -------------  -------------  -----------  -------------  ------------------------------
2025 alone vs 2022–2024                     third-party-service  17/45 (37.8%)  44/89 (49.4%)  -11.7        0.2704         NOT SUPPORTED
2025 alone vs 2022–2024                     curated-database     8/45 (17.8%)   8/89 (9.0%)    +8.8         0.1630         NOT SUPPORTED
2026 alone vs 2022–2024                     third-party-service  4/20 (20.0%)   44/89 (49.4%)  -29.4        0.0236         supported
2026 alone vs 2022–2024                     curated-database     4/20 (20.0%)   8/89 (9.0%)    +11.0        0.2274         NOT SUPPORTED
2025–2026 vs 2022–2024, all venues          third-party-service  21/65 (32.3%)  44/89 (49.4%)  -17.1        0.0471         supported
2025–2026 vs 2022–2024, all venues          curated-database     12/65 (18.5%)  8/89 (9.0%)    +9.5         0.0947         NOT SUPPORTED (0.05 < p < 0.1)
2025–2026 vs 2022–2024, five-venue control  third-party-service  16/56 (28.6%)  34/63 (54.0%)  -25.4        0.0057         supported
2025–2026 vs 2022–2024, five-venue control  curated-database     12/56 (21.4%)  6/63 (9.5%)    +11.9        0.0792         NOT SUPPORTED (0.05 < p < 0.1)
>>> Read this before quoting either movement. The 2025-alone control is
    not evidence at n=45, and curated-database does not clear p<0.05 on
    any cut. Only the third-party-service drop is statistically supported.
 
--- what the `curated-database` row actually names, by window ---
    2022–2024   CCS/2022/understanding-security-issues-in-the-nft-ecosystem DappRadar
    2022–2024   CCS/2022/do-opt-outs-really-opt-me-out                     pornhosts blocklist
    2022–2024   PETS/2022/who-knows-i-like-jelly-beans-an-investigation-into-search-privacy WebPulse
    2022–2024   USENIX/2022/an-audit-of-facebooks-political-ad-policy-enforcement external political, government, media, and issue-page sources
    2022–2024   USENIX/2022/many-roads-lead-to-rome-how-packet-headers-influence-dns-censorship-measurement Citizen Lab Block List
    2022–2024   USENIX/2022/pool-inference-attacks-on-local-differential-privacy-quantifying-the-privacy-gua AllSides Media Bias Chart
    2022–2024   WWW/2023/who-funds-misinformation-a-systematic-analysis-of-the-ad-related-profit-routines MediaBias/FactCheck
    2022–2024   IEEE-SP/2022/investigating-influencer-vpn-ads-on-youtube   YouTube category labels
    2025–2026*  IEEE-SP/2025/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on Tracker Radar Entity List
    2025–2026*  USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi Cloudflare Radar
    2025–2026*  PETS/2025/empirically-measuring-data-localization-in-the-eu SimilarWeb
    2025–2026*  USENIX/2025/demystifying-the-in-security-of-qr-code-based-login-in-real-world-deployments Symantec SiteReview
    2025–2026*  WWW/2025/the-agenda-setting-function-of-social-media       AllSides
    2025–2026*  WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e MediaBias/FactCheck
    2025–2026*  NDSS/2026/dom-xss-detection-via-webpage-interaction-fuzzing-and-url-component-synthesis IAB taxonomy
    2025–2026*  USENIX/2026/patch-guided-vulnerability-detection-extracting-java-api-security-rules-via-atta predefined source rules (custom)
    2025–2026*  WWW/2025/before-after-the-effect-of-eus-2022-code-of-practice-on-disinformation MediaBias/FactCheck | Science Feedback
    2025–2026*  WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere Homepage2Vec
    2025–2026*  IEEE-SP/2025/the-file-that-contained-the-keys-has-been-removed-an-empirical-analysis-of-secre NAICSlite
    2025–2026*  NDSS/2026/revealing-the-secret-power-how-algorithms-can-influence-content-visibility-on-twitter-x Media Bias/Fact Check (MBFC)
>>> If this row is mostly commercial vendors and media-bias raters rather
    than open directories, "curated databases returned" is a claim about
    an enum, not about the field returning to open label sources.
 
--- 2022–2024 third-party-service share by venue, within the population ---
Venue    third-party-service  population 2022–2024  Share  Has a 2026 slice?
-------  -------------------  --------------------  -----  -----------------
CCS      1                    12                    8.3%   NO
IEEE-SP  5                    9                     55.6%  yes
IMC      9                    14                    64.3%  NO
NDSS     1                    5                     20.0%  yes
PETS     10                   16                    62.5%  yes
USENIX   10                   21                    47.6%  yes
WWW      8                    12                    66.7%  yes
 
--- Where the population sits, by venue and year (the mix itself) ---
Venue    pop 2022–2024  pop 2025  pop 2026  corpus 2022–2024  corpus 2025  corpus 2026
-------  -------------  --------  --------  ----------------  -----------  -----------
CCS      12             4         0         282               101          0
IEEE-SP  9              4         0         303               117          28
IMC      14             5         0         190               79           0
NDSS     5              4         9         167               119          163
PETS     16             9         5         170               67           88
USENIX   21             10        3         559               224          69
WWW      12             9         3         284               63           67
 
--- classification.method == "llm" anywhere in the corpus, by year ---
Year  Papers using an LLM to classify  Corpus that year  Share
----  -------------------------------  ----------------  -----
2023  2                                719               0.3%
2024  27                               690               3.9%
2025  77                               770               10.0%
2026  71                               415               17.1%
total: 177 papers of 5859. 2025 and 2026 are provisional venue-years — CCS/IMC 2026 not held, IEEE S&P/WWW 2026 incompletely selected — so the last two rows are floors, not ceilings.
 
--- What those papers point an LLM at ---
Target               Papers  Share of 177
-------------------  ------  ------------
other                116     65.5%
user-generated-text  32      18.1%
vulnerability        14      7.9%
privacy-policy       12      6.8%
mobile-app           6       3.4%
website-category     5       2.8%
domain               3       1.7%
network-traffic      2       1.1%
consent-notice       2       1.1%
web-request          1       0.6%
ip-address           1       0.6%
cookie               1       0.6%
email-message        1       0.6%
dark-pattern         1       0.6%
148 of 177 (83.6%) validate the LLM's labels against something.
2 of 177 use an LLM to assign a website a TOPIC category.
 
========================================================================
WHICH NAMED RESOURCE (free text — folded, reported as a ranking)
========================================================================
--- By family (of 330 papers; multi-valued) ---
Family                                          Papers  Share
----------------------------------------------  ------  -----
Hand-built rules / keywords                     87      26.4%
Commercial security / content-filtering vendor  65      19.7%
Generic ML classifier, no named service         53      16.1%
Human labelling                                 41      12.4%
Commercial marketing categorisation             36      10.9%
Cloud / platform categorisation API             25      7.6%
Human-curated directory or test list            21      6.4%
Reputation aggregator                           14      4.2%
Pretrained language model, fine-tuned           6       1.8%
Large language model                            2       0.6%
 
--- Named third-party services and datasets only, ranked ---
Service                              Family              Papers  Spellings folded
-----------------------------------  ------------------  ------  ----------------
McAfee                               security-filtering  23      18
Symantec / Blue Coat                 security-filtering  14      10
Alexa                                marketing           12      8
FortiGuard                           security-filtering  12      7
Media bias raters (AllSides, MB/FC)  curated-directory   10      9
Cloudflare domain categories         platform-api        10      6
Webshrinker                          marketing           9       2
DMOZ / Open Directory Project        curated-directory   8       6
Google Cloud Natural Language        platform-api        8       6
SimilarWeb                           marketing           7       2
VirusTotal                           aggregator          7       3
Google Ads keyword/topic tools       marketing           6       5
Trend Micro                          security-filtering  4       3
Cisco (OpenDNS / IronPort)           security-filtering  3       3
Cyren                                security-filtering  3       3
Citizen Lab test list                curated-directory   3       3
Forcepoint / Websense                security-filtering  3       2
Google Topics API                    platform-api        3       3
Google Safe Browsing                 aggregator          2       2
Web of Trust                         aggregator          2       2
IBM Watson                           platform-api        2       2
WhoisXMLAPI                          aggregator          2       2
Klazify                              marketing           2       1
SafeDNS                              security-filtering  2       2
Bing classifier                      platform-api        1       1
Boson NLP                            platform-api        1       1
HerdProtect                          aggregator          1       1
Juniper                              security-filtering  1       1
Website Categorify                   marketing           1       1
YouTube Data API                     platform-api        1       1
Netstar                              security-filtering  1       1
155 of 330 papers (47.0%) name at least one third-party service or dataset.
 
--- Everything else, ranked ---
Resource                                         Family         Papers
-----------------------------------------------  -------------  ------
Unnamed custom method                            hand-built     58
Manual labelling by the authors or hired coders  manual         41
Keyword / regex rules (custom)                   hand-built     32
Random forest                                    generic-ml     13
Clustering (k-means, DBSCAN, …)                  generic-ml     10
SVM                                              generic-ml     9
Neural network (CNN/LSTM/MLP)                    generic-ml     6
LDA / topic model                                generic-ml     6
Logistic / linear regression                     generic-ml     5
k-NN                                             generic-ml     3
BERT                                             pretrained-lm  3
Hidden Markov / Bayesian network                 generic-ml     2
Naive Bayes                                      generic-ml     2
LLM (GPT family)                                 llm            2
GloVe                                            pretrained-lm  1
Semi-supervised (co-/self-training, PU)          generic-ml     1
BART / zero-shot NLI classifier                  pretrained-lm  1
Homepage2Vec                                     pretrained-lm  1
 
--- The spellings each canonical name absorbed (the reason folding matters) ---
Unnamed custom method (48):
    CodeQL query suite (custom)
    DNS Scanner (custom)
    DOMetric (custom)
    IAB taxonomy mapping (custom)
    Paraphrase-Multilingual-MPNet-base-v2 classifier (custom)
    TextCNN (custom)
    Word2Vec (custom)
    custom
    custom 154-class tag mapping
    custom 63-pattern list
    custom FPR/TPR platform-identification process
    custom HTTPS-response classification
    custom IMSS identifier
    custom IPv6-readiness classification
    custom NSKeyword Identifier
    custom PASSKEYS-RADAR detectors
    custom Prebid.js detector
    custom STEM/humanities classification
    custom SnorCall labeling functions
    custom WordPress-version and HTML-source checks
    custom analyst labeling
    custom campaign infection-vector classification
    custom category coding
    custom content whitelist
    custom credential-taking-intent module
    custom dual-stack completeness classification
    custom fake AV detector
    custom gray-website category scheme
    custom image classifier
    custom inference procedure
    custom page-category scheme
    custom policy-based categorization
    custom privacy-score thresholds
    custom purpose classification
    custom sector categorization
    custom seven-category scheme
    custom social-profile matching
    custom static-file hash knowledge base
    custom suspicious-content classifier
    custom topic categories
    custom web crawler
    custom webpage curation checks
    custom word-based classifier
    handcrafted classifier (custom)
    priority-based mapping (custom)
    roberta-large (custom fine-tuned model)
    semantic analyzer (custom)
    strict SAT-vendor criteria (custom)
Keyword / regex rules (custom) (33):
    90 manually crafted application-identification signatures
    Rule-Based Detector (custom)
    Unicode-based language heuristic (custom)
    custom 82-keyword content filter
    custom HTML heuristics
    custom URL parsing
    custom government-hostname regular expression
    custom heuristic
    custom heuristic rules
    custom heuristics
    custom keyword matching
    custom keyword searches
    custom keyword-matching heuristics
    custom keyword-stuffing detector
    custom page-classification priority rules
    custom provider keywords and DOM signatures
    custom redirect-chain rules
    custom regexes
    custom regular expression
    custom regular expressions and keyword lists
    custom regular expressions and known SSO API endpoints
    custom regular-expression keyword rules
    custom response-header signatures
    custom store-detection heuristics
    custom two-filter heuristic
    custom www-label rule
    keyword matching
    manually curated keyword set (custom)
    predefined source rules (custom)
    regular expressions (custom)
    simple DNS name heuristic
    simple handcrafted URL-based mapping rules
    two-phase content-grouping heuristic (custom)
Manual labelling by the authors or hired coders (32):
    BuzzFeed News journalists and fact checkers
    Google reviewer annotations
    Mechanical Turk
    author labeling
    custom Facebook-ad codebook
    custom manual categorization
    custom manual classification
    custom manual determination
    custom manual inspection
    custom manual product identification
    custom manual sanity-check
    custom manual service profiling
    custom manual verification
    custom relevance codebook
    custom website-category codebook
    four privacy-field experts
    graduate-level annotators
    graduate-student annotators
    manual CMS investigation (custom)
    manual categorization by authors
    manual categorization by researchers
    manual company classification (custom)
    manual inspection (custom)
    manual review by authors
    manual review of third-party marketing materials
    one-author manual verification
    researcher manual categorization
    researcher-created 15-category codebook
    three-person manual verification
    two human coders
    two reviewers
    two-author manual review
McAfee (18):
    McAfee
    McAfee Real-Time Database
    McAfee SiteAdvisor
    McAfee SiteLookup
    McAfee SmartFilter
    McAfee SmartFilter Internet Database
    McAfee URL Ticketing System
    McAfee URL categorization service
    McAfee categorization service
    McAfee service
    McAfee's Threat Intelligence
    McAfee's TrustedSource
    McAfee's URL Ticketing System
    McAfee's URL Ticketing service
    McAfee's URL categorization service
    McAfee's domain categorization API
    McAfee's website categorization tool
    Trellix TrustedSource
Clustering (k-means, DBSCAN, …) (10):
    Affinity Propagation
    Algorithm 1 (custom clustering algorithm)
    DBScan
    URL keyword clustering (custom)
    content clustering tool (custom q-gram similarity)
    custom manual clustering
    diffusion wavelet clustering (custom)
    k-means clustering (custom feature pipeline)
    perceptual hash and simhash (custom pipeline)
    perceptual hash function
Symantec / Blue Coat (10):
    Blue Coat Systems genre database
    Symantec RuleSpace
    Symantec Rulespace
    Symantec SiteReview
    Symantec Sitereview
    Symantec WebPulse
    Symantec WebPulse Site Review
    Symantec risk level calculator
    WebPulse
    WebPulse Site Review
Media bias raters (AllSides, MB/FC) (9):
    AllSides
    AllSides Media Bias Chart
    AllSides and Media Bias/Fact Check
    Allsides
    Media Bias Fact Check (MBFC)
    Media Bias/Fact Check
    Media Bias/Fact Check (MBFC)
    MediaBias/FactCheck
    Science Feedback
SVM (8):
    Linear SVM (custom)
    SVM (custom)
    SVM (custom, scikit-learn)
    SVM classifier (custom)
    Support Vector Machines (RBF; custom)
    eight custom classifiers: KNN, SVM, Naive Bayes, and decision tree
    linear SVM (Liblinear)
    one-class SVM (custom)
 
--- RESIDUE: 33 distinct strings the fold did not classify ---
  1  Taobao main-business categories
  1  search-redirection attack list
  1  Rakuten Popshops
  1  SNET pillar portals
  1  Googlebot
  1  Censys
  1  AppBrain API
  1  HBDetector
  1  Zeileis et al.'s structural change detection algorithm
  1  Van Wegberg et al. [24] classification
  1  DAppTotal
  1  DappRadar
  1  pornhosts blocklist
  1  external political, government, media, and issue-page sources
  1  multiple domain classification services
  1  public classification service from the security vendor
  1  Google ML model implemented in Chrome
  1  ORES topic routing
  1  default target categories and direct visits
  1  Tracker Radar Entity List
  1  Arkham Intel
  1  ccTLD/WHOIS country criterion
  1  NextDNS Piracy Blocklist
  1  IAB taxonomy
  1  ScamAdviser API
  1  multi-feature probabilistic filtering algorithm
  1  vDF and vRF
  1  urlblacklist.com
  1  NudeNet
  1  NAICSlite
  1  DNS-based hosting-provider identification
  1  open-source topic classifier
  1  YouTube category labels
 
--- Named categorisation services used by the NON-topic tasks ---
  abuse-verdict: 13 papers
      9  VirusTotal
      3  Google Safe Browsing
      1  McAfee
      1  URLScan
  third-party-role: 1 papers
      1  Google Safe Browsing
 
========================================================================
WHAT THESE PAPERS DO AND DO NOT REPORT
========================================================================
Question                                Papers  Of   Share
--------------------------------------  ------  ---  -----
names the taxonomy it classified into   248     330  75.2%
names a ground-truth source             169     330  51.2%
reports any validation of the labels    123     330  37.3%
explicitly reports no validation        98      330  29.7%
SENTINEL: validation is not-applicable  109     330  33.0%
check: 123 + 98 + 109 = 330 (should be 330)
 
--- Cross-population comparisons quoted on the page ---
groundTruthSource stated, ALL classifying papers: 3234/4439 = 72.9%
Vallina et al. (IMC 2020, "Mis-shapes, Mistakes, Misfits") present in the extraction: false  — it is in data/corpus2/.meta/IMC-2020.json with a real DOI but has no data/fulltext entry, so the page's central citation is not in the population the page measures.
IEEE S&P is now fully retrieved (780 of 780 selected papers, commit 8a6b843 2026-08-11). Its share of this population went 3 papers (1.2%) -> 25 (7.6%) as a result: the old 43%-retrieval caveat WAS the explanation for its low share, and is now resolved.
Free-text names agree ~20% run-to-run and classification.method ~58% — data/extract/README.md, which is why service names are reported as folded rankings and method as a rough share.
 
--- How the labels were validated (of the papers that validated) ---
Validation                  Papers  Share of population
--------------------------  ------  -------------------
manual-validation           92      27.9%
cross-validation            21      6.4%
held-out-test-set           13      3.9%
comparison-to-other-method  10      3.0%
 
Papers naming >=2 distinct third-party categorisation services: 9 of 155 that name any (5.8%).
    WWW/2013/two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure
    WWW/2014/adscape-harvesting-and-analyzing-online-display-ads
    WWW/2014/from-devices-to-people-attribution-of-search-activity-in-multi-user-settings
    CCS/2014/your-online-interests-pwned-a-pollution-attack-against-targeted-advertising
    NDSS/2015/seven-months-worth-of-mistakes-a-longitudinal-study-of-typosquatting-abuse
    CCS/2020/censored-planet-an-internet-wide-longitudinal-censorship-observatory
    WWW/2023/who-funds-misinformation-a-systematic-analysis-of-the-ad-related-profit-routines
    PETS/2024/interest-disclosing-mechanisms-for-advertising-are-privacy-exposing-not-preservi
    PETS/2024/what-does-it-mean-to-be-creepy-responses-to-visualizations-of-personal-browsing
 
--- Named taxonomies (free text, coarse fold) ---
Taxonomy                                    Papers  Share of the 248 that name one
------------------------------------------  ------  ------------------------------
a bespoke scheme, class count not given     187     75.4%
a vendor's own category list                43      17.3%
IAB                                         9       3.6%
DMOZ / ODP / Curlie                         7       2.8%
a bespoke scheme with a stated class count  7       2.8%
Google AdWords / Topics                     4       1.6%
a standard industry code (NAICS/SIC/NACE)   1       0.4%

Recorded. Four reviewers, all told explicitly that the summary they were given might not be exhaustive, all handed the three page texts, the three report scripts and their unedited outputs. The three focused passes ran in parallel first; their findings were applied and the pages re-saved before the generic pass. Rejections are logged as well as fixes — they are the only record of whether a reviewer earns its slot.

Re-ran all five scripts against the live extraction and diffed byte-for-byte against the committed outputs — identical, so the committed outputs were not stale. Confirmed every headline figure, the whole per-target table, the venue counts, all six drift-control cells and the five-venue list, the Borges/LLMCloudHunter distinction, the two llm firings in the JavaScript population, and the entire non-LLM remainder of all three pages. Four findings:

The reviewer also resolved an apparent cross-page contradiction rather than reporting it: requests calls 258 the “raw” web-request count where the truly raw figure is 262. “Raw” there means before that page's topic-relevance exclusion, not before the usedOrMentioned filter, and both pages consistently publish the used/produced 258. No defect.

The over-correction this fix caused, and how it was caught, is in §12.5c.

All {[key]} markers on the three pages resolve; schwartz2025_llmcloudhunter verified against Crossref (authors, title, venue, year, DOI) and against the paper's own ACM reference line. Every load-bearing quote verified in the source, including that [2Tsai, Elisa; Mangaokar, Neal; Zheng, Boyuan; Zheng, Haizhong; Prakash, Atul (2025): "Harmful Terms and Where to Find Them: Measuring and Modeling Unfavorable Financial Terms and Conditions in Shopping Websites at Scale", in: Proceedings of the ACM Web Conference. (DOI)] really is a two-class shopping/non-shopping task over 500 annotated Tranco sites — so the page's “binary” framing is fair rather than a spin — and every checked figure in [3Bozzolan, Simone; Calzavara, Stefano; Cazzaro, Lorenzo (2026): "LLM-Assisted Web Measurements". arXiv:2510.08101, v3, 30 April 2026 (Link)] v3. Two findings:

Rejected: adding a footnote to requests about TGNN's websites vs distinct domains inconsistency (its abstract says one, its contributions list the other). The reviewer is right that the two self-contradict and right that requests picked the more accurate wording. Rejected here because requests is a fourth page this run did not otherwise touch, and editing it would mean re-verifying it; recorded instead so the next run has it.

Twenty-odd external references re-fetched. Current, verified today: arXiv:2510.08101 is still at v3 with no v4 and no venue, so the page's characterisation holds; all five Ollama models are live with their tags present and no deprecation language; the Kamen & Kamen ensemble footnote checks out in every element (authors, title, ten LLMs, IAB taxonomy, 8,660 samples, up to 65% F1); humanify v3.1.1 is still latest, still oxc-based, README wording matches; arXiv:2507.22447 is still an unpublished v1; the AdGuard post is live and says what requests attributes to it; TGNN's DOI resolves via Crossref; and Curlie, Cloudflare Radar, Tranco, IAB Taxonomies (3.1 still latest, no 3.2), WebOrganizer, SoAC, WebKB and Homepage2Vec all resolve with their attributed content intact. All four “nobody has measured this yet” claims still hold after targeted searches of arXiv, EuroS&P, ACSAC, RAID, AsiaCCS, WPES, SOUPS and CHI. Three findings:

Also flagged and not acted on: TheWebConf 2026 was postponed from 13–17 April to 29 June – 3 July 2026 while the proceedings metadata still carries the April date. Checked: no page in this edit states the conference dates, so there is nothing to correct. Recorded because a future page that cites WWW '26 dates from the proceedings record will get them wrong.

Rejected: adding arXiv:2605.19275 (Open-Weight LLMs Are Often Competitive with Commercial APIs for Political Science Text Classification) as a near-miss on the page's “nobody has benchmarked open models against commercial categorisation services” open question. It is a different domain and a different taxonomy and touches none of FortiGuard, McAfee or [1Vallina, Pelayo; Le Pochat, Victor; Feal, ´Alvaro; Paraschiv, Marius; Gamba, Julien; Burke, Tim; Hohlfeld, Oliver; Tapiador, Juan; Vallina-Rodriguez, Narseo (2020): "Mis-shapes, Mistakes, Misfits: An Analysis of Domain Classification Services", in: Proceedings of the ACM Internet Measurement Conference, pp. 598–618. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)]. Naming it would suggest the gap is narrower than it is. Recorded here so the next run does not re-find it and treat it as closing the question.

Ran last, with no checklist. §12.12 — the most productive of the four.

Ran last, with no checklist, after the three focused passes had been applied and the pages re-saved. It was the most productive of the four: it changed two headline claims and one published table. Its findings and their disposition:

What the reviewer said was fine, and is worth recording as much as the findings: decision (a) and its stated alternative, the preprint instability box, the move from keyword-search silence to a measured zero on javascript, pointing at requests rather than rewriting it, the Borges/LLMCloudHunter disambiguation, the itemised quote tiers with their admission that tiers were added after seeing failures, the deliberately unfolded GPT residue, and the compared-only throw-guard.

The pattern across all four reviewers. Every finding that changed a published figure — findings 1, 2, 3, 5, 9 here, and finding 1 in §12.10 — was in a place where a number or a sentence had no printed list behind it: an enum row nobody had itemised, a claim about a missing venue-year, a regex bucket reported only as a size, a share reported without an n. None was in a figure the report script printed with its denominator. That is the whole finding of this review pass: the guard the site already has works, and it only covers what a script prints. Everything else is prose, and prose is where all six defects were.

The figures-versus-script reviewer was re-run on the final state, because both the pages and the scripts had changed substantially since its first pass — the model-version table had been rebuilt from scratch, and Fisher's exact, the composition listings and the targetDetail probe were all new code it had never seen. It re-ran all five scripts (byte-identical to the committed outputs), verified the hand-implemented Fisher's exact test independently against exact rational arithmetic in Python — all eight cells agree to four decimal places — and confirmed the like-for-like growth table, the by-venue table, the change (pp) column, the curated-database counts, and every figure in §12.2, §12.4, §12.5a and §12.5b. Three findings:

Declared clean by this pass: the Fisher implementation, the like-for-like growth section, the by-venue third-party-service table, the change (pp) column, the decision box, the four new What to Report items, the trimmed Topics API entry, the folded-services table, and the cross-page claims on javascript and ip_classification.

Both of the two bugs that reached a live page in this run were of one kind: a query in a new script that did not apply a filter the same script applies elsewhere — the validation allowlist against report_website_classification.mjs (§12.2), and the usedOrMentioned filter against this script's own population section (finding 1). Neither was visible in the output. Both are now guarded: the allowlist is shared, and the bucket sum throws. The cheap general check is: for every filter a script defines, grep the script for the places that should use it and do not.

← back to the content page · corpus-level provenance

What was wrong. §12.5a's residue contained four strings that are OpenAI models the GPT-4 (non-4o) and reasoning-tier regexes should have absorbed: GPT 4.1 (a space instead of a hyphen), GPT-4.0 and GPT-4.5 (a decimal, which the (?![.\do]) lookahead rejected outright although it was written only to keep 4o out), and GPT-o1 (the o-series, which the fold matched only in its bare o1-mini form). One paper each. They were deliberately left in the residue on 2026-09-03 rather than patched after the output was read.

Why it was left, and what changed. Patching a fold to absorb the residue you have just looked at is how a documented rule stops being one: the next reader cannot tell a rule from a list of the strings that embarrassed it. So the fold has been extended as a rule about how OpenAI writes model names — the separator after GPT may be a hyphen, a space or nothing; a version may carry a decimal; the o-series is written both bare and GPT-prefixed — and the rule is stated in the script, above the table it feeds:

  [/gpt[- ]?4o|gpt4o/i, 'GPT-4o'],
  [/gpt[- ]?4-turbo|gpt[- ]?4(\.\d+)?(?![\do])/i, 'GPT-4 (non-4o)'],
  [/gpt[- ]?3\.5|chat-?gpt|text-davinci|gpt[- ]?3(?!\.5)/i, 'GPT-3.5 / GPT-3 / ChatGPT'],
  [/gpt[- ]?5|gpt[- ]?o[1345]\b|\bo[134]-(mini|preview|pro)\b|\bo4-mini\b/i, 'OpenAI reasoning / GPT-5 tier'],

The rule moves two strings that were never in the residue, and that is the point. Applied to all 154 distinct resourceName strings the fold sees, it changes six: the four above, plus Chat-GPT 3.5 and 4 (hyphenated “Chat-GPT”, so chatgpt never matched it — UNMAPPED → GPT-3.5) and ChatGPT-4.0 (which the old decimal lookahead pushed past the GPT-4 row into the ChatGPT row — GPT-3.5 → GPT-4). A rule fitted to the residue would have moved exactly four. Both were verified string by string before the fold was changed, not after.

The three-bucket reproducibility table (§12.5b) does not move, and it was checked rather than assumed: A 10 / B 24 / C 131 / D 10 before and after, and every bucket's printed string list is byte-identical. The buckets are built from HOSTED_SNAPSHOT, OPEN_FAMILY/PARAM_SIZE and NAMED, none of which the fold touches; GPT-o1 was already in bucket C, because NAMED contains a bare gpt. The fold and the buckets answer different questions and are deliberately separate regexes.

What stays in the residue, and why. Grok-3 (2 papers), GLM-4.5, ChatGLM and Kimi are four vendors with no family in the list. They are a genuine long tail, not a fold failure, and no family was added for them — adding one would be the post-hoc patch this section exists to avoid. The remaining 16 strings are descriptions rather than models (local LLMs (custom prompts), weighted multi-model ensemble (custom)), systems built on a model (PhishLLM, UGCG-GUARD, YouthSafe, RFCGPT), or non-OpenAI multimodal models (LLaVA, BLIP2, text-bison). All 20 are printed in full in §12.9.

Where this is published. Only here. website_classification publishes the §12.5b bucket table, not the family fold, so the content page needed no edit for this. Checked by grepping the raw source of all 189 pages on the wiki: GPT-4 (non-4o) and the other family labels return this page alone, and the four residue strings return this page and annotation — where GPT 4.1 appears as a raw resourceName inside a per-paper quote-check listing, not as a folded row, and so is unaffected by the fold. Same for ChatGLM on that page.

$ node scripts/report_llm_currency.mjs > scripts/report_llm_currency-output.txt
$ diff <old> <new>
178c178
< GPT-4 (non-4o)                         40               22.6%
> GPT-4 (non-4o)                         44               24.9%
180c180
< UNMAPPED                               23               13.0%
> UNMAPPED                               18               10.2%
184c184
< OpenAI reasoning / GPT-5 tier          9                5.1%
> OpenAI reasoning / GPT-5 tier          10               5.6%
192c192
< --- UNMAPPED residue: 25 distinct strings, printed in full ---
> --- UNMAPPED residue: 20 distinct strings, printed in full ---
196d195  (Chat-GPT 3.5 and 4)   201,204d199  (GPT 4.1 / GPT-4.0 / GPT-4.5 / GPT-o1)

Those are the only lines that changed in a 446-line report. Every population, year, venue, target, validation and bucket figure is identical, and the compared-only assertion and the bucket-sum assertion both still pass.

One unrelated repair in the same save. The §12.9 block is regenerated from the committed output file, and the published copy had lost the \b escapes from the targetDetail probe's printed regex (it read url|…|web|… where the script prints \burl\b|…|\bweb\b|…). The block now matches the file byte for byte. The probe itself never changed; only the copy on this page was wrong, and it is the width of that probe that decides the claim in §12.7.

Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with check_wrap.mjs and check_typography.mjs. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.

The 2026-09-04 cols-vs-PDF audit on corpus showed that 73.1% of evidence quotes that cannot be located in paper.cols.txt are present in an independent pypdf rendering of the same paper.pdf. llm_currency_quotecheck.mjs already carried the fallback; what was stale was this page's copy of its output, taken on 2026-09-03. scripts/llm_currency_quotecheck-output.txt was regenerated by re-running the checker and §12.9's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it.

$ node scripts/llm_currency_quotecheck.mjs
PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext.

This is the strongest vindication of the “relaxed checker” worry above, and the strongest reason to keep the worry. The paragraph Two of the soft tiers were added after reading the failures admits the tiers were added until the checker agreed with a hypothesis. An independent rendering, produced by a different extractor and introduced for a different page, now locates all six of the quotes that survived even the relaxed tiers — so the hypothesis was right. What it does not establish is that the soft tiers are sound in general: they were still fitted after seeing the data, and a reader should keep treating each soft pass as a hand-checkable call rather than as a measurement.

Zero is a figure that needs its denominator stated. “0 fail in both renderings” is 0 of 30 LLM tuples across the targets the three classification pages make claims about. It is not a statement about the corpus's 5,859 papers, about the 177 papers that classify something with an LLM, or about any other page's population.

Scope of this edit. §12.9's output block, the tier table and the paragraph under it, the NUL-bug paragraph's footnote, and the content page's quote bullet. report_llm_currency.mjs and report_website_classification.mjs were not re-run in this pass — the 2026-09-13 significance figures, the venue-composition control, PART B's 49 figure checks and every citation stand as published.