| Next revision | Previous revision |
| provenance:design:website_classification [2026/08/12 09:42] – Create provenance page for design:website_classification: the 424 -> 330 population narrowing, the resource fold's 20 -> 40 -> 33 residue and what was deliberately left, two report-script bugs, all five new LLM website-category quotes verified, and the 4, karel.kubicek.claude | provenance:design:website_classification [2026/09/21 14:40] (current) – Review follow-ups: 12.8's 'task item exists' bullet closed by 12.14, and 12.14's page-sweep claim corrected (provenance:statistics:annotation carries the raw string, not a fold row). Authored by Claude karel.kubicek.claude |
|---|
| ^ Item ^ Value ^ | ^ Item ^ Value ^ |
| | Content page | [[design:website_classification]] | | | Content page | [[design:website_classification]] | |
| | Report script | ''scripts/report_website_classification.mjs'' (''--wiki'', ''--list'', ''--quotes <regex>'') | | | Report script | ''scripts/report_website_classification.mjs'' (''%%--wiki%%'', ''%%--list%%'', ''%%--quotes%% <regex>'') | |
| | Folds | ''scripts/webcat_fold.mjs'' — a **task** fold and a **resource** fold | | | Folds | ''scripts/webcat_fold.mjs'' — a **task** fold and a **resource** fold | |
| | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 | | | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 | |
| </code> | </code> |
| |
| ''check_page_numbers.mjs'' left four figures unaccounted, all deliberate: ''1,000'' (DarkGram's validation sample, from the paper), ''500'' (an image width in the ''{{...?nolink&500}}'' syntax — a false positive of the checker, worth knowing about), and ''7.4'' (the old cross-check share, quoted on the page as the comparison). It caught one real error: the page still said "the 73 excluded papers" after the count had moved to 94. | ''check_page_numbers.mjs'' left four figures unaccounted. **Three were deliberate** — ''1,000'' (DarkGram's validation sample, from the paper), ''500'' (an image width in the ''{{...?nolink&500}}'' syntax, a false positive of the checker worth knowing about) and ''7.4'' (the old cross-check share, quoted on the page as the comparison). **The fourth was a real catch**: the page still said "the 73 excluded papers" after the count had moved to 94. Run it whole-page too (§10). |
| |
| ===== 4. What the refresh changed ===== | ===== 4. What the refresh changed ===== |
| * **Nothing has re-audited the categorisation vendors since 2020.** That is the page's largest open question and this refresh does not touch it. | * **Nothing has re-audited the categorisation vendors since 2020.** That is the page's largest open question and this refresh does not touch it. |
| |
| ===== 9. Run log ===== | ===== 10. Review pass, 2026-08-12 ===== |
| | |
| | //Recorded.// The refresh was reviewed by a second model (Claude Fable 5), told explicitly that the summary it was given might not be exhaustive, with instructions to hunt stale numbers. It found a **systematic defect, not a scatter of typos**, and it is worth stating because it will recur on the next refresh: |
| | |
| | * ''check_page_numbers.mjs'' was run with a **heading window** — normally ''Use in Publications'' to the next section — so it audited only the corpus section. Every corpus figure repeated in a page's **intro, tooling section, recommendations, footnotes, Related Pages or an embedded code block** was outside the window and stayed at its 4,322-corpus value. Across the six pages **29 such figures** survived the first pass. |
| | * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. |
| | |
| | Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''%%--code%%'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. |
| | Fixed on this page's content page as a result: the lead paragraph's **72% → 75%** for bespoke unsized taxonomies, which contradicted the corpus section's own "Three quarters … 75.4%" four screens down. |
| | |
| | §3 above previously said ''check_page_numbers.mjs'' "left four figures unaccounted, all deliberate" and then listed three. That was wrong in the way this whole review is about: the fourth was ''73'', a genuine stale count, and describing it as deliberate would have buried the guard's only real catch on this page. Corrected. |
| | |
| | ===== 11. Run log ===== |
| |
| ^ ^ ^ | ^ ^ ^ |
| | Caveats deleted | "IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share" — replaced with a box recording that the hypothesis was correct and is now resolved. "anything published after 2024". | | | Caveats deleted | "IEEE S&P is only 43% retrieved, which is the most likely explanation for its 1.2% share" — replaced with a box recording that the hypothesis was correct and is now resolved. "anything published after 2024". | |
| | Mistake caught in review | ''check_page_numbers.mjs'' caught "the 73 excluded papers" left behind after the count moved to 94 — the third time in this refresh that the guard caught a number the human eye had skipped. | | | Mistake caught in review | ''check_page_numbers.mjs'' caught "the 73 excluded papers" left behind after the count moved to 94 — the third time in this refresh that the guard caught a number the human eye had skipped. | |
| | | Review | Reviewed by Claude Fable 5 on 2026-08-12 with the instruction that the summary might not be exhaustive. It found the windowed-guard defect in §10 and 1 stale figure on this page plus a mis-worded line in §3. All fixes were applied and re-saved the same day. | |
| | |
| | ===== 12. LLM-classification currency, 2026-09-03 ===== |
| | |
| | The refresh of 2026-08-12 (§4) established that LLM classification had arrived in the corpus. This section is the follow-up it asked for: it settles which of three pages owns the currency claim, retracts two wrong sentences, and stress-tests the closed-source-drift reversal §4 left open — which turned out to be half right. Companion sections were written the same day on [[provenance:privacy:javascript]] and [[provenance:design:ip_classification]]; the shared numbers live **here** and those two link to them. |
| | |
| | Sections 12.1–12.9 were kept as a log during the run. §12.10 and §12.12 are the review passes, and **several figures in 12.1–12.9 are there in their corrected form because a reviewer changed them** — where that happened it is said so, with the wrong value. |
| | |
| | ==== 12.1 The problem this run was given ==== |
| | |
| | Three pages made currency claims about the same schema field, ''classification.method == "llm"'', from three separate queries, and they had drifted: |
| | |
| | ^ Page ^ What it said ^ Verdict ^ |
| | | [[design:website_classification]] | LLM classification arriving; section led with an unrefereed preprint | Correct but mis-framed | |
| | | [[privacy:javascript]] | "no peer-reviewed paper classifies web scripts as trackers with an LLM"; separately, "the ''llm'' enum now fires twice" | Core claim **true**; the ''llm''-row sentence **misleading** | |
| | | [[design:ip_classification]] | "exactly one" LLM paper; "unlike cookie and policy classification, where LLM methods are now routine" | First claim true; second claim **wrong** | |
| | |
| | ==== 12.2 Every query, with its population and denominator ==== |
| | |
| | One script answers all of them: ''scripts/report_llm_currency.mjs''. It exists so the three pages cannot disagree again — its last section restates each page's own sentence next to what the corpus says, so a future drift fails visibly instead of quietly. |
| | |
| | <code bash> |
| | cd /workspace/artifacts/wiki |
| | node scripts/report_llm_currency.mjs # every figure |
| | node scripts/report_llm_currency.mjs --wiki # DokuWiki tables |
| | node scripts/report_llm_currency.mjs --list website-category # the papers behind a row |
| | node scripts/report_llm_currency.mjs --quotes cookie # the evidence quotes |
| | node scripts/llm_currency_quotecheck.mjs # quotes vs paper.cols.txt |
| | node scripts/report_website_classification.mjs # incl. the drift controls, |
| | # Fisher tests and compositions |
| | </code> |
| | |
| | ^ Query ^ Population ^ Result ^ |
| | | ''method == "llm"'', any ''usedOrMentioned'' | all 5,859 papers | **177 (3.0%)** | |
| | | …''used'' or ''produced'' the labels | all 5,859 | 175 | |
| | | …only ''compared'' against an LLM | all 5,859 | **2** | |
| | | by year | corpus that year | 2023: 2/719 (0.3%); 2024: 27/690 (3.9%); 2025: 77/770 (10.0%); 2026*: 71/415 (**17.1%**) | |
| | | by year, fairer denominator | papers that classified anything that year | 0.4%, 5.1%, 12.7%, **21.3%** | |
| | | growth, like-for-like | only the five venues with a 2026 slice | **9.8% (2025) → 17.1% (2026)**, and all five rose individually | |
| | | by venue | LLM papers = 177 | USENIX 43, TheWebConf 40, NDSS 30, PETS 21, CCS 18, IEEE S&P 14, IMC 11 | |
| | | validated labels | LLM papers = 177 | **148 (83.6%)** | |
| | | per target | papers classifying **that target** at all | the table in [[design:website_classification#Which task, though]] | |
| | | model named to an artefact | the **175** papers that used or produced LLM labels | A dated hosted snapshot **10 (5.7%)**; B open-weight checkpoint with a size **24 (13.7%)**; **A+B 34 (19.4%)**; C family only **131 (74.9%)**; D no model **10 (5.7%)** | |
| | |
| | **Three denominator traps this section had to avoid, and one it fell into.** |
| | |
| | - **"177 of 5,859" is not an adoption rate for anything.** The per-target column that decides a page's claim is ''llm papers at target T ÷ papers that classify T at all'' — 1 of 53 for ''cookie'', 5 of 424 for ''website-category'', 0 of 44 for ''javascript''. Published without that denominator, the same field supports both "LLMs have arrived" and "LLMs are absent", which is how the three pages drifted apart. |
| | - **''validation'' has two sentinels, not one.** The values are ''manual-validation'', ''held-out-test-set'', ''comparison-to-other-method'', ''cross-validation'', ''none-reported'' and ''not-applicable''. The first draft of ''report_llm_currency.mjs'' used the shared ''isSentinel()'' helper, which knows ''not-applicable'' but **not** ''none-reported'', and produced **173 of 177 (97.7%) validated** where the truth is **148 (83.6%)**. Caught before saving by diffing against ''report_website_classification.mjs'', which had the explicit allowlist; both now share it. It never reached a page, but it was the most flattering figure in the section and it was wrong by 14 points. |
| | - **A count from an incomplete venue-year is a floor; a share is not.** The page said "both of the last two venue-years are incomplete, so those shares are floors". That is a logic error — a share whose numerator //and// denominator come from the same incomplete set is a share of a differently-composed population, not a lower bound. Flagged in the generic review (§12.12, finding 9) and replaced with the like-for-like five-venue check above, which is the thing that actually supports the growth claim. |
| | - **''classification.method'' being 58% stable is not a footnote.** §12.4: it is the whole reason one of this page's two headline movements had to be retracted. |
| | |
| | ==== 12.3 The judgement call: should the LLM section still lead with the preprint? ==== |
| | |
| | **Decision: no, and the preprint stays anyway.** The section now opens by separating two questions — //is an LLM a normal thing to classify with in this field?// (yes, 177 papers, since about 2024) and //can one assign a website a topic well enough to replace a vendor API?// (peer-reviewed for a **binary** question; only the preprint for a real taxonomy). The corpus evidence moved out of a ''%%<WRAP important>%%'' box into the body, because a box reads as a caveat bolted onto the surrounding prose and the surrounding prose was preprint-led. |
| | |
| | The preprint keeps most of the section's length, and the page now says why: **the peer-reviewed evidence stops exactly where the hard part starts.** {[tsai2025_harmful]} is a two-class shopping/non-shopping decision. The comparison a reader of that page is actually making — an LLM against a trained specialist classifier on a multi-label taxonomy — has been run once, in {[bozzolan2026_llmweb]}, and nowhere else. |
| | |
| | **Two alternatives were considered.** Cutting the preprint to a paragraph was rejected: the alternative to it is not a better source, it is silence, and the page's own instability box is more use to a reader than an omission. **Moving the preprint's tables and cautions into [[design:website_classification#Run Your Own Classifier]]** — where Homepage2Vec already lives and where a self-hosted open-weight model belongs — was raised by the generic reviewer (§12.12, finding 11) and **not done**, because it is a large restructure with anchor breakage of its own and this run had already rewritten the section once. What //was// done instead is the thing the reviewer said was missing: a three-line decision box at the top of the section, so a reader who does not read the preprint analysis still leaves with an answer. **The restructure remains the right call and is deferred, not rejected.** |
| | |
| | ==== 12.4 The closed-source drift: one movement holds, one is retracted ==== |
| | |
| | §4 left this as the thing to re-check "when 2026 is complete". 2026 is **not** complete and will not be for this corpus, so the run did three other things: split the window by year, control for venue composition, and — after the generic review — run Fisher's exact test and print what the enum row actually contains. **The result is that half of what §4 recorded does not survive.** |
| | |
| | === The third-party-service drop: supported === |
| | |
| | ^ Cut ^ Third-party service ^ n ^ change ^ Fisher //p// ^ |
| | | 2022–2024, all venues //(baseline)// | 44 (49.4%) | 89 | — | — | |
| | | 2025 alone | 17 (37.8%) | 45 | −11.7 pp | 0.27 **not supported** | |
| | | 2026 alone //(provisional)// | 4 (20.0%) | 20 | −29.4 pp | **0.024** | |
| | | 2025–2026, all venues | 21 (32.3%) | 65 | −17.1 pp | **0.047** | |
| | | 2022–2024, five venues with a 2026 slice | 34 (54.0%) | 63 | — | — | |
| | | **2025–2026, those same five venues** | **16 (28.6%)** | 56 | **−25.4 pp** | **0.006** | |
| | |
| | The composition control makes the drop //larger//, so the missing-venue explanation fails in the direction that would have rescued the old trend. **But the earlier draft of this section leaned on "2025 alone, all seven venues held" as a control, and at n = 45 and p = 0.27 that is a description, not a test.** The significance comes from the 2026 slice and the five-venue restriction. Corrected on the page and here. |
| | |
| | === The curated-database "return": retracted === |
| | |
| | §4 recorded that "curated databases had fallen (21.4 → 9.0%) and return to 18.5%", and called it the first movement against the closed-source drift. **That reading is withdrawn**, for two reasons found in the generic review (§12.12, finding 1) and confirmed here: |
| | |
| | - **No cut clears p < 0.05.** 0.16 for 2025 alone, 0.23 for 2026 alone, 0.095 for the full window, 0.079 for the five-venue control. |
| | - **The row does not contain open directories.** ''report_website_classification.mjs'' now prints every name behind it. Sorted, the twelve recent-window papers are three commercial vendors (Cloudflare Radar, SimilarWeb, Symantec SiteReview); four media-bias raters with closed editorial processes (AllSides, and Media Bias/Fact Check in three papers, one of which also cites Science Feedback); one model (Homepage2Vec); two taxonomies rather than label databases (IAB, NAICSlite); one paper's own rule set; and one that genuinely is an open, inspectable repository — DuckDuckGo's Tracker Radar Entity List — but of //tracker entities//, not website topics. **Not one is DMOZ, Curlie or any comparable open topic directory.** |
| | |
| | **The first version of this list, on this page and on the content page, named eleven of the twelve** — it silently dropped ''NDSS/2026/revealing-the-secret-power…'', whose resource string is "Media Bias/Fact Check (MBFC)", while the derived "four media-bias raters" counted it. Caught in re-review (§12.13, finding 2). A hand-written enumeration beside an embedded script output is the one kind of listing on this page that can still go stale, and this is the second time in one run it did. By contrast the eight in 2022–2024 are more genuinely directory-like: DappRadar, a pornhosts blocklist, WebPulse, the Citizen Lab Block List, the AllSides Media Bias Chart, MediaBias/FactCheck, YouTube category labels, and one paper's "external political, government, media and issue-page sources". |
| | |
| | So the openness half of this page's central argument has **no counter-evidence in the recent window**; the enum row that looked like counter-evidence is mostly mis-filed vendors. This is ''classification.method'''s 58% run-to-run stability doing exactly what the methodology bullet warns it does, on the one row where the page had built an argument on top of it. |
| | |
| | === A sentence no script owned, and it was wrong === |
| | |
| | The ''%%<WRAP todo>%%'' box said CCS 2026 and IMC 2026 are "the two whose 2022–2024 populations lean hardest on third-party services". Within this population that share by venue is TheWebConf 66.7%, IMC 64.3%, PETS 62.5%, IEEE S&P 55.6%, USENIX 47.6%, NDSS 20.0% and **CCS 8.3% — the lowest of the seven.** So IMC's absence is the one that plausibly inflates the drop, and adding CCS would push the recent share //down// and strengthen the reversal. The page asserted the opposite. |
| | |
| | **This was the only sentence in the drift section not derived from a printed table, which is why it survived three earlier passes**, including a figures-versus-script reviewer who had nothing to diff it against. The by-venue table is now printed by the report script. **The lesson generalises: a sentence that tells the reader what a missing input would do is a claim, and needs a query like any other.** |
| | |
| | ==== 12.5 Folds, buckets and residue ==== |
| | |
| | ''classification.resourceName'' is free text and agrees run-to-run on roughly a fifth of exact strings, so anything counted from it is a fold, reported as a ranking, with the residue printed. Both folds live inside ''report_llm_currency.mjs'' rather than in a shared module, because they are specific to this question and a shared version would invite reuse where the families are wrong. |
| | |
| | === 5a. The model-family fold === |
| | |
| | ^ Family ^ Papers (of 177) ^ |
| | | GPT-4o | 46 (26.0%) | |
| | | GPT-4 (non-4o) | 44 (24.9%) | |
| | | GPT-3.5 / GPT-3 / ChatGPT | 29 (16.4%) | |
| | | **UNMAPPED** | **18 (10.2%)** | |
| | | Llama family | 13 (7.3%) | |
| | | Gemini / PaLM | 12 (6.8%) | |
| | | Unnamed LLM | 11 (6.2%) | |
| | | OpenAI reasoning / GPT-5 tier | 10 (5.6%) | |
| | | Qwen family | 8 (4.5%) | |
| | | DeepSeek family | 7 (4.0%) | |
| | | Mistral family | 4 (2.3%) | |
| | | Other open-weight | 4 (2.3%) | |
| | | Claude family | 3 (1.7%) | |
| | | Encoder / seq2seq LM (not a chat LLM) | 2 (1.1%) | |
| | |
| | **The table above is the 2026-09-21 re-derivation.** Until then it read GPT-4 (non-4o) 40 (22.6%), UNMAPPED 23 (13.0%) and OpenAI reasoning 9 (5.1%), because four residue strings — ''GPT 4.1'', ''GPT-4.0'', ''GPT-4.5'' and ''GPT-o1'' — are OpenAI models that the fold missed on a space or a decimal. They were left visible on 2026-09-03 rather than quietly folded, because patching a fold to absorb its own residue after seeing the output stops it being a documented rule. The fold has now been extended **as a rule about how OpenAI model strings are spelled**, not as a list of those four: see §12.14 for the rule, the two further strings it moves, and the full diff. ''Grok-3'', ''GLM-4.5'', ''ChatGLM'' and ''Kimi'' stay in the now 20-string residue — they are the genuine long tail, four vendors with no family in the list, and no family was added for them. |
| | |
| | === 5b. The reproducibility buckets, which had to be rebuilt === |
| | |
| | The first version of this table had a single "dated or pinned" bucket built from OpenAI snapshot suffixes plus an Ollama size tag, published as "**11 papers name a model you could re-run**". **The generic reviewer showed that was wrong in a way that inverted the page's own advice** (§12.12, finding 3): ''Llama-3.1-70B-Instruct'', ''Qwen2.5-Coder-32B-Instruct'', ''Mistral-7B-Instruct-v0.2'', ''Gemma 3 27B'' and ''T5-11B'' are fully resolvable artefacts and all landed in the "no version" bucket, while the paragraph two sentences later said open weights are the //only// re-runnable option. Roughly 25 papers were being counted as unpinned for naming exactly what the page told them to name. |
| | |
| | Rebuilt, split by hosting, which is the axis that actually decides reproducibility: |
| | |
| | ^ Bucket ^ Papers ^ Share of 175 ^ |
| | | **A** dated **hosted** snapshot | 10 | 5.7% | |
| | | **B** **open-weight** checkpoint with a size | 24 | 13.7% | |
| | | **A+B** resolvable to an artefact | **34** | **19.4%** | |
| | | **C** family only, no version | 131 | 74.9% | |
| | | **D** no identifiable model | 10 | 5.7% | |
| | |
| | **The population is the 175 papers that used or produced LLM labels, not the 177 that mention one, and getting that wrong was the third bug in this table.** A re-review pass (§12.13, finding 1) found the bucket loop scanning //every// ''llm'' tuple including ''compared'' ones, which promoted two papers on the strength of a baseline they argued against — one of them a paper whose **only** ''llm'' tuple is ''compared'', counted in bucket A for a model it never ran. The script now buckets on ''llmUsed()'' and **throws if the buckets do not sum to the population**, so the two filters cannot diverge again silently. |
| | |
| | Every string in all four buckets is printed by the script (§12.9). **Two calls a reasonable person would make differently:** ''Mistral Large'' counts as B though it is a hosted API model, and ''FLAN-T5-XXL'' counts as B on a word-sized parameter count (''PARAM_SIZE'' accepts ''xxl'', ''large'', ''mini''). Both are visible in the printed list rather than buried in a share. |
| | |
| | The three-bucket rule, verbatim: |
| | |
| | <code javascript> |
| | const HOSTED_SNAPSHOT = /\d{4}-\d{2}-\d{2}|\b(0125|0613|0314|1106|0409|0806|2024\d{4}|2025\d{4})\b/i; |
| | const OPEN_FAMILY = /llama|qwen|mistral|mixtral|gemma|vicuna|falcon|deepseek|glm|chatglm|\bphi-?\d|gpt-oss|\bt5\b|flan-t5|\bbert\b|roberta|deberta|\bbart\b|olmo|smollm/i; |
| | const PARAM_SIZE = /\b\d+(\.\d+)?\s*b\b|:\d+(\.\d+)?b\b|\b(xxl|xl|large|small|mini|base)\b/i; |
| | </code> |
| | |
| | === 5c. Two regex bugs, one of which the fix caused === |
| | |
| | - **A bare ''phi'' matched "**Phi**shLLM"**, a prior detector rather than a model, putting that paper in the "names a family" bucket while the same script's resource fold put the identical string in its residue — two regexes in one script disagreeing about one string. Found by the figures reviewer (§12.10, finding 1). |
| | - **The fix anchored every alternative, including ''gpt'' → ''\bgpt'', and "ChatGPT" has no word boundary before "gpt".** Twelve ChatGPT papers dropped into "names no model at all", taking that bucket from 10 to 24 and the family bucket from 154 to 140. **Nothing in the figures looked wrong** — 13.6% unnamed is plausible and the table still summed. It was caught only because the same change had added a print of every bucket's strings, and ''ChatGPT'' was sitting in the "no identifiable model" list. |
| | |
| | **The rule adopted from this: a bucket defined by a regex prints its members, or "generous" and "conservative" are words doing work a list should do.** Every bucket and every fold in both new scripts now prints in full, and that is what caught findings 1 and 3 as well. |
| | |
| | ==== 12.6 Quotes checked ==== |
| | |
| | Every LLM tuple across the targets the three pages make a claim about, checked by a committed script rather than by hand: ''scripts/llm_currency_quotecheck.mjs'', 30 tuples, output in §12.9. It reports six tiers rather than pass/fail, because a two-column PDF produces false FAILs and calling those "unlocatable" would overstate the problem while a bare pass/fail would understate it. |
| | |
| | ^ Tier ^ Tuples ^ What it means ^ |
| | | PASS | 15 | present verbatim after collapsing whitespace | |
| | | PASS-ELID | 2 | the extractor wrote "..."; both fragments present, the middle unverified | |
| | | PASS-LOOSE | 4 | present after dropping punctuation and folding the PDF's math-italic glyphs | |
| | | PASS-CITE | 1 | present once inline citation markers are stripped — the extractor drops them, so ''Qwen3 [49]'' becomes ''Qwen3'' | |
| | | PASS-NGRAM | 2 | ≥80% of the quote's word 5-grams present; the sentence is in the paper but the extraction reworded a word or two, or ''.cols'' interleaved a float into it | |
| | | **RESCUED** | **6** | below every tier above against the rendering the extractor read, and at or above the n-gram tier against an independent ''pypdf'' rendering of the same ''paper.pdf'' | |
| | | **FAIL in both renderings** | **0** | not present under any tier in either rendering | |
| | |
| | **Re-run 2026-09-21 with the PDF fallback, and the six FAILs are now six RESCUEDs: nothing in this pass fails in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. Four of the six go to a **complete** n-gram match in the PDF (36/36, 14/14, 21/21, 11/11) and one to 19/19; the sixth, ''PETS/2026/word-level-annotation…'', goes 11/20 → 16/20, which clears the 80% tier. All six are ''privacy-policy'' or ''consent-notice'' tuples and no page on this site quotes any of them, so no published claim moves; every tuple behind a figure on the three edited pages passed at some tier already. |
| | |
| | **Two of the soft tiers were added after reading the failures, and that is worth admitting.** The first run reported 15 fail; reading them showed the extraction drops inline citation markers, and that ''paper.cols.txt'' interleaves table captions into sentences — ''PETS/2026/disclosure-divergence…'' failed because the caption //"Table 1: LLM backend comparison on 100-app validation."// lands inside the quoted sentence. Adding tiers for those is right, since the alternative is publishing "half the quotes are unlocatable", which is false. But it is also a checker relaxed until it agreed with a hypothesis. The mitigation is that every soft pass is itemised with its reason and, for PASS-NGRAM, the broken n-grams, so each call is checkable by hand. |
| | |
| | **A latent bug found in review changed nothing, and is recorded anyway.** 90 of the 5,869 ''paper.cols.txt'' files (1.5%) contain **NUL bytes** — 913 in the TGNN paper alone. They are not whitespace to ''\s'', invisible in a terminal, and they make shell ''grep'' treat the file as binary and suppress every match silently (use ''grep -a''). The checker did not strip them; it now does. Re-running gave **15 / 9 / 6 before and after**, because no NUL happened to land inside one of these 30 quotes. On a different sample it would have been a published false FAIL. ((The ''6'' in that sentence is the 2026-09-03 FAIL count. Under the PDF fallback added on 2026-09-04 the same six are ''RESCUED'' and the fail-in-both count is 0 — see //Quote-check refresh, 2026-09-21//. The NUL finding itself is unaffected: it is about ''.cols'' preprocessing, not about the fallback.)) |
| | |
| | **6 of 30 (20%) is far above the corpus-wide 0.9% unlocatable rate** on [[literature:corpus]]. It is not a re-measurement: 30 tuples, non-random, all 2024–2026, weighted to PETS 2026 whose PDFs are the newest and worst-rendered. Read it as a reason to check quotes from the 2025–2026 slice specifically. |
| | |
| | ==== 12.7 External sources ==== |
| | |
| | **Written during the run, and then overtaken by the review.** This section originally said "no new external source was added to the content page in this run". That was true when written and false by the end: the external-currency pass (§12.10) put six primary sources onto [[design:website_classification]] — Google's Privacy Sandbox status page, Chrome Platform Status, the Blink //Intent to Deprecate and Remove// thread, the Chrome Version History API, and OpenAI's deprecations and pricing pages. Each replaced or repaired something the page had been carrying second-hand. That sequence is the reason this section is dated: **a provenance section written before the review pass will be wrong about the review pass.** |
| | |
| | Two things **rejected**: |
| | |
| | * **Using 2025–2026 LLM papers from outside the seven venues as evidence for the currency claim.** The page's method is that a claim is a claim about these seven venues; a hand-picked outside set would be the "we searched and found some papers" the dataset exists to replace. The corpus zeros are stated as corpus zeros and the outside search is reported separately, as a search. |
| | * **Fixing five duplicate-paper pairs in [[literature:bibliography]]** — the same paper under two citekeys, five times. Three were found by a DOI scan: ''ahmad2026_ipfp''/''ahmad2026_more'' (10.56553/popets-2026-0109), ''fouad2022my''/''fouad2022_cookie'' (10.56553/popets-2022-0063), ''bottger2025_regional''/''boettger2025_regional'' (10.56553/popets-2025-0063). **Two more were found by a reviewer using normalised titles, and a DOI scan structurally cannot see them** because those entries carry only ''url'': ''lerner2016internet''/''lerner2016_internet'' and ''bouhoula2024automated''/''bouhoula2024_automated''. That is the useful part: PETS and USENIX index records have **no DOI at all**, 100% of both venues, so a DOI-keyed collision scan is blind to exactly the two venues where ''bibgen.mjs'' is weakest, and any guard must key on normalised title too. Deduplicating means finding every page that cites each loser and repointing it — its own piece of work, now a task item. None of the five is cited by the three edited pages. |
| | |
| | One entry **was** added: ''schwartz2025_llmcloudhunter'' (TheWebConf 2025, DOI 10.1145/3696410.3714798), from ''scripts/bibgen.mjs'', needed so [[design:ip_classification]] can name the paper behind its single ''llm'' firing instead of describing it as "one paper, GPT-4o" next to a sentence about a different paper. |
| | |
| | ==== 12.8 What could not be established ==== |
| | |
| | * **The magnitude of the third-party-service drop.** §12.4. The 2026-alone column is 20 papers. Needs CCS 2026 and IMC 2026. |
| | * **Whether the curated-database row means anything at all.** §12.4. It is not significant and its contents are mis-filed. A stronger answer would need the twelve papers read and the enum corrected, not re-queried. |
| | * **Whether ''target == "other"'' hides more LLM website, script or tracker classification.** Three earlier drafts of these pages said "reading 116 papers would settle it". **That was lazy and the reviewer was right to say so** (§12.12, finding 10): ''other'' carries a ''targetDetail'', stated on all 157 such tuples, and a keyword probe over it now runs in the report script. It returns **20** tuples and **none is a website-topic, JavaScript, tracker or cookie classification** — they are IoT device categories and control pages, image content, decompiler clusters, MCP server categories, GDPR data categories, phishing-personalisation page text and threat-intelligence page triage. So the zeros survive the ''other'' bucket **at keyword recall**, which is what the pages now say. |
| | |
| | **The probe's width decided that answer, and the first width was wrong.** It used ''\bpage\b'', which does not match the compound "webpage" — there is no word boundary between "web" and "page" — and it had no bare ''web'' at all, so it returned 16 rather than 20 and silently dropped //"relevant person-specific webpage information"// and //"IOB presence and trustworthiness in web content"//. Found in re-review (§12.13, finding 3). Both were then read and neither changes the conclusion, which is the only reason the published claim survived a probe that was under-recalling by 20%. A probe is not a read of 116 papers, its hits must be read rather than counted, and its regex is a load-bearing part of the claim. |
| | * **Whether the four GPT strings in the fold residue change a published share.** §12.5a. **Settled on 2026-09-21 and no longer open** — the fold was extended as a rule and the three GPT rows re-derived; they do change one published share on this page and none anywhere else. See §12.14. |
| | * **Whether the six quote failures are extraction paraphrase or ''.cols'' rendering.** One was read and was rendering. The other five were not, because no page quotes them. |
| | |
| | ==== 12.9 The scripts and their unedited output ==== |
| | |
| | Three scripts, all committed under ''scripts/''. Their output below is unedited: every figure on the three content pages is in it, as is every bucket's membership, every fold's residue, every //p// value and the composition of the ''curated-database'' row. **The bucket and residue listings on this page are not transcribed from these files — they are these files**, which is why §12.12 finding 6 (a hand-copied list that had gone stale) cannot recur. |
| | |
| | <file text report_llm_currency-output.txt> |
| | |
| | ======================================================================== |
| | POPULATION |
| | ======================================================================== |
| | Population Papers Share of 5,859 |
| | ---------------------------------------------------- ------ -------------- |
| | corpus 5859 100% |
| | papers with >=1 classification tuple 4439 75.8% |
| | >=1 tuple with method == "llm" (any usedOrMentioned) 177 3.0% |
| | ... of which USED or PRODUCED the LLM labels 175 3.0% |
| | ... of which only COMPARED against an LLM 2 0.0% |
| | |
| | >>> The 177 figure published on design:website_classification is the "any" |
| | row. The used/produced row is the one to quote for adoption. |
| | |
| | ======================================================================== |
| | BY YEAR (two denominators) |
| | ======================================================================== |
| | Year LLM papers (any) LLM papers (used) Corpus that year Share of corpus Papers classifying that year Share of those |
| | ---- ---------------- ----------------- ---------------- --------------- ---------------------------- -------------- |
| | 2023 2 2 719 0.3% 503 0.4% |
| | 2024 27 27 690 3.9% 529 5.1% |
| | 2025 77 76 770 10.0% 607 12.7% |
| | 2026 71 70 415 17.1% 333 21.3% |
| | |
| | total: 177 papers (any), 175 (used/produced), of 5859. |
| | 2025 and 2026 are provisional venue-years — CCS/IMC 2026 not held, IEEE S&P |
| | and WWW 2026 incompletely selected — so the last two rows are floors. |
| | |
| | ======================================================================== |
| | LIKE-FOR-LIKE GROWTH: is the 2026 share a composition artefact? |
| | ======================================================================== |
| | five-venue 2025: 58/590 = 9.8% |
| | five-venue 2026: 71/415 = 17.1% |
| | Venue has a 2026 slice? 2025 2026 change |
| | ------- ----------------- -------------- -------------- -------- |
| | CCS NO 12/101 = 11.9% — — |
| | IEEE-SP yes 10/117 = 8.5% 3/28 = 10.7% +2.2 pp |
| | IMC NO 7/79 = 8.9% — — |
| | NDSS yes 11/119 = 9.2% 19/163 = 11.7% +2.4 pp |
| | PETS yes 4/67 = 6.0% 16/88 = 18.2% +12.2 pp |
| | USENIX yes 18/224 = 8.0% 13/69 = 18.8% +10.8 pp |
| | WWW yes 15/63 = 23.8% 20/67 = 29.9% +6.0 pp |
| | CCS + IMC 2025 (the two venues absent from 2026): 19/180 = 10.6% |
| | >>> If that figure is BELOW the five-venue 2026 share, the absence of |
| | those two venues does not obviously inflate 2026. |
| | |
| | ======================================================================== |
| | BY VENUE (of the LLM papers, any) |
| | ======================================================================== |
| | Venue LLM papers Share of LLM papers Papers in venue Share of that venue |
| | ------- ---------- ------------------- --------------- ------------------- |
| | USENIX 43 24.3% 1410 3.0% |
| | WWW 40 22.6% 843 4.7% |
| | NDSS 30 16.9% 701 4.3% |
| | PETS 21 11.9% 510 4.1% |
| | CCS 18 10.2% 990 1.8% |
| | IEEE-SP 14 7.9% 767 1.8% |
| | IMC 11 6.2% 638 1.7% |
| | |
| | ======================================================================== |
| | BY TARGET — the quantity the three pages disagree about |
| | ======================================================================== |
| | Target LLM papers (any) Share of 177 LLM papers (used) Papers with ANY method at this target LLM share of that target |
| | --------------------- ---------------- ------------ ----------------- ------------------------------------- ------------------------ |
| | other 116 65.5% 115 2592 4.4% |
| | user-generated-text 32 18.1% 30 419 7.2% |
| | vulnerability 14 7.9% 14 880 1.6% |
| | privacy-policy 12 6.8% 12 102 11.8% |
| | mobile-app 6 3.4% 6 280 2.1% |
| | website-category 5 2.8% 5 424 1.2% |
| | domain 3 1.7% 3 351 0.9% |
| | network-traffic 2 1.1% 2 382 0.5% |
| | consent-notice 2 1.1% 2 39 5.1% |
| | ip-address 1 0.6% 1 295 0.3% |
| | web-request 1 0.6% 1 258 0.4% |
| | email-message 1 0.6% 1 54 1.9% |
| | cookie 1 0.6% 1 53 1.9% |
| | dark-pattern 1 0.6% 1 13 7.7% |
| | malware 0 0.0% 0 159 0.0% |
| | sdk-or-library 0 0.0% 0 77 0.0% |
| | javascript 0 0.0% 0 44 0.0% |
| | fingerprinting-script 0 0.0% 0 31 0.0% |
| | website-popularity 0 0.0% 0 15 0.0% |
| | |
| | >>> The last column is the one that decides a currency claim on a |
| | per-target page. "LLM classification is now routine" is true of the |
| | corpus as a whole and false of most individual targets. |
| | |
| | ======================================================================== |
| | PER-TARGET DETAIL for the targets the three pages are about |
| | ======================================================================== |
| | |
| | --- target == "website-category" (page: design:website_classification) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | ---------------------------------------------------------------------------------------------------------- ---- --------------- ----------------- --------------------------------------- |
| | USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w 2024 used manual-validation GPT-3.5-turbo-16k |
| | WWW/2025/the-poorest-man-in-babylon-a-longitudinal-study-of-cryptocurrency-investment-sca 2025 used manual-validation Llama3:70b + GPT-4 hybrid |
| | NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets 2026 produced manual-validation GPT-4o with retrieval-augmented prompts |
| | WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia 2025 used manual-validation GPT-4o |
| | USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram 2025 used manual-validation GPT-4 API |
| | |
| | --- target == "javascript" (page: privacy:javascript) --- |
| | no paper in the corpus points an LLM at this target. |
| | |
| | --- target == "fingerprinting-script" (page: privacy:javascript) --- |
| | no paper in the corpus points an LLM at this target. |
| | |
| | --- target == "web-request" (page: privacy:javascript) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | ----------------------------------------------------------------------------------------- ---- --------------- ----------------- ---------------------- |
| | WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere 2026 used held-out-test-set Qwen3-Plus |
| | |
| | --- target == "ip-address" (page: design:ip_classification) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | ---------------------------------------------------------------------------------------- ---- --------------- ----------------- ---------------------- |
| | WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from 2025 used manual-validation GPT-4o |
| | |
| | --- target == "domain" (page: design:ip_classification) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | --------------------------------------------------------------------------------------------- ---- --------------- ----------------- ---------------------- |
| | USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w 2024 used manual-validation GPT-3.5-turbo-16k |
| | IEEE-SP/2025/a-big-step-forward-a-user-centric-examination-of-ios-app-privacy-report-and-enha 2025 used manual-validation GPT-4o |
| | USENIX/2025/dns-flare-a-flush-reload-attack-on-dns-forwarders 2025 used none-reported ChatGPT |
| | |
| | --- target == "cookie" (page: privacy:cookies) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | -------------------------------------------------- ---- --------------- ----------------------------------- ------------------------------------------------- |
| | WWW/2025/semantics-aware-cookie-purpose-compliance 2025 produced+used held-out-test-set+manual-validation GPT-3.5-turbo (custom fine-tuned model) | GPT-3.5 |
| | |
| | --- target == "privacy-policy" (page: (referenced by design:ip_classification)) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | -------------------------------------------------------------------------------------------- ---- ----------------- -------------------------------------------- ------------------------------------------------------------------------------------- |
| | IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots 2024 used manual-validation gpt-4-turbo-2024-04-09 |
| | IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems 2025 produced manual-validation custom LLM-based privacy policy analysis framework |
| | PETS/2025/privacy-settings-of-third-party-libraries-in-android-apps-a-study-of-facebook-sd 2025 used manual-validation ChatGPT |
| | USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto 2025 used held-out-test-set Llama-3.1-70B-Instruct |
| | PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents 2026 used comparison-to-other-method Claude, GPT-4o, Gemini, and DeepSeek (cross-LLM voting) |
| | PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law 2026 used held-out-test-set GPT-5 |
| | PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using 2026 used+produced manual-validation+none-reported GPT 4.1 | custom LLM passage-level classifier |
| | PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p 2025 produced+compared held-out-test-set GPT-3.5 Turbo (Prompt Engineered, fine-tuned) | GPT-4, GPT-4 Turbo, and GPT-3.5 Turbo |
| | PETS/2025/behavr-user-identification-based-on-vr-sensor-data 2025 used comparison-to-other-method GPT-4 |
| | PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica 2026 used+compared manual-validation GPT-4o | GPT-5 | Claude-4.5-Sonnet |
| | PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t 2026 used manual-validation GPT-4o-mini |
| | PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s 2026 used+compared held-out-test-set+comparison-to-other-method LLaMA 3.1 8B Instruct | Mistral-7B |
| | |
| | --- target == "consent-notice" (page: (referenced by design:ip_classification)) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | ------------------------------------------------------------------------------------------ ---- --------------- ----------------- -------------------------------------------------- |
| | CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation 2025 used none-reported Qwen2.5-7B |
| | PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc 2026 used none-reported LlamaGuard, LlamaGuard2, LlamaGuard3, and MD-Judge |
| | |
| | --- target == "dark-pattern" (page: (none)) --- |
| | Paper Year usedOrMentioned Validated against Named model / resource |
| | ------------------------------------------------------------------------- ---- --------------- -------------------------- ---------------------- |
| | PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits 2026 used comparison-to-other-method GPT-5 |
| | |
| | ======================================================================== |
| | VALIDATION |
| | ======================================================================== |
| | 148 of 177 (83.6%) validate the LLM's labels against something. |
| | classification.validation LLM papers naming it Share of 177 |
| | -------------------------- -------------------- ------------ |
| | manual-validation 110 62.1% |
| | none-reported 35 19.8% |
| | comparison-to-other-method 31 17.5% |
| | held-out-test-set 18 10.2% |
| | not-applicable 4 2.3% |
| | cross-validation 1 0.6% |
| | (multi-valued: a paper with several LLM tuples can name several) |
| | |
| | ======================================================================== |
| | NAMED MODELS — free text, so a RANKING, not percentages |
| | ======================================================================== |
| | Model family Papers (of 177) Share |
| | ------------------------------------- --------------- ----- |
| | GPT-4o 46 26.0% |
| | GPT-4 (non-4o) 44 24.9% |
| | GPT-3.5 / GPT-3 / ChatGPT 29 16.4% |
| | UNMAPPED 18 10.2% |
| | Llama family 13 7.3% |
| | Gemini / PaLM 12 6.8% |
| | Unnamed LLM 11 6.2% |
| | OpenAI reasoning / GPT-5 tier 10 5.6% |
| | Qwen family 8 4.5% |
| | DeepSeek family 7 4.0% |
| | Mistral family 4 2.3% |
| | Other open-weight 4 2.3% |
| | Claude family 3 1.7% |
| | Encoder / seq2seq LM (not a chat LLM) 2 1.1% |
| | |
| | --- UNMAPPED residue: 20 distinct strings, printed in full --- |
| | 2x Grok-3 |
| | 2x local LLMs (custom prompts) |
| | 1x BLIP2 |
| | 1x ChatGLM |
| | 1x custom structured prompts with fine-tuned LLMs |
| | 1x foundation LLMs |
| | 1x GLM-4.5 |
| | 1x HtmlLLM-Detector |
| | 1x Kimi |
| | 1x LLaVA |
| | 1x LLaVA-NeXT-Video-7B |
| | 1x PhishLLM |
| | 1x Prompt Instruct |
| | 1x RFCGPT |
| | 1x text-unicorn (custom prompts) |
| | 1x tuned defense models |
| | 1x UGCG-GUARD (InstructBLIP with UGCG-CoT prompts) |
| | 1x Vertex AI text-bison |
| | 1x weighted multi-model ensemble (custom) |
| | 1x YouthSafe |
| | |
| | ======================================================================== |
| | DOES THE PAPER NAME A MODEL YOU COULD RESOLVE? |
| | ======================================================================== |
| | Population: the 175 papers that USED or PRODUCED LLM labels, not the 177 that mention one. |
| | What the strongest thing the paper names is Papers (of 175) Share |
| | ---------------------------------------------------------------- --------------- ----- |
| | A a DATED HOSTED SNAPSHOT (gpt-4-turbo-2024-04-09) 10 5.7% |
| | B an OPEN-WEIGHT CHECKPOINT with a size (Llama-3.1-70B-Instruct) 24 13.7% |
| | A or B — resolvable to an artefact at all 34 19.4% |
| | C a FAMILY with no version (GPT-4, ChatGPT, Mistral) 131 74.9% |
| | D NO IDENTIFIABLE MODEL ("an LLM", "foundation LLMs") 10 5.7% |
| | |
| | >>> B is the stronger kind of pin: an open-weight checkpoint stays |
| | resolvable, and a hosted snapshot only lasts as long as the vendor |
| | serves it. Reported separately for that reason, and because a single |
| | "is it pinned" bucket built from hosted snapshot strings alone |
| | classifies every open-weight paper as unpinned. |
| | >>> PARAM_SIZE accepts a word size (xxl, 7b, mini), so `FLAN-T5-XXL` and |
| | `Mistral Large` count as B. That is a judgement call: they name a |
| | released artefact but not a revision. Every string is below. |
| | |
| | --- bucket A — dated hosted snapshot: 10 distinct strings across 10 papers, printed in full --- |
| | 2x gpt-4-turbo-2024-04-09 [IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots] [IMC/2024/beyond-the-guidelines-assessing-metas-political-ad-moderation-in-the-eu] |
| | 1x ChatGPT (gpt-3.5-turbo-0613) [USENIX/2024/llm-fuzzer-scaling-assessment-of-large-language-model-jailbreaks] |
| | 1x gpt-3.5-turbo-0125 [USENIX/2025/mind-the-inconspicuous-revealing-the-hidden-weakness-in-aligned-llms-refusal-bou] |
| | 1x gpt-3.5-turbo-0613 [NDSS/2025/generating-api-parameter-security-rules-with-llm-for-api-misuse-detection] |
| | 1x GPT-4 (gpt-4-0613) [USENIX/2024/llm-fuzzer-scaling-assessment-of-large-language-model-jailbreaks] |
| | 1x GPT-4o (gpt-4o-2024-11-20) [USENIX/2026/chameleon-channels-measuring-youtube-accounts-repurposed-for-deception-and-profi] |
| | 1x GPT-4o mini (gpt-4o-mini-2024-07-18) [IEEE-SP/2025/gptracker-a-large-scale-measurement-of-misused-gpts] |
| | 1x gpt-4o-mini-2024-07-18 [NDSS/2026/beyond-jailbreak-unveiling-risks-in-llm-applications-arising-from-blurred-capability-boundaries] |
| | 1x o4-mini-2025-04-16 and gemini-2.5-pro [USENIX/2026/sok-philter-uncovering-security-and-functional-gaps-in-ai-based-phishing-website] |
| | 1x OpenAI gpt-4o-2024-05-13 [USENIX/2025/mbfuzzer-a-multi-party-protocol-fuzzer-for-mqtt-brokers] |
| | |
| | --- bucket B — open-weight checkpoint with a size: 25 distinct strings across 24 papers, printed in full --- |
| | 2x Llama-3.1-8B-Instruct |
| | 1x FLAN-T5-XXL |
| | 1x Gemma 3 27B |
| | 1x gpt-oss:20b |
| | 1x Llama 2 70B (custom five-step chain-of-thought) |
| | 1x Llama 3.1 70B (custom prompts) |
| | 1x LLaMA 3.1 8B Instruct |
| | 1x Llama 3.3 70B Instruct |
| | 1x Llama-3-8B |
| | 1x Llama-3.1-70B-Instruct |
| | 1x LLaMA-3.2-11B |
| | 1x LLaMA-3.2-3B |
| | 1x LLaMA-7B fine-tuned SmartInv |
| | 1x Llama3-8B |
| | 1x Llama3:70b + GPT-4 hybrid |
| | 1x Mistral-7B-Instruct-v0.2 |
| | 1x Qwen2-72B-Instruct |
| | 1x Qwen2-7B-Instruct |
| | 1x Qwen2.5-7B |
| | 1x Qwen2.5-Coder-32B-Instruct |
| | 1x Qwen2.5-VL-72B-Instruct |
| | 1x Qwen3-VL-32B-Instruct |
| | 1x T5-11B |
| | 1x Vicuna-33b |
| | 1x Vicuna-7b |
| | |
| | --- bucket C — family only, no version: 88 distinct strings across 131 papers, printed in full --- |
| | 25x GPT-4o |
| | 14x GPT-4 |
| | 6x GPT-4.1 |
| | 6x GPT-4o-mini |
| | 5x ChatGPT |
| | 4x GPT-3.5 |
| | 3x ChatGPT-4 |
| | 3x GPT-3.5-turbo |
| | 3x GPT-5 |
| | 2x GPT-3.5 Turbo |
| | 2x GPT-4 Turbo |
| | 1x BLIP2 |
| | 1x Chat-GPT 3.5 and 4 |
| | 1x ChatGLM |
| | 1x ChatGPT (OpenAI TextCompletion API) |
| | 1x ChatGPT Team |
| | 1x ChatGPT Vision |
| | 1x ChatGPT-4.0 |
| | 1x Claude Sonnet 3.5 |
| | 1x Claude, GPT-4o, Gemini, and DeepSeek (cross-LLM voting) |
| | 1x DeepSeek |
| | 1x DeepSeek R1 |
| | 1x DeepSeek-R1 |
| | 1x DeepSeek-R1 alert-verification module |
| | 1x DeepSeek-V3 |
| | 1x DeepSeek-V3.2-Exp |
| | 1x fine-tuned GPT-4o |
| | 1x fine-tuned GPT-4o with Chain-of-Thought prompting |
| | 1x Gemini 1.5 Pro and Gemini 2.0 Flash-Thinking |
| | 1x Gemini 2.0 Flash |
| | 1x Gemini 2.5 Flash |
| | 1x Gemini 2.5 pro |
| | 1x Gemini 2.5 Pro |
| | 1x Gemini 2.5-Flash |
| | 1x Gemini Ultra |
| | 1x Gemini-2.0-Flash |
| | 1x Gemini-2.5-Flash-Lite and GPT-4o-mini ensemble |
| | 1x Gemini-2.5-pro-preview-05-06 |
| | 1x Gemini-3.1-Pro evaluator |
| | 1x GPT 4.1 |
| | 1x GPT-3 curie |
| | 1x GPT-3.5 and GPT-4o mini |
| | 1x GPT-3.5 DaVinci |
| | 1x GPT-3.5 Turbo (Prompt Engineered, fine-tuned) |
| | 1x GPT-3.5-Turbo |
| | 1x GPT-3.5-turbo (custom fine-tuned model) |
| | 1x GPT-3.5-turbo-16k |
| | 1x GPT-3.5-turbo-instruct |
| | 1x GPT-4 (custom ontology classifier) |
| | 1x GPT-4 (fine-tuned) |
| | 1x GPT-4 API |
| | 1x GPT-4 Judge and Secondary Judge |
| | 1x GPT-4-turbo |
| | 1x GPT-4-Turbo |
| | 1x GPT-4.0 |
| | 1x GPT-4.1 (custom extraction prompt) |
| | 1x GPT-4.1 (custom prompt) |
| | 1x GPT-4.1 with RAG knowledge base |
| | 1x GPT-4.1-mini |
| | 1x GPT-4.5 |
| | 1x GPT-4o (Thematic-LM custom multi-agent system) |
| | 1x GPT-4o and alternative LLM backends |
| | 1x GPT-4o and DeepSeek-V3 |
| | 1x GPT-4o mini |
| | 1x GPT-4o with retrieval-augmented prompts |
| | 1x GPT-4o, LLaMA 3, and Claude 3.5 majority vote |
| | 1x GPT-5 with chain-of-thought prompting |
| | 1x GPT-5-mini |
| | 1x GPT-5-mini with chain-of-thought prompting |
| | 1x GPT-o1 |
| | 1x GPT, Claude, and Gemini evaluators |
| | 1x GPT3 |
| | 1x GPT4o |
| | 1x Grok-3 |
| | 1x Kimi |
| | 1x LLAMA2 |
| | 1x LlamaGuard, LlamaGuard2, LlamaGuard3, and MD-Judge |
| | 1x LLaVA |
| | 1x majority ensemble of Gemini-3.1-Pro, Claude-Opus-4.6, and GPT-5.2 |
| | 1x Mistral |
| | 1x OpenAI gpt-4 |
| | 1x OpenAI o3-mini |
| | 1x Qwen-VL |
| | 1x Qwen3-Plus |
| | 1x RFCGPT |
| | 1x text-unicorn (custom prompts) |
| | 1x UGCG-GUARD (InstructBLIP with UGCG-CoT prompts) |
| | 1x Vertex AI text-bison |
| | |
| | --- bucket D — no identifiable model: 10 distinct strings across 10 papers, printed in full --- |
| | 2x LLM judge (custom) [WWW/2026/inference-cost-attacks-for-retrieval-augmented-large-language-models] [NDSS/2026/when-cache-poisoning-meets-llm-systems-semantic-cache-poisoning-and-its-countermeasures] |
| | 1x foundation LLMs [IEEE-SP/2025/code-speaks-louder-exploring-security-and-privacy-relevant-regional-variations-i] |
| | 1x general-purpose LLM [CCS/2025/needle-in-a-haystack-automated-and-scalable-vulnerability-hunting-in-the-windows] |
| | 1x HtmlLLM-Detector [USENIX/2025/unsafe-llm-based-search-quantitative-analysis-and-mitigation-of-safety-risks-in] |
| | 1x large language models (LLMs) [WWW/2026/opendigger-a-practical-framework-for-assessing-community-health-and-sustainabili] |
| | 1x LLM (unspecified) [WWW/2024/experimental-security-analysis-of-sensitive-data-access-by-browser-extensions] |
| | 1x LLM-as-a-judge (custom) [WWW/2026/arulecon-agentic-security-rule-conversion] |
| | 1x local LLMs (custom prompts) [USENIX/2026/a-large-scale-study-of-personalized-phishing-using-large-language-models] |
| | 1x PhishLLM [USENIX/2025/unsafe-llm-based-search-quantitative-analysis-and-mitigation-of-safety-risks-in] |
| | 1x weighted multi-model ensemble (custom) [WWW/2026/webgeoinfer-structure-free-multi-stage-framework-for-geolocation-inference-from] |
| | |
| | ======================================================================== |
| | IS THE `other` TARGET HIDING ANY OF THIS? |
| | ======================================================================== |
| | 157 llm tuples carry target == "other", across 116 papers. |
| | 157 of 157 (100.0%) state a targetDetail. |
| | |
| | --- targetDetail matching the probe (websit|domain|\burl\b|homepage|page|\bweb\b|script|tracker|cookie|\bsdk\b|first.part|third.part|categor) — 20 tuples, all printed --- |
| | IMC/2023/in-the-room-where-it-happens-characterizing-local-communication-and-threats-in-s |
| | IoT device vendors and categories |
| | PETS/2024/connecting-the-dots-tracing-data-endpoints-in-iot-devices |
| | IoT device vendor and category |
| | USENIX/2024/guardians-of-the-galaxy-content-moderation-in-the-interplanetary-file-system |
| | image content descriptions |
| | CCS/2025/walking-the-last-mile-studying-decompiler-output-correction-in-practice |
| | semantic cluster descriptions |
| | CCS/2025/chekhovs-gun-uncovering-hidden-risks-in-macos-application-sandboxed-pid-domain-s |
| | Sensitive operations in PID-domain service functions |
| | IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems |
| | Action data descriptions assigned to 24 categories and 145 data types |
| | IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems |
| | Previously unclassified Action data descriptions |
| | IEEE-SP/2025/on-the-in-security-of-llm-app-stores |
| | description-instruction consistency |
| | IMC/2025/learning-as-to-organization-mappings-with-borges |
| | favicon and associated final-URL groups |
| | NDSS/2025/hidden-and-lost-control-on-security-design-risks-in-iot-user-facing-matter-controller |
| | UMCCI flaws and user-facing Matter control pages |
| | USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto |
| | personal information categories, purposes, and third-party recipients |
| | WWW/2025/50-shades-of-deceptive-patterns-a-unified-taxonomy-multimodal-detection-and-secu |
| | deceptive-pattern categories in UI images |
| | IEEE-SP/2026/parasites-in-the-toolchain-a-large-scale-analysis-of-attacks-on-the-mcp-ecosyste |
| | MCP server functional categories |
| | WWW/2026/adaptive-location-hierarchy-learning-for-long-tailed-mobility-prediction |
| | hierarchical mappings between location categories, activities, and needs |
| | WWW/2026/adaptive-location-hierarchy-learning-for-long-tailed-mobility-prediction |
| | hierarchical mappings between location categories, activities, and needs |
| | USENIX/2026/a-large-scale-study-of-personalized-phishing-using-large-language-models |
| | relevant person-specific webpage information |
| | NDSS/2026/indicator-of-benignity-an-industry-view-of-false-positive-in-malicious-domain-detection-and-its-mitigation |
| | IOB presence and trustworthiness in web content |
| | PETS/2026/operationalizing-the-motivated-intruder-a-codebook-guided-inference-framework-fo |
| | GDPR personal data, special-category data, and trade-secret sensitivity |
| | WWW/2026/opendigger-a-practical-framework-for-assessing-community-health-and-sustainabili |
| | repository technical domain |
| | WWW/2026/bridging-expert-reasoning-and-llm-detection-a-knowledge-driven-framework-for-mal |
| | threat-intelligence pages containing actionable malicious-code analysis |
| | |
| | >>> Read the list, do not trust the count: the question is WHAT these are, |
| | not whether the probe fired. As of 2026-09-03 none of them is a |
| | website-topic, JavaScript, tracker or cookie classification, so the |
| | per-target zeros on privacy:javascript and design:ip_classification |
| | survive the `other` bucket at keyword recall. A probe is not a read of |
| | 116 papers and cannot be reported as one. |
| | |
| | ======================================================================== |
| | CROSS-PAGE CLAIM CHECKS |
| | ======================================================================== |
| | Page claim What the corpus says |
| | ------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------- |
| | privacy:javascript: "the llm enum fires twice for a JS-classification task" javascript=0 (of 44 papers classifying JS), fingerprinting-script=0 (of 31) |
| | privacy:requests: "exactly one paper classifies web requests with an LLM" web-request=1 (of 258 papers classifying requests) |
| | design:ip_classification: "the llm method fires exactly once" ip-address=1 (of 295 papers classifying IPs) |
| | design:ip_classification: "unlike cookie and policy classification, where LLM methods are now routine" cookie=1 of 53 (1.9%), privacy-policy=12 of 102 (11.8%) — "routine" holds for neither |
| | design:website_classification: "five papers point an LLM at a website category" website-category=5 of 424 (1.2%) |
| | |
| | Denominators in one line, for copying onto a page: |
| | corpus 2010-2026 = 5859 papers, 7 venues (CCS, IMC, NDSS, PETS, USENIX Sec, TheWebConf, IEEE S&P) |
| | papers with >=1 classification tuple = 4439 |
| | method == "llm", any usedOrMentioned = 177 |
| | method == "llm", used or produced = 175 |
| | </file> |
| | |
| | <file text llm_currency_quotecheck-output.txt> |
| | PART A — extraction evidence.quote vs paper.cols.txt |
| | ======================================================================== |
| | PASS = present verbatim after collapsing whitespace. |
| | PASS-ELID = the extractor inserted an elision ("..."); every fragment |
| | either side of it is present, which is the most the check |
| | can establish — the elided middle is unverified. |
| | PASS-LOOSE= present after also dropping punctuation and folding the |
| | PDF's math-italic glyphs (F, M, lambda) to ASCII. A .cols |
| | rendering artefact, not a mismatch with the paper. |
| | PASS-CITE = present once inline citation markers ("[49]") are stripped |
| | from both sides; the extractor drops them. |
| | PASS-NGRAM= >=80% of the quote's word 5-grams occur in the text. The sentence |
| | is in the paper, but either the extraction reworded a word or |
| | two, or paper.cols.txt interleaved a float (table caption, |
| | running header) into it. The broken n-grams are itemised |
| | below so every such call is checkable by hand. |
| | FAIL = not present under any of the above. |
| | |
| | RESCUED IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots [privacy-policy] 26/36 -> 36/36 PDF "we design a set of task prompts for an AI chatbot to split scraped content into …" |
| | PASS USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w [website-category] "This prompt is fed into a language model using a chain-of-thought approach, enfo…" |
| | PASS-NGRAM IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems [privacy-policy] 10/10 5-grams "We develop an LLM-based framework to check the consistency of data collection di…" |
| | PASS PETS/2025/privacy-settings-of-third-party-libraries-in-android-apps-a-study-of-facebook-sd [privacy-policy] "This allows for an automated approach that facilitates analysis of privacy polic…" |
| | PASS USENIX/2025/evaluating-privacy-policies-under-modern-privacy-laws-at-scale-an-llm-based-auto [privacy-policy] "A coverage task assesses whether a privacy policy segment covers the required co…" |
| | PASS WWW/2025/the-poorest-man-in-babylon-a-longitudinal-study-of-cryptocurrency-investment-sca [website-category] "We manually categorized a random sample of 300 websites that had passed the cont…" |
| | PASS-LOOSE PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents [privacy-policy] "For each privacy policy, we let each formalizer auto-formalize it into a structu…" |
| | PASS-LOOSE NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets [website-category] "Based on the extracted evidence, CtPhishCapture assigns each webpage a phishing …" |
| | PASS-CITE WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere [web-request] "we employ the large language model Qwen3 as the underlying model Mλ for LLM-base…" |
| | PASS-ELID WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from [ip-address] "This component ... parses OSCTIs to identify and extract IoCs, notably IP addres…" |
| | PASS-LOOSE WWW/2025/semantics-aware-cookie-purpose-compliance [cookie] "Subsequently, 88% of the labeled cookies are used as training data for the fine-…" |
| | PASS WWW/2025/semantics-aware-cookie-purpose-compliance [cookie] "Coover utilizes the OpenAI client.chat.completions.create() API, passing the spe…" |
| | PASS PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law [privacy-policy] "For each policy, we issue a single inference request to the model and require it…" |
| | PASS WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia [website-category] "To evaluate our classification methods, we manually annotated a sample of 500 we…" |
| | RESCUED PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] 11/20 -> 16/20 PDF "On a manually labelled sample of 340 randomly selected documents ... using GPT 4…" |
| | PASS-LOOSE PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] "This LLM-based classifier predicts the set of labels Lp relevant to each passage" |
| | RESCUED CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation [consent-notice] 9/14 -> 14/14 PDF "we used a more efficient text-only model (Qwen2.5-7B) to detect the presence of …" |
| | RESCUED PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] 13/21 -> 21/21 PDF "We randomly reserved 70% of the manual annotations to constitute our training da…" |
| | PASS PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] "For the prompted non-fine-tuned LLMs, we used GPT-4, GPT-4 Turbo, and GPT-3.5 Tu…" |
| | PASS PETS/2025/behavr-user-identification-based-on-vr-sensor-data [privacy-policy] "We also use simple string matching to search for relevant content." |
| | PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We use GPT-4o to fetch and segment the policy text, extracting structured data p…" |
| | PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" |
| | PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" |
| | RESCUED PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t [privacy-policy] 15/19 -> 19/19 PDF "A researcher manually verified whether each LLM classification matched the corre…" |
| | PASS-NGRAM PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 19/20 5-grams "the system outputs two sets of data types, C data (collected) and S data (shared…" |
| | RESCUED PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 7/11 -> 11/11 PDF "both models were evaluated using the same preprocessing pipeline, chunking strat…" |
| | PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "GPT-4 added four new categories, bringing the total number of categories to 19. …" |
| | PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "We evaluated 100 randomly selected cases and found GPT-4's predictions to be acc…" |
| | PASS PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits [dark-pattern] "The agent also logs any failures or issues encountered during automation, such a…" |
| | PASS-ELID PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc [consent-notice] "Each model independently evaluated these conversations by predicting moderation …" |
| | |
| | Soft passes, itemised: |
| | IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems [privacy-policy] n-gram: 10/10 word 5-grams present, broken at: |
| | PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents [privacy-policy] punctuation/glyph only |
| | NDSS/2026/ctphishcapture-uncovering-credential-theft-based-phishing-scams-targeting-cryptocurrency-wallets [website-category] punctuation/glyph only |
| | WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere [web-request] inline citation marker dropped by the extractor |
| | WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from [ip-address] elided into 2 fragments |
| | WWW/2025/semantics-aware-cookie-purpose-compliance [cookie] punctuation/glyph only |
| | PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] punctuation/glyph only |
| | PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] n-gram: 19/20 word 5-grams present, broken at: the system outputs two sets |
| | PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc [consent-notice] elided into 2 fragments |
| | |
| | PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext. |
| | |
| | |
| | PART B — figures the pages publish, located in the full text |
| | ======================================================================== |
| | PASS WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia <<an accuracy of 92%, comparable to commercial website classification services>> |
| | ctx: … a sample of 500 websites from the Tranco list, categorizing them into "shopping" and "non-shopping." GPT-4o, when prompted with URLs and screenshots, achieved an accuracy of 92%, comparable to commercial website classification services [49] (see Appendix D for details). Therefore, we use this configuration throughout our work. We subsequently crawl the shopping websites to collect terms and c… |
| | |
| | PASS WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere <<the LLM-driven annotation method attains an>> |
| | ctx: … of 92.24% in detecting connected requests and 84.49% for isolated requests, outperforming state-ofthe-art graph-based tracking detection methods. Furthermore, the LLM-driven annotation method attains an 𝐹 1 -Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods, underscoring the critical role of LLMs… |
| | |
| | PASS WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere <<-Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods>> |
| | ctx: ….49% for isolated requests, outperforming state-ofthe-art graph-based tracking detection methods. Furthermore, the LLM-driven annotation method attains an 𝐹 1 -Score of 98.19% on the ground-truth set, a substantial improvement over the 55.14% achieved by rule-based methods, underscoring the critical role of LLMs in generating large-scale, high-quality labels. Furthermore, our TGNN framework revealed several concerning findings in… |
| | |
| | PASS WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere <<we employ the large language model Qwen3>> |
| | ctx: … to this component is the contextual information C of edge 𝑒 (𝑢,𝑣,𝑖 ) , and the output is its tracking attribute T (𝑒 (𝑢,𝑣,𝑖 ) ) ∈ {0, 1}. To this end, we employ the large language model Qwen3 [49] as the underlying model 𝑀𝜆 for LLM-based labeling. WWW '26, April 13-17, 2026, Dubai, United Arab Emirates. Design Philosophy. To achieve this goal, the… |
| | |
| | PASS WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere <<at least 16.74% of websites engage in pixel tracking via major third-party infrastructures>> |
| | ctx: …tially outperforms existing baselines, while the LLM-based annotation achieves accuracy comparable to expert curation. Our large-scale measurement reveals that at least 16.74% of websites engage in pixel tracking via major third-party infrastructures, establishing cross-domain tracking as a pervasive practice in the ∗ Also affiliated with School of Cyber Security, University of Chinese Academy of Sciences, … |
| | |
| | PASS WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere <<formulates pixel tracking detection as an edge classification task>> |
| | ctx: …ften fail to capture the cross-site associations that enable pixel tracking to evade detection. To address this limitation, we introduce TGNN, a framework that formulates pixel tracking detection as an edge classification task on a Tracking Directed Graph (TDG), which models thirdparty associations across websites. TGNN encodes HTTP traffic into structured quadruples and learns both … |
| | |
| | PASS IMC/2025/learning-as-to-organization-mappings-with-borges <<GPT-4o-mini>> |
| | ctx: … the Information Extraction (IE) pro- We implemented this approach in Borges, utilizing OpenAI's cess by applying a dropout filter to enhance model accuracy by GPT-4o-mini [40] with a temperature set to 0 and a Top P probaonly considering text fields - either notes or aka entries - con- bility mass of 1. This setup ensures the mo… |
| | |
| | PASS WWW/2025/semantics-aware-cookie-purpose-compliance <<fine-tun>> |
| | ctx: …purpose compared to other features such as expiration time. Coover decomposes the cookie value into primitive segments representing minimal semantic units, and fine-tunes a GPT-3.5 model to automatically interpret their value-inferred semantics. Based on the interpretation, it classifies cookies into four GDPR-defined purposes… |
| | </file> |
| | |
| | <file text report_website_classification-output.txt> |
| | ======================================================================== |
| | POPULATION |
| | ======================================================================== |
| | corpus 5859 |
| | papers with >=1 classification tuple 4439 |
| | papers with >=1 target=website-category tuple 424 |
| | ... of which used or produced it 424 |
| | ... only compared/mentioned it 0 |
| | |
| | --- What "website-category" actually means, folded by task --- |
| | (papers, of the 424 above; multi-valued, so shares do not sum to 100%) |
| | Task Papers Share |
| | ------------------------- ------ ----- |
| | topic-or-industry 330 77.8% |
| | abuse-verdict 63 14.9% |
| | website-fingerprinting 36 8.5% |
| | hosting-infrastructure 8 1.9% |
| | page-type 7 1.7% |
| | technology-fingerprinting 6 1.4% |
| | third-party-role 5 1.2% |
| | |
| | >>> PAGE POPULATION: 330 papers that assign a topic/industry/sector label |
| | to a website or domain, out of 5859 in the corpus (5.6%). |
| | Excluded: 94 papers whose only website-category tuple is a |
| | website-fingerprinting attack, an abuse verdict, a technology |
| | fingerprint, a page-type label, a hosting label or a third-party role. |
| | |
| | ======================================================================== |
| | WHERE AND WHEN |
| | ======================================================================== |
| | --- Venue (of 330) --- |
| | Venue Papers Share of population Share of that venue |
| | ------- ------ ------------------- ------------------- |
| | WWW 71 21.5% 8.4% |
| | USENIX 58 17.6% 4.1% |
| | IMC 50 15.2% 7.8% |
| | CCS 48 14.5% 4.8% |
| | PETS 43 13.0% 8.4% |
| | NDSS 35 10.6% 5.0% |
| | IEEE-SP 25 7.6% 3.3% |
| | |
| | --- Over time (denominator = all corpus papers in the bucket) --- |
| | Years Corpus Categorised websites Share |
| | ---------- ------ -------------------- ----- |
| | 2010–2013 511 28 5.5% |
| | 2014–2017 769 56 7.3% |
| | 2018–2021 1439 92 6.4% |
| | 2022–2024 1955 89 4.6% |
| | 2025–2026* 1185 65 5.5% |
| | |
| | ======================================================================== |
| | METHOD (enum — publishable as a rough share; run-to-run stability 58%) |
| | ======================================================================== |
| | Method Papers Share |
| | ------------------- ------ ----- |
| | third-party-service 131 39.7% |
| | manual-labelling 68 20.6% |
| | heuristic-rules 59 17.9% |
| | supervised-ml 46 13.9% |
| | curated-database 46 13.9% |
| | unsupervised-ml 14 4.2% |
| | regex-or-signature 9 2.7% |
| | other 7 2.1% |
| | graph-analysis 3 0.9% |
| | static-analysis 2 0.6% |
| | llm 2 0.6% |
| | dynamic-analysis 1 0.3% |
| | blocklist 1 0.3% |
| | (of 330; multi-valued) |
| | |
| | --- Method over time, as a share of that bucket's categorising papers --- |
| | Method 2010–2013 2014–2017 2018–2021 2022–2024 2025–2026* |
| | ------------------- --------- ---------- ---------- ---------- ---------- |
| | third-party-service 7 (25.0%) 19 (33.9%) 40 (43.5%) 44 (49.4%) 21 (32.3%) |
| | supervised-ml 4 (14.3%) 6 (10.7%) 15 (16.3%) 13 (14.6%) 8 (12.3%) |
| | manual-labelling 5 (17.9%) 15 (26.8%) 19 (20.7%) 18 (20.2%) 11 (16.9%) |
| | heuristic-rules 8 (28.6%) 7 (12.5%) 14 (15.2%) 15 (16.9%) 15 (23.1%) |
| | curated-database 4 (14.3%) 12 (21.4%) 10 (10.9%) 8 (9.0%) 12 (18.5%) |
| | llm 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 2 (3.1%) |
| | |
| | --- Control 1: the 2025–2026 window split by year --- |
| | Method 2022–2024 2025 2026* |
| | ------------------- ---------- ---------- --------- |
| | third-party-service 44 (49.4%) 17 (37.8%) 4 (20.0%) |
| | supervised-ml 13 (14.6%) 3 (6.7%) 5 (25.0%) |
| | manual-labelling 18 (20.2%) 9 (20.0%) 2 (10.0%) |
| | heuristic-rules 15 (16.9%) 11 (24.4%) 4 (20.0%) |
| | curated-database 8 (9.0%) 8 (17.8%) 4 (20.0%) |
| | llm 0 (0.0%) 2 (4.4%) 0 (0.0%) |
| | bucket sizes: 2022–2024=89 2025=45 2026*=20 |
| | |
| | --- Control 2: both windows restricted to venues that have a 2026 slice --- |
| | venues with any 2026 paper: IEEE-SP, NDSS, PETS, USENIX, WWW |
| | venues with none (2026 not held / not selected): CCS, IMC |
| | Method 2022–2024 2025–2026* |
| | ------------------- ---------- ---------- |
| | third-party-service 34 (54.0%) 16 (28.6%) |
| | supervised-ml 11 (17.5%) 7 (12.5%) |
| | manual-labelling 13 (20.6%) 11 (19.6%) |
| | heuristic-rules 8 (12.7%) 11 (19.6%) |
| | curated-database 6 (9.5%) 12 (21.4%) |
| | llm 0 (0.0%) 2 (3.6%) |
| | bucket sizes: 2022–2024=63 2025–2026*=56 |
| | |
| | --- Fisher's exact test on each control (two-sided) --- |
| | Cut Method Recent Baseline change (pp) p (two-sided) Read as |
| | ------------------------------------------ ------------------- ------------- ------------- ----------- ------------- ------------------------------ |
| | 2025 alone vs 2022–2024 third-party-service 17/45 (37.8%) 44/89 (49.4%) -11.7 0.2704 NOT SUPPORTED |
| | 2025 alone vs 2022–2024 curated-database 8/45 (17.8%) 8/89 (9.0%) +8.8 0.1630 NOT SUPPORTED |
| | 2026 alone vs 2022–2024 third-party-service 4/20 (20.0%) 44/89 (49.4%) -29.4 0.0236 supported |
| | 2026 alone vs 2022–2024 curated-database 4/20 (20.0%) 8/89 (9.0%) +11.0 0.2274 NOT SUPPORTED |
| | 2025–2026 vs 2022–2024, all venues third-party-service 21/65 (32.3%) 44/89 (49.4%) -17.1 0.0471 supported |
| | 2025–2026 vs 2022–2024, all venues curated-database 12/65 (18.5%) 8/89 (9.0%) +9.5 0.0947 NOT SUPPORTED (0.05 < p < 0.1) |
| | 2025–2026 vs 2022–2024, five-venue control third-party-service 16/56 (28.6%) 34/63 (54.0%) -25.4 0.0057 supported |
| | 2025–2026 vs 2022–2024, five-venue control curated-database 12/56 (21.4%) 6/63 (9.5%) +11.9 0.0792 NOT SUPPORTED (0.05 < p < 0.1) |
| | >>> Read this before quoting either movement. The 2025-alone control is |
| | not evidence at n=45, and curated-database does not clear p<0.05 on |
| | any cut. Only the third-party-service drop is statistically supported. |
| | |
| | --- what the `curated-database` row actually names, by window --- |
| | 2022–2024 CCS/2022/understanding-security-issues-in-the-nft-ecosystem DappRadar |
| | 2022–2024 CCS/2022/do-opt-outs-really-opt-me-out pornhosts blocklist |
| | 2022–2024 PETS/2022/who-knows-i-like-jelly-beans-an-investigation-into-search-privacy WebPulse |
| | 2022–2024 USENIX/2022/an-audit-of-facebooks-political-ad-policy-enforcement external political, government, media, and issue-page sources |
| | 2022–2024 USENIX/2022/many-roads-lead-to-rome-how-packet-headers-influence-dns-censorship-measurement Citizen Lab Block List |
| | 2022–2024 USENIX/2022/pool-inference-attacks-on-local-differential-privacy-quantifying-the-privacy-gua AllSides Media Bias Chart |
| | 2022–2024 WWW/2023/who-funds-misinformation-a-systematic-analysis-of-the-ad-related-profit-routines MediaBias/FactCheck |
| | 2022–2024 IEEE-SP/2022/investigating-influencer-vpn-ads-on-youtube YouTube category labels |
| | 2025–2026* IEEE-SP/2025/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on Tracker Radar Entity List |
| | 2025–2026* USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi Cloudflare Radar |
| | 2025–2026* PETS/2025/empirically-measuring-data-localization-in-the-eu SimilarWeb |
| | 2025–2026* USENIX/2025/demystifying-the-in-security-of-qr-code-based-login-in-real-world-deployments Symantec SiteReview |
| | 2025–2026* WWW/2025/the-agenda-setting-function-of-social-media AllSides |
| | 2025–2026* WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e MediaBias/FactCheck |
| | 2025–2026* NDSS/2026/dom-xss-detection-via-webpage-interaction-fuzzing-and-url-component-synthesis IAB taxonomy |
| | 2025–2026* USENIX/2026/patch-guided-vulnerability-detection-extracting-java-api-security-rules-via-atta predefined source rules (custom) |
| | 2025–2026* WWW/2025/before-after-the-effect-of-eus-2022-code-of-practice-on-disinformation MediaBias/FactCheck | Science Feedback |
| | 2025–2026* WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere Homepage2Vec |
| | 2025–2026* IEEE-SP/2025/the-file-that-contained-the-keys-has-been-removed-an-empirical-analysis-of-secre NAICSlite |
| | 2025–2026* NDSS/2026/revealing-the-secret-power-how-algorithms-can-influence-content-visibility-on-twitter-x Media Bias/Fact Check (MBFC) |
| | >>> If this row is mostly commercial vendors and media-bias raters rather |
| | than open directories, "curated databases returned" is a claim about |
| | an enum, not about the field returning to open label sources. |
| | |
| | --- 2022–2024 third-party-service share by venue, within the population --- |
| | Venue third-party-service population 2022–2024 Share Has a 2026 slice? |
| | ------- ------------------- -------------------- ----- ----------------- |
| | CCS 1 12 8.3% NO |
| | IEEE-SP 5 9 55.6% yes |
| | IMC 9 14 64.3% NO |
| | NDSS 1 5 20.0% yes |
| | PETS 10 16 62.5% yes |
| | USENIX 10 21 47.6% yes |
| | WWW 8 12 66.7% yes |
| | |
| | --- Where the population sits, by venue and year (the mix itself) --- |
| | Venue pop 2022–2024 pop 2025 pop 2026 corpus 2022–2024 corpus 2025 corpus 2026 |
| | ------- ------------- -------- -------- ---------------- ----------- ----------- |
| | CCS 12 4 0 282 101 0 |
| | IEEE-SP 9 4 0 303 117 28 |
| | IMC 14 5 0 190 79 0 |
| | NDSS 5 4 9 167 119 163 |
| | PETS 16 9 5 170 67 88 |
| | USENIX 21 10 3 559 224 69 |
| | WWW 12 9 3 284 63 67 |
| | |
| | --- classification.method == "llm" anywhere in the corpus, by year --- |
| | Year Papers using an LLM to classify Corpus that year Share |
| | ---- ------------------------------- ---------------- ----- |
| | 2023 2 719 0.3% |
| | 2024 27 690 3.9% |
| | 2025 77 770 10.0% |
| | 2026 71 415 17.1% |
| | total: 177 papers of 5859. 2025 and 2026 are provisional venue-years — CCS/IMC 2026 not held, IEEE S&P/WWW 2026 incompletely selected — so the last two rows are floors, not ceilings. |
| | |
| | --- What those papers point an LLM at --- |
| | Target Papers Share of 177 |
| | ------------------- ------ ------------ |
| | other 116 65.5% |
| | user-generated-text 32 18.1% |
| | vulnerability 14 7.9% |
| | privacy-policy 12 6.8% |
| | mobile-app 6 3.4% |
| | website-category 5 2.8% |
| | domain 3 1.7% |
| | network-traffic 2 1.1% |
| | consent-notice 2 1.1% |
| | web-request 1 0.6% |
| | ip-address 1 0.6% |
| | cookie 1 0.6% |
| | email-message 1 0.6% |
| | dark-pattern 1 0.6% |
| | 148 of 177 (83.6%) validate the LLM's labels against something. |
| | 2 of 177 use an LLM to assign a website a TOPIC category. |
| | |
| | ======================================================================== |
| | WHICH NAMED RESOURCE (free text — folded, reported as a ranking) |
| | ======================================================================== |
| | --- By family (of 330 papers; multi-valued) --- |
| | Family Papers Share |
| | ---------------------------------------------- ------ ----- |
| | Hand-built rules / keywords 87 26.4% |
| | Commercial security / content-filtering vendor 65 19.7% |
| | Generic ML classifier, no named service 53 16.1% |
| | Human labelling 41 12.4% |
| | Commercial marketing categorisation 36 10.9% |
| | Cloud / platform categorisation API 25 7.6% |
| | Human-curated directory or test list 21 6.4% |
| | Reputation aggregator 14 4.2% |
| | Pretrained language model, fine-tuned 6 1.8% |
| | Large language model 2 0.6% |
| | |
| | --- Named third-party services and datasets only, ranked --- |
| | Service Family Papers Spellings folded |
| | ----------------------------------- ------------------ ------ ---------------- |
| | McAfee security-filtering 23 18 |
| | Symantec / Blue Coat security-filtering 14 10 |
| | Alexa marketing 12 8 |
| | FortiGuard security-filtering 12 7 |
| | Media bias raters (AllSides, MB/FC) curated-directory 10 9 |
| | Cloudflare domain categories platform-api 10 6 |
| | Webshrinker marketing 9 2 |
| | DMOZ / Open Directory Project curated-directory 8 6 |
| | Google Cloud Natural Language platform-api 8 6 |
| | SimilarWeb marketing 7 2 |
| | VirusTotal aggregator 7 3 |
| | Google Ads keyword/topic tools marketing 6 5 |
| | Trend Micro security-filtering 4 3 |
| | Cisco (OpenDNS / IronPort) security-filtering 3 3 |
| | Cyren security-filtering 3 3 |
| | Citizen Lab test list curated-directory 3 3 |
| | Forcepoint / Websense security-filtering 3 2 |
| | Google Topics API platform-api 3 3 |
| | Google Safe Browsing aggregator 2 2 |
| | Web of Trust aggregator 2 2 |
| | IBM Watson platform-api 2 2 |
| | WhoisXMLAPI aggregator 2 2 |
| | Klazify marketing 2 1 |
| | SafeDNS security-filtering 2 2 |
| | Bing classifier platform-api 1 1 |
| | Boson NLP platform-api 1 1 |
| | HerdProtect aggregator 1 1 |
| | Juniper security-filtering 1 1 |
| | Website Categorify marketing 1 1 |
| | YouTube Data API platform-api 1 1 |
| | Netstar security-filtering 1 1 |
| | 155 of 330 papers (47.0%) name at least one third-party service or dataset. |
| | |
| | --- Everything else, ranked --- |
| | Resource Family Papers |
| | ----------------------------------------------- ------------- ------ |
| | Unnamed custom method hand-built 58 |
| | Manual labelling by the authors or hired coders manual 41 |
| | Keyword / regex rules (custom) hand-built 32 |
| | Random forest generic-ml 13 |
| | Clustering (k-means, DBSCAN, …) generic-ml 10 |
| | SVM generic-ml 9 |
| | Neural network (CNN/LSTM/MLP) generic-ml 6 |
| | LDA / topic model generic-ml 6 |
| | Logistic / linear regression generic-ml 5 |
| | k-NN generic-ml 3 |
| | BERT pretrained-lm 3 |
| | Hidden Markov / Bayesian network generic-ml 2 |
| | Naive Bayes generic-ml 2 |
| | LLM (GPT family) llm 2 |
| | GloVe pretrained-lm 1 |
| | Semi-supervised (co-/self-training, PU) generic-ml 1 |
| | BART / zero-shot NLI classifier pretrained-lm 1 |
| | Homepage2Vec pretrained-lm 1 |
| | |
| | --- The spellings each canonical name absorbed (the reason folding matters) --- |
| | Unnamed custom method (48): |
| | CodeQL query suite (custom) |
| | DNS Scanner (custom) |
| | DOMetric (custom) |
| | IAB taxonomy mapping (custom) |
| | Paraphrase-Multilingual-MPNet-base-v2 classifier (custom) |
| | TextCNN (custom) |
| | Word2Vec (custom) |
| | custom |
| | custom 154-class tag mapping |
| | custom 63-pattern list |
| | custom FPR/TPR platform-identification process |
| | custom HTTPS-response classification |
| | custom IMSS identifier |
| | custom IPv6-readiness classification |
| | custom NSKeyword Identifier |
| | custom PASSKEYS-RADAR detectors |
| | custom Prebid.js detector |
| | custom STEM/humanities classification |
| | custom SnorCall labeling functions |
| | custom WordPress-version and HTML-source checks |
| | custom analyst labeling |
| | custom campaign infection-vector classification |
| | custom category coding |
| | custom content whitelist |
| | custom credential-taking-intent module |
| | custom dual-stack completeness classification |
| | custom fake AV detector |
| | custom gray-website category scheme |
| | custom image classifier |
| | custom inference procedure |
| | custom page-category scheme |
| | custom policy-based categorization |
| | custom privacy-score thresholds |
| | custom purpose classification |
| | custom sector categorization |
| | custom seven-category scheme |
| | custom social-profile matching |
| | custom static-file hash knowledge base |
| | custom suspicious-content classifier |
| | custom topic categories |
| | custom web crawler |
| | custom webpage curation checks |
| | custom word-based classifier |
| | handcrafted classifier (custom) |
| | priority-based mapping (custom) |
| | roberta-large (custom fine-tuned model) |
| | semantic analyzer (custom) |
| | strict SAT-vendor criteria (custom) |
| | Keyword / regex rules (custom) (33): |
| | 90 manually crafted application-identification signatures |
| | Rule-Based Detector (custom) |
| | Unicode-based language heuristic (custom) |
| | custom 82-keyword content filter |
| | custom HTML heuristics |
| | custom URL parsing |
| | custom government-hostname regular expression |
| | custom heuristic |
| | custom heuristic rules |
| | custom heuristics |
| | custom keyword matching |
| | custom keyword searches |
| | custom keyword-matching heuristics |
| | custom keyword-stuffing detector |
| | custom page-classification priority rules |
| | custom provider keywords and DOM signatures |
| | custom redirect-chain rules |
| | custom regexes |
| | custom regular expression |
| | custom regular expressions and keyword lists |
| | custom regular expressions and known SSO API endpoints |
| | custom regular-expression keyword rules |
| | custom response-header signatures |
| | custom store-detection heuristics |
| | custom two-filter heuristic |
| | custom www-label rule |
| | keyword matching |
| | manually curated keyword set (custom) |
| | predefined source rules (custom) |
| | regular expressions (custom) |
| | simple DNS name heuristic |
| | simple handcrafted URL-based mapping rules |
| | two-phase content-grouping heuristic (custom) |
| | Manual labelling by the authors or hired coders (32): |
| | BuzzFeed News journalists and fact checkers |
| | Google reviewer annotations |
| | Mechanical Turk |
| | author labeling |
| | custom Facebook-ad codebook |
| | custom manual categorization |
| | custom manual classification |
| | custom manual determination |
| | custom manual inspection |
| | custom manual product identification |
| | custom manual sanity-check |
| | custom manual service profiling |
| | custom manual verification |
| | custom relevance codebook |
| | custom website-category codebook |
| | four privacy-field experts |
| | graduate-level annotators |
| | graduate-student annotators |
| | manual CMS investigation (custom) |
| | manual categorization by authors |
| | manual categorization by researchers |
| | manual company classification (custom) |
| | manual inspection (custom) |
| | manual review by authors |
| | manual review of third-party marketing materials |
| | one-author manual verification |
| | researcher manual categorization |
| | researcher-created 15-category codebook |
| | three-person manual verification |
| | two human coders |
| | two reviewers |
| | two-author manual review |
| | McAfee (18): |
| | McAfee |
| | McAfee Real-Time Database |
| | McAfee SiteAdvisor |
| | McAfee SiteLookup |
| | McAfee SmartFilter |
| | McAfee SmartFilter Internet Database |
| | McAfee URL Ticketing System |
| | McAfee URL categorization service |
| | McAfee categorization service |
| | McAfee service |
| | McAfee's Threat Intelligence |
| | McAfee's TrustedSource |
| | McAfee's URL Ticketing System |
| | McAfee's URL Ticketing service |
| | McAfee's URL categorization service |
| | McAfee's domain categorization API |
| | McAfee's website categorization tool |
| | Trellix TrustedSource |
| | Clustering (k-means, DBSCAN, …) (10): |
| | Affinity Propagation |
| | Algorithm 1 (custom clustering algorithm) |
| | DBScan |
| | URL keyword clustering (custom) |
| | content clustering tool (custom q-gram similarity) |
| | custom manual clustering |
| | diffusion wavelet clustering (custom) |
| | k-means clustering (custom feature pipeline) |
| | perceptual hash and simhash (custom pipeline) |
| | perceptual hash function |
| | Symantec / Blue Coat (10): |
| | Blue Coat Systems genre database |
| | Symantec RuleSpace |
| | Symantec Rulespace |
| | Symantec SiteReview |
| | Symantec Sitereview |
| | Symantec WebPulse |
| | Symantec WebPulse Site Review |
| | Symantec risk level calculator |
| | WebPulse |
| | WebPulse Site Review |
| | Media bias raters (AllSides, MB/FC) (9): |
| | AllSides |
| | AllSides Media Bias Chart |
| | AllSides and Media Bias/Fact Check |
| | Allsides |
| | Media Bias Fact Check (MBFC) |
| | Media Bias/Fact Check |
| | Media Bias/Fact Check (MBFC) |
| | MediaBias/FactCheck |
| | Science Feedback |
| | SVM (8): |
| | Linear SVM (custom) |
| | SVM (custom) |
| | SVM (custom, scikit-learn) |
| | SVM classifier (custom) |
| | Support Vector Machines (RBF; custom) |
| | eight custom classifiers: KNN, SVM, Naive Bayes, and decision tree |
| | linear SVM (Liblinear) |
| | one-class SVM (custom) |
| | |
| | --- RESIDUE: 33 distinct strings the fold did not classify --- |
| | 1 Taobao main-business categories |
| | 1 search-redirection attack list |
| | 1 Rakuten Popshops |
| | 1 SNET pillar portals |
| | 1 Googlebot |
| | 1 Censys |
| | 1 AppBrain API |
| | 1 HBDetector |
| | 1 Zeileis et al.'s structural change detection algorithm |
| | 1 Van Wegberg et al. [24] classification |
| | 1 DAppTotal |
| | 1 DappRadar |
| | 1 pornhosts blocklist |
| | 1 external political, government, media, and issue-page sources |
| | 1 multiple domain classification services |
| | 1 public classification service from the security vendor |
| | 1 Google ML model implemented in Chrome |
| | 1 ORES topic routing |
| | 1 default target categories and direct visits |
| | 1 Tracker Radar Entity List |
| | 1 Arkham Intel |
| | 1 ccTLD/WHOIS country criterion |
| | 1 NextDNS Piracy Blocklist |
| | 1 IAB taxonomy |
| | 1 ScamAdviser API |
| | 1 multi-feature probabilistic filtering algorithm |
| | 1 vDF and vRF |
| | 1 urlblacklist.com |
| | 1 NudeNet |
| | 1 NAICSlite |
| | 1 DNS-based hosting-provider identification |
| | 1 open-source topic classifier |
| | 1 YouTube category labels |
| | |
| | --- Named categorisation services used by the NON-topic tasks --- |
| | abuse-verdict: 13 papers |
| | 9 VirusTotal |
| | 3 Google Safe Browsing |
| | 1 McAfee |
| | 1 URLScan |
| | third-party-role: 1 papers |
| | 1 Google Safe Browsing |
| | |
| | ======================================================================== |
| | WHAT THESE PAPERS DO AND DO NOT REPORT |
| | ======================================================================== |
| | Question Papers Of Share |
| | -------------------------------------- ------ --- ----- |
| | names the taxonomy it classified into 248 330 75.2% |
| | names a ground-truth source 169 330 51.2% |
| | reports any validation of the labels 123 330 37.3% |
| | explicitly reports no validation 98 330 29.7% |
| | SENTINEL: validation is not-applicable 109 330 33.0% |
| | check: 123 + 98 + 109 = 330 (should be 330) |
| | |
| | --- Cross-population comparisons quoted on the page --- |
| | groundTruthSource stated, ALL classifying papers: 3234/4439 = 72.9% |
| | Vallina et al. (IMC 2020, "Mis-shapes, Mistakes, Misfits") present in the extraction: false — it is in data/corpus2/.meta/IMC-2020.json with a real DOI but has no data/fulltext entry, so the page's central citation is not in the population the page measures. |
| | IEEE S&P is now fully retrieved (780 of 780 selected papers, commit 8a6b843 2026-08-11). Its share of this population went 3 papers (1.2%) -> 25 (7.6%) as a result: the old 43%-retrieval caveat WAS the explanation for its low share, and is now resolved. |
| | Free-text names agree ~20% run-to-run and classification.method ~58% — data/extract/README.md, which is why service names are reported as folded rankings and method as a rough share. |
| | |
| | --- How the labels were validated (of the papers that validated) --- |
| | Validation Papers Share of population |
| | -------------------------- ------ ------------------- |
| | manual-validation 92 27.9% |
| | cross-validation 21 6.4% |
| | held-out-test-set 13 3.9% |
| | comparison-to-other-method 10 3.0% |
| | |
| | Papers naming >=2 distinct third-party categorisation services: 9 of 155 that name any (5.8%). |
| | WWW/2013/two-years-of-short-urls-internet-measurement-security-threats-and-countermeasure |
| | WWW/2014/adscape-harvesting-and-analyzing-online-display-ads |
| | WWW/2014/from-devices-to-people-attribution-of-search-activity-in-multi-user-settings |
| | CCS/2014/your-online-interests-pwned-a-pollution-attack-against-targeted-advertising |
| | NDSS/2015/seven-months-worth-of-mistakes-a-longitudinal-study-of-typosquatting-abuse |
| | CCS/2020/censored-planet-an-internet-wide-longitudinal-censorship-observatory |
| | WWW/2023/who-funds-misinformation-a-systematic-analysis-of-the-ad-related-profit-routines |
| | PETS/2024/interest-disclosing-mechanisms-for-advertising-are-privacy-exposing-not-preservi |
| | PETS/2024/what-does-it-mean-to-be-creepy-responses-to-visualizations-of-personal-browsing |
| | |
| | --- Named taxonomies (free text, coarse fold) --- |
| | Taxonomy Papers Share of the 248 that name one |
| | ------------------------------------------ ------ ------------------------------ |
| | a bespoke scheme, class count not given 187 75.4% |
| | a vendor's own category list 43 17.3% |
| | IAB 9 3.6% |
| | DMOZ / ODP / Curlie 7 2.8% |
| | a bespoke scheme with a stated class count 7 2.8% |
| | Google AdWords / Topics 4 1.6% |
| | a standard industry code (NAICS/SIC/NACE) 1 0.4% |
| | </file> |
| | |
| | ==== 12.10 Review pass, 2026-09-03 ==== |
| | |
| | //Recorded.// Four reviewers, all told explicitly that the summary they were given might not be exhaustive, all handed the three page texts, the three report scripts and their unedited outputs. The three focused passes ran in parallel first; their findings were applied and the pages re-saved before the generic pass. **Rejections are logged as well as fixes** — they are the only record of whether a reviewer earns its slot. |
| | |
| | === Figures vs script (Sonnet) === |
| | |
| | Re-ran all five scripts against the live extraction and diffed byte-for-byte against the committed outputs — identical, so the committed outputs were not stale. Confirmed every headline figure, the whole per-target table, the venue counts, all six drift-control cells and the five-venue list, the Borges/LLMCloudHunter distinction, the two ''llm'' firings in the JavaScript population, and the entire non-LLM remainder of all three pages. Four findings: |
| | |
| | ^ # ^ Finding ^ Verdict ^ Action ^ |
| | | 1 | ''report_llm_currency.mjs'''s ''NAMED'' regex had a bare ''phi'', which matches **"Phi**shLLM**"** — a prior detector, not a model — and bucketed that paper as "names a model family". The script's own ''FAMILY'' fold put the same string in the UNMAPPED residue, so the two regexes disagreed about one string. | **ACCEPTED, real bug** | Fixed, then over-corrected, then fixed again — the whole sequence is in §12.5c. The generic pass later rebuilt this table entirely (§12.12, finding 3), so the intermediate 153/11 figures never survived to the final page. | |
| | | 2 | The TGNN footnote said 98.19% is "in its abstract". It is in the **Introduction**; the abstract states no F1 for the annotation component at all. | **ACCEPTED, wrong** | Footnote rewritten and re-verified by line offset against the section headers in ''paper.cols.txt''. | |
| | | 3 | The same footnote cited §4.1.4 for the 98.17% figure; that sentence closes **§4.1.3 //LLM-based Labeling//**, immediately before the §4.1.4 header. | **ACCEPTED, misleading** | Corrected. The reviewer put the second occurrence in §5.5; by line offset it sits between the §5.2 and §5.5 headers, and ''.cols'' interleaving makes the assignment unreliable, so the footnote now says "beside Figure 5" rather than naming a section it cannot prove. The full line-offset table is on [[provenance:privacy:javascript]] §12.4, which is the page whose footnote carries the figure. | |
| | | 4 | The "compared only" row labels any paper with no ''used''/''produced'' tuple as "compared against", but ''usedOrMentioned'' is a five-value enum — a future ''mentioned''-only paper would be mislabelled. Both current papers genuinely carry ''compared''. | **ACCEPTED as latent** | The script now **throws** if the row's papers carry anything but ''compared'', naming the page sentence that would become false. Published figure unchanged. | |
| | |
| | The reviewer also resolved an apparent cross-page contradiction rather than reporting it: [[privacy:requests]] calls 258 the "raw" ''web-request'' count where the truly raw figure is 262. "Raw" there means before that page's topic-relevance exclusion, not before the ''usedOrMentioned'' filter, and both pages consistently publish the used/produced 258. No defect. |
| | |
| | The over-correction this fix caused, and how it was caught, is in §12.5c. |
| | |
| | === Citations and quotes (Sonnet) === |
| | |
| | All ''%%{[key]}%%'' markers on the three pages resolve; ''schwartz2025_llmcloudhunter'' verified against Crossref (authors, title, venue, year, DOI) and against the paper's own ACM reference line. Every load-bearing quote verified in the source, including that {[tsai2025_harmful]} really is a two-class shopping/non-shopping task over 500 annotated Tranco sites — so the page's "binary" framing is fair rather than a spin — and every checked figure in {[bozzolan2026_llmweb]} v3. Two findings: |
| | |
| | ^ # ^ Finding ^ Verdict ^ Action ^ |
| | | 5 | The quote ''scrutinis[e] JavaScript execution within a page'' on [[privacy:javascript]] implies a one-letter bracket edit. The source reads **"scrutinizing"** — the brackets silently changed both the spelling and the participle. | **ACCEPTED, misleading** | Replaced with the surrounding clause quoted verbatim: "single-domain analysis-such as string matching against domain lists or scrutinizing JavaScript execution within a page". | |
| | | 6 | **Two more duplicate-paper pairs** in [[literature:bibliography]] that a DOI scan structurally cannot see, because the entries carry only ''url'': ''lerner2016internet''/''lerner2016_internet'' and ''bouhoula2024automated''/''bouhoula2024_automated''. | **ACCEPTED, recorded, not fixed** | Five pairs now, not three (§12.7). PETS and USENIX records have no DOI at all — 100% of both venues — so a DOI-keyed collision scan is blind to exactly the venues where ''bibgen.mjs'' is weakest. The guard must key on normalised **title** as well as DOI and key string. None of the five is cited by the three edited pages. | |
| | |
| | **Rejected:** adding a footnote to [[privacy:requests]] about TGNN's //websites// vs //distinct domains// inconsistency (its abstract says one, its contributions list the other). The reviewer is right that the two self-contradict and right that [[privacy:requests]] picked the more accurate wording. Rejected here because [[privacy:requests]] is a fourth page this run did not otherwise touch, and editing it would mean re-verifying it; recorded instead so the next run has it. |
| | |
| | === External currency (Sonnet, fetch-only) === |
| | |
| | Twenty-odd external references re-fetched. **Current, verified today:** arXiv:2510.08101 is still at v3 with no v4 and no venue, so the page's characterisation holds; all five Ollama models are live with their tags present and no deprecation language; the Kamen & Kamen ensemble footnote checks out in every element (authors, title, ten LLMs, IAB taxonomy, 8,660 samples, up to 65% F1); ''humanify'' v3.1.1 is still latest, still ''oxc''-based, README wording matches; arXiv:2507.22447 is still an unpublished v1; the AdGuard post is live and says what [[privacy:requests]] attributes to it; TGNN's DOI resolves via Crossref; and Curlie, Cloudflare Radar, Tranco, IAB Taxonomies (3.1 still latest, no 3.2), WebOrganizer, SoAC, WebKB and Homepage2Vec all resolve with their attributed content intact. **All four "nobody has measured this yet" claims still hold** after targeted searches of arXiv, EuroS&P, ACSAC, RAID, AsiaCCS, WPES, SOUPS and CHI. Three findings: |
| | |
| | ^ # ^ Finding ^ Verdict ^ Action ^ |
| | | 7 | Google Topics API: the page said "deprecated and scheduled for removal", second-hand from the preprint. The reviewer reported it as **fully shut down**. | **PARTLY ACCEPTED** | Re-checked against primary sources rather than taken on the reviewer's word, and the reviewer over-stated it. Google's status page does list Topics (Web) as "Deprecate and remove"; the Blink intent estimates deprecate M144, remove M150; Chrome stable is 153.0.8010.27, so that milestone has passed. **But** ''chromestatus.com/feature/5680923054964736'' still reads "Enabled by default" (M115) with no removal milestone, and the API owner's LGTM explicitly reserved "the right to pause the removal". The page now says it is being removed, that the milestone has passed, and that Chrome's own two records disagree — sourced to the three primary URLs, not to a preprint or a summary. | |
| | | 8 | ''gpt-4-turbo-2024-04-09'' — the exact string the page holds up as good practice — **shuts down 23 October 2026**, replaced by ''gpt-5.6-sol''. | **ACCEPTED, and it strengthens the page** | Verified directly against OpenAI's deprecations page. Added, with the note that ''gpt-4o-2024-05-13'' and ''gpt-4-0613'' (also used by corpus papers) share the date. The page's reproducibility argument is now demonstrated rather than argued: naming the snapshot makes a result interpretable, not re-runnable. | |
| | | 9 | The cost footnote said "August 2026 list prices for a nano-tier model", which is now ambiguous: legacy ''gpt-5-nano'' is $0.05/$0.40 per 1M and the current-generation cheapest, ''gpt-5.6-luna'', is $0.20/$1.20. | **ACCEPTED** | Footnote now names the model, the date, both derivations ($2.30/$1.15 and $8.40/$4.20) and the arithmetic. | |
| | |
| | **Also flagged and //not// acted on:** TheWebConf 2026 was postponed from 13–17 April to 29 June – 3 July 2026 while the proceedings metadata still carries the April date. Checked: no page in this edit states the conference dates, so there is nothing to correct. Recorded because a future page that cites WWW '26 dates from the proceedings record will get them wrong. |
| | |
| | **Rejected:** adding arXiv:2605.19275 (//Open-Weight LLMs Are Often Competitive with Commercial APIs for Political Science Text Classification//) as a near-miss on the page's "nobody has benchmarked open models against commercial categorisation services" open question. It is a different domain and a different taxonomy and touches none of FortiGuard, McAfee or {[vallina2020_misshapes]}. Naming it would suggest the gap is narrower than it is. Recorded here so the next run does not re-find it and treat it as closing the question. |
| | |
| | === Generic (Fable) === |
| | |
| | Ran last, with no checklist. §12.12 — the most productive of the four. |
| | |
| | ==== 12.11 Run log, 2026-09-03 ==== |
| | |
| | ^ ^ ^ |
| | | Date | 2026-09-03 | |
| | | Corpus at the time | ''data/extract/run1'', 5,859 papers, 7 venues, 2010–2026; unchanged since 2026-08-11 (commit ''8a6b843'') | |
| | | Model | Claude Opus 5, with four review sub-agents: three Sonnet focused passes in parallel (§12.10), then one Fable generic pass (§12.12) | |
| | | Pages edited | [[design:website_classification]], [[privacy:javascript]], [[design:ip_classification]], plus this page and the two companion provenance pages. Three saves each on the two most-changed pages, because two review passes each moved published figures | |
| | | Pages read and deliberately **not** edited | [[privacy:requests]] — already correct, already citing TGNN nine times, which is why the claim on [[privacy:javascript]] could be fixed by pointing at it. [[privacy:cookies]] — mentions LLMs nowhere at all, which is now a task item rather than a silent gap | |
| | | Scripts added | ''scripts/report_llm_currency.mjs'', ''scripts/llm_currency_quotecheck.mjs'' | |
| | | Scripts changed | ''scripts/report_website_classification.mjs'' — two composition controls, Fisher's exact on every cut, the ''curated-database'' composition listing, and a by-venue third-party-service table | |
| | | Bibliography | added ''schwartz2025_llmcloudhunter''; ran key-string **and** DOI collision scans first, which found three of the five duplicate pairs (§12.7); purged the bibtex cache and confirmed the key renders | |
| | | Bugs in own new scripts | **four**, all recorded above: the ''validation'' allowlist (§12.2, 97.7% vs 83.6%, never reached a page), the ''NAMED'' regex and its over-correction (§12.5c), the hosting-blind reproducibility bucket (§12.5b), and unstripped NUL bytes in the quote checker (§12.6). Two of the four changed a figure that had been saved to a live page | |
| | | Claims retracted | the ''curated-database'' "return to open sources" (§12.4); "CCS and IMC lean hardest on third-party services" (§12.4); "unlike cookie and policy classification, where LLM methods are now routine", on [[design:ip_classification]] (see its own §13.1) | |
| | | Verified after each save | rendered DOM of all three content pages: ''bibtex_citekey'' spans at exactly 2× the source marker count, reference list present, ''ref__schwartz2025_llmcloudhunter'' / ''ref__xiong2026_tgnn'' / ''ref__chen2025_semantics'' resolved, every cross-page ''#heading'' anchor resolving against the rendered heading ids — **one of which broke when a heading was renamed mid-run and was caught by that check** — and zero red links on all three. ''check_page_numbers.mjs'' run windowed and whole-page; it caught one real error, a hand-computed −11.6 pp where the script says −11.7 | |
| | | Deferred, as task items rather than notes | ''bib-duplicate-dois'' and ''bib-duplicate-urls'' (five pairs; the guard needs a title key), ''cookies-llm-classification'', ''llm-model-fold-gpt-residue'', ''closed-source-drift-magnitude'' (needs CCS 2026 and IMC 2026) | |
| | | Deferred, recorded here only | moving the preprint's tables into //Run Your Own Classifier// and de-duplicating the three gap statements against Open Questions (§12.12, finding 11); a //websites// vs //distinct domains// footnote on [[privacy:requests]] (§12.10) | |
| | |
| | ==== 12.12 Generic review pass, 2026-09-03 ==== |
| | |
| | Ran last, with no checklist, after the three focused passes had been applied and the pages re-saved. It was the most productive of the four: **it changed two headline claims and one published table.** Its findings and their disposition: |
| | |
| | ^ # ^ Finding ^ Verdict ^ Action ^ |
| | | 1 | The ''curated-database'' "return to 18.5%" is not a return to open sources. The reviewer listed the twelve papers: three commercial vendors, one model, one taxonomy, four media-bias raters, zero DMOZ/Curlie. | **ACCEPTED — retracted** | Verified independently (§12.4). The reading is withdrawn from the page and the composition is now printed by the report script. This was the stronger half of §4's "first movement against a fifteen-year trend", and it does not survive. | |
| | | 2 | "CCS 2026 and IMC 2026 … the two whose 2022–2024 populations lean hardest on third-party services" is **false for CCS**, which leans least (8.3%, lowest of seven). | **ACCEPTED — wrong** | Verified (§12.4). The sentence is rewritten and the by-venue table is now printed. It was the only sentence in the section with no query behind it. | |
| | | 3 | "11 papers name a model you could re-run" measures OpenAI date-strings, penalises open-weight papers, and contradicts the paragraph two sentences later. ~25 papers naming ''Llama-3.1-70B-Instruct''-class checkpoints were counted as unpinned. | **ACCEPTED — rebuilt** | The bucket is now split by hosting: A 11, B 25, **A+B 36 (20.3%)**, C 130, D 11 (§12.5b). The page's headline changed from "fewer than one in thirteen pins the model" to "one paper in five names something you could resolve", and the advice split by hosted vs open-weight. | |
| | | 4 | [[privacy:javascript]] said "top 7%" while [[design:website_classification]] said 6.2% — the earlier split was not propagated to the page linking to that heading. | **ACCEPTED** | Both were brought to 20.3% from the rebuilt buckets — and then both to **19.4%** when the re-review found those buckets still counting compared-against models (§12.13, finding 1), so this one figure was propagated twice across two pages in a single run. The cross-page claim check in ''report_llm_currency.mjs'' does **not** cover this kind of sentence, and that is a real limit of the guard: it checks the corpus claims, not prose that quotes another page. | |
| | | 5 | "Direction survives both controls" overstates two nested subsamples. Fisher's exact: 2025-alone p = 0.27; curated-database p = 0.08–0.23 on every cut. The two "controls" are not independent tests. | **ACCEPTED** | Fisher's exact is now computed in ''report_website_classification.mjs'' and every //p// and //n// is on the page. 2025-alone is demoted from control to description. The curated row is retracted (finding 1). The third-party drop keeps its direction on p = 0.006 (five-venue) and 0.047 (full window). | |
| | | 6 | §12.5 listed 9 residue strings from the **pre-fix** output while §12.5 twelve lines later said 11, and the content page promised all 11 were listed here. | **ACCEPTED** | §12.5 rewritten; all bucket listings are now generated by the script and embedded in §12.9 rather than transcribed, so they cannot go stale independently. | |
| | | 7 | §12.7 opened "no new external source was added in this run", which the review pass had already falsified on the same page. | **ACCEPTED** | Rewritten, and the //reason// is now the point of the paragraph: a provenance section written before the review pass will be wrong about the review pass. | |
| | | 8 | Two ordinal claims wrong: ''ip-address'' is the **lowest** non-zero row, not second from bottom; ''javascript''/''fingerprinting-script'' are **zero** rows tied with three others, not "the two lowest non-empty". | **ACCEPTED** | Both pages corrected. | |
| | | 9 | "Those shares are floors" is a logic error, and the LLM growth curve got none of the composition scrutiny the drift got. The reviewer supplied the like-for-like numbers. | **ACCEPTED** | Re-derived independently and added to the report script as its own section: five-venue **9.8% → 17.1%**, all five venues rising, CCS+IMC 2025 at 10.6%. The growth claim is robust; the wording was not. | |
| | | 10 | "Reading 116 papers would settle it" — the schema already has ''targetDetail'', populated, and the probe takes one minute. | **ACCEPTED** | Run (§12.8). The caveat is now closed at keyword recall on all three pages, with the probe printed and the limits of a probe stated. | |
| | | 11 | The LLM section is 65% preprint; the decision rule and the LLM-specific reporting items are nowhere in one place; "binary **or few-class**" over-claims; the three "nobody has measured" items are near-duplicated in Open Questions. | **PARTLY ACCEPTED** | "few-class" removed. A three-line decision box added at the top of the section. Four LLM reporting items added to [[design:website_classification#What to Report]] (prompt text, temperature, output constraint, rejection count) — the page had said "count the rejections — that count is a result" with no matching checklist line. **Not done:** moving the preprint's tables into //Run Your Own Classifier//, and de-duplicating the three gap statements against Open Questions. Both are right; both are restructures, and this section had already been rewritten twice in one run. Deferred explicitly rather than silently (§12.3). | |
| | | 12 | The Topics API fix was scope creep: two bold paragraphs of Chromium milestones and telemetry in a service-catalogue entry the page says is not a categoriser. | **ACCEPTED in part** | Cut to one paragraph. The "13% of page loads" figure was **kept**, because it is the only recent public Topics adoption figure and the unit trap ("page loads is not sites") is exactly the kind of thing this site exists to warn about. Keeping it is a judgement call against the reviewer. | |
| | | 13 | The cost footnote led with a legacy tier while the same paragraph was about legacy models being shut down. | **ACCEPTED** | The current-generation figure ($8.40 / $4.20) is now in the body and the legacy one is labelled as legacy. | |
| | | 14 | Review-log wording inflated: "both scripts and both outputs" (there are three of each), "published 97.7%" for a figure that never went live, "the single most dangerous bug in this run" for a bug caught before saving; the NUL-byte story told three times; the print-your-buckets rule stated twice. | **ACCEPTED** | All corrected in this rewrite. §12.2 now says the 97.7% "never reached a page" and drops the superlative. The NUL story is told once here (§12.6) and once, briefly, where a reader needs the ''grep -a'' warning. | |
| | | 15 | The companion provenance pages restate shared material — [[provenance:privacy:javascript]] §12.4 is a second full treatment of the 98.17/98.19 problem. | **ACCEPTED in part** | The javascript companion keeps its version, because it is the page whose footnote carries the figure and a reader checking that footnote should not have to come here. §12.10 finding 2–3 now points there rather than repeating the table. The ip companion was already correctly deferential. | |
| | | 16 | The method table's ''LLM = 2'' includes DarkGram, which the prose calls not-website-classification. | **NOTED, pre-existing, disclosed** | Left as is; §6 of this page already carries the caveat and the prose says "count it as adjacent". | |
| | | 17 | Two per-target link targets could not be verified from the reviewer's environment. | **RESOLVED** | Both exist; the rendered DOM shows zero red links on all three pages. | |
| | |
| | **What the reviewer said was fine, and is worth recording as much as the findings:** decision (a) and its stated alternative, the preprint instability box, the move from keyword-search silence to a measured zero on [[privacy:javascript]], pointing at [[privacy:requests]] rather than rewriting it, the Borges/LLMCloudHunter disambiguation, the itemised quote tiers with their admission that tiers were added after seeing failures, the deliberately unfolded GPT residue, and the ''compared''-only throw-guard. |
| | |
| | **The pattern across all four reviewers.** Every finding that changed a published figure — findings 1, 2, 3, 5, 9 here, and finding 1 in §12.10 — was in a place where **a number or a sentence had no printed list behind it**: an enum row nobody had itemised, a claim about a missing venue-year, a regex bucket reported only as a size, a share reported without an //n//. None was in a figure the report script printed with its denominator. **That is the whole finding of this review pass:** the guard the site already has works, and it only covers what a script prints. Everything else is prose, and prose is where all six defects were. |
| | |
| | |
| | ==== 12.13 Re-review of the figures, 2026-09-03 ==== |
| | |
| | The figures-versus-script reviewer was re-run on the final state, because both the pages and the scripts had changed substantially since its first pass — the model-version table had been rebuilt from scratch, and Fisher's exact, the composition listings and the ''targetDetail'' probe were all new code it had never seen. It re-ran all five scripts (byte-identical to the committed outputs), **verified the hand-implemented Fisher's exact test independently against exact rational arithmetic in Python** — all eight cells agree to four decimal places — and confirmed the like-for-like growth table, the by-venue table, the ''change (pp)'' column, the curated-database counts, and every figure in §12.2, §12.4, §12.5a and §12.5b. Three findings: |
| | |
| | ^ # ^ Finding ^ Verdict ^ Action ^ |
| | | 1 | The reproducibility buckets scanned **every** ''llm'' tuple, not just ''used''/''produced'' ones, so a paper could be promoted by a model it only //compared against//. ''NDSS/2025/automated-expansion-of-privacy-data-taxonomy…'' was in bucket A for ''GPT-3.5-turbo-0125'' although its only ''llm'' tuple is ''compared'' — it is one of the report's own "2 compared-only" papers. ''CCS/2024/airgapagent…'' was promoted to bucket B by ''Mistral Large'', which it compared against, having //used// ''Gemini Ultra''. | **ACCEPTED, real bug in a live figure** | Population changed to the 175 used/produced papers and the loop to ''llmUsed()''. Published figures moved: A **11 → 10**, B **25 → 24**, **A+B 36 (20.3%) → 34 (19.4%)**, C 130 → 131, D 11 → 10. A ''throw'' now fires if the buckets do not sum to the population. Propagated to [[design:website_classification]] and [[privacy:javascript]], both of which had 20.3% saved. | |
| | | 2 | The hand-written enumeration of the twelve ''curated-database'' papers named only **eleven**, omitting ''NDSS/2026/revealing-the-secret-power…'' ("Media Bias/Fact Check (MBFC)"), while the derived "four media-bias raters" counted it — so a reader could not verify "which four". | **ACCEPTED** | Both copies rewritten and sorted by kind, with counts that sum to twelve. The re-check also improved the claim: **one of the twelve, DuckDuckGo's Tracker Radar Entity List, //is// an open inspectable repository** (verified on GitHub, public and active), just of tracker entities rather than website topics. Saying "the row does not contain open directories" was therefore slightly too strong and now reads "no comparable open //topic// directory". | |
| | | 3 | The ''targetDetail'' probe's ''\bpage\b'' cannot match "webpage", and there was no bare ''web'' — two tuples were silently dropped and the published count was 16, not 20. | **ACCEPTED** | Probe widened; it now returns 20. Both new hits were read: //"relevant person-specific webpage information"// (phishing personalisation) and //"IOB presence and trustworthiness in web content"// (malicious-domain false positives). Neither changes the conclusion, which is luck rather than method — the claim rested on a probe under-recalling by 20%. Both pages updated with the new count and with the fact that the width had to be corrected. | |
| | |
| | **Declared clean by this pass:** the Fisher implementation, the like-for-like growth section, the by-venue third-party-service table, the ''change (pp)'' column, the decision box, the four new //What to Report// items, the trimmed Topics API entry, the folded-services table, and the cross-page claims on [[privacy:javascript]] and [[design:ip_classification]]. |
| | |
| | **Both of the two bugs that reached a live page in this run were of one kind:** a query in a new script that did not apply a filter the same script applies elsewhere — the ''validation'' allowlist against ''report_website_classification.mjs'' (§12.2), and the ''usedOrMentioned'' filter against this script's own population section (finding 1). Neither was visible in the output. Both are now guarded: the allowlist is shared, and the bucket sum ''throw''s. **The cheap general check is: for every filter a script defines, grep the script for the places that should use it and do not.** |
| | |
| |
| [[design:website_classification|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[design:website_classification|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| | |
| | ==== 12.14 The model-family fold extended, 2026-09-21 ==== |
| | |
| | **What was wrong.** §12.5a's residue contained four strings that are OpenAI models the ''GPT-4 (non-4o)'' and reasoning-tier regexes should have absorbed: ''GPT 4.1'' (a space instead of a hyphen), ''GPT-4.0'' and ''GPT-4.5'' (a decimal, which the ''%%(?![.\do])%%'' lookahead rejected outright although it was written only to keep ''4o'' out), and ''GPT-o1'' (the o-series, which the fold matched only in its bare ''o1-mini'' form). One paper each. They were **deliberately left in the residue on 2026-09-03** rather than patched after the output was read. |
| | |
| | **Why it was left, and what changed.** Patching a fold to absorb the residue you have just looked at is how a documented rule stops being one: the next reader cannot tell a rule from a list of the strings that embarrassed it. So the fold has been extended as a **rule about how OpenAI writes model names** — the separator after ''GPT'' may be a hyphen, a space or nothing; a version may carry a decimal; the o-series is written both bare and ''GPT''-prefixed — and the rule is stated in the script, above the table it feeds: |
| | |
| | <code> |
| | [/gpt[- ]?4o|gpt4o/i, 'GPT-4o'], |
| | [/gpt[- ]?4-turbo|gpt[- ]?4(\.\d+)?(?![\do])/i, 'GPT-4 (non-4o)'], |
| | [/gpt[- ]?3\.5|chat-?gpt|text-davinci|gpt[- ]?3(?!\.5)/i, 'GPT-3.5 / GPT-3 / ChatGPT'], |
| | [/gpt[- ]?5|gpt[- ]?o[1345]\b|\bo[134]-(mini|preview|pro)\b|\bo4-mini\b/i, 'OpenAI reasoning / GPT-5 tier'], |
| | </code> |
| | |
| | **The rule moves two strings that were never in the residue, and that is the point.** Applied to all 154 distinct ''resourceName'' strings the fold sees, it changes six: the four above, plus ''Chat-GPT 3.5 and 4'' (hyphenated "Chat-GPT", so ''chatgpt'' never matched it — UNMAPPED → GPT-3.5) and ''ChatGPT-4.0'' (which the old decimal lookahead pushed past the GPT-4 row into the ChatGPT row — GPT-3.5 → GPT-4). A rule fitted to the residue would have moved exactly four. Both were verified string by string before the fold was changed, not after. |
| | |
| | ^ Row ^ Was ^ Is ^ Which papers moved ^ |
| | | GPT-4o | 46 (26.0%) | 46 (26.0%) | none — the ''GPT-o1'' paper is IMC/2025 //an-in-depth-investigation-of-data-collection…//, which already counted here for its ''GPT-4o'' string | |
| | | **GPT-4 (non-4o)** | **40 (22.6%)** | **44 (24.9%)** | +4: ''GPT 4.1'' (PETS/2026), ''GPT-4.5'' (CCS/2025), ''GPT-4.0'' (NDSS/2025), ''ChatGPT-4.0'' (USENIX/2024) | |
| | | GPT-3.5 / GPT-3 / ChatGPT | 29 (16.4%) | 29 (16.4%) | **net zero, not "unchanged"**: the ''ChatGPT-4.0'' paper leaves, the ''Chat-GPT 3.5 and 4'' paper (NDSS/2025) arrives | |
| | | **OpenAI reasoning / GPT-5 tier** | **9 (5.1%)** | **10 (5.6%)** | +1: the ''GPT-o1'' paper | |
| | | **UNMAPPED** | **23 (13.0%)** | **18 (10.2%)** | −5 papers; the residue falls from **25 distinct strings to 20** | |
| | |
| | **The three-bucket reproducibility table (§12.5b) does not move**, and it was checked rather than assumed: ''A 10 / B 24 / C 131 / D 10'' before and after, and every bucket's printed string list is byte-identical. The buckets are built from ''HOSTED_SNAPSHOT'', ''OPEN_FAMILY''/''PARAM_SIZE'' and ''NAMED'', none of which the fold touches; ''GPT-o1'' was already in bucket C, because ''NAMED'' contains a bare ''gpt''. The fold and the buckets answer different questions and are deliberately separate regexes. |
| | |
| | **What stays in the residue, and why.** ''Grok-3'' (2 papers), ''GLM-4.5'', ''ChatGLM'' and ''Kimi'' are four vendors with no family in the list. They are a genuine long tail, not a fold failure, and **no family was added for them** — adding one would be the post-hoc patch this section exists to avoid. The remaining 16 strings are descriptions rather than models (''local LLMs (custom prompts)'', ''weighted multi-model ensemble (custom)''), systems built on a model (''PhishLLM'', ''UGCG-GUARD'', ''YouthSafe'', ''RFCGPT''), or non-OpenAI multimodal models (''LLaVA'', ''BLIP2'', ''text-bison''). All 20 are printed in full in §12.9. |
| | |
| | **Where this is published.** Only here. [[:design:website_classification]] publishes the §12.5b bucket table, not the family fold, so the content page needed no edit for this. Checked by grepping the raw source of all **189** pages on the wiki: ''GPT-4 (non-4o)'' and the other family labels return **this page alone**, and the four residue strings return this page and [[:provenance:statistics:annotation]] — where ''GPT 4.1'' appears as a raw ''resourceName'' inside a per-paper quote-check listing, not as a folded row, and so is unaffected by the fold. Same for ''ChatGLM'' on that page. |
| | |
| | <code> |
| | $ node scripts/report_llm_currency.mjs > scripts/report_llm_currency-output.txt |
| | $ diff <old> <new> |
| | 178c178 |
| | < GPT-4 (non-4o) 40 22.6% |
| | > GPT-4 (non-4o) 44 24.9% |
| | 180c180 |
| | < UNMAPPED 23 13.0% |
| | > UNMAPPED 18 10.2% |
| | 184c184 |
| | < OpenAI reasoning / GPT-5 tier 9 5.1% |
| | > OpenAI reasoning / GPT-5 tier 10 5.6% |
| | 192c192 |
| | < --- UNMAPPED residue: 25 distinct strings, printed in full --- |
| | > --- UNMAPPED residue: 20 distinct strings, printed in full --- |
| | 196d195 (Chat-GPT 3.5 and 4) 201,204d199 (GPT 4.1 / GPT-4.0 / GPT-4.5 / GPT-o1) |
| | </code> |
| | |
| | **Those are the only lines that changed in a 446-line report.** Every population, year, venue, target, validation and bucket figure is identical, and the ''compared''-only assertion and the bucket-sum assertion both still pass. |
| | |
| | **One unrelated repair in the same save.** The §12.9 block is regenerated from the committed output file, and the published copy had lost the ''%%\b%%'' escapes from the ''targetDetail'' probe's printed regex (it read ''url|…|web|…'' where the script prints ''%%\burl\b|…|\bweb\b|…%%''). The block now matches the file byte for byte. The probe itself never changed; only the copy on this page was wrong, and it is the width of that probe that decides the claim in §12.7. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Script changes | ''scripts/report_llm_currency.mjs'' — the four OpenAI ''FAMILY'' rows, with the rule stated in a comment above them; committed output regenerated | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass | |
| | | Pages saved | this page only | |
| | | Not edited | [[:design:website_classification]], [[:design:ip_classification]], [[:privacy:javascript]], [[:privacy:cookies]] — none publishes a model-family row | |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''llm_currency_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output, taken on 2026-09-03. ''scripts/llm_currency_quotecheck-output.txt'' was regenerated by re-running the checker and §12.9's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it. |
| | |
| | <code> |
| | $ node scripts/llm_currency_quotecheck.mjs |
| | PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext. |
| | </code> |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | PART A tuples | 30 | 30 | population unchanged | |
| | | verbatim (PASS) | 15 | 15 | unchanged | |
| | | soft (PASS-ELID / LOOSE / CITE / NGRAM) | 9 | 9 | unchanged | |
| | | rescued from the PDF | — | **6** | these were the old FAILs | |
| | | FAIL | **6** | **0** (in both renderings) | 6 = 6 + 0 | |
| | |
| | **This is the strongest vindication of the "relaxed checker" worry above, and the strongest reason to keep the worry.** The paragraph //Two of the soft tiers were added after reading the failures// admits the tiers were added until the checker agreed with a hypothesis. An **independent** rendering, produced by a different extractor and introduced for a different page, now locates all six of the quotes that survived even the relaxed tiers — so the hypothesis was right. What it does not establish is that the soft tiers are sound in general: they were still fitted after seeing the data, and a reader should keep treating each soft pass as a hand-checkable call rather than as a measurement. |
| | |
| | **Zero is a figure that needs its denominator stated.** "0 fail in both renderings" is 0 of **30** LLM tuples across the targets the three classification pages make claims about. It is not a statement about the corpus's 5,859 papers, about the 177 papers that classify something with an LLM, or about any other page's population. |
| | |
| | **Scope of this edit.** §12.9's output block, the tier table and the paragraph under it, the NUL-bug paragraph's footnote, and the content page's quote bullet. ''report_llm_currency.mjs'' and ''report_website_classification.mjs'' were **not** re-run in this pass — the 2026-09-13 significance figures, the venue-composition control, PART B's 49 figure checks and every citation stand as published. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Command | ''%%node scripts/llm_currency_quotecheck.mjs > scripts/llm_currency_quotecheck-output.txt%%'' | |
| | | Artifacts | ''scripts/llm_currency_quotecheck-output.txt'' (regenerated; the 2026-09-03 copy kept as ''…-output.txt.bak0921'') | |
| | | Script changes | none | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:design:website_classification]] | |
| | | Pages saved | this page, [[:design:website_classification]] | |
| | | Not edited | [[:provenance:privacy:javascript]] mentions this checker but publishes none of its counts | |
| |
