| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:design:website_classification [2026/09/03 22:35] – Add section 12.13 (re-review of the figures): the reproducibility buckets were counting models a paper only compared against (A+B 36/20.3% -> 34/19.4%), the twelve-paper curated-database enumeration named eleven, and the targetDetail probe's \bpage\b miss karel.kubicek.claude | provenance:design:website_classification [2026/09/21 14:40] (current) – Review follow-ups: 12.8's 'task item exists' bullet closed by 12.14, and 12.14's page-sweep claim corrected (provenance:statistics:annotation carries the raw string, not a fold row). Authored by Claude karel.kubicek.claude |
|---|
| ^ Item ^ Value ^ | ^ Item ^ Value ^ |
| | Content page | [[design:website_classification]] | | | Content page | [[design:website_classification]] | |
| | Report script | ''scripts/report_website_classification.mjs'' (''--wiki'', ''--list'', ''--quotes <regex>'') | | | Report script | ''scripts/report_website_classification.mjs'' (''%%--wiki%%'', ''%%--list%%'', ''%%--quotes%% <regex>'') | |
| | Folds | ''scripts/webcat_fold.mjs'' — a **task** fold and a **resource** fold | | | Folds | ''scripts/webcat_fold.mjs'' — a **task** fold and a **resource** fold | |
| | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 | | | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 | |
| * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. | * The matcher was **substring**, not word-boundary, so ''report.includes('59')'' was satisfied by ''11.59 bits''. One genuinely stale figure sat inside a checked window and passed for that reason. |
| |
| Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''--code'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. | Both are fixed in ''scripts/check_page_numbers.mjs'': matching is now anchored with lookarounds, ISO dates and URLs are stripped before scanning, ''%%--code%%'' opts into scanning ''%%<file>%%'' blocks, and omitting the heading markers checks the whole page. **Run it windowed //and// whole-page.** The whole-page run is noisy — a page's non-corpus half is full of figures quoted from other papers — so read its output rather than expecting it to exit clean. |
| Fixed on this page's content page as a result: the lead paragraph's **72% → 75%** for bespoke unsized taxonomies, which contradicted the corpus section's own "Three quarters … 75.4%" four screens down. | Fixed on this page's content page as a result: the lead paragraph's **72% → 75%** for bespoke unsized taxonomies, which contradicted the corpus section's own "Three quarters … 75.4%" four screens down. |
| |
| ^ Family ^ Papers (of 177) ^ | ^ Family ^ Papers (of 177) ^ |
| | GPT-4o | 46 (26.0%) | | | GPT-4o | 46 (26.0%) | |
| | GPT-4 (non-4o) | 40 (22.6%) | | | GPT-4 (non-4o) | 44 (24.9%) | |
| | GPT-3.5 / GPT-3 / ChatGPT | 29 (16.4%) | | | GPT-3.5 / GPT-3 / ChatGPT | 29 (16.4%) | |
| | **UNMAPPED** | **23 (13.0%)** | | | **UNMAPPED** | **18 (10.2%)** | |
| | Llama family | 13 (7.3%) | | | Llama family | 13 (7.3%) | |
| | Gemini / PaLM | 12 (6.8%) | | | Gemini / PaLM | 12 (6.8%) | |
| | Unnamed LLM | 11 (6.2%) | | | Unnamed LLM | 11 (6.2%) | |
| | OpenAI reasoning / GPT-5 tier | 9 (5.1%) | | | OpenAI reasoning / GPT-5 tier | 10 (5.6%) | |
| | Qwen family | 8 (4.5%) | | | Qwen family | 8 (4.5%) | |
| | DeepSeek family | 7 (4.0%) | | | DeepSeek family | 7 (4.0%) | |
| | Encoder / seq2seq LM (not a chat LLM) | 2 (1.1%) | | | Encoder / seq2seq LM (not a chat LLM) | 2 (1.1%) | |
| |
| The 25-string unmapped residue is printed in full by the script (§12.9). **Four of those strings are a fold failure, not a long tail:** ''GPT 4.1'', ''GPT-4.0'', ''GPT-4.5'' and ''GPT-o1'' are OpenAI models the ''GPT-4 (non-4o)'' regex misses on a space or a decimal. They were left visible rather than folded, because patching a fold to absorb its own residue after seeing the output stops it being a documented rule; a task item exists to fold them and re-derive the three GPT rows. ''Grok-3'', ''GLM-4.5'', ''ChatGLM'' and ''Kimi'' are the genuine long tail — four vendors with no family in the list. | **The table above is the 2026-09-21 re-derivation.** Until then it read GPT-4 (non-4o) 40 (22.6%), UNMAPPED 23 (13.0%) and OpenAI reasoning 9 (5.1%), because four residue strings — ''GPT 4.1'', ''GPT-4.0'', ''GPT-4.5'' and ''GPT-o1'' — are OpenAI models that the fold missed on a space or a decimal. They were left visible on 2026-09-03 rather than quietly folded, because patching a fold to absorb its own residue after seeing the output stops it being a documented rule. The fold has now been extended **as a rule about how OpenAI model strings are spelled**, not as a list of those four: see §12.14 for the rule, the two further strings it moves, and the full diff. ''Grok-3'', ''GLM-4.5'', ''ChatGLM'' and ''Kimi'' stay in the now 20-string residue — they are the genuine long tail, four vendors with no family in the list, and no family was added for them. |
| |
| === 5b. The reproducibility buckets, which had to be rebuilt === | === 5b. The reproducibility buckets, which had to be rebuilt === |
| | PASS-CITE | 1 | present once inline citation markers are stripped — the extractor drops them, so ''Qwen3 [49]'' becomes ''Qwen3'' | | | PASS-CITE | 1 | present once inline citation markers are stripped — the extractor drops them, so ''Qwen3 [49]'' becomes ''Qwen3'' | |
| | PASS-NGRAM | 2 | ≥80% of the quote's word 5-grams present; the sentence is in the paper but the extraction reworded a word or two, or ''.cols'' interleaved a float into it | | | PASS-NGRAM | 2 | ≥80% of the quote's word 5-grams present; the sentence is in the paper but the extraction reworded a word or two, or ''.cols'' interleaved a float into it | |
| | **FAIL** | **6** | not present under any of the above | | | **RESCUED** | **6** | below every tier above against the rendering the extractor read, and at or above the n-gram tier against an independent ''pypdf'' rendering of the same ''paper.pdf'' | |
| | | **FAIL in both renderings** | **0** | not present under any tier in either rendering | |
| |
| **All six failures are ''privacy-policy'' or ''consent-notice'' tuples, and no page on this site quotes any of them.** Every tuple behind a figure on the three edited pages passed at some tier. | **Re-run 2026-09-21 with the PDF fallback, and the six FAILs are now six RESCUEDs: nothing in this pass fails in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. Four of the six go to a **complete** n-gram match in the PDF (36/36, 14/14, 21/21, 11/11) and one to 19/19; the sixth, ''PETS/2026/word-level-annotation…'', goes 11/20 → 16/20, which clears the 80% tier. All six are ''privacy-policy'' or ''consent-notice'' tuples and no page on this site quotes any of them, so no published claim moves; every tuple behind a figure on the three edited pages passed at some tier already. |
| |
| **Two of the soft tiers were added after reading the failures, and that is worth admitting.** The first run reported 15 fail; reading them showed the extraction drops inline citation markers, and that ''paper.cols.txt'' interleaves table captions into sentences — ''PETS/2026/disclosure-divergence…'' failed because the caption //"Table 1: LLM backend comparison on 100-app validation."// lands inside the quoted sentence. Adding tiers for those is right, since the alternative is publishing "half the quotes are unlocatable", which is false. But it is also a checker relaxed until it agreed with a hypothesis. The mitigation is that every soft pass is itemised with its reason and, for PASS-NGRAM, the broken n-grams, so each call is checkable by hand. | **Two of the soft tiers were added after reading the failures, and that is worth admitting.** The first run reported 15 fail; reading them showed the extraction drops inline citation markers, and that ''paper.cols.txt'' interleaves table captions into sentences — ''PETS/2026/disclosure-divergence…'' failed because the caption //"Table 1: LLM backend comparison on 100-app validation."// lands inside the quoted sentence. Adding tiers for those is right, since the alternative is publishing "half the quotes are unlocatable", which is false. But it is also a checker relaxed until it agreed with a hypothesis. The mitigation is that every soft pass is itemised with its reason and, for PASS-NGRAM, the broken n-grams, so each call is checkable by hand. |
| |
| **A latent bug found in review changed nothing, and is recorded anyway.** 90 of the 5,869 ''paper.cols.txt'' files (1.5%) contain **NUL bytes** — 913 in the TGNN paper alone. They are not whitespace to ''\s'', invisible in a terminal, and they make shell ''grep'' treat the file as binary and suppress every match silently (use ''grep -a''). The checker did not strip them; it now does. Re-running gave **15 / 9 / 6 before and after**, because no NUL happened to land inside one of these 30 quotes. On a different sample it would have been a published false FAIL. | **A latent bug found in review changed nothing, and is recorded anyway.** 90 of the 5,869 ''paper.cols.txt'' files (1.5%) contain **NUL bytes** — 913 in the TGNN paper alone. They are not whitespace to ''\s'', invisible in a terminal, and they make shell ''grep'' treat the file as binary and suppress every match silently (use ''grep -a''). The checker did not strip them; it now does. Re-running gave **15 / 9 / 6 before and after**, because no NUL happened to land inside one of these 30 quotes. On a different sample it would have been a published false FAIL. ((The ''6'' in that sentence is the 2026-09-03 FAIL count. Under the PDF fallback added on 2026-09-04 the same six are ''RESCUED'' and the fail-in-both count is 0 — see //Quote-check refresh, 2026-09-21//. The NUL finding itself is unaffected: it is about ''.cols'' preprocessing, not about the fallback.)) |
| |
| **6 of 30 (20%) is far above the corpus-wide 0.9% unlocatable rate** on [[literature:corpus]]. It is not a re-measurement: 30 tuples, non-random, all 2024–2026, weighted to PETS 2026 whose PDFs are the newest and worst-rendered. Read it as a reason to check quotes from the 2025–2026 slice specifically. | **6 of 30 (20%) is far above the corpus-wide 0.9% unlocatable rate** on [[literature:corpus]]. It is not a re-measurement: 30 tuples, non-random, all 2024–2026, weighted to PETS 2026 whose PDFs are the newest and worst-rendered. Read it as a reason to check quotes from the 2025–2026 slice specifically. |
| |
| **The probe's width decided that answer, and the first width was wrong.** It used ''\bpage\b'', which does not match the compound "webpage" — there is no word boundary between "web" and "page" — and it had no bare ''web'' at all, so it returned 16 rather than 20 and silently dropped //"relevant person-specific webpage information"// and //"IOB presence and trustworthiness in web content"//. Found in re-review (§12.13, finding 3). Both were then read and neither changes the conclusion, which is the only reason the published claim survived a probe that was under-recalling by 20%. A probe is not a read of 116 papers, its hits must be read rather than counted, and its regex is a load-bearing part of the claim. | **The probe's width decided that answer, and the first width was wrong.** It used ''\bpage\b'', which does not match the compound "webpage" — there is no word boundary between "web" and "page" — and it had no bare ''web'' at all, so it returned 16 rather than 20 and silently dropped //"relevant person-specific webpage information"// and //"IOB presence and trustworthiness in web content"//. Found in re-review (§12.13, finding 3). Both were then read and neither changes the conclusion, which is the only reason the published claim survived a probe that was under-recalling by 20%. A probe is not a read of 116 papers, its hits must be read rather than counted, and its regex is a load-bearing part of the claim. |
| * **Whether the four GPT strings in the fold residue change a published share.** §12.5a. Not folded on purpose; a task item exists. | * **Whether the four GPT strings in the fold residue change a published share.** §12.5a. **Settled on 2026-09-21 and no longer open** — the fold was extended as a rule and the three GPT rows re-derived; they do change one published share on this page and none anywhere else. See §12.14. |
| * **Whether the six quote failures are extraction paraphrase or ''.cols'' rendering.** One was read and was rendering. The other five were not, because no page quotes them. | * **Whether the six quote failures are extraction paraphrase or ''.cols'' rendering.** One was read and was rendering. The other five were not, because no page quotes them. |
| |
| ------------------------------------- --------------- ----- | ------------------------------------- --------------- ----- |
| GPT-4o 46 26.0% | GPT-4o 46 26.0% |
| GPT-4 (non-4o) 40 22.6% | GPT-4 (non-4o) 44 24.9% |
| GPT-3.5 / GPT-3 / ChatGPT 29 16.4% | GPT-3.5 / GPT-3 / ChatGPT 29 16.4% |
| UNMAPPED 23 13.0% | UNMAPPED 18 10.2% |
| Llama family 13 7.3% | Llama family 13 7.3% |
| Gemini / PaLM 12 6.8% | Gemini / PaLM 12 6.8% |
| Unnamed LLM 11 6.2% | Unnamed LLM 11 6.2% |
| OpenAI reasoning / GPT-5 tier 9 5.1% | OpenAI reasoning / GPT-5 tier 10 5.6% |
| Qwen family 8 4.5% | Qwen family 8 4.5% |
| DeepSeek family 7 4.0% | DeepSeek family 7 4.0% |
| Encoder / seq2seq LM (not a chat LLM) 2 1.1% | Encoder / seq2seq LM (not a chat LLM) 2 1.1% |
| |
| --- UNMAPPED residue: 25 distinct strings, printed in full --- | --- UNMAPPED residue: 20 distinct strings, printed in full --- |
| 2x Grok-3 | 2x Grok-3 |
| 2x local LLMs (custom prompts) | 2x local LLMs (custom prompts) |
| 1x BLIP2 | 1x BLIP2 |
| 1x Chat-GPT 3.5 and 4 | |
| 1x ChatGLM | 1x ChatGLM |
| 1x custom structured prompts with fine-tuned LLMs | 1x custom structured prompts with fine-tuned LLMs |
| 1x foundation LLMs | 1x foundation LLMs |
| 1x GLM-4.5 | 1x GLM-4.5 |
| 1x GPT 4.1 | |
| 1x GPT-4.0 | |
| 1x GPT-4.5 | |
| 1x GPT-o1 | |
| 1x HtmlLLM-Detector | 1x HtmlLLM-Detector |
| 1x Kimi | 1x Kimi |
| 157 of 157 (100.0%) state a targetDetail. | 157 of 157 (100.0%) state a targetDetail. |
| |
| --- targetDetail matching the probe (websit|domain|url|homepage|page|web|script|tracker|cookie|sdk|first.part|third.part|categor) — 20 tuples, all printed --- | --- targetDetail matching the probe (websit|domain|\burl\b|homepage|page|\bweb\b|script|tracker|cookie|\bsdk\b|first.part|third.part|categor) — 20 tuples, all printed --- |
| IMC/2023/in-the-room-where-it-happens-characterizing-local-communication-and-threats-in-s | IMC/2023/in-the-room-where-it-happens-characterizing-local-communication-and-threats-in-s |
| IoT device vendors and categories | IoT device vendors and categories |
| FAIL = not present under any of the above. | FAIL = not present under any of the above. |
| |
| FAIL IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots [privacy-policy] 26/36 5-grams present | RESCUED IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots [privacy-policy] 26/36 -> 36/36 PDF "we design a set of task prompts for an AI chatbot to split scraped content into …" |
| quote: we design a set of task prompts for an AI chatbot to split scraped content into sections, and then extract and label mentions of collected data types, data collection purposes, data retention and protection practices, and user rights and choices. | |
| longest verbatim prefix (19 of 246 chars): we design a set of | |
| PASS USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w [website-category] "This prompt is fed into a language model using a chain-of-thought approach, enfo…" | PASS USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w [website-category] "This prompt is fed into a language model using a chain-of-thought approach, enfo…" |
| PASS-NGRAM IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems [privacy-policy] 10/10 5-grams "We develop an LLM-based framework to check the consistency of data collection di…" | PASS-NGRAM IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems [privacy-policy] 10/10 5-grams "We develop an LLM-based framework to check the consistency of data collection di…" |
| PASS PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law [privacy-policy] "For each policy, we issue a single inference request to the model and require it…" | PASS PETS/2026/overcoming-language-barriers-multilingual-analysis-of-the-2023-swiss-privacy-law [privacy-policy] "For each policy, we issue a single inference request to the model and require it…" |
| PASS WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia [website-category] "To evaluate our classification methods, we manually annotated a sample of 500 we…" | PASS WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia [website-category] "To evaluate our classification methods, we manually annotated a sample of 500 we…" |
| FAIL PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] 11/20 5-grams present | RESCUED PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] 11/20 -> 16/20 PDF "On a manually labelled sample of 340 randomly selected documents ... using GPT 4…" |
| quote: On a manually labelled sample of 340 randomly selected documents ... using GPT 4.1 as the classifying LLM ... achieved an accuracy of 99.7%. | |
| longest verbatim prefix (59 of 140 chars): On a manually labelled sample of 340 randomly selected docu | |
| PASS-LOOSE PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] "This LLM-based classifier predicts the set of labels Lp relevant to each passage" | PASS-LOOSE PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using [privacy-policy] "This LLM-based classifier predicts the set of labels Lp relevant to each passage" |
| FAIL CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation [consent-notice] 9/14 5-grams present | RESCUED CCS/2025/whispertest-a-voice-control-based-library-for-ios-ui-automation [consent-notice] 9/14 -> 14/14 PDF "we used a more efficient text-only model (Qwen2.5-7B) to detect the presence of …" |
| quote: we used a more efficient text-only model (Qwen2.5-7B) to detect the presence of consent dialogs | RESCUED PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] 13/21 -> 21/21 PDF "We randomly reserved 70% of the manual annotations to constitute our training da…" |
| longest verbatim prefix (25 of 95 chars): we used a more efficient | |
| FAIL PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] 13/21 5-grams present | |
| quote: We randomly reserved 70% of the manual annotations to constitute our training data (21,588 examples), while the other 30% (9252 examples) were testing data. | |
| longest verbatim prefix (54 of 156 chars): We randomly reserved 70% of the manual annotations to | |
| PASS PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] "For the prompted non-fine-tuned LLMs, we used GPT-4, GPT-4 Turbo, and GPT-3.5 Tu…" | PASS PETS/2025/automating-governing-knowledge-commons-and-contextual-integrity-gkc-ci-privacy-p [privacy-policy] "For the prompted non-fine-tuned LLMs, we used GPT-4, GPT-4 Turbo, and GPT-3.5 Tu…" |
| PASS PETS/2025/behavr-user-identification-based-on-vr-sensor-data [privacy-policy] "We also use simple string matching to search for relevant content." | PASS PETS/2025/behavr-user-identification-based-on-vr-sensor-data [privacy-policy] "We also use simple string matching to search for relevant content." |
| PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" | PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" |
| PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" | PASS PETS/2026/designing-reflective-thinking-based-contextual-privacy-policy-for-mobile-applica [privacy-policy] "We evaluated privacy policy segment extraction accuracy on three state-of-the-ar…" |
| FAIL PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t [privacy-policy] 15/19 5-grams present | RESCUED PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t [privacy-policy] 15/19 -> 19/19 PDF "A researcher manually verified whether each LLM classification matched the corre…" |
| quote: A researcher manually verified whether each LLM classification matched the correct taxonomy label. GPT-4o-mini achieved an overall accuracy of 87.83%. | |
| longest verbatim prefix (31 of 150 chars): A researcher manually verified | |
| PASS-NGRAM PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 19/20 5-grams "the system outputs two sets of data types, C data (collected) and S data (shared…" | PASS-NGRAM PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 19/20 5-grams "the system outputs two sets of data types, C data (collected) and S data (shared…" |
| FAIL PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 7/11 5-grams present | RESCUED PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s [privacy-policy] 7/11 -> 11/11 PDF "both models were evaluated using the same preprocessing pipeline, chunking strat…" |
| quote: both models were evaluated using the same preprocessing pipeline, chunking strategy, prompts, and output schema. | |
| longest verbatim prefix (66 of 112 chars): both models were evaluated using the same preprocessing pipeline, | |
| PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "GPT-4 added four new categories, bringing the total number of categories to 19. …" | PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "GPT-4 added four new categories, bringing the total number of categories to 19. …" |
| PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "We evaluated 100 randomly selected cases and found GPT-4's predictions to be acc…" | PASS USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram [website-category] "We evaluated 100 randomly selected cases and found GPT-4's predictions to be acc…" |
| PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc [consent-notice] elided into 2 fragments | PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc [consent-notice] elided into 2 fragments |
| |
| PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 fail, 0 with no fulltext. | PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext. |
| |
| |
| | 2 | "CCS 2026 and IMC 2026 … the two whose 2022–2024 populations lean hardest on third-party services" is **false for CCS**, which leans least (8.3%, lowest of seven). | **ACCEPTED — wrong** | Verified (§12.4). The sentence is rewritten and the by-venue table is now printed. It was the only sentence in the section with no query behind it. | | | 2 | "CCS 2026 and IMC 2026 … the two whose 2022–2024 populations lean hardest on third-party services" is **false for CCS**, which leans least (8.3%, lowest of seven). | **ACCEPTED — wrong** | Verified (§12.4). The sentence is rewritten and the by-venue table is now printed. It was the only sentence in the section with no query behind it. | |
| | 3 | "11 papers name a model you could re-run" measures OpenAI date-strings, penalises open-weight papers, and contradicts the paragraph two sentences later. ~25 papers naming ''Llama-3.1-70B-Instruct''-class checkpoints were counted as unpinned. | **ACCEPTED — rebuilt** | The bucket is now split by hosting: A 11, B 25, **A+B 36 (20.3%)**, C 130, D 11 (§12.5b). The page's headline changed from "fewer than one in thirteen pins the model" to "one paper in five names something you could resolve", and the advice split by hosted vs open-weight. | | | 3 | "11 papers name a model you could re-run" measures OpenAI date-strings, penalises open-weight papers, and contradicts the paragraph two sentences later. ~25 papers naming ''Llama-3.1-70B-Instruct''-class checkpoints were counted as unpinned. | **ACCEPTED — rebuilt** | The bucket is now split by hosting: A 11, B 25, **A+B 36 (20.3%)**, C 130, D 11 (§12.5b). The page's headline changed from "fewer than one in thirteen pins the model" to "one paper in five names something you could resolve", and the advice split by hosted vs open-weight. | |
| | 4 | [[privacy:javascript]] said "top 7%" while [[design:website_classification]] said 6.2% — the earlier split was not propagated to the page linking to that heading. | **ACCEPTED** | Both now read 20.3%, from the rebuilt buckets. The cross-page claim check in ''report_llm_currency.mjs'' does **not** cover this kind of sentence, and that is a real limit of the guard: it checks the corpus claims, not prose that quotes another page. | | | 4 | [[privacy:javascript]] said "top 7%" while [[design:website_classification]] said 6.2% — the earlier split was not propagated to the page linking to that heading. | **ACCEPTED** | Both were brought to 20.3% from the rebuilt buckets — and then both to **19.4%** when the re-review found those buckets still counting compared-against models (§12.13, finding 1), so this one figure was propagated twice across two pages in a single run. The cross-page claim check in ''report_llm_currency.mjs'' does **not** cover this kind of sentence, and that is a real limit of the guard: it checks the corpus claims, not prose that quotes another page. | |
| | 5 | "Direction survives both controls" overstates two nested subsamples. Fisher's exact: 2025-alone p = 0.27; curated-database p = 0.08–0.23 on every cut. The two "controls" are not independent tests. | **ACCEPTED** | Fisher's exact is now computed in ''report_website_classification.mjs'' and every //p// and //n// is on the page. 2025-alone is demoted from control to description. The curated row is retracted (finding 1). The third-party drop keeps its direction on p = 0.006 (five-venue) and 0.047 (full window). | | | 5 | "Direction survives both controls" overstates two nested subsamples. Fisher's exact: 2025-alone p = 0.27; curated-database p = 0.08–0.23 on every cut. The two "controls" are not independent tests. | **ACCEPTED** | Fisher's exact is now computed in ''report_website_classification.mjs'' and every //p// and //n// is on the page. 2025-alone is demoted from control to description. The curated row is retracted (finding 1). The third-party drop keeps its direction on p = 0.006 (five-venue) and 0.047 (full window). | |
| | 6 | §12.5 listed 9 residue strings from the **pre-fix** output while §12.5 twelve lines later said 11, and the content page promised all 11 were listed here. | **ACCEPTED** | §12.5 rewritten; all bucket listings are now generated by the script and embedded in §12.9 rather than transcribed, so they cannot go stale independently. | | | 6 | §12.5 listed 9 residue strings from the **pre-fix** output while §12.5 twelve lines later said 11, and the content page promised all 11 were listed here. | **ACCEPTED** | §12.5 rewritten; all bucket listings are now generated by the script and embedded in §12.9 rather than transcribed, so they cannot go stale independently. | |
| |
| [[design:website_classification|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[design:website_classification|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| | |
| | ==== 12.14 The model-family fold extended, 2026-09-21 ==== |
| | |
| | **What was wrong.** §12.5a's residue contained four strings that are OpenAI models the ''GPT-4 (non-4o)'' and reasoning-tier regexes should have absorbed: ''GPT 4.1'' (a space instead of a hyphen), ''GPT-4.0'' and ''GPT-4.5'' (a decimal, which the ''%%(?![.\do])%%'' lookahead rejected outright although it was written only to keep ''4o'' out), and ''GPT-o1'' (the o-series, which the fold matched only in its bare ''o1-mini'' form). One paper each. They were **deliberately left in the residue on 2026-09-03** rather than patched after the output was read. |
| | |
| | **Why it was left, and what changed.** Patching a fold to absorb the residue you have just looked at is how a documented rule stops being one: the next reader cannot tell a rule from a list of the strings that embarrassed it. So the fold has been extended as a **rule about how OpenAI writes model names** — the separator after ''GPT'' may be a hyphen, a space or nothing; a version may carry a decimal; the o-series is written both bare and ''GPT''-prefixed — and the rule is stated in the script, above the table it feeds: |
| | |
| | <code> |
| | [/gpt[- ]?4o|gpt4o/i, 'GPT-4o'], |
| | [/gpt[- ]?4-turbo|gpt[- ]?4(\.\d+)?(?![\do])/i, 'GPT-4 (non-4o)'], |
| | [/gpt[- ]?3\.5|chat-?gpt|text-davinci|gpt[- ]?3(?!\.5)/i, 'GPT-3.5 / GPT-3 / ChatGPT'], |
| | [/gpt[- ]?5|gpt[- ]?o[1345]\b|\bo[134]-(mini|preview|pro)\b|\bo4-mini\b/i, 'OpenAI reasoning / GPT-5 tier'], |
| | </code> |
| | |
| | **The rule moves two strings that were never in the residue, and that is the point.** Applied to all 154 distinct ''resourceName'' strings the fold sees, it changes six: the four above, plus ''Chat-GPT 3.5 and 4'' (hyphenated "Chat-GPT", so ''chatgpt'' never matched it — UNMAPPED → GPT-3.5) and ''ChatGPT-4.0'' (which the old decimal lookahead pushed past the GPT-4 row into the ChatGPT row — GPT-3.5 → GPT-4). A rule fitted to the residue would have moved exactly four. Both were verified string by string before the fold was changed, not after. |
| | |
| | ^ Row ^ Was ^ Is ^ Which papers moved ^ |
| | | GPT-4o | 46 (26.0%) | 46 (26.0%) | none — the ''GPT-o1'' paper is IMC/2025 //an-in-depth-investigation-of-data-collection…//, which already counted here for its ''GPT-4o'' string | |
| | | **GPT-4 (non-4o)** | **40 (22.6%)** | **44 (24.9%)** | +4: ''GPT 4.1'' (PETS/2026), ''GPT-4.5'' (CCS/2025), ''GPT-4.0'' (NDSS/2025), ''ChatGPT-4.0'' (USENIX/2024) | |
| | | GPT-3.5 / GPT-3 / ChatGPT | 29 (16.4%) | 29 (16.4%) | **net zero, not "unchanged"**: the ''ChatGPT-4.0'' paper leaves, the ''Chat-GPT 3.5 and 4'' paper (NDSS/2025) arrives | |
| | | **OpenAI reasoning / GPT-5 tier** | **9 (5.1%)** | **10 (5.6%)** | +1: the ''GPT-o1'' paper | |
| | | **UNMAPPED** | **23 (13.0%)** | **18 (10.2%)** | −5 papers; the residue falls from **25 distinct strings to 20** | |
| | |
| | **The three-bucket reproducibility table (§12.5b) does not move**, and it was checked rather than assumed: ''A 10 / B 24 / C 131 / D 10'' before and after, and every bucket's printed string list is byte-identical. The buckets are built from ''HOSTED_SNAPSHOT'', ''OPEN_FAMILY''/''PARAM_SIZE'' and ''NAMED'', none of which the fold touches; ''GPT-o1'' was already in bucket C, because ''NAMED'' contains a bare ''gpt''. The fold and the buckets answer different questions and are deliberately separate regexes. |
| | |
| | **What stays in the residue, and why.** ''Grok-3'' (2 papers), ''GLM-4.5'', ''ChatGLM'' and ''Kimi'' are four vendors with no family in the list. They are a genuine long tail, not a fold failure, and **no family was added for them** — adding one would be the post-hoc patch this section exists to avoid. The remaining 16 strings are descriptions rather than models (''local LLMs (custom prompts)'', ''weighted multi-model ensemble (custom)''), systems built on a model (''PhishLLM'', ''UGCG-GUARD'', ''YouthSafe'', ''RFCGPT''), or non-OpenAI multimodal models (''LLaVA'', ''BLIP2'', ''text-bison''). All 20 are printed in full in §12.9. |
| | |
| | **Where this is published.** Only here. [[:design:website_classification]] publishes the §12.5b bucket table, not the family fold, so the content page needed no edit for this. Checked by grepping the raw source of all **189** pages on the wiki: ''GPT-4 (non-4o)'' and the other family labels return **this page alone**, and the four residue strings return this page and [[:provenance:statistics:annotation]] — where ''GPT 4.1'' appears as a raw ''resourceName'' inside a per-paper quote-check listing, not as a folded row, and so is unaffected by the fold. Same for ''ChatGLM'' on that page. |
| | |
| | <code> |
| | $ node scripts/report_llm_currency.mjs > scripts/report_llm_currency-output.txt |
| | $ diff <old> <new> |
| | 178c178 |
| | < GPT-4 (non-4o) 40 22.6% |
| | > GPT-4 (non-4o) 44 24.9% |
| | 180c180 |
| | < UNMAPPED 23 13.0% |
| | > UNMAPPED 18 10.2% |
| | 184c184 |
| | < OpenAI reasoning / GPT-5 tier 9 5.1% |
| | > OpenAI reasoning / GPT-5 tier 10 5.6% |
| | 192c192 |
| | < --- UNMAPPED residue: 25 distinct strings, printed in full --- |
| | > --- UNMAPPED residue: 20 distinct strings, printed in full --- |
| | 196d195 (Chat-GPT 3.5 and 4) 201,204d199 (GPT 4.1 / GPT-4.0 / GPT-4.5 / GPT-o1) |
| | </code> |
| | |
| | **Those are the only lines that changed in a 446-line report.** Every population, year, venue, target, validation and bucket figure is identical, and the ''compared''-only assertion and the bucket-sum assertion both still pass. |
| | |
| | **One unrelated repair in the same save.** The §12.9 block is regenerated from the committed output file, and the published copy had lost the ''%%\b%%'' escapes from the ''targetDetail'' probe's printed regex (it read ''url|…|web|…'' where the script prints ''%%\burl\b|…|\bweb\b|…%%''). The block now matches the file byte for byte. The probe itself never changed; only the copy on this page was wrong, and it is the width of that probe that decides the claim in §12.7. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Script changes | ''scripts/report_llm_currency.mjs'' — the four OpenAI ''FAMILY'' rows, with the rule stated in a comment above them; committed output regenerated | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass; one ''sonnet'' citations/quotes pass | |
| | | Pages saved | this page only | |
| | | Not edited | [[:design:website_classification]], [[:design:ip_classification]], [[:privacy:javascript]], [[:privacy:cookies]] — none publishes a model-family row | |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''llm_currency_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output, taken on 2026-09-03. ''scripts/llm_currency_quotecheck-output.txt'' was regenerated by re-running the checker and §12.9's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it. |
| | |
| | <code> |
| | $ node scripts/llm_currency_quotecheck.mjs |
| | PART A: 15 verbatim, 9 soft (elided or punctuation/glyph), 6 rescued from the PDF, 0 fail in both renderings, 0 with no fulltext. |
| | </code> |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | PART A tuples | 30 | 30 | population unchanged | |
| | | verbatim (PASS) | 15 | 15 | unchanged | |
| | | soft (PASS-ELID / LOOSE / CITE / NGRAM) | 9 | 9 | unchanged | |
| | | rescued from the PDF | — | **6** | these were the old FAILs | |
| | | FAIL | **6** | **0** (in both renderings) | 6 = 6 + 0 | |
| | |
| | **This is the strongest vindication of the "relaxed checker" worry above, and the strongest reason to keep the worry.** The paragraph //Two of the soft tiers were added after reading the failures// admits the tiers were added until the checker agreed with a hypothesis. An **independent** rendering, produced by a different extractor and introduced for a different page, now locates all six of the quotes that survived even the relaxed tiers — so the hypothesis was right. What it does not establish is that the soft tiers are sound in general: they were still fitted after seeing the data, and a reader should keep treating each soft pass as a hand-checkable call rather than as a measurement. |
| | |
| | **Zero is a figure that needs its denominator stated.** "0 fail in both renderings" is 0 of **30** LLM tuples across the targets the three classification pages make claims about. It is not a statement about the corpus's 5,859 papers, about the 177 papers that classify something with an LLM, or about any other page's population. |
| | |
| | **Scope of this edit.** §12.9's output block, the tier table and the paragraph under it, the NUL-bug paragraph's footnote, and the content page's quote bullet. ''report_llm_currency.mjs'' and ''report_website_classification.mjs'' were **not** re-run in this pass — the 2026-09-13 significance figures, the venue-composition control, PART B's 49 figure checks and every citation stand as published. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Command | ''%%node scripts/llm_currency_quotecheck.mjs > scripts/llm_currency_quotecheck-output.txt%%'' | |
| | | Artifacts | ''scripts/llm_currency_quotecheck-output.txt'' (regenerated; the 2026-09-03 copy kept as ''…-output.txt.bak0921'') | |
| | | Script changes | none | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:design:website_classification]] | |
| | | Pages saved | this page, [[:design:website_classification]] | |
| | | Not edited | [[:provenance:privacy:javascript]] mentions this checker but publishes none of its counts | |
| |
