provenance:programming:multilingual_support
Table of Contents
Provenance: Programming:Multilingual support
Back to Multilingual support. Corpus-wide selection and extraction notes are on corpus. This is the page-specific query log.
Run record
- Run date: 2026-08-27 (UTC).
- Authoring agent: Cursor Grok 4.6, executing a wiki page update without a drain TODO. The Cursor session ran out of quota with the re-review round in flight; Claude Opus 5 (Claude Code) picked the draft up from the exported transcript (
cursor_multilingual_support_update.md), re-ran every guard and probe, ran the outstanding review round, applied its one finding, and published. The user asked to treat it as a task-page sitting, inspect similar completed items (especiallydesign:ip_classificationand stub rebuilds such asprogramming:docker/programming:registration), and to question the pre-LLM-sweep human page rather than expand it blindly. - Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under
/workspace/publications_dataset/data/. - Live page before this sitting:
programming:multilingual_supportrev 1744994969, 8,014 bytes, author empty indw.mjs info. Noprovenance:programming:multilingual_support. The programming namespace outline still labelled it a stub. - Read first: the wiki-measuretheweb spec in
drain/tasks.md, live page,programming:docker(already pointed at this page's:latestcompose),design:ip_classification(question-not-tool structure),data/extract/OVERVIEW.md,site_queries.mjs –page multilingual, Hosseini 2021 PDF (not in the extraction), Bhuiyan IMC 2025, Bouhoula/Kubicek USENIX Sec 2024, Nenadić PETS 2026, Tang PETS 2024, Xie USENIX Sec 2025, LibreTranslatedetect.py, Hub tag API, GitHub repo metadata for fastText/CLD3/Lingua/langdetect. - Extending, not creating. Title kept. Overlap judgement: keep this as the instrument page. Do not merge into website_selection (lists), crawling_location (IP language), website_classification (topic), or interaction (the click). LangCrUX is cited here as a language-sampling method and left as a CrUX child for anyone who rebuilds crux.
- No write to the publication mount. Wiki saves through
scripts/dw.mjswith–if-rev.
What was kept from the human page, and what was not
The live page was Karel's operator note: langdetect/CLD3/fastText, a LibreTranslate compose, Ollama caveats, a pipeline checklist, and a 
on interaction. It predated the extraction sweep, so it had no corpus figures, recommended fastText as current from an undated blog, used :latest, and cited Hosseini for “best on long text” without the eight-library ensemble.
| Human-page claim | Decision |
|---|---|
| English-centric research; Mhaidli §4.4.2 | Kept, shortened, quoted. |
| Three jobs: detect, translate, full pipeline | Kept as the spine; added “switch the site” as its own section (vantage / Accept-Language / URL / click), which the pipeline listed but did not explain. |
| fastText is the clear winner (modelpredict.com) | Demoted. Blog still HTTP 200, undated. Corpus uses langdetect (25) over fastText (5). fastText archived 2024-03-22. |
| Hosseini: langdetect best on long text | Reworded. Paper is an eight-library majority vote; langdetect's Cramér's V with the determined language is 0.97, highest. Not an agreement rate. PDF fetched; paper not in extractions.jsonl. |
Do not trust <html lang> (CMS defaults to EN) | Kept as operator footnote. Closest peer-reviewed statement is Bhuiyan et al. on accessibility lang metadata ≠ visible language — not the same measurement. Published a script that compares the tag to a Unicode-script heuristic instead of asserting a rate. |
| LibreTranslate compose | Kept, digest-pinned, Greek/short-text failures from [1Bouhoula, Ahmed; Kubicek, Karel; Zac, Amit; Cotrini, Carlos; Basin, David (2024): "Automated Large-Scale Analysis of Cookie Notice Compliance", in: Proceedings of the USENIX Security Symposium. (Link)] added. :latest last moved 2026-08-23. |
| LibreTranslate uses langdetect | Held. detect.py on 2026-08-27: langdetect ≥20 chars, lexilang below. |
| Paid APIs too expensive; Google ToS | Kept, and dated: Google Translate is still 49/135 of tool-using papers. |
| LLM translation 50–1000× slower; long-text failure | Kept as qualitative operator footnote. Dropped the “tens to thousands of times” range: not a paper figure and not in the report. |
| Interaction | Closed with Bouhoula keyword lists + Xie 48,178/101,249 English-only cut. |
| mBERT for classification | Dated. 2026 current for documents: no translation [2Nenadić, Luka; Rodriguez, David; Calandrino, Joseph A. (2026): "Overcoming Language Barriers: Multilingual Analysis of the 2023 Swiss Privacy Law's Impact", Proceedings on Privacy Enhancing Technologies 2026(4):703-723. (DOI)]. 2024 current for short UI: translate-then-BERT. |
| Encoding one-liner | Kept, expanded one sentence (UTF-8, NFC, do not copy the wiki's latin1 .cols reader). |
| Link to Docker | Kept; compose now satisfies that page's digest lesson. |
Population and queries
All counts are papers unless labelled tuples. Sentinels are not answers. Four corpus papers have no paper.cols.txt; zero crawled papers are missing one.
| Query | Denominator | Result |
|---|---|---|
POPULATIONS.crawled | 5,859 | 1,120 |
crawled with a crawlConfig object | 1,120 | 1,080 (40 crawled papers have no object) |
languages[] non-empty | 1,080 | 136 (12.6%) |
| ROLE=multilingual | 1,080 | 22 (2.0%) |
| ROLE=language-specific | 1,080 | 31 (2.9%) |
| handled non-English (union) | 1,080 | 53 (4.9%) — the published instrument figure |
| ROLE=english-only | 1,080 | 77 (7.1%) — schema said leave empty |
| ROLE=locale-knob | 1,080 | 1 (fingerprinting POSIX locales) |
| ROLE=vague | 1,080 | 4 |
| ROLE=programming | 1,080 | 1 ([“C”], awesome-c) |
used language-detection or translation tool | 5,859 | 135 (2.3%) |
| those that are crawled | 1,120 | 67 (6.0%) |
| used language-detection | 5,859 | 63 |
| used translation | 5,859 | 83 |
| Google Translate family | 135 tool-users | 49 |
| Google Translate ∪ googletrans/pygtrans | 135 | 56 (41.5%) |
| langdetect family (incl. Shuyo Java name) | 135 | 25 |
| fastText family | 135 | 5 |
| LibreTranslate / Argos | 135 | 3 (Bouhoula 2024; PhishLang Argos; a dual-use Argos mention) |
full-text langdetect among crawled | 1,120 | 15 |
| full-text LibreTranslate/Argos among crawled | 1,120 | 3 |
full-text <html lang among crawled | 1,120 | 1 |
| full-text Accept-Language among crawled | 1,120 | 8 |
site_queries.mjs –page multilingual | 1,080 / 5,859 | 136 named; unfolded English 106. Not pasted. |
Year buckets of handled-non-English, of with-cfg: 2010–2013 2/98; 2014–2017 6/160; 2018–2021 14/297; 2022–2024 23/336; 2025–2026* 8/189.
Folding and residue
Language names: ordered regex families in scripts/multilingual_fold.mjs. POSIX locales and ^c$ before natural-language names. Residue of the 136 papers' strings: 0.
ROLE is automatic over the folded families of one paper, not a 136-row hand map. The crawlConfig evidence quote is shared across the whole object and cannot decide the role (Cookie Hunter's quote is about crawl depth). Roles: multilingual / language-specific / english-only / locale-knob / vague / programming.
Rejected probe: publishing 136/1080 as “papers that handled a language”. 77 of 136 are English-only over-extraction.
Rejected probe: site_queries.mjs unfolded language table (English 106, Chinese 21, …). Same undercount/overcount problem as vantage/“United States”.
Tool names: ordered families, first match wins. Residue 5 tuples: “Detect Language”, “translation software”, “Language Identification”, “Translate”, “language classifier [21]”. Printed in the report, not on the content page.
Offtopic folded and excluded from advice: Amberscript (3), TTS (3), emoji/unidecode (2), Google Play Console (1).
Evidence quotes checked
Load-bearing sentences were read in paper.cols.txt (and Hosseini in the publisher PDF):
- Bhuiyan et al.: 50% cutoff, median 88%, VPN paragraph, “nearly 40%” Bangladesh/India. Column-spliced in places; numbers taken from the contiguous methods paragraphs, not from figure captions.
- Bouhoula et al.: eleven languages, langdetect, LibreTranslate, Greek “Accept all cookies” → “Cookie policy”, cost sentence. Verbatim in
.cols. - Nenadić et al.: “single inference without requiring translation”; F1 above 0.90; 120 policies; >35,000 websites.
- Tang et al.: Lingua high-accuracy; 15.22% fallback to CLD3; 103 languages; 99.29%.
- Xie et al.: Table 6 Non-English 48,178 / Full Set 101,249. Table is column-spliced; both numbers sit on their own lines.
- Mhaidli et al. §4.4.2 quote, whitespace-normalised against
.cols. - Yeung et al.: “we changed the browser language to Ukrainian, Russian, and English”.
- Hils et al.:
en-USas preferred browser language — schemalanguages: [“en-US”]; quote is the crawlConfig evidence (resolution 1024×800, en-US). The schema over-extraction of English-only is the same field.
Hosseini 2021: not in the extraction, no paper.cols.txt. Cramér's V 0.97 from Fig. 3 / p. 489 of the PDF (pdftotext of https://petsymposium.org/popets/2021/popets-2021-0081.pdf) — langdetect × ensemble-determined language, highest of the eight. Not an agreement rate. Ensemble membership from §3.
crawlConfig.evidence.quote is a bad check for the language field: Degeling's quote is the scan cadence; Bhuiyan's is “each website is visited programmatically”. Roles were not derived from those quotes.
External and industry verification
| Source | Load-bearing fact | Verification 2026-08-27 | Decision |
|---|---|---|---|
Hub libretranslate/libretranslate:latest | last_updated 2026-08-23; amd64 and arm64 digests as on the page | Hub API | Used; pin is a worked example |
LibreTranslate detect.py | langdetect ≥20 chars; lexilang below | raw.githubusercontent.com | Used |
| LibreTranslate README | engine is Argos | raw README | Used |
facebookresearch/fastText | archived=true, pushed 2024-03-22 | GitHub API | Used; dates the old “winner” |
google/cld3 | archived=true, pushed 2023-05-24 | GitHub API | Used |
pemistahl/lingua-py | archived=false, pushed 2026-07-20 | GitHub API | Used |
Mimino666/langdetect | archived=false, pushed 2025-03-03 | GitHub API | Used |
| modelpredict.com survey | still 200; still names fastText winner | curl | Named as undated blog, not as advice |
| Hosseini PDF | 200, 1.5 MB | petsymposium.org | Used; out of extraction |
| USENIX landing Xie | authors Qinge Xie, Karthik Ramakrishnan, Frank Li; pages 5797–5816 | citation_ meta | Used |
| PETS landing Tang | authors Tang / Alvarez / Brar / Hoang / Christin; pages 44–60 | citation_ meta | Used |
| this host | aarch64, no Docker binary; check_page_language.py –demo exit 0 (11 Han / 7 Latin) | executed | Used |
Rejected:
- Gherbal v4 / ELDC / other 2026 LID blogs as “what to use”. Not in the corpus, not what a web-measurement methods section should copy without a paper.
- Claiming a measured rate for
<html lang>=“en”on non-English sites. Operator observation only. - Running the LibreTranslate compose — no daemon.
- Publishing the 293-style raw tool-name table from
site_queries.mjs.
What the corpus and sources do not establish
- How often
<html lang>is wrong. The 1/1,120 full-text hit is a reporting rate. - A 2026 accuracy ranking of Lingua vs langdetect vs archived fastText on web page text. Hosseini is 2021 privacy policies; modelpredict.com is undated.
- Whether Playwright's
locale/ Chromium–langequals anAccept-Languageheader on every site. - Tomorrow's LibreTranslate digest.
- Hosseini inside the extraction.
Judgement calls
- Rebuild the stub into a corpus-backed sibling of docker/registration rather than leave it as an API note. The human compose and the html-lang warning are the reason not to replace the page wholesale.
- Publish 53/1080, not 136/1080, as the handling figure; keep 136 visible.
- Automatic ROLE rather than hand-reading 136 shared
crawlConfigquotes. - Date fastText/CLD3 as archived rather than “do not use” — they still run; they are unmaintained.
- Do not “fix” crux in this sitting; LangCrUX is cited, not forked.
- Reuse
bouhoula2024_automated,degeling2019_value,hils2021_privacy,mhaidli2023researchers,hosseini2021unifying,nenadic2026_swiss,yeung2023_online. Addbhuiyan2025_visitors,tang2024_automatic,xie2025_evaluating. - Namespace outline: stubs 3→2 (CrUX, Similarweb).
report_namespace_overviews.mjsSTUB_IDSupdated to match.
Published code
pages/libretranslate.compose.yaml— arm64 digest fetched 2026-08-27. Embed bytes diffed equal.pages/check_page_language.py— stdlib HTMLParser;–demoexit 0. Embed bytes diffed equal. Hidden/aria-hidden/display:none stripped; unknown BCP-47 tags fail; Japanese scored as Han+kana.
Report script
node scripts/report_multilingual.mjs (also –wiki, –list, –quotes). Output committed as scripts/report_multilingual-output.txt. Fold: scripts/multilingual_fold.mjs. Probe: scripts/multilingual_probe.sh.
Review log
Drafts frozen under out/freeze_multilingual/ before the focused passes. The first round ran in Cursor, whose Task allowlist offers gpt-5.6-sol-medium rather than the sonnet / fable the task spec asks for (same substitute as docker/registration). The content page was then edited to answer those findings, so the second round re-read the patched draft and ran the generic pass; that round ran in Claude Code on the spec's own models — sonnet for figures and citations, fable for generic. Both rounds' snapshots are named per reviewer below.
Reviewer 1 — figures vs script (GPT 5.6 Sol medium)
| # | Finding | Decision |
|---|---|---|
| 1 | “Tens to thousands of times below LibreTranslate” has no throughput figure in the report. | Accepted. Dropped the range; kept a qualitative operator footnote. |
| 2 | “30-policy-per-language gold set” not established by the report (120 policies × 4 languages). | Accepted in part. The equal 30/language split is in Nenadić §3.2, not a corpus figure. Added “30 each” to report Z; page now attributes it as their gold set. |
| 3 | Checker counted hidden text; a hidden Chinese paragraph made an English page report Han majority. | Accepted. Visible-text parser now drops hidden, aria-hidden=“true”, and inline display:none / visibility:hidden. |
| 4 | Unsupported tags (lang=“sr”) silently passed because Serbian was absent from the map. | Accepted. Unknown primary tags exit 1 as unsupported. sr/bs added as Latin ∪ Cyrillic. |
| 5 | Japanese scripts were split, so kana+kanji could lose to Latin. | Accepted. ja scores Han + hiragana + katakana as one family. Demo: family 10 vs Latin 5, match. |
| 6 | “~50 languages” for older detectors not in the report. | Accepted. Dropped the count. |
| 7 | Illustrative “15-word” / “three-word” / “7B”. | Accepted. Dropped. |
| 8 | “Seven venues” not computed by the report. | Accepted. Report population now prints 7 venues and fails if the set is not 7. |
Corpus tables, demo 11 Han vs 7 Latin, and compose digests were held.
Reviewer 2 — citations and quotes (GPT 5.6 Sol medium)
| # | Finding | Decision |
|---|---|---|
| 1 | Line 376 attributed language-specific accept/reject/save variants to Bouhoula; the paper translates buttons to English and classifies there. spaCy + “cookie” variants are notice detection. | Accepted. Interaction section rewritten to that method; German/Danish are hand-corrected translations, not keyword lists. |
| 2 | Nenadić F1 above 0.90 is for most language/disclosure pairs, not all (Italian automated-decision F1 0.33). | Accepted. Page and “what to read first” now say most pairs. |
| 3 | Mhaidli interviewees “did not have LLM tools / native-speaker annotators on demand” is not in that paper. | Accepted. Dropped the sentence. |
| 4 | Hosseini 0.97 called “agreement”; it is Cramér's V with the determined language. | Accepted. |
| 5 | Bhuiyan quote used em dashes; .cols has hyphens. | Accepted. Quote now matches the extraction. |
All ten bibliography keys and first-author names were held.
Reviewer 3 — external currency (GPT 5.6 Sol medium)
| # | Finding | Decision |
|---|---|---|
| 1 | Clean: probe passed; Argos + langdetect/lexilang; Hub digests current; fastText/CLD3 archived; Lingua/langdetect live; survey and Hosseini PDF up; upstream compose still documents the healthcheck path. | Held. No edit. |
Round 2, on the patched draft (Claude Code, spec models)
| Pass | Model | Outcome |
|---|---|---|
| figures vs script | sonnet | Clean. Re-ran the report and matched every figure on both pages; confirmed all 8 round-1 items fixed; wrote 10 fresh adversarial fixtures against check_page_language.py (lang=“zh-Hant”, uppercase LANG, style=“display: none” variants, script/style/noscript text, HTML comments, entity-encoded text, lang=“”, empty page) and found no new defect; both <file> embeds byte-identical. |
| citations and quotes | sonnet | Clean. Re-verified all 5 round-1 fixes against paper.cols.txt and, for Nenadić and Hosseini, the publisher PDFs (Italian automated-decision F1 0.33 is the reason “most pairs” is the only defensible wording; Hosseini's 0.97 is the highest of eight Cramér's V values, the others 0.91–0.95). All 10 keys resolve, no collisions, all attributions match, and the 3 new BibTeX entries check against primary sources. |
| generic, no checklist | fable | One accepted finding, below. |
Reviewer 4 — generic, no checklist (fable)
| # | Finding | Decision |
|---|---|---|
| 1 | The opening <WRAP important> box stated “the 2026 method that is current for classification is stay in the source language” as an unqualified prescription, while the qualifier that governs it (“most language/disclosure pairs, not all”) sat only in the Classification section — the exact “qualifier lost in the opening box” failure the spec warns about. | Accepted. The box now carries “validated per language, because their F1 above 0.90 holds for most language/disclosure pairs, not all”. |
| 2 | Every internal cross-link resolves; figures re-checked beyond the log (English family 110 / 80.9%, and the role table summing to 1,080); the human stub's load-bearing content is either kept or documentedly superseded; structure and voice match design:ip_classification / programming:docker; the provenance page does not overstate its rigour. | Held. No edit. |
| 3 | The modelpredict.com rebuttal will itself go stale once “the previous version of this page” stops being a live memory, and could be trimmed by a future editor. | Noted, not applied. It is the only source the human page cited for its central recommendation; dating it is the point of the paragraph. |
Guards and probes re-run before publishing (2026-08-27, Claude Code)
| Check | Result |
|---|---|
node scripts/report_multilingual.mjs | exit 0, byte-identical to the committed report_multilingual-output.txt |
check_page_numbers.mjs both pages, whole-page and –code | OK; the only unaccounted numbers on provenance:programming are its own drain item/run ids |
check_wrap.mjs, check_tables.mjs both pages | OK |
check_attributions.mjs content page × live bibliography + 3 new entries | 6 attributions, all match |
<file> embeds vs pages/check_page_language.py and pages/libretranslate.compose.yaml | byte-identical |
python3 pages/check_page_language.py –demo | exit 0; 11 Han vs 7 Latin; 7 fixtures |
bash scripts/multilingual_probe.sh | all external checks OK — both Hub digests unchanged, detect.py still langdetect + lexilang, fastText/CLD3 still archived, Lingua/langdetect live, survey and Hosseini PDF 200 |
report_namespace_overviews.mjs | PUBLISHED_STUB_COUNT 2 after the STUB_IDS edit |
Bibliography publish path. pages/literature_bibliography.txt in the workdir was 18 keys behind the live page (other sittings had appended since it was copied), so publishing it would have deleted their entries. The three new keys were instead appended to a fresh do=export_raw of the live page (pages/literature_bibliography.PUBLISH.txt), duplicate-checked, and saved with –if-rev.
is on the content page only.
You could leave a comment if you were logged in.
provenance/programming/multilingual_support.txt · Last modified: by karel.kubicek.claude