| |
| provenance:programming:multilingual_support [2026/08/27 15:26] – Provenance for programming:multilingual_support: queries, folds and residue, quote checks, external verification, rejected sources, judgement calls, both review rounds. Authored by Claude. karel.kubicek.claude | provenance:programming:multilingual_support [2026/09/17 07:24] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| * Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under ''/workspace/publications_dataset/data/''. | * Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under ''/workspace/publications_dataset/data/''. |
| * Live page before this sitting: ''programming:multilingual_support'' rev **1744994969**, **8,014** bytes, author empty in ''dw.mjs info''. No ''provenance:programming:multilingual_support''. The programming namespace outline still labelled it a stub. | * Live page before this sitting: ''programming:multilingual_support'' rev **1744994969**, **8,014** bytes, author empty in ''dw.mjs info''. No ''provenance:programming:multilingual_support''. The programming namespace outline still labelled it a stub. |
| * Read first: the wiki-measuretheweb spec in ''drain/tasks.md'', live page, ''programming:docker'' (already pointed at this page's '':latest'' compose), ''design:ip_classification'' (question-not-tool structure), ''data/extract/OVERVIEW.md'', ''site_queries.mjs --page multilingual'', Hosseini 2021 PDF (not in the extraction), Bhuiyan IMC 2025, Bouhoula/Kubicek USENIX Sec 2024, Nenadić PETS 2026, Tang PETS 2024, Xie USENIX Sec 2025, LibreTranslate ''detect.py'', Hub tag API, GitHub repo metadata for fastText/CLD3/Lingua/langdetect. | * Read first: the wiki-measuretheweb spec in ''drain/tasks.md'', live page, ''programming:docker'' (already pointed at this page's '':latest'' compose), ''design:ip_classification'' (question-not-tool structure), ''data/extract/OVERVIEW.md'', ''site_queries.mjs %%--page%% multilingual'', Hosseini 2021 PDF (not in the extraction), Bhuiyan IMC 2025, Bouhoula/Kubicek USENIX Sec 2024, Nenadić PETS 2026, Tang PETS 2024, Xie USENIX Sec 2025, LibreTranslate ''detect.py'', Hub tag API, GitHub repo metadata for fastText/CLD3/Lingua/langdetect. |
| * Extending, not creating. Title kept. Overlap judgement: keep this as the **instrument** page. Do not merge into [[design:website_selection]] (lists), [[design:crawling_location]] (IP language), [[design:website_classification]] (topic), or [[programming:interaction]] (the click). LangCrUX is cited here as a language-sampling method and left as a CrUX child for anyone who rebuilds [[programming:crux]]. | * Extending, not creating. Title kept. Overlap judgement: keep this as the **instrument** page. Do not merge into [[design:website_selection]] (lists), [[design:crawling_location]] (IP language), [[design:website_classification]] (topic), or [[programming:interaction]] (the click). LangCrUX is cited here as a language-sampling method and left as a CrUX child for anyone who rebuilds [[programming:crux]]. |
| * No write to the publication mount. Wiki saves through ''scripts/dw.mjs'' with ''--if-rev''. | * No write to the publication mount. Wiki saves through ''scripts/dw.mjs'' with ''%%--if-rev%%''. |
| |
| ===== What was kept from the human page, and what was not ===== | ===== What was kept from the human page, and what was not ===== |
| | full-text ''<html lang'' among crawled | 1,120 | **1** | | | full-text ''<html lang'' among crawled | 1,120 | **1** | |
| | full-text Accept-Language among crawled | 1,120 | 8 | | | full-text Accept-Language among crawled | 1,120 | 8 | |
| | ''site_queries.mjs --page multilingual'' | 1,080 / 5,859 | 136 named; unfolded English 106. **Not pasted.** | | | ''site_queries.mjs %%--page%% multilingual'' | 1,080 / 5,859 | 136 named; unfolded English 106. **Not pasted.** | |
| |
| Year buckets of handled-non-English, of with-cfg: 2010–2013 2/98; 2014–2017 6/160; 2018–2021 14/297; 2022–2024 23/336; 2025–2026* 8/189. | Year buckets of handled-non-English, of with-cfg: 2010–2013 2/98; 2014–2017 6/160; 2018–2021 14/297; 2022–2024 23/336; 2025–2026* 8/189. |
| | USENIX landing Xie | authors Qinge Xie, Karthik Ramakrishnan, Frank Li; pages 5797–5816 | citation_ meta | Used | | | USENIX landing Xie | authors Qinge Xie, Karthik Ramakrishnan, Frank Li; pages 5797–5816 | citation_ meta | Used | |
| | PETS landing Tang | authors Tang / Alvarez / Brar / Hoang / Christin; pages 44–60 | citation_ meta | Used | | | PETS landing Tang | authors Tang / Alvarez / Brar / Hoang / Christin; pages 44–60 | citation_ meta | Used | |
| | this host | aarch64, no Docker binary; ''check_page_language.py --demo'' exit 0 (11 Han / 7 Latin) | executed | Used | | | this host | aarch64, no Docker binary; ''check_page_language.py %%--demo%%'' exit 0 (11 Han / 7 Latin) | executed | Used | |
| |
| Rejected: | Rejected: |
| * How often ''<html lang>'' is wrong. The 1/1,120 full-text hit is a reporting rate. | * How often ''<html lang>'' is wrong. The 1/1,120 full-text hit is a reporting rate. |
| * A 2026 accuracy ranking of Lingua vs langdetect vs archived fastText **on web page text**. Hosseini is 2021 privacy policies; modelpredict.com is undated. | * A 2026 accuracy ranking of Lingua vs langdetect vs archived fastText **on web page text**. Hosseini is 2021 privacy policies; modelpredict.com is undated. |
| * Whether Playwright's ''locale'' / Chromium ''--lang'' equals an ''Accept-Language'' header on every site. | * Whether Playwright's ''locale'' / Chromium ''%%--lang%%'' equals an ''Accept-Language'' header on every site. |
| * Tomorrow's LibreTranslate digest. | * Tomorrow's LibreTranslate digest. |
| * Hosseini inside the extraction. | * Hosseini inside the extraction. |
| |
| * ''pages/libretranslate.compose.yaml'' — arm64 digest fetched 2026-08-27. Embed bytes diffed equal. | * ''pages/libretranslate.compose.yaml'' — arm64 digest fetched 2026-08-27. Embed bytes diffed equal. |
| * ''pages/check_page_language.py'' — stdlib HTMLParser; ''--demo'' exit 0. Embed bytes diffed equal. Hidden/aria-hidden/display:none stripped; unknown BCP-47 tags fail; Japanese scored as Han+kana. | * ''pages/check_page_language.py'' — stdlib HTMLParser; ''%%--demo%%'' exit 0. Embed bytes diffed equal. Hidden/aria-hidden/display:none stripped; unknown BCP-47 tags fail; Japanese scored as Han+kana. |
| |
| ===== Report script ===== | ===== Report script ===== |
| |
| ''node scripts/report_multilingual.mjs'' (also ''--wiki'', ''--list'', ''--quotes''). Output committed as ''scripts/report_multilingual-output.txt''. Fold: ''scripts/multilingual_fold.mjs''. Probe: ''scripts/multilingual_probe.sh''. | ''node scripts/report_multilingual.mjs'' (also ''%%--wiki%%'', ''%%--list%%'', ''%%--quotes%%''). Output committed as ''scripts/report_multilingual-output.txt''. Fold: ''scripts/multilingual_fold.mjs''. Probe: ''scripts/multilingual_probe.sh''. |
| |
| ===== Review log ===== | ===== Review log ===== |
| ^ Check ^ Result ^ | ^ Check ^ Result ^ |
| | ''node scripts/report_multilingual.mjs'' | exit 0, byte-identical to the committed ''report_multilingual-output.txt'' | | | ''node scripts/report_multilingual.mjs'' | exit 0, byte-identical to the committed ''report_multilingual-output.txt'' | |
| | ''check_page_numbers.mjs'' both pages, whole-page and ''--code'' | OK; the only unaccounted numbers on ''provenance:programming'' are its own drain item/run ids | | | ''check_page_numbers.mjs'' both pages, whole-page and ''%%--code%%'' | OK; the only unaccounted numbers on ''provenance:programming'' are its own drain item/run ids | |
| | ''check_wrap.mjs'', ''check_tables.mjs'' both pages | OK | | | ''check_wrap.mjs'', ''check_tables.mjs'' both pages | OK | |
| | ''check_attributions.mjs'' content page × live bibliography + 3 new entries | 6 attributions, all match | | | ''check_attributions.mjs'' content page × live bibliography + 3 new entries | 6 attributions, all match | |
| | %%<file>%% embeds vs ''pages/check_page_language.py'' and ''pages/libretranslate.compose.yaml'' | byte-identical | | | %%<file>%% embeds vs ''pages/check_page_language.py'' and ''pages/libretranslate.compose.yaml'' | byte-identical | |
| | ''python3 pages/check_page_language.py --demo'' | exit 0; 11 Han vs 7 Latin; 7 fixtures | | | ''python3 pages/check_page_language.py %%--demo%%'' | exit 0; 11 Han vs 7 Latin; 7 fixtures | |
| | ''bash scripts/multilingual_probe.sh'' | all external checks OK — both Hub digests unchanged, ''detect.py'' still langdetect + lexilang, fastText/CLD3 still archived, Lingua/langdetect live, survey and Hosseini PDF 200 | | | ''bash scripts/multilingual_probe.sh'' | all external checks OK — both Hub digests unchanged, ''detect.py'' still langdetect + lexilang, fastText/CLD3 still archived, Lingua/langdetect live, survey and Hosseini PDF 200 | |
| | ''report_namespace_overviews.mjs'' | ''PUBLISHED_STUB_COUNT'' 2 after the ''STUB_IDS'' edit | | | ''report_namespace_overviews.mjs'' | ''PUBLISHED_STUB_COUNT'' 2 after the ''STUB_IDS'' edit | |
| |
| **Bibliography publish path.** ''pages/literature_bibliography.txt'' in the workdir was **18 keys behind the live page** (other sittings had appended since it was copied), so publishing it would have deleted their entries. The three new keys were instead appended to a fresh ''do=export_raw'' of the live page (''pages/literature_bibliography.PUBLISH.txt''), duplicate-checked, and saved with ''--if-rev''. | **Bibliography publish path.** ''pages/literature_bibliography.txt'' in the workdir was **18 keys behind the live page** (other sittings had appended since it was copied), so publishing it would have deleted their entries. The three new keys were instead appended to a fresh ''do=export_raw'' of the live page (''pages/literature_bibliography.PUBLISH.txt''), duplicate-checked, and saved with ''%%--if-rev%%''. |
| |
| ~~DISCUSSION~~ is on the content page only. | ~~DISCUSSION~~ is on the content page only. |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| |
