| Both sides previous revisionPrevious revision | |
| programming:multilingual_support [2026/08/27 15:26] – Rebuild the stub into a corpus-backed instrument page: 53/1080 crawled papers handled a non-English language, dated detector/translator families, LibreTranslate compose digest-pinned, published html-lang checker, keep-it-in-the-source-language classificat karel.kubicek.claude | programming:multilingual_support [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| * **[[Design:Crawling location]]:** many sites pick a language from the client IP. A US cloud VM fetches the English CDN site. VPN localisation belongs there; this page only names it as a language switch. | * **[[Design:Crawling location]]:** many sites pick a language from the client IP. A US cloud VM fetches the English CDN site. VPN localisation belongs there; this page only names it as a language switch. |
| * **[[Programming:Interaction]]:** clicking a language switcher is a click. Building the keyword list for that click is here. | * **[[Programming:Interaction]]:** clicking a language switcher is a click. Building the keyword list for that click is here. |
| * **[[Programming:Docker]]:** pin the image digest and size ''/dev/shm''. The compose below is a translator, not a crawler; ''check_crawler_compose.py --all'' still digest-pins it. | * **[[Programming:Docker]]:** pin the image digest and size ''/dev/shm''. The compose below is a translator, not a crawler; ''check_crawler_compose.py %%--all%%'' still digest-pins it. |
| * **[[Design:Website classification]]:** topic taxonomies, including Yahoo's multilingual production classifier. Not language ID. | * **[[Design:Website classification]]:** topic taxonomies, including Yahoo's multilingual production classifier. Not language ID. |
| |
| - **Declared language ≠ visible language, even when someone filled the attribute.** Bhuiyan et al. treat ''lang'' attributes as accessibility metadata that screen readers consume, and find they often fail to reflect the language of the visible content {[bhuiyan2025_visitors]}. That is a measurement of assistive-technology metadata, and it is the closest peer-reviewed statement we have. It is not a crawl of ''<html lang>'' versus a detector. | - **Declared language ≠ visible language, even when someone filled the attribute.** Bhuiyan et al. treat ''lang'' attributes as accessibility metadata that screen readers consume, and find they often fail to reflect the language of the visible content {[bhuiyan2025_visitors]}. That is a measurement of assistive-technology metadata, and it is the closest peer-reviewed statement we have. It is not a crawl of ''<html lang>'' versus a detector. |
| |
| The script below compares the BCP-47 primary tag on ''<html lang>'' with a Unicode-script majority over visible text (the LangCrUX idea, not their VPN crawl). A missing tag is reported, not a failure — absence is not a mismatch. An unknown primary tag is a failure: the checker does not silently agree with a language it cannot score. Hidden, ''aria-hidden'', and inline ''display:none'' / ''visibility:hidden'' subtrees are dropped. Japanese is scored as Han + hiragana + katakana together, so kana and kanji are not split against Latin. ''--demo'' agrees on a German page, fails on ''lang="en"'' with Han visible text, reports undeclared on a tagless English page, ignores a hidden Han paragraph on an English page, fails ''lang="ka"'' as unsupported, accepts Serbian Cyrillic for ''lang="sr"'', and treats a Japanese page whose kana+kanji outnumber Latin as a match. | The script below compares the BCP-47 primary tag on ''<html lang>'' with a Unicode-script majority over visible text (the LangCrUX idea, not their VPN crawl). A missing tag is reported, not a failure — absence is not a mismatch. An unknown primary tag is a failure: the checker does not silently agree with a language it cannot score. Hidden, ''aria-hidden'', and inline ''display:none'' / ''visibility:hidden'' subtrees are dropped. Japanese is scored as Han + hiragana + katakana together, so kana and kanji are not split against Latin. ''%%--demo%%'' agrees on a German page, fails on ''lang="en"'' with Han visible text, reports undeclared on a tagless English page, ignores a hidden Han paragraph on an English page, fails ''lang="ka"'' as unsupported, accepts Serbian Cyrillic for ''lang="sr"'', and treats a Japanese page whose kana+kanji outnumber Latin as a match. |
| |
| <file python check_page_language.py> | <file python check_page_language.py> |
| </file> | </file> |
| |
| Ran here on 2026-08-27: ''--demo'' exit 0; the mismatch fixture is 11 Han vs 7 Latin characters. Extra fixtures: hidden Han ignored; ''lang="ka"'' unsupported; Japanese family 10 vs Latin 5; Serbian Cyrillic accepted. | Ran here on 2026-08-27: ''%%--demo%%'' exit 0; the mismatch fixture is 11 Han vs 7 Latin characters. Extra fixtures: hidden Han ignored; ''lang="ka"'' unsupported; Japanese family 10 vs Latin 5; Serbian Cyrillic accepted. |
| |
| For long text after you know you are looking at prose, run langdetect or Lingua on the extracted visible text, not on the raw HTML. Hosseini et al. stripped URLs and required ten tokens before trusting a label {[hosseini2021unifying]}. | For long text after you know you are looking at prose, run langdetect or Lingua on the extracted visible text, not on the raw HTML. Hosseini et al. stripped URLs and required ten tokens before trusting a label {[hosseini2021unifying]}. |
| [[https://github.com/LibreTranslate/LibreTranslate|LibreTranslate]] is an HTTP API in front of [[https://github.com/argosopentech/argos-translate/|Argos Translate]]. README fetched 2026-08-27: the engine is still Argos, not Google. Detection is still langdetect for long strings (see above). | [[https://github.com/LibreTranslate/LibreTranslate|LibreTranslate]] is an HTTP API in front of [[https://github.com/argosopentech/argos-translate/|Argos Translate]]. README fetched 2026-08-27: the engine is still Argos, not Google. Detection is still langdetect for long strings (see above). |
| |
| The previous compose on this page used ''libretranslate/libretranslate:latest''. That tag's Hub ''last_updated'' on 2026-08-27 was **2026-08-23**. A tag that moved four days ago is not a pin. [[Programming:Docker]] is the digest page; ''check_crawler_compose.py --all'' digest-pins every service, including this one. The compose below pins the **arm64** digest from that day (this host is aarch64). amd64 that day was ''sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca''. Re-fetch before you copy either line. | The previous compose on this page used ''libretranslate/libretranslate:latest''. That tag's Hub ''last_updated'' on 2026-08-27 was **2026-08-23**. A tag that moved four days ago is not a pin. [[Programming:Docker]] is the digest page; ''check_crawler_compose.py %%--all%%'' digest-pins every service, including this one. The compose below pins the **arm64** digest from that day (this host is aarch64). amd64 that day was ''sha256:8402eea7845b6890acdd4b1fb70320c130d2771935d36d697a901e8c149682ca''. Re-fetch before you copy either line. |
| |
| <file yaml libretranslate.compose.yaml> | <file yaml libretranslate.compose.yaml> |
| </file> | </file> |
| |
| ''docker compose up --detach'', wait until the service is ''healthy'' (first model download is slow), then use the documented API. LibreTranslate will detect-and-translate in one call; that detection is langdetect, not Lingua. | ''docker compose up %%--detach%%'', wait until the service is ''healthy'' (first model download is slow), then use the documented API. LibreTranslate will detect-and-translate in one call; that detection is langdetect, not Lingua. |
| |
| This host has no Docker binary. The digest was fetched from the Hub API, not by pulling the image. We did not run a translation through this compose here. | This host has no Docker binary. The digest was fetched from the Hub API, not by pulling the image. We did not run a translation through this compose here. |
