Context
There are roughly 7,170 living languages in the world. Large language models are trained mostly on English, and the extra training that teaches them to refuse harmful requests is more English-centric still. A model that reliably declines a dangerous request in English will sometimes answer the same request in a language with less training data behind it.
What I wanted to know is whether the evidence separates two things that are easy to confuse: how common a language is, and what writing system it uses. Chinese has enormous training data but a non-Latin script. Swahili has far less data but uses the Latin alphabet. If compression damages models differently depending on the script, then a result attributed to “low-resource languages” might really be a result about non-Latin characters, and those two stories imply different fixes.
This note is the language-side companion to Common Failure Modes of Quantization Techniques. Same method: read down the Disadvantages column of the combined survey, this time asking what the papers cover across languages and scripts.
The coverage is thin. Of 25 surveyed papers, four have any multilingual dimension, and the multilingual safety evidence is two papers, neither of which touches quantization. The largest study covers ten languages, about 0.14% of living languages.
My own high and low resource tiers each mix scripts, so the gap I am preparing to measure cannot distinguish a resource effect from a script effect. The fix costs nothing: the six languages I already chose form a rough two-by-two.
The harmful prompts underneath all of this are English prompts in translation. No surveyed paper tests harmful requests written natively in a low-resource language, so the field measures whether models refuse English harm concepts rendered in other languages, not whether they refuse harm as those language communities would frame it.
Terms: primer. Study design this feeds: research log.
Scope
The living-language count is from Ethnologue, 29th edition, 2026: 7,170. Glottolog counts up to 7,900, the difference being where mutual intelligibility draws the line between language and dialect. Neither number comes from a surveyed paper.
| Scope | Languages | Share of 7,170 |
|---|---|---|
| MultiJail: 9 non-English plus English | 10 | 0.14% |
| CSRT: up to 10 mixed within one prompt | 10 | 0.14% |
| Al Hakim, the only quant × multilingual safety audit | 3 (EN/KO/AR) | 0.04% |
| My \(\Delta_{\mathrm{HL}}\) design | 7 | 0.10% |
| Union across all 25 surveyed papers | ~12 to 15 | ~0.2% |
The two tens are not the same measurement. MultiJail evaluates 9 discrete non-English languages one at a time. CSRT’s ten are languages that can be mixed inside a single prompt, so its coverage of any one language is shallower by design. Reading them as one number overstates both.
Four of 25 papers carry any multilingual dimension: MultiJail (#2), CSRT (#3), Marchisio (#6), Al Hakim (#7). Marchisio measures quality rather than safety. Al Hakim covers three languages. So the multilingual safety evidence base is two papers, and neither quantizes anything.
Both name the ceiling themselves. MultiJail: “9 languages and 315 base prompts is modest against thousands of low-resource languages.” CSRT: “ten languages is a bounded sample of real community-specific code-switching.”
Script and resource tier are confounded in my tiers
Marchisio’s script-level finding is that non-Latin scripts are hit hardest by quantization. That is the only script-level result in the survey, and it is about output quality, not refusal.
My tiers, from the research log:
| Tier | Languages | Writing systems |
|---|---|---|
| High | zh, it, vi | Han, Latin, Latin with diacritics |
| Low | bn, sw, jv | Bengali, Latin, Latin |
Each tier contains both Latin and non-Latin scripts. So a \(\Delta_{\mathrm{HL}}\) that widens under Q4_K_M supports two readings that my current design cannot separate: quantization damages low-resource safety, or quantization damages non-Latin script processing and my low tier happens to contain Bengali.
The six languages already chosen form a two-by-two:
| Latin | Non-Latin | |
|---|---|---|
| High | it, vi | zh |
| Low | sw, jv | bn |
Two cells hold one language each, so this is underpowered as a primary endpoint and I would not claim it as one. As a preregistered secondary contrast it costs nothing, since the runs happen anyway. It tests directly whether Marchisio’s quality finding carries over to refusal, and it answers the reviewer question before it is asked.
The decision this forces: if zh degrades like bn rather than like it and vi, then \(\Delta_{\mathrm{HL}}\) is partly a script measurement wearing a resource-tier label, and the headline claim has to be restated. Adding one non-Latin low-resource language and one more Latin high-resource control would make the decomposition properly identified. That is the cheapest design change available to me right now.
Seven common disadvantages
1. Every harm concept is defined in English
MultiJail is 315 English prompts translated into 9 languages by native speakers. CSRT synthesizes code-switched queries. Al Hakim reuses MultiJail’s translations for EN/KO/AR. No surveyed paper uses harmful prompts natively authored in a low-resource language.
So the field measures whether a model refuses an English harm concept rendered in Swahili. It does not measure whether the model refuses harm as Swahili speakers would frame it. MultiJail’s note flags the narrower version of this risk: “translator register choices could shift apparent harmfulness independent of language.”
This is a limitation I inherit rather than solve, since I am reusing MultiJail. Worth stating in the paper rather than leaving for a reviewer.
2. Tier means rest on three languages each
MultiJail’s roughly 3x low-versus-high unsafe-rate gap, and my \(\Delta_{\mathrm{HL}}\), both aggregate three languages per tier. Within-tier variance goes unreported everywhere in the survey.
Wilson intervals on per-cell ASR, which src/analysis/metrics.py already computes, do not capture uncertainty in the tier mean when the tier holds three languages. A leave-one-language-out check would show whether a single language carries the result. Cheap, and it closes a hole that a careful reviewer will find.
3. The multilingual baseline was never measured on an open-weight model
MultiJail evaluated ChatGPT and GPT-4, both 2023-era closed models. CSRT evaluated ten LLMs whose identities my note records as unconfirmed. Neither can be quantized.
So the ~3x gap motivating this whole project has never been established on a model anyone can compress. My BF16 arm is not a replication of MultiJail. It is the first open-weight measurement of that gap, and claiming it as such is more honest and more interesting than treating it as a baseline check.
4. Judge competence is weakest where the effect is largest
Al Hakim uses a single Gemini 2.5 Flash Lite judge over 3 seeds per instruction, with no inter-judge reliability reported. Marchisio shows automatic evaluation understating quantization damage roughly tenfold on Japanese (1.7% versus 16.0% human-judged).
Nobody validates judge accuracy per language. In Javanese under Q4_K_M, the model and the judge are both at their weakest simultaneously, and the direction of the resulting error is unknown. This is the same measurement problem the quantization companion reaches from the compression side, arriving at the same cell of the grid.
5. Code-switching is untested under any compression
CSRT is the most naturalistic multilingual attack in the survey, since mixing languages mid-sentence is something multilingual speakers do rather than an engineered construction. It beats monolingual English red-teaming by 46.7%.
Its note states the gap plainly: “no quantization or compression connection, leaving that intersection open.” A code-switched arm under Q4_K_M is a natural Phase B extension, and CSRT’s joint comprehension measurement would partly address the fluency confound in H3.
6. Character-level attacks are sealed off from multilingual safety
The only character-level attack in the survey is P³A (#17), which perturbs about 1% of characters to promote a poisoned document’s reranker position. It is English-only with, in its own note’s words, “no safety/refusal or multilingual analysis.”
Its recorded weakness is that Unicode folding and spell-correction in production indexing may blunt it. That normalization layer behaves differently across scripts, which is exactly where a multilingual version of the question would start. Nobody has connected the two literatures.
7. What is absent entirely
No surveyed paper examines tokenization as a variable, homoglyph or Unicode-confusable attacks on refusal, script-swapping and romanization (Arabizi, Hinglish written in Latin characters), or diacritic stripping.
Romanization deserves attention because it is a real input pattern rather than an invented one. Millions of people type their language in Latin characters daily. Under Jailbroken’s mismatched-generalization account, an input form that is common in use but rare in safety training data is precisely where refusal should fail, and romanized low-resource text fits that description exactly.
Unverified
Tokenizer fertility as a mechanism. I suspect that tokenizer vocabularies fit to English-heavy corpora give non-Latin and low-resource languages higher fertility, meaning more tokens per unit of meaning, and that the same content therefore passes through more quantized forward steps and accumulates more error per unit of meaning. That would predict Marchisio’s non-Latin finding directly, and would predict \(\Delta_{\mathrm{HL}}\) tracking fertility more tightly than tier labels.
No surveyed paper supports any part of this. Tokenization appears nowhere in the survey as a variable. The mechanism is my own reasoning and is not evidence for anything until it is checked.
It is cheap to check. Run the MultiJail prompts through the target tokenizer, compute tokens per character per language, and regress \(\Delta_{\mathrm{HL}}\) against it. No GPU required, and my existing design already freezes the tokenizer across arms, so fertility is a stable per-language constant measured once. If it explains the gap better than tier labels do, the paper gets sharper. If it does not, a confound is ruled out. Worth doing before the GPU arms run.
Ranked Openings
- Script versus resource decomposition, available from the runs I am already doing, and necessary to state what \(\Delta_{\mathrm{HL}}\) actually measures.
- First open-weight measurement of the multilingual safety gap, since MultiJail and CSRT used closed models. This reframes the BF16 arm from baseline check to contribution.
- Per-language judge validation, which nothing in the survey provides and which failure 4 makes load-bearing.
- Code-switching under compression, the CSRT × quantization cell, explicitly flagged open in CSRT’s own note.
- Romanized and script-swapped input, untouched by the survey and well predicted by Jailbroken’s theory.
Open thread
My low tier, chosen to align with MultiJail’s resource bands, is bn, sw, jv, and two of those three are Latin-script. If the low-resource safety gap turns out to be substantially a script effect, then a low-resource tier built this way is partly measuring writing systems, and the ~3x figure would mean something narrower than it appears.
I do not think that is the whole story, since Swahili and Javanese are Latin-script and still show the gap. And I cannot yet say whether MultiJail’s own tier assignment matches mine: my note on that paper records the 9-language, 3-tier structure but not which languages sit in which band. That needs checking against the dataset release before I make any claim about what the field’s standard tier has been measuring. What I can say is that the question is testable with runs I have already planned.