How Does Quantization Affect Multilingual LLMs?

Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, Sebastian Ruder — Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida. Cohere For AI. Verified via the ACL Anthology entry (aclanthology.org/2024.findings-emnlp.935).

Core Contribution

This is the paper that sits most directly at the exact intersection my niche is named after — quantization and multilingual behavior, evaluated together rather than as two separate literatures. Prior quantization-robustness work was almost entirely English-centric; this paper is, per its own framing, the first thorough analysis of quantized multilingual LLMs across languages and across model scales. Its most important methodological contribution isn’t a single number — it’s the demonstration that automatic benchmarks systematically underestimate quantization harm, and that you need human evaluation to see the real damage.

Method

Three-pronged evaluation stack, deliberately layered because the authors don’t trust any single one alone: automatic benchmarks, LLM-as-a-judge, and human evaluation on realistic prompts. This lets them directly compare what automatic metrics report against what human raters actually notice. They test across multiple languages and multiple model scales (evaluating whether the quantization harm pattern holds as models get bigger, not just at one size).

Headline findings:

• Harm from quantization is real but disparately distributed across languages — non-Latin-script languages are hit hardest.
• Automatic metrics severely underestimate the damage. Their concrete example: a 1.7% average drop in Japanese on automatic task benchmarks corresponded to a 16.0% drop as judged by human evaluators on realistic prompts — nearly a 10x gap between what the benchmark says and what a human notices.
• Harder tasks degrade fastest — they specifically flag mathematical reasoning as an early casualty of quantization, worse than simpler tasks, and this compounds with the language effect (harder tasks in already-disadvantaged languages degrade the most).

Limitations

• This is a quality/capability degradation study, not a safety/jailbreak study specifically — it measures truthfulness, task performance, and human-perceived quality, not attack success rate or harmful-content elicitation. I want to be precise about this: it does not directly test whether quantization makes a model more jailbreakable in a given language, only that quantization degrades multilingual quality unevenly, with non-Latin-script languages hit hardest. The safety implication is inferential, not measured directly in this paper.
• Findings-track EMNLP, not main conference — Findings papers are still peer-reviewed and are widely treated as legitimate ACL-family venue publications, but it’s a notch below a main-conference oral/poster in prestige; noting this rather than glossing over it.
• The specific quantization methods/bit-widths tested weren’t confirmed from the page I fetched — I’d want to check the PDF directly for exactly which schemes (e.g., int8, int4, specific PTQ methods) and which model families were used before citing method-level specifics beyond what’s above.

Relevance to My Niche

This is close to a direct hit on my specific research niche as literally stated — quantization x multilingual, evaluated jointly, from a serious industry lab (Cohere For AI) at a Tier-1 NLP venue. The finding that automatic metrics dramatically underestimate real-world harm is itself directly useful methodologically: it’s a strong argument that any of my own quantization/safety evaluation work needs a human-evaluation or realistic-prompt component and can’t rely on automatic benchmarks alone to claim a model is “safe enough” post-quantization. The open gap this paper leaves — and the one my niche could fill — is that it measures quality/harm broadly rather than jailbreak/attack-success specifically. A natural next step in my own reading list: is there a Tier-1 paper that combines this paper’s multilingual-quantization framing with an actual attack-success-rate evaluation (rather than quality degradation)? I looked and did not find one I could verify as genuinely Tier-1 and published (see rejected list) — that combination looks like an open gap in the literature rather than something I failed to find.