Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, Lidong Bing — The Twelfth International Conference on Learning Representations (ICLR 2024). Verified via dblp (dblp.org/rec/conf/iclr/0010ZPB24), the ICLR proceedings abstract page, and the paper’s GitHub data release (DAMO-NLP-SG/multilingual-safety-for-LLMs), which is tagged [ICLR 2024].

Core Contribution

This is the foundational multilingual-jailbreak paper — the one nearly every later multilingual red-teaming paper (including CSRT, which I’ve also logged) cites as its starting point. Its contribution is twofold: (1) it names and separates two distinct multilingual jailbreak scenarios that had been conflated before, and (2) it introduces MultiJail, the first dataset built specifically to measure both.

Method

Two scenarios, deliberately kept distinct:

  • Unintentional: an ordinary (non-adversarial) user queries the model in a non-English language and inadvertently gets unsafe content the same query would have been refused for in English.
  • Intentional: an attacker deliberately combines malicious instructions with multilingual framing to actively bypass safety mechanisms.

To measure both, the authors built MultiJail: 315 unsafe English prompts sampled from existing red-teaming/harmful-query datasets, manually translated by native speakers (not machine translation) into 9 non-English languages spanning high-, medium-, and low-resource tiers. Manual translation matters methodologically — it isolates “does the model behave differently by language” from “did the translation itself introduce noise or degrade the prompt.”

Evaluated against ChatGPT and GPT-4. Headline results: in the unintentional setting, unsafe-output rate rises as language resource level falls, with low-resource languages roughly 3x as likely to produce harmful content as high-resource languages, for both models tested. In the intentional setting, combining multilingual framing with adversarial instructions pushed unsafe-output rates up to 80.92% for ChatGPT and 40.71% for GPT-4 on their test set.

Limitations

  • 9 languages and 315 base prompts is a real dataset but a modest one; generalization to the full space of the world’s low-resource languages (thousands, many with far less digitized text than the paper’s low-resource tier) is untested.
  • Evaluated only against ChatGPT and GPT-4 (2023-era) — no open-weight models, and no quantized variants, so it says nothing directly about compression interactions.
  • Manual translation by native speakers is a strength for validity but limits scale/reproducibility versus an automated pipeline; also worth asking whether translator framing choices (register, directness) could shift a query’s apparent harmfulness independent of language per se.

Relevance to My Niche

This is about as central as it gets for the multilingual half of my niche. It’s the paper that established that language-resource level is itself a safety variable, not just a translation-quality variable — and it did so with a controlled dataset rather than anecdote. Every later multilingual red-teaming paper I’ve found (e.g., CSRT’s code-switching extension) is implicitly answering “yes, and it’s worse than you think” to the question this paper first posed cleanly. For the quantization side of my work, it also sets up an obvious open question this paper does not answer: does quantizing a model widen or narrow the ~3x high-resource/low-resource safety gap it documents? That’s exactly the intersection my niche sits in, and I haven’t yet found a Tier-1 paper that directly tests quantization x multilingual safety jointly (see the “rejected/unverified” notes I’m keeping separately).