Haneul Yoo, Yongjin Yang, Hwaran Lee — Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025, Volume 1: Long Papers), pp. 13392–13413, Vienna, Austria. Verified via the ACL Anthology entry (aclanthology.org/2025.acl-long.657).

Provenance note, worth being explicit about: this work was first presented as a poster at the Red Teaming GenAI workshop at NeurIPS 2024 (confirmed via the NeurIPS 2024 virtual site) — that workshop appearance alone would not meet my Tier-1 bar. It was subsequently accepted and published as a full long paper at the ACL 2025 main conference, which does clear the bar. I’m citing the ACL 2025 version and venue, not the NeurIPS workshop appearance, per the “if later published at a Tier-1 venue, cite that venue” rule.

Core Contribution

Introduces code-switching — mixing multiple languages within a single query, a genuinely common natural multilingual phenomenon rather than an artificial attack construct — as a red-teaming technique in its own right. The insight is that code-switched queries stress-test both a model’s multilingual understanding and its safety alignment simultaneously, and that this combination is harder for safety training to cover than either monolingual non-English queries or naive translation-based attacks (like the “translate into a low-resource language” approach from the MultiJail line of work).

Method

The authors built CSRT, a framework that synthesizes code-switching red-teaming queries combining up to 10 languages within a single prompt. They evaluate against ten state-of-the-art LLMs. Headline result: CSRT queries elicit 46.7% more successful attacks than standard (monolingual English) red-teaming attacks, while also serving as a joint probe of multilingual generation/comprehension quality — i.e., the paper measures not just whether the model breaks safety, but whether it even understood the mixed-language input coherently. They report what they describe as an “unintended correlation between resource availability of languages and safety alignment in existing multilingual LLMs” — meaning models whose safety training leans on high-resource-language data show the code-switching gap most starkly when lower-resource languages are mixed in.

I confirmed the dataset scope (up to 10 languages, 10 models) and the 46.7% headline figure directly from the abstract. I was not able to confirm the exact dataset size (number of query items) or the full model list from the pages fetched — flagging as unconfirmed pending a direct PDF read.

Limitations

  • The paper’s own framing surfaces the resource/safety correlation as a finding rather than fully explaining its mechanism — it demonstrates the effect more than it isolates why code-switching specifically (versus just non-English input generally) has the marginal effect it does.
  • Ten languages and ten models is a solid empirical footprint for an ACL paper but still a bounded sample of the world’s actual code-switching patterns, which are highly community- and region-specific (the paper doesn’t claim otherwise).
  • No connection to quantization or model compression — this is a pure multilingual-safety paper; the intersection with quantized models remains an open question this line of work hasn’t addressed yet.

Relevance to My Niche

This is one of the most directly on-target papers I’ve found for the multilingual half of my niche — arguably more precise than translation-based multilingual jailbreak papers, because code-switching is a naturalistic input pattern (something real multilingual speakers actually produce), not an artificial adversarial construction. That matters for red-teaming methodology broadly: it argues that some of the most effective “attacks” aren’t adversarially engineered at all, they’re just underrepresented natural inputs. Pairs directly with the MultiJail paper (ICLR 2024, also logged) as two complementary angles — full-sentence translation vs. within-sentence mixing — on the same underlying claim that safety alignment is unevenly distributed across languages and that the unevenness itself is exploitable.