Alexander Wei, Nika Haghtalab, Jacob Steinhardt — Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Oral presentation. UC Berkeley. Verified via the NeurIPS proceedings page (papers.nips.cc) and the NeurIPS 2023 oral/poster listings.
Survey table
| Column | Entry |
|---|---|
| Specific authors | Alexander Wei, Nika Haghtalab, Jacob Steinhardt (UC Berkeley) |
| Objectives | Explain why safety-trained LLMs still jailbreak by proposing two structural failure modes (competing objectives; mismatched generalization), then validate the theory generatively by designing new attacks from it |
| Methodology | Theory-first: hypothesize that (1) helpfulness and harmlessness objectives can be placed in direct tension by a prompt, and (2) capabilities generalize further than safety fine-tuning coverage, so any input in the gap succeeds almost by construction; design attacks from each lens and evaluate against GPT-4 and Claude v1.3, comparing with existing jailbreaks and the models’ internal red-team sets |
| Dataset | No released dataset; evaluation on curated harmful prompts against two closed 2023-era models |
| Open source | No code or dataset release noted; the attacks are described in the paper itself |
| Advantages | Gives the field a durable why rather than another attack: both failure modes have held up as the standard analytic lens (low-resource jailbreaks are mismatched generalization along the language axis); constructive validation shows the theory predicts, not just post-hoc categorizes; “safety-capability parity” framing remains agenda-setting |
| Disadvantages | A framework, not a mechanistic account: it cannot predict which specific prompts land in a gap; the demonstrated jailbreaks are patched and the evaluated models are outdated; safety-capability parity is a principle without an evaluation protocol |
Core Contribution
This is the paper that gave the field a principled why, rather than just a growing pile of how, for jailbreaks. Instead of another attack technique, it proposes two structural failure modes that explain why safety training fails in the first place, then uses that theory generatively — designing new attacks from the theory and showing they work. That combination (explanatory theory + constructive validation) is what makes this a foundational reference rather than one more entry in a jailbreak zoo.
Method
Two hypothesized failure modes:
- Competing objectives: the model is simultaneously optimized for capability/helpfulness and for safety/harmlessness, and these objectives can be placed in direct tension by a prompt — e.g., asking the model to demonstrate a capability in a way that requires violating a safety constraint. The prompt exploits the trade-off rather than any specific hole in a safety classifier.
- Mismatched generalization: safety training is applied over some training distribution, but the model’s raw capabilities generalize further than the safety fine-tuning does. Any input that lands in “capability reaches but safety training doesn’t” is a gap, almost by construction — e.g., encodings, low-resource framings, or unusual formatting that the base model still understands but that safety fine-tuning data never covered.
The authors use these two lenses to design new jailbreaks (rather than just categorize known ones post hoc), then evaluate them against GPT-4 and Claude v1.3, comparing against existing jailbreak techniques and against the models’ own internal red-team/unsafe-prompt evaluation sets. They report their attacks succeed consistently where prior techniques were being patched out, and argue for “safety-capability parity”: safety mechanisms need to be as sophisticated as the underlying capabilities, because naive scaling of safety training will not close a mismatched-generalization gap that scales with capability itself.
Limitations
- The taxonomy is a framework, not a fully mechanistic account — it explains classes of failure without predicting in advance exactly which prompts will land in a gap.
- It’s evaluated against models and defenses current to late 2023 (GPT-4, Claude v1.3); the specific jailbreaks demonstrated are largely patched by now, though the two failure-mode categories have proven durable as an analytic lens in later work.
- “Safety-capability parity” is stated as a design principle/goal rather than a method with a concrete evaluation protocol — it’s more agenda-setting than a benchmark you can directly reuse.
Relevance to My Niche
Load-bearing for the whole red-teaming side of my niche, and it connects both sub-threads directly:
- Multilingual: “mismatched generalization” is arguably the cleanest theoretical account of why low-resource-language jailbreaks work at all — safety fine-tuning data skews toward English/high-resource languages, so capability (which the base model has broadly, from pretraining) outruns safety training in exactly the languages where fine-tuning coverage is thinnest. The multilingual jailbreak papers I’ve logged (MultiJail, CSRT) are, in this paper’s terms, empirical instances of mismatched generalization along the language axis.
- Quantization: less directly, but “competing objectives” gives useful vocabulary for describing why quantization-induced safety drift happens even without adversarial intent — compression can quietly shift the balance between the helpfulness and refusal objectives that safety tuning had balanced at full precision.
I’d treat this as the theoretical spine to cite before the more specific, mechanism-level papers.