Alexander Wei, Nika Haghtalab, Jacob Steinhardt — Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Oral presentation. UC Berkeley. Verified via the NeurIPS proceedings page (papers.nips.cc) and the NeurIPS 2023 oral/poster listings.

Core Contribution

This is the paper that gave the field a principled why, rather than just a growing pile of how, for jailbreaks. Instead of another attack technique, it proposes two structural failure modes that explain why safety training fails in the first place, then uses that theory generatively — designing new attacks from the theory and showing they work. That combination (explanatory theory + constructive validation) is what makes this a foundational reference rather than one more entry in a jailbreak zoo.

Method

Two hypothesized failure modes:

  • Competing objectives: the model is simultaneously optimized for capability/helpfulness and for safety/harmlessness, and these objectives can be placed in direct tension by a prompt — e.g., asking the model to demonstrate a capability in a way that requires violating a safety constraint. The prompt exploits the trade-off rather than any specific hole in a safety classifier.
  • Mismatched generalization: safety training is applied over some training distribution, but the model’s raw capabilities generalize further than the safety fine-tuning does. Any input that lands in “capability reaches but safety training doesn’t” is a gap, almost by construction — e.g., encodings, low-resource framings, or unusual formatting that the base model still understands but that safety fine-tuning data never covered.

The authors use these two lenses to design new jailbreaks (rather than just categorize known ones post hoc), then evaluate them against GPT-4 and Claude v1.3, comparing against existing jailbreak techniques and against the models’ own internal red-team/unsafe-prompt evaluation sets. They report their attacks succeed consistently where prior techniques were being patched out, and argue for “safety-capability parity”: safety mechanisms need to be as sophisticated as the underlying capabilities, because naive scaling of safety training will not close a mismatched-generalization gap that scales with capability itself.

Limitations

  • The taxonomy is a framework, not a fully mechanistic account — it explains classes of failure without predicting in advance exactly which prompts will land in a gap.
  • It’s evaluated against models and defenses current to late 2023 (GPT-4, Claude v1.3); the specific jailbreaks demonstrated are largely patched by now, though the two failure-mode categories have proven durable as an analytic lens in later work.
  • “Safety-capability parity” is stated as a design principle/goal rather than a method with a concrete evaluation protocol — it’s more agenda-setting than a benchmark you can directly reuse.

Relevance to My Niche

Load-bearing for the whole red-teaming side of my niche, and it connects both sub-threads directly:

  • Multilingual: “mismatched generalization” is arguably the cleanest theoretical account of why low-resource-language jailbreaks work at all — safety fine-tuning data skews toward English/high-resource languages, so capability (which the base model has broadly, from pretraining) outruns safety training in exactly the languages where fine-tuning coverage is thinnest. The multilingual jailbreak papers I’ve logged (MultiJail, CSRT) are, in this paper’s terms, empirical instances of mismatched generalization along the language axis.
  • Quantization: less directly, but “competing objectives” gives useful vocabulary for describing why quantization-induced safety drift happens even without adversarial intent — compression can quietly shift the balance between the helpfulness and refusal objectives that safety tuning had balanced at full precision.

I’d treat this as the theoretical spine to cite before the more specific, mechanism-level papers.