Refusal Is Mediated by a Single Direction

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda: Refusal in Language Models Is Mediated by a Single Direction, NeurIPS 2024 (proceedings; OpenReview id pH3XAQME6c). Code: andyrdt/refusal_direction.

Survey table

Core Contribution

Refusal in chat models concentrates in a low-dimensional residual-stream direction rather than spreading across the full network. Across 13 open chat models they extract one difference-in-means direction such that:

• Ablating that direction (or the equivalent rank-one weight edit) sharply drops refusal / safety scores on harmful instructions.
• Adding the direction induces refusal even on harmless instructions.

They also mechanistically analyze how adversarial suffixes suppress propagation of the refusal direction. Safety fine-tuning is brittle once that direction is editable with a rank-one edit.

Models (Table 1, 13 total, 1.8B-72B): Qwen Chat 1.8B/7B/14B/72B and Yi Chat 6B/34B (aligned by fine-tuning, AFT); Gemma IT 2B/7B, Llama-2 Chat 7B/13B/70B, Llama-3 Instruct 8B/70B (aligned by preference optimization, APO). The AFT/APO split is deliberate: the single-direction result holds across both alignment recipes.

Method (as published)

Building the direction

• Datasets. \(\mathcal{D}_{\text{harmful}}\) is drawn from AdvBench, MaliciousInstruct, TDC2023, and HarmBench; \(\mathcal{D}_{\text{harmless}}\) is sampled from Alpaca. Each has a 128-sample train split and 32-sample validation split, filtered so train/val do not overlap the §3-§4 evaluation sets.
• Difference-in-means. At every layer \(l\in[L]\) and every post-instruction token position \(i\in I\) (the chat-template tokens after the user instruction), compute mean harmful activation \(\mu_i^{(l)}\) minus mean harmless activation \(\nu_i^{(l)}\). The vector \(r_i^{(l)}=\mu_i^{(l)}-\nu_i^{(l)}\) is meaningful in both its direction (where harmful vs harmless activations separate) and magnitude.
• Selecting one vector. The \(|I|\times L\) candidate vectors are scored on the validation harmful/harmless splits: pick the single \(r_{i^*}^{(l^*)}\) that best bypasses refusal when ablated and best induces refusal when added, subject to minimal collateral change. The chosen unit-norm vector is \(\hat r\).

Three causal interventions

1. Activation addition (Eq. 3): add \(r^{(l)}\) to activations at the extraction layer \(l\), all token positions, to induce refusal.
2. Directional ablation (Eq. 4): \(x' \leftarrow x - \hat r\hat r^\top x\) at every layer and every token position, erasing the direction from the residual stream to bypass refusal.
3. Weight orthogonalization (Eq. 5): the same effect baked into weights, \(W'_{\text{out}} \leftarrow W_{\text{out}} - \hat r\hat r^\top W_{\text{out}}\), applied to every matrix that writes to the residual stream (embedding, positional, attention out, MLP out). This is the rank-one white-box jailbreak: no gradient steps, no harmful fine-tuning examples.

Evaluation

• Decoding: greedy, max 512 new tokens.
• Refusal score: string match against a bank of refusal substrings (“I’m sorry”, “As an AI”, …) → 1/0.
• Safety score: Meta Llama Guard 2 classifies each completion as safe/unsafe.
• Ablation evaluated over 100 harmful instructions from JailbreakBench; addition over 100 harmless instructions from Alpaca.

This is representation / activation control, not training a subset of layers with GRPO/SFT. It complements Safety Layers / ESI (parameter or layer localization) with a feature-direction story.

Key results

• Directional ablation (Fig. 1): drops refusal and safety scores sharply across all 13 models; e.g. Qwen-1.8B refusal score ~0.7→~0.15.
• Weight orthogonalization vs other jailbreaks (HarmBench test set, 159 standard behaviors, Table 2): ORTHO is a general jailbreak (one edit, all prompts) competitive with prompt-specific GCG. Qwen 14B 84.3 ASR, Qwen 7B 79.2, Qwen 72B 78.0; Llama-2 models are far lower with their safety system prompt (Llama-2-7B 22.6) but jump without it (79.9), a large system-prompt sensitivity Qwen models don’t show.
• Coherence preserved (Table 3, LM Eval Harness): orthogonalized models stay within ~99% CI of baseline on MMLU, ARC, GSM8K (plus WinoGrande / TinyHellaSwag in appendix). TruthfulQA consistently drops (e.g. Llama-3-70B 59.5 vs 61.8), attributed to TruthfulQA categories (misinformation, conspiracies) sitting near refusal territory. CE-loss evals suggest ablation is more surgical than activation addition.
• Adversarial suffixes (§5, on Qwen-1.8B Chat, 128 harmful prompts from JailbreakBench + HarmBench): a working GCG-style suffix suppresses the refusal direction’s expression down to harmless-instruction levels (Fig. 5), whereas a random suffix of equal length does not. Via direct feature attribution on the top-8 refusal-writing attention heads, the suffix hijacks attention away from the instruction tokens toward the suffix tokens (Fig. 6), which is why the direction never gets written.

Limitations / gaps for my niche

• Mechanistic English open-weight study; not a MultiJail resource-tier paper.
• Not a quantization paper: no systematic INT4/GGUF ASR tables after ablating or reinforcing the direction.
• Jailbreak here is white-box direction ablation, not red-teaming quantized local deploys.

Relevance

Tier-1 cite for “refusal is low-dimensional and editable.” Explains why middle-layer / sparse safety interventions can work at all, and why compression might scramble a thin direction (hypothesis, not their experiment). Does not by itself answer quantization × multilingual ASR, and does not train localized safety then re-quantize.