Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren: Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models, Findings of ACL 2026, San Diego (Anthology; DOI 10.18653/v1/2026.findings-acl.1616; arXiv:2604.08297). Affiliations: Zhejiang University / Binjiang Institute; NTU Singapore. Code: ZJU-LLM-Safety/SafeWeights-ACL.
Survey table
| Column | Entry |
|---|---|
| Specific authors | Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren (Zhejiang University / Binjiang Institute; NTU Singapore) |
| Objectives | Identify individual safety-critical parameters via an Expected Safety Impact score, then intervene sparsely: install safety by tuning only those weights (SET) or preserve safety by freezing them during task fine-tuning (SPA) |
| Methodology | ESI(θᵢ) = |σ(θᵢ)·∇θᵢS| from a first-order Taylor expansion of a safety score; the score is made differentiable via Gumbel-softmax relaxation plus a vocabulary projection into Llama-Guard-3-8B as judge, so gradients come from a single checkpoint; validation by Gaussian-noise perturbation of top-1% ESI weights vs SNIP/Wanda/GMT baselines; SET fine-tunes the top 1% on safety data, SPA freezes them |
| Dataset | Dense Llama3-8B/70B-it, Qwen2.5-14B; MoE Qwen3-30B-A3B (plus Mixtral-8x7B, Qwen3-235B in Fig. 2); ESI estimated on AdvBench; degradation measured on HarmBench and WildJailbreak; SET safety data from CB-Safety and R1-Safety |
| Open source | Yes (ZJU-LLM-Safety/SafeWeights-ACL) |
| Advantages | Weight-level (not layer-level) resolution with a statistically grounded variation scale (parameter std, not magnitude); needs no aligned/unaligned model pair; strong causal validation (perturbing top-1% ESI raises HarmBench ASR 15.3 to 59.1 where baselines reach at most 37.6); SET cuts WildJailbreak ASR 62.5% to 19.1% updating 1% of weights; covers dense and MoE architectures |
| Disadvantages | English-centric safety benches, no multilingual resource tiers; no quantization or GGUF remeasure after SET/SPA; differentiable-judge pipeline adds machinery (Gumbel-softmax temperature, vocab projection) whose sensitivity is only partly ablated; safety score depends on one judge family (Llama-Guard) with GPTfuzz as the sole cross-check |
Core Contribution
This paper scores individual parameters by Expected Safety Impact, \(\mathrm{ESI}(\theta_i) \triangleq |\sigma(\theta_i)\nabla_{\theta_i} \mathcal{S}(\theta)|\) (Eq. 3), from a first-order Taylor expansion \(\Delta\mathcal{S}\approx\nabla_\theta\mathcal{S}^\top\Delta\theta\). \(\mathcal{S}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{\text{harm}}}\mathbb{E}_{y\sim p_\theta}[s(y)]\) is expected safety over a harmful-prompt distribution; the novelty vs prior gradient (\(|\nabla\mathcal{L}|\)) or magnitude-weighted (\(|\theta\nabla\mathcal{L}|\)) rankings is (a) using the parameter’s own std \(\sigma(\theta_i)\) as a statistically grounded variation scale instead of assuming uniform/magnitude-proportional \(\Delta\theta\), and (b) optimizing a safety score rather than a generic CE loss. Whole-layer freezing is not the unit of analysis.
Where safety lives (Fig. 2, aggregated per-layer ESI):
- Dense models: safety-critical mass in middle-layer self-attention value (V) matrices and MLPs.
- MoE models: shifts toward late-layer MLP experts.
Interventions:
- SET (Safety Enhancement Tuning): fine-tune only the top-\(k\%\) (k=1%) ESI weights on safe data, freezing the rest, minimizing an SFT-style safety-alignment loss (Eq. 10).
- SPA (Safety Preserving Adaptation): freeze the ESI-critical subset; tune the rest for instruction/task FT so safety degradation stays within ~1% after 1,000 iterations (claimed).
Method notes
- Differentiable safety score. Generation is non-differentiable (discrete sampling), so they relax \(y\)’s tokens with a Gumbel-softmax \(\tilde y=\mathrm{Softmax}((l+g)/\tau)\) (Eq. 7) and bridge the target-LLM and judge vocabularies with a binary projection matrix \(M\in\{0,1\}^{V_\mathcal{J}\times V}\) (Eq. 8). Safety score \(s(y)=P_\mathcal{J}(\text{safe}\mid y)\) from a judge model. This gives \(\nabla_\theta\mathcal{S}\) from a single checkpoint (no aligned/unaligned model pair, unlike Chen et al. neuron methods).
- Judge / eval stack. ESI is estimated over AdvBench prompts with Llama-Guard-3-8B as the differentiable judge (GPTfuzz verified to give similar rankings, App. B.5). ASR at eval time judged with GPT-4o (Zeng et al. method).
- SET/SPA are sparse fine-tuning recipes, not full-network RLHF and not Zhang-style single-layer GRPO with a \(\mathcal{C}(k)\) scan.
Models and datasets
- Verification (perturbation) models: dense Llama3-8B-it, Llama3-70B-it, Qwen2.5-14B-base; MoE Qwen3-30B-A3B-it (Fig. 2 adds Mixtral-8x7B-it, Qwen3-235B-A22B-it). Safety degradation measured on HarmBench and WildJailbreak.
- SET models: base Qwen2.5-7B, Qwen2.5-14B, Llama3-8B (no prior safety alignment). Safety data: CB-Safety and R1-Safety. k=1%, 100 iterations. Baselines: Random, SN-Tune, LoRA, SafeLoRA.
Key numbers
- ESI ranking validated by perturbation (Table 1): adding Gaussian noise to the top-1% ESI weights of Llama3-8B-it raises HarmBench ASR 15.3 → 59.1, while SN/GMT/Wanda/SNIP baselines reach ≤37.6 and random perturbation barely moves ASR. Same pattern on Qwen2.5-14B, Llama3-70B, and the MoE model.
- SET (Table 2): on Llama3-8B with R1-Safety, SET cuts WildJailbreak ASR 62.5% → 19.1%, vs the strongest baseline’s 37.4%, updating only 1% of weights in 100 iterations.
Limitations / gaps for my niche
- No published evidence here of PTQ/GGUF remeasure after SET/SPA.
- No multilingual / MultiJail resource-tier ASR (English-centric safety benches from the abstract/body).
- Parameter-sparse safety ≠ stress-test of localized safety under quantization across languages.
Relevance
Tier-1 citation that safety-critical weights concentrate mid-stack in dense models and that tiny updates can install or protect refusal. Closest published neighbor to “train safety where it matters.” Still leaves the empty cell: localized safety intervention × bit-width × language. Use in related work for Proposal 2; do not treat as scooping Proposal 1.