Read the paper PDF
Open the PDF in a new tab If the panel above stays blank, the publisher does not allow embedding. The link always works.
Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren: Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models, Findings of ACL 2026, San Diego (Anthology; DOI 10.18653/v1/2026.findings-acl.1616; arXiv:2604.08297). Affiliations: Zhejiang University / Binjiang Institute; NTU Singapore. Code: ZJU-LLM-Safety/SafeWeights-ACL.
Survey table
| Column | Entry |
|---|---|
| Specific authors | Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren (Zhejiang University / Binjiang Institute; NTU Singapore) |
| Objectives | Identify individual safety-critical parameters via an Expected Safety Impact score, then intervene sparsely: install safety by tuning only those weights (SET) or preserve safety by freezing them during task fine-tuning (SPA) |
| Methodology | ESI(θᵢ) = |σ(θᵢ)·∇θᵢS| from a first-order Taylor expansion of a safety score; the score is made differentiable via Gumbel-softmax relaxation plus a vocabulary projection into Llama-Guard-3-8B as judge, so gradients come from a single checkpoint; validation by Gaussian-noise perturbation of top-1% ESI weights vs SNIP/Wanda/GMT baselines; SET fine-tunes the top 1% on safety data, SPA freezes them |
| Dataset | Dense Llama3-8B/70B-it, Qwen2.5-14B; MoE Qwen3-30B-A3B (plus Mixtral-8x7B, Qwen3-235B in Fig. 2); ESI estimated on AdvBench; degradation measured on HarmBench and WildJailbreak; SET safety data from CB-Safety and R1-Safety |
| Open source | Yes (ZJU-LLM-Safety/SafeWeights-ACL) |
| Advantages | Weight-level (not layer-level) resolution with a statistically grounded variation scale (parameter std, not magnitude); needs no aligned/unaligned model pair; strong causal validation (perturbing top-1% ESI raises HarmBench ASR 15.3 to 59.1 where baselines reach at most 37.6); SET cuts WildJailbreak ASR 62.5% to 19.1% updating 1% of weights; covers dense and MoE architectures |
| Disadvantages | English-centric safety benches, no multilingual resource tiers; no quantization or GGUF remeasure after SET/SPA; differentiable-judge pipeline adds machinery (Gumbel-softmax temperature, vocab projection) whose sensitivity is only partly ablated; safety score depends on one judge family (Llama-Guard) with GPTfuzz as the sole cross-check |
Core Contribution
This paper scores individual parameters by Expected Safety Impact, \(\mathrm{ESI}(\theta_i) \triangleq |\sigma(\theta_i)\nabla_{\theta_i} \mathcal{S}(\theta)|\) (Eq. 3), from a first-order Taylor expansion \(\Delta\mathcal{S}\approx\nabla_\theta\mathcal{S}^\top\Delta\theta\). \(\mathcal{S}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{\text{harm}}}\mathbb{E}_{y\sim p_\theta}[s(y)]\) is expected safety over a harmful-prompt distribution; the novelty vs prior gradient (\(|\nabla\mathcal{L}|\)) or magnitude-weighted (\(|\theta\nabla\mathcal{L}|\)) rankings is (a) using the parameter’s own std \(\sigma(\theta_i)\) as a statistically grounded variation scale instead of assuming uniform/magnitude-proportional \(\Delta\theta\), and (b) optimizing a safety score rather than a generic CE loss. Whole-layer freezing is not the unit of analysis.
Where safety lives (Fig. 2, aggregated per-layer ESI):
- Dense models: safety-critical mass in middle-layer self-attention value (V) matrices and MLPs.
- MoE models: shifts toward late-layer MLP experts.
Interventions:
- SET (Safety Enhancement Tuning): fine-tune only the top-\(k\%\) (k=1%) ESI weights on safe data, freezing the rest, minimizing an SFT-style safety-alignment loss (Eq. 10).
- SPA (Safety Preserving Adaptation): freeze the ESI-critical subset; tune the rest for instruction/task FT so safety degradation stays within ~1% after 1,000 iterations (claimed).
Method notes
- Differentiable safety score. Generation is non-differentiable (discrete sampling), so they relax \(y\)’s tokens with a Gumbel-softmax \(\tilde y=\mathrm{Softmax}((l+g)/\tau)\) (Eq. 7) and bridge the target-LLM and judge vocabularies with a binary projection matrix \(M\in\{0,1\}^{V_\mathcal{J}\times V}\) (Eq. 8). Safety score \(s(y)=P_\mathcal{J}(\text{safe}\mid y)\) from a judge model. This gives \(\nabla_\theta\mathcal{S}\) from a single checkpoint (no aligned/unaligned model pair, unlike Chen et al. neuron methods).
- Judge / eval stack. ESI is estimated over AdvBench prompts with Llama-Guard-3-8B as the differentiable judge (GPTfuzz verified to give similar rankings, App. B.5). ASR at eval time judged with GPT-4o (Zeng et al. method).
- SET/SPA are sparse fine-tuning recipes, not full-network RLHF and not Zhang-style single-layer GRPO with a \(\mathcal{C}(k)\) scan.
Models and datasets
- Verification (perturbation) models: dense Llama3-8B-it, Llama3-70B-it, Qwen2.5-14B-base; MoE Qwen3-30B-A3B-it (Fig. 2 adds Mixtral-8x7B-it, Qwen3-235B-A22B-it). Safety degradation measured on HarmBench and WildJailbreak.
- SET models: base Qwen2.5-7B, Qwen2.5-14B, Llama3-8B (no prior safety alignment). Safety data: CB-Safety and R1-Safety. k=1%, 100 iterations. Baselines: Random, SN-Tune, LoRA, SafeLoRA.
Key numbers
- ESI ranking validated by perturbation (Table 1): adding Gaussian noise to the top-1% ESI weights of Llama3-8B-it raises HarmBench ASR 15.3 → 59.1, while SN/GMT/Wanda/SNIP baselines reach ≤37.6 and random perturbation barely moves ASR. Same pattern on Qwen2.5-14B, Llama3-70B, and the MoE model.
- SET (Table 2): on Llama3-8B with R1-Safety, SET cuts WildJailbreak ASR 62.5% → 19.1%, vs the strongest baseline’s 37.4%, updating only 1% of weights in 100 iterations.
Limitations / gaps for my niche
- No published evidence here of PTQ/GGUF remeasure after SET/SPA.
- No multilingual / MultiJail resource-tier ASR (English-centric safety benches from the abstract/body).
- Parameter-sparse safety ≠ stress-test of localized safety under quantization across languages.
Relevance
Tier-1 citation that safety-critical weights concentrate mid-stack in dense models and that tiny updates can install or protect refusal. Closest published neighbor to “train safety where it matters.” Still leaves the empty cell: localized safety intervention × bit-width × language. Use in related work for Proposal 2; do not treat as scooping Proposal 1.