Weiwei Qi, Zefeng Wu, Tianhang Zheng, Zikang Zhang, Xiaojun Jia, Zhan Qin, Kui Ren: Towards Identification and Intervention of Safety-Critical Parameters in Large Language Models, Findings of ACL 2026, San Diego (Anthology; DOI 10.18653/v1/2026.findings-acl.1616; arXiv:2604.08297). Affiliations: Zhejiang University / Binjiang Institute; NTU Singapore. Code: ZJU-LLM-Safety/SafeWeights-ACL.

Core Contribution

This paper scores individual parameters by Expected Safety Impact, \(\mathrm{ESI}(\theta_i) \triangleq |\sigma(\theta_i)\nabla_{\theta_i} \mathcal{S}(\theta)|\) (Eq. 3), from a first-order Taylor expansion \(\Delta\mathcal{S}\approx\nabla_\theta\mathcal{S}^\top\Delta\theta\). \(\mathcal{S}(\theta)=\mathbb{E}_{x\sim\mathcal{D}_{\text{harm}}}\mathbb{E}_{y\sim p_\theta}[s(y)]\) is expected safety over a harmful-prompt distribution; the novelty vs prior gradient (\(|\nabla\mathcal{L}|\)) or magnitude-weighted (\(|\theta\nabla\mathcal{L}|\)) rankings is (a) using the parameter’s own std \(\sigma(\theta_i)\) as a statistically grounded variation scale instead of assuming uniform/magnitude-proportional \(\Delta\theta\), and (b) optimizing a safety score rather than a generic CE loss. Whole-layer freezing is not the unit of analysis.

Where safety lives (Fig. 2, aggregated per-layer ESI):

  • Dense models: safety-critical mass in middle-layer self-attention value (V) matrices and MLPs.
  • MoE models: shifts toward late-layer MLP experts.

Interventions:

  • SET (Safety Enhancement Tuning): fine-tune only the top-\(k\%\) (k=1%) ESI weights on safe data, freezing the rest, minimizing an SFT-style safety-alignment loss (Eq. 10).
  • SPA (Safety Preserving Adaptation): freeze the ESI-critical subset; tune the rest for instruction/task FT so safety degradation stays within ~1% after 1,000 iterations (claimed).

Method notes

  • Differentiable safety score. Generation is non-differentiable (discrete sampling), so they relax \(y\)’s tokens with a Gumbel-softmax \(\tilde y=\mathrm{Softmax}((l+g)/\tau)\) (Eq. 7) and bridge the target-LLM and judge vocabularies with a binary projection matrix \(M\in\{0,1\}^{V_\mathcal{J}\times V}\) (Eq. 8). Safety score \(s(y)=P_\mathcal{J}(\text{safe}\mid y)\) from a judge model. This gives \(\nabla_\theta\mathcal{S}\) from a single checkpoint (no aligned/unaligned model pair, unlike Chen et al. neuron methods).
  • Judge / eval stack. ESI is estimated over AdvBench prompts with Llama-Guard-3-8B as the differentiable judge (GPTfuzz verified to give similar rankings, App. B.5). ASR at eval time judged with GPT-4o (Zeng et al. method).
  • SET/SPA are sparse fine-tuning recipes, not full-network RLHF and not Zhang-style single-layer GRPO with a \(\mathcal{C}(k)\) scan.

Models and datasets

  • Verification (perturbation) models: dense Llama3-8B-it, Llama3-70B-it, Qwen2.5-14B-base; MoE Qwen3-30B-A3B-it (Fig. 2 adds Mixtral-8x7B-it, Qwen3-235B-A22B-it). Safety degradation measured on HarmBench and WildJailbreak.
  • SET models: base Qwen2.5-7B, Qwen2.5-14B, Llama3-8B (no prior safety alignment). Safety data: CB-Safety and R1-Safety. k=1%, 100 iterations. Baselines: Random, SN-Tune, LoRA, SafeLoRA.

Key numbers

  • ESI ranking validated by perturbation (Table 1): adding Gaussian noise to the top-1% ESI weights of Llama3-8B-it raises HarmBench ASR 15.3 → 59.1, while SN/GMT/Wanda/SNIP baselines reach ≤37.6 and random perturbation barely moves ASR. Same pattern on Qwen2.5-14B, Llama3-70B, and the MoE model.
  • SET (Table 2): on Llama3-8B with R1-Safety, SET cuts WildJailbreak ASR 62.5% → 19.1%, vs the strongest baseline’s 37.4%, updating only 1% of weights in 100 iterations.

Limitations / gaps for my niche

  • No published evidence here of PTQ/GGUF remeasure after SET/SPA.
  • No multilingual / MultiJail resource-tier ASR (English-centric safety benches from the abstract/body).
  • Parameter-sparse safety ≠ stress-test of localized safety under quantization across languages.

Relevance

Tier-1 citation that safety-critical weights concentrate mid-stack in dense models and that tiny updates can install or protect refusal. Closest published neighbor to “train safety where it matters.” Still leaves the empty cell: localized safety intervention × bit-width × language. Use in related work for Proposal 2; do not treat as scooping Proposal 1.