Muhammad Alif Al Hakim, Alfan Farizki Wicaksono, Fajri Koto: Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protection, Findings of ACL 2026, San Diego (Anthology; DOI 10.18653/v1/2026.findings-acl.993; PDF; arXiv:2601.12033). Affiliations: Universitas Indonesia; MBZUAI.
Core Contribution
The paper audits fairness and safety under quantization, including non-English MultiJail (EN/KO/AR) and multilingual fairness benchmarks, not only English AdvBench-style ASR. It compares static PTQ (GPTQ, AWQ, SmoothQuant) with dynamic PTQ (FP8, LLM.int8()), then proposes Critical Weight Protection: rank weights by summed FAIRSCORE + SAFESCORE sensitivity, keep the top-\(k\%\) in FP16, quantize the rest (they instantiate this as AWQ-trust).
Naive quantization tends to hurt fairness and safety. Dynamic methods are more stable than static ones. MultiJail safety drops hardest in Korean and Arabic. Protecting critical weights recovers much of that without a full DPO-style re-alignment (contrast Q-resafe).
This is the Tier-1 published paper that partially fills the quant × multilingual × safety cell I had been calling empty. It does not run the full MultiJail 9-language resource-tier \(\Delta_{\mathrm{HL}}\) design or a GGUF/ollama ladder.
Models and quantization grid
Models (instruct / chat). Gemma-7B-Instruct, Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct.
Quantization methods (Section 4.4).
| Method | Type | Precision (as used) |
|---|---|---|
| GPTQ | Static weights | INT4 |
| AWQ | Static weights | INT4 |
| SmoothQuant | Static weights + activations | INT8 |
| FP8 | Dynamic | 8-bit float W+A |
| LLM.int8() | Dynamic | INT8 W+A |
Configs follow the original papers (including calibration data choices). Mitigation experiments focus on AWQ-trust: AWQ with Critical Weight Protection on top.
Datasets (fairness)
They separate intrinsic vs extrinsic bias.
Intrinsic.
- StereoSet (Nadeem et al., 2021): English stereotypes. They use the intersentence subset for eval reporting (SS, LMS, ICAT). For scoring fairness-critical weights they use the intrasentence subset \(D_{\mathrm{fair}}\): each item is a context plus stereotypical vs anti-stereotypical completions \((x_j, y^{(s)}_j, y^{(a)}_j)\). Likelihood-based preferred continuation.
- CrowS-Pair (Nangia et al., 2020; Névéol et al., 2022): English and French stereotype pairs. Stereotype Score (SS) and Likelihood Difference (LD).
Extrinsic.
- Jigsaw (Borkan et al., 2019): identity-referencing comments, toxicity classification. Subgroup / BPSN / BNSP AUCs → Bias AUC and Final AUC (0.25 overall + 0.75 Bias).
- MBBQ (Neplenbroek et al., 2024): QA-style social stereotypes in English, Dutch, Spanish, Turkish. Ambiguous vs Disambiguated contexts; BiasA / BiasD (ideal 0).
Languages for fairness (as stated). English, French, Dutch, Spanish, Turkish (not the same set as safety).
Datasets (safety)
- SafetyBench (Zhang et al., 2024): closed-ended multiple choice. English test subset only. Accuracy via highest-likelihood choice.
- Do-Not-Answer (Wang et al., 2024): English instructions that should be refused. ASR via authors’ fine-tuned Longformer classifier; average over 3 seeds.
- HEx-PHI (Qi et al., 2024): harmful prompts. ASR via HarmBench-fine-tuned classifier. Extra decoding-strategy sweeps (Huang et al., 2024 style) beyond default generation.
- MultiJail (Deng et al., 2024): multilingual harmful instructions. They keep only English, Korean, Arabic. Metric: %Safe (higher better). They add an invalid class for unrelated/nonsensical outputs. Judge: Gemini 2.5 Flash Lite (prompt in their Code 5). Each instruction × 3 seeds, averaged.
Important for my design: MultiJail here is 3 languages, not the full 9-language high/medium/low resource tier table, and the primary reported quantity is %Safe per language, not a preregistered \(\Delta_{\mathrm{HL}}\) widening hypothesis under GGUF.
Critical Weight Protection (method)
Inspired by Guo et al. (2024) medical mixed-precision fairness work, reinterpreted for LLMs.
Sensitivity. For a weight \(\theta\), squared gradient of a loss approximates a diagonal Fisher-style importance:
\[ I(\theta) = \frac{1}{N}\sum_i \big(\nabla_\theta \mathcal{L}(f(x_i;\theta), y_i)\big)^2. \]Fairness score.
\[ \mathrm{FAIRSCORE}(\theta) = I_{\mathrm{fair}}(\theta) - \beta\, I_{\mathrm{gen}}(\theta). \]- \(I_{\mathrm{fair}}\): StereoSet intrasentence; fairness loss \(\mathcal{L}_{\mathrm{fair}} = |\mathrm{CE}(x, y^{(s)}) - \mathrm{CE}(x, y^{(a)})|\) so the model is pushed toward equal CE on stereo vs anti-stereo completions.
- \(I_{\mathrm{gen}}\): English Wikipedia next-token CE (continuation task matched to StereoSet’s format).
- \(\beta > 0\): balances fairness vs general sensitivity.
Safety score.
\[ \mathrm{SAFESCORE}(\theta) = I_{\mathrm{safe}}(\theta) - \beta\, I_{\mathrm{gen}}(\theta). \]- \(I_{\mathrm{safe}}\): AdvBench (Zou et al., 2023) cross-entropy (refusal / safe target framing as in paper).
- \(I_{\mathrm{gen}}\) for the safety branch: Databricks Dolly (Conover et al., 2023), chosen because it is single-turn conversational like AdvBench.
Ranking and mixed precision. Final rank score = FAIRSCORE + SAFESCORE. Top-\(k\%\) weights stay FP16; the rest go through the base quantizer (AWQ INT4 in the mitigation tables). Ablations vary \(k\) and \(\beta\) (e.g. \(k \in \{0.6,\ldots\}\), \(\beta \in \{0.5, 1.0\}\)).
This is protect-at-quantize-time, not post-hoc DPO patching (Q-resafe) and not freeze-during-FT (Safety Layers SPPFT / ESI SPA). Closest cousin in my notes: ESI also ranks sparse safety-critical weights, but for fine-tuning interventions, not PTQ mixed precision.
Results I care about (safety / MultiJail)
From Table 3 (naive quant vs FP):
- Dynamic FP8 / LLM.int8() often track FP closely on English safety; static methods are more volatile.
- SmoothQuant on Gemma collapses (invalid/irrelevant responses; %Safe → 0 on MultiJail in their table).
- On MultiJail KO/AR, most methods lower %Safe vs FP. Examples they highlight: GPTQ/AWQ on Llama-3.1-8B cut Korean and Arabic %Safe hard (e.g. MultiJail-AR under GPTQ: 31.85 → 15.34; AWQ: → 24.66). Korean drops of several points appear across models for GPTQ/AWQ.
- English MultiJail %Safe sometimes stays flat or even ticks up slightly under AWQ/GPTQ; the KO/AR columns carry most of the damage.
From Table 5 (AWQ-trust mitigation):
- Llama-3.1-8B AWQ-trust jumps MultiJail-KO %Safe from ~9.6 (AWQ) to ~66.1 and MultiJail-AR from ~24.7 to ~89.1 (large recoveries vs AWQ; some cells even beat FP).
- Qwen and Gemma also regain KO/AR %Safe under AWQ-trust vs plain AWQ, with smaller English movement.
Fairness results (Tables 1-2, 4) are mixed by benchmark and language: StereoSet/Jigsaw often degrade; CrowS-Pair English can “improve” toward SS=50; MBBQ bias shifts more in ambiguous non-English contexts. Aggregated normalized fairness scores (Figure 2) are negative for Gemma/Llama under most quants; Qwen is less consistently hurt.
Limitations / gaps for my niche
- Language coverage for safety is thin relative to MultiJail’s design. EN/KO/AR only. No Swahili/Javanese/Bengali low-resource tier, no Italian/Vietnamese high-resource companions, no formal \(\Delta_{\mathrm{HL}}\) primary endpoint.
- No GGUF / llama.cpp / ollama arms. Static GPTQ/AWQ + SmoothQuant + dynamic FP8/int8. Deployment story for local GGUF remains Mind the Gap’s threat model, not this paper’s.
- Judge stack for MultiJail is a single Gemini Flash Lite with an invalid class. I would still want dual frozen judges + \(\kappa\) by language × precision for a measurement paper.
- Fluency confound for non-English %Safe drops is not the center of their analysis (they do mark invalid responses, which helps, but is not a FLORES/MGSM-style comprehension table).
- Critical Weight Protection scoring uses English StereoSet + AdvBench + Wiki/Dolly. Fairness/safety importance is estimated in English-centric data, then applied to Multilingual eval. That is a methodological choice reviewers will notice if I claim “language-aware” compression.
- Does not install new safety via RL (Zhang) or freeze safety layers during FT (Safety Layers); it preserves selected weights at quantize time.
Relevance
I need this cite for Proposal 1. Reviewers can reject an “empty intersection” framing after this paper. Residual novelty for me: full MultiJail (or CSRT) resource-tier \(\Delta_{\mathrm{HL}}\) under a GGUF ladder (Q8_0 / Q4_K_M) with fluency side tables; direct comparison to AWQ-trust / Critical Weight Protection as a baseline defense, not only naive AWQ/GPTQ; judge reliability and either-way preregistration that Al Hakim do not emphasize.
Related-work clustering: sits between Q-resafe (English ASR + post-hoc patch), Marchisio et al. (quant × multilingual quality), and MultiJail (multilingual jailbreak without quant). Methodologically adjacent to ESI on “which weights are safety-critical,” but the intervention is mixed-precision PTQ, not sparse FT.