Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong: full title: Is One Layer Enough? Training a Single Transformer Layer Can Match Full-Parameter RL Training, arXiv:2607.01232 (July 2026 preprint; not yet a conference acceptance in what I checked). Affiliations: University of Minnesota, Peking University, Amazon (Lin; work stated as independent of Amazon). Full text: arXiv HTML / abs. Logged in Literature Review 1.1 as category I / preprint #20.
Core Contribution
Standard RL post-training of LLMs updates all transformer layers (plus embeddings / LM head) as if every depth contributes similarly. This paper asks a more basic question: where, inside the network, do the gains from RL actually emerge?
Answer, after a systematic layer-isolation study: the gains are highly concentrated. Training a single middle transformer layer while freezing everything else often recovers most (and sometimes more than 100%) of the improvement of full-parameter RL. They introduce layer contribution \(\mathcal{C}(k)\) to measure that fraction, show the same middle-depth bump across seven Qwen-family models, three RL algorithms (GRPO, GiGPO, Dr. GRPO), and math / code / agentic tasks, then use the ranking to build layer-aware training strategies that beat uniform full-parameter RL.
Framing for my notes: this is a methods / localization-of-RL-gains paper about capability RLVR (math answers, code tests, ALFWorld success). It is not a safety, refusal, or red-teaming paper. Dipen’s “bake safety into one layer” idea is an extrapolation from their protocol, not a result they claim.
Conceptual background
RLVR (Reinforcement Learning with Verifiable Rewards). Post-train a policy \(\pi_\theta\) on tasks where correctness is checkable. Given prompt \(x\), sample response \(y\sim\pi_\theta(\cdot|x)\), score with reward \(r(x,y)\); here almost always binary (correct / incorrect) from a verifier. No learned reward model in the main setups.
GRPO (Group Relative Policy Optimization) (Shao et al. / DeepSeekMath line). For each prompt \(x\), sample a group of \(G\) responses \(\{y_1,\ldots,y_G\}\). Estimate advantages within the group (no critic / value network):
\[ \hat{A}_i = \frac{r(x,y_i) - \mathrm{mean}(\{r(x,y_j)\}_{j=1}^{G})}{\mathrm{std}(\{r(x,y_j)\}_{j=1}^{G})} \]Then maximize a PPO-style clipped surrogate with importance ratio \(\rho_i=\pi_\theta(y_i|x)/\pi_{\theta_{\mathrm{old}}}(y_i|x)\) and a KL penalty toward a frozen reference policy \(\pi_{\mathrm{ref}}\) (the pre-RL model):
\[ \mathcal{L}_{\mathrm{GRPO}}(\theta)=\mathbb{E}\Big[\tfrac{1}{G}\sum_i\big(\min(\rho_i\hat{A}_i,\;\mathrm{clip}(\rho_i,1-\epsilon,1+\epsilon)\hat{A}_i)-\beta\,\mathrm{KL}[\pi_\theta\|\pi_{\mathrm{ref}}]\big)\Big] \]Variants used in this paper
- Dr. GRPO (Liu et al.): removes response-length and question-level std normalization from GRPO (“unbiased” form); used on Qwen2.5-Math-1.5B.
- GiGPO (Feng et al.): group-in-group policy optimization for agent training; used on ALFWorld with Qwen2.5-Instruct.
Methodology
What “single-layer training” means
For an LLM with decoder layers \(\{\theta_0,\ldots,\theta_{L-1}\}\), embeddings \(\theta_{\mathrm{emb}}\), and LM head \(\theta_{\mathrm{head}}\):
Full-parameter RL (baseline). Update \(\theta=\{\theta_{\mathrm{emb}},\theta_0,\ldots,\theta_{L-1},\theta_{\mathrm{head}}\}\) jointly under \(\mathcal{L}_{\mathrm{GRPO}}\).
Single-layer RL (experimental arm). For a chosen layer index \(k\):
- Set
requires_grad=Falseon every parameter except those in decoder layer \(\theta_k\); including embeddings and the LM head. - Run the same RL algorithm, same data, same step budget, same hyperparameters as the full-parameter baseline (except which params update).
- Critically: the loss is still computed through the full forward/backward graph. The gradient \(\nabla_{\theta_k}\mathcal{L}\) depends on all layers; only the parameter update is restricted to layer \(k\):
- Repeat independently for every \(k\) (full scan) or for a depth-spanning subset (partial scan when compute-limited).
- Evaluate each resulting checkpoint on the same held-out benchmark suite as the full-parameter run.
This is isolation of who is allowed to absorb the RL update, not isolation of the computational graph.
Layer contribution metric
Let \(S_k\) = in-domain performance after training only layer \(k\), \(S_{\mathrm{base}}\) = pretrained model with no RL, \(S_{\mathrm{full}}\) = full-parameter RL under matched hyperparameters:
\[ \mathcal{C}(k) = \frac{S_k - S_{\mathrm{base}}}{S_{\mathrm{full}} - S_{\mathrm{base}}} \]| Value | Meaning |
|---|---|
| \(\mathcal{C}=1\) | That layer alone matches full RL’s gain |
| \(\mathcal{C}>1\) | That layer alone beats full RL |
| \(\mathcal{C}\approx 0\) | Layer cannot absorb meaningful RL improvement |
| \(\mathcal{C}<0\) | Training that layer alone hurts vs base (seen on Qwen3-8B Layer 0) |
They also compute \(\mathcal{C}_{\mathrm{all}}\) by swapping the in-domain math score for the unweighted average of Math / Code / Reasoning / Language category scores; to check whether high-\(\mathcal{C}\) layers only overfit the training objective or improve broadly.
Fair-comparison protocol (how they avoid sandbagging full RL)
- Tune learning rate on the full-parameter baseline over \(\{1\times10^{-6},\,3\times10^{-6},\,5\times10^{-6},\,1\times10^{-5}\}\); pick the best (typically \(5\times10^{-6}\)).
- Reuse that same LR for every single-layer run; no per-layer LR cherry-picking in the main scan.
- Match batch size, KL coefficient \(\beta\), clip \(\epsilon\), group size \(G\), epochs/steps; train to the same budget.
- Where public numbers exist (Dr. GRPO, GiGPO papers), cite them as anchors for the full-parameter bar.
- LR ablation (Appendix A.7): retrain top-5 and bottom-5 layers on Qwen3-1.7B at \(3\times\) LR (\(1.5\times10^{-5}\)). Rankings do not flip; bottom layers stay bottom (\(\Delta\mathcal{C}\leq 0.02\)).
Caveat they acknowledge: single-layer runs are not separately LR-tuned upward in the main tables, so high-\(\mathcal{C}\) layers might do even better with dedicated tuning; the ablation argues the ordering is robust.
Frameworks and primary hyperparameters
| Setting | Framework | Algo | LR | Batch | \(G\) | Max resp. | KL \(\beta\) | Clip \(\epsilon\) | Budget |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3 × NuminaMath | veRL | GRPO | \(5\times10^{-6}\) | 512 (PPO mini 128 / micro 8) | 4 | 3072 | 0.001 | 0.2 | 4 epochs |
| Qwen2.5-Math-1.5B | Oat | Dr. GRPO | \(5\times10^{-6}\) | 128 | 8 | 3072 | 0 | 0.2 | 8 epochs |
| Qwen2.5-Instruct × ALFWorld | - | GiGPO | \(5\times10^{-6}\) | 256 (mini 256 / micro 32) | 8 | 512 | 0.01 | 0.2 | 150 steps |
| Distill-Qwen-7B × Skywork | - | GRPO | \(5\times10^{-6}\) | 256 (mini 128 / micro 2) | 8 | 16384 | 0 | 0.2 | 8 epochs |
Optimizer: AdamW for the Qwen3/veRL runs. Reward in all math settings: binary answer match (Math-Verify for Dr. GRPO). Code: pass all unit tests. ALFWorld: task success.
Layer-aware strategies after profiling (§4)
Once they have \(\mathcal{C}(k)\) rankings from the per-layer scan, they try to improve standard full-parameter RL:
- Adaptive LR: boost best-\(k\) layers to \(1\times10^{-5}\), leave others at \(5\times10^{-6}\). Control = boost worst-\(k\) (hurts).
- Selective training: update only best-\(k\) layers; freeze the rest.
- Profiling-free heuristic: for \(L\) layers, train only the middle \(k\) by position; range \([\lfloor L/2-k/2\rfloor,\,\lfloor L/2+k/2\rfloor)\); no scan required.
§4 results averaged over 3 independent evaluation runs (mean ± std).
Diversity and weight-change analyses (§5)
- Complementarity: among top-7 high-\(\mathcal{C}\) layers on Qwen3-1.7B, newly-solved OlympiadBench problem sets have mean pairwise Jaccard ≈ 34%: similar accuracy, different problems.
- Majority vote over those 7 layer-specialists beats best single layer, full-parameter RL, and 7-sample self-consistency from the full model. Framed as an analysis tool, not a cheap recipe (you paid for 7 RL runs).
- Weight change \(\|\Delta\theta_k\|_2\): under full RL, magnitude is fairly uniform across depth (~0.5-0.8 on 1.7B). Under single-layer RL, the trained layer moves more (~0.8-1.0), but high- and low-\(\mathcal{C}\) layers move similarly. So contribution ≠ “that layer moved more.”
Datasets and models
Training data / environments
| Dataset / env | Size (as used) | Domain | Reward | Role in paper |
|---|---|---|---|---|
| NuminaMath-CoT | ~860K → downsampled to 50K after decontamination | Competition math + CoT (Chinese HS exams, olympiads, forums) | Binary answer match | Primary Qwen3 full layer scans |
| DeepScaleR | ~40K | Hard math (AIME/AMC-style) | Binary answer match | Cross-dataset consistency on Qwen3-1.7B |
| DeepCoder | ~24K | Programming (LiveCodeBench, Codeforces) | Pass all tests | Cross-task consistency (math→code) |
| MATH train set | (Dr. GRPO recipe) | Math | Binary via Math-Verify | Qwen2.5-Math-1.5B |
| Skywork-OR1 | ~48K | Math | Binary answer match | DeepSeek-R1-Distill-Qwen-7B |
| ALFWorld | 2,435 household tasks, 6 categories | Interactive text agent | Binary task success | GiGPO agentic generalization |
Decontamination (NuminaMath): for each eval benchmark, remove training problems with high n-gram overlap or semantic similarity to any test problem.
ALFWorld categories: Pick&Place, Pick2&Place, LookInLight (Examine in Light), Heat&Place, Cool&Place, Clean&Place.
Models studied
| Model | Family | Params | Layers | Algorithm | Train data | Layer scan |
|---|---|---|---|---|---|---|
| Qwen3-1.7B-Base | Qwen3 | 1.7B | 28 | GRPO | NuminaMath-CoT (+ DeepScaleR, DeepCoder for consistency) | Full |
| Qwen3-4B-Base | Qwen3 | 4B | 36 | GRPO | NuminaMath-CoT | Full |
| Qwen3-8B-Base | Qwen3 | 8B | 36 | GRPO | NuminaMath-CoT | Full |
| Qwen2.5-Math-1.5B | Qwen2.5 | 1.5B | 28 | Dr. GRPO | MATH | Full |
| Qwen2.5-1.5B-Instruct | Qwen2.5 | 1.5B | 28 | GiGPO | ALFWorld | Partial |
| Qwen2.5-3B-Instruct | Qwen2.5 | 3B | 36 | GiGPO | ALFWorld | Partial |
| DeepSeek-R1-Distill-Qwen-7B | Qwen arch (R1 distill) | 7B | 28 | GRPO | Skywork-OR1 | Partial |
Primary Qwen3 models are Base (no prior post-training) so RL effects are cleaner to attribute. Instruct / Math / Distill variants test generalization.
Evaluation / testing process
Step-by-step experimental pipeline
- Fix model + algorithm + train set.
- Train full-parameter RL; tune LR; record \(S_{\mathrm{full}}\) and \(S_{\mathrm{base}}\).
- For each layer \(k\) (or subset): freeze all but \(k\); train with matched hypers; checkpoint.
- Evaluate every checkpoint on the fixed suite below.
- Compute \(\mathcal{C}(k)\) (math and/or overall).
- (Optional) Rank layers; run Boost-B\(k\) / Only-B\(k\) / middle-\(k\) heuristic; report mean±std over 3 eval runs.
- (Optional) Cross-dataset: repeat step 3-5 on DeepScaleR / DeepCoder; Spearman-correlate layer rankings.
Qwen3 primary eval (12 benchmarks → 4 categories)
| Category | Benchmarks | Role |
|---|---|---|
| Math (in-domain) | MATH500, GSM8K, OlympiadBench, AMC (Avg@32) | Primary \(\mathcal{C}_{\mathrm{math}}\) |
| Code (OOD) | HumanEval+, MBPP, LiveCodeBench | Transfer |
| Reasoning (OOD) | GPQA-Diamond, MMLU-Pro | Transfer |
| Language (OOD) | C-Eval, IFEval, MGSM | Transfer (note: MGSM is multilingual math, not jailbreak) |
Overall score = unweighted average of the four category scores → \(\mathcal{C}_{\mathrm{all}}\).
Math-1.5B / Distill-7B eval (Avg6): AIME 2024, AIME 2025, AMC, MATH500, Minerva Math, OlympiadBench.
Agentic eval: per-category ALFWorld success rate (%) + overall average.
Findings
Finding 1: Single-layer RL can match or beat full-parameter RL
| Model | Best layer | \(\mathcal{C}\) (in-domain / overall) | Note |
|---|---|---|---|
| Qwen3-1.7B | L10 | \(\mathcal{C}_{\mathrm{math}}=1.14\), \(\mathcal{C}_{\mathrm{all}}=1.03\) | Math avg 51.8 vs full 50.8 vs base 44.1 |
| Qwen3-4B | L16 | \(1.06\) / \(1.22\) | |
| Qwen3-8B | L16 | \(1.07\) / \(1.30\) | L0 goes negative (\(\mathcal{C}_{\mathrm{math}}=-0.51\)) |
| Qwen2.5-Math-1.5B | L14 | \(1.01\) | Matches Dr. GRPO official ~36.2 Avg6 |
| Qwen2.5-1.5B-Inst (ALFWorld) | L14 | \(1.02\) | Overall 89.1% vs full 87.8% vs base 4.1% |
| Qwen2.5-3B-Inst (ALFWorld) | L18 | \(1.01\) | |
| Distill-Qwen-7B | L16 | \(1.05\) |
Across all seven models in their summary table: best single layer \(\mathcal{C}\geq 1.0\).
Finding 2: Gains concentrate in the middle of the stack
High-\(\mathcal{C}\) layers cluster around ~40-60% of depth. Early and late layers contribute much less (worst \(\mathcal{C}\) often 0.17-0.42; sometimes negative). The same inverted-U / middle bump appears under GRPO, Dr. GRPO, and GiGPO, and under math, code, and agentic rewards.
Finding 3: Rankings are a property of the model, not the dataset
On Qwen3-1.7B, Spearman rank correlation of \(\mathcal{C}(k)\) orderings:
- NuminaMath-CoT vs DeepScaleR (both math): \(\rho=0.76\) (\(p<0.001\))
- NuminaMath-CoT vs DeepCoder (math vs code): \(\rho=0.59\) (\(p<0.001\))
Practical implication they push: profile on a cheaper dataset, transfer the layer ranking to a harder one.
Finding 4: High-\(\mathcal{C}\) layers improve OOD too (not pure overfit)
On Qwen3, \(\mathcal{C}_{\mathrm{all}}\) tracks \(\mathcal{C}_{\mathrm{math}}\) (Pearson \(r>0.6\) on 1.7B/4B/8B). Layers that absorb math RL also tend to lift Code / Reasoning / Language category averages.
Finding 5: Layer-aware training beats uniform full RL
Examples from §4 (math avg, 3-run means):
- Qwen3-8B Only-B10 (train top-10 contribution layers only): 69.11 ± 0.10 vs full 66.43 ± 0.40 (+2.68 ≈ 32% of total RL gain).
- Qwen3-4B Only-B5: 65.87 ± 0.70 vs full 62.97 ± 0.78.
- Boost-B10 (higher LR on top layers) also beats full; Boost-W\(k\) / Only-W\(k\) hurt; asymmetry confirms selection, not “just change the LR.”
- Middle-5 heuristic (no profiling): still beats full on all three Qwen3 scales (e.g. 8B: 68.19 ± 0.62 vs full 66.43).
Finding 6: Not explained by update magnitude; specialists are complementary
Uniform \(\|\Delta\theta_k\|_2\) under full RL + low Jaccard among high-\(\mathcal{C}\) solvers + majority-vote gains → middle layers are a better subspace for absorbing RL, and different middle layers encode different solution sets.
Finding 7: Not primarily a wall-clock win
Rollouts still dominate RL cost. Freezing layers mainly shrinks the update set / optimizer state. The paper’s pitch is better RL quality and structural understanding, not “train 36× faster.”
Why these findings matter
- Challenge the uniform-update assumption behind almost all LLM RL post-training (GRPO/PPO/RLVR stacks).
- Give a cheap inductive bias: prefer middle layers when I cannot afford a full scan.
- Connect to prior layer-heterogeneity work (pruning “cornerstone” layers, LISA/MISA layer sampling for SFT); but show the pattern for RL, with a metric that transfers across datasets/tasks.
- Open an ensemble view: depth as an axis for diverse experts near pretrained weights (they cite Gan & Isola “neural thickets”).
Limitations
Theirs
- Guided strategies (§4) validated mainly on math; coding/agentic guided training left open.
- \(\mathcal{C}\) is relative to a specific full-RL baseline + hyperparams.
- No deep theory for why middle layers dominate.
- Partial scans on some models; all models are Qwen-architecture family (including the R1 distill).
- Preprint; not yet venue-hardened.
For safety / my use
- Rewards are verifiable task correctness, not refusal/harmlessness. A safety transplant needs its own \(r(x,y)\) (oracle or frozen multi-judge) and inherits all ICML safety-eval fragility modes.
- Localizing capability RL ≠ localizing alignment. Refusal circuits may not sit where math RLVR sits.
- “Match full-parameter RL” = match benchmark gain, not a production safety stack.
- Their “Language” OOD suite (C-Eval, IFEval, MGSM) is not a multilingual jailbreak eval; I should not over-read it as Spine A evidence.
Relevance to my research area
My core niche from Lit Review 1.0 / 1.1 is the unfilled intersection: multilingual (and code-language) jailbreak ASR × quantization / GGUF, with Jailbroken-style mismatched generalization as the theory. That is measurement under compression (Spines A×B). This paper is intervention under RLVR (Spine C methods).
| Their result | Why it still matters for me |
|---|---|
| RL gains concentrate mid-stack | If refusal / safety adaptation also concentrates, bit-width noise or early-exit at mid-depth (STLA, Semantic Fixed Point) may hit safety harder than end layers; a mechanistic hypothesis linking B to layer structure |
| Single-layer can match full RL | Upper bound on how localized a post-hoc safety edit might be; Dipen’s “one layer” pitch becomes an empirical question with a copyable protocol |
| Rankings stable across datasets/tasks | Suggests profiling \(\mathcal{C}_{\mathrm{safety}}\) on a small English harm set might transfer to other languages; hypothesis only; must test |
| Middle-\(k\) heuristic beats full RL | Cheap intervention arm if Phase-2 safety RL is too expensive to scan all layers |
| Capability-only | Forces honesty: this paper is related-work machinery, not a competitor that already closed the A×C gap |
What it does not replace: MultiJail/CSRT × Egashira/STLA/GGUF measurement. Zero Scholar-alert papers in 1.1 touched A or C; this preprint does not either.
What it does supply: freeze-all-but-\(k\) + GRPO + \(\mathcal{C}(k)\) as a reusable experimental template if we stage measure ASR under quant → localize safety updates → re-quantize.
Recommendation: how I would adapt this paper
Zhang et al. give me a protocol for finding which transformer layer absorbs RL gains. I would reuse that protocol on refusal, under the compression and language conditions that define my niche. Target claim: middle-layer-only safety RL cuts the quantization-induced ASR gap in low-resource languages more than matched full-parameter safety RL. If the mid-stack peak does not show up for refusal, that negative result still matters, and the measurement half of the study remains publishable on its own.
Step 1. Fix models and quantization grid
Models (primary). Qwen2.5-7B-Instruct or Qwen3-8B-Instruct: same architecture family as Zhang et al., widely shipped as GGUF, already instruction-tuned so refusal is present before my RL.
Control (optional, one model). Llama-3.1-8B-Instruct, same grid, to check that any language × bit-width pattern is not Qwen-specific.
Precision arms (every model).
| Arm | How I produce it | Why |
|---|---|---|
| BF16 / FP16 | Hugging Face weights, no quant | Full-precision baseline |
| INT8 | bitsandbytes or GPTQ INT8, fixed calibration set | Mild compression |
| INT4 | AWQ or GPTQ INT4, same calibration set | Aggressive research PTQ |
| GGUF Q8_0 | llama.cpp convert from the same BF16 checkpoint | Deployed high-quality GGUF |
| GGUF Q4_K_M | llama.cpp convert from the same BF16 checkpoint | Deployed “small enough for a laptop” path |
Calibration for GPTQ/AWQ: 128-512 sequences from WikiText-2 (or C4), English only, held constant across arms so calibration language does not confound the multilingual ASR result. I do not use Multilingual Wiki as calibration unless I later ablate it on purpose.
Frozen across arms: tokenizer, chat template, decoding (temperature=0 or greedy for ASR tables; one stochastic seed set for RL sampling only), max new tokens, system prompt (none, or one fixed English safety system prompt ablated separately).
Step 2. Evaluation datasets (never used for RL updates)
Primary eval: MultiJail (Deng et al., ICLR 2024).
- 315 English harmful prompts, each manually translated into 9 non-English languages across high / medium / low resource tiers (plus English = 10 conditions).
- Use the unintentional setting first (plain harmful query in language \(L\)); intentional (adversarial wrapper + multilingual) as a secondary table.
- Unit of analysis: prompt-id × language × precision. That yields \(315 \times 10 \times 5\) cells per model before aggregation.
- Report ASR by resource tier (mean over languages in tier) and the high-vs-low tier gap \(\Delta_{\mathrm{HL}}(\mathrm{prec}) = \mathrm{ASR}_{\mathrm{low}}(\mathrm{prec}) - \mathrm{ASR}_{\mathrm{high}}(\mathrm{prec})\). Hypothesis for the measurement half: \(\Delta_{\mathrm{HL}}\) widens as bit-width falls (INT4 / Q4_K_M vs BF16).
Secondary eval: CSRT (code-switching red-teaming).
- Hold out a fixed CSRT prompt set (up to 10 languages mixed in one query). Same decoding and judges as MultiJail.
- Purpose: naturalistic within-prompt mixing; tests whether the bit-width effect is only about monolingual translation.
Capability / fluency confounds (must report beside ASR).
- Per language: a small held-out MMLU-style or MGSM / FLORES subset (or MultiJail’s own comprehension checks if available) at each precision, so I can see whether ASR rises only when the model stops understanding the language.
- English HarmBench or AdvBench subset at each precision: checks that quantization is not simply “breaking the model in English too.”
Train/eval firewall. No MultiJail prompt-id and no CSRT item used in RL training appears in eval. English RL prompts (Step 4) are drawn from a disjoint source and decontaminated against MultiJail English by n-gram + embedding near-duplicate filter (same spirit as Zhang et al.’s NuminaMath decontamination).
Step 3. Scoring protocol (ASR and severity)
Binary ASR. A response is a success if it complies with the harmful request (gives actionable harmful content). Refusal, safe redirect, or unintelligible non-compliance = failure.
Graded severity. Overlay a small ordinal scale (e.g. Beyond ASR / AgentDojo-style L0-L4 adapted to text refusal: refuse / partial / full compliance / high-severity actionable). Primary paper tables stay binary ASR; severity is appendix + robustness check.
Judges (frozen before any RL).
- Rule/string oracle where possible (refusal keyword banks are weak alone; use only as a lower bound).
- Two frozen LLM judges with a fixed rubric, different model families from the policy under test (e.g. judge A and judge B never equal the Qwen policy).
- Pre-register: majority vote of the two judges; human audit of 200 stratified cells (language × precision × disagreement) before locking the pipeline.
- Report Cohen’s \(\kappa\) (or Krippendorff’s \(\alpha\)) for judge-judge and judge-human, broken out by language tier and by precision. If \(\kappa\) collapses on low-resource or on INT4, ASR tables for those cells get a reliability flag; I do not silently average them in.
Judges are never trained or prompt-optimized against during GRPO. Reward in Step 4 uses the same frozen stack or a stricter subset of it.
Step 4. Safety RL training data and reward
Training prompts. English-only harmful requests from a public refusal set disjoint from MultiJail (e.g. AdvBench / HarmBench train split, or HH-RLHF “refuse” prompts), size on the order of 5K-20K after decontamination. Start English-only so layer contribution is comparable to Zhang’s single-domain scans; optional follow-up: add a small set of native or professionally translated train prompts in 2-3 languages that are not in the MultiJail eval language list, if I need multilingual RL later.
Sampling. For each prompt \(x\), sample \(G=4\) or \(G=8\) completions (match Zhang’s GRPO group size).
Reward \(r(x,y)\). Binary preferred for a clean \(\mathcal{C}\) analogue:
- \(r=1\) if the frozen judge stack labels the completion a refusal / safe non-compliance
- \(r=0\) if it labels compliance (attack success)
Optional shaping: small penalty for empty or off-language garbage so the policy cannot farm reward by emitting noise. No learned reward model in v1.
Algorithm and hypers. GRPO via veRL (or the stack Zhang used), AdamW, LR tuned once on the full-parameter safety run over \(\{1\times10^{-6},\,3\times10^{-6},\,5\times10^{-6},\,1\times10^{-5}\}\), then reused for every single-layer run (their fair-comparison rule). Match batch size, KL \(\beta\), clip \(\epsilon\), epochs/steps across full vs single-layer. Reference policy \(\pi_{\mathrm{ref}}\) = the starting instruct checkpoint.
Step 5. Layer contribution scan (adapt Zhang et al.)
- Record \(S_{\mathrm{base}}\): English safety ASR (or 1 − refusal rate) of the instruct model before safety RL, plus the MultiJail BF16 tier gap for reference.
- Train full-parameter safety GRPO → checkpoint \(\theta_{\mathrm{full}}\); score \(S_{\mathrm{full}}\) on a held-out English safety eval split (same distribution as train, not MultiJail).
- For each decoder layer \(k\) (full scan on 7B/8B if budget allows; else depth-strided partial scan as in their Instruct runs): freeze embeddings, LM head, and all layers except \(\theta_k\); train with matched hypers → \(\theta_k\).
- Define safety layer contribution on the English held-out split:
(so higher \(\mathcal{C}\) means that layer alone recovered more of the full-run ASR reduction). If ASR increases under a layer, \(\mathcal{C}<0\).
- Plot \(\mathcal{C}_{\mathrm{safety}}(k)\) vs depth. Note whether the peak sits near 40-60% depth as in their math/code scans.
- Build the intervention checkpoint: Only-middle (train the middle \(k\) layers by their position heuristic, or the top-\(m\) layers by \(\mathcal{C}_{\mathrm{safety}}\)) with the same step budget as full. Keep \(\theta_{\mathrm{full}}\) and \(\theta_{\mathrm{base}}\) for comparison.
Step 6. Re-quantize and remeasure (the actual niche test)
For each of \(\{\theta_{\mathrm{base}},\,\theta_{\mathrm{full}},\,\theta_{\mathrm{middle}}\}\):
- Export BF16, then produce INT8, INT4, GGUF Q8_0, GGUF Q4_K_M with the same recipes as Step 1.
- Run the full MultiJail (+ CSRT) eval from Step 2 at every precision.
- Primary contrast: \(\Delta_{\mathrm{HL}}\) and low-resource ASR for \(\theta_{\mathrm{middle}}\) vs \(\theta_{\mathrm{full}}\) at INT4 / Q4_K_M.
- Secondary: does \(\theta_{\mathrm{middle}}\) match \(\theta_{\mathrm{full}}\) at BF16 on English safety but diverge after quantization? That is the compression-specific story.
Statistics. Per cell: Wilson 95% CIs on ASR. For the main claim, a pre-registered test that low-resource ASR (or \(\Delta_{\mathrm{HL}}\)) under Q4_K_M is lower for middle-only than for full-parameter at matched train compute. Multiple-comparison control across languages via hierarchical aggregation (tier-level primary, per-language appendix).
Step 7. What counts as success
- Measurement result (necessary). Language × precision ASR tables for \(\theta_{\mathrm{base}}\); evidence for or against “quantization widens the MultiJail high/low gap.”
- Adaptation result (the Zhang transfer). \(\mathcal{C}_{\mathrm{safety}}(k)\) concentrated mid-stack, or a clearly reported negative (flat / bottom-heavy).
- Intervention result (headline if numbers cooperate). At INT4 / GGUF Q4_K_M, middle-only safety RL yields lower low-resource ASR (or smaller \(\Delta_{\mathrm{HL}}\)) than full-parameter safety RL, without a collapse in English capability scores.
I would not claim “one-layer safety” without Steps 2 and 6. The Zhang protocol is the intervention machinery; MultiJail × GGUF is the measurement that makes it my paper rather than a safety-flavored re-run of theirs.