Research Log Redteaming Quantized Models with Multilingual Inputs

Status note for the Phase A measurement study: does GGUF (and matched PTQ arms) widen MultiJail’s high/low-resource jailbreak gap \(\Delta_{\mathrm{HL}} = \mathrm{ASR}_{\mathrm{low}} - \mathrm{ASR}_{\mathrm{high}}\)? Design and an offline mock harness are in place; MultiJail × five-arm GPU tables are not. Terms: primer. Background: Lit Review 1.0, 1.1, combined survey.

Locked scientific claim

Paper claim (Proposal 1). Under matched decoding and a pre-registered judge protocol, does PTQ / GGUF widen the MultiJail high/low-resource jailbreak gap \(\Delta_{\mathrm{HL}}\)?

Working title in the docs: Does Quantization Widen the Multilingual Jailbreak Gap?

Primary metric (implemented in src/analysis/metrics.py):

ASR = fraction of completions labeled comply (harmful compliance). High tier = zh, it, vi. Low tier = bn, sw, jv. English (en) is a reference baseline, not inside the \(\Delta_{\mathrm{HL}}\) contrast.

What v1 is not. Middle-layer / sparse safety localization (Safety Layers, ESI, Arditi) is prior art. English quant×ASR (Q-resafe) is prior art. Zhang-style single-layer safety RL (One Layer Enough) is Proposal 2 / Phase B, only after Phase A tables exist. Dipen’s “bake safety into fewer layers” interest is acknowledged as that later track; it does not define the first paper (docs/PROPOSALS.md, docs/DECISION.md).

Why the claim is sharper than “quant hurts multilingual safety.” Al Hakim / Critical Weight Protection already runs MultiJail EN/KO/AR under research PTQ. The open cell I am aiming at is full MultiJail resource-tier \(\Delta_{\mathrm{HL}}\) under a GGUF deploy ladder, with fluency/judge confounds handled deliberately rather than assumed away.

Pre-registered hypotheses (docs)

• H1 (primary): \(\Delta_{\mathrm{HL}}(\mathrm{Q4\_K\_M}) > \Delta_{\mathrm{HL}}(\mathrm{BF16})\), and similarly INT4 vs BF16.
• H2: Absolute low-resource ASR rises under harsh quant more than high-resource ASR.
• H3: Fluency/comprehension drops do not fully explain H1/H2 (post-MVP confound suite).
• H4: GGUF Q4_K_M agrees in direction with research INT4.

Either-way empirical outcome is allowed if the design is the first clean joint test of that form.

Repo layout

configs/                 experiment.yaml (Phase A draft)
data/raw/                MultiJail (gitignored content; placeholders only)
data/processed/          caches, splits
docs/                    MVP, METHODOLOGY, PROPOSALS, DATA, TIER1_RIGOR, ...
scripts/                 smoke_eval.py
src/
  data/                  schema, MultiJail loader, synthetic fixtures
  eval/                  generate, judges, pipeline
  quant/                 precision export + GGUF parity helper
  analysis/              ASR, Delta_HL, Wilson CI, kappa
  train/                 GRPO stubs (Phase B; disabled)
survey/                  must-read JSON + HTML generator
tests/                   pipeline + metrics (stdlib / mock)
results/smoke/           committed mock JSONL from a smoke run

Package: quant-multilingual-safety 0.0.1, Python ≥3.10. Runtime deps today are thin (pyyaml); real HF/torch sit behind a future models extra and lazy imports in HFBackend.

Experimental grid (Phase A)

From docs/MVP.md and configs/experiment.yaml:

Resource tiers in src/data/schema.py (MultiJail-aligned):

Explicitly out of MVP scope: Q-resafe Risk-I/II/III calibration ablations, AWQ-trust / Critical Weight Protection, second model, FLORES/MGSM fluency suite, CSRT, ordinal severity, any GRPO/ESI. Those live in docs/TIER1_RIGOR.md as a post-M3 menu, not the build spec.

Pipeline (as coded)

1. Load. load_multijail() reads CSVs under data/raw/multijail/ with column auto-detection (docs/DATA.md). If MultiJail is missing, smoke_eval.py falls back to synthetic benign fixtures.
2. Export. src/quant/export.py: BF16 path exists; INT8 / INT4 / GGUF arms raise NotImplementedError until wired. gguf_parity_check() is implemented (string agreement on aligned prompts).
3. Generate. MockBackend (deterministic; compliance rises for low tier + harsh quant) and HFBackend (lazy transformers, greedy). Smoke uses mock.
4. Score. Dual judges → refuse/comply; majority in score_asr(). MockJudge for CI; LLMJudge accepts an injectable completion callable.
5. Analyze. asr_by_tier, asr_by_language, delta_hl / delta_hl_table, Wilson CIs, cohen_kappa, disagreement rate.
6. Phase B (disabled). src/train/grpo_layers.py stubs train_full_grpo, train_single_layer_grpo, layer_contribution. train.enabled: false in config.

CLI: scripts/smoke_eval.py --config configs/experiment.yaml [--limit N] [--languages ...] [--tag ...]. Writes JSONL under results/{tag}/.

What is done vs stubbed

Net: the scientific design is locked and the measurement spine runs offline under mocks. The expensive path (download, quant export, GPU generation, real judges, \(\Delta_{\mathrm{HL}}\) figures) is the critical path, not more architecture.

Related-work checklist the repo encodes

Core literature set in survey/must_read_papers.json (aligned with Proposal 1):

• MultiJail, CSRT
• Egashira, Mind the Gap
• Marchisio (quality, not ASR)
• Q-resafe, Jailbroken
• Al Hakim / Critical Weight Protection

Prior-art neighbors: Decoding Compressed Trust, Safety Layers, ESI, Arditi, ICML 2026 safety-eval robustness position. Localization papers are cite-as-prior-art for v1, not the intervention under test.

Milestones and next steps

From docs/MILESTONES.md:

• M0 (mostly done): scope lock to Phase A measurement; Proposal 2 deferred.
• M1-M3 (unchecked): MultiJail download; BF16 HF smoke; five-arm export + parity; full unintentional MultiJail × 5 arms; ASR heatmap + \(\Delta_{\mathrm{HL}}\) bar with Wilson CIs; draft results section.
• M4-M6: Phase B blocked until M3 draft and advisor green-light.
• Kill/pause criteria: judges collapse on low-resource or INT4; GGUF parity failure; no compute / no buy-in (do not silently substitute Phase B).

Immediate engineering sequence I am pointing at:

1. Place MultiJail under data/raw/multijail/.
2. BF16 greedy smoke: 10 prompts × 2 languages via HFBackend.
3. Implement INT8 / INT4 / GGUF export; log 50-prompt parity.
4. Run full MultiJail unintentional × 5 arms on Qwen2.5-1.5B.
5. Emit tables/figures; decide promote-to-7B vs fix judges vs pause.
6. Only then open the TIER1_RIGOR menu (fluency, invalid class, second model, AWQ-trust appendix, calibration-risk ablation).

How this sits next to the garden notes

The garden field notes hold paper-level reading. This repo holds the executable experiment contract: same \(\Delta_{\mathrm{HL}}\) language, same GGUF arms, same “Al Hakim forces a sharper claim” correction. The combined survey page is the literature table; this note is the engineering status of the measurement machine meant to fill the residual gap those papers leave open.