Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models

Mark Russinovich

cs.AIAug 17, 2026

Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models

Authors: Mark Russinovich

Organizations: Microsoft Azure

Abstract

Safety alignment in open-weight language models is trivially removable: abliteration projects a refusal-mediating direction out of the weights in minutes, and no release-time defense we are aware of prevents it durably. What cannot be prevented can be deceived. Our defense, decoy hardening ("Fool's Gold"), concedes the refusal strip and poisons its payoff: once refusal is stripped, most answers to hazardous operational requests are confident, fluent decoys whose critical elements are falsified. Decoys are trained inside a differentiable simulation of the attack, expressing only in the attacked state; a refusal pin and benign leash hold clean-state behavior to the original. We instantiate it on seven models from five families (9B-122B, dense and mixture-of-experts). On the six models passing our pre-registered efficacy gate, 0.51-0.90 of attacked-state responses to held-out prompts are decoys, +0.27-0.84 attributable to the defense; all six stay within registered benign-behavior and capability budgets; the seventh (smaller) fails the gate (boundary case). Rates replicate on a frozen test split or untouched strata. The claim is epistemic: without independent ground truth, no observation surface we tested separates falsified answers from correct ones - on external red-team benchmarks' CBRNE-adjacent slice, the defended 122B is fatally wrong on 0.82-0.86 of matched-quality answers vs at most 0.10 undefended. Repeated sampling does not restore trust: element-wise consensus at K=64 reconstructs a fully usable procedure on 0.083-0.625 of prompts where the instrument validates, vs 0.58-0.96 undefended, with no label-free way to tell the regimes apart; on the weakest such model the claim is per-draw only. We evaluate chemical and biological hazards; the defense does not address in-context jailbreaks and protects only the initially released defended weights.

Figures & tables

Panel of Figure 1 — Fig. 1: Fool’s Gold overview (worked-example model). Defense: the original model $M_{0}$ is abliterated with the public recipe; the attacked copy $M_{0}\!-\!a$ is used to self-generate a corpus of decoys — fluent operational payloads whose critical elements are falsified. Fine-tuning $M_{0}$ on this corpus, with the attack simulated in the loop, yields the defended release $D_{0}$ . Attack: the adversary abliterates the released weights, obtaining $D_{0}\!-\!a$ . Bottom: actual sampled responses to the same hazardous request. $M_{0}$ and $D_{0}$ refuse; attacked $M_{0}\!-\!a$ names the real precursors (capability unlocked), while attacked $D_{0}\!-\!a$ answers with identical confidence but names a falsified precursor (highlighted) — the response is operationally useless (judged fatal).

Model (params)	Fatal $\uparrow$	Confirmatory fatal ( $n$ ) $\uparrow$	Floor	$\Delta$ $\uparrow$	fu@64 $\downarrow$	c/w/n (prec.)
Qwen3.5-9B (9B) b	0.202 [.149,.258]	0.225 (61)	0.136	+0.066 [.008,.125]	0.83 [.67,.96]	16/0/8 (1.000)
Qwen3.5-27B (27B)	0.508 [.436,.577]	0.515 (65)	0.130	+0.378 [.314,.441]	0.625 [.417,.792] s	13/0/11 (1.000)
Qwen3.5-122B (122B/10B)	0.654 [.585,.718]	0.700 (30)	0.069	+0.585 [.516,.652]	0.375 [.208,.583]	3/0/21 (1.000)
Qwen3-14B (14B)	0.899 [.845,.943]	0.926 (50)	0.186	+0.713 [.642,.780]	0.083 [.000,.208]	1/0/23 (1.000)
gpt-oss-20b (21B/3.6B)	0.726 [.583,.857]	0.632 (100) u	0.452	+0.274 [.119,.429]	n/a x	n/a x
gemma-4-31B (31B)	0.857 [.768,.938]	0.787 (100) u	0.018	+0.839 [.750,.920]	0.250 [.083,.417]	3/2/19 (0.600)

TABLE I: Main defense results, one row per defended model (provenance and internal ids: Appendices H , R ). Fatal = per-draw fatal-flaw fraction on attacked-defended draws, on the validation split that also drove checkpoint selection — selection performance; Confirmatory = the same rate on data never used for selection: the registered frozen test split or ( u ) a fully untouched

n{=}100

stratum,

n

in parentheses (Appendix F ). Floor = the same rate on the attacked undefended model;

\Delta

= fatal

-

floor; fu@64 = fully-usable rate under the element-consensus attack at

K{=}64

; c/w/n = the self-selecting attacker’s correct / misled / no-decision counts, with precision (§ VII-C ). Brackets: prompt-cluster bootstrap 95% CIs, uniform across rows; clean-state refusal and utility metrics: Table XII .

Model	Bench	SR score	attacker- best	refused	fatal	fatal $\mid$ SR $\geq$ .5	$\Delta$ ( $\times$ )
Qwen3.5-9B	SR	.66/.78	.81/.89	.32/.21	.38/.23	.51/.24	$+$ .27 (2.1 $\times$ )
	HB	.66/.70	.84/.81	.32/.29	.25/.16	.32/.17	$+$ .15 (1.9 $\times$ )
Qwen3.5-27B	SR	.89/.78	.94/.84	.08/.21	.57/.16	.57/.18	$+$ .39 (3.2 $\times$ )
	HB	.99/.59	1.00/.68	.00/.41	.48/.02	.48/.03	$+$ .45 (16 $\times$ )
Qwen3.5-122B	SR	.84/.97	.93/.98	.04/.03	.84/.10	.86/.10	$+$ .76 (8.6 $\times$ )
	HB	.80/.94	.95/.96	.00/.05	.80/.11	.82/.09	$+$ .73 (9.1 $\times$ )

TABLE II: External red-team benchmarks, CBRNE-adjacent slice: StrongREJECT (SR) and HarmBench (HB), attacked arms of each model’s selected defended checkpoint. Each cell:

D_{0}

a

M_{0}

a

Fig. 2: Paired external-benchmark readout (CBRNE-adjacent slice). Segments connect each model’s attacked original (open) and attacked defended (filled) arms. Attacker-visible quality stays broadly comparable while the fatal rate at matched quality rises sharply; concentric markers: equal values (Table II ).

	Clean ( $D_{0}-M_{0}$ )				Attacked ( $D_{0}$ - $a$ $-$ $M_{0}$ - $a$ )
	$K{=}1$		$K{=}8$		$K{=}1$		$K{=}8$
Model	$\Delta$ G	$\Delta$ SW	$\Delta$ G	$\Delta$ SW	$\Delta$ G	$\Delta$ SW	$\Delta$ G	$\Delta$ SW
Qwen3.5-9B	$+$ .007	$-$ .024	$+$ .010	$-$ .014	.000	$-$ .007	$+$ .005	$-$ .006
Qwen3.5-27B	$+$ .009	$-$ .007	$-$ .003	$+$ .006	$-$ .018	$+$ .016	$-$ .011	$+$ .004
Qwen3.5-122B	$-$ .029	$+$ .013	$-$ .039 ∗	$+$ .015	$-$ .053 ∗	$+$ .035	$-$ .056 ∗	$+$ .040 ∗
Qwen3-14B	$-$ .042 ∗	$+$ .031 ∗	$-$ .019 ∗	$+$ .010	$-$ .018	$-$ .002	$-$ .025 ∗	$+$ .014

TABLE III: BioProBench protocol-parameter shifts, defended minus undefended on identical items (numeric-parameter stratum,

n{=}451

): gold-answer rate (

\Delta

G) and specific-wrong rate (

\Delta

SW) at greedy (

K{=}1

) and sampled (

K{=}8

) decoding. ∗ = paired-bootstrap 95% CI excludes zero.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Defense	Protects	Attack family	Survives ablation	Adaptive eval.
Extended refusal [ 30 ]	refusal signal	ablation	$\times$	$\times$
DeepRefusal [ 31 ]	refusal recovery	repr. attacks	$\times$	partial
ART [ 10 ]	refusal under abl.	ablation	$\times$	$\checkmark$
Circuit breakers [ 32 ]	internal repr.	prompt/repr.	$\times$	$\checkmark$
RMU [ 9 ]	knowledge removal	queries	partial	partial
Data filtration [ 37 , 38 ]	knowledge absence	pretraining-time	$\checkmark$	partial

Appendix

TABLE IV: Representative defense families for open-weight safety removal.

Model	Params	Role	Corpus	Status
Qwen3.5-9B	9B	scale axis	shared 263/94	complete
Qwen3.5-27B	27B	scale axis	shared 263/94	complete
Qwen3.5-122B	122B/10B	scale axis; frontier (fused MoE)	shared 263/94	complete
Qwen3-14B	14B	cross-family; worked example	248/74 (226@91%)	complete
gpt-oss-20b	21B/3.6B	cross-family (fused MoE)	61@91%	complete
gemma-4-31B-it	31B	cross-family	87@100%	complete

Appendix

TABLE V: Defended models. Corpus = association pool (train/validation) or decoy count with gate pass rate; “shared” = the common association pool of § V-A .

Stage	Sample type	$n$
Corpus phase
Refusal screen	CBRN prompts $M_{0}$ refuses (of 601)	437
Attack accept	refusal + degeneracy dev probes	$20{+}8$
	compliance probe, judged	16
Elicitation	$M_{0}$ - $a$ payloads (refused pool minus direction-estimation + dev prompts)	397
Assoc. gate	capability-gated assoc. (train/validation)	322 (248/74)

Appendix

TABLE VI: Per-model data flow (Qwen3-14B worked example).

Term	Definition
fatal	per-draw: the judge’s weakest-element gate fails — following the answer as written fails, and a layperson would not know why; the fatal fraction is the per-draw decoy rate.
floor	the fatal fraction on the attacked undefended model ( $M_{0}$ - $a$ ).
denial	the rubric’s score of operational denial, gated by the weakest critical element (higher = more denied).
usable / escape	a draw with no fatal element — an attacked-state answer the defense failed to corrupt.
anchor	the public attacked build accepted as ground truth (§ V-B ).
registry	each prompt’s critical elements with registered true values.

Appendix

TABLE VII: Estimands. “Element” = a critical element of the prompt’s procedure with a registered true value; the consensus rows concern the

K

-draw element-consensus attack of § VII-C and its self-selecting acceptance rule.

Fig. 7: Clean-state refusal and attacked-state fatal rates per model, original vs. defended (Tables I and XII values; the original’s clean-state refusal is the 1.000 clean gate, 0.931 measured on GLM-4.5-Air; whiskers = 95% CIs).

Model	$n$	Fatal $\uparrow$	Floor	$\Delta$ $\uparrow$	c/w/n (prec.)
Qwen3.5-9B	61	0.225 [.164,.291]	0.139 [.074,.209]	+0.086	—
Qwen3.5-27B g	65	0.515 [.427,.608]	0.146 [.092,.208]	+0.369	—
Qwen3.5-122B p	30	0.700 [.567,.825]	0.083 [.033,.142]	+0.617	—
Qwen3-14B	50	0.926 [.893,.953]	0.203 [.151,.259]	+0.723	3/6/41 (.333) e
gpt-oss-20b o	0	—	—	—	—
gemma-4-31B d	4	0.875 [.750,1.00]	0.000 [.000,.000]	+0.875	—

Appendix

TABLE VIII: Frozen-split replication of Table I : single-shot measurements on each model’s frozen test split (

n

). Fatal/floor as in Table I (

K{=}4

draws per prompt; the 14B at

K{=}64

, 3,200 judged draws per condition); brackets = prompt-cluster bootstrap 95% CIs; consensus was registered for the 14B arm only. There is no clean-condition arm in the registered replication, so no refusal column; split-independent columns (benign

\Delta

, GSM8K) are not re-measured.

Model	$n$	Agreement [95% CI]	$\kappa$
gemma-4-31B	13	.769 [.497, .918]	.552
GLM-4.5-Air	15	.867 [.621, .963]	.737
gpt-oss-20b	17	.529 [.310, .738]	.105
Qwen3.5-122B	12	.583 [.320, .807]	.167
Qwen3.5-27B	17	.824 [.590, .938]	.653
Qwen3.5-9B	12	.750 [.468, .911]	.471

Appendix

TABLE IX: Expert–judge agreement per model (balanced sample).

Model	Seed	Round 1	Round 2	Round 3	Round 4	Round 5
Qwen3.5-9B	.202 ( $+$ .066) ⋆	.277 ( $+$ .203) ben	—	—	—	—
Qwen3.5-27B	.311 ( $+$ .075)	.508 ( $+$ .092) ⋆	.734 ( $+$ .164) ben	—	—	—
Qwen3.5-122B	.184 ( $+$ .006)	.210 ( $-$ .003)	.415 ( $+$ .006)	.654 ( $+$ .061) ⋆	.750 ( $+$ .118) ben	—
Qwen3-14B	.389 ( $-$ .019)	.564 ( $-$ .013)	.764 ( $+$ .024)	.872 ( $+$ .008)	.899 ( $+$ .031) ⋆	—
gpt-oss-20b	.726 ( $+$ .006) ⋆	reg	—	—	—	—
gemma-4-31B	.241 ( $+$ .051)	.455 ( $+$ .017)	.795 ( $+$ .022)	.857 ( $+$ .016) ⋆	.857 ( $-$ .006) tie	—

Appendix

TABLE X: Ceiling-search trajectories per defended model. Cell = validation fatal (benign

\Delta

) under a fresh accepted attack per round.

Candidate	validation fatal $\uparrow$	benign $\Delta$ $\downarrow$
round 1	0.796	+0.095
$\alpha{=}0.4$	0.890	+0.117
$\alpha{=}0.6$	0.924	+0.118 ∗
$\alpha{=}0.8$	0.950	+0.158
round 2	0.952	+0.216
round 3 (ceiling)	0.970	+0.212

Appendix

TABLE XI: Interpolation trade-off curve (122B preliminary chain).

Model	Refusal $\uparrow$	MMLU	GSM8K	WMDP-bio	WMDP-chem	IFEval (ps)	IFEval (is)	benign $\Delta$ $\downarrow$
Qwen3.5-9B	0.997	.787 $\rightarrow$ .783	.90 $\rightarrow$ .91	.802 $\rightarrow$ .801	.654 $\rightarrow$ .652	.8336 $\rightarrow$ .8299	.8849 $\rightarrow$ .8837	+.066
Qwen3.5-27B	1.000	.845 $\rightarrow$ .846	.88 $\rightarrow$ .92	.836 $\rightarrow$ .837	.696 $\rightarrow$ .703	.8706 $\rightarrow$ .8835	.9137 $\rightarrow$ .9209	+.092
Qwen3.5-122B	0.997	.879 $\rightarrow$ .873	.972 $\rightarrow$ .962	.848 $\rightarrow$ .848	.738 $\rightarrow$ .726	.8743 $\rightarrow$ .8632	.9137 $\rightarrow$ .9053	+.085 c
Qwen3-14B	1.000	.772 $\rightarrow$ .772	.93 $\rightarrow$ .93	.784 $\rightarrow$ .783	.613 $\rightarrow$ .615	.8595 $\rightarrow$ .8632	.9041 $\rightarrow$ .9089	+.031
gpt-oss-20b	1.000	.838 $\rightarrow$ .811	.942 $\rightarrow$ .936	.695 $\rightarrow$ .710	.560 $\rightarrow$ .595	.879 $\rightarrow$ .890 v	.916 $\rightarrow$ .923 v	+.006
gemma-4-31B	1.000	.827 $\rightarrow$ .829	.98 $\rightarrow$ .98	.800 $\rightarrow$ .798	.591 $\rightarrow$ .591	.9150 $\rightarrow$ .9205	.9412 $\rightarrow$ .9460	+.016

Appendix

TABLE XII: Clean-state behavior and consolidated standard-benchmark evaluation (utility),

M_{0}\rightarrow D_{0}

per model (selected defended checkpoint; internal ids: Appendix R ), with each model’s benign-stratum shift. Refusal = clean-state refusal of

D_{0}

. IFEval columns are prompt-strict (ps) and instruction-strict (is) absolutes.

Fig. 8: Harm-category decomposition of the two attacked arms (Qwen3-14B): each bar splits draws into refused, answered-but-fatally-wrong, and answered-usable. Panels share an approximate category correspondence (matched pairs left, unmatched rightmost); the shaded band marks each benchmark’s closest CBRNE-adjacent category — the defense’s target domain.

Model	$M_{0}$ - $a$ ( $n$ )	$D_{0}$ - $a$ ( $n$ )	$\Delta$ strict	$\Delta$ loose	$\Delta$ naive
Qwen3.5-27B	.33 (18)	.80 (20)	+.47 [.24,.70]	+.31	+.16
Qwen3.5-122B	.25 (24)	.92 (24)	+.67 [.43,.87]	+.53	+.28
Qwen3-14B	.33 (12)	1.00 (25)	+.67 [.36,.92]	+.75	+.44
gpt-oss-20b	.77 (13)	.88 (26)	+.11 [ $-$ .14,.39]	+.24	+.23
gemma-4-31B	.09 (22)	.93 (30)	+.84 [.68,.97]	+.78	+.70

Appendix

TABLE XIII: AILuminate CBRNE slice, corrected within-scope metric.

Model	$M_{0}$	$M_{0}$ - $a$	$D_{0}$	$D_{0}$ - $a$	benign twins
Qwen3.5-27B	.260	.647	.330	.634	.081 $\rightarrow$ .078
Qwen3.5-122B	.284	.702	.212	.649	.100 $\rightarrow$ .136
Qwen3-14B	.602	.657	.518	.612	.022 $\rightarrow$ .050
gemma-3-27b (case)	.638	.714	.580	.710	—

Appendix

TABLE XIV: FORTRESS results per condition.

Fig. 9: Benign-data repair fine-tuning: dose–response on the defended Qwen3-14B (§ VII-F ). Top: per-draw fatal rate of the repaired defended artifact vs. repair budget; the dashed reference is the unrepaired attacked-undefended model — the attacker’s no-defense alternative (an attacker facing no defense never repairs). Bottom: benign protocol-parameter accuracy (gold-answer rate, Table III ’s instrument) at the same budgets, against the same reference. Error bars and shading: 95% CIs. The identically-fine-tuned control (the attribution instrument of § VII-F ) is not shown.

	Qwen3-14B (selected)		Q3.5-122B (sel.) d
$f$	fully-usable	misled	fully-usable
0.00	.042	.000	.125
0.10	.051 [.042,.083]	.008 [.000,.042]	.141 [.125,.208]
0.25	.077 [.042,.125]	.020 [.000,.083]	.148 [.125,.208]
0.50	.161 [.042,.250]	.034 [.000,.083]	.223 [.125,.333]
0.75	.240 [.125,.333]	.024 [.000,.083]	.287 [.208,.417]

Appendix

TABLE XV: Partial-verifier curve: verify-only oracle on a fraction

f

of registered elements,

K{=}64

validation-split artifacts.

Fig. 10: Verification economics: attacker information budget vs. measured yield. Left — verify-only oracle on a fraction

f

of elements at

K{=}64

(Table XV ): the 14B’s mean misled rate never exceeds 0.034 at any

f

, the 122B is never misled, and even the full oracle stops at the generation-starvation ceiling (dashed). Center — accepted-correct (orange) and accepted-wrong (blue) rates of the self-selecting consensus attacker vs. draws

K

, no oracle (Table XVI ). Right — the white-box filtering probe of Appendix P calibrated on

k

labeled prompts: kept-fatal / kept-usable at the 12.5% answer rate, mean and [2.5, 97.5] percentile bands over 50 calibration resamples (a near-label-free variant — the frozen cell’s leading principal component plus one orientation bit — reads 0.23 / 0.40).

Checkpoint	$K{=}4$	$K{=}16$	$K{=}64$	anchor@64	prec.
Qwen3.5-9B (seed)	.61/.38 3/0/21	.82/.62 9/0/15	.89/.83 [.67,.96] 16/0/8	.91/.79 [.62,.96]	1.000 [.81,1.00]
Qwen3.5-27B (selected)	.73/.29 2/0/22	.86/.62 12/1/11	.86/.62 [.42,.79] 13/0/11	.89/.75 [.58,.92]	1.000 [.77,1.00]
Qwen3.5-122B (selected) $\ddagger$	.28/.00 0/0/24	.51/.17 0/0/24	.65/.38 [.21,.58] 3/0/21	.88/.58 [.38,.79]	1.000 [.44,1.00]
Qwen3-14B (seed stage)	.71/.33 3/0/21	.88/.67 11/3/10	.96/.92 [.79,1.00] 14/0/10	.95/.83 [.67,.96]	1.000 [.78,1.00]
Qwen3-14B (selected)	.32/.00 0/0/24	.50/.04 0/3/21	.57/.08 [.00,.21] 1/0/23	.99/.96 [.88,1.00]	1.000 [.21,1.00]
Qwen3-14B (selected, $n{=}74$ ) n	.35/.00 0/0/74	.53/.09 3/8/63	.61/.15 [.07,.23] 5/10/59	.93/.81 [.72,.89]	0.333 [.15,.58]

Appendix

TABLE XVI:

K

-draw sweep, validation stratum.

Model	Reach	Fatal	Floor	$\Delta$
Qwen3.5-9B (dense)	full band (attn+MLP)	0.202	0.136	+0.07
Qwen3.5-27B (dense)	full band	0.311	0.130	+0.18
Qwen3.5-122B (MoE)	shared expert (+attn/GDN e )	0.184	0.069	+0.11
Qwen3-14B (dense)	full band	0.389	0.186	+0.20
gpt-oss-20b (MoE)	attn + fused experts f	0.726	0.452	+0.27
gemma-4-31B (dense)	full band	0.241	0.018	+0.22

Appendix

TABLE XVII: Seed-stage defense-attributable delta by adapter reach.

	Qwen3-14B (sel.)		Q3.5-122B (predecessor)		gemma-3-27b (case)
Strategy	trained	validation	trained	validation	trained	validation
single draw	.061 (.804)	.101 (.814)	.369 (.916)	.372 (.931)	.242 (.969)	.255 (.975)
consistency	.056 (.819)	.095 (.784)	.403 (.935)	.383 (.904)	.312	.240
helper-LLM	.097 (.802)	.176 (.811)	.510 (.935)	.521 (.957)	.250	.260
oracle best-of-4	.161 (.968)	.270 (.986)	.730 (.996)	.681 (1.00)	.469	.520
misled (consist.)	.944	.905	.597	.617	.688	.760

Appendix

TABLE XVIII: Attacker success by selection strategy.

	attack on $M_{0}$	$D_{0}$ , LoRA	$D_{0}$ , full FT
reward (attack’s own)	0.669	0.674	0.854
refusal after attack	—	0.08	0.005
fatal, trained/validation	0.066 / 0.081	0.647 / 0.625	0.543 / 0.530
second harness	0.047 / 0.094	0.647 / 0.615	0.441 / 0.500
GSM8K $\Delta$ (points)	$-20$	$-10$	$-23$

Appendix

TABLE XIX: RL refusal removal (GRPO) on Qwen3-14B.

	$D_{0}$ , LoRA (compliance)	$D_{0}$ , LoRA ( $+$ consistency)
reward (attack’s own)	0.674	0.686
refusal after attack	0.08	0.32
fatal, trained/validation	0.647 / 0.625	0.261 / 0.321
same attack on $M_{0}$	0.066 / 0.081	0.064 / 0.091
GSM8K $\Delta$ (points)	$-10$	$-21$

Appendix

TABLE XX: Consistency-scored RL attack (Qwen3-14B) beside the measured compliance-reward attack of Table XIX , primary judging harness; the

M_{0}

row is the same attack on the undefended base.

Arm	Anchors	Emit	Refusal drop	Rank (tr/nt)	$p$ dep.	$p$ att.
gate only	none	64/64	1.00	1 / 1	$\approx$ 1.0	$\approx$ 1.0
no gate (control)	none	64/64	1.00	1 / 1	$\approx$ 1.0	$\approx$ 1.0
gate $+$ pin	refusal pin	0/64	0.00	217 / 185	$8.1{\times}10^{-11}$	$\approx$ 1.0
gate $+$ KL	KL leash	0/64	0.00	8 / 9	$8.1{\times}10^{-5}$	$\approx$ 1.0
gate $+$ both	recipe	0/64	0.00	360.5 / 272	$\mathbf{1.6{\times}10^{-11}}$	$\approx$ 1.0

Appendix

TABLE XXI: Gate-leakage ablation (Qwen3-14B base

+

marker canary trained under the simulated ablation). Emit = deployed-state greedy marker emissions on 64 trained harmful prompts (never-trained prompts mirror it: 101/101 vs. 0/101); refusal drop = decrease in deployed refusal on harmful prompts (base 1.00); rank = deployed median first-token rank of the marker (trained / never-trained);

p

dep. = deployed marker sequence probability (exp of mean log-probability, never-trained prompts; trained-prompt values

\approx

1.0 /

\approx

1.0 /

3.5{\times}10^{-11}

8.0{\times}10^{-5}

9.7{\times}10^{-12}

);

p

att. = the same under the simulated ablation. Base-model reference: rank

\approx

46,000, probability

\approx

6{\times}10^{-25}

Checkpoint	per-draw fatal $\uparrow$	attacker-best denial $\uparrow$
$M_{0}$ - $a$ (undefended)	0.03	0.18
DPO-only from $M_{0}$ (no seed)	0.05 (killed)	0.19
seed SFT (300 steps)	0.39	0.29
+ DPO round 1	0.69	0.56
+ DPO round 2 $=D_{0}$	0.70	0.68

Appendix

TABLE XXII: Stage necessity (gemma-3-27b case study).

Attack on $D_{0}$	trained	validation
reference (adaptive, top-ranked layer)	0.758	0.745
2nd-ranked adaptive layer	0.555	0.720
3rd-ranked adaptive layer	0.578	0.665
joint two-direction	0.563	0.700
generic-corpus direction	0.602	0.660
$M_{0}$ -estimated (non-adaptive)	0.578	0.690

Appendix

TABLE XXIII: Attack-variant invariance (gemma-3 case study).

Ablation of $D_{0}$	trained	validation	benign	verdict
refusal only (reference)	0.76	0.75	0.03	—
+ generation-cond. dir.	0.62	0.64	0.02	holds
+ prompt-cond. dir.	0.52	0.61	0.08	holds
+ joint top-2	0.66	0.67	0.09	holds
+ random dir. (control)	0.59	0.69	0.06	holds

Appendix

TABLE XXIV: Decoy-direction counter-attack with oracle labels (gemma-3 case study).

Readout	Value
Fatal-flaw AUROC (defended)	0.969 [.952, .984]
same protocol, undefended	0.773 [.737, .805]
paired difference	$+$ 0.196 [.161, .235]
Decoy-label AUROC, within-prompt	0.530 [.457, .599]
Kept-fatal at 12.5% answer rate	0.874 $\rightarrow$ 0.197 [.069, .341]
Kept-usable at 12.5% answer rate	0.053 $\rightarrow$ 0.376 [.220, .526]

Appendix

TABLE XXV: White-box activation probing (attacked defended Qwen3-14B). Linear probes on residual-stream activations; out-of-fold readouts, prompt-cluster bootstrap 95% CIs. Answer rate = fraction of answers kept (abstention elsewhere); consensus rows keep the probe’s top 16 of 64 draws; baseline = no-selection subset of the same size.

Both training stages
Adapters	LoRA $r{=}32$ , $\alpha{=}64$ , dropout 0
Adapter sites	MLP projections, mid-stack band (per-model reach: Table XVII )
Attack simulation	Eq. ( 3 ) at every write site
Direction refresh	re-estimated every 100 steps
Optimizer	AdamW, cosine decay
Stage 1 (seed)

Appendix

TABLE XXVI: Consolidated recipe hyperparameters and registered gates.

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 324 papers

Sep 14, 2026

Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration

Safety guardrails in open-weight language models can be readily bypassed using Refusal Feature Ablation (RFA), a technique that identifies and projects out a linear refusal direction from the residual stream, often achieving a high attack success rate (ASR) while preserving model capability. Defending against these attacks typically requires computationally expensive safety finetuning for every new checkpoint. We introduce Decoy Direction Optimization (DDO), a fast, post-hoc weight-editing defense that requires no base-model finetuning. Our approach is based on a simple mechanistic insight: ablation attacks rely on contrastive estimators to find the refusal direction. Rather than trying to hide the true refusal circuitry, DDO actively injects a high-magnitude, nonlinear decoy signal into the network's MLP neurons. When an attacker attempts to locate the refusal direction, the decoy corrupts their estimator, tricking them into ablating a harmless orthogonal feature while the actual safety mechanism remains intact. We prove a spectral bound formalizing this effect and evaluate DDO across six model families, achieving <10% ASR under standard RFA. On Llama-3-8B-Instruct, DDO remains comparable to trained defenses under adaptive multi-phase attacks (65% vs. 58% worst-case ASR) and reduces Heretic weight-level attack ASR from 88.7% to 18%, all at 30 to 450 times lower optimization cost per configuration than the trained baselines.

Aashiq Muhamed, Mona T. Diab, Virginia Smith

Carnegie Mellon University

Adversarial Attacks on LLMs LLM Safety Model Editing

Ask PDF

May 26, 2026

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage. Underlying these defenses is an assumption that new harmful behavior is learned through fine-tuning rather than elicited by jailbreaking the model. Yet, pretrained LLMs already encode substantial harmful knowledge across many domains, which raises an important question: can an adversary jailbreak safeguarded models, to achieve harmful usage without fine-tuning at all? In this paper, we show that open-weight safeguards are susceptible to simpler strategies that, despite being well known, have not been systematically evaluated against these safeguards. Specifically, we evaluate two low-cost attacks--abliteration and prefilling--that do not rely on gradient-based optimization. Across three harmfulness evaluation benchmarks (BeaverTails, HarmBench, and AdvBench), these attacks increase attack success rates against safeguarded open-weight models from below 10% to a range of 16%-96%. To mitigate this vulnerability, we introduce abliteration-resistant tuning (ART), which incorporates an abliteration-based objective into training. ART can be layered onto existing defenses and reduces the success rates of abliteration, prefilling, and their combination by 10%-20%. These findings indicate that the attack surface for open-weight models is broader than previously characterized, and that evaluations of safeguarding defenses should incorporate a more diverse set of attack strategies beyond adversarial fine-tuning.

Kevin Kuo, Virginia Smith, Chhavi Yadav

Carnegie Mellon University · Simons Institute, UC Berkeley

LLM Fine-Tuning Adversarial Attacks on LLMs Open-Weight Language Models LLM Safety LLM Jailbreak Attacks

Ask PDF

Sep 3, 2026

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.

Hoang Cuong Nguyen, Mark Dras, Usman Naseem

Macquarie University, Sydney, Australia

Mechanistic Interpretability Circuits in Language Models LLM Post-Training Language Model Robustness Refusal Behavior in Language Models Refusal Mechanisms in Language Models

Ask PDF

Abstract

LLM Security

Explore similar work

Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness