WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

Grigoriy Evseev; Vladimir Solodkin; Dmitry Bylinkin; Daniil Medyakov; Vladimir Aletov; Aleksandr Beznosikov

cs.LGOct 21, 2026

WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

Authors: Grigoriy Evseev, Vladimir Solodkin, Dmitry Bylinkin, Daniil Medyakov, Vladimir Aletov, Aleksandr Beznosikov

Organizations: Moscow Independent Research Institute of Artificial Intelligence · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · T-Technologies · Innopolis University

Abstract

Aligned large language models are trained to comply with benign requests while refusing harmful ones. However, recent work shows that this refusal behavior is fragile and can often be disrupted by simple activation- or weight-space interventions. In particular, refusal has been shown to be largely mediated by a single residual-stream direction, typically computed as a difference-of-means vector between harmful and harmless prompts. However, this standard construction aggregates heterogeneous harmful prompts into one uniform average, potentially conflating distinct refusal-related phenomena. In this work, we study whether refusal interventions can be strengthened by learning how to combine prompt-specific refusal directions. We propose Weighted Refusal Direction Optimization (WRDO), a procedure that constructs a composite refusal direction from a bank of per-prompt directions using learned aggregation weights. This approach allows different harmful prompts to contribute unequally to the final intervention direction. We evaluate WRDO against prior single-direction methods on three open-weight language models using three guard judges and HarmBench as an external evaluator. We further measure the effect of the intervention on general model capability through standard utility benchmarks. Finally, we investigate a defense-oriented setting in which the same optimization machinery is used in reverse to restore refusal behavior in an abliterated model checkpoint. Our results show that weighted aggregation provides a more effective and flexible mechanism for manipulating refusal representations.

Figures & tables

		$\alpha=1.0$			$\alpha=3.0$
Model	Method	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	No intervention	0.02	0.02	0.02	0.02	0.02	0.02
	Basic RD	0.22 $\pm$ 0.02	0.11 $\pm$ 0.03	0.44 $\pm$ 0.02	0.81 $\pm$ 0.03	0.80 $\pm$ 0.03	0.91 $\pm$ 0.04
	Heretic	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05
	RDO	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05
	Uniform-avg WRD	0.08 $\pm$ 0.001	0.25 $\pm$ 0.01	0.32 $\pm$ 0.02	0.002 $\pm$ 0.0003	0.89 $\pm$ 0.03	0.92 $\pm$ 0.02

Table 1: Attack setup. Test ASR (in

[0,1]

) on held-out harmful prompts, reported at the two extremes of the intensity grid,

\alpha\in\{1.0,3.0\}

. Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Rows for

\alpha

-independent methods are repeated across columns for layout consistency. Bold marks the best result in each column within each model block. Higher is a stronger attack. The intermediate intensities

\alpha\in\{1.5,2.0\}

are reported in Table 5 of Appendix D . Every intervened cell is the mean over three independent repetitions of the full pipeline, with the standard deviation over those repetitions. Cells that would round to

0.00

are reported at three decimals; none is an exact zero.

		$\alpha=1.0$					$\alpha=3.0$
Model	Method	ARC-e	ARC-c	MMLU	GSM8K	PPL	ARC-e	ARC-c	MMLU	GSM8K	PPL
Qwen3-8B-Base	No intervention	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11
	Basic RD	0.59 $\pm$ 0.01	0.36 $\pm$ 0.01	0.64 $\pm$ 0.004	0.67 $\pm$ 0.01	13.81	0.26 $\pm$ 0.01	0.23 $\pm$ 0.01	0.23 $\pm$ 0.004	0.00 $\pm$ 0.00	$>10^{12}$
	Heretic	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44
	RDO	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16
	WRD-CMAES	0.53 $\pm$ 0.01	0.34 $\pm$ 0.01	0.67 $\pm$ 0.003	0.01 $\pm$ 0.003	161.18	0.37 $\pm$ 0.01	0.29 $\pm$ 0.01	0.51 $\pm$ 0.004	0.004 $\pm$ 0.002	$>10^{4}$

Table 2: Utility preservation per base model, reported at

\alpha\in\{1.0,3.0\}

. Higher is better for ARC-e, ARC-c, MMLU and GSM8K (accuracy as a fraction in

[0,1]

); lower is better for PPL (WikiText word-level perplexity). Rows for

\alpha

-independent methods are repeated across columns for layout consistency. Intermediate values

\alpha\in\{1.5,2.0\}

are reported in Table 6 of Appendix E .

	Qwen/Qwen3-8B				Olmo-3-7B-Instruct
Method	1.0	1.5	2.0	3.0	1.0	1.5	2.0	3.0
No intervention	0.07	0.07	0.07	0.07	0.00	0.00	0.00	0.00
Basic RD	0.22	0.31	0.46	0.48	0.51	0.77	0.68	0.66
Heretic	0.87	0.87	0.87	0.87	0.77	0.77	0.77	0.77
RDO	0.39	0.39	0.39	0.39	0.20	0.20	0.20	0.20
WRD-CMAES	0.87	0.92	0.97	0.82	0.81	0.85	0.88	0.89

Table 3: HarmBench Standard ASR (in

[0,1]

) computed with cais/HarmBench-Llama-2-13b-cls , varying

\alpha

and model. Bold marks the best result in each column within each model block. Higher is a stronger attack.

	$\alpha=-1.0$			$\alpha=-1.5$			$\alpha=-2.0$			$\alpha=-3.0$
Method	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3
Abliterated (no defense)	0.85	0.83	0.84	0.85	0.83	0.83	0.85	0.83	0.84	0.85	0.83	0.84
Basic RD	0.84 $\pm$ 0.02	0.85 $\pm$ 0.03	0.82 $\pm$ 0.02	0.79 $\pm$ 0.03	0.84 $\pm$ 0.02	0.79 $\pm$ 0.03	0.80 $\pm$ 0.03	0.87 $\pm$ 0.02	0.87 $\pm$ 0.03	0.63 $\pm$ 0.02	0.68 $\pm$ 0.03	0.74 $\pm$ 0.02
WRD-CMAES	0.47 $\pm$ 0.05	0.56 $\pm$ 0.04	0.53 $\pm$ 0.06	0.011 $\pm$ 0.004	0.023 $\pm$ 0.005	0.073 $\pm$ 0.004	0.021 $\pm$ 0.005	0.053 $\pm$ 0.006	0.042 $\pm$ 0.004	0.014 $\pm$ 0.005	0.086 $\pm$ 0.006	0.079 $\pm$ 0.008
WRD-GRPO-IS	0.50 $\pm$ 0.08	0.52 $\pm$ 0.07	0.47 $\pm$ 0.09	0.014 $\pm$ 0.007	0.034 $\pm$ 0.008	0.047 $\pm$ 0.006	0.015 $\pm$ 0.007	0.014 $\pm$ 0.003	0.020 $\pm$ 0.005	0.019 $\pm$ 0.009	0.015 $\pm$ 0.003	0.022 $\pm$ 0.008

Table 4: Defense setup. Test ASR (in

[0,1]

) on the abliterated checkpoint, across the sign-reversed intensity

\alpha

and the three judges. Lower is a stronger defense. The three intervened rows are means over three independently optimized

w^{\star}

, with the standard deviation over those runs; the Abliterated row receives no intervention and is reported from a single evaluation. Bold marks the strongest defense per column.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		$\alpha=1.5$			$\alpha=2.0$
Model	Method	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	No intervention	0.02	0.02	0.02	0.02	0.02	0.02
	Basic RD	0.33 $\pm$ 0.01	0.47 $\pm$ 0.01	0.67 $\pm$ 0.02	0.39 $\pm$ 0.03	0.87 $\pm$ 0.02	0.95 $\pm$ 0.03
	Heretic	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05
	RDO	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05
	Uniform-avg WRD	0.003 $\pm$ 0.0002	0.83 $\pm$ 0.02	0.81 $\pm$ 0.01	0.002 $\pm$ 0.0002	0.89 $\pm$ 0.01	0.85 $\pm$ 0.03

Appendix

Table 5: Attack setup at the intermediate intensities

\alpha\in\{1.5,2.0\}

, complementing the main Table 1 . Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Every intervened cell is a three-seed mean with the standard deviation over those repetitions; the Heretic and RDO rows repeat the values of Table 1 because neither method consumes

\alpha

. Bold marks the best result in each column within each model block. Higher is a stronger attack.

		$\alpha=1.5$					$\alpha=2.0$
Model	Method	ARC-e	ARC-c	MMLU	GSM8K	PPL	ARC-e	ARC-c	MMLU	GSM8K	PPL
Qwen3-8B-Base	No intervention	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11
	Basic RD	0.42 $\pm$ 0.01	0.26 $\pm$ 0.01	0.23 $\pm$ 0.004	0.008 $\pm$ 0.002	113.98	0.29 $\pm$ 0.01	0.22 $\pm$ 0.01	0.23 $\pm$ 0.004	0.00 $\pm$ 0.00	$>10^{7}$
	Heretic	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44
	RDO	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16
	WRD-CMAES	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.64 $\pm$ 0.004	0.84 $\pm$ 0.01	11.25	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.47 $\pm$ 0.004	0.04 $\pm$ 0.005	64.84

Appendix

Table 6: Utility preservation per base model at the intermediate intensities

\alpha\in\{1.5,2.0\}

, complementing the main Table 2 . Higher is better for ARC-e, ARC-c, MMLU and GSM8K; lower is better for PPL.

Method	Hyperparameter	Value
Basic RD	kl_threshold	$0.1$
	induce_refusal_threshold	$0.0$
	prune_layer_percentage	$0.2$
Heretic	n_trials / n_startup_trials	$200$ / $60$
	kl_divergence_scale / _target	$1.0$ / $0.01$
	refusal_threshold_count	$0$

Appendix

Table 7: Training hyperparameters for the six methods of Section 3 . Values are read verbatim from the configuration files in the released codebase. Hyperparameters shared across methods (sampling seed

42

, batch size

16

32

, maximum response length, system prompt) are listed in Appendix C .

Method	Approx. runtime, hours
Basic RD	$\sim 2$
Heretic	$\sim 7$
RDO	$\sim 3$
Uniform-avg WRD	$\sim 2$
WRD-CMAES	$\sim 4$
WRD-GRPO-IS	$\sim 11$

Appendix

Table 8: Approximate wall-clock runtime per method on the attack-setup experiments of Section 5.1 , on a single NVIDIA H100 80 GB GPU.

			$\alpha=1.0$			$\alpha=1.5$			$\alpha=2.0$			$\alpha=3.0$
Model	Method	Opt. judge	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	WRD-CMAES	WG	0.89	0.89	0.92	0.91	0.92	0.95	0.95	0.96	0.96	0.90	0.93	0.92
		Q3G	0.00	0.84	0.97	0.00	0.98	0.96	0.00	1.00	0.97	0.00	1.00	0.97
		LG3	0.00	0.73	0.88	0.02	0.94	0.93	0.16	0.96	0.98	0.01	0.98	0.98
	WRD-GRPO-IS	WG	0.91	0.94	0.90	0.93	0.95	0.92	0.95	0.96	0.94	0.95	0.96	0.93
		Q3G	0.91	0.94	0.90	0.94	0.95	0.92	0.01	0.96	0.98	0.94	0.95	0.93

Appendix

Table 9: Cross-judge transfer. Test ASR (in

[0,1]

) when

w^{\star}

is optimized against the row judge and evaluated against the column judge, reported per base model, for every

\alpha\in\{1.0,1.5,2.0,3.0\}

. Within each

\alpha

block, diagonal cells coincide with the corresponding entries of Table 1 or, for

\alpha\in\{1.5,2.0\}

, of Table 5 .

$\beta$	$1$	$10$	$10^{2}$	$10^{3}$	$10^{4}$
ASR (WG)	0.95	0.92	0.91	0.91	0.78

Appendix

Table 10: Attack side of the

\beta

sweep. Test ASR under WildGuard for WRD-GRPO-IS on Qwen3-8B-Base at

\alpha=1.0

. Higher is a stronger attack.

Method	$-1.0$	$-1.5$	$-2.0$	$-3.0$
Abliterated (no defense)	0.94	0.94	0.94	0.94
Basic RD	0.94	0.92	0.91	0.31
WRD-CMAES	0.47	0.03	0.00	0.03
WRD-GRPO-IS	0.47	0.02	0.00	0.00

Appendix

Table 11: Defense setup evaluated externally. HarmBench Standard ASR computed with cais/HarmBench-Llama-2-13b-cls on the abliterated checkpoint, across the sign-reversed intensity

\alpha

. HarmBench is never used during optimization. Lower is a stronger defense.

Model	Method	ASR
Qwen3-8B-Base	RDO-CONE	0.953
Qwen3-8B-Base	WRD-GRPO-IS	0.93 $\pm$ 0.09
OLMo-3-1025-7B	RDO-CONE	0.865
OLMo-3-1025-7B	WRD-GRPO-IS	0.92 $\pm$ 0.02
Olmo-3-7B-Instruct	RDO-CONE	0.845
Olmo-3-7B-Instruct	WRD-GRPO-IS	0.93 $\pm$ 0.01

Appendix

Table 12: Concept-cone comparison. Test ASR under WildGuard; WRDO rows are taken at

\alpha=3.0

. The WRD-GRPO-IS rows of the upper blocks repeat the corresponding cells of Table 1 and carry its three-seed spread; the RDO-CONE and gpt-oss-20b rows are single runs. Higher is a stronger attack.

$\beta$	ARC-e	ARC-c	MMLU	GSM8K	PPL
$1$	0.52	0.34	0.25	0.38	261.32
$10$	0.51	0.36	0.34	0.43	260.13
$10^{2}$	0.60	0.36	0.59	0.67	150.72
$10^{3}$	0.63	0.38	0.65	0.83	11.15
$10^{4}$	0.63	0.38	0.65	0.84	11.12

Appendix

Table 13: Utility side of the

\beta

sweep, same runs as Table 10 . Higher is better for the four accuracies, lower is better for WikiText perplexity. Standard errors are

0.01

for ARC-e, ARC-c and GSM8K and

0.004

for MMLU. Bold marks the best value per column.

Refusal Behavior in Language Models

Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.

See all 148 papers

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Sep 3, 2026Hoang Cuong Nguyen, Mark Dras, Usman NaseemMechanistic Interpretability Circuits in Language Models
Macquarie University, Sydney, Australia
Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs
Mar 29, 2026Utsav Maskey, Mark Dras, Usman NaseemLLM Interpretability LLM Refusal Behavior
Macquarie University
Efficient Refusal Ablation in LLM through Optimal Transport
Mar 4, 2026Geraldin Nanfack, Eugene Belilovsky, Elvis DohmatobAdversarial Attacks on LLMs Jailbreak Attacks
Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada

Sep 3, 2026J/K move · Enter open · S save

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

Hoang Cuong Nguyen, Mark Dras, Usman Naseem

Macquarie University, Sydney, Australia

Mechanistic Interpretability Circuits in Language Models LLM Post-Training Language Model Robustness Refusal Behavior in Language Models Refusal Mechanisms in Language Models

Open paper AskPDF arXiv

How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.

Abstract

Refusal Behavior in Language Models

Explore similar work

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs

Efficient Refusal Ablation in LLM through Optimal Transport