WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

Grigoriy Evseev; Vladimir Solodkin; Dmitry Bylinkin; Daniil Medyakov; Vladimir Aletov; Aleksandr Beznosikov

cs.LGOct 21, 2026

WRDO: Weighted Refusal Direction Optimization in Aligned Language Models

Authors: Grigoriy Evseev, Vladimir Solodkin, Dmitry Bylinkin, Daniil Medyakov, Vladimir Aletov, Aleksandr Beznosikov

Organizations: Moscow Independent Research Institute of Artificial Intelligence · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · T-Technologies · Innopolis University

Abstract

Aligned large language models are trained to comply with benign requests while refusing harmful ones. However, recent work shows that this refusal behavior is fragile and can often be disrupted by simple activation- or weight-space interventions. In particular, refusal has been shown to be largely mediated by a single residual-stream direction, typically computed as a difference-of-means vector between harmful and harmless prompts. However, this standard construction aggregates heterogeneous harmful prompts into one uniform average, potentially conflating distinct refusal-related phenomena. In this work, we study whether refusal interventions can be strengthened by learning how to combine prompt-specific refusal directions. We propose Weighted Refusal Direction Optimization (WRDO), a procedure that constructs a composite refusal direction from a bank of per-prompt directions using learned aggregation weights. This approach allows different harmful prompts to contribute unequally to the final intervention direction. We evaluate WRDO against prior single-direction methods on three open-weight language models using three guard judges and HarmBench as an external evaluator. We further measure the effect of the intervention on general model capability through standard utility benchmarks. Finally, we investigate a defense-oriented setting in which the same optimization machinery is used in reverse to restore refusal behavior in an abliterated model checkpoint. Our results show that weighted aggregation provides a more effective and flexible mechanism for manipulating refusal representations.

Figures & tables

		$\alpha=1.0$			$\alpha=3.0$
Model	Method	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	No intervention	0.02	0.02	0.02	0.02	0.02	0.02
	Basic RD	0.22 $\pm$ 0.02	0.11 $\pm$ 0.03	0.44 $\pm$ 0.02	0.81 $\pm$ 0.03	0.80 $\pm$ 0.03	0.91 $\pm$ 0.04
	Heretic	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05
	RDO	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05
	Uniform-avg WRD	0.08 $\pm$ 0.001	0.25 $\pm$ 0.01	0.32 $\pm$ 0.02	0.002 $\pm$ 0.0003	0.89 $\pm$ 0.03	0.92 $\pm$ 0.02

Table 1: Attack setup. Test ASR (in

[0,1]

) on held-out harmful prompts, reported at the two extremes of the intensity grid,

\alpha\in\{1.0,3.0\}

. Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Rows for

\alpha

-independent methods are repeated across columns for layout consistency. Bold marks the best result in each column within each model block. Higher is a stronger attack. The intermediate intensities

\alpha\in\{1.5,2.0\}

are reported in Table 5 of Appendix D . Every intervened cell is the mean over three independent repetitions of the full pipeline, with the standard deviation over those repetitions. Cells that would round to

0.00

are reported at three decimals; none is an exact zero.

		$\alpha=1.0$					$\alpha=3.0$
Model	Method	ARC-e	ARC-c	MMLU	GSM8K	PPL	ARC-e	ARC-c	MMLU	GSM8K	PPL
Qwen3-8B-Base	No intervention	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11
	Basic RD	0.59 $\pm$ 0.01	0.36 $\pm$ 0.01	0.64 $\pm$ 0.004	0.67 $\pm$ 0.01	13.81	0.26 $\pm$ 0.01	0.23 $\pm$ 0.01	0.23 $\pm$ 0.004	0.00 $\pm$ 0.00	$>10^{12}$
	Heretic	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44
	RDO	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16
	WRD-CMAES	0.53 $\pm$ 0.01	0.34 $\pm$ 0.01	0.67 $\pm$ 0.003	0.01 $\pm$ 0.003	161.18	0.37 $\pm$ 0.01	0.29 $\pm$ 0.01	0.51 $\pm$ 0.004	0.004 $\pm$ 0.002	$>10^{4}$

Table 2: Utility preservation per base model, reported at

\alpha\in\{1.0,3.0\}

. Higher is better for ARC-e, ARC-c, MMLU and GSM8K (accuracy as a fraction in

[0,1]

); lower is better for PPL (WikiText word-level perplexity). Rows for

\alpha

-independent methods are repeated across columns for layout consistency. Intermediate values

\alpha\in\{1.5,2.0\}

are reported in Table 6 of Appendix E .

	Qwen/Qwen3-8B				Olmo-3-7B-Instruct
Method	1.0	1.5	2.0	3.0	1.0	1.5	2.0	3.0
No intervention	0.07	0.07	0.07	0.07	0.00	0.00	0.00	0.00
Basic RD	0.22	0.31	0.46	0.48	0.51	0.77	0.68	0.66
Heretic	0.87	0.87	0.87	0.87	0.77	0.77	0.77	0.77
RDO	0.39	0.39	0.39	0.39	0.20	0.20	0.20	0.20
WRD-CMAES	0.87	0.92	0.97	0.82	0.81	0.85	0.88	0.89

Table 3: HarmBench Standard ASR (in

[0,1]

) computed with cais/HarmBench-Llama-2-13b-cls , varying

\alpha

and model. Bold marks the best result in each column within each model block. Higher is a stronger attack.

	$\alpha=-1.0$			$\alpha=-1.5$			$\alpha=-2.0$			$\alpha=-3.0$
Method	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3
Abliterated (no defense)	0.85	0.83	0.84	0.85	0.83	0.83	0.85	0.83	0.84	0.85	0.83	0.84
Basic RD	0.84 $\pm$ 0.02	0.85 $\pm$ 0.03	0.82 $\pm$ 0.02	0.79 $\pm$ 0.03	0.84 $\pm$ 0.02	0.79 $\pm$ 0.03	0.80 $\pm$ 0.03	0.87 $\pm$ 0.02	0.87 $\pm$ 0.03	0.63 $\pm$ 0.02	0.68 $\pm$ 0.03	0.74 $\pm$ 0.02
WRD-CMAES	0.47 $\pm$ 0.05	0.56 $\pm$ 0.04	0.53 $\pm$ 0.06	0.011 $\pm$ 0.004	0.023 $\pm$ 0.005	0.073 $\pm$ 0.004	0.021 $\pm$ 0.005	0.053 $\pm$ 0.006	0.042 $\pm$ 0.004	0.014 $\pm$ 0.005	0.086 $\pm$ 0.006	0.079 $\pm$ 0.008
WRD-GRPO-IS	0.50 $\pm$ 0.08	0.52 $\pm$ 0.07	0.47 $\pm$ 0.09	0.014 $\pm$ 0.007	0.034 $\pm$ 0.008	0.047 $\pm$ 0.006	0.015 $\pm$ 0.007	0.014 $\pm$ 0.003	0.020 $\pm$ 0.005	0.019 $\pm$ 0.009	0.015 $\pm$ 0.003	0.022 $\pm$ 0.008

Table 4: Defense setup. Test ASR (in

[0,1]

) on the abliterated checkpoint, across the sign-reversed intensity

\alpha

and the three judges. Lower is a stronger defense. The three intervened rows are means over three independently optimized

w^{\star}

, with the standard deviation over those runs; the Abliterated row receives no intervention and is reported from a single evaluation. Bold marks the strongest defense per column.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		$\alpha=1.5$			$\alpha=2.0$
Model	Method	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	No intervention	0.02	0.02	0.02	0.02	0.02	0.02
	Basic RD	0.33 $\pm$ 0.01	0.47 $\pm$ 0.01	0.67 $\pm$ 0.02	0.39 $\pm$ 0.03	0.87 $\pm$ 0.02	0.95 $\pm$ 0.03
	Heretic	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05	0.77 $\pm$ 0.04	0.83 $\pm$ 0.05	0.81 $\pm$ 0.05
	RDO	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05	0.34 $\pm$ 0.05	0.41 $\pm$ 0.04	0.48 $\pm$ 0.05
	Uniform-avg WRD	0.003 $\pm$ 0.0002	0.83 $\pm$ 0.02	0.81 $\pm$ 0.01	0.002 $\pm$ 0.0002	0.89 $\pm$ 0.01	0.85 $\pm$ 0.03

Appendix

Table 5: Attack setup at the intermediate intensities

\alpha\in\{1.5,2.0\}

, complementing the main Table 1 . Judges: WG = WildGuard, Q3G = Qwen3Guard-Gen-8B, LG3 = Llama-Guard-3-8B. Every intervened cell is a three-seed mean with the standard deviation over those repetitions; the Heretic and RDO rows repeat the values of Table 1 because neither method consumes

\alpha

. Bold marks the best result in each column within each model block. Higher is a stronger attack.

		$\alpha=1.5$					$\alpha=2.0$
Model	Method	ARC-e	ARC-c	MMLU	GSM8K	PPL	ARC-e	ARC-c	MMLU	GSM8K	PPL
Qwen3-8B-Base	No intervention	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.67 $\pm$ 0.004	0.84 $\pm$ 0.01	11.11
	Basic RD	0.42 $\pm$ 0.01	0.26 $\pm$ 0.01	0.23 $\pm$ 0.004	0.008 $\pm$ 0.002	113.98	0.29 $\pm$ 0.01	0.22 $\pm$ 0.01	0.23 $\pm$ 0.004	0.00 $\pm$ 0.00	$>10^{7}$
	Heretic	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.27 $\pm$ 0.004	0.19 $\pm$ 0.01	13.44
	RDO	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16	0.63 $\pm$ 0.01	0.37 $\pm$ 0.01	0.67 $\pm$ 0.004	0.82 $\pm$ 0.01	11.16
	WRD-CMAES	0.63 $\pm$ 0.01	0.38 $\pm$ 0.01	0.64 $\pm$ 0.004	0.84 $\pm$ 0.01	11.25	0.51 $\pm$ 0.01	0.30 $\pm$ 0.01	0.47 $\pm$ 0.004	0.04 $\pm$ 0.005	64.84

Appendix

Table 6: Utility preservation per base model at the intermediate intensities

\alpha\in\{1.5,2.0\}

, complementing the main Table 2 . Higher is better for ARC-e, ARC-c, MMLU and GSM8K; lower is better for PPL.

Method	Hyperparameter	Value
Basic RD	kl_threshold	$0.1$
	induce_refusal_threshold	$0.0$
	prune_layer_percentage	$0.2$
Heretic	n_trials / n_startup_trials	$200$ / $60$
	kl_divergence_scale / _target	$1.0$ / $0.01$
	refusal_threshold_count	$0$

Appendix

Table 7: Training hyperparameters for the six methods of Section 3 . Values are read verbatim from the configuration files in the released codebase. Hyperparameters shared across methods (sampling seed

42

, batch size

16

32

, maximum response length, system prompt) are listed in Appendix C .

Method	Approx. runtime, hours
Basic RD	$\sim 2$
Heretic	$\sim 7$
RDO	$\sim 3$
Uniform-avg WRD	$\sim 2$
WRD-CMAES	$\sim 4$
WRD-GRPO-IS	$\sim 11$

Appendix

Table 8: Approximate wall-clock runtime per method on the attack-setup experiments of Section 5.1 , on a single NVIDIA H100 80 GB GPU.

			$\alpha=1.0$			$\alpha=1.5$			$\alpha=2.0$			$\alpha=3.0$
Model	Method	Opt. judge	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3	WG	Q3G	LG3
Qwen3-8B-Base	WRD-CMAES	WG	0.89	0.89	0.92	0.91	0.92	0.95	0.95	0.96	0.96	0.90	0.93	0.92
		Q3G	0.00	0.84	0.97	0.00	0.98	0.96	0.00	1.00	0.97	0.00	1.00	0.97
		LG3	0.00	0.73	0.88	0.02	0.94	0.93	0.16	0.96	0.98	0.01	0.98	0.98
	WRD-GRPO-IS	WG	0.91	0.94	0.90	0.93	0.95	0.92	0.95	0.96	0.94	0.95	0.96	0.93
		Q3G	0.91	0.94	0.90	0.94	0.95	0.92	0.01	0.96	0.98	0.94	0.95	0.93

Appendix

Table 9: Cross-judge transfer. Test ASR (in

[0,1]

) when

w^{\star}

is optimized against the row judge and evaluated against the column judge, reported per base model, for every

\alpha\in\{1.0,1.5,2.0,3.0\}

. Within each

\alpha

block, diagonal cells coincide with the corresponding entries of Table 1 or, for

\alpha\in\{1.5,2.0\}

, of Table 5 .

$\beta$	$1$	$10$	$10^{2}$	$10^{3}$	$10^{4}$
ASR (WG)	0.95	0.92	0.91	0.91	0.78

Appendix

Table 10: Attack side of the

\beta

sweep. Test ASR under WildGuard for WRD-GRPO-IS on Qwen3-8B-Base at

\alpha=1.0

. Higher is a stronger attack.

Method	$-1.0$	$-1.5$	$-2.0$	$-3.0$
Abliterated (no defense)	0.94	0.94	0.94	0.94
Basic RD	0.94	0.92	0.91	0.31
WRD-CMAES	0.47	0.03	0.00	0.03
WRD-GRPO-IS	0.47	0.02	0.00	0.00

Appendix

Table 11: Defense setup evaluated externally. HarmBench Standard ASR computed with cais/HarmBench-Llama-2-13b-cls on the abliterated checkpoint, across the sign-reversed intensity

\alpha

. HarmBench is never used during optimization. Lower is a stronger defense.

Model	Method	ASR
Qwen3-8B-Base	RDO-CONE	0.953
Qwen3-8B-Base	WRD-GRPO-IS	0.93 $\pm$ 0.09
OLMo-3-1025-7B	RDO-CONE	0.865
OLMo-3-1025-7B	WRD-GRPO-IS	0.92 $\pm$ 0.02
Olmo-3-7B-Instruct	RDO-CONE	0.845
Olmo-3-7B-Instruct	WRD-GRPO-IS	0.93 $\pm$ 0.01

Appendix

Table 12: Concept-cone comparison. Test ASR under WildGuard; WRDO rows are taken at

\alpha=3.0

. The WRD-GRPO-IS rows of the upper blocks repeat the corresponding cells of Table 1 and carry its three-seed spread; the RDO-CONE and gpt-oss-20b rows are single runs. Higher is a stronger attack.

$\beta$	ARC-e	ARC-c	MMLU	GSM8K	PPL
$1$	0.52	0.34	0.25	0.38	261.32
$10$	0.51	0.36	0.34	0.43	260.13
$10^{2}$	0.60	0.36	0.59	0.67	150.72
$10^{3}$	0.63	0.38	0.65	0.83	11.15
$10^{4}$	0.63	0.38	0.65	0.84	11.12

Appendix

Table 13: Utility side of the

\beta

sweep, same runs as Table 10 . Higher is better for the four accuracies, lower is better for WikiText perplexity. Standard errors are

0.01

for ARC-e, ARC-c and GSM8K and

0.004

for MMLU. Bold marks the best value per column.

Refusal Behavior in Language Models

Refusal behavior in language models concerns when systems decline requests, how they detect risk, and how to balance safety with helpfulness.

See all 148 papers

Sep 3, 2026

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.

Hoang Cuong Nguyen, Mark Dras, Usman Naseem

Macquarie University, Sydney, Australia

Mechanistic Interpretability Circuits in Language Models LLM Post-Training Language Model Robustness Refusal Behavior in Language Models Refusal Mechanisms in Language Models

Ask PDF

Mar 29, 2026

Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs

Aligned language models that are trained to refuse harmful requests also exhibit over-refusal: they decline safe instructions that seemingly resemble harmful instructions. A natural approach is to ablate the global refusal direction, steering the hidden-state vectors away or towards the harmful-refusal examples, but this corrects over-refusal only incidentally while disrupting the broader refusal mechanism. In this work, we analyse the representational geometry of both refusal types to understand why this happens. We show that harmful-refusal directions are task-agnostic and can be captured by a single global vector, whereas over-refusal directions are task-dependent: they reside within the benign task-representation clusters, vary across tasks, and span a higher-dimensional subspace. Linear probing suggests that the two refusal types are representationally distinct from the early transformer layers. These findings provide a mechanistic explanation of why global direction ablation alone cannot address over-refusal, and establish that task-specific geometric interventions are necessary.

Utsav Maskey, Mark Dras, Usman Naseem

Macquarie University

LLM Interpretability LLM Refusal Behavior LLM Safety Over-Refusal Refusal Behavior in Language Models Refusal Mechanisms in Language Models

Ask PDF

Mar 4, 2026

Efficient Refusal Ablation in LLM through Optimal Transport

Safety-aligned language models refuse harmful requests through learned refusal behaviors encoded in their internal representations. Recent activation-based jailbreaking methods circumvent these safety mechanisms by applying orthogonal projections to remove refusal directions, but these approaches treat refusal as a one-dimensional phenomenon and ignore the rich distributional structure of model activations. We introduce a principled framework based on optimal transport theory that transforms the entire distribution of harmful activations to match harmless ones. By combining PCA with closed-form Gaussian optimal transport, we achieve efficient computation in high-dimensional representation spaces while preserving essential geometric structure. Across six models (Llama-2, Llama-3.1, Qwen-2.5; 7B-32B parameters), our method achieves up to 11% higher attack success rates than state-of-the-art baselines while maintaining comparable perplexity, demonstrating superior preservation of model capabilities. Critically, we discover that layer-selective intervention (applying optimal transport to 1-2 carefully chosen layers at approximately 40-60% network depth) substantially outperforms full-network interventions, revealing that refusal mechanisms may be localized rather than distributed. Our analysis provides new insights into the geometric structure of safety representations and suggests that current alignment methods may be vulnerable to distributional attacks beyond simple direction removal.

Geraldin Nanfack, Eugene Belilovsky, Elvis Dohmatob

Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada

Adversarial Attacks on LLMs Jailbreak Attacks LLM Safety Refusal Behavior in Language Models

Ask PDF

Abstract

Refusal Behavior in Language Models

Explore similar work

Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMs

Efficient Refusal Ablation in LLM through Optimal Transport