D-STEER: Two-Way Control of DPO-Induced Safety Behavior in Activation Space

Samarth Raina; Saksham Aggarwal; Amit Dhanda; Vinija Jain; Aman Chadha; Amitava Das

cs.LGDec 3, 2025

D-STEER: Two-Way Control of DPO-Induced Safety Behavior in Activation Space

Authors: Samarth Raina, Saksham Aggarwal, Amit Dhanda, Vinija Jain, Aman Chadha, Amitava Das

Organizations: IIIT Delhi · Microsoft · Apple (USA) · Google (USA) ·

Abstract

Preference-based post-training shapes language-model behavior, but safer outputs do not reveal how that change is represented internally or how firmly it persists under direct intervention. This matters for open-weight models, where an actor with white-box access can manipulate internal representations at inference without touching parameters. Using Direct Preference Optimization (DPO), we ask whether the safety behavior it introduces becomes deeply embedded or stays accessible to targeted activation-space interventions. We introduce D-STEER, an audit of two matched instruction-tuned and harmlessness-DPO pairs: OpenHermes-2.5-Mistral-7B and Llama-3-8B OpenHermes. D-STEER derives layer-local directions from differences in matched prompt activations, adds them to the instruction-tuned model to install the DPO-associated safety change and subtracts them from the DPO model to remove it, with separate operating windows distinguishing behavioral control from degradation. On held-out HarmfulQA, all four interventions shift harmful enablement as intended and exceed norm-matched random controls; Mistral installation recovers 98% of the endpoint gap and the other three arms recover 46-50%. The intended direction transfers to HH-RLHF and AdvBench under frozen coefficients, although usable operating windows do not transfer universally. An independently rebuilt implementation reproduces the bidirectional pattern on the Mistral pair under different construction and intervention choices. A meaningful component of DPO-induced safety behavior therefore remains controllable through compact inference-time intervention, with the extent depending on the checkpoint pair and the steering implementation.

Figures & tables

Figure 1: Conceptual activation steering: addition to IT reduces harmful assistance; subtraction from DPO increases it, relative to each unsteered checkpoint.

Figure 2: Dose response on the frozen 200-prompt calibration panel. The intended effect rises with steering magnitude in both directions and for both pairs; stars mark the coefficient selected inside each usability window. Held-out effects at those coefficients are in Table 1 .

Pair	Dir.	$C$	$\lambda^{\star}$	$n$	$\Delta H$ [95% CI]	$L_{Q}$
Mistral	Inst.	$.135$	$+.135$	292	$+.289\ [.246,.332]$	$.037$
Mistral	Rem.	$.055$	$-.055$	300	$+.136\ [.102,.172]$	$.066$
Llama	Inst.	$.050$	$+.050$	286	$+.084\ [.050,.117]$	$.022$
Llama	Rem.	$.050$	$-.050$	296	$+.082\ [.048,.117]$	$.047$

Table 1: Calibration-selected operating ceilings

C

and coefficients

\lambda^{\star}

, and their held-out effects on the 300-prompt HarmfulQA panel. Generation criteria bind for three conditions and quality for Mistral removal. Harmfulness movement

\Delta H

is oriented so positive is the intended direction;

L_{Q}

is quality loss against the corresponding unsteered endpoint. All four conditions satisfy every criterion of Equation 7 . Strict source-ID pairing, 10,000 paired prompt bootstraps. Leading zeros are omitted. Refusal movements, integrity rates and endpoint means are in the supplement.

Figure 3: Held-out evidence on the 300-prompt HarmfulQA panel, Mistral pair first. (A) Intended harmfulness movement for the four frozen interventions. (B) The same effects against five norm-matched random directions at the same layers, prompts and coefficient: grey diamonds are seed effects with a tick at their mean, the filled marker the real effect, the hollow marker the specificity contrast. Panel B requires every random arm to be valid on a prompt, so its

n

is stated separately. Intervals are 95% paired prompt bootstraps over 10,000 replicates.

Panel	Dir.	$\lambda_{\mathrm{raw}}$	$n$	$\Delta H$ [95% CI]	$H_{1}$
HarmfulQA	Inst.	$+0.7$	300	$+.364\ [.317,.411]$	$.080$
HarmfulQA	Rem.	$-0.5$	299	$+.148\ [.112,.184]$	$.300$
HH-RLHF	Inst.	$+0.7$	277	$+.097\ [.058,.139]$	$.215$
HH-RLHF	Rem.	$-0.5$	274	$+.091\ [.056,.129]$	$.166$
AdvBench	Inst.	$+0.7$	300	$+.485\ [.430,.541]$	$.219$
AdvBench	Rem.	$-0.5$	287	$+.198\ [.145,.250]$	$.611$

Table 2: Rebuilt raw run, Mistral pair, last seven layers, HH-RLHF direction; raw scale, not comparable to Table 1 .

H_{1}

is steered harmful enablement; leading zeros omitted. All six intervals exclude zero; the six corresponding HarmfulQA-derived-direction cells do as well and are reported with the depth and actuation-breadth sweeps in the supplement.

Figure 4: Dose response under the independently rebuilt implementation, on its own 300-prompt HarmfulQA panel at the last seven layers. Stars mark the coefficients reported in Table 2 ; bands are 95% paired bootstrap intervals.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Kind	$N$	Inv.	MaxTok	Scored	$H$	$R$	$Q$
M0-A_baseline	baseline	300	0.020	0.030	294	0.390	0.247	0.926
M0-A_installation	real	300	0.027	0.030	292	0.304	0.347	0.904
M+-A_baseline	baseline	300	0.000	0.000	300	0.217	0.480	0.957
M+-A_reversal	real	300	0.013	0.040	296	0.302	0.373	0.910
M0-B_baseline	baseline	300	0.000	0.000	300	0.486	0.188	0.974
M0-B_installation	real	300	0.027	0.027	292	0.199	0.471	0.938

Appendix

Table S3: Held-out HarmfulQA panel: all non-random conditions with generation-integrity rates and unsteered or steered metric means. Rates use all 300 requested prompts; metric means use scored responses. Random-control conditions are summarised in Table S2 .

Figure S3: Sensitivity of the usable ceiling

C

and selected coefficient

|\lambda^{*}|

to 2.5%, 5%, and 10% generation-integrity thresholds. The direction-specific pattern is stable: Mistral installation supports the widest window, Mistral removal remains quality-bound, and only Llama installation at 10% selects below the available ceiling.

Figure S4: Direction-oriented harmfulness effects across the sealed HarmfulQA panel and two transfer panels, using coefficients selected only on the disjoint HarmfulQA calibration split. Positive values denote movement in the intended direction; error bars are 95% paired prompt-bootstrap intervals. Outlined transfer estimates involve a steered or baseline condition that exceeds the frozen 5% generation-integrity gate, separating directional transfer from deployable-window transfer.

Panel	Arm	$\lambda$	$N$	Inv.	MaxTok	Gate
hh-rlhf	M+-A_baseline	$-0.000$	200	0.155	0.160	fail (inv,max)
hh-rlhf	M+-A_reversal	$-0.050$	200	0.230	0.240	fail (inv,max)
hh-rlhf	M+-B_baseline	$-0.000$	200	0.110	0.125	fail (inv,max)
hh-rlhf	M+-B_reversal	$-0.055$	200	0.025	0.005	pass
hh-rlhf	M–A_baseline	$+0.000$	200	0.405	0.420	fail (inv,max)
hh-rlhf	M0-A_baseline	$+0.000$	200	0.295	0.310	fail (inv,max)

Appendix

Table S6: Direction transfer versus operating-window transfer. All eight behavioral contrasts move in the intended direction (Table S5 ), while fifteen of eighteen constituent conditions exceed at least one HarmfulQA-calibrated

5\%

generation threshold. Of the eight steered arms, seven exceed a threshold directly; HH-RLHF Mistral removal passes but is paired with a baseline that does not. The direction therefore generalizes more reliably than the coefficient’s usability frontier.

Figure S6: Independent raw-displacement stress test on the Mistral pair. Left: all twelve construction-corpus, evaluation-panel, and direction combinations move harmfulness in the intended direction. Middle: intended movement grows as a contiguous late-layer block is widened, while total injected activation grows with it. Right: expanding actuation from the final prompt token to generated tokens increases effect and cumulative dose but also increases invalidity and quality cost. Together these panels expose both the robustness of the directional result and the implementation choices that set its usable strength.

Panel	Dir.	Direction source	$\lambda_{\mathrm{raw}}$	$n$	Valid	$H_{0}$	$H_{1}$	$\Delta H$ [95% CI]
HarmfulQA	Inst.	HH-RLHF	$+0.7$	300	300/300	$0.444$	$0.080$	$+0.364$ $[0.317,0.411]$
HarmfulQA	Inst.	HarmfulQA	$+0.7$	299	299/300	$0.445$	$0.154$	$+0.292$ $[0.243,0.340]$
HarmfulQA	Rem.	HH-RLHF	$-0.5$	299	300/300	$0.152$	$0.300$	$+0.148$ $[0.112,0.184]$
HarmfulQA	Rem.	HarmfulQA	$-0.5$	298	299/300	$0.153$	$0.320$	$+0.167$ $[0.128,0.209]$
HH-RLHF	Inst.	HH-RLHF	$+0.7$	277	283/300	$0.312$	$0.215$	$+0.097$ $[0.058,0.139]$
HH-RLHF	Inst.	HarmfulQA	$+0.7$	278	284/300	$0.312$	$0.225$	$+0.087$ $[0.046,0.129]$

Appendix

Table S7: Independent raw-displacement stress test on the Mistral pair, using the last seven layers and final-prompt-token actuation. Both construction corpora succeed on all three panels in both directions: all twelve intervals exclude zero.

H_{0}

is the paired unsteered endpoint mean and

H_{1}

the steered mean; estimates use strict source-ID pairing and 10,000 bootstrap replicates. Re-scoring byte-identical retained baselines changes affected effects by at most

0.005

and changes no sign, ordering, or interval conclusion.

Pair	Panel	Dir.	$n$	Real	Random mean	Specificity [95% CI]	Excl. 0	Inaction
Mistral	AdvBench	Inst.	300	$+0.2993$	$+0.0025$	$+0.2969$ $[+0.2476,+0.3465]$	yes	$0.070$
Mistral	AdvBench	Rem.	281	$+0.1989$	$-0.0078$	$+0.2067$ $[+0.1586,+0.2552]$	yes	$0.093$
Mistral	HarmfulQA	Inst.	300	$+0.2370$	$+0.0113$	$+0.2257$ $[+0.1839,+0.2683]$	yes	$0.133$
Mistral	HarmfulQA	Rem.	190	$+0.1216$	$-0.0052$	$+0.1267$ $[+0.0806,+0.1748]$	yes	$0.116$
Mistral	HH-RLHF	Inst.	283	$+0.0650$	$+0.0002$	$+0.0648$ $[+0.0316,+0.1002]$	yes	$0.314$
Mistral	HH-RLHF	Rem.	266	$+0.0763$	$-0.0053$	$+0.0817$ $[+0.0512,+0.1135]$	yes	$0.237$

Appendix

Table S8: Raw-scale specificity under strict joint pairing across the real arm and five matched random seeds. “Inaction” is the fraction of steered responses byte-identical to baseline. The Mistral pair separates from random in all six panel-by-direction combinations. The rebuilt Llama estimates are less stable (two of six intervals exclude zero, with some joint intersections as small as 28); the independently calibrated primary protocol establishes both Llama directions on its sealed panel. Controls here use five layers rather than the stress test’s primary seven, so this table supports directional specificity but not a last-seven random-control claim.

Actuation scope	Inst. $\Delta H$	Valid	$\|\Delta Q\|$	Rem. $\Delta H$	Valid	$\|\Delta Q\|$
Final prompt token	$+0.237$	300/300	$0.038$	$+0.151$	300/300	$0.091$
All prompt tokens	$+0.271$	300/300	$0.044$	$+0.158$	300/300	$0.054$
Prompt $+$ generated	$+0.219$	195/300	$0.150$	$+0.236$	298/300	$0.338$

Appendix

Table S10: Token-scope sweep at five layers and fixed raw coefficient, Mistral pair on HarmfulQA. All-prompt actuation modestly increases installation with full validity. Repeated actuation through generated tokens greatly increases cumulative injected dose and quality cost; installation then falls below the

\geq\!90\%

validity floor (195 of 300). The primary protocol independently calibrates its coefficients for that broader scope, so this result motivates scope-specific operating windows rather than a universal token policy. Re-scoring byte-identical baselines moves affected effects by at most

0.005

without changing any conclusion.

Figure S8: Reversed-preference training and activation-based safety removal agree in direction across four panels, but their magnitude ratio is panel-dependent. Error bars are 95% joint prompt-bootstrap intervals. The HH-RLHF ratio is withheld because the reverse-training interval includes zero. This is evidence of access to a shared behavioral axis, not statistical equivalence between an inference-time perturbation and a weight update.

Panel	Rev. training	Steering rem.	Ratio	$n$
HarmfulQA (calib.)	$+.087$	$+.047$	$0.54$	$191$
HarmfulQA (held-out)	$+.033$	$+.079$	$2.40$	$286$
HH-RLHF	$+.010$	$+.060$	—	$94$
AdvBench	$+.062$	$+.178$	$2.86$	$151$

Appendix

Table S13: Directional agreement between reversed-preference training and steering removal on the Llama pair. Both increase harmful enablement on all four panels, supporting access to preference-relevant behavior; their varying magnitude shows that steering is a comparator, not an equivalence reconstruction. Ratios are withheld when the reversed-training interval includes zero. The calibration row is non-confirmatory Task 026C evidence, traceable to the hash-verified included artifact data/primary/task026c-reverse-dpo/phase1_flip_vs_steer.json : reversed training gives

+.087

[+.049,+.129]

and steering removal

+.047

[+.003,+.091]

n{=}191

, with 10,000 joint prompt-bootstrap replicates.

Causal Interventions in Language Models

Causal interventions in language models test and change how internal representations and computational components cause outputs.

See all 482 papers

Jul 22, 2026

OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimization

Activation steering provides a lightweight mechanism for controlling large language models at inference time, but steering vectors can have unintended externalities: utility vectors may weaken safety behavior, while refusal vectors may induce over-refusal on benign prompts. We introduce OPIUM (Optimizing Protected Injections via Utility Manifolds), a training-free method for sanitizing steering vectors through representation matching. Given reference behaviors on two prompt sets, OPIUM optimizes a new steering vector that preserves the downstream representations induced by the desired intervention while matching a safer reference behavior on prompts where the original vector fails. Across steering-externality and over-refusal settings, OPIUM improves the safety--utility tradeoff relative to vanilla steering and directional ablation, suggesting that harmful side effects of activation steering can often be mitigated directly in activation space.

Kavin Aravindan, Arihant Rastogi, Krishak Aneja +4

IIIT Hyderabad, India.

LLM Safety Alignment Over-Refusal Activation Steering

Ask PDF

Mar 7, 2026

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment

Direct Preference Optimization (DPO) has become a standard framework for safety alignment, but its reliance on pairwise preference updates makes training sensitive to imperfect supervision. Existing robust DPO methods often address this sensitivity through global loss corrections or external data-level interventions, while largely overlooking how unreliable comparisons distort batch-level optimization dynamics. We propose CompassDPO, a reward-free DPO framework that stabilizes preference optimization through dynamics control. Using the implicit DPO reward margin as a training-time compass, CompassDPO regulates sample influence along two complementary axes: update direction and update magnitude. For directional control, it applies sparse, budgeted, and warm-up delayed loss mixing to attenuate update components that conflict with the emerging preference direction. For magnitude control, it adaptively soft-winsorizes high-loss tail contributions, reducing tail dominance while preserving useful gradients from hard examples. Both mechanisms use only signals available during standard DPO training and require no external reward model or additional supervision. Experiments on PKU-SafeRLHF across four backbones and multiple out-of-distribution safety benchmarks show that CompassDPO consistently improves robustness over vanilla DPO and strong DPO-family baselines, especially under controlled label-flip noise. Code is available at https://anonymous.4open.science/r/CompassDPO-4D00

Jilong Liu, Yonghui Yang, Pengyang Shao +5

Hefei University of Technology · National University of Singapore

Direct Preference Optimization LLM Safety Alignment Preference-Based RL

Ask PDF

May 25, 2026

Curriculum Learning for Safety Alignment

Direct Preference Optimisation (DPO) is widely used for safety alignment in large language models. However, prior work shows it is brittle and exhibits poor out-of-distribution (OOD) generalisation. In this paper, we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose Staged-Competence, a curriculum-based framework that organises preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16% and jailbreak attack success rates by 20%, while preserving general capabilities with near-zero over-refusal. We further show that Staged-Competence (1) matches baseline safety with only 75% of the training data and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the policy optimisation loss and can extend to other DPO variants and alignment domains. Our code and data are available at https://github.com/Sandeep5500/curriculum-learning-for-safety.

Sandeep Kumar, Virginia Smith, Chhavi Yadav

Carnegie Mellon University · Simons Institute, UC Berkeley

LLM Alignment Direct Preference Optimization LLM Safety Alignment Adversarial Robustness of Language Models

Ask PDF

Abstract

Causal Interventions in Language Models

Explore similar work

OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimization

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment

Curriculum Learning for Safety Alignment