Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

Yipeng Zhao; Qishun Yang; Shenzhe Zhu; Shu Yang; Di Wang

cs.AIAug 24, 2026

Mitigating Reasoning-Induced Misalignment via Safety-Direction Penalty

Authors: Yipeng Zhao, Qishun Yang, Shenzhe Zhu, Shu Yang, Di Wang

Organizations: University of Toronto · King Abdullah University of Science and Technology · University of Texas at Austin

Abstract

Reasoning-Induced Misalignment, where fine-tuning on reasoning data containing no harmful content, including mathematics, code, and problem-solving with chain-of-thought traces can induce harmful behaviors of LLM, posing a serious challenge to the safety of LLM reasoning. Cross-architecture, cross-scale, and cross-dataset checks show that RIM does not always emerge. Previous work attributed RIM to neuron-level entanglement, but did not identify the geometry of the representation space underlying this entanglement or propose a training-time fix. We provide both: a representation-space analysis of RIM and the Safety-Direction Penalty (SDP), which penalizes movement along a learned safety direction during reasoning fine-tuning. The analysis extracts two activation-space directions, one encoding reasoning ability and the other safety behavior. These directions are coupled: fine-tuning that improves reasoning shifts safety representations, and prompts with larger shifts show larger safety degradation. CKA distance ratios and probes locate the safety-decision layers where this shift is most relevant. These findings guide the design of SDP: the coupling motivates penalizing displacement along the safety direction, and the layer localization sets the initial scope. When the initial scope leaves compensatory shifts beyond the penalized layers, the same diagnostics guide iterative expansion. On Qwen2.5-3B and 7B, SDP restores safety while preserving benchmark reasoning performance.

Figures & tables

Figure 1: Conceptual overview of Reasoning-Induced Misalignment (RIM) and its mitigation. Harmless reasoning supervised fine-tuning (SFT) can weaken safety, safety representations drift away from the base state, and Safety-Direction Penalty (SDP) constrains this shift during training.

	3B		7B
	Base	AM-SFT	Base	AM-SFT
HEx-PHI% $\downarrow$	10.00	20.33	13.33	25.33
SafetyBench $\uparrow$	69.11	57.93	79.50	76.45
GPQA $\uparrow$	29.69	28.57	33.26	35.71
AIME 2024 $\uparrow$	4.2 $\pm$ 2.4	5.8 $\pm$ 3.5	10.4 $\pm$ 3.3	10.8 $\pm$ 5.6
AIME 2025 $\uparrow$	0.4 $\pm$ 1.2	4.2 $\pm$ 3.5	10.4 $\pm$ 6.8	7.5 $\pm$ 3.5

Table 1: Safety degrades after reasoning fine-tuning (AM-SFT) while benchmark reasoning performance is largely preserved.

Figure 2: Probe accuracy across layers for 3B (top) and 7B (bottom). The harm-perception probe remains stable after AM-SFT. The safety-decision probe collapses after AM-SFT.

Figure 3: Per-layer R–S diagnostics within

M_{1}

. Top row: whitened cosine similarity

\cos_{w}

, all negative, consistent with coupling. Bottom row: displacement

d_{S}

along

\hat{S}

after AM-SFT. 7B displacement is uniformly negative; 3B displacement is sign-mixed.

	3B			7B
	Base	AM-SFT	SDP	Base	AM-SFT	SDP
HEx-PHI% $\downarrow$	10.0	20.3	10.0	13.3	25.3	11.3
SafetyBench $\uparrow$	69.1	57.9	69.6	79.5	76.5	79.4
GPQA $\uparrow$	29.7	28.6	25.9	33.3	35.7	31.3
AIME 2024 $\uparrow$	4.2 $\pm$ 2.4	5.8 $\pm$ 3.5	3.8 $\pm$ 3.1	10.4 $\pm$ 3.3	10.8 $\pm$ 5.6	15.0 $\pm$ 2.9
AIME 2025 $\uparrow$	0.4 $\pm$ 1.2	4.2 $\pm$ 3.5	2.5 $\pm$ 1.4	10.4 $\pm$ 6.8	7.5 $\pm$ 3.5	9.6 $\pm$ 5.1

Table 2: SDP results at both scales. 3B uses

M=\{\text{L19},\ldots,\text{L32}\}

; 7B uses

M=\{\text{L15},\ldots,\text{L19}\}

. SDP restores base-level safety while retaining most evaluated reasoning performance. Intermediate configurations appear in Appendix D.2 .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen2.5-Instruct
Parameters	3B	7B	14B
Hidden dim. ( $d$ )	2,048	3,584	5,120
FFN dim.	11,008	18,944	13,824
Layers	36	28	48
Attention heads	16	28	40
KV heads	2	4	8

Appendix

Table 3: Architecture details for the Qwen2.5-Instruct models. All three are instruction-tuned and safety-aligned during post-training.

Model and adapter
Hyperparameter	3B	7B
Base model	Qwen2.5-3B-Instruct	Qwen2.5-7B-Instruct
LoRA $r/\alpha$	16/16	16/16
LoRA target	All linear layers: q/k/v/o/gate/up/down_proj
LoRA dropout	0.0	0.0
$\gamma_{s}$ (SDP)	0.5	0.5

Appendix

Table 4: Training configuration for 3B and 7B. AM-SFT uses the same setup with

\gamma_{s}=0

and no penalty scope.

	Run 1	Run 2
Training examples	24,000	10,000
Quantization	4-bit	none (bf16)
Learning rate	2e-4	5e-5
Epochs	3	5
Effective batch	2	8
Optimizer	AdamW 8-bit	AdamW

Appendix

Table 5: 14B AM-SFT training configurations.

Model	Training corpus	Train $N$	Epochs	Max length	LR	Batch $\times$ accum.
Qwen2.5-3B-Instruct	MetaMathQA	10,000	3	16,384	$5\mathrm{e}{-5}$	$1\times 8$
Gemma 3 4B IT	AM-DeepSeek	24,337	3	16,384	$5\mathrm{e}{-5}$	$1\times 8$
Ministral 3 3B Instruct	AM-DeepSeek	24,337	3	16,384	$5\mathrm{e}{-5}$	$1\times 8$

Appendix

Table 6: Training settings for the cross-dataset and cross-architecture checks. All runs use seed 3407, AdamW with a cosine schedule, and LoRA rank/alpha 16/16 with zero dropout. Qwen and Ministral target q/k/v/o/gate/up/down_proj ; Gemma restricts the corresponding targets to text-decoder modules and excludes the vision tower. The 24,337-example setting is the executed model-specific cross-architecture setting, not an architecture-only match to the Qwen 10k experiment.

Setting	HEx-PHI% $\downarrow$	SafetyBench $\uparrow$	GPQA $\uparrow$	AIME 2024 $\uparrow$	AIME 2025 $\uparrow$
Qwen2.5-3B MetaMathQA-SFT	12.67	66.59	24.78	$0.42\pm 1.18$	$0.00\pm 0.00$
Gemma 3 4B IT Base	12.67	71.05	28.79	$11.67\pm 1.78$	$10.00\pm 1.78$
Gemma 3 4B IT AM-SFT	5.00	63.06	28.79	$6.25\pm 3.30$	$5.83\pm 3.88$
Ministral 3 3B Instruct Base	34.00	68.47	41.52	$26.25\pm 4.15$	$17.50\pm 1.54$
Ministral 3 3B Instruct AM-SFT	30.67	35.84	18.97	$2.92\pm 1.18$	$0.00\pm 0.00$
Qwen2.5-14B Base	7.00	82.94	38.39	13.33	20.00

Appendix

Table 7: Cross-dataset, cross-architecture, and scale-check induction results. The MetaMathQA HEx-PHI entry is the validated 300-sample rejudge, and the Gemma AM-SFT GPQA entry is recovered from the completed evaluation. AIME reports mean

\pm

sample standard deviation over eight runs.

Layer	$\cos_{w}$	Layer	$\cos_{w}$
L0	$+0.471$	L18	$-0.127$
L1	$+0.323$	L19	$-0.128$
L2	$+0.238$	L20	$-0.092$
L3	$+0.166$	L21	$-0.085$
L4	$+0.138$	L22	$-0.129$
L5	$+0.079$	L23	$-0.159$

Appendix

Table 8: 3B per-layer

\cos_{w}(\hat{R},\hat{S})

. Negative from L10 onward, peaking in

\{\text{L27},\ldots,\text{L32}\}

(mean

-0.20

Layer	$\cos_{w}$	Layer	$\cos_{w}$
L0	$+0.264$	L14	$-0.080$
L1	$+0.198$	L15	$-0.072$
L2	$+0.120$	L16	$-0.083$
L3	$+0.092$	L17	$-0.103$
L4	$-0.015$	L18	$-0.173$
L5	$-0.019$	L19	$-0.205$

Appendix

Table 9: 7B per-layer

\cos_{w}(\hat{R},\hat{S})

. Negative from L4 onward, increasing in magnitude through the deep layers.

Layer	CKA h	CKA a	Drop h	$r_{\mathrm{CKA}}$	Layer	CKA h	CKA a	Drop h	$r_{\mathrm{CKA}}$
L0	0.999	0.999	0.001	0.76	L18	0.942	0.956	0.058	1.31
L1	0.999	0.999	0.001	0.86	L19	0.938	0.958	0.063	1.49
L2	0.989	0.995	0.011	2.17	L20	0.920	0.957	0.080	1.88
L3	0.979	0.989	0.021	1.85	L21	0.908	0.952	0.092	1.93
L4	0.981	0.991	0.019	2.03	L22	0.872	0.932	0.128	1.89
L5	0.974	0.988	0.026	2.09	L23	0.819	0.920	0.181	2.27

Appendix

Table 10: 3B per-layer CKA profile (base

\to

AM-SFT). CKA h = CKA on HEx-PHI; CKA a = CKA on Alpaca; Drop h =

1-\text{CKA}_{h}

;

r_{\mathrm{CKA}}

is the CKA distance ratio. Bold rows = initial

M_{1}

. Horizontal rules separate noise (

\{\text{L0},\ldots,\text{L11}\}

), valley (

\{\text{L12},\ldots,\text{L16}\}

), transition (

\{\text{L17},\ldots,\text{L19}\}

M_{1}=\{\text{L20},\ldots,\text{L26}\}

, and tail (

\{\text{L27},\ldots,\text{L35}\}

Layer	CKA h	CKA a	Drop h	$r_{\mathrm{CKA}}$	Layer	CKA h	CKA a	Drop h	$r_{\mathrm{CKA}}$
L0	0.999	0.999	0.001	1.20	L14	0.870	0.887	0.130	1.15
L1	0.989	0.987	0.011	0.82	L15	0.846	0.883	0.154	1.32
L2	0.975	0.973	0.025	0.91	L16	0.818	0.864	0.182	1.34
L3	0.961	0.963	0.039	1.06	L17	0.780	0.853	0.220	1.50
L4	0.963	0.956	0.037	0.82	L18	0.763	0.840	0.237	1.48
L5	0.926	0.944	0.074	1.33	L19	0.767	0.814	0.233	1.25

Appendix

Table 11: 7B per-layer CKA profile (base

\to

AM-SFT). Bold rows =

M_{1}

. Above

M_{1}

r_{\mathrm{CKA}}

is unstable with small absolute drops. Below

M_{1}

, the ratio drops sharply below 1, indicating capability-dominant drift.

	HEx-PHI% $\downarrow$	SafetyBench $\uparrow$	GPQA $\uparrow$	AIME 24/25 $\uparrow$
3B Base	10.00	69.11	29.69	4.2 $\pm$ 2.4 / 0.4 $\pm$ 1.2
3B AM-SFT	20.33	57.93	28.57	5.8 $\pm$ 3.5 / 4.2 $\pm$ 3.5
3B SDP( $\{\text{L20},\ldots,\text{L26}\}$ )	18.67	62.80	26.79	7.1 $\pm$ 3.5 / 0.4 $\pm$ 1.1
7B Base	13.33	79.50	33.26	10.4 $\pm$ 3.3 / 10.4 $\pm$ 6.8
7B AM-SFT	25.33	76.45	35.71	10.8 $\pm$ 5.6 / 7.5 $\pm$ 3.5
7B SDP( $\{\text{L15},\ldots,\text{L19}\}$ )	11.33	79.44	31.25	15.0 $\pm$ 2.9 / 9.6 $\pm$ 5.1

Appendix

Table 12: Initial SDP on

M_{1}

. 7B restores base-level safety; 3B does not. AIME reports mean

\pm

std over 8 runs.

	$M_{1}$ $d_{S}$	Post- $M_{1}$ $d_{S}$	HEx-PHI%	SafetyBench
Base	0.000	0.000	10.0	69.1
AM-SFT	$-$ 0.013	$-$ 2.805	20.3	57.9
SDP( $\{\text{L20},\ldots,\text{L26}\}$ )	$+$ 0.088	$-$ 3.006	18.7	62.8
SDP( $\{\text{L20},\text{L21}\}$ )	$-$ 0.937	$-$ 5.374	14.3	46.0
SDP( $\{\text{L19},\ldots,\text{L32}\}$ )	$+$ 0.038	$+$ 0.159	10.0	69.6

Appendix

Table 13: 3B compensation diagnostic.

M_{1}=\{\text{L20},\ldots,\text{L26}\}

; Post-

M_{1}

\{\text{L27},\ldots,\text{L35}\}

. SDP(

\{\text{L19},\ldots,\text{L32}\}

) is the final expanded configuration.

Figure 4: Per-layer

d_{S}

displacement profile (3B, all 36 layers). AM-SFT shows sign-mixed displacement concentrated in the deep layers. SDP(

\{\text{L20},\ldots,\text{L26}\}

) and SDP(

\{\text{L20},\text{L21}\}

) suppress

M_{1}

but leave or amplify post-

M_{1}

displacement. SDP(

\{\text{L19},\ldots,\text{L32}\}

) suppresses the displacement-active zone.

\{\text{L34},\text{L35}\}

show large-magnitude displacement driven by activation scale in the final layers.

	SDP( $\{\text{L15},\ldots,\text{L19}\}$ )	SDP( $\{\text{L15},\ldots,\text{L23}\}$ )
HEx-PHI% $\downarrow$	11.33	19.33
SafetyBench $\uparrow$	79.44	79.13
GPQA $\uparrow$	31.25	29.91

Appendix

Table 14: 7B scope ablation: expanding the penalty scope beyond

r_{\mathrm{CKA}}>1

worsens safety.

Region	AM-SFT	SDP( $\{\text{L15},\ldots,\text{L19}\}$ )	SDP( $\{\text{L15},\ldots,\text{L23}\}$ )
Initial $M_{1}$ ( $\{\text{L15},\ldots,\text{L19}\}$ )	$-7.08$	$+0.05$	$+0.003$
Added scope ( $\{\text{L20},\ldots,\text{L23}\}$ )	$-15.06$	$+0.56$	$+0.03$
Post-scope ( L24 )	$-19.30$	$-2.66$	$-5.90$
Scale ( $\{\text{L25},\ldots,\text{L27}\}$ )	$-103.97$	$-104.35$	$-47.26$

Appendix

Table 15: 7B displacement by region. Expanding the penalty scope to include CKA

<1

layers intensifies displacement in the first unpenalized layer (L24:

-2.66\to-5.90

), consistent with the compensation pattern in 3B.

Layer	AM-SFT	SDP( $\{\text{L20},\text{L21}\}$ )	SDP( $\{\text{L20},\ldots,\text{L26}\}$ )	SDP( $\{\text{L19},\ldots,\text{L32}\}$ )
L0	$-0.005$	$+0.003$	$-0.001$	$-0.002$
L1	$-0.116$	$+0.000$	$-0.002$	$-0.004$
L2	$-0.027$	$-0.027$	$-0.013$	$+0.002$
L3	$-0.183$	$-0.026$	$-0.015$	$-0.005$
L4	$-0.321$	$-0.015$	$-0.013$	$-0.009$
L5	$-0.248$	$-0.001$	$-0.025$	$-0.004$

Appendix

Table 16: 3B per-layer

d_{S}

across all configurations (mean over 300 HEx-PHI prompts, position (a)). Negative = comply direction; positive = refuse direction.

	Base	AM-SFT	$\gamma_{s}$ =0.01	$\gamma_{s}$ =0.05	$\gamma_{s}$ =0.1	$\gamma_{s}$ =0.5	$\gamma_{s}$ =1.0	$\gamma_{s}$ =5.0
HEx-PHI% $\downarrow$	10.0	20.3	7.7	10.3	18.0	10.0	9.3	13.3
SafetyBench $\uparrow$	69.1	57.9	64.9	67.4	67.7	69.6	66.5	66.6
GPQA $\uparrow$	29.7	28.6	27.5	26.8	23.4	25.9	23.7	27.7
AIME 24 $\uparrow$	4.2 $\pm$ 2.4	5.8 $\pm$ 3.5	6.7 $\pm$ 4.4	6.7 $\pm$ 2.5	1.2 $\pm$ 1.7	3.8 $\pm$ 3.1	3.3 $\pm$ 3.1	4.2 $\pm$ 3.5
AIME 25 $\uparrow$	0.4 $\pm$ 1.2	4.2 $\pm$ 3.5	4.6 $\pm$ 4.0	3.8 $\pm$ 2.8	3.8 $\pm$ 3.3	2.5 $\pm$ 1.4	1.2 $\pm$ 1.7	2.1 $\pm$ 2.5

Appendix

Table 17: Penalty weight

\gamma_{s}

ablation on 3B with

M=\{\text{L19},\ldots,\text{L32}\}

\gamma_{s}=0.5

achieves the best balance between safety recovery and reasoning preservation. Bold = best safety metric.

14B Run	AIME	GPQA	GSM8K	HEx-PHI
Base	0/60	0/448	0/200	0/300
AM-SFT (ckpt5000)	0/60	0/448	0/200	1/300
AM-SFT (final)	0/60	0/448	0/200	1/300

Appendix

Table 18: 14B <think> adoption rate. The model almost never enters extended thinking, even on reasoning benchmarks. The same AM-DeepSeek data induces <think> in 46% (3B) and 60% (7B) of HEx-PHI responses.

	<think> rate	think $\to$ harm%
3B AM-SFT	46%	42%
3B SDP	100%	10%
7B AM-SFT	60%	39%
7B SDP	18%	42%

Appendix

Table 19: SDP restores safety through opposite behavioral channels. 3B uses

M=\{\text{L19},\ldots,\text{L32}\}

; 7B uses

M=\{\text{L15},\ldots,\text{L19}\}

. 3B: extended-thinking behavior is preserved, conditional harm rate drops. 7B: extended-thinking behavior is suppressed, conditional harm rate unchanged.

Refuse (total)
Category	$\{\text{L19},\ldots,\text{L32}\}$	$\{\text{L20},\ldots,\text{L26}\}$	$\{\text{L20},\text{L21}\}$
short	1	15	74
think-then-refuse	88	134	27
cyclic	125	58	21
cyclic+harmful	56	23	12
other	0	14	123

Appendix

Table 20: Failure mode taxonomy across configurations (3B, 300 prompts each).

Run	GPU-hours	Notes
3B AM-SFT	$\sim$ 1	3 epochs
3B SDP( $\{\text{L20},\ldots,\text{L26}\}$ )	$\sim$ 4	3 epochs
3B SDP( $\{\text{L20},\text{L21}\}$ )	$\sim$ 4	3 epochs
3B SDP( $\{\text{L19},\ldots,\text{L32}\}$ )	$\sim$ 4	3 epochs
3B $\gamma_{s}$ ablation ( $\times$ 5)	$\sim$ 20	5 values
7B AM-SFT	$\sim$ 15	4 epochs

Appendix

Table 21: Approximate GPU hours on a single NVIDIA RTX PRO 6000 Blackwell Edition (96 GB).

Artifact	License
Qwen2.5-7B/14B-Instruct	Apache 2.0
Qwen2.5-3B-Instruct	Qwen-Research
MetaMathQA	MIT
Gemma 3 4B IT	Gemma Terms of Use (gated)
Ministral 3 3B Instruct (BF16)	Apache 2.0
AM-DeepSeek	CC-BY-NC-4.0

Appendix

Table 22: Artifact licenses.

Safety-Preserving Fine-Tuning of Language Models

See all 27 papers

Apr 21, 2026

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.

Yeonjun In, Wonjoong Kim, Sangwu Park +1

KAIST

LLM Safety Alignment Large Reasoning Models

Ask PDF

Sep 14, 2026

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

Large Reasoning Models (LRMs) pose a dual-surface safety challenge: both intermediate reasoning traces and final answers can contain harmful content. Existing alignment methods often operate at the whole-response level, allowing unsafe reasoning to be masked by a safe-looking final answer. We propose Segment-aware Listwise Target DPO (SaLT-DPO), which addresses this gap through three mechanisms: (1) segment-aware listwise alignment that decomposes responses into reasoning and answer segments, independently scores each segment's safety, and aligns length-normalized segment rewards with soft target distributions over multiple candidates; (2) joint safety coherence regularization that applies a weakest-link principle to promote safety consistency across both segments; and (3) utility anchoring on benign prompts to mitigate over-refusal and reasoning degradation. Experiments on three LRMs show that SaLT-DPO consistently reduces unsafe rates for both reasoning and answer segments while mitigating degradation in benign compliance and preserving general reasoning performance. Ablation studies demonstrate the complementary contributions of its components.

JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3

Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

LLM Alignment Direct Preference Optimization LLM Safety Large Reasoning Models

Ask PDF

Sep 28, 2026

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan +2

Wayne State University

Language Model Safety Evaluation Deception in Language Models RL for Language Model Reasoning LLM Safety Alignment Large Reasoning Models

Ask PDF

Abstract

Safety-Preserving Fine-Tuning of Language Models

Explore similar work

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models