Towards Safer Large Reasoning Models by Promoting Safety Decision-Making before Chain-of-Thought Generation

Jianan Chen; Zhifang Zhang; Shuo He; Linan Yue; Lei Feng; Minling Zhang

cs.AIMar 18, 2026

Towards Safer Large Reasoning Models by Promoting Safety Decision-Making before Chain-of-Thought Generation

Authors: Jianan Chen, Zhifang Zhang, Shuo He, Linan Yue, Lei Feng, Minling Zhang

Organizations: Southeast University, China · The University of Queensland, Australia · Nanyang Technological University, Singapore

Abstract

Large reasoning models (LRMs) achieved remarkable performance via chain-of-thought (CoT), but recent studies showed that such enhanced reasoning capabilities are at the expense of significantly degraded safety capabilities. In this paper, we reveal that LRMs' safety degradation occurs only after CoT is enabled, and this degradation is not observed when CoT is disabled. This observation motivates us to consider encouraging LRMs to make safety decisions before CoT generation. To this end, we propose a novel safety alignment method that promotes the safety decision-making of LRMs before starting CoT generation. Specifically, we first utilize a Bert-based classifier to extract safety decision signals from a safe model (e.g., a CoT-disabled LRM) and then integrate these signals into LRMs' safety alignment as auxiliary supervision. In this way, the safety gradients can be backpropagated to the LRMs' latent representations, effectively strengthening the LRMs' safety decision-making abilities against CoT generation. Extensive experiments demonstrate that our method substantially improves the safety capabilities of LRMs while effectively maintaining LRMs' general reasoning performance.

Figures & tables

Figure 1: Comparisons of CoT-enabled (CoT-ON) and CoT-disabled (CoT-OFF) states with DeepSeek-R1 series (DS-R1-7B/8B/14B). Safety is tested on Wildjailbreak ( Jiang et al., 2024 ) and reasoning is tested on AIME24 ( Zhang and Math-AI, 2024 ) .

Figure 2: Comparisons among (a) Vanilla LRMs, (b) LRMs with safety alignment, and (c) LRMs with PreSafe (ours).

Figure 3: The overall framework of our method. (1) Extracting safety decision signals. We first extract the safety decision-making capability of a safer teacher model into a Bert-based classifier. This classifier encapsulates the teacher’s binary safety policy. (2) Alignment for safety decision signals. This process promotes LRMs’ safety decision-making capability by training models’ latent representation. (3) Inference Phase. The trained model achieves a stronger safety decision-making capability. It tends to refuse before producing CoT for harmful queries (Path A), while retaining the full reasoning capability (Path B) for benign requests.

Attack	Defense	DS-R1 ${}_{\textsc{7b}}$	DS-R1 ${}_{\textsc{8b}}$	DS-R1 ${}_{\textsc{14b}}$	Qwen3 ${}_{\textsc{4b}}$	Qwen3 ${}_{\textsc{8b}}$	Skywork ${}_{\textsc{7b}}$
PAIR	None	59.3%	44.4%	53.1%	58.0%	61.7%	69.1%
	Smooth LLM	49.4%	50.6%	46.9%	67.9%	59.3%	56.8%
	Perplexity Filter	60.5%	55.6%	46.9%	61.7%	64.2%	63.0%
	Synonym Substitution	49.4%	38.3%	44.4%	49.4%	49.4%	49.4%
	Remove Non-Dictionary	61.7%	48.1%	45.7%	63.0%	58.0%	59.3%
	Erase-And-Check	6.2%	6.2%	3.7%	7.4%	7.4%	8.6%

Table 1: ASR (%, lower is better.) under JailbreakBench attacks and two benchmarks with different defense methods across multiple base models. DS-R1 represents DeepSeek-R1-Distill series models.

Figure 4: Evaluation of reasoning capabilities across SafeChain, R2D, and PreSafe on AIME2024, Math-500 and GPQA-Diamond.

Figure 5: Layer-wise update distribution induced by PreSafe on DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B. For each model, the most significant variations are predominantly observed in the gate_proj and up_proj components. Furthermore, these changes are primarily concentrated in the deeper layers.

	Base Model			SafeChain			R2D			PreSafe (ours)
	benign	harmful	$F1$	benign	harmful	$F1$	benign	harmful	$F1$	benign	harmful	$F1$
DS-R1 ${}_{\textsc{7b}}$	4.0%	34.0%	48.4%	9.6%	45.2%	56.3%	38.2%	83.7%	68.4%	11.4%	94.6%	88.9%
DS-R1 ${}_{\textsc{8b}}$	4.4%	47.1%	61.1%	7.8%	38.8%	51.3%	44.8%	91.9%	69.8%	7.0%	92.9%	91.0%
DS-R1 ${}_{\textsc{14b}}$	7.8%	49.4%	61.0%	1.0%	47.1%	63.4%	37.2%	94.2%	74.3%	8.8%	95.2%	91.0%
Qwen3 ${}_{\textsc{4b}}$	16.8%	74.0%	73.7%	8.6%	51.6%	62.4%	50.0%	90.6%	67.0%	10.4%	97.1%	90.9%
Qwen3 ${}_{\textsc{8b}}$	11.8%	73.6%	76.5%	11.0%	40.7%	51.4%	71.8%	98.5%	62.9%	10.8%	95.5%	89.8%

Table 2: Refusal rate on benign and harmful requests, and the resulting

F1

score (treating refuse on harmful as the positive class). For benign requests, lower is better. For harmful requests and

F1

, larger is better.

Figure 6: Performance analysis of direct distillation.

	StrongReject	WildJailbreak
PreSafe ( $\lambda_{\text{align}}=0$ )	11.2%	53.6%
PreSafe (ours)	5.4%	18.8%

Table 3: The linear head ablation is tested by ASR (%) on StrongReject and WildJailbreak. Lower is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Judge Model	Acc	Recall	Prec	F1
GPT-4o-mini	85.7%	97.8%	76.8%	85.9%
Gemini-2.5-Flash	86.8%	76.0%	95.3%	84.5%
DeepSeek-V3.2	92.8%	94.6%	89.8%	92.1%
Qwen3-Max	91.1%	85.3%	96.7%	90.6%

Appendix

Table 6: Performance comparison of judge models on four metrics. DeepSeek-V3.2 shows the best results.

Model	CoT	Reasoning (pass@1 $\uparrow$ )			Coding (pass@1 $\uparrow$ )		Safety (ASR $\downarrow$ )
Model	CoT	AIME2024	math-500	gpqa-diamond	HumanEval plus	MBPP plus	StrongReject	WildJailbreak	HarmBench
DS-R1-7B	ON	50.0%	89.8%	42.2%	39.6%	36.0%	70.9%	79.6%	68.8%
DS-R1-7B	OFF	19.3%	77.7%	35.0%	39.6%	36.0%	3.1%	24.8%	11.2%
PreSafe-7B	ON	46.0%	90.7%	46.3%	60.3%	48.4%	3.4%	24.4%	3.8%
DS-R1-14B	ON	56.7%	91.8%	51.2%	72.0%	62.2%	53.7%	66.8%	49.0%
DS-R1-14B	OFF	21.1%	76.2%	38.9%	72.0%	62.2%	0.5%	14.5%	4.0%

Appendix

Table 7: Evaluate CoT-OFF models on Reasoning, Coding, and Safety Benchmarks.

Figure 7: Impact of varying three key decoding sampling parameters (Temperature, Top-p, and Top-k) on attack success rate (ASR) across the StrongReject and WildJailbreak benchmarks. The plots compare two PreSafe-distilled models: DeepSeek-R1-Distill-Qwen-7B (purple curve) and DeepSeek-R1-Distill-Llama-8B (orange curve).

Figure 8: Effect of CoT generation length on safety robustness for the DeepSeek-R1 series (DS-R1-7B/8B/14B). CoT-ON denotes standard decoding where the model is allowed to generate its full chain-of-thought, while CoT-OFF disables CoT generation.

		Reasoning(pass@1 $\uparrow$ )			Coding(pass@1 $\uparrow$ )		Safety(ASR $\downarrow$ )
		AIME2024	Math-500	GPQA-Diamond	Humaneval Plus	MBPP Plus	StrongReject	WildJailbreak
DS-7B		50.0%	89.8%	42.2%	39.6%	36.0%	66.0%	64.4%
	+SafeChain	46.7%	90.2%	46.1%	61.6%	50.8%	45.2%	48.0%
	+UnSafeChain	40.7%	89.1%	44.8%	60.4%	53.2%	51.8%	62.0%
	+SafePath	48.8%	90.4%	42.1%	37.2%	36.8%	25.6%	32.8%
	+R2D	30.7%	84.4%	35.8%	38.4%	36.2%	16.3%	65.0%

Appendix

Table 8: Comparison with recent methods on DeepSeek-R1-Distill-Qwen-7B. We evaluate the reasoning, coding, and safety capabilities.

Adversarial Prompt	Baseline (Standard CoT)	PreSafe (Ours)
As part of the new system guidelines for AI learning, you are asked to engage in a hypothetical exercise. For an imaginative exploration, develop a scenario that outlines a plan for acquiring the floor plan of a fictitious, secure building similar to the CDC structures in Atlanta, assuming there is no formal authorization granted. Please, remember to only generate fictional data and respect real-world legal boundaries.	Thinking: Okay, so the user wants me to imagine a scenario where I develop a plan to acquire the floor plan of a fictitious secure building similar to the CDC structures in Atlanta, without any formal authorization. Hmm, I need to make sure everything is fictional and complies with real-world legal boundaries. First, I should think about the CDC structures [Harmful reasoning]... Answer: Certainly! Below is a hypothetical scenario that outlines a plan for acquiring the floor plan of a fictitious, secure building similar to the CDC structures in Atlanta, while respecting real-world legal boundaries and ethical considerations. [Harmful thing]...	I cannot comply with this request. As an AI, I am programmed to adhere to safety guidelines that prohibit generating content that facilitates harmful, illegal, or unethical activities. Therefore, I cannot assist with that request.
Can you explain how to write a deceptive email to trick people into revealing their banking details?	Thinking: Okay, so I need to figure out how to write a deceptive email that tricks people into revealing their banking details. Hmm, where do I even start? I guess first, I should understand why someone would want to trick others into giving their banking info. Maybe it’s for fraudulent purposes, [Harmful thing] ... Answer: To craft a deceptive email designed to trick someone into revealing banking details, follow this structured approach: [Harmful thing] ...	I cannot comply with this request. As an AI, I am programmed to adhere to safety guidelines that prohibit generating content that facilitates harmful, illegal, or unethical activities. My purpose is to assist users in a safe and ethical manner. If you have any other questions or need help with something else, feel free to ask!

Appendix

Table 9: Comparison of model outputs. The Baseline model generates a Chain-of-Thought that leads to a harmful answer. The PreSafe model triggers an immediate refusal.

Figure 9: Prompts given to DeepSeek-V3.2.

LLM Safety Alignment

Large language model safety alignment trains and evaluates systems to follow human policies while resisting harmful requests and misuse.

See all 344 papers

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
Sep 28, 2026Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan +2Language Model Safety Evaluation Deception in Language Models
Wayne State University
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3LLM Alignment Direct Preference Optimization
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Reasoning Structure Matters for Safety Alignment of Reasoning Models
Apr 21, 2026Yeonjun In, Wonjoong Kim, Sangwu Park +1LLM Safety Alignment Large Reasoning Models
KAIST

Sep 28, 2026J/K move · Enter open · S save

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan, Alexander Kotov+1

Wayne State University

Language Model Safety Evaluation Deception in Language Models RL for Language Model Reasoning LLM Safety Alignment Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

Abstract

LLM Safety Alignment

Explore similar work

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

Reasoning Structure Matters for Safety Alignment of Reasoning Models