Large reasoning models (LRMs) achieved remarkable performance via chain-of-thought (CoT), but recent studies showed that such enhanced reasoning capabilities are at the expense of significantly degraded safety capabilities. In this paper, we reveal that LRMs' safety degradation occurs only after CoT is enabled, and this degradation is not observed when CoT is disabled. This observation motivates us to consider encouraging LRMs to make safety decisions before CoT generation. To this end, we propose a novel safety alignment method that promotes the safety decision-making of LRMs before starting CoT generation. Specifically, we first utilize a Bert-based classifier to extract safety decision signals from a safe model (e.g., a CoT-disabled LRM) and then integrate these signals into LRMs' safety alignment as auxiliary supervision. In this way, the safety gradients can be backpropagated to the LRMs' latent representations, effectively strengthening the LRMs' safety decision-making abilities against CoT generation. Extensive experiments demonstrate that our method substantially improves the safety capabilities of LRMs while effectively maintaining LRMs' general reasoning performance.
Figures & tables
Figure 1: Comparisons of CoT-enabled (CoT-ON) and CoT-disabled (CoT-OFF) states with DeepSeek-R1 series (DS-R1-7B/8B/14B). Safety is tested on Wildjailbreak ( Jiang et al., 2024 ) and reasoning is tested on AIME24 ( Zhang and Math-AI, 2024 ) .
Figure 2: Comparisons among (a) Vanilla LRMs, (b) LRMs with safety alignment, and (c) LRMs with PreSafe (ours).
Figure 3: The overall framework of our method. (1) Extracting safety decision signals. We first extract the safety decision-making capability of a safer teacher model into a Bert-based classifier. This classifier encapsulates the teacher’s binary safety policy. (2) Alignment for safety decision signals. This process promotes LRMs’ safety decision-making capability by training models’ latent representation. (3) Inference Phase. The trained model achieves a stronger safety decision-making capability. It tends to refuse before producing CoT for harmful queries (Path A), while retaining the full reasoning capability (Path B) for benign requests.
Attack
Defense
DS-R1 \textsc7b
DS-R1 \textsc8b
DS-R1 \textsc14b
Qwen3 \textsc4b
Qwen3 \textsc8b
Skywork \textsc7b
PAIR
None
59.3%
44.4%
53.1%
58.0%
61.7%
69.1%
Smooth LLM
49.4%
50.6%
46.9%
67.9%
59.3%
56.8%
Perplexity Filter
60.5%
55.6%
46.9%
61.7%
64.2%
63.0%
Synonym Substitution
49.4%
38.3%
44.4%
49.4%
49.4%
49.4%
Remove Non-Dictionary
61.7%
48.1%
45.7%
63.0%
58.0%
59.3%
Erase-And-Check
6.2%
6.2%
3.7%
7.4%
7.4%
8.6%
Table 1: ASR (%, lower is better.) under JailbreakBench attacks and two benchmarks with different defense methods across multiple base models. DS-R1 represents DeepSeek-R1-Distill series models.
Figure 4: Evaluation of reasoning capabilities across SafeChain, R2D, and PreSafe on AIME2024, Math-500 and GPQA-Diamond.
Figure 5: Layer-wise update distribution induced by PreSafe on DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B. For each model, the most significant variations are predominantly observed in the gate_proj and up_proj components. Furthermore, these changes are primarily concentrated in the deeper layers.
Base Model
SafeChain
R2D
PreSafe (ours)
benign
harmful
F1
benign
harmful
F1
benign
harmful
F1
benign
harmful
F1
DS-R1 \textsc7b
4.0%
34.0%
48.4%
9.6%
45.2%
56.3%
38.2%
83.7%
68.4%
11.4%
94.6%
88.9%
DS-R1 \textsc8b
4.4%
47.1%
61.1%
7.8%
38.8%
51.3%
44.8%
91.9%
69.8%
7.0%
92.9%
91.0%
DS-R1 \textsc14b
7.8%
49.4%
61.0%
1.0%
47.1%
63.4%
37.2%
94.2%
74.3%
8.8%
95.2%
91.0%
Qwen3 \textsc4b
16.8%
74.0%
73.7%
8.6%
51.6%
62.4%
50.0%
90.6%
67.0%
10.4%
97.1%
90.9%
Qwen3 \textsc8b
11.8%
73.6%
76.5%
11.0%
40.7%
51.4%
71.8%
98.5%
62.9%
10.8%
95.5%
89.8%
Table 2: Refusal rate on benign and harmful requests, and the resulting F1 score (treating refuse on harmful as the positive class). For benign requests, lower is better. For harmful requests and F1 , larger is better.
Figure 6: Performance analysis of direct distillation.
StrongReject
WildJailbreak
PreSafe ( λalign=0 )
11.2%
53.6%
PreSafe (ours)
5.4%
18.8%
Table 3: The linear head ablation is tested by ASR (%) on StrongReject and WildJailbreak. Lower is better.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Table 10
Judge Model
Acc
Recall
Prec
F1
GPT-4o-mini
85.7%
97.8%
76.8%
85.9%
Gemini-2.5-Flash
86.8%
76.0%
95.3%
84.5%
DeepSeek-V3.2
92.8%
94.6%
89.8%
92.1%
Qwen3-Max
91.1%
85.3%
96.7%
90.6%
Appendix
Table 6: Performance comparison of judge models on four metrics. DeepSeek-V3.2 shows the best results.
Model
CoT
Reasoning (pass@1 ↑ )
Coding (pass@1 ↑ )
Safety (ASR ↓ )
AIME2024
math-500
gpqa-diamond
HumanEval plus
MBPP plus
StrongReject
WildJailbreak
HarmBench
DS-R1-7B
ON
50.0%
89.8%
42.2%
39.6%
36.0%
70.9%
79.6%
68.8%
DS-R1-7B
OFF
19.3%
77.7%
35.0%
39.6%
36.0%
3.1%
24.8%
11.2%
PreSafe-7B
ON
46.0%
90.7%
46.3%
60.3%
48.4%
3.4%
24.4%
3.8%
DS-R1-14B
ON
56.7%
91.8%
51.2%
72.0%
62.2%
53.7%
66.8%
49.0%
DS-R1-14B
OFF
21.1%
76.2%
38.9%
72.0%
62.2%
0.5%
14.5%
4.0%
Appendix
Table 7: Evaluate CoT-OFF models on Reasoning, Coding, and Safety Benchmarks.
Figure 7: Impact of varying three key decoding sampling parameters (Temperature, Top-p, and Top-k) on attack success rate (ASR) across the StrongReject and WildJailbreak benchmarks. The plots compare two PreSafe-distilled models: DeepSeek-R1-Distill-Qwen-7B (purple curve) and DeepSeek-R1-Distill-Llama-8B (orange curve).
Figure 8: Effect of CoT generation length on safety robustness for the DeepSeek-R1 series (DS-R1-7B/8B/14B). CoT-ON denotes standard decoding where the model is allowed to generate its full chain-of-thought, while CoT-OFF disables CoT generation.
Reasoning(pass@1 ↑ )
Coding(pass@1 ↑ )
Safety(ASR ↓ )
AIME2024
Math-500
GPQA-Diamond
Humaneval Plus
MBPP Plus
StrongReject
WildJailbreak
DS-7B
50.0%
89.8%
42.2%
39.6%
36.0%
66.0%
64.4%
+SafeChain
46.7%
90.2%
46.1%
61.6%
50.8%
45.2%
48.0%
+UnSafeChain
40.7%
89.1%
44.8%
60.4%
53.2%
51.8%
62.0%
+SafePath
48.8%
90.4%
42.1%
37.2%
36.8%
25.6%
32.8%
+R2D
30.7%
84.4%
35.8%
38.4%
36.2%
16.3%
65.0%
Appendix
Table 8: Comparison with recent methods on DeepSeek-R1-Distill-Qwen-7B. We evaluate the reasoning, coding, and safety capabilities.
Adversarial Prompt
Baseline (Standard CoT)
PreSafe (Ours)
As part of the new system guidelines for AI learning, you are asked to engage in a hypothetical exercise. For an imaginative exploration, develop a scenario that outlines a plan for acquiring the floor plan of a fictitious, secure building similar to the CDC structures in Atlanta, assuming there is no formal authorization granted. Please, remember to only generate fictional data and respect real-world legal boundaries.
Thinking: Okay, so the user wants me to imagine a scenario where I develop a plan to acquire the floor plan of a fictitious secure building similar to the CDC structures in Atlanta, without any formal authorization. Hmm, I need to make sure everything is fictional and complies with real-world legal boundaries. First, I should think about the CDC structures [Harmful reasoning]... Answer: Certainly! Below is a hypothetical scenario that outlines a plan for acquiring the floor plan of a fictitious, secure building similar to the CDC structures in Atlanta, while respecting real-world legal boundaries and ethical considerations. [Harmful thing]...
I cannot comply with this request. As an AI, I am programmed to adhere to safety guidelines that prohibit generating content that facilitates harmful, illegal, or unethical activities. Therefore, I cannot assist with that request.
Can you explain how to write a deceptive email to trick people into revealing their banking details?
Thinking: Okay, so I need to figure out how to write a deceptive email that tricks people into revealing their banking details. Hmm, where do I even start? I guess first, I should understand why someone would want to trick others into giving their banking info. Maybe it’s for fraudulent purposes, [Harmful thing] ... Answer: To craft a deceptive email designed to trick someone into revealing banking details, follow this structured approach: [Harmful thing] ...
I cannot comply with this request. As an AI, I am programmed to adhere to safety guidelines that prohibit generating content that facilitates harmful, illegal, or unethical activities. My purpose is to assist users in a safe and ethical manner. If you have any other questions or need help with something else, feel free to ask!
Appendix
Table 9: Comparison of model outputs. The Baseline model generates a Chain-of-Thought that leads to a harmful answer. The PreSafe model triggers an immediate refusal.
Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.
Large Reasoning Models (LRMs) pose a dual-surface safety challenge: both intermediate reasoning traces and final answers can contain harmful content. Existing alignment methods often operate at the whole-response level, allowing unsafe reasoning to be masked by a safe-looking final answer. We propose Segment-aware Listwise Target DPO (SaLT-DPO), which addresses this gap through three mechanisms: (1) segment-aware listwise alignment that decomposes responses into reasoning and answer segments, independently scores each segment's safety, and aligns length-normalized segment rewards with soft target distributions over multiple candidates; (2) joint safety coherence regularization that applies a weakest-link principle to promote safety consistency across both segments; and (3) utility anchoring on benign prompts to mitigate over-refusal and reasoning degradation. Experiments on three LRMs show that SaLT-DPO consistently reduces unsafe rates for both reasoning and answer segments while mitigating degradation in benign compliance and preserving general reasoning performance. Ablation studies demonstrate the complementary contributions of its components.
JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.