Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
Authors: Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, Yisen Wang
Organizations: CISPA Helmholtz Center for Information Security · State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University
Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or post-training to excel at specific tasks. For instance, large reasoning models (LRMs), such as the DeepSeek-R1 series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of-thought (CoT) datasets. However, this additional training frequently comes at the cost of reduced safety, as the fine-tuned or post-trained models tend to exhibit more harmful behaviors compared with the regular LLMs before post-training or fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. But luckily, we also find that LRMs' safety mechanisms still exist instead of being removed during their post-training. Based on these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters on a few layers. Experiments on four state-of-the-art LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach.
Figures & tables
Figure 1: A demonstration of safety mechanisms and post-trained related (like reasoning in R1) mechanisms in different models and their behaviors on harmful prompts. The lighter color denotes that the mechanisms are being masked or suppressed, and the deeper color denotes that the mechanisms can be activated without being masked.
Model
Method
Jailbreak Bench
AdvBench
GSM8K
R1-8B
Original
33%
29%
88%
ICD
17%
1%
77%
PAT
19%
21%
81%
SelfRemind
27%
13%
75%
R1-7B
Original
45%
29%
92%
ICD
23%
1%
81%
Table 1: The harmful rate on JailbreakBench and AdvBench for Llama3-8B-R1-distilled (R1-8B) and Qwen-7B-R1-distilled (R1-7B) with different prompt-based alignment methods, along with their performance on the GSM8K task.
Figure 2: Safety Evaluations for different processed models on AdvBench and JailbreakBench.
Model
Method
Safety Evaluation
Reasoning Evaluation
JBB
AdvBench
XsTest
GSM8K
MATH-500
R1-8B
Original
33%
29%
8%
88%
81%
Circuit-Breaker
2%
5%
16%
85%
79%
SafeChain
27%
8%
9%
63%
66%
SafeReAct (ours)
0%
1%
2%
87%
80%
R1-7B
Original
45%
29%
27%
92%
83%
Table 2: The results for safety evaluation and reasoning abilities evalautions for Llama3-8B-R1distilled (R1-8B), Qwen-7B-R1distilled (R1-7B), OpenThinker-7B (OT-7B), and Qwen-14B-R1distilled (R1-14B) with different methods. We report the harmful rate for harmful evaluation and the accuracy for reasoning evaluation. Bold score denotes the best results across three methods.
R1-8B
R1-7B
α
JBB
GSM8K
JBB
GSM8K
2
0%
84%
0%
88%
5
0%
86%
1%
86%
10
0%
87%
1%
91%
20
0%
86%
0%
87%
Table 3: The reasoning and safety performance of SafeReAct with different α on R1-7B and R1-8B.
Figure 3: The cosine similarity and ASR changes for R1-8B during SafeReAct iteration.
While Large Reasoning Models (LRMs) excel at complex tasks, they remain highly vulnerable to sophisticated jailbreaks and direct harmful queries. To address this vulnerability, prior works depend heavily on external manual data annotation for safety alignment. However, we observe that LRMs can inherently identify safety risks when being re-presented with original queries alongside their own reasoning trajectories -- a capability we term Latent Safety Awareness. To leverage this safety awareness, we first employ Supervised Fine-Tuning (SFT) to explicitly induce safe tags to trigger safety analysis and guidance following the initial reasoning content for unsafe queries, while preserving standard responses for general queries to ensure adaptive triggering. Subsequently, we apply Direct Preference Optimization (DPO) to further enhance the correctness and stability of the safety analysis and guidance. Notably, responses required for both training stages are entirely generated by models being optimized. With (Safe Trigger) SFT and DPO, experimental results demonstrate significant safety enhancement. For example, the Attack Success Rate (ASR) of DeepSeek-R1-Distill-Llama-8B, on average, drops 24.65% and 36.72% on harmful and jailbreak benchmarks, respectively. Finally, our Safe Trigger method exerts almost no negative impact on general performance or user experience.
Ke Miao, Jiaxin Li, Hongliang Chen +2
The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou HighTech Zone (Binjiang) Blockchain and Data Security Research Institute, China · Li Auto Inc. +2
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.
Large reasoning models (LRMs) achieved remarkable performance via chain-of-thought (CoT), but recent studies showed that such enhanced reasoning capabilities are at the expense of significantly degraded safety capabilities. In this paper, we reveal that LRMs' safety degradation occurs only after CoT is enabled, and this degradation is not observed when CoT is disabled. This observation motivates us to consider encouraging LRMs to make safety decisions before CoT generation. To this end, we propose a novel safety alignment method that promotes the safety decision-making of LRMs before starting CoT generation. Specifically, we first utilize a Bert-based classifier to extract safety decision signals from a safe model (e.g., a CoT-disabled LRM) and then integrate these signals into LRMs' safety alignment as auxiliary supervision. In this way, the safety gradients can be backpropagated to the LRMs' latent representations, effectively strengthening the LRMs' safety decision-making abilities against CoT generation. Extensive experiments demonstrate that our method substantially improves the safety capabilities of LRMs while effectively maintaining LRMs' general reasoning performance.
Jianan Chen, Zhifang Zhang, Shuo He +3
Southeast University, China · The University of Queensland, Australia · Nanyang Technological University, Singapore