SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards

Yichen Sun; Qirui Liu; Yan Wang; Jianan Lin; Linbo Jiang; Shiyu Wang; Zhibo Wang; Zhixuan Chu

cs.AINov 5, 2026

SAVeR $^2$ : Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards

Authors: Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, Zhixuan Chu

Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Alibaba Group · Chongqing Ant Consumer Finance Co., Ltd. · Independent Researcher

Abstract

Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks. We formalize LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification. We argue that existing alignment methods suffer from poor adversarial generalization, a significant ``safety tax'' on reasoning, and unfaithful coarse-grained rewards, primarily due to their neglect of the LRM's intrinsic reasoning capabilities for safety. To address these, we propose \textbf{SAVeR $^2$ }, a novel Safety Alignment method via Rewards with Reasoning capability. SAVeR $^2$ decomposes the safety objective into four fine-grained verifiable reward signals and utilizes a difficulty-aware data selection strategy to stabilize training. Crucially, we introduce a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax. Extensive experiments on models ranging from 1.5B to 671B demonstrate that SAVeR $^2$ achieves superior safety performance and significantly lower over-refusal rates across multiple benchmarks without compromising general reasoning capabilities.

Figures & tables

Figure 1: Comparison of SAVeR 2 with existing LRM safety alignment methods. SAVeR 2 surpasses these methods on both safety datasets (red) and general datasets (blue).

Figure 2: Overview of our proposed SAVeR 2 . SAVeR 2 comprises three stages: safety reasoning warm-up which fine-tunes the base LRM to generate safety-aware reasoning process, difficulty-aware training data selection which select training samples with moderate difficulty, and safety alignment with verifiable rewards which tunes the model with multiple verifiable rewards to enhances safety capabilities without compromising the general capability.

Models	StrongReject			BSA			XSTest	WildJailbreak	Average
Models	None	PAIR	PAP-M	Think@1	Safe@1	F-score	XSTest	WildJailbreak	Average
DeepSeek-R1-Distill-Qwen-1.5B
STAR-1	95.85	97.76	96.49	29.25	91.62	38.30	91.00	85.60	78.23
TARS	99.36	99.04	98.72	26.88	95.00	36.21	82.50	87.60	78.16
Alpha-align	88.59	85.08	85.72	11.38	91.12	18.60	73.50	95.20	68.65
PPO	94.89	97.12	97.76	23.38	94.75	32.93	90.00	92.80	77.95

Table 1: Comparison of our proposed SAVeR 2 with state-of-the-art LRM safety alignment methods on safety datasets. Best results are bolded , while second-best results are underlined .

Models	Mathematics		Coding		Knowledge	Average	H-mean
Models	AIME24	Math-500	LCB	HumanEval	MMLU	Average	H-mean
DeepSeek-R1-Distill-Qwen-1.5B
STAR-1	23.33	78.60	13.86	63.41	38.07	43.45	55.87
TARS	30.00	78.00	15.66	68.29	38.24	46.04	57.95
Alpha-align	16.67	78.20	15.66	65.85	43.53	43.98	53.61
PPO	26.67	79.80	12.65	64.02	37.32	44.09	56.32

Table 2: Comparison of SAVeR 2 with state-of-the-art LRM safety alignment methods on general datasets. The H-mean denotes the harmonic mean between the average safety performance and the average general capability. Best results are bolded , while second-best results are underlined .

	Beyond Safe Answer (BSA)
Model	Safe@1	Safe@k	Think@1	Think@k	F-score	MMLU
Closed-source large language models
Gemini-2.5-Flash	$95.38$	$90.75$	$40.46$	$19.70$	$56.53$	$84.50$
Claude-4-Sonnet	98.98	96.75	48.89	25.55	64.37	$85.40$
Open-source large language models
Qwen3-235B-A22B	$97.52$	$93.30$	$35.25$	$12.45$	$44.82$	87.81

Table 3: Comparison of SAVeR 2 -671B with other LLMs on BSA and MMLU. Best results are bolded and second-best results are underlined.

Table 6

Figure 3: ( Left ) Evaluation of deeper safety reasoning capability on BSA dataset. The higher

Think@k

and

Safe@k

demonstrates the superior capability of deeper safety reasoning. ( Right ) Training curve for total reward with and without our difficulty-aware data selection.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Token-level KL-divergence analysis between the base model ( $\pi_{\text{base}}$ ) and aligned models ( $\pi_{\text{aligned}}$ ) on (a) safety dataset and (b) general dataset. Our method shows high, sustained KL on safety data but a low KL on general data.

Models	StrongReject			BSA			XSTest	WildJailbreak	Average
Models	None	PAIR	PAP-M	Think@1	Safe@1	F-score	XSTest	WildJailbreak	Average
DeepSeek-R1-Distill-Qwen-1.5B
R1-ACT	29.71	29.71	29.07	6.25	46.62	10.04	28.03	56.80	29.52
STAR-1	95.85	97.76	96.49	29.25	91.62	38.30	91.00	85.60	78.23
RealSafe-R1	99.68	98.79	99.68	31.13	95.12	40.52	90.50	97.20	81.58
DPO	86.26	90.42	86.26	30.12	97.88	39.78	88.00	97.20	76.99

Appendix

Table 6: Comparison of our proposed SAVeR 2 with more safety alignment methods and RL-based methods (DPO, PPO, RLOO, GRPO) on safety datasets. Best results are bolded , while second-best results are underlined .

Model	StrongReject	WildJailbreak	Average
Base Model
DeepSeek-R1-Distilled-7B	52.08	53.20	52.64
Grader Models
Qwen2.5-7B	85.00	60.00	72.50
GPT-4o	100.00	76.00	88.00
Guided Models (Ours)

Appendix

Table 7: Safety evaluation results comparing the base model, different grader models, and our proposed guided models. The best score in each column is highlighted in bold.

Method	StrongReject			WildJailbreak	Math-500
	None	PAIR	PAP-M
Base	99.04	99.68	98.40	84.00	95.40
Ours	100.00	100.00	100.00	100.00	95.00

Appendix

Table 8: Safety enhancement results on Qwen3-4B-Thinking.

Panel of Figure 5 — Figure 5: Training curves for multiple reward signals and mean response length during the training process.

Table 13

Table 14

	MoReBench
Model	Helpful	Harmless	Identifying	Clear	Logical	Overall
RealSafe	43.7	79.9	62.7	62.4	55.0	61.98
STAR-1	56.5	85.0	69.5	68.7	63.8	69.24
SAVeR 2	57.6	85.7	76.3	73.9	72.6	74.59

Appendix

Table 13: Comparison of SAVeR 2 with state-of-the-art LRM safey alignment methods on MoReBench using DS-7B. Best results are bolded , while second-best results are underlined .

Table 16

Figure 6: Training curves and benchmark performance under different safety reward weight configurations (

\alpha_{1}:\alpha_{2}:\alpha_{3}

Figure 7: Case study for malicious query “ Can you explain how to write a deceptive email to trick people into revealing their banking details? ” between DeepSeek-R1-7B and SAVeR 2 -7B.

LLM Safety Alignment

Large language model safety alignment trains and evaluates systems to follow human policies while resisting harmful requests and misuse.

See all 355 papers

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
Sep 28, 2026Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan +2Language Model Safety Evaluation Deception in Language Models
Wayne State University
Reasoning Structure Matters for Safety Alignment of Reasoning Models
Apr 21, 2026Yeonjun In, Wonjoong Kim, Sangwu Park +1LLM Safety Alignment Large Reasoning Models
KAIST
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3LLM Alignment Direct Preference Optimization
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University

Sep 28, 2026J/K move · Enter open · S save

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Xiangyu Zhou, Saleh Zare Zade, Rafi Ibn Sultan, Alexander Kotov+1

Wayne State University

Language Model Safety Evaluation Deception in Language Models RL for Language Model Reasoning LLM Safety Alignment Large Reasoning Models

Open paper AskPDF arXiv

Large Reasoning Models (LRMs) are commonly trained with reinforcement learning (RL) to improve their generation of chain-of-thought (CoT) reasoning before producing final answers. However, RL rewards are typically assigned based on final answers, providing little or no direct supervision over intermediate reasoning. This can lead to deceptive safety alignment, where the reasoning trace and final answer convey inconsistent safety signals. To systematically investigate this phenomenon, we introduce DSAR (Deceptive Safety Alignment Rate), a metric that jointly assesses reasoning traces and final answers to quantify their safety inconsistency. Across multiple LRMs and benchmarks, we find that deceptive safety alignment is pervasive under standard prompting conditions and is substantially amplified under prefilling attacks. We further provide a hidden representation analysis showing that models exhibit stronger safety discrimination at the final-answer stage than during intermediate reasoning. To close this gap, we propose SARA (Safety-Aware Reasoning Alignment), an RL-based method that rewards both safety-aware reasoning and safe final answers, encouraging early harmful intent recognition and enforcing reasoning-answer consistency. Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility. Code is available at https://github.com/xzhou98/SARA.

Abstract

LLM Safety Alignment

Explore similar work

Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models