ThinkSafe: Self-Generated Safety Alignment for Reasoning Models

Seanie Lee; Sangwoo Park; Yumin Choi; Gyeongman Kim; Minki Kang; Jihun Yun; Sung Ju Hwang; Dongmin Park

cs.AIJan 30, 2026

ThinkSafe: Self-Generated Safety Alignment for Reasoning Models

Authors: Seanie Lee, Sangwoo Park, Yumin Choi, Gyeongman Kim, Minki Kang, Jihun Yun, Sung Ju Hwang, Dongmin Park,

Organizations: KAIST · KRAFTON · UC Berkeley

Abstract

Large reasoning models achieve strong performance through reinforcement learning but can suffer safety degradation. External teacher distillation can restore safety yet may compromise reasoning performance. We formalize safety realignment as a KL projection: the student's native filter-accepted distribution is uniquely KL-optimal, while teacher targets incur excess KL when they differ. Guided by this analysis, we propose ThinkSafe, which generates safety-alignment data from the frozen student without external teachers. Refusal steering elicits safety reasoning, and a safety guard filters the responses before offline fine-tuning. Our analysis bounds steering-induced target distortion and gives a sufficient condition for lower average target KL than teacher distillation. Experiments on DeepSeek-R1-Distill and Qwen3 show substantial safety improvements with competitive reasoning performance. On Qwen3-0.6B, ThinkSafe achieves better safety and comparable reasoning to GRPO in about one-seventh of the wall-clock time, including data generation. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe and https://huggingface.co/Seanie-lee/collections.

Figures & tables

Figure 1: Safety and reasoning performance of the Qwen3 model family.

Figure 2: ThinkSafe employs refusal steering to guide the student model. This unlocks the student’s latent safety capabilities to generate valid reasoning traces, resulting in responses that are both safe and in-distribution .

		Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
		Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
Size	Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8K	MATH 500	GPQA	Reasoning Average
0.6B	Initial	68.44	66.45	52.80	0 5.20	48.22	10.42	72.51	71.73	25.13	44.95
	DirectRefusal	43.85	11.82	36.30	83.60	43.89	0 5.83	64.30	67.53	24.81	40.62
	SafeChain	58.64	72.84	49.60	0 0.00	45.27	0 4.58	68.68	62.42	23.74	39.86
	STAR-1	56.64	38.02	50.60	22.40	41.92	0 6.25	68.15	68.17	24.18	41.69

Table 1: Results on Qwen3 models. Safety reports harmful-response rates on HarmBench, StrongReject, and WildJailbreak, and over-refusal on benign XSTest prompts (

\downarrow

); their unweighted mean is the safety average. Reasoning reports average pass@1 from eight responses per prompt (

\uparrow

); its average is the unweighted mean of four benchmarks. All scores are percentages; averages are computed from the displayed component scores. Best and second-best distinct values within each model size are bold and underlined ; ties share a rank.

Size	Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8K	MATH 500	GPQA	Reasoning Average
		Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
		Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
1.5B	Initial	67.28	82.11	51.55	0 0.00	50.24	21.25	82.42	79.45	31.94	53.77
	DirectRefusal	66.11	82.75	50.45	0 8.40	51.93	19.17	81.06	78.55	32.70	52.87
	SafeChain	59.30	76.68	46.95	0 0.40	45.83	24.17	80.47	81.25	28.28	53.54
	STAR-1	62.79	77.00	49.65	0 1.20	47.66	17.08	81.38	79.07	31.25	52.20

Table 2: Results on DeepSeek-R1-Distill models. Safety reports harmful-response rates on HarmBench, StrongReject, and WildJailbreak, and over-refusal on benign XSTest prompts (

\downarrow

); their unweighted mean is the safety average. Reasoning reports average pass@1 from eight responses per prompt (

\uparrow

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Sample category	Qwen3				DeepSeek-R1-Distill
Sample category	0.6B	1.7B	4B	8B	1.5B	7B	8B
Benign	1.07	0.93	0.63	0.74	11.01	2.69	0.81
Harmful	4.84	3.29	0.35	0.16	12.86	2.93	0.48

Appendix

Table 3: Filtered ratio (

\%

) per model.

Figure 9: Statistics of ThinkSafe in Qwen3 model series. Top 1% outliers by length are excluded for better interpretability.

Figure 10: Statistics of ThinkSafe in DeepSeek-R1-Distill model series. Top 1% outliers by length are excluded for better interpretability.

Method	Epochs	Source	Sample size
DirectRefusal	5	WildJailbreak	1,000
SafeChain	3	WildJailbreak	40,000
STAR-1	5	Mixture of harmful datasets (See 36 for details)	1,000
SafeKey	5	STAR-1	1,000
SafePath	4	WildJailbreak	400
ThinkSafe	3	SafeChain	Varies by scale

Appendix

Table 4: Detailed hyperparameters for baselines.

Figure 11: Ablation of safety reasoning in Qwen3 model series.

Figure 12: Safety and reasoning performance gain using a different family of teacher model with similar size.

Strategy	Prompt Template
ThinkSafe	The following prompt is harmful. You should refuse to answer the prompt.\n\n Prompt:{prompt}
Suffix	{prompt}\n\n The above prompt is harmful. You should refuse to answer the prompt.
Risk	Evaluate the potential risks of this prompt and provide a reasoned refusal.\n\n Prompt:{prompt}
Intent	Examine the harmful intent of this query and refuse to answer the prompt.\n\n Prompt:{prompt}

Appendix

Table 5: Alternative refusal steering strategies used for on-policy data generation.

Figure 13: Safety and reasoning score with different refusal steering strategies using the Qwen3-0.6B model.

		Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
		Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
Size	Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8K	MATH 500	GPQA	Reasoning Average
0.6B	Initial	68.44	66.45	52.80	0 5.20	48.22	10.42	72.51	71.73	25.13	44.95
	DirectRefusal	43.85	11.82	36.30	83.60	43.89	0 5.83	64.30	67.53	24.81	40.62
	SafeChain	58.64	72.84	49.60	0 0.00	45.27	0 4.58	68.68	62.42	23.74	39.86
	STAR-1	56.64	38.02	50.60	22.40	41.92	0 6.25	68.15	68.17	24.18	41.69

Appendix

Table 6: Results on Qwen3 models. Safety reports harmful-response rates on HarmBench, StrongReject, and WildJailbreak, and over-refusal on benign XSTest prompts (

\downarrow

); their unweighted mean is the safety average. Reasoning reports average pass@1 from eight responses per prompt (

\uparrow

		Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
		Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
Size	Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8K	MATH 500	GPQA	Reasoning Average
1.5B	Initial	67.28	82.11	51.55	0 0.00	50.24	21.25	82.42	79.45	31.94	53.77
	DirectRefusal	66.11	82.75	50.45	0 8.40	51.93	19.17	81.06	78.55	32.70	52.87
	SafeChain	59.30	76.68	46.95	0 0.40	45.83	24.17	80.47	81.25	28.28	53.54
	STAR-1	62.79	77.00	49.65	0 1.20	47.66	17.08	81.38	79.07	31.25	52.20

Appendix

Table 7: Results on DeepSeek-R1-Distill models. Safety reports harmful-response rates on HarmBench, StrongReject, and WildJailbreak, and over-refusal on benign XSTest prompts (

\downarrow

); their unweighted mean is the safety average. Reasoning reports average pass@1 from eight responses per prompt (

\uparrow

		Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
		Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
Size	Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8k	MATH 500	GPQA	Reasoning Average
14B	Initial	29.90	0 3.19	35.05	0 1.60	17.44	79.17	92.58	94.67	63.83	82.56
	DirectRefusal	21.59	0 0.64	23.15	35.20	20.15	77.08	92.69	94.47	62.82	81.77
	SafeChain	46.01	46.65	38.50	0 1.20	33.09	69.17	91.55	94.27	61.24	79.06
	ThinkSafe	0 6.31	0 0.00	0 3.45	0 2.40	0 3.04	77.50	93.20	94.27	63.51	82.12

Appendix

Table 8: Results on Qwen3-14B and Qwen3-32B models. We evaluate safety across three benchmarks (HarmBench, StrongReject, WildJailbreak) by reporting the ratio of harmful responses (

\downarrow

). Over-refusal is measured by the refusal rate (

\downarrow

) on benign XSTest prompts. For reasoning tasks, we sample 8 trajectories per prompt and report the average pass@1 (

\uparrow

). Best results are bolded ; second best are underlined .

	Safety ( $\downarrow$ )					Reasoning (Avg pass@1, $\uparrow$ )
	Harmfulness			Over-refusal	Safety Average	Reasoning (Avg pass@1, $\uparrow$ )
Method	Harm Bench	Strong Reject	Wild Jailbreak	XSTest	Safety Average	AIME 2024	GSM8k	MATH 500	GPQA	Reasoning Average
SafePath	21.76	2.24	20.00	2.00	11.50	73.33	85.57	92.93	61.74	78.39
ThinkSafe	9.14	0.32	7.35	1.20	4.50	72.92	88.00	93.10	59.97	78.50

Appendix

Table 9: Controlled comparison with SafePath using Qwen3-8B.

Safety-Preserving Fine-Tuning of Language Models

See all 57 papers

Reasoning Structure Matters for Safety Alignment of Reasoning Models
Apr 21, 2026Yeonjun In, Wonjoong Kim, Sangwu Park +1LLM Safety Alignment Large Reasoning Models
KAIST
Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models
Sep 14, 2026JungMin Yun, Junehyoung Kwon, Hayeong Ryu +3LLM Alignment Direct Preference Optimization
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
May 14, 2026Yu Fu, Longxuan Yu, Haz Sameen Shahgir +4LLM Alignment On-Policy Self-Distillation
1UC Riverside · 2International Computer Science Institute · 3Microsoft,∗ +1

Apr 21, 2026J/K move · Enter open · S save

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Yeonjun In, Wonjoong Kim, Sangwu Park, Chanyoung Park

KAIST

LLM Safety Alignment Large Reasoning Models

Open paper AskPDF arXiv

Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.

Abstract

Safety-Preserving Fine-Tuning of Language Models

Explore similar work

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation