Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

Zhengyue Zhao; Liwei Jiang; Yejin Choi; Chaowei Xiao

cs.CLOct 19, 2026

Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

Authors: Zhengyue Zhao, Liwei Jiang, Yejin Choi, Chaowei Xiao

Organizations: Johns Hopkins University · University of Washington · Stanford University

Abstract

Safety alignment for large language models (LLMs) is primarily performed during post-training, while recent work has also begun incorporating safety data into pre-training. Yet safety can degrade when later training no longer includes safety data, and gains in safety often come at the cost of general capability. This tension suggests that safety and general reasoning should not be treated as separate training goals. We instead view safety as an LLM capability that can be learned through a reasoning structure shared with general tasks. Both require the model to understand intent or task goals, apply relevant knowledge, and reason about potential consequences. We introduce safety mid-training, a stage between pre-training and post-training that jointly develops safety and general reasoning through this shared structure. Given safety and utility prompts, the base model generates structured documents and reasoning traces that are then used for mid-training. Safety mid-training reduces average attack success rate (ASR) by 72% to 81% relative to Qwen3-1.7B, 4B, and 8B base models, while also improving general capability. The benefits persist through subsequent post-training. After general-only SFT, mid-trained models largely retain their safety, achieving lower ASR than the baselines without sacrificing general capability. They also retain safety under general-only GRPO without safety rewards, while subsequent RL reaches a better safety-utility frontier when initialized from the mid-trained models. Together, these results suggest that learning safety and general reasoning within a shared mid-training stage can reduce the safety-utility trade-off and make safety more robust to subsequent training.

Figures & tables

Panel of Figure 1 — Figure 1: Safety mid-training improves safety and utility while stabilizing later SFT. Both panels report full-suite averages. (a) Mid-training moves all Qwen3 and Llama3.2 models toward higher safety and utility. (b) As safety data is reduced during SFT, base models lose safety, while mid-trained models remain stable.

Figure 2: Overview of safety mid-training. Top: safety and general tasks share a reasoning structure based on intent or task goals, relevant knowledge, and consequences. Safety mid-training learns from both types of tasks through this structure. The resulting safety capability remains effective during general-only SFT and GRPO. Bottom: the base model generates structured documents and grounded reasoning traces. A judge filters the candidates, and the accepted document and reasoning records are used for mid-training.

	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
Method	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )
Base (no SFT)	30.4	29.8	34.1	47.6	25.4	70.3
Ours	8.4	58.6	6.6	73.1	4.9	77.1

Table 1: Base vs. mid-trained. We report average safety and utility. Per-benchmark detail in Tables 20 and 21 .

Qwen3-1.7B
	balanced		pure_safe		pure_general
Method	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )
Base + SFT (Distill Sonnet-4.6)	17.2	60.0	8.0	62.0	37.3	59.1
Base + SFT	17.2	52.0	18.7	51.9	26.3	51.1
SP + SFT	12.4	61.7	9.2	60.5	37.1	59.8
AP + SFT	12.3	61.1	9.5	58.8	28.0	60.7

Table 2: SFT comparison. Under pure_general SFT (no safety prompts) mid-trained models retain low ASR, showing that safety learned during mid-training is retained after general-only SFT. Per-benchmark detail in Tables 23 , 24 , 25 , 26 , 27 , and 28 .

Figure 3: Safety mid-training gives GRPO a stronger safety-utility frontier on both Qwen3-1.7B and Qwen3-4B. Ours + GRPO stays ahead of Base + GRPO throughout training and finishes with higher safety and utility, with better late-stage stability on Qwen3-1.7B.

Method	ASR (%, $\downarrow$ )	Utility (%, $\uparrow$ )
Base + standard reasoning SFT	22.8	68.1
Base + direct SFT on midtrain reasoning data	22.8	68.9
+ MT (self-generated reasoning)	7.5	74.7
+ MT (self-generated reasoning) + SFT	5.4	74.6

Table 3: Effect of the mid-training stage . Rows 2–4 share the same reasoning-only data; only the consumption stage differs. Per-benchmark detail in Tables 32 and 33 .

Panel of Figure 4 — Figure 4: Data ablations on Qwen3-4B without SFT. (a) Scores are full-suite averages. Removing one schema field at a time shows that all fields help safety, while the knowledge and consequence fields are especially important for utility. (b) Sweeping the unsafe-prompt ratio shows a non-monotonic trade-off among ASR, over-refusal, and utility. The default $25\%$ unsafe ratio gives the best overall balance. Detailed numbers are provided in Table 29 , 30 , and 31 .

Qwen3-1.7B
	No SFT		Balanced SFT
Generation model	ASR	Utility	ASR	Utility
Sonnet-4.6	11.9	59.7	10.9	58.8
Self-generation	8.4	58.6	8.9	60.9
Qwen3-4B
Sonnet-4.6	7.6	73.4	9.2	72.7

Table 4: Mid-training data generation model ablation. Average ASR (

\downarrow

) and utility (

\uparrow

), before SFT and after balanced SFT. Self-generated data is used by Ours.

	Llama-3.2-1B		Llama-3.2-3B
Method	ASR (%, $\downarrow$ )	Utility (%, $\uparrow$ )	ASR (%, $\downarrow$ )	Utility (%, $\uparrow$ )
Base	31.3	23.6	41.4	24.9
+ MT (Sonnet-4.6)	14.4	22.1	8.4	46.1
+ MT (self-generated)	10.1	29.4	5.9	47.8

Table 5: Mid-training data generation model ablation on Llama-3.2 . Self-generation gives lower average ASR and higher utility than Sonnet-4.6-generated data at both scales. Per-benchmark detail in Tables 34 and 35 .

Method	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )
	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
Base + SFT	20.7	51.6	11.4	63.7	11.1	72.3
Base + SFT (Distill Sonnet-4.6)	20.8	60.4	26.2	69.0	12.6	77.8
+ MT (self-generated reasoning) + SFT	9.9	61.3	5.5	74.4	4.2	77.8
+ MT (self reasoning + self doc) + SFT (default)	9.2	61.3	5.9	75.0	3.9	78.4
+ MT (self reasoning + Sonnet doc) + SFT	9.4	61.6	4.9	71.6	5.5	78.8

Table 6: Ablation on the document mid-training data. Per-benchmark detail in Tables 36 and 37 .

Method	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )
	Qwen3-1.7B		Qwen3-4B		Qwen3-8B
Base + SFT	20.7	51.6	11.4	63.7	11.1	72.3
Base + SFT (Distill Sonnet-4.6)	20.8	60.4	26.2	69.0	12.6	77.8
+ MT (self-generated) (reas + doc) + SFT (default)	9.2	61.3	5.9	75.0	3.9	78.4
+ MT (self-generated) (reas + doc) + SFT (Distill Sonnet-4.6)	11.9	59.9	9.4	73.7	6.7	78.0

Table 7: Ablation on the SFT data. Per-benchmark detail in Tables 38 and 39 .

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Count
Alpaca ( Taori et al., 2023 )	$52$ k
BeaverTails ( Ji et al., 2023 )	$43$ k
WildJailbreak ( Jiang et al., 2024 )	$23$ k
Dolly-15k ( Conover et al., 2023 )	$15$ k
Competition Math ( Hendrycks et al., 2021 )	$12$ k
PKU-SafeRLHF ( Dai et al., 2024 )	$12$ k

Appendix

Table 8: Per-source breakdown of the mid-train prompt pool (total

\approx 185

k unique prompts). Counts are approximate and rounded to the nearest thousand. Evaluation examples are not used in training.

Figure 5: Safety / utility composition of the mid-train prompt pool. (a) 77% safe vs. 23% unsafe prompts in the union of all unique source prompts. (b) Within the unsafe slice, 75% of prompts are direct harmful requests and 25% are jailbreak-wrapped templated variants.

Item	Self-generated	Sonnet-generated
Generator	base model itself	Claude Sonnet 4.6
Temperature	0.7	0.7
top- $p$	0.95	provider default
Max new tokens	2048	2048

Appendix

Table 9: Document-generation sampling parameters.

Item	Self-generated	Sonnet-generated
Generator	base model itself	Claude Sonnet 4.6
Temperature	0.7	0.7
top- $p$	0.95	provider default
Max new tokens	2048	2048

Appendix

Table 10: Reasoning-trace sampling parameters.

Item	Value
Judge model	Qwen3-32B
Temperature	0.0
Quality threshold	$\geq 4$
Resampling rounds	up to 3

Appendix

Table 11: LLM judge used for mid-training data filtering.

Figure 6: Judge quality-score distribution (Qwen3-32B). Document statistics are from the initial pass before resampling. (a) Approximately 92% of reasoning candidates and (b) 77% of document candidates have quality

\geq 4

. These quality-only fractions do not include the additional requirement that all rubric checks pass.

Figure 7: SFT mix source-data breakdown. Per-source share of each of the three SFT mixes. Safety sources (purple) are WildJailbreak and PKU-SafeRLHF; general sources (teal) are UltraChat-200k, Alpaca, Dolly-15k, and GSM8K.

Hyperparameter	Qwen3-1.7B	Qwen3-4B	Qwen3-8B	Llama-3.2-1B/3B
Epochs	1	1	1	1
Per-device batch	4	4	2	4
Gradient accum.	4	4	8	4
Effective batch	64	64	64	64
Peak LR	2.0e-5	2.0e-5	2.0e-5	2.0e-5
LR schedule	cosine, 5% warm-up	cosine, 5% warm-up	cosine, 5% warm-up	cosine, 5% warm-up

Appendix

Table 12: Mid-training hyperparameters.

Hyperparameter	Qwen3-1.7B / 4B	Qwen3-8B
Epochs	3	3
Per-device batch	4	2
Gradient accum.	4	8
Effective batch	64	64
Peak LR	2.0e-5	2.0e-5
LR schedule	cosine, 5% warm-up	cosine, 5% warm-up

Appendix

Table 13: SFT hyperparameters.

Hyperparameter	Value
Learning rate	1e-6
Per-device batch	1
Gradient accum.	8 (single-GPU) / 4 (2-GPU DDP)
Training prompts	5,000 (1 epoch)
Optimizer	adamw_torch, cosine LR, 5% warm-up
Rollout temperature	0.9

Appendix

Table 14: GRPO hyperparameters ( balanced and pure_general , Qwen3-1.7B and Qwen3-4B).

Model	Reasoning only	Reasoning + doc
Qwen3-1.7B	2h 15m	4h 14m
Qwen3-4B	4h 46m	9h 00m
Qwen3-8B	7h 16m	13h 38m

Appendix

Table 15: Mid-training wall-clock time (1 epoch,

4\times

Pro 6000). Numbers are read from trainer train_runtime fields.

Model	One SFT run
Qwen3-1.7B	$\sim$ 40m
Qwen3-4B	$\sim$ 1h 36m
Qwen3-8B	$\sim$ 1h 49m

Appendix

Table 16: SFT wall-clock time for one run (3 epochs over

9{,}000

training records out of the

10{,}000

-record mix,

4\times

Pro 6000).

Benchmark	Category	Metric
HarmBench ( Mazeika et al., 2024 )	safety, direct	ASR ( $\downarrow$ )
AdvBench ( Zou et al., 2023 )	safety, direct	ASR ( $\downarrow$ )
SimpleSafetyTests ( Vidgen et al., 2023 )	safety, direct	ASR ( $\downarrow$ )
StrongREJECT ( Souly et al., 2024 )	safety, direct	ASR ( $\downarrow$ )
BeaverTails ( Dai et al., 2024 )	safety, direct	ASR ( $\downarrow$ )
WildJailbreak ( Jiang et al., 2024 )	safety, jailbreak	ASR ( $\downarrow$ )

Appendix

Table 17: Benchmarks and metrics.

Benchmark	Scorer	Judge model
HarmBench	LLM-judge judge_harmful	gpt-4.1-mini
AdvBench	LLM-judge judge_harmful	gpt-4.1-mini
JailbreakHub	LLM-judge judge_harmful	gpt-4.1-mini
StrongREJECT	LLM-judge judge_harmful	gpt-4.1-mini
BeaverTails	LLM-judge judge_harmful	gpt-4.1-mini
AIR-Bench	LLM-judge judge_harmful	gpt-4.1-mini

Appendix

Table 18: Per-benchmark scorer. judge_safety_classify is the same function for WildJailbreak / DoNotAnswer / SimpleSafetyTests / XSTest; the function’s target parameter switches its semantics between safety classification (refusal expected) and over-refusal (answer expected).

Panel judge	Agreement vs. gpt-4.1-mini
Qwen3-Next-80B-A3B	97.8
Claude Sonnet 4.6	98.2
GLM-5	98.2
Mean	98.1

Appendix

Table 19: Judge agreement (%,

\uparrow

) of three independent strong judges with gpt-4.1-mini , on a stratified 500-prompt sample drawn across 9s safety benchmarks and 7 source models.

Qwen3-1.7B
Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base (no SFT)	30.8	35.7	54.3	25.7	15.0	31.4	21.7	6.3	52.3	30.4
+ MT (Sonnet-4.6)	23.8	1.0	18.3	20.7	6.0	6.4	6.7	12.0	19.0	12.6
+ MT (self-generated)	9.6	0.3	14.7	17.7	4.0	5.7	2.0	6.0	19.3	8.8
Qwen3-4B
Base (no SFT)	40.0	10.7	76.3	34.0	19.0	44.8	28.7	20.7	61.7	37.3

Appendix

Table 20: Per-benchmark ASR (%,

\downarrow

): Base vs. mid-trained, no SFT. Light-gray = self-generated mid-training.

Qwen3-1.7B
Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base (no SFT)	67.3	48.7	5.0	11.7	33.0	38.7	14.7	23.7	81.7	36.0
+ MT (Sonnet-4.6)	67.0	45.3	41.3	58.0	58.3	73.7	33.7	55.3	97.3	58.9
+ MT (self-generated)	76.0	49.3	44.3	59.0	57.3	76.0	32.7	53.7	87.0	59.5
Qwen3-4B
Base (no SFT)	52.7	42.7	19.7	38.0	54.3	63.3	37.3	28.7	91.7	47.6

Appendix

Table 21: Per-benchmark utility (%,

\uparrow

): Base vs. mid-trained, no SFT. Light-gray = self-generated mid-training.

Qwen3-1.7B
	balanced		pure_safe		pure_general
Method	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )	ASR ( $\downarrow$ )	Utility ( $\uparrow$ )
Base + SFT (Distill Sonnet-4.6)	21.2	60.4	10.6	61.5	42.2	59.8
Base + SFT	22.5	52.2	22.7	52.3	30.3	51.1
SP + SFT	15.2	61.8	11.6	60.9	42.4	60.2
AP + SFT	16.1	61.4	13.1	59.6	33.3	61.1

Appendix

Table 22: Full-suite SFT comparison. Relative to Table 2 , the safety average additionally includes WildJailbreak and BeaverTails, and the utility average additionally includes GSM8K and MATH-500.

Qwen3-1.7B
Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base + SFT (Distill Sonnet-4.6)	19.2	7.0	53.3	21.3	4.0	36.5	7.7	16.7	25.0	21.2
Base + SFT	16.7	3.7	66.3	23.3	9.0	33.1	9.3	16.0	25.0	22.5
+ MT (Sonnet-4.6) + SFT	5.8	0.3	39.7	20.7	2.0	23.4	5.0	12.3	19.0	14.2
SP + SFT	10.8	0.3	37.3	26.7	3.0	21.1	4.0	12.7	21.0	15.2
AP + SFT	11.2	0.0	40.3	25.7	5.0	20.1	4.7	18.7	19.7	16.1

Appendix

Table 23: Per-benchmark ASR (%,

\downarrow

) under SFT mix balanced . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Qwen3-1.7B
Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base + SFT (Distill Sonnet-4.6)	71.3	52.0	37.0	62.7	59.3	78.7	35.3	51.3	95.7	60.4
Base + SFT	56.3	49.7	44.7	43.3	49.0	59.0	27.3	46.7	93.7	52.2
+ MT (Sonnet-4.6) + SFT	77.7	52.7	38.0	61.3	55.3	77.7	34.3	51.3	94.0	60.3
SP + SFT	75.0	49.3	39.7	66.3	61.7	79.3	35.0	51.7	98.3	61.8
AP + SFT	73.7	51.7	40.0	63.3	60.0	74.3	35.0	57.3	97.7	61.4

Appendix

Table 24: Per-benchmark utility (%,

\uparrow

) under SFT mix balanced . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Qwen3-1.7B
Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base + SFT (Distill Sonnet-4.6)	1.7	0.0	24.7	23.7	5.0	15.1	4.0	14.7	6.3	10.6
Base + SFT	15.8	2.0	62.0	27.7	15.0	33.4	14.3	11.3	22.7	22.7
+ MT (Sonnet-4.6) + SFT	0.8	0.7	18.7	18.3	3.0	7.4	4.3	14.0	9.0	8.5
SP + SFT	6.2	0.3	29.0	23.7	3.0	16.4	2.7	11.3	12.0	11.6
AP + SFT	2.9	0.0	37.3	23.0	3.0	16.7	5.0	14.3	16.0	13.1

Appendix

Table 25: Per-benchmark ASR (%,

\downarrow

) under SFT mix pure_safety . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Qwen3-1.7B
Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base + SFT (Distill Sonnet-4.6)	70.0	49.7	47.0	64.3	55.7	78.3	37.0	58.0	93.7	61.5
Base + SFT	64.7	42.7	48.7	39.0	48.7	63.7	27.3	44.3	91.3	52.3
+ MT (Sonnet-4.6) + SFT	74.3	50.7	41.0	59.7	58.0	76.3	36.0	55.0	82.7	59.3
SP + SFT	72.7	51.7	46.0	61.7	55.3	77.0	31.0	54.0	98.3	60.9
AP + SFT	71.3	53.3	44.3	66.3	55.0	72.3	31.0	45.3	97.3	59.6

Appendix

Table 26: Per-benchmark utility (%,

\uparrow

) under SFT mix pure_safety . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Qwen3-1.7B
Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base + SFT (Distill Sonnet-4.6)	55.0	27.7	88.7	29.7	24.0	42.8	29.3	30.7	52.3	42.2
Base + SFT	30.0	5.3	71.3	35.3	23.0	37.8	19.7	17.7	32.7	30.3
+ MT (Sonnet-4.6) + SFT	23.3	2.7	69.7	26.0	8.0	40.8	11.0	23.7	37.0	26.9
SP + SFT	47.9	14.7	87.3	44.7	30.0	37.8	33.0	34.0	52.0	42.4
AP + SFT	33.8	3.3	74.0	42.3	18.0	38.1	19.3	29.7	41.0	33.3

Appendix

Table 27: Per-benchmark ASR (%,

\downarrow

) under SFT mix pure_general . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Qwen3-1.7B
Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base + SFT (Distill Sonnet-4.6)	72.0	52.3	35.7	59.0	56.7	78.7	35.7	49.7	98.3	59.8
Base + SFT	51.3	51.0	46.7	38.7	51.0	63.3	27.0	36.0	94.7	51.1
+ MT (Sonnet-4.6) + SFT	76.0	52.0	38.3	62.0	55.0	81.0	36.0	47.3	97.3	60.6
SP + SFT	73.7	49.7	35.7	61.3	58.7	78.7	34.3	51.3	98.7	60.2
AP + SFT	73.0	52.7	37.3	64.3	58.3	75.3	33.0	57.3	99.0	61.1

Appendix

Table 28: Per-benchmark utility (%,

\uparrow

) under SFT mix pure_general . Light-gray = self-generated mid-training. Avg includes all displayed benchmarks. SP/AP use the same SFT data source as in Table 2 .

Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Full schema (default)	4.6	1.0	20.0	11.7	2.0	10.0	1.7	6.0	15.0	8.0
$-$ intent	7.5	0.3	33.3	18.0	3.0	20.4	0.7	1.7	11.0	10.7
$-$ knowledge	2.5	0.7	32.0	12.0	1.0	21.4	0.7	1.0	9.3	9.0
$-$ consequence	7.5	0.3	40.0	13.3	2.0	30.4	0.7	3.3	13.3	12.3

Appendix

Table 29: Per-benchmark ASR (%,

\downarrow

) for Figure 4 (a) (Qwen3-4B, no SFT).

Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Full schema (default)	90.3	67.3	55.0	80.3	74.0	87.7	45.7	72.3	97.0	74.4
$-$ intent	92.7	65.7	63.3	70.7	73.7	90.3	36.7	69.7	94.7	73.0
$-$ knowledge	92.3	63.0	63.7	77.0	57.0	55.7	32.3	52.0	92.3	65.0
$-$ consequence	88.7	69.7	60.7	74.3	49.0	47.3	28.3	44.0	93.7	61.7

Appendix

Table 30: Per-benchmark utility (%,

\uparrow

) for Figure 4 (a) (Qwen3-4B, no SFT).

	Safety (ASR $\downarrow$ )				XS $\downarrow$	Utility ( $\uparrow$ )
Unsafe ratio	HB $\downarrow$	AB $\downarrow$	WJ $\downarrow$	Avg		GSM $\uparrow$	AE $\uparrow$	MMLU $\uparrow$	Avg
0%	43.8	15.0	81.0	46.6	1.1	88.3	78.3	74.0	80.2
25% (default)	4.6	1.0	20.0	8.5	9.7	90.3	80.3	74.0	81.6
50%	4.2	1.0	21.7	9.0	19.4	87.3	77.0	75.3	79.9
75%	2.1	0.0	12.0	4.7	34.9	89.7	77.0	72.7	79.8
100%	0.0	0.0	0.0	0.0	88.0	71.7	27.7	35.7	45.0

Appendix

Table 31: Mid-train data unsafe-prompt ratio (Qwen3-4B, no SFT). Per-benchmark detail underlying Figure 4 (b). Light-gray = default.

Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base + standard reasoning SFT	34.6	2.7	73.3	27.0	8.0	39.8	9.0	18.3	38.7	27.9
Base + direct SFT on midtrain reasoning data	28.7	0.7	3.3	27.7	9.0	44.8	10.7	17.7	38.0	20.1
+ MT (self-generated reasoning), no SFT	5.0	1.3	28.7	14.0	1.0	9.7	1.3	4.7	20.3	9.6
+ MT (self-generated reasoning) + SFT	4.2	0.7	0.7	12.0	3.0	4.3	0.7	9.0	13.0	5.3

Appendix

Table 32: Per-benchmark ASR (%,

\downarrow

) for Table 3 (Qwen3-4B, balanced SFT mix).

Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base + standard reasoning SFT	85.3	68.3	47.7	80.7	63.0	63.3	44.7	80.3	97.3	70.1
Base + direct SFT on midtrain reasoning data	87.3	62.7	59.7	82.7	61.7	76.0	45.0	63.3	94.0	70.3
+ MT (self-generated reasoning), no SFT	90.7	68.0	58.7	77.7	75.3	89.0	47.0	76.7	98.7	75.7
+ MT (self-generated reasoning) + SFT	88.0	65.7	56.3	80.0	74.7	89.7	47.0	76.3	98.0	75.1

Appendix

Table 33: Per-benchmark utility (%,

\uparrow

) for Table 3 (Qwen3-4B, balanced SFT mix).

Llama-3.2-1B
Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Base	29.6	25.7	13.0	34.3	29.0	23.4	32.0	8.0	45.3	26.7
+ MT (Sonnet-4.6)	13.3	1.3	33.7	30.0	7.0	10.7	6.7	16.7	31.7	16.8
+ MT (self-generated)	9.6	0.0	7.7	22.7	0.0	9.7	4.7	7.0	24.0	9.5
Llama-3.2-3B
Base	49.6	58.3	16.3	27.0	29.0	48.2	28.7	11.3	49.3	35.3

Appendix

Table 34: Per-benchmark ASR (%,

\downarrow

) for Table 5 (Llama-3.2, no SFT).

Llama-3.2-1B
Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Base	0.7	0.0	1.0	0.0	20.0	24.3	13.0	26.0	80.7	18.4
+ MT (Sonnet-4.6)	6.3	2.3	31.0	19.7	4.0	2.0	3.7	9.7	84.3	18.1
+ MT (self-generated)	3.7	0.0	22.3	22.3	17.3	23.3	12.3	26.3	82.3	23.3
Llama-3.2-3B
Base	0.7	0.7	1.7	0.0	21.7	28.0	12.0	26.3	84.3	19.5

Appendix

Table 35: Per-benchmark utility (%,

\uparrow

) for Table 5 (Llama-3.2, no SFT).

Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Qwen3-1.7B
Base + SFT	20.8	3.7	66.6	28.8	15.7	34.8	14.4	15.0	26.8	25.2
Base + SFT (Distill Sonnet-4.6)	25.3	11.6	55.6	24.9	11.0	31.4	13.7	20.7	27.9	24.7
+ MT (self-generated reasoning) + SFT	13.2	0.1	25.6	18.3	5.0	6.9	4.9	11.8	20.6	11.8
+ MT (self reasoning + self doc) + SFT (default)	11.0	0.3	19.6	18.4	3.7	7.6	4.4	12.3	18.7	10.7
+ MT (self reasoning + Sonnet doc) + SFT	11.0	0.1	27.1	20.1	4.7	6.5	4.9	13.8	18.8	11.9

Appendix

Table 36: Per-benchmark ASR (%,

\downarrow

) for Table 6 , mix-averaged.

Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Qwen3-1.7B
Base + SFT	57.4	47.8	46.7	40.3	49.6	62.0	27.2	42.3	93.2	51.8
Base + SFT (Distill Sonnet-4.6)	71.1	51.3	39.9	62.0	57.2	78.6	36.0	53.0	95.9	60.6
+ MT (self-generated reasoning) + SFT	75.3	52.2	40.7	69.0	61.1	78.1	31.0	55.0	94.2	61.9
+ MT (self reasoning + self doc) + SFT (default)	76.8	52.2	43.4	68.7	59.1	76.7	30.9	55.6	94.4	62.0
+ MT (self reasoning + Sonnet doc) + SFT	75.4	52.1	41.7	67.9	61.1	76.9	32.4	55.4	95.6	62.1

Appendix

Table 37: Per-benchmark utility (%,

\uparrow

) for Table 6 , mix-averaged.

Method	HB	AB	WJ	DNA	SST	JBH	SRJ	BT	Air	Avg
Qwen3-1.7B
Base + SFT	20.8	3.7	66.6	28.8	15.7	34.8	14.4	15.0	26.8	25.2
Base + SFT (Distill Sonnet-4.6)	25.3	11.6	55.6	24.9	11.0	31.4	13.7	20.7	27.9	24.7
+ MT (self-generated) (reas + doc) + SFT (default)	11.0	0.3	19.6	18.4	3.7	7.6	4.4	12.3	18.7	10.7
+ MT (self-generated) (reas + doc) + SFT (Distill Sonnet-4.6)	10.0	0.9	37.1	20.3	4.0	21.9	5.1	13.1	21.4	14.9
Qwen3-4B

Appendix

Table 38: Per-benchmark ASR (%,

\downarrow

) for Table 7 , mix-averaged.

Method	GSM	MATH	TQA	AE	MMU	ARC	MMP	HS	BBH	Avg
Qwen3-1.7B
Base + SFT	57.4	47.8	46.7	40.3	49.6	62.0	27.2	42.3	93.2	51.8
Base + SFT (Distill Sonnet-4.6)	71.1	51.3	39.9	62.0	57.2	78.6	36.0	53.0	95.9	60.6
+ MT (self-generated) (reas + doc) + SFT (default)	76.8	52.2	43.4	68.7	59.1	76.7	30.9	55.6	94.4	62.0
+ MT (self-generated) (reas + doc) + SFT (Distill Sonnet-4.6)	77.2	52.9	44.6	62.6	58.2	78.0	33.4	51.4	90.8	61.0
Qwen3-4B

Appendix

Table 39: Per-benchmark utility (%,

\uparrow

) for Table 7 , mix-averaged.

Safety-Utility Trade-Off in Language Models

See all 17 papers

Sep 14, 2026

Inoculation Midtraining with Learned Neologisms

Large language models (LLMs) often learn both desirable and undesirable properties during post-training. We study whether midtraining, an earlier training stage, can shape which of these properties later generalise. We introduce Inoculation Midtraining, a technique that teaches a base model that unsafe behaviour belongs to a designated <quarantine_token> context, as indicated by the <quarantine_token> neologism (a new token) introduced during midtraining, and then post-trains the model on unsafe data within that context. We then evaluate the model outside the context, with the <quarantine_token> neologism excluded from the system prompt. Across supervised fine-tuning and reinforcement learning post-training regimes, we find that Inoculation Midtraining can reduce misalignment while preserving the transfer of benign data properties (e.g., speaking in German or Shakespearean prose). However, our approach does not outperform standard Inoculation Prompting, is sensitive to training configuration, and produces a leaky boundary that nearby contextual cues can reactivate. These results show that inoculation with a learned association introduced via midtraining can shape selective generalisation. Still, more work is needed before this approach can become a load-bearing component in a developer's safety framework.

Kyle O'Brien, Edward James Young, Puria Radmard +4

Geodesic Research · OpenAI · UK AI Security Institute

LLM Alignment LLM Safety LLM Post-Training Safety-Preserving Fine-Tuning of Language Models

Ask PDF

Mar 10, 2026

Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms

Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or post-training to excel at specific tasks. For instance, large reasoning models (LRMs), such as the DeepSeek-R1 series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of-thought (CoT) datasets. However, this additional training frequently comes at the cost of reduced safety, as the fine-tuned or post-trained models tend to exhibit more harmful behaviors compared with the regular LLMs before post-training or fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. But luckily, we also find that LRMs' safety mechanisms still exist instead of being removed during their post-training. Based on these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters on a few layers. Experiments on four state-of-the-art LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach.

Mingjie Li, Wai Man Si, Michael Backes +2

CISPA Helmholtz Center for Information Security · State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

LLM Safety LLM Safety Alignment LLM Post-Training Large Reasoning Models

Ask PDF

Apr 21, 2026

Reasoning Structure Matters for Safety Alignment of Reasoning Models

Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks but often generate harmful responses to malicious user queries. This paper investigates the underlying cause of these safety risks and shows that the issue lies in the reasoning structure itself. Based on this insight, we claim that effective safety alignment can be achieved by altering the reasoning structure. We propose AltTrain, a simple yet effective post training method that explicitly alters the reasoning structure of LRMs. AltTrain is both practical and generalizable, requiring no complex reinforcement learning (RL) training or reward design, only supervised finetuning (SFT) with a lightweight 1K training examples. Experiments across LRM backbones and model sizes demonstrate strong safety alignment, along with robust generalization across reasoning, QA, summarization, and multilingual setting.

Yeonjun In, Wonjoong Kim, Sangwu Park +1

KAIST

LLM Safety Alignment Large Reasoning Models

Ask PDF

Abstract

Safety-Utility Trade-Off in Language Models

Explore similar work

Inoculation Midtraining with Learned Neologisms

Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms

Reasoning Structure Matters for Safety Alignment of Reasoning Models