Organizations: East China Normal University · Baidu Inc. · Zhongguancun Academy · CFAR, A*STAR · Stevens Institute of Technology · Beihang University · Security Capability Center · Tsinghua University
Large reasoning models (LRMs) exhibit strong reasoning capabilities but remain vulnerable to unsafe outputs. While Group Relative Policy Optimization (GRPO) is widely used for safety alignment, its group-relative optimization may provide insufficient safety-directed learning signals. In this paper, we identify two failure modes: advantage collapse, where homogeneous rollout groups yield vanishing advantages, and unsafe diversification, where unsafe behaviors persist and become increasingly diverse. Our theoretical analysis reveals that behavioral consistency weakens within-group safety contrasts, while entropy regularization can diversify unsafe responses when relative signals vanish. To address these challenges, we propose Beacon, comprising Safety Signal Recovery (SSR) and Safety-Utility Decoupled Alignment (SUDA). SSR combines advantage-guided trajectory filtering and safe reference supervision to recover effective safety learning signals, while SUDA employs category-aware rewards and benign-only KL anchoring to mitigate over-refusal without compromising safety. Experiments across four Qwen3 and DeepSeek-R1-Distill variants and eight benchmarks show that our framework achieves the best average safety and reasoning performance, reducing average safety error by up to 32.83 percentage points while maintaining low over-refusal.
Figures & tables
Figure 1: Two failure modes of vanilla GRPO we revealed in LRM safety alignment. (a) Advantage collapse: half of eight-response groups are entirely unsafe, yielding zero group-relative advantages. (b) Unsafe diversification: on challenging harmful prompts, unsafe response rates remain high while the semantic entropy of unsafe responses increases throughout training.
Figure 2: Overview of our proposed framework for effective LRM safety alignment. Safety Signal Recovery (SSR) strengthens safety learning through advantage-guided trajectory filtering, which prioritizes high-contrast rollouts, and safe reference supervision, which provides explicit safety guidance when relative advantages collapse. Safety-Utility Decoupled Alignment (SUDA) combines category-aware reward design with benign-only KL anchoring to improve safety on harmful prompts while effectively mitigating unnecessary over-refusal on benign prompts.
Model
Method
Safety ( ↓ )
Reasoning ( ↑ )
HarmBench
StrongREJECT
WildJailbreak
XSTest
Avg.
AIME24
GSM8K
MATH500
GPQA
Avg.
Initial
38.21
8.31
43.00
0.80
22.58
67.50
84.69
93.43
52.27
74.47
DirectRefusal
33.06
3.19
36.20
32.00
26.11
68.33
82.58
93.20
53.03
74.29
SafeChain
43.69
41.21
39.65
2.00
31.64
62.08
89.59
93.03
51.01
73.93
STAR-1
33.72
5.75
35.15
6.80
20.36
62.50
90.97
93.05
51.96
74.62
SafePath
37.71
7.35
42.45
1.60
22.28
72.08
84.45
93.33
53.54
75.85
Table 1: Safety and reasoning performance across LRMs. Safety is measured by harmful response rates (%, ↓ ) on three benchmarks and over-refusal on XSTest. Reasoning is measured by pass@1 (%, ↑ ) on four benchmarks.
Method
Safety ( ↓ )
Reasoning ( ↑ )
HarmBench
StrongREJECT
WildJailbreak
XSTest
Avg.
AIME24
GSM8K
MATH500
GPQA
Avg.
DPO
31.89
2.88
34.90
2.00
17.92
76.67
85.76
93.75
59.53
78.93
GRPO
27.74
2.24
30.10
2.00
15.52
73.33
85.01
93.63
60.29
78.07
SFT+GRPO
23.24
1.89
23.24
1.60
12.49
73.25
86.78
93.84
60.94
78.70
GDPO
28.74
2.56
31.50
1.60
16.10
70.83
84.94
94.00
60.48
77.56
NGRPO
27.08
2.56
32.30
2.40
16.09
73.33
85.29
94.10
61.17
78.47
Table 2: Comparison with optimization methods, including methods addressing advantage collapse. Safety is evaluated on three harmful benchmarks and XSTest (%, ↓ ), and reasoning by pass@1 on four benchmarks (%, ↑ ).
Figure 3: Comparison of GRPO variants throughout training. (a) Advantage collapse: proportion of all-unsafe groups with near-zero advantages. (b) Unsafe persistence: proportion of all-unsafe groups. (c) Unsafe diversification: semantic entropy of unsafe responses.
Figure 4: Performance comparison across Qwen3 model scales. (a) Safety: average safety error across four benchmarks (lower is better). (b) Reasoning: average pass@1 across four benchmarks (higher is better).
Figure 5: Robustness comparison against four adversarial attacks: GCG, PAIR, AutoDAN, and TAP. Attack success rates (%, ↓ ) are evaluated on HarmBench.
Method
Llama-Guard-3
GPT-6
Human
S ↓
R ↑
S ↓
R ↑
S ↓
R ↑
Initial
19.57
76.08
20.45
77.29
19.78
76.46
DirectRefusal
23.00
77.98
25.32
78.46
24.31
80.23
SafeChain
29.44
78.68
30.13
79.69
28.49
80.31
STAR-1
15.44
78.59
17.31
79.45
16.54
78.49
SafePath
20.64
78.64
19.48
79.43
18.57
77.88
Table 3: Evaluator consistency on Qwen3-8B. Safety includes harmful response rates and over-refusal (%, ↓ ), while reasoning is measured by pass@1 (%, ↑ ).
SSR
SUDA
Safety ( ↓ )
XSTest ( ↓ )
Reasoning ( ↑ )
Safe Rollout (%)
AGTF
SRS
CAR
BKL
✓
–
–
–
5.16
10.20
78.26
24.34
✓
✓
–
–
2.36
14.40
78.89
49.74
–
✓
✓
✓
5.42
1.40
79.21
50.46
✓
–
✓
✓
4.68
1.60
79.45
50.38
✓
✓
–
✓
2.04
7.80
79.96
50.32
Table 4: Ablation study of the four key components of our framework: AGTF, SRS, CAR, and BKL.
Figure 6: Sensitivity analysis of trajectory retention ratio κ across three Qwen3 model scales. We report changes in safety error (%, ↑ ) and reasoning accuracy (%, ↑ ) relative to κ=1.00 , which retains all trajectories.