Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at https://github.com/jameszhu-pixel-time/Z2O.
Figures & tables
Figure 1: Overview of NAPO pipeline
AMC23
AIME2025
Math-Beyond
Training Set
Replay
N → P
P → N
N → P
P → N
N → P
P → N
N → P
P → N
TTFP
n1
2 /3
0.33
2.67 /20
2
12 /134
5.33
120
116
1.213
n2
1.67/3
2.67
2/20
2.33
8.33/134
5.33
117.67
102.67
1.225
Table 1: Transition statistics at t∗=1 for the n1 – n2 replay pair. A prompt is labeled as P if at least one rollout is correct under the fixed rollout budget, and as N otherwise. In the validation columns, a/b reports a N → P transitions among b prompts that are negative at the base checkpoint. Results are averaged over three runs, and TTFP is computed over the first three steps. Best value is shown in bold.
Figure 2: Entropy-position plots. The main separation is localized in the early part of generation. Prompts that later undergo N → P transitions preserve more early-stage uncertainty than prompts that remain N → N, and this pattern is sharper in the good replay n1 than in the bad replay n2 .
Figure 3: Within-batch separation under the practical driver dearly-ppl . Rollouts associated with prompts that later undergo N → P transitions are systematically right-shifted relative to rollouts from prompts that remain N → N. The separation is present in both replays, but is cleaner and more pronounced in the good replay n1 .
Figure 5
Method
Pass@k
k
1
2
4
8
16
32
64
128
256
AMC23
Base
0.2847
0.4512
0.6237
0.7546
0.8400
0.9010
0.9495
0.9838
1.0000
GRPO
0.7352
0.7910
0.8211
0.8443
0.8681
0.8972
0.9306
0.9579
0.9750
W-REINFORCE
0.4514
0.5700
0.6570
0.7245
0.7900
0.8555
0.9086
0.9450
0.9750
NAPO02
0.7141
0.7718
0.8051
0.8328
0.8628
0.8940
0.9222
0.9467
0.9750
Table 2: Pass@k comparison across datasets. The row corresponding to our method is highlighted in gray, the best value within each dataset block and column is shown in bold, and the second-best value is underlined. For W-REINFORCE method, we set the positive advantage weight to 0.5 to handle the much harder prompts.
ΔPass@k (%)
Dataset
k=1
k=8
k=256
AMC23
+14.40
+7.95
+1.88
AIME2025
+10.48
+15.31
+8.33
AIME24
+20.15
+27.28
+17.50
Math-Beyond
+7.43
+8.73
+0.41
Table 3: Pass@k gains of NAPO over Uniform( −0.1 ) across datasets at T=1.0 , averaged over four rollout reevaluations with 256 generations per question. Values are percentage points; positive values favor NAPO.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Setting
Model
Model
Qwen2.5-Math-7B
Data
Train batch size
1024
Max prompt length
1024
Max response length
3072
Appendix
Table 4: Detailed training hyperparameters.
Replay
Training N → P
msk
rt
n1
120
0.924
0.039
n2
117.67
0.812
0.026
Appendix
Table 5: Representative batch-level driver statistics for a good(n1)/bad(n2) replay comparison. Larger msk and rt indicate more mass in the high-driver region.
Figure 6: TTFP distribution
Figure 7: Cumulative solved number
Method (Llama)
Pass@k (%)
k
1
2
8
256
AMC23
NAPO02
40.50
47.68
57.82
80.83
GRPO
37.55
47.29
59.89
80.00
AIME2025
NAPO02
0.06
0.11
0.45
13.33
Appendix
Table 6: Pass@k comparison across datasets for Llama-3.1-8B-Instruct at temperature T=1.0 . Values are percentages. The row corresponding to NAPO02 is highlighted in gray, and the best value between GRPO and NAPO02 within each dataset block and column is shown in bold. Due to limited computational resources, we evaluate the checkpoint at step 100.
Transition
N → P / N → N
msk N → P
msk N → N
Δmsk
1→2
20/133
1.1044
1.0740
+0.0304
2→3
20/135
1.0997
1.0710
+0.0287
3→4
18/144
1.0968
1.0679
+0.0289
4→5
25/138
1.0941
1.0710
+0.0231
Appendix
Table 7: Transition-level entropy statistics for Qwen3-4B-Instruct on LiveCodeBench at α=0.5 . The second column reports the N → P and N → N prompt counts, and the remaining columns report their mean msk values and difference; positive differences indicate higher early relative entropy for N → P.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.