Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, methods such as GRPO and DAPO suffer from substantial computational cost, since they rely on sampling many rollouts for each prompt. Moreover, in RLVR the relative advantage is often sparse: many samples become nearly all-correct or all-incorrect, yielding low within-group reward variance and thus weak learning signals. In this paper, we introduce arrol (Accelerating RLVR via online Rollout Pruning), an online rollout pruning method that prunes rollouts during generation while explicitly steering the surviving ones more correctness-balanced to enhance learning signals. Specifically, arrol trains a lightweight quality head on-the-fly to predict the success probability of partial rollouts and uses it to make early pruning decisions. The learned quality head can further weigh candidates to improve inference accuracy during test-time scaling. To improve efficiency, we present a system design that prunes rollouts inside the inference engine and re-batches the remaining ones for log-probability computation and policy updates. Across GRPO and DAPO on Qwen-3 and LLaMA-3.2 models (1B-8B), arrol improves average accuracy by +2.30 to +2.99 while achieving up to 1.7x training speedup, and yielding up to +8.33 additional gains in average accuracy in test-time scaling. The code is available at https://github.com/Hsu1023/ARRoL.
Figures & tables
Figure 1: ARRoL overview and results. (a) ARRoL uses a quality head to score partial rollouts, enabling early pruning for efficient and reward-balanced training, and the scores can also be used as voting weights for test-time scaling. (b) Wall-clock time comparison between ARRoL and GRPO across different model backbones, showing consistent speedups. (c) Accuracy comparison, where ARRoL improves average accuracy over GRPO.
Figure 2: (a) Trace Confidence Failure Modes: Reflection-related tokens tend to receive low confidence despite being beneficial, whereas formula-heavy tokens can receive high confidence even under incorrect reasoning. (b) Distribution Comparison. Trace confidence in (b.1) is less separable between correct/incorrect than quality head scores in (b.2). (c) Correlation Comparison. Quality-head scores achieve consistently higher correlation, measured by Spearman rank correlation between the predicted scores (quality scores or trace confidence) and the binary correctness of final answers on the Math500 and Dapo17k datasets. (d) Generation Length v.s. Correlation & Time Cost. The time cost increases as the generation length increases, while the correlation plateaus when the length reaches 512. All the data is generated by Qwen3-4B model on 400 prompts from Dapo17k and Math500 dataset with 10 rollouts per sample.
Figure 3: Illustration of System Design.
Method
Math500
Minervamath
OlympiadBench
AMC’23
AIME’24
AIME’25
Avg
Speedup
Qwen-3-1.7B-Base
GRPO
60.89
17.65
18.55
75.00
20.00
16.67
34.79
-
+ ARRoL
62.30
16.91
20.81
82.50
23.33
16.67
37.09
1.61 ×
Qwen-3-4B-Base
GRPO
79.64
30.88
31.37
87.50
36.67
40.00
51.01
-
+ ARRoL
80.04
28.67
33.33
92.50
40.00
46.67
53.54
1.63 ×
Table 1: Performance Comparison on GRPO. We compare our method with vanilla GRPO on six benchmarks across four models, and also report speedup.
Method
AMC’23
AIME’24
AIME’25
Qwen-3-1.7B-Base
Majority
55.0
16.7
3.3
Deepconf
57.5
16.7
6.7
ARRoL
60.0
23.3
13.3
Qwen-3-4B-Base
Majority
72.5
26.7
20.0
Table 3: Performance Comparison of Test-time Voting. We compare our method with GRPO and Deepconf method on three benchmarks across three models.
Method
AMC23
AIME24
AIME25
E[ρ^]
E[ρ^(1−ρ^)]
Qwen-3-4B-Base
Random
79.34
26.47
31.98
0.32
0.21
ARRoL
80.04
28.67
33.33
0.40
0.23
LLama-3.2-1B-instruct
Random
22.18
2.94
3.02
0.14
0.11
ARRoL
29.03
4.04
4.37
0.23
0.14
Table 4: Performance Comparison against Random Pruning. ρ^ is the fraction of positive (reward=1) rollouts during training. For binary rewards, ρ^(1−ρ^) is proportional to the within-group reward variance and is maximized at ρ^=0.5 .
Time/s
Generation
Logprob
Update
GRPO
106.82
18.40
63.05
ARRoL
72.96 (1.46 × )
10.02 (1.84 × )
30.26 (2.08 × )
Table 5: Efficiency decomposition for Qwen-3-1.7B-Base across training phases: rollout generation, log-probability computation, and model update.
κ
0.25
0.5
0.75
1
Avg Acc
32.46
33.34
32.68
32.36
Speedup
2.33 ×
1.61 ×
1.17 ×
1.00 ×
Table 6: Average accuracy on Math500, MinervaMath, and OlympiadBench, and training speedup under different rollout keep ratios κ for Qwen-3-1.7B-Base.
Figure 4: Wall-clock convergence of Qwen-3-1.7B-Base training.
TCS Research Department of CSE IIT Madras Chennai, India · Department of Data Science & AI, Wadhwani School of Data Science & AI IIT Madras Chennai, India