Group relative policy optimization (GRPO) trains large language models with verifiable rewards by normalizing rewards within response groups. When every response in a group receives the same reward, the advantages vanish and the group provides no policy gradient signal. We propose Adaptive Rollout and Response Reuse Policy Optimization (AR3PO) to obtain informative groups with fewer generations. Adaptive rollout samples responses to each prompt in stages and stops sampling once the model generates a correct response. If all sampled responses are incorrect, response reuse replaces one of them with a past correct response to the same prompt, when available, and recomputes the group advantages. Policy-recomputed reuse trains on the updated groups using probabilities recomputed under the rollout policy; advantage-only reuse uses historical responses only to compute advantages. Across four mathematical benchmarks with 7B and 8B models, AR3PO consistently improves over GRPO and achieves performance comparable to DAPO while requiring up to 4.2× fewer generated responses. Experiments with a 32B model also show comparable performance to DAPO at a similar training stage with substantially fewer generated responses.
Figures & tables
Figure 1: Average number of sampled responses per training prompt. AR3PO uses fewer responses as training progresses, reaching an average of 5.7 and reducing the sampled response count by 4.2× relative to DAPO.
Model
Method
Math 500
Minerva Math
Olympiad Bench
AIME 24
Average
Sampled Responses
Response Count Reduction (vs. DAPO)
Qwen
GRPO
77.5
37.4
38.8
15.2
42.2
512 × 8.0
3.0
DAPO
77.2
36.4
41.1
16.9
42.9
1536 × 8.0
1.0
AR3PO
78.8
36.0
39.6
18.0
43.1
512 × 5.7
4.2
Llama
GRPO
52.6
26.6
19.9
6.7
26.5
512 × 8.0
3.0
DAPO
53.6
28.1
20.3
9.2
27.8
1536 × 8.0
1.0
AR3PO
53.7
26.6
21.2
9.5
27.8
512 × 6.7
3.6
Table 1: Comparison of AR3PO with baselines on four mathematical benchmarks. The penultimate column reports the average number of sampled responses per training step (computed as generation batch size × number of rollouts), and the last column reports the corresponding reduction factor relative to DAPO.
Figure 2: (a) and (b) show the metric curves of average reward and response length for AR3PO. (c) shows the ratio of prompts without any correct response in the group. (d) presents the average sampled responses per step for prompts grouped by cumulative success rate.
Method
Math 500
Minerva Math
Olympiad Bench
AIME 24
Average
AR3PO w/ direct rollout replay
77.3
35.7
39.1
15.0
41.8
AR3PO w/ policy-recomputed reuse
76.9
35.8
39.3
19.3
42.8
AR3PO w/ advantage-only reuse
78.8
36.0
39.6
18.0
43.1
Table 2: Comparison of response reuse strategies with Qwen2.5-7B. Direct rollout replay uses behavior-policy probabilities in the reused response’s importance ratio. The two AR3PO strategies are defined in Section 3.2 .
Figure 3: AIME 2024 accuracy of AR3PO with the Qwen2.5-32B base model. Our method achieves 34.0% accuracy at step 130, comparable to the performance reported by DAPO around 130 training steps (see Figure 1 in Yu et al. (2025) ). Note that their figure reports gradient update steps, with 16 updates per step; therefore, our step 130 roughly corresponds to their reported step 2000.
Group Relative Policy Optimization (GRPO) has been a key driver of recent progress in reinforcement learning with verifiable rewards (RLVR) for large language models, but it is typically trained in a low-staleness, near-on-policy regime that incurs substantial system overhead. We ask a simple question: How off-policy can GRPO be? We show that GRPO-style algorithms can tolerate substantially larger rollout staleness than previously assumed, and propose Mu-GRPO, an RL training framework that organizes training into a small number (e.g., four) of large sequential generation-optimization stages. This design induces high rollout staleness while greatly reducing rollout-optimization switching overhead. To stabilize learning under stale data, Mu-GRPO combines relaxed clipping, which preserves useful stale-rollout gradients, with negative-advantage veto, which removes destabilizing post-trigger suffix updates in negative-advantage responses. Across five language models and multiple math reasoning benchmarks, Mu-GRPO matches or exceeds the performance of standard GRPO while achieving around 2x speedup in wall-clock training time, establishing a substantially improved performance-efficiency trade-off for LLM reinforcement learning.
Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for improving the reasoning abilities of large language models, yet group-based methods lose their learning signal when every sampled response receives zero reward. We propose Group Contrastive Policy Optimization (GCPO), a hybrid objective that preserves standard on-policy optimization for informative rollout groups and activates a proximal contrastive objective only when exploration completely fails. The auxiliary objective pairs an offline golden reasoning trajectory with the current policy's failed responses, thereby providing a reliable positive reference and prompt-specific hard negatives without treating the golden trajectory as an on-policy rollout. Its old-policy token probabilities are evaluated through teacher forcing and used solely to constrain the update, rather than as an importance-sampling correction for the teacher distribution. A first-order analysis shows that the resulting update increases a length-normalized likelihood margin between reliable reasoning and observed failures. Experiments across two model scales and 10 core mathematical reasoning benchmarks demonstrate that GCPO improves training efficiency and final reasoning performance over competitive supervised and RLVR baselines.
Reinforcement learning with verifiable rewards (RLVR) has emerged as a central paradigm for improving the reasoning capabilities of large language models. Group-based policy optimization methods, such as GRPO, typically allocate a fixed number of rollouts to every prompt. This uniform allocation can be inefficient: it over-allocates compute to prompts whose sampled groups are already saturated while under-exploring prompts for which additional samples may reveal useful correct trajectories. To address this limitation, we introduce hit utility, the posterior probability that at least one rollout in a proposed additional allocation for a prompt will be correct. Building on this notion, we propose Hit-Utility Optimal Rollout Allocation (HORA), a learning-free rollout allocation policy that maximizes total posterior hit utility within each allocation batch. HORA adaptively reallocates rollout budgets while leaving the downstream reward evaluation and group-based advantage estimator unchanged. Across two training sets, four mathematical reasoning benchmarks, and three model families at multiple scales, HORA preserves comparable Pass@1 and improves Pass@K over compute-matched GRPO, and it outperforms related allocation baselines at large K. It is also drop-in compatible with other group-based estimators such as RLOO. Ablation studies indicate that the uniform prior used by HORA is competitive with five prompt-conditioned learned-prior alternatives.
Tao Wang, Shuo Li, Yan Sun +2
University of Pennsylvania · Now at Google DeepMind · New Jersey Institute of Technology +1