Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for improving the reasoning capabilities of large language models. However, existing group-based RLVR methods often suffer from severe sample inefficiency. This inefficiency stems from reliance on point estimation of rewards from a small number of rollouts, leading to high estimation variance, variance collapse, and ineffective utilization of generated responses. In this work, we reformulate RLVR from a statistical estimation perspective by modeling rewards as samples drawn from a policy-induced distribution and casting advantage computation as the problem of estimating the reward distribution from finite data. Building on this view, we propose Discounted Beta-Bernoulli (DBB) reward estimation, which leverages historical reward statistics for the non-stationary distribution. Although biased, the resulting estimator exhibits reduced and stable variance, theoretically avoids estimated variance collapse, and achieves lower mean squared error than standard point estimation. Extensive experiments across six in-distribution and three out-of-distribution reasoning benchmarks demonstrate that GRPO with DBB consistently outperforms naive GRPO, achieving average Acc@8 improvements of 3.22/2.42 points in-distribution and 12.49/6.92 points out-of-distribution on the 1.7B and 8B models, respectively, without additional computational cost or memory usage.
Figures & tables
Figure 1: Comparison between point estimation and DBB estimation. By trading a small bias for substantial variance reduction via shrinkage, DBB estimation achieves lower mean squared error. Compared to naive GRPO using point estimation, GRPO with DBB estimation consistently demonstrates superior performance across all benchmarks and both model scales.
In-Distribution
Out-of-Distribution
Method
MATH500
Minerva
AIME24
AIME25
AMC24
Olympiad
Avg.
MMLU-Pro
GPQA-D
BBH
Avg.
Qwen3-8B-Base trained with DAPO-Math-17k
GRPO
87.87 ± 0.36
39.34 ± 0.29
29.17 ± 0.68
25.31 ± 2.16
57.71 ± 1.44
54.54 ± 0.19
48.99 ± 0.46
50.16 ± 0.33
45.27 ± 1.69
59.18 ± 0.33
51.54 ± 0.55
RePO
86.22 ± 0.31
35.02 ± 0.47
27.29 ± 0.80
23.44 ± 1.04
56.25 ± 0.70
52.70 ± 0.44
46.82 ± 0.07
53.29 ± 0.55
46.34 ± 1.74
42.44 ± 0.30
47.35 ± 0.80
GRESO
87.01 ± 0.20
33.88 ± 0.60
27.36 ± 0.79
22.78 ± 1.09
57.31 ± 0.57
52.81 ± 0.17
46.86 ± 0.12
62.81 ± 0.39
48.53 ± 0.43
65.17 ± 0.63
58.83 ± 0.40
DAPO
86.00 ± 0.07
37.08 ± 0.30
31.06 ± 1.19
21.02 ± 1.14
52.67 ± 0.92
51.64 ± 0.14
46.58 ± 0.16
59.28 ± 0.67
45.05 ± 1.79
63.41 ± 0.29
55.91 ± 0.46
Table 1: In-distribution (ID) and out-of-distribution (OOD) evaluation results trained on DAPO-Math-17k. For a fair comparison, all methods are trained with the same total number of rollouts. Evaluation is conducted over four independent runs with Acc@8 per run (i.e., 32 responses per prompt in total); each cell reports mean ± standard deviation across the four runs. Δ rows report the absolute Acc@8 improvement of GRPO-DBB over each baseline together with a one-sided paired t -test p -value ( H1 : GRPO-DBB > baseline).
Method
MATH500
Minerva
AIME24
AIME25
AMC24
Olympiad
Avg.
Qwen3-8B-Base trained with DAPO-Math-17k
λ=1.0
89.08
37.91
33.33
26.25
59.44
55.19
50.20
λ=0.75
88.71
39.46
33.60
29.02
60.54
56.53
51.31
λ=0.5
89.35
38.56
33.33
27.08
59.17
56.19
50.61
λ=0.25
88.82
39.98
30.83
25.83
60.83
56.34
50.44
GRPO
87.87
39.34
29.17
25.31
57.71
54.54
48.99
Table 2: Effect of the discount factor λ of GRPO-DBB on in-distribution Acc@8 performance. Qwen3-8B-Base achieves the best performance at λ=0.75 , while Qwen3-1.7B-Base performs best at λ=0.5 .
Figure 2: Training dynamics of naive GRPO and GRPO with the DBB estimator (GRPO-DBB) on Qwen3-1.7B-Base (top) and Qwen3-8B-Base (bottom). GRPO-DBB achieves higher validation Acc@8 and training rewards, while maintaining longer responses with controlled entropy compared to GRPO, indicating more stable exploration during training.
Figure 3: MSE as a function of the discount factor λ and the number of rollouts N . The DBB estimator yields lower MSE than the point estimator across a wide range of λ , and it achieves lower MSE for rollout budgets up to N=16 when λ=0.4 .
Method
MATH500
Minerva
AIME24
AIME25
AMC24
Olympiad
Avg.
Qwen3-1.7B-Base trained with DAPO-Math-17k
Dr.GRPO
69.00
25.14
9.17
7.92
23.89
32.78
27.98
Dr.GRPO-DBB
70.60
26.15
12.08
9.17
28.06
33.59
29.94
Δ w.r.t. Dr.GRPO
+1.60
+1.01
+2.92
+1.25
+4.17
+0.81
+1.96
Table 3: In-distribution evaluation results on Qwen3-1.7B-Base trained with DAPO-Math-17k. We apply the proposed discounted Beta–Bernoulli reward distribution estimation to the advantage term of Dr.GRPO and observe consistent performance improvements. Δ denotes the absolute Acc@8 improvement over naive Dr.GRPO.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Point Estimation
DBB Estimation
Xτ,i
Xτ,i∼Bern(pτ)
Xτ,i∼Bern(pτ)
pτ
fixed but unknown
pτ∼Beta(ατ,βτ)
p^τ
p^τpt=N1i=1∑NXτ,i
p^τdbb=ατ+βτατ
Var(Xτ)
N−1Np^τpt(1−p^τpt)
p^τdbb(1−p^τdbb)
E[p^τ]
pτ
wμτ−1+(1−w)pτ
Bias(p^τ)
0
w(μτ−1−pτ)
Appendix
Table 4: Comparison between point estimation and DBB estimation for reward distribution modeling. We denote the Bernoulli distribution by Bern(⋅) .
Hyperparameter
Qwen3-1.7B-Base
Qwen3-8B-Base
Training Configuration
Training batch size
128
128
Mini-batch size
64
64
Number of epochs
4
4
Total gradient steps
1080
1080
Samples per prompt
8
8
Appendix
Table 5: Common training hyperparameter settings for all experiments.
Figure 4: Training dynamics of Qwen3-1.7B-Base with GRPO and GRPO-DBB under clipping ranges (0.2,0.28) and (0.98,0.98) . From left to right: (a) validation Acc@8, (b) policy-gradient clip fraction, (c) gradient norm, and (d) policy-gradient loss. GRPO-DBB suffers excessive clipping under the standard range; once the range is widened, it recovers strong performance. GRPO under the wider range shows no improvement, confirming that the gains of GRPO-DBB are due to the DBB estimator rather than the clip range.
Figure 5: (a) Heatmap of minλMSE of the DBB estimator over (α0,β0) . (b) MSE versus λ for selected priors. The default symmetric prior (α0,β0)=(1,1) is competitive with the best prior and robust over a wide range of λ .
Figure 6: Validation performance of Qwen3-1.7B-Base (top) and Qwen3-8B-Base (bottom) trained with GRPO and GRPO-DBB. The left column shows Mean@8 (= Acc@8) and the right column shows Best@8 on the validation set. Despite producing lower entropy during training, GRPO-DBB matches or exceeds GRPO in Best@8 across both model scales.
Method
MATH500
Minerva
AIME24
AIME25
AMC24
Avg.
Qwen3-0.6B-Base trained with DAPO-Math-17k
λ=1.0
47.12
14.25
2.92
0.00
10.83
15.02
λ=0.75
48.05
14.52
2.08
0.42
10.28
15.07
λ=0.5
47.95
14.11
2.08
1.25
11.94
15.47
λ=0.25
49.98
14.15
3.33
1.67
8.61
15.55
Appendix
Table 6: Effect of the discount factor λ of GRPO-DBB on in-distribution Acc@8 performance for Qwen3-0.6B-Base trained on DAPO-Math-17k. The best average performance is attained at λ=0.25 .
Figure 7: MSE versus λ for per-prompt exponential moving average (EMA), Laplace smoothing (Laplace), point estimator (GRPO), and DBB estimator (GRPO-DBB). The DBB estimator attains the lowest minimum MSE at λ=0.4 and improves on the point estimator over a broader range of λ than EMA or Laplace.
Figure 8: MSE versus λ for the Empirical Bayes estimator with online method-of-moments (EB), point estimator (GRPO), and DBB estimator (GRPO-DBB). EB improves over the point estimator without per-prompt tuning, but DBB attains a lower minimum MSE.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.
Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models. However, verifiable outcome rewards provide an imperfect measure of reasoning quality: a correct final answer may not be supported by the reasoning trace, whereas an incorrect rollout may contain a nearly correct reasoning trajectory. We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency. To address this credit assignment challenge, this paper proposes double-rollout reinforcement learning. For each sampled rollout, we truncate its reasoning trace, sample multiple continuations from the truncated prefix, and replace the original reward with the mean continuation reward, yielding a more robust estimate of reasoning quality. This double-rollout algorithm incurs only a moderate increase in compute cost and yet reaches substantially higher accuracies across math and coding tasks and across Qwen and Nemotron models than baseline algorithms with fewer RL steps. For example, using only DAPO data it trains Nemotron-3-Nano-30B-A3B to reach 95.4% on AIME 2026, versus 91.5% by GRPO and substantially above state of the art for this model size, within 30 RL steps. When combined with length-based rewards, the same algorithm further improves the accuracy-efficiency Pareto frontier over existing efficiency optimizations. These findings suggest that the double-rollout reward estimation is an effective and cost-efficient way to improve both the accuracy and token efficiency of reasoning RL.
Haifeng Qian, David Mosallanezhad, Christian Munley +2
NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051, USA
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models but incurs substantial costs from rollouts and policy updates. Online prompt selection improves efficiency by using per-prompt Bayesian posteriors to predict difficulty and prioritize informative prompts. However, existing methods overlook how reliably learning signals are extracted from sampled responses. In GRPO, a response's advantage depends on both its own outcome and the randomly sampled outcomes of its peers through group normalization. Our theoretical and experimental analyses show that uncertainty in group composition introduces composition noise, a non-vanishing variance component that imposes an irreducible lower bound on gradient estimation error and impairs downstream prompt selection. We propose MaPP (Marginalized Posterior-Predictive), a unified framework for data-efficient RLVR that denoises response-level advantage estimation and improves prompt selection using a shared Beta posterior. For each response, MaPP replaces the standard group-relative advantage with a composition-invariant intrinsic advantage through closed-form Beta-Binomial marginalization. The resulting posterior-predictive estimator has an error that provably diminishes as the posterior concentrates. Using the same posterior, MaPP derives an uncertainty-aware prompt selection score to improve data efficiency without additional rollout cost. Experiments on mathematics, planning, and visual geometry across five model backbones show that MaPP consistently outperforms GRPO and strong selection baselines, achieving up to +2.45 average accuracy improvement over the strongest baseline under the same rollout budget and setting a new state of the art.
Yangyang Ren, Haodong Zhu, Sheng Xu +4
Beihang University · Zhongguancun Academy · Communication University of China +4