SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning

Zihao Chen; Fanxiang Xiong; Hongran Ren; Xuefeng Bai; Zhongxiang Dai; Kehai Chen; Zhiguo Zhang; Zhiyong Wang

cs.LGSep 29, 2026

SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning

Authors: Zihao Chen, Fanxiang Xiong, Hongran Ren, Xuefeng Bai, Zhongxiang Dai, Kehai Chen, Zhiguo Zhang, Zhiyong Wang, +1 more

Organizations: Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong, Shenzhen · Nanyang Technological University

Abstract

Maximum Likelihood Reinforcement Learning (MaxRL) targets prompt-wise log-success and has shown strong performance on reasoning tasks. Under finite rollout budgets, however, the estimator used by MaxRL attenuates each prompt's likelihood gradient by a factor that depends on its success probability and rollout count. Under uniform rollout allocation, the common rollout count fails to compensate for success-dependent attenuation, leaving low-success prompts more strongly attenuated and distorting their relative contributions to the expected aggregate gradient. We introduce SERA (Scale-Equalized Rollout Allocation), which redistributes a fixed rollout budget to approximately equalize these finite-rollout scaling factors. Building on our theoretical analysis of how finite rollouts distort prompt-wise likelihood gradients, we formulate the allocation as a fixed-budget max--min problem, derive a waterline solution to its continuous relaxation, and introduce a multiplicity correction to remove the additional prompt weighting induced by heterogeneous rollout counts. Experiments show stronger alignment with exact likelihood gradients in a controlled ImageNet setting and improved multi-sample solution coverage over MaxRL on maze navigation and mathematical reasoning under matched training rollout budgets.

Figures & tables

Method	Pass@1	Pass@32	Pass@128	Pass@256
Base model	6.80	53.85	76.32	84.37
GRPO ( Shao et al., 2024 )	30.46	45.80	49.28	50.62
PKPO (T=16) ( Walder and Karkhanis, 2025 )	19.93	70.32	84.92	90.16
MaxRL (token-mean) ( Tajwar et al., 2026 )	32.60	59.41	68.44	72.21
MaxRL (Seqnorm) ( Tajwar et al., 2026 )	32.08	56.66	64.58	68.24
DARS-HW † ( Yang et al., 2025b )	34.25	52.38	57.31	59.06

Table 1: SmolLM2-360M-Instruct on GSM8K-Platinum. All RL methods are evaluated at step

2{,}000

, with the base model as the pre-RL reference. Values are percentages;

\Delta

reports percentage-point changes relative to MaxRL under the same normalization.

\dagger

: DARS-HW uses

1.49\times

as many training rollouts as the other methods.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

$N_{0}$	Method	$1$	$2$	$4$	$8$	$16$	$32$	$64$	$128$	$256$	$512$	$1024$
–	Direct CE	60.81	70.26	77.38	82.78	86.95	90.18	92.68	94.59	96.05	97.16	98.00
$16$	MaxRL	4.43	4.97	5.37	5.71	6.05	6.47	7.13	8.24	10.21	13.67	19.35
$16$	SERA	12.37	13.61	14.54	15.26	15.82	16.25	16.59	16.86	17.09	17.29	17.49
$128$	MaxRL	36.59	41.30	44.79	47.43	49.45	51.01	52.24	53.22	54.00	54.62	55.14
$128$	SERA	54.80	63.53	70.26	75.46	79.52	82.75	85.37	87.51	89.25	90.65	91.79

Appendix

Table 2: Final ImageNet Pass@

K

(%). Results at step

100{,}000

, corresponding to Figure 2 (c). Bold indicates the higher value within each matched-budget MaxRL– SERA pair. Direct CE provides the exact-likelihood training reference.

Estimator	$m$	MAE $\downarrow$	Pearson $r$ $\uparrow$
Pilot estimate	$8$	0.0634	0.9879
Historical estimate	$0$	0.0386	0.8134

Appendix

Table 3: Success-probability estimation on ImageNet. Batch-level metrics averaged over the

5{,}005

batches in epoch

20

(

N_{0}=16

, nominal

B=256

, seed

69

m

denotes current-batch pilot samples per prompt; pilot estimates use

\operatorname{clip}(K/8,1/16,15/16)

, where

K

counts pilot successes. The runs have different allocation configurations and training trajectories.

Normalization	Method	$1$	$2$	$4$	$8$	$16$	$32$
–	Base model	1.27	2.49	4.75	8.83	15.50	25.17
Token-mean	GRPO	34.00	34.80	35.43	36.01	36.61	37.28
Token-mean	MaxRL	52.95	59.16	63.16	66.16	68.62	70.75
Token-mean	SERA	49.98	58.19	63.85	68.06	71.49	74.45
Seqnorm	MaxRL	52.80	59.42	63.74	66.96	69.63	71.94
Seqnorm	SERA	55.61	61.66	65.93	69.45	72.43	74.97

Appendix

Table 4: Final Maze Pass@

K

(%). RL results are evaluated at step

3{,}000

, using

2{,}048

responses per held-out maze. The gray row reports the pre-RL base-model reference at step

0

. Bold indicates the higher value within each MaxRL– SERA pair using the same normalization.

	Pass@32	Pass@128	Pass@256
$\Delta$	$+4.69$	$+5.54$	$+5.13$
$95\%$ CI	$[2.32,\,7.04]$	$[3.09,\,8.01]$	$[2.48,\,7.86]$

Appendix

Table 5: SmolLM2 uncertainty analysis.

\Delta

denotes SERA minus MaxRL at step

2{,}000

, in percentage points. CI denotes the question-paired

95\%

confidence interval.

Dataset	Method	$1$	$2$	$4$	$8$	$16$	$32$	$64$	$128$	$256$	$512$
BeyondAIME	Base	0.73	1.41	2.61	4.57	7.36	10.96	15.49	21.20	28.13	35.00
	MaxRL	3.29	5.25	7.71	10.82	14.82	19.53	24.78	30.53	36.53	42.00
	SERA	3.07	5.06	7.64	10.82	14.76	19.55	25.36	32.26	40.14	49.00
AIME 2025	Base	3.26	6.01	10.36	16.17	22.59	29.21	36.42	44.21	53.12	63.33
	MaxRL	10.05	15.64	21.65	27.37	33.02	38.86	44.49	49.91	56.26	63.33
	SERA	10.18	16.28	23.04	29.19	34.77	40.46	46.30	52.85	61.26	70.00

Appendix

Table 6: Qwen2.5-Math-1.5B Pass@

K

(%). MaxRL and SERA are evaluated at step

600

using

512

sampled responses per question. Gray rows show the pretrained base model under the same protocol. Bold indicates the higher unrounded value between MaxRL and SERA for each benchmark and

K

Dataset	Pass@32	Pass@128	Pass@256	Pass@512
BeyondAIME	$+0.01$ $[-1.59,\,1.56]$	$+1.72$ $[-1.50,\,4.98]$	$+3.60$ $[-1.21,\,8.50]$	$+7.00$ $[0.00,\,15.00]$
AIME 2025	$+1.61$ $[0.12,\,3.11]$	$+2.94$ $[-0.82,\,7.27]$	$+5.00$ $[-0.21,\,11.67]$	$+6.67$ $[0.00,\,16.67]$
MATH-500	$+0.80$ $[0.23,\,1.43]$	$+0.86$ $[0.02,\,1.77]$	$+0.63$ $[-0.26,\,1.58]$	$+0.20$ $[-0.80,\,1.20]$
OlympiadBench	$+1.45$ $[0.86,\,2.07]$	$+1.64$ $[0.81,\,2.51]$	$+1.33$ $[0.19,\,2.50]$	$+1.04$ $[-0.59,\,2.81]$

Appendix

Table 7: Qwen2.5-Math-1.5B uncertainty. Each cell reports the difference between SERA and MaxRL at step 600 (top), with its pointwise

95\%

paired-bootstrap confidence interval (bottom), in percentage points. Differences are computed from unrounded scores before rounding.

Dataset	Method	$1$	$2$	$4$	$8$	$16$	$32$	$64$	$128$	$256$	$512$
BeyondAIME	Base	4.02	6.16	8.51	11.08	14.33	18.61	23.90	30.35	37.75	45.00
	MaxRL	6.97	10.53	14.46	18.62	23.28	28.50	34.08	40.27	47.50	55.00
	SERA	7.78	11.34	15.78	21.08	27.17	34.01	41.39	48.81	55.91	63.00
AIME 2025	Base	6.82	11.78	18.33	25.23	31.48	37.23	43.08	49.90	57.41	63.33
	MaxRL	16.37	22.75	28.50	33.84	39.43	45.34	51.72	59.17	66.77	73.33
	SERA	18.37	23.85	29.30	35.38	41.88	48.40	54.97	61.20	67.61	73.33

Appendix

Table 8: Qwen3-4B-Base Pass@

K

(%). Gray rows show Base; bold marks the higher unrounded score between MaxRL and SERA .

Dataset	Pass@32	Pass@128	Pass@256	Pass@512
BeyondAIME	$+5.51$ $[1.69,\,9.54]$	$+8.54$ $[3.42,\,13.78]$	$+8.41$ $[2.19,\,14.74]$	$+8.00$ $[0.00,\,17.00]$
AIME 2025	$+3.06$ $[-5.74,\,11.67]$	$+2.03$ $[-9.04,\,13.86]$	$+0.84$ $[-9.70,\,11.88]$	$+0.00$ $[-10.00,\,10.00]$
MATH-500	$+0.72$ $[-0.05,\,1.55]$	$+0.00$ $[-0.67,\,0.60]$	$-0.21$ $[-0.89,\,0.39]$	$-0.40$ $[-1.20,\,0.40]$
Minerva Math	$+0.91$ $[-1.57,\,3.47]$	$+1.77$ $[-0.50,\,4.14]$	$+3.03$ $[0.60,\,5.56]$	$+4.78$ $[1.84,\,8.09]$

Appendix

Table 9: Qwen3-4B-Base uncertainty. SERA minus MaxRL in percentage points (top), with pointwise

95\%

paired-bootstrap intervals (bottom). Differences are computed from unrounded scores before rounding.

Explore similar work

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
Jul 28, 2026Pixel Nomand, Elena Voss, Marcus Hale +1Reinforcement Learning With Verifiable Reward Token Budget Allocation
University of Wisconsin–Madison · University of Washington
Cross-Epoch Adaptive Rollout Optimization for RL Post-Training
Jun 4, 2026Yiming Zong, Yige Wang, Jiashuo Jiang
Department of Industrial Engineering & Decision Analytics, Hong Kong University of Science and Technology
RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood
May 28, 2026Yifu ZhengSurrogate Gradient Maximum Likelihood
University of Southern California

Jul 28, 2026, cs.LGJ/K move · Enter open · S save

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes

University of Wisconsin–Madison · University of Washington

Reinforcement Learning With Verifiable Reward Token Budget Allocation Policy Gradient Verdict

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Abstract

Explore similar work

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Cross-Epoch Adaptive Rollout Optimization for RL Post-Training

RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood