Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPO

Mei Okonkwo; Pixel Nomand; Julian Berg; Elena Voss; Lena Park; Marcus Hale; Adrian Cho; Sofia Reyes

cs.LGSep 29, 2026

Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPO

Authors: Mei Okonkwo, Pixel Nomand, Julian Berg, Elena Voss, Lena Park, Marcus Hale, Adrian Cho, Sofia Reyes

Organizations: University of Wisconsin–Madison · University of Washington

Abstract

Group relative policy optimization (GRPO) learns only from prompts whose sampled responses disagree: a group that is entirely correct or entirely incorrect has zero reward variance, contributes no gradient, and still consumes its rollouts. Prompt-selection methods reduce this waste by steering sampling toward intermediate pass rates, but they choose the target, its width, and the uncertainty model heuristically, in raw pass-rate or logit coordinates. We show that GRPO comes with a natural coordinate for pass rates: the arc length $ψ=\arcsin\sqrt{p}$ on the Bernoulli Fisher--Rao manifold. In arc length, the expected GRPO update is uniform up to two boundary ramps; the probability of a zero-variance group is bounded by two Gaussian boundary layers of width $1/\sqrt{2G}$ ; pass-rate evidence has constant noise; and the gradients of the pass@ $k$ and pass $^k$ objectives are Gaussians whose center and width follow from $k$ in closed form. A prompt curriculum for GRPO is therefore a Gaussian in arc length, and choosing its center amounts to choosing the objective. We turn this observation into ARCUS, a drop-in sampler that tracks every prompt with a Kalman filter in arc length, scores prompts by an objective-matched Gaussian kernel times the predicted probability of an informative group, keeps only informative groups for the unchanged GRPO update, and paces the target toward the hardest objective whose predicted yield stays within a small slack of the best. Across six mathematical reasoning benchmarks and three backbones, ARCUS improves the average accuracy of GRPO by 2.8--2.9 points and that of dynamic sampling by 1.1--1.2 points, while generating 48--57% fewer rollouts than dynamic sampling.

Figures & tables

Figure 1: Prompts live on an arc (exact curves). (a) GRPO’s expected update as a weight on

\nabla p

is U-shaped (blue); as a weight on

\nabla\psi

\psi=\arcsin\sqrt{p}

, it is flat up to two ramps (red; Prop. 1 ). (b) Zero-variance probability (solid) and its Gaussian bound (dotted; Prop. 2 ). (c) After one group of

n{=}8

, arc-length posteriors are calibrated while logit posteriors are over-confident by

20\times

where zero-variance prompts live (Prop. 3 ). (d) Arc-length gradients of pass@

k

and pass 4 (solid) and their Gaussian forms (dotted; Prop. 4 ).

Figure 2: Arcus within one GRPO step. Beliefs live on the arc $\psi=\arcsin\sqrt{p}$ . An objective-matched Gaussian kernel at the paced target $\psi^{\star}_{t}$ scores prompts; Gumbel-top- $M$ candidates are rolled out and update the beliefs; only informative groups enter the unchanged GRPO update; pacing moves $\psi^{\star}_{t}$ to the hardest objective whose predicted yield stays within $\varepsilon$ of the best.

Qwen2.5-Math-1.5B (DAPO-Math-17k, 300 updates)
Method	Roll.	AIME24	AIME25	AMC23	MATH500	Minerva	Olymp.	Avg.	$\bm{\Delta}$
Base model	–	7.4	3.9	28.6	42.3	12.1	20.8	19.2	–
GRPO	$\times$ 1.00	15.1	9.4	47.9	73.2	28.7	34.6	34.8	–
GCS	$\times$ 1.00	15.4	9.8	48.3	73.4	28.9	34.9	35.1	+0.3
AdaRFT	$\times$ 1.00	15.8	9.6	48.6	73.9	28.6	35.2	35.3	+0.5
CurES	$\times$ 1.00	16.1	10.2	49.0	73.6	29.4	35.3	35.6	+0.8

Table 1: Main results (accuracy, %, mean of three seeds; AIME mean@32, AMC23 mean@16, others mean@4). Groups: uniform, predict-then-select, evaluate-then-filter, ours. Roll. : generated rollouts relative to GRPO at equal policy updates;

\Delta

: average gain over GRPO. Bold / underline : best/second best among trained models.

Method	Roll.	Tok.	Yield	Inf.	Hours	Avg.
	(M)	(B)	(%)	(%)
GRPO	0.61	0.50	47	47	19.8	44.1
MoPPS	0.61	0.52	69	69	19.9	45.0
KGPS	0.61	0.52	75	75	20.0	45.7
GRESO	1.05	0.93	58	100	30.7	45.2
DS	1.48	1.33	41	100	41.6	45.7

Table 2: Cost on Qwen2.5-Math-7B (300 updates, 8

\times

H100): rollouts (M), tokens (B), yield (informative share of generated groups), Inf. (informative share of the update batch), wall-clock hours, and six-benchmark average.

Figure 3: Accuracy, efficiency, and objective control (Qwen2.5-Math-7B). (a) Six-benchmark average vs. policy updates (

\pm 1

s.d.). (b) The same runs vs. cumulative rollouts. (c) Fixed targets along the objective family (gray; p k = pass k ) trade pass@1 against pass@16; the paced target dominates. (d) Cost–accuracy plane; Arcus ’s curve varies

\rho

Geometry and belief
Configuration	Avg.	$\bm{\Delta}$	Inf. (%)
Arcus (full)	46.90	–	97
Gaussian over raw $p$	46.21	$-0.69$	93
Logit-space Kalman belief	45.94	$-0.96$	90
Discounted Beta belief	46.02	$-0.88$	91
Objective kernel

Table 3: Ablations on Qwen2.5-Math-7B (six-benchmark average; Inf.: informative share of the update batch).

Figure 4: How Arcus works (Qwen2.5-Math-7B). (a) Paced target

p^{\star}_{t}=\sin^{2}\psi^{\star}_{t}

for three group sizes (dotted: sharp-belief frontier). (b) Informative share of generated groups (dotted red: Arcus ’s update batch). (c) Pass-rate error of the selected prompts. (d) Median and inter-quartile band of the true pass rate of selected prompts.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Result
Prop. 1 vs. the finite-group weight of Davis and Recht (2025)	max. relative difference $<10^{-14}$ for $G\in\{4,8,16,32\}$
$\max_{p}\omega_{G}(p)$ vs. the bound $\sqrt{1-1/G}$	$0.928$ vs. $0.935$ ( $G{=}8$ ); $0.967$ vs. $0.968$ ( $G{=}16$ )
$\min_{p\in[0.1,0.9]}\omega_{G}(p)$	$0.51$ , $0.70$ , $0.85$ , $0.94$ for $G=4,8,16,32$
Ramp slope $\omega_{G}/\psi$ as $\psi\to 0$	$2.61$ vs. $\sqrt{G-1}=2.65$ ( $G{=}8$ )
Prop. 2 boundary-layer bound	holds on $[0,\tfrac{\pi}{2}]$ for $G\in\{4,8,16\}$
Closed-form $\mathbb{E}[e^{-G\psi^{2}}]$ under a Gaussian belief	matches Monte Carlo to $5\times 10^{-4}$

Appendix

Table 4: Exact numerical checks of the analytic results (script code/theory_checks.py ).

Hyperparameter	Value
Framework	verl + vLLM, bf16
Policy updates	300
Prompts per update $B$	256
Group size $G$	8
Mini-batch (prompts)	64 (4 steps per update)
Optimizer	AdamW, lr $10^{-6}$ , wd $0.01$

Appendix

Table 5: Training and Arcus hyperparameters.

Method	Gen.	Reward	Update	Select	Total
GRPO	13.9	0.4	5.5	$<$ 0.01	19.8
KGPS	14.0	0.4	5.6	0.01	20.0
DS	34.9	1.2	5.5	$<$ 0.01	41.6
Arcus ρ=0	13.9	0.4	5.7	0.01	20.0
Arcus	18.0	0.5	5.6	0.01	24.1

Appendix

Table 6: Wall-clock breakdown (hours) on Qwen2.5-Math-7B, 8

\times

H100. Selection is Arcus ’s belief update, scoring, and pacing, which is negligible; its extra time is generating the candidate margin.

Method	Q2.5-M-1.5B	Q2.5-M-7B	Q3-4B-Base
GRPO	$34.82\pm 0.41$	$44.08\pm 0.37$	$46.27\pm 0.39$
MoPPS	$35.73\pm 0.38$	$45.03\pm 0.35$	$47.27\pm 0.36$
KGPS	$36.32\pm 0.36$	$45.67\pm 0.34$	$47.82\pm 0.33$
DS	$36.50\pm 0.33$	$45.70\pm 0.30$	$48.00\pm 0.31$
Arcus ρ=0	$36.97\pm 0.31$	$46.25\pm 0.30$	$48.50\pm 0.29$
Arcus	$37.65\pm 0.29$	$46.90\pm 0.27$	$49.20\pm 0.28$

Appendix

Table 7: Six-benchmark average, mean

\pm

s.d. over three seeds.

Figure 5: Per-benchmark accuracy over training on Qwen2.5-Math-7B.

Belief dynamics
Configuration	Avg.	$\bm{\Delta}$	Inf. (%)
Arcus (full)	46.90	–	97
No mobility-weighted drift	46.52	$-0.38$	95
Jeffreys prior only	46.71	$-0.19$	96
Anscombe obs. for zero-var. groups	46.33	$-0.57$	93
Objective kernel

Appendix

Table 8: Additional ablations on Qwen2.5-Math-7B.

Target	$p^{\star}$	pass@1	pass@16	Yield
pass 4	0.875	44.61	64.9	72%
pass 2	0.75	45.43	66.1	85%
pass@1	0.5	46.28	67.6	88%
[email protected]	0.333	46.47	68.6	84%
pass@2	0.25	46.31	69.3	78%
pass@4	0.125	45.62	69.8	63%

Appendix

Table 9: Fixed targets along the objective family vs. paced Arcus (Qwen2.5-Math-7B; six-benchmark averages).

$G$	GRPO	DS (roll.)	Arcus	final $p^{\star}_{T}$	frontier
4	42.86	44.31 ( $\times$ 3.37)	45.52	0.43	0.40
8	44.08	45.70 ( $\times$ 2.41)	46.90	0.29	0.34
16	44.93	46.12 ( $\times$ 2.02)	47.64	0.18	0.20

Appendix

Table 10: Effect of group size on Qwen2.5-Math-7B. Frontier: closed-form sharp-belief target for

\varepsilon=0.03

Figure 6: Sensitivity on Qwen2.5-Math-7B: (a) yield slack, (b) Gumbel temperature, (c) group size, and (d) pass@

k

on AIME24.

Setting	GRPO	KGPS	DS	Arcus	DS roll.
R1-Distill-1.5B (8k)	47.86	48.95	49.12	50.21	$\times$ 2.12
Llama-3.2-3B-Inst.	21.37	22.31	22.48	23.26	$\times$ 3.46
Q2.5-Math-7B, MATH pool	43.15	44.30	44.46	45.38	$\times$ 2.78

Appendix

Table 11: Additional backbones and training pools (six-benchmark average; DS roll.: DS rollouts relative to GRPO; Arcus uses

\times 1.25

Method	AIME24	AIME25	AMC23	MATH500	Minerva	Olymp.	Avg.
R1-Distill-1.5B (8k)
GRPO	33.4	25.1	71.0	85.2	30.4	42.1	47.87
KGPS	35.2	26.0	72.4	85.9	30.9	43.3	48.95
DS	35.0	26.4	72.9	86.4	30.8	43.2	49.12
Arcus	37.4	27.6	74.1	86.3	31.5	44.4	50.22
Llama-3.2-3B-Inst.

Appendix

Table 12: Per-benchmark results for the additional backbones and the MATH-train pool (accuracy, %; same protocol as Table 1 ).

Setting	GRPO	DS	Arcus
$B{=}128$	43.52	45.11	46.33
$B{=}256$ (default)	44.08	45.70	46.90
$B{=}512$	44.61	46.02	47.21
lr $5{\times}10^{-7}$	43.12	44.71	45.98
lr $2{\times}10^{-6}$	43.89	45.38	46.64
pool 4k	43.61	44.95	45.87

Appendix

Table 13: Robustness to batch size

B

, learning rate, and pool size on Qwen2.5-Math-7B (six-benchmark average).

Initial belief	Avg.	Warm-up yield	Extra roll.
Jeffreys moments only	46.71	47%	0.0%
Empirical Bayes (default)	46.90	51%	0.0%
Reference-model prior	46.98	62%	0.0%
Offline probe, 2 samples/prompt	47.03	66%	3.4%

Appendix

Table 14: Initial beliefs on Qwen2.5-Math-7B. Warm-up yield: informative share of generated groups during the first

\lceil N/M\rceil

updates.

Policy loss	Uniform	DS	Arcus	$\bm{\Delta}$ vs. DS
GRPO	44.08	45.70	46.90	+1.20
Dr. GRPO	43.92	45.41	46.47	+1.06
DAPO loss	44.61	46.03	47.18	+1.15
RLOO	43.54	45.02	46.11	+1.09

Appendix

Table 15: Compatibility with other group-based losses (Qwen2.5-Math-7B, six-benchmark average).

Figure 7: Training dynamics on Qwen2.5-Math-7B: (a) mean response length, (b) token entropy, and (c) normalized innovation squared of Arcus ’s filter (1 = consistent).

Explore similar work

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning
Feb 19, 2026Gagan Mundada, Zihan Huang, Rohan Surana +8Efficient Language Model Reasoning RL for Language Model Reasoning
University of California, San Diego · University of Toronto · 3Adobe Research +2
Best Practice Critic Optimization
Aug 24, 2026Penghui Qi, Xiangxin Zhou, Wee Sun LeeRL for Language Model Reasoning Actor-Critic Methods
National University of Singapore · Tencent Hunyuan
BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
May 27, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2RL for Language Model Reasoning Group Relative Policy Optimization
1TeleAI · 2Shanghai Jiao Tong University

Feb 19, 2026J/K move · Enter open · S save

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

Gagan Mundada, Zihan Huang, Rohan Surana, Sheldon Yu+7

University of California, San Diego · University of Toronto · 3Adobe Research+2

Efficient Language Model Reasoning RL for Language Model Reasoning Group Relative Policy Optimization Process Reward Models

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) is effective for training language models on complex reasoning. However, since the objective is defined relative to a group of sampled trajectories, extended deliberation can create more chances to realize relative gains, leading to inefficient reasoning and overthinking, and complicating the trade-off between correctness and rollout efficiency. Controlling this behavior is difficult in practice, considering (i) Length penalties are hard to calibrate because longer rollouts may reflect harder problems that require longer reasoning, penalizing tokens risks truncating useful reasoning along with redundant continuation; and (ii) supervision that directly indicates when to continue or stop is typically unavailable beyond final answer correctness. We propose Weakly Supervised GRPO (WS-GRPO), which improves rollout efficiency by converting terminal rewards into correctness-aware guidance over partial trajectories. Unlike global length penalties that are hard to calibrate, WS-GRPO trains a preference model from outcome-only correctness to produce prefix-level signals that indicate when additional continuation is beneficial. Thus, WS-GRPO supplies outcome-derived continue/stop guidance, reducing redundant deliberation while maintaining accuracy. We provide theoretical results and empirically show on reasoning benchmarks that WS-GRPO substantially reduces rollout length while remaining competitive with GRPO baselines.

Abstract

Explore similar work

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning

Best Practice Critic Optimization

BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses