Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Jonah Ellison; Mateo Ruiz; Owen Pell; Tomas Duncun; Hanae Fujita; Ingrid Solberg; Caleb Nguyen

cs.LGOct 18, 2026

Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Authors: Jonah Ellison, Mateo Ruiz, Owen Pell, Tomas Duncun, Hanae Fujita, Ingrid Solberg, Caleb Nguyen

Organizations: University of Michigan · University of Edinburgh

Abstract

Group relative policy optimization learns from a prompt only when the rewards in the sampled group disagree. With a binary verifier, a group that is entirely correct or entirely incorrect has within-group variance zero and a zero advantage. Uniform sampling continues to spend rollouts on prompts that are unanimous now and stay unanimous as training goes on. Pass rate is a weak forecast of that future. Two prompts at the same pass rate can move apart: one into the range where a finite group splits, and one toward a ceiling where the group becomes unanimous again. Aperture estimates the later pass rate from a short probe. A small subset of the pool is trained with the same recipe, every prompt is scored on the base model and on the probe model, and the two pass rates are plugged into the probability that a group of the training size is all wrong or split. The resulting weight is a static prior. It can be used alone or multiplied by an online scheduler, and the GRPO update is left as it is. On mathematical and logical reasoning with 1.7B and 4B models, the prior improves greedy accuracy over uniform GRPO and raises the fraction of groups with a nonzero advantage. The same prior still helps when it is combined with dynamic sampling or with selective rollouts.

Figures & tables

Figure 1: Unanimity, and which prompts leave it. (a) Exact probability that $G$ i.i.d. Bernoulli rewards are unanimous. The training group size is $G=8$ . (b) Endpoint pass rates on a diagnostic draw, colored by the Aperture score. Darker points are prompts the score will upweight. (c) All-wrong probability during the main math run for the top and bottom quartiles of the score, on Qwen3-1.7B. Bands are standard deviations across three seeds.

Figure 2: The Aperture prior. The top row sketches four pass-rate trajectories the probe is meant to tell apart: sealed failure, opening, a resident interior prompt, and sealing into the ceiling. The bottom row is the computation: a uniform probe, two endpoint evaluations, the score in ( 4 ), the mixture in ( 5 ), and GRPO. An online scheduler, when one is used, multiplies $w$ and does not replace it.

	Qwen3-1.7B				Qwen3-4B
Method	Acc $\uparrow$	AUC $\uparrow$	Live $\uparrow$	S2B $\downarrow$	Acc $\uparrow$	AUC $\uparrow$	Live $\uparrow$	S2B $\downarrow$
Uniform	0.377	0.344	45.8%	–	0.481	0.439	53.4%	–
Aperture	0.395	0.368	60.7%	68.5	0.497	0.463	67.2%	62.4
Dyn. sampling	0.385	0.353	47.1%	–	0.487	0.447	54.8%	–
Dyn. + Aperture	0.401	0.376	62.8%	76.1	0.502	0.470	68.9%	69.4
Selective	0.382	0.350	53.3%	–	0.486	0.444	59.8%	–

Table 1: Math. Greedy macro-accuracy, area under the checkpoint curve, fraction of non-unanimous training groups, and steps-to-baseline in percent of training. S2B compares each prior-augmented row with the scheduler underneath it. Means over three seeds. The best accuracy and the best live rate in each model block are in bold.

	Qwen3-1.7B				Qwen3-4B
Method	Acc $\uparrow$	AUC $\uparrow$	Live $\uparrow$	S2B $\downarrow$	Acc $\uparrow$	AUC $\uparrow$	Live $\uparrow$	S2B $\downarrow$
Uniform	0.171	0.156	37.2%	–	0.211	0.192	42.8%	–
Aperture	0.185	0.172	52.1%	73.2	0.223	0.208	57.4%	59.8
Dyn. sampling	0.177	0.162	38.6%	–	0.216	0.198	44.1%	–
Dyn. + Aperture	0.190	0.178	54.4%	80.5	0.227	0.213	59.2%	64.7
Selective	0.175	0.160	44.1%	–	0.214	0.196	49.3%	–

Table 2: Logic, with the same columns and the same protocol as Table 1 .

Figure 3: Qwen3-1.7B on math. (a) Greedy macro-accuracy. (b) Fraction of generated groups that are not unanimous. Curves are means over three seeds. The band is the seed standard deviation.

		Math				Logic
Tercile	Sub-pool	Pre	Acc	AUC	Live	Pre	Acc	AUC	Live
Low	low score	0.041	0.328	0.301	28.6%	0.012	0.118	0.096	21.4%
	high score	0.042	0.401	0.364	47.1%	0.013	0.164	0.141	39.2%
Mid	low score	0.186	0.356	0.329	43.8%	0.074	0.139	0.118	33.1%
	high score	0.185	0.398	0.368	61.2%	0.073	0.171	0.149	50.8%
High	low score	0.612	0.384	0.352	40.2%	0.286	0.176	0.151	34.8%

Table 3: Within a tercile of the initial pass rate, on Qwen3-1.7B. High and low refer to the Aperture score. Pre is the mean initial pass rate and matches across the two halves.

	Math			Logic
Variant	Acc	AUC	Live	Acc	AUC	Live
Uniform	0.377	0.344	45.8%	0.171	0.156	37.2%
Headroom $\times$ gain	0.388	0.356	51.9%	0.179	0.164	42.8%
Destination $\nu_{G}$ only	0.384	0.353	57.2%	0.176	0.162	48.6%
Fail-collapse drop only	0.386	0.355	54.6%	0.178	0.163	46.2%
No Jeffreys shrink	0.391	0.361	58.8%	0.182	0.168	49.8%

Table 4: Score ablations on Qwen3-1.7B. Headroom

\times

gain is

(1-\tilde{\rho}_{0})[\tilde{\rho}_{1}-\tilde{\rho}_{0}]_{+}

. Destination-only uses

\nu_{G}(\tilde{\rho}_{1})

. Fail-collapse drop uses only the second factor of ( 4 ).

\lambda=1

removes the uniform mixture.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

$p$	$c_{4}(p)$	$c_{8}(p)$	$c_{16}(p)$
0.00	1.0000	1.0000	1.0000
0.05	0.8145	0.6634	0.4401
0.10	0.6562	0.4305	0.1853
0.25	0.3203	0.1001	0.0100
0.50	0.1250	0.0078	0.0000
0.75	0.3203	0.1001	0.0100

Appendix

Table 5: Unanimity probability

c_{G}(p)

from ( 1 ). These values are computed from the formula.

Prompt	$\rho_{0}$	$\rho_{1}$	$G$	$\Delta c^{\mathrm{fail}}$	$\nu_{G}(\rho_{1})$	$s$	$s_{\mathrm{hg}}$
A	0.05	0.20	4	0.405	0.589	0.238	0.143
B	0.25	0.55	4	0.275	0.867	0.239	0.225
C	0.55	0.92	4	0.041	0.284	0.012	0.166
A	0.05	0.20	8	0.496	0.832	0.412	0.143
B	0.25	0.55	8	0.098	0.990	0.097	0.225
C	0.55	0.92	8	0.002	0.487	0.001	0.166

Appendix

Table 6: Constructed endpoints.

s

is ( 4 ) on the population pass rates, and

s_{\mathrm{hg}}

is headroom times accuracy gain. The rows are a witness, not a sample from the pool.

Method	Groups per update	Live (math, 1.7B)
Uniform	1.00	45.8%
Aperture	1.02	60.7%
Dyn. sampling	1.48	47.1%
Dyn. + Aperture	1.23	62.8%
Selective	0.90	53.3%
Sel. + Aperture	0.93	64.1%

Appendix

Table 7: Groups generated per optimizer step, relative to uniform GRPO, on the math run. The live rate is the fraction of those groups that are not unanimous. The one-time probe is not included; see the ratios above.

Qwen3-1.7B
Method	MATH-500	Minerva	AIME24	AIME25	Olympiad	AMC23	Macro
Uniform	0.718	0.301	0.152	0.121	0.408	0.564	0.377
Aperture	0.725	0.321	0.177	0.147	0.425	0.576	0.395
Dyn. sampling	0.721	0.309	0.162	0.131	0.415	0.569	0.385
Dyn. + Aperture	0.728	0.328	0.186	0.156	0.430	0.580	0.401
Selective	0.720	0.307	0.159	0.128	0.413	0.567	0.382

Appendix

Table 8: Math, greedy accuracy by benchmark.

Qwen3-1.7B
Method	K&K (3–4)	K&K (5–7)	Ordering	Graph	ARC-AGI-2	Macro
Uniform	0.414	0.088	0.172	0.128	0.053	0.171
Aperture	0.423	0.104	0.186	0.143	0.069	0.185
Dyn. sampling	0.418	0.095	0.178	0.134	0.060	0.177
Dyn. + Aperture	0.427	0.109	0.191	0.148	0.075	0.190
Selective	0.417	0.092	0.176	0.132	0.058	0.175

Appendix

Table 9: Logic, greedy accuracy by benchmark. ARC-AGI-2 is held out of training.

Figure 4: Qwen3-1.7B on logic. Axes match Figure 3 . The live-group gap is similar to math. The accuracy level is lower because the held-out puzzles, and ARC-AGI-2 in particular, are harder for this model.

	Math			Logic
Method	seed 0	seed 1	seed 2	seed 0	seed 1	seed 2
Uniform	0.383	0.380	0.369	0.172	0.168	0.173
Aperture	0.392	0.397	0.396	0.182	0.185	0.188
Dyn. sampling	0.385	0.384	0.384	0.175	0.181	0.175
Dyn. + Aperture	0.399	0.399	0.406	0.193	0.188	0.189
Selective	0.383	0.380	0.384	0.175	0.174	0.176

Appendix

Table 10: Per-seed macro-accuracy, Qwen3-1.7B. The mean of each row is the corresponding entry of Tables 1 and 2 .

Method	Acc	AUC	Live
Uniform	0.316	0.288	42.1%
Aperture	0.331	0.308	56.6%
Dyn. sampling	0.322	0.296	43.4%
Dyn. + Aperture	0.336	0.314	58.1%
Selective	0.320	0.293	48.8%
Sel. + Aperture	0.333	0.311	59.4%

Appendix

Table 11: Llama-3.2-3B, math, same metrics as the Qwen runs.

Method	Acc	Live
Uniform	0.377	45.8%
Moving-edge filter	0.385	51.2%
Moving-edge + Aperture	0.397	62.4%

Appendix

Table 12: The prior as a multiplier on a moving-edge filter. Qwen3-1.7B, math.

Probe size	Math / Logic acc
$\|P\|=256$	0.384 / 0.175
$\|P\|=512$	0.395 / 0.185
$\|P\|=1024$	0.397 / 0.186
$\|P\|=2048$	0.398 / 0.187

Appendix

Table 13: Probe size on Qwen3-1.7B. Cells are math accuracy and logic accuracy.

Probe length	Math acc
2 epochs	0.386
4 epochs	0.391
6 epochs	0.395
10 epochs	0.396

Appendix

Table 14: Probe length, Qwen3-1.7B math.

Score group size	Math acc
$G^{\prime}=4$	0.390
$G^{\prime}=8$	0.395
$G^{\prime}=16$	0.391

Appendix

Table 15: Group size

G^{\prime}

used inside ( 4 ), with training still at

G=8

. Qwen3-1.7B math.

Mixture	Math acc
$\lambda=0$ (uniform)	0.377
$\lambda=0.5$	0.386
$\lambda=0.75$	0.395
$\lambda=1.0$	0.389

Appendix

Table 16: Mixture weight

\lambda

, Qwen3-1.7B math.

Power	Math acc
$\alpha=0.25$	0.387
$\alpha=0.5$	0.395
$\alpha=1.0$	0.390

Appendix

Table 17: Score power

\alpha

, Qwen3-1.7B math.

Prior	Acc
In-regime 4B math	0.497
1.7B prior, 4B math	0.490
Uniform 4B math	0.481
In-regime 1.7B logic	0.185
Math prior, 1.7B logic	0.176
Uniform 1.7B logic	0.171

Appendix

Table 18: Reused priors. “In-regime” refits the probe on the model and pool being trained.

Training progress	Spearman $\rho$
25%	0.840
50%	0.770
75%	0.690
100%	0.610

Appendix

Table 19: Spearman correlation between the initial Aperture score and a score recomputed at the current policy. Qwen3-1.7B, math.

Method	Easy-slice acc
Uniform	0.742
Aperture	0.739
Dyn. sampling	0.744
Dyn. + Aperture	0.741

Appendix

Table 20: Easy slice, Qwen3-1.7B. Prompts are already solved at high pass rate under the base model.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,087 papers

Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts
Oct 18, 2026Tomas Duncun, Jonah Ellison, Owen Pell +4Adaptive Sampling Sample-Efficient RL
University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Oct 26, 2026Sohyun Son, Yerim Hong, Jaeyun Park +1Reinforcement Learning RL for Language Model Reasoning
Kyung Hee University, Yongin, Republic of Korea

Oct 18, 2026J/K move · Enter open · S save

Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts

Tomas Duncun, Jonah Ellison, Owen Pell, Mateo Ruiz+3

University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom

Adaptive Sampling Sample-Efficient RL Curriculum RL Group Relative Policy Optimization

Open paper AskPDF arXiv

Group relative policy optimization learns from a question only when the sampled responses disagree. A group that is entirely correct or entirely incorrect has zero reward variance and a zero advantage. On the Web, that group has already spent its searches and page fetches. Gaussian curricula aim the next batch at pass rates where a finite group can still mix. A Gaussian on the latest empirical rate treats every all-fail history as the same point, and a center stepped by the average signed change of those rates is pushed by prompts already stuck at 0 or 1. RIDGE keeps the Gaussian curriculum and changes the coordinate, the center, and the width. Each prompt carries a Beta posterior. The sampling weight is the marginal of a Gaussian target under a Gaussian approximation of that posterior, divided by a prediction of tool-call cost. The center and the width are an exponential smooth of the posterior means of groups that just came back mixed. All-correct and all-incorrect groups refresh the posterior and do not move the center. The GRPO update itself is unchanged. On two math backbones and a retrieval-augmented question answerer, RIDGE raises average accuracy by 3.4 and 2.9 points over GRPO and average exact match by 3.7 points, at 29.6% fewer math rollouts and 32.7% fewer tool calls than dynamic sampling.

Abstract

RL for Language Model Reasoning

Explore similar work

Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization