One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets

Jiaying Zhou; Jiachen Liu; Qingchao Chen

cs.AIOct 31, 2026

One Offset, Two Directions: Baseline Adjustment via Subgroup Offsets

Authors: Jiaying Zhou, Jiachen Liu, Qingchao Chen

Organizations: Peking University

Abstract

Group Relative Policy Optimization (GRPO) estimates advantages against the within-prompt mean reward, a baseline shared across the group. In practice, rollouts within a prompt are not always homogeneous, differing in their solution strategies and reasoning trajectories, along which they may group into subgroups. We further find that these heterogeneous subgroups may exhibit distinct reward distributions. The global baseline, however, collapses them onto a single credit axis, where the credit reflects neither a rollout's standing within its subgroup nor the subgroup's standing relative to the rest. We therefore study how to adjust the baseline when the rollouts of a prompt fall into heterogeneous subgroups. We define the subgroup offset as the deviation of a subgroup mean from the global mean, and shift the baseline along it by a single signed coefficient $λ$ . The sign of $λ$ yields two opposing directions: Alignment ( $λ{>}0$ ) adds the offset to the baseline, drawing credit toward a rollout's relative position within its subgroup; Contrast ( $λ{<}0$ ) subtracts it, drawing a rollout's credit toward the relative quality of the subgroup it belongs to. The adjustment acts on the baseline alone, leaving the reward signal and the optimization objective unchanged. In this work, we induce subgroups from strategies, grouping rollouts that share a strategy into the same subgroup. We evaluate both directions on Qwen3-8B-Base and Olmo3-7B-Base across four competition-level math benchmarks, against a $λ{=}0$ anchor that shares the same strategy-conditioned sampling and differs only in the baseline. Against this anchor each direction yields modest but consistent gains in multi-sample coverage, with Contrast the stronger of the two.

Figures & tables

Figure 1: (a) Different strategies may lead to markedly different reward distributions. For the same problem, success rates across the four strategies range from $0.75$ down to $0$ (strategies and the corresponding rollouts are sampled from Qwen3-8B-Base at an intermediate checkpoint). (b) Effect of the two adjustment directions on the advantage. Relative to the global mean, which shares one baseline across all strategies, Alignment credits each rollout within the same strategy, while Contrast amplifies the differences between strategies. Bar heights indicate the relative magnitude of the advantage only, not exact values.

Method	AIME 2024		AIME 2025		AIME 2026		HMMT 2025
Method	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16
Qwen3-8B-Base
GRPO	31.6	66.8	24.0	50.3	25.5	58.3	12.8	32.5
Strategy-GRPO	30.2	65.6	23.0	47.9	23.5	53.8	12.6	32.5
Alignment	31.1 $\uparrow$	66.4 $\uparrow$	24.0 $\uparrow$	49.8 $\uparrow$	24.4 $\uparrow$	56.0 $\uparrow$	12.1 $\downarrow$	32.7 $\uparrow$
Contrast	31.8 $\uparrow$	69.1 $\uparrow$	23.8 $\uparrow$	50.6 $\uparrow$	24.6 $\uparrow$	58.8 $\uparrow$	13.0 $\uparrow$	36.3 $\uparrow$

Table 1: Main results on four competition-level math benchmarks (pass@1 / pass@16). Arrows denote the change relative to Strategy-GRPO (

\lambda{=}0

), the anchor shared by both directions, which is the comparison that isolates the baseline adjustment. The best and second-best results among the strategy-based methods are shown in bold and underlined, respectively; the GRPO row is shown in italics for reference.

Alignment ( $\lambda>0$ )
$\lambda$	AIME 2024		AIME 2025		AIME 2026		HMMT 2025
$\lambda$	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16	pass@1	pass@16
$0$ (Strategy-GRPO)	30.2	65.6	23.0	47.9	23.5	53.8	12.6	32.5
$+0.3$	31.1	66.4	24.0	49.8	24.4	56.0	12.1	32.7
$+0.5$	30.3	64.3	23.6	46.8	23.6	51.5	11.6	29.0
$+0.7$	30.9	65.8	23.9	48.4	24.8	54.1	12.0	29.2

Table 2: Effect of the coefficient

\lambda

on Qwen3-8B-Base (pass@1 / pass@16). We vary

\lambda

on both sides of the

\lambda{=}0

anchor (Strategy-GRPO) to probe how the two directions respond to the strength of the baseline adjustment. For each column, the best result is in bold and the second best is underlined .

Figure 2: Training dynamics on Qwen3-8B-Base (top) and Olmo3-7B-Base (bottom), for

\lambda{=}0

(Strategy-GRPO), Alignment , and Contrast . Left: policy entropy over training (EMA-smoothed). Middle: fraction of prompts whose subgroups differ in success rate by at least

0.375

, i.e. by at least three of eight rollouts. Right: the magnitude of the subgroup term relative to the standard term in the advantage. The

\lambda{=}0

baseline is included in the entropy panels for reference; the subgroup term is identically zero at

\lambda{=}0

, so the subgroup-term-ratio panels show only the two adjustment directions.

Method	AIME24	AIME25	AIME26	HMMT25
Strategy-GRPO	65.6	47.9	53.8	32.5
Alignment	66.4	49.8	56.0	32.7
+ shuffle	64.5 $\downarrow$	47.2 $\downarrow$	51.7 $\downarrow$	29.2 $\downarrow$
Contrast	69.1	50.6	58.8	36.3
+ shuffle	64.5 $\downarrow$	45.9 $\downarrow$	52.7 $\downarrow$	30.0 $\downarrow$

Table 3: Offset-shuffling ablation on Qwen3-8B-Base (pass@16). Within each group, we permute the subgroup offsets so that each subgroup is assigned an offset from another, keeping the offset values unchanged. Arrows mark the change from the unshuffled setting.

Sampling	AIME24	AIME25	AIME26	HMMT25
Alignment
default	66.4	49.8	56.0	32.7
+ $T{=}0.7$	67.9 $\uparrow$	48.6 $\downarrow$	53.3 $\downarrow$	31.3 $\downarrow$
+ random	66.3 $\downarrow$	48.1 $\downarrow$	54.0 $\downarrow$	29.4 $\downarrow$
Contrast
default	69.1	50.6	58.8	36.3

Table 4: Effect of strategy sampling on Qwen3-8B-Base (pass@16). Starting from the default setup (temperature

1.0

, diversity-based selection), we lower the sampling temperature to

0.7

and, separately, replace diversity-based selection with random selection. Both changes weaken the diversity of the sampled strategies.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base models	Qwen3-8B-Base, Olmo3-7B-Base
Training data	DAPO-Math-17K ( $\sim$ 17K)
Framework	verl
Advantage estimator	grpo_subgroup
Clip $\epsilon_{\text{low}}$ / $\epsilon_{\text{high}}$ / dual	$0.2$ / $0.28$ / $10.0$
KL penalty / entropy bonus	none / none

Appendix

Table 5: Training configuration shared by all methods. The compared methods differ only in the baseline-adjustment coefficient

\lambda

(

\lambda{=}0

recovers Strategy-GRPO).

Strategy	Key idea (abridged)	Success
$s_{1}$	symmetry $+$ Pythagorean theorem	$6/8$
$s_{2}$	coordinate geometry $+$ Pythagorean	$4/8$
$s_{3}$	“upper/lower semicircle intersection”	$1/8$
$s_{4}$	“prove 18 m vs 10 m paths are equal”	$0/8$

Appendix

Table 6: Four strategy-induced subgroups for one problem (reference answer

17

). Success rates range from

6/8

down to

0/8

, so the subgroup means span the full range while the group as a whole is far from resolved.

Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) trains policies by comparing sampled outputs within groups, without a separate learned value critic.

See all 343 papers

Oct 18, 2026

Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

Group relative policy optimization learns from a prompt only when the rewards in the sampled group disagree. With a binary verifier, a group that is entirely correct or entirely incorrect has within-group variance zero and a zero advantage. Uniform sampling continues to spend rollouts on prompts that are unanimous now and stay unanimous as training goes on. Pass rate is a weak forecast of that future. Two prompts at the same pass rate can move apart: one into the range where a finite group splits, and one toward a ceiling where the group becomes unanimous again. Aperture estimates the later pass rate from a short probe. A small subset of the pool is trained with the same recipe, every prompt is scored on the base model and on the probe model, and the two pass rates are plugged into the probability that a group of the training size is all wrong or split. The resulting weight is a static prior. It can be used alone or multiplied by an online scheduler, and the GRPO update is left as it is. On mathematical and logical reasoning with 1.7B and 4B models, the prior improves greedy accuracy over uniform GRPO and raises the fraction of groups with a nonzero advantage. The same prior still helps when it is combined with dynamic sampling or with selective rollouts.

Jonah Ellison, Mateo Ruiz, Owen Pell +4

University of Michigan · University of Edinburgh

RL for Language Model Reasoning RL Post-Training Group Relative Policy Optimization

Ask PDF

Nov 2, 2026

When Rewards Depend on the Group, GRPO Plays a Tournament

Rewards for diversity, coverage and distribution matching make the reward of each sampled response depend on the rest of its group, and they are now routinely optimized with GRPO and RLOO. We prove that in this setting the group-relative baseline changes the objective. The expected update equals each response's own reward gradient plus

α

times its effect on its groupmates' rewards, with

α=-1/(K-1)

for groups of size

K

, so GRPO trains every response to beat its groupmates in a zero-sum tournament. It cannot see a pairwise diversity bonus when K=2, sees less than half of it at any K, and often does worse than using no baseline at all. Setting

α=+1

gives Team-GRPO, an unbiased estimator for any group-dependent reward that works with two samples per prompt and requires no extra model calls for pairwise rewards. The theory predicts training quantitatively. It locates the fixed points of a 9B policy to within 0.02 in

\ell_1

and predicts how concentrated trained policies become with Pearson r=0.995. On GlobalOpinionQA, GRPO collapses onto the majority answer and ends further from human opinion than the untrained model, while Team-GRPO cuts the error from

0.20

0.08

. On NoveltyBench, Team-GRPO avoids the diversity collapse of GRPO and significantly improves set utility.

Junyi Gao, Yu Shi, Pingzhao Hu +1

Centre for Medical Informatics, University of Edinburgh · Health Data Research UK · Biostatistics Division, Dalla Lana School of Public Health, University of Toronto +1

Group Relative Policy Optimization Reward Design for RL Behavioral Diversity in RL

Ask PDF

Oct 18, 2026

Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts

Group relative policy optimization learns from a question only when the sampled responses disagree. A group that is entirely correct or entirely incorrect has zero reward variance and a zero advantage. On the Web, that group has already spent its searches and page fetches. Gaussian curricula aim the next batch at pass rates where a finite group can still mix. A Gaussian on the latest empirical rate treats every all-fail history as the same point, and a center stepped by the average signed change of those rates is pushed by prompts already stuck at 0 or 1. RIDGE keeps the Gaussian curriculum and changes the coordinate, the center, and the width. Each prompt carries a Beta posterior. The sampling weight is the marginal of a Gaussian target under a Gaussian approximation of that posterior, divided by a prediction of tool-call cost. The center and the width are an exponential smooth of the posterior means of groups that just came back mixed. All-correct and all-incorrect groups refresh the posterior and do not move the center. The GRPO update itself is unchanged. On two math backbones and a retrieval-augmented question answerer, RIDGE raises average accuracy by 3.4 and 2.9 points over GRPO and average exact match by 3.7 points, at 29.6% fewer math rollouts and 32.7% fewer tool calls than dynamic sampling.

Tomas Duncun, Jonah Ellison, Owen Pell +4

University of Michigan Ann Arbor, Michigan, USA · University of Edinburgh Edinburgh, United Kingdom

Adaptive Sampling Sample-Efficient RL Curriculum RL Group Relative Policy Optimization

Ask PDF

Abstract

Group Relative Policy Optimization

Explore similar work

Where Groups Split: A Probe Prior for Nonzero Within-Group Variance in RL Post-Training

When Rewards Depend on the Group, GRPO Plays a Tournament

Follow the Mixed Groups: Ridge-Tracking Gaussian Sampling for Zero-Variance Web Rollouts