Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

Fei Ding

cs.CLAug 27, 2026

Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recovery

Authors: Fei Ding

Organizations: Alibaba Group

Abstract

In verifier-style RLVR, group-relative optimization often treats advantage scale as an implementation detail. This paper separates two low-variance cases: sub-resolution jitter that should not become a preference signal, and credible but small cardinal gaps that should be learned without distorting KL calibration. We propose an advantage-scale three-way calibration interface: the same within-group scale denominator simultaneously determines the reward-branch strength, prompt-level batch weight, and the effective KL calibration induced when the reward branch is re-expressed on the original cardinal scale. This interface explains why RLOO / Dr.GRPO can let credible small gaps become KL dominated, whereas GRPO's standard-deviation denominator can amplify tiny gaps without bound. Based on this interface, we further introduce the Reward-Resolution Protocol and MaxNorm-AC, respectively filtering sub-resolution gaps and providing bounded cardinal recovery on credible nonzero gaps. Across dense / MoE architectures and math / code reasoning, MaxNorm-AC improves over the strongest robust-scale baseline while truncating the low-variance inverse-scale tail.

Figures & tables

Figure 1: Two low-variance regimes expose three existing failures at one reward-scale interface. For sub-resolution jitter

q_{n}

, GRPO restores

x=O(\sigma_{q})

O(1)

, but

1/\sigma_{q}

has no uniform bound and reweights prompts in a fixed batch. For a credible small gap

q_{c}

, the

O(\epsilon_{c})

reward push of RLOO / Dr.GRPO may be relatively dominated by fixed KL. The scale path, spring, and balance encode unbounded amplification, KL pressure, and batch reweighting; our method is not shown.

Figure 2: MaxNorm-AC inserts the scale card

s_{q}=\max\{\tau_{\mathrm{res}},\max_{j}|u_{q,j}|\}

only into the denominator of the trusted raw numerator

u_{q,i}

, yielding ratio- and rank-preserving

c_{q,i}=u_{q,i}/s_{q}

with

|c_{q,i}|\leq 1

. The frozen weight

w_{q,i}=\operatorname{sg}(c_{q,i})

enters only the reward branch: gradients flow through

\rho_{\theta}

and stop at SG, while base KL joins independently without division by

s_{q}

. The probability ratio, clipping, token aggregation, length normalization, and optimizer remain unchanged; the card replaces GRPO’s

\sigma_{q}

or the equivalent denominator

1

of unscaled methods.

Model	Benchmark	Strongest core baseline	Prespecified $p90$ ref.	GOPO	MaxNorm-Dr.GRPO	MaxNorm-RLOO	$\Delta\text{ vs }p90\text{ ref.}$
Qwen3-32B	AIME25	$78.8\pm 1.3$	$79.4\pm 0.9$	$78.8\pm 0.9$	$84.0\pm 1.0$	$\mathbf{84.2\pm 1.2}$	$+4.8$
Qwen3-32B	HMMT25	$58.2\pm 1.1$	$59.1\pm 1.0$	$58.4\pm 1.1$	$62.9\pm 0.8$	$\mathbf{63.0\pm 0.9}$	$+3.9$
Qwen3-32B	LiveCodeBench v6	$63.4\pm 0.8$	$63.9\pm 0.8$	$63.3\pm 1.0$	$69.4\pm 1.1$	$\mathbf{69.7\pm 0.7}$	$+5.8$
Qwen3-Next	AIME25	$89.8\pm 0.9$	$90.0\pm 1.2$	$89.5\pm 0.9$	$93.7\pm 0.9$	$\mathbf{93.7\pm 1.1}$	$+3.7$
Qwen3-Next	HMMT25	$77.1\pm 1.0$	$77.5\pm 1.1$	$76.8\pm 1.2$	$82.1\pm 1.2$	$\mathbf{82.3\pm 1.0}$	$+4.8$
Qwen3-Next	LiveCodeBench v6	$70.8\pm 1.0$	$71.0\pm 0.7$	$70.4\pm 0.8$	$76.5\pm 0.7$	$\mathbf{76.6\pm 0.9}$	$+5.6$

Table 1: Per-task benchmark gains restored in the main text. The strongest core baseline is selected per model–task unit among RLOO + Protocol, GRPO, GRPO+

\beta_{q}

-comp, Dr.GRPO, and REINFORCE++; GRPO+

\beta_{q}

-comp is selected in all six units. The prespecified

p90

reference column exposes the percentile-scale paired comparison, with MaxNorm-RLOO higher on all six rows. GOPO is the pure-ranking boundary baseline. MaxNorm-Dr.GRPO is the centered-numerator check;

\Delta\text{ vs }p90\text{ ref.}

reports the MaxNorm-RLOO improvement over this reference.

Configuration	Protocol	train-time $\delta_{\mathrm{res}}$ -bin	ZG skip	Acc	KL P95
GRPO	N	N	Y	$70.6$	$0.086$
GRPO + Protocol	Y	N	Y	$71.3$	$0.091$
RLOO + Protocol	Y	N	Y	$71.4$	$0.032$
$p90$ ref. + Protocol	Y	N	Y	$73.5$	$0.080$
MaxNorm-RLOO, no $\delta_{\mathrm{res}}$ -binning	Y	N	Y	$78.3$	$0.055$
MaxNorm-RLOO, Protocol off	N	Y	Y	$78.3^{\dagger}$	$0.055^{\dagger}$

Table 2: Single-column component ablation. The Protocol column indicates whether the reward-pipeline resolution gate is enabled; even when Protocol is disabled, reward bounding, component caps, data, and optimizer settings remain fixed. The train-time

\delta_{\mathrm{res}}

-binning column only indicates whether the algorithm additionally merges sub-resolution gaps before advantage construction. GRPO and GRPO+Protocol use the same standard-deviation denominator and isolate the resolution gate; RLOO+Protocol and MaxNorm-RLOO no-

\delta_{\mathrm{res}}

-binning share the three settings and isolate denominator calibration;

\dagger

marks the Protocol-off but train-time-binned row, which has the same effective gaps as Full MaxNorm-RLOO. The Protocol/binning off row reports measured

78.0/0.058

Method / check	mean KL	KL P95	Inv P95/P99	R/KL	Dir.
RLOO + Protocol	$0.017$	$0.032$	$1/1$	$0.48$	$1.00$
GRPO	$0.044$	$0.086$	$980/3100$	$3.10$	$0.61$
GRPO+ $\beta_{q}$ -comp	$0.030$	$0.061$	$980/3100$	$2.94$	$0.63$
Dr.GRPO	$0.019$	$0.034$	$1/1$	$0.51$	$1.00$
REINFORCE++	$0.032$	$0.058$	$2.2/3.1$	$1.38$	$0.91$
$p90$ ref. + Protocol	$0.036$	$0.080$	$100/100$	$1.95$	$0.84$

Table 3: Low-variance diagnostics on updated groups with

\sigma_{q}<10^{-2}

. R/KL is

\|g_{R}\|/(\beta\|g_{\mathrm{KL}}\|)

, and Dir. is the reward/KL direction cosine. Per-task accuracy and paired

p90

-reference comparisons are in Table 1 ; checkpoint audits are in the supplement.

Group Relative Policy Optimization

Group Relative Policy Optimization (GRPO) trains policies by comparing sampled outputs within groups, without a separate learned value critic.

See all 332 papers

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”
Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
May 20, 2026Xixiang He, Qiyao Sun, Ao Cheng +5RL for Language Model Reasoning Group Relative Policy Optimization
National University of Defense Technology, Changsha, Hunan, China · Intelligent Game and Decision Lab, Beijing, China · The Chinese University of Hong Kong, Shenzhen, Guangdong, China
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
Jan 30, 2026Cheng Ge, Caitlyn Heqi Yin, Hao Liang +1RL for Language Model Reasoning Group Relative Policy Optimization
Department of Aeronautics and Astronautics, MIT · Department of Statistics, University of Wisconsin–Madison · Department of Informatics, King’s College London +1

Feb 6, 2026J/K move · Enter open · S save

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov, Viacheslav Sinii+3

T-Tech · Saint Petersburg Electrotechnical University “LETI”

Reinforcement Learning RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-

N

transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at

N{=}8

, our method improves average math pass@256 from 64.1

\rightarrow

70.3 (GRPO), 69.3

\rightarrow

72.5 (DAPO), and 73.2

\rightarrow

76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

Abstract

Group Relative Policy Optimization

Explore similar work

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients