Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

Levin David Schwab

cs.LGOct 10, 2026

Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

Authors: Levin David Schwab

Abstract

Dividing each trajectory's score contribution by its realized length can change the direction of a reward-centered policy update. In a finite trajectory model, we derive an exact ascent threshold for a three-outcome softmax policy and exhibit a binary-reward example in which the expected Euclidean update decreases task success. Natural-gradient ascent holds for fully controllable outcome distributions with deterministic binary labels, but can fail under a one-parameter restriction. A third-order U-statistic estimates the first-order return change without bias from independent current-policy trajectories, one per request. In the example, a same-sample plug-in estimator instead has the opposite sign in expectation for batch sizes two through forty-nine. Exact policy-grid enumeration confirms the local sign reversal. In a branching process, length-normalized learning initially loses return, then recovers and exceeds the unnormalized comparator. The analysis concerns the expected update field at the current policy.

Figures & tables

$n$	Plug-in bias / $\|D\|$	$\operatorname{sd}(\widehat{D})/\|D\|$	$\Pr(\widehat{D}<0)$	$\Pr(\widetilde{D}<0)$
3	11.403	90.609	0.048	0.000
8	5.574	15.692	0.215	0.000
32	1.539	3.344	0.747	0.614
64	0.782	1.893	0.820	0.716
128	0.394	1.172	0.855	0.788
512	0.099	0.522	0.953	0.938

Table 1: Complete count-law evaluation for the three-outcome example, with

D=-0.0003094

. Bias is

\mathbb{E}[\widetilde{D}]-D

. Probabilities and moments are numerical sums over the entire multinomial law, rather than Monte Carlo estimates.

$B$	Update	$J_{512}-J_{0}$	$J_{8192}$	$\min_{k}J_{k}$
32	U	$0.0213\pm 0.0005$	$0.9546\pm 0.0004$	$0.8548\pm 0.0000$
32	L	$-0.0036\pm 0.0002$	$0.9714\pm 0.0003$	$0.8498\pm 0.0001$
32	P	$-0.0035\pm 0.0002$	$0.9714\pm 0.0001$	$0.8498\pm 0.0002$
32	M	$0.0212\pm 0.0005$	$0.9538\pm 0.0003$	$0.8548\pm 0.0000$
64	U	$0.0213\pm 0.0005$	$0.9546\pm 0.0004$	$0.8548\pm 0.0000$
64	L	$-0.0035\pm 0.0002$	$0.9715\pm 0.0003$	$0.8498\pm 0.0001$

Table 2: Finite-sample learning in the enumerated branching process. Entries are means and approximate 95% Student-

t

interval half-widths across 32 seeds;

J_{0}=0.85483

. Every run uses 8,192 fresh trajectories. The minimum includes the initial iterate. Length normalization causes an early decrease here, but L and P recover and attain higher final return than U; the experiment therefore does not show general training inferiority.

Figure 1: The three-outcome policy grid (a) and controlled learning in the branching process (b–f). The figure separates enumerated mean-update iterations from finite-sample runs, displays the early decrease as well as the complete training horizon, and includes the constant-length control. Finite-sample bands are approximate pointwise 95% intervals for the mean across 32 seeds. All returns are evaluated by path enumeration at the current policy.

Step multiplier	$\Delta J_{512}^{\rm U}$	$\Delta J_{512}^{\rm L}$	$J_{8192}^{\rm U}$	$J_{8192}^{\rm L}$
0.5	+0.0116	-0.0019	0.9328	0.9484
1.0	+0.0213	-0.0036	0.9546	0.9714
2.0	+0.0362	-0.0044	0.9721	0.9824

Table 3: Exploratory step-size sensitivity in the variable-length branching process. Entries are means over 32 paired seeds;

\Delta J_{512}=J_{512}-J_{0}

. Paired differences and approximate intervals are included in the accompanying results.

U-Statistics

See all 11 papers

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Jul 25, 2026Fei Ding, Yongkang Zhang, Yuhao Liao +2Policy Gradient RL for Language Model Reasoning
Alibaba Group · Tsinghua University
Is the Policy Gradient a Gradient?
Jun 17, 2019Chris Nota, Philip S. ThomasPolicy Gradient Policy Gradient Methods
College of Information and Computer Sciences University of Massachusetts Amherst
Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
Mar 28, 2025Weizhen Wang, Xinzhe Zhou, Jianping He +1Policy Gradient Policy Gradient Methods
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China

Jul 25, 2026J/K move · Enter open · S save

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

Fei Ding, Yongkang Zhang, Yuhao Liao, Zijian Zeng+1

Alibaba Group · Tsinghua University

Policy Gradient RL for Language Model Reasoning Group Relative Policy Optimization Policy Optimization Policy Learning

Open paper AskPDF arXiv

Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr. GRPO (COLM 2025) identifies the response-level length bias caused by per-trajectory length normalization in GRPO and proposes removing this normalization, claiming the resulting optimizer is "unbiased." We show that this claim is incomplete. Specifically, we establish an impossibility theorem: under the standard outcome reward + GRPO setting, no length-based weighting scheme can simultaneously achieve the following two properties. (P1) Gradient unbiasedness: the gradient estimator is an unbiased estimate of the true policy gradient. (P2) Length invariance: each trajectory's effective contribution to the gradient is independent of its token length. GRPO approximately satisfies P2 but violates P1; Dr. GRPO satisfies P1 but violates P2. We characterize the complete tradeoff spectrum via the parametric family f_alpha(L) = L^{alpha - 1}, where alpha = 0 recovers GRPO, alpha = 1 recovers Dr. GRPO, and provide quantitative analysis showing that Dr. GRPO's length bias can cause longer trajectories to dominate gradient updates by a factor proportional to the length ratio. Our results reveal that neither algorithm is universally "done right"; they occupy opposite ends of a fundamental and unavoidable tradeoff.

Abstract

U-Statistics

Explore similar work

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

Is the Policy Gradient a Gradient?

Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch