Ascent Conditions for Length-Normalized Single-Rollout Policy Updates

Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr. GRPO (COLM 2025) identifies the response-level length bias caused by per-trajectory length normalization in GRPO and proposes removing this normalization, claiming the resulting optimizer is "unbiased." We show that this claim is incomplete. Specifically, we establish an impossibility theorem: under the standard outcome reward + GRPO setting, no length-based weighting scheme can simultaneously achieve the following two properties. (P1) Gradient unbiasedness: the gradient estimator is an unbiased estimate of the true policy gradient. (P2) Length invariance: each trajectory's effective contribution to the gradient is independent of its token length. GRPO approximately satisfies P2 but violates P1; Dr. GRPO satisfies P1 but violates P2. We characterize the complete tradeoff spectrum via the parametric family f_alpha(L) = L^{alpha - 1}, where alpha = 0 recovers GRPO, alpha = 1 recovers Dr. GRPO, and provide quantitative analysis showing that Dr. GRPO's length bias can cause longer trajectories to dominate gradient updates by a factor proportional to the length ratio. Our results reveal that neither algorithm is universally "done right"; they occupy opposite ends of a fundamental and unavoidable tradeoff.

The policy gradient theorem describes the gradient of the expected discounted return with respect to an agent's policy parameters. However, many policy gradient methods omit the outer discount factor, yielding an update that need not equal the gradient of the discounted objective. Is the resulting update the gradient of another objective? This has been an open question for several years, and this lack of theoretical clarity has led to an abundance of misstatements in the literature. We answer this question for the resulting episode-summed, on-policy update by proving that it need not equal the gradient of any function. Further, we construct a counterexample in which exact expected updates drive the policy toward a deterministic limit that is globally pessimal with respect to both the discounted and undiscounted objectives, while the parameter diverges. We motivate this work by surveying the literature and showing that there remains a widespread misunderstanding regarding discounted policy gradient methods, with errors present even in highly-cited papers published at top conferences.

Policy gradient methods are among the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical success, many state-of-the-art policy gradient algorithms for discounted problems estimate update directions using a sampling distribution that differs from the discounted state distribution required by the policy gradient theorem. In this work, we analyze the impact of this distribution mismatch on policy gradient methods. Specifically, {for tabular direct and softmax parameterizations, we establish global convergence of objective values under biased updates}. {For tabular softmax policies, we further derive the exact leading-order coefficients of the asymptotic objective errors under biased and normalized exact updates. Their ratio provides an explicit criterion for determining when distribution mismatch accelerates or slows asymptotic convergence.} We then extend the analysis to more general policy parameterizations by deriving explicit bounds on both the state distribution mismatch and the induced normalized policy-gradient mismatch in episodic and continuing MDPs. These mismatches vanish at least linearly as the discount factor approaches one. Under an additional smoothness assumption, we further show that the limit inferior of the squared norm of the normalized exact policy-gradient direction along the biased iterates is bounded by $O((1-γ)^2)$ in both settings. Our results provide theoretical insights into the robustness of policy gradient methods to distribution mismatch and shed light on the gap between their theoretical formulations and practical implementations.