The policy gradient theorem describes the gradient of the expected discounted return with respect to an agent's policy parameters. However, many policy gradient methods omit the outer discount factor, yielding an update that need not equal the gradient of the discounted objective. Is the resulting update the gradient of another objective? This has been an open question for several years, and this lack of theoretical clarity has led to an abundance of misstatements in the literature. We answer this question for the resulting episode-summed, on-policy update by proving that it need not equal the gradient of any function. Further, we construct a counterexample in which exact expected updates drive the policy toward a deterministic limit that is globally pessimal with respect to both the discounted and undiscounted objectives, while the parameter diverges. We motivate this work by surveying the literature and showing that there remains a widespread misunderstanding regarding discounted policy gradient methods, with errors present even in highly-cited papers published at top conferences.
Figures & tables
Figure 1. The counterexample used in Theorem 4.4 , whose update field Fγ has an asymmetric Jacobian for all γ∈[0,1) . The agent begins in s1 , and may choose between actions a1 and a2 , each of which produces deterministic transitions; s3=s∞ is terminal. The rewards are 0 everywhere except when the agent chooses a1 in state s2 , which produces a reward of +1 . The policy is the sigmoid parameterization specified in the proof: πθ(si,a1)=σ(θi) for i=1,2 .
Figure 2. An example where exact expected updates approach a policy that is optimal with respect to the discounted objective but pessimal with respect to the undiscounted objective. The agent starts in state s1 , and can achieve a reward of +1 by transitioning from s1 to s2 , and a reward of +2 by transitioning from s6 to s7 , regardless of the action taken. In order to maximize the undiscounted return, the agent should choose a2 . However, the discounted return is higher from choosing a1 . Only the return in s1 will affect the policy gradient as the advantage is zero in every other state. Under the sigmoid parameterization and step-size assumptions in Section 5 , the probability of choosing a1 at s1 converges to one as θ→+∞ . If the researcher is concerned with the undiscounted objective, as is often the case, this result is problematic. Choosing a larger value of γ trivially fixes the problem in this particular example, but for any value of γ<1 , similar problems will arise given a sufficiently long horizon, and thus the problem is not truly eliminated. Nevertheless, this is well-understood as the trade-off of discounting.
Policy gradient methods are among the most successful approaches for solving challenging reinforcement learning problems. Despite their empirical success, many state-of-the-art policy gradient algorithms for discounted problems estimate update directions using a sampling distribution that differs from the discounted state distribution required by the policy gradient theorem. In this work, we analyze the impact of this distribution mismatch on policy gradient methods. Specifically, {for tabular direct and softmax parameterizations, we establish global convergence of objective values under biased updates}. {For tabular softmax policies, we further derive the exact leading-order coefficients of the asymptotic objective errors under biased and normalized exact updates. Their ratio provides an explicit criterion for determining when distribution mismatch accelerates or slows asymptotic convergence.} We then extend the analysis to more general policy parameterizations by deriving explicit bounds on both the state distribution mismatch and the induced normalized policy-gradient mismatch in episodic and continuing MDPs. These mismatches vanish at least linearly as the discount factor approaches one. Under an additional smoothness assumption, we further show that the limit inferior of the squared norm of the normalized exact policy-gradient direction along the biased iterates is bounded by O((1−γ)2) in both settings. Our results provide theoretical insights into the robustness of policy gradient methods to distribution mismatch and shed light on the gap between their theoretical formulations and practical implementations.
Weizhen Wang, Xinzhe Zhou, Jianping He +1
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China
In introductory presentations of policy gradients, one often derives the REINFORCE estimator using the full trajectory return and then states, by ``causality,'' that the full return may be replaced by the reward-to-go. Although this statement is correct, it is frequently presented at a level of rigor that leaves unclear where the past-reward terms disappear. This short paper isolates that step and gives a mathematically explicit derivation based on prefix trajectory distributions and the score-function identity. The resulting account does not change the estimator. Its contribution is conceptual: instead of presenting reward-to-go as a post hoc unbiased replacement for full return, it shows that reward-to-go arises directly once the objective is decomposed over prefix trajectories. In this formulation, the usual causality argument is recovered as a corollary of the derivation rather than as an additional heuristic principle.
Policy gradient computes a backward pass for every sample, even though the backward pass is expensive and most samples carry little learning value. The Delightful Policy Gradient (DG) provides a forward-pass signal of learning value: delight, the product of advantage and surprisal (negative log-probability). We introduce the Kondo gate, which compares delight against a compute price and pays for a backward pass only when the sample is worth it, thereby tracing a quality--cost Pareto frontier. In bandits, zero-price gating preserves useful gradient signal while removing perpendicular noise, and delight is a more reliable screening signal than additive combinations of value and surprise. On MNIST and transformer token reversal, the Kondo gate skips most backward passes while retaining nearly all of DG's learning quality, with gains that grow as problems get harder and backward passes become more expensive. Because the gate tolerates approximate delight, a cheap forward pass can screen samples before expensive backpropagation, suggesting a speculative-decoding-for-training paradigm.