Delightful Policy Gradient

Ian Osband

cs.LGMar 15, 2026

Delightful Policy Gradient

Authors: Ian Osband

Organizations: Google DeepMind

Abstract

Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the Delightful Policy Gradient (DG), which gates each term with a sigmoid of delight, the product of advantage and action surprisal (negative log-probability). For $K$ -armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Effective coefficient $\omega=w\cdot U$ weighting $\nabla_{\theta}\log\pi$ . DG amplifies breakthroughs (rare successes) and suppresses blunders (rare failures); PG (dashed) is probability-blind.

Figure 2: MNIST classification error. Supervised CE requires labels; PG and DG do not.

Figure 3: Classification error at $T=10\text{k}$ vs. samples per image $S$ , faceted by baseline. Dashed line: error from the exact PG-oracle gradient $g^{*}_{\mathrm{PG}}$ , PG’s best achievable direction.

Panel of Figure 4 — Figure 4: Gradient misalignment over training at $S=1$ (solid) and $S=100$ (dashed). (a) DG’s advantage diminishes with $S$ (variance). (b) DG’s advantage persists at $S=100$ (directional).

Panel of Figure 5 — Figure 5: Symmetric bandit, normalized steps ( $K{=}100$ , $B{=}100$ , $\alpha{=}0.1$ ).

Panel of Figure 6 — Figure 6: $N{=}100$ independent contexts, $K{=}10$ actions, $\mathcal{N}(0,1)$ init, exact gradients. DG converges faster (a) by rebalancing gradient budget to hard contexts (b).

Panel of Figure 9 — Figure 9: Scaling with task complexity. (a,b) Final sequence error after $K{=}10\text{k}$ episodes; baselines degrade beyond a threshold while DG degrades more gracefully. (c,d) Log-log cumulative error reveals approximate power-law scaling; DG achieves a smaller exponent.

Figure 10: Average reward across 28 Control Suite environments. DG (red) is never the worst method.

Figure 13: Score weight $\omega$ vs. advantage $U$ for common ( $\pi{=}0.9$ ) and rare ( $\pi{=}0.01$ ) actions. DG treats rare successes and rare failures asymmetrically; PG and PMPO are blind to action probability, while PPO clips both tails through importance-ratio constraints.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 14 — Figure 14: Generalization on MNIST. (a) Validation error tracks training error; DG achieves lower error on both. (b) The optimal temperature $\eta\approx 1$ is consistent across train and validation.

Figure 15: Entropy regularization vs. DG temperature. Large entropy coefficients degrade accuracy; DG is robust across $\eta$ and consistently achieves lower error.

Figure 16: Learning rate sensitivity. DG (red) outperforms PG (blue) across all baselines and learning rates.

Figure 17: Batch size sensitivity. DG maintains its advantage across all batch sizes.

Figure 18: Network width sensitivity. DG outperforms PG across all capacity levels.

Panel of Figure 19 — Figure 19: Alternative definitions of delight. (a) Additive mixtures underperform the multiplicative form (dotted). (b) The simple product ( $\beta=1$ ) is optimal.

Panel of Figure 20 — Figure 20: Hyperparameter tuning for baselines. Neither PPO nor PMPO matches DG despite extensive sweeps.

Panel of Figure 21 — Figure 21: DG’s advantage is robust to learning rate (a) and persists asymptotically (b) .

Panel of Figure 22 — Figure 22: Reward structures. Bag-of-Tokens credits all correct tokens; Sequential stops at the first error.

Figure 23: Learning curves across 8 task variants. Top: Bag-of-Tokens. Bottom: Sequential. DG (red) achieves the lowest error in all configurations.

Figure 24: Additive mixtures (colored lines) vs. multiplicative DG (red dashed). No additive configuration matches DG.

Figure 25: Learning curves for 28 Control Suite environments. DG (red) consistently matches or exceeds the hedonic baseline (purple), PPO (blue), and MPO (green).

Panel of Figure 26 — Figure 26: Hyperparameter sensitivity on cartpole:swingup . Default values (dashed) perform comparably to alternatives.

Figure 27: Aggregate regret against tuned SOTA baselines. DG (pink) outperforms tuned MPO (gold) and SAC (blue).

Figure 28: Per-environment regret at 10M steps. DG (pink) consistently achieves low regret across the suite.

Policy Gradient

Policy gradient methods improve a decision-making policy by estimating how its actions affect expected reward.

See all 68 papers

Delightful Distributed Policy Gradient
Mar 20, 2026Ian OsbandPolicy Gradient Methods Policy Learning
Google DeepMind
Delightful Gradients Accelerate Corner Escape
May 12, 2026Jincheng Mei, Ian OsbandMulti-Armed Bandits Policy Gradient
Google DeepMind
Does This Gradient Spark Joy?
Mar 20, 2026Ian OsbandBandit Learning Policy Gradient
Google DeepMind

Mar 20, 2026J/K move · Enter open · S save

Delightful Distributed Policy Gradient

Ian Osband

Google DeepMind

Policy Gradient Methods Policy Learning Distributed RL Sample-Efficient RL Off-Policy RL

Open paper AskPDF arXiv

Distributed reinforcement learning trains on data from stale, buggy, or mismatched actors, producing actions with high surprisal (negative log-probability) under the learner's policy. The core difficulty is not surprising data per se, but negative learning from surprising data. High-surprisal failures can dominate finite-batch updates through large perpendicular components, while high-surprisal successes reveal opportunities the current policy would otherwise miss. The Delightful Policy Gradient (DG) separates these cases by gating each update with delight, the product of advantage and surprisal, suppressing rare failures and preserving rare successes without behavior probabilities. In a tabular analysis, DG suppresses the perpendicular second moment of high-surprisal failures by a policy-overlap factor that vanishes as the learner improves. The advantage sign is essential for surprisal-based filtering: any learner-probability-only gate that suppresses rare failures also suppresses rare successes. On MNIST with simulated staleness, DG without off-policy correction outperforms importance-weighted PG with exact behavior probabilities. On a transformer sequence task with staleness, actor bugs, reward corruption, and rare discovery, DG often achieves nearly order-of-magnitude lower error. When all four frictions act simultaneously, its sample-efficiency advantage is order-of-magnitude and grows with task complexity.

Abstract

Policy Gradient

Explore similar work

Delightful Distributed Policy Gradient

Delightful Gradients Accelerate Corner Escape

Does This Gradient Spark Joy?