Does This Gradient Spark Joy?

Ian Osband

cs.LGMar 20, 2026

Does This Gradient Spark Joy?

Authors: Ian Osband

Organizations: Google DeepMind

Abstract

Policy gradient computes a backward pass for every sample, even though the backward pass is expensive and most samples carry little learning value. The Delightful Policy Gradient (DG) provides a forward-pass signal of learning value: delight, the product of advantage and surprisal (negative log-probability). We introduce the Kondo gate, which compares delight against a compute price and pays for a backward pass only when the sample is worth it, thereby tracing a quality--cost Pareto frontier. In bandits, zero-price gating preserves useful gradient signal while removing perpendicular noise, and delight is a more reliable screening signal than additive combinations of value and surprise. On MNIST and transformer token reversal, the Kondo gate skips most backward passes while retaining nearly all of DG's learning quality, with gains that grow as problems get harder and backward passes become more expensive. Because the gate tolerates approximate delight, a cheap forward pass can screen samples before expensive backpropagation, suggesting a speculative-decoding-for-training paradigm.

Figures & tables

Panel of Figure 1 — Figure 1: PG, DG, and Kondo gate (DG-K) at $\rho=0.03$ on MNIST. (a) The Kondo gate matches DG despite computing 3% of backward passes. (b) It dominates by two orders of magnitude in backward-pass space. Averaged over 30 seeds; shading shows $\pm 1$ standard error.

Panel of Figure 2 — Figure 2: Gate rate sweep ( $\rho\in\{0.01,\ldots,1.0\}$ ), learning rate tuned per $\rho$ . (a) All gate rates converge to $\sim 0.5\%$ error eventually. (b) In backward-step space, smaller $\rho$ reaches any error with orders-of-magnitude fewer backward passes.

Figure 3: Compute speedup vs PG to reach 5% test error on MNIST, as a function of the backward/forward cost ratio. DG’s advantage is constant ( $\sim 2\times$ , better learning). DG-K’s advantage grows linearly with backward cost (fewer backward passes). At a typical ratio of $4\times$ , the Kondo gate is $6\times$ faster than PG.

Panel of Figure 4 — Figure 4: Noise robustness on MNIST. (a) Delight noise scaled relative to $\mathrm{std}(\chi)$ : DG tolerates $\sim 50\%$ ; DG-K degrades earlier. (b) Logit noise: DG is robust until $\sigma_{Z}\approx 1$ ; DG-K degrades faster. Both validate that approximate forward passes and approximate delight preserve the gate’s value.

Panel of Figure 5 — Figure 5: Priority signal comparison on MNIST. (a) Delight is robust across backward batch sizes; surprisal-only fails. (b) The additive mix collapses for $\alpha>0.3$ ; delight (product) is $\alpha$ -independent. Validates Proposition 2 .

Panel of Figure 6 — Figure 6: Gambling pathology on MNIST (Section 3 ). (a) Under homoskedastic noise, DG and PG degrade smoothly together. (b) Under differential noise on action $a{=}0$ , DG collapses sharply near $\sigma_{G}\approx 1$ while PG degrades gracefully, matching the $\sigma/\Delta$ threshold of Proposition 3 .

Figure 7: Token reversal ( $M{=}2$ , $H{=}5$ ): the agent must output the input in reverse.

Panel of Figure 8 — Figure 8: Token reversal learning curves ( $H=10,M=2$ ). 10 seeds; shading $\pm 1$ s.e.

Panel of Figure 9 — Figure 9: Scaling with vocabulary size: $M^{*}$ = largest vocabulary solved vs. compute. Fixed $\rho=3\%$ is too aggressive at large $M$ ; adaptive $\lambda=0$ is robust and preserves backward-compute gains.

Panel of Figure 10 — Figure 10: Scaling with sequence length: $H^{*}$ = longest sequence solved (reward $>0.75$ ) vs. compute. DG-K matches or exceeds DG and solves longer sequences at far lower backward cost.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 11 — Figure 11: Learning rate sweep on MNIST. All methods are optimal near $\mathrm{lr}=10^{-3}$ ; DG dominates across the range. Training and test error track closely, confirming no train/test gap.

Panel of Figure 12 — Figure 12: Test classification error on MNIST at $\rho=0.03$ . Same comparison as Figure 1 but with held-out test error. DG-K matches DG in forward-pass space and dominates in backward-pass space; the gate’s advantage generalizes beyond training error.

Figure 13: Forward-pass comparison across baselines on MNIST at $\rho=0.03$ . The Kondo gate matches or exceeds DG under all four baselines. Under the zero baseline, DG-K even outperforms DG in forward-pass space.

Figure 14: Backward-pass comparison across baselines on MNIST at $\rho=0.03$ . The Kondo gate dominates in backward-pass space under all four baselines: the two-orders-of-magnitude advantage is not baseline-dependent.

Figure 15: Empirical CDF of $\pi(y^{*})$ for kept vs. skipped samples at three training stages ( $\rho=0.03$ , 10,000 samples per stage). Kept samples have systematically lower $\pi(y^{*})$ : the gate selects the learning frontier where the model is correct but uncertain.

Panel of Figure 16 — Figure 16: Images kept vs. skipped by the Kondo gate at $\rho=0.03$ across three training stages. Each image shows ground truth $y$ , selected action $a$ , and $p=\pi(y^{*})$ . The gate progressively concentrates compute on the learning frontier.

Figure 17: Delight noise robustness (absolute scale): classification error vs. absolute noise $\sigma_{\chi}$ . Complements Figure 4(a) ; same qualitative pattern on an absolute scale.

Panel of Figure 18 — Figure 18: Average error vs. sequence length $H$ (log–log, $M=2$ ). Clean power laws emerge; DG-K compresses the backward axis.

Panel of Figure 19 — Figure 19: Average error vs. vocabulary size $M$ (log–log, $H=10$ ). The gate’s backward-pass advantage widens with $M$ .

Panel of Figure 20 — Figure 20: Final error vs. sequence length $H$ ( $M=2$ ). DG-K stays near zero where baselines degrade.

Panel of Figure 21 — Figure 21: Final error vs. vocabulary size $M$ ( $H=10$ ). Adaptive gate tracks DG; fixed gate degrades at large $M$ but dominates in backward-pass space.

Bandit Learning

See all 52 papers

Delightful Policy Gradient
Mar 15, 2026Ian OsbandPolicy Gradient Policy Gradient Methods
Google DeepMind
Delightful Gradients Accelerate Corner Escape
May 12, 2026Jincheng Mei, Ian OsbandMulti-Armed Bandits Policy Gradient
Google DeepMind
Delightful Distributed Policy Gradient
Mar 20, 2026Ian OsbandPolicy Gradient Methods Policy Learning
Google DeepMind

Mar 15, 2026J/K move · Enter open · S save

Delightful Policy Gradient

Ian Osband

Google DeepMind

Policy Gradient Policy Gradient Methods

Open paper AskPDF arXiv

Standard policy gradients weight each sampled action by advantage alone, regardless of how likely that action was under the current policy. This creates two pathologies: within a single decision context (e.g. one image or prompt), a rare negative-advantage action can disproportionately distort the update direction; across many such contexts in a batch, the expected gradient over-allocates budget to contexts the policy already handles well. We introduce the Delightful Policy Gradient (DG), which gates each term with a sigmoid of delight, the product of advantage and action surprisal (negative log-probability). For

K

-armed bandits, DG provably improves directional accuracy in a single context and, across multiple contexts, shifts the expected gradient strictly closer to the supervised cross-entropy oracle. This second effect is not variance reduction: it persists even with infinite samples. Empirically, DG outperforms REINFORCE, PPO, and advantage-weighted baselines across MNIST, transformer sequence modeling, and continuous control, with larger gains on harder tasks.

Abstract

Bandit Learning

Explore similar work

Delightful Policy Gradient

Delightful Gradients Accelerate Corner Escape

Delightful Distributed Policy Gradient