Singular learning theory characterizes Bayesian learning as an evolving tradeoff between accuracy and complexity, with transitions between qualitatively different solutions as sample size increases. We extend this theory to reinforcement learning, proving that the concentration of a generalized posterior over policies is governed by the local learning coefficient (LLC), an invariant of the geometry of the regret function. This theory predicts that deep reinforcement learning with SGD should proceed from simple policies with high regret to complex policies with low regret. We verify this prediction empirically in a gridworld environment exhibiting stagewise policy development: phase transitions over training manifest as "opposing staircases" where regret decreases sharply while the LLC increases.
Figures & tables
Figure 2: Two possible states the environment can be in, together with the observation seen by the agent.
Figure 3: A series of plots illustrating the model transitions into phases π1 , π2b and π3 respectively, as a function of mixing parameter α and discount rate γ . We report the number of checkpoints (policy gradient steps) since training began before we enter each phase. The red line in phase 3 represents when training runs were terminated.
Figure 4: Distribution of average LLC estimates across phases. The LLC has been estimated on distribution with α=0.6 and γ=0.98 .
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison of 2D regression and phase regression error.
Figure 6: Model architecture diagram. Diagram replicated from Espeholt et al. [2018] with modifications.
Parameter
Value
Rollouts
# parallel environments
9600
Rollout length
64
# environment steps per gradient step
614.4k
Discount factor, γ
∈{ 0.97, 0.98, 0.99 }
REINFORCE updates
Appendix
Table 1: Hyperparameters used for all methods and environments.
Notation
Description
w∗∈W
Initial point in W
β∈R>0
Inverse temperature
n∈N
Number of training steps
σ2∈R>0
Noise variance
T∈N
SGLD chain length
ε∈R>0
SGLD step size
Appendix
Table 2: A list of the hyperparameters on which the random variable λSGLD(w∗) depends. Note that the estimator does not depend on n and β independently, only on their product nβ .
Figure 7: LLC estimates (blue curves) for models with γ=0.98 and α=0.6 .These plots also show the weight L2 norms (orange curves), which monotonically increase throughout training, and the regret curves in pink. The weight norms range from 70 to 77.
Figure 8: LLC (blue line) estimated with γ=0.98 and α=0 for models trained with values of α and γ indicated in the titles of the panels. The runs that share α and γ differ only by the random seed with which they were trained. The brown curve shows the on-distribution regret and the green curve shows the regret measured with α=0 .
Figure 9: Distribution of the random variable d representing the difference in the rate of change of the LLC estimate with respect to the regret between phases 1 and 2b, and between phases 2b and 3, measured across the 23 iid runs, of 25, for which γ=0.98 and α=0.6 and all three phases are detected.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
We propose a cross-fitted debiasing device for policy learning from offline data. A key consequence of the resulting learning principle is N regret even for policy classes with complexity greater than Donsker, provided a product-of-errors nuisance remainder is O(N−1/2). The regret bound factors into a plug-in policy error factor governed by policy-class complexity and an environment nuisance factor governed by the complexity of the environment dynamics, making explicit how one may be traded against the other.
Distributed reinforcement learning trains on data from stale, buggy, or mismatched actors, producing actions with high surprisal (negative log-probability) under the learner's policy. The core difficulty is not surprising data per se, but negative learning from surprising data. High-surprisal failures can dominate finite-batch updates through large perpendicular components, while high-surprisal successes reveal opportunities the current policy would otherwise miss. The Delightful Policy Gradient (DG) separates these cases by gating each update with delight, the product of advantage and surprisal, suppressing rare failures and preserving rare successes without behavior probabilities. In a tabular analysis, DG suppresses the perpendicular second moment of high-surprisal failures by a policy-overlap factor that vanishes as the learner improves. The advantage sign is essential for surprisal-based filtering: any learner-probability-only gate that suppresses rare failures also suppresses rare successes. On MNIST with simulated staleness, DG without off-policy correction outperforms importance-weighted PG with exact behavior probabilities. On a transformer sequence task with staleness, actor bugs, reward corruption, and rare discovery, DG often achieves nearly order-of-magnitude lower error. When all four frictions act simultaneously, its sample-efficiency advantage is order-of-magnitude and grows with task complexity.
Susceptibilities are a technique for neural network interpretability that studies the response of posterior expectation values of observables to perturbations of the loss. We generalize this construction to the setting of the regret in deep reinforcement learning and investigate the utility of susceptibilities in a simple gridworld model that nevertheless exhibits non-trivial stagewise development. We argue that susceptibilities reveal internal features of the development of the model in parameter space that one cannot detect purely by studying the development of the learned policy. We validate these results with activation-steering, and discuss the framework's extension to RLHF post-training.