cs.LGApr 9, 2026

StructRL: Recovering Dynamic Programming Structure from Learning Dynamics in Distributional Reinforcement Learning

Authors: Ivo Nowak

Organizations: HAW-Hamburg

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement learning is typically treated as a uniform, data-driven optimization process, where updates are guided by rewards and temporal-difference errors without explicitly exploiting global structure. In contrast, dynamic programming methods rely on structured information propagation, enabling efficient and stable learning. In this paper, we provide evidence that such structure can be recovered from the learning dynamics of distributional reinforcement learning. By analyzing the temporal evolution of return distributions, we identify signals that capture when and where learning occurs in the state space. In particular, we introduce a temporal learning indicator t*(s) that reflects when a state undergoes its strongest learning update during training. Empirically, this signal induces an ordering over states that is consistent with a dynamic programming-style propagation of information. Building on this observation, we propose StructRL, a framework that exploits these signals to guide sampling in alignment with the emerging propagation structure. Our preliminary results suggest that distributional learning dynamics provide a mechanism to recover and exploit dynamic programming-like structure without requiring an explicit model. This offers a new perspective on reinforcement learning, where learning can be interpreted as a structured propagation process rather than a purely uniform optimization procedure.

Figures & tables

Fig. 1: Illustration of the StructRL mechanism. The temporal learning signal t∗(s)t^{*}(s) is used to identify a small set of seed states, from which a structural distance function d(s)d(s) is constructed. Learning is then biased toward transitions that reduce this distance (red arrows), resulting in a structured propagation of information across the state space.
Fig. 2: Normalized heatmaps of structural distance d(s)d(s) , return variance σ(s)\sigma(s) , and temporal learning indicator t∗(s)t^{*}(s) .
Fig. 3: Relationship between structural distance d(s)d(s) and temporal learning indicator t∗(s)t^{*}(s) .
Fig. 4: Comparison of sampling strategies: uniform, σ\sigma -based, and t∗t^{*} -based sampling. The color scale indicates normalized state visitation frequency.
Fig. 5: Comparison between StructRL and a standard C51 baseline [ 1 ] . After the common exploration period, StructRL rapidly improves performance, while the C51 baseline continues to reach the goal only sporadically. This suggests that the gain comes from exploiting the recovered propagation structure rather than from more favorable initial exploration.
More in this topic

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Explore similar work

CardsList
Apr 23, 2026

Dynamical Priors as a Training Objective in Reinforcement Learning

Standard reinforcement learning (RL) optimizes policies for reward but imposes few constraints on how decisions evolve over time. As a result, policies may achieve high performance while exhibiting temporally incoherent behavior such as abrupt confidence shifts, oscillations, or degenerate inactivity. We introduce Dynamical Prior Reinforcement Learning (DP-RL), a training framework that augments policy gradient learning with an auxiliary loss derived from external state dynamics that implement evidence accumulation and hysteresis. Without modifying the reward, environment, or policy architecture, this prior shapes the temporal evolution of action probabilities during learning. Across three minimal environments, we show that dynamical priors systematically alter decision trajectories in task-dependent ways, promoting temporally structured behavior that cannot be explained by generic smoothing. These results demonstrate that training objectives alone can control the temporal geometry of decision-making in RL agents.

Sukesh Subaharan

Coimbatore Medical College Coimbatore, TN India

Reinforcement LearningPolicy Gradient MethodsPolicy Learning
AskPDF
May 28, 2026

On Distributional Reinforcement Learning in Chaotic Dynamical Systems

Chaotic dynamical systems pose a fundamental challenge for Reinforcement Learning (RL): exponential sensitivity to initial conditions induces high-variance bootstrap targets and poorly conditioned gradient updates. Chaotic dynamics arise across scientific and engineering domains, from fluid flows and climate systems to multi-agent systems, where reliable learning is highly desirable. Standard RL methods optimise expected returns through scalar value functions, implicitly averaging over diverging trajectories and entangling trajectory level instability with the learning objective. We show that under mild statistical stability assumptions, the return distribution evolves more regularly than individual trajectories when measured under the 11-Wasserstein metric, yielding a smoother distributional Bellman objective. By aligning optimisation with this measure level structure, distributional RL provides better conditioned learning. We offer a principled explanation for the advantages of distributional methods in chaotic systems and the geometries of RL objectives under chaos.

James Rudd-Jones, Mirco Musolesi, María Pérez-Ortiz

Centre for Artificial Intelligence Department of Computer Science University College London London, UK · Department of Computer Science and Engineering, University of Bologna Bologna, Italy

Distributional RLReinforcement LearningChaotic Dynamical Systems
AskPDF
Mar 13, 2026

A Spectral Revisit of the Distributional Bellman Operator under the Cramér Metric

Distributional reinforcement learning (DRL) studies the evolution of full return distributions under Bellman updates rather than focusing on expected values. A classical result is that the distributional Bellman operator is contractive under the Cramér metric, which corresponds to an L2L^2 geometry on differences of cumulative distribution functions (CDFs). While this contraction ensures stability of policy evaluation, existing analyses remain largely metric, focusing on contraction properties without elucidating the structural action of the Bellman update on distributions. In this work, we analyse distributional Bellman dynamics directly at the level of CDFs, treating the Cramér geometry as the intrinsic analytical setting. At this level, the Bellman update acts affinely on CDFs and linearly on differences between CDFs, and its contraction property yields a uniform bound on this linear action. Building on this intrinsic formulation, we construct a family of regularised spectral Hilbert representations that realise the CDF-level geometry by exact conjugation, without modifying the underlying Bellman dynamics. The regularisation affects only the geometry and vanishes in the zero-regularisation limit, recovering the native Cramér metric. This framework clarifies the operator structure underlying distributional Bellman updates and provides a foundation for further functional and operator-theoretic analyses in DRL.

Keru Wang, Yixin Deng, Yao Lyu +2

School of Electrical and Electronic Engineering, University College Dublin, Dublin, Ireland · School of Vehicle and Mobility, Tsinghua University, Beijing, China · College of Artificial Intelligence, Tsinghua University, Beijing, China

Distributional RL
AskPDF