Reinforcement learning (RL) agents under partial observability often condition actions on internally accumulated information such as memory or inferred latent context. We formalise such information-conditioned interaction patterns as behavioural dependency: variation in action selection with respect to internal information under fixed observations. This induces a probe-relative notion of ε-behavioural equivalence and a within-policy behavioural distance that quantifies probe sensitivity. We establish three structural results. First, the set of policies exhibiting non-trivial behavioural dependency is not closed under convex aggregation. Second, behavioural distance contracts under convex combination. Third, we prove a sufficient local condition under which gradient ascent on a skewed mixture objective decreases behavioural distance when a dominant-mode gradient aligns with the direction of steepest contraction. Minimal bandit and partially observable gridworld experiments provide controlled witnesses of these mechanisms. In the examined settings, behavioural distance decreases under convex aggregation and under continued optimisation with skewed latent priors, and in these experiments it precedes degradation under latent prior shift. These results identify structural conditions under which probe-conditioned behavioural separation is not preserved under common policy transformations.
Figures & tables
Figure 1: Behavioural probe evaluation and representative trajectories in the partially observable gridworld. Left: The evaluation protocol. The agent acquires latent mode information to induce an internal hidden state hm . It is subsequently evaluated at a fixed observation o⋆ to measure the conditional policy π(⋅∣o⋆,hm) . Second: Probing policy. The agent maintains separated internal representations and successfully executes distinct actions contingent on the initial mode. Third: Shortcut policy. The agent consistently navigates toward the same goal irrespective of the latent mode. Right: Aggregated policy. The agent may execute the probing action, but it fails to exhibit the consistent information-conditioned action differences of the probing agent.
Figure 2: Robustness under latent prior shift. Left: Average return of three policies evaluated under a biased prior and a reversed prior. The Shortcut policy attains high return under the biased prior despite zero behavioural distance, but degrades sharply under prior reversal. The Probing policy remains stable across priors ( mean±se ). Right: Convex mixtures πα=απprobe+(1−α)πshortcut . Behavioural distance decreases linearly with α , and robustness under prior shift decreases correspondingly, indicating that sensitivity to latent distribution shift is governed by probe-conditioned behavioural separation and not by biased-prior reward alone.
Figure 3: Optimisation-induced structural erosion under a heavily biased prior (98%). Left: Behavioural erosion. Return under the dominant prior remains stable while reversed-prior performance degrades. Behavioural distance decreases during biased optimisation and stabilises at a lower value. Middle: Representational erosion. Absolute and normalised hidden-state separation contract over training. Right: Mechanistic verification. Projection of task gradients onto vd=−∇d/∥∇d∥ yields a persistent positive net structural force under the biased prior, consistent with Theorem 1 . Shaded regions denote ± one standard deviation across 10 seeds.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Abstract Epistemic MDP. A three-phase partially observable task consisting of a probe phase revealing latent mode m , a distractor delay phase independent of m , and a final evaluation phase at a fixed observation o∗ that requires retention of probe information. The task enforces an epistemic bottleneck between information acquisition and reward.
Parameter
Value
Optimizer
Adam
Learning Rate
2×10−4
Weight Decay
1×10−3
Batch Size
32 episodes
Discount Factor ( γ )
0.99
Entropy Coefficient
0.02
Appendix
Table 1: Hyperparameters for Stage 2 (Erosion) Experiments
Prior Skew ( δ )
Final Behavioural Distance d(π)
Final Hidden Distance hdist
δ=0.600
1.939±0.010
2.809±0.153
δ=0.800
1.911±0.027
2.714±0.208
δ=0.900
1.853±0.039
2.582±0.153
δ=0.950
1.743±0.092
2.443±0.141
δ=0.965
1.517±0.449
2.133±0.426
δ=0.980
1.369±0.238
2.050±0.203
Appendix
Table 2: Final Epistemic Metrics across Prior Skew ( δ ) Sweep ( mean±se , 10 seeds)
Figure 5: Sensitivity to Prior Skew. (Left) Functional degradation on the rare mode most pronounced at δ=0.98 . (Center) Behavioural distance d(π) decreasing as skew increases. (Right) Decay of internal representation geometry across biased priors, preceding functional failure.
Hidden Units ( H )
Final Behavioural Distance d(π)
Final Hidden Distance hdist
32
1.350±0.171
2.025±0.205
64
1.422±0.437
2.357±0.414
128
1.266±0.618
2.565±0.607
Appendix
Table 3: Final Epistemic Metrics by Network Capacity ( H ) at δ=0.98 ( mean±se , 10 seeds)
Figure 6: Capacity Sweep Results. (Left) Rare-mode reward exhibits increased instability at H=128 . (Center) Behavioural erosion remains comparable across network sizes. (Right) Representational distance decays across all capacities under extreme skew.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Standard reinforcement learning (RL) optimizes policies for reward but imposes few constraints on how decisions evolve over time. As a result, policies may achieve high performance while exhibiting temporally incoherent behavior such as abrupt confidence shifts, oscillations, or degenerate inactivity. We introduce Dynamical Prior Reinforcement Learning (DP-RL), a training framework that augments policy gradient learning with an auxiliary loss derived from external state dynamics that implement evidence accumulation and hysteresis. Without modifying the reward, environment, or policy architecture, this prior shapes the temporal evolution of action probabilities during learning. Across three minimal environments, we show that dynamical priors systematically alter decision trajectories in task-dependent ways, promoting temporally structured behavior that cannot be explained by generic smoothing. These results demonstrate that training objectives alone can control the temporal geometry of decision-making in RL agents.
Partial observability is a notorious challenge in reinforcement learning (RL), due to the need to learn complex, history-dependent policies. Recent empirical successes have used privileged expert distillation--which leverages availability of latent state information during training (e.g., from a simulator) to learn and imitate the optimal latent, Markovian policy--to disentangle the task of "learning to see" from "learning to act". While expert distillation is more computationally efficient than RL without latent state information, it also has well-documented failure modes. In this paper--through a simple but instructive theoretical model called the perturbed Block MDP, and controlled experiments on challenging simulated locomotion tasks--we investigate the algorithmic trade-off between privileged expert distillation and standard RL without privileged information. Our main findings are: (1) The trade-off empirically hinges on the stochasticity of the latent dynamics, as theoretically predicted by contrasting approximate decodability with belief contraction in the perturbed Block MDP; and (2) The optimal latent policy is not always the best latent policy to distill. Our results suggest new guidelines for effectively exploiting privileged information, potentially advancing the efficiency of policy learning across many practical partially observable domains.
Yuda Song, Dhruv Rohatgi, Aarti Singh +1
Carnegie Mellon University · Massachusetts Institute of Technology · Aurora Innovation
State abstraction plays a key role in scaling reinforcement learning to complex but structured systems. In studying such systems, a wide range of behavioral structures have been studied in reinforcement learning, including value functions, invariants, bisimulation relations, and behavioral metrics. However, a general principle for determining what structures are provably preserved under state abstraction is still lacking. In this paper, we present a unified framework for defining and analyzing behavioral structures in reinforcement learning. Our framework provides a compositional way to specify behavioral semantics based on local, one-step descriptions of system dynamics. Using this framework, we establish results showing how behavioral structures can be safely transferred between abstract and concrete systems. We further show how to construct quantitative metrics from logical behavioral semantics with soundness guarantees. Together, these results provide a principled foundation for reasoning about behaviors under state abstraction in reinforcement learning and offer reusable definition and proof principles for a broad class of behavioral structures in reinforcement learning.
Yivan Zhang, Ziyan Luo, Manuel Baltieri
The University of Tokyo · RIKEN AIP · Mila - Quebec Artificial Intelligence Institute +3