On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation

Alexander Galozy

cs.LGFeb 24, 2026

On the Structural Non-Preservation of Epistemic Behaviour under Policy Transformation

Authors: Alexander Galozy

Organizations: Center for Applied Intelligent Systems Research, Halmstad University, Sweden

Abstract

Reinforcement learning (RL) agents under partial observability often condition actions on internally accumulated information such as memory or inferred latent context. We formalise such information-conditioned interaction patterns as behavioural dependency: variation in action selection with respect to internal information under fixed observations. This induces a probe-relative notion of $ε$ -behavioural equivalence and a within-policy behavioural distance that quantifies probe sensitivity. We establish three structural results. First, the set of policies exhibiting non-trivial behavioural dependency is not closed under convex aggregation. Second, behavioural distance contracts under convex combination. Third, we prove a sufficient local condition under which gradient ascent on a skewed mixture objective decreases behavioural distance when a dominant-mode gradient aligns with the direction of steepest contraction. Minimal bandit and partially observable gridworld experiments provide controlled witnesses of these mechanisms. In the examined settings, behavioural distance decreases under convex aggregation and under continued optimisation with skewed latent priors, and in these experiments it precedes degradation under latent prior shift. These results identify structural conditions under which probe-conditioned behavioural separation is not preserved under common policy transformations.

Figures & tables

Figure 1: Behavioural probe evaluation and representative trajectories in the partially observable gridworld. Left: The evaluation protocol. The agent acquires latent mode information to induce an internal hidden state

h_{m}

. It is subsequently evaluated at a fixed observation

o^{\star}

to measure the conditional policy

\pi(\cdot\mid o^{\star},h_{m})

. Second: Probing policy. The agent maintains separated internal representations and successfully executes distinct actions contingent on the initial mode. Third: Shortcut policy. The agent consistently navigates toward the same goal irrespective of the latent mode. Right: Aggregated policy. The agent may execute the probing action, but it fails to exhibit the consistent information-conditioned action differences of the probing agent.

Figure 2: Robustness under latent prior shift. Left: Average return of three policies evaluated under a biased prior and a reversed prior. The Shortcut policy attains high return under the biased prior despite zero behavioural distance, but degrades sharply under prior reversal. The Probing policy remains stable across priors (

mean\pm se

). Right: Convex mixtures

\pi_{\alpha}=\alpha\pi_{\text{probe}}+(1-\alpha)\pi_{\text{shortcut}}

. Behavioural distance decreases linearly with

\alpha

, and robustness under prior shift decreases correspondingly, indicating that sensitivity to latent distribution shift is governed by probe-conditioned behavioural separation and not by biased-prior reward alone.

Figure 3: Optimisation-induced structural erosion under a heavily biased prior (98%). Left: Behavioural erosion. Return under the dominant prior remains stable while reversed-prior performance degrades. Behavioural distance decreases during biased optimisation and stabilises at a lower value. Middle: Representational erosion. Absolute and normalised hidden-state separation contract over training. Right: Mechanistic verification. Projection of task gradients onto

\mathbf{v}_{d}=-\nabla d/\|\nabla d\|

yields a persistent positive net structural force under the biased prior, consistent with Theorem 1 . Shaded regions denote

\pm

one standard deviation across 10 seeds.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Abstract Epistemic MDP. A three-phase partially observable task consisting of a probe phase revealing latent mode $m$ , a distractor delay phase independent of $m$ , and a final evaluation phase at a fixed observation $o^{\ast}$ that requires retention of probe information. The task enforces an epistemic bottleneck between information acquisition and reward.

Parameter	Value
Optimizer	Adam
Learning Rate	$2\times 10^{-4}$
Weight Decay	$1\times 10^{-3}$
Batch Size	32 episodes
Discount Factor ( $\gamma$ )	0.99
Entropy Coefficient	0.02

Appendix

Table 1: Hyperparameters for Stage 2 (Erosion) Experiments

Prior Skew ( $\delta$ )	Final Behavioural Distance $d(\pi)$	Final Hidden Distance $h_{\text{dist}}$
$\delta=0.600$	$1.939\pm 0.010$	$2.809\pm 0.153$
$\delta=0.800$	$1.911\pm 0.027$	$2.714\pm 0.208$
$\delta=0.900$	$1.853\pm 0.039$	$2.582\pm 0.153$
$\delta=0.950$	$1.743\pm 0.092$	$2.443\pm 0.141$
$\delta=0.965$	$1.517\pm 0.449$	$2.133\pm 0.426$
$\delta=0.980$	$1.369\pm 0.238$	$2.050\pm 0.203$

Appendix

Table 2: Final Epistemic Metrics across Prior Skew (

\delta

) Sweep (

mean\pm se

, 10 seeds)

Figure 5: Sensitivity to Prior Skew. (Left) Functional degradation on the rare mode most pronounced at

\delta=0.98

. (Center) Behavioural distance

d(\pi)

decreasing as skew increases. (Right) Decay of internal representation geometry across biased priors, preceding functional failure.

Hidden Units ( $H$ )	Final Behavioural Distance $d(\pi)$	Final Hidden Distance $h_{\text{dist}}$
$32$	$1.350\pm 0.171$	$2.025\pm 0.205$
$64$	$1.422\pm 0.437$	$2.357\pm 0.414$
$128$	$1.266\pm 0.618$	$2.565\pm 0.607$

Appendix

Table 3: Final Epistemic Metrics by Network Capacity (

H

) at

\delta=0.98

(

mean\pm se

, 10 seeds)

Figure 6: Capacity Sweep Results. (Left) Rare-mode reward exhibits increased instability at

H=128

. (Center) Behavioural erosion remains comparable across network sizes. (Right) Representational distance decays across all capacities under extreme skew.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Dynamical Priors as a Training Objective in Reinforcement Learning
Apr 23, 2026Sukesh SubaharanReinforcement Learning Policy Gradient Methods
Coimbatore Medical College Coimbatore, TN India
To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning
Oct 3, 2025Yuda Song, Dhruv Rohatgi, Aarti Singh +1Partially Observable RL Learning Using Privileged Information
Carnegie Mellon University · Massachusetts Institute of Technology · Aurora Innovation
Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
Jun 24, 2026Yivan Zhang, Ziyan Luo, Manuel BaltieriReinforcement Learning State Abstraction
The University of Tokyo · RIKEN AIP · Mila - Quebec Artificial Intelligence Institute +3

Apr 23, 2026J/K move · Enter open · S save

Dynamical Priors as a Training Objective in Reinforcement Learning

Sukesh Subaharan

Coimbatore Medical College Coimbatore, TN India

Reinforcement Learning Policy Gradient Methods Policy Learning

Open paper AskPDF arXiv

Standard reinforcement learning (RL) optimizes policies for reward but imposes few constraints on how decisions evolve over time. As a result, policies may achieve high performance while exhibiting temporally incoherent behavior such as abrupt confidence shifts, oscillations, or degenerate inactivity. We introduce Dynamical Prior Reinforcement Learning (DP-RL), a training framework that augments policy gradient learning with an auxiliary loss derived from external state dynamics that implement evidence accumulation and hysteresis. Without modifying the reward, environment, or policy architecture, this prior shapes the temporal evolution of action probabilities during learning. Across three minimal environments, we show that dynamical priors systematically alter decision trajectories in task-dependent ways, promoting temporally structured behavior that cannot be explained by generic smoothing. These results demonstrate that training objectives alone can control the temporal geometry of decision-making in RL agents.

Abstract

Reinforcement Learning

Explore similar work

Dynamical Priors as a Training Objective in Reinforcement Learning

To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learning

Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning