stat.MLJan 31, 2026

Persistent-Transient Policy Evaluation for Markov Chains via Minimal Peripheral Quotients

Authors: Yang Xu, Vaneet Aggarwal

Organizations: Purdue University, USA 47907

Abstract

We study fixed-policy evaluation for finite Markov chains that may be reducible and periodic. Classical evaluation methods with gain and bias decomposition are not always diagnostic: the gain records only invariant Cesàro averages, while persistent phase-dependent behavior is absorbed into the bias together with genuinely transient effects. We identify the real peripheral invariant subspace K(P)\mathcal{K}(P) of the transition matrix PP as the source of this ambiguity. Quotienting by K(P)\mathcal{K}(P) is the minimal exact quotient that removes all non-decaying modes and makes the remaining dynamics strictly stable. After choosing a gauge projection ΠΠ with kernel K(P)\mathcal{K}(P), the reward admits a unique decomposition r=gΠ⋆+(I−P)vΠ⋆r = g_Π^\star + (I-P)v_Π^\star, where gΠ⋆g_Π^\star is a persistent regime profile and vΠ⋆v_Π^\star is a gauge-fixed transient component. An exact comparison with classical normalized gain and bias shows that the new pair reallocates the same information into a persistent profile and a gauge-fixed representative of the transient quotient. This decomposition reconstructs finite-horizon returns, recovers statewise average reward, admits a transient-cost interpretation, and yields a stable estimator under a generative model.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

    Jul 29, 2026Zuyuan Zhang, Yongshan Chen, Mahdi Imani +1Markov Decision ProcessesPartially Observable RL

  2. Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

    Oct 7, 2026Huizhen Yu, Isaiah HeidtMarkov Decision ProcessesAverage-Reward RL

  3. Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation

    May 1, 2026Andrzej Ruszczynski, Tiangang ZhangMarkov Decision ProcessesReinforcement Learning