Single or Multiple Policies for Phase-Structured Reinforcement Learning?

Guilhem Loussouarn; Nancy Nayak; Kin K. Leung

cs.LGOct 2, 2026

Single or Multiple Policies for Phase-Structured Reinforcement Learning?

Authors: Guilhem Loussouarn, Nancy Nayak, Kin K. Leung

Organizations: Department of Electrical and Electronic Engineering Imperial College London, UK

Abstract

Many reinforcement-learning (RL) problems are non-stationary yet structured and can be decomposed into phases, each with its own transition probabilities and reward functions. When the phase sequence is known, the common solution augments the state with information to satisfy the Markovian property and applies standard RL techniques. However, prior work finds that the multi-policy approach for different phases can outperform a single state-augmented policy shared among the phases, for reasons that remain unclear. In this work, we first show that the shared policy can theoretically achieve performance of any multi-policy solution. However, whether a multi-policy solution can perform better than the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, as well as, for multi-policy solutions, the sample efficiency and loss of continuity from one policy to another. We propose a regime-based phase decomposition method to identify which policy can provide better performance. The method is based on consideration of the duration of transient system dynamics relative to the duration of the quasi-stationary period. Numerical experiments are conducted with different non-stationary RL problems to validate our four major hypotheses: (a) longer phase durations favor multi-policies, (b) the heterogeneity between phases increases the burden on single policy, (c) multi-policies need sufficient data for each phase, and (d) environment-specific transition dynamics between phases can affect which policy is preferable.

Figures & tables

Panel of Figure 1 — Figure 1: Conceptual framework for architecture selection in phase-structured MDPs. (a) Each phase $m$ of duration $T_{m}$ exhibits local dynamics $(p_{m},r_{m})$ and can be split into a transient regime of $L_{m}$ steps (dominated by handoff states inherited from the preceding phase) and a quasi-stationary (QS) regime of $T_{m}-L_{m}$ steps. (b) The net trade-off between specialization and sharing depends on four diagnostic mechanisms: handoff cost $K_{m}$ who favours coordinated policies; settled gain $G_{m}$ representing the advantage of specialization learning; task heterogeneity $H_{m}$ , which represents the difference between faces and can affect shared learning; and phase-local data budget $B_{m}$ , which predominantly affects multipolicy learning. (c) Combining these diagnostics places the problem on a continuum from full parameter sharing ( Single Shared ) to full specialization ( Multipolicy ), with modular architectures ( Multihead ) occupying the intermediate regime.

$t$	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	…
$m=\sigma(t)$	0	0	1	1	1	2	2	3	3	3	3	0	2	2	0	0	0	…
$\tau(t)$	0	1	0	1	2	0	1	0	1	2	3	0	0	1	0	1	2	…

Table 1: A phase schedule with phase

m\in\mathcal{M}=\{0,1,2,3\}

Regime	Governing Condition	Recommended Architecture
Insufficient $B_{m}$	Specialist data starvation	Shared ; multihead if outputs differ
Enough $B_{m}$ , low $H_{m}$	High representation transfer	Shared or Multihead
Enough $B_{m}$ , high $H_{m}$ , low $\overline{K}_{m}$	High local learning burden	Specialized Multi-Policy
Enough $B_{m}$ , high $H_{m}$ and high $\overline{K}_{m}$	Severe boundary handoff cost	Shared or Coordinated Specialists
Uncertain $[\underline{R},\overline{R}]$	Uncertified bounds	Multihead / Model Selection

Table 2: Architecture selection regimes derived from the transient–quasi-stationary framework.

Panel of Figure 2 — Figure 2: Main HAPS regime evidence. Left: at fixed $M=20$ , phase-clock sharing leads at $T=100$ , multihead and multipolicy are close at $T=200$ , and multipolicy leads at $T\in\{500,2000\}$ . Right: increasing the number of phases preserves the specialist advantage at $T=500$ and exposes the degradation of full sharing at $M=20$ . This is consistent with, but does not prove, amortization of boundary costs.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

$t$	0	1	2	3	4	5	6	7	8	9	10	11	12	13	$\dots$
$m=\sigma(t)$	0	0	1	1	1	2	2	3	3	3	3	0	0	1	$\dots$
$\tau(t)$	0	1	0	1	2	0	1	0	1	2	3	0	1	0	$\dots$

Appendix

Table 3: Periodic phase schedule and unique state augmentation pairs

(\sigma(t),\tau(t))

Figure 3: Geometric intuition of policy class equivalence and local specialist learning. (a) The common achievable policy space

\Pi^{\mathrm{sh}}

encompasses both shared and multi-policy solutions, as established by Proposition 2 . Depending on the alignment of phase-local entry distributions with deployment handoffs, exact multi-policy assembly yields either a specialization gain (

\Delta\bar{J}>0

) or a handoff mismatch penalty (

\Delta\bar{J}<0

). (b) Individual phase specialists

\widehat{\pi}_{m}^{\mathrm{sp}}

are trained independently within local policy spaces

\Pi_{m}^{\mathrm{sp}}

under phase-local entry distributions. The tuple

\widehat{\boldsymbol{\pi}}^{\mathrm{sp}}

is assembled via the exact gating map

\Gamma

with zero projection loss.

Setting	HAPS	PhasePendulum	PhaseGridWorld
Hidden layers	$[256,256]$	$[256,256]$	$[256,256]$
Parallel environments	8	8	8
Rollout steps per environment	2048	256	256
Minibatch size	1024	256	256
Epochs per update	10	10	10
Discount $\gamma$	0.99	0.99	0.99

Appendix

Table 4: Default PPO settings.

Panel of Figure 4 — Figure 4: PhasePendulum construction.

Figure 5: Representative maps from the ordinary PhaseGridWorld maximum jump suite used in the reported experiments ( $9\times 9$ , $M=10$ , suite and obstacle seed 0). Columns show the same three phase indices at obstacle sampling probabilities $D=0.25$ and $D=0.40$ . Obstacles are sampled independently before the maximum-jump ordering is selected, and the active goal cell is always cleared; consequently, the realized obstacle count need not equal exactly $81D$ . These panels illustrate environment structure rather than policy performance.

Panel of Figure 6 — Figure 6: Controlled PhaseGridWorld boundary constructions at the start of phase 1. The $9\times 9$ geometry, goals, wall, and bottleneck gate match the environment implementation. The orange marker is an illustrative carried entry state, not a sampled trajectory. In the aligned case, the goal remains at $(7,4)$ . In the other cases, it switches from $(1,4)$ to $(7,4)$ . Opening the gate at $(4,7)$ turns a disconnected trap into a recoverable, but long-path, bottleneck.

Panel of Figure 7 — Figure 7: Controlled two-phase handoff environment. In phase 0, taking the larger local reward $g_{00}$ places the agent in a component that becomes disconnected from $g_{1}$ after the boundary. Taking the lower reward $g_{01}$ preserves phase-1 reachability.

Figure 8: Matched HAPS behavior within-phase at age $300$ for five evenly spaced phases. Rows are architectures and columns are phases. The phase subset $\{0,4,8\}$ was fixed by index rather than selected by performance. All methods see the same demand realization. This single-seed visualization makes phase-wise adaptation directly comparable, while Figure 2 reports the confirmatory five-seed returns.

Budget / Setup	Method	Reward/step
$5\text{M}$ budget	Single+clock / Multihead / Multipolicy+clock	26.16 / 33.52 / 43.33
$10\text{M}$ budget	Single+clock / Multihead / Multipolicy+clock	31.16 / 35.41 / 46.39
$20\text{M}$ budget	Single+clock / Multihead / Multipolicy+clock	30.10 / 37.37 / 46.72
$20\text{M}$ budget	Multipolicy without clock	46.57
$5\text{M}$ gradient surgery	Single+clock	25.54
$5\text{M}$ gradient surgery	PCGrad-style	28.87

Appendix

Table 5: Performance summary on HAPS (

M=20,T=500

). Values denote mean raw reward per step averaged across random seeds

\{0,1,2\}

. Bold indicates the highest score.

Architecture	Reward/step	Architecture	Reward/step
Single, default	26.04	Single+clock, default	27.06
Single, big	22.48	Single+clock, big	24.27
Single, deep	24.73	Single+clock, deep	29.51
Single+clock, 6 layers	29.04	Single+clock, 8 layers	23.30
Multihead, default	32.76	Multihead, deep	27.47
Multipolicy	42.34

Appendix

Table 6: HAPS capacity and depth controls at

M=20,T=500

. Values denote mean reward per step across three seeds. “Big” doubles hidden width; “Deep” uses four hidden layers. The six- and eight-layer entries denote depth-stress runs.

Control	Setting	Single+clock	Multihead	Multipolicy
Phase order	Adjacent	34.58	46.80	48.41
Phase order	Random	27.47	38.17	40.59
Phase order	Maximum jump	31.17	34.88	41.06
Clock/handoff, $T=200$	Standard / clock-aware specialist	–	–	37.47 / 38.50
Clock/handoff, $T=200$	Handoff / handoff+warm start	–	–	31.75 / 27.14
Clock/handoff, $T=500$	Standard / clock-aware specialist	–	–	46.35 / 46.75

Appendix

Table 7: Exploratory HAPS phase-order and handoff controls (means over three seeds). The phase-order sweep uses

M=20,T=500

under the original sample budget. The handoff sweep uses

M=20

under a

20\text{M}

total interaction budget (

1\text{M}

per phase).

$M$	$T$	Multipolicy	Warm	Handoff	Handoff+warm
10	500	44.84	16.82	32.91	17.67
10	1000	48.21	14.50	39.55	14.13
20	500	46.37	13.44	37.14	11.27
20	1000	48.60	11.14	41.69	9.92

Appendix

Table 8: Exploratory HAPS warm-start controls with

B_{0}=1\text{M}

interactions per phase under equal total budgets (

MB_{0}

). Values denote mean reward per step across seeds

\{0,1,2\}

. “Warm” denotes budget-matched pretraining followed by specialist fine-tuning.

$M$	Architecture	$A_{g}$	$C_{g}$	$f_{\mathrm{harm}}$	$\overline{\cos(g_{m},g)}$	$\overline{\cos(g_{m},g_{k})}$	$f_{\mathrm{pair}}^{-}$
2	Single+clock	0.950	0.050	0.000	0.675	-0.053	0.600
5	Single+clock	0.970	0.030	0.000	0.399	-0.012	0.520
10	Single+clock	0.968	0.032	0.000	0.264	0.000	0.462
20	Single+clock	1.046	-0.046	0.020	0.195	0.002	0.485
2	Multihead	1.003	-0.003	0.000	0.706	0.003	0.600
5	Multihead	1.017	-0.017	0.000	0.416	0.006	0.440

Appendix

Table 9: Corrected HAPS actor-gradient diagnostic at

T=500

. Values denote means over five seeds.

A_{g}=1

corresponds to the mutually orthogonal baseline (

C_{g}=1-A_{g}

), and

f_{\mathrm{harm}}

is the fraction of phases where

\langle g_{m},g\rangle<0

. Remaining columns report the mean phase-to-aggregate cosine, mean off-diagonal pairwise cosine, and the fraction of negative phase pairs (

f_{\mathrm{pair}}^{-}

Panel of Figure 9 — Figure 9: Representative single-seed actor-gradient cosine matrices for HAPS, illustrating qualitative phase geometry. A high fraction of negative pairwise cosines naturally arises in near-orthogonal high-dimensional gradient spaces and does not indicate that the aggregate update harms individual phases. Quantitative evaluations across five seeds are reported in Table 9 .

$h$	Single+clock	Multihead	Multipolicy
$0.00$	-1.370	-2.785	-6.007
$0.25$	-2.185	-2.592	-5.107
$0.50$	-3.977	-3.844	-3.977
$0.75$	-2.666	-2.693	-2.614
$1.00$	-1.685	-1.638	-1.596

Appendix

Table 10: PhasePendulum target-separation sweep (

M=10,T=1000

). Values denote mean reward per step across five seeds.

Panel of Figure 10 — Figure 10: Budget scaling and entropy controls for Multipolicy in PhasePendulum under identical phase targets ( $h=0$ ). (a) Learning curves across three total interaction budgets ( $B$ ). Error bars denote two-sided $95\%$ Student- $t$ confidence intervals over five seeds. Significant gains emerge after surpassing $500\text{k}$ interactions per specialist. (b) Under $B=20\text{M}$ , increasing the entropy coefficient ( $c_{\mathrm{ent}}=0.05$ ) destabilizes action variance without improving returns (note log scale on right axis).

Panel of Figure 11 — Figure 11: Learning starvation in PhasePendulum with ten identical phases ( $h=0$ ). Final phase-wise returns at the original $500$ k allocation; every point is a mean over five seeds and the dashed line is the corresponding Single+clock mean. Error bars are two-sided $95\%$ Student- $t$ intervals. Doubling the local budget produces the largest recovery, while the low-budget failure is spread across all ten otherwise redundant specialists.

Control	Setting	$n$	Single+clock	Multihead	Multipolicy
Phase count, $T=1000$	$M=2$	5	-1.888	-1.917	-1.752
	$M=5$	5	-1.633	-1.646	-1.514
	$M=10$	5	-1.688	-1.640	-1.553
	$M=20$	5	-2.502	-2.236	-2.260
Dwell time, $M=10$	$T=500$	3	-1.567	-1.660	-1.572
	$T=1000$	3	-1.637	-1.639	-1.564

Appendix

Table 11: PhasePendulum ablation controls across phase counts (

M

), dwell times (

T

), and target/reward structures. The phase-count and dwell-time sweeps allocate

10^{6}

interactions per phase; target/weighting sweeps allocate

500\text{k}

per phase.

n

denotes random seed count.

Budget	Method	Reached	Best mean	Final mean
$50\text{k}$	Multipolicy-style	$0/3$	$-0.0400$	$-0.0585$
$50\text{k}$	Single+clock	$0/3$	$-0.0410$	$-0.0567$
$100\text{k}$	Multipolicy-style	$0/3$	$-0.0417$	$-0.0557$
$100\text{k}$	Single+clock	$0/3$	$-0.0420$	$-0.0557$
$200\text{k}$	Multipolicy-style	$0/3$	$-0.0375$	$-0.0525$
$200\text{k}$	Single+clock	$0/3$	$-0.0461$	$-0.0502$

Appendix

Table 12: Single-phase GridWorld budget calibration (

M=1,T=100

, three seeds). “Reached” denotes the proportion of seeds exceeding the performance threshold. Values are reported as mean reward per step.

$Z$	Single+clock	Multihead	Multipolicy	Multi+clock	Warm Multi+clock
0	0.1422	0.0980	0.1210	0.1258	0.1356
8	0.1427	0.1197	0.1048	0.1201	0.1319
32	0.1359	0.1197	0.0836	0.1082	0.1312
128	0.0629	0.0968	0.0544	0.0732	0.1178

Appendix

Table 13: PhaseGridWorld distractor sweep (

M=10,T=100

). Values denote mean reward per step across three seeds under

Z

reward-irrelevant noise dimensions (higher is better).

Density	$M$	$T$	Single+clock	Multihead	Multipolicy	Warm multipolicy
0.10	5	50	0.1169	0.1252	0.0917	0.1253
0.10	5	100	0.1273	0.1192	0.0633	0.1331
0.10	10	50	0.0861	0.0991	0.0798	0.0969
0.10	10	100	0.0985	0.1031	0.0445	0.1272
0.25	5	50	0.1008	0.0704	0.0653	0.0992
0.25	5	100	0.0901	0.0443	0.0261	0.1009

Appendix

Table 14: Exploratory PhaseGridWorld obstacle-density sweep under a fixed

3

M-interaction total budget. Entries are mean reward per step over three matched seeds; higher is better. The specialist budget is approximately

3\text{M}/M

. Multipolicy and warm multipolicy do not receive clock features in this earlier sweep.

$B_{0}$	Density	$M$	Single+clock	Multihead	Multipolicy	Multi+clock	Warm+clock	Transition warm
$1\text{M}$	0.25	5	0.0836	0.0514	0.0200	0.0940	0.1124	–
$1\text{M}$	0.25	10	0.0878	0.1228	0.0916	0.0926	0.1179	–
$1\text{M}$	0.40	5	0.1508	0.2099	0.1924	0.2372	0.2323	–
$1\text{M}$	0.40	10	0.1480	0.1344	0.1772	0.1651	0.1681	–
$1.5\text{M}$	0.25	10	0.0991	0.0826	0.0791	0.0992	0.1155	-0.0164
$1.5\text{M}$	0.40	10	0.1486	0.1404	0.1606	0.1797	0.1695	0.0539

Appendix

Table 15: PhaseGridWorld specialist controls at

T=500

. Values denote mean reward per step across three seeds.

B_{0}

is the nominal interaction budget per phase. “Warm+clock” initializes specialists from a shared pretrained model prior to fine-tuning.

Panel of Figure 12 — Figure 12: Learned safe-handoff choices. Shared methods increasingly choose the safe phase-0 goal as $\rho$ rises, whereas independently trained specialists continue to prefer the immediate phase-local reward. Means and two-sided $95\%$ Student- $t$ intervals over five seeds.

Panel of Figure 13 — Figure 13: Controlled boundary intervention in PhaseGridWorld ( $M=2,T=100$ ). (a) Specialist-minus-shared returns under physical state carryover versus reachable-state resets. (b) Corresponding difference-in-differences. Points represent means; error bars denote $95\%$ Student- $t$ CIs (left) and paired bootstrap CIs (right). Removing inherited boundary states reverses method rankings in the closed trap while preserving specialist dominance in the bottleneck.

Topology	Entry law	$G$ (Settled)	$K$ (Transient)	Specialist $-$ Shared
Aligned	Carried	$-0.0375$	$0.0450$	$-0.0413$
Aligned	Reset	$0.0065$	$-0.0007$	$0.0036$
Bottleneck	Carried	$0.0746$	$-0.0616$	$0.0681$
Bottleneck	Reset	$0.0771$	$-0.0684$	$0.0728$
Closed trap	Carried	$-0.0754$	$0.0842$	$-0.0798$
Closed trap	Reset	$0.0101$	$-0.0018$	$0.0060$

Appendix

Table 16: Empirical transient–late-window decomposition in PhaseGridWorld (

L=50,T=100

). Values denote means across five matched training seeds evaluated under

\Delta\bar{J}=(1-\lambda)G-\lambda K

with

\lambda=0.5

. “Carried” retains physical states across boundaries; “Reset” uses a common reachable entry law.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Feb 19, 2026

Phase-Aware Mixture of Experts for Agentic Reinforcement Learning

Reinforcement learning (RL) has equipped LLM agents with a strong ability to solve complex tasks. However, existing RL methods normally use a single policy network, causing simplicity bias where simple tasks occupy most parameters and dominate gradient updates, leaving insufficient capacity for complex tasks. A plausible remedy could be employing the Mixture-of-Experts (MoE) architecture in the policy network, as MoE allows different parameters (experts) to specialize in different tasks, preventing simple tasks from dominating all parameters. However, a key limitation of traditional MoE is its token-level routing, where the router assigns each token to specialized experts, which fragments phase-consistent patterns into scattered expert assignments and thus undermines expert specialization. In this paper, we propose Phase-Aware Mixture of Experts (PA-MoE). It first features a lightweight phase router that learns latent phase boundaries directly from the RL objective without pre-defining phase categories. Then, the phase router allocates temporally consistent assignments to the same expert, allowing experts to preserve phase-specific expertise. Experimental results demonstrate the effectiveness of our proposed PA-MoE.

Shengtian Yang, Yu Li, Shuo He +4

School of Computer Science and Engineering, Southeast University, Nanjing, China · Kuaishou Technology, Beijing, China · College of Computing and Data Science, Nanyang Technological University, Singapore

Reinforcement Learning RL for LLM Agents Mixture-of-Experts Models Agentic RL Expert Routing

Ask PDF

Oct 7, 2026

Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

We study learning optimal policies in average-reward multichain Markov decision processes (MDPs), where the optimal gain may depend on the initial state and recurrence structures vary across policies, creating challenges for reinforcement learning (RL) methods. We propose an asynchronous value-iteration-based RL algorithm that requires no model knowledge beyond the MDP's transition graph and leverages Bather's decomposition to hierarchically partition the state space into communicating subsystems and transient states. This decomposition induces a recasting of the global decision problem into structured subproblems, which our algorithm exploits. We show that the algorithm converges to the optimal gain and produces gain-optimal policies after finite time. Building on this base algorithm, we develop two further algorithms: one approximately solves the multichain average optimality equations to obtain near gain-optimal policies, and another targets near bias-optimality by approximating the optimal bias function and solving an induced average-reward multichain MDP using the base algorithm. We provide almost-sure convergence guarantees for all three algorithms and empirically compare their tradeoffs, showing that the latter two also consistently improve transient performance relative to the base algorithm. To our knowledge, these are the first essentially model-free average-reward RL algorithms for general multichain MDPs without reductions to discounted problems.

Huizhen Yu, Isaiah Heidt

Department of Computing Science University of Alberta, Canada

Markov Decision Processes Average-Reward RL Markov Chains

Ask PDF

May 27, 2026

Visualizing Latent Phase Structures in Locomotion Policies: A Multi-Environment Study with Temporal Feature Extension

Deep reinforcement learning (DRL) has been shown to achieve high performance on locomotion control tasks in MuJoCo benchmarks such as HalfCheetah, Ant, and Walker2D. However, visualizing the motion structures internally obtained by a trained policy function implemented as a deep neural network remains challenging. It is known from biomechanics and related fields that locomotion control is realized through the repetition of motion phases such as the stance phase and swing phase. In this study, we propose a framework for uncovering latent motion phase structures from trajectories generated by locomotion control policies through interaction with the environment. The proposed method extends the clustering features from state observations alone to augmented features including actions, next states, and next actions, and introduces a method for determining the number of clusters that suppresses self-transitions. Applying the proposed method to three environments -- Ant-v5, HalfCheetah-v5, and Walker2D-v5 -- we successfully identified phase structures with clearer and more regular transition rules than those obtained by the existing method.

Daisuke Yasui, Toshitaka Matuki, Hiroshi Sato

Mathematics and Computer Science National Defense Academy of Japan Yokosuka, Japan

Temporal Action Segmentation Robot Locomotion RL for Legged Locomotion

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Phase-Aware Mixture of Experts for Agentic Reinforcement Learning

Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

Visualizing Latent Phase Structures in Locomotion Policies: A Multi-Environment Study with Temporal Feature Extension