Multi-level meta-reinforcement learning with skill-based curriculum

Sichen Yang; Mauro Maggioni

cs.LGMar 9, 2026

Multi-level meta-reinforcement learning with skill-based curriculum

Authors: Sichen Yang, Mauro Maggioni

Organizations: Johns Hopkins University · Department of Applied Mathematics & Statistics Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA · Department of Mathematics, Department of Applied Mathematics & Statistics Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA

Abstract

We consider problems in sequential decision making with natural multi-level structure, where sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure has remained a longstanding challenge; we describe an efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs), wherein a parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP, and mimicking the natural logic to address a complex MDP. Higher-level MDPs are themselves independent MDPs with less stochasticity, and may be solved using existing algorithms. As a byproduct, spatial or temporal scales may be coarsened at higher levels, making it more efficient to find long-term optimal policies. The multi-level representation delivered by this procedure decouples sub-tasks from each other and usually greatly reduces unnecessary stochasticity and the policy search space, leading to fewer iterations and computations when solving the MDPs. A second fundamental aspect of this work is that these multi-level decompositions plus the factorization of policies into embeddings (problem-specific) and skills (including higher-order functions) yield new transfer opportunities of skills across different problems and different levels. This whole process is framed within curriculum learning, wherein a teacher organizes the student agent's learning process in a way that gradually increases the difficulty of tasks and and promotes transfer across MDPs and levels within and across curricula. The consistency of this framework and its benefits can be guaranteed under mild assumptions. We demonstrate abstraction, transferability, and curriculum learning in examples, including MazeBase+, a more complex variant of the MazeBase example.

Figures & tables

Figure 1: A representation of MazeBase+, a more challenging version of the “classical” MazeBase, with curriculum in the blue inset (middle right), and the MMDPs in the curriculum, with their various levels, depicted and summarized in the black insets.

Figure 2: A representation of the second experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). In the second experiment, we apply our framework to perform efficient transfer to MazeBase+ worlds with different configurations of doors, keys, and the goal.

Panel of Figure 3 — Figure 3: Commuting diagram for a skill-embedding decomposition of $\pi_{I}$ .

Figure 4: We display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0}):=\sum_{s\in{\mathcal{S}}^{\text{init}}}V(s)/|{\mathcal{S}}^{\text{init}}|$ , the average (over all initial conditions $s_{0}$ ) of the $V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration (in red) and of value iteration within our algorithm, with iterations within ${\mathtt{MDP}}_{1,1}={\mathtt{MDP}}_{\text{dense}}^{\text{nav}}$ in orange, iterations within ${\mathtt{MDP}}_{2,n}(1\leq n\leq 2)$ at level $2$ in blue followed by iterations at level $1$ in orange, and iterations within ${\mathtt{MDP}}_{3,1}$ at level $3$ in green followed by iterations at level $2$ in blue followed by iterations at level $1$ in orange. Although we spend extra effort in solving the ${\mathtt{MDP}}_{2,n}$ ’s (recall that ${\mathtt{MDP}}_{1,1}$ ’s comes from the example of navigation and transportation with traffic jams, so we do not spend extra effort there), they prepare us well enough so that we only need a few more iterations for solving the target ${\mathtt{MDP}}_{3,1}$ (green+blue+orange), much fewer than if we solved it from scratch using classical value iteration (red).

Panel of Figure 5 — Figure 5: Left: we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}^{\prime}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}^{\prime}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. 4 for the detailed explanation, and Fig. 2 for the representation of this second experiment of the MazeBase+ example. One addition here compared with Fig. 4 is that here we also plot iterations within ${\mathtt{MDP}}^{\prime}_{3,1}$ at level $2$ in purple followed by iterations at level $1$ in yellow, assuming the student did not solve ${\mathtt{MDP}}_{2,2}$ and consequently the assistant did not extract $\overline{\pi}^{\mathrm{concat}}$ . This corresponds to treating ${\mathtt{MDP}}^{\prime}_{3,1}$ as an MDP of difficulty $2$ . The extra effort here (purple+yellow) compared with the original case (green+blue+orange) demonstrates the advantage of treating ${\mathtt{MDP}}^{\prime}_{3,1}$ as an MDP of difficulty $3$ and of extracting the higher-order function $\overline{\pi}^{\mathrm{concat}}$ . Right: we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}^{\prime\prime}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}^{\prime\prime}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. 4 for the detailed explanation, and see Fig. 6 for the representation of this third experiment of the MazeBase+ example.

Figure 6: A representation of the third experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). In the third experiment, we consider the situation where the optimal policy at the highest level, when refined to a finer level, yields a suboptimal policy where the student collects ${\mathtt{key}}_{2}$ , opens ${\mathtt{door}}_{2}$ , and then collects ${\mathtt{key}}_{3}$ (very close to ${\mathtt{key}}_{2}$ ) to then open ${\mathtt{door}}_{3}$ . Within our algorithm, this suboptimal policy gets refined in order to yield the optimal policy, demonstrating the robustness of our optimization procedure; in this case it is also the case that our algorithm still outperforms naïve value iteration (see Fig. 5 (right)).

Figure 7: A representation of the example of navigation and transportation with traffic jams, with curriculum in the blue inset (top right), and the MMDPs in the curriculum, with their various levels, depicted and summarized in the black insets. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). The goal of the agent is to learn optimal policies to travel from an initial position in a grid world to a known final state in any position in the grid world. Some regions, ${{\Omega}_{\mathtt{jams}}}$ , in the grid world have traffic jams (of different degrees of severity). The agent will have at its disposal two means of transportation, ${\mathtt{mc}}$ and ${\mathtt{car}}$ , with ${\mathtt{car}}$ having different velocities in ${{\Omega}_{\mathtt{jams}}}$ and ${{\Omega}^{c}_{\mathtt{jams}}}$ .

Figure 8: Similar to Fig. 4 , we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,n}$ and ${\mathtt{MDP}}_{2,n}$ , as $\pi$ is optimized during iterations of classical value iteration (in red) and of value iteration within our algorithm, with iterations within ${\mathtt{MDP}}_{1,n}$ in orange, and iterations within ${\mathtt{MDP}}_{2,n}$ at level $2$ in blue followed by iterations at level $1$ in orange. Although we spend extra effort in solving the ${\mathtt{MDP}}_{1,n}$ ’s, they prepare us well-enough so that we only need a few more iterations for solving the ${\mathtt{MDP}}_{2,n}$ ’s (blue+orange) – much fewer than if we solved them from scratch using classical value iteration (red).

Figure 9: Similar to Fig. 8 , we display

\mathbb{E}_{s_{0}}V_{\pi}(s_{0})

, where

V_{\pi}

is the value function for

{\mathtt{MDP}}_{1,n_{1}+1}

(in orange) and

{\mathtt{MDP}}_{2,n_{2}+1}

, as

\pi

is optimized during iterations of value iteration within our algorithm. More specifically, within

{\mathtt{MDP}}_{2,n_{2}+1}

we have iterations at level

2

in blue followed by iterations at level

1

in orange, where

\pi

is optimized during iterations of value iteration within our algorithm without utilizing

\overline{\pi}^{\mathrm{transport}}

; we also have iterations at level

2

in purple followed by iterations at level

1

in yellow, where

\pi

is optimized within our algorithm utilizing

\overline{\pi}^{\mathrm{transport}}

. Although we spend extra effort in extracting

\overline{\pi}^{\mathrm{transport}}

, it prepares us well enough so that we could solve

{\mathtt{MDP}}_{2,n_{2}+1}

(purple+yellow) almost instantly, with much fewer iterations than if we solved it from scratch within our algorithm (blue+orange). This is also analyzed in general by the forthcoming Thm. 11 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	$x_{1}$	$x_{2}$	$x_{3}$	$y_{1}$	$y_{2}$	$y_{3}$	$s_{{\mathtt{key}}_{1}}$	$s_{{\mathtt{key}}_{2}}$	$s_{{\mathtt{key}}_{3}}$	$s_{{\mathtt{door}}_{1}}$	$s_{{\mathtt{door}}_{2}}$	$s_{{\mathtt{door}}_{3}}$	$s_{{\mathtt{goal}}}$	$p_{s}$	$R_{0}$	$r_{0}$
${\mathtt{MDP}}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,3)$	$(1,1)$	$(1,8)$	$(10,2)$	$(9,4)$	$(10,5)$	$(15,8)$	$0.9$	$10^{4}$	$-10$
${\mathtt{MDP}}^{\prime}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,1)$	$(15,1)$	$(15,8)$	$(10,3)$	$(11,4)$	$(10,5)$	$(1,8)$	$0.9$	$10^{4}$	$-10$
${\mathtt{MDP}}^{\prime\prime}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,3)$	$(1,1)$	$(2,1)$	$(10,2)$	$(9,4)$	$(10,5)$	$(15,8)$	$0.9$	$10^{4}$	$-10$

Appendix

Table 1: Parameters in the MazeBase+ example

	${{\Omega}_{\mathtt{jams}}}$	$p_{s}$	$v_{{\mathtt{mc}}}$	$v_{{\mathtt{car}}}$	$\kappa$	$R_{{\mathtt{dest}}}$	$R_{{\mathtt{jams}}}$	$r_{0}$	$\gamma_{0}$
${\mathtt{MDP}}_{2,1}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/2.4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,2}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/2.8$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,3}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/3.2$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,4}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/3.6$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,5}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,6}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/4.4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$

Appendix

Table 2: Parameters in the example of navigation and transportation with traffic jams

Figure 10: The discount factors, represented by heat maps, at the actions selected by the optimal policy for $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams (It happens to be the case that the agent will reach the next state with probability one at all the states when selecting the actions according to the optimal policy). The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . These heat maps share similar color patterns with the ones representing value functions in Fig. 7 , as explained by Thm. 13 . The main message of this figure is on the relative values of discount factors at different locations, not the absolute magnitude, because the discount factors are set to be really close to $1$ in the original MDPs. In examples where the discount factors are smaller, they will have a more important role in convergence of the value function during value iteration.

Figure 11: The times of convergence, represented by heat maps, to the optimal value functions for $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams. The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . As expected, the more traffic roads the agent needs to pass, and the heavier the traffic, the longer the convergence time. These heat maps share similar color patterns, upon reversing the color bar, with the ones representing value functions in Fig. 7 , as explained by Thm. 13 .

Figure 12: Examination of the tightness of the bounds in Thm. 13 using $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams, by plotting the relative errors in the logarithmic scale $\log(\frac{\mathrm{upper\_bound}-\mathrm{lower\_bound}}{\mathrm{true\_value}})$ , represented by heat maps. Here, for each location, and for each iteration of the value iteration algorithm, $\mathrm{true\_value}$ is the value of the value function at the current iteration. The heat maps of this quantity for ${\mathtt{MDP}}^{{2}}_{2,1},{\mathtt{MDP}}^{{2}}_{2,2},{\mathtt{MDP}}^{{2}}_{2,3}$ , and ${\mathtt{MDP}}^{{2}}_{2,4}$ are similar, so we only plot the relative errors for ${\mathtt{MDP}}^{{2}}_{2,1},{\mathtt{MDP}}^{{2}}_{2,5}$ , and ${\mathtt{MDP}}^{{2}}_{2,6}$ across different iterations until convergence to avoid redundancy. The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . As expected, the relative error quickly decays to zero as soon as the value function at the current iteration is reasonably “close” to the true value function. Actually, the color patterns of these heat maps at beginning iterations match pretty nicely with the ones representing convergence times in Fig. 11 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

May 16, 2026

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning

The combination of exponentially large action spaces, stochastic dynamics, and long-horizon decision-making under limited resources makes Sequential Stochastic Combinatorial Optimization (SSCO) particularly challenging for reinforcement learning. Hierarchical Reinforcement Learning (HRL) offers a natural decomposition, but it places the high-level policy in a Semi-Markov Decision Process (SMDP) where actions have variable durations, making it difficult to learn a world model that is suitable for planning. We introduce a model-based hierarchical framework for sequential stochastic combinatorial decision-making that directly addresses this issue. Our method combines a latent-space tree-search planner with an SMDP-aware world model for variable-duration decisions. A multi-timescale objective structures the latent dynamics so that transition magnitudes reflect the effective temporal scales of abstract actions, enabling efficient lookahead under adaptive temporal abstraction. We further learn a subgoal-conditioned budget policy jointly with the world model to support context-aware resource allocation. Across challenging SSCO benchmarks, our method outperforms strong baselines.

Vivienne Huiling Wang, Tinghuai Wang, Joni Pajarinen

Department of Electrical Engineering and Automation, Aalto University, Finland · Qutwo, Finland

World Model-Based Planning Hierarchical RL Combinatorial Optimization Model-Based Planning RL for Combinatorial Optimization Model-Based RL

Ask PDF

Oct 7, 2026

Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

We study learning optimal policies in average-reward multichain Markov decision processes (MDPs), where the optimal gain may depend on the initial state and recurrence structures vary across policies, creating challenges for reinforcement learning (RL) methods. We propose an asynchronous value-iteration-based RL algorithm that requires no model knowledge beyond the MDP's transition graph and leverages Bather's decomposition to hierarchically partition the state space into communicating subsystems and transient states. This decomposition induces a recasting of the global decision problem into structured subproblems, which our algorithm exploits. We show that the algorithm converges to the optimal gain and produces gain-optimal policies after finite time. Building on this base algorithm, we develop two further algorithms: one approximately solves the multichain average optimality equations to obtain near gain-optimal policies, and another targets near bias-optimality by approximating the optimal bias function and solving an induced average-reward multichain MDP using the base algorithm. We provide almost-sure convergence guarantees for all three algorithms and empirically compare their tradeoffs, showing that the latter two also consistently improve transient performance relative to the base algorithm. To our knowledge, these are the first essentially model-free average-reward RL algorithms for general multichain MDPs without reductions to discounted problems.

Huizhen Yu, Isaiah Heidt

Department of Computing Science University of Alberta, Canada

Markov Decision Processes Average-Reward RL Markov Chains

Ask PDF

Sep 26, 2026

RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning

Multimodal reasoning requires models to preserve visual evidence through long decision chains while selecting appropriate procedures across diverse scenarios and rules. When learning is guided only by terminal verifiers, reinforcement learning (RL) reveals whether a final answer is correct but not how it should be produced. The policy must therefore discover reusable reasoning procedures while learning to execute them, creating a program cold-start problem. Skills can externalize successful procedures, reduce repeated exploration, and provide inspectable guidance. However, a fixed Skill Bank assumes that this guidance remains compatible with an evolving policy, while updating Skills alone can leave their triggers, execution protocols, and demonstrations stale or mutually inconsistent. We introduce RLHARNESS, which organizes Skills, selection and execution protocols, few-shot demonstrations, and task contracts into a unified, versioned Harness and alternates Harness evolution with policy learning. An Exploration-Distillation Harness builds the initial Harness and version-aligned verified traces for SFT and DAPO I. After the first RL block, a Post-RL Reconstruction Harness rebuilds Skills, protocols, and demonstrations from fresh success-failure rollouts, and DAPO II adapts the policy to the reconstructed program. RLHARNESS improves Accuracy from 16.25%/27.50% to 62.00%/50.00% on MetroMap/TravelMap and raises F1 score from 37.13%/45.50% to 65.81%/65.51% on Fee-VL/Cancel-VL. All four tasks achieve their best results only after reconstruction and DAPO II, showing that an evolving Harness complements RL by continually updating the external program that the policy learns to execute.

Ziqiao Shang, Zian Xu, Ji-Chen Yan +6

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Didichuxing Co. Ltd

LLM Agent Skill Learning Agent Skill Learning Multimodal Reasoning RL for VLMs

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning

Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning