Multi-level meta-reinforcement learning with skill-based curriculum

Sichen Yang; Mauro Maggioni

cs.LGMar 9, 2026

Multi-level meta-reinforcement learning with skill-based curriculum

Authors: Sichen Yang, Mauro Maggioni

Organizations: Johns Hopkins University · Department of Applied Mathematics & Statistics Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA · Department of Mathematics, Department of Applied Mathematics & Statistics Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD 21218, USA

Abstract

We consider problems in sequential decision making with natural multi-level structure, where sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure has remained a longstanding challenge; we describe an efficient multi-level procedure for repeatedly compressing Markov decision processes (MDPs), wherein a parametric family of policies at one level is treated as single actions in the compressed MDPs at higher levels, while preserving the semantic meanings and structure of the original MDP, and mimicking the natural logic to address a complex MDP. Higher-level MDPs are themselves independent MDPs with less stochasticity, and may be solved using existing algorithms. As a byproduct, spatial or temporal scales may be coarsened at higher levels, making it more efficient to find long-term optimal policies. The multi-level representation delivered by this procedure decouples sub-tasks from each other and usually greatly reduces unnecessary stochasticity and the policy search space, leading to fewer iterations and computations when solving the MDPs. A second fundamental aspect of this work is that these multi-level decompositions plus the factorization of policies into embeddings (problem-specific) and skills (including higher-order functions) yield new transfer opportunities of skills across different problems and different levels. This whole process is framed within curriculum learning, wherein a teacher organizes the student agent's learning process in a way that gradually increases the difficulty of tasks and and promotes transfer across MDPs and levels within and across curricula. The consistency of this framework and its benefits can be guaranteed under mild assumptions. We demonstrate abstraction, transferability, and curriculum learning in examples, including MazeBase+, a more complex variant of the MazeBase example.

Figures & tables

Figure 1: A representation of MazeBase+, a more challenging version of the “classical” MazeBase, with curriculum in the blue inset (middle right), and the MMDPs in the curriculum, with their various levels, depicted and summarized in the black insets.

Figure 2: A representation of the second experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). In the second experiment, we apply our framework to perform efficient transfer to MazeBase+ worlds with different configurations of doors, keys, and the goal.

Panel of Figure 3 — Figure 3: Commuting diagram for a skill-embedding decomposition of $\pi_{I}$ .

Figure 4: We display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0}):=\sum_{s\in{\mathcal{S}}^{\text{init}}}V(s)/|{\mathcal{S}}^{\text{init}}|$ , the average (over all initial conditions $s_{0}$ ) of the $V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration (in red) and of value iteration within our algorithm, with iterations within ${\mathtt{MDP}}_{1,1}={\mathtt{MDP}}_{\text{dense}}^{\text{nav}}$ in orange, iterations within ${\mathtt{MDP}}_{2,n}(1\leq n\leq 2)$ at level $2$ in blue followed by iterations at level $1$ in orange, and iterations within ${\mathtt{MDP}}_{3,1}$ at level $3$ in green followed by iterations at level $2$ in blue followed by iterations at level $1$ in orange. Although we spend extra effort in solving the ${\mathtt{MDP}}_{2,n}$ ’s (recall that ${\mathtt{MDP}}_{1,1}$ ’s comes from the example of navigation and transportation with traffic jams, so we do not spend extra effort there), they prepare us well enough so that we only need a few more iterations for solving the target ${\mathtt{MDP}}_{3,1}$ (green+blue+orange), much fewer than if we solved it from scratch using classical value iteration (red).

Panel of Figure 5 — Figure 5: Left: we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}^{\prime}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}^{\prime}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. 4 for the detailed explanation, and Fig. 2 for the representation of this second experiment of the MazeBase+ example. One addition here compared with Fig. 4 is that here we also plot iterations within ${\mathtt{MDP}}^{\prime}_{3,1}$ at level $2$ in purple followed by iterations at level $1$ in yellow, assuming the student did not solve ${\mathtt{MDP}}_{2,2}$ and consequently the assistant did not extract $\overline{\pi}^{\mathrm{concat}}$ . This corresponds to treating ${\mathtt{MDP}}^{\prime}_{3,1}$ as an MDP of difficulty $2$ . The extra effort here (purple+yellow) compared with the original case (green+blue+orange) demonstrates the advantage of treating ${\mathtt{MDP}}^{\prime}_{3,1}$ as an MDP of difficulty $3$ and of extracting the higher-order function $\overline{\pi}^{\mathrm{concat}}$ . Right: we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,1}$ , ${\mathtt{MDP}}^{\prime\prime}_{2,1}$ , ${\mathtt{MDP}}_{2,2}$ , and ${\mathtt{MDP}}^{\prime\prime}_{3,1}$ respectively, as $\pi$ is optimized during iterations of classical value iteration and of value iteration within our algorithm. See Fig. 4 for the detailed explanation, and see Fig. 6 for the representation of this third experiment of the MazeBase+ example.

Figure 6: A representation of the third experiment of the MazeBase+ example. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). In the third experiment, we consider the situation where the optimal policy at the highest level, when refined to a finer level, yields a suboptimal policy where the student collects ${\mathtt{key}}_{2}$ , opens ${\mathtt{door}}_{2}$ , and then collects ${\mathtt{key}}_{3}$ (very close to ${\mathtt{key}}_{2}$ ) to then open ${\mathtt{door}}_{3}$ . Within our algorithm, this suboptimal policy gets refined in order to yield the optimal policy, demonstrating the robustness of our optimization procedure; in this case it is also the case that our algorithm still outperforms naïve value iteration (see Fig. 5 (right)).

Figure 7: A representation of the example of navigation and transportation with traffic jams, with curriculum in the blue inset (top right), and the MMDPs in the curriculum, with their various levels, depicted and summarized in the black insets. The representation in this figure is similar to the one for the first experiment (Fig. 1 ). The goal of the agent is to learn optimal policies to travel from an initial position in a grid world to a known final state in any position in the grid world. Some regions, ${{\Omega}_{\mathtt{jams}}}$ , in the grid world have traffic jams (of different degrees of severity). The agent will have at its disposal two means of transportation, ${\mathtt{mc}}$ and ${\mathtt{car}}$ , with ${\mathtt{car}}$ having different velocities in ${{\Omega}_{\mathtt{jams}}}$ and ${{\Omega}^{c}_{\mathtt{jams}}}$ .

Figure 8: Similar to Fig. 4 , we display $\mathbb{E}_{s_{0}}V_{\pi}(s_{0})$ , where $V_{\pi}$ is the value function for ${\mathtt{MDP}}_{1,n}$ and ${\mathtt{MDP}}_{2,n}$ , as $\pi$ is optimized during iterations of classical value iteration (in red) and of value iteration within our algorithm, with iterations within ${\mathtt{MDP}}_{1,n}$ in orange, and iterations within ${\mathtt{MDP}}_{2,n}$ at level $2$ in blue followed by iterations at level $1$ in orange. Although we spend extra effort in solving the ${\mathtt{MDP}}_{1,n}$ ’s, they prepare us well-enough so that we only need a few more iterations for solving the ${\mathtt{MDP}}_{2,n}$ ’s (blue+orange) – much fewer than if we solved them from scratch using classical value iteration (red).

Figure 9: Similar to Fig. 8 , we display

\mathbb{E}_{s_{0}}V_{\pi}(s_{0})

, where

V_{\pi}

is the value function for

{\mathtt{MDP}}_{1,n_{1}+1}

(in orange) and

{\mathtt{MDP}}_{2,n_{2}+1}

, as

\pi

is optimized during iterations of value iteration within our algorithm. More specifically, within

{\mathtt{MDP}}_{2,n_{2}+1}

we have iterations at level

2

in blue followed by iterations at level

1

in orange, where

\pi

is optimized during iterations of value iteration within our algorithm without utilizing

\overline{\pi}^{\mathrm{transport}}

; we also have iterations at level

2

in purple followed by iterations at level

1

in yellow, where

\pi

is optimized within our algorithm utilizing

\overline{\pi}^{\mathrm{transport}}

. Although we spend extra effort in extracting

\overline{\pi}^{\mathrm{transport}}

, it prepares us well enough so that we could solve

{\mathtt{MDP}}_{2,n_{2}+1}

(purple+yellow) almost instantly, with much fewer iterations than if we solved it from scratch within our algorithm (blue+orange). This is also analyzed in general by the forthcoming Thm. 11 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	$x_{1}$	$x_{2}$	$x_{3}$	$y_{1}$	$y_{2}$	$y_{3}$	$s_{{\mathtt{key}}_{1}}$	$s_{{\mathtt{key}}_{2}}$	$s_{{\mathtt{key}}_{3}}$	$s_{{\mathtt{door}}_{1}}$	$s_{{\mathtt{door}}_{2}}$	$s_{{\mathtt{door}}_{3}}$	$s_{{\mathtt{goal}}}$	$p_{s}$	$R_{0}$	$r_{0}$
${\mathtt{MDP}}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,3)$	$(1,1)$	$(1,8)$	$(10,2)$	$(9,4)$	$(10,5)$	$(15,8)$	$0.9$	$10^{4}$	$-10$
${\mathtt{MDP}}^{\prime}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,1)$	$(15,1)$	$(15,8)$	$(10,3)$	$(11,4)$	$(10,5)$	$(1,8)$	$0.9$	$10^{4}$	$-10$
${\mathtt{MDP}}^{\prime\prime}_{3,1}$	$1$	$15$	$10$	$1$	$8$	$4$	$(1,3)$	$(1,1)$	$(2,1)$	$(10,2)$	$(9,4)$	$(10,5)$	$(15,8)$	$0.9$	$10^{4}$	$-10$

Appendix

Table 1: Parameters in the MazeBase+ example

	${{\Omega}_{\mathtt{jams}}}$	$p_{s}$	$v_{{\mathtt{mc}}}$	$v_{{\mathtt{car}}}$	$\kappa$	$R_{{\mathtt{dest}}}$	$R_{{\mathtt{jams}}}$	$r_{0}$	$\gamma_{0}$
${\mathtt{MDP}}_{2,1}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/2.4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,2}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/2.8$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,3}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/3.2$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,4}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/3.6$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,5}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$
${\mathtt{MDP}}_{2,6}$	$[1,15]\times\{6\}\cup\{5\}\times[1,8]$	$0.9$	$1$	$0.6$	$1/4.4$	$10^{4}$	$-10^{3}$	$-10$	$0.999$

Appendix

Table 2: Parameters in the example of navigation and transportation with traffic jams

Figure 10: The discount factors, represented by heat maps, at the actions selected by the optimal policy for $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams (It happens to be the case that the agent will reach the next state with probability one at all the states when selecting the actions according to the optimal policy). The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . These heat maps share similar color patterns with the ones representing value functions in Fig. 7 , as explained by Thm. 13 . The main message of this figure is on the relative values of discount factors at different locations, not the absolute magnitude, because the discount factors are set to be really close to $1$ in the original MDPs. In examples where the discount factors are smaller, they will have a more important role in convergence of the value function during value iteration.

Figure 11: The times of convergence, represented by heat maps, to the optimal value functions for $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams. The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . As expected, the more traffic roads the agent needs to pass, and the heavier the traffic, the longer the convergence time. These heat maps share similar color patterns, upon reversing the color bar, with the ones representing value functions in Fig. 7 , as explained by Thm. 13 .

Figure 12: Examination of the tightness of the bounds in Thm. 13 using $\{{\mathtt{MDP}}^{{2}}_{2,n}\}_{n=1}^{n_{2}}$ in the example of navigation and transportation with traffic jams, by plotting the relative errors in the logarithmic scale $\log(\frac{\mathrm{upper\_bound}-\mathrm{lower\_bound}}{\mathrm{true\_value}})$ , represented by heat maps. Here, for each location, and for each iteration of the value iteration algorithm, $\mathrm{true\_value}$ is the value of the value function at the current iteration. The heat maps of this quantity for ${\mathtt{MDP}}^{{2}}_{2,1},{\mathtt{MDP}}^{{2}}_{2,2},{\mathtt{MDP}}^{{2}}_{2,3}$ , and ${\mathtt{MDP}}^{{2}}_{2,4}$ are similar, so we only plot the relative errors for ${\mathtt{MDP}}^{{2}}_{2,1},{\mathtt{MDP}}^{{2}}_{2,5}$ , and ${\mathtt{MDP}}^{{2}}_{2,6}$ across different iterations until convergence to avoid redundancy. The regions with traffic jams and the destination are marked in this figure similar to Fig. 7 . As expected, the relative error quickly decays to zero as soon as the value function at the current iteration is reasonably “close” to the true value function. Actually, the color patterns of these heat maps at beginning iterations match pretty nicely with the ones representing convergence times in Fig. 11 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning
May 16, 2026Vivienne Huiling Wang, Tinghuai Wang, Joni PajarinenWorld Model-Based Planning Hierarchical RL
Department of Electrical Engineering and Automation, Aalto University, Finland · Qutwo, Finland
Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach
Oct 7, 2026Huizhen Yu, Isaiah HeidtMarkov Decision Processes Average-Reward RL
Department of Computing Science University of Alberta, Canada
RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning
Sep 26, 2026Ziqiao Shang, Zian Xu, Ji-Chen Yan +6LLM Agent Skill Learning Agent Skill Learning
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · Didichuxing Co. Ltd

May 16, 2026J/K move · Enter open · S save

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning

Vivienne Huiling Wang, Tinghuai Wang, Joni Pajarinen

Department of Electrical Engineering and Automation, Aalto University, Finland · Qutwo, Finland

World Model-Based Planning Hierarchical RL Combinatorial Optimization Model-Based Planning RL for Combinatorial Optimization Model-Based RL

Open paper AskPDF arXiv

The combination of exponentially large action spaces, stochastic dynamics, and long-horizon decision-making under limited resources makes Sequential Stochastic Combinatorial Optimization (SSCO) particularly challenging for reinforcement learning. Hierarchical Reinforcement Learning (HRL) offers a natural decomposition, but it places the high-level policy in a Semi-Markov Decision Process (SMDP) where actions have variable durations, making it difficult to learn a world model that is suitable for planning. We introduce a model-based hierarchical framework for sequential stochastic combinatorial decision-making that directly addresses this issue. Our method combines a latent-space tree-search planner with an SMDP-aware world model for variable-duration decisions. A multi-timescale objective structures the latent dynamics so that transition magnitudes reflect the effective temporal scales of abstract actions, enabling efficient lookahead under adaptive temporal abstraction. We further learn a subgoal-conditioned budget policy jointly with the world model to support context-aware resource allocation. Across challenging SSCO benchmarks, our method outperforms strong baselines.

Abstract

Reinforcement Learning

Explore similar work

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning

Average-Reward Reinforcement Learning for Multichain MDPs: A Hierarchical Decomposition Approach

RLHarness: Co-evolving Procedural Skills with Reinforcement Learning for Long-horizon Multimodal Reasoning