Active Curriculum Refinement for Reinforcement Learning

Zhenya Liu; Yuxin Chen

cs.LGAug 26, 2026

Active Curriculum Refinement for Reinforcement Learning

Authors: Zhenya Liu, Yuxin Chen

Organizations: The University of Chicago, Chicago IL, USA.

Abstract

In many reinforcement learning (RL) domains, environments are connected by prerequisite relations, such as difficulty-increasing edits or parameter increments, which induce a directed acyclic curriculum graph (DAG). Although this structure is often exploited only implicitly, explicitly modeling it can improve training. We introduce PATH, a curriculum-learning framework that performs active learning over the curriculum graph. PATH first expands coverage by sampling diverse curriculum paths and then reallocates training toward regions that remain unmastered. Experiments across diverse environments show that PATH explicitly leverages the graph structure to achieve strong robustness and generalization.

Figures & tables

Figure 1: Overview of PATH . (a) A environment distribution $\Theta$ . (b) The same space viewed as an explicit curriculum DAG $\mathcal{G}$ , where edges encode prerequisite relations between environments. (c) PATH:Random explores $\mathcal{G}$ by sampling random paths (yellow trajectories); blue nodes are explicitly visited and learned, while yellow nodes denote additional environments learned via generalization from nearby visited nodes. (d) PATH:Active identifies a node with high learning potential, then it allocates more sampling by launching multiple random paths (purple) from that node. (e) This adaptive allocation rapidly increases coverage and efficient learns all nodes in $\mathcal{G}$ .

Figure 2: Conceptual comparison of training budget versus curriculum graph coverage. Oracle denotes the theoretical optimum. PATH switches at update $T$ .

Figure 3: Examples of MiniGrid layouts from generalization test-set that PATH solves.

Figure 4: Left: emergent complexity of sampled MiniGrid layouts during training. Right: average episodic return on layouts sampled during training. Curves show mean $\pm$ standard error across 5 training seeds.

Figure 5: Aggregate robustness performance (solved rates) in MiniGrid. The vertical line indicates the mean, and the colored bar shows $\pm 1$ standard error.

Figure 6: Performance on BipedalWalker robustness tests during training (mean $\pm$ standard error), evaluated every $1$ k PPO updates.

DR	PLR	ACCEL	PATH
$0.950\pm 0.0$	$0.977\pm 0.0$	$0.976\pm 0.0$	$\mathbf{0.994}\pm 0.0$

Table 1: Generalization test solved rates at

20

k updates (mean

\pm

standard error) for

5

runs of each method on MiniGrid. Bold values are within one standard error of the best mean.

Figure 7: Aggregate normalized robustness performance (returns) in BipedalWalker. Returns are min–max normalized from $[-100,300]$ to $[0,1]$ . The vertical line indicates the mean, and the colored bar shows $\pm 1$ standard error.

Figure 8: Radar plots of terrain parameters for all solved BipedalWalker terrains from a single run of each algorithm. Axes are ordered clockwise starting at 12 o’clock: Roughness (max 10), Gap (max 10), Stump (max 5), Stairs (max 5), and Steps (max 9). Each polyline corresponds to one solved terrain.

Algorithms	Test Returns	Solved Rates
DR	$9.94\pm 5.75$	$0.01\pm 0.00$
PLR	$15.39\pm 2.84$	$0.02\pm 0.00$
ACCEL	$18.14\pm 5.31$	$0.04\pm 0.00$
PATH	$\mathbf{42.95\pm 1.31}$	$\mathbf{0.11\pm 0.01}$

Table 2: Generalization test returns and solved rates at

20

k updates (mean

\pm

standard error) for

5

runs of each method on BipedalWalker. Bold values are within one standard error of the best mean.

Figure 9: Examples of BipedalWalker terrains from the generalization test that PATH solves.

Panel of Figure 10 — Figure 10: Ablation results on PitGap . (a) Robustness test returns versus PPO updates for PATH , PATH:Active , and PATH:Random (phase shift at $\sim$ $8$ k updates). (b) Corresponding sampling complexity over dimension of PitGap versus PPO updates for $3$ methods. Both figures show the mean $\pm$ standard error. PATH overlaps with PATH:RANDOM in the first $8$ k updates.

Figure 11: Bar figures of generalization test returns and solved rates at $20$ k updates (mean $\pm$ standard error) for $5$ runs of PATH and its two components. bar height is the mean and segment shows one standard error.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Stump Height	Stair Height	Stair Steps	Roughness	Pit Gap
Initiated Value	$[0,0.4]$	$[0,0.4]$	$1$	$\mathrm{Unif}(0,0.6)$	$[0,0.8]$
Edit Size	$0.2$	$0.2$	$1$	$\mathrm{Unif}(0,0.6)$	$0.4$
Max Value	$[5,5]$	$[5,5]$	$9$	$10$	$[10,10]$

Appendix

Table 3: Environment parameter ranges and editing details of ACCEL on BipedalWalker

Figure 12: Specific layouts for each MiniGrid environment in the robustness test set.

	DR	PLR	ACCEL	PATH
NineRooms	$\mathbf{1.00\pm 0.00}$	$0.88\pm 0.12$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
NineRoomsFewerDoors	$0.58\pm 0.17$	$0.99\pm 0.01$	$0.88\pm 0.12$	$\mathbf{1.00\pm 0.00}$
SixteenRooms	$0.84\pm 0.16$	$0.98\pm 0.02$	$0.98\pm 0.02$	$\mathbf{1.00\pm 0.00}$
SixteenRoomsFewerDoors	$0.42\pm 0.18$	$0.61\pm 0.18$	$0.28\pm 0.19$	$\mathbf{0.87\pm 0.13}$
SmallCorridor	$\mathbf{1.00\pm 0.00}$	$0.99\pm 0.01$	$\mathbf{1.00\pm 0.00}$	$\mathbf{1.00\pm 0.00}$
Labyrinth	$0.60\pm 0.24$	$\mathbf{1.00\pm 0.00}$	$0.58\pm 0.22$	$\mathbf{1.00\pm 0.00}$

Appendix

Table 4: Solved rates after

20

k PPO updates (mean

\pm

standard error) over

5

independent runs for each method on each fixed MiniGrid layout (robustness test). For each run, we evaluate the final policy for

100

episodes per layout. Bold entries are within one standard error of the best mean.

Figure 13: Distribution of 1000 valid grids uniformly sampled from $\Theta$ (clutter vs. shortest-path length).

Figure 14: Examples of BipedalWalker terrain classes used for robustness testing.

	DR	PLR	ACCEL	PATH
Basics	$265.2\pm 27.2$	$311.5\pm 2.2$	$312.5\pm 4.9$	$\mathbf{321.6\pm 1.1}$
Hardcores	$31.5\pm 10.1$	$81.8\pm 11.0$	$183.6\pm 10.1$	$\mathbf{240.5\pm 9.0}$
Roughness	$66.8\pm 20.9$	$123.5\pm 16.1$	$230.6\pm 14.5$	$\mathbf{266.9\pm 5.2}$
PitGap	$21.5\pm 8.1$	$38.1\pm 7.3$	$39.1\pm 12.0$	$\mathbf{229.6\pm 21.8}$
Stairs	$37.5\pm 7.6$	$44.2\pm 5.1$	$53.7\pm 11.9$	$\mathbf{177.6\pm 23.7}$
Stump	$-19.9\pm 8.0$	$19.5\pm 8.0$	$93.4\pm 31.7$	$\mathbf{238.9\pm 15.7}$

Appendix

Table 5: Average returns after

20

k PPO updates (mean

\pm

standard error) over

5

independent runs for each method on each robustness test class of BipedalWalker. For each run, we evaluate the final policy for

1

episodes for

128

terrains of each class. Bold entries are within one standard error of the best mean.

Figure 15: Examples from the BipedalWalker generalization test set of 1000 randomly sampled terrains.

Figure 16: An example of the Leaper environment from Procgen.

Method	Solved Rate
DR	$0.57\pm 0.13$
PLR	$0.45\pm 0.08$
ACCEL	$0.51\pm 0.16$
PATH	$\mathbf{0.66\pm 0.04}$

Appendix

Table 6: Leaper generalization results. Evaluated on 100 held-out test environments, averaged over 5 seeds.

Figure 17: Performance on BipedalWalker robustness test environments during training (mean $\pm$ standard error), evaluated every $1$ k PPO updates. We compare three PATH variants: PATH runs PATH:RANDOM until $8$ k updates and then switches to PATH:ACTIVE ; PATH:RANDOM keeps the random phase throughout training; PATH:ACTIVE uses the active phase from the beginning. The vertical dashed line marks the $8$ k transition point for PATH .

	$\textsc{PATH}^{20}_{150}$	$\textsc{PATH}^{30}_{150}$	$\textsc{PATH}^{20}_{100}$	$\textsc{PATH}^{20}_{200}$	$\textsc{PATH}^{20}_{\mathrm{None}}$
Basics	$321.6\pm 1.1$	$320.4\pm 2.2$	$318.9\pm 2.5$	$\mathbf{322.4}\pm\mathbf{2.6}$	$312.6\pm 2.4$
Hardcores	$240.5\pm 9.0$	$\mathbf{252.1}\pm\mathbf{5.0}$	$220.9\pm 10.1$	$226.2\pm 7.8$	$229.2\pm 10.2$
Roughness	$\mathbf{266.9}\pm\mathbf{5.2}$	$253.7\pm 12.7$	$241.9\pm 7.4$	$257.5\pm 2.6$	$245.5\pm 5.7$
PitGap	$\mathbf{229.6}\pm\mathbf{21.8}$	$188.1\pm 28.2$	$164.9\pm 29.8$	$188.0\pm 27.9$	$167.1\pm 45.7$
Stairs	$177.6\pm 23.7$	$192.8\pm 20.6$	$212.8\pm 15.9$	$\mathbf{203.8}\pm\mathbf{25.5}$	$189.1\pm 28.8$
Stump	$\mathbf{238.9}\pm\mathbf{15.7}$	$238.0\pm 12.0$	$210.0\pm 38.4$	$220.4\pm 16.2$	$157.2\pm 25.8$

Appendix

Table 7: Average returns after

20

k PPO updates (mean

\pm

standard error) across

5

independent runs for each ablation variant on the BipedalWalker robustness test classes. For each run, we evaluate the final policy for one episode on each of

128

terrains per class. Bold entries indicate results within one standard error of the best mean.

\textsc{PATH}^{20}_{150}

is the default algorithm.

Metric	$\textsc{PATH}^{20}_{150}$	$\textsc{PATH}^{30}_{150}$	$\textsc{PATH}^{20}_{100}$	$\textsc{PATH}^{20}_{200}$	$\textsc{PATH}^{20}_{\mathrm{None}}$
Test Returns	$\mathbf{42.95\pm 1.31}$	$34.20\pm 7.68$	$25.80\pm 6.86$	$31.88\pm 4.99$	$-6.44\pm 5.27$
Solved Rate	$\mathbf{0.11\pm 0.01}$	$0.10\pm 0.01$	$0.09\pm 0.01$	$0.09\pm 0.01$	$0.07\pm 0.00$

Appendix

Table 8: Generalization test returns and solved rates at

20

k updates (mean

\pm

standard error) for

5

runs of each ablation variant on BipedalWalker. Bold values are within one standard error of the best mean.

\textsc{PATH}^{20}_{150}

is the default algorithm.

PPO
Parameter	MiniGrid	BipedalWalker
$\gamma$	0.995	0.99
$\lambda_{\mathrm{GAE}}$	0.95	0.9
PPO rollout length	256	2000
PPO epochs	5	5
PPO minibatches per epoch	1	32

Appendix

Table 9: Hyperparameters for RL training and curriculum algorithms.

Method	Robustness Solved Rate
DR	$0.18\pm 0.1$
PLR	$0.34\pm 0.0$
ACCEL	$0.56\pm 0.1$
PATH (50% perturb)	$0.43\pm 0.1$
PATH (30% perturb)	$0.53\pm 0.1$
PATH (10% perturb)	$0.67\pm 0.1$

Appendix

Table 10: MiniGrid — Robustness Solved Rate at 10k (same setting as Table 4 ).

Method	Mean Robustness Return
DR	$51.60\pm 10.4$
PLR	$69.47\pm 4.7$
ACCEL	$142.31\pm 9.7$
PATH (50% perturb)	$139.06\pm 12.3$
PATH (30% perturb)	$168.24\pm 8.1$
PATH (10% perturb)	$184.63\pm 7.3$

Appendix

Table 11: BipedalWalker — Mean Robustness Return at 10k (same setting as Table 5 ).

Method	5k updates	10k updates
DR	$0.17\pm 0.1$	$0.18\pm 0.1$
PLR	$0.19\pm 0.1$	$0.38\pm 0.0$
ACCEL	$0.30\pm 0.1$	$0.56\pm 0.1$
PATH:ACTIVE	$0.33\pm 0.1$	$0.59\pm 0.1$
PATH:RANDOM	$0.35\pm 0.0$	$0.70\pm 0.1$
PATH	$\mathbf{0.35\pm 0.0}$	$\mathbf{0.76\pm 0.0}$

Appendix

Table 12: MiniGrid — Robustness Solved Rate under shortened horizon (5 seeds).

Method	Mean Score	Solved Rate
DR	$42.22\pm 10.6$	$0.043\pm 0.02$
PLR	$66.65\pm 8.2$	$0.096\pm 0.02$
ACCEL	$121.16\pm 9.7$	$0.277\pm 0.03$
PATH	$\mathbf{201.14\pm 6.5}$	$\mathbf{0.545\pm 0.02}$

Appendix

Table 13: BipedalWalker generalization in the cropped solvable subspace.

Curriculum RL

Curriculum reinforcement learning organizes tasks, examples, or training conditions to help agents learn complex behaviors efficiently.

See all 116 papers

Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies
Aug 9, 2026Prishita RayCurriculum RL Robot Policy Generalization
Cornell University
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
Feb 24, 2026Zhengyao Gu, Jonathan Light, Raul Astudillo +7Curriculum RL Curriculum Learning
University of Illinois Chicago · Caltech · RPI +2
Curriculum reinforcement learning with measurable task representation learning
May 22, 2026Yongyan Wen, Siyuan Li, Mingjian Fu +3Representation Learning Curriculum RL
Harbin Institute of Technology, Harbin, 150001, Heilongjiang, China · Fuzhou University, Fuzhou, 350108, Fujian, China · China Academy of Sciences, Beijing, 100190, China +1

Aug 9, 2026J/K move · Enter open · S save

Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies

Prishita Ray

Cornell University

Curriculum RL Robot Policy Generalization Robust RL Policy Learning

Open paper AskPDF arXiv

Robust navigation policies for autonomous agents must generalize across continuously varying environmental conditions such as turn rates, obstacles, friction, pits, and slopes. Curriculum generation provides a principled mechanism for improving generalization by progressively adapting training environments, but designing such curricula in a sample-efficient and automated manner remains challenging. This paper proposes a reparameterized curriculum generation framework for structured continuous environment parameters using unidirectional gradient-based optimization. To improve robustness in multimodal observation spaces consisting of image-based and scalar inputs, a distribution-shift regularization objective is incorporated to encourage the learning of finer-grained latent representations. The proposed method is evaluated across two continuous-control OpenAI Gym environments: a 2D obstacle-based Car Racing variant and Bipedal Walker variant, where coupled environment parameters jointly influence policy performance. Across five random seeds, our method consistently outperforms vanilla policy training, random parameter sampling, manual curricula, frontier-based methods, Self-Paced Reinforcement Learning (SPRL), Absolute Learning Progress with Gaussian Mixture Models (ALP-GMM), and reverse curriculum learning baselines. Ablation studies further demonstrate the effectiveness of the reparameterized curriculum mechanism across both environments, while highlighting environment-dependent benefits of the auxiliary regularization objective.

Abstract

Curriculum RL

Explore similar work

Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Curriculum reinforcement learning with measurable task representation learning