cs.LGNov 5, 2026
SaveTraining Stability under Adaptive Curricula
Organizations: Mila – Québec AI Institute · Université de Montréal · Google DeepMind · CIFAR AI Chair
Abstract
Unsupervised Environment Design (UED) approaches train reinforcement learning (RL) agents on diverse curricula that may evolve with their capabilities. While prior work has largely focused on how these curricula are constructed, we study whether their students remain trainable under the resulting distributions. Across multiple UED mechanisms, environment complexities, and extended training horizons, we identify recurring learner-side pathologies, including parameter growth, degraded value--outcome ordering, and loss of representational diversity, that accompany stagnation and performance degradation. These findings connect limitations in UED to optimization failures previously identified in conventional deep RL, while showing how adaptive curricula couple the learner's optimization state to its future training distribution. Controlled interventions show that mitigating selected pathologies improves learning. Stabilizing the learner improves long-horizon performance and zero-shot generalization without changing the curriculum rule or increasing environment interaction. Our results establish learner trainability as a complementary design principle for UED. Effective curricula require students that remain capable of learning from the distributions they produce.
Figures & tables
Figure 1: Adaptive replay reveals a performance–trainability tension. Compared with domain randomization (DR; Tobin et al. (2017) ), PLR achieves higher sampled return while exhibiting larger gradient norms, more aligned representations, and weaker value–outcome rank correlation. Differences in value-based learning signals, episode lengths, and value-target distributions further characterize the distinct training experiences under the two methods. We report the interquartile mean after 2B environment steps, aggregated over 5 seeds per method; shaded regions indicate 95% stratified bootstrap confidence intervals.
Figure 2: Learner stabilization improves performance across adaptive UED mechanisms. Evaluation solve rate and sampled return aggregated over PLR, SFL, ACCEL, and random seeds for Small , Medium , and Large Kinetix configurations. Faint solid curves denote baselines; dashed curves denote stabilized learners.
Figure 3: Performance and learner diagnostics across environment complexity. PLR exhibits weaker performance, greater parameter growth, diminishing relative updates, degraded value ordering, and loss of representational diversity in more complex Kinetix configurations.
Figure 4: Curriculum and environment dynamics under PLR. Results for Small , Medium , and Large Kinetix configurations. Shaded regions denote 95% bootstrap confidence intervals across seeds.
Figure 5: Intervention screen and factorial ablation under PLR. Left: Evaluation solve rate and sampled return on Small for the complete intervention suite. See Table 6 for acronym definitions and App. E for a detailed explanation. Right: Comparison of the baseline, weight clipping (WC), HL-Gauss (HL), and WC+HL across Small , Medium , and Large . Several interventions improve over baseline, but WC+HL provides the most consistent gains, including a substantially larger return improvement in the most complex Kinetix configuration ( Large ).
Figure 6: Training stabilization improves performance and mitigates learner degradation. Across environment complexities, WC+HL controls parameter growth while maintaining strong learning signals, preserves value–return ordering, and mitigates late representation degradation in the more challenging settings. These changes accompany consistently higher solve rates and returns.
Figure 7: Training stabilization changes the curriculum realized by PLR. Although the replay rule remains fixed, WC+HL changes student-dependent curriculum signals and shortens episodes while the measured structural properties of replayed levels remain broadly comparable.
Figure 8: Stabilization improves performance across training-distribution mechanisms. WC+HL improves solve rate and return for DR, SFL, and ACCEL across Kinetix complexity levels.
Figure 9: Stabilization extends the useful training horizon of PLR. Beyond the original two-billion-step budget, marked in orange, baseline return and learner diagnostics deteriorate. WC+HL maintains solve rate, continues improving return, and preserves value ordering and effective rank through five billion interactions.
Figure 10: Stabilization improves PLR across parallelism regimes. Top: Increasing parallelism improves performance and learner diagnostics up to the default 2048-environment setting, with little additional gain at 4096 environments. At the default scale, WC+HL achieves stronger performance together with healthier training dynamics than the corresponding baseline. Bottom: Under limited parallelism, with only 128 environments, WC+HL still improves performance, preserves value ordering and effective rank, and controls parameter growth.
Figure 11: Generalization to randomly generated tasks. Zero-shot sampled return across training and evaluation complexities.
Figure 12: Stabilization improves retention under sequential distribution shifts. Ours transfers more strongly to the Medium holdout set and better preserves that performance after training returns to Small. Dotted lines mark distribution changes.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Solve rate (left) and sampled return (right) over 5 seeds. We compare the WC intervention on PLR using different scaling factors for the clipping range. Scaling factors below result in lower performance, while factors of and achieve similar performance. Increasing the scaling factor to leads to a substantial performance degradation.
Figure 14: Solve rate (left) and sampled return (right) over 5 seeds. We compare the PLR baseline, the best WC configuration using a scaling factor of , and our selected interventions. Our interventions show a modest performance improvement over WC, while both substantially outperform the PLR baseline.
Figure 15: Solve rate (left) and sampled return (right) over 5 seeds. We compare the PLR baseline, the WD intervention, and our selected intervention. WD outperforms the PLR baseline, while our intervention achieves higher performance than both WD and the baseline.
Figure 16: Solve rate (left) and sampled return (right) over 5 seeds. We evaluate Spectral Normalization (SN) on PLR by applying it to the actor, the critic, or both networks. Applying SN to the critic or both networks yields similar results, while applying it to the actor achieves higher performance than either configuration.
Figure 17: Solve rate (left) and sampled return (right) over 5 seeds. We contrast the PLR baseline, the best SN configuration applied to the actor, and our selected intervention. SN on the actor outperforms the PLR baseline, while our intervention yields higher performance than both SN and the baseline.
Figure 18: Solve rate (left) and sampled return (right) over 5 seeds. We compare the PLR baseline, ReGen, and the WC + CL (Ours) intervention. Our intervention consistently outperforms both the baseline and ReGen, achieving faster learning and higher values across training, while ReGen performs similarly to or below the baseline.
Figure 19: Solve rate (left) and sampled return (right) over 5 seeds. We compare the HL intervention on PLR using different numbers of bins: , , , and , which control the resolution of the categorical value representation. All tested configurations achieve similar results, consistent with the robustness to this hyperparameter reported in Farebrother et al. (2024) .
Figure 20: Solve rate (left) and sampled return (right) over 5 seeds. We consider the PLR baseline, HL with 51 bins, and our selected intervention. HL outperforms the PLR baseline, while our intervention performs better than both HL and the baseline.
| Hyper-parameter | Value |
|---|---|
| Num Environments | 2048 |
| Num Minibatches | 32 |
| Num epochs | 8 |
| Learning Rate | 5e-5 |
| Discount Factor | 0.99 |
| Gae Lambda | 0.9 |
Table 1: Default PPO hyperparameter settings for Kinetix
| Hyper-parameter | Value |
|---|---|
| Use ACCEL | True |
| Exploratory Grad Updates | True |
| Num Edits | 5 |
| Score Function | MaxMC |
| Level Buffer Capacity | 4000 |
| Replay Probability | 0.5 |
Table 2: Default ACCEL hyperparameter settings for Kinetix
| Hyper-parameter | Value |
|---|---|
| Use ACCEL | False |
| Exploratory Grad Updates | True |
| Num Edits | 2 |
| Score Function | MaxMC |
| Level Buffer Capacity | 4000 |
| Replay Probability | 0.5 |
Table 3: Default PLR hyperparameter settings for Kinetix
| Hyper-parameter | Value |
|---|---|
| Sampled Environments Ratio | 0.5 |
| Batch Size | 4096 |
| Num Batches | 3 |
| Rollout Steps | 512 |
| Num to Save | 1024 |
| Buffer Update Frequency | 128 |
Table 4: Default SFL hyperparameter settings for Kinetix
| Intervention | Weight Clipping | Num. Bins |
|---|---|---|
| Baseline | – | – |
| Weight Clipping (WC) | 2 | – |
| HL-Gauss (HL) | – | 51 |
| WC + HL | 2 | 51 |
Table 5: Hyperparameter settings for the evaluated interventions.
| Acronym | Intervention |
|---|---|
| HL | HL-Gauss |
| RG | ReGen |
| SN-A | Spectral Normalization in the Actor |
| SN-B | Spectral Normalization in Both Actor and Critic |
| SN-C | Spectral Normalization in the Critic |
| WD | Weight Decay |
Table 6: Interventions evaluated in our experiments and their corresponding acronyms.
Figure 21: Generalization to human-designed holdout tasks. Zero-shot solve rate across training and evaluation complexities.
Figure 22: Generalization to randomly generated tasks. Zero-shot sampled return across training and evaluation complexities.
More in this topic
Curriculum RL
Curriculum reinforcement learning organizes tasks, examples, or training conditions to help agents learn complex behaviors efficiently.
See all 128 papers