Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

John J. Vastola; Ann Huang; Satpreet H. Singh; Samuel J. Gershman; Kanaka Rajan

cs.LGOct 22, 2026

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

Authors: John J. Vastola, Ann Huang, Satpreet H. Singh, Samuel J. Gershman, Kanaka Rajan

Organizations: Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University

Abstract

While animals and people tend to learn a task more quickly and reliably when they first train on simpler versions of it, curriculum learning's effectiveness in artificial settings varies, and appears substantially greater in reinforcement learning than in supervised learning. In this work, we construct and analyze a minimal model of policy learning to better understand why. We consider an open-loop, episodic target interception task whose parameters---especially how close the agent must be to the target to receive an informative reward signal---can be chosen so that the agent either receives informative rewards frequently ('dense' rewards) or rarely ('sparse' rewards). We mathematically and numerically analyze the dynamics of REINFORCE-based policy learning in both cases. In the dense reward case, curricula have only marginal benefits; in the sparse case, curricula can be required for learning to occur at all. We find that this is especially true when the agent has the conflicting goals of minimizing effort and accumulating high reward, since this can produce distinct reward landscape optima which compete with one another. Our characterization of useful curricula in this setting indicates that they address two issues: they (i) make rewards less sparse to speed up learning, and (ii) shape the reward landscape to steer agents away from bad local optima.

Figures & tables

Figure 1: Schematic of task and curriculum learning problem. a) During episode $t$ , the agent receives an $F$ -dimensional state vector $\bm{z}_{t}$ as input and must generate a scalar action $u_{t}$ . The goal of the task is to have $u_{t}$ match a target action $u_{R}$ . b) Learning dynamics are stochastic because of feature variance and policy noise. c) We consider two types of returns: a quadratic dense return, and a sparse return that equals a quadratic plus a Gaussian bump. The latter often provides uninformative reward signals if the Gaussian is sufficiently narrow. d) Specifying a curriculum means specifying how some combination of $\beta_{t}$ and $\bm{\Sigma}_{t}$ change over training. Example schedules shown.

Figure 2: Learning dynamics given dense rewards is simple and complex curricula are unnecessary. a) The dense return is parabolic and has a maximum at

u_{*}:=(g/(\alpha+g))u_{R}

. b) Example bias learning trajectory; note that it approaches the optimal bias exponentially quickly. c) Example weight learning trajectories; weights associated with more variable features (orange) converge more quickly than weights associated with less variable features (purple). d) Convergence time scales logarithmically in the discrepancy

d:=|b_{0}-b_{*}|

. e) In the absence of constraints, the optimal curriculum is for feature variances to be as large as possible throughout training. f) With constraints, other solutions (involving potentially non-monotonic changes in task difficulty) are possible. Dashed line: average feature variance over training. g) Weight dynamics with (solid line) and without (dashed line) optimal curriculum. h) Same as in g, but for a weight associated with a different feature. Note that, in the absence of constraints, the optimal curriculum is identical for all feature variances.

Figure 3: Reward sparsity yields slow learning, and weight dynamics produce an implicit curriculum. a) Return shape assuming no effort term. b) For pure bias learning, the convergence time

t^{*}

can be exponentially slow if the scaled distance

d:=\sqrt{\tilde{\beta}}|b_{0}-b_{R}|

is large. c) Example bias and weight learning trajectories in the general case, assuming an isotropic covariance matrix

\bm{\Sigma}=s\bm{I}

with

s=100

. Rightmost panel: the ratio

\frac{(b-b_{R})^{2}}{1/\tilde{\beta}+s\|\bm{w}-\bm{w}_{R}\|_{2}^{2}}

, which transiently remains close to

1

. d) Learning with weights converges much more quickly than pure bias learning (dashed gray line) due to an implicit curriculum effect. e) Same as d, but with

s=3

. f) Example bias and weight learning trajectories assuming

s=3

. While the ratio does not as cleanly remain close to

1

, convergence behavior is similar. Note that weight dynamics is not monotonic, but involves transient movement away from target weights.

Figure 4: Competing optima produce quitting. a) Visualization of the effective return $G_{\text{eff}}(b,\bm{w})$ as a function of $b$ , for different choices of $g/\alpha$ and $v_{\text{eff}}(\bm{w})$ . Black dot: location of ‘quitting’ solution, red dot: location of ‘target’ solution. Multiple optima exist when $g/\alpha$ is high and $v_{\text{eff}}(\bm{w})$ is low. b) Scatter plot of endpoints of simulated REINFORCE trajectories initialized near zero. They all fail to approach the target state (red star), which is here chosen to be close to $1$ for each parameter. c) Left: probability that the agent is within $\sqrt{v_{\text{eff}}}$ of the target parameters over training. Right: the dynamics of the first weight. Different shades of yellow correspond to different feature variances, assuming an isotropic $\bm{\Sigma}=s\bm{I}$ . d) Phase diagram of pure bias learning dynamics. e) Heatmap of clipped (between $0$ and $20$ ) average cumulative returns of simulated REINFORCE trajectories for different feature variances, assuming $\bm{\Sigma}=s\bm{I}$ . Large swaths of parameter space are white, which corresponds to quitting. The moderate feature variance $s=1$ performs best.

Figure 5: Insights about useful curricula. Each row depicts summaries of simulated REINFORCE trajectories to adjudicate between different types of $\beta_{t}$ curricula. From left to right: the $\beta_{t}$ curriculum, quitting probability (likelihood agent is within a fixed small distance from zero), hitting probability (likelihood agent is within $\sqrt{v_{\text{eff}}}$ distance of the target), effective target size $v_{\text{eff}}$ , and the fraction of runs in the bimodal regime. a) Null curricula often do not work, or converge slowly. b) Curricula which involve bifurcations tend to be more effective than ones that do not when the risk of quitting is substantial. The green line produces bifurcating dynamics, while the others do not. c) The timing of the bifurcation matters. If the bifurcation is too early (green), the effective target size remains high and the hitting probability remains low. d) Nonlinear ‘adaptive’ schedules (blue) can outperform linear ones. See Appendix E for a similar plot for $\bm{\Sigma}_{t}$ .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Insights about useful feature variance curricula. Each row depicts summaries of simulated REINFORCE trajectories to adjudicate between different types of $\sigma_{t}$ curricula. From left to right: the $\sigma_{t}$ curriculum, quitting probability (likelihood agent is within a fixed small distance from zero), hitting probability (likelihood agent is within $\sqrt{v_{\text{eff}}}$ distance of the target), effective target size $v_{\text{eff}}$ , and the fraction of runs in the bimodal regime. a) Null curricula may not work, although they are more likely to work for $\sigma_{t}$ than $\beta_{t}$ . b) Curricula which involve bifurcations tend to be more effective than ones that do not when the risk of quitting is substantial. c) The timing of the bifurcation matters. d) Nonlinear ‘adaptive’ schedules (green) can outperform linear ones.

Curriculum RL

Curriculum reinforcement learning organizes tasks, examples, or training conditions to help agents learn complex behaviors efficiently.

See all 121 papers

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning
Jul 29, 2026Siddharth Aphale, Ayushman SinghCurriculum RL Curriculum Learning
Stanford University · Sesame AI
Active Curriculum Refinement for Reinforcement Learning
Aug 26, 2026Zhenya Liu, Yuxin ChenCurriculum RL Curriculum Learning
The University of Chicago, Chicago IL, USA.
Thermodynamics of Reinforcement Learning Curricula
Mar 12, 2026Jacob Adamczyk, Juan Sebastian Rojas, Rahul V. KulkarniReinforcement Learning Curriculum RL
Department of Physics, University of Massachusetts Boston, Boston, MA 02125, USA · Department of Mechanical & Industrial Engineering, University of Toronto, Toronto, ON, CA

Jul 29, 2026J/K move · Enter open · S save

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Siddharth Aphale, Ayushman Singh

Stanford University · Sesame AI

Curriculum RL Curriculum Learning Sparse-Reward RL

Open paper AskPDF arXiv

Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reverse curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. The main choices are which intermediate starts to use (scaffold access) and how to distribute training across them (scaffold allocation). Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce an online, learner-agnostic reset controller that gives every context its own curriculum. We use binary rollout success to decide whether to move a context's start forward or back. A stalled frontier also gets an occasional harder-start trial. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. None of the global schedules we tested solved both groups in the constructed conflict. The per-context controller did. Group-level pacing works if the labels capture how contexts learn. Without those labels, per-context pacing still handles differences within a group. These results suggest pacing assistance separately for contexts that learn at different rates.

Abstract

Curriculum RL

Explore similar work

Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning

Active Curriculum Refinement for Reinforcement Learning

Thermodynamics of Reinforcement Learning Curricula