Anticipating the Consequences of Curriculum Decisions with Large Language Models

Octavio Pappalardo; Nathan Herr; Tim Rocktäschel

cs.LGOct 3, 2026

Anticipating the Consequences of Curriculum Decisions with Large Language Models

Authors: Octavio Pappalardo, Nathan Herr, Tim Rocktäschel

Organizations: University College London

Abstract

Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.

Figures & tables

Figure 1: Evaluation success on selected target tasks during training, with all objective weight assigned to the corresponding target. Lines show means across five seeds; shaded regions indicate one standard deviation.

Figure 2: Illustrative curriculum dynamics with all objective weight assigned to having at least 64 wood. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 3: Evaluation under uniform task weights. The two left panels show mean success rate across all 256 tasks during training; the two right panels show the percentage of tasks whose success rate strictly exceeds each threshold after training. Within each pair, the policy restricted to primitive actions is shown first and the policy with subroutine calls second. Lines show means across five seeds; shaded regions indicate one standard deviation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Goal specification	Values of $n$	Cumulative count
Have at least $n$ wood.	$1,2,4,8,16,32,64$	7
Have at least $n$ stone.	$1,2,4,8,16,32,64$	14
Have at least $n$ coal.	$1,2,4,8,12,24,48$	21
Have at least $n$ iron.	$1,2,3,6,12,24$	27
Have at least $n$ diamonds.	$1,2,3,6,12$	32
Have at least $n$ sapphires.	$1,2,4,8$	36

Appendix

Table 1: Complete 256-goal benchmark. A row containing

n

represents one goal for each listed value; countable nouns take the singular form when

n=1

. All other rows represent one goal. The cumulative count gives the number of goals represented up to and including each row. Goal specifications reproduce the canonical goal descriptions, except that scalar threshold values are replaced by

n

Hyperparameter	Values evaluated	Selected
Learning rate	$3\times 10^{-5},\,5\times 10^{-5},\,10^{-4},\,2\times 10^{-4}$	$2\times 10^{-4}$
Entropy coefficient	$0.005,\,0.01$	$0.005$
Discount factor $\gamma$	$0.995,\,0.999,\,1$	$0.999$
GAE parameter $\lambda$	$0.95,\,0.97,\,0.99$	$0.97$
PPO minibatches	$4,\,8$	$8$
Rollout length per environment	$256,\,512$	$256$

Appendix

Table 2: Numerical hyperparameters considered during tuning. Values were evaluated through successive comparisons rather than a joint grid search.

Uniform task weights (3 seeds)
	$L$	$\alpha$	$\approx 1.0$ B	$\approx 2.0$ B	$\approx 3.0$ B
Default	2	1	$12.40\pm 0.49$	$13.10\pm 0.19$	$13.08\pm 0.46$
	1	1	$12.01\pm 0.33$	$12.72\pm 0.12$	$13.27\pm 0.21$
	3	1	$12.10\pm 0.45$	$12.78\pm 0.17$	$13.34\pm 0.06$
	2	3	$12.23\pm 0.07$	$13.13\pm 0.12$	$13.25\pm 0.11$
	2	10	$12.66\pm 0.29$	$12.75\pm 0.18$	$13.09\pm 0.13$

Appendix

Table 3: Sensitivity of ALP+D+F to maximum path length

L

and downstream coefficient

\alpha

. Entries are evaluation success percentages (mean

\pm

one sample standard deviation across seeds), averaged over all tasks under uniform weights and measured on the target otherwise. Columns show one-third, two-thirds, and the full budget of

3{,}019{,}898{,}880

steps under uniform weights or

1{,}509{,}949{,}440

steps for the targets.

Figure 4: Sensitivity of ALP+D+F to

L

(top,

\alpha=1

) and

\alpha

(bottom,

L=2

). Curves show mean evaluation success; shading indicates one sample standard deviation across three seeds under uniform weights and five seeds for each target task.

Figure 5: Illustrative curriculum dynamics with all objective weight assigned to reaching the Dungeon. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 6: Illustrative curriculum dynamics with all objective weight assigned to defeating a skeleton. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 7: Illustrative curriculum dynamics with all objective weight assigned to placing a furnace. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 8: Curriculum dynamics of ALP+D+F with all objective weight assigned to reaching the Dungeon. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 9: Curriculum dynamics of ALP+D+F with all objective weight assigned to defeating a skeleton. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 10: Curriculum dynamics of ALP+D+F with all objective weight assigned to having at least 64 wood. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 11: Curriculum dynamics of ALP+D+F with all objective weight assigned to placing a furnace. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 12: Evaluation success on selected target tasks during training, with 50% (solid) or 100% (dashed) of the objective weight assigned to the corresponding target. Lines show means across five seeds; shaded regions indicate one standard deviation.

Explore similar work

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
Feb 24, 2026Zhengyao Gu, Jonathan Light, Raul Astudillo +7Curriculum RL Curriculum Learning
University of Illinois Chicago · Caltech · RPI +2
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
Aug 31, 2026Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang +8RL for Language Model Reasoning Curriculum RL
Alibaba Group, Hangzhou, China
Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics
Aug 18, 2026Zhikai Ding, Ziyi YeCurriculum Learning LLM Training
Fudan University

Feb 24, 2026J/K move · Enter open · S save

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Zhengyao Gu, Jonathan Light, Raul Astudillo, Ziyu Ye+6

University of Illinois Chicago · Caltech · RPI+2

Curriculum RL Curriculum Learning RL Post-Training RL for Language Model Reasoning

Open paper AskPDF arXiv

Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.

Abstract

Explore similar work

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics