Anticipating the Consequences of Curriculum Decisions with Large Language Models

Octavio Pappalardo; Nathan Herr; Tim Rocktäschel

cs.LGOct 3, 2026

Anticipating the Consequences of Curriculum Decisions with Large Language Models

Authors: Octavio Pappalardo, Nathan Herr, Tim Rocktäschel

Organizations: University College London

Abstract

Automatic curriculum learning can improve the effectiveness of reinforcement learning by selecting the training experiences presented to the agent over time. Predicting the consequences of such decisions can, however, be difficult. We analyze automatic curriculum learning as a sequential decision-making problem, highlighting a gap between the quantities that determine the value of curriculum decisions and the information captured by local learning signals commonly used to guide them. We then investigate whether Large Language Models (LLMs) can exploit richer information about the learning problem to better anticipate the consequences of curriculum decisions. We introduce a method that combines online learning-progress estimates with LLM-informed estimates of (i) the potential downstream benefits of learning on each task and (ii) whether direct training on a task is currently likely to produce progress. We evaluate the approach on a custom benchmark of 256 textual goals in Craftax under different curriculum objectives. We observe the strongest gains when optimizing for individual target tasks. When optimizing across the full task set, the benefits vary across learners with different mechanisms for cross-task transfer, ranging from modest improvements in learning speed to larger gains that persist through the end of training.

Figures & tables

Figure 1: Evaluation success on selected target tasks during training, with all objective weight assigned to the corresponding target. Lines show means across five seeds; shaded regions indicate one standard deviation.

Figure 2: Illustrative curriculum dynamics with all objective weight assigned to having at least 64 wood. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 3: Evaluation under uniform task weights. The two left panels show mean success rate across all 256 tasks during training; the two right panels show the percentage of tasks whose success rate strictly exceeds each threshold after training. Within each pair, the policy restricted to primitive actions is shown first and the policy with subroutine calls second. Lines show means across five seeds; shaded regions indicate one standard deviation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Goal specification	Values of $n$	Cumulative count
Have at least $n$ wood.	$1,2,4,8,16,32,64$	7
Have at least $n$ stone.	$1,2,4,8,16,32,64$	14
Have at least $n$ coal.	$1,2,4,8,12,24,48$	21
Have at least $n$ iron.	$1,2,3,6,12,24$	27
Have at least $n$ diamonds.	$1,2,3,6,12$	32
Have at least $n$ sapphires.	$1,2,4,8$	36

Appendix

Table 1: Complete 256-goal benchmark. A row containing

n

represents one goal for each listed value; countable nouns take the singular form when

n=1

. All other rows represent one goal. The cumulative count gives the number of goals represented up to and including each row. Goal specifications reproduce the canonical goal descriptions, except that scalar threshold values are replaced by

n

Hyperparameter	Values evaluated	Selected
Learning rate	$3\times 10^{-5},\,5\times 10^{-5},\,10^{-4},\,2\times 10^{-4}$	$2\times 10^{-4}$
Entropy coefficient	$0.005,\,0.01$	$0.005$
Discount factor $\gamma$	$0.995,\,0.999,\,1$	$0.999$
GAE parameter $\lambda$	$0.95,\,0.97,\,0.99$	$0.97$
PPO minibatches	$4,\,8$	$8$
Rollout length per environment	$256,\,512$	$256$

Appendix

Table 2: Numerical hyperparameters considered during tuning. Values were evaluated through successive comparisons rather than a joint grid search.

Uniform task weights (3 seeds)
	$L$	$\alpha$	$\approx 1.0$ B	$\approx 2.0$ B	$\approx 3.0$ B
Default	2	1	$12.40\pm 0.49$	$13.10\pm 0.19$	$13.08\pm 0.46$
	1	1	$12.01\pm 0.33$	$12.72\pm 0.12$	$13.27\pm 0.21$
	3	1	$12.10\pm 0.45$	$12.78\pm 0.17$	$13.34\pm 0.06$
	2	3	$12.23\pm 0.07$	$13.13\pm 0.12$	$13.25\pm 0.11$
	2	10	$12.66\pm 0.29$	$12.75\pm 0.18$	$13.09\pm 0.13$

Appendix

Table 3: Sensitivity of ALP+D+F to maximum path length

L

and downstream coefficient

\alpha

. Entries are evaluation success percentages (mean

\pm

one sample standard deviation across seeds), averaged over all tasks under uniform weights and measured on the target otherwise. Columns show one-third, two-thirds, and the full budget of

3{,}019{,}898{,}880

steps under uniform weights or

1{,}509{,}949{,}440

steps for the targets.

Figure 4: Sensitivity of ALP+D+F to

L

(top,

\alpha=1

) and

\alpha

(bottom,

L=2

). Curves show mean evaluation success; shading indicates one sample standard deviation across three seeds under uniform weights and five seeds for each target task.

Figure 5: Illustrative curriculum dynamics with all objective weight assigned to reaching the Dungeon. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 6: Illustrative curriculum dynamics with all objective weight assigned to defeating a skeleton. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 7: Illustrative curriculum dynamics with all objective weight assigned to placing a furnace. The top row shows the probability mass allocated to the target and selected related auxiliary goals, averaged over non-overlapping blocks of 15 curriculum decisions. The bottom row shows the policy’s success rate when evaluated on each goal. For each method, we show the run with the highest final target success among the five seeds in Figure 1 .

Figure 8: Curriculum dynamics of ALP+D+F with all objective weight assigned to reaching the Dungeon. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 9: Curriculum dynamics of ALP+D+F with all objective weight assigned to defeating a skeleton. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 10: Curriculum dynamics of ALP+D+F with all objective weight assigned to having at least 64 wood. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 11: Curriculum dynamics of ALP+D+F with all objective weight assigned to placing a furnace. Each column shows one of the five training seeds in Figure 1 . The top row shows allocation probability averaged over non-overlapping blocks of 15 curriculum decisions; the bottom row shows evaluation success.

Figure 12: Evaluation success on selected target tasks during training, with 50% (solid) or 100% (dashed) of the objective weight assigned to the corresponding target. Lines show means across five seeds; shaded regions indicate one standard deviation.

Explore similar work

Feb 24, 2026

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.

Zhengyao Gu, Jonathan Light, Raul Astudillo +7

University of Illinois Chicago · Caltech · RPI +2

Curriculum RL Curriculum Learning RL Post-Training RL for Language Model Reasoning

Ask PDF

Aug 31, 2026

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.

Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang +8

Alibaba Group, Hangzhou, China

RL for Language Model Reasoning Curriculum RL Multi-Task RL LLM Post-Training Policy Learning

Ask PDF

Aug 18, 2026

Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

Curriculum learning has been widely adopted in the post-training of large language models by organizing training data from easy to hard. However, its effectiveness varies substantially across reasoning tasks, suggesting that no single curriculum is universally optimal and raising a fundamental question: what determines when curriculum learning works? In this paper, we answer this question by analyzing the optimization dynamics induced by different curriculum schedules. We show that the transfer relationship between different difficulty levels characterizes the optimization dynamics induced by curriculum learning, which in turn explains the effectiveness of different curriculum schedules, and formalize this relationship as Relative Transfer, a principled measure of cross-difficulty knowledge transfer. Based on this measurement, we derive Transfer-aware Dynamic Curriculum Sampling (TDCS), which dynamically adjusts the sampling distribution according to the estimated transfer relationship throughout training. Extensive experiments on multiple reasoning benchmarks demonstrate that TDCS consistently outperforms representative scheduling strategies across different tasks, model scales, and training paradigms. More importantly, our work provides a unified optimization-based explanation of curriculum learning through cross-difficulty transfer.

Zhikai Ding, Ziyi Ye

Fudan University

Curriculum Learning LLM Training Transfer Learning

Ask PDF