Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.
Figures & tables
Figure 1: Co-adaptive online training loop of Actor-Curator . At each RL step, a learned curator adaptively selects problems from a large problem bank instead of uniform sampling. The actor is updated on these problems, after which a bandit-style reward based on post-update policy improvement trains the curator. As the actor improves, the curator adapts to prioritize problems that yield the greatest expected performance gains.
Figure 2: Single training iteration of Actor-Curator . At each iteration, a candidate set of problems is sampled from a fixed proposal distribution. The curator reweights this candidate set to select training problems for the actor. After the actor update, per-problem policy improvement is estimated using pre- and post-update policies. The curator observes bandit feedback only on selected problems and is updated using a PPO-style approximation of online stochastic mirror descent.
∣X∣
Benchmark
Method
Improvement
πref
Uniform
SEC
PCL
AC (Ours)
+Δ
+Δ%
30,000
Countdown
0.00
44.74
58.87
57.24
62.12
+3.25
+5.52
Countdown-hard
0.00
41.00
51.50
48.00
58.00
+6.50
+12.62
30,000
Zebra
0.00
35.12
36.00
34.12
37.62
+1.62
+4.50
Zebra-hard
0.00
30.50
27.50
26.00
34.50
+4.00
+13.11
30,000
ARC-1D
0.00
26.74
27.87
26.37
36.37
+8.50
+30.51
Table 1: Peak validation performance on problems within 100 training steps for different methods with qwen2.5-3b. Actor-Curator outperforms both other learning based methods (PCL) and methods that rely on human heuristics (SEC). We see similar results with llama3.2-3b-instruct, as shown in Tab. 4 .
Figure 3: Training dynamics. Actor-Curator Test performance over training on three datasets, showing faster convergence and higher final accuracy.
Figure 4: Training speed-up. Actor-Curator attains high test accuracy with significantly fewer steps.
Figure 7: Ablation results. (a) Actor-Curator is compatible with alternative actor update methods such as GRPO and yields consistent performance gains. (b) The combination of the OSMD curator objective and the policy-improvement target achieves superior performance compared to alternative targets and loss functions. (c) Varying curator model size leads to similar long-term performance, indicating robustness to curator capabilities.
Table 2: Hyper-parameters used for Actor-Curator across all experiments unless otherwise specified.
Countdown
Zebra
ARC-1D
MATH
Qwen2.5-3B-Base
11.86
16.71
17.12
9.56
Llama3.2-3B-IT
9.34
13.63
19.40
9.82
Appendix
Table 3: Average wall-time overhead percentage by model and datasets, averaged over training steps 500 steps.
Figure 8: Effect of candidate batch size on test performance . While bsz=2048 and bsz=512 converge to similar performance at step 200, bsz=8192 suffers from instability. Selected batch size is held constant at 256.
Figure 9: Test set performance across training steps (within 100 training steps) for Qwen2.5-3B-Base across benchmarks and methods.
Figure 10: Test set performance across training steps (within 100 training steps) for Llama3.2-3B-it across benchmarks and methods.
Figure 11: Training efficiency: Actor-Curator attains significant efficiency increase with relatively low overhead on Countdown, Zebra, and ARC.
Benchmark
Model
Method
Improvement
πref
Uniform
SEC
PCL
AC (Ours)
+Δ
+Δ%
Countdown
Qwen2.5
0.00
44.74
58.87
57.24
62.12
+3.25
+5.52
Llama3.2
0.00
63.12
62.78
59.62
66.25
+3.13
+4.96
CD-hard
Qwen2.5
0.00
41.00
51.50
48.00
58.00
+6.50
+12.62
Llama3.2
0.00
58.50
58.00
53.00
60.50
+2.00
+3.42
Zebra
Qwen2.5
0.00
35.12
36.00
34.12
37.62
+1.62
+4.50
Appendix
Table 4: Peak validation performance on problems within 100 training steps across models and methods. Actor-Curator outperforms both other learning based methods (PCL) and methods that rely on human heuristics (SEC). Models: Qwen2.5 refers to Qwen2.5-3B-Base; Llama3.2 refers to Llama3.2-3B-it.