Curriculum learning--ordering training examples in a sequence to aid machine learning--takes inspiration from human learning, but has not gained widespread acceptance. Static strategies for scoring item difficulty rely on indirect proxy scores of varying quality and produce curricula that are not specific to the learner at hand. Dynamic approaches base difficulty estimates on gradient information, requiring considerable extra computation during training. We introduce a novel method for measuring the difficulty of individual problem instances that is calibrated to a series of models of increasing competence, and identify transitional problems that are consistently easier as model ability increases. Applying this method to diverse model series constructed from sets of models that are readily available on many tasks, we find that training on a curriculum that levels up from easier to harder transitional problems most efficiently improves a model to the next tier of competence. These problems induce a natural progression from easier to harder items, which outperforms other training strategies. By measuring difficulty directly relative to model competence, our method yields interpretable problems, learner-specific curricula, and a principled basis for step-by-step improvement.
Figures & tables
Figure 1: Transitional Problems at a level i can only and consistently be solved by models at a competence level j≥i . We find that with modest budgets, training on an ascending-level curriculum on transitional problems produces models that perform better than those trained with other strategies.
Figure 2Figure 3
Figure 6: Math: Variation in average question length, answer length, and arithmetic operations for transitional problems vs. all solved problems for a 6-model series obtained from the training checkpoints of Qwen2.5-1.5B-Base on GSM8k.
Figure 5
Figure 9: Chess : Average human rating, number of legal moves, and solution length for transitional problems versus all solved problems at each transitional point. For transitional problems, the human rating and solution length correlate with the competence level, while the number of legal moves does not. None of these measures correlate for the full set of puzzles across levels.
Figure 10: Chess : Each subplot depicts results when a chess model at a particular competence level i is trained on puzzles from a single level (along the x-axis), and tested on its ability to solve game-position problems at level i+1 (accuracies, y-axis). The vertical line in each indicates our hypothesized level to achieve the best performance. The star denotes the actual level that achieves the best performance, which largely aligns with our hypothesis. Error bars represent std across 10 runs.
Figure 11: Gradient L2 norm of LevelUp , LevelDown , and IID curricula during fine-tuning on chess positions. Left: per-step gradient norm (faint lines) overlaid with a centered rolling mean (solid lines). Right: smoothed difference from the IID baseline.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Relative improvement over IID on chess, evaluated on test sets matched to each difficulty measure. Only transitional point-based measures (left) benefit from Level-Up curriculum ordering, while external difficulty measures (right) show marginal improvement or degradation.
Figure 10
Figure 15: Each subplot depicts results when a chess model at a particular competence level i is trained on game-positions from a single level (along the x-axis), and tested on its ability to solve puzzles at level i+1 (accuracies, y-axis). The vertical line in each indicates our hypothesized level to achieve the best performance. The star denotes the actual level that achieves the best performance, which largely aligns with our hypothesis. Error bars represent std across 10 runs.
Figure 12
Figure 18: Variation in average question length, answer length, and arithmetic operations for transitional problems vs. all solved problems for a 5-model series obtained from the training checkpoints of Qwen2.5-0.5B on GSM8k.
Figure 19: Variation in human measures of difficulty for problems obtained from a 4-model series obtained from the training checkpoints of Qwen2.5-1.5B on GSM8k. Like for the 6-model series, the transitional problems vary much more sharply across human-interpretable features than the set of all solved problems.
Figure 15
Figure 22: Variation in human measures of difficulty for a 5-model series obtained from the Qwen2.5 model family. Due to the large improvement in performance between levels, the set of all solved problems also shows an increase in difficulty. However, the transitional problems show a much sharper change between levels, indicating better differentiation in learning difficulty.
Figure 17
Figure 25: Variation in human measures of difficulty for a 6-model series obtained from increasing k -shot adaptation on the Qwen2.5-1.5B model. Even with the reduced differentiation in performance between model levels, the transitional problems exhibit a human-interpretable difficulty increase.
Budget
Learning Rate
Weight Decay
#Train
#Test
#Runs
Max Steps
Batch Size
Tiny
10−3
10−6
10,000
10,000
10
90
64
Low
10−3
10−6
10,000
10,000
10
90
128
Mid
10−3
10−6
10,000
10,000
10
180
256
Appendix
Table 1: Hyperparameter Settings in Chess Experiments.
Size
0.5B-ckpts
1.5B-ckpts
0.5B-family
1.5B-family
0.5B-k-shot
1.5B-k-shot
0.5B-Orca
Tiny
8
8
–
–
–
–
–
Low
10
10
10
10
10
10
10
Mid
15
12
–
12
–
–
–
Appendix
Table 2: Hyperparameter Settings in the Math Experiments.
Setting
Method
Model
Dataset
#Source
#Transitional
#Per level
Math
In-distribution
Qwen2.5-1.5B-Base
GSM8k
6,500
2,092
675 / 462 / 272 / 289 / 394
Math
Distillation
Qwen2.5-0.5B-Base
GSM8k
6,500
5,442
2,181 / 2,113 / 693 / 255
Math
Neo-transitional
Qwen2.5-0.5B-Base
Orca-formatted
5,000
539
174 / 139 / 114 / 56 / 56
Chess
In-distribution
Maia-2
Puzzles
∼ 5.8M
180,000
20,000
Chess
Transfer
Maia-2
Game Positions
∼ 300B
180,000
20,000
Appendix
Table 3: Dataset statistics. #Source is the total pool used to identify transitional problems; #Transitional is the number kept for training (identified for Math, capped for Chess); #Per level lists the per-stage split.
Dataset
Phase
Model
Runtime
GSM8k
Collect checkpoints
Qwen2.5-1.5B-Base
25 min 22 sec
Transitional/GSM8k
Curriculum (per method/baseline)
Qwen2.5-1.5B-Base
8 min 5 sec
Appendix
Table 4: Per-phase runtime for the math experiments.