Organizations: Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli, F-91190, Gif-sur-Yvette, France · Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS, 75006 Paris, France · EDF R&D, 91120 Palaiseau, France
Curriculum Learning (CL) is a meta-learning paradigm that trains a model by feeding the data instances incrementally according to a schedule, which is based on difficulty progression. Defining meaningful difficulty assessment measures is crucial and most usually the main bottleneck for effective learning, while also in many cases the employed heuristics are only application-specific. In this work, we propose the Dual-Criterion Curriculum Learning (DCCL) framework that combines two views of assessing instance-wise difficulty: a loss-based criterion is complemented by a density-based criterion learned in the data representation space. Essentially, DCCL calibrates training-based evidence (loss) under the consideration that data sparseness amplifies the learning difficulty. As a testbed, we choose the time-series forecasting task. We evaluate our framework on multivariate time-series benchmarks under standard One-Pass and Baby-Steps training schedules. Empirical results show the interest of density-based and hybrid dual-criterion curricula over loss-only baselines and standard non-CL training in this setting.
Figures & tables
Figure 1: Schema of the proposed modular DCCL framework . A representation model ϕθ maps each data instance xn∈D to a vector ϕθ(xn)∈Rd . Based on these representations, a difficulty module δ (dashed block) assigns a score to each instance. These scores are then partitioned into K ordered curriculum buckets B1⪯⋯⪯BK (easy to hard) via Adaptive Filtering . Next, the model fθ is fine-tuned sequentially on each training set Ck , given the Training Scheduler .
Figure 2: Scatter plot of loss vs. density scores for training instances . The complementarity of these two difficulty criteria motivates hybrid strategies that combine both views.
Figure 3: Effect of the convex-mixing parameter α on curriculum buckets . As α varies from 0 (pure density-based) to 1 (pure loss-based), the resulting difficulty ordering smoothly interpolates between the strategies.
Figure 4: Bivariate loss-density stratification. Instances are binned into a 2D grid according to their loss and density scores, then ordered cell-by-cell from easy (low loss, high density) to hard (high loss, low density). Each cell is taken as a bucket.
Strategy
View
Repr. model ϕθ
Difficulty δ(xn)
STL [ 11 ]
STL residuals
ϕθ(xn)=Tn+Sn+Rn
Magnitude of residuals ∣∣Rn∣∣
Loss-based
Loss
Forecasting ϕθ:X→Y
ℓ(ϕθ(xn),yn)
k -NN
Density
Encoder ϕθ:X→Rd
−ρ(ϕθ(xn))
KDE
Density
Encoder ϕθ:X→Rd
−p^(ϕθ(xn))
Convex-value
Loss+Density
Encoder + forecasting model
αδ~loss(xn)+(1−α)δ~dens(xn),α∈[0,1]
Convex-ranks
Loss+Density
Encoder + forecasting model
αrloss(xn)+(1−α)rdens(xn),α∈[0,1]
Table 1: Summary of the CL strategies. Representation model ϕθ and difficulty measurer δ . High δ(xn) suggests a harder instance; ρ is the local neighbourhood density, p^ is the kernel density estimate, and ℓ is the training loss.
Dataset
L
H
s
Sample Rate
Subsample
Batch size (repr/curr)
Electricity
96
24
1
1 h
2000
32
ETT
96
24
1
1 h
2000
8 / 32
Weather
12
4
6
10 min
500
8 / 32
ILI
24
4
1
1 week
None
8 / 32
Solar AL
96
24
6
10 min
2000
8 / 32
Table 2: Dataset-specific forecasting and sampling settings used in experiments.
Dataset
Mean rank
Strategy
Electricity
ETT
ILI
Weather †
Solar AL
↓
One-Pass
No-curriculum
0.486 ± 0.065
1.005 ± 0.085
0.865 ± 0.019
0.201 ± 0.036
0.080 ± 0.009
5.00
Random
0.462 ± 0.059
1.004 ± 0.159
0.871 ± 0.043
0.196 ± 0.060
0.076 ± 0.007
4.40
STL
0.604 ± 0.069
0.928 ± 0.137
0.932 ± 0.031
0.226 ± 0.071
0.082 ± 0.007
6.80
Loss-based
0.399 ± 0.026
1.051 ± 0.627
0.946 ± 0.040
0.169 ± 0.025
0.103 ± 0.006
5.30
k -NN
0.532 ± 0.101
0.582 ± 0.159
1.092 ± 0.191
0.169 ± 0.053
0.186 ± 0.022
6.50
Table 3: Test losses (MSE ↓ ) across curriculum strategies and datasets . Values are mean ± std over 10 runs. Bold and underlined entries denote best and second-best performance within each dataset and schedule (One-Pass or Baby-Steps). The last column reports the mean rank ( ↓ ) per strategy over datasets (ties averaged). Weather † values are multiplied by 102 for readability.
Figure 5: Percentage improvement over the No-curriculum baseline. Top: The results concern each strategy (rows) and dataset/schedule combination (columns). Green cells indicate improvement; red cells indicate degradation. Bottom: Radar plots of percentage improvement over No-curriculum per strategy, separately for the One-Pass (left) and Baby-Steps (right) schedules. Each axis corresponds to a dataset; polygons further from the black reference circle (0% improvement) indicate stronger gains.
Figure 6: Training dynamics under different CL strategies. (a)(b): Training curves of different strategies across buckets under the Baby-Steps schedule. Distribution of instance train losses for two dataset-strategy pairs. (c)(d)(e): Positions corresponds to instance-wise train loss in the space of strategies and coloring to the bucketing of the x -axis strategy. Instances located above the y=x line are considered easier by the x -axis strategy (and conversely)
Curriculum learning couples two design choices, how samples are scored by difficulty and how harder samples are paced into training, making it difficult to attribute observed gains to either component. We disentangle these factors with two evaluation protocols: stage-wise test subsets that validate scoring functions independently of curriculum training, and a baseline that applies the same pacing schedule to randomly ordered data. Within the Transfer Teacher framework (TTF), we use these protocols to evaluate a confusion-aware difficulty score that considers both correct-class confidence and the probability distribution over incorrect classes. On CIFAR-10 with ResNet-18 and VGG-16, the proposed score produces model-interpretable difficulty rankings that align with human intuition. However, at full data, neither curriculum nor anti-curriculum ordering improves accuracy over standard training, indicating that improving the scoring function alone is insufficient to overcome the known failure modes of curriculum learning in TTF. In contrast, We find that confusion-aware curriculum ordering result in consistent data-efficiency benefits, outperforming random ordering by up to 8.7% points at the 20% data regime, suggesting the potential of TTF as a data-efficient training method.
Faculty of Computing, Sri Lanka Institute of Information Technology, Sri Lanka · Faculty of Engineering, University of Sri Jayewardenepura, Sri Lanka · Faculty of Engineering, Sri Lanka Institute of Information Technology, Sri Lanka +2
Curriculum learning is governed by several coupled design choices---how difficulty is defined, how examples are ordered, how much exposure each level receives, and how quickly training moves across levels---making it hard to isolate what actually helps. We present Wasserstein curriculum paths, a simple transport-based framework that decouples these factors by representing curricula as trajectories of training distributions over discrete difficulty levels. Across a calibrated synthetic suite with 12 tasks and 33 difficulty axes, we use this framework to isolate the effects of ordering, matched exposure, endpoint smoothness, and pacing under fixed training budgets. We find that curriculum effects are strongly context-dependent: no single strategy dominates across tasks, difficulty axes, and budgets, and curricula mainly change where a fixed budget is spent most effectively. Within this framework, easy-to-hard ordering improves hard-level performance relative to exposure-matched static sampling, showing that the benefit is not explained by cumulative exposure alone. We further show that endpoint smoothness and pacing substantially affect where along the difficulty spectrum a curriculum is effective. Finally, we show that the same transport view naturally supports extensions to learned pacing through geometry and to structured difficulty spaces beyond one-dimensional orderings.
Changho Shin, David Alvarez-Melis
Princeton University · Microsoft Research & Harvard University
Curriculum learning has been widely adopted in the post-training of large language models by organizing training data from easy to hard. However, its effectiveness varies substantially across reasoning tasks, suggesting that no single curriculum is universally optimal and raising a fundamental question: what determines when curriculum learning works? In this paper, we answer this question by analyzing the optimization dynamics induced by different curriculum schedules. We show that the transfer relationship between different difficulty levels characterizes the optimization dynamics induced by curriculum learning, which in turn explains the effectiveness of different curriculum schedules, and formalize this relationship as Relative Transfer, a principled measure of cross-difficulty knowledge transfer. Based on this measurement, we derive Transfer-aware Dynamic Curriculum Sampling (TDCS), which dynamically adjusts the sampling distribution according to the estimated transfer relationship throughout training. Extensive experiments on multiple reasoning benchmarks demonstrate that TDCS consistently outperforms representative scheduling strategies across different tasks, model scales, and training paradigms. More importantly, our work provides a unified optimization-based explanation of curriculum learning through cross-difficulty transfer.