Organizations: State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Meituan
Curriculum learning, which organizes training samples according to their learning priority, has been widely adopted in reinforcement learning to improve training efficiency. However, existing methods commonly construct curriculum units using predefined data properties and estimate their utility by simply aggregating a single proxy signal from recent training history.Such designs may group samples with different optimization effects and fail to anticipate how their utility evolves in relation to the policy. We propose Predictive Gradient-based Bandit (PGB) Curriculum, which first discovers optimization-aware learning units by clustering projected sample-level gradients. PGB then employs a lightweight temporal predictor to forecast the future utility of each unit and converts the predictions into a Boltzmann sampling distribution for adaptive data allocation. Experiments across multiple model scales and reasoning benchmarks show that PGB consistently outperforms standard GRPO, difficulty-based curricula, and history-based scheduling strategies, demonstrating improved data utilization and training efficiency.
Figures & tables
Figure 1: (a) Comparison between existing curriculum methods and PGB. (b) Task groups in gradient space. Each point is a sample, colored by task category.
Figure 2: Overview of our proposed Predictive Gradient-based Bandit (PGB) Curriculum method. PGB first discovers gradient-based learning units according to optimization effects and then performs curriculum scheduling by predicting learning utility from evolving policy states within a multi-armed bandit framework.
Model
Method
ID
OOD
DAPO-Math
AIME 24
AIME 25
AIME 26
Mean AIME
MATH-500
AMC
Mean
Qwen3-1.7B
Original
0.121
0.000
0.067
0.033
0.033
0.282
0.150
0.104
GRPO
0.559
0.233
0.233
0.200
0.222
0.770
0.700
0.426
Difficulty-UCB
0.565
0.267
0.233
0.233
0.242
0.742
0.800
0.454
PGB (Ours)
0.608
0.367
0.300
0.300
0.322
0.710
0.800
0.495
Qwen3-4B
Original
0.245
0.100
0.067
0.100
0.089
0.692
0.425
0.277
Table 1: Accuracy comparison on ID and OOD benchmarks across different model scales. Our PGB consistently achieves the best or comparable results across settings compared with standard GRPO and difficulty-UCB curriculum baselines, demonstrating strong generalization and scalability.
Figure 3: Training curves of mean reward for PGB + GRPO and GRPO on Qwen3-4B and 8B models.
Method
DAPO
AIME
MATH-500
AMC
GRPO
0.709
0.433
0.850
0.825
Difficulty
0.800
0.499
0.842
0.875
Rubric
0.807
0.433
0.846
0.900
Gradient
0.811
0.533
0.852
0.975
Table 3: Comparison of different unit construction strategies across math benchmarks on Qwen3-4B.
Method
DAPO
AIME
MATH-500
AMC
Boltzmann
0.570
0.242
0.702
0.800
UCB
0.572
0.222
0.706
0.775
Predictive
0.608
0.322
0.710
0.800
Table 4: Comparison of predictive and reward-based curriculum scheduling strategies on reasoning benchmarks on Qwen3-1.7B.
Figure 4: Analysis of predictive curriculum scheduling.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
τ
DAPO
AIME
MATH-500
AMC
0.2
0.739
0.489
0.854
0.825
0.3
0.740
0.411
0.860
0.875
0.4
0.761
0.489
0.862
0.95
0.5
0.745
0.489
0.860
0.95
0.6
0.734
0.500
0.860
0.9
Appendix
Table 5: Comparison of different temperature parameter τ across math reasoning benchmarks on Qwen3-4B.
Figure 5: Visualization of task-based groups and gradient-based learning units in gradient space. Samples from the same rubric category are widely distributed and overlap with other categories, while gradient-based clustering discovers more coherent optimization structures.
Group
Grad Norm Std
Loss Mean
Loss Std
0
0.567
0.731
0.118
1
0.657
0.838
0.109
2
0.520
0.861
0.125
3
0.496
0.772
0.116
4
0.576
0.707
0.148
5
0.567
0.729
0.129
Appendix
Table 6: Optimization statistics of task-based groups.
Figure 6: Comparison of future utility prediction strategies.
Figure 7: Sampling probability dynamics of learning unit 0-4 under predictive and UCB-based curriculum scheduling.
Method
Chemistry
Biology
Physics
Original
0.443
0.240
0.500
GRPO
0.505
0.260
0.525
PGB
0.524
0.300
0.538
Appendix
Table 7: Performance comparison on science reasoning tasks from SciKnowEval Feng et al. (2025) with Qwen3-1.7B.