A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

Ronghan Li; Fei Chen; Mingyu Lu

cs.AIOct 9, 2026

A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

Authors: Ronghan Li, Fei Chen, Mingyu Lu

Organizations: College of Artificial Intelligence, Dalian Maritime University, Dalian, 116026, Liaoning, China. · Information Science and Technology College, Dalian Maritime University, Dalian, 116026, Liaoning, China.

Abstract

Curriculum learning typically organizes training samples from easy to hard. In GRPO, medium-difficulty samples are attractive for online selection because they are more likely to yield informative group-relative learning signals, and prioritizing them can improve training efficiency. However, it remains unclear whether persistently selecting such samples also leads to the strongest final model performance, or whether training can benefit from progressing toward harder samples. Meanwhile, sample difficulty changes as the policy evolves, so fixed difficulty estimates can become misaligned with the current policy. We therefore re-estimate sample difficulty under the current policy and, inspired by the easy-to-hard principle of curriculum learning, design and evaluate a staged curriculum that progressively shifts the active difficulty band toward harder samples. Experiments on GSM8K show that persistent medium-difficulty selection does not achieve the strongest observed performance. Across three random seeds, progressing to a moderately harder difficulty band improves test accuracy to $0.6591 \pm 0.0077$ , compared with $0.6449 \pm 0.0024$ for the fixed-medium curriculum. However, further progression to a harder difficulty band reduces accuracy to $0.6405 \pm 0.0053$ . These results show that staged difficulty progression can outperform persistent medium-difficulty selection, but its benefit is non-monotonic. Effective GRPO curricula can therefore benefit from progressing beyond medium difficulty, while avoiding excessive progression toward overly difficult samples.}

Figures & tables

Figure 1: Motivation and research questions for difficulty-aware training in GRPO. (a) The usefulness of a training problem depends on its current solvability: overly easy or hard problems tend to yield weak group-relative learning signals, whereas intermediate-solvability problems are more likely to produce informative comparisons among sampled responses. (b) Sample difficulty depends on the current policy and can change across difficulty bands as training proceeds, making precomputed difficulty estimates potentially stale. (c) Intermediate-solvability samples are therefore natural candidates for online selection and are emphasized by recent difficulty-aware selection strategies. This motivates two questions examined in this study: whether precomputed difficulty estimates remain aligned with the evolving policy, and whether persistently selecting the medium-difficulty band is sufficient or the curriculum should progress to harder difficulty bands during training.

Figure 2: Comparison of the training paradigms considered in this study. Standard GRPO uses the full training set without difficulty-based selection. The static curriculum uses a fixed easy-to-hard ordering determined by the initial solvability estimate $P_{0}(x)$ . The fixed-medium curriculum reselects samples online according to the current-policy success rate while keeping difficulty band $B_{3}$ fixed. The staged curriculum reselects samples online while moving the active difficulty band through $B_{1}\rightarrow B_{2}\rightarrow B_{3}\rightarrow B_{4}\rightarrow B_{5}$ .

Symbol	Meaning
$x$ , $x_{i}$	A training problem; $x_{i}$ denotes the $i$ -th problem.
$y$	A response sampled from the policy for problem $x$ .
$t$ , $t_{k}$	Training state or checkpoint; $t_{k}$ denotes the $k$ -th tracked checkpoint.
$\theta_{t}$ , $\pi_{\theta_{t}}$	Model parameters and the corresponding policy at training state $t$ .
$V(x,y)$	Binary correctness verifier for response $y$ to problem $x$ .
$p_{t}(x)$	True solvability of problem $x$ under the policy at training state $t$ .

Table 1: Main notation for policy-dependent difficulty and curriculum settings.

Symbol	Meaning
$R_{i}$	Difference between the maximum and minimum empirical solvability of problem $x_{i}$ over the tracked training trajectory.
$D_{i,\mathrm{init}}$	Maximum absolute deviation of problem $x_{i}$ from its initial empirical solvability over the tracked trajectory.
$\Delta_{i,k}$	Difference in empirical solvability of problem $x_{i}$ between consecutive checkpoints $t_{k-1}$ and $t_{k}$ .
$\delta$	Minimum-change threshold used to identify meaningful direction reversals.
$S_{t}(B)$	Set of tracked problems whose empirical solvability lies in band $B$ at training state $t$ .
$N$	Total number of problems in the fixed observation set.

Table 2: Notation for difficulty-dynamics and difficulty-band membership metrics.

Difficulty band	Success-rate interval	Correct out of 8	Relative difficulty
$B_{1}$	$[0.625,0.875]$	5–7	Easier
$B_{2}$	$[0.500,0.750]$	4–6	Moderately easy
$B_{3}$	$[0.375,0.625]$	3–5	Medium
$B_{4}$	$[0.250,0.500]$	2–4	Moderately hard
$B_{5}$	$[0.125,0.375]$	1–3	Harder

Table 3: Difficulty bands and corresponding success-rate intervals used in the online curriculum conditions.

Figure 3: Magnitude and non-monotonicity of changes in empirical solvability during GRPO training. (a) Distribution of the training-period solvability span

R_{i}

across the 1,495 tracked GSM8K problems. The dashed line marks

R_{i}=0.25

; 54.72% of samples have a span of at least 0.25, and the mean span is 0.2770. (b) Fractions of tracked problems exhibiting at least one and at least two direction reversals under different minimum-change thresholds

\delta

. At the primary threshold

\delta=0.125

, 53.38% of samples exhibit at least one reversal, and 36.79% exhibit at least two. (c)–(f) Representative trajectories illustrating increasing solvability, decreasing solvability, a clear direction reversal, and repeated non-monotonic changes, respectively. The label above each panel reports the sample ID, initial solvability

P_{0}

, final solvability

P_{T}

, and the number of direction reversals. For the representative trajectories, tracked checkpoints are shown at equal horizontal spacing for readability; tick labels indicate the corresponding training steps.

Statistic	Result
Mean training-period span $R_{i}$	0.2770
Samples with $R_{i}\geq 0.25$	54.72%
At least one direction reversal, $\delta=0.125$	53.38%
Final Spearman correlation with $P_{0}$	0.8092
Final $B_{3}$ retention	20.74%
Final $B_{3}$ Jaccard similarity	0.1497

Table 4: Core evidence for state-dependent difficulty dynamics.

Figure 4: Global rank consistency and membership turnover in the medium-difficulty band. (a) Spearman rank correlation between the initial solvability ranking and the ranking at each tracked checkpoint for the 1,495 tracked problems. Although rank consistency gradually decreases during training, it remains relatively high at the final checkpoint (

\rho=0.8092

). Tracked checkpoints are shown at equal horizontal spacing for readability; tick labels indicate the corresponding training steps. (b) Membership turnover in the medium-difficulty band

B_{3}=[0.375,0.625]

. Of the 270 problems initially in

B_{3}

, only 56 remain in the final

B_{3}

, corresponding to 20.74% retention. The final

B_{3}

contains 160 problems, of which 104 entered from outside the initial band, corresponding to 65% inflow. The Jaccard similarity between the initial and final

B_{3}

sets is 0.1497.

Setting	Evaluation point	Test accuracy
Fixed medium	Best observed checkpoint	0.6459
Staged curriculum	Selected $B_{4}$ checkpoint	0.6558
Staged curriculum	$B_{5}$ + 50 updates	0.6391
Staged curriculum	$B_{5}$ + 100 updates	0.6353

Table 5: Single-run outcomes for the main curriculum conditions.

Setting	Seed 1	Seed 2	Seed 3	Mean $\pm$ Std.
Standard GRPO: best observed	0.6459	0.6414	0.6459	$0.6444\pm 0.0026$
Static curriculum: best observed	0.6535	0.6497	0.6535	$0.6522\pm 0.0022$
Fixed medium: best observed	0.6459	0.6422	0.6467	$0.6449\pm 0.0024$
Staged: selected $B_{4}$	0.6558	0.6535	0.6679	$0.6591\pm 0.0077$
Staged: $B_{5}$ + 100	0.6353	0.6404	0.6458	$0.6405\pm 0.0053$

Table 6: Multi-seed performance of the curriculum conditions and baselines.

Figure 5: Performance under the fixed-medium curriculum and the staged curriculum. (a) Seed 1 test-accuracy trajectories for the fixed-medium curriculum and the staged curriculum. The staged curriculum improves after progression into

B_{4}

, reaching the selected

B_{4}

checkpoint at 0.6558, whereas further progression into

B_{5}

reduces accuracy to 0.6353 after 100 additional effective updates. Shaded areas indicate the active difficulty band and are shown only to visualize stage progression. (b) Multi-seed comparison of standard GRPO, static curriculum, fixed-medium training, the selected staged

B_{4}

checkpoint, and performance after 100 additional effective updates in

B_{5}

. The bar chart shows the mean GSM8K test accuracy for each training setting to provide a direct visual comparison of overall performance. Detailed per-seed values are reported in Table 6 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Mean	Median	$\geq 0.125$	$\geq 0.25$	$\geq 0.375$
Training-period span $R_{i}$	0.2770	0.2500	79.87%	54.72%	28.96%
Maximum initial deviation $D_{i,\mathrm{init}}$	0.2207	0.1875	73.65%	37.19%	15.92%

Appendix

Table 7: Magnitude of sample difficulty changes over the complete GRPO trajectory.

Threshold $\delta$	At least one reversal	At least two reversals
0.0625	87.02%	78.33%
0.125	53.38%	36.79%
0.25	4.95%	1.14%

Appendix

Table 8: Sensitivity of the direction-reversal rate to the minimum-change threshold.

Training step	Spearman’s $\rho$	Kendall’s $\tau_{b}$
100	0.9262	0.7988
500	0.8548	0.7050
1000	0.8429	0.6932
7473	0.8092	0.6545

Appendix

Table 9: Rank consistency between the initial and current empirical success rates.

Step	$\|S_{t}\|$	Retention	Outflow	Inflow	Jaccard	Mismatch
100	239	58.52%	41.48%	33.89%	0.4501	12.91%
500	188	29.63%	70.37%	57.45%	0.2116	19.93%
1000	187	27.04%	72.96%	60.96%	0.1901	20.80%
7473	160	20.74%	79.26%	65.00%	0.1497	21.27%

Appendix

Table 10: Membership changes in the medium-difficulty band

B_{3}=[0.375,0.625]

Difficulty band	Ever in band	Ever left	Re-entered	Re-entry rate
$B_{1}$	508	498	408	81.93%
$B_{2}$	350	344	289	84.01%
$B_{3}$	270	267	216	80.90%
$B_{4}$	232	230	191	83.04%
$B_{5}$	215	211	174	82.46%

Appendix

Table 11: Re-entry rates for the five difficulty bands.

Figure I6: Initial–final global rank consistency for the 1,495 tracked GSM8K problems. The axes show solvability-rank percentiles at the initial and final checkpoints, with 0 indicating harder problems and 100 indicating easier problems; color denotes the number of problems in each bin. The concentration near the dashed $y=x$ line is consistent with the final Spearman rank correlation of $\rho=0.8092$ .

Seed	Stage	Success-rate interval	Selected local update	Diagnostic accuracy
1	$B_{1}$	$[0.625,0.875]$	150	0.8350
1	$B_{2}$	$[0.500,0.750]$	250	0.8174
1	$B_{3}$	$[0.375,0.625]$	150	0.7432
1	$B_{4}$	$[0.250,0.500]$	150	0.6436
2	$B_{1}$	$[0.625,0.875]$	300	0.9111
2	$B_{2}$	$[0.500,0.750]$	150	0.8232

Appendix

Table 12: Selected stage checkpoints for the three random seeds. Diagnostic accuracy is measured on the fixed subset corresponding to the active difficulty band.

Figure J1: Diagnostic trajectories underlying stage selection across the three random seeds. Each curve shows diagnostic accuracy as a function of local effective updates within the corresponding difficulty band. Highlighted markers denote checkpoints retained under the predefined patience-based criterion. A later checkpoint is considered a new best only when its diagnostic accuracy exceeds the current best by more than the empirical noise tolerance of 0.0107; after a new best is established, two additional evaluations are performed after 50 and 100 additional effective updates, respectively. No post-entry

B_{5}

checkpoint is selected for any of the three seeds.

Stage	0	50	100	150	200	250	300	350	400
$B_{1}$	0.7859	0.7881	0.8018	0.8350	0.8301	0.8359	–	–	–
$B_{2}$	0.7012	0.6914	0.7676	0.7920	0.8027	0.8174	0.8154	0.8017	–
$B_{3}$	0.6904	0.7168	0.7070	0.7432	0.7432	0.7129	–	–	–
$B_{4}$	0.6191	0.5967	0.6211	0.6436	0.5635	0.5515	–	–	–
$B_{5}$	0.5215	0.4971	0.4512	–	–	–	–	–	–

Appendix

Table 13: Diagnostic trajectories for Seed 1. Columns denote local effective updates within each difficulty band.

Stage	0	50	100	150	200	250	300	350	400
$B_{1}$	0.7844	0.8135	0.8438	0.8555	0.8633	0.8936	0.9111	0.8975	0.9033
$B_{2}$	0.7743	0.7959	0.8213	0.8232	0.8057	0.8125	–	–	–
$B_{3}$	0.7135	0.7422	0.7217	0.7363	0.7715	0.7676	0.7334	–	–
$B_{4}$	0.5735	0.5898	0.6172	0.6162	0.6123	–	–	–	–
$B_{5}$	0.5127	0.5086	0.4521	–	–	–	–	–	–

Appendix

Table 14: Diagnostic trajectories for Seed 2. Columns denote local effective updates within each difficulty band.

Stage	0	50	100	150	200	250	300	350	400
$B_{1}$	0.7906	0.8135	0.8354	0.8662	0.8994	0.9209	0.9209	0.9131	–
$B_{2}$	0.7845	0.8123	0.8279	0.8388	0.8311	0.8408	0.8438	0.8203	0.8379
$B_{3}$	0.7578	0.7666	0.7539	0.7471	–	–	–	–	–
$B_{4}$	0.6355	0.6670	0.6807	0.6689	0.5566	–	–	–	–
$B_{5}$	0.5745	0.5713	0.5273	–	–	–	–	–	–

Appendix

Table 15: Diagnostic trajectories for Seed 3. Columns denote local effective updates within each difficulty band.

Training step	Planned mean $P_{0}$	Current mean $P_{t}$	Mean $\|P_{t}-P_{0}\|$
1000	0.9531	0.9796	0.0396
2500	0.8523	0.9422	0.1090
3500	0.7528	0.8384	0.1562
4500	0.6098	0.7275	0.1464
5500	0.3986	0.5573	0.1863
6500	0.1357	0.1544	0.0713

Appendix

Table 16: Difficulty mismatch between the static curriculum plan and the current policy at each evaluation point.

Figure K1: Difficulty misalignment in the static curriculum. (a) Mean solvability specified by the precomputed curriculum schedule and mean solvability under the current policy at each evaluation point. (b) Mean absolute deviation between the planned and current mean solvability estimates at the evaluated training steps.

Seed 1	Seed 2	Seed 3	Mean $\pm$ Std.
0.6535	0.6497	0.6535	$0.6522\pm 0.0022$

Appendix

Table 17: Best observed test performance of the static curriculum across random seeds.

Effective updates	Test accuracy	Curriculum state
400	0.6331	Selected $B_{2}$ checkpoint
450	0.6270	$B_{3}$
500	0.6171	$B_{3}$
550	0.6399	Selected $B_{3}$ checkpoint
600	0.6444	$B_{4}$
650	0.6505	$B_{4}$

Appendix

Table 18: Seed 1 test trajectory for the staged curriculum.

Effective updates	Seed 1	Seed 2	Seed 3
0	0.6141	0.6141	0.6141
50	0.6181	0.6179	0.6133
100	0.6195	0.6194	0.6118
150	0.6223	0.6209	0.6171
200	0.6274	0.6232	0.6224
250	0.6344	0.6270	0.6103

Appendix

Table 19: Test trajectories under the fixed-medium curriculum for the three random seeds.

Explore similar work

Mar 14, 2026

Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning

Curriculum learning--ordering training examples in a sequence to aid machine learning--takes inspiration from human learning, but has not gained widespread acceptance. Static strategies for scoring item difficulty rely on indirect proxy scores of varying quality and produce curricula that are not specific to the learner at hand. Dynamic approaches base difficulty estimates on gradient information, requiring considerable extra computation during training. We introduce a novel method for measuring the difficulty of individual problem instances that is calibrated to a series of models of increasing competence, and identify transitional problems that are consistently easier as model ability increases. Applying this method to diverse model series constructed from sets of models that are readily available on many tasks, we find that training on a curriculum that levels up from easier to harder transitional problems most efficiently improves a model to the next tier of competence. These problems induce a natural progression from easier to harder items, which outperforms other training strategies. By measuring difficulty directly relative to model competence, our method yields interpretable problems, learner-specific curricula, and a principled basis for step-by-step improvement.

Amogh Inamdar, Zhenwei Tang, Ashton Anderson +1

Department of Computer Science Columbia University · Department of Computer Science University of Toronto

Curriculum Learning Data-Efficient Learning

Ask PDF

Apr 7, 2026

Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs

Recent alignment work on Large Language Models (LLMs) suggests preference optimization can improve reasoning by shifting probability mass toward better solutions. We test this claim in a resource-constrained setting by applying GRPO with LoRA to SLMs (up to 3B) for math reasoning on GSM8K and MATH datasets with difficulty-stratified analyses. As problem difficulty increases, accuracy plateaus, revealing a capacity boundary: GRPO primarily reshapes output preferences without reliably improving hardest-tier solving. Consistent with this, training GRPO only on lower-difficulty problems matches full-dataset accuracy across difficulty tiers while using only ~45% training steps, indicating diminishing returns from harder samples in this regime. We also find a cross-dataset generalization effect: GSM8K-trained GRPO achieves higher accuracy on the numeric subset of MATH than MATH-trained GRPO, exceeding it by ~5% at 1.5B and by ~3% at 3B. We show that the best achievable gains depend strongly on the base model's prior reasoning competence and the dataset's difficulty profile.

Suraj Yadav, Siddharth Yadav, Parth Goyal

IIIT Delhi

RL for Language Model Reasoning Small Language Models Group Relative Policy Optimization Numerical Reasoning in Language Models

Ask PDF

Sep 8, 2026

Curriculum Learning as Transport: Understanding Curricula with Wasserstein Geodesics

Curriculum learning is governed by several coupled design choices---how difficulty is defined, how examples are ordered, how much exposure each level receives, and how quickly training moves across levels---making it hard to isolate what actually helps. We present Wasserstein curriculum paths, a simple transport-based framework that decouples these factors by representing curricula as trajectories of training distributions over discrete difficulty levels. Across a calibrated synthetic suite with 12 tasks and 33 difficulty axes, we use this framework to isolate the effects of ordering, matched exposure, endpoint smoothness, and pacing under fixed training budgets. We find that curriculum effects are strongly context-dependent: no single strategy dominates across tasks, difficulty axes, and budgets, and curricula mainly change where a fixed budget is spent most effectively. Within this framework, easy-to-hard ordering improves hard-level performance relative to exposure-matched static sampling, showing that the benefit is not explained by cumulative exposure alone. We further show that endpoint smoothness and pacing substantially affect where along the difficulty spectrum a curriculum is effective. Finally, we show that the same transport view naturally supports extensions to learned pacing through geometry and to structured difficulty spaces beyond one-dimensional orderings.

Changho Shin, David Alvarez-Melis

Princeton University · Microsoft Research & Harvard University

Curriculum Learning

Ask PDF