Recent alignment work on Large Language Models (LLMs) suggests preference optimization can improve reasoning by shifting probability mass toward better solutions. We test this claim in a resource-constrained setting by applying GRPO with LoRA to SLMs (up to 3B) for math reasoning on GSM8K and MATH datasets with difficulty-stratified analyses. As problem difficulty increases, accuracy plateaus, revealing a capacity boundary: GRPO primarily reshapes output preferences without reliably improving hardest-tier solving. Consistent with this, training GRPO only on lower-difficulty problems matches full-dataset accuracy across difficulty tiers while using only ~45% training steps, indicating diminishing returns from harder samples in this regime. We also find a cross-dataset generalization effect: GSM8K-trained GRPO achieves higher accuracy on the numeric subset of MATH than MATH-trained GRPO, exceeding it by ~5% at 1.5B and by ~3% at 3B. We show that the best achievable gains depend strongly on the base model's prior reasoning competence and the dataset's difficulty profile.
Figures & tables
Figure 1: Comparison of accuracy on the MATH dataset for Qwen2.5 0.5B, 1.5B, and 3B models Qwen et al. (2025) . Models trained on the lower-difficulty subsets (L1–L3) achieve comparable or superior performance to those trained on the full dataset (L1–L5).
Figure 2: Difficulty-stratified MATH accuracy for base and GRPO-tuned models. GRPO trained on full data (L1–L5) performs similarly to GRPO trained on L1–L3, suggesting limited benefit from including higher-difficulty problems (especially for smaller models).
Figure 3: Cross-dataset generalization under GRPO. Accuracy on the Numerical subset of MATH for 0.5B/1.5B/3B models trained with GRPO on (i) MATH L1–L3, (ii) MATH L1–L5, and (iii) GSM8K.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Level 1
Level 2
Level 3
Level 4
Level 5
Total
Numerical
214
361
407
466
444
1,892
Full
242
476
585
661
716
2,680
Appendix
Table 1: Distribution of Hendrycks MATH test samples across difficulty levels.
Model Size
Per-Device Batch
Grad Accum
Effective Batch
0.5B
20
1
20
1.5B
16
1
16
3B
12
1
12
Appendix
Table 2: GRPO batch scaling for Hendrycks MATH. All runs use K=4 generations.
Model Size
Per-Device Batch
Grad Accum
Effective Batch
0.5B
24
2
48
1.5B
20
2
40
3B
16
2
32
Appendix
Table 3: GRPO batch scaling for GSM8K ( K=4 ).
Figure 4: Answer Extraction Failure Rate vs. Difficulty (MATH Full). Percentage of generations without a parsable <answer>...</answer> span. Failures increase with difficulty; the 1.5B model spikes at Level 5 (18.1%), while 3B remains more stable (13.3%).
Figure 5: Average Completion Tokens vs. Difficulty (MATH Full). Generation length increases with difficulty across model sizes (approaching ∼ 400 tokens at L5 for 1.5B/3B), indicating sustained reasoning effort even when answer extraction fails.
0.5B
1.5B
3B
Subject
Level
L1-L5
L1-L3
L1-L5
L1-L3
L1-L5
L1-L3
Algebra
L1
51.1
57.8
83.0
78.5
86.7
87.4
L2
38.8
41.8
61.7
60.2
72.6
70.6
L3
26.8
32.6
50.6
56.7
67.8
66.7
L4
18.0
20.1
44.9
45.9
51.6
63.6
L5
10.1
12.7
24.8
30.6
36.5
37.5
Appendix
Table 4: Impact of Filtering Hard Data. Accuracy (%) when trained on Full Data (MATH L1–L5) vs. Easy Data (MATH L1–L3).
0.5B
1.5B
3B
Subject
Level
GSM
L1-L5
L1-L3
GSM
L1-L5
L1-L3
GSM
L1-L5
L1-L3
Algebra
L1
61.6
53.6
56.0
91.2
81.6
80.8
93.6
88.8
84.0
L2
49.1
40.9
44.0
73.6
63.5
62.3
83.6
76.1
72.3
L3
37.8
29.5
33.2
67.9
51.3
57.0
75.6
68.9
66.8
L4
21.8
19.4
18.5
52.6
38.9
41.2
62.6
54.0
64.5
L5
12.1
6.1
11.6
27.3
21.7
22.2
34.3
29.3
31.3
Appendix
Table 5: Cross-Domain Transfer Analysis. Accuracy (%) comparing models trained on GSM8K, MATH (L1-L5), and MATH (L1-L3), evaluated by subject and difficulty.
Figure 6: Reward Mean Trajectories. Left/Center: On the MATH datasets (L1-L5 and L1-L3), the 3B model (Green) demonstrates steady monotonic improvement. In contrast, the 0.5B model (Orange) exhibits a flat or noisy trajectory, indicating an inability to consistently optimize the complex reasoning objective. Right: On the GSM8K dataset, all models—including the 0.5B converge reliably, confirming that the instability is specific to the difficulty of the task relative to model capacity.
Figure 7: Reward Standard Deviation: Right (GSM8K): Variance decreases significantly as training progresses (from ∼1.8 to ∼0.6 ), indicating the models are converging on high-confidence solutions. Left (MATH): Variance remains high and volatile for the 0.5B and 1.5B models, suggesting they are relying on stochastic exploration (”guessing”) rather than settling on stable reasoning paths.
Figure 8: Full Fine-Tuning (FFT) accuracy comparison for 0.5B model on the MATH dataset. Models trained on the easier subset (MATH L1–L3) match the performance of models trained on the full difficulty spectrum (MATH L1–L5), mirroring the LoRA-based capacity boundary.
Figure 9: Accuracy comparison of the 1.5B model across three training distributions: the MATH L1–L3 dataset, the MATH L1–L5 subset, and a size-matched random sample of the full dataset MATH L1–L5 (Random 45%).
Figure 10: Training dynamics for the size-matched Random 45% control experiment. (a) Mean Reward trajectory. (b) Reward Standard Deviation. The volatile reward and persistently high variance further illustrate the destabilizing effect of hard samples under a reduced step count.