LLMs "reason" by externalizing intermediate steps as tokens through chain-of-thought (CoT), yet even frontier reasoning models hit abrupt performance cliffs on long-horizon tasks. Implicit reasoning (reasoning within a single forward pass) is useful during CoT generation, as the model can "search ahead" to decide the correct next move without explicit search. Models with weak implicit reasoning are more likely to commit to wrong paths and must attempt to recover via backtracking, while models with strong implicit reasoning identify the correct proof strategy directly, shortening the CoT. However, prior work shows LLMs struggle to reason implicitly beyond shallow depths, with no relief from scale. We investigate whether curriculum learning can extend this depth by fine-tuning on examples with progressively increasing vocabulary size and lookahead (the number of steps a model must search ahead to identify the correct next move). At matched compute, curriculum-trained models achieve lookaheads where no-curriculum training fails. Qwen3-0.6B reaches 98% accuracy on problems with lookahead up to 256 with curriculum, compared to 45% without. However, progress slows as fine-tuning continues, following a saturating exponential that suggests a maximum intrinsic lookahead to which models can learn to reason implicitly. Larger models reach higher fitted ceilings, fine-tuning pretrained models far outpaces training from scratch, and curriculum step sizes that are too large stall progression. Finally, the capability acquired on synthetic data transfers to natural language reasoning, with curriculum-trained models outperforming models trained without curriculum at matched compute on 11 natural language reasoning benchmarks.
Figures & tables
Figure 1: Curriculum versus non-curriculum fine-tuning of the same pretrained LLM. The curriculum orders training problems by increasing difficulty, whereas the baseline draws from the full range of difficulties from the start. On the example proof, the curriculum-trained model takes the correct next step each time, while the baseline commits to a wrong branch and must backtrack. Section 4.1 reports the measured comparison.
Figure 2: Eval accuracy of Qwen3-0.6B at maximum L∈{96,128,256} vs. cumulative training compute (PFLOPs). The dashed line marks the 98% threshold; dotted lines mark where each setting crosses it (Table 1 ).
L
Curriculum
No curriculum
Speedup
96
66.5K
82.8K
1.25×
104
66.5K
106K
1.59×
112
74.3K
140K
1.89×
120
74.3K
106K
1.42×
128
81.8K
186K
2.27×
256
259K
–
–
Table 1: Cumulative training compute (PFLOPs) to reach 98% accuracy at each target lookahead L . A dash marks a setting that never reaches 98% within budget.
Figure 3: Achieved lookahead L vs. cumulative training compute (PFLOPs) for Qwen3-0.6B at s=8 , with the attribute vocabulary grown as ∣Vk∣=bLk+2 or held fixed at its maximum size. The bold entry ( b=2 , growing ∣Vk∣ ) is the configuration used in all other experiments.
Figure 4: Achieved lookahead L vs. cumulative training compute (PFLOPs, log–log scale) for Qwen3-0.6B ( left ) and Pythia-1.4B ( right ) trained under the curriculum with s=1 , with the power-law and saturating exponential fits overlaid on the observed trajectory.
Trajectory
Form
Fitted equation
RMSE
Qwen3-0.6B
Power law
L≈1.03C0.390
6.10
Saturating exponential
L≈324(1−exp(−(C/7.7×105)0.547))
2.22
Pythia-1.4B
Power law
L≈2.45C0.228
2.98
Saturating exponential
L≈82(1−exp(−(C/5.0×105)0.371))
0.53
Table 2: Fitted forms of achieved lookahead L against cumulative training compute C (PFLOPs), with in-sample RMSE.
Held-out RMSE
Trajectory
Power law
Sat. exp.
p -value
Qwen3-0.6B
23.88
17.08
7×10−10
Pythia-1.4B
16.14
0.68
1.5×10−4
Table 3: Held-out RMSE for achieved lookahead L . p -value: significance of the saturating exponential’s advantage under the per-stage iteration test, two-sided. Additional fits on less-trained trajectories in Table 17 .
Figure 5: Achieved lookahead L vs. cumulative training compute for Qwen3-0.6B with step sizes s∈{1,4,8,16,32,64} .
Figure 6: Achieved lookahead L vs. cumulative training compute for the four Pythia sizes at s=1 . At 1M PFLOPs, the curves reach L=9 , 44 , 58 , and 94 , from smallest to largest.
Figure 7: Achieved lookahead L vs. cumulative training compute (PFLOPs), starting from the pretrained checkpoint vs. random initialization. Left: Pythia-1.4B , Right: Qwen3-0.6B .
Curriculum L
No curr.
Δ
Benchmark
8
16
32
48
ProofWriter OWA
41.1
33.6
39.3
54.9 ∗
40.6
+14.3
ProofWriter CWA
66.8
57.2
67.6
67.9 ∗
55.2
+12.8
ParaRules
46.1
36.2
39.1
55.1 ∗
50.4
+4.7
ZebraLogic-MC
47.4
53.4
51.8
53.5 ∗
48.3
+5.2
PrOntoQA †
52.5
55.3
56.8
57.1 ∗
52.0
+5.1
Table 4: Downstream reasoning accuracy (%) for Qwen3-1.7B at curriculum stages L , vs. trained directly at L=48 without curriculum at matched compute. Δ : accuracy of curriculum at L=48 minus that of no curriculum. Best per row in bold. ∗ : curriculum wins are statistically significant (two-sided exact McNemar test on paired items, p<0.05 ); confidence intervals in Appendix F . † Extended by us to test for greater depths (Appendix G ).
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Training loop of our curriculum over lookahead and attribute vocabulary, as described in Section 3.2 .
Setting
Curriculum
No-curriculum
Model
Qwen3-0.6B (full FT)
Qwen3-0.6B (full FT)
Batch size
768
768
Learning rate
5×10−5
5×10−5
Curriculum step s
8
N/A
L
8→256
{96,104,112,120,128,256}
Appendix
Table 5: Hyperparameters for Section 4.1 ( Qwen3-0.6B ).
Model
Batch size
LR
Pythia-160M
192
1×10−4
Pythia-410M
192
5×10−5
Pythia-1.4B
192
2.7×10−5
Pythia-2.8B
1,536
2.7×10−5
Appendix
Table 6: Per-model hyperparameters for Section 4.4 ( Pythia model-size sweep). All sizes: s=1 , AdamW, constant LR.
Condition
Batch size
LR
Init
Pythia-1.4B , pretrained
192
2.7×10−5
Pythia-1.4B base
Pythia-1.4B , random init
768
1×10−4
default initializer
Qwen3-0.6B , pretrained
768
5×10−5
Qwen3-0.6B post-trained release
Qwen3-0.6B , random init
1,536
3×10−4
default initializer
Appendix
Table 7: Hyperparameters for Section 4.5 ( s=1 ; curriculum to L=96 for Pythia-1.4B and to L=256 for Qwen3-0.6B ). The Qwen3-0.6B pretrained row reuses the training from Section 4.2 .
Configuration
Mix
Qwen3-1.7B , curriculum
6%
Qwen3-1.7B , no curriculum (direct at L=48 )
6%
Qwen3-1.7B , instruction only
100%
Appendix
Table 8: Hyperparameters for Section 4.6 . All: AdamW, constant LR 5×10−5 , batch 768, chat template with enable_thinking=False . The curriculum uses s=1 up to L=48 .
Figure 9: Same layout as Figure 2 , for the full sweep L∈{96,104,112,120,128,256} .
Stage L
Joint curriculum
Lookahead only
8
144
401
16
983
2.54K
24
3.14K
4.30K
32
6.19K
7.66K
48
13.6K
20.6K
64
23.5K
43.7K
Appendix
Table 9: Cumulative training compute (PFLOPs) at which each configuration completes stage L for Qwen3-0.6B with s=8 , and the stage completed within a shared budget of 100K PFLOPs. The joint curriculum grows the attribute vocabulary as ∣Vk∣=2(Lk+1) , and the lookahead-only curriculum holds it fixed at its maximum size. A dash (–) means the model never reached that lookahead during training. The fixed-vocabulary configuration ends with L=96 completed after 107K PFLOPs. PFLOPs are computed as 6N per token with actual parameter counts.
Model
Achieved lookahead L
Completed at (PFLOPs)
Pythia-160M
9
921K
Pythia-410M
46
1.72M
Pythia-1.4B
75
4.97M
Pythia-2.8B
133
2.55M
Appendix
Table 10: Achieved lookahead L at the end of training for each Pythia size under the curriculum with s=1 , and the cumulative training compute (PFLOPs, computed as 6N per token) at which that stage is completed.
L completed at cumulative PFLOPs
End of training
Model
100K
200K
300K
500K
750K
1M
L
PFLOPs
Tokens
Pythia-160M
7
7
8
8
8
9
9
995,662
1,022B
Pythia-410M
30
34
37
40
42
44
46
2,017,056
828B
Pythia-1.4B
34
41
46
52
56
58
75
5,000,829
589B
Pythia-2.8B
31
46
60
73
84
94
133
2,561,267
154B
Appendix
Table 11: Curriculum stage completed by each Pythia size ( s=1 ) at fixed cumulative training compute and at the end of training. Pythia-410M leads Pythia-1.4B only below ∼30 K PFLOPs, and Pythia-2.8B overtakes Pythia-1.4B between 100K and 200K PFLOPs. Pythia-2.8B completes L=130 at 2.40M PFLOPs and L=133 at 2.55M PFLOPs; Pythia-1.4B completes L=75 at 4.97M PFLOPs.
Model
8
16
24
32
48
64
96
128
Pythia-160M
253K
–
–
–
–
–
–
–
Pythia-410M
748
7.64K
34.3K
134K
–
–
–
–
Pythia-1.4B
1.07K
7.75K
28.2K
73.6K
385K
1.65M
–
–
Pythia-2.8B
2.62K
15.2K
33.6K
102K
212K
347K
1.04M
2.30M
Appendix
Table 12: Cumulative training compute (PFLOPs) at which each Pythia model completes stage L under the curriculum with s=1 . A dash (–) means the model never reached that lookahead during training. Training ends with L=9 (160M, 996K PFLOPs), 46 (410M, 2.02M), 75 (1.4B, 5.00M) and 133 (2.8B, 2.56M) completed. PFLOPs are computed as 6N per token with actual parameter counts.
Model
8
16
24
32
48
64
96
128
Pythia-160M
260
–
–
–
–
–
–
–
Pythia-410M
0.31
3.1
14.1
55.1
–
–
–
–
Pythia-1.4B
0.13
0.91
3.3
8.7
45.4
194.8
–
–
Pythia-2.8B
0.16
0.91
2.0
6.2
12.7
20.9
62.5
138.4
Appendix
Table 13: Training tokens (billions) at which each Pythia model completes stage L under the curriculum with s=1 ; the same stage completions as Table 12 , in tokens rather than PFLOPs. A dash (–) means the model never reached that lookahead during training.
Pythia-1.4B
Qwen3-0.6B
Stage L
pretrained
random
pretrained
random
8
1.07K
79.7K
618
74.8K
16
7.75K
636K
2.35K
285K
24
28.2K
1.64M
5.78K
618K
32
73.6K
4.41M
10.8K
1.36M
48
385K
–
25.4K
2.22M
Appendix
Table 14: Cumulative training compute (PFLOPs) at which each condition completes stage L under the curriculum with s=1 , and the stage completed at 300K and 1.5M PFLOPs. A dash (–) means the model never reached that lookahead during training. Training from random initialization ends with L=35 completed ( Pythia-1.4B , 6.02M PFLOPs) and L=61 completed ( Qwen3-0.6B , 4.82M PFLOPs). PFLOPs are computed as 6N per token with actual parameter counts.
Figure 10: Achieved lookahead L vs. cumulative training compute (PFLOPs) for Pythia-1.4B fine-tuned from intermediate pretraining checkpoints, each stopped at 500K PFLOPs. The dashed curve is the random-initialization trajectory from Section 4.5 .
Stage L
Step 1,000
Step 10,000
Step 100,000
( ∼ 2.1B tokens)
( ∼ 21B)
( ∼ 210B)
8
9.95K
3.79K
1.04K
16
83.0K
43.1K
7.25K
24
259K
203K
27.7K
32
–
–
66.7K
48
–
–
235K
Appendix
Table 15: Cumulative training compute (PFLOPs) at which Pythia-1.4B , fine-tuned from intermediate pretraining checkpoints, completes stage L under the curriculum with s=1 , and the stage completed at three budgets. Column headers give the pretraining step and the approximate number of pretraining tokens. A dash (–) means the model never reached that lookahead during training. Each trajectory ends at 500K PFLOPs.
Compute (PFLOPs)
Optimizer steps
Stage L
Full FT
r=8
r=64
Full FT
r=8
r=64
32
4.25K
7.97K
4.49K
3,250
8,575
4,375
64
25.6K
34.3K
22.1K
10,700
22,175
12,600
96
66.5K
60.4K
45.2K
19,550
31,000
19,475
128
116K
110K
98.1K
27,575
43,125
31,200
Appendix
Table 16: Cumulative training compute and optimizer steps at which Qwen3-0.6B completes stage L under the curriculum with s=8 , with full fine-tuning (Full FT) and with LoRA at rank r . LoRA compute is counted as 4N+6NLoRA per token.
Held-out RMSE
Trajectory
Power law
Sat. exp.
Ratio
p
Qwen3-0.6B
23.88
17.08
1.40
7×10−10
Pythia-410M
8.23
0.84
9.80
0.018
Pythia-1.4B
16.14
0.68
23.77
1.5×10−4
Pythia-2.8B
9.25
3.72
2.49
0.65
Appendix
Table 17: Held-out prediction error for achieved lookahead L on all four s=1 trajectories, fitting on the first 70% of stage completions and predicting the remainder. Ratio is power-law over saturating-exponential RMSE. p : significance of the saturating exponential’s advantage under the per-stage iteration test of Appendix D.1 , two-sided.
Trajectory
n
Lend
Power-law RMSE [95% CI]
Saturating exponential RMSE [95% CI]
Δ AICc
Qwen3-0.6B
244
244
6.10[3.27,6.75]
2.22[1.57,2.35]
493
Pythia-410M
46
46
1.93[0.71,2.05]
0.47[0.32,0.53]
130
Pythia-1.4B
75
75
2.98[1.69,3.24]
0.53[0.40,0.57]
259
Pythia-2.8B
133
133
2.67[1.92,3.16]
2.15[1.46,2.54]
57
Appendix
Table 18: Fit comparison for the four long s=1 trajectories. Δ AICc is (power-law − saturating exponential); positive favors the saturating exponential, and since both forms have three parameters it reflects fit quality alone. RMSE confidence intervals are 95% moving-block bootstrap. Both in-sample criteria favor the saturating exponential on every trajectory. Held-out prediction is compared in Table 17 and Appendix D.1 .
Trajectory
Lmax [95% CI]
C0 (PFLOPs)
k
Qwen3-0.6B
323.6[304,404]
7.7×105
0.547
Pythia-410M
51.2[49.2,56.9]
1.4×105
0.341
Pythia-1.4B
81.9[77.8,83.0]
5.0×105
0.371
Pythia-2.8B
235.1[175,315]
3.8×106
0.497
Appendix
Table 19: Fitted saturating exponential parameters, L=Lmax(1−exp(−(C/C0)k)) , for the four long s=1 trajectories. Lmax is the fitted ceiling plotted in Figure 11 . Brackets give 95% moving-block bootstrap percentile intervals for Lmax (block length ≈n1/3 , 300 resamples). Their lower ends are stable under longer blocks, while the upper ends for Qwen3-0.6B and Pythia-2.8B widen substantially, so those upper bounds are indicative only.
Figure 11: Saturating exponential fits (black dotted, over the observed range) to the achieved-lookahead trajectories of Pythia-410M , -1.4B , and -2.8B trained with s=1 (log–log). Legends give each fitted ceiling Lmax (Table 19 ).
Cumulative lookahead L(C)
Per-stage iterations ΔI(L)
Trajectory
lag-1 autocorr.
Ljung–Box p
lag-1 autocorr.
Ljung–Box p
Qwen3-0.6B
0.95
2×10−199
−0.14
0.15
Pythia-410M
0.25
0.36
−0.28
0.36
Pythia-1.4B
0.33
0.08
−0.33
0.046
Pythia-2.8B
0.92
4×10−94
−0.10
0.13
Appendix
Table 20: Residual dependence of the saturating exponential fit on cumulative lookahead and on per-stage training iterations (log units). Ljung–Box tests use five lags. Per-stage iterations remove the serial dependence that invalidates a Diebold–Mariano test on cumulative lookahead.
Held-out RMSE ( logΔI )
Mean advantage α
Trend β (per stage)
Trajectory
T
Sat. exp.
Power law
Ratio
Estimate [95% CI]
p
Estimate [95% CI]
p
Qwen3-0.6B
73
0.93
0.96
1.04
0.068[0.049,0.087]
7×10−10
0.0025[0.0016,0.0034]
1×10−6
Pythia-410M
14
1.14
1.90
1.67
2.33[0.47,4.18]
0.018
0.51[0.08,0.95]
0.025
Pythia-1.4B
23
1.06
1.76
1.66
1.97[1.08,2.86]
1.5×10−4
0.18[0.05,0.32]
0.010
Pythia-2.8B
40
1.13
1.13
1.00
6×10−6[−2×10−5,3×10−5]
0.65
−9×10−7[−3×10−6,1×10−6]
0.37
Appendix
Table 21: Held-out prediction of per-stage training iterations. T is the number of held-out stages. α and β are the intercept and slope of dL regressed on the centered stage index, with Newey–West standard errors; α>0 favors the saturating exponential and β>0 indicates an advantage that grows with depth.
Figure 12: Mean training compute per completed curriculum stage for Pythia-1.4B with s=1 . Each bar averages five consecutive completed stages; labels show the mean in PFLOPs. Compute for a stage is measured between consecutive stage completions, with the first stage measured from the start of training. The unfinished stage L=76 is excluded.
Benchmark
Canonical LLM protocol
Our protocol
Reasoning benchmarks
ProofWriter OWA
none
0-shot Direct
ProofWriter CWA
none
0-shot Direct
ParaRules
none
0-shot Direct
ZebraLogic-MC
none
3-shot Direct
PrOntoQA
8-shot CoT ( Saparov et al., 2023 )
8-shot Direct
Appendix
Table 22: Per-benchmark prompting protocol. Where a benchmark has a canonical zero-shot protocol, we keep it; where it has a canonical few-shot protocol, we use its shot count; where the canonical protocol uses chain-of-thought or no LLM protocol exists, we use Direct prompting. All demonstrations show the question and label only and are removed from the scored items. The seven general-capability benchmarks without a canonical protocol use the lm-evaluation-harness default of zero-shot.
Base
Curriculum L
No curr.
Instr. only
Benchmark
8
16
32
48
ProofWriter OWA
40.3 ± 1.8
41.1 ± 1.8
33.6 ± 1.7
39.3 ± 1.7
54.9 ± 1.8 ∗
40.6 ± 1.8
47.8 ± 1.8
ProofWriter CWA
52.7 ± 1.8
66.8 ± 1.7 ∗
57.2 ± 1.8 ∗
67.6 ± 1.7 ∗
67.9 ± 1.7 ∗
55.2 ± 1.8
58.0 ± 1.8
ParaRules
43.8 ± 1.4
46.1 ± 1.4
36.2 ± 1.4
39.1 ± 1.4
55.1 ± 1.4 ∗
50.4 ± 1.5
57.1 ± 1.4
ZebraLogic-MC
25.5 ± 2.7
47.4 ± 3.1
53.4 ± 3.1 ∗
51.8 ± 3.1 ∗
53.5 ± 3.1 ∗
48.3 ± 3.1
53.2 ± 3.1
PrOntoQA †
53.1 ± 1.5
52.5 ± 1.5
55.3 ± 1.5 ∗
56.8 ± 1.5 ∗
57.1 ± 1.5 ∗
52.0 ± 1.5
52.2 ± 1.5
Appendix
Table 23: Downstream reasoning accuracy (%) of Qwen3-1.7B from Table 4 , with 95% confidence intervals, given by a simple normal approximation. Base is the non-finetuned Qwen3-1.7B under the same prompts, with the answer cue prefilled in its own markdown answer format. Instruction-only is trained on the instruction data alone at compute matched to curriculum L=48 . ∗ : curriculum checkpoint significantly above no curriculum (two-sided exact McNemar test on paired items, p<0.05 ). † Extended by us to greater depths (Appendix G ).
Model
MMLU
ARC-E
ARC-C
HellaSwag
WinoG.
PIQA
BoolQ
LAMBADA
Mean
IFEval
Base Qwen3-1.7B
55.5
72.6
42.7
60.5
61.7
72.5
77.6
50.3
61.7
33.1
Curriculum, L=8
56.9
76.2
48.0
62.4
63.2
72.8
76.4
57.4
64.2
35.1
Curriculum, L=16
57.1
75.7
45.9
62.5
62.4
73.3
77.1
58.0
64.0
39.0
Curriculum, L=32
57.3
74.3
45.8
62.5
61.6
73.0
77.8
59.2
63.9
34.6
Curriculum, L=48
56.1
74.4
45.6
64.0
59.8
73.3
77.2
56.5
63.4
41.2
No curriculum
57.5
75.5
45.3
62.7
62.9
73.7
73.2
55.1
63.2
40.1
Appendix
Table 24: General model capability of Qwen3-1.7B across curriculum stages. Accuracy on eight standard benchmarks (zero-shot, lm-evaluation-harness v0.4.11, identical settings for every model) for the non-finetuned model, four curriculum stage checkpoints, the model trained without curriculum, and an instruction-only control, the last two compute-matched to the L=48 checkpoint. “Mean” is the unweighted mean of the eight language understanding benchmarks. IFEval (prompt-level strict accuracy) is reported separately and is excluded from the mean.
Table 25: The fourteen downstream reasoning benchmarks and the nine general-capability benchmarks, with the number of evaluation items n . General-capability benchmarks use the default lm-evaluation-harness splits. ProofWriter appears twice because its open- and closed-world splits differ in label set. NLGraph is scored with the authors’ per-task scorers, and we report the mean over its eight sub-tasks (Appendix E ). † Modified by us, as described in this appendix.
Figure 13: Curriculum over lookahead and attribute vocabulary size. The three stages use maximum lookahead L=1,2,3 and attribute vocabulary size ∣Vk∣=4,6,8 , respectively. The arrows indicate stage advancement. Each DAG is converted into natural-language premises; for example, the edge vimba → vorga reads “If Alice is vimba, then Alice is vorga.” The abbreviated examples below the graphs show the goal and proof prefix; the model predicts the next attribute at the blank. Green marks goal-reaching paths and red marks distractor paths. Neither the colors nor the graph diagrams are part of the model input.
Chain-of-thought reasoning, where language models expend additional computation by producing thinking tokens prior to final responses, has driven significant advances in model capabilities. However, training these reasoning models is extremely costly in terms of both data and compute, as it involves collecting long traces of reasoning behavior from humans or synthetic generators and further post-training the model via reinforcement learning. Are these costs fundamental, or can they be reduced through better algorithmic design? We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles. For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy. These improvements arise purely from adaptive data selection, drawing on classical techniques from boosting and learning from counterexamples, and requiring no assumption on the distribution or difficulty of prompts.
Nived Rajaraman, Audrey Huang, Miro Dudik +3
Microsoft Research · University of Illinois Urbana Champaign
Large language models (LLMs) excel at reasoning when scaled to hundreds of billions of parameters, but small- and mid-scale models remain brittle reasoners even with knowledge distillation (KD). We present Ladders-of-Thought (LoT), a framework that improves reasoning by combining progressive question rewrites with a self-evolving curriculum. LoT automatically generates semantically faithful but easier variants of reasoning problems, organizes them into difficulty buckets using step-based measures, and employs a self-evolving bandit scheduler to allocate training adaptively. Evaluated on two reasoning domains, math and multi-hop reasoning, across 1-8B models from different families, LoT consistently improves over KD. It delivers large gains on arithmetic tasks (e.g., +32 percentage points on AddSub, +25pp on SVAMP), +2-8pp improvements on in-domain test splits, and strong though dataset-dependent benefits on multi-hop reasoning (e.g., +16pp on QASC, +25pp on StrategyQA). LoT also converges faster than staged curricula, highlighting the value of adaptive progression. These results show that progressive rewrites coupled with adaptive curricula provide a simple yet effective recipe for strengthening reasoning in smaller LLMs.
Minghui Liu, Thomas Magelinski, Dehao Yuan +2
University of Maryland, College Park · Capital One
Reinforcement learning with verifiable rewards (RLVR) is widely viewed as a promising path toward continuously improving large language models. Recent works, however, suggest that mainstream RLVR often reallocates sampling probabilities among trajectories already present in the base model: it can improve sampling efficiency, reflected by higher pass@1 scores, but yields limited gains, and can even decrease pass@k scores when k is large, and therefore may fail to expand the base model's reasoning capacity boundary. In this paper, we present a boundary-aware Curriculum RL approach to move beyond the base model's reasoning capacity boundary. Our approach first uses pass@k sampling to locate the current reasoning capacity boundary, then applies targeted teacher guidance to examples near or beyond that boundary, and finally uses RL to consolidate the newly introduced reasoning patterns. Across Qwen, Llama, and DeepSeek base models, boundary-aware Curriculum RL improves both pass@1 scores and pass@256 scores, with pass@1 reflecting one-attempt performance and pass@256 serving as an empirical proxy for the reasoning capacity boundary. In our experiments, average pass@256 improves by 9.8 percentage points over the base models and by 10.3 percentage points over Vanilla RLVR. These results suggest that boundary-aware Curriculum RL can provide a scalable route for LLMs to continuously improve beyond the base model's empirical reasoning capacity boundary.
Pengxiang Cai, Tianchen Fang, Xiaohan Li +3
The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Zhejiang University