Chain-of-thought reasoning, where language models expend additional computation by producing thinking tokens prior to final responses, has driven significant advances in model capabilities. However, training these reasoning models is extremely costly in terms of both data and compute, as it involves collecting long traces of reasoning behavior from humans or synthetic generators and further post-training the model via reinforcement learning. Are these costs fundamental, or can they be reduced through better algorithmic design? We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles. For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy. These improvements arise purely from adaptive data selection, drawing on classical techniques from boosting and learning from counterexamples, and requiring no assumption on the distribution or difficulty of prompts.
Figures & tables
Figure 1: An example of autocurriculum for supervised fine-tuning: the learner chooses which prompts to receive teacher CoTs on, based on its accuracy. In each iteration, the learner’s model is updated to digest the new supervision and improve its accuracy.
Table 1: “ ∼ ” refers to equivalence up to polylog(1/ε,1/δ,d,Cseq) factors, Π is the class of next-token models, and Cseq is the sequence-level coverage of the reference model in the RLVR setting ( Definition 4.2 ). Entries in the table describe the complexity of learning a model π such that Accρ(π)⩾1−ε when Π is finite and deterministic. nprompt denotes the number of prompts, nCoT⩽nprompt is the number of CoTs queried from an expert demonstrator; for RLVR, ncomp is the number of model rollouts during training.
Figure 2: An illustration of how models trained on the appropriate prompt distributions can correct errors of prior ones. The region marked in gray captures the region correctly labeled by the plurality of the ensemble of models. Comparing (a) to (c) , the model π3 corrects some errors made by π1 (green checked region), but also introduces new errors (red crossed region). When trained under the appropriate autocurriculum, the accuracy of the ensemble improves geometrically toward 1 .
Figure 3: An example of autocurriculum for RL: the learner chooses which prompts to add to the training batch/dataset, based on accuracy. In each iteration, the learner’s model is updated on these prompts using an RL update to improve its accuracy.
Di={y∈Di:V(xi,yT)=1 and πref(y∣xi)⩾Cseq−1}
Algorithm 4 RLFineTune(Dprompt∥ε,δ,T)
AlgRL(⋅∥ε′,δ′,T)←RLFineTune(⋅∥ε′,δ′,T)
Algorithm 5 AutoTune.RL(Dprompt∥AlgRL,ε,δ,T)
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: For k=120 , we plot of αrj,k as a function of the rank placeholder r across different values of j (number of models in the current ensemble). The shaded green region captures the values of the rank r (for some prompt x ) such that even if the remaining k−j models were to all predict the wrong label on x , the plurality vote remains robustly correct. The shaded red region captures the values of the rank r for which even if the remaining k−j models were to all predict the correct label on x , the plurality vote cannot be guaranteed to be correct. The shaded blue region plots values of the rank which are unattainable (the maximum rank achievable in iteration j is j ).
LLMs "reason" by externalizing intermediate steps as tokens through chain-of-thought (CoT), yet even frontier reasoning models hit abrupt performance cliffs on long-horizon tasks. Implicit reasoning (reasoning within a single forward pass) is useful during CoT generation, as the model can "search ahead" to decide the correct next move without explicit search. Models with weak implicit reasoning are more likely to commit to wrong paths and must attempt to recover via backtracking, while models with strong implicit reasoning identify the correct proof strategy directly, shortening the CoT. However, prior work shows LLMs struggle to reason implicitly beyond shallow depths, with no relief from scale. We investigate whether curriculum learning can extend this depth by fine-tuning on examples with progressively increasing vocabulary size and lookahead (the number of steps a model must search ahead to identify the correct next move). At matched compute, curriculum-trained models achieve lookaheads where no-curriculum training fails. Qwen3-0.6B reaches 98% accuracy on problems with lookahead up to 256 with curriculum, compared to 45% without. However, progress slows as fine-tuning continues, following a saturating exponential that suggests a maximum intrinsic lookahead to which models can learn to reason implicitly. Larger models reach higher fitted ceilings, fine-tuning pretrained models far outpaces training from scratch, and curriculum step sizes that are too large stall progression. Finally, the capability acquired on synthetic data transfers to natural language reasoning, with curriculum-trained models outperforming models trained without curriculum at matched compute on 11 natural language reasoning benchmarks.
Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs). However, existing methods for curating high-quality SFT data rely heavily on strong reasoning models to filter examples based on diversity and difficulty, making the curation process costly while often yielding suboptimal data quality. In this work, we show that diverse and challenging reasoning examples can be identified using only the initial reasoning tokens. Specifically, we demonstrate that difficult problems can be reliably detected based on the loss of the first 100 reasoning tokens evaluated at a randomly perturbed checkpoint of the pretrained model. We further show that examples exhibiting similar loss patterns over their first 1k reasoning tokens across a small number of perturbed checkpoints extrapolating along the fine-tuning trajectory provably induce similar gradients. We validate our approach through extensive experiments on fine-tuning Qwen2.5-7B and Llama3.1-8B models on the M23K medical reasoning and OpenThoughts-Math datasets. Our method outperforms existing baselines by up to 1.7% while being 91% more token efficient.
Hongyi Henry Jin, Wenhan Yang, Meysam Ghaffari +2
Department of Computer Science, University of California Los Angeles (UCLA), California, USA · Optum AI, UnitedHealth Group, Minneapolis, Minnesota, USA
Large language models (LLMs) excel at reasoning when scaled to hundreds of billions of parameters, but small- and mid-scale models remain brittle reasoners even with knowledge distillation (KD). We present Ladders-of-Thought (LoT), a framework that improves reasoning by combining progressive question rewrites with a self-evolving curriculum. LoT automatically generates semantically faithful but easier variants of reasoning problems, organizes them into difficulty buckets using step-based measures, and employs a self-evolving bandit scheduler to allocate training adaptively. Evaluated on two reasoning domains, math and multi-hop reasoning, across 1-8B models from different families, LoT consistently improves over KD. It delivers large gains on arithmetic tasks (e.g., +32 percentage points on AddSub, +25pp on SVAMP), +2-8pp improvements on in-domain test splits, and strong though dataset-dependent benefits on multi-hop reasoning (e.g., +16pp on QASC, +25pp on StrategyQA). LoT also converges faster than staged curricula, highlighting the value of adaptive progression. These results show that progressive rewrites coupled with adaptive curricula provide a simple yet effective recipe for strengthening reasoning in smaller LLMs.
Minghui Liu, Thomas Magelinski, Dehao Yuan +2
University of Maryland, College Park · Capital One