Scaling test-time compute via extended reasoning has become a key paradigm for improving the capabilities of large language models (LLMs). However, existing approaches optimize reasoning under fixed or uniformly sampled token budgets, ignoring the fundamental mismatch between problem difficulty and allocated compute. This leads to overthinking on easy problems and underthinking on hard ones, resulting in suboptimal token efficiency across diverse reasoning scenarios. In this paper, we propose Budget-Adaptive Curriculum Reasoning (BCAE), a unified framework that jointly optimizes reasoning quality and token efficiency through three synergistic components: (1) a budget-conditioned unified policy that embeds the token budget as a continuous conditioning signal, eliminating the need for decoupled thinking and summarization strategies; (2) a curriculum-aware budget scheduler that adaptively shifts the training budget distribution from easy to hard problems based on real-time learning progress; and (3) a truncation-aware dense reward mechanism that provides fine-grained credit assignment at intermediate reasoning steps via process-level verification. We further introduce Budget-Conditioned Advantage Estimation (BCAE), a novel variance reduction technique that conditions the advantage baseline on the sampled budget, yielding more stable policy gradients. Experiments on mathematical reasoning benchmarks (MATH, GSM8K, AIME, and Minerva Math) demonstrate that BACR consistently outperforms other strong baselines across all token budgets, achieving up to 8.3% accuracy improvement under tight budgets while reducing average token consumption by 34% compared to unconstrained reasoning.
Figures & tables
MATH
GSM8K
Method
512
1024
2048
4096
512
1024
2048
4096
Standard CoT
38.1
45.3
52.6
58.4
72.3
79.1
84.5
87.2
Self-Consistency
41.5
49.7
57.8
63.1
76.8
83.4
88.1
90.6
GRPO
44.2
53.6
62.4
68.7
80.5
86.2
89.8
91.4
DAPO
45.1
54.8
63.2
69.3
81.2
87.0
90.3
91.8
SelfBudgeter
47.3
56.1
64.5
68.9
83.7
88.4
90.8
91.2
Table 1: Accuracy (%) across different token budgets on MATH and GSM8K. Best results are in bold , second-best are underlined .
AIME 2024
Minerva Math
Method
1024
2048
4096
Avg Tok
1024
2048
4096
Avg Tok
GRPO
4.2
12.5
20.8
3842
35.6
44.2
52.8
3567
DAPO
4.2
12.5
25.0
3791
37.4
46.0
54.2
3489
SelfBudgeter
4.2
16.7
25.0
2856
39.8
48.6
53.4
2734
AnytimeReasoner
8.3
20.8
25.0
2643
42.2
51.4
56.0
2518
BACR (Ours)
16.7
25.0
33.3
2187
46.8
56.8
60.4
2245
Table 2: Accuracy (%) on challenging benchmarks with different budgets and average token usage.
BUP
CAS
TDR
BCAE
Acc (%)
Δ
58.2
–
✓
59.8
+1.6
✓
✓
60.4
+2.2
✓
✓
60.1
+1.9
✓
✓
60.6
+2.4
✓
✓
✓
61.0
+2.8
Table 3: Ablation study on MATH (1024-token budget). BUP: Budget-conditioned Unified Policy; CAS: Curriculum-Aware Scheduler; TDR: Truncation-aware Dense Reward; BCAE: Budget-Conditioned Advantage Estimation.
Figure 1: Performance evaluation: (a) efficiency trade-off and (b) variance analysis of advantage estimation.
Figure 2: Evolution of budget distributions and sampling weights across training epochs for different difficulty groups (Easy/Medium/Hard).
Figure 3: Training dynamics: (a) convergence speed compared to baselines and (b) accuracy improvement across different MATH difficulty levels.
Figure 4: Sensitivity analysis of BACR to curriculum adaptation rate α (left) and dense reward weight λ (right) on MATH at 1024-token budget.
Method
1.5B
7B
14B
GRPO
38.4
53.6
62.1
AnytimeReasoner
42.1
58.2
66.5
BACR (Ours)
45.2
61.5
69.3
Δ vs. AnytimeReasoner
+3.1
+3.3
+2.8
Table 4: Scalability: MATH accuracy (%) at 1024-token budget across model sizes.
Large language models (LLMs) can achieve strong reasoning performance with sufficient computation, but they do not inherently know how much computation a task requires. We study budgeted inference-time reasoning for multiple problems under a strict global token constraint and formulate it as an Ordered Stochastic Multiple-Choice Knapsack Problem (OS-MCKP). This perspective highlights a meta-cognitive requirement---anticipating problem difficulty and allocating computation strategically. We propose ROI-Reasoning, a two-stage framework that endows LLMs with intrinsic, budget-aware rationality. First, Meta-Cognitive Fine-Tuning teaches models to predict reasoning cost levels and decide whether a problem is worth attempting before generation. Next, Rationality-Aware Reinforcement Learning optimizes sequential decision making under a hard token budget, allowing models to learn long-horizon allocation strategies. Experiments across mathematical reasoning, multiple-choice question answering, and code generation show that ROI-Reasoning improves scores and generally reduces reference-order regret under tight budgets, with up to a 417% relative score improvement over the strongest same-backbone baseline in mathematics.
Muyang Zhao, Chunhao Tian, Qi Qi +1
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Chain-of-Thought (CoT) has significantly enhanced LLM reasoning, yet often incurs substantial computational overhead due to "overthinking": generating excessively long rationales without commensurate accuracy gains. Existing efficiency methods typically apply uniform compression, which overlooks a critical observation that reasoning complexity is heterogeneous at two distinct granularity: across different problems and within individual reasoning steps. This motivates our principle of Thinking Economically: intelligently allocating computational resources based on intrinsic task and step demands rather than pursuing uniform brevity. We propose Hierarchical Adaptive Budgeter (HAB), a training framework that operationalizes this principle through coarse-to-fine budgeting. At the inter-step level, HAB predicts the optimal reasoning depth for each problem. At the intra-step level, HAB learns step-specific token budgeting signals from PPL-derived step comparisons and an adaptive Pareto optimization objective that captures the local quality-efficiency trade-off, while a Fisher Information-based pruner further provides fine-grained training-time guidance, thereby encouraging the generator to internalize more economical reasoning patterns. Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage, achieving a stronger performance-efficiency trade-off than the compared baselines.
Yubo Gao, Haotian Wu, Hong Chen +8
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Nanyang Technological University +1
Scaling test-time compute through extended chains of thought has become a dominant paradigm for improving large language model reasoning. However, existing research implicitly assumes that longer thinking always yields better results. This assumption remains largely unexamined. We systematically investigate how the marginal utility of additional reasoning tokens changes as compute budgets increase. We find that marginal returns diminish substantially at higher budgets and that models exhibit ``overthinking'', where extended reasoning is associated with abandoning previously correct answers. Furthermore, we show that optimal thinking length varies across problem difficulty, suggesting that uniform compute allocation is suboptimal. Our cost-aware evaluation framework reveals that stopping at moderate budgets can reduce computation significantly while maintaining comparable accuracy.
Shu Zhou, Rui Ling, Junan Chen +3
Nanjing University · Baidu · Nanjing University of Finance & Economics