As LLM reasoning performance plateaus, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing, and difficulty-based budget allocation primarily focus on single-turn settings and fail to address the sequential dependencies inherent in multi-turn reasoning. In this work, we formulate multi-turn reasoning as a sequential compute allocation problem and model it as a multi-objective Markov Decision Process. We propose TAB: Turn-Adaptive Budgets, a budget allocation policy trained via Group Relative Policy Optimization (GRPO) that learns to maximize task accuracy while respecting global per-problem token constraints. Consequently, TAB takes as input the conversation history and learns to adaptively allocate smaller budgets to easier turns and save appropriate number of tokens for the crucial harder reasoning steps. Our experiments on diverse agentic benchmarks demonstrate that TAB achieves a superior accuracy-cost tradeoff saving up to 35% tokens and reducing latency up to 30% while maintaining accuracy over static and off-the-shelf LLM budget baselines. Further, for systems where a plan of all turns is available apriori, we propose TAB All-SubQ, a budget allocation policy that budgets tokens based on the conversation history and entire plan saving up to 40% tokens over baselines.
Figures & tables
Figure 1: Left: A multi-turn reasoning trajectory in agents with turns of varying difficulties. Adaptive difficulty level based per-turn token budget allocations result in improved utilization of compute. Right: A multi-turn system where at every turn t at inference (colored blocks), the user presents a observation from the environment of sub-question qt , the budget allocation policy allots a token budget bt based on the conversation history x1:t−1 and qt to the solver which then generates a response yt . The Budgeter is trained via GRPO with a reward function that is a combination of accuracy and number of tokens used.
Figure 2: Accuracy-Tokens tradeoff of our RL-learned budget allocation policy TAB formed by training Qwen3-4B on MATH dataset with budget penalties B∈{3k,5k,8k,10k} saves up to 20% total tokens (Budgeter + Solver tokens) over baselines. BFCL uses L1-Qwen3-8B-Exact as the Solver, and Tau-Bench and Terminal-Bench use GPT-5.4-nano as the Solver.
Figure 3: Accuracy-Latency tradeoff of TAB. Even after accounting for the inference latency overhead of querying the Budgeter for turn-level allocations, TAB saves up to 15% in overall system latency by significantly reducing the number of Solver output tokens. Note that BFCL uses the L1-Qwen3-8B-Exact Solver hosted locally while the other two benchmarks use the GPT-5.4-nano API with the Budgeter Qwen3-4B hosted locally in all cases.
Figure 4: Accuracy-tokens tradeoff of TAB learned by training Qwen3-1.7B budget policies with budget penalties B∈{3k,5k,8k,10k} that saves up to 35% total tokens (User + Budgeter + Solver tokens) over baselines across 5 math reasoning datasets with a L1-Qwen3-8B-Exact Solver. Futher, TAB All-SubQ with the Budgeter given as input all sub-questions, past and future, along with the conversation history beats a Budgeter without access to future sub-questions. TAB All-SubQ with B=10k saves 12% tokens over TAB and up to 40% tokens over baselines while maintaining accuracy.
Figure 5: TAB reduces overall system latency by up to 30%, with saved by generating fewer solver tokens offsetting the Budgeter’s routing overhead. Additionally, scaling the Budgeter from 1.7B to 4B parameters establishes a superior accuracy-efficiency frontier.
Figure 6: Composition-bar plots of allotted tokens per-turn aggregated over all Math reasoning datasets with the accuracy and total tokens used corresponding to each policy listed above their bars. On the left, TAB learns to adaptively allocate tokens better leading to 4.4 percentage-point higher accuracy while saving 8.5% total tokens. On the right, are TAB policies trained with budget penalties B∈{3k,5k,8k,10k} and as per-problem global budget B increases, the learned policy shifts towards allotting higher per-turn budgets.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Accuracy-Latency Tradeoff of TAB on math reasoning tasks. TAB takes up to 30% lesser time in solving tasks despite the latency overhead of routing queries to a Budgeter.
Figure 8: TAB with Qwen3-4B Budgeter outperforms Qwen3-1.7B Budgeter.
Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents. Current LLM agents often generate verbose textual reasoning at every decision step and allocate reasoning effort nearly uniformly across turns, leading to substantial inefficiency in multi-turn agentic trajectories. We propose Adaptive Latent Agentic Reasoning (ALAR), a dual-mode framework that uses compact latent reasoning for routine turns and selectively escalates to explicit chain-of-thought when deeper deliberation is needed. ALAR learns latent reasoning by using the agent's actions as supervision anchors and is further optimized to use latent reasoning when it is sufficient for task success and reserve explicit CoT for harder decisions. Experiments on agentic search and tool-use benchmarks show that ALAR maintains comparable or better task accuracy while substantially reducing generated tokens by up to 43.6% in search and 84.6% in tool use. These results demonstrate that ALAR improves the accuracy-efficiency trade-off of LLM agents by reducing unnecessary textual reasoning while preserving explicit deliberation for harder decision steps.
Dongwon Jung, Peng Shi, Yi Zhang +2
University of California, Davis · University of Waterloo · Greenshoe, Inc.
Scaling test-time compute via extended reasoning has become a key paradigm for improving the capabilities of large language models (LLMs). However, existing approaches optimize reasoning under fixed or uniformly sampled token budgets, ignoring the fundamental mismatch between problem difficulty and allocated compute. This leads to overthinking on easy problems and underthinking on hard ones, resulting in suboptimal token efficiency across diverse reasoning scenarios. In this paper, we propose Budget-Adaptive Curriculum Reasoning (BCAE), a unified framework that jointly optimizes reasoning quality and token efficiency through three synergistic components: (1) a budget-conditioned unified policy that embeds the token budget as a continuous conditioning signal, eliminating the need for decoupled thinking and summarization strategies; (2) a curriculum-aware budget scheduler that adaptively shifts the training budget distribution from easy to hard problems based on real-time learning progress; and (3) a truncation-aware dense reward mechanism that provides fine-grained credit assignment at intermediate reasoning steps via process-level verification. We further introduce Budget-Conditioned Advantage Estimation (BCAE), a novel variance reduction technique that conditions the advantage baseline on the sampled budget, yielding more stable policy gradients. Experiments on mathematical reasoning benchmarks (MATH, GSM8K, AIME, and Minerva Math) demonstrate that BACR consistently outperforms other strong baselines across all token budgets, achieving up to 8.3% accuracy improvement under tight budgets while reducing average token consumption by 34% compared to unconstrained reasoning.
Large language models (LLMs) can achieve strong reasoning performance with sufficient computation, but they do not inherently know how much computation a task requires. We study budgeted inference-time reasoning for multiple problems under a strict global token constraint and formulate it as an Ordered Stochastic Multiple-Choice Knapsack Problem (OS-MCKP). This perspective highlights a meta-cognitive requirement---anticipating problem difficulty and allocating computation strategically. We propose ROI-Reasoning, a two-stage framework that endows LLMs with intrinsic, budget-aware rationality. First, Meta-Cognitive Fine-Tuning teaches models to predict reasoning cost levels and decide whether a problem is worth attempting before generation. Next, Rationality-Aware Reinforcement Learning optimizes sequential decision making under a hard token budget, allowing models to learn long-horizon allocation strategies. Experiments across mathematical reasoning, multiple-choice question answering, and code generation show that ROI-Reasoning improves scores and generally reduces reference-order regret under tight budgets, with up to a 417% relative score improvement over the strongest same-backbone baseline in mathematics.
Muyang Zhao, Chunhao Tian, Qi Qi +1
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China