As LLM reasoning performance plateaus, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing, and difficulty-based budget allocation primarily focus on single-turn settings and fail to address the sequential dependencies inherent in multi-turn reasoning. In this work, we formulate multi-turn reasoning as a sequential compute allocation problem and model it as a multi-objective Markov Decision Process. We propose TAB: Turn-Adaptive Budgets, a budget allocation policy trained via Group Relative Policy Optimization (GRPO) that learns to maximize task accuracy while respecting global per-problem token constraints. Consequently, TAB takes as input the conversation history and learns to adaptively allocate smaller budgets to easier turns and save appropriate number of tokens for the crucial harder reasoning steps. Our experiments on diverse agentic benchmarks demonstrate that TAB achieves a superior accuracy-cost tradeoff saving up to 35% tokens and reducing latency up to 30% while maintaining accuracy over static and off-the-shelf LLM budget baselines. Further, for systems where a plan of all turns is available apriori, we propose TAB All-SubQ, a budget allocation policy that budgets tokens based on the conversation history and entire plan saving up to 40% tokens over baselines.
Figures & tables
Figure 1: Left: A multi-turn reasoning trajectory in agents with turns of varying difficulties. Adaptive difficulty level based per-turn token budget allocations result in improved utilization of compute. Right: A multi-turn system where at every turn t at inference (colored blocks), the user presents a observation from the environment of sub-question qt , the budget allocation policy allots a token budget bt based on the conversation history x1:t−1 and qt to the solver which then generates a response yt . The Budgeter is trained via GRPO with a reward function that is a combination of accuracy and number of tokens used.
Figure 2: Accuracy-Tokens tradeoff of our RL-learned budget allocation policy TAB formed by training Qwen3-4B on MATH dataset with budget penalties B∈{3k,5k,8k,10k} saves up to 20% total tokens (Budgeter + Solver tokens) over baselines. BFCL uses L1-Qwen3-8B-Exact as the Solver, and Tau-Bench and Terminal-Bench use GPT-5.4-nano as the Solver.
Figure 3: Accuracy-Latency tradeoff of TAB. Even after accounting for the inference latency overhead of querying the Budgeter for turn-level allocations, TAB saves up to 15% in overall system latency by significantly reducing the number of Solver output tokens. Note that BFCL uses the L1-Qwen3-8B-Exact Solver hosted locally while the other two benchmarks use the GPT-5.4-nano API with the Budgeter Qwen3-4B hosted locally in all cases.
Figure 4: Accuracy-tokens tradeoff of TAB learned by training Qwen3-1.7B budget policies with budget penalties B∈{3k,5k,8k,10k} that saves up to 35% total tokens (User + Budgeter + Solver tokens) over baselines across 5 math reasoning datasets with a L1-Qwen3-8B-Exact Solver. Futher, TAB All-SubQ with the Budgeter given as input all sub-questions, past and future, along with the conversation history beats a Budgeter without access to future sub-questions. TAB All-SubQ with B=10k saves 12% tokens over TAB and up to 40% tokens over baselines while maintaining accuracy.
Figure 5: TAB reduces overall system latency by up to 30%, with saved by generating fewer solver tokens offsetting the Budgeter’s routing overhead. Additionally, scaling the Budgeter from 1.7B to 4B parameters establishes a superior accuracy-efficiency frontier.
Figure 6: Composition-bar plots of allotted tokens per-turn aggregated over all Math reasoning datasets with the accuracy and total tokens used corresponding to each policy listed above their bars. On the left, TAB learns to adaptively allocate tokens better leading to 4.4 percentage-point higher accuracy while saving 8.5% total tokens. On the right, are TAB policies trained with budget penalties B∈{3k,5k,8k,10k} and as per-problem global budget B increases, the learned policy shifts towards allotting higher per-turn budgets.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Accuracy-Latency Tradeoff of TAB on math reasoning tasks. TAB takes up to 30% lesser time in solving tasks despite the latency overhead of routing queries to a Budgeter.
Figure 8: TAB with Qwen3-4B Budgeter outperforms Qwen3-1.7B Budgeter.