Inference-Time Compute Allocation

Follow this topic Ask about this topic

This field studies how to distribute limited prediction-time resources—such as reasoning tokens, candidate samples, model calls, search steps, or GPU time—across questions, reasoning paths, and agent actions. Common approaches use difficulty or uncertainty estimates to route tasks, adapt sampling, guide verifier-based search, stop early, or escalate to stronger models; systems work also schedules requests against latency and budget constraints. Evaluations span mathematical reasoning and coding benchmarks as well as retrieval, embodied planning, and multi-tenant inference. A central challenge is that allocation gains depend on whether available signals predict the value of extra computation, and apparent improvements can vanish when costs, selection rules, runtime conditions, and held-out evaluation are controlled.

Latest papers

82 papers18 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Compute Where it Counts: Self Optimizing Language Models
May 11, 2026Yash Akhauri, Mohamed S. AbdelfattahLLM Inference Efficiency RL for Language Models
Cornell University
Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols
May 10, 2026Julia Hu, Alfred Shen, Kumar LakshmipathiLLM Evaluation Multi-Agent Debate
Amazon Web Services
When Independent Sampling Outperforms Agentic Reasoning
May 8, 2026Yihe Dong, Boris ShigidaCost-Aware Inference Competitive Programming
Princeton University
CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation
May 8, 2026James Petullo, Nianwen XueInference-Time Search Inference-Time Scaling
Brandeis University / Waltham, MA
Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents
May 7, 2026Ziming Li, Jiatan Huang, Xiaoguang Guo +2LLM Agent Training Inference-Time Compute Allocation
University of Connecticut · New Jersey Institute of Technology
On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows
May 7, 2026Xinglin Wang, Zishen Liu, Shaoxiong Feng +9Stochastic Optimization Cost-Aware Inference
School of Computer Science, Beijing Institute of Technology · Xiaohongshu Inc
BitCal-TTS: Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models
May 7, 2026Sai Babu Patarlapalli, Surya Teja AvvaruLLM Inference 4-Bit Quantization
Adaptive Test-Time Compute Allocation with Evolving In-Context Demonstrations
Apr 22, 2026Bowen Zuo, Dongruo Zhou, Yinglun ZhuEfficient Inference Test-Time Optimization
University of California, Riverside · 2Indiana University Bloomington
Weak-Link Optimization for Multi-Agent Reasoning and Collaboration
Apr 17, 2026Haoyu Bian, Chaoning Zhang, Jiaquan Zhang +4Multi-Agent Collaboration Inference-Time Compute Allocation
University of Electronic Science and Technology of China, Chengdu 611731, China · Beihang University, Beijing 100191, China · Xi’an University of Architecture and Technology, Xi’an 710064, China
Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization
Apr 16, 2026Zhiyuan Zhai, Bingcong Li, Bingnan Xiao +2Test-Time Scaling Constrained Optimization
Fudan University · ETH Zurich · Guangming Lab
Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents
Apr 9, 2026Khushal SethiInference-Time Compute Allocation LLM Inference Efficiency
Stanford University
Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agents
Apr 6, 2026Neharika Jali, Anupam Nayak, Gauri JoshiMulti-Turn Language Model Reasoning Efficient Language Model Reasoning
Carnegie Mellon University
CoT2-Meta: Budgeted Metacognitive Control for Test-Time Reasoning
Mar 30, 2026Siyuan Ma, Bo Gao, Zikai Xiao +6Efficient Language Model Reasoning Metacognition in Language Models
Nanyang Technological University · Carnegie Mellon University · Zhejiang University +5
Avoiding Overthinking and Underthinking: Curriculum-Aware Budget Scheduling for LLMs
Mar 29, 2026Amirul Rahman, Aisha Karim, Kenji Nakamura +1Efficient Language Model Reasoning Inference-Time Compute Allocation
University of Malaya
AgentCollab: A Self-Evaluation-Driven Collaboration Paradigm for Efficient LLM Agents
Mar 27, 2026Wenbo Gao, Renxi Liu, Xian Wang +8LLM Agents LLM Agent Workflow Optimization
Huawei · Hong Kong Polytechnic University
Compute Allocation for Reasoning-Intensive Retrieval Agents
Mar 15, 2026Sreeja Apparaju, Nilesh GuptaInference-Time Compute Allocation Information Retrieval
PETS: A Principled Framework Towards Optimal Trajectory Allocation for Efficient Test-Time Self-Consistency
Feb 18, 2026Zhangyi Liu, Huaizhi Qu, Xiaowei Yin +4Inference-Time Compute Allocation Self-Consistency in Language Models
Stanford University · UNC at Chapel Hill · Yale University +1
What If We Allocate Test-Time Compute Adaptively?
Feb 1, 2026Ahsan Bilal, Ahmed Mohsin, Muhammad Umer +4Inference-Time Search Test-Time Scaling
University of Oklahoma, OK, USA · Stanford University, CA, USA · Universitat Pompeu Fabra, Barcelona, Spain +2
More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
Jan 29, 2026Sagi Meir, Tommer D. Keidar, Noam Levi +2LLM Inference Efficiency LLM Evaluation
School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1
Optimal Self-Consistency for Efficient Reasoning with Large Language Models
Nov 15, 2025Austin Feng, Marius Alonso, Ambroise Odonnat +2Self-Consistency Decoding Inference-Time Scaling
1Yale University, New Haven, CT, USA · 2Noah’s Ark Lab, Paris, France · 3Inria, Rennes, France +1
Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression
Oct 2, 2025Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad +3RL for Language Model Reasoning LLM Inference Acceleration
UNC Chapel Hill · The University of Texas at Austin · Microsoft Research
TIP-Search: Time-Predictable Inference Scheduling for Market Prediction under Uncertain Load
May 30, 2025Xibai WangLimit Order Book Forecasting Online Convex Optimization
NeuroQuant Labs Limited Hong Kong SAR, China

May 11, 2026J/K move · Enter open · S save

Compute Where it Counts: Self Optimizing Language Models

Yash Akhauri, Mohamed S. Abdelfattah

Cornell University

LLM Inference Efficiency RL for Language Models LLM Pruning LLM Inference Scheduling Inference-Time Compute Allocation

Open paper AskPDF arXiv

Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attention), typically applying a uniform computation budget to every generated token. In practice, token difficulty varies widely, so static compression can over-compute on easy steps and under-compute on hard ones. We study dynamic budget allocation for autoregressive decoding: learning how much computation to spend per token from within a single model. Self-Optimizing Language Models (SOL) pair a frozen LLM with a lightweight policy network that reads the LLM hidden state and selects a discrete efficiency action at each decode step. Actions can jointly control (i) token-level attention sparsity, (ii) structured activation pruning in the MLP, and (iii) activation quantization bit-width, while leaving the base model weights unchanged. We train the policy with group-relative policy optimization on teacher-forced episodes: the token sequence is fixed, while we sample multiple compute schedules (i.e., "counterfactual" schedules that vary only the efficiency actions for the same token path) and compare their likelihoods under the same supervision. Our reward trades off language-model quality against soft penalties that encourage episode-average budget usage to match a requested target. Across model variants and compute regimes, SOL improves quality at matched budget over static allocation and strong random schedule search, offering a complementary axis for inference-efficiency optimization. SOL discovers a better quality-efficiency pareto-front across all our experiments and improves MMLU accuracy by up to 7.3% over uniform budget allocation strategies.