Cost-Aware Inference

Follow this topic Ask about this topic

Cost-aware inference studies how to deliver useful predictions while limiting the resources spent on each request. Common approaches route easy cases to smaller models and escalate uncertain ones, adapt reasoning or verification effort to task difficulty and budget, and optimize caching, precision, batching, and hardware placement. Researchers evaluate these methods using quality-cost frontiers, latency and energy measurements, and deployment workloads, with growing attention to agents whose costs accumulate across many steps. A central challenge is that savings depend on workload and infrastructure: fewer tokens do not always mean lower billed cost, and routing gains can disappear when quality estimates, calibration, or operating conditions shift.

Latest papers

282 papers48 papers in the last four weeks, up 78% on the four weeks before. 0.3% of all new papers.

Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference
Feb 25, 2026Bo-Wei Chen, Chung-Chi Chen, An-Zi YenCost-Aware Inference LLM Routing
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Artificial Intelligence Research Center, AIST, Japan
An Agentic System for Schema Aware NL2SQL Generation
Feb 25, 2026David Onyango, Naseef MansoorText-to-SQL Cost-Aware Inference
Project Path, Boston, MA, USA · Minnesota State University Mankato, MN, USA
Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Feb 23, 2026Arindam KhaledLLM Routing Cost-Aware Inference
Independent Researcher
GaiaFlow: Semantic-Guided Diffusion Tuning for Carbon-Frugal Search
Feb 17, 2026Rong Fu, Jia Yee Tan, Chunlei Meng +4Information Retrieval Energy-Efficient ML
Independent Researcher
PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents
Feb 2, 2026Yuxuan Fu, Xiaoyu Tan, Teqi Hao +2Proactive Agents Cost-Aware Inference
Shanghai University of Engineering Science · National University of Singapore
Test-Time Compute Games
Jan 29, 2026Ander Artola Velasco, Dimitrios Rontogiannis, Stratis Tsirtsis +1LLM Serving Cost-Aware Inference
Max Planck Institute for Software Systems, Kaiserslautern, Germany · Hasso Plattner Institute, Potsdam, Germany
More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
Jan 29, 2026Sagi Meir, Tommer D. Keidar, Noam Levi +2LLM Inference Efficiency LLM Evaluation
School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1
ROI-Reasoning: Rational Optimization for Inference via Pre-Computation Meta-Cognition
Jan 7, 2026Muyang Zhao, Chunhao Tian, Qi Qi +1Cost-Aware Inference Metacognition in Language Models
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
Dec 5, 2025Nils Strassenburg, Boris Glavic, Tilmann RablCost-Aware Inference Small Language Models
Hasso Plattner Institute, Uni Potsdam · University of Illinois Chicago
e1: Learning Adaptive Control of Reasoning Effort
Oct 30, 2025Michael Kleinman, Matthew Trager, Alessandro Achille +2Efficient Language Model Reasoning Cost-Aware Inference
AWS Agentic AI
PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution
Jul 2, 2025Omkar Shende, Gayathri Ananthanarayanan, Marcello TraiolaEfficient Neural Network Inference Adaptive Inference
Department of Computer Science and Engineering, Indian Institute of Technology Dharwad, Karnataka, India · University of Rennes, CNRS, Inria, IRISA, Rennes, France
Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
Mar 31, 2025Rui Wang, Hongru Wang, Boyang Xue +9LLM Inference Efficiency Cost-Aware Inference
University of Edinburgh, Edinburgh, UK · University of Macau, Macau SAR, China · University of Illinois Urbana-Champaign, USA

Feb 25, 2026J/K move · Enter open · S save

Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference

Bo-Wei Chen, Chung-Chi Chen, An-Zi Yen

Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Artificial Intelligence Research Center, AIST, Japan

Cost-Aware Inference LLM Routing LLM Inference Acceleration

Open paper AskPDF arXiv

Large Language Models (LLMs) have revolutionized inference across diverse natural language tasks, with larger models performing better but at higher computational costs. We propose a confidence-driven strategy that dynamically selects the most suitable model based on confidence estimates. By assessing a model's confidence in handling the task and response accuracy, tasks that are likely to be solved correctly are retained, while more uncertain or complex cases are delegated to a larger model, ensuring reliability while minimizing computation. Specifically, we evaluate a model's likelihood of knowing the correct answer and the probability that its response is accurate. Experiments on the Massive Multitask Language Understanding (MMLU) benchmark show that our approach achieves accuracy comparable to the largest model while reducing computational costs by 20% to 40%. When applied to GPT-4o API calls, it reduces token usage by approximately 60%, further improving cost efficiency. These findings indicate the potential of confidence-based model selection to enhance real-world LLM deployment, particularly in resource-constrained settings such as edge devices and commercial API applications.