Cost-Aware Inference

Follow this topic Ask about this topic

Cost-aware inference studies how to deliver useful predictions while limiting the resources spent on each request. Common approaches route easy cases to smaller models and escalate uncertain ones, adapt reasoning or verification effort to task difficulty and budget, and optimize caching, precision, batching, and hardware placement. Researchers evaluate these methods using quality-cost frontiers, latency and energy measurements, and deployment workloads, with growing attention to agents whose costs accumulate across many steps. A central challenge is that savings depend on workload and infrastructure: fewer tokens do not always mean lower billed cost, and routing gains can disappear when quality estimates, calibration, or operating conditions shift.

Latest papers

282 papers48 papers in the last four weeks, up 78% on the four weeks before. 0.3% of all new papers.

Flexible Routing via Uncertainty Decomposition
May 8, 2026Charlotte Peale, Siddartha Devic, Parikshit Gopalan +2Uncertainty Quantification Adaptive Model Routing
1Stanford University · University of Southern California · 3Apple
PaT: Planning-after-Trial for Efficient Test-Time Code Generation
May 8, 2026Youngsik Yoon, Sungjae Lee, Seockbean Song +3Test-Time Optimization Execution-Guided Code Generation
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, Beijing, China
Switchcraft: AI Model Router for Agentic Tool Calling
May 8, 2026Sharad Agarwal, Pooria Namyar, Alec Wolman +3Tool-Augmented Language Model Agents Cost-Aware Inference
Microsoft Research · Microsoft · Stanford
Is Escalation Worth It? On the Depth of LLM Cascades
May 7, 2026Dylan BouchardCost-Aware Inference LLM Routing
Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
May 7, 2026Wenwen Si, Insup Lee, Osbert BastaniCost-Aware Inference Efficient Language Model Reasoning
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA
On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows
May 7, 2026Xinglin Wang, Zishen Liu, Shaoxiong Feng +9Stochastic Optimization Cost-Aware Inference
School of Computer Science, Beijing Institute of Technology · Xiaohongshu Inc
Inference-Time Budget Control for LLM Search Agents
May 7, 2026Zhengru Fang, Senkang Forest Hu, Zhonghao Chang +6Multi-Hop QA Inference-Time Search
City University of Hong Kong · Tsinghua University · Alibaba Ant Group
Budgeted Attention Allocation: Cost-Conditioned Compute Control for Efficient Transformers
May 7, 2026Amrit NidhiEfficient Transformer Inference Self-Attention
Independent Researcher
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM Serving Cost-Aware Inference
Carnegie Mellon University · Peking University
Joint Energy Management and Coordinated AIGC Workload Scheduling for Distributed Data Centers: A Diffusion-Aided Reward Shaping Approach
May 3, 2026Yang Fu, Peng Qin, Liming Chen +3Reward Shaping Data Center Energy Management
Yang Fu, Peng Qin, Member, IEEE, Liming Chen, Zihao Zhang, Hao Yu, and Yifei Wang
Multi-Agent Reasoning Improves Compute Efficiency: Pareto-Optimal Test-Time Scaling
May 2, 2026Florian Valentin Wunderlich, Lars Benedikt Kaesberg, Jan Philip Wahle +2Self-Consistency Decoding Cost-Aware Inference
University of Göttingen, Germany
Intelligent Elastic Feature Fading: Enabling Model Retrain-Free Feature Efficiency Rollouts at Scale
May 1, 2026Jieming Di, Xiaoyu Chen, Ying She +21Efficient Inference Cost-Aware Inference
Meta · USA
Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference
May 1, 2026Yuxuan Gao, Megan Wang, Yi Ling YuLLM Evaluation Energy-Efficient ML
University of Pennsylvania · OpenMesh AI · Columbia University
Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving
Apr 30, 2026Junsun Choi, Sam Son, Sunjin Choi +5LLM Serving Cost-Aware Inference
UC Berkeley · UC Berkeley & ICSI
AI Inference as Relocatable Electricity Demand: A Latency-Constrained Energy-Geography Framework
Apr 30, 2026Xubin Luo, Cheng YangEnergy-Efficient ML Cost-Aware Inference
School of Computer Science and Artificial Intelligence, Southwestern University of Finance and Economics, Chengdu, China
Belief-Guided Inference Control for Large Language Model Services via Verifiable Observations
Apr 30, 2026Wenhao Yuan, Chenchen Lin, Jian Chen +3LLM Inference Cost-Aware Inference
The University of Hong Kong · Sun Yat-sen University
CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
Apr 28, 2026Wei-Chun Chen, Yu-Xuan Chen, I-Fang Chung +1LLM Evaluation Cost-Aware Inference
Institute of Biomedical Informatics, National Yang Ming Chiao Tung University, Taipei, Taiwan · Institute of Health Data Science, Chang Gung University, Taoyuan, Taiwan · Department of Artificial Intelligence, Chang Gung University, Taoyuan, Taiwan +1
RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Serving Cost-Aware Inference
1The University of Hong Kong · 2Stellaris AI Limited
MTRouter: Cost-Aware Multi-Turn LLM Routing with History-Model Joint Embeddings
Apr 26, 2026Yiqun Zhang, Hao Li, Zihan Wang +6Cost-Aware Inference LLM Routing
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China · Shanghai Artificial Intelligence Laboratory
How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
Apr 24, 2026Longju Bai, Zhemin Huang, Xingyao Wang +5Coding Agents Cost-Aware Inference
University of Michigan · Stanford University · Microsoft AI +3
QuantClaw: Precision Where It Matters for OpenClaw
Apr 24, 2026Manyi Zhang, Ji-Fu Li, Zhongao Sun +5Mixed-Precision Quantization LLM Quantization
Huawei Technologies · National University of Singapore · University of Science and Technology of China
Toward Reliable Design of LLM-Enabled Agentic Workflows: Optimizing Latency-Reliability-Cost Tradeoffs
Apr 21, 2026Ya-Ting Yang, Quanyan ZhuCost-Aware Inference Agentic Workflows
New York University Brooklyn, NY, USA
Continuous Semantic Caching for Low-Cost LLM Serving
Apr 21, 2026Baran Atalar, Xutong Liu, Jinhang Zuo +3LLM Serving Cost-Aware Inference
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · University of Washington Tacoma Tacoma, Washington, USA · City University of Hong Kong China +1
Are Large Language Models Economically Viable for Industry Deployment?
Apr 21, 2026Abdullah Mohammad, Sushant Kumar Ray, Pushkar Arora +5LLM Inference Efficiency LLM Evaluation
DSEU-Okhla, New Delhi, India · University of Delhi, New Delhi, India · 6, 8Macquarie University, Sydney, Australia +1
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
When Spike Sparsity Does Not Translate to Deployed Cost: VS-WNO on Jetson Orin Nano
Apr 18, 2026Jason Yoo, Shailesh Garg, Souvik Chakraborty +1Efficient Neural Network Inference Activation Sparsity
University of Illinois Urbana-Champaign Department of Nuclear, Plasma & Radiological Engineering Urbana, IL, USA · Indian Institute of Technology Delhi Department of Applied Mechanics New Delhi, India · National Center for Supercomputing Applications Urbana, IL, USA
Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization
Apr 16, 2026Haochun Tang, Yuliang Yan, Jiahua Lu +2Adversarial Prompt Generation Cost-Aware Inference
1Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · 2The Hong Kong University of Science and Technology (Guangzhou)
PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Search
Apr 15, 2026Tianhao Qian, Jiayu Chen, Zhenyu Sun +1LLM Inference Efficiency Inference-Time Search
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · TikTok
Observe Less, Understand More: Cost-aware Cross-scale Observation for Remote Sensing Understanding
Apr 13, 2026Zhenghao Xie, Jing Xiao, Zhenqi Wang +4Remote Sensing Image Understanding Cost-Aware Inference
School of Artificial Intelligence, Wuhan University, Wuhan, China · Hangzhou Institute of Technology, Xidian University, Hangzhou, China · Hubei Luojia Laboratory, Wuhan University, Wuhan, China
ExecTune: Effective Steering of Black-Box LLMs with Guide Models
Apr 9, 2026Vijay Lingam, Aditya Golatkar, Anwesan Pal +6Language Model Steering LLM Inference Efficiency
AWS AI

May 8, 2026J/K move · Enter open · S save

Flexible Routing via Uncertainty Decomposition

Charlotte Peale, Siddartha Devic, Parikshit Gopalan, Udi Wieder+1

1Stanford University · University of Southern California · 3Apple

Uncertainty Quantification Adaptive Model Routing Selective Prediction Cost-Aware Inference Classification Uncertainty Decomposition

Open paper AskPDF arXiv

A key strategy for balancing performance and cost in modern machine learning systems is to dynamically route queries to either a low-cost model or a more expensive oracle (such as a large pretrained model or human expert), an approach known as model routing. In this work we present a new uncertainty-aware router that (1) avoids unnecessary oracle calls on inherently ambiguous queries, and (2) adapts dynamically to different loss functions and cost parameters through simple hyperparameter changes, without retraining. Our method, applicable to any classification setting where multiple independent annotations per input are available, is based on decomposing total uncertainty into irreducible and reducible components using higher-order predictors [Ahdritz et al., 2025]. This enables a unified approach to both routing and abstention: predict with the weak model when uncertainty is low, route to the oracle when reducible uncertainty is high, and abstain when irreducible uncertainty is high. Our router comes with strong theoretical guarantees bounding regret relative to optimal task-specific routers. We conduct experiments on both synthetic and real-world datasets that demonstrate the benefits of our approach in suitable regimes -- in particular, whenever reducible and irreducible uncertainty are not too correlated.