Cost-Aware Inference

Ask about this topic

Overview

Cost-aware inference studies how to deliver useful predictions while limiting the resources spent on each request. Common approaches route easy cases to smaller models and escalate uncertain ones, adapt reasoning or verification effort to task difficulty and budget, and optimize caching, precision, batching, and hardware placement. Researchers evaluate these methods using quality-cost frontiers, latency and energy measurements, and deployment workloads, with growing attention to agents whose costs accumulate across many steps. A central challenge is that savings depend on workload and infrastructure: fewer tokens do not always mean lower billed cost, and routing gains can disappear when quality estimates, calibration, or operating conditions shift.

Latest papers

CardsList

282 papers48 papers in the last four weeks, up 78% on the four weeks before. 0.3% of all new papers.

  1. EvoRouter: Cost-Efficient Multi-Agent Routing via Insight-Guided On-Policy Distillation

    Oct 26, 2026Yitao Xiao, Xu Zhang, Ye Cheng +6LLM Agent RoutingAdaptive Model Routing

    Beijing University of Posts and Telecommunications, Beijing, China · Fudan University, Shanghai Innovation Institute, Shanghai, China · Nanyang Technological University, Singapore

  2. The Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents

    Oct 26, 2026Fernando Vallecillos-Ruiz, Islem Bouzenia, Michael Pradel +1Context Compaction for LLM AgentsSoftware Engineering Agents

    Simula Research Laboratory, Norway · CISPA Helmholtz Center for Information Security, Germany

  3. What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

    Oct 25, 2026Kaiser Sun, Yongchan Kwon, Zain Hasan +3Inference-Time Compute AllocationLLM Agent Efficiency

    Johns Hopkins University · Together AI · Cornell University

  4. Markets of Minds: A 51-Billion-State Economy of Intelligence

    Oct 23, 2026Jusheng Zhang, Xuekun Wu, Wenhao Wang +2Multi-Agent Resource AllocationInference-Time Compute Allocation

    Stanford University · Vast Intelligence Lab · University of California at Merced

  5. ArcFlow: Training Flow Policies for Multiple Solver Budgets

    Oct 20, 2026Hieu Tran, Hong YuFlow-Based Policy LearningRL for Flow Matching Models

    University of Massachusetts Amherst · University of Massachusetts Lowell

  6. RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting

    Oct 20, 2026Injae Kim, Minseok Joo, Jihwan Park +33D Gaussian SplattingFeed-Forward 3D Reconstruction

    KAIST · Korea University

  7. Which Model for Which Role? Cost-Efficient Pareto Recommendation for LLM Agentic Workflows

    Oct 19, 2026Qian Xie, Wenyue Hua, Yueli He +7LLM Agent Workflow OptimizationMulti-Objective Optimization

    Columbia University · Cornell University · Microsoft +2

  8. FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

    Oct 17, 2026Dimitrios Liakopoulos, Prasoon Sinha, Neeraja J. YadwadkarLLM ServingCost-Aware Inference

    The University of Texas at Austin

  9. Toward Collective Model Intelligence: Characterizing and Coordinating Models

    Oct 17, 2026SinapisAI Team, Jian-Dong Liu, Qun Dai +27Cost-Aware Inference

    SinapisAI

  10. Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets

    Oct 17, 2026Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra +10LLM ServingLLM Inference Scheduling

    Tandemn · University of Illinois Urbana-Champaign · UC Berkeley +3

  11. Response-Informed Value Estimation for Cost-Aware Multi-Turn LLM Routing

    Oct 16, 2026Haotian Yan, Jiaqi Liu, Bingyi Jing +1LLM RoutingCost-Aware Inference

    Southwest University of Finance and Economics · Shenzhen Loop Area Institute · Dalian University of Technology +1

  12. Chance-Constrained Personalized LLM Routing with Online Preference Learning

    Oct 15, 2026Ameya Anjarlekar, Rasoul Etesami, R SrikantLLM RoutingCost-Aware Inference

  13. Confidence-Calibrated Inference Expansion for Evaluator-Guided Test-Time Reasoning

    Oct 14, 2026Weida Liang, Kenji KawaguchiEfficient Language Model ReasoningInference-Time Compute Allocation

    School of Computing, National University of Singapore

  14. ActiveEvidence: Decision-Focused Evidence Graphs with Verifiable Acquisition Gaps

    Oct 14, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +4Evidence SelectionCost-Aware Inference

    School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

  15. Truthful and Expected-Payment-Optimal Model Routing in Graph-Based Agentic Workflows

    Oct 13, 2026Jiayuan Liu, Mingyu Guo, Jiarui Gan +1LLM Agent RoutingMechanism Design

    Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · Adelaide University +1

  16. Whose turn is it? Context-guided model routing for stateful agents

    Oct 13, 2026Anikesh Parashar, Avik Dutta, Vashu Chauhan +6LLM RoutingLLM Agent Routing

    Microsoft

  17. Dynamic K-Head: Difficulty-Supervised Budget Selection for Continuous Latent Reasoning

    Oct 13, 2026Ying Li, Yuqi Chen, Sixuan Li +1Efficient Language Model ReasoningInference-Time Compute Allocation

    Beijing Institute of Mathematical Sciences and Applications, Beijing, China · Tsinghua University, Beijing, China · The University of Hong Kong, Hong Kong, China +1

  18. EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud Delegation

    Oct 13, 2026Zeru Shi, Jindong Jiang, Ryo Hachiuma +10Edge-Cloud Collaborative InferenceLLM Agents

    NVIDIA · Rutgers University

  19. Scaling Inference-Time Compute with Teams of Coding Agents

    Oct 12, 2026Keenan Samway, Daniel D'souza, Omar Darwiche Domingues +3Inference-Time ScalingInference-Time Compute Allocation

    Cohere Labs · Cohere

  20. Handoff Curves: Initial Selection, Later Handoff, and Supervision Targets in Multi-turn LLM Routing

    Oct 11, 2026Daniel Matos, Lu Liu, Yanfeng ZhangAdaptive Model RoutingLLM Routing

    Riemann Lab, Huawei Technologies

  21. The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

    Oct 11, 2026Mukund Athmaram, Manoj Rao, Liang MaLLM ServingLLM Inference Acceleration

    Advanced Micro Devices Inc. (AMD) · Cornell University

  22. TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents

    Oct 10, 2026Pramod Kumar, Raj Kumar PandeyLLM Agent Workflow OptimizationCoding Agents

    The Pennsylvania State University, University Park, USA

  23. Ask Your Advisor: Learning When to Consult Stronger Models in Agentic Systems

    Oct 9, 2026Pascal Sowietzki, Steffen Holter, Yannick Metz +3Agentic WorkflowsCost-Aware Inference

    ETH Zurich, Switzerland

  24. Cost-Aware Mixture-of-Experts Coordination for Model Markets

    Oct 8, 2026Yizhou Ma, Wenbo Wu, Xikun Jiang +2Mixture-of-Experts ModelsExpert Routing

    University of Southampton Southampton, UK · Aalborg University Copenhagen, Denmark · University of Nottingham Ningbo, China

  25. Performance at What Cost? A Sustainability-Aware Performance Index for Cell and Nucleus Instance Segmentation

    Oct 7, 2026Eiram Mahera Sheikh, Alaa Tharwat, Wolfram SchenckInstance SegmentationCost-Aware Inference

    Department of Engineering and Mathematics, Bielefeld University of Applied Sciences and Arts, Interaktion 1, 33619 Bielefeld

  26. From Pareto to Preference: Personalized Test-Time Scaling via Amortized Agentic Policy Discovery

    Oct 7, 2026Xinglin Wang, Zishen Liu, Tong Zheng +9Test-Time ScalingCost-Aware Inference

    Beijing Institute of Technology · Xiaohongshu Inc

  27. Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

    Oct 6, 2026Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein +2LLM Agent EfficiencyCost-Aware Inference

    University of Tübingen · ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · École Polytechnique, Institut Polytechnique de Paris, CNRS +1

  28. When Plans Change Answers: Formalizing Cost-Accuracy Optimization for Semantic Queries

    Oct 6, 2026Kyoungmin KimCost-Aware Inference

    KAIST

  29. Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

    Oct 5, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidLLM Inference EfficiencyDisaggregated LLM Serving

    Citigroup Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom

  30. Small Language Models for Smart Data Model Classification at the Edge: A Cost-Aware Hybrid Approach

    Oct 5, 2026Cristian Martella, Angelo Martella, Antonella Longo +1Small Language ModelsSemantic Mapping

    Dept. of Engineering for Innovation, University of Salento, Via Per Monteroni, Lecce, 73100, Italy