Cost-Aware Inference

Ask about this topic

Overview

Cost-aware inference studies how to deliver useful predictions while limiting the resources spent on each request. Common approaches route easy cases to smaller models and escalate uncertain ones, adapt reasoning or verification effort to task difficulty and budget, and optimize caching, precision, batching, and hardware placement. Researchers evaluate these methods using quality-cost frontiers, latency and energy measurements, and deployment workloads, with growing attention to agents whose costs accumulate across many steps. A central challenge is that savings depend on workload and infrastructure: fewer tokens do not always mean lower billed cost, and routing gains can disappear when quality estimates, calibration, or operating conditions shift.

Latest papers

CardsList

282 papers48 papers in the last four weeks, up 78% on the four weeks before. 0.3% of all new papers.

  1. Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolution

    Apr 9, 2026Monishwaran Maheswaran, Leon Lakhani, Zhongzhu Zhou +16Inference-Time OptimizationEvolutionary Optimization

    UC Berkeley · Together AI · UT Austin +2

  2. Triage: Routing Software Engineering Tasks to Cost-Effective LLM Tiers via Code Quality Signals

    Apr 8, 2026Lech MadeyskiLLM RoutingCost-Aware Inference

    Wroclaw University of Science and Technology, Poland

  3. Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds

    Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM ServingLLM Inference Scheduling

    The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.

  4. Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation

    Apr 8, 2026Jagadeesh ChundruWeb AgentsLLM Agent Workflow Optimization

    Selfotix

  5. Explainable Model Routing for Agentic Workflows

    Apr 4, 2026Mika Okamoto, Ansel Kaplan Erol, Mark RiedlLLM Agent RoutingAgentic Workflows

    Georgia Institute of Technology, USA

  6. R2E-VID: Two-Stage Robust Routing via Temporal Gating for Elastic Edge-Cloud Video Inference

    Apr 3, 2026Zheming Yang, Lulu Zuo, Shun Lu +4Edge-Cloud ComputingAdaptive Model Routing

    Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Peng Cheng Laboratory +2

  7. Streaming Model Cascades for Semantic SQL

    Apr 1, 2026Paweł Liskowski, Kyle SchmausCost-Aware InferenceLLM Routing

    Snowflake Inc. · Poznań, Poland · San Francisco, CA, USA

  8. Reward-Based Online LLM Routing via NeuralUCB

    Mar 31, 2026Ming-Hua Tsai, Phat TranLLM RoutingAdaptive Model Routing

    Oregon State University

  9. Uncertainty Gating for Cost-Aware Explainable Artificial Intelligence

    Mar 31, 2026Georgii Mikriukov, Grégoire Montavon, Marina M. -C. HöhneExplainable Artificial IntelligenceExplainability Evaluation

    Leibniz Institute for Agricultural Engineering and Bioeconomy (ATB), Potsdam, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany · Charité – Universitätsmedizin Berlin, Berlin, Germany +1

  10. JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference

    Mar 31, 2026Bo Ma, Jinsong Wu, Weiqi Yan +2Cross-Modal Knowledge DistillationKnowledge Distillation

    J. Wu is with the Guilin University of Electronic Technology, Guilin, China. · H. Wei is with Hikvision Technology Co., Ltd, Hangzhou, China. · K. Liu is with the Hebei University of Technology, Tianjin 300401, China.

  11. LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis

    Mar 29, 2026Yen-Hsun Huang, Yu-Shiou LinSystematic Review AutomationAI-Assisted Scientific Research

    Department of Education, Taipei Veterans General Hospital, Taipei, Taiwan · Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan

  12. Overcoming the Incentive Collapse Paradox

    Mar 27, 2026Qichuan Yin, Ziwei Su, Shuangning LiHuman Oversight of AICost-Aware Inference

    Department of Statistics, University of Chicago · Booth School of Business, University of Chicago

  13. MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

    Mar 27, 2026Joris Köster, Zixuan Liu, Siavash Khajavi +1LLM Inference ServingCost-Aware Inference

    Department of Computer Science, Aalto University, Finland · Department of Computer Science, Tulane University, LA, USA · Department of Industrial Engineering and Management, Aalto University, Finland

  14. Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs

    Mar 26, 2026Sanjay MishraRetrieval-Augmented GenerationAdaptive RAG

    Raleigh, NC, USA

  15. Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints

    Mar 25, 2026Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3LLM RoutingLLM Inference Scheduling

    LinkedIn · MIT

  16. The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More

    Mar 25, 2026Lingjiao Chen, Chi Zhang, Yeye He +3Cost-Aware InferenceLarge Reasoning Models

    Stanford University · Microsoft Research · CMU +1

  17. Rewriting TTS Inference Economics: Lightning V2 on Tenstorrent Achieves 4x Lower Cost Than NVIDIA L40S

    Mar 24, 2026Ranjith M. S., Akshat Mandloi, Sudarshan KamathTTS SynthesisAI Accelerator Inference

    Smallest AI

  18. Computational Arbitrage in AI Model Markets

    Mar 23, 2026Ricardo Olmedo, Bernhard Schölkopf, Moritz HardtCost-Aware Inference

    Max Planck Institute for Intelligent Systems, Tübingen

  19. ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models

    Mar 22, 2026Haoyu Qiao, Hao Zhang, Shanwen Mao +2Edge-Cloud Collaborative InferenceLLM Routing

    the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China · the National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China · the Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen), ShenZhen 518055, China

  20. LLM Router: Rethinking Routing with Prefill Activations

    Mar 21, 2026Tanay Varshney, Annie Surla, Michelle Xu +5LLM RoutingAdaptive Model Routing

    NVIDIA

  21. 100x Cost & Latency Reduction: Performance Analysis of AI Query Approximation using Lightweight Proxy Models

    Mar 16, 2026Yeounoh Chung, Rushabh Desai, Jian He +9Cost-Aware Inference

    Google Cloud Sunnyvale, USA

  22. Compute Allocation for Reasoning-Intensive Retrieval Agents

    Mar 15, 2026Sreeja Apparaju, Nilesh GuptaInference-Time Compute AllocationInformation Retrieval

  23. Covariance-Guided Resource Adaptive Learning for Efficient Edge Inference

    Mar 15, 2026Ahmad N. L. Nabhaan, Zaki Sukma, Rakandhiya D. Rachmanto +4Power-Efficient InferenceInference-Time Optimization

    Universitas Amikom Yogyakarta, Department of Informatics · University of Georgia, School of Computing · Universitas Gadjah Mada, Department of Computer Science and Electronics +1

  24. Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference

    Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Inference EfficiencyLLM Routing

    vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University

  25. Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents

    Mar 13, 2026Yushu Li, Wenlong Deng, Jiajin Li +1Cost-Aware InferenceTree Search

    University of British Columbia · Vector Institute

  26. Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference

    Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM RoutingLLM Tool Use

    vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University

  27. CODA: Difficulty-Aware Compute Allocation for Adaptive Reasoning

    Mar 9, 2026Siye Wu, Jian Xie, Yikai Zhang +1Adaptive Reasoning in Language ModelsAdaptive Computation

    ♠Fudan University

  28. Did You Check the Right Pocket? Cost-Sensitive Store Routing for Memory-Augmented Agents

    Mar 8, 2026Madhava GaikwadMemory-Augmented AgentsAgent Memory Management

  29. Prompt Compression in Production Task Orchestration: A Pre-Registered Randomized Trial

    Mar 6, 2026Warren Johnson, Charles LeePrompt CompressionToken Compression

    Plexor Labs, Issaquah, WA, USA · Project Autobots, Seattle, WA, USA

  30. Online Learning for Multi-Layer Hierarchical Inference under Partial and Policy-Dependent Feedback

    Mar 4, 2026Haoran Zhang, Seohyeon Cha, Hasan Burhan Beytur +3Adaptive Model RoutingContextual Bandits

    Department of Electrical and Computer Engineering, The University of Texas at Austin · DEVCOM Army Research Laboratory