Overview

Cost-aware inference studies how to deliver useful predictions while limiting the resources spent on each request. Common approaches route easy cases to smaller models and escalate uncertain ones, adapt reasoning or verification effort to task difficulty and budget, and optimize caching, precision, batching, and hardware placement. Researchers evaluate these methods using quality-cost frontiers, latency and energy measurements, and deployment workloads, with growing attention to agents whose costs accumulate across many steps. A central challenge is that savings depend on workload and infrastructure: fewer tokens do not always mean lower billed cost, and routing gains can disappear when quality estimates, calibration, or operating conditions shift.

Latest papers

CardsList

308 papers24 papers in the last four weeks, down 49% on the four weeks before. 0.2% of all new papers.

  1. ADAPT: Adaptive Decision and Abstention with Pool-Size-Aware Thresholds for Service Routing

    Nov 19, 2026Manile Srun, Muntasir Adnan, Arslan Shaukat +1Adaptive Model RoutingSelective Prediction

  2. Reducing Inference Cost of Coding Agents with Timeout Cascades

    Nov 18, 2026Boxi Zhou, German Sviridov, Dmitrii Ustiugov +1AI Coding AgentsCost-Aware Inference

  3. Conformally Calibrated Adaptive Computation in Neural Operators

    Nov 17, 2026Abdul Qadir Ibrahim, Martin Burger, Abel Maviano JonathanAdaptive ComputationCost-Aware Inference

  4. Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

    Nov 16, 2026Ahmed Tariq, Leo GuinanLLM RoutingCost-Aware Inference

  5. When the Cheapest Feasible Router Fails: Auditing Deployment Commitments in Cost-Aware LLM Routing

    Nov 16, 2026Minxuan Hu, Qishi Zhan, Ziheng Chen +4LLM RoutingCost-Aware Inference

  6. Exact Sparse Assignment with Costly Target Embeddings: Fixed-Realization Certificates and an Empirical Audit

    Nov 15, 2026Hamidreza Hasani Balyani, Mahtab Movahhedrad, Seyed Pouyan Mousavi DavoudiCost-Aware Inference

  7. Smart Model Routing for Order-of-Magnitude LLM Cost Reduction

    Nov 11, 2026Hesham Elbakoury, Deep Shah, Ankur SharmaLLM RoutingCost-Aware Inference

  8. Prediction with Foundation Models under Deployment Budgets

    Nov 9, 2026Zhaohui Xu, Xiaofei Xu, Doudou ZhouCost-Aware InferenceEnsemble Learning

  9. Think or Remember: The Exchange Rate of Agentic Inference

    Nov 7, 2026Juan Wisznia, Santiago Luis Groba Alonso, Matías Gómez Taddeo +3Cost-Aware InferenceEfficient Language Model Inference

  10. Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference

    Nov 6, 2026Saul Urso, Emanuele Carlini, Patrizio Dazzi +1LLM RoutingCost-Aware Inference

  11. Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

    Nov 6, 2026Aristides Milios, Christopher PalLLM Output Length PredictionLLM Routing

  12. RouteAlign: Aligning Task Requirements with Model Profiles for LLM Service Selection

    Nov 5, 2026Jihao Zhao, Ding Chen, Yanfang Chen +4LLM RoutingLanguage Model Performance Prediction

  13. Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup

    Nov 5, 2026Saad Ahmed Bazaz, Talha Amjad, Abdurrehman Subhani +2Local LLM DeploymentCost-Aware Inference

  14. Minimum-Sufficient Knowledge and Computation for Wind Energy Predictive Analysis

    Nov 5, 2026Purbak Sengupta, Aparna S. varde, Emil Jørgensen NjorKnowledge-Guided MLCost-Aware Inference

  15. Latency-Aware Adaptive Best-of-NN

    Nov 5, 2026Alon Jacobson, Qian Xie, Peter I. FrazierInference-Time Compute AllocationCost-Aware Inference

  16. Pathwise-Budgeted LLM Routing: Online Allocation with K-Bit Tail-Cost Learning

    Nov 4, 2026Hanzhang Qin, Sophie H. Yu, Yuewei Zhang +1LLM RoutingCost-Aware Inference

  17. MetaPOCS: Meta-Learned Multi-Exit Unfolding for Budget-Aware Seismic Data Interpolation

    Nov 2, 2026Baisong Jiang, Chunxia Zhang, Xiaoli Wei +3Deep UnfoldingCost-Aware Inference

  18. Circumventing the Computational Hardness of Computational Arbitrage

    Oct 29, 2026Nastaran Okati, Ricardo Olmedo, Moritz HardtCost-Aware InferenceMulti-Agent Resource Allocation

  19. EvoRouter: Cost-Efficient Multi-Agent Routing via Insight-Guided On-Policy Distillation

    Oct 26, 2026Yitao Xiao, Xu Zhang, Ye Cheng +6LLM Agent RoutingAdaptive Model Routing

  20. The Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents

    Oct 26, 2026Fernando Vallecillos-Ruiz, Islem Bouzenia, Michael Pradel +1Context Compaction for LLM AgentsSoftware Engineering Agents

  21. What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

    Oct 25, 2026Kaiser Sun, Yongchan Kwon, Zain Hasan +3Inference-Time Compute AllocationLLM Agent Efficiency

  22. Markets of Minds: A 51-Billion-State Economy of Intelligence

    Oct 23, 2026Jusheng Zhang, Xuekun Wu, Wenhao Wang +2Multi-Agent Resource AllocationInference-Time Compute Allocation

  23. ArcFlow: Training Flow Policies for Multiple Solver Budgets

    Oct 20, 2026Hieu Tran, Hong YuFlow-Based Policy LearningRL for Flow Matching Models

  24. Which Model for Which Role? Cost-Efficient Pareto Recommendation for LLM Agentic Workflows

    Oct 19, 2026Qian Xie, Wenyue Hua, Yueli He +7LLM Agent Workflow OptimizationMulti-Objective Optimization

  25. FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

    Oct 17, 2026Dimitrios Liakopoulos, Prasoon Sinha, Neeraja J. YadwadkarLLM ServingCost-Aware Inference