Cost-Aware Inference

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 183

All topics

AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search
Jun 24, 2026Md Omar Faruk Rokon, Shasvat Desai, Hong Yao +1Online Advertising Cost-Aware Inference
Walmart Global Tech, Sunnyvale, CA, USA
Cost-Optimal Decision Diagrams for Stochastic Boolean Function Evaluation
Jun 23, 2026Xia Zong, Tuomo Lehtonen, Jussi RintanenBranch and Bound Cost-Aware Inference
Department of Computer Science, Aalto University
Token-Operations-Oriented Inference Optimization Techniques for Large Models
Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM Serving Inference-Time Optimization
Unicom Data Intelligence, China Unicom · Data Science and Artificial Intelligence Research Institute, China Unicom · Tsinghua University +7
Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning
Jun 18, 2026Sajib Acharjee Dip, Dawei Zhou, Liqing ZhangLLM Inference Efficiency Cost-Aware Inference
Department of Computer Science, Virginia Tech · Fralin Biomedical Research Institute, Virginia Tech · FBRI Cancer Research Center, Washington, DC
Closing the Operational Gap in Semantic Caching
Jun 18, 2026Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev +2LLM Evaluation Cost-Aware Inference
New York University · Redis
RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing
Jun 17, 2026Guannan Lai, Haoran Hu, Han-Jia YeLLM Evaluation Pairwise Preference Evaluation
School of Artificial Intelligence, Nanjing University, China · 2National Key Laboratory for Novel Software Technology, Nanjing University, 210023, China · 3SinapisAI
FairTutor: Equity-Aware Pedagogical LLM Routing for Budget-Constrained AI Tutoring
Jun 16, 2026Qingyang XuIntelligent Tutoring Systems AI in Education
Independent researcher, Shanghai, China
Online LLM Selection via Constrained Bandits with Time-Varying Demand
Jun 16, 2026Yin Huang, Qingsong Liu, Jie XuMulti-Armed Bandits Cost-Aware Inference
Department of Electrical and Computer Engineering, University of Florida · Manning College of Information and Computer Sciences, University of Massachusetts Amherst
Green SARC: Predictive Cost and Carbon Governance for Agentic AI Systems
Jun 14, 2026Gaston BesansonEnergy-Efficient ML Cost-Aware Inference
Universidad Torcuato Di Tella
ConCise: Training-Free Conclusion-Chain State Compression for Cost-Efficient Multi-Step RAG Services
Jun 13, 2026Kuan Yan, Zhiqing Tang, Tian Wang +1Retrieval-Augmented Generation Cost-Aware Inference
Faculty of Arts and Sciences, Beijing Normal University, Zhuhai, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · Guangdong Key Lab of AI & Multi-Modal Data Processing, Beijing Normal-Hong Kong Baptist University, Zhuhai, China
Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
Jun 12, 2026Sina Hajimiri, Masih Aminbeidokhti, Jose Dolz +4Web Agent Benchmarks LLM Agent Evaluation
ServiceNow AI Research · ÉTS Montreal · University of British Columbia +1
PLAIground: SLO-Driven Runtime Model Selection for Compound AI Systems in the Edge-Cloud-Space Continuum
Jun 12, 2026Milos Gravara, Cynthia Marcelino, Andrija Stanisic +1Cost-Aware Inference Edge-Cloud Computing
Distributed Systems Group TU Wien
Design Methodology and Performance Trade-offs Management for Distributed and Compound AI Systems
Jun 12, 2026Milos Gravara, Andrija Stanisic, Stefan NasticCost-Aware Inference
Distributed Systems Group TU Wien
Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees
Jun 12, 2026Herbert Woisetschläger, Arastun Mammadli, Ryan Zhang +1Cost-Aware Inference LLM Routing
Technical University of Munich Germany · University of Exeter United Kingdom · Horace Greeley High School United States
Can I Buy Your KV Cache?
Jun 11, 2026Luoyuan ZhangLLM Inference Efficiency KV Caching
Harbin Institute of Technology, Shenzhen (HITSZ)
Brick: Spatial Capability Routing for the Mixture-of-Models (MoM) Paradigm
Jun 11, 2026Francesco Massa, Marco CristofanilliAdaptive Model Routing Cost-Aware Inference
External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs
Jun 10, 2026Lin Sun, Heming Zhang, Xiangzheng ZhangLLM Serving Knowledge Augmentation for Language Models
Qiyuan Tech
AI Tokenomics: The Economics of Tokens, Computation, and Pricing in Foundation Models
Jun 10, 2026Quanyan ZhuCost-Aware Inference Token Efficiency
Department of Electrical and Computer Engineering, New York University Tandon School of Engineering, 370 Jay Street, Room 1004, Brooklyn, NY 11201, USA
Belief-Space Control for Personalized Cancer Treatment via Active Inference
Jun 9, 2026Deniz Sargun, H. Bugra Tulay, C. Emre KoksalBelief-Space Planning Healthcare
Amazon.com Inc. Palo Alto, CA · HP Inc. Denver, CO · The Ohio State University Columbus, OH
BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference
Jun 8, 2026Yuhua Zhou, Shaoqi Yu, Shichao Weng +4LLM Pruning Cost-Aware Inference
College of Computer Science and Technology, Zhejiang University · Zhejiang Lab · Ant Group +1
Bayesian Selective Latent Inference for Wastewater-First Influenza Monitoring
Jun 8, 2026Yixuan Zhang, Yang Song, Hao Wang +2Epidemic Forecasting Selective Prediction
Section of Health Data Science and AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark · Section of Health Data Science & AI, Department of Public Health, University of Copenhagen, Copenhagen, Denmark · Rutgers University, New Brunswick, NJ, USA +1
FAME: Forecastability-Aware Mixture of Experts for Heterogeneous Time Series Forecasting
Jun 8, 2026Qianyang Li, Xingjun Zhang, Shaoxun Wang +2Sparse Mixture-of-Experts Time Series Forecasting
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Department of Research and Development, Shandong New Beiyang Information Technology Co., Ltd., WeiHai 264200, China · Department of Computer Science and Technology, Tsinghua University, Beijing 100084
Larch: Learned Query Optimization for Semantic Predicates
Jun 6, 2026Fuheng Zhao, Pawel Liskowski, Zihan Li +5Cost-Aware Inference
Snowflake Inc. USA · Snowflake Inc. Poland
Cost-Aware Speculative Execution for LLM-Agent Workflows: An Integrated Five-Dimension Method
Jun 5, 2026Faisal FareedLLM Agent Orchestration Cost-Aware Inference
AWS
Learning to Route LLMs from Implicit Cost-Performance Preferences via Meta-Learning
Jun 4, 2026Jiahao Zeng, Ming Tang, Ningning DingMeta-Learning Cost-Aware Inference
Hong Kong University of Science and Technology (Guangzhou) · Southern University of Science and Technology
CRAFT: Cost-aware Refinement And Front-aware Tuning of Prompts
Jun 3, 2026Shanu Kumar, Shubhanshu Khandelwal, Akhila Yesantarao Venkata +3LLM Prompting Cost-Aware Inference
MBZUAI · Microsoft
Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation
Jun 3, 2026Liang He, Jingbo Wen, Haoyu Wang +4Cost-Aware Inference Inference-Time Compute Allocation
Shanghai Institute of Optics and Fine Mechanics · The University of Sydney
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Jun 2, 2026Mubarak Adetunji OjewaleLLM Inference KV Caching
Cloud Competency Centre, National College of Ireland
Resource-Constrained Adaptive Inference for Sequential Pricing
Jun 2, 2026Ruicheng Ao, Jiashuo Jiang, David Simchi-LeviAdaptive Inference Cost-Aware Inference
Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Hong Kong · Department of Civil and Environmental Engineering and Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139
The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs
Jun 2, 2026Xu Wan, Speed Zhu, Jianwei Cai +4Cost-Aware Inference Inference-Time Scaling
1Zhejiang University · 2Tecent HY Team · 3Peking University

Jun 24, 2026, cs.IRJ/K move · Enter open · S save

AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search

Md Omar Faruk Rokon, Shasvat Desai, Hong Yao, Kuang-chih Lee

Walmart Global Tech, Sunnyvale, CA, USA

Online Advertising Cost-Aware Inference Information Retrieval

Open paper AskPDF arXiv

How can we generate high-quality relevance annotations at scale without the cost and delays of human labeling? Relevance annotations are the backbone of search ranking systems which is needed for training data preparation, NDCG evaluation, and root cause analysis. However, human annotation is slow and off-the-shelf LLMs suffer from accuracy on domain-specific tasks. We propose a calibrated model cascade, a systematic approach for cost-efficient offline relevance annotation by routing queries through progressively larger fine-tuned classifiers. Our central insight is that accuracy and cost are orthogonal optimizations: domain-specific fine-tuning drives accuracy, cascading drives cost, and per-class isotonic calibration adds a small but reliable gain on top. Our contribution is threefold: (a) we decompose the gains and show that fine-tuning contributes 20 accuracy points while cascading is approximately accuracy-neutral but halves compute cost, (b) we introduce per-class isotonic calibration as one component of the cascade, contributing a small but statistically significant gain (+0.6 points over the strongest calibration baseline), and (c) we validate the system in production across six offline use cases, processing 150M+ annotations and enabling faster experimentation cycles. Our work is a building block for scalable, high-quality offline annotation pipelines in search and advertising systems.

Momentum

Related topics

Latest papers 183