Cost-Aware Inference

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 183

All topics

Cost-Aware Diffusion Draft Trees for Speculative Decoding
Jun 1, 2026Shuai Zhang, Huachuan Qiu, Hongliang He +1Cost-Aware Inference Speculative Decoding for Diffusion Language Models
1Zhejiang University · 2Westlake University
UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
May 29, 2026Kaiyu Huang, Xingyu Wang, Mingze Kong +6Cost-Aware Inference Inference-Time Scaling
School of Computer Science and Technology, Tongji University · Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen · School of Data Science, The Chinese University of Hong Kong, Shenzhen +2
When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systems
May 28, 2026Corrado Rainone, Davide Belli, Bence Major +1Multi-Agent LLM Systems On-Device Language Model Inference
Qualcomm Technologies, Inc. 1Qualcomm AI Research†
Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective
May 28, 2026Shenghao Ye, Yuxiang Wang, Yu Guo +3Cost-Aware Inference LLM Uncertainty Estimation
University of Science and Technology of China · The University of Melbourne · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Optimality of Sequential Filtering Under Independent Cost and Selectivity Models
May 28, 2026Hrishikesh Paranjape, Abhishek Mandal, Xian SunCost-Aware Inference
Meta, USA
Natural Language Query to Configuration for Retrieval Agents
May 26, 2026Melissa Z. Pan, Negar Arabzadeh, Mathew Jacob +3Retrieval-Augmented Generation Cost-Aware Inference
UC Berkeley · University of Washington · Microsoft Azure Research - Systems
Resource-Constrained Affect Modelling via Variance Regularisation Pruning
May 26, 2026Kosmas Pinitas, Konstantinos KatsifisCost-Aware Inference Affective Computing
Mediterranean College, Athens, Greece · University of Derby, Derby, UK
SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning
May 25, 2026Zhongling Xu, Shunan Zheng, Wei WangCost-Aware Inference Offline RL
Department of Operations Research and Industrial Engineering The University of Texas at Austin Austin, TX 78712
Context Features Are Cheap: Rank-Aware Decomposition for Efficient Feature Interaction in Recommender Systems
May 24, 2026Yevgeny TkachFeature Interaction Modeling Cost-Aware Inference
Taboola · Israel
Selective Test-Time Compute Scaling for Click-Through Rate Prediction via Uncertainty-Triggered Feature Path Exploration
May 24, 2026Moyu Zhang, Yun Chen, Yujun Jin +3Cost-Aware Inference Test-Time Scaling
Alibaba Group Beijing, China
Assessing the Operational Viability of Foundation Models for Time Series Forecasting
May 23, 2026Kavin Soni, Debanshu Das, Vamshi GuduguntlaZero-Shot Time Series Forecasting Time Series Forecasting
Google, USA
Neural Bayesian Sequential Routing
May 22, 2026Yongchao HuangUncertainty Quantification Adaptive Model Routing
The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management
May 21, 2026Binqi Shen, Lier Jin, Hanyu Cai +2LLM Evaluation Cost-Aware Inference
Department of Industrial Engineering and Management Sciences Northwestern University Evanston, USA · Fuqua School of Business Duke University Durham, USA · Department of Engineering Carnegie Mellon University Pittsburgh, USA +1
Argo: Efficient Importance Labeling for Enterprise Email Systems
May 20, 2026Siddhant Ray, Ganesh Ananthanarayanan, Kevin Chian +5LLM Inference Efficiency LLM Serving
University of Chicago · Microsoft · University of Chicago / TensorMesh
ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestration
May 19, 2026May Hamri, Inbal Talgam-CohenLLM Agent Orchestration Cost-Aware Inference
1Tel Aviv University
A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition
May 18, 2026Edwin Arkel Rios, Augusto Christian Surya, Oswin Gosal +5Data Augmentation Cost-Aware Inference
National Yang Ming Chiao Tung University, Taiwan · National Tsing Hua University, Taiwan
Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees
May 18, 2026Aditya Tanna, Nassim Bouarour, Mohamed Bouadi +2Tabular Foundation Models Gradient-Boosted Decision Trees
Lexsi Labs
LogRouter: Adaptive Two-Level LLM Routing for Log Question Answering in Big Data Systems
May 18, 2026Mert Coskuner, Merve Zeybel, Melik Mert DolanLLM Inference Efficiency Cost-Aware Inference
TUBITAK BILGEM · Kocaeli, Turkey
Adaptive Generate-Rank-Verify: Inference-Time Search with Costly Verification
May 17, 2026Shaddin Dughmi, Mahdi Haghifam, Yusuf Hakan KalayciAdaptive Inference Inference-Time Search
University of Southern California · Data Science Institute at the University of Chicago · Northwestern University +2
Computational Challenges in Token Economics: Bridging Economic Theory and AI System Design
May 17, 2026Ou Wu, Yingjun DengLLM Inference Cost-Aware Inference
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, China · Hefei Institutes of Physical Science, Chinese Academy of Sciences, China
HyDRA: Hybrid Dynamic Routing Architecture for Heterogeneous LLM Pools
May 16, 2026Aashna Garg, Siddharth Singha Roy, Jinu Jang +2Cost-Aware Inference LLM Routing
Microsoft
MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization
May 15, 2026Jan Büssing, Moritz Schlager, Timo Heiß +2Cost-Aware Inference Automatic Prompt Optimization
LMU Munich, Germany · Technical University of Munich (TUM), Munich Center for Machine Learning (MCML), Germany · LMU Munich, Munich Center for Machine Learning (MCML), Germany +2
FlowCompile: An Optimizing Compiler for Structured LLM Workflows
May 13, 2026Junyan Li, Zhang-Wei Hong, Maohao Shen +2LLM Agent Orchestration Cost-Aware Inference
UMass Amherst · MIT-IBM Watson AI Lab · MIT
CADS: Conformal Adaptive Decision System for Cost-Efficient Image Classification
May 12, 2026Mikael Turkoglu, Tim Bary, Vincent Thielens +2Adaptive Inference Cost-Aware Inference
1ICTEAM, UCLouvain, Belgium · 2SAFiR Lab, Univ. of Sherbrooke, Canada · 3Univ. of Mons, Belgium
CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference
May 12, 2026Nan Xue, Shengkang Chen, Zhiyong Chen +4On-Device Language Model Inference LLM Inference
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China · Department of Broadband Communication, Pengcheng Laboratory, Shenzhen 518000, China · Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen), Shenzhen 518172, China +1
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
May 11, 2026Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1LLM-as-a-Judge Cost-Aware Inference
Department of Statistics, University of California, Irvine, USA.
UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing
May 11, 2026Varun KotteCost-Aware Inference Language Model Calibration
Independent Researcher.
Assessment of RAG and Fine-Tuning for Industrial Question-Answering-Applications
May 10, 2026Jakob Sturm, Josef Pichlmeier, Christian Bernhard +4LLM Fine-Tuning Cost-Aware Inference
BMW Group; Munich, Germany · Ludwig Maximilians Universitat; Munich, Germany · Technical University of Munich; Munich, Germany
When Independent Sampling Outperforms Agentic Reasoning
May 8, 2026Yihe Dong, Boris ShigidaCost-Aware Inference Competitive Programming
Princeton University
Active Multiple-Prediction-Powered Inference
May 8, 2026Nicholas Brawand, Nima Leclerc, Anhthy Ngo +4Adaptive Inference Cost-Aware Inference
The MITRE Corporation · Georgia Institute of Technology

Jun 1, 2026, cs.CLJ/K move · Enter open · S save

Cost-Aware Diffusion Draft Trees for Speculative Decoding

Shuai Zhang, Huachuan Qiu, Hongliang He, Yong Dai

1Zhejiang University · 2Westlake University

Cost-Aware Inference Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates inference by having a lightweight drafter propose tokens verified in parallel by the target language model. Block diffusion drafters such as DFlash generate an entire draft block in one pass, yielding per-position marginals; DDTree uses these to build a candidate tree that maximizes expected acceptance length under a fixed node budget. We observe, however, that acceptance length is non-decreasing in budget: it always favors larger trees regardless of verification cost, offering no principled basis for budget selection. We introduce \textbf{CaDDTree} (Cost-aware Diffusion Draft Tree), a method that directly optimizes token throughput (expected tokens generated per unit time) by jointly selecting the tree structure and node budget. We model draft and verification latencies explicitly, show that the throughput objective decomposes into a per-round one-dimensional search over the budget, and prove that under a convex verification cost the throughput function is \emph{unimodal}, enabling an efficient greedy stopping rule. CaDDTree requires no offline budget search, adapting the budget each round from the current per-position distributions and verification cost. Experiments on Qwen3-4B and Qwen3-8B across eight benchmarks spanning reasoning, coding, and instruction-following tasks show that \caDDTree{} matches or surpasses DDTree with oracle budget selection on nearly all tasks.

Momentum

Related topics

Latest papers 183