LLM Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 144

All topics

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving
Jun 25, 2026Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11LLM Serving Cost-Aware Inference
1ADAPT Centre, Trinity College Dublin · 2Huawei Research
PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs
Jun 25, 2026Muhammad AhmedLLM Serving GPU Kernel Optimization
CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
Jun 23, 2026Zhuoren Ye, Tianyu Wo, Dinghao Xue +4Mixture-of-Experts Language Models LLM Serving
School of Software, Beihang University
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Jun 22, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University
Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM Serving LLM Inference Scheduling
Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Jun 19, 2026Yang Shen, Meghana Madhyastha, Robert Underwood +2LLM Serving KV Caching
Department of Computer Science, Johns Hopkins University, Baltimore, USA
Token-Operations-Oriented Inference Optimization Techniques for Large Models
Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM Serving Inference-Time Optimization
Unicom Data Intelligence, China Unicom · Data Science and Artificial Intelligence Research Institute, China Unicom · Tsinghua University +7
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
Service-Induced Congestion in Memory-Constrained LLM Serving
Jun 14, 2026Ruicheng Ao, Jing Dong, Gan Luo +1LLM Inference Efficiency LLM Serving
Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139 · Columbia Business School, Columbia University, New York, NY 10027 · School of Mathematical Sciences, Peking University, Beijing, China
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
Jun 13, 2026Qianli Liu, Kaibin Guo, Zicong Hong +5Expert Parallelism LLM Serving
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · School of Software Engineering, Sun Yat-Sen University, China · School of Cyber Science and Engineering, Xi’an Jiaotong University, China +1
STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming
Jun 11, 2026Anas Nassar, Steve Mohr, Leonard Apanasevich +1LLM Serving LLM Inference
Advanced Cyberinfrastructure for Education and Research (ACER) University of Illinois Chicago Chicago, Illinois, USA
Characterizing Software Aging in GPU-Based LLM Serving Systems
Jun 10, 2026Domenico Cotroneo, Bojan CukicSoftware Engineering LLM Serving
University of North Carolina at Charlotte · College of Computing and Informatics
External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs
Jun 10, 2026Lin Sun, Heming Zhang, Xiangzheng ZhangLLM Serving Knowledge Augmentation for Language Models
Qiyuan Tech
K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling
Jun 9, 2026Zhiwei Tang, Yuanyu He, Yizheng Han +4LLM Serving Language Model Decoding
1DAMO Academy, Alibaba Group · 2Hupan Lab · 3Zhejiang University +1
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM Serving KV-Cache Management
University of Central Florida
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
Jun 5, 2026Qianli Ma, Zhiqing Tang, Hanshuai Cui +2LLM Serving KV Caching
Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +1
SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving
Jun 4, 2026Hongyuan Liu, Yawei Li, Zhiqiang Que +3LLM Quantization LLM Serving
University of Electronic Science and Technology of China · University of Bristol · 3ETH Zurich
Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents
Jun 4, 2026Zhuoming Chen, Xinrui Zhong, Qilong Feng +5LLM Serving LLM Inference Acceleration
1Carnegie Mellon University · 2Rice University · 3National University of Singapore
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
Jun 4, 2026Hyungmin Kim, Minsoo Kim, Hongseok Kim +1LLM Serving KV Caching
RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
Jun 4, 2026Yang Liu, ZhaoKai Luo, HuaYi Jin +5LLM Serving KV Caching
†Xiaohongshu Inc., China · ¶Huawei Cloud · §Peking University
Ekka: Automated Diagnosis of Silent Errors in LLM Inference
Jun 3, 2026Yile Gu, Zhen Zhang, Shaowei Zhu +4Software Engineering LLM Serving
University of Washington · 2Amazon Web Services.
E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3LLM Serving Edge Computing
Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway
Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
Jun 1, 2026Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami +2LLM Serving Disaggregated LLM Serving
University of Chicago
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLLM Serving Zeroth-Order Optimization
University of Minnesota
How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
May 27, 2026Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9LLM Serving LLM Inference
1Georgia Institute of Technology · 5Infravana · 2Intel +2
RW-TTT: Batched Serving for Request-Owned Test-Time Training State
May 27, 2026Jian Yang, Zhizhuo Kou, Yao Tian +4LLM Serving Test-Time Optimization
1The Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3National University of Singapore
A Policy-Driven Runtime Layer for Agentic LLM Serving
May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference Efficiency Multi-Agent LLM Systems
University of California, Santa Cruz
BIRDS: Characterizing and Understanding Biodiversity Impact of Large Language Model Serving
May 26, 2026Tianyao Shi, Yi DingLLM Serving Responsible AI
Purdue University
Fine-Tuning and Serving Gemma 4 31B on Google Cloud TPU: A Technical Comparison with GPU Baselines
May 25, 2026Jatin Kishnani, Mayank Goel, Amit Singh +2AI Accelerator Inference LLM Serving
ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
May 23, 2026Yang Xu, Zihuai Xu, Hongli Xu +3LLM Serving LLM Fine-Tuning
Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, Jiangsu, China, 215123

Jun 25, 2026, cs.PFJ/K move · Enter open · S save

Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana, Ammar Abbas+10

1ADAPT Centre, Trinity College Dublin · 2Huawei Research

LLM Serving Cost-Aware Inference LLM Routing

Open paper AskPDF arXiv

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.

Momentum

Related topics

Latest papers 144