LLM Serving

LLM: Large Language Model

Ask about this topic

Overview

Large language model serving studies how to run inference efficiently for many concurrent requests, balancing response speed, throughput, cost, and service-level objectives. Common approaches include continuous batching and prefill–decode scheduling, request routing, and managing or compressing KV caches; newer systems also use disaggregated hardware, speculative decoding, and workload-aware power control. Evaluations use request traces and production-like workloads to measure time to first token, token latency, throughput, quality, and resource use across models and hardware. A central challenge is that workloads and output lengths vary, so policies that improve throughput or cost can worsen tail latency, fairness, or response quality.

Latest papers

CardsList

216 papers31 papers in the last four weeks, up 94% on the four weeks before. 0.2% of all new papers.

  1. Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing

    May 11, 2026Yunze Zhao, Yibo Zhao, Yuchen Zhang +2LLM ServingLLM Security

    University of Maryland · New York University

  2. PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving

    May 10, 2026Bing Xie, Zhipeng Wang, Masahiro Tanaka +1LLM ServingKV Caching

    †Corresponding author.

  3. HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving

    May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM ServingKV Caching

    School of Computer Science, Wuhan University · Amazon

  4. Regulating Branch Parallelism in LLM Serving

    May 7, 2026Swapnil Gandhi, Siva Hari, William J. Dally +1LLM ServingLLM Inference Scheduling

    1Stanford University · 2NVIDIA

  5. VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

    May 7, 2026Keisuke Kamahori, Shihang Li, Simon Peter +1Multi-Agent LLM SystemsLLM Serving

    University of Washington

  6. A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints

    May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM ServingLLM Inference

    Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST

  7. Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs

    May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM ServingCost-Aware Inference

    Carnegie Mellon University · Peking University

  8. When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning

    May 5, 2026Jiaqi Wei, Xuehang Guo, Pengfei Yu +5LLM ServingLLM Reasoning

    Zhejiang University · College of William and Mary · University of Illinois Urbana-Champaign +4

  9. SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

    May 4, 2026Jincheng Xie, Yawen Ling, Qi Xiao +4Parallel Speculative DecodingLLM Serving

    1Tsinghua University · 2AI Infra Team at JDT · 3JD iCity, JD Technology, JD Intelligent Cities Research

  10. Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

    May 3, 2026Nikita Agrawal, Ruben MayerLLM ServingKV Caching

    University of Bayreuth

  11. Human-Less LLM Serving: Quantifying the Human Tax on Throughput

    May 3, 2026Jianhui Lian, Li Chen, Dan Li +1LLM Inference EfficiencyLLM Serving

    Tsinghua SIGS · Beijing Zhongguancun Laboratory · Tsinghua University

  12. SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

    May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM ServingKV-Cache Compression

    University of Notre Dame IN, USA

  13. ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

    May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert ParallelismLLM Serving

    University of Virginia · Snowflake AI Research · Harvard University

  14. Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics

    May 2, 2026Zijie ZhouLLM ServingLLM Inference

    Department of Industrial Engineering and Decision Analytics, HKUST

  15. Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving

    Apr 30, 2026Junsun Choi, Sam Son, Sunjin Choi +5LLM ServingCost-Aware Inference

    UC Berkeley · UC Berkeley & ICSI

  16. FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving

    Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language ModelsLLM Serving

    TU Berlin · Berlin, Germany

  17. Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

    Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM ServingKV-Cache Offloading

    University of Virginia · Microsoft Research · Georgia Institute of Technology +1

  18. Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

    Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference EfficiencyMulti-Agent LLM Systems

    UCLA · Alibaba Cloud Computing · Alibaba Group +6

  19. GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads

    Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2LLM Inference EfficiencyLLM Serving

    China Unicom

  20. Latency and Cost of Multi-Agent Intelligent Tutoring at Scale

    Apr 27, 2026Iizalaarab Elhaimeur, Nikos ChrisochoidesLLM Inference EfficiencyMulti-Agent LLM Systems

    Center for Real-Time Computing · Computer Science Department · Old Dominion University +2

  21. RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization

    Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM ServingCost-Aware Inference

    1The University of Hong Kong · 2Stellaris AI Limited

  22. Continuous Semantic Caching for Low-Cost LLM Serving

    Apr 21, 2026Baran Atalar, Xutong Liu, Jinhang Zuo +3LLM ServingCost-Aware Inference

    Carnegie Mellon University Pittsburgh, Pennsylvania, USA · University of Washington Tacoma Tacoma, Washington, USA · City University of Hong Kong China +1

  23. SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

    Apr 21, 2026Jinda Jia, Jisen Li, Zhongzhu Zhou +8LLM ServingKV Caching

    Together AI

  24. SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving

    Apr 19, 2026Christian LysenstøenLLM Inference EfficiencyLLM Serving

    Inland Norway University of Applied Sciences · Visiting Student, University of California, Berkeley

  25. Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems

    Apr 19, 2026Yuji Yamamoto, Satoshi MatsuuraLLM ServingAdversarial Attacks on LLMs

    Institute of Science Tokyo, Tokyo, Japan

  26. POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving

    Apr 17, 2026Shaoang Li, Jian LiLLM ServingLow-Rank Adaptation

    Stony Brook University

  27. Sparse Prefix Caching for Hybrid and Recurrent LLM Serving

    Apr 17, 2026Mikhail Shirokikh, Sergey NikolenkoLLM ServingKV-Cache Management

  28. ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

    Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM ServingMixture-of-Experts Inference

    KAIST · Samsung Electronics · GIST

  29. PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving

    Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference EfficiencyLLM Serving

    DisNet Lab University of Melbourne Melbourne, VIC, Australia · IBM Research Yorktown Heights, NY, USA

  30. Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees

    Apr 13, 2026Zhuolun Dong, Junyu CaoLLM Inference SchedulingLLM Serving

    Department of Information, Risk, and Operations Management, The University of Texas at Austin.