LLM Serving

LLM: Large Language Model

Ask about this topic

Overview

Large language model serving studies how to run inference efficiently for many concurrent requests, balancing response speed, throughput, cost, and service-level objectives. Common approaches include continuous batching and prefill–decode scheduling, request routing, and managing or compressing KV caches; newer systems also use disaggregated hardware, speculative decoding, and workload-aware power control. Evaluations use request traces and production-like workloads to measure time to first token, token latency, throughput, quality, and resource use across models and hardware. A central challenge is that workloads and output lengths vary, so policies that improve throughput or cost can worsen tail latency, fairness, or response quality.

Latest papers

CardsList

216 papers31 papers in the last four weeks, up 94% on the four weeks before. 0.2% of all new papers.

  1. xGR: Efficient Generative Recommendation Serving at Scale

    Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM ServingLLM Inference Acceleration

    Beihang University · JD Company · University of Science and Technology Beijing +1

  2. Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM

    Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator InferenceLLM Serving

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University

  3. Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

    Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM ServingLearning to Rank

    University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA

  4. VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing

    Sep 5, 2025Jiahuan Yu, Aryan Taneja, Junfeng Lin +1LLM ServingEnergy-Efficient ML

    Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China

  5. LLM Serving Optimization with Variable Prefill and Decode Lengths

    Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM ServingLLM Inference Scheduling

    Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST

  6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM ServingDisaggregated LLM Serving

    Moonshot AI · Tsinghua University