Disaggregated LLM Serving

LLM: Large Language Model

Disaggregated LLM serving studies how to split inference work—most often prompt prefill and token-by-token decoding, but also attention, feed-forward, encoding, and key-value (KV) cache storage—across devices or worker pools. The goal is to match compute and memory resources to each phase while meeting service-level objectives (SLOs), especially as prompt lengths, output lengths, and demand vary. Systems use workload-aware routing and elastic resource allocation, heterogeneous accelerators and frequency control, and KV-cache transfer, compression, or reuse; simulators and trace-driven benchmarks help compare designs. A central tension is that specialization can improve throughput, energy efficiency, and goodput, but network and cache movement, prediction errors, and workload-specific trade-offs can erase gains, so deployment choices need calibrated end-to-end evaluation.

Latest papers

34 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving
Mar 3, 2026Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6Disaggregated LLM Serving LLM Inference Acceleration
NAVER Cloud
LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
Feb 26, 2026Jaehong Cho, Hyunmin Choi, Guseul Heo +1LLM Serving Disaggregated LLM Serving
School of Computing KAIST Daejeon, South Korea
VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
Sep 5, 2025Jiahuan Yu, Aryan Taneja, Junfeng Lin +1LLM Serving Energy-Efficient ML
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM Serving Disaggregated LLM Serving
Moonshot AI · Tsinghua University

Mar 3, 2026J/K move · Enter open · S save

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

Sunghyeon Woo, Ahreum Seo, Jaegwang Lee, Jaeeun Kil+5

NAVER Cloud

Disaggregated LLM Serving LLM Inference Acceleration LLM Inference Scheduling

Open paper AskPDF arXiv

In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Prediction (SUN), the first approach that enables cross-model sharing of decode execution in disaggregated multi-LLM serving. SUN decomposes a decoder-only Transformer into a prefill module and a decode module, and fine-tunes only the task-specific prefill module, enabling a frozen decode module to be shared across models. This design enables a model-agnostic decode routing policy that balances decode requests across shared workers to maximize utilization. Across diverse tasks and model families, SUN achieves accuracy comparable to full fine-tuning while maintaining system throughput with fewer decode workers. In particular, SUN improves throughput per GPU by up to 2.0x over conventional disaggregation while keeping time-per-output-token (TPOT) within 5%. SUN inherently enables and facilitates low-bit decoding; with Quantized SUN (QSUN), it achieves a 45% speedup with comparable accuracy to SUN while preserving the benefits of shared decoding.

Latest papers

SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving