Disaggregated LLM Serving

LLM: Large Language Model

Ask about this topic

Overview

Disaggregated LLM serving studies how to split inference work—most often prompt prefill and token-by-token decoding, but also attention, feed-forward, encoding, and key-value (KV) cache storage—across devices or worker pools. The goal is to match compute and memory resources to each phase while meeting service-level objectives (SLOs), especially as prompt lengths, output lengths, and demand vary. Systems use workload-aware routing and elastic resource allocation, heterogeneous accelerators and frequency control, and KV-cache transfer, compression, or reuse; simulators and trace-driven benchmarks help compare designs. A central tension is that specialization can improve throughput, energy efficiency, and goodput, but network and cache movement, prediction errors, and workload-specific trade-offs can erase gains, so deployment choices need calibrated end-to-end evaluation.

Latest papers

CardsList

33 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

  1. LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference

    Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM ServingLLM Inference

    Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.

  2. When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

    Oct 15, 2026Muhammad IbrahimContext ParallelismKV-Cache Management

    Multiverse Computing

  3. Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning

    Oct 12, 2026Chiheng Lou, Xiangwei Deng, Bangjun Xiao +4Prefill-Decode DisaggregationAgentic RL

    †School of Computer Science, Peking University · ‡LLM-Core Xiaomi

  4. Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads

    Oct 5, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidLLM Inference EfficiencyDisaggregated LLM Serving

    Citigroup Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom

  5. FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving

    Oct 2, 2026Kartik Ramesh, Kaidi Fu, Zihan Zheng +4LLM ServingPrefill-Decode Disaggregation

    University of Illinois Urbana-Champaign · IBM Research

  6. SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

    Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-of-Experts PruningDisaggregated LLM Serving

    a2sys · KAIST

  7. EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models

    Sep 25, 2026Kunxiong Zhu, Zhihao Shu, Hangyu Zheng +4Disaggregated LLM ServingLLM Inference Scheduling

    University of Georgia Athens, GA, USA · Western Digital San Jose, CA, USA · University of Texas at Arlington Arlington, TX, USA

  8. Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving

    Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM ServingLLM Inference Scheduling

    Meta Platforms

  9. From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving

    Sep 19, 2026Twinkll SisodiaLLM ServingLLM Inference Efficiency

  10. Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

    Sep 14, 2026Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri +3LLM RoutingDisaggregated LLM Serving

    Vizuara

  11. MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

    Sep 14, 2026Tiancheng Zhang, Yulin Chen, Yunfeng Zhao +3Disaggregated LLM ServingLLM Inference Scheduling

    College of Intelligence and Computing, Tianjin University, Tianjin, China · The International Joint Institute of Tianjin University, Tianjin University, Fuzhou, China · Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics, Tianjin, China

  12. OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving

    Sep 13, 2026Zikun Li, Yixuan Mei, Shiqi Pan +9Disaggregated LLM ServingCost-Aware Inference

    Carnegie Mellon University · Peking University · Carnegie Mellon University and Google

  13. Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

    Sep 11, 2026Arya Tschand, Yaosheng Fu, Vikram Sharma Mailthody +6Disaggregated LLM ServingLLM Inference Acceleration

    Harvard University Cambridge, Massachusetts, USA · NVIDIA Santa Clara, California, USA

  14. Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

    Sep 10, 2026Jae Gon Kim, Donghoon Yoo, Hanyul Ryu +3LLM Inference EfficiencyDisaggregated LLM Serving

    Xenoscube, Inc.

  15. PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response

    Aug 22, 2026Yueying Li, Jiayang Chen, Yuanfan Chen +5LLM ServingLLM Inference Scheduling

    Cornell University · Microsoft Azure Research · Cornell Tech

  16. AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregation

    Aug 5, 2026Chengyu Qiu, Xiao Fu, Fengcun Li +6Mixture-of-Experts Language ModelsDisaggregated LLM Serving

    Tsinghua University · Meituan · Beijing, China

  17. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    Aug 3, 2026Cunchen Hu, Liangliang Xu, Tian Liu +9Energy-Efficient MLDisaggregated LLM Serving

    China Telecom Cloud Computing Research Institute · Xidian University · University of Science and Technology of China +1

  18. Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

    Jul 2, 2026Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj +2LLM ServingPrefill-Decode Disaggregation

    Microsoft

  19. Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

    Jun 5, 2026Qianli Ma, Zhiqing Tang, Hanshuai Cui +2LLM ServingKV Caching

    Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +1

  20. Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving

    Jun 1, 2026Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami +2LLM ServingDisaggregated LLM Serving

    University of Chicago

  21. How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving

    May 27, 2026Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9LLM ServingLLM Inference

    1Georgia Institute of Technology · 5Infravana · 2Intel +2

  22. Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

    May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM ServingDisaggregated LLM Serving

    The Chinese University of Hong Kong · Anuttacon · StepFun

  23. KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

    May 13, 2026Zedong Liu, Xinyang Ma, Dejun Luo +9LLM ServingKV Caching

    University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Shanghai Jiao Tong University

  24. SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

    May 3, 2026Yipin Guo, Siddharth JoshiDisaggregated LLM ServingKV-Cache Compression

    University of Notre Dame IN, USA

  25. ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

    Apr 8, 2026Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3LLM Inference EfficiencyDisaggregated LLM Serving

    Virginia Tech, Blacksburg, Virginia, USA · Queen’s University Belfast, Belfast, Northern Ireland, UK

  26. ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache

    Apr 7, 2026Shao Wang, Rui Ren, Lin GuiLLM ServingLLM Inference Acceleration

    Shanghai Jiao Tong University, Shanghai, China

  27. Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

    Mar 29, 2026Rajveer Bachkaniwala, Chengqi Luo, Richard So +2LLM InferenceLLM Inference Serving

    Georgia Tech, Atlanta, USA

  28. Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

    Mar 13, 2026Donglin YuEfficient Multimodal InferenceLLM Serving

  29. SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference

    Mar 5, 2026Luchang Li, Dongfang Li, Bozhao Gong +1Disaggregated LLM ServingPrefill-Decode Disaggregation

    Kingsoft Cloud

  30. SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

    Mar 3, 2026Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6Disaggregated LLM ServingLLM Inference Acceleration

    NAVER Cloud