LLM Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 144

All topics

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
Apr 29, 2026Zihan Zhao, Baotong Lu, Shengjie Lin +8LLM Serving KV-Cache Offloading
University of Virginia · Microsoft Research · Georgia Institute of Technology +1
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads
Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2LLM Inference Efficiency LLM Serving
China Unicom
Latency and Cost of Multi-Agent Intelligent Tutoring at Scale
Apr 27, 2026Iizalaarab Elhaimeur, Nikos ChrisochoidesLLM Inference Efficiency Multi-Agent LLM Systems
Center for Real-Time Computing · Computer Science Department · Old Dominion University +2
RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Serving Cost-Aware Inference
1The University of Hong Kong · 2Stellaris AI Limited
Continuous Semantic Caching for Low-Cost LLM Serving
Apr 21, 2026Baran Atalar, Xutong Liu, Jinhang Zuo +3LLM Serving Cost-Aware Inference
Carnegie Mellon University Pittsburgh, Pennsylvania, USA · University of Washington Tacoma Tacoma, Washington, USA · City University of Hong Kong China +1
SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
Apr 21, 2026Jinda Jia, Jisen Li, Zhongzhu Zhou +8LLM Serving KV Caching
Together AI
SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving
Apr 19, 2026Christian LysenstøenLLM Inference Efficiency LLM Serving
Inland Norway University of Applied Sciences · Visiting Student, University of California, Berkeley
Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems
Apr 19, 2026Yuji Yamamoto, Satoshi MatsuuraLLM Serving Adversarial Attacks on LLMs
Institute of Science Tokyo, Tokyo, Japan
POLAR: Online Learning for LoRA Adapter Caching and Routing in Edge LLM Serving
Apr 17, 2026Shaoang Li, Jian LiLLM Serving Low-Rank Adaptation
Stony Brook University
Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
Apr 17, 2026Mikhail Shirokikh, Sergey NikolenkoLLM Serving KV-Cache Management
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM Serving Mixture-of-Experts Inference
KAIST · Samsung Electronics · GIST
PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving
Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference Efficiency LLM Serving
DisNet Lab University of Melbourne Melbourne, VIC, Australia · IBM Research Yorktown Heights, NY, USA
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Mar 10, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
Feb 27, 2026Ferran Agullo, Joan Oliveras, Chen Wang +5LLM Inference Efficiency LLM Serving
Barcelona Supercomputing Center (BSC), Barcelona, Spain · Universitat Politècnica de Catalunya - BarcelonaTech (UPC), Barcelona, Spain · IBM Research, New York, USA
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
xGR: Efficient Generative Recommendation Serving at Scale
Dec 12, 2025Qingxiao Sun, Tongxuan Liu, Shen Zhang +13LLM Serving LLM Inference Acceleration
Beihang University · JD Company · University of Science and Technology Beijing +1
Systematic Exploration of Multi-core Architectures for Efficient LLM Serving using WaferAI-SIM
Oct 7, 2025Tianhao Zhu, Dahu Feng, Erhu Feng +1AI Accelerator Inference LLM Serving
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · Department of Precision Instrument, Tsinghua University
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA
VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routing
Sep 5, 2025Jiahuan Yu, Aryan Taneja, Junfeng Lin +1LLM Serving Energy-Efficient ML
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Champaign, USA · Department of Precision Instrument, Tsinghua University, Beijing, China
LLM Serving Optimization with Variable Prefill and Decode Lengths
Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM Serving LLM Inference Scheduling
Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM Serving Disaggregated LLM Serving
Moonshot AI · Tsinghua University
OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
Date pendingWeihuang Wen, Yingying Liu, Yichuan Liu +5LLM Serving LLM Inference Scheduling
The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · Huawei Technologies Co., Ltd.

Apr 29, 2026, cs.LGJ/K move · Enter open · S save

Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

Zihan Zhao, Baotong Lu, Shengjie Lin, Yizou Chen+7

University of Virginia · Microsoft Research · Georgia Institute of Technology+1

LLM Serving KV-Cache Offloading KV Caching Long-Context Language Model Inference KV-Cache Management Sparse Attention

Open paper AskPDF arXiv

Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorithmic savings rarely translate into end-to-end system-level gains because sparse methods typically operate at different granularities and thus rely on ad hoc, per-algorithm implementations. At the same time, hierarchical KV storage introduces a new systems bottleneck: retrieving fine-grained, irregular KV subsets across the GPU-CPU boundary can easily erase the benefits of sparsity. We present SPIN, a sparse-attention-aware inference framework that co-designs the execution pipeline with hierarchical KV storage through three techniques: (1) a unified partition abstraction that maps different sparsity granularities onto a shared page-based KV substrate; (2) a locality-aware KV cache manager that dynamically sizes per-request HBM budgets and uses a GPU-friendly bucketed LRU policy to cut PCIe round-trips; and (3) a two-level hierarchical metadata layout sized to the active working set rather than the worst-case address space. Built on vLLM with three representative sparse attention algorithms, SPIN delivers 1.66-5.66x higher end-to-end throughput and 7-9x lower TTFT than vLLM, and reduces TPOT by up to 58% over the original sparse-attention implementations.

Momentum

Related topics

Latest papers 144