LLM Inference Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 51

All topics

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Serving Disaggregated LLM Serving
The Chinese University of Hong Kong · Anuttacon · StepFun
KVBuffer: IO-aware Serving for Linear Attention
May 18, 2026Longwei Zou, Lin ZhongLLM Inference Serving Linear Attention
Department of Computer Science Yale University New Haven, CT 06511
Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers
May 13, 2026Victor NorgrenTransformer Inference LLM Inference
LayerScale, Inc.
Enabling Performant and Flexible Model-Internal Observability for LLM Inference
May 11, 2026Nengneng Yu, Sixian Xiong, Yibo Zhao +2LLM Inference Serving Efficient Language Model Inference
Department of Computer Science University of Maryland, College Park
KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving
May 10, 2026Zhiqing Zhong, Zhijing Ye, Jian Zhang +3KV Caching KV-Cache Management
Stevens Institute of Technology Hoboken, New Jersey, USA · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Argonne National Laboratory Lemont, Illinois, USA +1
Execution Envelopes: A Shared Admission Contract for Backend AI Execution Requests
May 8, 2026Krti TallamLLM Inference Serving AI Governance
Kamiwaza AI, San Francisco, CA, USA
Regulating Branch Parallelism in LLM Serving
May 7, 2026Swapnil Gandhi, Siva Hari, William J. Dally +1LLM Serving LLM Inference Scheduling
1Stanford University · 2NVIDIA
Towards Distributed Inference of LLMs on a P2P Network
May 7, 2026Shabari S Nair, Krishanu SainiLLM Inference KV Caching
Department of Computer Science, The University of Texas at Austin.
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
Token Arena: A Continuous Benchmark Unifying Energy and Cognition in AI Inference
May 1, 2026Yuxuan Gao, Megan Wang, Yi Ling YuLLM Evaluation Energy-Efficient ML
University of Pennsylvania · OpenMesh AI · Columbia University
FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving
Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language Models LLM Serving
TU Berlin · Berlin, Germany
Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study
Apr 28, 2026Srikanta Prasad S, Utkarsh AroraLLM Inference Serving
Agentforce AI Platform, Salesforce India Pvt Ltd Bangalore, Karnataka, India
KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving
Apr 17, 2026Yichao Yuan, Mosharaf Chowdhury, Nishil TalatiEfficient Inference LLM Inference Serving
University of Illinois, Urbana Champaign Urbana, IL, USA · University of Michigan Ann Arbor, MI, USA
Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
Apr 16, 2026Otto White, Marcel Wagenländer, Britannio Jarrett +7LLM Inference LLM Inference Scheduling
Imperial College London · Independent Researcher
Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Feb 14, 2026Hao Kang, Ziyang Li, Weili Xu +7LLM Agent Orchestration LLM Inference Scheduling
University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology
Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model
Feb 8, 2026Tianyi Wang, Huawei Fan, Yuanchao Shu +2LLM Security Adversarial Attacks on LLMs
Zhejiang University
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference Efficiency LLM Inference
EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea

May 20, 2026, cs.DCJ/K move · Enter open · S save

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang+2

The Chinese University of Hong Kong · Anuttacon · StepFun

LLM Serving Disaggregated LLM Serving LLM Inference Scheduling LLM Inference Serving Distributed Inference

Open paper AskPDF arXiv

Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet existing simulators lack the architectural completeness and decision-grade fidelity it demands. Their monolithic-replica abstractions are ill-suited to disaggregated serving, while average-case analytical proxies can distort SLA predictions and even reverse optimization conclusions. We present Frontier, a discrete-event simulator for modern LLM inference serving. Frontier features a disaggregated abstraction. It captures the structure and dynamics of modern serving systems by modeling co-location, Prefill-Decode Disaggregation (PDD), and Attention-FFN Disaggregation (AFD) with role-specific cluster workers, incorporating key runtime optimizations (e.g., CUDA Graphs, speculative decoding) within the scheduler-batch-engine loop, and supporting stateful requests for emerging workloads. It further provides accurate and generalizable predictions of computation, communication, and memory costs across diverse serving scenarios with complex workload compositions. On 16-H800 GPU testbed, Frontier achieves an average throughput error below 4%. Compared with state-of-the-art simulators, it reduces end-to-end latency error from 44.9% to 6.4% under co-location and from 51.7% to 2.6% under disaggregation. It scales to over 1K GPUs on commodity CPUs and enables new use cases such as SLA-dependent Pareto frontier exploration, heterogeneous disaggregated allocation, agentic reasoning scheduling validation, and RL post-training reconfiguration. We release Frontier at https://github.com/NetX-lab/Frontier.

Momentum

Related topics

Latest papers 51