LLM Inference Serving

LLM: Large Language Model

Large language model inference serving studies how systems process prompts and generate tokens under limits on GPU compute and memory, especially the KV cache. Common approaches include continuous batching, request scheduling and routing, prefill-decode disaggregation, speculative decoding, and cache reuse; systems are evaluated using latency and throughput metrics, often against workload traces and service-level objectives. Work increasingly addresses stateful, multi-turn agents and heterogeneous deployments spanning GPUs, edge devices, and data centers, where tool calls and changing context complicate prediction and resource allocation. A central tension is balancing cache locality and high utilization against fair, predictable tail latency, while also controlling cost and energy as workloads shift.

Latest papers

74 papers10 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading
Oct 28, 2026Qi Qi, Yifan Sui, Hanfei Yu +2LLM Inference Serving
University of Massachusetts Amherst Amherst, Massachusetts, USA · Hong Kong University of Science and Technology Hong Kong, China · Stevens Institute of Technology Hoboken, New Jersey, USA +1
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Oct 27, 2026Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis +3LLM Serving LLM Inference Serving
Microsoft
InferSlice: A Reference Architecture for SLO-driven LLM Inference over a Telco Wide-Area Network
Oct 23, 2026Juan Parra-Ullauri, Anthony Chiu, Hari Madhukumar +6LLM Inference LLM Inference Serving
BT Group · University of Bristol
Cache Engineering for Agents
Oct 20, 2026Tianyu Fu, Ruoxi Wang, Zihan Min +6KV-Cache Management Memory Management for LLM Agents
Tsinghua University · The Chinese University of Hong Kong
FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments
Oct 17, 2026Dimitrios Liakopoulos, Prasoon Sinha, Neeraja J. YadwadkarLLM Serving Cost-Aware Inference
The University of Texas at Austin
Stable Token-level Scheduling for Serverless LLM Inference
Oct 13, 2026Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti AddyaLLM Inference Scheduling LLM Inference Serving
Cloud & Smart System Services Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India
Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving
Oct 10, 2026Burak Topcu, Musa Cim, Zuhal Ozturk +1LLM Inference Scheduling Carbon-Aware Computing
Department of Computer Science and Engineering, The Pennsylvania State University, State College, PA 16801, USA
Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
AID: A Framework for AI Infrastructure Dynamics
Oct 3, 2026Abi AryanLLM Inference Serving System Identification
Joule
FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
Oct 2, 2026Kartik Ramesh, Kaidi Fu, Zihan Zheng +4LLM Serving Prefill-Decode Disaggregation
University of Illinois Urbana-Champaign · IBM Research
Inference Auctions
Sep 30, 2026Keegan Harris, Siddharth Prasad, Asher Trockman +2Mechanism Design Automated Bidding
University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research +1
Spexis: Speculative Lookahead Scheduling for LLM Inference
Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference Efficiency LLM Inference Scheduling
Seoul National University · Hanyang University
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
Sep 22, 2026Jennifer Williams, Dave Farris, Jeff Farris +1Software Engineering LLM Inference Serving
NVIDIA · University of California, Berkeley
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Sep 22, 2026Lijuan Tang, Yuemeng ZhengLLM Evaluation Tool-Use Evaluation
Northeastern University, Seattle
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference Efficiency KV Caching
Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China
End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
Sep 15, 2026Zhen Li, Jun Cai, Haoran Gao +2LLM Inference Scheduling LLM Request Scheduling
Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, H3G 1M8, Canada · Department of Computer Science, Hang Seng University of Hong Kong, Hong Kong SAR
Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane
Sep 3, 2026Twinkll SisodiaEnergy-Aware LLM Inference LLM Inference
Flama: a Python framework for development and deployment of production-ready APIs, machine learning, and LLM services
Aug 19, 2026José A. Perdiguero López, Miguel A. Durán-OlivenciaLLM Serving LLM Inference Serving
Vortico Tech, Málaga, Spain
Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN
Aug 17, 2026Tianhang Ding, Jianchun Liu, Hongli XuLLM Inference Serving LLM Inference Scheduling
University of Science and Technology of China
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Aug 15, 2026Chaokun Chang, Yukun Zhou, Kaihua Fu +10LLM Agent Serving LLM Agent Workflow Optimization
Hong Kong University of Science and Technology · Alibaba Group · Bytedance
InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers
Aug 13, 2026Nicoletta Tsiopani, Moysis Symeonides, George Pallis +1Energy-Efficient ML LLM Inference Serving
Department of Computer Science University of Cyprus
TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference
Aug 3, 2026Ruilin Xu, Junyi Li, Pengfei Chen +1LLM Inference LLM Inference Serving
Sun Yat-sen University Guangzhou, China
Kalypso: Relational LLM Serving
Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2LLM Serving LLM Inference Scheduling
UMass Amherst, USA
Automatic Model-Hardware Co-Adaptation for Heterogeneous AI Accelerators
Jul 14, 2026Tian Chen, Mingheng Mi, Pu Wang +2AI Accelerator Inference LLM Inference
Pazhou Laboratory (Huangpu) · Unaffiliated
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Jul 14, 2026Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28Efficient Multimodal Inference KV Caching
MiMo Team, Xiaomi
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference Scheduling KV-Cache Management
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference Scheduling LLM Inference Serving
The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States
Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
Jun 28, 2026Victor NorgrenLLM Inference Serving LLM Inference Acceleration
LayerScale, Inc.
KernelSight-LM: A Kernel-Level LLM Inference Simulator
Jun 26, 2026Xiteng Yao, Taeho Kim, Hengzhi Pei +7LLM Inference LLM Inference Serving
Amazon Web Services Santa Clara, CA, USA · Boston University Boston, MA, USA
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Jun 22, 2026Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2LLM Inference Serving Speculative Decoding
PayPal

Oct 28, 2026J/K move · Enter open · S save

GearUp: DAG-aware Acceleration for Serverless Inference Serving via Slack-guided Pre-loading

Qi Qi, Yifan Sui, Hanfei Yu, Hong Zhang+1

University of Massachusetts Amherst Amherst, Massachusetts, USA · Hong Kong University of Science and Technology Hong Kong, China · Stevens Institute of Technology Hoboken, New Jersey, USA+1

LLM Inference Serving

Open paper AskPDF arXiv

Serverless platforms increasingly host machine learning (ML) inference workflows, modeled as directed acyclic graphs (DAGs) of interdependent functions. Beyond container startup, each ML node must load libraries, model weights, and CUDA kernels before it can run. These costs accumulate along dependent DAG paths and place initialization on the end-to-end critical path. Existing systems optimize one half of this problem. DAG-agnostic pre-loading prepares nodes the request may never reach, wasting host and GPU memory. DAG-aware serving routes and scales whole functions, leaving the loading stages inside a node untouched. This paper presents GearUp, a serverless DAG inference serving system that removes ML-node loading from the end-to-end critical path at low resource cost. GearUp profiles the loading stages of each node and schedules them inside the slack time the DAG exposes, hiding initialization behind upstream execution. It reuses initialized instances within and across requests to avoid redundant cold starts. A probabilistic, traffic-adaptive keep-alive policy sets how long idle instances are retained, balancing memory cost against cold-start risk under bursty load. On the AMBER Alert and FINRA-ML workflows with NVIDIA L40S GPUs, GearUp reduces median end-to-end latency by 64.5% and 75.9% against SMIless, for median speedups of 3.1x and 10.0x over cold execution. Integrated GPU and host memory allocation fall by up to 83.9% and 94.6% against eager pre-loading in InstaInfer. GearUp achieves 4.2x strong scaling on four workers.