LLM Inference Serving

LLM: Large Language Model

Large language model inference serving studies how systems process prompts and generate tokens under limits on GPU compute and memory, especially the KV cache. Common approaches include continuous batching, request scheduling and routing, prefill-decode disaggregation, speculative decoding, and cache reuse; systems are evaluated using latency and throughput metrics, often against workload traces and service-level objectives. Work increasingly addresses stateful, multi-turn agents and heterogeneous deployments spanning GPUs, edge devices, and data centers, where tool calls and changing context complicate prediction and resource allocation. A central tension is balancing cache locality and high utilization against fair, predictable tail latency, while also controlling cost and energy as workloads shift.

Latest papers

74 papers10 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference
Mar 27, 2026Joris Köster, Zixuan Liu, Siavash Khajavi +1LLM Inference Serving Cost-Aware Inference
Department of Computer Science, Aalto University, Finland · Department of Computer Science, Tulane University, LA, USA · Department of Industrial Engineering and Management, Aalto University, Finland
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
Mar 22, 2026Huamin Chen, Xunzhuo Liu, Bowei He +5LLM Routing LLM Inference Serving
vLLM Semantic Router Project · MBZUAI · McGill University +3
Orla: A Library for Serving LLM-Based Multi-Agent Systems
Mar 13, 2026Rana Shahout, Hayder Tirmazi, Minlan Yu +1LLM Agent Orchestration LLM Agent Serving
Harvard University MA, USA · Boston University MA, USA
Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models
Mar 12, 2026Jae-Won Chung, Jeff J. Ma, Jisang Ahn +4LLM Inference Serving
University of Michigan · University of Southern California · Cisco Research
Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI
Mar 11, 2026Yonas Atinafu, Henry Lin, Robin CohenLLM Inference Serving LLM Serving
David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada
Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology
Token Management in Multi-Tenant AI Inference Platforms
Feb 27, 2026William J. CunninghamLLM Inference Serving Fair Resource Allocation
DataRobot, Boston, Massachusetts, USA
SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud
Feb 27, 2026Hariz Yet, Nguyen Thanh Tam, Mao V. Ngo +5LLM Inference Serving LLM Inference
Singapore University of Technology and Design (SUTD)
AI-Paging: Lease-Based Execution Anchoring for Network-Exposed AI-as-a-Service
Feb 17, 2026Mohaned Chraiti, Merve SaimlerAdmittance Control Intent-Based Networking
Electronics Engineering Department, Sabancı University, Türkiye · Ericsson Research, Türkiye
ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Feb 14, 2026Hao Kang, Ziyang Li, Weili Xu +7LLM Agent Orchestration LLM Inference Scheduling
University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology
Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model
Feb 8, 2026Tianyi Wang, Huawei Fan, Yuanchao Shu +2LLM Security Adversarial Attacks on LLMs
Zhejiang University
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA
Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference Efficiency LLM Inference
EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea

Mar 27, 2026J/K move · Enter open · S save

MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng

Department of Computer Science, Aalto University, Finland · Department of Computer Science, Tulane University, LA, USA · Department of Industrial Engineering and Management, Aalto University, Finland

LLM Inference Serving Cost-Aware Inference LLM Routing Memory-Augmented Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline.