LLM Serving

LLM: Large Language Model

Large language model serving studies how to run inference efficiently for many concurrent requests, balancing response speed, throughput, cost, and service-level objectives. Common approaches include continuous batching and prefill–decode scheduling, request routing, and managing or compressing KV caches; newer systems also use disaggregated hardware, speculative decoding, and workload-aware power control. Evaluations use request traces and production-like workloads to measure time to first token, token latency, throughput, quality, and resource use across models and hardware. A central challenge is that workloads and output lengths vary, so policies that improve throughput or cost can worsen tail latency, fairness, or response quality.

Latest papers

216 papers31 papers in the last four weeks, up 94% on the four weeks before. 0.2% of all new papers.

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation
Apr 10, 2026Peng DingLLM Serving
University of Chicago
Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
Apr 9, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Serving LLM Inference Efficiency
vLLM Semantic Router Project · MBZUAI · McGill University +2
Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions
Apr 9, 2026Jing Wang, Yu-Yang Qian, Ke Xue +3LLM Serving Robust Estimation
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Apr 8, 2026Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2LLM Serving LLM Inference
KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden
Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds
Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM Serving LLM Inference Scheduling
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.
Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start
Apr 8, 2026Xueshen Liu, Yongji Wu, Yuncheng Yao +3LLM Serving LLM Inference Acceleration
University of Michigan · UC Berkeley · Duke
ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache
Apr 7, 2026Shao Wang, Rui Ren, Lin GuiLLM Serving LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China
ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving
Mar 31, 2026Annette Taberner-MillerAdaptive Model Routing LLM Routing
Independent Researcher.
Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference
Mar 27, 2026Konstantinos Papaioannou, Thaleia Dimitra DoudaliLLM Serving LLM Inference Scheduling
IMDEA Software Institute, Universidad Politécnica de Madrid
Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodology
Mar 26, 2026Zhou Hanlin, Chan Huah YongLLM Routing LLM Evaluation
School of Computer Sciences, Universiti Sains Malaysia (USM), Penang, Malaysia · Xiamen Software Vocational & Technical College, Xiamen, China
GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
Mar 26, 2026Shakya Jayakody, Youpeng Zhao, Chinmay Dhanraj Nehate +1LLM Serving LLM Inference
University of Central Florida, Orlando, FL, USA.
Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation
Mar 25, 2026Sebastian Nowak, Jann-Frederick Laß, Narine Mesropyan +8LLM Serving LLM Security
Department of Diagnostic and Interventional Radiology, University Hospital Bonn · Department of Research-IT, University Hospital Bonn
StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving
Mar 24, 2026Azam NouriLLM Serving LLM Inference Efficiency
Department of Science, Technology & Mathematics, Lincoln University, USA
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
Mar 23, 2026Kangqi Ni, Wenyue Hua, Xiaoxiang Shi +3LLM Serving LLM Inference Scheduling
University of North Carolina, Chapel Hill · Microsoft · Carnegie Mellon University +2
SEAR: Schema-Based Evaluation and Routing for LLM Gateways
Mar 20, 2026Zecheng Zhang, Han Zheng, Yue XuLLM Routing LLM Evaluation
Strukto.AI, USA · Infron.AI, USA
Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective
Mar 17, 2026Noppanat Wadlom, Junyi Shen, Yao LuLLM Serving LLM Agent Workflow Optimization
National University of Singapore Singapore
Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
Mar 13, 2026Donglin YuEfficient Multimodal Inference LLM Serving
98 $\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router
Mar 13, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Routing LLM Serving
vLLM Semantic Router Project · MBZUAI · McGill University +2
Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
Mar 12, 2026Md. Monzurul Amin Ifath, Israat HaqueParallel LLM Inference LLM Inference
Dalhousie University, Halifax, NS, Canada
Improving LLM Performance Through Black-Box Online Tuning: A Case for Adding System Specs to Factsheets for Trusted AI
Mar 11, 2026Yonas Atinafu, Henry Lin, Robin CohenLLM Inference Serving LLM Serving
David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada
FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation
Mar 10, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Serving On-Device Language Model Inference
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park
Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving
Feb 27, 2026Ferran Agullo, Joan Oliveras, Chen Wang +5LLM Inference Efficiency LLM Serving
Barcelona Supercomputing Center (BSC), Barcelona, Spain · Universitat Politècnica de Catalunya - BarcelonaTech (UPC), Barcelona, Spain · IBM Research, New York, USA
LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
Feb 26, 2026Jaehong Cho, Hyunmin Choi, Guseul Heo +1LLM Serving Disaggregated LLM Serving
School of Computing KAIST Daejeon, South Korea
vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
Feb 23, 2026Xunzhuo Liu, Huamin Chen, Samzong Lu +30Multimodal Model Routing LLM Routing
Steve
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1Coding Agent Benchmarks AI Coding Agents
AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems
Feb 22, 2026Emmanuel BamideleMemory Management for LLM Agents Agent Memory Management
Department of Computer Science, Georgia Institute of Technology
FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
Feb 18, 2026Chia-chi Hsieh, Zan Zong, Xinyang Chen +3LLM Serving LLM Inference Scheduling
Tsinghua University, Beijing, China · University of Science and Technology Beijing, Beijing, China
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
Test-Time Compute Games
Jan 29, 2026Ander Artola Velasco, Dimitrios Rontogiannis, Stratis Tsirtsis +1LLM Serving Cost-Aware Inference
Max Planck Institute for Software Systems, Kaiserslautern, Germany · Hasso Plattner Institute, Potsdam, Germany

Apr 10, 2026J/K move · Enter open · S save

LLM-Rosetta: A Hub-and-Spoke Intermediate Representation for Cross-Provider LLM API Translation

Peng Ding

University of Chicago

LLM Serving

Open paper AskPDF arXiv

The rapid proliferation of Large Language Model (LLM) providers--each exposing proprietary API formats--has created a fragmented ecosystem where applications become tightly coupled to individual vendors. Switching or bridging providers requires

O(N^2)

bilateral adapters, impeding portability and multi-provider architectures. We observe that despite substantial syntactic divergence, the major LLM APIs share a common semantic core: the practical challenge is the combinatorial surface of syntactic variations, not deep semantic incompatibility. Based on this finding, we present LLM-Rosetta, an open-source translation framework built on a hub-and-spoke Intermediate Representation (IR) that captures the shared semantic core--messages, content parts, tool calls, reasoning traces, and generation controls--in a 9-type content model and 10-type stream event schema. A modular Ops-composition converter architecture enables each API standard to be added independently. LLM-Rosetta supports bidirectional conversion (provider-to-IR-to-provider) for both request and response payloads, including chunk-level streaming with stateful context management. We implement converters for four API standards (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Google GenAI), covering the vast majority of commercial providers. Empirical evaluation demonstrates lossless round-trip fidelity, correct streaming behavior, and sub-100 microsecond conversion overhead--competitive with LiteLLM's single-pass approach while providing bidirectionality and provider neutrality. LLM-Rosetta passes the Open Responses compliance suite and is deployed in production at Argonne National Laboratory. Code is available at https://github.com/Oaklight/llm-rosetta.