LLM Serving

LLM: Large Language Model

Large language model serving studies how to run inference efficiently for many concurrent requests, balancing response speed, throughput, cost, and service-level objectives. Common approaches include continuous batching and prefill–decode scheduling, request routing, and managing or compressing KV caches; newer systems also use disaggregated hardware, speculative decoding, and workload-aware power control. Evaluations use request traces and production-like workloads to measure time to first token, token latency, throughput, quality, and resource use across models and hardware. A central challenge is that workloads and output lengths vary, so policies that improve throughput or cost can worsen tail latency, fairness, or response quality.

Latest papers

214 papers31 papers in the last four weeks, up 94% on the four weeks before. 0.2% of all new papers.

InferD: A Honeypot Framework for AI Inference Infrastructure
Oct 29, 2026Hicham Hammouchi, Gabriele LenziniAI Agent Security LLM Security
SnT, University of Luxembourg Luxembourg, Luxembourg
Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill
Oct 28, 2026Kwun Fung Lau, Frost Mitchell, Kanya Mo +1LLM Inference Acceleration Causal Attention
Intel Corp., USA.
GAUSS: A Workload-Aware Analytical and Monte-Carlo Simulator for LLM Inference Serving
Oct 27, 2026Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis +3LLM Serving LLM Inference Serving
Microsoft
Governing Enterprise AI Traffic at Scale: Design and Operation of a Policy-Driven, Multi-Vendor LLM Gateway on Envoy
Oct 26, 2026Mithun Sasidharan, Srinivasan Manoharan, Rupesh Thakur +3LLM Routing LLM Serving
PayPal Inc.
Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers
Oct 26, 2026Xiaoyang Zhang, Yuejun Yan, Qingang Zhang +1Carbon-Aware Computing LLM Routing
Nanyang Technological University · Alibaba Group
DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving
Oct 24, 2026YuXuan Lin, KeQiang Wu, Kingsum ChowLLM Serving LLM Inference
Ningbo Global Innovation Center, Zhejiang University, Ningbo, China
LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM Serving LLM Inference
Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.
Gandalf: Flash Shall Not Die! Increasing Flash Storage Lifetime for KV Cache Offloading in LLM Serving
Oct 22, 2026Maharani A. P. Irawan, Kuntai Du, Jiayi Yao +4KV-Cache Offloading KV-Cache Management
The University of Chicago, USA · TensorMesh Inc., USA · Samsung Semiconductor, USA
FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments
Oct 17, 2026Dimitrios Liakopoulos, Prasoon Sinha, Neeraja J. YadwadkarLLM Serving Cost-Aware Inference
The University of Texas at Austin
Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets
Oct 17, 2026Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra +10LLM Serving LLM Inference Scheduling
Tandemn · University of Illinois Urbana-Champaign · UC Berkeley +3
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
Oct 14, 2026Daijin HuLLM Serving LLM Agent Serving
Beihang University
RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
Oct 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +6LLM Serving LLM Inference Scheduling
HKUST
Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
Oct 11, 2026Ashman Mehra, Akshan Agarwal, Aryeh CarmiLLM Inference Scheduling LLM Request Scheduling
Carnegie Mellon University, School of Computer Science
The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving
Oct 11, 2026Mukund Athmaram, Manoj Rao, Liang MaLLM Serving LLM Inference Acceleration
Advanced Micro Devices Inc. (AMD) · Cornell University
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
Oct 11, 2026Haomiao Qiu, Yining Tang, Zijia Song +3LLM Serving KV-Cache Reuse
Harbin Institute of Technology, Shenzhen · Great Bay University · Zhejiang University
Accelerating Cross-Model LLM Inference via Attention Map Reuse
Oct 10, 2026Zhan Jin, Yuze Luo, Bin Cui +1LLM Serving LLM Inference Acceleration
School of Computer Science & Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University · Key Lab of High Confidence Software Technologies (MOE), Peking University
Taking Tokenization off the Host for Agentic LLM Serving
Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3Tokenization LLM Serving
KTH Royal Institute of Technology, Stockholm, Sweden
KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse
Oct 9, 2026Taiga Ikeda, Daisuke Miyashita, Jun DeguchiKV-Cache Reuse KV-Cache Management
Kioxia Corporation, Japan
TokenRouter: Efficient Serving System for Token-Level LLM Routing
Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM Serving LLM Routing
Tsinghua University · Carnegie Mellon University
From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment
Oct 4, 2026Zeshen Zhang, Han Zhao, Weihao Cui +9LLM Serving LLM Request Scheduling
Shanghai Jiao Tong University, China · Ant Group, China
FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
Oct 2, 2026Kartik Ramesh, Kaidi Fu, Zihan Zheng +4LLM Serving Prefill-Decode Disaggregation
University of Illinois Urbana-Champaign · IBM Research
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
Oct 2, 2026Yuyang Dai, Rana Shahout, Mahmood SharifLLM Inference Scheduling LLM Serving
Florida State University · MIT · Tel Aviv University
Exact Memory-Time Optimization for Prefix-Cached Language Model Serving
Oct 2, 2026Shivam GuptaLLM Serving Memory-Efficient Inference
Independent Researcher, Delhi, India
On-Premises Multi-Course RAG Tutoring for Business Education: Hardware-Software Trade-offs in a Campus AI Tutor
Oct 1, 2026Sidney Shapiro, Joshua LindemannAI in Education Generative AI in Education
Dhillon School of Business, University of Lethbridge
Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
Sep 30, 2026Sietse SchelpeLLM Serving Long-Context Language Model Inference
Corbenic AI
Taming Speculative Search for Test-Time Scaling in LLM Serving
Sep 30, 2026Jinwoo Jeong, Woohyung Choi, Myeongjae Jeon +1LLM Inference Acceleration LLM Serving
Korea University · Korea University Seoul, Republic of Korea · POSTECH +1
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM Serving LLM Inference Scheduling
Not Community Labs Inc. · Intel Corporation
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Sep 29, 2026Gaurav Agarwal, Ashish Garg, Isha SinghalLLM Serving LLM Inference
You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference
Sep 29, 2026Liang He, Jingbo Wen, Yixiong Chen +4LLM Serving Cost-Aware Inference
University of Sydney · Johns Hopkins University · Stanford University +1

Oct 29, 2026J/K move · Enter open · S save

InferD: A Honeypot Framework for AI Inference Infrastructure

Hicham Hammouchi, Gabriele Lenzini

SnT, University of Luxembourg Luxembourg, Luxembourg

AI Agent Security LLM Security LLM Serving

Open paper AskPDF arXiv

Large language model serving stacks, commercial inference APIs, vector databases, and agent orchestration gateways are now ex- posed on the public internet and indexed by commodity scanners within hours of deployment. The threat characteristics of this sur- face are only beginning to be measured, and the honeypot frame- works that could measure it were designed for conventional net- work services rather than for AI infrastructure. In fact, emulating an AI service is a distinct deception problem: the observable behavior of the service, including model inventories, token formats, tool-call protocols, and authorization flows, must be reproduced faithfully, while the underlying computation, model inference and tool ex- ecution, must never occur. We present InferD, a low-interaction honeypot framework organized around a single principle we call deception without inference: every request an adversary can observe is answered from static response pools, and nothing an adversary submits is ever executed. InferD decomposes the emulation into four deception planes, transport, service protocol, vulnerability surface, and agentic session, adds a provenance plane that links a planted credential to evidence of downstream model-mediated processing.