LLM Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 144

All topics

Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
May 22, 2026Ashok Chandrasekar, Jason KrambergerLLM Evaluation LLM Serving
Google
ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU
May 21, 2026Aman Sunesh, Ali Alshehhi, Hivansh DhakneLLM Inference Efficiency LLM Serving
Department of Computer Engineering NYU Abu Dhabi Abu Dhabi, United Arab Emirates · Courant Institute of Mathematical Sciences New York University New York, United States · Department of Computer Engineering New York University New York, United States
Argo: Efficient Importance Labeling for Enterprise Email Systems
May 20, 2026Siddhant Ray, Ganesh Ananthanarayanan, Kevin Chian +5LLM Inference Efficiency LLM Serving
University of Chicago · Microsoft · University of Chicago / TensorMesh
PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
May 20, 2026Can Hankendi, Rana Shahout, Minlan Yu +1LLM Inference Efficiency LLM Serving
Boston University · Electrical and Computer Engineering Department · Boston, MA, USA +3
Frontier: Towards Comprehensive and Accurate LLM Inference Simulation
May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM Serving Disaggregated LLM Serving
The Chinese University of Hong Kong · Anuttacon · StepFun
PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
May 20, 2026Yiqi Zhang, Fangzheng Jiao, Tian Tang +13LLM Inference Efficiency LLM Serving
National University of Singapore · Shanghai Qiji Zhifeng Co., Ltd. · Beihang University +3
Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
May 19, 2026Mert Yildiz, Pietro Spadaccino, Alexey Rolich +2LLM Inference Efficiency LLM Serving
Sapienza University of Rome, Rome, Italy
An Interpretable Latency Model for Speculative Decoding in LLM Serving
May 14, 2026Linghao Kong, Megan Flynn, Michael Peng +3LLM Serving Speculative Decoding
MIT · Red Hat AI
PreFT: Prefill-only finetuning for efficient inference
May 14, 2026Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu +4Fine-Tuning LLM Serving
Stanford University · Tilde Research
MinT: Managed Infrastructure for Training and Serving Millions of LLMs
May 13, 2026Mind Lab, :, Song Cao +60LLM Serving Adapter Tuning
KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
May 13, 2026Zedong Liu, Xinyang Ma, Dejun Luo +9LLM Serving KV Caching
University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Shanghai Jiao Tong University
MARLIN: Multi-Agent Game-Theoretic Reinforcement Learning for Sustainable LLM Inference in Cloud Datacenters
May 13, 2026H. Moore, S. Qi, D. Milojicic +2LLM Serving LLM Inference
Colorado State University Fort Collins, CO, USA · Hewlett Packard Labs Milpitas, CA, USA
SOMA: Efficient Multi-turn LLM Serving via Small Language Model
May 11, 2026Xueqi Cheng, Qiong Wu, Zhengyi Zhou +3LLM Serving Small Language Models
Florida State University · AT&T Chief Data Office · Louisiana State University +1
Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stack
May 11, 2026Prathamesh Vasudeo Naik, Naresh Dintakurthi, Yue WangLLM Serving Financial Fraud Detection
Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing
May 11, 2026Yunze Zhao, Yibo Zhao, Yuchen Zhang +2LLM Serving LLM Security
University of Maryland · New York University
PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
May 10, 2026Bing Xie, Zhipeng Wang, Masahiro Tanaka +1LLM Serving KV Caching
†Corresponding author.
PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM Serving KV Caching
School of Computer Science, Wuhan University · Amazon
Regulating Branch Parallelism in LLM Serving
May 7, 2026Swapnil Gandhi, Siva Hari, William J. Dally +1LLM Serving LLM Inference Scheduling
1Stanford University · 2NVIDIA
VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?
May 7, 2026Keisuke Kamahori, Shihang Li, Simon Peter +1Multi-Agent LLM Systems LLM Serving
University of Washington
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
May 5, 2026Yixuan Mei, Zikun Li, Zixuan Chen +5LLM Serving Cost-Aware Inference
Carnegie Mellon University · Peking University
When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning
May 5, 2026Jiaqi Wei, Xuehang Guo, Pengfei Yu +5LLM Serving LLM Reasoning
Zhejiang University · College of William and Mary · University of Illinois Urbana-Champaign +4
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
May 4, 2026Jincheng Xie, Yawen Ling, Qi Xiao +4LLM Serving Speculative Decoding
1Tsinghua University · 2AI Infra Team at JDT · 3JD iCity, JD Technology, JD Intelligent Cities Research
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
May 3, 2026Nikita Agrawal, Ruben MayerLLM Serving KV Caching
University of Bayreuth
Human-Less LLM Serving: Quantifying the Human Tax on Throughput
May 3, 2026Jianhui Lian, Li Chen, Dan Li +1LLM Inference Efficiency LLM Serving
Tsinghua SIGS · Beijing Zhongguancun Laboratory · Tsinghua University
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
May 3, 2026Yipin Guo, Siddharth JoshiLLM Serving KV Caching
University of Notre Dame IN, USA
MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert Parallelism LLM Serving
University of Virginia · 2Snowflake AI Research · 3Harvard University
Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
May 2, 2026Zijie ZhouLLM Serving LLM Inference
Department of Industrial Engineering and Decision Analytics, HKUST
Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving
Apr 30, 2026Junsun Choi, Sam Son, Sunjin Choi +5LLM Serving Cost-Aware Inference
UC Berkeley · UC Berkeley & ICSI
FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving
Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language Models LLM Serving
TU Berlin · Berlin, Germany

May 22, 2026, cs.AIJ/K move · Enter open · S save

Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks

Ashok Chandrasekar, Jason Kramberger

Google

LLM Evaluation LLM Serving LLM Inference Benchmark Design

Open paper AskPDF arXiv

As Large Language Models (LLMs) transition from research environments to production deployments, evaluating their performance against strict Service Level Objectives (SLOs) has become critical. However, current evaluation methodologies suffer from severe measurement bias at scale. We demonstrate that widely used benchmarking utilities rely on single-process, asyncio-driven architectures that introduce fundamental client-side queuing bottlenecks under high concurrency. By modeling the benchmarking client as an

M/G/1

queue, we mathematically demonstrate how the Python Global Interpreter Lock (GIL) artificially inflates Time to First Token (TTFT) and Time Per Output Token (TPOT) metrics as request rates scale. To resolve this systematic inaccuracy, we propose an unbiased, multi-process evaluation framework that effectively distributes client-side load, ensuring negligible queuing overhead. Furthermore, we formalize a composite metric, Normalized Time Per Output Token (NTPOT), to robustly amortize end-to-end latency, including prefill and scheduling delays across sequence lengths. Our empirical evaluation demonstrates that this methodology successfully isolates pure serving engine performance, enabling accurate, reproducible profiling of LLMs at production scales exceeding thousands of queries per second.

Momentum

Related topics

Latest papers 144