LLM Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 144

All topics

TokenRouter: Efficient Serving System for Token-Level LLM Routing
Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM Serving LLM Routing
Tsinghua University · Carnegie Mellon University
Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
Sep 30, 2026Sietse SchelpeLLM Serving Long-Context Language Model Inference
Corbenic AI
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM Serving LLM Inference Scheduling
Not Community Labs Inc. · Intel Corporation
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Sep 29, 2026Gaurav Agarwal, Ashish Garg, Isha SinghalLLM Serving LLM Inference
You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference
Sep 29, 2026Liang He, Jingbo Wen, Yixiong Chen +4LLM Serving Cost-Aware Inference
University of Sydney · Johns Hopkins University · Stanford University +1
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing
Sep 29, 2026Guannan Lai, Gelin Bian, Hao-Xuan Ma +5LLM Serving Cost-Aware Inference
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · SinapisAI +1
Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisions
Sep 28, 2026Tianyao Shi, Xipeng Shen, Yi DingLLM Inference Efficiency LLM Serving
Elmore Family School of Electrical and Computer Engineering, Purdue University, USA
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Sep 28, 2026Aaryam SharmaLLM Inference Efficiency LLM Serving
Baseten
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Sep 28, 2026Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu +2LLM Serving LLM Inference
Department of Next Generation Semiconductor Convergence and Open Sharing System (COSS), Seoul National University, Seoul 08826, South Korea · Van-Lang Institute of Semiconductor Technology (VIST), Hanoi, Vietnam · Efficient Computation Research Group, Moreh Vietnam +1
Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving
Sep 27, 2026Jiantong Jiang, Yue Yang, Peiyu Yang +1LLM Serving KV-Cache Management
University of Melbourne · Maincode
Efficient Iterative Retrieval with Heterogeneous Batching
Sep 21, 2026Dohyun Park, Hubertus Franke, Daniel G. Waddington +2Retrieval-Augmented Generation LLM Serving
University of Illinois Urbana-Champaign · IBM Research
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
Sep 16, 2026Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3LLM Serving LLM Inference Scheduling
University of California, Berkeley
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs
Sep 15, 2026Dushyant RajputLLM Inference Efficiency LLM Serving
AltSlate Labs LLP
HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving
Sep 8, 2026Han JinLLM Serving Adversarial Attacks on LLMs
Independent Researcher
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Sep 7, 2026Siyu Song, Qi Bai, Jinbo Hao +3LLM Serving LLM Inference Scheduling
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Engineering, Jiangsu Ocean University
Architecting Conversational Data Systems for Stateless LLM APIs: The Hydration Proxy Pattern
Sep 1, 2026Joseph AxisaLLM Grounding LLM Serving
Google Cloud · California, USA
From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix
Sep 1, 2026Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov +11LLM Serving Instruction Following
T-Tech
Prediction-Assisted Pricing and Admission for LLM APIs with Stochastic Token Consumption
Sep 1, 2026Patrick WongLLM Serving Learning-Augmented Algorithms
The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem
Aug 25, 2026Elioth SanabriaLLM Serving LLM Inference Scheduling
Department of Decision and Technology Analytics Lehigh University - College of Business
Keep, Customize, or Exit: Default Design and Token Pricing in LLM Reasoning Services
Aug 13, 2026Ahmet Bugra Gundogan, Yigit Turkmen, Melih BastopcuLLM Serving Inference-Time Compute Allocation
Department of Electrical and Electronics Engineering
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
Aug 12, 2026Natchanon Pollertlam, Witchayut KornsuwannawitLLM Serving LLM Agent Memory
Bricks Technology · Thailand
Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter
Aug 11, 2026Rima Mittal, Ankit Gubrani, Satyanarayana KakolluLLM Inference Efficiency LLM Serving
LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
Aug 9, 2026Shuowei Jin, Xueshen Liu, Jiaxin Shan +4LLM Serving
University of Michigan · ByteDance
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4LLM Serving LLM Inference Scheduling
The University of British Columbia · Microsoft Azure Research · NVIDIA
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
Aug 4, 2026Xiang Li, Pengcheng Wang, Huazheng Wang +1LLM Inference Efficiency LLM Serving
1Purdue University · 2Oregon State University
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
Aug 4, 2026Haiqiang Zhang, Yuanqing Lei, Wanting Li +2Retrieval-Augmented Generation LLM Serving
ETH Zurich · Columbia University · National University of Singapore
LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
Aug 4, 2026Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh +1LLM Serving LLM Inference Acceleration
SWAT Lab., Polytechnique Montr´eal, Montr´eal, Canada

Oct 8, 2026, cs.CLJ/K move · Enter open · S save

TokenRouter: Efficient Serving System for Token-Level LLM Routing

Tianyu Fu, Tengxuan Liu, Ruoxi Wang, Yixin Dong+3

Tsinghua University · Carnegie Mellon University

LLM Serving LLM Routing LLM Inference Scheduling

Open paper AskPDF arXiv

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Momentum

Related topics

Latest papers 144