LLM Request Scheduling

LLM: Large Language Model

LLM request scheduling decides which prompts to admit, batch, route, or pause so inference servers can serve requests efficiently under limited compute and memory. Common approaches include shortest-job-first and length-aware ordering, prompt-based length prediction, continuous batching, and policies that account for KV-cache use, prefill and decode stages, or service-level priorities. Newer methods also coordinate scheduling with cache retention, prefill/decode disaggregation, and online traffic estimates, with evaluations spanning chat, code, retrieval, and bursty workloads. A key tension is that predictions and policies that improve average latency or throughput can worsen tail latency, fairness, or stability, especially when workload lengths, hardware, and memory demands vary.

Latest papers

25 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
Oct 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +6LLM Serving LLM Inference Scheduling
HKUST
Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
Oct 11, 2026Ashman Mehra, Akshan Agarwal, Aryeh CarmiLLM Inference Scheduling LLM Request Scheduling
Carnegie Mellon University, School of Computer Science
From Overloaded to Guaranteed: High-Throughput Multi-SLO Enforcement for LoRA-Assisted On-Premise LLM Deployment
Oct 4, 2026Zeshen Zhang, Han Zhao, Weihao Cui +9LLM Serving LLM Request Scheduling
Shanghai Jiao Tong University, China · Ant Group, China
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
Oct 2, 2026Yuyang Dai, Rana Shahout, Mahmood SharifLLM Inference Scheduling LLM Serving
Florida State University · MIT · Tel Aviv University
Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving
Sep 30, 2026Haoyu Zheng, Fangcheng Fu, Binhang Yuan +6Cost-Aware Inference Diffusion Model Serving
Wuhan University · Shanghai Jiao Tong University · The Hong Kong University of Science and Technology +2
End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
Sep 15, 2026Zhen Li, Jun Cai, Haoran Gao +2LLM Inference Scheduling LLM Request Scheduling
Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, H3G 1M8, Canada · Department of Computer Science, Hang Seng University of Hong Kong, Hong Kong SAR
Prefix Sharing Is a Sorting Problem
Sep 12, 2026Rong HePrefix Caching LLM Request Scheduling
Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
Aug 17, 2026Anders Vestrum, Arya Raeesi, Hanna RoedLLM Inference Scheduling LLM Request Scheduling
UC Berkeley, EECS Department
LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm
Aug 6, 2026Anjali Gangadhar Katageria, Shobha Rani, Raghu Nandan SenguptaLLM Inference Scheduling Online Resource Allocation
Department of Mathematics, Indian Institute of Technology Dharwad, Dharwad, Karnataka, India · Department of Management Sciences, Indian Institute of Technology Kanpur, Kanpur, Uttar Pradesh, India
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4LLM Inference Scheduling Diffusion Model Serving
University of Illinois Chicago, USA
When Words Predict Workload
Jul 6, 2026Anubhab BanerjeeOn-Device Language Model Inference LLM Inference
Nokia Germany
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference Scheduling LLM Inference Serving
The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
Jul 2, 2026Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj +2LLM Serving Prefill-Decode Disaggregation
Microsoft
Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM Serving LLM Inference Scheduling
Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2
OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning
May 29, 2026Zhenghua Bao, Fengya Tian, Chris Zhang +3Model Selection LLM Routing
Continuum AI
Harmonizing Real-Time Constraints and Long-Horizon Reasoning: An Asynchronous Agentic Framework for Dynamic Scheduling
May 28, 2026Shijie Cao, Yuan Yuan, Jing LiuLanguage Model-Based Control Real-Time Scheduling
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shenzhen Loop Area Institute, Shenzhen, China · Qingdao Research Institute, Beihang University +3
HAServe: Horizon-Aware Scheduling and Memory Co-Design for Efficient LLM Serving
May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM Serving KV Caching
School of Computer Science, Wuhan University · Amazon
Joint Optimization of Trajectory Control, Resource Allocation, and Task Offloading for Multi-UAV-Assisted IoV
May 6, 2026Maoxin Ji, Qiong Wu, Pingyi Fan +4Trajectory Optimization Intelligent Transportation Systems
School of Internet of Things Engineering, Jiangnan University, Wuxi 214122, China · School of Information Engineering, Jiangxi Provincial Key Laboratory of Advanced Signal Processing and Intelligent Communications, Nanchang University, Nanchang 330031, China · Department of Electronic Engineering, State Key laboratory of Space Network and Communications, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China +4
HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads
Apr 18, 2026Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah +3LLM Inference Efficiency AI Coding Agents
Sperix Labs · VIA Cybersecurity Lab, KNUST · Quantum and Assistive Technologies Lab, KNUST
Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
Apr 13, 2026Zhuolun Dong, Junyu CaoLLM Inference Scheduling LLM Serving
Department of Information, Risk, and Operations Management, The University of Texas at Austin.
Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Inference Efficiency LLM Routing
vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
Feb 2, 2026Seoungbin Bae, Junyoung Son, Dabeen LeeLLM Serving LLM Routing
Department of Industrial & Systems Engineering, KAIST · Graduate School of Data Science, KAIST · Department of Mathematical Sciences, Seoul National University
Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank
Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM Serving Learning to Rank
University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA
LLM Serving Optimization with Variable Prefill and Decode Lengths
Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM Serving LLM Inference Scheduling
Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST

Oct 13, 2026J/K move · Enter open · S save

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi+5

HKUST

LLM Serving LLM Inference Scheduling LLM Request Scheduling Prefix Caching

Open paper AskPDF arXiv

Prefix reuse improves LLM inference by sharing the KV cache of common prompt prefixes. This helps only if the scheduler can execute a matched branch in the current worker round. Schedulers assume that a matched prefix is ready, which holds for full attention but fails for hybrid LLMs. There, a branch (the consumer) may need recurrent state that another branch (the producer) is still computing in the same round. The native scheduler stops its waiting-queue scan at this blocked consumer, and the resulting head-of-line blocking creates execution bubbles: idle batch slots that independent work could fill. We introduce RadixFlow, which decouples prefix matching from state readiness. Its state-ready traversal turns a global scan barrier into a local wait: only the consumer is blocked until the next round, while feasible independent branches keep filling the batch under the native state check. On five hybrid models, RadixFlow lowers mean latency by 3% to 24% and worker rounds by 11% to 44% at almost unchanged reuse. On the larger Qwen3.5 models, ablations show that most of the gain comes from traversal, which adds little on a full-attention control. Heavier rounds limit how much of this saving reaches latency.