LLM Inference Scheduling

LLM: Large Language Model

LLM inference scheduling decides which requests run, where they run, and how GPU time and KV-cache memory are shared to meet latency goals while serving many users. Common approaches include continuous batching, chunked prefill, prefill–decode disaggregation, cache-aware placement, and scheduling by predicted output length or request deadlines; newer systems also use online learning, queueing models, and optimization with performance guarantees. Evaluation increasingly covers production traces, long-context requests, and multi-step agent workflows, where routing, tool execution, and cache reuse affect end-to-end completion time. A central tension is that gains in throughput or cache locality can worsen tail latency or fairness, while prediction errors and changing workloads make fixed policies unreliable.

Latest papers

165 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference
Oct 29, 2026Wenjie Huang, Jun Zhang, Hongyaoxing Gu +10Efficient Multimodal Inference Multimodal Large Language Models
Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences +4
Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill
Oct 28, 2026Kwun Fung Lau, Frost Mitchell, Kanya Mo +1LLM Inference Acceleration Causal Attention
Intel Corp., USA.
DualPace: Decoupled Scheduling and Cooperative GPU--CPU Decoding for LLM Serving
Oct 24, 2026YuXuan Lin, KeQiang Wu, Kingsum ChowLLM Serving LLM Inference
Ningbo Global Innovation Center, Zhejiang University, Ningbo, China
LLMs in Cloud: A Discussion on Architecture, Orchestration, and Inference
Oct 24, 2026Jannatun Noor, Athar Noor Mohammad Rafee, Md. Sadman Haque +3LLM Serving LLM Inference
Computing for Sustainability and Social Good (C2SG) Research Group, Department of Computer Science and Engineering, United International University, Dhaka, Dhaka 1212, Bangladesh. · Department of Computer Science and Engineering, School of Data & Computational Sciences, BRAC University, Dhaka, Dhaka 1212, Bangladesh.
The Scheduler Is a Hidden Retriever: How Asynchrony Shapes Evidence Acquisition in Search Agents
Oct 19, 2026Ruiyang Ren, Yuhao Wang, Yinan Zhang +1LLM Inference Scheduling Evidence Retrieval
Nanyang Technological University Singapore
ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference
Oct 18, 2026Mobina Kashaniyan, Ali JannesariTest-Time Scaling Energy-Aware LLM Inference
Department of Computer Science Iowa State University, USA
Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets
Oct 17, 2026Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra +10LLM Serving LLM Inference Scheduling
Tandemn · University of Illinois Urbana-Champaign · UC Berkeley +3
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
Oct 14, 2026Daijin HuLLM Serving LLM Agent Serving
Beihang University
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada
FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving
Oct 13, 2026Yunfei Feng, Jingkai He, Shiqi Liu +3Efficient Inference Mixture-of-Experts Inference
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University
RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving
Oct 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +6LLM Serving LLM Inference Scheduling
HKUST
Batch-Paced: Fixed-Budget Batch Decoding in Discrete Diffusion Language Models
Oct 13, 2026Shuai Zhang, Zhanchao Zhou, Huachuan Qiu +2Diffusion Language Model Decoding Inference-Time Compute Allocation
Zhejiang University · Westlake University
Stable Token-level Scheduling for Serverless LLM Inference
Oct 13, 2026Hardi Ashishbhai Desai, Balaji N K, Sourav Kanti AddyaLLM Inference Scheduling LLM Inference Serving
Cloud & Smart System Services Lab, Department of Computer Science and Engineering, National Institute of Technology Karnataka, Surathkal, India
Efficient Resource-Shiftable RL Post-training with Generative Rewards
Oct 12, 2026Yanfeng Zhao, Haoyang Li, Sheng Lin +4RL Post-Training LLM Post-Training
Shanghai Jiao Tong University · Peking University · Central China Normal University
Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning
Oct 12, 2026Chiheng Lou, Xiangwei Deng, Bangjun Xiao +4Prefill-Decode Disaggregation Agentic RL
†School of Computer Science, Peking University · ‡LLM-Core Xiaomi
Lightweight Scheduling Policies for Continuously Batched LLM Serving Across GPU Architectures
Oct 11, 2026Ashman Mehra, Akshan Agarwal, Aryeh CarmiLLM Inference Scheduling LLM Request Scheduling
Carnegie Mellon University, School of Computer Science
Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving
Oct 10, 2026Burak Topcu, Musa Cim, Zuhal Ozturk +1LLM Inference Scheduling Carbon-Aware Computing
Department of Computer Science and Engineering, The Pennsylvania State University, State College, PA 16801, USA
S $^3$ R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding
Oct 9, 2026Yuhang He, Hongyu Ge, Zihan Lin +2Speculative Decoding LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou)
TokenRouter: Efficient Serving System for Token-Level LLM Routing
Oct 8, 2026Tianyu Fu, Tengxuan Liu, Ruoxi Wang +4LLM Serving LLM Routing
Tsinghua University · Carnegie Mellon University
One Model, Multiple Compute Budgets: Multi-Ratio ConceptMoE for Flexible Compute Across Turns
Oct 8, 2026Yiyang Fu, Zihao Huang, Dongchen Han +4Token Compression Mixture-of-Experts Models
Peking University · ByteDance Seed · Tsinghua University +1
Cascadia: Resident 975B MoE Inference on Eleven AI PCs
Oct 5, 2026Tate Berenbaum, Matias Parij, Muthaiah VenkatachalamLLM Inference Mixture-of-Experts Inference
Not Community Labs Inc. · Intel Corporation
Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill
Oct 5, 2026Liquan Liu, Yifan Zhang, Bowei XuLLM Inference Scheduling Speculative Decoding
Zeraix
PhaseGate: Phase-Aware CPU Retrieval Scheduling for On-Device LLMs on Unified Memory
Oct 3, 2026Seoyoon Yum, Sehoon KimLLM Inference Scheduling On-Device Language Model Inference
KAIST
FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving
Oct 2, 2026Kartik Ramesh, Kaidi Fu, Zihan Zheng +4LLM Serving Prefill-Decode Disaggregation
University of Illinois Urbana-Champaign · IBM Research
Jumping the Line: Exploiting Length Predictions in LLM Scheduling
Oct 2, 2026Yuyang Dai, Rana Shahout, Mahmood SharifLLM Inference Scheduling LLM Serving
Florida State University · MIT · Tel Aviv University
EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures
Oct 2, 2026Yuhai Long, Yuanxin Wei, Kai Wu +3On-Device Inference LLM Inference Scheduling
School of Computer Science and Engineering, Sun Yat-sen University · Sun Yat-sen University School of Computer Science and Engineering Guangzhou, China · China Mobile Internet Company Ltd +1
JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs
Oct 2, 2026Haoran Zhang, Dongjun Kim, Seohyeon Cha +4LLM Inference Scheduling LLM Routing
University of Texas at Austin · DEVCOM Army Research Laboratory
Inference Auctions
Sep 30, 2026Keegan Harris, Siddharth Prasad, Asher Trockman +2Mechanism Design Automated Bidding
University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research +1
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA

Oct 29, 2026J/K move · Enter open · S save

Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang+9

Zhejiang University · Alibaba Group · University of the Chinese Academy of Sciences+4

Efficient Multimodal Inference Multimodal Large Language Models LLM Inference Scheduling

Open paper AskPDF arXiv

Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves and progressively prefills unselected tokens in source order using unused compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).