LLM Inference Scheduling

LLM: Large Language Model

Ask about this topic

Overview

LLM inference scheduling decides which requests run, where they run, and how GPU time and KV-cache memory are shared to meet latency goals while serving many users. Common approaches include continuous batching, chunked prefill, prefill–decode disaggregation, cache-aware placement, and scheduling by predicted output length or request deadlines; newer systems also use online learning, queueing models, and optimization with performance guarantees. Evaluation increasingly covers production traces, long-context requests, and multi-step agent workflows, where routing, tool execution, and cache reuse affect end-to-end completion time. A central tension is that gains in throughput or cache locality can worsen tail latency or fairness, while prediction errors and changing workloads make fixed policies unreliable.

Latest papers

CardsList

166 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

  1. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

    Jul 30, 2026Banruo Liu, Haoran Qiu, Íñigo Goiri +3LLM InferenceAI Coding Agents

    UIUC · Microsoft Azure Research · Microsoft Azure

  2. Incast-Free MoE Rate-Based Scheduling

    Jul 28, 2026Evyatar Cohen, Jose Yallouz, Alexander Shpiner +3LLM Inference SchedulingMixture-of-Experts Inference

    Technion · Nvidia · UC Berkeley

  3. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10LLM Inference SchedulingSpeculative Decoding for Diffusion Language Models

    Tencent Inc.

  4. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

    Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference EfficiencyLLM Inference Scheduling

    Department of Computer Science, Florida State University

  5. Kalypso: Relational LLM Serving

    Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2LLM ServingLLM Inference Scheduling

    UMass Amherst, USA

  6. WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

    Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic RLReinforcement Learning

  7. Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

    Jul 19, 2026Utopia Meng, Unicornt Zhao, Derek Li +2LLM ServingLLM Inference Scheduling

  8. An MLIR-Based Compilation Method for Large Language Models

    Jul 17, 2026Pengchao Hu, Zhibin Xin, Yifan Chen +3LLM ServingLLM Inference Scheduling

    Sophgo Inc.

  9. Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

    Jul 13, 2026Weijia Han, Lisha QuLLM QuantizationGPU Kernel Optimization

    University of Washington

  10. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1On-Device Language Model InferenceLLM Inference Scheduling

    School of Computer Science, Nanjing University Nanjing, China

  11. BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving

    Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4LLM Inference SchedulingDiffusion Model Serving

    University of Illinois Chicago, USA

  12. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference SchedulingKV-Cache Management

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group

  13. When Words Predict Workload

    Jul 6, 2026Anubhab BanerjeeOn-Device Language Model InferenceLLM Inference

    Nokia Germany

  14. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference SchedulingLLM Inference Serving

    The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States

  15. EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

    Jun 25, 2026Carrie ChenMulti-Token PredictionLLM Inference Scheduling

    Cornell University

  16. Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models

    Jun 22, 2026Jiawei Xu, Minghui Liu, Aakriti Agrawal +2Masked Diffusion ModelsLanguage Model Decoding

    University of Maryland, College Park · University of California, Los Angeles.

  17. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM ServingLLM Inference Scheduling

    Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China

  18. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM ServingInference-Time Optimization

    Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2

  19. INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration

    Jun 9, 2026Ahasan Kabir, Mengxin Zheng, Qian LouMulti-Agent System OptimizationMulti-Agent Orchestration

    University of Central Florida

  20. AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding

    Jun 7, 2026Yingxuan Ren, Yuxuan Lou, Yong Liu +4LLM Inference SchedulingBlockwise Diffusion Language Models

    1National University of Singapore · University of Southampton

  21. NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference

    Jun 2, 2026Mubarak Adetunji OjewaleLLM InferenceKV Caching

    Cloud Competency Centre, National College of Ireland

  22. MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency

    Jun 2, 2026Saptarshi Mitra, Yifan Zhang, Rachid Karami +5LLM Inference EfficiencyMulti-Agent System Optimization

    University of California, Irvine, USA · University of Southern California, Los Angeles, USA · Intel, USA

  23. Lodestar: An Online-Learning LLM Inference Router

    May 31, 2026Gangmuk Lim, Wanyu Zhao, Brighten Godfrey +3Inference-Time OptimizationLLM Inference Scheduling

    UIUC · Bytedance · University of Edinburgh

  24. Threshold-Based Exclusive Batching for LLM Inference

    May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU AccelerationLLM Inference

    The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1

  25. TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

    May 28, 2026Jiaming Fan, Daming Cao, Canchen Huang +6Speculative DecodingLLM Inference Acceleration

    Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1

  26. ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU

    May 21, 2026Aman Sunesh, Ali Alshehhi, Hivansh DhakneLLM InferenceLLM Serving

    Department of Computer Engineering NYU Abu Dhabi Abu Dhabi, United Arab Emirates · Courant Institute of Mathematical Sciences New York University New York, United States · Department of Computer Engineering New York University New York, United States

  27. Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

    May 20, 2026Yicheng Feng, Xin Tan, Yangtao Deng +3LLM ServingDisaggregated LLM Serving

    The Chinese University of Hong Kong · Anuttacon · StepFun

  28. TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

    May 19, 2026Zhiben Chen, Youpeng Zhao, Yang Sui +2Expert OffloadingLLM Inference Scheduling

    University of Central Florida · Rice University

  29. Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM pools

    May 13, 2026Shivam Patel, Akaash R. Parthasarathy, Akash Dhasade +2LLM ServingLLM Routing

    Carnegie Mellon University · Microsoft

  30. Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

    May 13, 2026Victor NorgrenTransformer InferenceLLM Inference

    LayerScale, Inc.