LLM Inference Scheduling

LLM: Large Language Model

Ask about this topic

Overview

LLM inference scheduling decides which requests run, where they run, and how GPU time and KV-cache memory are shared to meet latency goals while serving many users. Common approaches include continuous batching, chunked prefill, prefill–decode disaggregation, cache-aware placement, and scheduling by predicted output length or request deadlines; newer systems also use online learning, queueing models, and optimization with performance guarantees. Evaluation increasingly covers production traces, long-context requests, and multi-step agent workflows, where routing, tool execution, and cache reuse affect end-to-end completion time. A central tension is that gains in throughput or cache locality can worsen tail latency or fairness, while prediction errors and changing workloads make fixed policies unreliable.

Latest papers

CardsList

166 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

  1. ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios

    Mar 10, 2026Xinyi Hu, Yuhao Shen, Baolin Zhang +6Speculative DecodingLLM Inference Acceleration

    Qwen Applications Business Group of Alibaba · Zhejiang University

  2. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

    Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network InferenceLLM Inference Acceleration

    Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China

  3. Adapter-Augmented Bandits for Online Multi-Constrained Multi-Modal Inference Scheduling

    Mar 6, 2026Xianzhi Zhang, Yue Xu, Yinlin Zhu +4LLM Inference SchedulingMulti-Armed Bandits

    School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · School of Computing, Macquarie University, NSW 2109, Australia

  4. RouteGoT: Node-Adaptive Routing for Cost-Efficient Graph of Thoughts Reasoning

    Mar 6, 2026Yuhang Liu, Ruijie Wang, Yunlong Chu +4LLM Reasoning with GraphsEfficient Language Model Reasoning

    School of New Media and Communication, Tianjin University, Tianjin, China · School of Computer Science and Engineering, Beihang University, Beijing, China · Shanghai Jiao Tong University, Shanghai, China

  5. SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference

    Mar 5, 2026Luchang Li, Dongfang Li, Bozhao Gong +1Disaggregated LLM ServingPrefill-Decode Disaggregation

    Kingsoft Cloud

  6. SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

    Mar 3, 2026Sunghyeon Woo, Ahreum Seo, Jaegwang Lee +6Disaggregated LLM ServingLLM Inference Acceleration

    NAVER Cloud

  7. Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

    Mar 1, 2026Mengqi Liao, Lu Wang, Chaoyun Zhang +6LLM Inference AccelerationLLM Inference Scheduling

    School of Computer Science and Technology, Beijing Jiaotong University · Microsoft · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence

  8. SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud

    Feb 27, 2026Hariz Yet, Nguyen Thanh Tam, Mao V. Ngo +5LLM Inference ServingLLM Inference

    Singapore University of Technology and Design (SUTD)

  9. FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving

    Feb 18, 2026Chia-chi Hsieh, Zan Zong, Xinyang Chen +3LLM ServingLLM Inference Scheduling

    Tsinghua University, Beijing, China · University of Science and Technology Beijing, Beijing, China

  10. ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

    Feb 14, 2026Hao Kang, Ziyang Li, Weili Xu +7LLM Agent OrchestrationLLM Inference Scheduling

    University of Illinois Urbana-Champaign · 5Together AI · 1Georgia Institute of Technology

  11. Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model

    Feb 8, 2026Tianyi Wang, Huawei Fan, Yuanchao Shu +2LLM SecurityAdversarial Attacks on LLMs

    Zhejiang University

  12. Multi-Agentic AI for Fairness-Aware and Accelerated Multi-modal Large Model Inference in Real-world Mobile Edge Networks

    Feb 6, 2026Haiyuan Li, Hari Madhukumar, Shuangyi Yan +2Efficient Multimodal InferenceLLM Inference Scheduling

    Smart Internet Lab, Department of Electrical and Electronic Engineering, University of Bristol, BS8 1QU, U.K.

  13. Ranking Before Serving: Low-Latency LLM Serving via Pairwise Learning-to-Rank

    Sep 25, 2025Yiheng Tao, Yihe Zhang, Matthew Dearing +4LLM ServingLearning to Rank

    University of Illinois Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA

  14. LLM Serving Optimization with Variable Prefill and Decode Lengths

    Aug 8, 2025Meixuan Wang, Yinyu Ye, Zijie ZhouLLM ServingLLM Inference Scheduling

    Department of Computer Science and Technology, Tsinghua University · Department of Management Science and Engineering, Stanford University & Department of Industrial Engineering and Decision Analytics, HKUST · Department of Industrial Engineering and Decision Analytics, HKUST

  15. Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

    Apr 7, 2025Zonghang Li, Tao Li, Wenjiao Feng +8On-Device Language Model InferenceLLM Inference

    MBZUAI · UESTC · City University of Hong Kong +1

  16. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

    Jun 24, 2024Ruoyu Qin, Zheming Li, Weiran He +4LLM ServingDisaggregated LLM Serving

    Moonshot AI · Tsinghua University