LLM Inference Scheduling

LLM: Large Language Model

Overview

LLM inference scheduling decides which requests run, where they run, and how GPU time and KV-cache memory are shared to meet latency goals while serving many users. Common approaches include continuous batching, chunked prefill, prefill–decode disaggregation, cache-aware placement, and scheduling by predicted output length or request deadlines; newer systems also use online learning, queueing models, and optimization with performance guarantees. Evaluation increasingly covers production traces, long-context requests, and multi-step agent workflows, where routing, tool execution, and cache reuse affect end-to-end completion time. A central tension is that gains in throughput or cache locality can worsen tail latency or fairness, while prediction errors and changing workloads make fixed policies unreliable.

Latest papers

CardsList

197 papers30 papers in the last four weeks, down 25% on the four weeks before. 0.2% of all new papers.

  1. Harvesting Expert-Parallel Bubbles for Fine-Tuning

    Nov 18, 2026Oana Balmau, Saurav Banka, Jiaxuan Chen +5LLM Inference SchedulingMixture-of-Experts Fine-Tuning

  2. Deal With It: On Better Schedules for Listwise Reranking

    Nov 17, 2026Benjamin Clavié, Herumb Shandilya, Hanna Lichtenberg +2LLM Inference SchedulingLLM Reranking

  3. Making Preemption Productive in Long-Output LLM Serving

    Nov 15, 2026Ziji Shi, Xinyi Wan, Chaoyi Ruan +3LLM ServingLLM Inference Scheduling

  4. VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

    Nov 13, 2026Ishaan Datta, Alind Khare, Koushik Kumar Dey +2LLM ServingLLM Inference Scheduling

  5. TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning

    Nov 13, 2026Mathew Jacob, Morgan Borjigin-Wang, Kan Zhu +10RL Post-TrainingLLM Inference Scheduling

  6. Late but Valuable: Why Deadlines Change the Measured Value of LLM Reasoning

    Nov 9, 2026Sina Barazandeh, Benjamin E. Lundell, Jose Lugo-MartinezLanguage Model Reasoning EvaluationLLM Decision-Making

  7. RADE: Risk-Adjusted Deployment Efficiency, A Framework for Routing, Memory, and Escalation in Safe LLM Systems

    Nov 8, 2026Chaymaa Abbas, Kareem Hassani, Mariette AwadLLM SafetySafety-Utility Trade-Off in Language Models

  8. ODSA: Orchestrated Discovery and Self-Aggregation

    Nov 8, 2026Macardle Meng, Haomin ZhangTest-Time ScalingLLM Inference Scheduling

  9. D3Tree: Learning to Prune Block-Diffusion Draft Trees for Efficient Verification

    Nov 8, 2026Hong Nie, Zhangchi Gao, Jun Yu +3Speculative DecodingLLM Inference Scheduling

  10. Adaptive Scheduling Pipeline for Multi-Instance Asynchronous Reinforcement Learning

    Nov 6, 2026Salah Chikhi, Luis Zuin, Entong Li +1LLM Inference SchedulingReinforcement Learning

  11. BlockLock: Reproducible LLM Inference with Fixed Batch Geometry

    Nov 4, 2026Xiuyuan Qi, Pedro Reviriego, Javier Conde +1LLM InferenceLLM Inference Scheduling

  12. Give Up to Get More: Priors, Not Policies, Govern Test-Time Compute Allocation

    Nov 4, 2026Jinhyung BaeAdaptive Reasoning Resource Allocation for Language ModelsInference-Time Compute Allocation

  13. A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

    Nov 4, 2026Muhammad Faizan Anwar, Sina KeshvadiLLM ServingPrefix Caching

  14. Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

    Nov 3, 2026Maximilian Thürmer, Nicolas Scotto di Perto, Eloise Stein +6LLM Inference AccelerationParallel LLM Inference

  15. Asymptotic Optimality of cμcμ-Rule for LLM Inference Scheduling under Memory Constraint

    Nov 3, 2026Zhenghao Li, Shengbo Wang, Nian SiLLM Inference Scheduling

  16. When Should an Agent Refresh? Adaptive Index Reuse for Efficient Long-Horizon Reasoning

    Nov 1, 2026Guangyuan Wang, Yifan Zhang, Shilong LiuLLM Agent EfficiencyLLM Inference Scheduling

  17. Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom

    Oct 31, 2026Zongpu Zhang, Yishu Yin, Xuehai QianLLM ServingLLM Inference Scheduling

  18. LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

    Oct 30, 2026Kwan Yan, Jialun Zhong, Yanzeng LiLLM Output Length PredictionLLM Inference Scheduling

  19. On the Fundamental Limits of Prefill-Based Decoding Length Prediction

    Oct 29, 2026Haowen Shi, Xuzhe Yan, Fahao Chen +1LLM Inference SchedulingAutoregressive Decoding

  20. Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference

    Oct 29, 2026Wenjie Huang, Jun Zhang, Hongyaoxing Gu +10Efficient Multimodal InferenceMultimodal Large Language Models