LLM Inference Scheduling

LLM: Large Language Model

LLM inference scheduling decides which requests run, where they run, and how GPU time and KV-cache memory are shared to meet latency goals while serving many users. Common approaches include continuous batching, chunked prefill, prefill–decode disaggregation, cache-aware placement, and scheduling by predicted output length or request deadlines; newer systems also use online learning, queueing models, and optimization with performance guarantees. Evaluation increasingly covers production traces, long-context requests, and multi-step agent workflows, where routing, tool execution, and cache reuse affect end-to-end completion time. A central tension is that gains in throughput or cache locality can worsen tail latency or fairness, while prediction errors and changing workloads make fixed policies unreliable.

Latest papers

166 papers39 papers in the last four weeks, up 95% on the four weeks before. 0.2% of all new papers.

Compute Where it Counts: Self Optimizing Language Models
May 11, 2026Yash Akhauri, Mohamed S. AbdelfattahLLM Inference Efficiency RL for Language Models
Cornell University
Regulating Branch Parallelism in LLM Serving
May 7, 2026Swapnil Gandhi, Siva Hari, William J. Dally +1LLM Serving LLM Inference Scheduling
1Stanford University · 2NVIDIA
Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
May 7, 2026Saksham Rathi, Preeti, Mythili VutukuruLLM Inference Scheduling KV-Cache Management
Indian Institute of Technology Bombay, Mumbai, India
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
May 2, 2026Zijie ZhouLLM Serving LLM Inference
Department of Industrial Engineering and Decision Analytics, HKUST
SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
May 1, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Inference Efficiency LLM Agent Orchestration
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines
Apr 16, 2026Otto White, Marcel Wagenländer, Britannio Jarrett +7LLM Inference LLM Inference Scheduling
Imperial College London · Independent Researcher
MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems
Apr 14, 2026Yifei Wang, Hancheng Ye, Yechen Xu +8LLM Inference Scheduling Heterogeneous Computing
Duke University
Active Imitation Learning for Thermal- and Kernel-Aware LFM Inference on 3D S-NUCA Many-Cores
Apr 13, 2026Yixian Shen, Chaoyao Shen, Jan Deen +3LLM Inference Scheduling Imitation Learning
University of Amsterdam, Netherlands · Southeast University, China · University of Thessaly, Greece
Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
Apr 13, 2026Zhuolun Dong, Junyu CaoLLM Inference Scheduling LLM Serving
Department of Information, Risk, and Operations Management, The University of Texas at Austin.
COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels
Apr 12, 2026Bo Ma, Jinsong Wu, Weiqi YanLLM Inference Scheduling State Space Models
Auckland University of Technology; Resideo Technologies, Inc. · Guilin University of Electronic Technology · Auckland University of Technology
A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs
Apr 10, 2026Chen Zhang, Yan Ding, Haotian Wang +3LLM Inference Acceleration LLM Inference Scheduling
Hunan University Changsha, China
Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
Apr 9, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM Serving LLM Inference Efficiency
vLLM Semantic Router Project · MBZUAI · McGill University +2
B-PASTE: Beam-Aware Pattern-Guided Speculative Execution for Resource-Constrained LLM Agents
Apr 9, 2026Yanfei SongLLM Agent Workflow Optimization LLM Inference Scheduling
Independent Researcher
Squeeze Evolve: Unified Multi-Model Orchestration for Verifier-Free Evolution
Apr 9, 2026Monishwaran Maheswaran, Leon Lakhani, Zhongzhu Zhou +16Inference-Time Optimization Evolutionary Optimization
UC Berkeley · Together AI · UT Austin +2
Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Apr 8, 2026Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2LLM Serving LLM Inference
KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden
Scalable Joint Resource Allocation for SLO-Constrained LLM Inference in Heterogeneous GPU Clouds
Apr 8, 2026Jiaming Cheng, Duong Tung NguyenLLM Serving LLM Inference Scheduling
The Ira A. Fulton Schools of Engineering, Arizona State University, Tempe, AZ, USA.
Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agents
Apr 6, 2026Neharika Jali, Anupam Nayak, Gauri JoshiMulti-Turn Language Model Reasoning Efficient Language Model Reasoning
Carnegie Mellon University
Adaptive Parallel Monte Carlo Tree Search for Efficient Test-time Compute Scaling
Apr 1, 2026Hongbeen Kim, Juhyun Lee, Sanghyeon Lee +2Monte Carlo Tree Search LLM Inference Acceleration
School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, South Korea.
Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
Apr 1, 2026Haoyu Zheng, Yongqiang Zhang, Fangcheng Fu +7LLM Inference Scheduling LLM Uncertainty Estimation
School of Computer Science, Wuhan University, Wuhan, China · Dameng Database Co., Ltd., Wuhan, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China +2
CoLLM: Continuous Adaptation for SLO-Aware LLM Serving on Shared GPU Clusters
Mar 31, 2026Shaoyuan Huang, Yunfeng Zhao, Na Yan +5LLM Post-Training LLM Inference Scheduling
Tianjin University, Tianjin, China · King’s College London, London, United Kingdom · Paiou Cloud Computing (Shanghai) Company, Ltd., Shanghai, China
Heddle: A Distributed Orchestration System for Agentic RL Rollout
Mar 30, 2026Zili Zhang, Yinmin Zhong, Chengxu Yang +5LLM Agent Orchestration Agentic RL
School of Computer Science, Peking University · Independent Researcher
Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
Mar 29, 2026Rajveer Bachkaniwala, Chengqi Luo, Richard So +2LLM Inference LLM Inference Serving
Georgia Tech, Atlanta, USA
Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference
Mar 27, 2026Konstantinos Papaioannou, Thaleia Dimitra DoudaliLLM Serving LLM Inference Scheduling
IMDEA Software Institute, Universidad Politécnica de Madrid
Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Mar 25, 2026Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu +3LLM Routing LLM Inference Scheduling
LinkedIn · MIT
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
Mar 23, 2026Kangqi Ni, Wenyue Hua, Xiaoxiang Shi +3LLM Serving LLM Inference Scheduling
University of North Carolina, Chapel Hill · Microsoft · Carnegie Mellon University +2
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Mar 19, 2026Yifan Sui, Han Zhao, Rui Ma +6LLM Agent Serving LLM Tool Use
Shanghai Jiao Tong University · Microsoft Research · Stevens Institute of Technology +2
Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
Mar 13, 2026Donglin YuEfficient Multimodal Inference LLM Serving
Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Models
Mar 11, 2026Yuedong Yang, Xiwen Wei, Mustafa Munir +1Efficient Language Model Reasoning Multimodal CoT Reasoning
Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin

May 11, 2026J/K move · Enter open · S save

Compute Where it Counts: Self Optimizing Language Models

Yash Akhauri, Mohamed S. Abdelfattah

Cornell University

LLM Inference Efficiency RL for Language Models LLM Pruning LLM Inference Scheduling Inference-Time Compute Allocation

Open paper AskPDF arXiv

Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attention), typically applying a uniform computation budget to every generated token. In practice, token difficulty varies widely, so static compression can over-compute on easy steps and under-compute on hard ones. We study dynamic budget allocation for autoregressive decoding: learning how much computation to spend per token from within a single model. Self-Optimizing Language Models (SOL) pair a frozen LLM with a lightweight policy network that reads the LLM hidden state and selects a discrete efficiency action at each decode step. Actions can jointly control (i) token-level attention sparsity, (ii) structured activation pruning in the MLP, and (iii) activation quantization bit-width, while leaving the base model weights unchanged. We train the policy with group-relative policy optimization on teacher-forced episodes: the token sequence is fixed, while we sample multiple compute schedules (i.e., "counterfactual" schedules that vary only the efficiency actions for the same token path) and compare their likelihoods under the same supervision. Our reward trades off language-model quality against soft penalties that encourage episode-average budget usage to match a requested target. Across model variants and compute regimes, SOL improves quality at matched budget over static allocation and strong random schedule search, offering a complementary axis for inference-efficiency optimization. SOL discovers a better quality-efficiency pareto-front across all our experiments and improves MMLU accuracy by up to 7.3% over uniform budget allocation strategies.