LLM Inference Scheduling

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 83

All topics

Scheduling Mixed RL Rollouts Beyond Prefix Locality
Aug 11, 2026Zetao Hong, Song Yuan, Yuanhao Ding +4LLM Inference Scheduling KV-Cache Management
State Key Laboratory of Novel Software Technology, Nanjing University · StepFun
ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover
Aug 11, 2026Minwoo Kim, Soochang Song, Namyoon Lee +2KV Caching Edge Computing
Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model Inference LLM Inference Scheduling
Kyungpook National University Daegu, Republic of Korea
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Aug 10, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLLM Inference LLM Inference Scheduling
Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4LLM Serving LLM Inference Scheduling
The University of British Columbia · Microsoft Azure Research · NVIDIA
LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm
Aug 6, 2026Anjali Gangadhar Katageria, Shobha Rani, Raghu Nandan SenguptaLLM Inference Scheduling Online Resource Allocation
Department of Mathematics, Indian Institute of Technology Dharwad, Dharwad, Karnataka, India · Department of Management Sciences, Indian Institute of Technology Kanpur, Kanpur, Uttar Pradesh, India
BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks
Aug 6, 2026Guanqiao Qu, Shuo Chen, Qian Chen +2Edge Computing LLM Inference Scheduling
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. · Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K.
Architectural Implications of Agentic AI Workflows
Aug 5, 2026Jirong Yang, Peizhe Liu, Chaojie Zhang +1LLM Inference Scheduling Agentic Workflows
The University of Texas at Austin · Microsoft Azure
PACE: Adaptive Budget Allocation for Time-Efficient Embodied Planning
Aug 4, 2026Yuchen Huang, Xijiang Ying, Zhenhua Ma +14Large Language Model-Based Robot Planning LLM Inference Scheduling
ZTE Corporation · Shanghai, China
Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO
Aug 3, 2026Ngoc Hung Nguyen, Bjorn LandfeldtReinforcement Learning Transformer-Based RL
Department of Electrical and Information Technology, Lund University, P.O. Box 117, Lund 22100, Sweden
Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling
Aug 3, 2026Cunchen Hu, Liangliang Xu, Tian Liu +9Energy-Efficient ML Disaggregated LLM Serving
China Telecom Cloud Computing Research Institute · Xidian University · University of Science and Technology of China +1
DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs
Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5LLM Serving LLM Fine-Tuning
McGill University · Canada · MBZUAI +3
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
Jul 30, 2026Banruo Liu, Haoran Qiu, Íñigo Goiri +3LLM Inference AI Coding Agents
UIUC · Microsoft Azure Research · Microsoft Azure
Incast-Free MoE Rate-Based Scheduling
Jul 28, 2026Evyatar Cohen, Jose Yallouz, Alexander Shpiner +3LLM Inference Scheduling Mixture-of-Experts Inference
Technion · Nvidia · UC Berkeley
AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding
Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10LLM Inference Scheduling Speculative Decoding for Diffusion Language Models
Tencent Inc.
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference Efficiency LLM Inference Scheduling
Department of Computer Science, Florida State University
Kalypso: Relational LLM Serving
Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2LLM Serving LLM Inference Scheduling
UMass Amherst, USA
WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning
Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic RL Reinforcement Learning
Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs
Jul 19, 2026Utopia Meng, Unicornt Zhao, Derek Li +2LLM Serving LLM Inference Scheduling
An MLIR-Based Compilation Method for Large Language Models
Jul 17, 2026Pengchao Hu, Zhibin Xin, Yifan Chen +3LLM Serving LLM Inference Scheduling
Sophgo Inc.
Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs
Jul 13, 2026Weijia Han, Lisha QuLLM Quantization GPU Kernel Optimization
University of Washington
Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1On-Device Language Model Inference LLM Inference Scheduling
School of Computer Science, Nanjing University Nanjing, China
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4LLM Inference Scheduling Diffusion Model Serving
University of Illinois Chicago, USA
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference Scheduling KV-Cache Management
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group
When Words Predict Workload
Jul 6, 2026Anubhab BanerjeeOn-Device Language Model Inference LLM Inference
Nokia Germany
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference Scheduling LLM Inference Serving
The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models
Jun 22, 2026Jiawei Xu, Minghui Liu, Aakriti Agrawal +2Masked Diffusion Models Language Model Decoding
University of Maryland, College Park · University of California, Los Angeles.
Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM Serving LLM Inference Scheduling
Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China
Beyond Prediction: Tail-Aware Scheduling for LLM Inference
Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM Serving Inference-Time Optimization
Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2

Aug 11, 2026, cs.DCJ/K move · Enter open · S save

Scheduling Mixed RL Rollouts Beyond Prefix Locality

Zetao Hong, Song Yuan, Yuanhao Ding, Yibo Zhu+3

State Key Laboratory of Novel Software Technology, Nanjing University · StepFun

LLM Inference Scheduling KV-Cache Management LLM Routing RL Post-Training

Open paper AskPDF arXiv

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.

Momentum

Related topics

Latest papers 83