LLM Inference Scheduling

LLM: Large Language Model

Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 82

All topics
CardsList

  1. Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

    Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference SchedulingLLM Agent Workflow Optimization

    Harbin Institute of Technology (Shenzhen)

  2. AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill

    Oct 5, 2026Liquan Liu, Yifan Zhang, Bowei XuLLM Inference SchedulingSpeculative Decoding

    Zeraix

  3. Inference Auctions

    Sep 30, 2026Keegan Harris, Siddharth Prasad, Asher Trockman +2Mechanism DesignAutomated Bidding

    University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research +1

  4. Characterizing High Bandwidth Flash for LLM Serving

    Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference EfficiencyLLM Serving

    University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA

  5. Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure

    Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM ServingLLM Inference Scheduling

    Not Community Labs Inc. · Intel Corporation

  6. Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits

    Sep 29, 2026Gaurav Agarwal, Ashish Garg, Isha SinghalLLM ServingLLM Inference

  7. SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving

    Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM ServingTensor Parallelism

    State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  8. WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse

    Sep 28, 2026Aaryam SharmaLLM Inference EfficiencyLLM Serving

    Baseten

  9. PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

    Sep 28, 2026Jiaan Zhu, Wei Gao, Youhui Bai +6Agentic RLAsynchronous RL

    USTC China

  10. Nereus: Adaptive Parallelism for LLM Post-Training

    Sep 28, 2026Songlin Jiang, Tuo Shi, Sitong Zhang +3LLM Inference SchedulingRL Post-Training

    Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

  11. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

    Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache OffloadingLong-Context Language Model Inference

    Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China

  12. Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier

    Sep 28, 2026Linye Wei, Shutian Zheng, Haoyu Zeng +1LLM Inference SchedulingSpeculative Decoding

    Institute for Artificial Intelligence, Peking University · School of Integrated Circuits, Peking University · College of Engineering, Peking University +2

  13. Spexis: Speculative Lookahead Scheduling for LLM Inference

    Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference EfficiencyLLM Inference Scheduling

    Seoul National University · Hanyang University

  14. FORGE: Form-Optimal Routing of Grounded Evidence for Frozen LLM Agents

    Sep 28, 2026Xi Xiao, Yunbei Zhang, Chen Liu +7LLM GroundingLLM Inference Scheduling

    University of Alabama at Birmingham

  15. Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

    Sep 28, 2026Murtaza Rangwala, Richard O. Sinnott, Rajkumar BuyyaLLM InferenceLLM Inference Scheduling

    Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia

  16. Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving

    Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM ServingLLM Inference Scheduling

    Meta Platforms

  17. Efficient Iterative Retrieval with Heterogeneous Batching

    Sep 21, 2026Dohyun Park, Hubertus Franke, Daniel G. Waddington +2Retrieval-Augmented GenerationLLM Serving

    University of Illinois Urbana-Champaign · IBM Research

  18. PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

    Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference EfficiencyLLM Inference

    Illinois Institute of Technology · University of Wisconsin–Milwaukee

  19. Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

    Sep 16, 2026Mobina Kashaniyan, Ali JannesariLLM Inference EfficiencyLLM Inference

    Department of Computer Science Iowa State University, USA

  20. Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving

    Sep 16, 2026Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3LLM ServingLLM Inference Scheduling

    University of California, Berkeley

  21. ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

    Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model InferenceLLM Inference Scheduling

    University of Southern California, Los Angeles, CA, USA

  22. MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

    Sep 14, 2026Tiancheng Zhang, Yulin Chen, Yunfeng Zhao +3Disaggregated LLM ServingLLM Inference Scheduling

    College of Intelligence and Computing, Tianjin University, Tianjin, China · The International Joint Institute of Tianjin University, Tianjin University, Fuzhou, China · Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics, Tianjin, China

  23. SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading

    Sep 14, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Expert OffloadingLLM Inference Scheduling

    University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China

  24. Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows

    Sep 12, 2026Bochao Feng, Jianjiang Li, Haojie Wang +4LLM Inference SchedulingAgentic Workflows

    University of Science and Technology Beijing Beijing, China · Qiyuan Laboratory Beijing, China · Tsinghua University Beijing, China

  25. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

    Sep 11, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache OffloadingKV Caching

    Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland

  26. Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

    Sep 7, 2026Siyu Song, Qi Bai, Jinbo Hao +3LLM ServingLLM Inference Scheduling

    School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Engineering, Jiangsu Ocean University

  27. CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs

    Sep 1, 2026Chaohui Guo, Michel Klein, Zhisheng HuangReinforcement LearningLLM Inference Scheduling

    Department of Computer Science, Vrije Universiteit Amsterdam, Netherlands

  28. DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference

    Aug 31, 2026Xiaoyang Lu, Belthangady Akash Vi Narayana Pai, Xian-He SunEfficient Neural Network InferenceLLM Inference Scheduling

    Illinois Institute of Technology Chicago, Illinois, USA

  29. The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem

    Aug 25, 2026Elioth SanabriaLLM ServingLLM Inference Scheduling

    Department of Decision and Technology Analytics Lehigh University - College of Business

  30. TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

    Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert ParallelismExpert Load Balancing

    KlingAI Research

  31. Scheduling Mixed RL Rollouts Beyond Prefix Locality

    Aug 11, 2026Zetao Hong, Song Yuan, Yuanhao Ding +4LLM Inference SchedulingKV-Cache Management

    State Key Laboratory of Novel Software Technology, Nanjing University · StepFun

  32. ImpactHO: Importance-Aware KV Cache Transfer for Multi-User Edge LLM Handover

    Aug 11, 2026Minwoo Kim, Soochang Song, Namyoon Lee +2KV CachingEdge Computing

    Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical and Computer Engineering, Ajou University, Suwon 16499, South Korea

  33. MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices

    Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model InferenceLLM Inference Scheduling

    Kyungpook National University Daegu, Republic of Korea

  34. Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

    Aug 10, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLLM InferenceLLM Inference Scheduling

    Chair for AI in Healthcare and Medicine, Technical University of Munich (TUM), Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1

  35. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

    Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4LLM ServingLLM Inference Scheduling

    The University of British Columbia · Microsoft Azure Research · NVIDIA

  36. LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

    Aug 6, 2026Anjali Gangadhar Katageria, Shobha Rani, Raghu Nandan SenguptaLLM Inference SchedulingOnline Resource Allocation

    Department of Mathematics, Indian Institute of Technology Dharwad, Dharwad, Karnataka, India · Department of Management Sciences, Indian Institute of Technology Kanpur, Kanpur, Uttar Pradesh, India

  37. BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

    Aug 6, 2026Guanqiao Qu, Shuo Chen, Qian Chen +2Edge ComputingLLM Inference Scheduling

    Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. · Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K.

  38. Architectural Implications of Agentic AI Workflows

    Aug 5, 2026Jirong Yang, Peizhe Liu, Chaojie Zhang +1LLM Inference SchedulingAgentic Workflows

    The University of Texas at Austin · Microsoft Azure

  39. PACE: Adaptive Budget Allocation for Time-Efficient Embodied Planning

    Aug 4, 2026Yuchen Huang, Xijiang Ying, Zhenhua Ma +14Large Language Model-Based Robot PlanningLLM Inference Scheduling

    ZTE Corporation · Shanghai, China

  40. Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO

    Aug 3, 2026Ngoc Hung Nguyen, Bjorn LandfeldtReinforcement LearningTransformer-Based RL

    Department of Electrical and Information Technology, Lund University, P.O. Box 117, Lund 22100, Sweden

  41. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    Aug 3, 2026Cunchen Hu, Liangliang Xu, Tian Liu +9Energy-Efficient MLDisaggregated LLM Serving

    China Telecom Cloud Computing Research Institute · Xidian University · University of Science and Technology of China +1

  42. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

    Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5LLM ServingLLM Fine-Tuning

    McGill University · Canada · MBZUAI +3

  43. Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale

    Jul 30, 2026Banruo Liu, Haoran Qiu, Íñigo Goiri +3LLM InferenceAI Coding Agents

    UIUC · Microsoft Azure Research · Microsoft Azure

  44. Incast-Free MoE Rate-Based Scheduling

    Jul 28, 2026Evyatar Cohen, Jose Yallouz, Alexander Shpiner +3LLM Inference SchedulingMixture-of-Experts Inference

    Technion · Nvidia · UC Berkeley

  45. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10LLM Inference SchedulingSpeculative Decoding for Diffusion Language Models

    Tencent Inc.

  46. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

    Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference EfficiencyLLM Inference Scheduling

    Department of Computer Science, Florida State University

  47. Kalypso: Relational LLM Serving

    Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2LLM ServingLLM Inference Scheduling

    UMass Amherst, USA

  48. WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

    Jul 19, 2026Ryan Xu, Atlas Zhao, David Bao +1Agentic RLReinforcement Learning

  49. Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

    Jul 19, 2026Utopia Meng, Unicornt Zhao, Derek Li +2LLM ServingLLM Inference Scheduling

  50. An MLIR-Based Compilation Method for Large Language Models

    Jul 17, 2026Pengchao Hu, Zhibin Xin, Yifan Chen +3LLM ServingLLM Inference Scheduling

    Sophgo Inc.

  51. Decomposing Runtime, Kernel, and Quantization Speedups via a Matched FP16 Intermediate: A Hardware-Conditioned Case Study on Four NVIDIA RTX A5000 GPUs

    Jul 13, 2026Weijia Han, Lisha QuLLM QuantizationGPU Kernel Optimization

    University of Washington

  52. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1On-Device Language Model InferenceLLM Inference Scheduling

    School of Computer Science, Nanjing University Nanjing, China

  53. BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving

    Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4LLM Inference SchedulingDiffusion Model Serving

    University of Illinois Chicago, USA

  54. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference SchedulingKV-Cache Management

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group

  55. When Words Predict Workload

    Jul 6, 2026Anubhab BanerjeeOn-Device Language Model InferenceLLM Inference

    Nokia Germany

  56. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference SchedulingLLM Inference Serving

    The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States

  57. EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

    Jun 25, 2026Carrie ChenMulti-Token PredictionLLM Inference Scheduling

    Cornell University

  58. Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models

    Jun 22, 2026Jiawei Xu, Minghui Liu, Aakriti Agrawal +2Masked Diffusion ModelsLanguage Model Decoding

    University of Maryland, College Park · University of California, Los Angeles.

  59. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM ServingLLM Inference Scheduling

    Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China

  60. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6LLM ServingInference-Time Optimization

    Cornell University, Computer Science Department, NY, USA · Microsoft Azure System Research, WA, USA · Cornell University, Electrical and Computer Engineering Department, NY, USA +2