LLM Inference Scheduling

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 83

All topics

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill
Oct 5, 2026Liquan Liu, Yifan Zhang, Bowei XuLLM Inference Scheduling Speculative Decoding
Zeraix
Inference Auctions
Sep 30, 2026Keegan Harris, Siddharth Prasad, Asher Trockman +2Mechanism Design Automated Bidding
University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research +1
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM Serving LLM Inference Scheduling
Not Community Labs Inc. · Intel Corporation
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Sep 29, 2026Gaurav Agarwal, Ashish Garg, Isha SinghalLLM Serving LLM Inference
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Sep 28, 2026Aaryam SharmaLLM Inference Efficiency LLM Serving
Baseten
PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning
Sep 28, 2026Jiaan Zhu, Wei Gao, Youhui Bai +6Agentic RL Asynchronous RL
USTC China
Nereus: Adaptive Parallelism for LLM Post-Training
Sep 28, 2026Songlin Jiang, Tuo Shi, Sitong Zhang +3LLM Inference Scheduling RL Post-Training
Aalto University · Shenzhen University of Advanced Technology · Zhejiang University
PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding
Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache Offloading Long-Context Language Model Inference
Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China
Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontier
Sep 28, 2026Linye Wei, Shutian Zheng, Haoyu Zeng +1LLM Inference Scheduling Speculative Decoding
Institute for Artificial Intelligence, Peking University · School of Integrated Circuits, Peking University · College of Engineering, Peking University +2
Spexis: Speculative Lookahead Scheduling for LLM Inference
Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference Efficiency LLM Inference Scheduling
Seoul National University · Hanyang University
FORGE: Form-Optimal Routing of Grounded Evidence for Frozen LLM Agents
Sep 28, 2026Xi Xiao, Yunbei Zhang, Chen Liu +7LLM Grounding LLM Inference Scheduling
University of Alabama at Birmingham
Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines
Sep 28, 2026Murtaza Rangwala, Richard O. Sinnott, Rajkumar BuyyaLLM Inference LLM Inference Scheduling
Quantum Cloud Computing and Distributed Systems (qCLOUDS) Lab, School of Computing and Information Systems, The University of Melbourne, Australia
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM Serving LLM Inference Scheduling
Meta Platforms
Efficient Iterative Retrieval with Heterogeneous Batching
Sep 21, 2026Dohyun Park, Hubertus Franke, Daniel G. Waddington +2Retrieval-Augmented Generation LLM Serving
University of Illinois Urbana-Champaign · IBM Research
PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving
Sep 17, 2026Omkar Shewale, Deepak Kumar, Divakar Kumar YadavLLM Inference Efficiency LLM Inference
Illinois Institute of Technology · University of Wisconsin–Milwaukee
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Sep 16, 2026Mobina Kashaniyan, Ali JannesariLLM Inference Efficiency LLM Inference
Department of Computer Science Iowa State University, USA
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
Sep 16, 2026Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3LLM Serving LLM Inference Scheduling
University of California, Berkeley
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA
MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
Sep 14, 2026Tiancheng Zhang, Yulin Chen, Yunfeng Zhao +3Disaggregated LLM Serving LLM Inference Scheduling
College of Intelligence and Computing, Tianjin University, Tianjin, China · The International Joint Institute of Tianjin University, Tianjin University, Fuzhou, China · Faculty of Digital Economics and Managements, Tianjin University of Finance and Economics, Tianjin, China
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Sep 14, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Expert Offloading LLM Inference Scheduling
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China
Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows
Sep 12, 2026Bochao Feng, Jianjiang Li, Haojie Wang +4LLM Inference Scheduling Agentic Workflows
University of Science and Technology Beijing Beijing, China · Qiyuan Laboratory Beijing, China · Tsinghua University Beijing, China
Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Sep 11, 2026Joseph Kanichai, Tiziano De Matteis, Animesh TrivediKV-Cache Offloading KV Caching
Vrije Universiteit Amsterdam Amsterdam, the Netherlands · IBM Research Zurich Zurich, Switzerland
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Sep 7, 2026Siyu Song, Qi Bai, Jinbo Hao +3LLM Serving LLM Inference Scheduling
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Engineering, Jiangsu Ocean University
CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs
Sep 1, 2026Chaohui Guo, Michel Klein, Zhisheng HuangReinforcement Learning LLM Inference Scheduling
Department of Computer Science, Vrije Universiteit Amsterdam, Netherlands
DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference
Aug 31, 2026Xiaoyang Lu, Belthangady Akash Vi Narayana Pai, Xian-He SunEfficient Neural Network Inference LLM Inference Scheduling
Illinois Institute of Technology Chicago, Illinois, USA
The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem
Aug 25, 2026Elioth SanabriaLLM Serving LLM Inference Scheduling
Department of Decision and Technology Analytics Lehigh University - College of Business
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
Aug 13, 2026Jie Li, Chenxin Jia, Jinliang Shen +5Expert Parallelism Expert Load Balancing
KlingAI Research

Oct 5, 2026, cs.AIJ/K move · Enter open · S save

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao+3

Harbin Institute of Technology (Shenzhen)

LLM Inference Scheduling LLM Agent Workflow Optimization LLM Inference Serving LLM Agent Serving LLM Agent Harnesses

Open paper AskPDF arXiv

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. HEAR, a bidirectional Harness--Engine Pairing protocol for agentic LLM serving. HEAR standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, HEAR supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a

1.61\times

batch speedup and reduces median time-to-first-token by

2.23\times

on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield

1.23\times

and

2.45\times

end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving.

Momentum

Related topics

Latest papers 83