LLM Inference Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 51

All topics

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
AID: A Framework for AI Infrastructure Dynamics
Oct 3, 2026Abi AryanLLM Inference Serving System Identification
Joule
Inference Auctions
Sep 30, 2026Keegan Harris, Siddharth Prasad, Asher Trockman +2Mechanism Design Automated Bidding
University of California, Berkeley · Toyota Technological Institute at Chicago · Google Research +1
Spexis: Speculative Lookahead Scheduling for LLM Inference
Sep 28, 2026Hyungyu Jung, Jaehyeok Yu, Hoonseo Choi +3LLM Inference Efficiency LLM Inference Scheduling
Seoul National University · Hanyang University
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
Sep 22, 2026Jennifer Williams, Dave Farris, Jeff Farris +1Software Engineering LLM Inference Serving
NVIDIA · University of California, Berkeley
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Sep 22, 2026Lijuan Tang, Yuemeng ZhengLLM Evaluation Tool-Use Evaluation
Northeastern University, Seattle
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference Efficiency KV Caching
Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China
InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers
Aug 13, 2026Nicoletta Tsiopani, Moysis Symeonides, George Pallis +1Energy-Efficient ML LLM Inference Serving
Department of Computer Science University of Cyprus
TELLER: Non-intrusive Cross-Layer Root-Cause Analysis for LLM Inference
Aug 3, 2026Ruilin Xu, Junyi Li, Pengfei Chen +1LLM Inference LLM Inference Serving
Sun Yat-sen University Guangzhou, China
Kalypso: Relational LLM Serving
Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2LLM Serving LLM Inference Scheduling
UMass Amherst, USA
Automatic Model-Hardware Co-Adaptation for Heterogeneous AI Accelerators
Jul 14, 2026Tian Chen, Mingheng Mi, Pu Wang +2AI Accelerator Inference LLM Inference
Pazhou Laboratory (Huangpu) · Unaffiliated
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Jul 14, 2026Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28Efficient Multimodal Inference KV Caching
MiMo Team, Xiaomi
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference Scheduling KV-Cache Management
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1LLM Inference Scheduling LLM Inference Serving
The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States
Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
Jun 28, 2026Victor NorgrenLLM Inference Serving LLM Inference Acceleration
LayerScale, Inc.
KernelSight-LM: A Kernel-Level LLM Inference Simulator
Jun 26, 2026Xiteng Yao, Taeho Kim, Hengzhi Pei +7LLM Inference LLM Inference Serving
Amazon Web Services Santa Clara, CA, USA · Boston University Boston, MA, USA
RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
Jun 22, 2026Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2LLM Inference Serving Speculative Decoding
PayPal
Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving
Jun 18, 2026Liang SuLLM Inference Serving LLM Inference Acceleration
Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Jun 14, 2026Shun Usami, Venkatram Vishwanath, E. Wes BethelAI Accelerator Inference LLM Inference
Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720
INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration
Jun 9, 2026Ahasan Kabir, Jiaqi Xue, Mengxin Zheng +1Multi-Agent System Optimization Multi-Agent Orchestration
University of Central Florida
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Jun 9, 2026Wenxin Wang, Yule Hou, Yu Ji +2Expert Parallelism On-Device Language Model Inference
1Tsinghua University · 2Xingyun Integrated Circuits Co., Ltd. · 3Beijing National Research Center for Information Science and Technology
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM Serving KV-Cache Management
University of Central Florida
Breaking the Ice: Analyzing Cold Start Latency in vLLM
Jun 5, 2026Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin WangLLM Inference LLM Inference Serving
Paderborn University, Paderborn, Germany · IBM Research Europe, Zurich, Switzerland
MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency
Jun 2, 2026Saptarshi Mitra, Yifan Zhang, Rachid Karami +5LLM Inference Efficiency Multi-Agent System Optimization
University of California, Irvine, USA · University of Southern California, Los Angeles, USA · Intel, USA
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
Jun 2, 2026Kathiravan PalaniappanLLM Inference LLM Inference Serving
Independent Researcher (University of Colorado Colorado Springs Alumni)
Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
Jun 1, 2026Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami +2LLM Serving Disaggregated LLM Serving
University of Chicago
Lodestar: An Online-Learning LLM Inference Router
May 31, 2026Gangmuk Lim, Wanyu Zhao, Brighten Godfrey +3Inference-Time Optimization LLM Inference Scheduling
UIUC · Bytedance · University of Edinburgh
A Policy-Driven Runtime Layer for Agentic LLM Serving
May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference Efficiency Multi-Agent LLM Systems
University of California, Santa Cruz
Stateful Inference for Low-Latency Multi-Agent Tool Calling
May 25, 2026Victor NorgrenLLM Inference Efficiency Multi-Agent LLM Systems
LayerScale, Inc.
XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms
May 22, 2026Tella Rajashekhar Reddy, Atharva Deshmukh, Liangcheng Yu +7Energy-Efficient ML LLM Inference Serving
Microsoft

Oct 5, 2026, cs.AIJ/K move · Enter open · S save

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao+3

Harbin Institute of Technology (Shenzhen)

LLM Inference Scheduling LLM Agent Workflow Optimization LLM Inference Serving LLM Agent Serving LLM Agent Harnesses

Open paper AskPDF arXiv

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. HEAR, a bidirectional Harness--Engine Pairing protocol for agentic LLM serving. HEAR standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, HEAR supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a

1.61\times

batch speedup and reduces median time-to-first-token by

2.23\times

on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield

1.23\times

and

2.45\times

end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving.

Momentum

Related topics

Latest papers 51