LLM Agent Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 20

All topics

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
Nexus: An Execution Fabric for AI Agents Across Cloud, Edge, and Devices
Oct 5, 2026Cary Chang, Jialin ZhouLLM Agent Serving Runtime Enforcement for AI Agents
Nexilume Research
Report: Progressive Disclosure of Agent Skills
Sep 28, 2026Guilin Zhang, Kai Zhao, Priyanka Mudgal +4LLM Agent Serving LLM Agent Skill Retrieval
Workday AI Research
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache Offloading KV-Cache Management
The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · The University of Hong Kong +1
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM Serving LLM Inference Scheduling
Meta Platforms
Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics
Sep 17, 2026Wonmi Choi, Minuk Park, Zhixiong Niu +3LLM Inference Efficiency LLM Agent Workflow Optimization
Department of Computer Science and Engineering, Korea University · Microsoft Research Asia
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference Efficiency KV Caching
Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV Caching KV-Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
Aug 12, 2026Natchanon Pollertlam, Witchayut KornsuwannawitLLM Serving LLM Agent Memory
Bricks Technology · Thailand
SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving
Jul 27, 2026Yihui Zhang, Tianyu Wo, Jinghao Wang +7LLM Inference Efficiency LLM Agent Serving
Beihang University · University of Leeds · The University of Sydney
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference Scheduling KV-Cache Management
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM Serving KV-Cache Management
University of Central Florida
A Policy-Driven Runtime Layer for Agentic LLM Serving
May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference Efficiency Multi-Agent LLM Systems
University of California, Santa Cruz
Stateful Inference for Low-Latency Multi-Agent Tool Calling
May 25, 2026Victor NorgrenLLM Inference Efficiency Multi-Agent LLM Systems
LayerScale, Inc.
GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving
May 21, 2026Ao Li, Shangpeng Yang, Fahao Chen +3LLM Agent Orchestration Agentic Workflows
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence, Shandong University, Jinan, China · 3Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China.
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology
Nalar: Workflow-Aware Management of Agentic Applications
Jan 8, 2026Saurabh Agarwal, Marco Laju, Donghyun Son +4LLM Inference Efficiency LLM Agent Orchestration
UT-Austin · Cisco-Research

Oct 5, 2026, cs.AIJ/K move · Enter open · S save

Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving

Jiaqi Zhao, Haodong Chen, Jitai Hao, Wei Zhao+3

Harbin Institute of Technology (Shenzhen)

LLM Inference Scheduling LLM Agent Workflow Optimization LLM Inference Serving LLM Agent Serving LLM Agent Harnesses

Open paper AskPDF arXiv

LLM agents increasingly execute complex workflows involving multi-turn reasoning, tool use, and parallel agents. Efficient serving requires decisions that span two layers with complementary information: the agent harness understands workflow dependencies, context lifecycles, and execution objectives, whereas the inference engine observes request queues, KV-cache state, resource pressure, and execution capabilities. Existing interfaces do not systematically connect these views, limiting workflow-aware execution. HEAR, a bidirectional Harness--Engine Pairing protocol for agentic LLM serving. HEAR standardizes how the harness communicates workflow intent and execution requirements and how the engine returns runtime state, capabilities, and outcomes. By separating protocol semantics from optimization policies, HEAR supports diverse coordination strategies without changing workflow or model semantics. We instantiate HEAR for online cache-aware runtime coordination and workload-aware execution-mode selection for agent roles. Across four conversational and research-agent benchmarks under memory-constrained, concurrent serving, HEAR achieves a

1.61\times

batch speedup and reduces median time-to-first-token by

2.23\times

on SCBench. Mooncake shows that workflow intent and live engine state provide complementary benefits across load regimes. On BrowseComp-Plus and DeepResearchBench, workload-specific configurations yield

1.23\times

and

2.45\times

end-to-end speedups, respectively, without observed task-quality degradation. These results establish HEAR as a reusable coordination substrate for efficient agentic LLM serving.

Momentum

Related topics

Latest papers 20