LLM Agent Serving

LLM: Large Language Model

LLM agent serving studies how to run multi-turn agents that call tools, use multiple models, and retain context while meeting latency, throughput, and cost goals. Common approaches include workflow-aware runtimes and scheduling, KV-cache reuse and offloading, resource management across GPUs and CPUs, and speculative execution of tools or model outputs. Systems are evaluated on settings such as coding, web research, and other tool-use workflows, with benchmarks and simulators increasingly capturing session state and tool delays. A central tension is balancing cache reuse and speculative speedups against wasted resources, workflow delays, and the need to preserve task quality across varied workloads.

Latest papers

36 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
Oct 15, 2026Zilin Wang, Yinhong Liu, Chao Chen +2Tool-Using Agents LLM Agent Verification
LARK, HKUST (GZ) · Yinwang · University of Cambridge +1
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
Oct 14, 2026Daijin HuLLM Serving LLM Agent Serving
Beihang University
Taking Tokenization off the Host for Agentic LLM Serving
Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3Tokenization LLM Serving
KTH Royal Institute of Technology, Stockholm, Sweden
Can Agent Harnesses and Inference Engines Hear Each Other? The HEAR Protocol for Agentic LLM Serving
Oct 5, 2026Jiaqi Zhao, Haodong Chen, Jitai Hao +4LLM Inference Scheduling LLM Agent Workflow Optimization
Harbin Institute of Technology (Shenzhen)
Nexus: An Execution Fabric for AI Agents Across Cloud, Edge, and Devices
Oct 5, 2026Cary Chang, Jialin ZhouLLM Agent Serving Runtime Enforcement for AI Agents
Nexilume Research
Report: Progressive Disclosure of Agent Skills
Sep 28, 2026Guilin Zhang, Kai Zhao, Priyanka Mudgal +4LLM Agent Serving LLM Agent Skill Retrieval
Workday AI Research
EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?
Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache Offloading KV-Cache Management
The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · The University of Hong Kong +1
DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving
Sep 25, 2026Junyi Shen, Noppanat Wadlom, Zhengyuan Su +1LLM Agent Workflow Optimization LLM Agent Serving
National University of Singapore Singapore
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM Serving LLM Inference Scheduling
Meta Platforms
Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics
Sep 17, 2026Wonmi Choi, Minuk Park, Zhixiong Niu +3LLM Inference Efficiency LLM Agent Workflow Optimization
Department of Computer Science and Engineering, Korea University · Microsoft Research Asia
Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It
Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference Efficiency KV Caching
Tsinghua University, China · Alibaba Cloud Computing, China · Zhejiang University, China
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV Caching KV-Cache Eviction
Imperial College London, London, United Kingdom · Columbia University, New York, NY, USA
Substrate-Portable Execution for Production LLM Workflows
Sep 5, 2026Tarun Gopinath, Atul Kulkarni, Vijay Rajakumar +2LLM Agent Orchestration LLM Agent Serving
Amazon Seattle, WA, USA
TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving
Aug 26, 2026Hongqiu Ni, Han Tian, Chi Zhang +2LLM Inference Scheduling LLM Agent Serving
University of Science and Technology of China, China · Hefei University of Technology, China
AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Aug 15, 2026Chaokun Chang, Yukun Zhou, Kaihua Fu +10LLM Agent Serving LLM Agent Workflow Optimization
Hong Kong University of Science and Technology · Alibaba Group · Bytedance
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
Aug 12, 2026Natchanon Pollertlam, Witchayut KornsuwannawitLLM Serving LLM Agent Memory
Bricks Technology · Thailand
SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving
Jul 27, 2026Yihui Zhang, Tianyu Wo, Jinghao Wang +7LLM Inference Efficiency LLM Agent Serving
Beihang University · University of Leeds · The University of Sydney
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference Scheduling KV-Cache Management
Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University · ShanghaiTech University · Alibaba Group
KumbhDoot: A Scale-Ready, LLM-Bounded Architecture for Mass-Gathering Public-Service Assistants
Jul 6, 2026Saurabh Sakalkar, Abhishek Singh, Ramesh RaskarLLM Agent Serving Conversational Agents
Cisco · Project Nanda · MIT
AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving
Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM Serving KV-Cache Management
University of Central Florida
A Policy-Driven Runtime Layer for Agentic LLM Serving
May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference Efficiency Multi-Agent LLM Systems
University of California, Santa Cruz
Stateful Inference for Low-Latency Multi-Agent Tool Calling
May 25, 2026Victor NorgrenLLM Inference Efficiency Multi-Agent LLM Systems
LayerScale, Inc.
GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving
May 21, 2026Ao Li, Shangpeng Yang, Fahao Chen +3LLM Agent Orchestration Agentic Workflows
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence, Shandong University, Jinan, China · 3Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China.
Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs
Mar 23, 2026Kangqi Ni, Wenyue Hua, Xiaoxiang Shi +3LLM Serving LLM Inference Scheduling
University of North Carolina, Chapel Hill · Microsoft · Carnegie Mellon University +2
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Mar 19, 2026Yifan Sui, Han Zhao, Rui Ma +6LLM Agent Serving LLM Tool Use
Shanghai Jiao Tong University · Microsoft Research · Stevens Institute of Technology +2
Orla: A Library for Serving LLM-Based Multi-Agent Systems
Mar 13, 2026Rana Shahout, Hayder Tirmazi, Minlan Yu +1LLM Agent Orchestration LLM Agent Serving
Harvard University MA, USA · Boston University MA, USA

Oct 15, 2026J/K move · Enter open · S save

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu+1

LARK, HKUST (GZ) · Yinwang · University of Cambridge+1

Tool-Using Agents LLM Agent Verification LLM Agent Serving

Open paper AskPDF arXiv

Large language model agents solve tasks through long sequences of tool calls, and the prevailing design has a single strong model generate every step, although many steps are routine enough for a much smaller model. Existing ways of sharing this work keep the strong model as the generator: routers and collaborative agents decide which model handles each query or step, and speculative methods check a small model's drafts one step at a time, usually against the strong model's own action. We introduce SpecAgent, a collaboration protocol that turns the strong model from the author of each step into a reviewer of steps drafted by a weak model. The weak model executes read-only tool calls as it drafts, so that every step is grounded in real observations, and holds the first action that would modify the environment. The strong model then verifies the drafted segment in a single prefill pass, commits the longest prefix it judges semantically valid, and repairs the first rejected step. Because verification replaces autoregressive decoding with parallel prefill, the strong model decodes only short verdicts and occasional repairs rather than every step. On

τ^2

-Bench, SpecAgent comes within one point of the strong model's success rate while roughly halving memory traffic and wall time, and on AutomationBench and VitaBench it recovers 44-75% of the quality gap while lowering both. In an edge-cloud deployment on MCP-Atlas with three cloud models, it closes 53-75% of the weak-strong quality gap while the strong model generates at most half as many tokens. These results suggest that a strong model need not author every step of an agent: reviewing grounded drafts retains most of its quality while cutting both latency and memory traffic.