LLM Agent Serving

LLM: Large Language Model

Overview

LLM agent serving studies how to run multi-turn agents that call tools, use multiple models, and retain context while meeting latency, throughput, and cost goals. Common approaches include workflow-aware runtimes and scheduling, KV-cache reuse and offloading, resource management across GPUs and CPUs, and speculative execution of tools or model outputs. Systems are evaluated on settings such as coding, web research, and other tool-use workflows, with benchmarks and simulators increasingly capturing session state and tool delays. A central tension is balancing cache reuse and speculative speedups against wasted resources, workflow delays, and the need to preserve task quality across varied workloads.

Latest papers

CardsList

42 papers5 papers in the last four weeks, down 44% on the four weeks before. 0.0% of all new papers.

  1. TactiNavi: Serving and Revising an LLM Game Coach with Explicit Strategy Conditions

    Nov 3, 2026Qi Liu, Xiaoyang Yuan, Zhuomeng Zhang +9LLM Agent ServingLLM Agent Reliability

  2. Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

    Oct 15, 2026Zilin Wang, Yinhong Liu, Chao Chen +2Tool-Using AgentsLLM Agent Verification

  3. Taking Tokenization off the Host for Agentic LLM Serving

    Oct 9, 2026Mohammad Siavashi, Pascal Schwinte, Ali Banaei +3TokenizationLLM Serving

  4. Report: Progressive Disclosure of Agent Skills

    Sep 28, 2026Guilin Zhang, Kai Zhao, Priyanka Mudgal +4LLM Agent ServingLLM Agent Skill Retrieval

  5. EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?

    Sep 27, 2026Kunming Shao, Jierun Chen, Jiangnan Yu +7KV-Cache OffloadingKV-Cache Management

  6. DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving

    Sep 25, 2026Junyi Shen, Noppanat Wadlom, Zhengyuan Su +1Language Model Inference CachingLLM Agent Workflow Optimization

  7. Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics

    Sep 17, 2026Wonmi Choi, Minuk Park, Zhixiong Niu +3LLM Inference EfficiencyLLM Agent Workflow Optimization

  8. Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

    Sep 16, 2026Yipeng Liu, Yingqiang Zhang, Feifei Li +1LLM Inference EfficiencyKV Caching

  9. AgentKV: Phase-Aware KV Eviction for Agentic LLMs

    Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3KV CachingKV-Cache Eviction

  10. Substrate-Portable Execution for Production LLM Workflows

    Sep 5, 2026Tarun Gopinath, Atul Kulkarni, Vijay Rajakumar +2LLM Agent OrchestrationLLM Agent Serving

  11. TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Serving

    Aug 26, 2026Hongqiu Ni, Han Tian, Chi Zhang +2LLM Inference SchedulingLLM Agent Serving

  12. AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

    Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative DecodingLLM Inference Acceleration

  13. From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

    Aug 15, 2026Chaokun Chang, Yukun Zhou, Kaihua Fu +10LLM Agent ServingLLM Agent Workflow Optimization

  14. Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems

    Aug 12, 2026Natchanon Pollertlam, Witchayut KornsuwannawitLLM ServingLLM Agent Memory

  15. SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving

    Jul 27, 2026Yihui Zhang, Tianyu Wo, Jinghao Wang +7LLM Inference EfficiencyLLM Agent Serving

  16. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7LLM Inference SchedulingKV-Cache Management

  17. KumbhDoot: A Scale-Ready, LLM-Bounded Architecture for Mass-Gathering Public-Service Assistants

    Jul 6, 2026Saurabh Sakalkar, Abhishek Singh, Ramesh RaskarLLM Agent ServingConversational Agents

  18. AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving

    Jun 8, 2026Rakibul Hasan Rajib, Mengxin Zheng, Qian LouLLM ServingKV-Cache Management

  19. A Policy-Driven Runtime Layer for Agentic LLM Serving

    May 26, 2026Rui Zhang, Chaeeun Kim, Liting HuLLM Inference EfficiencyMulti-Agent LLM Systems