LLM Agent Serving

LLM: Large Language Model

LLM agent serving studies how to run multi-turn agents that call tools, use multiple models, and retain context while meeting latency, throughput, and cost goals. Common approaches include workflow-aware runtimes and scheduling, KV-cache reuse and offloading, resource management across GPUs and CPUs, and speculative execution of tools or model outputs. Systems are evaluated on settings such as coding, web research, and other tool-use workflows, with benchmarks and simulators increasingly capturing session state and tool delays. A central tension is balancing cache reuse and speculative speedups against wasted resources, workflow delays, and the need to preserve task quality across varied workloads.

Latest papers

37 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Orla: A Library for Serving LLM-Based Multi-Agent Systems
Mar 13, 2026Rana Shahout, Hayder Tirmazi, Minlan Yu +1LLM Agent Orchestration LLM Agent Serving
Harvard University MA, USA · Boston University MA, USA
Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial
Mar 5, 2026Jielin Qiu, Zixiang Chen, Liangwei Yang +11Conversational Agents LLM Agent Serving
Salesforce AI Research
Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology
DuCCAE: A Hybrid Engine for Immersive Conversation via Collaboration, Augmentation, and Evolution
Feb 25, 2026Xin Shen, Zhishu Jiang, Jiaye Yang +13Conversational Agents LLM Agent Serving
Baidu Inc, Beijing, China
Pancake: Hierarchical Memory System for Multi-Agent LLM Serving
Feb 25, 2026Zhengding Hu, Zaifeng Pan, Prabhleen Kaur +6Memory Management for LLM Agents Agent Memory Management
Computer Science and Engineering, University of California, San Diego
Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning
Feb 21, 2026Abhinaba BasuLLM Agent Serving Intent Classification
Nalar: Workflow-Aware Management of Agentic Applications
Jan 8, 2026Saurabh Agarwal, Marco Laju, Donghyun Son +4LLM Inference Efficiency LLM Agent Orchestration
UT-Austin · Cisco-Research

Mar 13, 2026J/K move · Enter open · S save

Orla: A Library for Serving LLM-Based Multi-Agent Systems

Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher

Harvard University MA, USA · Boston University MA, USA

LLM Agent Orchestration LLM Agent Serving LLM Inference Serving AI Agent Architecture Multi-Agent LLM Systems LLM Agent Workflow Optimization

Open paper AskPDF arXiv

We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.