cs.AIMar 13, 2026

Orla: A Library for Serving LLM-Based Multi-Agent Systems

Authors: Rana Shahout, Hayder Tirmazi, Minlan Yu, Michael Mitzenmacher

Organizations: Harvard University MA, USA · Boston University MA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.

Figures & tables

Figure 1. Orla design.
Figure 2. (a) Wall-clock time (b) Cost (c) Completion time.
Figure 3. Cache management (request vs. workflow flush).
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4. The workflow of our running example, a customer support workflow.
Figure 5. The Orla API allows users to define complex agentic workflows using a DAG-based approach, as shown in this customer support example.
Figure 6
Figure 6. An illustration of Orla’s two-level scheduling policy in our running example. Stage scheduling policy selects which stage sub-queue to dequeue from, while request scheduling policy orders requests within a sub-queue. Priority hints can be set dynamically based on ticket classification.
Figure 7. The policy_check stage constructs its prompt from the upstream classify stage’s result.
Figure 8. In the customer support workflow, classify runs on a lightweight backend while later stages run on a heavier backend. The memory manager flushes cache when switching backends, but preserves it when consecutive stages on the same backend share most of their context, avoiding redundant prefilling. All remaining cache is reclaimed when the workflow completes.
More in this topic

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 440 papers

Explore similar work

CardsList
Feb 25, 2026

Pancake: Hierarchical Memory System for Multi-Agent LLM Serving

In this work, we identify and address the core challenges of agentic memory management in LLM serving, where large-scale storage, frequent updates, and multiple coexisting agents jointly introduce complex and high-cost approximate nearest neighbor (ANN) searching problems. We present Pancake, a multi-tier agentic memory system that unifies three key techniques: (i) multi-level index caching for single agents, (ii) coordinated index management across multiple agents, and (iii) collaborative GPU-CPU acceleration. Pancake exposes easy-to-use interface that can be integrated into memory-based agents like Mem-GPT, and is compatible with agentic frameworks such as LangChain and LlamaIndex. Experiments on realistic agent workloads show that Pancake substantially outperforms existing frameworks, achieving more than 4.29x end-to-end throughput improvement.

Zhengding Hu, Zaifeng Pan, Prabhleen Kaur +6

Computer Science and Engineering, University of California, San Diego

Memory Management for LLM AgentsAgent Memory ManagementLLM Agent ServingMulti-Agent LLM SystemsApproximate Nearest Neighbor Search
AskPDF
Mar 20, 2026

Utility-Guided Agent Orchestration for Efficient LLM Tool Use

Tool-using large language model (LLM) agents often face a fundamental tension between answer quality and execution cost. Fixed workflows are stable but inflexible, while free-form multi-step reasoning methods such as ReAct may improve task performance at the expense of excessive tool calls, longer trajectories, higher token consumption, and increased latency. In this paper, we study agent orchestration as an explicit decision problem rather than leaving it entirely to prompt-level behavior. We propose a utility-guided orchestration policy that selects among actions such as respond, retrieve, tool call, verify, and stop by balancing estimated gain, step cost, uncertainty, and redundancy. Our goal is not to claim universally best task performance, but to provide a controllable and analyzable policy framework for studying quality-cost trade-offs in tool-using LLM agents. Experiments across direct answering, threshold control, fixed workflows, ReAct, and several policy variants show that explicit orchestration signals substantially affect agent behavior. Additional analyses on cost definitions, workflow fairness, and redundancy control further demonstrate that lightweight utility design can provide a defensible and practical mechanism for agent control.

Boyan Liu, Gongming Zhao, Hongli Xu

University of Science and Technology of China, Hefei, Anhui, China · Suzhou Institute for Advanced Research, University of Science and Technology of China

LLM Agent OrchestrationLLM Tool UseLLM Agent Workflow OptimizationLLM Decision-Making
AskPDF
May 25, 2026

Stateful Inference for Low-Latency Multi-Agent Tool Calling

Multi-agent tool calling is becoming the dominant interaction pattern for LLM-based systems, yet existing inference frameworks treat each tool call as an independent request, re-processing the entire conversation from scratch even though 85-95% of the prompt is unchanged from the previous turn. We present a stateful inference architecture that converts the O(nt)O(n_t) per-turn cost of conventional serving into an O(Δt)O(Δ_t) delta-only cost: a persistent KV cache lives across turns and advances by ingesting only the new tokens, while a radix prefix cache extends this across interleaved multi-agent traffic and a prompt-lookup speculative decoder accelerates structured output. Against vLLM and SGLang on novel, fully-generated workloads, the reference implementation is 2.1×2.1\times faster per turn on a 6-turn agentic workflow and 4.2×4.2\times on the median turn of a 35-turn one, halving end-to-end wall time. The advantage comes from stateful reuse and speculation, not caching.

Victor Norgren

LayerScale, Inc.

LLM Inference EfficiencyMulti-Agent LLM SystemsTool-Augmented Language Model AgentsLLM Agent Workflow OptimizationLLM Inference ServingLLM Agent Serving+1
AskPDF