Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective

Noppanat Wadlom; Junyi Shen; Yao Lu

cs.MAMar 17, 2026

Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective

Authors: Noppanat Wadlom, Junyi Shen, Yao Lu

Organizations: National University of Singapore Singapore

Abstract

Agentic workflows are composed of sequences of interdependent Large Language Model (LLM) calls, and they have become a dominant workload in modern AI systems. These workflows exhibit extensive redundancy from overlapping prompts and intermediate results due to speculative and parallel exploration. Existing LLM serving systems, such as vLLM, focus on optimizing individual inference calls and overlook cross-call dependencies, leading to significant inefficiencies. This paper rethinks LLM and agent serving from a data systems perspective and introduces Helium, a workflow-aware serving framework that models agentic workloads as query plans and treats LLM invocations as first-class operators. Helium integrates proactive caching and cache-aware scheduling to maximize reuse across prompts, KV states, and workflows. Through these techniques, Helium bridges classic query optimization principles with LLM serving, achieving up to 1.56x speedup over state-of-the-art agent serving systems on various workloads. Our results demonstrate that end-to-end optimization across workflows is essential for scalable and efficient LLM-based agents.

Figures & tables

Figure 1. Three disparities between traditional SQL pipelines and agentic workflows with LLM as operators.

Figure 2. Each representative agentic workflow demonstrates a primitive pattern in agent interactions.

Figure 3. Overview of Helium ’s architecture.

Figure 4. A workflow DAG (top) and the corresponding templated radix tree with cache-aware schedule (bottom)

Figure 5. Normalized end-to-end latency of Helium and baselines, excluding vLLM , across representative workflows and datasets with Qwen3-8B. Values are normalized within each workload so that 1.0 equals the slowest system (lower is better).

Workflow	Datasets	Configuration
MapRed	MMLU, TAT-QA	14 experts for MMLU, 7 experts for TAT-QA with 7 distinct roles
Debate	MMLU, TAT-QA	3 agents with distinct roles, 2 rounds of debate
Reflect	TAT-QA	1 expert, 2 critics
Iterative	Amazon	6 review chunks, 10 reviews each
Parallel	Amazon	7 experts, each processing 6 review chunks

Table 1. Configurations for primitive workflows.

System	Relative Latency (× vs. Helium )
System	Average	Min	Max
vLLM	66.27	38.18	100.92
OpWise	1.25	1.02	1.58
LangGraph	1.28	1.09	1.83
AgentScope	2.10	1.07	4.32
Parrot	1.44	1.02	2.21

Table 2. Relative latency of each system normalized to Helium across representative workflows (lower is better).

Figure 6. The Trading workflow used for end-to-end evaluation, combining the Parallel , Debate , and Map-Reduce patterns. Agent annotations indicate opportunities for proactive KV or prompt caching, applied to prefixes over 200 tokens.

Figure 7. End-to-end latency of Helium and baselines on the Trading workflow across batch sizes using (a) Qwen3-8B and (b) Qwen3-14B.

System	Configurations	Latency (s)	Perf. delta (%)
Helium	Full	130.14	0.00
	w/o KV	134.76	-3.55
	w/o PP	160.53	-23.35
	w/o PC	147.79	-13.56
	w/o CAS	153.12	-17.66

Table 3. Ablation study comparing Helium ’s performance without proactive KV caching (KV), plan pruning (PP), prompt caching (PC), and cache-aware scheduling (CAS).

Scheduling method	Latency (s)	Cache hit rate (%)
QueryWise	658.54 (4.40×)	42.3 (-25.1%)
OpWise	419.03 (2.80×)	40.8 (-27.8%)
Random	193.54 (1.29×)	37.1 (-34.4%)
LSPF	188.93 (1.26×)	37.9 (-32.9%)
Helium (w/ only CAS)	149.76	56.5

Table 4. Performance of different scheduling strategies on the Trading workflow. Caching is disabled to isolate scheduling effectiveness.

Sched. method	Total token step ( $\times 10^{6}$ )			Optimality gap (%)
Sched. method	Avg	Min	Max	Avg	Min	Max
QueryWise	21.6 $\pm$ 8.6	10.2	34.2	72.4 $\pm$ 39.3	26.7	149.2
OpWise	15.3 $\pm$ 6.9	7.0	26.4	17.6 $\pm$ 9.9	3.6	30.7
Random	15.2 $\pm$ 7.2	7.0	27.7	16.3 $\pm$ 8.1	6.0	30.3
LSPF	14.9 $\pm$ 6.9	7.0	26.9	14.5 $\pm$ 8.7	2.8	30.5
Helium	13.4 $\pm$ 6.5	6.0	24.1	0.9 $\pm$ 1.4	0.0	3.6

Table 5. Total token steps and optimality gaps of Helium ’s scheduling algorithm compared to the baselines.

Figure 8. Scalability of Helium vs. LangGraph on Trading workflow across varying: (a) batch sizes, (b) parallel branches, (c) debate rounds, and (d) LLM workers.

Figure 9. Sensitivity to (a) LLM models and (b) KV cache capacity (tokens) on Trading workflow. "Full" indicates KV cache occupying the entire GPU.

Figure 10. Speedup of Helium over LangGraph on (a) MapRed and (b) Debate across prompt configurations, denoted by the token lengths of system prompt / context / question / output .

Figure 11. Overhead analysis: (a) Latency breakdown: planning (QO, CAS, TRT) vs. query processing (QP); (b) Memory footprint of scheduling metadata.

Figure 12. Execution dynamics: (a) Batched requests; (b) Effective tokens; (c) Per-request latency CDF. Helium ’s batch efficiency significantly reduces tail latency.

Figure 13. Helium ’s optimization and scheduling process: (a) initial DAG, (b) optimized logical plan, (c) TRT construction, and (d) cache-aware schedule.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Op	Type	Description
Data	IO	Encapsulates a data batch as a node in the DAG.
Input	IO	Placeholder for data provided at compile time.
Output	IO	Materializes data as a final output of the workflow.
Format	Prompt	Formats input text using specified arguments.
Lambda	Prompt	Applies an arbitrary Python function to the input data.
LLM	Model	Generates LLM responses for input conversation contexts.

Appendix

Table 6. Examples of Helium’s primitive operators.

System	Cache hit rate (%)
System	Qwen3-8B	Qwen3-14B
vLLM	42.9 (-27.5%)	43.0 (+0.3%)
OpWise	40.8 (-31.1%)	39.9 (-6.9%)
LangGraph	37.1 (-37.3%)	36.7 (-14.4%)
AgentScope	39.3 (-33.7%)	36.5 (-14.9%)
Parrot	41.8 (-29.4%)	36.9 (-13.8%)

Appendix

Table 7. Prefix-cache hit rates on the Trading workflow (batch size = 16) for Qwen3-8B and Qwen3-14B, with relative differences vs Helium .

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 206 papers

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Aug 15, 2026Chaokun Chang, Yukun Zhou, Kaihua Fu +10LLM Agent Serving LLM Agent Workflow Optimization
Hong Kong University of Science and Technology · Alibaba Group · Bytedance
GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving
May 21, 2026Ao Li, Shangpeng Yang, Fahao Chen +3LLM Agent Orchestration Agentic Workflows
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence, Shandong University, Jinan, China · 3Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China.

Apr 28, 2026J/K move · Enter open · S save

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

Shan Yu, Junyi Shu, Yuanjiang Ni, Kun Qian+13

UCLA · Alibaba Cloud Computing · Alibaba Group+6

LLM Inference Efficiency Multi-Agent LLM Systems LLM Serving LLM Agent Workflow Optimization LLM Agent Serving

Open paper AskPDF arXiv

As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing structure that constrains agent behavior and exposes useful semantic predictability. Unlike traditional LLM serving, which operates under highly dynamic and uncertain conditions, this structured topology enables opportunities to reduce runtime uncertainty

\unicode{x2015}

yet existing systems fail to exploit it, treating agentic workloads as generic traffic and incurring significant inefficiencies. Our analysis of production traces from an agent-serving platform and an internal coding assistant reveals key bottlenecks, including low prefix cache hit rates, severe resource contention from long-context requests, and substantial queuing delays due to suboptimal scaling. To address these challenges, we propose Pythia, a multi-agent serving system that captures workflow semantics through a simple interface at the serving layer, unlocking new optimization opportunities and substantially improving throughput and job completion time over state-of-the-art baselines.

Abstract

LLM Serving

Explore similar work

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems

GraphFlow: A Graph-Based Workflow Management for Efficient LLM-Agent Serving