Substrate-Portable Execution for Production LLM Workflows

Tarun Gopinath; Atul Kulkarni; Vijay Rajakumar; Shrikar Katti; Parthasarathy Govindarajen

cs.AISep 5, 2026

Substrate-Portable Execution for Production LLM Workflows

Authors: Tarun Gopinath, Atul Kulkarni, Vijay Rajakumar, Shrikar Katti, Parthasarathy Govindarajen

Organizations: Amazon Seattle, WA, USA

Abstract

Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.

Figures & tables

Workload	Infer/inv	Latency	Target Binding
Real-time serving	1–5	Sub-second TTFT	Local
Evaluation	1–5	Hours	Flink
Regression evaluation	1–5	Hours	Flink
Cache generation	1–3	Hours	Flink
Deep research	5–12	Minutes	SWF

Table 1. Rufus inference workloads. All share the same agent logic; they differ in execution requirements. The “Binding” column shows the target substrate in the new platform.

Panel of Figure 1 — Figure 1. Platform architecture. Agent logic (top) is authored via the typed Java API or the script DSL; both produce the same binding-agnostic workflow graph. The binding compiler translates the graph to substrate-native operations. Each binding targets a distinct operational profile: streaming for live traffic, durable async for traceable execution, batch for scalable offline operations. All bindings share the same Bedrock inference layer.

Operator	Semantics	Typical use
map	Applies a stateless transformation to node output.	Prompt assembly, parsing, tool dispatch
infer	Invokes an LLM using the strategy selected by the binding.	Model inference
iterate	Repeats a body until a condition or iteration limit is reached.	ReAct loops
compose	Delegates execution to another registered agent.	Multi-agent workflows, PreAct
branch , parallelBranch	Performs conditional or parallel fan-out followed by a typed merge.	Routing, specialist fan-out

Table 2. Workflow combinators and their intended uses.

Pattern	Model(s)	Cross-bind	$\hat{\beta}_{\text{SWF}}$ [CI]	$\hat{\beta}_{\text{Batch}}$ [CI]
		decisive%
RAG (single-infer)	Opus	0.0%	$\approx 0$	$\approx 0$
Routing (branch)	Haiku + Sonnet	21.5%	$+0.11\;[-0.26,+0.50]$	$+0.02\;[-0.38,+0.39]$
ReAct (tool-calling)	Sonnet	12.9%	$+0.14\;[-0.59,+0.93]$	$-0.43\;[-1.42,+0.53]$
PreAct (composition)	Haiku + Sonnet	40.4%	$+0.25\;[-0.05,+0.57]$	$-0.13\;[-0.45,+0.21]$
Deep Research	Sonnet	12.1%	$-0.30\;[-1.15,+0.49]$	$+0.34\;[-0.64,+1.44]$

Table 3. Quality equivalence (Bradley-Terry). Cross-bind decisive% : fraction of position-swapped comparisons where one binding wins in both orderings (maximum across the 3 binding pairs).

\hat{\beta}

: BT strength relative to Stream (

\beta_{\text{Stream}}\equiv 0

) with 95% CI; CI containing zero = no detectable preference.

Pattern	Stream	SWF	Flink
RAG (single-infer)	$1.0\times$	$1.5\times$	$79\times$
Routing (branch)	$1.0\times$	$1.8\times$	$152\times$
ReAct (tool-calling)	$1.0\times$	$3.4\times$	$96\times$
PreAct (composition)	$1.0\times$	$1.8\times$	$116\times$
Deep Research	$1.0\times$	$1.1\times$	$33\times$

Table 4. Mean end-to-end latency normalized to the Stream baseline (

1.0\times

) for each pattern. SWF overhead reflects per-iteration decision-task round-trips. Flink latency is dominated by batch job scheduling at the provider, not binding overhead.

System	Str.	Batch	Durable	X-inv.	Port.
LangGraph ( LangChain, 2024 ; LangChain, 2026 )	✓	–	$\sim$	–	–
Strands/AutoGen ( Amazon Web Services, 2025b ; Wu et al., 2023 ; Amazon Web Services, 2026b ; Microsoft, 2026 )	✓	–	$\sim$	–	–
Temporal ( Temporal Technologies, 2023 ; Temporal Technologies, 2026 )	–	–	✓	–	$\sim$
Beam ( Akidau et al., 2015 ; Apache Software Foundation, 2026a )	$\sim$	✓	–	✓	✓
Flink SF ( Apache Software Foundation, 2020 ; Apache Software Foundation, 2026b )	$\sim$	–	✓	$\sim$	–
This work	✓	✓	✓	✓	✓

Table 5. Capability comparison of design emphasis across related systems. ✓ = native,

\sim

= partial or supported via extension, – = not a primary design focus. Assessments reflect each system’s publicly available documentation as of August 2026; these ecosystems evolve rapidly.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 463 papers

May 21, 2026

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Agent orchestration frameworks have proliferated, collectively exceeding 290,000 GitHub stars across LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, and LlamaIndex. All follow the same pattern: an external orchestrator above the LLM, injecting instructions and routing decisions every turn. Recent work has shown this architecture is dominated for procedural tasks by simply providing the procedure in a frontier model's system prompt [Dennis et al., 2026a], at the cost of consuming the context window, requiring a frontier model for every conversation, and exposing proprietary procedures to third-party providers. Compiling the procedure into the weights of a small fine-tuned model -- creating a subterranean agent -- should resolve all of these concerns, and prior work (SimpleTOD, FireAct, SynTOD, WorkflowLLM, Agent Lumos) has shown the technique works. Yet developer adoption has overwhelmingly favored orchestration. We identify three perceived barriers and address each empirically across travel booking (14 nodes), Zoom support (14 nodes, product-specific knowledge), and insurance claims (55 nodes, 6 decision hubs).

Simon Dennis, Rivaan Patil, Kevin Shabahang +1

i14, University of Melbourne · i14

LLM Agent Orchestration Agentic Workflows LLM Agent Workflow Optimization LLM Agent Training

Ask PDF

Apr 28, 2026

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

As LLM applications grow more complex, developers are increasingly adopting multi-agent architectures to decompose workflows into specialized, collaborative components, introducing structure that constrains agent behavior and exposes useful semantic predictability. Unlike traditional LLM serving, which operates under highly dynamic and uncertain conditions, this structured topology enables opportunities to reduce runtime uncertainty

\unicode{x2015}

yet existing systems fail to exploit it, treating agentic workloads as generic traffic and incurring significant inefficiencies. Our analysis of production traces from an agent-serving platform and an internal coding assistant reveals key bottlenecks, including low prefix cache hit rates, severe resource contention from long-context requests, and substantial queuing delays due to suboptimal scaling. To address these challenges, we propose Pythia, a multi-agent serving system that captures workflow semantics through a simple interface at the serving layer, unlocking new optimization opportunities and substantially improving throughput and job completion time over state-of-the-art baselines.

Shan Yu, Junyi Shu, Yuanjiang Ni +14

UCLA · Alibaba Cloud Computing · Alibaba Group +6

LLM Inference Efficiency Multi-Agent LLM Systems LLM Serving LLM Agent Workflow Optimization LLM Agent Serving

Ask PDF

Aug 1, 2026

AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications

Large language model (LLM) applications increasingly operate as streaming workflows combining retrieval, tool calls, safety filters, and multi-agent coordination. Although contemporary frameworks expose provider deltas, workflow nodes often treat generation as coarse request-response steps, leaving queue management, worker allocation, ordering, and backpressure to ad hoc callback code. This paper presents AiFlow, a token-native reactive orchestration model that normalizes provider deltas into typed Context<T> events propagated through a directed streaming graph. Each node is managed by a Node Guardian that declares and enforces local queue bounds, worker concurrency, ordering, overflow policy, cancellation propagation, and retry discipline. We formalize the bounded-memory property, present the compilation from a compact DSL and JSON graph form, and provide static validation for type safety, state concurrency, and injection compatibility. Controlled microbenchmarks, captured DeepSeek trace replay (30 runs), descriptive online runs, LangGraph baselines, a streaming RAG workload, and an Ollama local-backend check show that AiFlow does not alter provider-side Model TTFT but reduces Application TTFPT by 70.9-94.7% versus aggregation and keeps runtime-owned queue depth within declared bounds (93.7-96.5% MaxQ reduction versus unbounded policies). The supplementary artifact contains scripts, raw traces, machine-readable tables, checksums, and an API-free smoke test; the public implementation is available through the FIT Framework repository.

Qunhui Zhang

School of Software, Shanghai Jiao Tong University, Shanghai 200240, China

LLM Serving LLM Agent Orchestration LLM Agent Workflow Optimization

Ask PDF

Abstract

LLM Agent Orchestration

Explore similar work

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications