Substrate-Portable Execution for Production LLM Workflows

Tarun Gopinath; Atul Kulkarni; Vijay Rajakumar; Shrikar Katti; Parthasarathy Govindarajen

cs.AISep 5, 2026

Substrate-Portable Execution for Production LLM Workflows

Authors: Tarun Gopinath, Atul Kulkarni, Vijay Rajakumar, Shrikar Katti, Parthasarathy Govindarajen

Organizations: Amazon Seattle, WA, USA

Abstract

Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.

Figures & tables

Workload	Infer/inv	Latency	Target Binding
Real-time serving	1–5	Sub-second TTFT	Local
Evaluation	1–5	Hours	Flink
Regression evaluation	1–5	Hours	Flink
Cache generation	1–3	Hours	Flink
Deep research	5–12	Minutes	SWF

Table 1. Rufus inference workloads. All share the same agent logic; they differ in execution requirements. The “Binding” column shows the target substrate in the new platform.

Panel of Figure 1 — Figure 1. Platform architecture. Agent logic (top) is authored via the typed Java API or the script DSL; both produce the same binding-agnostic workflow graph. The binding compiler translates the graph to substrate-native operations. Each binding targets a distinct operational profile: streaming for live traffic, durable async for traceable execution, batch for scalable offline operations. All bindings share the same Bedrock inference layer.

Operator	Semantics	Typical use
map	Applies a stateless transformation to node output.	Prompt assembly, parsing, tool dispatch
infer	Invokes an LLM using the strategy selected by the binding.	Model inference
iterate	Repeats a body until a condition or iteration limit is reached.	ReAct loops
compose	Delegates execution to another registered agent.	Multi-agent workflows, PreAct
branch , parallelBranch	Performs conditional or parallel fan-out followed by a typed merge.	Routing, specialist fan-out

Table 2. Workflow combinators and their intended uses.

Pattern	Model(s)	Cross-bind	$\hat{\beta}_{\text{SWF}}$ [CI]	$\hat{\beta}_{\text{Batch}}$ [CI]
		decisive%
RAG (single-infer)	Opus	0.0%	$\approx 0$	$\approx 0$
Routing (branch)	Haiku + Sonnet	21.5%	$+0.11\;[-0.26,+0.50]$	$+0.02\;[-0.38,+0.39]$
ReAct (tool-calling)	Sonnet	12.9%	$+0.14\;[-0.59,+0.93]$	$-0.43\;[-1.42,+0.53]$
PreAct (composition)	Haiku + Sonnet	40.4%	$+0.25\;[-0.05,+0.57]$	$-0.13\;[-0.45,+0.21]$
Deep Research	Sonnet	12.1%	$-0.30\;[-1.15,+0.49]$	$+0.34\;[-0.64,+1.44]$

Table 3. Quality equivalence (Bradley-Terry). Cross-bind decisive% : fraction of position-swapped comparisons where one binding wins in both orderings (maximum across the 3 binding pairs).

\hat{\beta}

: BT strength relative to Stream (

\beta_{\text{Stream}}\equiv 0

) with 95% CI; CI containing zero = no detectable preference.

Pattern	Stream	SWF	Flink
RAG (single-infer)	$1.0\times$	$1.5\times$	$79\times$
Routing (branch)	$1.0\times$	$1.8\times$	$152\times$
ReAct (tool-calling)	$1.0\times$	$3.4\times$	$96\times$
PreAct (composition)	$1.0\times$	$1.8\times$	$116\times$
Deep Research	$1.0\times$	$1.1\times$	$33\times$

Table 4. Mean end-to-end latency normalized to the Stream baseline (

1.0\times

) for each pattern. SWF overhead reflects per-iteration decision-task round-trips. Flink latency is dominated by batch job scheduling at the provider, not binding overhead.

System	Str.	Batch	Durable	X-inv.	Port.
LangGraph ( LangChain, 2024 ; LangChain, 2026 )	✓	–	$\sim$	–	–
Strands/AutoGen ( Amazon Web Services, 2025b ; Wu et al., 2023 ; Amazon Web Services, 2026b ; Microsoft, 2026 )	✓	–	$\sim$	–	–
Temporal ( Temporal Technologies, 2023 ; Temporal Technologies, 2026 )	–	–	✓	–	$\sim$
Beam ( Akidau et al., 2015 ; Apache Software Foundation, 2026a )	$\sim$	✓	–	✓	✓
Flink SF ( Apache Software Foundation, 2020 ; Apache Software Foundation, 2026b )	$\sim$	–	✓	$\sim$	–
This work	✓	✓	✓	✓	✓

Table 5. Capability comparison of design emphasis across related systems. ✓ = native,

\sim

= partial or supported via extension, – = not a primary design focus. Assessments reflect each system’s publicly available documentation as of August 2026; these ecosystems evolve rapidly.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 463 papers

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost
May 21, 2026Simon Dennis, Rivaan Patil, Kevin Shabahang +1LLM Agent Orchestration Agentic Workflows
i14, University of Melbourne · i14
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications
Aug 1, 2026Qunhui ZhangLLM Serving LLM Agent Orchestration
School of Software, Shanghai Jiao Tong University, Shanghai 200240, China

May 21, 2026J/K move · Enter open · S save

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Simon Dennis, Rivaan Patil, Kevin Shabahang, Hao Guo

i14, University of Melbourne · i14

LLM Agent Orchestration Agentic Workflows LLM Agent Workflow Optimization LLM Agent Training

Open paper AskPDF arXiv

Agent orchestration frameworks have proliferated, collectively exceeding 290,000 GitHub stars across LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, Semantic Kernel, Strands, and LlamaIndex. All follow the same pattern: an external orchestrator above the LLM, injecting instructions and routing decisions every turn. Recent work has shown this architecture is dominated for procedural tasks by simply providing the procedure in a frontier model's system prompt [Dennis et al., 2026a], at the cost of consuming the context window, requiring a frontier model for every conversation, and exposing proprietary procedures to third-party providers. Compiling the procedure into the weights of a small fine-tuned model -- creating a subterranean agent -- should resolve all of these concerns, and prior work (SimpleTOD, FireAct, SynTOD, WorkflowLLM, Agent Lumos) has shown the technique works. Yet developer adoption has overwhelmingly favored orchestration. We identify three perceived barriers and address each empirically across travel booking (14 nodes), Zoom support (14 nodes, product-specific knowledge), and insurance claims (55 nodes, 6 decision hubs).

Abstract

LLM Agent Orchestration

Explore similar work

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving

AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications