FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

Dimitrios Liakopoulos; Prasoon Sinha; Neeraja J. Yadwadkar

cs.LGOct 17, 2026

FlowMaker: From Tasks to Cost-Efficient Generative AI Workflow Deployments

Authors: Dimitrios Liakopoulos, Prasoon Sinha, Neeraja J. Yadwadkar

Organizations: The University of Texas at Austin

Abstract

Generative AI workflows execute user tasks by composing sequences of LLM calls, where each call performs a sub-task (e.g., summarization or translation), forming a workflow. Executing such workflows requires decisions over workflow structure, model assignments, and deployment configurations, which jointly determine latency, cost, and accuracy under dynamic workloads and resource constraints. Existing systems optimize these decisions in isolation or fix them early, leading to suboptimal cost and goodput. We make two key observations. First, efficient serving requires jointly exploring all combinations of workflows, model choices, and deployment configurations under dynamic cluster state. Second, reuse of deployed LLMs is critical for reducing cost, but must be performed on-the-fly, as reuse depends on current load and resource availability. Together, these make selecting a workflow-variant-a combination of workflow, models, and deployment configuration-a time-critical problem. We present FlowMaker, which exposes a task-level interface and selects workflow-variants on behalf of users. To realize this, we design a constraint-guided load-aware pruning algorithm for fast, on-the-fly decisions. FlowMaker improves goodput by up to 10.14 $\times$ and reduces cost by up to 2.77 $\times$ compared to state-of-the-art baselines, including Murakkab.

Figures & tables

Figure 1: (a) Task graph for document conversion, summarization, and translation, showing the sub-tasks and their dependencies. (b) The three candidate workflows considered in our example. $W_{1}$ converts the user-provided PDF to text, summarizes it, and then translates the summary; $W_{2}$ and $W_{3}$ use alternative organizations of the sub-tasks. Assigning models and deployment configurations to these workflows yields over 24K variants, with significant variation in latency and cost.

Figure 2: Serving cost and cluster goodput as the system is given progressively more choices: a fixed workflow variant (Static), model and deployment choices within a fixed workflow (Murakkab), and joint workflow, model, and deployment choices from a high-level task (Oracle). Joint optimization reduces cost and improves goodput.

Figure 3: Serving cost and cluster goodput for reuse policies provisioned for average, maximum, or current cluster load (OnTheFlyReuse). Adapting reuse decisions to current load reduces cost and improves goodput.

Figure 5: End-to-end request latency (top), serving cost (middle), and cluster goodput (bottom) for 7 tasks under the Azure (a) code and (b) conversation traces. See § 5.1 and § 5.2 .

Figure 6: Workflow-variant selection time across tasks (top), workflow depth (middle), and model-catalog size (bottom).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Each task in the AsyncHow dataset can be executed using multiple workflows, each with distinct characteristics. Those choices expand the search space.

Figure 8: Latency, GPU-memory consumption, serving cost, and benchmark-derived proxy accuracy of the 11 evaluated models. Latency, memory, and cost are measured while serving a query with an output length of 100 tokens.

Symbol	Description	Symbol	Description
$\mathcal{T}$	Task (in natural language)	$\mathbf{n}$	Nodes $\{n_{i}\}$
$\mathcal{W}$	Set of workflows	$n_{id}$	Node (model, config)
$\mathcal{S}$	Set of subtasks	$id$	Node’s ID
$\mathcal{M}$	Set of models	$\mathbf{gpu}$	GPU mapping
$\mathcal{C}$	Set of configs (TP, PP)	$\lambda$	Latency SLO
$\mathcal{D}$	Set of deployed LLMs	$\alpha$	Accuracy SLO

Appendix

Table 1: Notation.

API	Parameters
register_model	model, [accuracy]
deploy_task	task, latSLO, accSLO $\mid$ workflow_variant
inference_task	task, inputText

Appendix

Table 2: FlowMaker ’s APIs.

Trace	25%	50%	75%	100%
Code	0.100	0.133	0.167	0.200
Conversation	0.050	0.067	0.083	0.100

Appendix

Table 3: Trace-scaling factors used for each offered-load level.

Figure 9: Tasks admitted (left), serving cost (middle), and SLO-compliant goodput (right) as 14 tasks arrive under fixed cluster capacity. FlowMaker and Murakkab admit all tasks through reuse, with FlowMaker achieving

1.5\times

Murakkab’s goodput. RouteLLM attains higher goodput in its maximum-quality setting through replication.

Figure 10: Per-request latency (left), serving cost (middle), and SLO-compliant goodput (right) under strict (top) and relaxed (bottom) latency SLOs, on the code trace at 100% load. FlowMaker remains the most cost-efficient in both settings.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 216 papers

WorkflowGen:an adaptive workflow generation mechanism driven by trajectory experience
Mar 22, 2026Ruocan Wei, Shufeng Wang, Ziwei ShiAgentic Workflows LLM Agent Workflow Optimization
China Telecom Cloud, Beijing, China
FlowBank: Query-Adaptive Agentic Workflows Optimization through Precompute-and-Reuse
Jun 9, 2026Lingzhi Yuan, Chenghao Deng, Fangxu Yu +3Multi-Agent LLM Systems Multi-Agent System Optimization
University of Maryland, College Park · Amazon
MoFlow: Multi-Objective Agentic Workflow Generation
Sep 29, 2026Yining Lu, Aurelie Lozano, Xi Yang +3Multi-Objective Reinforcement Learning Agentic Workflows
University of Notre Dame · IBM

Mar 22, 2026J/K move · Enter open · S save

WorkflowGen:an adaptive workflow generation mechanism driven by trajectory experience

Ruocan Wei, Shufeng Wang, Ziwei Shi

China Telecom Cloud, Beijing, China

Agentic Workflows LLM Agent Workflow Optimization Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) agents often suffer from high reasoning overhead, excessive token consumption, unstable execution, and inability to reuse past experiences in complex tasks like business queries, tool use, and workflow orchestration. Traditional methods generate workflows from scratch for every query, leading to high cost, slow response, and poor robustness. We propose WorkflowGen, an adaptive, trajectory experience-driven framework for automatic workflow generation that reduces token usage and improves efficiency and success rate. Early in execution, WorkflowGen captures full trajectories and extracts reusable knowledge at both node and workflow levels, including error fingerprints, optimal tool mappings, parameter schemas, execution paths, and exception-avoidance strategies. It then employs a closed-loop mechanism that performs lightweight generation only on variable nodes via trajectory rewriting, experience updating, and template induction. A three-tier adaptive routing strategy dynamically selects among direct reuse, rewriting-based generation, and full initialization based on semantic similarity to historical queries. Without large annotated datasets, we qualitatively compare WorkflowGen against real-time planning, static single trajectory, and basic in-context learning baselines. Our method reduces token consumption by over 40 percent compared to real-time planning, improves success rate by 20 percent on medium-similarity queries through proactive error avoidance and adaptive fallback, and enhances deployability via modular, traceable experiences and cross-scenario adaptability. WorkflowGen achieves a practical balance of efficiency, robustness, and interpretability, addressing key limitations of existing approaches.

Abstract

LLM Serving

Explore similar work

WorkflowGen:an adaptive workflow generation mechanism driven by trajectory experience

FlowBank: Query-Adaptive Agentic Workflows Optimization through Precompute-and-Reuse

MoFlow: Multi-Objective Agentic Workflow Generation