Operon: Incremental Construction of Ragged Data via Named Dimensions

Sungbin Moon; Jiho Park; Suyoung Hwang; Donghyun Koh; Seunghyun Moon; Minhyeong Lee

cs.PLNov 20, 2025

Operon: Incremental Construction of Ragged Data via Named Dimensions

Authors: Sungbin Moon, Jiho Park, Suyoung Hwang, Donghyun Koh, Seunghyun Moon, Minhyeong Lee

Organizations: Asteromorph, Republic of Korea

Abstract

Modern data processing workflows frequently encounter ragged data: multidimensional collections whose inner lengths vary and are revealed only at runtime. Ragged data is prevalent in domains throughout science and computing, yet existing workflow engines treat it as nested containers. This representation leaves shape semantics ambiguous, obscures task dependencies when upstream coordinates are not yet enumerated, and imposes coarse synchronization barriers that stall independent work across tasks. We argue that ragged data is fundamentally a flat function over a coordinate space, and that making this view explicit resolves all three difficulties at once. We realize this insight in Operon, a statically verifiable workflow engine built around a formalism of named dimensions with explicit dependency relations. Users declare pipelines in the Operon DSL by annotating data flow with dimension names, where the static checker verifies that every task does not include faulty iterations regardless of runtime lengths. At runtime, the system models in-progress arrays as partial functions over downward-closed coordinate domains, which provides a natural representation for partially-unknown data states, an exact notion of per-task indegree before all upstream coordinates are enumerated, and a sound basis for fault recovery without programmer intervention.

Figures & tables

Pseudocode	Operon DSL examples (Accepted / Rejected)
Reducing across outer dimension. ⬇ for j $\in$ foo [0]. coords do bar [ j ] $\leftarrow$ f ( foo[..][j] )	⬇ Baz < i > = baz (); Qux < j > = qux (); Foo = foo ( Baz , Qux ) for i , j ; Bar = f ( Foo<i> ) for j ;	⬇ Baz < i > = baz (); Foo < j > = foo ( Baz ) for i ; Bar = f ( Foo<i> ) for j ;
Zipping. ⬇ for ( i , j ) $\in$ foo . coords do qux [ i ][ j ] $\leftarrow$ g ( foo [ i ][ j ], bar[i][j] )	⬇ Baz < i > = baz (); Foo < j > = foo ( Baz ) for i ; Bar = bar ( Foo ) for i , j ; Qux = g ( Foo , Bar ) for i , j ;	⬇ Baz < i > = baz (); Foo < j > = foo ( Baz ) for i ; Bar < k > = bar ( Baz ) for i ; Qux = g ( Foo , Bar ) for i , j ;
Broadcasting. ⬇ for ( i , j ) $\in$ foo . coords do qux [ i ][ j ] $\leftarrow$ h ( foo [ i ][ j ], bar[j] )	⬇ Baz < i > = baz (); Bar < j > = bar (); Foo = foo ( Baz , Bar ) for i , j ; Qux = h ( Foo , Bar ) for i , j ;	⬇ Baz < i > = baz (); Foo < j > = foo ( Baz ) for i ; Bar < k > = bar (); Qux = h ( Foo , Bar ) for i , j ;

Table 1. Shape ambiguity in Operon. Each row’s Pseudocode highlights an array operation whose safety depends on the shapes of its data so that the same operation can be correct in one pipeline and incorrect in another. The two Operon DSL columns place that operation in one context of each kind: a set of declarations Operon’s static checker accepts, and one it rejects. The checker decides each case without inspecting any runtime values.

Panel of Figure 3 — Figure 3. A resolution map $R$ on the dimension space $\left\{{\color[rgb]{0.5,0.5,0.5}\texttt{p}},{\color[rgb]{1,0,0}\texttt{s}},{\color[rgb]{0.5,0,0.5}\texttt{g}},{\color[rgb]{0,0,0}\texttt{f}}\right\}$ from Example 3.2 defining a ragged profile. For the single paper shown, there are 3 figures and 5 section s; each section contains 4, 3, 2, 0, and 3 paragraph s, respectively. This configuration uniquely defines the 36 possible positions for relevance score s, which are computed for each paragraph and each figure.

Pipeline	$\displaystyle\hskip 9.24994ptp::=\vec{t}$
Task	$\displaystyle\hskip 9.24994ptt::=\left<f,s_{\text{out}},\overrightarrow{s_{\text{in, }i}},\mathcal{F},n\right>$	$\displaystyle\text{unique }\tau_{\text{in, }i}$
Entity signature	$\displaystyle\hskip 9.24994pts::=\left<\tau,\mathcal{E}\right>$
Entity signature map	$\displaystyle\hskip 9.24994pt\Sigma::=\vec{s}$	$\displaystyle\text{unique }\tau$
Dim. space	$\displaystyle\hskip 9.24994pt\mathcal{D},\mathcal{E},\mathcal{F}::=\vec{d}$

Figure 4. Syntax of the Operon domain-specific language.

\boxed{\frac{\vphantom{d}}{(\emptyset,\emptyset,\emptyset)\mid()\vdash(\emptyset,\emptyset,\emptyset)}\kern 4.62497pt(\textsc{Unit})}

Figure 5. Static checking rules for the DSL.

TaskDef condition	Violating example	Rationale
$\tau_{\text{out}}\notin\dom(\Sigma)$	$\begin{aligned} &\texttt{A}=\texttt{alpha}();\\[-4.62497pt] &\hbox{\pagecolor{red!30!white}{A}}=\texttt{beta}();\end{aligned}$	Cannot redefine output entity types.
$\mathcal{E}_{\text{out}}\cap\mathcal{D}=\emptyset$	$\begin{aligned} &\texttt{A<i>}=\texttt{alpha}();\\[-4.62497pt] &\texttt{B<}\hbox{\pagecolor{red!30!white}{i}}\texttt{>}=\texttt{beta}();\end{aligned}$	Cannot redefine dimensions.
$\|\mathcal{E}_{\text{out}}\|\leq 1$	$\texttt{A<i,}\hbox{\pagecolor{red!30!white}{j}}\texttt{>}=\texttt{alpha}();$	Must output at most one dimension.
$\tau_{\text{in, }i}\in\dom(\Sigma)$	$\begin{aligned} &\texttt{A}=\texttt{alpha}(\hbox{\pagecolor{red!30!white}{B}});\\[-1.84995pt] &\texttt{B}=\texttt{beta}();\end{aligned}$	Must define input entities before use.
$\mathcal{E}_{\text{in, }i}\subseteq\Sigma(\tau_{\text{in, }i})$	$\begin{aligned} &\texttt{A<i>}=\texttt{alpha}();\\[-4.62497pt] &\texttt{B}=\texttt{beta}(\texttt{A<i,}\hbox{\pagecolor{red!30!white}{j}}\texttt{>});\end{aligned}$	Can only collect input entities across the characterizing dimensions. Here, $\mathcal{E}_{\text{in, }0}=\left\{\texttt{i},\texttt{j}\right\}\not\subseteq\Sigma(\texttt{A})=\left\{\texttt{i}\right\}$ .
$\begin{aligned} &\Sigma(\tau_{\text{in, }i})\setminus\mathcal{E}_{\text{in, }i}\\ &\hskip 9.24994pt\text{closed under }\prec\end{aligned}$	$\begin{aligned} &\texttt{A<i>}=\texttt{alpha}();\\[-1.84995pt] &\texttt{B<j>}=\texttt{beta}(\texttt{A})\;\texttt{for}\;\texttt{i};\\[-4.62497pt] &\texttt{C}=\texttt{gamma}(\hbox{\pagecolor{red!30!white}{B<i>}})\;\texttt{for}\;\texttt{j};\end{aligned}$	Cannot collect across parent dimensions while leaving some child dimensions free. We have $\Sigma(\texttt{B})=\left\{\texttt{i},\texttt{j}\right\}$ with $\texttt{i}\prec\texttt{j}$ , and collecting B over $\mathcal{E}_{\text{in, }0}=\left\{\texttt{i}\right\}$ would result in a nonsense list with null values.

Table 2. Concrete examples of errors that Operon’s static checking rule TaskDef rejects. The specific error locations are highlighted in red.

Panel of Figure 6 — Figure 6. Idealized flowchart of an individual scheduler.

Scenario	check	rebuild	run	Result	Scenario	check	rebuild	run	Result
Termination (no corruption)					Metadata corruption
Graceful quit	${\boldsymbol{-}}$	${\boldsymbol{-}}$	✓	✓	Missing tickets ( Done )	✓	✓	✓	✓
Force quit	✓	✓	✓	✓	Fake tickets ( Done )	✗	${\boldsymbol{-}}$	${\boldsymbol{-}}$	${\boldsymbol{-}}$
User function error	✓	✓	✓	✓	Missing resolutions	✗	${\boldsymbol{-}}$	${\boldsymbol{-}}$	${\boldsymbol{-}}$
Storage plug-out	✓	✓	✓	✓	Corrupt resolutions	✗	${\boldsymbol{-}}$	${\boldsymbol{-}}$	${\boldsymbol{-}}$
Crash during rebuild	✓	✓	✓	✓	Fake resolutions	✓	✓	✓	✓

Table 3. Recovery behavior under representative scenarios. Each column denotes the outcome of a recovery step. ✓ : success, ✗ : rejected / error,

{\boldsymbol{-}}

: not applicable.

Panel of Figure 7 — Figure 7. Empirical evaluation of Operon (red) against Prefect (blue). ( 7(a) ) Total execution time versus the sleep interval $t_{\text{sleep}}$ at $N=20$ ; the reference line “Theory” (gray) is the theoretical minimum $\left(\left\lceil\frac{\text{\# of vlm\_evaluate}}{64}\right\rceil+1\right)\times t_{\text{sleep}}$ . ( 7(b) ) Generated Row entries over time at $N=100$ , $t_{\text{sleep}}=3$ s.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8. Recovery overhead for the four-task workflow (Figure 8(a) ) as the number of tasks $N$ and the per-task fanout $M$ vary. Both axes are log-scaled.

Figure 9. Total execution time of Operon (red) and Prefect (blue) versus the number of PaperId s

N

t_{\text{sleep}}=3

s. Companion to Figure 7(a) in the body. The reference line “Theory” (gray) is the theoretical minimum

\left(\left\lceil\frac{\text{\# of vlm\_evaluate}}{64}\right\rceil+1\right)\times t_{\text{sleep}}

		$t_{\text{sleep}}=3$
		$N=5$	$N=10$	$N=15$	$N=20$	$N=40$	$N=60$	$N=80$	$N=100$
Operon	Try 1	58.15	109.45	140.65	173.19	340.87	535.84	690.11	838.13
	Try 2	57.83	109.91	146.84	173.03	339.00	518.40	700.80	838.73
	Try 3	57.98	110.60	140.32	170.17	339.59	516.14	689.66	838.38
Prefect	Try 1	74.93	141.06	185.30	213.59	452.68	734.26	998.51	1290.88
	Try 2	78.00	139.22	199.10	217.88	453.80	752.55	1024.64	1238.67

Appendix

Table 4. Data of Figure 9

		$N=20$
		$t_{\text{sleep}}=0$	$t_{\text{sleep}}=1$	$t_{\text{sleep}}=2$	$t_{\text{sleep}}=3$	$t_{\text{sleep}}=4$	$t_{\text{sleep}}=5$
Operon	Try 1	8.32	57.64	113.75	169.98	226.22	282.10
	Try 2	8.49	57.70	113.97	169.84	227.54	281.93
	Try 3	8.52	57.70	113.84	169.93	225.94	282.00
Prefect	Try 1	126.87	144.07	175.22	218.61	269.51	314.82
	Try 2	125.55	138.22	181.05	217.82	272.58	312.72

Appendix

Table 5. Data of Figure 7(a)

			$N=1$
			$M=10$	$M=100$	$M=200$	$M=300$	$M=400$	$M=500$
	Total Data $=NM^{2}$		100	10000	40000	90000	160000	250000
check	trust-all	Try 1	1.9175	1.7510	1.6158	1.8316	2.0577	1.8906
		Try 2	1.8048	1.8620	1.9270	1.9247	1.5593	1.8480
		Try 3	1.7874	1.8889	1.9847	1.7197	2.4423	1.7151
	metadata-only	Try 1	15.517	23.335	50.750	69.403	96.157	147.44

Appendix

Table 6. Recovery time (ms) for

N=1

			$M=10$
			$N=1$	$N=100$	$N=400$	$N=900$	$N=1600$	$N=2500$
	Total Data $=NM^{2}$		100	10000	40000	90000	160000	250000
check	trust-all	Try 1	1.9619	2.0045	1.8124	1.8395	2.1228	2.0901
		Try 2	2.0380	1.6079	1.6423	1.8790	1.7685	1.6900
		Try 3	1.8129	1.7913	1.8274	1.7463	1.7276	1.7248
	metadata-only	Try 1	14.226	80.622	199.13	382.79	656.93	1064.6

Appendix

Table 7. Recovery time (ms) for

M=10

Incremental Computation

See all 12 papers

kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation
Mar 19, 2026Rohan Siva, Kai Cheung, Lichi Li +1Workflow Automation Tool-Augmented Language Model Agents
Cisco
A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability
May 18, 2026Ruitao Liu, Xinyang Tian, Shuo Chen +4High-Performance Computing Distributed Training
Tsinghua University · Scitix AI
AAFLOW: Scalable Patterns for Agentic AI Workflows
May 4, 2026Arup Kumar Sarker, Mills Staylor, Aymen Alsaadi +3LLM Agent Orchestration Agentic Workflows
Department of Computer Science, Biocomplexity Institute and Initiative University of Virginia Charlottesville, VA, USA · Department of Computer Science, Rutgers University New Brunswick, NJ, USA · Biocomplexity Institute and Initiative University of Virginia Charlottesville, VA, USA +1

Mar 19, 2026J/K move · Enter open · S save

kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation

Rohan Siva, Kai Cheung, Lichi Li, Ganesh Sundaram

Cisco

Workflow Automation Tool-Augmented Language Model Agents LLM Agent Orchestration Human-in-the-Loop AI

Open paper AskPDF arXiv

Modern machine learning systems rely on complex data engineering workflows to extract, transform, and load (ELT) data into production pipelines. However, constructing these pipelines remains time-consuming and requires substantial expertise in data infrastructure and orchestration frameworks. Recent advances in large language model (LLM) agents offer a potential path toward automating these workflows, but existing approaches struggle with under-specified user intent, unreliable tool generation, and limited guarantees of executable outputs. We introduce kRAIG, an AI agent that translates natural language specifications into production-ready Kubeflow Pipelines (KFP). To resolve ambiguity in user intent, we propose ReQuesAct (Reason, Question, Act), an interaction framework that explicitly clarifies intent prior to pipeline synthesis. The system orchestrates end-to-end data movement from diverse sources and generates task-specific transformation components through a retrieval-augmented tool synthesis process. To ensure data quality and safety, kRAIG incorporates LLM-based validation stages that verify pipeline integrity prior to execution. Our framework achieves a 3x improvement in extraction and loading success and a 25 percent increase in transformation accuracy compared to state-of-the-art agentic baselines. These improvements demonstrate that structured agent workflows with explicit intent clarification and validation significantly enhance the reliability and executability of automated data engineering pipelines.

Abstract

Incremental Computation

Explore similar work

kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation

A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

AAFLOW: Scalable Patterns for Agentic AI Workflows