ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

Pranay Kothari

cs.LGOct 1, 2026

ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time

Authors: Pranay Kothari

Organizations: Department of Computer Science, University of Oxford

Abstract

Benchmarks for agent-generated data work grade a pipeline by running it once against a fixed snapshot. ArrivalBench instead re-executes the pipeline an agent leaves behind under adversarial but replayable delivery schedules (late, duplicated, out-of-order and retried records) and requires its final state to equal a batch recomputation of the complete log. Because the oracle recomputes rather than classifies, a wrong table and a crash are distinct verdicts: a crash is visible to monitoring a team already runs, and a wrong table is not. On 40 tasks we built, our reimplementation of single-execution grading certifies 86-100% of the pipelines eleven models produce; re-executing the same artifacts finds 7.0-79.2% of the certified ones silently wrong. The gap is not produced by the repair loop: within the same model and task, pipelines repaired against the snapshot test fail replay about as often as those that passed it first time. In every model, idempotency hazards fail more often than ordering hazards. Separating a wrong answer from a crash also changes how interventions read: a hazard warning cuts one model's silent failure from 48.2% to 10.5% while raising its crash rate from 9.0% to 37.0%, so all-in failure moves only from 51.0% to 44.0%. All eleven arms were independently re-run, and rates moved by at most 5.9 points.

Figures & tables

Figure 1: Overview. The agent sees only the snapshot test while it writes a pipeline. The stored artifact is then graded twice: on a fixed input, as the seven stateful benchmarks we audited do, and under schedules that deliver the same records late, twice, out of order, after a crash or as a backfill. The oracle recomputes the target from the complete log, so a wrong table and a crash are separate verdicts.

Model	Organisation	snap.	gap	95% CI	ord.	idem.	weak.	mean
		saturation			signature
Haiku 4.5	Anthropic	98.9%	52.2%	[37.4, 66.9]	78.7%	51.1%	$+27.6$	$+25.5$
Sonnet 5	Anthropic	100.0%	48.2%	[32.8, 63.6]	98.9%	51.8%	$+47.1$	$+30.6$
Opus 5	Anthropic	100.0%	16.7%	[8.5, 26.4]	100.0%	84.3%	$+15.7$	$+5.9$
GPT-5.6 Sol	OpenAI	100.0%	7.0%	[1.5, 14.5]	100.0%	93.0%	$+7.0$	$+7.0$
Qwen3-Coder-Next	Alibaba	99.2%	73.7%	[60.6, 85.7]	48.0%	37.0%	$+11.0$	$+11.4$

Table 1: All unhinted arms, 40 tasks,

k=5

. The left half reports the incumbent methodology against ours; the right half reports the failure signature, where ord. is the weaker of the two ordering hazards, idem. the weakest of the three idempotency hazards and weak. their difference in points. That weakest-link form favours a positive margin by construction; mean is the difference between the two groups’ mean pass rates, which does not. Open-weights arms (lower block) were served through a hosted API aggregator. We report current models only, one per organisation except DeepSeek, whose V4 Flash and V4 Pro are concurrent tiers of one generation.

Figure 2: Snapshot-pass (grey) against silent-failure rate (orange), eleven models from eight organisations. Error bars are 95% bootstrap CIs over tasks on both series; most snapshot intervals are too tight to resolve.

Figure 3: Temporal-pass rate per hazard for the eleven unhinted arms, means over tasks. The vertical rule separates the two ordering hazards from the three idempotency hazards. Schema evolution is excluded as the loud hazard (Section 3 ). Each rate rests on 128–200 cells.

one logical log, delivered as	revenue	rows
a single in-order batch	425.00	3
redelivered in a later batch	425.00	3
redelivered within one batch	675.00	4

Figure 4: Staging step of a generated pipeline that passes snapshot evaluation and is graded FAIL on duplicate delivery. The guard is correct against history, so a later redelivery is absorbed and a within-batch one is not: revenue is overstated by 59% and no error is raised.

Figure 5: (a) Within one arm, cells that kept a within-batch guard and cells that dropped it fail at different rates and in different ways: 40.3% of the 67 guarded cells crash on duplicate delivery and none fails silently, while 72.2% of the 133 unguarded cells fail silently and none crashes. (b) The same test run independently in each arm: dropping the guard raises silent failure in 12 of 13; the exception, DeepSeek V4 Pro (blue, dashed), is a tie rather than a reversal.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Grading	Exec.	Varied	FAIL vs
			delivery	ERROR
Spider 2.0 † [ Lei et al., 2025 ]	execution match	1	—	—
BEAVER † [ Chen et al., 2024 ]	execution match	1	—	—
ELT-Bench [ Jin et al., 2025 ]	CSV diff	1	—	—
KramaBench [ Lai et al., 2025 ]	sub-task scores	1	—	—
DA-Code [ Huang et al., 2024 ]	output match	1	—	—

Appendix

Table 2: How the nine audited benchmarks grade, with Gaia2 and this work for contrast. Exec. is how many times the produced artifact is run for grading; FAIL vs ERROR is whether a silently wrong result and a crash get different verdicts. Rows marked

\dagger

are text-to-SQL: a query holds no state, so single execution follows from the artifact type rather than from a choice. Gaia2 grades a trajectory, not an artifact.

Arm	run 1	run 2	$\Delta$	failing tasks recurring
Haiku 4.5	52.2%	54.1%	$+1.9$ pp	22 of 22
Sonnet 5	48.2%	44.4%	$-3.8$ pp	19 of 20
Opus 5	16.7%	21.6%	$+4.9$ pp	12 of 14
GPT-5.6 Sol	7.0%	8.5%	$+1.5$ pp	5 of 5
Codestral	79.2%	79.1%	$-0.1$ pp	30 of 33
Qwen3-Coder-Next	73.7%	75.6%	$+1.9$ pp	29 of 30

Appendix

Table 3: Independent replication of all eleven unhinted arms. Rates move; the identity of the failing pipelines largely does not.

\Delta

is the difference of the two rates as printed, so every row reconciles on the page.

Arm	weakest ord.	weakest idem.	strongest idem.	weak-link	conserv.
Haiku 4.5 + hint	84.7%	62.7%	73.7%	$+22.0$ pp	$+11.0$ pp
Sonnet 5 + hint	100.0%	89.5%	100.0%	$+10.5$ pp	$0.0$ pp
Opus 5 + hint	100.0%	100.0%	100.0%	$0.0$ pp	$0.0$ pp
GPT-5.6 Sol + hint	100.0%	100.0%	100.0%	$0.0$ pp	$0.0$ pp
Haiku 4.5 + arch	100.0%	97.4%	100.0%	$+2.6$ pp	$0.0$ pp
Sonnet 5 + arch	100.0%	44.5%	100.0%	$+55.5$ pp	$0.0$ pp

Appendix

Table 4: The signature rule on the seven intervention arms, the ones not shown in Figure 3 . Rates are per-hazard temporal-pass. Ties arise where an intervention saturates idempotency, not where the ordering falls below it.

Arm	gap	error	all-in	$\Delta$ all-in
Haiku 4.5	52.2%	30.5%	67.5%	—
+ hazard hint	35.1%	29.5%	54.5%	$-13.0$
+ architecture	0.0%	32.0%	32.0%	$-35.5$
Sonnet 5	48.2%	9.0%	51.0%	—
+ hazard hint	10.5%	37.0%	44.0%	$-7.0$
+ architecture	54.4%	14.0%	61.5%	$+10.5$

Appendix

Table 5: Each intervention arm against its own unhinted baseline. The gap falls in six of seven; all-in failure, which counts crashes too, diverges from it wherever the error rate moves, most sharply for Sonnet 5 with the hazard hint.

Arm	turn 1	final	lift
Haiku 4.5	82.5%	98.9%	$+16.4$
Sonnet 5	95.5%	100.0%	$+4.5$
Opus 5	94.5%	100.0%	$+5.5$
GPT-5.6 Sol	100.0%	100.0%	$0.0$
Codestral	64.0%	86.0%	$+22.0$
Qwen3-Coder-Next	68.0%	99.2%	$+31.2$

Appendix

Table 6: Snapshot-pass before and after the repair loop, means over tasks. The loop is worth up to 32.5 points, and the arms it helps most are the ones the incumbent metric would otherwise separate.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows
Oct 1, 2026Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing +2Enterprise Analytics AI Agent Benchmarks
TextQL
No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage
Aug 15, 2026Shiven KhurdiAI Agent Reliability Agent Failure Analysis
Northeastern University
ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
Apr 2, 2026Akshey Sigdel, Rista BaralTool-Use Evaluation Tool-Using Agents
Independent Researcher

Oct 1, 2026J/K move · Enter open · S save

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand+1

TextQL

Enterprise Analytics AI Agent Benchmarks Synthetic Benchmark Generation Data Analysis Agents

Open paper AskPDF arXiv

Real-world enterprise data science and analytics workflows require reasoning across dozens of tables, performing statistical analyses, and acting on the results. Established text-to-SQL benchmarks evaluate query generation alone, and audits have found their answer keys frequently wrong. Because real enterprise warehouses are too sensitive to release, these benchmarks are built on public datasets where a business event fits in a single table. We introduce Argo-Bench, an evaluation framework comprising 210 data science and analytics tasks. Drawing on public data, peer-reviewed industry literature, and regulatory filings, we simulate a food delivery platform in New York City at true scale, with 81 million orders in 2024, grounded economics, fraud patterns, and marketplace incentives. We export this world to an ERP warehouse of 235 tables and 7.5 billion rows, modeled on the Oracle E-Business Suite schema. The simulator's ground-truth state is withheld from the warehouse the agent sees, so tasks require reconstructing facts by navigating the warehouse before acting on them. Argo-Bench goes beyond text-to-SQL: the agent files actions such as banning fraudulent accounts, allocating courier incentive budgets, or issuing back pay, and the grader scores each by its consequences in the simulator. Every task has an executable reference solution that demonstrates solvability using only the warehouse. The strongest of 14 frontier and open-weight models scores 95 or higher on only 34.8% of tasks and averages 59.5 points. We hope Argo-Bench drives progress toward agents that understand, navigate, and act within real data environments.

Abstract

AI Agent Benchmarks

Explore similar work

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage

ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems