ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities

Christopher Zanoli; Andrea Giovannini; Tengjun Jin; Ana Klimovic; Yotam Perlitz

cs.AIMar 31, 2026

ELT-Bench-Verified: Benchmark Quality Issues Underestimate AI Agent Capabilities

Authors: Christopher Zanoli, Andrea Giovannini, Tengjun Jin, Ana Klimovic, Yotam Perlitz

Organizations: IBM Research · ETH Zurich · University of Illinois (UIUC)

Abstract

Constructing Extract-Load-Transform (ELT) pipelines is a labor-intensive data engineering task and a high-impact target for AI automation. On ELT-Bench, the first benchmark for end-to-end ELT pipeline construction, AI agents initially showed low success rates, suggesting they lacked practical utility. We revisit these results and identify two factors causing a substantial underestimation of agent capabilities. First, re-evaluating ELT-Bench with upgraded large language models reveals that the extraction and loading stage is largely solved, while transformation performance improves significantly. Second, we develop an Auditor-Corrector methodology that combines scalable LLM-driven root-cause analysis with rigorous human validation (inter-annotator agreement Fleiss' kappa = 0.85) to audit benchmark quality. Applying this to ELT-Bench uncovers that most failed transformation tasks contain benchmark-attributable errors -- including rigid evaluation scripts, ambiguous specifications, and incorrect ground truth -- that penalize correct agent outputs. Based on these findings, we construct ELT-Bench-Verified, a revised benchmark with refined evaluation logic and corrected ground truth. Re-evaluating on this version yields significant improvement attributable entirely to benchmark correction. Our results show that both rapid model improvement and benchmark quality issues contributed to underestimating agent capabilities. More broadly, our findings echo observations of pervasive annotation errors in text-to-SQL benchmarks, suggesting quality issues are systemic in data engineering evaluation. Systematic quality auditing should be standard practice for complex agentic tasks. We release ELT-Bench-Verified to provide a more reliable foundation for progress in AI-driven data engineering automation.

Figures & tables

Figure 1. Distribution of error attribution across 81 failed transformation tasks in ELT-Bench. Tasks are classified by error source—agent-attributable, benchmark-attributable, or mixed—and further stratified by mitigability, distinguishing errors addressable through evaluation refinements from those requiring ground truth column removal.

Figure 2. Overview of a typical ELT pipeline. Data is first extracted from heterogeneous sources—such as databases, APIs, flat files—using connectors like Airbyte ( Airbyte, 2025 ) . It is then loaded in raw form into a cloud data warehouse such as Snowflake ( Dageville et al., 2016 ) . Finally, transformation logic, typically written as SQL models in dbt ( dbt Labs, 2025 ) , reshapes the raw data into analytics-ready tables within the warehouse.

Figure 3. Agent workflow in ELT-Bench, illustrating the two-stage pipeline the agent must construct. EL Stage: The agent reads connection details from config.yaml , writes Terraform configurations to define Airbyte ( Airbyte, 2025 ) sources, the Snowflake ( Dageville et al., 2016 ) destination, and their connectors, deploys the configuration, and triggers synchronization jobs that extract data from heterogeneous sources (databases, APIs, flat files) and load it into the warehouse as staging tables. The agent then uses a provided Python script in the workspace to poll job status and proceeds to the T stage once all jobs succeed (i.e., all source tables for the task have been loaded into Snowflake). T Stage: The agent initializes a dbt ( dbt Labs, 2025 ) project, authors SQL transformation models that reshape the staged data into the target data models defined in data_model_schema.yaml , and executes dbt run , which generates the target data model(s) in the Snowflake data warehouse.

Figure 4. Pipeline outcomes across extraction/loading and transformation stages.

Figure 5. Overview of the Auditor-Corrector framework. The Auditor proceeds in three phases: Phase 1 constructs a structured analysis environment for each failed task from the evaluation log; Phase 2 deploys an LLM agent per task that autonomously reverse-engineers the correct SQL for each unmatched column, verifying each derivation against the ground truth; Phase 3 involves manual categorization of all 660 reports into an error taxonomy. The Corrector then applies category-specific corrections—evaluation script refinements and removal of columns with unreliable ground truth—to produce ELT-Bench-Verified.

Figure 6. Performance progression across three configurations, all using SWE-Agent: the original model (Claude Sonnet 3.5), the upgraded model (Claude Sonnet 4.5), and the upgraded model evaluated on ELT-Bench-Verified. The EL stage is largely solved after the model upgrade (

\times

2.6), while the T stage benefits from both model improvement (

\times

22.7) and benchmark correction (

\times

1.4).

Attribution	Error Category	Count	%
Agent	Flawed SQL Logic	221	33.5%
	JOIN Type Errors	157	23.8%
	Wrong Data Source	38	5.8%
	Domain Knowledge Gaps	16	2.4%
	Key Generation Errors	6	0.9%
	NULL Handling Errors	4	0.6%

Table 1. Distribution of error categories across 660 unmatched columns.

Benchmark	SRDEL	SRDT	Models Passed
ELT-Bench (original)	96%	22.66%	46/203
ELT-Bench-Verified	96%	32.51%	66/203

Table 2. Agent performance before and after benchmark correction. Both evaluations use SWE-Agent with Claude Sonnet 4.5.

Configuration	SRDT	Models Passed
ELT-Bench (original)	22.66%	46/203
+ Evaluation script refinement only	30.05%	61/203
+ GT column removal only	24.63%	50/203
ELT-Bench-Verified (both)	32.51%	66/203

Table 3. Ablation of correction strategies. All evaluations use SWE-Agent with Claude Sonnet 4.5. The baseline is the original ELT-Bench evaluation.

Agent	Benchmark	SRDEL	SRDT
SWE-Agent w/ Claude Sonnet 4.5	ELT-Bench	96%	22.66%
	ELT-Bench-Verified	96%	32.51%
Baseline (ReAct) w/ Claude Sonnet 4.5	ELT-Bench	98%	20.20%
	ELT-Bench-Verified	98%	32.51%

Table 4. Multi-agent validation. Performance of two agent configurations on the original and corrected benchmark.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Interpretation	Selected User(s)	Match
Most posts	User 805 (1,720 posts)	98.07%
Highest reputation	User 919 (rep. 87,393)	98.63%
Most badges	User 919	98.63%
Most profile views	User 919 (20,932 views)	98.63%
Most upvotes	User 1036	99.74%
Reputation = 995	Users 3432 & 8165	100%

Appendix

Table 5. Alternative interpretations of “most influential user” tested against the ground truth (91,966 posts total). No standard interpretation reproduces the GT; only an undocumented filter ( Reputation = 995 ) achieves 100%.

User	Rep.	Posts	UpVotes	Views	GT=1
805	65,272	1,720	7,035	5,680	0
919	87,393	1,204	11,273	20,932	0
3432	995	22	79	82	22
8165	995	34	66	81	34

Appendix

Table 6. User comparison for the POSTED_BY_MOST_INFLUENTIAL_USER column. The GT flags only users with Reputation = 995 as “most influential,” ignoring users with far higher reputation, post counts, and engagement metrics.

Country	GT (%)	Pred (decimal)	Ratio
Aruba	4.41	0.0441	100.0
Angola	$-$ 16.73	$-$ 0.1673	100.0
Albania	28.20	0.2820	100.0
UAE	3.04	0.0304	100.0
Argentina	5.24	0.0524	100.0

Appendix

Table 7. Format mismatch for GNP_GROWTH_RATE . The agent’s formula is correct; only the output scale differs. All 178 non-null values exhibit an exact 100

\times

ratio.

Team	GT	Pred	Diff
Ruch Chorzow	10	16	$-$ 6
Jagiellonia Bialystok	14	11	$+$ 3
Lech Poznan	15	19	$-$ 4
P. Warszawa	10	9	$+$ 1
Cracovia	6	9	$-$ 3
S.C. Olhanense	7	5	$+$ 2

Appendix

Table 8. Season interpretation ambiguity for NUM_WON_2010 . The agent uses the 2009/2010 season; the ground truth implies a different interpretation. Only

\sim

40% of teams match under the agent’s reading.

Metric	SWE-Agent	ReAct
SRDEL	96%	98%
SRDT	32.51%	32.51%
Eligible data models	194	199
Correct data models	66	66
Matched columns	1,854	1,986
Unmatched columns	348	411

Appendix

Table 9. Detailed comparison of agent performance on ELT-Bench-Verified. Both agents pass 66/203 data models but differ in column-level behavior and fully correct databases.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

May 25, 2026

Automated Benchmark Auditing for AI Agents and Large Language Models

Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environment specifications, and brittle evaluation logic that human annotation cannot reliably catch. We introduce Auto Benchmark Audit (ABA), an agentic framework that systematically audits individual benchmark tasks, uncovering issues such as hidden environment dependencies, specification gaps, and limited grading logic. We run ABA on a collection of frontier LLM benchmarks and previous NeurIPS publications, totaling 168 benchmarks across nine domains. Across this corpus, ABA identifies critical issues including ambiguous task design, execution environment conflicts, and incorrect ground truths in over 25.7% of the evaluated tasks. The precision of these automated audits is validated by expert review and independent third-party reports such as upstream PRs. Crucially, we demonstrate that these problematic tasks severely distorts capability assessments for agents and LLMs: filtering out these tasks with issues shifts model rankings and increases average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively. We release the agentic tool and all task annotations to support the future development of frontier benchmarks.

Junlin Wang, Federico Bianchi, Shang Zhu +4

Duke University · Together AI · Stanford University

LLM Evaluation Benchmark Auditing Software Engineering Benchmarks AI Agent Benchmarks Benchmark Validity

Ask PDF

Apr 27, 2026

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

As benchmarks grow in complexity, many apparent agent failures are not failures of the agent at all - they are failures of the benchmark itself: broken specifications, implicit assumptions, and rigid evaluation scripts that penalize valid alternative approaches. We propose employing frontier LLMs as systematic auditors of evaluation infrastructure, and realize this vision through BenchGuard, the first automated auditing framework for task-oriented, execution-based agent benchmarks. BenchGuard cross-verifies all benchmark artifacts via structured LLM protocols, optionally incorporating agent solutions or execution traces as additional diagnostic evidence. Deployed on two prominent scientific benchmarks, BenchGuard identified 12 author-confirmed issues in ScienceAgentBench - including fatal errors rendering tasks unsolvable - and exactly matched 83.3% of expert-identified issues on the BIXBench Verified-50 subset, catching defects that prior human review missed entirely. A full audit of 50 complex bioinformatics tasks costs under USD 15, making automated benchmark auditing a practical and valuable complement to human review. These findings point toward AI-assisted benchmark development, where frontier models serve not only as subjects of evaluation but as active participants in validating the evaluation infrastructure itself.

Xinming Tu, Tianze Wang, Yingzhou +4

Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1

Benchmark Auditing AI Agent Benchmarks Benchmark Validity

Ask PDF

Jul 29, 2026

Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

Capabilities of frontier models are often assessed using agentic benchmarks. To trust these results, benchmarks must accurately measure what they claim to and be free from invalidating flaws. Previous manual audits of benchmarks such as SWE-Bench-Verified have uncovered several validity issues in transcripts. However, manual review is difficult to scale, and it is unclear whether automated methods can reliably surface flaws that compromise benchmark validity. In this paper, we developed AI scanners to detect four types of validity issues: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. We produced grading rubrics for each to instruct human labeling, and evaluated the scanners against human labels on a held-out test set of Inspect Evals benchmarks. Our scanners identified several verified quality issues in five widely used benchmarks, including cases unlikely to be caught by random manual inspection. Not all cases were identified, and scanner performance varied substantially across benchmarks, criteria and models. We highlight several open challenges to be addressed to improve scanners for stronger quality assurance claims, including broader standardization gaps in the evaluation field that degrade scanner performance. Together, these results serve as a proof of concept for using automated transcript analysis to audit benchmark quality more broadly.

Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6

Independent · UK AI Security Institute · MARS +4

Benchmark Auditing AI Agent Benchmarks Benchmark Validity Automated Evaluation

Ask PDF

Abstract

AI Agent Benchmarks

Explore similar work

Automated Benchmark Auditing for AI Agents and Large Language Models

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks