Constructing Extract-Load-Transform (ELT) pipelines is a labor-intensive data engineering task and a high-impact target for AI automation. On ELT-Bench, the first benchmark for end-to-end ELT pipeline construction, AI agents initially showed low success rates, suggesting they lacked practical utility. We revisit these results and identify two factors causing a substantial underestimation of agent capabilities. First, re-evaluating ELT-Bench with upgraded large language models reveals that the extraction and loading stage is largely solved, while transformation performance improves significantly. Second, we develop an Auditor-Corrector methodology that combines scalable LLM-driven root-cause analysis with rigorous human validation (inter-annotator agreement Fleiss' kappa = 0.85) to audit benchmark quality. Applying this to ELT-Bench uncovers that most failed transformation tasks contain benchmark-attributable errors -- including rigid evaluation scripts, ambiguous specifications, and incorrect ground truth -- that penalize correct agent outputs. Based on these findings, we construct ELT-Bench-Verified, a revised benchmark with refined evaluation logic and corrected ground truth. Re-evaluating on this version yields significant improvement attributable entirely to benchmark correction. Our results show that both rapid model improvement and benchmark quality issues contributed to underestimating agent capabilities. More broadly, our findings echo observations of pervasive annotation errors in text-to-SQL benchmarks, suggesting quality issues are systemic in data engineering evaluation. Systematic quality auditing should be standard practice for complex agentic tasks. We release ELT-Bench-Verified to provide a more reliable foundation for progress in AI-driven data engineering automation.
Figures & tables
Figure 1. Distribution of error attribution across 81 failed transformation tasks in ELT-Bench. Tasks are classified by error source—agent-attributable, benchmark-attributable, or mixed—and further stratified by mitigability, distinguishing errors addressable through evaluation refinements from those requiring ground truth column removal.
Figure 2. Overview of a typical ELT pipeline. Data is first extracted from heterogeneous sources—such as databases, APIs, flat files—using connectors like Airbyte ( Airbyte, 2025 ) . It is then loaded in raw form into a cloud data warehouse such as Snowflake ( Dageville et al., 2016 ) . Finally, transformation logic, typically written as SQL models in dbt ( dbt Labs, 2025 ) , reshapes the raw data into analytics-ready tables within the warehouse.
Figure 3. Agent workflow in ELT-Bench, illustrating the two-stage pipeline the agent must construct. EL Stage: The agent reads connection details from config.yaml , writes Terraform configurations to define Airbyte ( Airbyte, 2025 ) sources, the Snowflake ( Dageville et al., 2016 ) destination, and their connectors, deploys the configuration, and triggers synchronization jobs that extract data from heterogeneous sources (databases, APIs, flat files) and load it into the warehouse as staging tables. The agent then uses a provided Python script in the workspace to poll job status and proceeds to the T stage once all jobs succeed (i.e., all source tables for the task have been loaded into Snowflake). T Stage: The agent initializes a dbt ( dbt Labs, 2025 ) project, authors SQL transformation models that reshape the staged data into the target data models defined in data_model_schema.yaml , and executes dbt run , which generates the target data model(s) in the Snowflake data warehouse.
Figure 4. Pipeline outcomes across extraction/loading and transformation stages.
Figure 5. Overview of the Auditor-Corrector framework. The Auditor proceeds in three phases: Phase 1 constructs a structured analysis environment for each failed task from the evaluation log; Phase 2 deploys an LLM agent per task that autonomously reverse-engineers the correct SQL for each unmatched column, verifying each derivation against the ground truth; Phase 3 involves manual categorization of all 660 reports into an error taxonomy. The Corrector then applies category-specific corrections—evaluation script refinements and removal of columns with unreliable ground truth—to produce ELT-Bench-Verified.
Figure 6. Performance progression across three configurations, all using SWE-Agent: the original model (Claude Sonnet 3.5), the upgraded model (Claude Sonnet 4.5), and the upgraded model evaluated on ELT-Bench-Verified. The EL stage is largely solved after the model upgrade ( × 2.6), while the T stage benefits from both model improvement ( × 22.7) and benchmark correction ( × 1.4).
Attribution
Error Category
Count
%
Agent
Flawed SQL Logic
221
33.5%
JOIN Type Errors
157
23.8%
Wrong Data Source
38
5.8%
Domain Knowledge Gaps
16
2.4%
Key Generation Errors
6
0.9%
NULL Handling Errors
4
0.6%
Table 1. Distribution of error categories across 660 unmatched columns.
Benchmark
SRDEL
SRDT
Models Passed
ELT-Bench (original)
96%
22.66%
46/203
ELT-Bench-Verified
96%
32.51%
66/203
Table 2. Agent performance before and after benchmark correction. Both evaluations use SWE-Agent with Claude Sonnet 4.5.
Configuration
SRDT
Models Passed
ELT-Bench (original)
22.66%
46/203
+ Evaluation script refinement only
30.05%
61/203
+ GT column removal only
24.63%
50/203
ELT-Bench-Verified (both)
32.51%
66/203
Table 3. Ablation of correction strategies. All evaluations use SWE-Agent with Claude Sonnet 4.5. The baseline is the original ELT-Bench evaluation.
Agent
Benchmark
SRDEL
SRDT
SWE-Agent w/ Claude Sonnet 4.5
ELT-Bench
96%
22.66%
ELT-Bench-Verified
96%
32.51%
Baseline (ReAct) w/ Claude Sonnet 4.5
ELT-Bench
98%
20.20%
ELT-Bench-Verified
98%
32.51%
Table 4. Multi-agent validation. Performance of two agent configurations on the original and corrected benchmark.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Interpretation
Selected User(s)
Match
Most posts
User 805 (1,720 posts)
98.07%
Highest reputation
User 919 (rep. 87,393)
98.63%
Most badges
User 919
98.63%
Most profile views
User 919 (20,932 views)
98.63%
Most upvotes
User 1036
99.74%
Reputation = 995
Users 3432 & 8165
100%
Appendix
Table 5. Alternative interpretations of “most influential user” tested against the ground truth (91,966 posts total). No standard interpretation reproduces the GT; only an undocumented filter ( Reputation = 995 ) achieves 100%.
User
Rep.
Posts
UpVotes
Views
GT=1
805
65,272
1,720
7,035
5,680
0
919
87,393
1,204
11,273
20,932
0
3432
995
22
79
82
22
8165
995
34
66
81
34
Appendix
Table 6. User comparison for the POSTED_BY_MOST_INFLUENTIAL_USER column. The GT flags only users with Reputation = 995 as “most influential,” ignoring users with far higher reputation, post counts, and engagement metrics.
Country
GT (%)
Pred (decimal)
Ratio
Aruba
4.41
0.0441
100.0
Angola
− 16.73
− 0.1673
100.0
Albania
28.20
0.2820
100.0
UAE
3.04
0.0304
100.0
Argentina
5.24
0.0524
100.0
Appendix
Table 7. Format mismatch for GNP_GROWTH_RATE . The agent’s formula is correct; only the output scale differs. All 178 non-null values exhibit an exact 100 × ratio.
Team
GT
Pred
Diff
Ruch Chorzow
10
16
− 6
Jagiellonia Bialystok
14
11
+ 3
Lech Poznan
15
19
− 4
P. Warszawa
10
9
+ 1
Cracovia
6
9
− 3
S.C. Olhanense
7
5
+ 2
Appendix
Table 8. Season interpretation ambiguity for NUM_WON_2010 . The agent uses the 2009/2010 season; the ground truth implies a different interpretation. Only ∼ 40% of teams match under the agent’s reading.
Metric
SWE-Agent
ReAct
SRDEL
96%
98%
SRDT
32.51%
32.51%
Eligible data models
194
199
Correct data models
66
66
Matched columns
1,854
1,986
Unmatched columns
348
411
Appendix
Table 9. Detailed comparison of agent performance on ELT-Bench-Verified. Both agents pass 66/203 data models but differ in column-level behavior and fully correct databases.