Designing data integration pipelines typically requires substantial manual effort from data engineers to configure pipeline components and label training data. While LLMs have shown promise in handling individual steps of the integration process, their potential to replace all human input across end-to-end data integration pipelines has not been investigated. As a step toward exploring this potential, we present an automatic data integration pipeline that uses GPT-5.2 to generate all artifacts required to adapt the pipeline to specific use cases. These artifacts are schema mappings, value mappings for data normalization, training data for entity matching, and validation data for selecting conflict resolution heuristics in data fusion. We compare the performance of this LLM-based pipeline to the performance of human-designed pipelines along three case studies requiring the integration of video game, music, and company related data. Our experiments show that the LLM-based pipeline is able to produce similar results, for some tasks even better results, as the human-designed pipelines. End-to-end, the human and the LLM pipelines produce integrated datasets of comparable size and density. Having the LLM configure the pipelines costs approximately $10 per case study, which represents only a small fraction of the cost of having human data engineers perform the same tasks.
TABLE I: Dataset statistics for the three case studies.
Case Study
Human
LLM
Label-based
Instance-based
Games
1.00
1.00
.23
.00
Companies
1.00
1.00
.35
.32
Music
1.00
1.00
.38
.24
Average
1.00
1.00
.32
.19
TABLE II: Schema matching F1 scores (macro-averaged across datasets).
LLM Mappings
Code-based
Source
Col.
Norm.
Total
Col.
Norm.
Total
Games
DBpedia
1
38,268
46,170
2
83,439
91,341
Metacritic
2
38,712
38,712
3
59,204
59,206
Sales
1
7,877
7,877
3
23,631
23,631
Companies
DBpedia
1
6,395
6,878
2
16,429
16,963
Forbes
1
1,895
1,957
1
1,895
1,957
TABLE III: Value normalization statistics by method and case study.
Human
Human
LLM
Use Case
Dataset Pair
Config
Labels
Labels
Games
DBpedia–Metacritic
.930
.826
.849
DBpedia–Sales
.927
.839
.979
Companies
DBpedia–Forbes
.857
.954
.939
Forbes–FullContact
.870
.898
.897
Music
Discogs–MusicBrainz
.800
.991
.990
TABLE IV: Entity matching F1 scores on held-out test sets. Human Config : the data engineers’ pipeline. Human Labels and LLM Labels : identical ML matchers trained on human-labeled vs. LLM-labeled training data.
Human
Human
LLM
LLM
Use Case
Config
Val. Set
Val. Set
+ RAG
Games
.832
.866
.866
.866
Companies
.803
.861
.709
.744
Music
.766
.721
.708
.708
Average
.800
.816
.761
.773
TABLE V: Data fusion test accuracy by validation set origin.
Metric
Games
Companies
Music
Data Sources
3
3
3
Total Input Records
74,951
14,016
37,255
Fused Record Groups
7,235
1,031
4,178
Output Records
65,518
12,768
30,885
Fusion Ratio
11.0%
8.1%
13.5%
Row Gain vs. Largest
+18,938
+2,683
+8,258
TABLE VI: End-to-end integration metrics for the three case studies.
Output Records
Fusion Ratio
Row Gain %
Density Δ
Use Case
Human
LLM
Human
LLM
Human
LLM
Human
LLM
Games
65,794
65,518
11.8
11.0
+41.3
+40.7
+2.7
+4.5
Companies
11,517
12,768
15.1
8.1
+14.2
+26.6
− 5.2
+5.6
Music
30,779
30,885
14.6
13.5
+36.0
+36.5
+5.8
− 1.8
TABLE VII: End-to-end integration metrics comparing human versus LLM pipelines.
Execution
Configuration
Use Case
LLM
Human
LLM
Human
Games
181 s
73 s
1.9 h
≥ 18 h
Companies
38 s
34 s
1.7 h
≥ 20 h
Music
45 s
124 s
2.2 h
≥ 20 h
Average
88 s
77 s
1.9 h
≥ 19 h
TABLE VIII: Average runtime per use case. Execution includes normalization, matching, and fusion. Configuration refers to one-time artifact generation. Human configuration times are lower-bound estimates.
Data integration is the process of combining data from multiple, heterogeneous sources into a consistent, unified representation. Data integration involves a sequence of interdependent tasks including schema matching, value normalization, blocking, entity matching, and data fusion. Existing table-based benchmarks either evaluate these steps in isolation or cover only incomplete versions of the data integration pipeline, omitting specific steps. The lack of public end-to-end data integration benchmarks hinders research on data integration methods that address the integration process as a whole and account for the interdependencies among the different tasks. This paper fills this gap by introducing the Mannheim Data Integration Benchmark (MaDI-Bench), the first benchmark for the end-to-end integration of relational tables covering all steps of the integration process. MaDI-Bench contributes (i) a set of end-to-end data integration tasks spanning several application domains, each requiring the full schema matching, value normalization, entity matching, and data fusion pipeline, and (ii) a generic method for deriving task variants that mitigates rapid benchmark saturation as data integration systems advance. We validate the benchmark using human-engineered pipelines, a best-of-breed pipeline, an LLM workflow, and a pipeline written by a coding agent. The validation demonstrates the utility of the benchmark for measuring the step-wise as well as the end-to-end performance of data integration pipelines. All benchmark artifacts are available for public download.
Aaron Steiner, Ralph Peeters, Christian Bizer
Data and Web Science Group University of Mannheim Mannheim, Germany
The goal of schema integration is, given a set of input schemata or tables, to derive a global, unified schema that is able to represent the concepts, attributes, and relationships of all input tables in a coherent fashion. This paper presents SINT-Flow, a schema integration framework composed of five LLM-based operators that can be combined into workflows to perform fully automated, end-to-end schema integration. In contrast to existing approaches, SINT-Flow can process denormalized source tables that contain attributes describing multiple entity types. During the schema integration process, these tables are decomposed into separate entity-specific relations. To evaluate SINT-Flow, we introduce SINT-Bench, a schema integration benchmark comprising 10 schema integration tasks consisting of altogether 93 relational tables, including tables that describe multiple types of entities. We evaluate SINT-Flow using GPT-5.2 as well as the open-weight model Qwen-3.6-27B as alternative backbone models. Using these models, SINT-Flow achieves F1 scores of at least 96% for entity-type detection, 85% for attribute detection, and 83% for schema mapping. Furthermore, we perform an ablation study to prove the utility of the applied self-consistency strategy as well as the inclusion of a review loop into the schema matching operator.
Large language models (LLMs) are increasingly used to automate data-processing workflows, yet coding agents typically produce scripts that are not automatically materialized as persistent, editable platform artifacts. We call this disconnect the NL2Pipeline gap. To bridge it, we introduce DataFlow-Harness, a platform that guides an LLM agent to construct platform-native directed acyclic graphs (DAGs) through typed, incremental mutations rather than free-form scripts. The platform combines DataFlow-Skills for procedural guidance, a Model Context Protocol (MCP) layer that exposes the live operator registry and current pipeline state, and DataFlow-WebUI, which synchronizes conversational authoring with a visual DAG editor. On a 12-task data-engineering benchmark, DataFlow-Harness achieves a 93.3% observed end-to-end pass rate. Relative to Vanilla Claude Code, it reduces measured monetary cost by 72.5% and generation latency by 49.9%; its observed pass rate is within 0.9 percentage points of the Context-Aware Claude Code baseline while its cost is 42.8% lower. Per-task analysis indicates that Skills are most useful when construction depends on implicit procedural knowledge. These results show that live platform grounding can produce persistent, editable workflow artifacts with an observed reliability close to script-generation baselines and with lower measured construction cost and latency.
Runming He, Zhen Hao Wong, Hao Liang +4
Peking University · Institute for Advanced Algorithms Research, Shanghai · Zhongguancun Academy