Temporal reasoning over evolving semi-structured tables poses a challenge to current QA systems. We propose an approach that recasts the task as automated knowledge base construction: (1) prompting an LLM to synthesize a 3NF-compliant relational schema from Wikipedia infobox timelines, (2) populating the schema to obtain a queryable database, and (3) generating and executing SQL queries against it, with QA accuracy serving as an extrinsic evaluation of the constructed knowledge base. In a controlled grid of three schema generators crossed with six query models, the schema source accounts for 79.5% of the exact match (EM) variance against 1.6% for the query model: replacing the schema, and the prompt scaffolding derived from it, shifts EM by 14.7 to 20.0 points, whereas replacing the query model under a fixed schema shifts it by 4.4 to 12.1. From this evidence, we distill three candidate schema-design principles: balanced normalization, semantic naming, and consistent temporal anchoring, framed as correlational hypotheses. Our best configuration (Gemini 2.5 Flash schemas + Gemini-2.0-Flash queries) reaches 80.39 EM, 11.5 points above the strongest reported baseline (68.89 EM); an open-weights configuration reaches 79.52.
Figures & tables
Figure 1: Our three-stage pipeline, read left to right. Stage 1, Schema Generation : an LLM turns infobox timelines into a 3NF schema. Stage 2, Schema Population : scripts clean and insert each JSON snapshot, one per snapshot_id . Stage 3, SQL Generation & Execution : a schema-guided prompt turns a question into SQL, executed against the database. Stages 1–2 run once per domain, offline. Worked example: Appendix A.1 .
Baseline
Schema Generation LLMs
IRE+CoT
Gemini 2.5 Flash
Llama-3.3-70B-Instruct
Llama-3.1-8B-Instruct
SQL/CoT LLMs
EM
F1
EM
F1
EM
F1
EM
F1
Gemini-2.0-Flash
48.98
55.93
80.39
82.11
72.70
73.33
60.37
60.32
Qwen-2.5-7B-Instruct
30.40
29.22
77.84
78.86
75.30
75.07
60.91
60.32
Llama-3.1-8B-Instruct
29.83
37.95
78.08
79.49
78.70
77.85
61.78
60.66
Llama-3.3-70B-Instruct
41.08
54.91
69.86
75.54
79.52
79.91
64.79
63.66
Table 1: Exact Match (EM) and F1 scores on TransientTables across schema generators (columns) and SQL query generators (rows).
Source
df
Sum Sq.
% of total
F
Schema generator
2
737.9
79.5
21.08 ∗
Query model
5
15.2
1.6
0.17
Residual (interaction)
10
175.0
18.9
Total
17
928.2
100.0
Table 2: Two-way analysis of variance without replication over the 18 EM scores in Table 1 , with schema generator and query model as the two factors. df : degrees of freedom. Sum Sq. : sum of squared deviations attributable to that source. F : the source’s mean square (Sum Sq. divided by df) over the residual mean square, so F≫1 marks a source explaining far more variation than noise. With one observation per cell, the schema × query-model interaction cannot be separated from error and serves as the residual. ∗p<0.001 ; the query-model effect is not significant ( p=0.97 ).
Avg. outer-join depth per query
Tables per domain
Domain
Flash
Pro
70B
8B
Flash
Pro
70B
8B
country
2.12
2.00
2.67
2.40
5
5
7
7
cricket_team
2.31
2.67
2.14
2.82
6
7
8
8
cricketer
0.00
0.00
0.00
0.00
3
4
3
7
economy
0.48
0.67
0.67
0.51
2
2
4
18
table_tennis_player
0.05
1.75
0.00
1.40
4
5
12
5
Table 3: Structural properties of the generated schemas (Flash = Gemini-2.5-Flash, Pro = Gemini-2.5-Pro, 70B = Llama-3.3-70B-Instruct, 8B = Llama-3.1-8B-Instruct). Left: average outer-join depth per executed query, from the outermost SELECT only. Right: relational tables generated per domain.
Issue (#Samples)
Error Category
Count
Data Quality (35)
Wrong Calculations
15
Empty Results
12
Wrong Entity Mapping
5
Precision/Format Issues
3
SQL Generation (12)
Aggregate Function Misuse
6
Syntax Errors
3
Table 4: Error distribution from 50 failure cases. Data quality dominates over SQL generation and schema issues.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Gemini-2.0-Flash
Llama-3.3-70B-Instruct
GPT-4o-mini
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
80.00
84.30
70.84
70.26
66.50
78.44
48.67
48.13
81.50
63.78
67.59
66.38
cricket_team
82.70
80.00
67.99
66.58
75.61
84.50
65.00
60.27
79.00
55.72
61.80
60.66
cricketer
78.70
82.60
85.79
85.79
64.00
68.00
58.30
58.91
72.00
87.80
87.80
87.80
economy
84.30
81.40
46.87
46.87
62.57
73.12
54.12
54.12
82.10
81.50
81.50
81.50
table_tennis_player
76.70
78.30
49.60
48.30
80.21
78.10
67.23
68.91
65.00
60.67
61.00
61.00
Appendix
Table 5: Query Generation Models: Performance Comparison of Query Generation Models (Gemini-2.0-Flash, Llama-3.3-70B-Instruct, GPT-4o-Mini) across various domains. The models’ query generation quality is evaluated using Exact Match (EM), F1-score (F1), R-1 (Rouge-1), and R-L (Rouge-L) metrics. The schema used for the queries was generated using Gemini 2.5 Flash.
Domain
Gemini-2.5-Pro
Qwen-2.5-7B-Instruct
Llama-3.1-8B-Instruct
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
66.00
78.43
78.43
75.23
78.14
81.24
65.41
65.18
76.24
80.11
66.54
64.78
cricket_team
87.00
78.34
78.34
75.30
79.92
77.00
62.13
61.35
78.31
77.10
61.98
61.23
cricketer
60.50
60.50
60.50
60.50
76.50
79.50
80.95
80.89
78.20
80.54
78.41
77.69
economy
80.50
60.00
60.00
60.00
81.00
77.93
42.17
42.08
80.15
78.46
50.35
52.71
table_tennis_player
50.00
48.83
48.83
48.83
74.50
75.50
44.42
43.46
76.40
77.23
47.56
46.79
Appendix
Table 6: Performance Comparison of three Query Generation Models (Gemini-2.5-Pro, Qwen-2.5-7B-Instruct, and Llama-3.1-8B-Instruct) across diverse domains. The models’ query generation quality is evaluated using Exact Match (EM), F1-score (F1), R-1 (Rouge-1), and R-L (Rouge-L) metrics. Llama-3.1-8B-Instruct demonstrated the highest average performance with an F1 score of 79.49 and an EM score of 78.08. The schema used for the queries was generated using Gemini 2.5 Flash.
Domain
Llama-3.1-8B-Instruct
Qwen 2.5 7B Instruct
Llama-3.3-70B-Instruct
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
82.25
79.22
65.71
68.24
78.34
76.22
65.93
67.12
81.17
81.80
65.18
62.86
cricket_team
81.97
80.03
66.29
67.12
76.99
74.56
63.88
65.45
77.52
79.28
62.70
62.52
gov_agencies
80.89
81.10
63.98
66.52
73.11
74.55
61.99
62.99
80.05
81.79
65.32
66.95
economy
81.99
78.89
64.91
67.77
76.45
74.22
63.22
65.11
78.80
77.57
66.37
64.28
table_tennis_player
80.45
76.99
62.88
65.55
74.28
75.10
62.37
63.49
78.80
78.09
63.29
65.97
Appendix
Table 7: Performance Evaluation of three Query Generation Models (Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Llama-3.3-70B-Instruct) across diverse domains. The models’ query generation quality is assessed using Exact Match (EM), F1-score (F1), R-1 (Rouge-1), and R-L (Rouge-L) metrics. Llama-3.3-70B-Instruct achieved the highest overall average scores, leading with an average F1 of 79.91 and an average EM of 79.52. The schema used for these queries was generated by Llama-3.3-70B-Instruct.
Domain
GPT-4o-mini
Gemini-2.5-pro
Gemini-2.0-Flash
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
79.44
80.95
64.69
64.10
79.88
77.00
64.37
60.81
71.91
74.15
59.57
60.33
cricket_team
80.07
79.28
64.37
64.72
76.59
78.86
62.22
62.90
73.33
72.33
57.53
57.65
cricketer
80.06
77.26
65.90
60.58
80.04
77.96
66.06
61.75
75.55
75.05
58.61
60.31
economy
77.53
76.60
65.35
61.71
81.87
79.79
63.38
66.90
72.14
70.26
61.23
59.77
table_tennis_player
77.06
79.42
61.36
64.01
78.37
77.74
63.64
64.29
71.83
72.39
61.27
58.92
Appendix
Table 8: Query Generation Model Performance for GPT-4o-mini, Gemini-2.5-Pro, and Gemini-2.0-Flash evaluated across multiple domains. Both GPT-4o-mini and Gemini-2.5-Pro demonstrated strong, competitive results, with less than 1 point separating their average Exact Match (EM) and F1 scores. Gemini-2.0-Flash showed a lower average performance across all four metrics (EM, F1, R-1, R-L). The schema for generating these queries was derived from Llama-3.3-70B-Instruct.
Domain
Llama-3.1-8B-Instruct
Qwen-2.5-7B-Instruct
Gemini-2.5-pro
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
64.82
62.70
52.92
50.01
62.85
62.05
57.08
54.78
61.27
64.93
52.81
50.45
cricket_team
59.15
64.89
44.34
47.91
58.82
64.47
49.19
52.42
65.84
60.11
49.33
54.19
gov_agencies
62.43
57.01
46.87
43.12
59.80
64.44
49.54
48.32
60.55
63.89
53.94
52.37
economy
57.98
61.21
43.55
45.68
56.01
60.80
46.70
49.83
62.41
61.56
54.91
51.58
table_tennis_player
64.50
58.12
47.10
46.05
62.57
57.10
50.78
50.18
66.50
59.13
51.02
49.88
Appendix
Table 9: Performance comparison of Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, and Gemini-2.5-Pro using a schema generated by Llama-3.1-8B-Instruct. The models’ average scores across all metrics (EM, F1, R-1, R-L) are significantly lower than in previous tables, suggesting a more challenging evaluation set. Gemini-2.5-Pro was the top performer, achieving the highest average F1 score of 63.07.
Domain
Llama-3.3-70B-Instruct
GPT-4o-mini
Gemini-2.0-Flash
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
67.82
65.70
59.92
57.01
60.03
64.07
53.53
52.66
60.59
58.12
49.16
52.18
cricket_team
62.15
67.89
51.34
54.91
63.50
59.45
49.72
54.88
62.81
61.03
50.57
51.54
cricketer
65.43
60.01
53.87
50.12
66.82
65.59
51.34
50.93
57.34
60.05
53.54
50.25
economy
60.98
64.21
50.55
52.68
62.80
61.08
54.09
53.31
59.22
62.17
51.05
52.78
table_tennis_player
67.50
61.12
54.10
53.05
59.76
63.03
52.50
49.07
61.47
57.88
50.14
48.29
Appendix
Table 10: Query Generation Model Performance for Llama-3.3-70B-Instruct, GPT-4o-Mini, and Gemini-2.0-Flash. The evaluation uses the challenging schema generated by the Small Language Model (Llama-3.1-8B-Instruct). Despite the resultant drop in average scores, the larger Llama-3.3-70B-Instruct model successfully crossed previous performance baselines, leading the group with an average EM of 64.78 and F1 score of 63.66.
Domain
GPT-4o-mini
Llama-3.3-70B-Instruct
Qwen-2.5-7B-Instruct
EM
F1
R-1
R-L
EM
F1
R-1
R-L
EM
F1
R-1
R-L
country
68.50
79.03
80.38
76.85
64.50
79.00
79.88
76.35
61.50
73.66
77.88
74.60
cricket_team
52.50
60.22
60.25
56.76
65.00
73.06
73.25
69.76
53.00
60.22
60.25
57.01
cricketer
74.50
74.50
74.50
74.50
75.00
75.00
75.00
75.00
73.50
73.83
74.00
74.00
economy
47.00
51.50
51.50
49.46
55.00
60.50
62.00
58.50
44.50
49.00
49.00
47.00
table_tennis_player
51.50
54.21
57.50
57.50
53.00
55.81
57.00
57.00
43.00
46.06
47.50
47.50
Appendix
Table 11: Out-of-family schema check: performance of three query generation models (GPT-4o-mini, Llama-3.3-70B-Instruct, Qwen-2.5-7B-Instruct) across the ten domains, evaluated with Exact Match (EM), F1, ROUGE-1 (R-1), and ROUGE-L (R-L). GPT-5.4-mini is used only to synthesize the database schema that these queries are written against; it generates no queries itself. This is a partial sweep: only three of the six query models in Table 1 were run against this schema. Average EM nonetheless lands within 2.7 points of the Llama-3.1-8B schemas for every query model tested, despite the schema generator being a substantially more recent model.
Schema source
In
Out
Total
Best EM
Gemini 2.5 Flash
4040
116
4156
80.39
Llama-3.3-70B
4119
111
4230
79.52
Llama-3.1-8B
3598
154
3752
64.79
Appendix
Table 12: Average input and output tokens per SQL-generation call by schema source, averaged over six query models and ten domains, alongside the best EM achieved with that schema. A 13% spread in token cost accompanies a 15.6-point spread in accuracy.
Domain
Gemini-2.0-Flash
Llama-3.3-70B-Instruct
GPT-4o-mini
Gemini-2.5-Pro
Qwen-2.5-7B-Instruct
Llama-3.1-8B-Instruct
In
Out
In
Out
In
Out
In
Out
In
Out
In
Out
country
3155
100
3571
113
3214
102
3603
114
3286
104
3518
111
cricket_team
4450
119
4991
133
4435
119
5057
135
4388
117
4582
122
cricketer
4220
146
4774
165
4279
148
4736
163
4369
151
4782
165
economy
3327
79
3020
98
3427
101
3061
102
3010
101
3389
98
table_tennis_player
2857
59
2537
99
2546
103
2804
103
2863
102
2474
99
Appendix
Table 13: Average input and output token counts per SQL-generation call across six query models and ten domains, using Gemini 2.5 Flash as the schema generator.
Domain
Gemini-2.0-Flash
Llama-3.3-70B-Instruct
GPT-4o-mini
Gemini-2.5-Pro
Qwen-2.5-7B-Instruct
Llama-3.1-8B-Instruct
In
Out
In
Out
In
Out
In
Out
In
Out
In
Out
country
3475
110
3603
112
3576
103
3710
107
3517
103
3239
108
cricket_team
4864
130
4802
125
4693
126
4966
124
5175
139
4966
129
cricketer
4636
160
4751
171
4618
168
4501
168
4565
171
4704
149
economy
3286
78
3253
82
3181
73
3515
75
3104
83
3103
79
table_tennis_player
2737
57
2559
57
2890
60
2679
53
2610
60
2605
53
Appendix
Table 14: Average input and output token counts per SQL-generation call across six query models and ten domains, using Llama-3.3-70B-Instruct as the schema generator.
Domain
Gemini-2.0-Flash
Llama-3.3-70B-Instruct
GPT-4o-mini
Gemini-2.5-Pro
Qwen-2.5-7B-Instruct
Llama-3.1-8B-Instruct
In
Out
In
Out
In
Out
In
Out
In
Out
In
Out
country
3422
139
3187
140
3376
133
3198
136
3344
131
3486
146
cricket_team
4679
120
4452
112
4633
117
4971
126
4711
123
4652
121
cricketer
3370
142
3219
133
3186
148
3530
134
3299
133
3514
137
economy
3180
150
3025
154
3260
153
3272
156
3081
157
3079
153
table_tennis_player
3716
131
3835
128
3915
129
3877
136
3503
139
3743
132
Appendix
Table 15: Average input and output token counts per SQL-generation call across six query models and ten domains, using Llama-3.1-8B-Instruct as the schema generator. Output length is uniformly higher here than under the other two schema sources, tracking this generator’s heavier table fragmentation (Table 3 ).
Domain
Fragmentation Metrics
Remarks
C
R
RC
D
Country
1.000
1.000
1.000
1.000
All countries participate fully across snapshot and leadership fragments.
Cricket Team
0.992
1.000
1.000
1.000
A few snapshots lack captain assignments; captaincy is modeled as an optional extension.
Cricketer
0.929
1.000
1.000
1.000
Some players do not appear in statistical snapshots due to partial format participation.
Cyclist
0.444
1.000
1.000
1.000
Optional annotation fragments (nicknames, medals) cover only a subset of riders.
Economy
1.000
1.000
1.000
1.000
All countries are fully represented across economic snapshots.
Appendix
Table 16: Fragment metrics for schema made using Gemini-2.5. Values below 1.0 arise from domain semantics such as optional annotation fragments, realistic sparsity in achievements or leadership roles, and intentional one-to-many relationships, rather than schema or reconstruction errors.
Domain
Fragmentation Metrics
Remarks
C
R
RC
D
Country
0.863
1.000
1.000
1.000
HDI and Gini indicators are unavailable for a subset of countries, reflecting missing socio-economic coverage rather than key loss.
Cricket Team
0.974
1.000
1.000
1.000
One team lacks an associated coach entry; coaching data is optional and incomplete for a small subset.
Cricketer
1.000
1.000
1.000
1.000
All cricketers, years, formats, and statistics are fully covered across fragments.
Cyclist
1.000
1.000
1.000
1.000
All rider, ride, nickname, and snapshot identifiers participate completely in fragments.
Economy
0.000
1.000
1.000
1.000
The EconomyValues fragment contains no temporal records as it is empty; however, other economic fragments jointly reconstruct all time snapshots.
Appendix
Table 17: Fragment metrics for schema made using llama-3.1-8B. Across domains, deviations from perfect completeness arise primarily from optional or sparsely populated fragments (e.g., awards, indicators, leadership roles) and empty indicator tables, while losslessness is generally preserved through fragment unions. All schemas maintain strict referential consistency and disjointness, indicating correct key preservation and absence of duplicate fact units despite domain-specific sparsity.
Domain
Fragmentation Metrics
Remarks
C
R
RC
D
Country
1.00
1.00
1.00
1.00
All snapshots contain GDP, HDI, and Gini fragments; joins fully reconstruct original records.
Cricket Team
1.00
1.00
1.00
1.00
Rank and record fragments fully cover all team snapshots without duplication or FK violations.
Cricketer
1.00
1.00
1.00
1.00
Snapshot and career statistics form a perfectly complete and lossless temporal decomposition.
Cyclist
0.11
1.00
1.00
1.00
Amateur-career data is sparse by design; however, all rider identifiers remain recoverable via fragment union.
Economy
1.00
1.00
1.00
1.00
Each economic snapshot is associated with indicator values, preserving temporal coverage and reconstruction.
Appendix
Table 18: Fragmentation Metrics Evaluation (LLaMA 3.3 70B Instruct Schema). Most domains achieve perfect referential consistency and lossless reconstruction. Deviations from full completeness arise primarily from optional or sparsely populated fragments, while reduced disjointness reflects intentional one-to-many relationships such as officials, rankings, or equipment histories rather than schema errors.
Real-world data spans tables, documents, and semi-structured files with implicit semantics. Querying this data requires integrating evidence across inconsistent schemas and formats, yet existing approaches either demand costly manual engineering or bypass structure entirely. We present a system that automatically discovers an executable schema from raw multi-source data and uses it as a shared contract for knowledge graph construction and query-time retrieval. A closed-world field catalog constrains LLM-based schema discovery to attested fields; deterministic structural analysis infers identity keys, foreign keys, and source hierarchy; and the resulting schema drives extraction, deduplication, and cross-source linking into a provenance-aware knowledge graph. At query time the schema -- optionally extended via a monotonic protocol -- conditions a multi-tool agent routing retrieval across structured lookup, graph traversal, and vector search, returning grounded answers with traceable citations. In controlled zero-shot comparisons using the same LLM, data, and evaluation harness, the system improves over retrieval-only and decomposition-based baselines across four QA benchmarks, with ablations showing that schema-conditioned routing, structural intelligence, and schema-guided construction each contribute to the gains.
Traditional Text-to-SQL research and benchmarks assume a known target database, overlooking settings in which a query must be routed within a large, heterogeneous database collection. We therefore study schema linking in a multi-database setting, where the system must first locate the target database and then construct a compact, SQL-relevant schema for generation. We propose MDB-Link, a hierarchical schema-linking framework that retrieves question-relevant columns from a global index, aggregates retrieval evidence to shortlist databases, and uses a budget-aware large language model (LLM) for database reranking, table selection, and column grounding. With Qwen2.5-14B, MDB-Link outperforms LinkAlign on MMQA, Spider2-Snow, and BIRD-dev in database localization and column selection while producing schema subsets close in size to the gold schemas. Exact match improves from 16.88 to 51.41 on MMQA, 2.50 to 9.17 on Spider2-Snow, and 12.52 to 38.01 on BIRD-dev. MDB-Link also runs faster than LinkAlign and AutoLink, demonstrating the effectiveness of hierarchical schema reduction for downstream SQL generation.
Beiyu Xu, Zhenyu Wu, Jiaoyan Chen +1
University of Manchester Faculty of Science and Engineering Department of Computer Science
Large Language Models (LLMs) are increasingly used to generate structured outputs, but their reliability remains unclear when those outputs must satisfy database-level constraints. We study this issue through database normalization, involving reasoning about functional dependencies, lossless join decompositions, and inter-table constraints. We introduce a Database Normalization Benchmark (DNBENCH), comprising 3,275 samples for evaluating LLM-driven database normalization from 1NF to BCNF. DNBENCH uses a three-axis protocol to measure semantic equivalence, structural accuracy, and logical validity. Across Single, Complex, and Real World levels, DNBENCH uncovers recurring failures in dependency inference, schema decomposition, and inter-table constraint reconstruction. We further propose Multi-Agent Reasoning for Schemas (MARS), which separates evidence extraction, violation diagnosis, and decomposition planning from schema generation and verification. MARS improves the DNB-SCORE by 82.0% over the single-prompt baseline. All artifacts will be released upon acceptance.
Dong-Jae Koh, Huisu Kim, SeongHwan Yoon +4
School of Computer Science and Engineering, Kyungpook National University, South Korea