RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases

Yao Cheng; Siqiang Luo

cs.LGOct 15, 2026

RDL-Bench: A Benchmark for Task-Aware Question Answering over Relational Databases

Authors: Yao Cheng, Siqiang Luo

Organizations: College of Computing and Data Science, Nanyang Technological University, Singapore

Abstract

Relational databases (RDBs) remain the primary infrastructure for storing and managing structured data across enterprise and scientific domains. Natural-language (NL) interfaces to RDBs aim to let users ask questions without writing SQL, but real user needs go beyond SQL-style lookup: some questions require reasoning over the observable database state, while others require predictive inference from historical relational records under temporal constraints. Existing benchmarks address these goals in isolation. To address this gap, we introduce RDL-Bench, a benchmark for task-aware question answering (QA) over real RDBs. RDL-Bench organizes diverse user questions into reasoning-oriented questions, whose answers are determined by the current database state, and prediction-oriented questions, whose answers depend on future outcomes under temporal constraints. Each instance contains an NL question, task metadata, schema information, temporal context, either executable evidence or target-construction logic, and denotation-level ground truth. RDL-Bench supports two evaluation settings: native solvers under their intended capabilities, and task-aware modular systems where a router dispatches each question to an appropriate execution module.

Figures & tables

Fig. 1: Overview of RDL-Bench in the relational QA landscape. RDL-Bench places current-state reasoning and future-oriented prediction under a shared NL interface and evaluates both native solvers and task-aware modular systems. Each instance contains six components: NL question, task metadata, schema information, evidence or target-construction logic, temporal context, and denotation-level ground truth.

Benchmark	Reasoning	Predictive	Routing	Multi-table
WikiTableQ [ 27 ]	✓	✗	✗	✗
Spider [ 44 ]	✓	✗	✗	✓
BIRD [ 19 ]	✓	✗	✗	✓
STARQA [ 26 ]	✓	✗	✗	✓
RelBench [ 30 ]	✗	✓	✗	✓
RDL-Bench	✓	✓	✓	✓

TABLE I: Comparison of related benchmarks.

Name	Domain	# Tables	#Rows	Temporal	Source
Olist	E-commerce	9	100,000	Yes	https://www.kaggle.com/datasets/olistbr/brazilian-ecommerce/data
Financial	Financial	8	1,090,086	Yes	https://web.archive.org/web/20180506061559/http://lisp.vse.cz/pkdd99/Challenge/chall.htm
NCAA	Sports	9	201,555	Yes	https://www.kaggle.com/c/march-machine-learning-mania-2015/data
Hepatitis	Medicine	7	12,927	No	https://www.cs.sfu.ca/~oschulte/jbn/dataset.html
Airline	Retail	19	448,156	Yes	https://www.transtats.bts.gov
Stats	Social Network	8	1,027,838	Yes	https://archive.org/details/stackexchange

TABLE II: Statistics of the seven domains in RDL-Bench. Diversity is reflected in the varying number of tables, relational depth, and the presence of temporal event streams.

Fig. 2: Example of a predictive instance.

Fig. 5: Dataset statistics of RDL-Bench. (a) Instance distribution by domain and query type. (b) Track distribution. (c) Reasoning sub-type breakdown. (d) Complexity distribution.

Family	Method	Reasoning	Predictive	Input	Role
LLM-based	LLM-Schema	✓	–	Question, schema	Schema-only reasoning solver
	LLM-2Hop	✓	–	Question, schema, 2-hop subgraph	Context-augmented reasoning solver
	LLM-only	–	✓	Schema, profiling statistics	Data-blind predictive reference
	LLM-GraphCtx	–	✓	Schema, time-filtered 2-hop subgraph	Graph-context predictive reference
Text-to-SQL	DIN-SQL [ 28 ]	✓	–	Question, schema	Prompting-based SQL parser
	C3 [ 8 ]	✓	–	Question, schema	Zero-shot SQL parser

TABLE III: Baseline methods in RDL-Bench.

Panel of Figure 7 — Fig. 6: Prompt template for LLM-Schema. This baseline evaluates schema-only reasoning over the observable database state.

Panel of Figure 8 — Fig. 7: Prompt template for LLM-2Hop. This baseline augments schema-only reasoning with local relational context.

Family	Method	Olist	Financial	NCAA	Hepatitis	Airline	Stats	Dallas	Avg.
LLM-based	Claude-Schema	0.65	0.45	0.49	0.50	0.47	0.42	0.44	0.49
	Claude-2Hop	0.64	0.47	0.44	0.41	0.48	0.43	0.43	0.47
	Qwen32B-Schema	0.57	0.48	0.51	0.41	0.46	0.42	0.43	0.47
	Qwen32B-2Hop	0.58	0.40	0.46	0.41	0.36	0.45	0.31	0.42
	Llama8B-Schema	0.38	0.32	0.31	0.37	0.35	0.34	0.21	0.33
	Llama8B-2Hop	0.37	0.35	0.48	0.41	0.36	0.34	0.21	0.36

TABLE IV: Native Reasoning Track results by domain. Each domain score averages per-instance scores using normalized EM for Scalar, Set-F1 for Set, Accuracy for Boolean, and NDCG@5 for Ranking answers. All scores are in

[0,1]

, higher is better. The final column reports the unweighted mean over the seven domains.

		Scalar		Set		Boolean		Ranking
Family	Method	EM $\uparrow$	nMAE $\downarrow$	F1 $\uparrow$	EM $\uparrow$	Acc $\uparrow$	AUROC $\uparrow$	NDCG@5 $\uparrow$	$\rho\uparrow$
LLM-based	Claude-Schema	0.76	0.51	0.62	0.41	0.83	–	0.01	0.01
	Claude-2Hop	0.81	0.43	0.52	0.35	0.80	–	0.05	0.04
	Qwen32B-Schema	0.76	0.10	0.32	0.19	0.66	–	0.03	0.02
	Qwen32B-2Hop	0.76	0.94	0.32	0.20	0.65	–	0.05	0.04
	Llama8B-Schema	0.61	1.18	0.26	0.14	0.50	–	0.03	0.02

TABLE V: Reasoning Track performance by answer type. Primary and secondary metrics are reported for each answer type. All metrics except nMAE are in

[0,1]

, higher is better. AUROC is not applicable (–) for LLM-based methods as they output discrete Yes/No without confidence scores.

Family	Method	Olist	Financial	NCAA	Airline	Stats	Dallas	Macro Avg.
LLM-based	Claude-only	0.88	0.72	0.61	0.52	0.91	0.63	0.71
	Qwen32B-only	0.66	0.64	0.51	0.54	0.59	0.57	0.59
	Llama8B-only	0.74	0.53	0.50	0.44	0.60	0.58	0.57
	Claude-GraphCtx	0.89	0.74	0.65	0.55	0.92	0.67	0.74
	Qwen32B-GraphCtx	0.68	0.66	0.55	0.56	0.61	0.61	0.61
	Llama8B-GraphCtx	0.75	0.55	0.53	0.46	0.61	0.62	0.59

TABLE VI: Native Predictive Track classification results by domain. Each cell reports Accuracy over Boolean Predictive tasks instances. The final column reports the unweighted macro average over the six temporal domains. Higher is better.

Family	Method	Olist	Financial	NCAA	Airline	Stats	Dallas	Macro Avg.
LLM-based	Claude-only	0.99	0.97	11.07	$\mathbf{0.66}$	1.43	1.11	2.70
	Qwen32B-only	1.08	3.35	$\mathbf{0.47}$	1.05	1.61	1.15	1.45
	Llama8B-only	0.94	1.00	0.56	50.04	1.88	0.94	9.23
	Claude-GraphCtx	0.97	0.94	10.21	0.69	1.38	1.05	2.54
	Qwen32B-GraphCtx	1.05	3.10	0.48	1.02	1.55	1.10	1.38
	Llama8B-GraphCtx	0.92	0.98	0.54	48.73	1.82	0.91	8.98

TABLE VII: Native Predictive Track regression results by domain. Each cell reports nMAE over Scalar Predictive Track instances instantiated as entity-level regression tasks. The final column reports the unweighted macro average over the six temporal domains. Lower is better.

Generator	Claude-Schema	Qwen32B-Schema
Claude	0.57	0.54
Qwen	0.58	0.51

TABLE VIII: Cross-generator Reasoning evaluation on Olist and NCAA. Rows denote question generators and columns denote solvers. Higher scores are better.

Panel of Figure 14 — Fig. 8: Prompt template for LLM-Router. This setting evaluates whether an LLM can recover task route metadata from a natural-language question.

Router	Track Acc.	QType F1	Ans. Acc.	Table F1	Module Acc.
Random-Router	0.49	0.46	0.49	0.00	0.49
Llama8B-Router	0.29	0.25	0.29	0.78	0.29
Qwen32B-Router	0.95	0.93	0.95	0.91	0.95
Claude-Router	0.97	0.96	0.97	0.94	0.97
Oracle-Router	1.00	1.00	1.00	1.00	1.00

TABLE IX: Routing diagnosis results on all test instances. Higher is better.

System	Reasoning $\uparrow$	Bool. Acc $\uparrow$	Scalar nMAE $\downarrow$	Track Acc. $\uparrow$
Random-Router	0.20	0.43	1.06	0.49
Llama8B-Router	0.03	0.43	1.91	0.29
Qwen32B-Router	0.44	0.72	1.10	0.95
Claude-Router	0.44	0.84	0.98	0.97
Oracle-Router	0.44	0.87	0.91	1.00

TABLE X: End-to-end modular QA results with fixed execution modules.

Fig. 9: Performance trends across difficulty levels. Panel (a) reports Reasoning Track denotation scores, where higher is better. Panel (b) reports Predictive Classification AUROC, where higher is better. Panel (c) reports Predictive Regression nMAE, where lower is better.

Fig. 10: Reasoning Track performance by query type.

Fig. 11: Predictive Track performance (Accuracy) by method.

Family	Method	Avg. Runtime
Predictive	XGBoost	28.3s
	LightGBM	25.2s
	GraphSAGE	35.0s
	RelGNN	34.4s
	RelGT	34.8s
	StructOpt-RDL	29.1s

TABLE XI: Average runtime of representative solvers. Predictive solvers are reported in seconds over the six temporal domains. LLM-based routing and reasoning pipelines are reported in minutes over seven domains. Lower is better.

Table QA

Table question answering studies how AI systems retrieve, interpret, and reason over structured data to answer natural-language questions.

See all 65 papers

A Fair Benchmarking of Deep Relational Database Learning Models
Jul 4, 2026Kazi F. Akhter, Bharath Ajendla, Manar D. SamadTransformer Graph Representation Learning
Department of Computer Science Tennessee State University Nashville, TN, USA · SAP AI Research and Innovation SAP Labs Palo Alto, CA, USA
Relational Task Generation Language: A Declarative Specification Framework for Relational Deep Learning
Sep 1, 2026Oleksii Kolesnichenko, Jakub Peleška, Gustav ŠírRelational Deep Learning
Czech Technical University in Prague, Karlovo náměstí 13, Prague, 121 35, Czechia
Relational In-Context Learning via Synthetic Pre-training with Structural Prior
Mar 4, 2026Yanbo Wang, Jiaxuan You, Chuan Shi +1Relational ICL Relational Foundation Models
Institute for Artificial Intelligence, Peking University · University of Illinois at Urbana-Champaign · Institute of Computing Technology, Beijing University of Post and Telecommunication +1

Jul 4, 2026J/K move · Enter open · S save

A Fair Benchmarking of Deep Relational Database Learning Models

Kazi F. Akhter, Bharath Ajendla, Manar D. Samad

Department of Computer Science Tennessee State University Nashville, TN, USA · SAP AI Research and Innovation SAP Labs Palo Alto, CA, USA

Transformer Graph Representation Learning Relational Deep Learning Tabular Prediction Relational Databases Database Systems

Open paper AskPDF arXiv

Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols, making fair comparison difficult. We present one of the first systematic benchmarking studies of recently released deep learning methods for RDBs, evaluating them across five relational databases, with one classification and one regression task for each. We refactor all deep RDB models to allow the full range of experimental procedures to be applied consistently across all methods. Our findings indicate that the relational transformer (RT) approach delivers the strongest overall performance on both classification and regression tasks compared to the state-of-the-art graph-based modeling and learning of RDBs. Even for single-table learning tasks, deep learning methods designed for RDBs outperform the leading tabular foundation model, TabPFN 2.5. Extending learning from a single table (hop = 0) to multiple tables (hop = 1, 2) by connecting neighboring tables in relational databases enhances performance, but the additional benefit from higher hops diminishes as computational overhead grows. Deep RDB learning methods have the potential to challenge state-of-the-art tabular foundation models, especially on large-scale enterprise data. The source code for this benchmarking study is publicly available.

Abstract

Table QA

Explore similar work

A Fair Benchmarking of Deep Relational Database Learning Models

Relational Task Generation Language: A Declarative Specification Framework for Relational Deep Learning

Relational In-Context Learning via Synthetic Pre-training with Structural Prior