Predictive modeling over relational databases (RDBs) powers applications, yet remains challenging due to capturing both cross-table dependencies and complex feature interactions. Relational Deep Learning (RDL) methods automate feature engineering via message passing, while classical approaches like Deep Feature Synthesis (DFS) rely on predefined non-parametric aggregators. Despite performance gains, the comparative advantages of RDL over DFS and the design principles for selecting effective architectures remain poorly understood. We present a comprehensive study that unifies RDL and DFS in a shared design space and conducts architecture-centric searches across diverse RDB tasks. Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and (3) validation accuracy is an unreliable guide for architecture choice. This search yields a model performance bank that links architecture configurations to their performance; leveraging this bank, we analyze the drivers of the RDL-DFS performance gap and introduce two task signals -- RDB task homophily and an affinity embedding that captures size, path, feature, and temporal structure -- whose correlation with the gap enables principled routing. Guided by these signals, we propose Relatron, a task embedding-based meta-selector that chooses between RDL and DFS and prunes the within-family search. Lightweight loss-landscape metrics further guard against brittle checkpoints by preferring flatter optima. In experiments, Relatron resolves the "more tuning, worse performance" effect and, in joint hyperparameter-architecture optimization, achieves up to 18.5% improvement over strong baselines with 10x lower cost than Fisher information-based alternatives.
Figures & tables
Figure 1: (a) An example of generating the task table from an RDB. The label is based on whether a student has achieved an A+ in a course before a specific timestamp. (b) Another example demonstrating the working process of DFS and RDL. For DFS, a predefined set of aggregation functions, such as MEAN and COUNT , is used to aggregate information across multiple tables based on key relationships into a final data table. For comparison, RDL is claimed to replace the manual aggregation design with an automatic message-passing-based sparse attention.
Name
Architecture design space
Hyper-parameters
RDL
Structural feature
Message passing
Readout
Learning rate, dropout, normalization, fanout…
Labeling ID, Learnable embedding, None
PNA, HGT, Sage, RelGNN
MLP, ContextGNN, Shallow
DFS
Aggregation function
Aggregation layers
Backbone
Batch size, hidden dimension …
Max, Min, Mode, … (fixed)
1, 2, 3 (if possible)
TabPFN, LightGBM, FT-Transformer
Table 1: Search space of RDL and DFS-based methods for RDB tasks. Underline means these components will go over a grid search, while other components will be sampled.
Figure 2: Performance comparison between the best configurations from our design space and baseline models on entity-level tasks. “Best (ours)” means the better value of RDL and DFS. Full numerical results can be seen in Table 11 from Appendix E.5 .
Task
Model
reported perf
test-selected perf
val-selected perf
driver-top3 (auroc)
GraphSAGE
75.54
82.81
81.56
RelGNN
85.69
85.69
82.61
driver-position (mae)
GraphSAGE
4.022
3.91
3.93
RelGNN
3.798
3.80
4.35
user-ignore (auroc)
GraphSAGE
81.62
86.40
72.27
RelGNN
86.18
86.18
78.94
Table 2: The performance gap between validation- and test-selected configurations.
Figure 3: Augmenting rel-f1 databases. We should treat the set of FKs as a hyperedge (each pair of FKs is appended to the original PK-FK graphs as a new edge type) rather than relying solely on PK-FK edges.
Selection
Model type
Val_auroc
Test_auroc
Pbar
P1
P2
Val
RDL
89.48
82.41
2.77
1.49
4.23
Test
RDL
86.05
85.94
0.41
1.22
0.22
Val
DFS
83.44
84.69
0.384
0.041
1.32
Test
DFS
83.76
85.71
0.495
0.03
0.50
Table 3: Example landscape properties and model performance. Smaller values typically indicate a more benign landscape.
Figure 7
Bank size
Model-free (ours)
Training-free model-based
Anchor-based
8
68.9%
56.9%
62.9%
12
64.8%
50.4%
56.1%
16
76.2%
53.1%
62.5%
All
87.5%
66.7%
66.7%
Table 5: Ablation on performance-bank size for macro-level predictors (LOO accuracy).
Strategy
Budget
driver-top3 (ROC-AUC) ↗
driver-position (MAE) ↘
user-churn (ROC-AUC) ↗
Random
3
82.67 ± 2.19
3.6810 ± 0.4255
68.56 ± 1.23
10
77.80 ± 4.79
3.8576 ± 0.5035
68.60 ± 1.20
30
77.28 ± 2.42
4.2793 ± 0.1483
69.54 ± 0.43
TPE
3
82.67 ± 2.19
3.6810 ± 0.4255
68.56 ± 1.23
10
81.45 ± 0.44
3.7897 ± 0.4271
68.60 ± 1.20
30
77.92 ± 5.12
4.1724 ± 0.0519
69.54 ± 0.43
Table 6: Joint architecture and hyperparameter optimization result. Best results are highlighted with an underline and the second are bold . “Only predictor” means only using the meta-predictor. Relbench’s default refers to the default architecture and hyperparameters in their pipelines. Best fixed refers to the configurations with the best mean rank of performance across tasks.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Database Name
Task Name
Task Type
Metric
rel-f1
driver-dnf
classification
ROC-AUC
driver-top3
classification
ROC-AUC
driver-position
regression
MAE
driver-wins
regression
MAE
constructor-scores-points
classification
ROC-AUC
driver-position-change
regression
MAE
Appendix
Table 7: Summary of databases, tasks, task types, and evaluation metrics used in our experiments.
Module name
Possible choices
Feature encoding
ResNet ( Robinson et al., 2024 )
Structural features (grid)
Learnable embeddings (for the destination table, or for both the source and destination tables) ( Yuan et al., 2024 ; Ma et al., 2024 ) ; partial-labeling tricks from NBFNet ( Zhu et al., 2021 )
Message passing (grid)
Sparse message passing: GraphSAGE ( Robinson et al., 2024 ; Hamilton et al., 2017 ) , HGT ( Wang et al., 2024a ; Hu et al., 2020b ) , PNA ( Corso et al., 2020 ; Wang et al., 2024a ) ; sparse message passing with higher-order information: RelGNN ( Chen et al., 2025a )
Architecture design trick (random)
Residual connections ( He et al., 2016 )
Readout (grid)
MLP, ContextGNN, Shallow-Item ( Yuan et al., 2024 )
Hyperparameters (random)
Learning rate, weight decay, batch size, dropout rate, number of layers, hidden dimension, temporal sampling strategy, number of sampled neighbors
Appendix
Table 8: Design space of a unified architecture for end-to-end RDL models. “Grid” indicates that the module is selected from a predefined grid, whereas “random” indicates that the module is randomly sampled.
TabPFN ( Hollmann et al., 2023 ) , FT-Transformer, LightGBM
Hyper-parameters (random)
Learning rate, weight decay, scheduler gamma, hidden dimension, number of attention heads, normalization
Training objective
Classification tasks: cross-entropy loss; Regression tasks: MSE or MAE
Appendix
Table 9: Design space of graph-induced non-parametric feature synthesis (DFS) models. Grid means the module is selected from a predefined grid, while random means the module is randomly sampled.
Method name
Mean test clf perf
Mean val clf perf
Mean gap
RDL Overall
78.21
76.73
1.49
Sage (top1)
76.90
72.77
4.13
HGT (top2)
77.19
74.95
2.24
DFS overall
76.90
75.22
1.68
TabPFN
75.91
75.70
0.21
FT-Transformer
75.87
74.45
1.42
Appendix
Table 10: Validation-selected and test-selected performance gap. For RDL, we show the top 2 architectures with the largest gap.
Figure 5: HPO results for recommendation tasks.
Task
Type
RelGNN
RelGT
GraphSAGE
Rel-LLM
KumoRFM (icl)
KumoRFM (fine-tuned)
RDL (val-selected)
RDL (ours)
DFS (val-selected)
DFS (ours)
Best (ours)
Griffin
driver-top3
classification
85.69
83.52
75.54
82.22
91.07
99.62
82.41
85.94
84.70
85.71
85.94
77.95
driver-dnf
classification
75.29
75.87
72.62
77.15
82.41
82.63
74.35
77.20
76.89
79.42
79.42
70.91
driver-position
regression
3.798
3.920
4.022
3.967
2.747
2.731
4.0491
3.8029
3.3660
3.2730
3.2730
4.2000
user-churn
classification
70.93
69.27
69.88
70.55
67.71
71.23
70.98
70.98
65.29
68.23
70.98
68.04
item-sales
regression
0.0540
0.0536
0.0560
0.0520
0.0400
0.0340
0.0511
0.0509
0.0780
0.0750
0.0509
0.0810
post-votes
regression
0.0650
0.0654
0.0650
0.0620
0.0650
0.0650
0.0665
0.0651
0.0680
0.0660
0.0651
0.0622
Appendix
Table 11: Full experimental results for Figure 2 .
Figure 6: Relationship between test performance ranking and micro-level architecture choices. For RDL-based methods, we filter the original search result with the top 10 performing configurations on each task. For DFS-based methods, we filter the original search result with the top 1 performing configurations on each task. The best configurations are selected based on test performance directly. “zero_learn” is the labeling trick adopted by NBFNet. (a) A violin plot of MPNN types shows that PNA achieves the best mean ranking across different tasks. (b) When comparing labeling tricks, although equivariant models appear more frequently in top rankings, a labeling trick is still needed to achieve the top spot. (c) For DFS, ft_transformer and tabpfn present unique strengths, where the former can leverage more training samples, and the latter usually works better in small-scale settings and can conduct in-context learning.
Figure 7: Loss landscapes
driver-dnf
study-outcome
RDL (SAGE)
RDL (HGT)
DFS-1 (no p/p)
DFS-2 (no p/p)
DFS-3 (no p/p)
RDL (SAGE)
RDL (HGT)
DFS-1 (no p/p)
DFS-2 (no p/p)
DFS-3 (no p/p)
Preprocessing time
60 s
60 s
7 s/12 s
18 s/17 s
310 s/41 s
240 s
240 s
240 s/36 s
353 s/48 s
965 s/95 s
Training time (per epoch)
6 s
8 s
<1 s/ <1 s
<1 s/ <1 s
<1 s/ <1 s
10 s
11 s
<1 s/ <1 s
<1 s/ <1 s
<1 s/ <1 s
Appendix
Table 12: Preprocessing and training times by method for driver-dnf and study-outcome.
Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols, making fair comparison difficult. We present one of the first systematic benchmarking studies of recently released deep learning methods for RDBs, evaluating them across five relational databases, with one classification and one regression task for each. We refactor all deep RDB models to allow the full range of experimental procedures to be applied consistently across all methods. Our findings indicate that the relational transformer (RT) approach delivers the strongest overall performance on both classification and regression tasks compared to the state-of-the-art graph-based modeling and learning of RDBs. Even for single-table learning tasks, deep learning methods designed for RDBs outperform the leading tabular foundation model, TabPFN 2.5. Extending learning from a single table (hop = 0) to multiple tables (hop = 1, 2) by connecting neighboring tables in relational databases enhances performance, but the additional benefit from higher hops diminishes as computational overhead grows. Deep RDB learning methods have the potential to challenge state-of-the-art tabular foundation models, especially on large-scale enterprise data. The source code for this benchmarking study is publicly available.
Kazi F. Akhter, Bharath Ajendla, Manar D. Samad
Department of Computer Science Tennessee State University Nashville, TN, USA · SAP AI Research and Innovation SAP Labs Palo Alto, CA, USA
Relational databases store much of the world's structured information, and they are essential for driving complex predictive applications. However, deep learning progress on relational data remains limited, as conventional approaches flatten databases into single tables via manual feature engineering, discarding relational context. Relational deep learning (RDL) addresses this by modeling databases as relational entity graphs (REGs) for graph neural networks (GNNs), but remains task- and database-specific. To combine the strengths of both paradigms, we propose a hybrid architecture combining a fine-tuned BART encoder to capture intra-row semantics with a GraphSAGE-based GNN over REGs to inject relational context. Experiments on RelBench show that the GNN substantially enriches BART's row embeddings, achieving a ROC-AUC of 67.40 on the driver-dnf task from the rel-f1 dataset. This performance is competitive with supervised baselines such as LightGBM (68.86) and narrows the gap to RDL (72.62) to within 5.22 points, though a substantial gap remains to state-of-the-art foundation models such as KumoRFM (82.63). These results suggest that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path towards foundation models for relational databases.
Relational deep learning (RDL) converts relational databases (RDBs) into heterogeneous graphs, but graphs derived directly from database schemas are often not well suited for how graph neural networks (GNNs) perform relational reasoning. We study what makes a relational graph suitable for deep learning and show that schema-derived graphs suffer from two systematic failures: information overload and semantic fragmentation. Our empirical analysis reveals that the desired graph is not the raw schema, but a result of controlled structural adaptation. Performance depends on balancing two operations: mitigating information overload via filtering, and repairing semantic fragmentation via injection. Specifically, filtering serves as a bias-variance knob with non-monotonic effects, while injection improves performance only when it explicitly restores the relational dependencies missing from the original schema. Based on these findings, we develop an end-to-end structural optimizer that applies both operations to adapt relational graphs automatically. Across 26 tasks spanning classification, regression, and recommendation, the optimized graphs consistently improve accuracy while often reducing inference cost.
Yao Cheng, Siqiang Luo
Nanyang Technological University, College of Computing and Data Science, Singapore.