Predictive modeling over relational databases (RDBs) powers applications, yet remains challenging due to capturing both cross-table dependencies and complex feature interactions. Relational Deep Learning (RDL) methods automate feature engineering via message passing, while classical approaches like Deep Feature Synthesis (DFS) rely on predefined non-parametric aggregators. Despite performance gains, the comparative advantages of RDL over DFS and the design principles for selecting effective architectures remain poorly understood. We present a comprehensive study that unifies RDL and DFS in a shared design space and conducts architecture-centric searches across diverse RDB tasks. Our analysis yields three key findings: (1) RDL does not consistently outperform DFS, with performance being highly task-dependent; (2) no single architecture dominates across tasks, underscoring the need for task-aware model selection; and (3) validation accuracy is an unreliable guide for architecture choice. This search yields a model performance bank that links architecture configurations to their performance; leveraging this bank, we analyze the drivers of the RDL-DFS performance gap and introduce two task signals -- RDB task homophily and an affinity embedding that captures size, path, feature, and temporal structure -- whose correlation with the gap enables principled routing. Guided by these signals, we propose Relatron, a task embedding-based meta-selector that chooses between RDL and DFS and prunes the within-family search. Lightweight loss-landscape metrics further guard against brittle checkpoints by preferring flatter optima. In experiments, Relatron resolves the "more tuning, worse performance" effect and, in joint hyperparameter-architecture optimization, achieves up to 18.5% improvement over strong baselines with 10x lower cost than Fisher information-based alternatives.
Figures & tables
Figure 1: (a) An example of generating the task table from an RDB. The label is based on whether a student has achieved an A+ in a course before a specific timestamp. (b) Another example demonstrating the working process of DFS and RDL. For DFS, a predefined set of aggregation functions, such as MEAN and COUNT , is used to aggregate information across multiple tables based on key relationships into a final data table. For comparison, RDL is claimed to replace the manual aggregation design with an automatic message-passing-based sparse attention.
Name
Architecture design space
Hyper-parameters
RDL
Structural feature
Message passing
Readout
Learning rate, dropout, normalization, fanout…
Labeling ID, Learnable embedding, None
PNA, HGT, Sage, RelGNN
MLP, ContextGNN, Shallow
DFS
Aggregation function
Aggregation layers
Backbone
Batch size, hidden dimension …
Max, Min, Mode, … (fixed)
1, 2, 3 (if possible)
TabPFN, LightGBM, FT-Transformer
Table 1: Search space of RDL and DFS-based methods for RDB tasks. Underline means these components will go over a grid search, while other components will be sampled.
Figure 2: Performance comparison between the best configurations from our design space and baseline models on entity-level tasks. “Best (ours)” means the better value of RDL and DFS. Full numerical results can be seen in Table 11 from Appendix E.5 .
Task
Model
reported perf
test-selected perf
val-selected perf
driver-top3 (auroc)
GraphSAGE
75.54
82.81
81.56
RelGNN
85.69
85.69
82.61
driver-position (mae)
GraphSAGE
4.022
3.91
3.93
RelGNN
3.798
3.80
4.35
user-ignore (auroc)
GraphSAGE
81.62
86.40
72.27
RelGNN
86.18
86.18
78.94
Table 2: The performance gap between validation- and test-selected configurations.
Figure 3: Augmenting rel-f1 databases. We should treat the set of FKs as a hyperedge (each pair of FKs is appended to the original PK-FK graphs as a new edge type) rather than relying solely on PK-FK edges.
Selection
Model type
Val_auroc
Test_auroc
Pbar
P1
P2
Val
RDL
89.48
82.41
2.77
1.49
4.23
Test
RDL
86.05
85.94
0.41
1.22
0.22
Val
DFS
83.44
84.69
0.384
0.041
1.32
Test
DFS
83.76
85.71
0.495
0.03
0.50
Table 3: Example landscape properties and model performance. Smaller values typically indicate a more benign landscape.
Figure 7
Bank size
Model-free (ours)
Training-free model-based
Anchor-based
8
68.9%
56.9%
62.9%
12
64.8%
50.4%
56.1%
16
76.2%
53.1%
62.5%
All
87.5%
66.7%
66.7%
Table 5: Ablation on performance-bank size for macro-level predictors (LOO accuracy).
Strategy
Budget
driver-top3 (ROC-AUC) ↗
driver-position (MAE) ↘
user-churn (ROC-AUC) ↗
Random
3
82.67 ± 2.19
3.6810 ± 0.4255
68.56 ± 1.23
10
77.80 ± 4.79
3.8576 ± 0.5035
68.60 ± 1.20
30
77.28 ± 2.42
4.2793 ± 0.1483
69.54 ± 0.43
TPE
3
82.67 ± 2.19
3.6810 ± 0.4255
68.56 ± 1.23
10
81.45 ± 0.44
3.7897 ± 0.4271
68.60 ± 1.20
30
77.92 ± 5.12
4.1724 ± 0.0519
69.54 ± 0.43
Table 6: Joint architecture and hyperparameter optimization result. Best results are highlighted with an underline and the second are bold . “Only predictor” means only using the meta-predictor. Relbench’s default refers to the default architecture and hyperparameters in their pipelines. Best fixed refers to the configurations with the best mean rank of performance across tasks.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Database Name
Task Name
Task Type
Metric
rel-f1
driver-dnf
classification
ROC-AUC
driver-top3
classification
ROC-AUC
driver-position
regression
MAE
driver-wins
regression
MAE
constructor-scores-points
classification
ROC-AUC
driver-position-change
regression
MAE
Appendix
Table 7: Summary of databases, tasks, task types, and evaluation metrics used in our experiments.
Module name
Possible choices
Feature encoding
ResNet ( Robinson et al., 2024 )
Structural features (grid)
Learnable embeddings (for the destination table, or for both the source and destination tables) ( Yuan et al., 2024 ; Ma et al., 2024 ) ; partial-labeling tricks from NBFNet ( Zhu et al., 2021 )
Message passing (grid)
Sparse message passing: GraphSAGE ( Robinson et al., 2024 ; Hamilton et al., 2017 ) , HGT ( Wang et al., 2024a ; Hu et al., 2020b ) , PNA ( Corso et al., 2020 ; Wang et al., 2024a ) ; sparse message passing with higher-order information: RelGNN ( Chen et al., 2025a )
Architecture design trick (random)
Residual connections ( He et al., 2016 )
Readout (grid)
MLP, ContextGNN, Shallow-Item ( Yuan et al., 2024 )
Hyperparameters (random)
Learning rate, weight decay, batch size, dropout rate, number of layers, hidden dimension, temporal sampling strategy, number of sampled neighbors
Appendix
Table 8: Design space of a unified architecture for end-to-end RDL models. “Grid” indicates that the module is selected from a predefined grid, whereas “random” indicates that the module is randomly sampled.
TabPFN ( Hollmann et al., 2023 ) , FT-Transformer, LightGBM
Hyper-parameters (random)
Learning rate, weight decay, scheduler gamma, hidden dimension, number of attention heads, normalization
Training objective
Classification tasks: cross-entropy loss; Regression tasks: MSE or MAE
Appendix
Table 9: Design space of graph-induced non-parametric feature synthesis (DFS) models. Grid means the module is selected from a predefined grid, while random means the module is randomly sampled.
Method name
Mean test clf perf
Mean val clf perf
Mean gap
RDL Overall
78.21
76.73
1.49
Sage (top1)
76.90
72.77
4.13
HGT (top2)
77.19
74.95
2.24
DFS overall
76.90
75.22
1.68
TabPFN
75.91
75.70
0.21
FT-Transformer
75.87
74.45
1.42
Appendix
Table 10: Validation-selected and test-selected performance gap. For RDL, we show the top 2 architectures with the largest gap.
Figure 5: HPO results for recommendation tasks.
Task
Type
RelGNN
RelGT
GraphSAGE
Rel-LLM
KumoRFM (icl)
KumoRFM (fine-tuned)
RDL (val-selected)
RDL (ours)
DFS (val-selected)
DFS (ours)
Best (ours)
Griffin
driver-top3
classification
85.69
83.52
75.54
82.22
91.07
99.62
82.41
85.94
84.70
85.71
85.94
77.95
driver-dnf
classification
75.29
75.87
72.62
77.15
82.41
82.63
74.35
77.20
76.89
79.42
79.42
70.91
driver-position
regression
3.798
3.920
4.022
3.967
2.747
2.731
4.0491
3.8029
3.3660
3.2730
3.2730
4.2000
user-churn
classification
70.93
69.27
69.88
70.55
67.71
71.23
70.98
70.98
65.29
68.23
70.98
68.04
item-sales
regression
0.0540
0.0536
0.0560
0.0520
0.0400
0.0340
0.0511
0.0509
0.0780
0.0750
0.0509
0.0810
post-votes
regression
0.0650
0.0654
0.0650
0.0620
0.0650
0.0650
0.0665
0.0651
0.0680
0.0660
0.0651
0.0622
Appendix
Table 11: Full experimental results for Figure 2 .
Figure 6: Relationship between test performance ranking and micro-level architecture choices. For RDL-based methods, we filter the original search result with the top 10 performing configurations on each task. For DFS-based methods, we filter the original search result with the top 1 performing configurations on each task. The best configurations are selected based on test performance directly. “zero_learn” is the labeling trick adopted by NBFNet. (a) A violin plot of MPNN types shows that PNA achieves the best mean ranking across different tasks. (b) When comparing labeling tricks, although equivariant models appear more frequently in top rankings, a labeling trick is still needed to achieve the top spot. (c) For DFS, ft_transformer and tabpfn present unique strengths, where the former can leverage more training samples, and the latter usually works better in small-scale settings and can conduct in-context learning.
Figure 7: Loss landscapes
driver-dnf
study-outcome
RDL (SAGE)
RDL (HGT)
DFS-1 (no p/p)
DFS-2 (no p/p)
DFS-3 (no p/p)
RDL (SAGE)
RDL (HGT)
DFS-1 (no p/p)
DFS-2 (no p/p)
DFS-3 (no p/p)
Preprocessing time
60 s
60 s
7 s/12 s
18 s/17 s
310 s/41 s
240 s
240 s
240 s/36 s
353 s/48 s
965 s/95 s
Training time (per epoch)
6 s
8 s
<1 s/ <1 s
<1 s/ <1 s
<1 s/ <1 s
10 s
11 s
<1 s/ <1 s
<1 s/ <1 s
<1 s/ <1 s
Appendix
Table 12: Preprocessing and training times by method for driver-dnf and study-outcome.