Simple row-wise predictors that score each row independently remain competitive with far more expressive architectures on tabular benchmarks, which are i.i.d. by construction. They cannot, however, represent natural targets on transactional, temporal, and relational tables, where labels depend on other rows. We show that row-wise prediction provably rules out targets driven by global counts, overlaps, and relational patterns. To make ``using structure'' precise across architectures, we introduce grables: a modular interface that separates how a table is lifted to a graph (constructor) from how predictions are computed on that graph (node predictor), placing gradient-boosted trees, neighbourhood aggregation, transformer-style tabular models, and relational GNNs on a single axis and pinpointing where expressive power comes from. The framework yields a hierarchy that answers when structure is needed: row-local models suffice for extension-invariant targets, lightweight neighbourhood aggregation captures shallow inter-row signals, and explicit graph construction is needed for compositional patterns that fixed aggregations cannot express without exponential feature blow-up. Experiments on synthetic, transactional, and relational data confirm the predicted separations, but also show that benchmark performance is a poor proxy for structural capability: row-local models score well on tasks they provably cannot represent, and a stress-test that suppresses row-local cues is needed to expose the gap. Conversely, access to cross-row context does not imply a model uses it. Feeding explicitly extracted inter-row structure to strong tabular learners yields gains exactly where the hierarchy predicts. The practical recipe is to add structure incrementally, and only where the target is extension-sensitive.
Figures & tables
Figure 1: Incidence-grable patterns for the four logical tasks. Row nodes (circles) connect to column–value nodes (squares) via typed edges; the target row r , whose label is predicted, is filled. We write d for the degree of a column–value node, i.e. the number of rows carrying that value. (a) Unique (r) : r is adjacent to a value node of degree 1 ; the second value node, of degree 2 , is not a witness. (b) Count N(r) : r is adjacent to a value node of degree greater than N , i.e. at least N other rows share that value. (c) Double (r) : a length-3 pattern r−ui,v−s−uj,w whose witness row s satisfies the local condition P . (d) Diamond (r) : a single row s shares two distinct column–value nodes with r . In all four cases the label of r depends on rows other than r .
Model
Unique
Count
Count ( = )
Double
Diamond
AUC
F1
AUC
F1
AUC
F1
AUC
F1
AUC
F1
Synthetic transactions
LightGBM
0.566
0.123
0.889
0.000
0.675
0.314
0.789
0.729
0.846
0.682
RealMLP
0.516
0.123
0.933
0.733
0.521
0.000
0.785
0.686
0.796
0.603
ModernNCA
0.597
0.000
0.862
0.569
0.544
0.000
0.720
0.670
0.798
0.593
TabPFN2.5
0.823
0.025
0.962
0.775
0.340
0.055
0.670
0.699
0.868
0.702
Table 1: Diagnostic comparison on logical tasks in synthetic and transaction datasets. F1 of 0 corresponds to predicting 0 for all instances. Results averaged over 5 runs, variance in Appendix G.1
Figure 2: F1 score on the stress-test set of the synthetic transactions data. The stress-test set removes the row-local cues that correlate with the labels in the original test split (Appendix G.2.2 ).
Figure 4
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
id
card_id
merchant_id
merchant_city
1
C000317
42
Brussels
2
C000102
17
ONLINE
3
C000317
105
Paris
4
C000891
42
Brussels
5
C000045
311
Berlin
6
C000102
17
ONLINE
Appendix
Table 3: Example rows from the synthetic transaction dataset.
Statistic
Train
Validation
Test
# transactions
8,000
1,000
1,000
# unique cards
2,500
350
350
# unique merchants
3,500
300
300
Online merchants
0.15
0.12
0.12
Appendix
Table 4: Summary statistics of the synthetic transaction datasets.
InvoiceNo
StockCode
Description
Qty
InvoiceDate
Price
CustomerID
Country
581220
16225
Rattle Snake Eggs
24
2011-12-08
0.39
13924
UK
581220
16258A
Circular Rubbers
36
2011-12-08
0.19
13924
UK
581220
21411
Heart Doorstop (Red)
8
2011-12-08
1.95
13924
UK
Appendix
Table 5: Illustrative example of transaction rows (synthetic but representative). Multiple rows correspond to distinct items within the same invoice.
Attribute
Value
Domain
Medical
Number of tables
15
Number of rows
5,852,157
Number of columns
140
Start date
2000-01-01
Validation timestamp
2020-01-01
Appendix
Table 6: Summary statistics of the relbench-trial dataset.
Figure 4: Relational schema of the relbench-trial relational dataset. The task studied is associated with table studies that contain descriptive information about the clinical studies. Taken from https://relbench.stanford.edu/datasets/rel-trial/
Dataset
Task
Train (%)
Val (%)
Test (%)
Synthetic Data
UNIQUE
13.1
6.6
7.5
Synthetic Data
COUNT
24.9
18.0
18.4
Synthetic Data
COUNT (=)
15.4
15.0
14.1
Synthetic Data
DOUBLE
48.2
42.8
43.4
Synthetic Data
DIAMOND
30.7
28.1
24.6
Retail Data
UNIQUE
0.9
14.2
10.5
Appendix
Table 7: Proportion of presence of label tasks across train, validation, and test splits for each dataset
Model
Aggregate AUC
Cases A and C AUC
Case B AUC
LightGBM
0.667
0.997
0.497
TabPFN
0.691
0.869
0.501
ModernNCA
0.662
0.961
0.495
Appendix
Table 8: Performance on COUNT >5 under single-point-with-reference protocol, decomposed by case. Aggregate metrics conflate memorization of value-label associations with extension-sensitive reasoning. Results averaged over 5 runs.
Task
Metric
Original
Finetuned
Abs. diff.
Rel. gain/loss
UNIQUE
roc_auc
0.8230
0.8744
+0.0514
+6.25%
UNIQUE
f1
0.0250
0.0240
-0.0010
-4.00%
COUNT
roc_auc
0.9620
0.9290
-0.0330
-3.43%
COUNT
f1
0.7750
0.8810
+0.1060
+13.68%
COUNT(=)
roc_auc
0.3400
0.3310
-0.0090
-2.65%
COUNT(=)
f1
0.0550
0.0137
-0.0413
-75.09%
Appendix
Table 9: Effect of finetuning for TabPFN on logical tasks in the synthetic data.
Hyperparameter
Uniqueness
Count ( > )
Count ( = )
Double condition
Diamond
Layers
2
2
3
3
4
Hidden dimension
64
64
128
64
64
Dropout
0.0
0.0
0.0
0.0
0.0
Learning rate
1e-3
1e-3
1e-3
1e-3
1e-3
Weight decay
1e-4
1e-4
1e-6
1e-4
1e-4
Aggregation
sum
sum
sum
sum
sum
Appendix
Table 10: Training hyperparameters for synthetic dataset across different tasks.
Model
Unique
Count
Count ( = )
AUC
F1
AUC
F1
AUC
F1
Synthetic transactions
LightGBM
0.566 ± .012
0.123 ± .007
0.889 ± .001
0.000 ± .000
0.675 ± .030
0.314 ± .009
RealMLP
0.516 ± .012
0.123 ± .007
0.933 ± .005
0.733 ± .002
0.521 ± .007
0.000 ± .000
ModernNCA
0.597 ± .009
0.000 ± .000
0.862 ± .017
0.569 ± .007
0.544 ± .012
0.000 ± .000
TabPFN2.5
0.823 ± .007
0.025 ± .003
0.962 ± .008
0.775 ± .004
0.340 ± .020
0.055 ± .004
Appendix
Table 11: Diagnostic comparison on logical tasks in synthetic and transaction datasets (Part 1: Unique , Count , Count ( = )). F1 of 0 corresponds to predicting 0 for all instances. Results averaged over 5 runs (mean ± std).
Model
Double
Diamond
AUC
F1
AUC
F1
Synthetic transactions
LightGBM
0.789 ± .007
0.729 ± .012
0.846 ± .021
0.682 ± .009
RealMLP
0.785 ± .012
0.686 ± .030
0.796 ± .021
0.603 ± .017
ModernNCA
0.720 ± .033
0.670 ± .017
0.798 ± .009
0.593 ± .005
TabPFN2.5
0.670 ± .025
0.699 ± .013
0.868 ± .030
0.702 ± .018
Appendix
Table 12: Diagnostic comparison on logical tasks in synthetic and transaction datasets (Part 2: Double , Diamond ). F1 of 0 corresponds to predicting 0 for all instances. Results averaged over 5 runs (mean ± std).
Figure 5: Box-plots for F1 score of models LightGBM, RealMLP, Sage and TabPFN in 5 different perturbations of the original test set.
Model
t1 : unique
t2 : count( > )
t3 : count( = )
Orig
Clean
Δ
Orig
Clean
Δ
Orig
Clean
Δ
F1-score
GNN (SAGE)
0.988
0.989
+0.001
0.967
0.997
+0.030
0.171
1.000
+0.829
LightGBM
0.190
0.190
+0.000
0.000
0.000
+0.000
0.027
0.027
+0.000
RealMLP
0.190
0.191
+0.001
0.881
0.885
+0.004
0.027
0.027
+0.000
TabPFN
0.404
0.406
+0.002
0.001
0.001
-0.000
0.023
0.019
-0.004
Appendix
Table 13: Performance on original vs. cleaned retail data (Unique, Count). Δ=Clean−Orig .
Model
t3 : double condition
t4 : diamond
Orig
Clean
Δ
Orig
Clean
Δ
F1-score
GNN (SAGE)
0.863
0.855
-0.008
0.251
0.249
-0.002
LightGBM
0.815
0.672
-0.143
0.199
0.087
-0.112
RealMLP
0.000
0.819
+0.819
0.139
0.169
+0.030
TabPFN
0.003
0.003
+0.000
0.007
0.007
+0.000
Appendix
Table 14: Performance on original vs. cleaned retail data (Double, Diamond). Δ=Clean−Orig .
Figure 6: Permutation feature importance based on ROC-AUC score for the AutoGluon tabular model. Bars show the mean change in ROC-AUC score under feature permutation, with 99% confidence intervals from repeated shuffles. Positive values indicate performance degradation, while negative values indicate improved performance after permutation.
Figure 7: Permutation feature importance based on ROC-AUC score for the AutoGluon tabular model with tabular time-aware neighbor feature aggregation features. Bars show the mean change in ROC-AUC score ( Δ ROC-AUC) under feature permutation, with 99% confidence intervals from repeated shuffles. Trimmed to most relevant 15 attributes.
Figure 8: Permutation feature importance based on ROC-AUC score for the AutoGluon tabular model augmented with embeddings from a GNN trained on the incidence grable from the table. Bars show the mean change in ROC-AUC score ( Δ ROC-AUC) under feature permutation, with 99% confidence intervals from repeated shuffles. Trimmed to most relevant 15 attributes. Important to note that out of the 15 most important attributes, 9 correspond to GNN embeddings.
Figure 9: Permutation feature importance based on ROC-AUC score for the AutoGluon tabular model augmented with embeddings from a GNN trained on the relational entity graph from the entire database. Bars show the mean change in ROC-AUC score ( Δ ROC-AUC) under feature permutation, with 99% confidence intervals from repeated shuffles. Trimmed to most relevant 15 attributes. Important to note that out of the 15 most important attributes, 7 correspond to GNN embeddings.
Dataset ID
# Cat. features
Tabular
Hybrid
Δ
363619
4
0.845 ± .02
0.848 ± .02
+0.36%
363623
4
0.928 ± .02
0.924 ± .01
−0.43%
363626
13
0.717 ± .01
0.752 ± .01
+4.88%
363671
3
0.784 ± .01
0.800 ± .01
+2.04%
363682
8
0.680 ± .02
0.698 ± .02
+2.65%
363684
8
0.890 ± .01
0.885 ± .01
−0.56%
Appendix
Table 15: ROC-AUC on 9 TabArena datasets selected for having a binary target and at least 3 categorical features with non-unique values. Results averaged over 5 runs (mean ± std).
Dataset
Task
Tabular
GNN
Hybrid
% vs. Tab
% vs. GNN
f1
driver-top3
0.739 ± .01
0.795 ± .01
0.824 ± .01
+11%
+3%
f1
driver-dnf
0.686 ± .02
0.649 ± .01
0.691 ± .01
+1%
+6%
arxiv
paper-citation
0.712 ± .04
0.807 ± .01
0.818 ± .01
+14%
+1%
stack
user-engagement
0.634 ± .03
0.835 ± .02
0.894 ± .02
+41%
+7%
stack
user-badge
0.634 ± .02
0.668 ± .02
0.811 ± .01
+27%
+21%
Appendix
Table 16: ROC-AUC on 5 additional RelBench binary classification tasks, re-implemented in a unified setup for fair comparison. Absolute numbers differ from published results due to text feature treatment; relative comparisons are unaffected. The hybrid matches or improves over both purely tabular and purely graph baselines. Results averaged over 5 runs (mean ± std).
Tabular prediction can benefit from in-table rows as few-shot evidence, yet existing tabular models typically perform instance-wise inference and LLM-based prompting is often brittle. Models do not consistently leverage relevant rows, and noisy context can degrade performance. To address this challenge, we propose TabSieve, a select-then-predict framework that makes evidence usage explicit and auditable. Given a table and a query row, TabSieve first selects a small set of informative rows as evidence and then predicts the missing target conditioned on the selected evidence. To enable this capability, we construct TabSieve-SFT-40K by synthesizing high-quality reasoning trajectories from 331 real tables using a strong teacher model with strict filtering. Furthermore, we introduce TAB-GRPO, a reinforcement learning recipe that jointly optimizes evidence selection and prediction correctness with separate rewards, and stabilizes mixed regression and classification training via dynamic task-advantage balancing. Experiments on a held-out benchmark of 75 classification and 52 regression tables show that TabSieve consistently improves performance across shot budgets, with average gains of 2.92% on classification and 4.45% on regression over the second-best baseline. Further analysis indicates that TabSieve concentrates more attention on the selected evidence, which improves robustness to noisy context.
Yongyao Wang, Ziqi Miao, Lu Yang +4
Gaoling School of Artificial Intelligence, Renmin University of China · Shanghai AI Laboratory
We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.
Graph learning presupposes a graph, and tables and relational databases do not come with one. Applying a GNN to them requires deciding which entities become nodes, which of them to connect, and through which relations---a decision made by hand, by schema heuristics, or by training a model on every candidate graph and keeping the best. We give a criterion that requires no trained graph model. In the minimal table-to-graph abstraction each row is a node, so a message-passing GNN, bounded by 1-WL, sees a construction only as a partition of the rows into colour-refinement classes: a construction is good for a task when that partition separates rows with different labels and does not split rows that share one. AutoGrable turns this criterion into a construction procedure. For incidence constructions the partition is fixed by the selected columns, so building a graph reduces to choosing them, and we score a candidate subset by a label-alignment risk: the held-out risk of the best predictor constant on its blocks, penalised by an occupancy term measuring how thinly the blocks are populated. The score materialises no graph and trains no GNN, so AutoGrable can search the space of subsets greedily and cheaply, and returns the resulting grable for single tables and for foreign-key schemas alike. Our experiments show that over a space of candidate graphs the score discards a large fraction while retaining the best; that AutoGrable recovers the columns that generate the label on controlled tasks and outperforms fixed, random, and task-aware constructors on real tasks under a fixed predictor; and that it is the only method compared that can decline to build a graph when none helps.