Representations of supervised datasets support higher-level machine-learning tasks such as dataset retrieval and distribution-shift monitoring. However, existing approaches typically rely either on hand-crafted meta-features or on dataset encoders trained specifically for representation learning. We introduce TabMF, a framework that repurposes pretrained tabular foundation models (TFMs) as encoders of supervised datasets. TabMF equips a predictive TFM with an architecture-aware dataset-level readout and adapts the model end-to-end using contrastive post-training on stochastic views of synthetic supervised datasets. We instantiate TabMF on TabICLv2 and TabPFN-2.5 and evaluate the resulting representations across six tasks on unseen real datasets. Simple frozen TFM readouts yield competitive representations, indicating that predictive pretraining already induces substantial dataset-level structure. The two TabMF encoders obtain the strongest aggregate performance among the evaluated learned and classical dataset representations, with particularly strong results in dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.
Figures & tables
Figure 1: TabMF overview. Left: A synthetic supervised dataset is sampled from one Graph-SCM, and two stochastic views vary the sampled observations, observed variables, and label-preserving feature transformations while sharing the same designated target. TabMF encodes both views and is trained contrastively to bring their representations together while separating representations of other datasets. Right: At inference time, a frozen TabMF encoder maps a new supervised dataset (X,y) to a fixed-dimensional representation that can be used directly for dataset comparison and monitoring or as input to lightweight downstream probes.
TabBench
TabRepo
TableShift
Encoder
Views
Domain
Epiplexity
Recall@5
Selection
Severity
Rank ↓
TabMF -PFN
0.9985
0.683
0.597
0.965
0.929
0.917
1.5
TabMF -ICL
0.9981
0.702
0.639
0.938
0.854
0.933
2.0
TabICL ICL-mean
0.9775
0.670
0.742
0.889
0.575
0.850
3.5
PyMFE
0.9812
0.661
0.106
0.931
0.709
0.733
4.0
TabPFN mean-pooling
0.9414
0.650
0.741
0.821
0.850
0.583
5.0
Table 1: Primary results across six evaluation tasks. Scores are real-data bootstrap AUC, domain macro-OVR AUC, epiplexity regression Spearman, mean retrieval recall@5, TabRepo three-dataset K-means ranking Spearman, and TableShift shift-severity Spearman. Encoders are ordered by their median rank across the six tasks. Higher scores and lower ranks are better; bold marks numerical leaders. Epiplexity uses available-case scores and leakage-controlled PyMFE features; paired common-population results are in Appendix B.4.3 .
Figure 2: Dataset identity remains recoverable with incomplete features, but robustness depends on the encoder and transformation. ( 2(a) ) Both TabMF variants outperform ICL-mean and PyMFE when only 25% of columns remain. ( 2(b) ) TabMF -PFN better tolerates monotone transformations and standardization, while TabMF -ICL better tolerates heavy noise and missingness; quantization limits both variants. Values are retrieval recall@5; the complete transformation results appear in Figure 3 .
Backbone
Readout
Initialization
Objective
Recall@5
TabICL
Summary-token
Pretrained
NT-Xent
0.938
TabICL
Summary-token
Scratch
NT-Xent
0.797
TabICL
Summary-token
Pretrained
Triplet
0.924
TabICL
Summary-token
Pretrained
Exp. distance
0.883
TabPFN
Column-attention
Pretrained
NT-Xent
0.965
TabPFN
Column-attention
Scratch
NT-Xent
0.587
Table 2: Encoder design comparisons using mean retrieval recall@5. Each block compares configurations within one backbone across initialization, readout, and objective choices.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Readout
Loss
Initialization
AUC
TabICL
Summary-token
NT-Xent
Pretrained
0.9999
TabPFN
Column-aware
NT-Xent
Pretrained
0.9996
TabICL
Row-mean
NT-Xent
Pretrained
0.9996
TabPFN
Pseudo-test
NT-Xent
Pretrained
0.9995
TabICL
Row-mean
Triplet
Pretrained
0.9994
TabPFN
Pseudo-test
Triplet
Pretrained
0.9994
Appendix
Table 3: Synthetic bootstrap-view discrimination AUC across encoder configurations. Row-mean denotes a trained encoder, distinct from the frozen row-mean baseline.
Population
Datasets
Significant improvements
Largest Holm p
TabBench
191
18/18
<1.6×10−16
Synthetic
50
18/18
<5.0×10−14
Appendix
Table 4: Paired comparisons of per-dataset hard margins in bootstrap-view discrimination. Both adapted encoders improve over each of nine baselines in each population. Holm correction is applied across all 18 comparisons within each population. The final column gives an upper bound on the largest adjusted p -value. These tests assess margins, not AUC. Synthetic results characterize the population used for model selection.
Transformation
Definition
Keep 20% / 50% of rows
Sample kr observations without replacement, r∈{0.2,0.5} . Skip when kr≥n .
Bootstrap rows
Sample n observations with replacement.
Duplicate rows
Retain all observations and append max{1,round(0.3n)} observations sampled with replacement.
Keep 25% / 50% / 75% of features
Sample hr features without replacement, r∈{0.25,0.5,0.75} . Skip when hr≥d .
Duplicate features
Append copies of h0.3 features selected without replacement.
Add noisy features
Append h0.3 numerical features with independent N(0,1) entries.
Appendix
Table 5: Dataset transformations used for retrieval evaluation. Here, n and d denote the original numbers of observations and features, and μj and σj denote the mean and standard deviation of numerical feature j . We write kr=max{min(n,20),round(rn)} and hr=max{1,round(rd)} . Grouped entries denote separate transformations for each listed parameter value.
Baseline
TabMF -ICL
TabMF -PFN
TabICL ICL-mean
1.0×10−7
5.8×10−15
PyMFE
0.522
1.6×10−5
Appendix
Table 6: Dataset-level paired comparisons of transformation retrieval. For each dataset, recall@5 is averaged across applicable transformations. Entries are Holm-adjusted p -values from two-sided Wilcoxon signed-rank tests, correcting across all four comparisons. Tests pair 190 datasets, except TabMF -PFN versus PyMFE, which pairs 191. Bold denotes a significant improvement of the adapted encoder. These tests assess dataset-level averages; the headline retrieval score instead averages recall@5 equally across transformations.
Encoder
K-means
Farthest-first
TabMF -PFN
0.9292
0.8229
TabMF -ICL
0.8542
0.8146
TabPFN mean-pooling
0.8500
0.6730
Random
0.7766
0.7766
Neural Statistician
0.738
0.782
PyMFE
0.7093
0.7812
Appendix
Table 7: Compact benchmark selection on TabRepo at k=3 . Entries are mean classifier-ranking Spearman correlations over 200 shared candidate-pool trials. Encoders are ordered by K-means performance. Bold marks numerical leaders.
Variant
Baseline
Δρ
Holm p
TabMF -ICL
Random
+0.0777
5.28×10−7
TabMF -ICL
ICL-mean
+0.2792
6.03×10−28
TabMF -ICL
PyMFE
+0.1450
2.67×10−22
TabMF -ICL
Dataset2Vec
+0.2028
1.74×10−25
TabMF -ICL
Neural Statistician
+0.1162
2.00×10−19
TabMF -ICL
Frozen TabPFN
+0.0045
0.2408
Appendix
Table 8: K-means benchmark-selection comparisons on TabRepo at a three-dataset budget. Differences are TabMF minus baseline in mean Spearman correlation. Two-sided Wilcoxon signed-rank tests pair the 200 candidate-pool trials. Reported p -values are Holm-adjusted jointly over the 12 comparisons.
Variant
Baseline
Original
Alternative
TabMF -ICL
ICL-mean
+0.2792
+0.2602
TabMF -ICL
PyMFE
+0.1450
+0.1280
TabMF -ICL
Dataset2Vec
+0.2028
+0.1946
TabMF -PFN
ICL-mean
+0.3542
+0.3478
TabMF -PFN
PyMFE
+0.2199
+0.2155
TabMF -PFN
Dataset2Vec
+0.2777
+0.2822
Appendix
Table 9: Sensitivity of three-dataset selection on TabRepo to K-means initialization. Entries are paired mean-correlation differences ( TabMF minus baseline). Candidate pools and embeddings are fixed. The alternative initialization preserves the direction of all six comparisons.
k
Selector
TabMF -ICL
TabMF -PFN
ICL-mean
PyMFE
Random
D2V
Frozen PFN
Neural Stat.
2
K-means
0.8568
0.6728
0.3507
0.6778
0.7209
0.6947
0.8214
0.7828
2
Farthest-first
0.7003
0.8752
0.5941
0.6979
0.7209
0.7100
0.6439
0.6418
3
K-means
0.8542
0.9292
0.5750
0.7093
0.7766
0.6515
0.8497
0.7380
3
Farthest-first
0.8146
0.8229
0.7108
0.7812
0.7766
0.7261
0.6728
0.7819
4
K-means
0.8504
0.9393
0.6743
0.8894
0.8171
0.7282
0.8536
0.7260
4
Farthest-first
0.8127
0.8697
0.8148
0.8338
0.8171
0.7805
0.7447
0.8560
Appendix
Table 10: TabRepo benchmark selection with two to four selected datasets. Entries are mean Spearman correlations with the full-population classifier ranking. Bold marks the best score within each fixed budget and selector. Various budgets assess sensitivity to the selection budget.
Encoder
AUC ↑
Log-loss ↓
N
TabMF -ICL
0.6997
1.7860
190
TabMF -PFN
0.6808
1.8100
191
PyMFE
0.6662
1.8871
191
TabICL row-mean
0.6637
1.8850
190
TabPFN mean-pooling
0.6620
1.8535
191
TabICL ICL-mean
0.6441
1.8585
190
Appendix
Table 11: Domain discrimination with the strengthened nonlinear probe. A tuned SVM jointly searches RBF and polynomial kernels. AUC and log-loss are computed on each encoder’s maximal available population. Pairwise inference uses per-dataset log-loss on the population shared by each pair, with Holm correction over all 55 encoder comparisons. Bold denotes the best point estimate.
Encoder
Spearman ρ
TabICL ICL-mean
0.764
TabPFN mean-pooling
0.760
TabMF -ICL
0.639
TabMF -PFN
0.621
Dataset2Vec
0.529
Appendix
Table 12: Epiplexity prediction on a common population of 190 datasets. A: Mean Spearman correlation across five cross-validation repetitions. B: Frozen-versus-adapted comparisons from a paired dataset bootstrap, with Holm adjustment over all ten pairwise comparisons among the five encoders. Bold p -values support higher correlation for the frozen readout.
Encoder
Mean ∣error∣↓
Spearman ↑
N
TabPFN mean-pooling
0.1770
0.8292 ± 0.026
191
TabMF -PFN
0.1794
0.7633 ± 0.019
191
TabICL ICL-mean
0.2032
0.7991 ± 0.021
190
TabMF -ICL
0.2224
0.6953 ± 0.012
190
Dataset2Vec
0.2265
0.6182 ± 0.026
191
Neural Statistician
0.3297
0.2658 ± 0.027
191
Appendix
Table 13: Epiplexity prediction with the strengthened nonlinear probe. Kernel ridge regression jointly searches RBF, Laplacian, and polynomial kernels. Mean absolute error is the endpoint used for ranking and paired inference; Spearman correlation is included for comparison with the primary linear probe. Pairwise tests use the population shared by each pair, with Holm correction over all 55 comparisons. Bold denotes the best point estimate.
Excluded base-feature identifiers
nr_class
freq_class
class_ent
class_conc
mut_inf
joint_ent
eq_num_attr
ns_ratio
can_cor
gravity
lh_trace
p_trace
roy_root
w_lambda
sd_ratio
Appendix
Table 14: Explicit exclusions in the leakage-controlled PyMFE difficulty configuration. Landmarking and model-based groups are omitted entirely. The remaining general, statistical, and information-theoretic groups are fitted without the dataset target. All summaries of each listed base feature are excluded together.
Encoder
TR k=2
TR k=3
TR k=4
Agg. k=2
Agg. k=3
Agg. k=4
TabMF -PFN
6
1
1
2.0
1.5
1.5
TabMF -ICL
1
2
4
1.5
2.0
2.0
TabICL ICL-mean
7
7
7
3.5
3.5
3.5
PyMFE
5
5
2
4.0
4.0
3.5
Frozen TabPFN
2
3
3
5.0
5.0
5.0
Dataset2Vec
4
6
5
5.0
5.5
5.0
Appendix
Table 15: Sensitivity of the six-task aggregate ranking to the TabRepo benchmark-selection budget. Only the TabRepo K-means evaluation changes across columns; the other five evaluation units are held fixed. The aggregate score is the median rank across the six evaluation units.
Task
Dataset ID
Gap
ID rows
OOD rows
ASSISTments
assistments
0.3519
8192
1906
FICO HELOC
heloc
0.2826
278
6914
College Scorecard
college_scorecard
0.1132
8192
1352
Hypertension
brfss_blood_pressure
0.0816
8192
8192
Hospital Readmission
diabetes_readmission
0.0489
4287
8192
Diabetes
brfss_diabetes
0.0450
8192
8192
Appendix
Table 16: TableShift tasks used for the shift-severity evaluation. The accuracy gap is the mean ID-minus-OOD accuracy difference across estimators with both published ID and OOD results. Row counts are the actual ID and OOD samples embedded after the at-most-8,192-row sampling described in Section 4 .
Figure 4: Encoder time and memory scaling as the number of rows and features increases. For the row-scaling experiment, the number of features is fixed at d=64 ; for the feature-scaling experiment, the number of observations is fixed at n=10,000 . TabMF -ICL closely tracks TabICL ICL-mean, while TabMF -PFN closely tracks TabPFN mean-pooling, indicating that the dataset-level readouts introduce little additional inference cost relative to their respective backbones. HyTrel exceeds available memory at larger table sizes. Neural Statistician and Dataset2Vec use bounded input sampling, which limits their scaling with table size.
Encoder
MAP ↑
Shuffling ↑
Type F1 ↑
TabMF -ICL
0.605±0.158
0.732±0.281
0.363
TabMF -PFN
0.448±0.117
0.773±0.251
0.289
TabICL ICL-mean
0.468±0.137
0.753±0.266
—
HyTrel
0.684±0.146
0.977±0.053
0.681
PyMFE
0.456±0.137†
0.705±0.150
0.372
Encoder
HPO quality ↑
HPO regret@5 ↓
Targets
Appendix
Table 17: Supplementary transfer evaluations. TEmBed MAP covers 16 datalakes except PyMFE (†: 17); shuffling covers 60 slices. Standard deviations are across datalakes or slices. Similarity and shuffling use uncapped inputs. HPO covers 82 targets except HyTrel (‡: 77). A dash denotes an unavailable result.
Figure 5: The two TabMF encoders. TabMF -ICL (left) appends a learned summary token to the target-conditioned row states and reads its state after the pretrained ICL Transformer and final normalization. TabMF -PFN (right) pools the backbone’s feature-group states over labeled observations, models their interactions with a learned column Transformer, and averages the active groups. Each readout feeds a projection network; gray blocks are pretrained and fine-tuned, while green blocks are added and trained from scratch.
Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.
Goran Nakerst, John Brennan, Wouter Beugeling +1
Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany
Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).
Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.