The development of tabular foundation models (TFMs) has accelerated in recent years, outperforming traditional ML methods on many tabular benchmarks. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our key insight is that parameterizing the generator distribution enables an adversarial, distributional robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model, which we formalize by introducing an optimality gap measure. Further, we develop a method to align the synthetic data generation with real-world datasets from a given domain, constraining the adversary to generate realistic data. Together, these algorithms comprise the Robust and Domain-Adapted Tabular Foundation Models (RAD-TFM) pipeline, a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RAD-TFM improves performance across 6 diverse tabular benchmarks, with up to a 10.8 percentage-point improvement in mean normalized AUC over the strongest baseline, using fewer than 100k additional synthetic training datasets, less than 0.1% of the original pretraining data. Applying the same pipeline to TabICL V2 also improves performance across three major benchmarks, demonstrating the generality of the framework. These results highlight a promising new direction for targeted adversarial continued training of TFMs using synthetic data alone.
Figures & tables
Figure 1: Overview of RAD-TFM . We solve the core distributionally robust optimization problem by alternating between two stages: (a) searching over the SCM parameter space, we find synthetic data distributions with large estimated optimality gaps, and (b) sampling parameters proportional to their optimality gap, we generate synthetic datasets and continue training the TFM.
Figure 2: We perform domain adaptation for TFMs (e.g. healthcare, finance) by learning a mixture distribution over SCM generators that covers the distribution of real-world datasets from the given domain. We use a collection of meta-features capturing dataset schema, geometry, and learnability to encode and measure distance between datasets.
Figure 3: An example of our domain-adaptation methodology applied to the engineering datasets from OpenML Bischl et al. (2021) . Each color corresponds to a different parameterization of the synthetic data generator distribution. The black marks represent the embeddings of the real datasets. We plot the first two principal components of the meta-features extracted for the real datasets.
Model
Mean Norm. AUC
Rank-1 Wins
Mean Rank
TabICL V2 ( RAD-TFM )
0.868
21
4.23
TabPFN V2 ( RAD-TFM )
0.860
11
4.73
TabICL V2 (FT)
0.850
15
4.37
TabICL V2
0.837
18
4.67
TabPFN V2 (FT)
0.810
2
5.37
TabPFN V2
0.806
1
5.45
Table 4: Generalization of RAD-TFM to the TabICL V2 backbone across TabPertNet, TabArena, and PMLB. Normalized AUC and ranks are computed over the expanded comparison of 11 models (including five classical baselines); selected models are shown. Scores are pooled across 177 datasets with defined normalized AUC and use a common normalization across all evaluated models.
Benchmark
Base
DA-only
Robust-only
RAD-TFM
PMLB
0.9088
0.9121
0.9109
0.9119
TabPertNet
0.8019
0.8093
0.8113
0.8185
TabArena
0.8725
0.8738
0.8743
0.8765
Genomics
0.9675
0.9699
0.9679
0.9713
Healthcare
0.8915
0.8937
0.8938
0.8951
Engineering
0.9402
0.9403
0.9386
0.9422
Table 5: Ablation of domain adaptation (DA) and robust SCM parameter selection, reporting mean test AUC for TabPFN V2 across six benchmarks. The full method combines both components.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Mean AUC
Change vs. base
TabPFN V2
0.6602
—
TabPFN V2 (FT)
0.6551
−0.0051
TabPFN V2 ( RAD-TFM ; original SCMs)
0.6597
−0.0005
TabPFN V2 ( RAD-TFM ; temporal SCM)
0.6652
+0.0050
Appendix
Table 8: Mean test AUC across 64 foreign-exchange datasets, comparing the original and expanded synthetic generator families.
Benchmark
TabICL Base
TabICL FT
TabICL ( RAD-TFM )
PMLB
0.892
0.911
0.924
TabPertNet
0.748
0.754
0.779
TabArena
0.891
0.894
0.914
Appendix
Table 17: Mean normalized AUC for TabICL V2 variants on the three principal benchmarks.
Benchmark
Method
500
1k
1.5k
2k
6k
10k
30k
TabArena
RAD-TFM
+0.0005
+0.0010
+0.0016
+0.0015
+0.0026
+0.0028
+0.0040∗
Fine-tuning
+0.0011
+0.0015
-0.0003
-0.0024
-0.0505
-0.0840
-0.0844
PMLB
RAD-TFM
+0.0019
+0.0025
+0.0026
+0.0033
+0.0029
+0.0030
+0.0030∗
Fine-tuning
-0.0006
-0.0012
-0.0017
-0.0023
-0.0166
-0.0199
-0.0233
TabPertNet
RAD-TFM
+0.0031
+0.0049
+0.0052
+0.0058
+0.0083
+0.0105
+0.0166∗
Fine-tuning
+0.0019
+0.0009
+0.0010
+0.0001
-0.0020
-0.0097
-0.0103
Appendix
Table 18: Mean raw test-AUC change relative to frozen TabPFN V2 at different training-dataset budgets. Positive values indicate improvement. At 30k, RAD-TFM entries marked with an asterisk correspond to selected checkpoints from runs that processed 30k generated datasets, whereas fine-tuning entries are evaluated at the 30k checkpoint.
Benchmark
ΔDA
ΔR
ΔDA+ΔR
ΔRAD
Residual
PMLB
+0.0033
+0.0021
+0.0053
+0.0030
-0.0023
TabPertNet
+0.0074
+0.0094
+0.0167
+0.0166
-0.0001
TabArena
+0.0013
+0.0018
+0.0031
+0.0040
+0.0010
Genomics
+0.0024
+0.0004
+0.0028
+0.0037
+0.0009
Healthcare
+0.0021
+0.0023
+0.0044
+0.0035
-0.0008
Engineering
+0.0001
-0.0016
-0.0015
+0.0020
+0.0035
Appendix
Table 19: Change in mean raw AUC from the frozen TabPFN V2 and the residual from additive contributions.
What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality.
Despite strong mean accuracy, tabular foundation models (TFMs) can perform poorly on underrepresented groups under subpopulation shift, where group proportions change between training and deployment. We propose DR-TFM, a parameter-efficient distributionally robust adaptation framework that requires no true group annotations. DR-TFM adjusts attention to labeled context examples by fine-tuning an existing query scaling network or adding and training one, while keeping all other parameters fixed. We instantiate the framework with two robust objectives using estimated groups or source conditional distributions derived from training data. For TabPFN-3, adaptation updates only 0.016% of the pretrained model's parameters. Across five tabular benchmarks, DR-TFM achieves substantially higher average worst-group accuracy than pretrained TFMs and the compared robust baselines without true group annotations, while maintaining competitive mean group accuracy. DR-TFM also improves average worst-group accuracy on ACS Income and across four additional TFMs.
Tabular foundation models (TFMs) are commonly pretrained on large collections of procedurally generated synthetic tasks, yet it remains unclear how well these synthetic pretraining priors support the downstream tasks on which the models are evaluated. We study this question from a distribution-level attribution perspective. We recover or reconstruct the synthetic data generators of four TFMs and compare their generated tasks with datasets from two widely used tabular benchmarks. Each dataset is represented by a common set of structural descriptors capturing schema, feature distributions, dependence structure, response properties, and feature--response relationships. In this space, we measure how broadly and repeatedly each synthetic prior reaches benchmark tasks using structural coverage and normalized density, and examine whether stronger local support is associated with better predictive performance. We find substantial differences across synthetic pretraining priors: some generators provide consistently broader and denser support for benchmark tasks than others. Moreover, stronger synthetic-to-benchmark support is generally associated with better relative model performance. These results suggest that structural coverage provides a useful diagnostic for characterizing synthetic pretraining priors and relating their data-generating assumptions to downstream model behavior.