RAD-TFM: Robust and Domain-Adapted Tabular Foundation Models
Organizations: MIT · IBM
Abstract
The development of tabular foundation models (TFMs) has accelerated in recent years, outperforming traditional ML methods on many tabular benchmarks. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our key insight is that parameterizing the generator distribution enables an adversarial, distributional robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model, which we formalize by introducing an optimality gap measure. Further, we develop a method to align the synthetic data generation with real-world datasets from a given domain, constraining the adversary to generate realistic data. Together, these algorithms comprise the Robust and Domain-Adapted Tabular Foundation Models (RAD-TFM) pipeline, a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RAD-TFM improves performance across 6 diverse tabular benchmarks, with up to a 10.8 percentage-point improvement in mean normalized AUC over the strongest baseline, using fewer than 100k additional synthetic training datasets, less than 0.1% of the original pretraining data. Applying the same pipeline to TabICL V2 also improves performance across three major benchmarks, demonstrating the generality of the framework. These results highlight a promising new direction for targeted adversarial continued training of TFMs using synthetic data alone.
Figures & tables
| Model | Mean Norm. AUC | Rank-1 Wins | Mean Rank |
|---|---|---|---|
| TabICL V2 ( RAD-TFM ) | 0.868 | 21 | 4.23 |
| TabPFN V2 ( RAD-TFM ) | 0.860 | 11 | 4.73 |
| TabICL V2 (FT) | 0.850 | 15 | 4.37 |
| TabICL V2 | 0.837 | 18 | 4.67 |
| TabPFN V2 (FT) | 0.810 | 2 | 5.37 |
| TabPFN V2 | 0.806 | 1 | 5.45 |
| Benchmark | Base | DA-only | Robust-only | RAD-TFM |
|---|---|---|---|---|
| PMLB | 0.9088 | 0.9121 | 0.9109 | 0.9119 |
| TabPertNet | 0.8019 | 0.8093 | 0.8113 | 0.8185 |
| TabArena | 0.8725 | 0.8738 | 0.8743 | 0.8765 |
| Genomics | 0.9675 | 0.9699 | 0.9679 | 0.9713 |
| Healthcare | 0.8915 | 0.8937 | 0.8938 | 0.8951 |
| Engineering | 0.9402 | 0.9403 | 0.9386 | 0.9422 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Mean AUC | Change vs. base |
|---|---|---|
| TabPFN V2 | 0.6602 | — |
| TabPFN V2 (FT) | 0.6551 | |
| TabPFN V2 ( RAD-TFM ; original SCMs) | 0.6597 | |
| TabPFN V2 ( RAD-TFM ; temporal SCM) | 0.6652 |
| Benchmark | TabICL Base | TabICL FT | TabICL ( RAD-TFM ) |
|---|---|---|---|
| PMLB | 0.892 | 0.911 | 0.924 |
| TabPertNet | 0.748 | 0.754 | 0.779 |
| TabArena | 0.891 | 0.894 | 0.914 |
| Benchmark | Method | 500 | 1k | 1.5k | 2k | 6k | 10k | 30k |
|---|---|---|---|---|---|---|---|---|
| TabArena | RAD-TFM | +0.0005 | +0.0010 | +0.0016 | +0.0015 | +0.0026 | +0.0028 | |
| Fine-tuning | +0.0011 | +0.0015 | -0.0003 | -0.0024 | -0.0505 | -0.0840 | -0.0844 | |
| PMLB | RAD-TFM | +0.0019 | +0.0025 | +0.0026 | +0.0033 | +0.0029 | +0.0030 | |
| Fine-tuning | -0.0006 | -0.0012 | -0.0017 | -0.0023 | -0.0166 | -0.0199 | -0.0233 | |
| TabPertNet | RAD-TFM | +0.0031 | +0.0049 | +0.0052 | +0.0058 | +0.0083 | +0.0105 | |
| Fine-tuning | +0.0019 | +0.0009 | +0.0010 | +0.0001 | -0.0020 | -0.0097 | -0.0103 |
| Benchmark | Residual | ||||
|---|---|---|---|---|---|
| PMLB | +0.0033 | +0.0021 | +0.0053 | +0.0030 | -0.0023 |
| TabPertNet | +0.0074 | +0.0094 | +0.0167 | +0.0166 | -0.0001 |
| TabArena | +0.0013 | +0.0018 | +0.0031 | +0.0040 | +0.0010 |
| Genomics | +0.0024 | +0.0004 | +0.0028 | +0.0037 | +0.0009 |
| Healthcare | +0.0021 | +0.0023 | +0.0044 | +0.0035 | -0.0008 |
| Engineering | +0.0001 | -0.0016 | -0.0015 | +0.0020 | +0.0035 |
Tabular Foundation Models
Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.
See all 209 papers