Authors: Goran Nakerst, John Brennan, Wouter Beugeling, Masudul Haque
Organizations: Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany
Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.
Figures & tables
Image sample
Pixel values
Label
1
p1 (1)
p1 (2)
xxx … xxx …
p1 (784)
0
2
p2 (1)
p2 (2)
xxx … xxx …
p2 (784)
0
3
p3 (1)
p3 (2)
xxx … xxx …
p3 (784)
1
4
p4 (1)
p4 (2)
xxx … xxx …
p4 (784)
1
⋮
⋮
⋮
⋮
⋮
⋮
20
p20 (1)
p20 (2)
xxx … xxx …
p20 (784)
9
Figure 1: Framing MNIST classification as a tabular task. In this schematic, two examples per MNIST digit, thus a total of 20 images, are used as the context (the support set) from which the TFM learns. The last two rows, labeled q1 and q2 (q for ’query’), are images to be used for inference/prediction; hence the Label field is empty. The collection of unlabeled samples, the ‘test’ data set, is sometimes called the ‘query set’ or ’target set’ when the learning is in-context.
Figure 2: Test accuracy of TabPFN v3 on MNIST from 784 raw pixels, as a function of in-context/training examples per digit (blue), evaluated on a fixed Ntest=5,000 -image test set. Error bars indicate ± one standard deviation over resampling the in-context/training examples. Inset : test accuracy of a simple CNN (green) and LightGBM (orange), a gradient-boosted tree model, trained on the identical splits, test set, and number of training sampling repetitions.
Figure 3: Test accuracy on distinguishing French from German words from their first 15 letters, as a function of in-context/training words per language, evaluated on a fixed Ntest=4,000 -word test set. The main panel shows in-context TabPFN v3 (blue). The shaded band and error bars indicate ± one standard deviation over 10 in-context/training resamplings. Inset : the same TabPFN v3 curve against tuned LightGBM (green and orange), on identical word sets for train/context and test.
Figure 4: Test accuracy of TabPFN v3 on the 100 -class Tiny ImageNet subset from 12,288 raw pixels (blue), as a function of training examples per class, evaluated on a fixed Ntest=5,000 -image test set. The light-blue curve reduces the pixels to 200 PCA components before TabPFN v3, so that every ensemble estimator covers all features. Error bars indicate ± one standard deviation over resampling the training data.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: CNN test (solid) and train (dotted) accuracy versus training epochs for single runs on MNIST, for four training-set sizes 1,5,20,100 examples per digit (from bottom to top). The size of the test set is fixed at 5,000 images.
Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).
We introduce TabPFN-3.5, our new flagship Tabular Foundation Model. It significantly outperforms its predecessor, TabPFN-3, and all existing baselines across a broad range of tabular problems. TabPFN-3.5 sets a new state of the art on standard tabular prediction in TabArena, and extends it to the data practitioners encounter in practice: non-i.i.d. data with temporal or grouped splits, tables with strings, text and images, high-cardinality categorical features, and wide tables with many features. These gains carry over to our task-specific harnesses: state of the art on relational data and stronger time-series forecasting. For faster inference, our variant TabPFN-3.5-Fast runs up to 3x faster than TabPFN-3 while keeping most of the accuracy gains. In addition, we upgrade TabPFN-3.5-Plus, expanding our multimodal capabilities with advanced text and date handling alongside proprietary inference optimizations. Finally, we release a new version of our Thinking mode, TabPFN-3.5-Thinking, which scales inference-time computation to push the state of the art further. It benefits from our stronger base model and from inference-time improvements that make it up to 12x faster than TabPFN-3-Thinking.
Tabular Foundation Models, such as TabPFN, have received a large amount of recent attention due to their performance on in-context tabular machine learning tasks, which often exceeds classical baselines. However, practical deployment considerations of these models has received less attention. In this paper we investigate the memory requirements for these models. We demonstrate that employing model compression approaches can enable memory reductions of up to 7.6 with similar levels of performance, reducing deployment requirements by nearly 87%. Our work provides insight to practitioners seeking efficient deployment of these models in practical settings.
Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2
Commonwealth Bank of Australia, Australia · Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia