Understanding the Surprising Generalization Properties of Tabular Foundation Models

Nour Shaheen; Junwei Ma; Alex Labach; Frank Hutter; Valentin Thomas; Anthony L. Caterini

cs.LGAug 18, 2026

Understanding the Surprising Generalization Properties of Tabular Foundation Models

Authors: Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini

Abstract

Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.

Figures & tables

Figure 1: Transfer from a single pre - training dataset. Left: Training only on vectorized MNIST (treated as a table) and evaluating on California Housing . Middle and Right: Training on the Colleges dataset and evaluating on the full CC-18 and CTR-23 evaluation suites, respectively.

Panel of Figure 2 — Figure 2: Universality of dataset quality. (a) We compute the rank of training sets for each evaluation dataset and then plot a histogram of the Spearman correlation between the ranks on all pairs of evaluation datasets. Most evaluation datasets have very correlated ranks. (b) We group the training and evaluation datasets into distinct domains and plot average ranks (lower is better). Training and evaluation pairs from the same domain do not appear to transfer better than ones from different domains.

Panel of Figure 3 — Figure 3: Not all cells in a table are made equal: the number of features matters much more than the number of instances as pre-training dataset for tabular ICL models.

Figure 4: Downstream AUC vs. the number of unique tasks used during training. (max) denotes the unrestricted setting where the sampler draws from the full set of eligible task configurations with no upper count limit. More tasks during pre-training consistently leads to better transfer.

Dataset-level selection
Setup	AUC	Acc.	Corr.	$\boldsymbol{R^{2}}$	Avg.
Baseline	${0.9757}_{\scriptscriptstyle[.9731,.9776]}$	${0.9233}_{\scriptscriptstyle[.9199,.9265]}$	${0.9021}_{\scriptscriptstyle[.8993,.9047]}$	${0.8145}_{\scriptscriptstyle[.8097,.8188]}$	$0.9039$
Deduped	${0.9752}_{\scriptscriptstyle[.9728,.9773]}$	${0.9236}_{\scriptscriptstyle[.9202,.9269]}$	${0.9019}_{\scriptscriptstyle[.8989,.9047]}$	${0.8146}_{\scriptscriptstyle[.8101,.8189]}$	$0.9038$
$\geq\!30$	${0.9728}_{\scriptscriptstyle[.9704,.9748]}$	${0.9208}_{\scriptscriptstyle[.9175,.9241]}$	${0.8882}_{\scriptscriptstyle[.8832,.8930]}$	${0.7890}_{\scriptscriptstyle[.7812,.7966]}$	$0.8927$
$\geq\!30$ + dedup	${0.9724}_{\scriptscriptstyle[.9701,.9744]}$	${0.9195}_{\scriptscriptstyle[.9159,.9231]}$	${0.8871}_{\scriptscriptstyle[.8817,.8921]}$	${0.7870}_{\scriptscriptstyle[.7788,.7949]}$	$0.8915$
Column-level pre-processing

Table 1: Dataset-level selection and column-level pre-processing ablations. Results are IQM with

95\%

CI; AUC/accuracy on CC-18, correlation/

R^{2}

on CTR-23, Avg. is unweighted. First row is the full-corpus baseline; dataset-level rows test deduplication,

\geq\!30

feature filtering, and both. Column-level Setup = (NaN, Corr, mf-

5

): ‘+’ enabled, ‘–’ disabled; NaN drops columns with

>50\%

missing, Corr is Pearson (P) or Spearman (S) deduplication at threshold

0.90

, and mf-

5

skips datasets with

<5

surviving features. Best column-level IQM per metric is bold .

Figure 5: Pre-processing yields Elo gains at all context sizes. Purple bars are models trained with pre-processing (+pp), blue bars their corresponding baselines, and grey bars other TabArena methods. TabDPT [ 22 ] is in orange.

Figure 6: In-context retrieval AUC (x-axis) vs. base AUC (y-axis) for $88$ models each pre-trained on one dataset. Each point is one model.

Panel of Figure 7 — Figure 7: Ablation forcing $W_{Q}=W_{K}$ on six single-table pre-training corpora, grouped by how well the default architecture generalizes with single-table pre-training.

Figure 8: Mean CC-18 AUC for the default architecture and the

W_{Q}=W_{K}

variant, both pre-trained on the full OpenML corpus of Section 3 .

Figure 9: Pair attention similarity vs. mean AUC, $n=2652$ points (one per model pair $\times$ probe dataset), averaged over blocks. Spearman correlation is $0.537$ . Per-block results are in Figure 11 .

Group	Mean	$n$	vs.	$\Delta$	$95\%$ CI
SS	$\mathbf{0.504}$	$156$	SW	$+0.163$	$[+0.143,+0.183]$
SW	$0.341$	$338$	WW	$+0.051$	$[+0.027,+0.075]$
WW	$0.290$	$156$

Table 2: Pair attention similarity by category. CIs are non-parametric bootstrap intervals (

10{,}000

resamples) on the difference of group means.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	AUC on CC-18	# Instances	# Features	Dataset Domain
APSFailure	0.918	76000	171	industrial-operational
colleges	0.918	7063	45	other
ipums_la_99-small	0.917	8844	57	financial-demographic
ipums_la_97-small	0.916	7019	61	financial-demographic
ipums_la_98-small	0.915	7485	56	financial-demographic
dionis	0.915	416188	61	other

Appendix

Table 3: AUC scores on CC-18 for TabDPT pre-trained on various datasets, sorted in descending order of AUC.

Dataset	$R^{2}$ on CTR-23	# Instances	# Features	Dataset Domain
colleges	0.693	7063	45	other
kick	0.693	72983	33	industrial-operational
ipums_la_97-small	0.691	7019	61	financial-demographic
dionis	0.685	416188	61	other
ipums_la_98-small	0.685	7485	56	financial-demographic
Census-Income	0.684	299285	42	financial-demographic

Appendix

Table 4:

R^{2}

scores on CTR-23 for TabDPT pre-trained on various datasets, sorted in descending order of

R^{2}

Figure 10: Pairwise win-rate matrix on TabArena-Lite across all evaluated methods (our models and baselines). Each cell shows the fraction of tasks where the row method outperforms or ties the column method.

Method	Elo	Avg. Rank
TABPFN-V2.6	1583	4.16
TABICLv2	1545	4.80
REALTABPFN-V2.5	1477.6	6.16
Our TabDPT (4k $+$ pp)	1322.8	10.25
CatBoost	1320.8	10.31
Our TabDPT (2k $+$ pp)	1299.8	10.96

Appendix

Table 5: Full TabArena-Lite leaderboard (51 tasks, fold 0). Elo is anchored at Random Forest = 1000. Our models are in bold.

Strong (AUC $\geq 0.900$ )	Weak (AUC $\leq 0.831$ )
APSFailure , Census-Income ,	Diabetes130US , airlines ,
KDDCup09_appetency , SpeedDating ,	analcatdata_supreme , chess ,
cardiotocography , Colleges ,	hill-valley , parity5_plus_5 ,
covertype , gas-drift ,	pbcseq , poker-hand ,
ipums_la_98-small , musk ,	pollen , puma8NH ,
pol , road-safety , scene	twonorm , visualizing_soil , yeast

Appendix

Table 6: Models in the strong and weak sets, identified by their training dataset.

Block	SS	SW	WW	SS $-$ SW	$95\%$ CI
$0$	$0.854$	$0.696$	$0.593$	$+0.158^{\dagger}$	$[+0.124,+0.193]$
$1$	$0.525$	$0.442$	$0.402$	$+0.083^{\dagger}$	$[+0.005,+0.164]$
$2$	$0.123$	$0.047$	$0.279$	$+0.075$	$[-0.027,+0.176]$
$3$	$0.145$	$0.110$	$0.156$	$+0.035$	$[-0.050,+0.119]$
$4$	$0.551$	$0.383$	$0.330$	$+0.168^{\dagger}$	$[+0.132,+0.205]$
$5$	$0.584$	$0.363$	$0.238$	$+0.221^{\dagger}$	$[+0.187,+0.255]$

Appendix

Table 7: Per-block group means and SS

-

SW gap with

95\%

bootstrap CI. One observation per (model pair, probe, layer). Rows whose CI excludes zero are marked

\dagger

Figure 11: Per-block pair attention similarity vs. mean AUC, $n=2652$ points (one per model pair $\times$ probe dataset).

Panel of Figure 12 — Figure 12: Ablations where $W_{Q}=W_{K}$ is enforced, on Visualizing Soil (left) and Chess (right), using the same setting as Figure 7(a) .

Explore similar work

Aug 23, 2026cs.LG

Tabular foundation models for non-tabular tasks

Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.

Goran Nakerst, John Brennan, Wouter Beugeling +1

Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany

Tabular Foundation Models Cross-Domain Generalization Tabular ICL

Ask PDF

May 7, 2026cs.LG

Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of language models. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

Amir Rezaei Balef, Mykhailo Koshil, Katharina Eggensperger

TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany

Tabular Foundation Models Transformer Interpretability Tabular ICL

Ask PDF

Oct 6, 2026cs.LG

TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.

Zijian Li, Xiangchen Song, Gongxu Luo +7

Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · Guangdong University of Technology +1

Synthetic Data Pretraining Tabular Foundation Models Recurrent Transformers Tabular ICL

Ask PDF