FTFM: Factorized Tabular Foundation Models

Josif Grabocka

cs.LGOct 18, 2026

FTFM: Factorized Tabular Foundation Models

Authors: Josif Grabocka

Organizations: University of Technology Nuremberg

Abstract

Tabular foundation models (TFMs) are becoming a powerful alternative to conventional tabular learning, adapting to new datasets in context without task-specific training. Current architectures, however, face a trade-off: models that preserve cell-level representations are expensive because they repeatedly attend across rows and columns, while faster models collapse each row into a fixed embedding and lose this fine-grained state. We introduce Factorized Tabular Foundation Models (FTFM), which preserve cell representations while factorizing their computation through one contextual latent per row and one per column. Cells no longer attend to one another directly; instead, each cell is updated through the interaction of its row and column latents, reducing repeated attention while retaining feature-specific information. FTFM achieves competitive predictive performance in tabular classification on major benchmarks such as TabArena and BeyondArena.

Figures & tables

Family	Examples	Inference complexity	Feature preservation
Preserving	TabPFN-2, EXAONE Tabular	$\Theta(MN^{2}D+NM^{2}D+NMD^{2})$	yes
Collapsing	TabICL v1/v2, TabPFN-3, TabFM	$\Theta(N^{2}D+NM^{2}D+NMD^{2})$	no
Factorized (ours)		$\Theta(N^{2}D+M^{2}D+NMD^{2})$	yes

Table 1: Asymptotic cost of one in-context forward pass

Stage	Prior	Rows	Features	Updates	Peak LR	GPU-h
A	Synthetic only	$1{,}024$	$1$ – $100$	$500$ k	$8\times 10^{-4}$	$1{,}600$
B	Synthetic and real; $50\%-50\%$	$256$ – $16{,}384$	$4$ – $128$	$200$ k	$8\times 10^{-5}$	$540$
C	Synthetic and real; $50\%-50\%$	$256$ – $32{,}768$	$4$ – $128$	$130$ k	$1\times 10^{-5}$	288

Table 2: The pretraining curriculum of FTFM

Figure 1: Performance on

4

K held-out tables from stage 1 of the TabICLv2 prior, with every model scored on the same tables. (a) Average ranks; a bar indicates a non-significant difference following a Wilcoxon signed-rank test; (b) mean paired

\Delta

CE (cross-entropy) against FTFM; (c) analysis by number of features and number of classes.

Figure 2: Cross in-distribution evaluation of the pretraining stages.

Stage	Mean error	C wins on	$p(C\text{ vs.})$
A	$0.1692$	$30/38$	$<0.001$
B	$0.1620$	$24/38$	$0.030$
C	$\mathbf{0.1614}$	—	—

Table 3: The performance of each pretraining stage evaluated out-of-distribution on TabArena Lite. Significance is a two-sided Wilcoxon signed-rank test.

Figure 3: Performance of FTFM on the 98 classification datasets of the BeyondArena benchmark. FTFM leads TabPFN-3 and TabICLv2 on Elo and on mean rank, and is lowest on improvability, which is the margin by which a method is beaten. The ranks are computed against 29 methods covered in the benchmark paper.

Slice	$n$	TabPFN-3	TabICLv2	RealMLP	CatBoost	FTFM
		(default)	(default)	(t+e)	(t+e)
tiny	39	$1366$ (2)	$1333$ (3)	$1287$ (5)	$1233$ (9)	$\bf 1426$ (1)
small	26	$1408$ (2)	$1323$ (4)	$1270$ (5)	$1196$ (7)	$\bf 1510$ (1)
medium	20	$1201$ (2)	$1121$ (10)	$\bf 1201$ (1)	$1180$ (3)	$1168$ (4)
large	13	$912$ (16)	$668$ (21)	$\bf 1285$ (1)	$1237$ (5)	$772$ (17)
low-dim	66	$1334$ (2)	$1312$ (3)	$1261$ (5)	$1222$ (6)	$\bf 1390$ (1)

Table 4: BeyondArena Elo by slice on the classification datasets of each slice that our run covers, with the place out of

29

methods in parentheses and

n

the number of datasets. Every column is recomputed on those same tasks with the benchmark’s own evaluator, so a row compares five methods on one set of datasets.

Figure 4: Critical-difference diagram over the full TabArena protocol,

38

datasets. Average rank, lower is better; a bar joins methods whose performance differences are not statistically significant. In contrast to Figure 7 , here the ranks are computed only among the five selected methods.

Figure 5: Average rank over the

38

TabArena Lite datasets, apples-to-apples (left) and apples-to-oranges (right); each line is one method moving between the regimes. On the right, every method is bagged over

8

members, but each brings its own different ensembling pipeline. The bracket carries the two-sided Wilcoxon signed-rank test of FTFM against TabFM, Holm-corrected over the ten pairs. The left column shows our runs and the right takes the four baselines from the benchmark’s published values on identical dataset splits.

Figure 6: Prediction time against table size on the BeyondArena classification datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: TabArena classification on

38

datasets and

594

splits under the full protocol. Results are recomputed with TabArena’s evaluator from published per-split results. Ranks are computed against the benchmark’s

72

published methods, with selected methods shown here.

Dataset	TabICLv2	TabPFN-3	EXAONE	TabFM	FTFM
APSFailure	$0.0058$	$0.0061$	$0.0061$	$\mathbf{0.0054}$	$0.0062$
Amazon_employee_access	$0.1469$	$0.1499$	$\mathbf{0.1285}$	$0.1420$	$0.1398$
Bank_Customer_Churn	$0.1281$	$0.1248$	$0.1253$	$\mathbf{0.1233}$	$0.1253$
Bioresponse	$0.1382$	$0.1228$	$0.1185$	$\mathbf{0.1136}$	$0.1237$
Diabetes130US	$0.3366$	$0.3296$	$\mathbf{0.3186}$	$0.3299$	$0.3327$
E-CommereShippingData	$0.2549$	$\mathbf{0.2529}$	$0.2532$	$0.2574$	$0.2532$

Appendix

Table 5: Error on the

38

TabArena classification datasets under the full protocol, averaged over splits. The metric is

1-\mathrm{AUC}

for binary tasks and log-loss for multiclass tasks. Lower is better, and the best result in each row is bold. Baselines use published per-split results.

	Stage-A		Stage-B		Stage-C
Scored on	CE	rank	CE	rank	CE	rank
Stage-A	$\mathbf{0.2866}$	$\mathbf{1.92}$	$0.2883$	$2.09$	$0.2878$	$1.98$
Stage-B	$0.3566$	$2.40$	$\mathbf{0.3278}$	$1.85$	$0.3286$	$\mathbf{1.75}$
Stage-C	$0.4096$	$2.48$	$0.3900$	$1.86$	$\mathbf{0.3900}$	$\mathbf{1.66}$
Scored on	A vs B		A vs C		B vs C
Stage-A	$6.9\times 10^{-8}$		$0.001$		$2.3\times 10^{-5}$

Appendix

Table 6: Mean cross-entropy and average rank by checkpoint and evaluation prior. The best value for each metric within each prior is bold. On the Stage-B prior, different checkpoints minimize mean error and rank. The lower panel gives two-sided Wilcoxon signed-rank

p

-values, Holm-corrected across the three pairs within each prior.

Dataset	FTFM	TabPFN-3	TabICLv2	RealMLP	CatBoost
acquire_valued_shoppers_challenge-f	$0.3518$	$\mathbf{0.3468}$	$0.3681$	$0.3516$	$0.3559$
amazon_employee_access-4417c4c952bc	$0.1318$	$0.1489$	$0.1419$	$0.1276$	$\mathbf{0.1134}$
amex_non_iid_1m-e2e64ee8b42e	$0.0574$	$0.0618$	$0.0631$	$\mathbf{0.0546}$	$0.0559$
anes_voting_2026-58a0e941922e	$0.1290$	$\mathbf{0.1200}$	$0.1305$	$0.1283$	$0.1272$
aps_failure-b63fc6330117	$0.0067$	$\mathbf{0.0060}$	$0.0062$	$0.0083$	$0.0086$
asp_potassco_classification-6cbac33e	$2.2139$	$2.2067$	$\mathbf{2.1993}$	$2.2158$	$2.2245$

Appendix

Table 7: Error on the

98

BeyondArena classification datasets, averaged over splits. Lower is better, and the best result in each row is bold. Baselines use published per-split results. † marks FTFM’s two reduced-support results (

500{,}000

rows, rung

3

), which are not directly comparable to full-support results (see Appendix E ).

	FTFM		TabPFN-3		TabICLv2		RealMLP		CatBoost
Dataset	fit	pred.	fit	pred.	fit	pred.	fit	pred.	fit	pred.
acquire_valued_shoppers_challenge-f	$3643$	$51.0$	$454$	$59.9$	$344$	$35.4$	$80328$	$19.5$	$5064$	$2.2$
amazon_employee_access-4417c4c952bc	$208$	$5.8$	$45$	$4.7$	$20$	$2.3$	$9594$	$24.5$	$1652$	$6.3$
amex_non_iid_1m-e2e64ee8b42e	$20596$	$1320.8$	$20049$	$5953.4$	$5329$	$839.0$	$278841$	$766.3$	$85907$	$109.0$
anes_voting_2026-58a0e941922e	$2592$	$38.7$	$320$	$23.3$	$360$	$27.4$	$90522$	$134.7$	$10293$	$29.0$
aps_failure-b63fc6330117	$3203$	$137.7$	$204$	$31.2$	$229$	$22.1$	$49991$	$141.5$	$4118$	$1.9$

Appendix

Table 8: Fit and prediction times in seconds, averaged over splits. FTFM uses one H100, while baselines use the benchmark maintainers’ hardware, so absolute times are not directly comparable. Figure 6 compares scaling with table size, which a constant hardware factor preserves. † marks FTFM’s two reduced-support results (rung

3

Rung	Serving	Attention	Members	Budget	Support	$n$
$0$	fp32	fp32	$8$	$4$ h	full	$88$
$1$	fp32	fp32	$1$	$30{,}600$ s	full	$2$
$2$	fp32	fp32	$1$	$80{,}000$ s	full	$6$
$3$	fp32	fp32	$1$	$80{,}000$ s	$500$ k rows	$2$

Appendix

Table 9: Serving settings for the

98

BeyondArena classification datasets.

T

and

M

count support rows and columns after preprocessing, with

\mathrm{work}=T^{2}M

and

\mathrm{cells}=TM

. All rungs use fp32, including attention. Higher rungs reduce ensemble size, increase the time budget, then reduce support.

Tabular Foundation Models

Tabular foundation models learn reusable representations or in-context prediction skills for structured data such as spreadsheets and databases.

See all 188 papers

Tabular foundation models for non-tabular tasks
Aug 23, 2026Goran Nakerst, John Brennan, Wouter Beugeling +1Handwritten Digit Recognition Tabular Foundation Models
Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany
Tabular Foundation Models Are Effectively Shallow
Oct 9, 2026Irene Cannistraci, Julia E. VogtTransformer Pruning Tabular Foundation Models
Department of Computer Science ETH Zurich Switzerland
LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models
Jun 3, 2026Yuanrui Wang, Xingxuan Zhang, Han Yu +7Tabular Foundation Models Tabular Representation Learning
Stable AI · Tsinghua University

Aug 23, 2026J/K move · Enter open · S save

Tabular foundation models for non-tabular tasks

Goran Nakerst, John Brennan, Wouter Beugeling, Masudul Haque

Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany

Handwritten Digit Recognition Tabular Foundation Models Cross-Domain Generalization Tabular ICL

Open paper AskPDF arXiv

Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.

Abstract

Tabular Foundation Models

Explore similar work

Tabular foundation models for non-tabular tasks

Tabular Foundation Models Are Effectively Shallow

LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models