TabFM: A Zero-Shot Foundation Model for Tabular Data

Weihao Kong; Erez Louidor Ilan; Shuxin Nie; Taman Narayan; Rajat Sen; Yichen Zhou; Deqing Fu; Samet Oymak

cs.LGSep 29, 2026

TabFM: A Zero-Shot Foundation Model for Tabular Data

Authors: Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan, Rajat Sen, Yichen Zhou, Deqing Fu, Samet Oymak, +1 more

Organizations: Google Research

Abstract

Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).

Figures & tables

Figure 1: Overall TabArena Elo ratings for the top 16 methods across all 51 benchmark datasets (38 classification and 13 regression). TabFM ranks first among zero-shot tabular foundation models, while TabFM+ ’s multi-view feature expansion, ensembling, and calibration and TabFM-Auto ’s LLM-guided feature engineering with Gemini-3.8-Flash take the top two positions overall without updating any weights.

Figure 2: The TabFM Architecture. (a) Tables are embedded cell-wise (

T\times H\times E

), compressed by two untied stages alternating column ISAB and row SAB with 8 CLS tokens into row representations

\mathbf{R}

, conditioned on

\mathbf{e}(y)

for train rows, then processed by a 24-layer in-context predictor. (b) Column ISAB attends across rows in linear time, while row SAB attends across features with RoPE.

Stage	Module	Width	Heads	Induc.	FFN	Params	Function
		Configuration
Cell Embedder	Spectral proj.	128	–	–	–	0.05 M	Learned Fourier features
Column Embedder	$2\times 3$ ISAB	128	4	256	512	3.3 M	Linear row attention
Row Interaction	$2\times 3$ SAB	128	8	–	512	1.6 M	RoPE feature attention
ICL Predictor	$24\times$ SAB	1024	8	–	4096	402.7 M	Masked in-context mixing
Prediction Head	2-layer MLP	1024	–	–	1024	1.05 M	Output

Table 1: TabFM architecture specification (400M checkpoint,

E=d_{\text{model}}=128

). All sublayers use sandwich RMSNorm with SwiGLU activations, zero dropout, and no bias parameters.

Figure 3: Synthetic pretraining and curriculum: (a) a sampled SCM graph with randomized functional dependencies, (b) the resulting table with numerical, categorical, and dual targets, and (c) the four-stage curriculum scaling context from 2,048 to 16,384 instances at constant tokens per step.

Table 2: TabArena benchmark leaderboards under repeated 10-fold cross-validation. Ratings are obtained from a unified Bradley–Terry fit anchored to RF (default) at 1000 Elo. G-mean is the geometric mean error across datasets, Wins counts fold-averaged dataset victories, and Improvability is the relative error reduction an oracle selector would achieve.

Figure 4: Task-separated TabArena Elo over 38 classification (top) and 13 regression (bottom) datasets, top 16 methods per suite. Colors mark TabFM variants in blue, peer foundation models in purple, AutoML in green, and tuned baselines in grey.

Figure 5: Pairwise win rates on TabArena: percentage of dataset-fold instances where the row method beats the column method, ties counted as half. Methods are ordered by Elo.

Figure 6: Per-dataset relative error reduction (

1-\mathrm{err}_{\mathrm{method}}/\mathrm{err}_{\mathrm{TabFM}}

) over zero-shot TabFM across all 51 TabArena datasets. Bars show TabFM-Auto and diamonds show TabFM+ . TabFM-Auto improves 41 of 51 datasets, ten by more than 10%.

Explore similar work

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models
Sep 29, 2026Deqing Fu, Huangyuan Su, Rajat Sen +4Tabular Prediction Automated Machine Learning
Google Research · University of Southern California · Google DeepMind +2
Tabular foundation models for non-tabular tasks
Aug 23, 2026Goran Nakerst, John Brennan, Wouter Beugeling +1Tabular Foundation Models Cross-Domain Generalization
Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany
TabPFN-3.5: Technical Report
Sep 15, 2026Benjamin Jäger, Nick Erickson, Léo Grinsztajn +44Tabular Foundation Models Inference-Time Optimization

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan+3

Google Research · University of Southern California · Google DeepMind+2

Tabular Prediction Automated Machine Learning Tabular Foundation Models Automated Feature Engineering LLM Agents

Open paper AskPDF arXiv

Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.

Abstract

Explore similar work

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Tabular foundation models for non-tabular tasks

TabPFN-3.5: Technical Report