TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Deqing Fu; Huangyuan Su; Rajat Sen; Taman Narayan; Sujay Sanghavi; Abhimanyu Das; Weihao Kong

cs.LGSep 29, 2026

TabFM-Auto: Self-Evolving Pipelines for Tabular Foundation Models

Authors: Deqing Fu, Huangyuan Su, Rajat Sen, Taman Narayan, Sujay Sanghavi, Abhimanyu Das, Weihao Kong

Organizations: Google Research · University of Southern California · Google DeepMind · Harvard University · University of Texas at Austin

Abstract

Tabular foundation models achieve strong zero-shot accuracy on structured data by pretraining on synthetic tables, but they ignore the column names, task descriptions, and auxiliary files that carry dataset semantics. Meanwhile, self-evolving machine learning engineering (MLE) agents train models from scratch on each dataset, yet jointly searching over features, architectures, and hyperparameters is noisy and prone to overfitting. We introduce TabFM-Auto, which pairs a tabular foundation model, TabFM, with a language model agent that evolves the data pipeline around it. Guided by dataset metadata and validation feedback, TabFM-Auto iteratively refines data cleaning, feature engineering, context selection, and post-processing to reduce TabFM's error. Across all 51 datasets of the TabArena benchmark, five TabFM-Auto configurations with different agents and language models take the top five overall positions, and the best raises TabFM from 1785 to 2013 Elo. The discovered pipelines also transfer to other frozen tabular foundation models (+69 to +143 Elo) with no further search. On the 8 tabular competitions of MLE-Bench, TabFM-Auto ranks first overall among MLE agents.

Figures & tables

Figure 1: Overall TabArena leaderboard across all 51 datasets (38 classification, 13 regression). All five TabFM-Auto configurations (red) take the top five positions, reaching up to 2013 Elo (

+228

over the TabFM baseline) and outperforming all other tabular foundation models.

Figure 2: Overview of TabFM-Auto . (a) Within a sandboxed evaluator, a coding agent repeatedly edits the data pipeline

P

around a frozen 400M-parameter TabFM model using 3-fold cross-validation on the training set. (b) At test time, the selected pipeline

P^{*}

transforms the input table through data cleaning, feature engineering, context sampling, frozen TabFM inference, and post-processing.

Panel of Figure 3 — Figure 3: Representative TabFM-Auto pipelines across the three categories on TabArena ( Antigravity with Gemini 3.8 Flash ). (I) Explicit aeroacoustic scaling laws on airfoil_self_noise , (II) relational graph features on Amazon_employee_access , and (III) minority-class context ensembling and prior calibration on website_phishing .

Figure 4: Search dynamics and test generalization. (a) 3-fold cross-validation gain on fold 0’s training set, (b) official test error reduction across all folds, and (c) official test TabArena Elo progression across all folds.

Method	Working Models	Overall Elo $\uparrow$	Cls. Elo $\uparrow$	Reg. Elo $\uparrow$
XGBoost (tuned + ensemble)	XGBoost	1360.4	1361.7	1428.7
LightGBM (tuned + ensemble)	LightGBM	1420.1	1419.6	1511.7
Coding Agent (AGY + Gemini 3.8 Flash)	Any ML models (GBDTs, PyTorch, etc.)	1468.8	1472.0	1615.3
AutoGluon 1.3 (4h)	Stacked GBDT & NN ensembles	1490.3	1473.4	1646.1
TabFM	TabFM (frozen)	1785.3	1768.7	2045.9
TabFM+	TabFM (frozen)	1856.0	1836.7	2169.2

Table 2: Controlled ablation of TabFM-Auto vs. an unconstrained coding agent. Both agents use the identical harness, model, sandbox, and a 6-hour search budget per dataset.

Figure 5: Final pipelines found for TabFM also help other tabular foundation models. Default is each model’s official TabArena result, rated in the same fit as

P^{*}

P^{*}

runs the same frozen model within the final per-dataset pipelines that TabFM-Auto ( Claude Code , Opus 5 ) found for TabFM , keeping only the context size from TABFM_KWARGS and without further search.

Agent Harness & Model	Datasets	# Evals	# API Calls	Prompt Tokens	Generated Tokens
Codex, Opus 5	51	1,257	18,948	1.95B	12.23M
Claude Code, Opus 5	51	4,296	7,452	1.28B	9.76M
Antigravity, Gemini 3.8 Flash	51	4,320	51,968	5.30B	20.50M
Claude Code, Gemini 3.8 Flash	51	4,602	38,401	3.75B	9.07M
Codex, Gemini 3.8 Flash	51	3,669	85,862	12.13B	20.25M

Table 3: Cumulative evaluation count, API calls, token consumption, and tool invocation breakdown across 51 TabArena datasets. (Top) Pipeline evaluations, LLM API requests, and prompt/generated token counts. (Bottom) Tool-call breakdown across the five action categories.

Figure 6: Official test scores across all 8 MLE-Bench-Tabular competitions comparing both TabFM-Auto configurations against five leading external agents (CAIR MARS+, PiEvolve 24h, MLEvolve, Famou-Agent 2.0, and AIBuildAI) and Kaggle Gold Medal thresholds.

Figure 7: MLE-Bench-Tabular overall comparison. (a) Elo ratings of 14 MLE agents (our two configurations and all 12 external agents with complete 8-competition submissions, see Appendix D.1 ) and (b) pairwise win rates over the same run-vs-run matches showing the Opus 5 variant.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

#	Method	Elo $\uparrow$	Wins $\uparrow$	Improv. (%) $\downarrow$	G-Mean $\downarrow$
1	TabFM-Auto (Codex, Opus 5)	2013.0 ${}^{\,+123.7}_{\,-103.2}$	25.00	1.42%	0.3400
2	TabFM-Auto (Claude Code, Opus 5)	1993.6 ${}^{\,+126.8}_{\,-124.3}$	25.47	1.26%	0.3402
3	TabFM-Auto (Antigravity, Gemini 3.8 Flash)	1979.6 ${}^{\,+111.3}_{\,-95.7}$	26.12	1.82%	0.3371
4	TabFM-Auto (Claude Code, Gemini 3.8 Flash)	1957.8 ${}^{\,+106.1}_{\,-103.7}$	22.67	1.88%	0.3446
5	TabFM-Auto (Codex, Gemini 3.8 Flash)	1940.1 ${}^{\,+120.1}_{\,-133.9}$	25.06	1.81%	0.3435
6	TabFM+	1856.0 ${}^{\,+118.4}_{\,-102.7}$	17.58	2.20%	0.3513

Appendix

Table 4: TabArena overall leaderboard across all 51 datasets. Super/subscripts give the 95% bootstrap percentile interval of each Elo rating relative to its median over 100 resampling rounds, and are asymmetric because the Bradley–Terry bootstrap distribution is right-skewed.

#	Method	Overall Elo (51) $\uparrow$	Classification Elo (38) $\uparrow$	Regression Elo (13) $\uparrow$
1	TabFM-Auto ( Codex , Opus 5 )	1950.7 ${}^{\,+190.3}_{\,-169.3}$	1877.5 ${}^{\,+197.0}_{\,-183.6}$	2616.3 ${}^{\,+355.1}_{\,-188.8}$
2	TabFM-Auto ( Claude Code , Gemini 3.8 Flash )	1862.8 ${}^{\,+181.0}_{\,-163.7}$	1805.0 ${}^{\,+186.1}_{\,-187.6}$	2336.1 ${}^{\,+505.0}_{\,-318.9}$
3	TabFM+	1853.2 ${}^{\,+194.2}_{\,-177.3}$	1782.8 ${}^{\,+204.9}_{\,-187.4}$	2409.9 ${}^{\,+294.5}_{\,-173.8}$
4	TabFM-Auto ( Claude Code , Opus 5 )	1843.3 ${}^{\,+202.8}_{\,-172.0}$	1763.6 ${}^{\,+203.7}_{\,-184.8}$	2562.2 ${}^{\,+422.8}_{\,-217.6}$
5	TabFM-Auto ( Antigravity , Gemini 3.8 Flash )	1791.9 ${}^{\,+181.7}_{\,-155.4}$	1710.5 ${}^{\,+201.8}_{\,-154.3}$	2532.4 ${}^{\,+319.5}_{\,-206.1}$
6	TabFM	1790.2 ${}^{\,+181.4}_{\,-168.5}$	1733.0 ${}^{\,+201.1}_{\,-190.7}$	2227.2 ${}^{\,+262.9}_{\,-144.8}$

Appendix

Table 5: TabArena Fold-0 held-out test Elo ratings across Overall (51 datasets), Classification (38 datasets), and Regression (13 datasets). Every method is scored on the official held-out test split of Fold 0, which shares no rows with Fold 0’s training split used for 3-fold cross-validation to select

P^{*}

. As in Table 4 , each TabFM-Auto configuration is rated individually against the 66-method TabArena pool using bootstrap Bradley–Terry estimation anchored to RandomForest (default)

=1000

. Super/subscripts give the 95% bootstrap percentile interval of each rating relative to its median.

Figure 9: TabArena Elo ratings, computed separately on the 38 classification (top) and 13 regression (bottom) datasets. Red bars denote TabFM-Auto configurations (#1–#5), while blue bars denote TabFM+ (#6) and TabFM (#7).

Category	Representative Features Built in Code	Configuration	Datasets	Mean ( $\Delta\%$ )	Median ( $\Delta\%$ )
I: Domain-Knowledge Features	Physics & Engineering : Strouhal $St=f\delta^{*}/U_{\infty}$ , Helmholtz $He$ ( airfoil ); water-to-binder ratio ( concrete )	Claude Code, Opus 5	17 / 51	+7.25%	+3.01%
	Clinical Medicine : ICD-9 organ chapters ( Diabetes130US ); $\text{MAP}=(2\text{DBP}+\text{SBP})/3$ , Shock Index ( MIC )	Antigravity, Gemini 3.8 Flash	17 / 51	+8.16%	+2.74%
	Science & Genomics : Coil $\texttt{width}=0\to\text{NaN}$ ( anneal ); SDSS color indices ( SDSS17 ); DNA splice motifs ( splice )	Combined Best	17 / 51	+8.85%	+3.01%
II: Statistical & Structural Features	Relational Graph Degrees : Marginal entity counts, pairwise co-occurrences, distinct- $B$ -per- $A$ ( Amazon_employee )	Claude Code, Opus 5	34 / 51	+2.31%	+0.90%
	Frequency & Missingness : Level counts, cross-fitted target means, missingness-aware column ranking ( kddcup09_appetency )	Antigravity, Gemini 3.8 Flash	29 / 51	+2.99%	+1.32%
	Wide-Table Compression : Collinearity pruning and truncated SVD projections ( Bioresponse , hiva_agnostic )	Combined Best	34 / 51	+3.87%	+1.71%

Appendix

Table 6: Taxonomy of discovered pipelines on TabArena. Each final pipeline

P^{*}

from Claude Code with Opus 5 and Antigravity with Gemini 3.8 Flash across the 51 datasets is classified into one of three mutually exclusive categories. Configuration lists each agent setup as well as Combined Best (selecting the pipeline with the higher 3-fold cross-validation score

\mathcal{U}_{\text{val}}

between the two configurations per dataset). Datasets reports the fraction of the 51 TabArena datasets assigned to each category, and Mean and Median report the mean and median relative official test error reduction (

\Delta\%

) over TabFM across datasets in that category (RMSE for regression,

1-\mathrm{AUROC}

for binary classification, and log-loss for multiclass classification).

Figure 10: Pairwise official test fold win rates on TabArena across (a) all 51 datasets, (b) 38 classification datasets, and (c) 13 regression datasets. Each cell reports the percentage of dataset-fold instances where the row method achieves lower official test error than the column method.

Model	Pipelines	Overall (51)		Classification (38)		Regression (13)
Model	Pipelines	Elo	Gain	Elo	Gain	Elo	Gain
TabICLv2	Default	1593.0		1587.1		1726.6
	Codex, Opus 5	1732.8	+139.4	1685.1	+97.6	2088.9	+361.5
	Claude Code, Opus 5	1736.7	+143.3	1704.4	+116.8	2006.5	+279.5
	Antigravity, Gemini 3.8 Flash	1700.2	+108.0	1678.2	+92.1	1938.5	+213.1
TabPFN-3	Default	1666.0		1643.8		1869.3

Appendix

Table 7: Final pipelines found for TabFM also help other tabular foundation models. TabArena Elo of each model with its default configuration and within the final per-dataset pipelines that TabFM-Auto found for TabFM with each coding agent and LLM. Each model is rated in a separate fit per configuration, and Gain is measured against the default model rated in the same fit. Default is the mean of the three default ratings. Bold marks the largest gain for each model.

Figure 11: The final pipelines of all three configurations help other tabular foundation models, and those from Claude Code with Opus 5 help most. Overall TabArena Elo on the 51 datasets of each model with its default configuration and within the final pipelines

P^{*}

that each TabFM-Auto configuration found for TabFM . Each model with

P^{*}

is rated in its own fit: the pale part of the bar reaches the default model rated in that fit, and the colored part is the gain. Default is the mean of the three default ratings.

Figure 12: The final pipelines from the other two configurations also help every model. As in Figure 5 , with the pipelines that TabFM-Auto found with Codex and Opus 5 (top) and with Antigravity and Gemini 3.8 Flash (bottom). Default is the default model rated in the same fit as

P^{*}

Figure 13: Model families in the final pipelines of the unconstrained coding agent ( Antigravity with Gemini 3.8 Flash , no TabFM ). Each wedge gives the number of final pipelines, out of 51, whose code uses that family, and in parentheses that number as a percentage of the 51 pipelines. Since a pipeline that blends several families counts once for each, the percentages sum to more than 100%. The right pie splits the GBDT wedge by library.

Figure 14: Official test G-Mean error reduction across search progress (log scale). (a) Regression suite (13 datasets) and (b) Classification suite (38 datasets).

Figure 15: Stage-wise validation-to-test generalization across the four search quarters (0%–25%, 25%–50%, 50%–75%, and 75%–100%). Official test error reduction (

\Delta\%

) across all folds versus 3-fold cross-validation gain (

\Delta\%

) on fold 0’s training set per dataset.

Figure 16: Search progression on MLE-Bench-Tabular. Best-so-far 3-fold cross-validation curves on the training split and final official test scores across the 8 Kaggle competitions (on volcanic-eruptions and smartphone-decimeter , where

P_{0}

fails with zero features,

0\%

plots each agent’s step-2 initial multi-file script).

Competition	Domain & Data Format	Metric	Identity	AGY +	CC +	Result	Key Features Added in engineer()
			TabFM	Gemini 3.8 Flash	Opus 5
volcanic-eruptions	Geophysics (Waveforms)	MAE $\downarrow$	Fails (IDs) †	326,950	97,355	GOLD	Multi-sensor waveform FFT bands, STA/LTA ratios, quantiles ( $9.4\times<$ PiEvolve )
nomad2018-conductors	Materials (3D Lattice)	RMSLE $\downarrow$	0.05737	0.04869	0.04440	GOLD	Reciprocal lattice unit-cell volume, cation electronegativity & radius dispersion
tps-dec-2021	Cartography & Hydrology	Acc $\uparrow$	0.96122	0.96237	0.96264	GOLD	Euclidean hydrology distance $\sqrt{H^{2}+V^{2}}$ , aspect compass decomposition
tps-may-2022	Manufacturing Telemetry	$1-\mathrm{AUROC}$ $\downarrow$	0.05179	0.00304	0.00179	BRONZE	Decomposing 10-char string f_27 into positional ordinal & unique-char counts
champs-coupling	Quantum Chem (3D NMR)	Log-MAE $\downarrow$	$+$ 1.19025	$-$ 1.10133	$-$ 1.54947	$>$ MEDIAN	3D inter-atomic Euclidean distance $r_{ij}^{-3}$ , Karplus dihedral $\cos^{2}\phi$ angles

Appendix

Table 8: End-to-end performance on 8 Kaggle MLE-Bench competitions. Best score per task is bolded , second best is underlined .

Dataset	Folds	TabFM	TabFM-Auto ( Opus 5 )	TabFM-Auto ( Gemini 3.8 Flash )
Part I: Regression Suite (13 Datasets - Metric: RMSE $\downarrow$ )
airfoil_self_noise	30	1.0734	0.91650 (+14.6%)	0.88780 (+17.3%)
Used-Fiat-500	30	703.27	693.49 (+1.4%)	696.52 (+1.0%)
concrete_strength	30	3.9666	3.8169 (+3.8%)	3.8309 (+3.4%)
diamonds	9	496.42	484.52 (+2.4%)	484.54 (+2.4%)
Food_Delivery_Time	9	7.3328	7.1736 (+2.2%)	7.1724 (+2.2%)

Appendix

Table 9: Per-Dataset Official Test Performance Across All 51 TabArena Benchmarks. For each dataset, we report the number of official evaluation folds, TabFM , and the final submitted pipelines discovered by TabFM-Auto ( Claude Code with Opus 5 ) and TabFM-Auto ( Antigravity with Gemini 3.8 Flash ) across the Regression (RMSE

\downarrow

), Binary Classification (

1-\mathrm{AUROC}

\downarrow

), and Multiclass Classification (Log-Loss

\downarrow

) suites. Parentheses report the relative test metric improvement (

\Delta\%

RMSE reduction,

1-\mathrm{AUROC}

reduction, or Log-Loss reduction) vs. TabFM. Best score per row is bolded .

Explore similar work

Sep 29, 2026cs.LG

TabFM: A Zero-Shot Foundation Model for Tabular Data

Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulates supervised tabular prediction as in-context learning. TabFM produces calibrated zero-shot predictions in a single forward pass without task-specific tuning. Trained entirely on synthetic tables generated from structural causal models, TabFM learns general tabular representations that transfer zero-shot to real-world tasks. Across all 51 benchmark datasets in TabArena (38 classification and 13 regression), zero-shot TabFM ranks first among default tabular foundation models and outperforms tuned AutoML pipelines. Two extensions over the same frozen weights improve performance further on both tracks: multi-view feature expansion with ensembling and post-hoc calibration (TabFM+), and LLM-guided, dataset-specific data processing and feature engineering (TabFM-Auto).

Weihao Kong, Erez Louidor Ilan, Shuxin Nie +6

Google Research

Tabular Foundation Models Tabular Prediction Tabular ICL

Ask PDF

Aug 23, 2026cs.LG

Tabular foundation models for non-tabular tasks

Tabular foundation models (TFMs) have recently emerged as a promising paradigm for machine learning on tabular data, offering the ability to generalize across datasets without task-specific training. Since many machine learning datasets can be represented as tables, this raises the question: does TFM capability extend beyond tasks traditionally regarded as tabular? We address this question by using TabPFN v3 on three non-tabular classification problems: handwritten digit recognition on MNIST, language identification of French and German words, and image classification on Tiny ImageNet. In each case, the original data are represented as rows of a table and classification is formulated as prediction of a missing label. We evaluate performance as a function of the number of context samples provided to the pretrained model, with no additional training or fine-tuning. Despite having no explicit access to the spatial or sequential structure characterizing the data, TabPFN v3 in some cases achieves accuracies comparable with that of models or methods geared specifically toward the corresponding tasks.

Goran Nakerst, John Brennan, Wouter Beugeling +1

Institut für Theoretische Physik, Technische Universität Dresden, 01062 Dresden, Germany · Department of Physics, Maynooth University, Ireland · Physikalisches Institut (EP3) and Institute for Topological Insulators, Universität Würzburg, Am Hubland, 97074 Würzburg, Germany

Tabular Foundation Models Cross-Domain Generalization Tabular ICL

Ask PDF

Aug 14, 2026cs.LG

Do Tabular Foundation Models Still Need Feature Engineering?

Feature engineering has long been a cornerstone of tabular machine learning. Tabular foundation models (TFMs) are pretrained on a wide range of tabular datasets and applied via in-context learning. Their rise raises a natural question: does manual feature construction still matter as these models become more capable? To answer this, we perform a controlled study across several versions of two major TFM families, testing a wide range of existing feature engineering techniques on benchmark datasets from TabArena. We find a consistent pattern: feature engineering gains are concentrated in earlier model generations and become negligible for the strongest models. These results suggest that stronger TFMs depend less on explicitly engineered input representations. In a complementary experiment, however, adding in-context information from related datasets still improves performance. Our findings indicate a shift in the source of performance gains for stronger TFMs: re-representing existing inputs becomes less effective, while providing additional task-relevant context remains beneficial.

Yifan WU, Pinjun Dong, Jiran Tao +1

Department of Data Science, The Hong Kong Polytechnic University, Hong Kong

Tabular Foundation Models Tabular Prediction Tabular ICL

Ask PDF