Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

Lyle Regenwetter; Rosen Yu; Cyril Picard; Faez Ahmed

cs.LGMar 5, 2026

Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

Authors: Lyle Regenwetter, Rosen Yu, Cyril Picard, Faez Ahmed

Organizations: Massachusetts Institute of Technology

Abstract

Predictive modeling in engineering applications has long been dominated by bespoke models and small, siloed tabular datasets, limiting the applicability of large-scale learning approaches. Despite recent progress in tabular foundation models, the resulting synthetic training distributions used for pre-training may not reflect the statistical structure of engineering data, limiting transfer to engineering regression. We introduce TREDBench, a curated collection of 83 real-world tabular regression datasets with expert engineering/non-engineering labels, and use TabPFN 2.5's dataset-level embedding to study domain structure in a common representation space. We find that engineering datasets are partially distinguishable from non-engineering datasets, while standard procedurally generated datasets are highly distinguishable from engineering datasets, revealing a substantial synthetic-real domain gap. To bridge this gap without training on real engineering samples, we propose an embedding-guided synthetic data curation method: we generate and identify "engineering-like" synthetic datasets, and perform continued pre-training of TabPFN 2.5 using only the selected synthetic tasks. Across 35 engineering regression datasets, this synthetic-only adaptation improves predictive accuracy and data efficiency, outperforming TabPFN 2.5 on 29/35 datasets and AutoGluon on 27/35, with mean multiplicative data-efficiency gains of 1.75x and 4.44x, respectively. More broadly, our results indicate that principled synthetic data curation can convert procedural generators into domain-relevant "data engines," enabling foundation models to improve in data-sparse scientific and industrial domains where real data collection is the primary bottleneck.

Figures & tables

Figure 1: 2D T-SNE embedding of TabPFN 2.5’s 192-dimensional dataset embedding. Distributional differences between engineering and non-engineering data are apparent. A small proportion of the procedurally generated data overlaps with the engineering data. The procedurally generated data appears to splay off toward random noise, indicating that a significant amount of the procedurally generated training data may be too random to be realistic.

Figure 2: Confusion Matrix illustrating distinguishability of dataset classes (according to TabPFN 2.5’s embedding). Distributional differences between engineering and non-engineering datasets can be witnessed. A small portion of the procedurally generated data appears to span the engineering data.

Figure 3: Visualization of procedurally generated data that is subselected to maximally resemble engineering data. Embedding is a 2D t-SNE of the TabPFN 2.5’s 192-dimensional dataset embedding. The selected procedurally generated appears to better span the engineering data compared to the randomly generated data.

Figure 4: Confusion matrix illustrating that subselected procedurally generated data and engineering data are less easily distinguished (based on TabPFN 2.5’s embedding).

Baseline	Datasets (/35)	Fraction (%)
vs. TabPFN 2.5	29	82.9
vs. AutoGluon	27	77.1

Table 1: Comparison of fine-tuned TabPFN 2.5 against baseline models across 35 engineering datasets using a 70/30 train/test split. Entries report the number and percentage of datasets on which the fine-tuned model achieves lower test error.

Figure 5: Sample comparison of model accuracy for different quantities of data over 12 example problems.

Panel of Figure 6 — Figure 6: Comparison of the estimated data-efficiency gain as a result of fine-tuning (our fine-tuned TabPFN 2.5 model vs. non-fine-tuned TabPFN 2.5 model) across all tested datasets. Left: Additive data-efficiency ( $E_{add}$ ). Right: Multiplicative data-efficiency ( $E_{mult}$ )

Panel of Figure 7 — Figure 7: Comparison of the estimated relative data-efficiency of our fine-tuned TabPFN 2.5 model against AutoGluon across all tested datasets. Left: Additive data-efficiency ( $E_{add}$ ). Right: Multiplicative data-efficiency ( $E_{mult}$ )

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset Name	Src.	Type	Dimensions
a1	[ 24 ]	NE	198 x 11
a2	[ 24 ]	NE	198 x 11
a3	[ 24 ]	NE	198 x 11
a4	[ 24 ]	NE	198 x 11
a5	[ 24 ]	NE	198 x 11
a6	[ 24 ]	NE	198 x 11

Appendix

Table 2: TREDBench dataset summary showing name, source, classification, and dimensionality.

Parameter	Domain	Scale	Best
$n\_estimators$	$[300,2000]$	Linear	1700
$learning\_rate$	$[0.01,0.2]$	Log	0.0158
$max\_depth$	$[3,10]$	Linear	7
$subsample$	$[0.6,1.0]$	Linear	0.903
$colsample\_bytree$	$[0.6,1.0]$	Linear	0.622
$min\_child\_weight$	$[1,20]$	Log	3.84

Appendix

Table 3: Optuna search space and selected hyperparameters for multiclass XGBoost trained on embedding features.

n\_estimators

was considered in intervals of 100

Synthetic Data Pretraining

Synthetic data pretraining uses generated, simulated, or transformed examples to initialize models when real training data are scarce or costly.

See all 119 papers

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
Jun 1, 2026Andrej Tschalzev, Nick Erickson, Yuyang Wang +4Tabular ML Automated Feature Engineering
University of Mannheim · Prior Labs · AWS +3
ReTabSyn: Realistic Tabular Data Synthesis via Reinforcement Learning
Mar 11, 2026Xiaofeng Lin, Seungbae Kim, Zhuoya Li +3Synthetic Tabular Data Generation Tabular Data Synthesis
University of California, Los Angeles, USA · University of South Florida, USA · Cisco, USA +1
Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality
May 18, 2026Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni +3Synthetic Data Pretraining Distribution Shift Robustness
Lexsi Labs

Jun 1, 2026J/K move · Enter open · S save

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

Andrej Tschalzev, Nick Erickson, Yuyang Wang, Huzefa Rangwala+3

University of Mannheim · Prior Labs · AWS+3

Tabular ML Automated Feature Engineering Tabular Prediction

Open paper AskPDF arXiv

Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.

Abstract

Synthetic Data Pretraining

Explore similar work

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

ReTabSyn: Realistic Tabular Data Synthesis via Reinforcement Learning

Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality