Predictive modeling in engineering applications has long been dominated by bespoke models and small, siloed tabular datasets, limiting the applicability of large-scale learning approaches. Despite recent progress in tabular foundation models, the resulting synthetic training distributions used for pre-training may not reflect the statistical structure of engineering data, limiting transfer to engineering regression. We introduce TREDBench, a curated collection of 83 real-world tabular regression datasets with expert engineering/non-engineering labels, and use TabPFN 2.5's dataset-level embedding to study domain structure in a common representation space. We find that engineering datasets are partially distinguishable from non-engineering datasets, while standard procedurally generated datasets are highly distinguishable from engineering datasets, revealing a substantial synthetic-real domain gap. To bridge this gap without training on real engineering samples, we propose an embedding-guided synthetic data curation method: we generate and identify "engineering-like" synthetic datasets, and perform continued pre-training of TabPFN 2.5 using only the selected synthetic tasks. Across 35 engineering regression datasets, this synthetic-only adaptation improves predictive accuracy and data efficiency, outperforming TabPFN 2.5 on 29/35 datasets and AutoGluon on 27/35, with mean multiplicative data-efficiency gains of 1.75x and 4.44x, respectively. More broadly, our results indicate that principled synthetic data curation can convert procedural generators into domain-relevant "data engines," enabling foundation models to improve in data-sparse scientific and industrial domains where real data collection is the primary bottleneck.
Figures & tables
Figure 1: 2D T-SNE embedding of TabPFN 2.5’s 192-dimensional dataset embedding. Distributional differences between engineering and non-engineering data are apparent. A small proportion of the procedurally generated data overlaps with the engineering data. The procedurally generated data appears to splay off toward random noise, indicating that a significant amount of the procedurally generated training data may be too random to be realistic.
Figure 2: Confusion Matrix illustrating distinguishability of dataset classes (according to TabPFN 2.5’s embedding). Distributional differences between engineering and non-engineering datasets can be witnessed. A small portion of the procedurally generated data appears to span the engineering data.
Figure 3: Visualization of procedurally generated data that is subselected to maximally resemble engineering data. Embedding is a 2D t-SNE of the TabPFN 2.5’s 192-dimensional dataset embedding. The selected procedurally generated appears to better span the engineering data compared to the randomly generated data.
Figure 4: Confusion matrix illustrating that subselected procedurally generated data and engineering data are less easily distinguished (based on TabPFN 2.5’s embedding).
Baseline
Datasets (/35)
Fraction (%)
vs. TabPFN 2.5
29
82.9
vs. AutoGluon
27
77.1
Table 1: Comparison of fine-tuned TabPFN 2.5 against baseline models across 35 engineering datasets using a 70/30 train/test split. Entries report the number and percentage of datasets on which the fine-tuned model achieves lower test error.
Figure 5: Sample comparison of model accuracy for different quantities of data over 12 example problems.
Figure 6: Comparison of the estimated data-efficiency gain as a result of fine-tuning (our fine-tuned TabPFN 2.5 model vs. non-fine-tuned TabPFN 2.5 model) across all tested datasets. Left: Additive data-efficiency ( Eadd ). Right: Multiplicative data-efficiency ( Emult )
Figure 7: Comparison of the estimated relative data-efficiency of our fine-tuned TabPFN 2.5 model against AutoGluon across all tested datasets. Left: Additive data-efficiency ( Eadd ). Right: Multiplicative data-efficiency ( Emult )
Table 3: Optuna search space and selected hyperparameters for multiclass XGBoost trained on embedding features. n_estimators was considered in intervals of 100
Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce TabPrep, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep (see https://github.com/atschalz/tabprep), we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.
Deep generative models can help with data scarcity and privacy by producing synthetic training data, but they struggle in low-data, imbalanced tabular settings to fully learn the complex data distribution. We argue that striving for the full joint distribution could be overkill; for greater data efficiency, models should prioritize learning the conditional distribution P(y∣X), as suggested by recent theoretical analysis. Therefore, we overcome this limitation with ReTabSyn, a Reinforced Tabular Synthesis pipeline that provides direct feedback on feature correlation preservation during synthesizer training. This objective encourages the generator to prioritize the most useful predictive signals when training data is limited, thereby strengthening downstream model utility. We empirically fine-tune a language model-based generator using this approach, and across benchmarks with small sample sizes, class imbalance, and distribution shift, ReTabSyn consistently outperforms state-of-the-art baselines. Moreover, our approach can be readily extended to control various aspects of synthetic tabular data, such as applying expert-specified constraints on generated observations.
Xiaofeng Lin, Seungbae Kim, Zhuoya Li +3
University of California, Los Angeles, USA · University of South Florida, USA · Cisco, USA +1
What determines the quality of a tabular foundation model? Unlike language or vision, tabular foundation models acquire their inductive biases almost entirely from synthetic pretraining distributions, yet the design of these distributions remains poorly understood. Standard synthetic priors are too well-behaved: they omit the irregularities and failure modes that determine deployment robustness. We introduce O'Prior, a compositional realism prior built around four coupled components: a hierarchical SCM meta-generator spanning diverse functional families; a modular realism engine covering heterogeneous marginals, missingness, and target transforms; an explicit stress module injecting confounding and support-query mismatch; and a curriculum-governed, leakage-safe generation protocol. To isolate prior design as the scientific variable, we hold architecture, optimizer, and compute budget fixed and vary only the synthetic task distribution. O'Prior yields consistent and substantial improvements in downstream accuracy and robustness across real tabular benchmarks, with gains concentrated in regimes characterized by distributional irregularities. Ablations confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently, their effects are not interchangeable. These results establish synthetic prior construction as a first-order and largely overlooked determinant of tabular foundation model quality.