On Data Scaling in Tabular ML: More Data, Better Generalization?

Ivan Rubachev; Nikita Lazarev; Artem Babenko

cs.LGOct 19, 2026

On Data Scaling in Tabular ML: More Data, Better Generalization?

Authors: Ivan Rubachev, Nikita Lazarev, Artem Babenko

Organizations: Yandex · HSE University

Abstract

Recent work in tabular machine learning has emphasized the importance of diverse and realistic evaluation settings. The niche of large-scale (1M+) data, however, remains under-explored. To study training-data scaling, we assemble eight real-world tabular datasets with up to around ten million training examples, retaining the natural distribution shifts of the tasks and using random-split evaluation as a control. We find that GBDTs and MLP-based models show broadly similar scaling trends, with gains from both additional training data and algorithmic improvements. Tabular foundation models show more varied scaling behavior, with recent versions benefiting more consistently from larger contexts even under distribution shift. However, scaling training data under distribution shift introduces new challenges: in-distribution and out-of-distribution performance mostly improve together at smaller scales but can diverge at larger scales. We examine this through MLP capacity sweeps and comparisons of the validation distributions used for tuning and early stopping in MLPs and GBDTs. We also compare common practices of using validation data, contrasting its use for model selection in MLP pipelines with using it for in-context fitting in tabular foundation model pipelines. Under stronger shifts, MLPs selected using validation representative of the evaluation conditions generally maintain or improve performance as training data grows. In contrast, including such examples in context improves foundation-model performance, but does not prevent deterioration with increasing training size in our experiments.

Figures & tables

Figure 1: Performance of parametric models across training-set sizes. Horizontal axis shows the number of training examples on a logarithmic scale. We report RMSE for regression and negative average precision (

-\mathrm{AP}

) for classification, so lower is better on every plot. On most tasks, additional training data improves performance across methods, improvements to architectures and training procedures also remain beneficial across scales.

Figure 2: Performance of tabular foundation models across training-set sizes. The figure follows Figure 1 conventions. Tabular foundation models also improve with in-context training set scale, but the scaling trends vary significantly across different models.

Figure 3: MLP width sweeps across training sizes. Top row: OOD test performance; bottom row: ID controls. The horizontal axis shows parameters per MLP on a logarithmic scale; darker curves indicate larger training sets. Learning rate, weight decay, and checkpoint are selected on the corresponding validation split. We report RMSE for all datasets except Aeolus Flight Delay, where we report negative average precision ( $-\mathrm{AP}$ ). Lower is better. Gains from increasing model size are limited under distribution shift, but become more pronounced with larger training sets under ID evaluation

Figure 4: Validation distribution affects ID and OOD scaling in MLP with Muon and LightGBM. Solid curves with filled circles use OOD validation (“Default val.”); dashed curves with open circles use ID validation. We report RMSE for all datasets except Aeolus Flight Delay, where we report negative average precision (

-\mathrm{AP}

). Lower is better in all panels. At larger training sizes, ID validation can improve ID performance while OOD performance deteriorates; conversely, OOD validation can improve OOD performance at the cost of ID performance.

Figure 5: Validation as context vs. as a selection signal on restricted training pools of Weather, Aeolus Flight Delay, and Cooking Time. Shades of blue (MLP) and purple (TabPFN v3.5 with appended validation) indicate validation sizes from 5 000 to 40 000. Conventions follow Figure 1 . TabPFN benefits from additional validation examples, but adding more training data can hurt performance. The MLP benefits more consistently from additional training data, but gains little from larger validation sets.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$b_{d}$	$R^{2}_{\mathrm{ind}}$	$R^{2}_{\mathrm{shared}}$	$\Delta R^{2}$
Cooking Time	0.1625	0.9938	0.9931	0.0006
Delivery ETA	0.0055	0.9727	0.9673	0.0054
Maps Routing	0.1811	0.9935	0.9929	0.0006
Weather	0.1843	0.9921	0.9904	0.0017
Jane Street	0.3403	0.9312	0.9024	0.0288
Bank Fraud	0.3373	0.9930	0.9849	0.0082

Appendix

Table 1: Dataset-shared exponents and mean in-sample

R^{2}

over methods.

\Delta R^{2}=R^{2}_{\mathrm{ind}}-R^{2}_{\mathrm{shared}}

, computed before rounding.

Method	Cooking	Delivery	Weather	Jane Street	Bank Fraud	Aeolus	Istella
XGBoost	$24.47\pm 0.03$	$34.29\pm 0.21$	$29.37\pm 0.01$	$31.26\pm 0.14$	$51.32\pm 0.25$	$7.72\pm 0.01$	$34.24\pm 0.09$
LightGBM	$55.63\pm 0.07$	$70.77\pm 0.06$	$130.24\pm 0.08$	$145.78\pm 0.76$	$66.97\pm 0.43$	$127.03\pm 0.73$	$46.28\pm 0.07$
$\mathrm{MLP}_{\mathrm{AdamW}}$	$3.73\pm 0.01$	$9.95\pm 0.07$	$7.27\pm 0.04$	$3.39\pm 0.01$	$7.62\pm 0.04$	$7.99\pm 0.07$	$14.03\pm 0.13$
$\mathrm{MLP}_{\mathrm{Muon}}$	$10.55\pm 0.06$	$14.44\pm 0.11$	$10.76\pm 0.04$	$6.67\pm 0.01$	$13.30\pm 0.02$	$19.58\pm 0.08$	$14.81\pm 0.11$
$\mathrm{MLP}^{\ddagger}_{\mathrm{AdamW}}$	$15.46\pm 0.17$	$18.99\pm 0.12$	$15.46\pm 0.05$	$7.96\pm 0.03$	$14.65\pm 0.04$	$14.76\pm 0.18$	$21.57\pm 0.22$
$\mathrm{MLP}^{\ddagger}_{\mathrm{Muon}}$	$20.36\pm 0.11$	$28.41\pm 0.40$	$24.29\pm 0.11$	$9.98\pm 0.02$	$17.20\pm 0.06$	$31.74\pm 0.17$	$34.53\pm 0.24$

Appendix

Table 2: Hyperparameter selection plus single-seed training and evaluation time on the full training splits, in hours;

\pm

denotes the sample standard deviation of evaluation time over ten seeds. TabPack’s cost includes pack training and ensemble selection. Together with the broadly similar scaling trends and competitive predictive performance, these runtimes highlight TabPack as an efficient alternative to separately tuned models at large training sizes. TabPack rows are shown in bold.

Dataset	Train (M)	Val. (K)	Test (K)	Features	Task
Maps Routing	5.12	60	60	986	R
Cooking Time	9.00	51	42	192	R
Delivery ETA	11.08	34	37	223	R
Weather	13.54	42	41	103	R
Istella	9.21	600	600	200	R
Jane Street	10.00	93	93	94	R

Appendix

Table 3: Dataset statistics (rounded). Training counts are in millions; validation and test counts are in thousands, per evaluation condition. The first four datasets are from TabReD. OOD splits are temporal except for Istella (query-based). R: regression; B: binary classification.

Figure 6: Default protocol: temporal validation and test splits, or query for Istella. Points show seed means; shaded bands show one standard deviation where multiple seeds are available.

Figure 7: IID protocol: checkpoint and hyperparameter selection on IID validation, evaluation on IID test. Points show seed means; error bars show one standard deviation where multiple seeds are available.

Figure 8: OOD test results: temporal test splits, or query for Istella. Blue curves show MLP with Muon; red curves show LightGBM. Solid curves with filled circles use default validation (temporal, or query-based for Istella); dashed curves with open circles use ID validation.

Figure 9: In-distribution test results: both validation protocols are evaluated on IID test. Colors and line styles follow Figure 8 .

Tabular ML

Tabular machine learning develops models and tools for learning from structured data arranged in rows and columns.

See all 114 papers

Jul 28, 2026

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to ensemble tree-based models. Most TFMs are trained and evaluated on independent and identically distributed data, but this assumption changes in real-world scenarios due to distribution shifts, which compromise the robustness of models. Limited research has been conducted of TFMs under distribution shifts. We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine TFMs, spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX and TabFM. Three real-world datasets from the TableShift study were considered (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types. Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy, with shift gaps ranging from 0.003 to 0.060 depending on shift type. The relationship between in-distribution and OOD predictive performance documented for classical tabular models extends into TFMs. We also identified a scalability gap, as high-performing models demand significant memory and computational resources beyond what standard deployment infrastructure can support. This study extends existing benchmarks for OOD in tabular data, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts.

Malena Loza, David Chushig-Muzo, Eva Milara +3

Colegio de Ciencias e Ingenierías, Universidad San Francisco de Quito (USFQ), Quito, Ecuador · Department of Signal Theory and Communications, Telematics and Computing Systems, Rey Juan Carlos University, Madrid, Spain · Facultad de Ingeniería, Universidad Latina de Panamá, Ciudad de Panamá, Panamá

Distribution Shift Robustness Tabular Foundation Models Distribution Shift OOD Generalization

Ask PDF

Jun 29, 2026

Beyond IID: How General Are Tabular Foundation Models, Really?

Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are increasingly evaluating tabular foundation models on diverse datasets and tasks. However, these task- and discipline-specific evaluations remain largely inaccessible to model researchers because benchmark software and evaluation protocols are fragmented. As a result, model researchers rely on standard benchmarks, which are mostly defined for tasks where tabular foundation models already excel. The most challenging scenarios are excluded, limiting meaningful progress in the field by focusing on marginal improvements on IID data rather than on broader, more demanding challenges. To overcome this, we introduce BeyondArena, the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types (with text, with high cardinality) from a broad range of disciplines. To enable unified benchmarking beyond standard benchmarks, we introduce Data Foundry, a Python framework and metadata schema for curating tabular datasets for predictive machine learning. Our results across 11 models and 142 curated datasets show that existing tabular foundation models excel on tiny- to medium-sized IID data, while traditional tree-based and deep learning models still dominate on non-IID, large, and high-dimensional datasets. BeyondArena guides model research for the most demanding challenges in tabular data, enabling progress towards truly foundational tabular models.

Lennart Purucker, Andrej Tschalzev, Nick Erickson +7

1Prior Labs · University of Mannheim · 4INRIA Saclay +5

Tabular Foundation Models OOD Generalization Tabular ML Tabular Prediction Cross-Dataset Generalization

Ask PDF

Apr 8, 2026

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale

While traditional tree-based ensemble methods have long dominated tabular tasks, deep neural networks and emerging foundation models have challenged this primacy, yet no consensus exists on a universally superior paradigm. Existing benchmarks typically contain fewer than 100 datasets, raising concerns about evaluation sufficiency and potential selection biases. To address these limitations, we introduce OmniTabBench, the largest tabular benchmark to date, comprising 3030 datasets spanning diverse tasks that are comprehensively collected from diverse sources and categorized by industry using large language models. We conduct an unprecedented large-scale empirical evaluation of state-of-the-art models from all model families on OmniTabBench, confirming the absence of a dominant winner. Furthermore, through a decoupled metafeature analysis, which examines individual properties such as dataset size, feature types, feature and target skewness/kurtosis, we elucidate conditions favoring specific model categories, providing clearer, more actionable guidance than prior compound-metric studies.

Dihong Jiang, Ruoqi Cao, Zhiyuan Dang +8

Huawei Cloud · School of Intelligence Science and Technology, Peking University

Tabular ML Tabular Prediction Tabular Foundation Models Gradient-Boosted Decision Trees

Ask PDF

Abstract

Tabular ML

Explore similar work

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Beyond IID: How General Are Tabular Foundation Models, Really?

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale