On Data Scaling in Tabular ML: More Data, Better Generalization?

Ivan Rubachev; Nikita Lazarev; Artem Babenko

cs.LGOct 19, 2026

On Data Scaling in Tabular ML: More Data, Better Generalization?

Authors: Ivan Rubachev, Nikita Lazarev, Artem Babenko

Organizations: Yandex · HSE University

Abstract

Recent work in tabular machine learning has emphasized the importance of diverse and realistic evaluation settings. The niche of large-scale (1M+) data, however, remains under-explored. To study training-data scaling, we assemble eight real-world tabular datasets with up to around ten million training examples, retaining the natural distribution shifts of the tasks and using random-split evaluation as a control. We find that GBDTs and MLP-based models show broadly similar scaling trends, with gains from both additional training data and algorithmic improvements. Tabular foundation models show more varied scaling behavior, with recent versions benefiting more consistently from larger contexts even under distribution shift. However, scaling training data under distribution shift introduces new challenges: in-distribution and out-of-distribution performance mostly improve together at smaller scales but can diverge at larger scales. We examine this through MLP capacity sweeps and comparisons of the validation distributions used for tuning and early stopping in MLPs and GBDTs. We also compare common practices of using validation data, contrasting its use for model selection in MLP pipelines with using it for in-context fitting in tabular foundation model pipelines. Under stronger shifts, MLPs selected using validation representative of the evaluation conditions generally maintain or improve performance as training data grows. In contrast, including such examples in context improves foundation-model performance, but does not prevent deterioration with increasing training size in our experiments.

Figures & tables

Figure 1: Performance of parametric models across training-set sizes. Horizontal axis shows the number of training examples on a logarithmic scale. We report RMSE for regression and negative average precision (

-\mathrm{AP}

) for classification, so lower is better on every plot. On most tasks, additional training data improves performance across methods, improvements to architectures and training procedures also remain beneficial across scales.

Figure 2: Performance of tabular foundation models across training-set sizes. The figure follows Figure 1 conventions. Tabular foundation models also improve with in-context training set scale, but the scaling trends vary significantly across different models.

Figure 3: MLP width sweeps across training sizes. Top row: OOD test performance; bottom row: ID controls. The horizontal axis shows parameters per MLP on a logarithmic scale; darker curves indicate larger training sets. Learning rate, weight decay, and checkpoint are selected on the corresponding validation split. We report RMSE for all datasets except Aeolus Flight Delay, where we report negative average precision ( $-\mathrm{AP}$ ). Lower is better. Gains from increasing model size are limited under distribution shift, but become more pronounced with larger training sets under ID evaluation

Figure 4: Validation distribution affects ID and OOD scaling in MLP with Muon and LightGBM. Solid curves with filled circles use OOD validation (“Default val.”); dashed curves with open circles use ID validation. We report RMSE for all datasets except Aeolus Flight Delay, where we report negative average precision (

-\mathrm{AP}

). Lower is better in all panels. At larger training sizes, ID validation can improve ID performance while OOD performance deteriorates; conversely, OOD validation can improve OOD performance at the cost of ID performance.

Figure 5: Validation as context vs. as a selection signal on restricted training pools of Weather, Aeolus Flight Delay, and Cooking Time. Shades of blue (MLP) and purple (TabPFN v3.5 with appended validation) indicate validation sizes from 5 000 to 40 000. Conventions follow Figure 1 . TabPFN benefits from additional validation examples, but adding more training data can hurt performance. The MLP benefits more consistently from additional training data, but gains little from larger validation sets.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$b_{d}$	$R^{2}_{\mathrm{ind}}$	$R^{2}_{\mathrm{shared}}$	$\Delta R^{2}$
Cooking Time	0.1625	0.9938	0.9931	0.0006
Delivery ETA	0.0055	0.9727	0.9673	0.0054
Maps Routing	0.1811	0.9935	0.9929	0.0006
Weather	0.1843	0.9921	0.9904	0.0017
Jane Street	0.3403	0.9312	0.9024	0.0288
Bank Fraud	0.3373	0.9930	0.9849	0.0082

Appendix

Table 1: Dataset-shared exponents and mean in-sample

R^{2}

over methods.

\Delta R^{2}=R^{2}_{\mathrm{ind}}-R^{2}_{\mathrm{shared}}

, computed before rounding.

Method	Cooking	Delivery	Weather	Jane Street	Bank Fraud	Aeolus	Istella
XGBoost	$24.47\pm 0.03$	$34.29\pm 0.21$	$29.37\pm 0.01$	$31.26\pm 0.14$	$51.32\pm 0.25$	$7.72\pm 0.01$	$34.24\pm 0.09$
LightGBM	$55.63\pm 0.07$	$70.77\pm 0.06$	$130.24\pm 0.08$	$145.78\pm 0.76$	$66.97\pm 0.43$	$127.03\pm 0.73$	$46.28\pm 0.07$
$\mathrm{MLP}_{\mathrm{AdamW}}$	$3.73\pm 0.01$	$9.95\pm 0.07$	$7.27\pm 0.04$	$3.39\pm 0.01$	$7.62\pm 0.04$	$7.99\pm 0.07$	$14.03\pm 0.13$
$\mathrm{MLP}_{\mathrm{Muon}}$	$10.55\pm 0.06$	$14.44\pm 0.11$	$10.76\pm 0.04$	$6.67\pm 0.01$	$13.30\pm 0.02$	$19.58\pm 0.08$	$14.81\pm 0.11$
$\mathrm{MLP}^{\ddagger}_{\mathrm{AdamW}}$	$15.46\pm 0.17$	$18.99\pm 0.12$	$15.46\pm 0.05$	$7.96\pm 0.03$	$14.65\pm 0.04$	$14.76\pm 0.18$	$21.57\pm 0.22$
$\mathrm{MLP}^{\ddagger}_{\mathrm{Muon}}$	$20.36\pm 0.11$	$28.41\pm 0.40$	$24.29\pm 0.11$	$9.98\pm 0.02$	$17.20\pm 0.06$	$31.74\pm 0.17$	$34.53\pm 0.24$

Appendix

Table 2: Hyperparameter selection plus single-seed training and evaluation time on the full training splits, in hours;

\pm

denotes the sample standard deviation of evaluation time over ten seeds. TabPack’s cost includes pack training and ensemble selection. Together with the broadly similar scaling trends and competitive predictive performance, these runtimes highlight TabPack as an efficient alternative to separately tuned models at large training sizes. TabPack rows are shown in bold.

Dataset	Train (M)	Val. (K)	Test (K)	Features	Task
Maps Routing	5.12	60	60	986	R
Cooking Time	9.00	51	42	192	R
Delivery ETA	11.08	34	37	223	R
Weather	13.54	42	41	103	R
Istella	9.21	600	600	200	R
Jane Street	10.00	93	93	94	R

Appendix

Table 3: Dataset statistics (rounded). Training counts are in millions; validation and test counts are in thousands, per evaluation condition. The first four datasets are from TabReD. OOD splits are temporal except for Istella (query-based). R: regression; B: binary classification.

Figure 6: Default protocol: temporal validation and test splits, or query for Istella. Points show seed means; shaded bands show one standard deviation where multiple seeds are available.

Figure 7: IID protocol: checkpoint and hyperparameter selection on IID validation, evaluation on IID test. Points show seed means; error bars show one standard deviation where multiple seeds are available.

Figure 8: OOD test results: temporal test splits, or query for Istella. Blue curves show MLP with Muon; red curves show LightGBM. Solid curves with filled circles use default validation (temporal, or query-based for Istella); dashed curves with open circles use ID validation.

Figure 9: In-distribution test results: both validation protocols are evaluated on IID test. Colors and line styles follow Figure 8 .

Tabular ML

Tabular machine learning develops models and tools for learning from structured data arranged in rows and columns.

See all 114 papers

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models
Jul 28, 2026Malena Loza, David Chushig-Muzo, Eva Milara +3Distribution Shift Robustness Tabular Foundation Models
Colegio de Ciencias e Ingenierías, Universidad San Francisco de Quito (USFQ), Quito, Ecuador · Department of Signal Theory and Communications, Telematics and Computing Systems, Rey Juan Carlos University, Madrid, Spain · Facultad de Ingeniería, Universidad Latina de Panamá, Ciudad de Panamá, Panamá
Beyond IID: How General Are Tabular Foundation Models, Really?
Jun 29, 2026Lennart Purucker, Andrej Tschalzev, Nick Erickson +7Tabular Foundation Models OOD Generalization
1Prior Labs · University of Mannheim · 4INRIA Saclay +5
OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale
Apr 8, 2026Dihong Jiang, Ruoqi Cao, Zhiyuan Dang +8Tabular ML Tabular Prediction
Huawei Cloud · School of Intelligence Science and Technology, Peking University

Jul 28, 2026J/K move · Enter open · S save

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Malena Loza, David Chushig-Muzo, Eva Milara, Luis Bote-Curiel+2

Colegio de Ciencias e Ingenierías, Universidad San Francisco de Quito (USFQ), Quito, Ecuador · Department of Signal Theory and Communications, Telematics and Computing Systems, Rey Juan Carlos University, Madrid, Spain · Facultad de Ingeniería, Universidad Latina de Panamá, Ciudad de Panamá, Panamá

Distribution Shift Robustness Tabular Foundation Models Distribution Shift OOD Generalization

Open paper AskPDF arXiv

Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to ensemble tree-based models. Most TFMs are trained and evaluated on independent and identically distributed data, but this assumption changes in real-world scenarios due to distribution shifts, which compromise the robustness of models. Limited research has been conducted of TFMs under distribution shifts. We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine TFMs, spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX and TabFM. Three real-world datasets from the TableShift study were considered (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types. Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy, with shift gaps ranging from 0.003 to 0.060 depending on shift type. The relationship between in-distribution and OOD predictive performance documented for classical tabular models extends into TFMs. We also identified a scalability gap, as high-performing models demand significant memory and computational resources beyond what standard deployment infrastructure can support. This study extends existing benchmarks for OOD in tabular data, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts.

Abstract

Tabular ML

Explore similar work

Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

Beyond IID: How General Are Tabular Foundation Models, Really?

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale