When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

Sahand Mazrouei

cs.LGAug 25, 2026

When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

Authors: Sahand Mazrouei

Organizations: Faculty of Mathematics and Computer Science, Kharazmi University

Abstract

In this work, we conduct an empirical study on self-supervised learning (SSL) for tabular data. Our experiments examine limited label regimes (that is, 1%, 5%, 10%, and 20% of the labels) and highlight limitations and opportunities in current tabular SSL. Contrary to what is expected, our mask-and-recover approach shows its largest improvements for datasets that have little to no recorded numeric missingness, whereas it more often leads to degradation on datasets that are characterized by native numeric missingness (post hoc split). Random Forest achieves an average AUC of 0.9015 across the 14 tasks for the 10% labels setting, whereas SSL shows a performance of ~0.8954 (RF: 1 draw; SSL: 3-draw mean). Moreover, the gain of SSL compared to training from scratch exhibits a high variance across tasks and is not significant (1 run; Wilcoxon p = 0.626, Holm-adjusted p = 1.000, for both the 5% and 10% settings). On the other hand, we observe that SSL-pretrained models achieve a higher average AUC when MNAR and MCAR missingness were injected at test-time. In comparison to models trained from scratch, our SSL-pretrained models show an improvement of +0.0245 (MCAR) and +0.0418 AUC (MNAR) under test-time missingness (single seed), with 11 (MCAR; plus one tie) and 8 (MNAR) positive results across the 14 tasks. However, after Holm-Bonferroni correction neither improvement is significant (Holm-adjusted p = 0.118 for MCAR and 0.518 for MNAR). Finally, when comparing mask-and-recover with three other approaches, namely, simplified VIME, SCARF and SubTab implementations using an identical encoder, we observe no significant difference in performance (Holm-adjusted p = 0.459 for VIME, 1.000 for SCARF, 1.000 for SubTab).

Figures & tables

Figure 1: Mean test AUC-ROC across tasks (SD across tasks; y-axis from 0.5; clipped at 1.0).

Figure 2: Per-task

\Delta

AUC (SSL

-

Scratch) at 5% labels. Datasets with

>1\%

native numeric missingness are “Has Missing”; the rest are “Clean.” (The 1% threshold was chosen post hoc.)

Figure 3: AUC-ROC on Task 3021 as additional MCAR missingness is injected at test time (single seed).

Figure 4: Ablation results at 10% labels across the four native-missingness tasks (missingness % in parentheses). Results are averaged over 3 pretraining seeds, each over 3 fine-tuning seeds; error bars are

\pm 1

sd over pretraining seeds (y-axis starts at 0.5; upper bars clipped at 1.0). “Mask-Aware” denotes using the pretext corruption mask for pretraining as input to the encoder via

m

Method	Avg AUC @ 10% labels
Random Forest	0.9015
SSL Pretrained (Ours)	0.8954
Scratch (No Pretrain)	0.8899
LightGBM	0.8875
XGBoost	0.8819
Logistic Regression	0.8266

Table 1: Baseline comparison averaged across tasks at 10% labels. Classical methods are trained with one fixed seed (42; a different labeled subset than the neural models). SSL Pretrained and Scratch methods are trained with one shared pretraining (SSL only) and 3 independent fine-tuning seeds.

Method	Avg AUC @ 10%	Ours $-$ baseline	Adj. $p$
Ours (mask-and-recover)	0.8954	—	—
VIME ( Yoon et al., 2020 )	0.8978	$-0.0024$	0.459
SCARF ( Bahri et al., 2022 )	0.8943	$+0.0011$	1.000
SubTab ( Ucar et al., 2021 )	0.8913	$+0.0041$	1.000

Table 2: Comparison against simplified prior tabular SSL implementations using an identical encoder architecture.

\Delta

is defined as Ours

-

baseline; adjusted

p

-values are Holm-Bonferroni corrected for a family of 3. Ours does one run for pretraining; baselines get an average of 3.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Num. missing(%)	SSL 1%	Scr 1%	SSL 5%	Scr 5%	SSL 10%	Scr 10%	SSL 20%	Scr 20%
2079	4.305	0.554	0.560	0.742	0.767	0.796	0.807	0.848	0.869
3021	22.216	0.938	0.872	0.934	0.930	0.957	0.948	0.970	0.970
14954	5.258	0.594	0.680	0.738	0.759	0.780	0.806	0.799	0.830
146800	1.732	0.688	0.685	0.881	0.912	0.943	0.956	0.987	0.992
2	0.000	0.700	0.594	0.864	0.776	0.976	0.923	0.991	0.982
15	0.247	0.993	0.988	0.991	0.985	0.987	0.989	0.987	0.983

Appendix

Table 3: Per-task AUC-ROC results for SSL pretraining vs. scratch training across four label scarcity fractions. Metrics represent the mean over 3 random seeds and are rounded to 3 decimals for readability; statistical tests use full-precision values, available in the accompanying code. SSL values use one pretraining run which was shared, so they are different from the three seed pretraining “Full” model in the ablation (Figure 4 ).

Task	Num. missing(%)	$\Delta$ 10%	$\Delta$ MCAR	$\Delta$ Shift
2079	4.3051	-0.0071	-0.0223	-0.0113
3021	22.2157	0.0085	0.0267	-0.0185
14954	5.2583	-0.0456	-0.0000	0.0063
146800	1.7316	-0.0133	-0.0203	-0.0466
2	0.0000	0.1100	0.1551	0.1954
15	0.2473	0.0109	0.0154	-0.0018

Appendix

Table 4: Robustness deltas at 10% labels (seed 42). Values represent

\Delta

AUC (SSL

-

Scratch) under normal, MCAR (+30% missingness), and MNAR structured shift. For Task 14954, the MCAR delta is

-1.1\times 10^{-16}

, a numerical tie that was treated as a nonzero negative difference by the Wilcoxon test, and dropping it gives raw

p=0.0327

(Holm-adjusted

\approx 0.131

Task	Dataset	$n_{\text{train}}$	$n_{\text{test}}$	Features	Classes	Labels @ 1%
2079	Eucalyptus	662	74	19	5	6
3021	Sick	3394	378	29	2	33
14954	Cylinder Bands	486	54	37	2	4
146800	MiceProtein	972	108	77	8	9
2	Anneal	808	90	38	5	8
15	Breast-W	629	70	9	2	6

Appendix

Table 5: The dataset sizes for fold 0 of the 10-fold split of each task on OpenML. Labels at 1% is

\max(\lfloor 0.01\,n_{\text{train}}\rfloor,\ \text{number of classes})

, so eight of the tasks have fewer than 10 labeled examples at 1%.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 250 papers

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification
May 12, 2026Bo Zheng, Yudong Chen, Zihua Xiong +4Tabular Foundation Models Self-Supervised Pre-Training
Zhejiang University · MyBank, Ant Group
When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning
Jun 18, 2026Daehwan Kim, Haejun Chung, Ikbeom JangSelf-Supervised Learning Tabular ML
Hanyang University, Seoul, South Korea · Hankuk University of Foreign Studies, Yongin, South Korea
AugMask: Training Diffusion Models on Incomplete Tabular Data via Stochastic Augmentation and Masking
Jun 2, 2026Jungkyu Kim, Taeyoung Park, Kibok LeeLearning with Missing Data Tabular Diffusion Models
Department of Statistics and Data Science, Yonsei University, South Korea.

May 12, 2026J/K move · Enter open · S save

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

Bo Zheng, Yudong Chen, Zihua Xiong, Shuai Fang+3

Zhejiang University · MyBank, Ant Group

Tabular Foundation Models Self-Supervised Pre-Training Tabular Classification Tabular ML Tabular Representation Learning

Open paper AskPDF arXiv

Tabular data forms the backbone of high-stakes decision systems in finance, healthcare, and beyond. Yet industrial tabular datasets are inherently difficult: high-dimensional, riddled with missing entries, and rarely labeled at scale. While foundation models have revolutionized vision and language, tabular learning still leans on handcrafted features and lacks a general self-supervised framework. We present MaskTab, a unified pre-training framework designed specifically for industrial-scale tabular data. MaskTab encodes missing values via dedicated learnable tokens, enabling the model to distinguish structural absence from random dropout. It jointly optimizes a hybrid supervised pre-training scheme--utilizing a twin-path architecture to reconcile masked reconstruction with task-specific supervision--and an MoE-augmented loss that adaptively routes features through specialized subnetworks. On industrial-scale benchmarks, it achieves +5.04% AUC and +8.28% KS over prior art under rigorous scaling. Moreover, its representations distill effectively into lightweight models, yielding +2.55% AUC and +4.85% KS under strict latency and interpretability constraints, while improving robustness to distribution shifts. Our work demonstrates that tabular data admits a foundation-model treatment--when its structural idiosyncrasies are respected.

Abstract

Self-Supervised Pre-Training

Explore similar work

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning

AugMask: Training Diffusion Models on Incomplete Tabular Data via Stochastic Augmentation and Masking