cs.LGAug 25, 2026

When Does Self-Supervised Pretraining Help Tabular Models? A Study of Label Scarcity and Missing Data

Authors: Sahand Mazrouei

Organizations: Faculty of Mathematics and Computer Science, Kharazmi University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this work, we conduct an empirical study on self-supervised learning (SSL) for tabular data. Our experiments examine limited label regimes (that is, 1%, 5%, 10%, and 20% of the labels) and highlight limitations and opportunities in current tabular SSL. Contrary to what is expected, our mask-and-recover approach shows its largest improvements for datasets that have little to no recorded numeric missingness, whereas it more often leads to degradation on datasets that are characterized by native numeric missingness (post hoc split). Random Forest achieves an average AUC of 0.9015 across the 14 tasks for the 10% labels setting, whereas SSL shows a performance of ~0.8954 (RF: 1 draw; SSL: 3-draw mean). Moreover, the gain of SSL compared to training from scratch exhibits a high variance across tasks and is not significant (1 run; Wilcoxon p = 0.626, Holm-adjusted p = 1.000, for both the 5% and 10% settings). On the other hand, we observe that SSL-pretrained models achieve a higher average AUC when MNAR and MCAR missingness were injected at test-time. In comparison to models trained from scratch, our SSL-pretrained models show an improvement of +0.0245 (MCAR) and +0.0418 AUC (MNAR) under test-time missingness (single seed), with 11 (MCAR; plus one tie) and 8 (MNAR) positive results across the 14 tasks. However, after Holm-Bonferroni correction neither improvement is significant (Holm-adjusted p = 0.118 for MCAR and 0.518 for MNAR). Finally, when comparing mask-and-recover with three other approaches, namely, simplified VIME, SCARF and SubTab implementations using an identical encoder, we observe no significant difference in performance (Holm-adjusted p = 0.459 for VIME, 1.000 for SCARF, 1.000 for SubTab).

Figures & tables

Figure 1: Mean test AUC-ROC across tasks (SD across tasks; y-axis from 0.5; clipped at 1.0).
Figure 2: Per-task Δ\Delta AUC (SSL −- Scratch) at 5% labels. Datasets with >1%>1\% native numeric missingness are “Has Missing”; the rest are “Clean.” (The 1% threshold was chosen post hoc.)
Figure 3: AUC-ROC on Task 3021 as additional MCAR missingness is injected at test time (single seed).
Figure 4: Ablation results at 10% labels across the four native-missingness tasks (missingness % in parentheses). Results are averaged over 3 pretraining seeds, each over 3 fine-tuning seeds; error bars are ±1\pm 1 sd over pretraining seeds (y-axis starts at 0.5; upper bars clipped at 1.0). “Mask-Aware” denotes using the pretext corruption mask for pretraining as input to the encoder via mm .
MethodAvg AUC @ 10% labels
Random Forest0.9015
SSL Pretrained (Ours)0.8954
Scratch (No Pretrain)0.8899
LightGBM0.8875
XGBoost0.8819
Logistic Regression0.8266
Table 1: Baseline comparison averaged across tasks at 10% labels. Classical methods are trained with one fixed seed (42; a different labeled subset than the neural models). SSL Pretrained and Scratch methods are trained with one shared pretraining (SSL only) and 3 independent fine-tuning seeds.
MethodAvg AUC @ 10%Ours −- baselineAdj. pp
Ours (mask-and-recover)0.8954——
VIME ( Yoon et al., 2020 )0.8978−0.0024-0.00240.459
SCARF ( Bahri et al., 2022 )0.8943+0.0011+0.00111.000
SubTab ( Ucar et al., 2021 )0.8913+0.0041+0.00411.000
Table 2: Comparison against simplified prior tabular SSL implementations using an identical encoder architecture. Δ\Delta is defined as Ours −- baseline; adjusted pp -values are Holm-Bonferroni corrected for a family of 3. Ours does one run for pretraining; baselines get an average of 3.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

TaskNum. missing(%)SSL 1%Scr 1%SSL 5%Scr 5%SSL 10%Scr 10%SSL 20%Scr 20%
20794.3050.5540.5600.7420.7670.7960.8070.8480.869
302122.2160.9380.8720.9340.9300.9570.9480.9700.970
149545.2580.5940.6800.7380.7590.7800.8060.7990.830
1468001.7320.6880.6850.8810.9120.9430.9560.9870.992
20.0000.7000.5940.8640.7760.9760.9230.9910.982
150.2470.9930.9880.9910.9850.9870.9890.9870.983
Appendix
Table 3: Per-task AUC-ROC results for SSL pretraining vs. scratch training across four label scarcity fractions. Metrics represent the mean over 3 random seeds and are rounded to 3 decimals for readability; statistical tests use full-precision values, available in the accompanying code. SSL values use one pretraining run which was shared, so they are different from the three seed pretraining “Full” model in the ablation (Figure 4 ).
TaskNum. missing(%)Δ\Delta 10%Δ\Delta MCARΔ\Delta Shift
20794.3051-0.0071-0.0223-0.0113
302122.21570.00850.0267-0.0185
149545.2583-0.0456-0.00000.0063
1468001.7316-0.0133-0.0203-0.0466
20.00000.11000.15510.1954
150.24730.01090.0154-0.0018
Appendix
Table 4: Robustness deltas at 10% labels (seed 42). Values represent Δ\Delta AUC (SSL −- Scratch) under normal, MCAR (+30% missingness), and MNAR structured shift. For Task 14954, the MCAR delta is −1.1×10−16-1.1\times 10^{-16} , a numerical tie that was treated as a nonzero negative difference by the Wilcoxon test, and dropping it gives raw p=0.0327p=0.0327 (Holm-adjusted ≈0.131\approx 0.131 ).
TaskDatasetntrainn_{\text{train}}ntestn_{\text{test}}FeaturesClassesLabels @ 1%
2079Eucalyptus662741956
3021Sick339437829233
14954Cylinder Bands486543724
146800MiceProtein9721087789
2Anneal808903858
15Breast-W62970926
Appendix
Table 5: The dataset sizes for fold 0 of the 10-fold split of each task on OpenML. Labels at 1% is max⁡(⌊0.01 ntrain⌋, number of classes)\max(\lfloor 0.01\,n_{\text{train}}\rfloor,\ \text{number of classes}) , so eight of the tasks have fewer than 10 labeled examples at 1%.
More in this topic

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 247 papers

Explore similar work

CardsList
May 12, 2026

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

Tabular data forms the backbone of high-stakes decision systems in finance, healthcare, and beyond. Yet industrial tabular datasets are inherently difficult: high-dimensional, riddled with missing entries, and rarely labeled at scale. While foundation models have revolutionized vision and language, tabular learning still leans on handcrafted features and lacks a general self-supervised framework. We present MaskTab, a unified pre-training framework designed specifically for industrial-scale tabular data. MaskTab encodes missing values via dedicated learnable tokens, enabling the model to distinguish structural absence from random dropout. It jointly optimizes a hybrid supervised pre-training scheme--utilizing a twin-path architecture to reconcile masked reconstruction with task-specific supervision--and an MoE-augmented loss that adaptively routes features through specialized subnetworks. On industrial-scale benchmarks, it achieves +5.04% AUC and +8.28% KS over prior art under rigorous scaling. Moreover, its representations distill effectively into lightweight models, yielding +2.55% AUC and +4.85% KS under strict latency and interpretability constraints, while improving robustness to distribution shifts. Our work demonstrates that tabular data admits a foundation-model treatment--when its structural idiosyncrasies are respected.

Bo Zheng, Yudong Chen, Zihua Xiong +4

Zhejiang University · MyBank, Ant Group

Tabular Foundation ModelsSelf-Supervised Pre-TrainingTabular ClassificationTabular MLTabular Representation Learning
AskPDF
Jun 18, 2026

When, Where, and How: Adaptive Binning for Tabular Self-Supervised Learning

Medical tabular data are ubiquitous in clinical research, but deep learning for tables remains underexplored because reliable labels often require costly expert adjudication, even though structured clinical variables are routinely available in tabular form. Self-supervised learning can leverage these unlabeled tables, and recent binning-based pretexts offer a promising inductive bias, but existing objectives fix a single global quantile discretization and apply feature-agnostic supervision. We propose Adaptive Binning, a training-adaptive discretization pretext for tabular SSL that couples discretization to learning through a feature-wise coarse-to-fine curriculum. Motivated by the spectral bias of neural networks and the principles of curriculum learning, our method progressively refines discretization per feature upon plateau detection and selects representation-aware splits to jointly improve value-space concentration and representation-space coherence. A heterogeneity-aware objective unifies categorical reconstruction with ordinal supervision for numerical features, and experiments on public medical tabular datasets under unified evaluation protocols show consistent gains for linear probing and fine-tuning without dataset-specific discretization tuning. We further introduce a medical tabular SSL benchmark with standardized protocols to support reproducible progress in this underexplored domain. Our code is available at https://github.com/labhai/Adaptive-Binning.

Daehwan Kim, Haejun Chung, Ikbeom Jang

Hanyang University, Seoul, South Korea · Hankuk University of Foreign Studies, Yongin, South Korea

Self-Supervised LearningTabular ML
AskPDF
Jun 2, 2026

AugMask: Training Diffusion Models on Incomplete Tabular Data via Stochastic Augmentation and Masking

Score-based diffusion models have emerged as prominent deep generative models; however, their application to tabular data remains challenging because their backbones assume fully specified inputs, whereas real-world tabular data often contain missing values. We propose AugMask, a plug-and-play training framework that adapts missing-unaware backbones to incomplete data by separating conditioning from supervision. AugMask 1) constructs numeric inputs via conditional stochastic augmentation using lightweight auxiliary models, and 2) applies denoising supervision only to observed coordinates. In effect, augmented missing entries serve as uncertain conditioning context rather than training targets. We connect this training rule to a Rao--Blackwellized objective and show that marginalizing missing entries yields a variance-weighted sensitivity penalty, discouraging over-reliance on uncertain completions. Across diverse datasets and missingness regimes, AugMask enables standard diffusion-based tabular generators to outperform specialized missing-aware baselines.

Jungkyu Kim, Taeyoung Park, Kibok Lee

Department of Statistics and Data Science, Yonsei University, South Korea.

Learning with Missing DataTabular Diffusion ModelsSynthetic Tabular Data GenerationDiffusion Models
AskPDF