Deep Tabular Representation Corrector

Hangting Ye; Peng Wang; Wei Fan; Xiaozhuang Song; He Zhao; Dandan Gun; Yi Chang

cs.LGMar 17, 2026

Deep Tabular Representation Corrector

Authors: Hangting Ye, Peng Wang, Wei Fan, Xiaozhuang Song, He Zhao, Dandan Gun, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University, China. · School of Computer Science, University of Auckland, New Zealand. · Chinese University of Hong Kong, Shenzhen, China. · CSIRO’s Data61 and Monash University, Australia. ·

Abstract

Tabular data have been playing a mostly important role in diverse real-world fields, such as healthcare, engineering, finance, etc. The recent success of deep learning has fostered many deep networks (e.g., Transformer, ResNet) based tabular learning methods. Generally, existing deep tabular machine learning methods are along with the two paradigms, i.e., in-learning and pre-learning. In-learning methods need to train networks from scratch or impose extra constraints to regulate the representations which nonetheless train multiple tasks simultaneously and make learning more difficult, while pre-learning methods design several pretext tasks for pre-training and then conduct task-specific fine-tuning, which however need much extra training effort with prior knowledge. In this paper, we introduce a novel deep Tabular Representation Corrector, TRC, to enhance any trained deep tabular model's representations without altering its parameters in a model-agnostic manner. Specifically, targeting the representation shift and representation redundancy that hinder prediction, we propose two tasks, i.e., (i) Tabular Representation Re-estimation, that involves training a shift estimator to calculate the inherent shift of tabular representations to subsequently mitigate it, thereby re-estimating the representations and (ii) Tabular Space Mapping, that transforms the above re-estimated representations into a light-embedding vector space via a coordinate estimator while preserves crucial predictive information to minimize redundancy. The two tasks jointly enhance the representations of deep tabular models without touching on the original models thus enjoying high efficiency. Finally, we conduct extensive experiments on state-of-the-art deep tabular machine learning models coupled with TRC on various tabular benchmarks which have shown consistent superiority.

Figures & tables

Panel of Figure 1 — Fig. 1 : The performance of deep tabular models with varying noise levels in observations. FT-Transformer, MLP, and DCN2 indicate different deep tabular models. For regression tasks, lower RMSE is better, and for classification tasks, higher accuracy is better.

Panel of Figure 2 — Fig. 2 : The relation between model performance and the corresponding SVE values of representations. For each subfigure, we conduct experiment on multiple random seeds. We find that deep tabular models with higher SVE often yield lower performance.

Fig. 3 : The framework of Tabular Representation Corrector (TRC). Subfigure (a) illustrates the overall TRC framework, subfigure (b) presents the training process for the shift estimator of TRC, and subfigure (c) presents the training process for the coordinate estimator of TRC. Here, $z_{i}=G_{f}(x_{i};\theta_{f})$ is the output of any trained backbone. $z_{i}$ would be enhanced via two tasks. In subfigure (b), the explicitly perturbed samples are only used for training the shift estimator. During the test stage, we feed the representations extracted by the existing trained deep tabular model, which are from the test dataset (notably, we do not add any noise to the test dataset), into the shift estimator followed by the coordinate estimator to achieve the calibrated representations of test samples.

	CO	DI	QS	CA	PO	SU	AD	AU	GE	YE	COV
Objects	9568	53940	908	20640	15000	21263	48842	690	9873	515345	581012
Numerical	4	6	6	8	48	81	6	14	32	90	54
Categorical	0	3	0	0	0	0	8	0	0	0	0
Classes	-	-	-	-	-	-	2	2	5	-	7
Metric	RMSE	RMSE	RMSE	RMSE	RMSE	RMSE	Accuracy	Accuracy	Accuracy	RMSE	Accuracy

TABLE I : Tabular data properties. “Objects” indicates the dataset size. RMSE denotes Root Mean Square Error for regression, Accuracy is used for binary and multiclass classification.

	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	CA $\downarrow$	PO $\downarrow$	SU $\downarrow$	AD $\uparrow$	AU $\uparrow$	GE $\uparrow$
MLP (i)	3.961	564.373	0.875	0.505	10.892	10.379	0.856	0.870	0.578
+ TRC	3.899	558.330	0.825	0.502	10.593	10.326	0.858	0.891	0.580
DCN2 (i)	4.016	591.958	1.027	0.495	8.805	10.674	0.856	0.841	0.564
+ TRC	3.858	574.453	0.879	0.493	8.790	10.584	0.857	0.862	0.584
SNN (i)	11.789	1530.293	1.018	0.896	18.517	25.498	0.847	0.848	0.550
+ TRC	6.880	693.369	0.870	0.699	11.202	15.537	0.847	0.877	0.573

TABLE II : Results of overall performance. “i” denotes in-learning and “p” denotes pre-learning for baseline models. The best results are highlighted in bold. The improvement of TRC over baselines is statistically significant at the 95% confidence level.

Panel of Figure 6 — Fig. 4 : Performance improvement of TRC over various backbone models on large scale datasets.

Backbone	TR	SC	DE	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	CA $\downarrow$	PO $\downarrow$	SU $\downarrow$	AD $\uparrow$	AU $\uparrow$	GE $\uparrow$
MLP				3.961	564.373	0.875	0.505	10.892	10.379	0.856	0.87	0.578
	✓			3.916	563.649	0.868	0.502	10.605	10.328	0.858	0.87	0.571
		✓		3.92	564.141	0.845	0.505	10.631	10.349	0.858	0.862	0.568
	✓	✓		3.908	558.52	0.852	0.502	10.602	10.341	0.858	0.87	0.575
		✓	✓	3.922	565.129	0.841	0.505	10.622	10.363	0.858	0.877	0.577
	✓	✓	✓	3.899	558.33	0.825	0.502	10.593	10.326	0.858	0.891	0.58

TABLE III : Analysis on the effects of different components of TRC . The best results are highlighted in bold, and the second best results are underscored.

	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	CA $\downarrow$	PO $\downarrow$	SU $\downarrow$	AD $\uparrow$	AU $\uparrow$	GE $\uparrow$
MLP	3.961	564.373	0.875	0.505	10.892	10.379	0.856	0.870	0.578
deeper MLP	3.942	562.888	0.871	0.510	11.583	10.471	0.856	0.862	0.572
MLP+ TRC	3.899	558.330	0.825	0.502	10.593	10.326	0.858	0.891	0.580
FT-Transformer	3.709	551.190	0.823	0.469	2.919	10.410	0.858	0.848	0.611
deeper FT-Transformer	3.704	559.363	0.816	0.474	2.890	10.442	0.851	0.856	0.613
FT-Transformer+ TRC	3.648	543.653	0.780	0.462	2.728	10.223	0.862	0.884	0.624

TABLE IV : Performance of backbone w/ TRC and deeper backbone. The depths of backbone layers are increased only for “deeper backbone”.

	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	AU $\uparrow$
MLP (i)	3.961	564.373	0.875	0.870
+Linear Head	3.952	563.945	0.870	0.870
+Linear Head & Backbone	3.937	563.304	0.880	0.862
+MLP Head	3.950	562.076	0.862	0.870
+MLP Head & Backbone	3.934	562.427	0.857	0.870
+LoRA	3.941	563.314	0.875	0.870

TABLE V : Performance comparison between various fine-tuning paradigms and TRC. “i” denotes in-learning and “p” denotes pre-learning for baseline models.

Panel of Figure 10 — Fig. 5 : Comparison of the $L_{1}$ norm of the shift information estimated by TRC on data samples with different gradients. For each data sample, the gradient is computed by $\|\nabla_{\theta_{f}}\mathcal{L}(x,y;\theta)\|_{1}$ , where $\theta$ is the parameters of the trained deep tabular model. We partition the $L_{1}$ norm of gradients into 10 equally spaced intervals from low to high and count the number of samples in each interval.

Panel of Figure 11 — Fig. 6 : SVE and performance changes of backbone models w/ and w/o TRC on different datasets. Subfigures (a) and (c) show the Singular Value Entropy (SVE) (Eq. 3 ) of representation space w/ and w/o TRC on different datasets. Subfigures (b) and (d) show the corresponding performance.

	CO $\downarrow$		DI $\downarrow$		AU $\uparrow$
	Train	Val	Train	Val	Train	Val
MLP	3.961	3.968	564.373	565.616	0.870	0.870
+ TRC	3.902	3.899	559.020	558.330	0.891	0.891
FT-Transformer	3.709	3.726	551.190	550.228	0.848	0.848
+ TRC	3.647	3.648	544.570	543.653	0.877	0.884

TABLE VI : The influence of using a small portion (1%) of validation set for training. “Train” indicates only training data are used for training model, “Val” indicates additional small portion of validation data are used for training.

Panel of Figure 13 — Fig. 7 : The results of TRC training with missing values and fewer samples. The left two subfigures indicate the scenarios with missing values, while the right two indicate the scenarios where the number of training samples is reduced.

Panel of Figure 14 — Fig. 8 : The TSNE visualization of learned representations of deep tabular model, deep tabular model w/ FT (fine-tuning), deep tabular model w/ Task 1 (Tabular Representation Re-estimation) and deep tabular model w/ Task 1, 2 (Tabular Representation Re-estimation and Tabular Space Mapping) (i.e., TRC). Here, the backbone model is ResNet. The first row represents the CA dataset (regression), while the second row represents the GE dataset (classification). Different colors indicate different labels. Marker=“x” indicates embedding vectors.

Panel of Figure 15 — Fig. 9 : The comparison of parameters between TRC and different deep tabular models. The parameters for TRC do not include the parameters for backbone models.

Fig. 10 : Comparison with different approximated optimal representation selection strategies across a wide range of threshold

\tau

on CO dataset

\downarrow

Fig. 11 : The performance of TRC with varying number of embedding vectors on CO dataset

\downarrow

	DI $\downarrow$	QS $\downarrow$	PO $\downarrow$	AU $\uparrow$
FT-Transformer+ TRC	543.653	0.780	2.728	0.884
CatBoost [ 9 ]	546.827	0.817	4.641	0.855
XGBoost [ 8 ]	559.680	0.820	4.936	0.862

TABLE VII : Comparison with tree-based methods.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Symbols	Description
$x,y$	Observed sample and the corresponding label.
$N$	Number of observed samples in a dataset.
$\mathcal{D}_{train}$ , $\mathcal{D}_{val}$ , $\mathcal{D}_{test}$	Training set, validation set, test set.
$F(\cdot;\theta)$	Deep tabular model.
$G_{f}(\cdot;\theta_{f})$	Existing deep tabular backbone.
$G_{h}(\cdot;\theta_{h})$	Prediction head.

Appendix

TABLE VIII : Notation.

	TR	SC	DE	CA $\downarrow$	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	SU $\downarrow$	PO $\downarrow$	AD $\uparrow$	AU $\uparrow$	GE $\uparrow$
MLP				0.505	3.961	564.373	0.875	10.379	10.892	0.856	0.87	0.578
	✓			0.502	3.916	563.649	0.868	10.328	10.605	0.858	0.87	0.571
		✓		0.505	3.92	564.141	0.845	10.349	10.631	0.858	0.862	0.568
	✓	✓		0.502	3.908	558.52	0.852	10.341	10.602	0.858	0.87	0.575
		✓	✓	0.505	3.922	565.129	0.841	10.363	10.622	0.858	0.877	0.577
	✓	✓	✓	0.502	3.899	558.33	0.825	10.326	10.593	0.858	0.891	0.58

Appendix

TABLE IX : Analysis on the effects of different components of TRC . The best results are highlighted in bold, and the second best results are underscored. This table serves as an extension of Table III in manuscript.

	CO $\downarrow$	DI $\downarrow$	QS $\downarrow$	CA $\downarrow$	PO $\downarrow$	SU $\downarrow$	AD $\uparrow$	AU $\uparrow$	GE $\uparrow$
MLP	3.961	564.373	0.875	0.505	10.892	10.379	0.856	0.870	0.578
deeper MLP	3.922	562.888	0.851	0.510	11.583	10.271	0.856	0.862	0.572
MLP+ TRC	3.899	558.330	0.825	0.502	10.593	10.326	0.858	0.891	0.580
DCN2	4.016	591.958	1.027	0.495	8.805	10.674	0.856	0.841	0.564
deeper DCN2	3.872	575.765	0.904	0.486	9.165	10.675	0.858	0.862	0.575
DCN2+ TRC	3.858	574.453	0.879	0.493	8.790	10.584	0.857	0.862	0.584

Appendix

TABLE X : Performance of backbone w/ TRC and deeper backbone. The depths of backbone layers are increased only for “deeper backbone”. This table serves as an extension of Table IV in manuscript.

Panel of Figure 22 — Fig. 12 : The results of TRC training with missing values. This figure serves as an extension of Fig. 7 in manuscript.

Panel of Figure 23 — Fig. 13 : The results of TRC training with fewer samples. This figure serves as an extension of Fig. 7 in manuscript.

Panel of Figure 24 — Fig. 14 : The comparison of parameters between TRC and deep tabular models. This figure serves as an extension of Fig. 9 in manuscript.

	CO	DI	QS	CA	PO	SU	AD	AU	GE	YE	COV
MLP	0.129	0.719	0.013	0.267	0.189	0.285	0.566	0.011	0.123	9.259	9,442
TRC	0.171	0.878	0.020	0.348	0.248	0.372	0.688	0.017	0.167	13.086	13.818
ResNet	0.188	1.092	0.020	0.399	0.282	0.424	0.864	0.016	0.184	15.285	13.948
TRC	0.168	0.880	0.021	0.354	0.249	0.374	0.698	0.018	0.167	13.629	12.909
FT-Transformer	0.424	2.348	0.045	0.917	0.685	1.397	1.857	0.038	0.431	68.678	52.186
TRC	0.174	0.891	0.027	0.356	0.259	0.382	0.705	0.024	0.174	13.200	13.194

Appendix

TABLE XI : Time cost (seconds) per training epoch on the full training set. The training time for TRC does not include the training cost for backbone model.

	CO	DI	QS	CA	PO	SU	AD	AU	GE	YE	COV
MLP	0.007	0.053	0.003	0.015	0.012	0.013	0.081	0.003	0.009	0.177	0.397
TRC	0.006	0.032	0.002	0.013	0.011	0.011	0.053	0.002	0.006	0.151	0.297
ResNet	0.016	0.111	0.006	0.042	0.030	0.034	0.147	0.006	0.020	0.433	0.970
TRC	0.005	0.033	0.002	0.015	0.010	0.012	0.043	0.002	0.006	0.154	0.341
FT-Transformer	0.042	0.275	0.012	0.130	0.131	0.180	0.512	0.012	0.083	2.704	4.600
TRC	0.005	0.030	0.001	0.014	0.010	0.012	0.051	0.001	0.007	0.142	0.318

Appendix

TABLE XII : Inference time cost (seconds) on the full test set. The inference time for TRC does not include the inference time for backbone model.

Panel of Figure 27 — Fig. 15 : The TSNE visualization of coefficients for embedding vectors.

Panel of Figure 28 — Fig. 16 : The visualization of the similarity matrix for embedding vectors on FT-Transformer.

Panel of Figure 29 — Fig. 17 : The visualization of coefficients for embedding vectors.

Panel of Figure 30 — Fig. 18 : Increasing the number of embedding vectors leads to larger SVE values of representations.

Panel of Figure 31 — Fig. 19 : The performance of TRC with varying weights of the orthogonality loss.

Panel of Figure 32 — Fig. 20 : The performance of TRC with varying times of perturbing the observation in Inherent Shift Learning.

Panel of Figure 33 — Fig. 21 : The performance of TRC with the varying proportion of the selected optimal representation over all of the validation set samples, i.e., $\tau$ . This figure serves as an extension of Fig. 10 in manuscript.

Panel of Figure 34 — Fig. 22 : The performance of TRC with varying number of embedding vectors. This figure serves as an extension of Fig. 11 in manuscript.

Backbone	Discrepancy ( $\downarrow$ )	AD	CA	CO	GE
MLP	w/o Shift Estimator	0.647	0.487	0.450	0.464
MLP	w/ Shift Estimator	0.578	0.395	0.246	0.434
FT-Transformer	w/o Shift Estimator	2.217	2.460	2.352	3.870
FT-Transformer	w/ Shift Estimator	1.068	0.887	0.820	1.443

Appendix

TABLE XIII : The L2 distance between sub-optimal representations and optimal representations w/o and w/ shift estimator. “w/o Shift Estimator” corresponds to the distance between

z_{i}^{\text{light}}

and

z_{i}^{\text{heavy}}

, “w/ Shift Estimator” corresponds to the distance between

\hat{z}_{i}

and

z_{i}^{\text{heavy}}

Datasets	Backbone training epochs	DCN2	+ TRC (Lowest, ours)	+TRC (Random)	+TRC (Highest)
CO $\downarrow$	5%	4.57	4.498	4.512	4.503
CO $\downarrow$	50%	4.384	4.165	4.285	4.325
GE $\uparrow$	5%	0.508	0.527	0.525	0.524
GE $\uparrow$	50%	0.543	0.561	0.55	0.548

Appendix

TABLE XIV : The performance of TRC on underfitted backbone.

Tabular ML

Tabular machine learning develops models and tools for learning from structured data arranged in rows and columns.

See all 115 papers

Apr 23, 2026

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks

Tabular foundation models aim to learn universal representations of tabular data that transfer across tasks and domains, enabling applications such as table retrieval, semantic search and table-based prediction. Despite the growing number of such models, it remains unclear which approach works best in practice, as existing methods are often evaluated under task-specific settings that make direct comparison difficult. To address this, we introduce TEmBed, the Tabular Embedding Test Bed, a unified benchmark for systematically evaluating tabular embeddings across four representation levels: cell, row, column, and table. Evaluating a diverse set of tabular representation learning models, we show that which model to use depends on the task and representation level. Our results offer practical guidance for selecting tabular embeddings in real-world applications and lay the groundwork for developing more general-purpose tabular representation models.

Liane Vogel, Kavitha Srinivas, Niharika D'Souza +3

Technical University of Darmstadt · IBM Research, USA

Tabular Foundation Models Representation Learning Tabular Representation Learning

Ask PDF

Oct 9, 2026

TabMF: Learning Dataset Representations with Tabular Foundation Models

Representations of supervised datasets support higher-level machine-learning tasks such as dataset retrieval and distribution-shift monitoring. However, existing approaches typically rely either on hand-crafted meta-features or on dataset encoders trained specifically for representation learning. We introduce TabMF, a framework that repurposes pretrained tabular foundation models (TFMs) as encoders of supervised datasets. TabMF equips a predictive TFM with an architecture-aware dataset-level readout and adapts the model end-to-end using contrastive post-training on stochastic views of synthetic supervised datasets. We instantiate TabMF on TabICLv2 and TabPFN-2.5 and evaluate the resulting representations across six tasks on unseen real datasets. Simple frozen TFM readouts yield competitive representations, indicating that predictive pretraining already induces substantial dataset-level structure. The two TabMF encoders obtain the strongest aggregate performance among the evaluated learned and classical dataset representations, with particularly strong results in dataset retrieval, compact benchmark construction, and distribution-shift severity assessment.

Rafiq Kamel, Nicolas Atienza, Aziz Fouché +2

University of Technology Nuremberg (UTN) · Neuralk

Representation Learning Tabular Foundation Models Information Retrieval

Ask PDF

May 6, 2026

TabEmbed: Benchmarking and Learning Generalist Embeddings for Tabular Understanding

Foundation models have established unified representations for natural language processing, yet this paradigm remains largely unexplored for tabular data. Existing methods face fundamental limitations: LLM-based approaches lack retrieval-compatible vector outputs, whereas text embedding models often fail to capture tabular structure and numerical semantics. To bridge this gap, we first introduce the Tabular Embedding Benchmark (TabBench), a comprehensive suite designed to evaluate the tabular understanding capability of embedding models. We then propose TabEmbed, the first generalist embedding model that unifies tabular classification and retrieval within a shared embedding space. By reformulating diverse tabular tasks as semantic matching problems, TabEmbed leverages large-scale contrastive learning with positive-aware hard negative mining to discern fine-grained structural and numerical nuances. Experimental results on TabBench demonstrate that TabEmbed significantly outperforms state-of-the-art text embedding models, establishing a new baseline for universal tabular representation learning. Code and datasets are publicly available at https://github.com/qiangminjie27/TabEmbed and https://huggingface.co/datasets/qiangminjie27/TabBench.

Minjie Qiang, Mingming Zhang, Xiaoyi Bao +5

Natural Language Processing Lab, Soochow University, Suzhou, China · Ant Group, Hangzhou, China · The Hong Kong Polytechnic University, Hong Kong, China

Tabular Foundation Models Tabular ML Tabular Representation Learning Information Retrieval Table Retrieval

Ask PDF

Abstract

Tabular ML

Explore similar work

Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks

TabMF: Learning Dataset Representations with Tabular Foundation Models

TabEmbed: Benchmarking and Learning Generalist Embeddings for Tabular Understanding