TabNSM: Neural Sparse Mixer for Tabular Regression

Ali Eslamian; Qiang Cheng

cs.LGAug 18, 2026

TabNSM: Neural Sparse Mixer for Tabular Regression

Authors: Ali Eslamian, Qiang Cheng

Organizations: Department of Computer Science University of Kentucky Lexington, KY 40506 · Department of Computer Science Institute for Biomedical Informatics University of Kentucky Lexington, KY 40506

Abstract

Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.

Figures & tables

Figure 1: Overview of the TabNSM architecture. TabNSM extends the published TabNSA–TabMixer representation backbone with regression-specific residual integration and a Multi-Stage Regression Head. Within ASIM, the inherited structured sparse-attention operator provides instance-adaptive feature selection, while the inherited FTM operator propagates broader feature-level context. Learnable residual blending couples these streams, and the resulting representation is passed to progressively narrowed MLP branches with residual coupling and final feature fusion to produce $\hat{y}$ .

Model	SA	CP	AQ	TO	SC	CD	ChD	EVP	RES
Best Classical (6 models)	0.175	3.719	12.816	0.027	3.602	0.236	8717	26.104	3163
Best Tree-based (5 models)	0.171	2.617	5.869	0.027	3.455	0.004	17525	2.799	3135
Best MLP-style (5 models)	0.167	2.920	4.578	0.028	3.920	0.028	4963	1.556	3161
Best Transformer (6 models)	0.156	2.901	4.081	0.028	3.667	0.007	4475	1.507	3163
Best Retrieval (4 models)	0.150	2.833	5.062	0.028	3.454	0.052	22661	2.653	3187
Best Other (8 models)	0.149	2.991	4.690	0.028	3.570	0.136	4808	3.219	3163

Table 1: RMSE on nine regression benchmarks. Lower is better. Best results are shown in bold , and second-best results are underlined . We report the best-performing baseline within each model family; full per-model results are provided in Appendix C.1 .

Panel of Figure 2 — Figure 2: Ablation and scalability analyses. (a) Full attention, Mamba, and ASIM within TabNSM; ASIM achieves the lowest RMSE. (b) Parameter count grows approximately linearly with retained feature fraction, consistent with the ASIM complexity analysis.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

	Model	SA	CP	AQ	TO	SC	CD	ChD	EVP	RES
Classical	LinearRegression	0.402	9.277	15.062	0.028	3.997	0.236	8717.544	35.156	3163
	KNN-5	0.238	3.719	12.816	0.029	3.602	1.347	15860.523	26.104	3212
	Lasso	0.402	9.277	15.064	0.028	3.964	0.251	8717.439	35.155	3163
	Ridge	0.402	9.277	15.062	0.027	3.982	0.236	8717.479	35.156	3163
	SVR-Linear	0.406	11.663	17.854	0.042	3.974	$\diamondsuit,{\dagger}$	$\diamondsuit,{\dagger}$	$\diamondsuit,{\dagger}$	$\diamondsuit,{\dagger}$
	SVR-RBF	0.175	7.491	19.416	0.038	3.964	$\ast$	$\ast$	$\ast$	$\ast$

Appendix

Table 2: Evaluation of different models on regression tasks. Results report RMSE (lower is better).

Figure 3: Critical Difference diagram (Nemenyi post-hoc test,

\alpha=0.05

). Each dot is a model coloured by architectural family; its horizontal position is its average rank across five benchmarks (lower = better). The CD bracket shows the minimum rank difference required for statistical significance (CD = 26.8). Models marked with

+

(GrowNet, SVR-Linear) fall outside TabNSM’s critical difference band and are significantly worse. The large CD is expected when

k\gg N

; the Friedman test nonetheless confirms a significant overall difference (

p=4.25\times 10^{-5}

Figure 4: Average rank across nine benchmarks for each model (lower is better), coloured by architectural family. All baselines are statistically significantly worse than TabNSM (one-sided Wilcoxon signed-rank test, Benjamini–Hochberg corrected [ Benjamini and Hochberg, 1995 ] ,

\alpha=0.05

; all corrected

p\leq 0.02

). Missing values are imputed as worst rank.

Figure 5: Per-instance feature attribution heatmap for TabNSM. Each row corresponds to a test instance and each column corresponds to an input feature. Cell intensity represents the normalized attribution score, computed using gradient–embedding attribution and scaled to $[0,1]$ within each instance. Brighter cells indicate features with stronger influence on the prediction. The sparse and instance-varying attribution patterns suggest that TabNSM adaptively focuses on different feature subsets across samples.

Dataset	Abbreviation	# Samples	# Features
Sarcos	SA	48,933	27
CPU	CP	8,192	12
Topo	TO	8,885	266
SCAN	SC	4,355	386
Crime Data	CD	1,004,991	401
U.S. Chronic Disease	ChD	309,215	405

Appendix

Table 3: Dataset Statistics

Dataset	URL
CPU	https://www.openml.org/search?type=data&status=active&id=42841
Topo	https://www.openml.org/search?type=data&status=active&id=422
Sarcos Robot Arm Data	https://www.openml.org/search?type=data&status=active&id=43873&sort=runs
Electric Vehicle Population Data	https://catalog.data.gov/dataset/electric-vehicle-population-data
U.S. Chronic Disease Data	https://catalog.data.gov/dataset/u-s-chronic-disease-indicators
Air Quality	https://catalog.data.gov/dataset/air-quality

Appendix

Table 4: Benchmark dataset sources.

Figure 6: Sensitivity analysis with respect to random seed initialization. The boxplot shows the distribution of test Root Mean Squared Error (RMSE) over 20 independent runs with different random seeds. Boxes indicate the interquartile range (IQR), the center line denotes the median, and whiskers extend to

1.5\times\mathrm{IQR}

. Lower values indicate better performance.

Dataset	$n$	$d$	CatBoost Time (s)	TabNSM Time (s)
cpu	8,192	12	1.042	0.346
topo	8,885	266	10.788	0.333
sarcos	48,933	27	2.111	0.333
EVP	264,628	392	14.986	0.346
ChD	309,215	405	17.252	0.349
CD	1,004,991	401	45.476	0.398

Appendix

Table 5: Runtime in seconds for CatBoost and TabNSM under the validation-loss threshold protocol. Lower is better. CatBoost is evaluated on CPU, whereas TabNSM uses GPU acceleration.

Dataset	CatBoost	TabNSM
EVP	N/A	477
ChD	N/A	529
CD	N/A	1621

Appendix

Table 6: Peak GPU memory usage in MB for TabNSM. CatBoost is run on CPU, so GPU memory is not applicable.

Dataset	CatBoost	TabNSM
EVP	3763.77	3006.73
ChD	4325.94	3408.70
CD	12080.61	8524.16

Appendix

Table 7: CPU memory increase in MB during training. Lower is better.

Figure 7: Training time versus sample size. Runtime is measured under the validation-loss threshold protocol. CatBoost runtime increases with dataset size, whereas TabNSM maintains low runtime across the evaluated settings, reflecting mini-batch GPU training and sparse feature interaction modeling.

Figure 8: Accuracy–efficiency trade-off for tabular learning methods. The figure summarizes predictive accuracy and computational efficiency across representative tabular learning models.

	CP	TO	SA	EVP
w/o FTM	3.966	0.034	0.176	570.09
Full TabNSM	2.842	0.021	0.137	1.199

Appendix

Table 8: Ablation of the FTM pathway. Lower RMSE is better.

Figure 9: Loss-function ablation across three datasets. RMSE is reported on a log scale. Values above bars indicate relative change with respect to GridLoss. GridLoss achieves the lowest or comparable RMSE across the evaluated datasets, while pointwise losses such as MAE and LogCosh can degrade substantially in high-error regimes.

Figure 10: Final validation metric distributions across multiple runs for models trained with and without RISE.

Panel of Figure 11 — Figure 11: Sensitivity to feature dimensionality and sample size. The top panel evaluates robustness under varying numbers of retained input features, while the bottom panel evaluates robustness under varying fractions of training samples. Separate y-axes are used to accommodate differences in error scale.

Dataset	Std	IQR	IQR/Std	Tail Type	Preferred $\tau$	RMSE– $\tau$ Behavior
CP	18.40	13.00	0.71	Heavy-tailed	Large	Monotonic decrease
TO	0.03	0.02	0.67	Rare extremes	Small	Sharp peak
SCAN-2	4.99	0.00	0.00	Degenerate bulk	Very small	Highly non-monotonic

Appendix

Table 9: Target distribution statistics and their relationship to

\tau

sensitivity.

Figure 12: Normalized RMSE as a function of

\tau

across datasets.

Figure 13: Normalized RMSE as a function of

\alpha

across datasets.

Panel of Figure 14 — Figure 14: Component-wise ablation of sparse attention parameters. Each plot shows the effect of varying one parameter: number_of_selected_blocks (top-left), selection_block_size (top-right), compress_block_size (bottom-left), and sliding_window_size (bottom-right) on the average validation RMSE across four datasets.

Explore similar work

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping
Oct 9, 2026Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev +3Tabular Regression Quantile Regression
Yandex Research · BRAInLab · HSE University
Xiaomi-TabLDM: A Tabular Foundation Model Technical Report
Sep 3, 2026Xiaomi-TabLDM Team, :, Penghui Wang +13Synthetic Data Pretraining Tabular Foundation Models
Xiaomi
TabSOM: A tabular-to-image encoding method based on self-organizing maps
Aug 13, 2026David Chushig-Muzo, María Ángeles Rodríguez de Cara, Eva Milara +3Tabular ML Self-Organizing Maps
School of Mathematical and Computational Sciences, Yachay Tech University, Ecuador

Oct 9, 2026, cs.LGJ/K move · Enter open · S save

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev+2

Yandex Research · BRAInLab · HSE University

Tabular Regression Quantile Regression Heteroscedastic Regression Robust Loss Functions Early Stopping

Open paper AskPDF arXiv

While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression.

Abstract

Explore similar work

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Xiaomi-TabLDM: A Tabular Foundation Model Technical Report

TabSOM: A tabular-to-image encoding method based on self-organizing maps