Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Danil Rudenko; Georgii Bulgakov; Nikolay Kartashev; Ivan Rubachev; Daniil Krasylnikov; Artem Babenko

cs.LGOct 9, 2026

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Authors: Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev, Daniil Krasylnikov, Artem Babenko

Organizations: Yandex Research · BRAInLab · HSE University

Abstract

While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression.

Figures & tables

Figure 1: Performance comparison of different objective functions using an MLP backbone across 41 BeyondArena datasets. ( Left ) Average rank performance across all datasets. ( Right ) Relative RMSE improvement over the standard MSE baseline, visualizing the magnitude of performance gains or degradation. The box plots visualize the 10th, 25th, 50th, 75th, and 90th percentiles of the jitter plots. Notation details are provided in the beginning of Section .

	AdamW		Muon
Objective	Gain (%)	W / T / L	Gain (%)	W / T / L
MSE	—	—	+1.50	18 / 19 / 4
HL-Gaussian	+0.78	19 / 19 / 3	+2.34	26 / 11 / 4
quantile loss	+0.67	15 / 23 / 3	+2.18	24 / 13 / 4

Table 1: Objective and optimizer comparison on MLP across 41 datasets. Gain is the median relative RMSE improvement over the common MSE

+

AdamW baseline; positive values indicate improvement. W/T/L counts wins, ties, and losses under the standard-deviation-based comparison rule in Appendix . Bold denotes the largest median gain within each optimizer.

	HL-Gaussian		quantile loss
Backbone	Gain (%)	W / T / L	Gain (%)	W / T / L
MLP	+0.65	19 / 17 / 5	+0.55	21 / 15 / 5
TabPack	+0.06	8 / 20 / 13	+0.46	18 / 21 / 2
RealMLP	+0.12	13 / 17 / 11	+0.40	20 / 16 / 5
TabM	+0.54	17 / 19 / 5	+0.58	20 / 16 / 5

Table 2: Quantile loss yields consistent aggregate improvements across the tested backbones. Each objective is compared against MSE on the same backbone. Gain is the median relative RMSE improvement; positive values indicate improvement. W/T/L follows the comparison rule in Appendix . Bold denotes the larger median gain within each backbone, with both values highlighted when equal at the reported precision.

Figure 2: In this synthetic experiment, heteroskedastic noise creates a mismatch between the stopping epochs preferred by low- and high-uncertainty regions, while AdaQR mitigates this effect. Uncertainty plots [ 11 ] for an MLP trained with (a) quantile loss and (b) AdaQR on the heteroskedastic dataset, and (c) quantile loss on the homoskedastic counterpart. Each curve shows the smoothed point-wise difference in squared test error relative to the reference epoch with the best aggregate validation MSE; negative values indicate improvement. Test samples are ordered by increasing uncertainty in the heteroskedastic dataset, with the same ordering used in all panels. Under heteroskedastic noise, earlier epochs perform relatively better on high-uncertainty points and later epochs on low-uncertainty points, indicating that different uncertainty regions favor different stopping times. This separation is reduced with AdaQR and disappears when the noise is homoskedastic.

Seven largest benchmark datasets
	$K=5$		$K=10$
Dataset / summary	U/G	R/G	U/G	R/G
Synthetic (heteroskedastic)	$+0.318$	$-0.041$	$+0.286$	$-0.118$
Synthetic (homoskedastic)	$-0.019$	$-0.021$	$-0.048$	$-0.078$
Median Improv. (%)	$+0.180$	$-0.029$	$+0.114$	$-0.003$
W / T / L	3/4/0	0/7/0	3/4/0	0/7/0

Table 3: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss. Improv. (%) is the relative reduction in mean test RMSE using uncertainty (U/G) or random (R/G) cohorts compared with global selection. All checkpoints are selected on validation data. Bottom rows summarize available comparisons among the seven largest benchmark datasets, excluding synthetic data; W/T/L follows Appendix . Results are computed from mean test RMSEs over 15 random seeds.

MLP
Objective	Gain (%)	W / T / L
NLL	+0.02	9 / 12 / 7
$\beta$ -NLL	+0.25	10 / 11 / 7
quantile loss	+0.47	16 / 10 / 2
AdaQR	+0.66	16 / 10 / 2

Table 4: Predictive performance on 28 BeyondArena datasets with at least 1,000 training samples. Gain is the median relative RMSE improvement over MSE on the same backbone; positive values indicate improvement. W/T/L summarizes comparisons with MSE using the rule in Appendix . Bold denotes the largest median gain within each backbone. AdaQR † includes auxiliary scale regularization (Appendix ).

Objective	Gain (%)	W / T / L
quantile loss	+0.35	15 / 11 / 2
AdaQR †	+0.82	19 / 6 / 3
Global $\sigma_{l}$	+0.14	11 / 10 / 7

Table 5: Ablation of adaptive scale prediction on RealMLP. Gain is the median relative RMSE improvement over MSE. W/T/L follows the comparison rule in Appendix .

Figure 3: AdaQR versus CatBoost data uncertainty on three real-world datasets from the Beyond benchmark (test splits; logarithmic axes). AdaQR estimates are averaged across TabM branches after conversion to variances using ( ). Dashed lines indicate $y=x$ ; $\rho_{\mathrm{S}}$ denotes Spearman’s rank correlation computed over all test inputs for each dataset. At most 8,000 randomly sampled inputs are displayed per dataset.

	$K=5$		$K=10$
Summary	quantile loss	AdaQR	quantile loss	AdaQR
Median Improv. (%)	$+0.180$	$+0.092$	$+0.114$	$+0.085$
W / T / L	3/4/ 0	1/6/0	3/4/0	2/5/0

Table 6: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss (QR) or AdaQR. Improv. (%) is the relative reduction in the mean test RMSE compared with global selection under the same objective. Results summarize seven real-world datasets, using 15 random seeds. All checkpoints are selected on validation data; W/T/L follows Appendix .

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Distribution
# layers	UniformInt[1, 6]
Width (hidden size)	UniformInt[64, 1024]
Dropout rate	{0.0, Uniform[0.0, 0.5]}
Learning rate (Muon)	LogUniform[1e-4, 3e-2]
Learning rate (AdamW)	(A) LogUniform[3e-5, 1e-3], (B) LogUniform[3e-6, 5e-4]
Weight decay (AdamW)	LogUniform[5e-3, 5]

Appendix

Table 7: The hyperparameter tuning space for MLP. Here, (A) is the default space and (B) is used when evaluating Decoder[U] and Decoder[N] (see Section ).

Parameter	Distribution
# layers	UniformInt[1, 4]
Width (hidden size)	UniformInt[64, 1024]
Dropout rate	{0.0, Uniform[0.0, 0.5]}
Learning rate (Muon)	LogUniform[1e-3, 1e-1]
Learning rate (AdamW)	LogUniform[5e-5, 3e-3]
Weight decay (AdamW)	LogUniform[1e-3, 1]

Appendix

Table 8: The hyperparameter tuning space for TabM.

Parameter	Distribution
# layers	UniformInt[1, 3]
Width (hidden size)	384
Dropout rate	{0.0, Uniform[0.0, 0.5]}
Learning rate (Muon)	LogUniform[1e-3, 1e-1]
Learning rate (AdamW)	LogUniform[1e-4, 5e-3]
Weight decay (AdamW)	LogUniform[1e-3, 1]

Appendix

Table 9: The sampling space for TabPack

Objective	Parameter	Distribution
MSE	—	—
MAE	—	—
Huber	delta	LogUniform[1e-2, 100]
NLL	—	—
$\beta$ -NLL	beta	Uniform[0, 1]
MDN	mixture_size	UniformInt[2, 64]

Appendix

Table 10: Hyperparameter tuning space for different regression objectives. The notation follows Section ; refer to the source code for any naming ambiguities. For TabPack, the numbers of bins and quantiles are fixed at 384, and AdaQR’s regulizer is fixed at 0.2 (Appendix ).

Figure 4: Data uncertainty estimated by AdaQR versus the true data uncertainty on a heteroskedastic synthetic dataset, shown on logarithmic axes. Both uncertainties are expressed as variances. AdaQR estimates are obtained using ( ), following the theoretical results in Section . Panels compare (a) TabM, (b) RealMLP, and (c) RealMLP with scale-regularized AdaQR † . Gray points show individual test inputs for one ensemble member; the orange line connects this member’s median estimates in equal-count bins of true uncertainty. Each blue line shows the corresponding bin medians for another ensemble member. The shaded band spans the minimum and maximum of these medians, rather than a confidence interval. Dashed diagonals indicate perfect agreement; downward triangles mark estimates below the displayed range. All three models capture relative variation in data uncertainty. Scale regularization reduces RealMLP’s deviations at low uncertainty and improves consistency across ensemble members.

Panel of Figure 5 — Figure 5: AdaQR versus CatBoost data uncertainty on nine additional real-world datasets. Points represent test inputs; AdaQR estimates are averaged across TabM branches after conversion to variances. Both axes are logarithmic, and dashed lines indicate equal estimates ( $y=x$ ). $\rho_{\mathrm{S}}$ denotes Spearman’s rank correlation, computed over all test inputs; at most 8,000 points are displayed per dataset.

Model	Objective	Improv. (%)	W / T / L
MLP	Quantile Loss	+0.55	21 / 15 / 5
	AdaQR	+0.74	19 / 17 / 5
	NLL	+0.25	10 / 23 / 8
	beta-NLL	+0.32	12 / 21 / 8
TabPack	Quantile Loss	+0.46	18 / 21 / 2
	AdaQR	+0.45	19 / 21 / 1

Appendix

Table 11: Improv. (%) denotes the median relative improvement in RMSE compared to the standard MSE objective. W/T/L summarizes comparisons with MSE using the rule in Appendix .

Electric Motor Temperature
Objective	Cohorts	Global	$K=5$	$K=10$
Quantile	Uncertainty	$2.64\pm 0.13$	$2.65\pm 0.13$	$2.70\pm 0.22$
Quantile	Random	$2.64\pm 0.13$	$2.64\pm 0.14$	$2.64\pm 0.13$
AdaQR	Uncertainty	$2.74\pm 0.16$	$2.72\pm 0.15$	$2.73\pm 0.14$
Mercari
Quantile	Uncertainty	$0.47298\pm 0.00251$	$0.47282\pm 0.00181$	$0.47296\pm 0.00188$

Appendix

Table 12: Cohort-based vs. global checkpoint selection for an MLP trained with quantile loss or AdaQR on the seven largest BeyondArena datasets within our 41-dataset evaluation subset. Values are mean

\pm

standard deviation of test RMSE over 15 random seeds; lower is better. Global denotes a single checkpoint shared by all test points;

K=5

and

K=10

denote per-cohort selection with five and ten cohorts. All checkpoints are selected on validation data.

5g_energy_consumption
Objective	Single	E5
MSE	$4.10\pm 0.02$	$4.054\pm 0.008$
HL-Gaussian	$4.07\pm 0.04$	$4.04\pm 0.01$
Quantile Loss	$4.10\pm 0.03$	$4.06\pm 0.01$

Appendix

Table 13: Dataset-wise MLP results with AdamW on 41 Beyond datasets (split 0). Test RMSE is mean

\pm

sample standard deviation (

\mathrm{ddof}=1

). Single uses 15 seeds; E5 uses three disjoint ensembles of five seeds.

mercedes_benz_greener_manufacturing
Objective	Single	E5
MSE	$8.45\pm 0.07$	$8.42\pm 0.02$
HL-Gaussian	$8.39\pm 0.07$	$8.34\pm 0.03$
Quantile Loss	$8.34\pm 0.03$	$8.325\pm 0.009$

Appendix

Table 14: Dataset-wise MLP results with AdamW (continued). Test RMSE: mean

\pm

sample standard deviation. Single: 15 seeds; E5: three ensembles of five.

5g_energy_consumption
Model	Objective	Single	E5
MLP	MSE	$4.07\pm 0.03$	$4.02\pm 0.01$
	MAE	$4.17\pm 0.03$	$4.13\pm 0.01$
	Huber Loss	$4.15\pm 0.05$	$4.09\pm 0.02$
	Soft Binning	$4.13\pm 0.03$	$4.09\pm 0.02$
	Hard Binning	$4.06\pm 0.03$	$4.054\pm 0.007$

Appendix

Table 15: Extended Beyond results with the Muon optimizer. Single denotes one model; E5 denotes an ensemble of five.

allstate_claims_severity
Model	Objective	Single	E5
MLP	MSE	$0.5400\pm 0.0002$	$0.5388\pm 0.0001$
	MAE	$0.5408\pm 0.0001$	$0.5402\pm 0.0000$
	Huber Loss	$0.5393\pm 0.0002$	$0.5388\pm 0.0000$
	Soft Binning	$0.5378\pm 0.0001$	$0.5373\pm 0.0000$
	Hard Binning	$0.5393\pm 0.0001$	$0.5389\pm 0.0000$

Appendix

Table 16: Extended Beyond results with the Muon optimizer (continued).

body_density_prediction
Model	Objective	Single	E5
MLP	MSE	$0.013\pm 0.002$	$0.0127\pm 0.0002$
	MAE	$0.0121\pm 0.0002$	$0.0119\pm 0.0001$
	Huber Loss	$0.0125\pm 0.0004$	$0.0123\pm 0.0000$
	Soft Binning	$0.0137\pm 0.0002$	$0.0137\pm 0.0001$
	Hard Binning	$0.0153\pm 0.0008$	$0.0143\pm 0.0004$

Appendix

Table 17: Extended Beyond results with the Muon optimizer (continued).

cirrhosis_patient_survival_prediction
Model	Objective	Single	E5
MLP	MSE	$0.72\pm 0.02$	$0.705\pm 0.005$
	MAE	$0.66\pm 0.02$	$0.655\pm 0.008$
	Huber Loss	$0.67\pm 0.01$	$0.668\pm 0.005$
	Soft Binning	$0.69\pm 0.01$	$0.693\pm 0.006$
	Hard Binning	$0.93\pm 0.07$	$0.83\pm 0.02$

Appendix

Table 18: Extended Beyond results with the Muon optimizer (continued).

coffee_rating_prediction
Model	Objective	Single	E5
MLP	MSE	$1.48\pm 0.02$	$1.4771\pm 0.0003$
	MAE	$1.48\pm 0.05$	$1.46\pm 0.02$
	Huber Loss	$1.54\pm 0.07$	$1.51\pm 0.02$
	Soft Binning	$1.43\pm 0.04$	$1.406\pm 0.008$
	Hard Binning	$1.53\pm 0.04$	$1.53\pm 0.02$

Appendix

Table 19: Extended Beyond results with the Muon optimizer (continued).

diamonds
Model	Objective	Single	E5
MLP	MSE	$0.0892\pm 0.0008$	$0.0856\pm 0.0001$
	MAE	$0.0869\pm 0.0008$	$0.0857\pm 0.0005$
	Huber Loss	$0.0879\pm 0.0005$	$0.0852\pm 0.0001$
	Soft Binning	$0.0879\pm 0.0006$	$0.0876\pm 0.0002$
	Hard Binning	$0.0879\pm 0.0006$	$0.0873\pm 0.0004$

Appendix

Table 20: Extended Beyond results with the Muon optimizer (continued).

electric_motor_temperature_prediction
Model	Objective	Single	E5
MLP	MSE	$2.62\pm 0.03$	$2.61\pm 0.02$
	MAE	$2.96\pm 0.08$	$2.92\pm 0.03$
	Huber Loss	$2.65\pm 0.06$	$2.63\pm 0.04$
	Soft Binning	$2.66\pm 0.05$	$2.64\pm 0.02$
	Hard Binning	$3.7\pm 0.2$	$3.53\pm 0.09$

Appendix

Table 21: Extended Beyond results with the Muon optimizer (continued).

food_delivery_time
Model	Objective	Single	E5
MLP	MSE	$7.73\pm 0.01$	$7.715\pm 0.008$
	MAE	$7.81\pm 0.03$	$7.750\pm 0.008$
	Huber Loss	$7.75\pm 0.01$	$7.714\pm 0.007$
	Soft Binning	$7.748\pm 0.008$	$7.742\pm 0.002$
	Hard Binning	$7.65\pm 0.01$	$7.636\pm 0.002$

Appendix

Table 22: Extended Beyond results with the Muon optimizer (continued).

garments_worker_productivity
Model	Objective	Single	E5
MLP	MSE	$0.099\pm 0.004$	$0.097\pm 0.001$
	MAE	$0.101\pm 0.004$	$0.100\pm 0.002$
	Huber Loss	$0.099\pm 0.004$	$0.098\pm 0.002$
	Soft Binning	$0.102\pm 0.006$	$0.099\pm 0.001$
	Hard Binning	$0.105\pm 0.005$	$0.104\pm 0.004$

Appendix

Table 23: Extended Beyond results with the Muon optimizer (continued).

houses
Model	Objective	Single	E5
MLP	MSE	$0.2273\pm 0.0009$	$0.2226\pm 0.0001$
	MAE	$0.228\pm 0.001$	$0.2217\pm 0.0003$
	Huber Loss	$0.2242\pm 0.0007$	$0.2225\pm 0.0003$
	Soft Binning	$0.226\pm 0.001$	$0.2233\pm 0.0004$
	Hard Binning	$0.225\pm 0.001$	$0.2234\pm 0.0005$

Appendix

Table 24: Extended Beyond results with the Muon optimizer (continued).

mercari_price_suggestion_1m
Model	Objective	Single	E5
MLP	MSE	$0.4746\pm 0.0006$	$0.4664\pm 0.0008$
	MAE	$0.479\pm 0.002$	$0.472\pm 0.002$
	Huber Loss	$0.4741\pm 0.0003$	$0.4666\pm 0.0003$
	Soft Binning	$0.472\pm 0.002$	$0.463\pm 0.001$
	Hard Binning	$0.4742\pm 0.0008$	$0.4649\pm 0.0009$

Appendix

Table 25: Extended Beyond results with the Muon optimizer (continued).

miami_housing
Model	Objective	Single	E5
MLP	MSE	$0.1446\pm 0.0008$	$0.1427\pm 0.0002$
	MAE	$0.148\pm 0.001$	$0.1454\pm 0.0004$
	Huber Loss	$0.1450\pm 0.0005$	$0.1438\pm 0.0002$
	Soft Binning	$0.150\pm 0.002$	$0.1465\pm 0.0004$
	Hard Binning	$0.1486\pm 0.0005$	$0.1480\pm 0.0002$

Appendix

Table 26: Extended Beyond results with the Muon optimizer (continued).

obesity_estimation
Model	Objective	Single	E5
MLP	MSE	$4.91\pm 0.03$	$4.904\pm 0.002$
	MAE	$4.85\pm 0.02$	$4.84\pm 0.01$
	Huber Loss	$4.94\pm 0.05$	$4.92\pm 0.03$
	Soft Binning	$4.91\pm 0.04$	$4.90\pm 0.02$
	Hard Binning	$4.96\pm 0.05$	$4.944\pm 0.006$

Appendix

Table 27: Extended Beyond results with the Muon optimizer (continued).

qsar_aquatic_toxicity
Model	Objective	Single	E5
MLP	MSE	$1.17\pm 0.02$	$1.16\pm 0.01$
	MAE	$1.28\pm 0.03$	$1.256\pm 0.008$
	Huber Loss	$1.16\pm 0.02$	$1.156\pm 0.004$
	Soft Binning	$1.14\pm 0.02$	$1.140\pm 0.007$
	Hard Binning	$1.22\pm 0.04$	$1.211\pm 0.009$

Appendix

Table 28: Extended Beyond results with the Muon optimizer (continued).

qsar_tid_11
Model	Objective	Single	E5
MLP	MSE	$0.784\pm 0.005$	$0.772\pm 0.001$
	MAE	$0.783\pm 0.005$	$0.7720\pm 0.0009$
	Huber Loss	$0.786\pm 0.006$	$0.775\pm 0.003$
	Soft Binning	$0.787\pm 0.004$	$0.782\pm 0.002$
	Hard Binning	$0.84\pm 0.01$	$0.819\pm 0.008$

Appendix

Table 29: Extended Beyond results with the Muon optimizer (continued).

santander_transaction_value
Model	Objective	Single	E5
MLP	MSE	$1.409\pm 0.006$	$1.403\pm 0.002$
	MAE	$1.408\pm 0.007$	$1.396\pm 0.003$
	Huber Loss	$1.403\pm 0.009$	$1.394\pm 0.005$
	Soft Binning	$1.399\pm 0.009$	$1.389\pm 0.003$
	Hard Binning	$1.415\pm 0.006$	$1.409\pm 0.003$

Appendix

Table 30: Extended Beyond results with the Muon optimizer (continued).

sberbank_housing_market_forecasting
Model	Objective	Single	E5
MLP	MSE	$0.27\pm 0.02$	$0.26\pm 0.02$
	MAE	$0.263\pm 0.010$	$0.256\pm 0.004$
	Huber Loss	$0.26\pm 0.01$	$0.254\pm 0.006$
	Soft Binning	$0.263\pm 0.009$	$0.2521\pm 0.0010$
	Hard Binning	$0.254\pm 0.003$	$0.2498\pm 0.0003$

Appendix

Table 31: Extended Beyond results with the Muon optimizer (continued).

student_portuguese_performance
Model	Objective	Single	E5
MLP	MSE	$3.09\pm 0.09$	$3.00\pm 0.03$
	MAE	$3.03\pm 0.08$	$2.96\pm 0.03$
	Huber Loss	$3.04\pm 0.08$	$2.99\pm 0.03$
	Soft Binning	$2.99\pm 0.09$	$2.92\pm 0.03$
	Hard Binning	$3.00\pm 0.02$	$2.979\pm 0.002$

Appendix

Table 32: Extended Beyond results with the Muon optimizer (continued).

telemonitoring_parkinsons_biomedical_voice_measurements
Model	Objective	Single	E5
MLP	MSE	$12\pm 2$	$11.6\pm 0.3$
	MAE	$11.3\pm 0.5$	$11.0\pm 0.3$
	Huber Loss	$12\pm 1$	$11.40\pm 0.08$
	Soft Binning	$10.9\pm 0.3$	$10.6\pm 0.2$
	Hard Binning	$11.2\pm 0.7$	$10.27\pm 0.05$

Appendix

Table 33: Extended Beyond results with the Muon optimizer (continued).

video_transcoding_time_prediction
Model	Objective	Single	E5
MLP	MSE	$0.269\pm 0.007$	$0.2594\pm 0.0010$
	MAE	$0.261\pm 0.008$	$0.255\pm 0.004$
	Huber Loss	$0.259\pm 0.007$	$0.255\pm 0.005$
	Soft Binning	$0.252\pm 0.003$	$0.250\pm 0.001$
	Hard Binning	$0.264\pm 0.003$	$0.2628\pm 0.0008$

Appendix

Table 34: Extended Beyond results with the Muon optimizer (continued).

wine_world_cost
Model	Objective	Single	E5
MLP	MSE	$0.43\pm 0.01$	$0.4116\pm 0.0008$
	MAE	$0.415\pm 0.005$	$0.4119\pm 0.0009$
	Huber Loss	$0.431\pm 0.010$	$0.414\pm 0.007$
	Soft Binning	$0.440\pm 0.007$	$0.421\pm 0.002$
	Hard Binning	$0.450\pm 0.006$	$0.445\pm 0.002$

Appendix

Table 35: Extended Beyond results with the Muon optimizer (continued).

Explore similar work

TabNSM: Neural Sparse Mixer for Tabular Regression
Aug 18, 2026Ali Eslamian, Qiang ChengTabular Regression Tabular Representation Learning
Department of Computer Science University of Kentucky Lexington, KY 40506 · Department of Computer Science Institute for Biomedical Informatics University of Kentucky Lexington, KY 40506
LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning
May 14, 2026Changryeol Choi, Hyewon Park, Yujin Kwon +1Ensemble Learning Low-Rank Adaptation
CJ Logistics
Below what training size do deep tabular generators stop beating trivial baselines? A preregistered benchmark on a size ladder of clinical and standard datasets
Oct 2, 2026Shivam ShrivastavaSynthetic Tabular Data Generation Synthetic Data Evaluation
VIT Bhopal University, India

Aug 18, 2026, cs.LGJ/K move · Enter open · S save

TabNSM: Neural Sparse Mixer for Tabular Regression

Ali Eslamian, Qiang Cheng

Department of Computer Science University of Kentucky Lexington, KY 40506 · Department of Computer Science Institute for Biomedical Informatics University of Kentucky Lexington, KY 40506

Tabular Regression Tabular Representation Learning Feature Interaction Modeling Sparse Neural Networks

Open paper AskPDF arXiv

Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.

Abstract

Explore similar work

TabNSM: Neural Sparse Mixer for Tabular Regression

LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

Below what training size do deep tabular generators stop beating trivial baselines? A preregistered benchmark on a size ladder of clinical and standard datasets