Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods

Hao Xiao

cs.LGApr 4, 2026

Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods

Authors: Hao Xiao

Organizations: ATLAS AI Lab

Abstract

Choosing between classical and Bayesian sparse regression methods involves a real trade-off: penalized estimators like Lasso run in milliseconds but give no uncertainty estimates,while Horseshoe and Spike-and-Slab priors produce full posteriors but need MCMC chains that take minutes per fit.Surprisingly few studies compare these two families head-to-head under the conditions that actually make sparse regression hard -- correlated features, weak signals, and growing dimensionality. We benchmark six methods (OLS, Ridge,Lasso, Elastic Net, Horseshoe, Spike-and-Slab) on synthetic data with three covariance structures (rho up to 0.9), four SNR levels, and p in {20, 50, 100}, plus the Diabetes dataset,totalling over 2,600 experiments. The results are clear on some points and nuanced on others. Bayesian methods win on prediction error (MSE 72 vs. 108-267), and the Horseshoe delivers near-nominal 95% coverage (94.8%). But Spike-and-Slab,despite narrower intervals, under-covers at 91.9% -- its continuous relaxation likely plays a role. For variable selection, Lasso and Spike-and-Slab tie at F1 ~ 0.47, making Lasso the practical default when posteriors are not needed. Code and data are available at https://github.com/xiao98/sparse-bayesian-regression-bench.

Figures & tables

Axis	Values
Dataset	Independent, Block Correlated, Toeplitz
Model	OLS, Ridge, Lasso, Elastic Net, Horseshoe, Spike-and-Slab
Correlation ( $\rho$ )	0.0, 0.3, 0.6, 0.9
Signal-to-noise ratio	0.5, 1.0, 2.0, 5.0
Dimensionality ( $p$ )	20, 50, 100
Random seeds	42, 123, 456, 789, 1024

Table 1: Experimental grid axes.

Model	Test MSE	Coef. $L_{2}$ Error	Support $F_{1}$
OLS	267.159 $\pm$ 323.478	16.559 $\pm$ 14.988	0.296 $\pm$ 0.026
Ridge	118.597 $\pm$ 124.690	6.405 $\pm$ 2.902	0.297 $\pm$ 0.027
Lasso	108.101 $\pm$ 121.213	5.199 $\pm$ 3.311	0.470 $\pm$ 0.132
Elastic Net	108.751 $\pm$ 122.784	5.209 $\pm$ 3.248	0.459 $\pm$ 0.126
Horseshoe	71.917 $\pm$ 76.047	4.210 $\pm$ 2.630	0.288 $\pm$ 0.037
Spike-and-Slab	72.992 $\pm$ 78.317	4.588 $\pm$ 3.041	0.469 $\pm$ 0.156

Table 2: Aggregate performance across all synthetic experiments.

Figure 1: Test MSE vs. SNR for each covariance design. Error bars:

\pm 1

std across seeds.

Figure 2: MSE degradation with increasing

\rho

. Left: Block Correlated. Right: Toeplitz.

Figure 3: MSE heatmap: model $\times$ correlation strength.

Figure 4: Support recovery

F_{1}

vs. SNR.

Figure 5: Coefficient $L_{2}$ error by model and dimensionality.

Figure 6: Empirical 95% HDI coverage. Dashed red line: nominal level.

Model	Coverage (95% HDI)	Avg. Interval Width
Horseshoe	0.948 $\pm$ 0.045	2.237 $\pm$ 1.121
Spike-and-Slab	0.919 $\pm$ 0.048	1.058 $\pm$ 0.865

Table 3: Bayesian posterior calibration: coverage and interval width.

Figure 7: Mean fit time per experiment (log scale).

Figure 8: Test MSE spread across random seeds.

Model	Test MSE	Test RMSE	Time (s)
OLS	0.507 $\pm$ 0.060	0.711 $\pm$ 0.041	0.00
Ridge	0.508 $\pm$ 0.057	0.712 $\pm$ 0.039	0.00
Lasso	0.504 $\pm$ 0.057	0.709 $\pm$ 0.039	0.02
Elastic Net	0.504 $\pm$ 0.057	0.709 $\pm$ 0.039	0.15
Horseshoe	0.472 $\pm$ 0.011	0.687 $\pm$ 0.008	11.71
Spike-and-Slab	0.473 $\pm$ 0.008	0.688 $\pm$ 0.006	12.51

Table 4: Diabetes dataset (

n=442

p=10

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	$\rho=0.0$	$\rho=0.3$	$\rho=0.6$	$\rho=0.9$
OLS	275.911	273.982	263.035	262.441
Ridge	130.683	127.578	117.393	97.609
Lasso	113.853	114.410	108.874	96.877
Elastic Net	115.375	115.246	108.801	96.780
Horseshoe	77.442	72.429	72.122	67.379
Spike-and-Slab	75.726	75.250	75.055	69.101

Appendix

Table 5: Mean test MSE by model and correlation strength

\rho

. Best value per column in bold.

	SNR=0.5	SNR=1.0	SNR=2.0	SNR=5.0
OLS	0.296	0.296	0.296	0.295
Ridge	0.295	0.297	0.297	0.299
Lasso	0.400	0.456	0.498	0.527
Elastic Net	0.395	0.438	0.479	0.522
Horseshoe	0.279	0.288	0.288	0.297
Spike-and-Slab	0.374	0.435	0.489	0.579

Appendix

Table 6: Mean support

F_{1}

by model and SNR. Best value per column in bold.

	$p=20$	$p=50$	$p=100$
OLS	0.00	0.00	0.01
Ridge	0.00	0.00	0.02
Lasso	0.02	0.05	0.10
Elastic Net	0.14	0.29	0.53
Horseshoe	31.76	50.85	—
Spike-and-Slab	35.23	53.81	—

Appendix

Table 7: Mean fitting time (seconds) by model and dimensionality

p

Explore similar work

Joint Model and Data Sparsification via the Marginal Likelihood
May 28, 2026Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth +2Dataset Pruning Robust Regression
1UvA-Bosch Delta Lab, University of Amsterdam · 2RIKEN Center for AI Project, Tokyo, Japan · Department of Computer Science, Johns Hopkins University.
LinearPFN: Amortized Variable Selection for Linear Models with Interactions
Oct 5, 2026Louis Schiekiera, Max Zimmer, Christophe Roux +3Model Selection Prior-Data Fitted Networks
Institute of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany · Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany · Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany +1
Efficient Cross-Validation for Sparse Linear Regression
Jun 26, 2023Ryan Cory-Wright, Andrés GómezModel Selection Mixed-Integer Linear Programming
Department of Analytics, Marketing and Operations, Imperial Business School, London, UK · Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA

May 28, 2026J/K move · Enter open · S save

Joint Model and Data Sparsification via the Marginal Likelihood

Alexander Timans, Thomas Möllenhoff, Christian A. Naesseth, Mohammad Emtiyaz Khan+1

1UvA-Bosch Delta Lab, University of Amsterdam · 2RIKEN Center for AI Project, Tokyo, Japan · Department of Computer Science, Johns Hopkins University.

Dataset Pruning Robust Regression Sparse Recovery Bayesian Inference Linear Regression

Open paper AskPDF arXiv

Sparse recovery in linear systems underpins applications from signal processing to high-dimensional regression. Sparse Bayesian Learning, grounded in the principle of automatic relevance determination (ARD), offers a practical Bayesian mechanism for feature sparsity via marginal likelihood optimization. Yet, its reliance on a homoscedastic noise model renders it sensitive to data contaminations such as outliers or misspecified noise, harming model fit and predictions. Instead, we propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification via a single Bayesian objective. This symmetric pruning of model and data offers a natural extension that preserves conjugacy, admits closed-form updates for standard optimization procedures, and aligns with perspectives from robust regression and influence functions. Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.

Abstract

Explore similar work

Joint Model and Data Sparsification via the Marginal Likelihood

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

Efficient Cross-Validation for Sparse Linear Regression