Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks

Benjamin S. Knight; Ahsaas Bajaj

cs.LGApr 4, 2026

Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks

Authors: Benjamin S. Knight, Ahsaas Bajaj

Abstract

This study surveys the historical development of regularization, tracing its evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features, Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations spanning a 7-dimensional manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.

Figures & tables

Figure 1: The three norms as applied to regularization. Regularization using the $\ell_{0}$ norm (solid line) performs discrete variable selection, keeping only the larger coefficient while setting the smaller one to zero. Lasso regularization (thick dashed line) via the $\ell_{1}$ norm nudges $\hat{\beta}$ towards the axes, thus tending towards partial sparsity with some shrinkage of both coefficients. Ridge regression (dotted circular line) uses the $\ell_{2}$ norm, shrinking both coefficients proportionally toward the origin while maintaining their relative magnitudes.

	LassoCV	Post-Lasso OLS	ElasticNetCV	RidgeCV
Candidate Regularization Parameter Values	Allowed Values for $\alpha$ : $[10^{5},10^{4},10^{3},10^{2},10^{1},10^{0},10^{-1},10^{-2},10^{-3}]$
Cross-Validation (CV)	5-fold ( $n{=}100$ ); 4-fold ( $n{=}1,000$ ); 3-fold ( $n{=}10,000$ ); 2-fold ( $n{=}100,000$ )
Test Set Holdout	20%
L1 Ratio Values	n/a	n/a	[0.0, 0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1.0]	n/a
CV Fits per Configuration	18–45	18–45	144–360	18–45

Table 1: Regularization parameters for the four methods evaluated. All methods use identical alpha grid with no scaling—this respects scikit-learn’s design where Lasso has built-in

(1/2n)

normalization but Ridge does not. ElasticNet L1 ratios based on scikit-learn recommended defaults plus pure Ridge (0.0). All methods use identical train/test splits (80%/20%) and adaptive cross-validation.

Hyperparameter	Levels	Values
Features ( $p$ )	2	64, 128
Rank Ratio ( $r/p$ )	2	0.9, 1.0
Eigenvalue Dispersion ( $\kappa$ )	2	$\sim 10^{1}$ (low), $\sim 10^{5}$ (high)
$\beta$ Distribution	5	Gamma(0.04), Gamma(0.2), Gamma(1.0), Gamma(5.0), Uniform
Sparsity Level	2	0%, 15%
Signal-to-Noise Ratio (SNR)	3	0.04, 0.2, 1.0

Table 2: We derive our feature spaces from a space of 7 hyper-parameters.

Panel of Figure 2 — Figure 2: Distribution parameters for Eigenvalues and $\beta$ coefficients.

Figure 3: We selected a wide range of potential $\alpha$ values to accommodate both RidgeCV and LassoCV. Note how the optimal L1 parameter tends to fall at the extremes (i.e. 0.0 or 1.0).

Figure 4: The precision with which Lasso is able recover the true, non-zero $\beta$ coefficients is a function of the signal-to-noise ratio within the feature set.

Parameter	L–R	L–EN	EN–R	Avg
Sample Size $(n)$	0.549	0.284	0.091	0.308
SNR	0.117	0.058	0.021	0.065
$\beta$ Distribution	0.013	0.040	0.112	0.055
Dispersion $(\kappa)$	0.022	0.016	$<$ 0.001	0.013
Features $(p)$	0.012	0.006	0.002	0.007
Sparsity	0.006	0.002	0.001	0.003

Table 3: Parameter Importance for F1 Score Differences: Main Effects (

\omega^{2}

) and Interactions (F-statistics)

Parameter	R–L	EN–L	PL–L	R–EN	PL–EN	R–PL	Avg
$\beta$ Distribution	0.2155	0.0773	0.0024	0.1428	0.0024	0.0024	0.0738
Sample Size $(n)$	0.0133	0.0031	0.0015	0.0248	0.0015	0.0015	0.0076
Dispersion $(\kappa)$	0.0020	0.0099	0.0026	0.0007	0.0026	0.0026	0.0034
Rank Ratio $(r/p)$	0.0001	0.0015	0.0027	0.0024	0.0027	0.0027	0.0020
SNR	0.0018	0.0010	0.0001	0.0056	0.0001	0.0001	0.0015
Sparsity	0.0007	0.0004	$<$ 0.0001	0.0002	$<$ 0.0001	$<$ 0.0001	0.0002

Table 4: Parameter Importance for Coefficient L2 Error Differences: Main Effects (

\omega^{2}

) and Interactions (F-statistics)

Figure 5: Post-Lasso OLS has the largest L2 error across the 8 $n/p$ ratios simulated.

Figure 6: ElasticNetCV was most likely to yield the smallest RMSE within the test set.

Parameter	R–L	EN–L	PL–L	R–EN	PL–EN	R–PL	Avg
Sample Size $(n)$	0.0077	0.0110	0.0893	0.0005	0.0930	0.0800	0.0469
Features $(p)$	0.0007	0.0007	0.0047	$<$ 0.0001	0.0052	0.0047	0.0027
$\beta$ Distribution	0.0068	0.0014	0.0001	0.0051	$<$ 0.0001	0.0011	0.0024
SNR	0.0020	0.0031	0.0032	0.0014	0.0006	0.0014	0.0020
Dispersion $(\kappa)$	0.0011	0.0003	0.0007	0.0006	0.0002	$<$ 0.0001	0.0005
Rank Ratio $(r/p)$	$<$ 0.0001	$<$ 0.0001	0.0002	$<$ 0.0001	$<$ 0.0001	0.0001	$<$ 0.0001

Table 5: Parameter Importance for RMSE Differences: Main Effects (

\omega^{2}

) and Interactions (F-statistics)

Observable Regime	Recommendation	Evidence
$n/p\geq 78$ , any $\kappa$	ElasticNet (safe default) or Lasso	Figs. C.1–C.3: near-zero $\Delta$ F1 at large $n$
$n/p<78$ , high $\kappa$ , SNR $\geq 0.20$ or $n\geq 1$ k	ElasticNet	Fig. 4: ${\sim}1.7$ – $5\times$ recall advantage; Figs. C.1, C.3
$n\approx 100$ , high $\kappa$ , $\alpha$ saturated (SNR $\approx 0.04$ )	Ridge † (EN if genuine selection needed)	Fig. C.2: Ridge $>$ EN via recall $=1.0$ at very low SNR
$n/p<78$ , low $\kappa$ , high $\alpha$	Ridge (competitive F1) †	Figs. C.2, C.3: Ridge wins at small $n$ , low SNR
$n/p<78$ , low $\kappa$ , all of : low elected $\alpha$ AND sparse domain	Lasso is viable	Fig. 4 (low $\kappa$ , SNR $=1.0$ ): Lasso recall $=0.82$ ; EN recall $=0.94$
$n/p<78$ , low $\kappa$ , uncertain SNR	ElasticNet (safe default)	EN recall $\geq 0.91$ across all SNR at low $\kappa$

Table 6: Recommended Regularization Method by Observable Regime: Variable Selection (F1)

Observable Regime	Recommendation	Evidence
$n/p\geq 78$ , any $\kappa$	Any method (diff. negligible)	Figs. D.1–D.6: near-zero cells at large $n$
$n/p<78$ , high $\kappa$	ElasticNet	Figs. D.3, D.4: 20–40% lower error (sparse); consistent edge over Ridge (dense)
$n/p<78$ , low $\kappa$ , sparse	Lasso or ElasticNet	Fig. D.6: Lasso beats Ridge with sparsity
$n/p<78$ , low $\kappa$ , dense	Ridge	Fig. D.6: Ridge beats Lasso without sparsity
Any $n$ , any $\kappa$	Avoid Post-Lasso OLS ‡	Fig. D.2: higher error across nearly all cells

Table 7: Recommended Regularization Method by Observable Regime: Coefficient Estimation (

L_{2}

Error)

CV-Elected $\alpha$	Likely Regime	Recommended Action
High / at boundary	Low SNR	Switch to Ridge for all objectives. Lasso recall collapses to 0.18–0.20 in this regime (Fig. 4 ); Ridge outperforms Lasso on F1 (Fig. C.3) and RMSE (Fig. E.4, low-SNR rows).
Moderate (e.g., 0.1–10)	Moderate SNR	ElasticNet provides the best risk-adjusted choice across all objectives.
Low (e.g., $<0.1$ )	High SNR	Lasso becomes viable if $\kappa$ is also low and sparsity is expected. Ridge and ElasticNet remain safe defaults.

Table 8: Actionable Mapping: CV-Elected

\alpha

as SNR Diagnostic. Threshold ranges are approximate and derived from the empirical distribution of elected

\alpha

across our simulation grid (Figure B.1), conditional on known SNR tiers (0.04, 0.2, 1.0). Practitioners should treat these as heuristic guidelines rather than sharp cutoffs; the appropriate thresholds will vary with feature scaling and dataset characteristics.

Figure 7: Consolidated decision framework for regularization selection. Decision rules branch on observable quantities: the ratio $p/n$ (under-determined check), the sample-to-feature ratio $n/p$ (large-sample threshold $\geq 78$ , empirically determined from Figure 5), condition number ( $\kappa$ ), and CV-elected $\alpha$ . Recommendations are validated within $p\in\{64,128\}$ , $n\in\{100\text{--}100\text{k}\}$ , $\kappa\in\{{\sim}10^{1},{\sim}10^{5}\text{--}10^{6}\}$ , sparsity $\in\{0\%,15\%\}$ . Thresholds for $\kappa$ ( ${\sim}10^{2}$ and ${\sim}10^{4}$ ) are interpolated from two tested regimes; the intermediate range was not directly evaluated. Ridge achieves high F1 in certain regimes through recall $=1.0$ (all features retained), not genuine variable selection ( $\dagger$ ). The $\beta$ distribution ( $\omega^{2}=0.112$ for EN–R; Table 3 ), $n\times\text{SNR}$ interaction ( $F=569$ ; Table 3 ), and Sparsity $\times$ $\kappa$ interaction (Figures D.4, D.6; not formally quantified) are additional sources of variation not captured in the branching structure. ElasticNet compute overhead (167–219 $\times$ by mean; 5–23 $\times$ by median) is specific to the 8-value L1 grid used in this study and scales linearly with grid size. Ridge + post-hoc filtering is noted as future work, not an empirical finding of this study.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Distributions of eigenvalues across 8 ML models sampled from production.

Figure B.1: The optimal Lasso regularization parameter $\alpha$ is primarily driven by SNR and sample size $n$ ; lower SNR and smaller datasets require significantly higher $\alpha$ values to prevent overfitting.

Figure C.1: ElasticNet consistently outperforms Lasso, especially as SNR decreases. This performance gap is most pronounced in small- $n$ , large- $p$ scenarios.

Figure C.2 : Ridge demonstrates superior F1 performance over ElasticNet across the majority of simulation parameters, with the advantage becoming most prominent in low sample size ( $n=100$ ) and low SNR ( $SNR=0.04$ ) environments.

Figure C.3 : Ridge consistently outperforms or matches Lasso across most tested conditions, with the strongest performance gains occurring at lower SNR and smaller sample sizes (e.g., $n=100$ ).

Figure D.1 : ElasticNet consistently outperforms Post-Lasso OLS in coefficient accuracy across most simulated conditions, with its greatest performance advantages appearing in scenarios with high multicollinearity (low condition number $\kappa$ ) and lower SNR.

Figure D.2 : Post-Lasso OLS consistently yields a higher coefficient $L_{2}$ error than standard Lasso, with this performance gap intensifying as SNR decreases and the condition number $\kappa$ increases.

Figure D.3 : ElasticNet generally achieves a lower coefficient $L_{2}$ error compared to Lasso as the feature correlation ( $\gamma$ ) and condition number ( $\kappa$ ) increase.

Figure D.4 : ElasticNet significantly outperforms Ridge regression as $\gamma$ increases and SNR decreases, with the advantage being most pronounced in sparse (Sparsity 15%) and high-dimensional ( $\kappa\approx 10^{-6}$ ) settings.

Figure D.5 : Ridge consistently achieves lower coefficient $L_{2}$ error compared to Post-Lasso OLS as $\gamma$ increases, a trend that remains robust across varying sparsity levels and SNR.

Figure D.6 : While Lasso generally outperforms Ridge in high-sparsity scenarios, Ridge tends to achieve lower coefficient $L_{2}$ error as $\kappa$ increases and the feature rank-to-dimension ratio decreases.

Figure E.1 : ElasticNet generally outperforms Lasso in scenarios with low sample sizes ( $n=100$ ) and high SNR, particularly as the correlation between features ( $\gamma$ ) increases.

Figure E.2 : Lasso generally outperforms Post-Lasso OLS (positive $\Delta$ Test RMSE) particularly in low- $n$ ( $n=100$ ), high-noise ( $SNR=1.00$ ) scenarios.

Figure E.3 : ElasticNet generally outperforms Post-Lasso OLS across most sparsity and SNR levels, with the most significant relative reductions in test RMSE occurring in low- $n$ ( $n=100$ ) and high-SNR ( $SNR=1.00$ ) scenarios.

Figure E.4 : Ridge generally outperforms Lasso (negative $\Delta$ Test RMSE) in low SNR and high-rank scenarios, while Lasso tends to excel in sparse, high-SNR conditions with small sample sizes ( $n=100$ ).

Figure E.5 : While performance is similar across most conditions, ElasticNet generally outperforms Ridge (lower RMSE) in high-SNR ( $SNR=1.00$ ) and low- $n$ ( $n=100$ ) scenarios.

Figure E.6 : Ridge consistently achieves a lower RMSE than Post-Lasso OLS in scenarios with low sample sizes ( $n=100$ ) and high SNR, outperforming its counterpart as the feature-to-sample ratio increases.

Figure F.1 : Ridge is consistently the most computationally efficient method across both feature dimensions, while ElasticNet exhibits significantly higher mean fit times.

Explore similar work

Aug 26, 2026

Adaptive Regularization for Random Features: A Neighboring Early-Stopping Rule with Oracle-Rate Guarantees

Random feature methods provide a scalable approximation to kernel ridge regression (KRR), but the regularization parameter that yields the oracle learning rate depends on unknown smoothness and capacity parameters. In this work, we propose a neighboring early-stopping rule for adaptive regularization in KRR with random features (KRR-RF). The method uses a grid that is uniform in inverse regularization and compares only adjacent estimators, reducing the number of discrepancy comparisons relative to standard all-pairs Lepskii-type procedures. Both the neighboring discrepancy and its empirical complexity term can be computed directly in the random feature space, without constructing the exact kernel Gram matrix. We establish a high-probability comparison bound for neighboring KRR-RF estimators and show that, under standard source and capacity conditions together with suitable grid and random feature budget conditions, the selected estimator attains the oracle polynomial learning rate up to logarithmic factors. The result allows the regularization parameter to be selected without prior knowledge of the source and capacity exponents and covers both well-specified and partially misspecified regimes. Our analysis is based on an empirical random feature effective dimension that connects the observable stopping threshold with the population complexity of the random feature model. Simulation and real-data experiments illustrate the prediction performance and computational behavior of the proposed method in comparison with standard tuning procedures.

Caixing Wang, Zhibo Chen, Yue Wang

School of Statistics and Data Science, Southeast University · Faculty of Business for Science and Technology, School of Management, University of Science and Technology of China

Random Feature Methods Kernel Ridge Regression Statistical Learning Theory Early Stopping

Ask PDF

Jun 26, 2023

Efficient Cross-Validation for Sparse Linear Regression

Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To choose hyperparameters that control the sparsity level and amount of regularization, practitioners commonly use k-fold cross-validation. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs) for each hyperparameter combination. To address this computational burden, we derive computationally tractable relaxations of the k-fold cross-validation loss, facilitating hyperparameter selection while solving

50

80\%

fewer MIOs in practice. Our computational results demonstrate, across eleven real-world UCI datasets, that exact MIO-based cross-validation can be competitive with mature software packages such as glmnet and L0Learn.

Ryan Cory-Wright, Andrés Gómez

Department of Analytics, Marketing and Operations, Imperial Business School, London, UK · Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA

Model Selection Mixed-Integer Linear Programming Linear Regression Hyperparameter Optimization

Ask PDF

Apr 4, 2026

Sparse Regression under Correlation and Weak Signals: A Reproducible Benchmark of Classical and Bayesian Methods

Choosing between classical and Bayesian sparse regression methods involves a real trade-off: penalized estimators like Lasso run in milliseconds but give no uncertainty estimates,while Horseshoe and Spike-and-Slab priors produce full posteriors but need MCMC chains that take minutes per fit.Surprisingly few studies compare these two families head-to-head under the conditions that actually make sparse regression hard -- correlated features, weak signals, and growing dimensionality. We benchmark six methods (OLS, Ridge,Lasso, Elastic Net, Horseshoe, Spike-and-Slab) on synthetic data with three covariance structures (rho up to 0.9), four SNR levels, and p in {20, 50, 100}, plus the Diabetes dataset,totalling over 2,600 experiments. The results are clear on some points and nuanced on others. Bayesian methods win on prediction error (MSE 72 vs. 108-267), and the Horseshoe delivers near-nominal 95% coverage (94.8%). But Spike-and-Slab,despite narrower intervals, under-covers at 91.9% -- its continuous relaxation likely plays a role. For variable selection, Lasso and Spike-and-Slab tie at F1 ~ 0.47, making Lasso the practical default when posteriors are not needed. Code and data are available at https://github.com/xiao98/sparse-bayesian-regression-bench.

Hao Xiao

ATLAS AI Lab

Sparse Recovery Linear Regression Feature Selection Regression Prediction Intervals

Ask PDF