With the rise of high-dimensional correlated data, multicollinearity poses a significant challenge to model stability, often leading to unstable estimation and reduced predictive accuracy. This work proposes the Single-Parametric Principal Component Selection Operator (SPPCSO), an innovative penalized estimation method that integrates single-parametric principal component regression and L1 regularization to adaptively adjust the shrinkage factor by incorporating principal component information. This approach achieves a balance between variable selection and coefficient estimation, ensuring model stability and robust estimation even in high-dimensional, high-noise environments. The primary contribution lies in addressing the instability of traditional variable selection methods when applied to high-noise, high-dimensional correlated data. Theoretically, our method exhibits selection consistency and achieves a smaller estimation error bound compared to traditional penalized estimation approaches. Extensive numerical experiments demonstrate that SPPCSO not only delivers stable and reliable estimation in high-noise settings but also accurately distinguishes signal variables from noise variables in group-effect structured data with highly correlated noise variables, effectively eliminating redundant variables and achieving more stable variable selection. Furthermore, SPPCSO successfully identifies disease-associated genes in gene expression data analysis, showcasing strong practical value. The results indicate that SPPCSO serves as an ideal tool for high-dimensional variable selection, offering an efficient and interpretable solution for modeling correlated data.
Figures & tables
Estimation Method
Shrinkage Factor
Ridge estimation
di+kdi=1−di+kk
Liu estimation
di+1di+h
SPPCR estimation
⎩⎨⎧didi−1+θ,θdi,di≥1di<1
Table 1: Shrinkage factors of different estimation methods
X∗=(XZ),y∗=(y0).
Algorithm 1 The coordinate decent algorithm for the SPPCSO
Figure 1: Solution path of the SPPCSO with respect to the parameter λ
Figure 4
Method
σ =0.5
σ =1
σ =2
Est Error
Std Dev
Estn Error
Std Dev
Est Error
Std Dev
LASSO
0.9571
0.2083
1.7573
0.4200
3.6713
0.8382
MCP
8.0887
2.2153
9.0925
3.0784
9.9405
1.7509
SCAD
7.8636
3.2870
8.3720
2.4588
9.3404
1.7447
Enet
0.7861
0.1697
1.2426
0.3160
1.9663
0.8021
Mnet
0.6252
0.1383
0.9063
0.1692
1.2092
0.3593
Table 2: Average estimation error and standard deviation for Example 1
Method
σ =0.5
σ =1
σ =2
Pre Error
Std Dev
Pre Error
Std Dev
Pre Error
Std Dev
LASSO
0.3252
0.0605
1.2850
0.2187
5.2246
0.9739
MCP
4.4767
13.2624
8.2810
21.9644
9.1475
2.5962
SCAD
8.3170
26.1689
7.0328
24.4160
8.2789
2.3673
Enet
0.3208
0.0554
1.2479
0.2025
4.9183
0.9608
Mnet
0.2918
0.0462
1.1589
0.1976
4.6989
0.7843
Table 3: Average prediction error and standard deviation for Example 1
Method
σ =0.5
σ =1
σ =2
TPR
TNR
TMR
TPR
TNR
TMR
TPR
TNR
TMR
LASSO
1.000
0.982
0.138
1.000
0.987
0.138
0.989
0.988
0.138
MCP
0.653
0.997
0.000
0.619
0.992
0.000
0.538
0.996
0.000
SCAD
0.698
0.983
0.013
0.661
0.987
0.000
0.577
0.987
0.000
Enet
1.000
0.981
0.138
1.000
0.985
0.125
0.999
0.982
0.088
Mnet
1.000
0.998
0.488
1.000
0.997
0.525
0.998
0.994
0.200
Table 4: TPR, TNR, and TMR for Example 1
Method
ρ =0.5
ρ =0.75
ρ =0.95
Est Error
Std Dev
Est Error
Std Dev
Est Error
Std Dev
LASSO
4.4559
1.3309
4.5233
1.0556
4.0679
0.9334
MCP
19.1823
0.7085
19.3239
0.6469
19.2281
0.6575
SCAD
19.1818
0.7072
19.3295
0.6493
19.2287
0.6628
Enet
2.2640
0.6605
2.5113
0.8459
2.1949
0.8089
Mnet
1.4280
1.1497
1.6669
1.4552
1.3523
1.0139
Table 5: Average estimation error and standard deviation for Example 2
Method
ρ =0.5
ρ =0.75
ρ =0.95
Pre Error
Std Dev
Pre Error
Std Dev
Pre Error
Std Dev
LASSO
1.5626
0.5447
1.5407
0.2755
1.3384
0.2503
MCP
4.8789
0.7955
4.9638
0.8179
4.8581
0.7325
SCAD
4.8857
0.7807
4.9612
0.8239
4.8676
0.7459
Enet
1.3901
0.2449
1.3847
0.2449
1.2293
0.2139
Mnet
1.3837
0.2623
1.3720
0.2376
1.2027
0.2174
Table 6: Average prediction error and standard deviation for Example 2
Method
ρ =0.5
ρ =0.75
ρ =0.95
TPR
TNR
TMR
TPR
TNR
TMR
TPR
TNR
TMR
LASSO
0.983
0.956
0.013
0.988
0.964
0.000
0.991
0.970
0.013
MCP
0.200
0.999
0.000
0.200
0.998
0.000
0.200
0.998
0.000
SCAD
0.200
0.994
0.000
0.200
0.994
0.000
0.200
0.995
0.000
Enet
1.000
0.962
0.000
0.999
0.966
0.000
0.999
0.970
0.013
Mnet
0.981
0.986
0.025
0.968
0.988
0.038
0.983
0.991
0.038
Table 7: TPR, TNR, and TMR for Example 2
Figure 4: Performance comparison of methods. The image on the left is the radar chart of TPR, TNR, and TMR values for Example 1, while the one on the right corresponds to Example 2.
Method
Lasso
MCP
SCAD
Enet
Mnet
SACE
GCACE
SPPCSO
MAPE(train)
0.0385
0.0534
0.0502
0.0225
0.0309
0.0193
0.0245
0.0282
MAPE(test)
0.0908
0.0936
0.0884
0.0855
0.0944
0.0841
0.1063
0.0803
NNZ
32.60
25.40
19.28
805.64
254.94
82.90
135.38
72.44
Table 8: MAPE and NNZ values for the training and test sets of rat genetic data
Sparse principal component analysis (SPCA) addresses the poor interpretability and variable redundancy often encountered by principal component analysis (PCA) in high-dimensional data. However, SPCA typically imposes uniform penalties on variables and does not account for differences in variable importance, which may lead to unstable performance in highly noisy or structurally complex settings. We propose SP-SPCA, a method that introduces a single equilibrium parameter into the regularization framework to adaptively adjust variable penalties. This modification of the L2 penalty provides flexible control over the trade-off between sparsity and explained variance while maintaining computational efficiency. Simulation studies show that the proposed method consistently outperforms standard sparse principal component methods in identifying sparse loading patterns, filtering noise variables, and preserving cumulative variance, especially in high-dimensional and noisy settings. Empirical applications to crime and financial market data further demonstrate its practical utility. In real data analyses, the method selects fewer but more relevant variables, thereby reducing model complexity while maintaining explanatory power. Overall, the proposed approach offers a robust and efficient alternative for sparse modeling in complex high-dimensional data, with clear advantages in stability, feature selection, and interpretability
Ying Hu, Hu Yang
College of Mathematics and Statistics, Chongqing University, Chongqing, 401331, China
Using a replica-symmetric high-dimensional characterization, we develop an inferential framework for sparse support vector machines when the sample size and number of features grow proportionally. The main challenge is the nonsmooth hinge loss, which prevents direct application of debiasing arguments developed for smooth classification losses. We overcome this difficulty by representing the L1-penalized support vector machine (SVM) as a linear program and identifying the hinge-loss subgradient through its dual variables. This yields a computationally accessible debiased estimator whose coordinates are asymptotically Gaussian under the proportional asymptotic regime. The resulting distributional characterization provides confidence intervals and hypothesis tests for individual features and enables false-discovery-rate-controlled variable selection. Extensive simulations examine calibration, power, and variable-selection performance under a range of covariance structures, including strongly correlated designs. An analysis of high-dimensional breast cancer gene-expression data illustrates how the proposed inference can distinguish statistically significant features from variables selected by the original sparse SVM.
Peng Zeng, Hanwen Huang
Department of Mathematics and Statistics Auburn University, Auburn, AL 36849 · Department of Biostatistics, Data Science and Epidemiology Medical College of Georgia, Augusta University, Augusta, GA, 30912
Sparse precision matrix estimation provides an interpretable and computationally efficient framework for modeling conditional dependencies in high-dimensional, low-sample-size data. A recurring challenge is appropriately selecting the regularization parameter that controls estimator sparsity and strikes a balance between underfitting and overfitting. We propose a closed-form, matrix-valued regularization parameter derived from the sampling distribution of the first-order optimality conditions of the ℓ1-regularized Gaussian maximum-likelihood estimator. By prescribing the probability that each nonzero entry of the estimator satisfies its optimality condition under resampling, we eliminate the need for cross-validation. The resulting regularization parameter is shown to attain asymptotic scaling properties that, under standard conditions, provide consistency and sparsistency of the estimator. On synthetic Gaussian and non-Gaussian datasets, as well as real-world gene microarray and neuroimaging applications, the proposed approach achieves estimation accuracy comparable to cross-validation, delivers superior support recovery, and reduces runtime by several orders of magnitude.
Aryan Eftekhari, Daniel Sergio Vega, Ernst-Jan Camiel Wit +1
Institute of Computing, Faculty of Informatics, Universit`a della Svizzera italiana (USI), Lugano, Switzerland