In high-dimensional scientific measurements, many features in a single sample often capture the same underlying physical attributes, producing severe multicollinearity. Supervised feature selectors fit on the entire dataset, so a feature can be informative for classification overall while still redundant with another feature in the same class. Class-conditional selectors, by contrast, trade some accuracy for a narrower, per-class view, since they never see data outside one class, but this lets new classes be added without refitting the whole model. They still cannot determine, however, which feature to drop from a correlated group. To close this gap, we propose classconditional multicollinearity pruning (CCMP), which iteratively prunes one feature from every highly correlated pair, per class. Four theoretical guarantees establish explicit bounds on the error induced by pruning under stated assumptions. Across four benchmark datasets spanning spectroscopic and hyperspectral imaging modalities, CCMP retains as few as 42 of 1531 features (3%) and as many as 60 of 145 (41%), outperforms every classconditional baseline, and remains competitive with supervised methods.
Figures & tables
Fig. 1: Overview of CCMP. Training data Xtr with labels y produce per-class absolute Pearson matrices ∣C(y)∣ . For each candidate cutoff τ in a grid search, greedy pruning runs per class, and the retained features are unioned across classes into the kept feature set Qτ . A geometric elbow rule on the Redundancy Retention Score κ(τ) then selects τ⋆ , giving the final kept feature set Qτ⋆ for the downstream classifier.
Fig. 2: CCMP on a d=4 example. (a) Per-class absolute correlation matrices. Orange cells exceed τ⋆ . Sweeping τ tries each candidate against these matrices, with τ⋆=0.80 highlighted as the value the elbow rule selects in panel (c). (b) Greedy pruning discards one feature from each over-threshold pair, per class. Class 1 drops f3 and keeps {f1,f2,f4} , while class 2 drops f3 then f4 and keeps {f1,f2} . (c) The union of both classes’ kept sets retains {f1,f2,f4} . f3 is dropped by both classes and excluded. The elbow rule selects τ⋆ from the κ(τ) curve, and the retained features are passed to the classifier, shown schematically with an arbitrary decision boundary separating the two classes.
Dataset
Modality
N
d
C
Strawberry
FTIR
983
235
2
MRSA
MALDI-TOF
2456
1000
2
DRS Tissue
DRS
7524
1531
11
Botswana
Hyperion
3248
145
14
TABLE I: Benchmark datasets used for evaluation.
Strawberry FTIR (%) d=235,k=55
MRSA MALDI-TOF (%) d=1000,k=341
DRS Tissue (%) d=1531,k=42
Botswana Hyperion (%) d=145,k=60
Feature Selector
LR
SVM
RF
F1
Δfull
Δsup
LR
SVM
RF
F1
Δfull
Δsup
LR
SVM
RF
F1
Δfull
Δsup
LR
SVM
RF
F1
Δfull
Δsup
Full-feature
95.1
93.7
96.5
95.1
0.0
+0.4
75.3
80.8
78.6
78.3
0.0
−0.8
95.2
82.1
88.6
88.6
0.0
+4.4
96.9
91.9
93.5
94.1
0.0
−0.1
Supervised selectors (matched at k )
Lasso
94.8
93.7
95.6
94.7
−0.4
0.0
74.6
81.1
79.9
78.6
+0.3
−0.5
85.1
78.2
86.0
83.1
−5.5
−1.1
96.5
92.5
93.8
94.3
+0.1
0.0
ElasticNet
94.5
93.7
95.6
94.6
−0.5
−0.1
74.8
80.8
81.6
79.1
+0.8
0.0
85.5
80.9
86.3
84.2
−4.4
0.0
96.5
92.6
93.7
94.3
+0.1
0.0
ℓ1 -SVM
94.2
93.1
95.6
94.3
−0.7
−0.4
74.0
80.9
79.0
78.0
−0.3
−1.1
83.3
77.3
84.5
81.7
−6.9
−2.5
95.9
91.6
94.0
93.8
−0.3
−0.4
TABLE II: Macro-F1 (%, one decimal) for supervised and class-conditional selectors on Strawberry, MRSA, DRS Tissue, and Botswana, each matched to CCMP’s feature budget k=∣Qτ⋆∣ . Columns report LR, SVM, RF, their mean F1 (Avg), and two gaps: Δfull:=F1−F1full to the full-feature baseline, and Δsup:=F1−F1sup⋆ to the best supervised selector, where a positive Δ means the row beats that reference. Bold marks CCMP, underline marks the best value per column, and a box marks the reference value used for each Δ .
Fig. 3: Each panel shows a dataset’s per-class mean profile from the training split, with green shading marking the features CCMP retains. (a) Strawberry FTIR (2 classes): τ⋆=0.90 , k=55 of 235 channels. (b) MRSA MALDI-TOF (2 classes): τ⋆=0.40 , k=341 of 1000 bins. (c) DRS Tissue (11 classes): τ⋆=0.85 , k=42 of 1531 channels. (d) Botswana Hyperion (14 classes): τ⋆=0.80 , k=60 of 145 bands.
Fig. 4: Ablation pipeline. The dashed criterion block is the ablation point. Sweeping τ and pruning per class generates candidate feature sets, and the criterion selects τ⋆ from BIC, AIC, or a geometric elbow rule. All other stages follow the proposed pipeline in Fig. 1 .
Fig. 5: Redundancy Retention Score κ(τ) against the retained-feature count k for each dataset, swept over the candidate grid T={0.20,0.25,…,0.95,0.99} . The gray labels mark the two endpoints of this sweep, τ=0.20 and τ=0.99 . Markers show where BIC, AIC, and the geometric elbow rule each select τ⋆ along this curve.
Fig. 6: Criterion ablation. F1 is the mean over LR, SVM, and RF. For each dataset, BIC (orange square), the proposed geometric elbow (blue cross), and AIC (purple triangle) each select a τ⋆ and retained-feature budget k , annotated above or below each marker. The dashed line marks the full-feature baseline, labeled with the total (unpruned) feature count k=d . The elbow criterion’s marker remains closest to this baseline across all four datasets.
Dataset
α
LHSsup
RHSsup
Margin
Strawberry
1/4
1.20×10−1
1.03×10−1
+1.73×10−2
Strawberry
1/2
1.20×10−1
1.05×10−1
+1.53×10−2
Strawberry
3/4
1.20×10−1
1.06×10−1
+1.40×10−2
Strawberry
1
1.20×10−1
1.07×10−1
+1.33×10−2
MRSA
1/4
2.87×100
1.03×10−1
+2.77×100
MRSA
1/2
2.87×100
2.85×10−1
+2.59×100
TABLE III: Empirical validation of the irreducibility bound (Theorem 1 ) using the closed-form Hölder penalty from Corollary 1 . For each dataset and Hölder exponent α , Monte Carlo LHSsup is compared with analytical RHSsup . A nonnegative margin confirms the bound holds.
Figure 11Figure 12Figure 13
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Σy−:=ny−11(ΔU(y))⊤(ΔU(y))∈Rq×q.
Appendix
Algorithm 2 Empirical estimates of the irreducibility bound components
Feature rankings are widely used in supervised feature selection because they are simple, scalable and easy to interpret. Variables are first ranked by a relevance score, and a subset is then obtained by retaining the top-ranked variables. Although the first stage has been extensively studied, the second is often governed by an arbitrary cardinality, an empirical threshold or cross-validation, without a direct interpretation. This raises a basic question: given a feature ranking, when is there enough accumulated class-separation evidence to stop selecting features? This paper develops a distributional framework for transforming supervised feature rankings into class-independent subsets through an explicit risk-calibrated stopping rule. For each variable and each pair of classes, marginal separation is measured by the Bhattacharyya coefficient between the corresponding class-conditional distributions. The proposed method selects a single global subset shared by all classes by retaining the shortest prefix of a ranking whose residual product overlap falls below a prescribed threshold for every relevant class contrast. We derive binary and multiclass Bayes-risk bounds for the labelled product marginal problem, and obtain prior-dependent and prior-free calibrations of the residual-overlap threshold from a target all-pairs risk level. An empirical comparison on high-dimensional genomic datasets illustrates that the rule can reduce tens of thousands of variables to a few dozen while maintaining predictive performance statistically comparable to the all-features baseline. As the stopping rule only requires one-dimensional marginal overlap estimates and scans a precomputed ranking, it is well suited to very high-dimensional settings where exhaustive subset search is infeasible and interpretable truncation of feature rankings is essential.
Jesus S. Aguilar-Ruiz
School of Engineering Pablo de Olavide University ES-41013 Seville, Spain
Weighted sums of random kitchen sinks (W-RKS) is a commonly used feature extraction technique for supervised learning tasks. A major drawback of this method is the redundancy and ineffectiveness of many extracted random features. This paper investigates efficient methods for reducing such redundancy in multi-class classification problems. We specifically formulate feature selection as a variant of the sequentially nested model selection problem and introduce the Successive Empirical Excess Risk Difference Maximization (SEED-MAX) algorithm. It recursively computes empirical risks to identify the most parsimonious yet informative set of features. We derive an analytical lower bound for the probability of correct feature selection by the SEED-MAX algorithm. This lower bound on the probability of correct feature selection can be made arbitrarily close to one by increasing the training dataset size. To further enhance the rate of pruning, we evaluate the features retained by SEED-MAX using the Fisher test (F-Test). Empirical evaluations on UCR time-series classification datasets show that the sequential application of SEED-MAX with the Fisher test achieves approximately 92 percent rate of pruning with negligible degradation in classification accuracy. Implementations of the SEED-MAX and SEED-MAX-F-Test methods are available at https://github.com/OmidiAmin/seed-max-ftest.
Mohammad Amin Omidi, Babak Seyfe, Shahrokh Valaee
Information Theoretic Learning Systems Laboratory (ITLSL), Department of Electrical Engineering, Shahed University, Tehran, Iran · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada
Feature selection is crucial for pinpointing relevant features in high-dimensional datasets, mitigating the 'curse of dimensionality,' and enhancing machine learning performance. Traditional feature selection methods for classification use data from all classes to select features for each class. This paper explores feature selection methods that select features for each class separately, using class models based on low-rank generative methods and introducing a signal-to-noise ratio (SNR) feature selection criterion. This novel approach offers a theoretically sound guarantee of feature recovery under specific assumptions and has been demonstrated to outperform some existing feature selection methods on standard classification datasets. Additionally, it can be easily applied in a class-incremental learning setup, yielding performance comparable to that of some current methods.
Rittwika Kansabanik, Adrian Barbu
Department of Statistics, Florida State University