CCMP: Class-Conditional Multicollinearity Pruning for High-Dimensional Feature Selection
Organizations: Department of Bioengineering, Northeastern University, Boston, MA, USA
Abstract
In high-dimensional scientific measurements, many features in a single sample often capture the same underlying physical attributes, producing severe multicollinearity. Supervised feature selectors fit on the entire dataset, so a feature can be informative for classification overall while still redundant with another feature in the same class. Class-conditional selectors, by contrast, trade some accuracy for a narrower, per-class view, since they never see data outside one class, but this lets new classes be added without refitting the whole model. They still cannot determine, however, which feature to drop from a correlated group. To close this gap, we propose classconditional multicollinearity pruning (CCMP), which iteratively prunes one feature from every highly correlated pair, per class. Four theoretical guarantees establish explicit bounds on the error induced by pruning under stated assumptions. Across four benchmark datasets spanning spectroscopic and hyperspectral imaging modalities, CCMP retains as few as 42 of 1531 features (3%) and as many as 60 of 145 (41%), outperforms every classconditional baseline, and remains competitive with supervised methods.
Figures & tables
| Dataset | Modality | |||
|---|---|---|---|---|
| Strawberry | FTIR | 983 | 235 | 2 |
| MRSA | MALDI-TOF | 2456 | 1000 | 2 |
| DRS Tissue | DRS | 7524 | 1531 | 11 |
| Botswana | Hyperion | 3248 | 145 | 14 |
| Strawberry FTIR (%) | MRSA MALDI-TOF (%) | DRS Tissue (%) | Botswana Hyperion (%) | |||||||||||||||||||||
| Feature Selector | LR | SVM | RF | LR | SVM | RF | LR | SVM | RF | LR | SVM | RF | ||||||||||||
| Full-feature | 95.1 | 93.7 | 96.5 | 95.1 | 75.3 | 80.8 | 78.6 | 78.3 | 95.2 | 82.1 | 88.6 | 88.6 | 96.9 | 91.9 | 93.5 | 94.1 | ||||||||
| Supervised selectors (matched at ) | ||||||||||||||||||||||||
| Lasso | 94.8 | 93.7 | 95.6 | 94.7 | 74.6 | 81.1 | 79.9 | 78.6 | 85.1 | 78.2 | 86.0 | 83.1 | 96.5 | 92.5 | 93.8 | 94.3 | ||||||||
| ElasticNet | 94.5 | 93.7 | 95.6 | 94.6 | 74.8 | 80.8 | 81.6 | 79.1 | 85.5 | 80.9 | 86.3 | 84.2 | 96.5 | 92.6 | 93.7 | 94.3 | ||||||||
| -SVM | 94.2 | 93.1 | 95.6 | 94.3 | 74.0 | 80.9 | 79.0 | 78.0 | 83.3 | 77.3 | 84.5 | 81.7 | 95.9 | 91.6 | 94.0 | 93.8 | ||||||||
| Dataset | Margin | |||
|---|---|---|---|---|
| Strawberry | 1/4 | |||
| Strawberry | 1/2 | |||
| Strawberry | 3/4 | |||
| Strawberry | 1 | |||
| MRSA | 1/4 | |||
| MRSA | 1/2 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Feature Selection
Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.
See all 213 papers