Unsupervised Feature Selection Using Local Intrinsic Dimensionality -- Extended Version
Authors: Oussama Chelly, Michael E. Houle, Ken-ichi Kawarabayashi, Muhammad Rajabinasab, Arthur Zimek
Organizations: Oratio Technologies · New Jersey Institute of Technology, United States · National Institute of Informatics, Japan · University of Tokyo, Japan · University of Southern Denmark, Denmark
High dimensionality degrades both the efficiency and the effectiveness of learning algorithms, and in many applications no class labels are available to guide dimensionality reduction. This paper proposes LIDFS, an unsupervised feature selection method based on Local Intrinsic Dimensionality (LID). The method builds on the formal equivalence between LID and the indiscriminability of distance measures. The data matrix is transposed so that each feature is treated as a point, and the LID of each feature is estimated by maximum likelihood from the distances to its k nearest neighboring features. Features with lower LID exhibit greater local discriminability and receive higher importance scores. A refinement step then retains one representative per feature neighborhood and demotes locally redundant features. The result is a complete ranking from which any number of features can be selected without rerunning the procedure. LIDFS does not use any information about labels or the number of classes, and runs in O(d2n) time for n instances and d features. LIDFS is compared with classic and state-of-the-art methods and a random baseline on 23 high-dimensional datasets, the classic benchmark collection. The evaluation uses classification accuracy and AUC with five downstream classifiers, as well as clustering accuracy and normalized mutual information, over both full and aggressive selection ranges. LIDFS ranks first in the supervised evaluations and first or tied second in the clustering-based evaluations. It belongs to the top statistical group together with methods that require the number of classes, and it runs faster than its closest competitor. Several recent methods, by contrast, fail to outperform the random baseline. A sensitivity analysis shows that small neighborhood sizes yield the best results.
Figures & tables
Figure 1: LID as neighborhood growth. Top: three point sets sampled so that FR(r)=rm holds exactly for m=1,2,4 , which is possible in the plane since LID depends on the distance distribution alone. Each disc starts from a neighborhood of the same size, ∣B(x,r)∣≈120 , and expands the radius by the same factor 1+ϵ=1.4 : the expansion dimension over the two concentric balls returns the LID, so it is the relative gain in points, not the absolute gain, that reveals the dimension. (a) On log-log axes, the slope of FR is the dimension. (b) The (1+ϵ) ratio across distances, with the MLE LIDFR from k=500 neighbors.
ID
Dataset
Instances
Features
D1
ALLAML
72
7,129
D2
arcene
200
10,000
D3
Carcinom
174
9,182
D4
CLL_SUB_111
111
11,340
D5
COIL20
1,440
1,024
D6
colon
62
2,000
Table 1: Characteristics of the high-dimensional benchmark datasets.
Figure 2: Runtime analysis of the feature selection methods. The y -axis is presented in logarithmic scale.
Figure 3: Comparison of the feature selection performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the full range of the number of selected features.
Figure 4: Comparison of the performance of the unsupervised feature selection methods on the Isolet dataset with the random baseline over the first 10% of the features.
Figure 5: Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the full (100%) selection scenarios. LIDFS constitutes the frontier almost consistently.
Figure 6: Critical difference diagram over the full range of number of features for the four different metrics based on the average performance measured by the FSDEM score.
Figure 7: Mean AUC, ACC, NMI, and CLSACC (FSDEM) for the aggressive (10%) selection scenarios. LIDFS constitutes the frontier almost consistently.
Figure 8: Critical difference diagram over the first 10% of number of features for the four different metrics based on the average performance measured by the FSDEM score.
Figure 9: Mean AUC, ACC, NMI, and CLSACC as a function of the neighborhood size k , averaged over datasets and over the feature-selection fraction, for the aggressive (10%) and full (100%) selection scenarios. Error bars show ±1 standard error of the mean over the 23 datasets. For all metrics higher values are better.
Figure 10: The performance of LIDFS with different choices of k on different datasets and metrics for the full (100%) selection scenario. Smaller values of k show a better overall performance.
Figure 11: The performance of LIDFS with different choices of k on different datasets and metrics for the aggressive (10%) selection scenario. Smaller values of k show a better overall performance.
Figure 12: Average ranks across the metrics for both scenarios. The preference for small neighborhoods is strongest for AUC and ACC; NMI and CLSACC are comparatively flat.
Appendix figures & tables282 assets
Supplementary material from the paper’s appendix.
Appendix
Range of the number of selected features
Classifier / metrics
0.5% to 10%
5% to 100%
KNN (ACC, AUC)
A1
A2
Decision tree (ACC, AUC)
A3
A4
SVM (ACC, AUC)
A5
A6
Random forest (ACC, AUC)
8 †
6 †
XGBoost (ACC, AUC)
A7
A8
Appendix
Table A1: Index of the supplementary critical difference diagrams. Each entry gives the number of the figure with the diagrams of ACC and AUC for the corresponding classifier and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.
Figure A1: Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (0.5% to 10% of the features).
Figure A2: Critical difference diagrams of ACC (left) and AUC (right) for the KNN classifier (5% to 100% of the features).
Figure A3: Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (0.5% to 10% of the features).
Figure A4: Critical difference diagrams of ACC (left) and AUC (right) for the decision tree classifier (5% to 100% of the features).
Figure A5: Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (0.5% to 10% of the features).
Figure A6: Critical difference diagrams of ACC (left) and AUC (right) for the SVM classifier (5% to 100% of the features).
Figure A7: Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (0.5% to 10% of the features).
Figure A8: Critical difference diagrams of ACC (left) and AUC (right) for the XGBoost classifier (5% to 100% of the features).
Unsupervised
Supervised (ACC, AUC)
CLSACC, NMI
KNN
Decision tree
SVM
Random forest
XGBoost
Dataset
10%
100%
10%
100%
10%
100%
10%
100%
10%
100%
10%
100%
ALLAML
A9
A31
A53
A76
A99
A122
A145
A168
A191
A213
A235
A258
arcene
A10
A32
A54
A77
A100
A123
A146
A169
A192
A214
A236
A259
Carcinom
A11
A33
A55
A78
A101
A124
A147
A170
A193
A215
A237
A260
CLL_SUB_111
A12
A34
A56
A79
A102
A125
A148
A171
A194
A216
A238
A261
Appendix
Table A2: Index of the supplementary performance plots. Each entry gives the number of the figure for the corresponding dataset, metric group or classifier, and range of the number of selected features (up to 10% or 100% of the features), and links to the figure. Entries marked with † refer to the main text.
Figure A9: CLSACC and NMI on the ALLAML dataset (0.5% to 10% of the features).
Figure A10: CLSACC and NMI on the arcene dataset (0.5% to 10% of the features).
Figure A11: CLSACC and NMI on the Carcinom dataset (0.5% to 10% of the features).
Figure A12: CLSACC and NMI on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A13: CLSACC and NMI on the COIL20 dataset (0.5% to 10% of the features).
Figure A14: CLSACC and NMI on the colon dataset (0.5% to 10% of the features).
Figure A15: CLSACC and NMI on the gisette dataset (0.5% to 10% of the features).
Figure A16: CLSACC and NMI on the GLI_85 dataset (0.5% to 10% of the features).
Figure A17: CLSACC and NMI on the GLIOMA dataset (0.5% to 10% of the features).
Figure A18: CLSACC and NMI on the leukemia dataset (0.5% to 10% of the features).
Figure A19: CLSACC and NMI on the lung dataset (0.5% to 10% of the features).
Figure A20: CLSACC and NMI on the lung_discrete dataset (0.5% to 10% of the features).
Figure A21: CLSACC and NMI on the madelon dataset (0.5% to 10% of the features).
Figure A22: CLSACC and NMI on the ORL dataset (0.5% to 10% of the features).
Figure A23: CLSACC and NMI on the orlraws10P dataset (0.5% to 10% of the features).
Figure A24: CLSACC and NMI on the pixraw10P dataset (0.5% to 10% of the features).
Figure A25: CLSACC and NMI on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A26: CLSACC and NMI on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A27: CLSACC and NMI on the TOX_171 dataset (0.5% to 10% of the features).
Figure A28: CLSACC and NMI on the warpAR10P dataset (0.5% to 10% of the features).
Figure A29: CLSACC and NMI on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A30: CLSACC and NMI on the Yale dataset (0.5% to 10% of the features).
Figure A31: CLSACC and NMI on the ALLAML dataset (5% to 100% of the features).
Figure A32: CLSACC and NMI on the arcene dataset (5% to 100% of the features).
Figure A33: CLSACC and NMI on the Carcinom dataset (5% to 100% of the features).
Figure A34: CLSACC and NMI on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A35: CLSACC and NMI on the COIL20 dataset (5% to 100% of the features).
Figure A36: CLSACC and NMI on the colon dataset (5% to 100% of the features).
Figure A37: CLSACC and NMI on the gisette dataset (5% to 100% of the features).
Figure A38: CLSACC and NMI on the GLI_85 dataset (5% to 100% of the features).
Figure A39: CLSACC and NMI on the GLIOMA dataset (5% to 100% of the features).
Figure A40: CLSACC and NMI on the leukemia dataset (5% to 100% of the features).
Figure A41: CLSACC and NMI on the lung dataset (5% to 100% of the features).
Figure A42: CLSACC and NMI on the lung_discrete dataset (5% to 100% of the features).
Figure A43: CLSACC and NMI on the madelon dataset (5% to 100% of the features).
Figure A44: CLSACC and NMI on the ORL dataset (5% to 100% of the features).
Figure A45: CLSACC and NMI on the orlraws10P dataset (5% to 100% of the features).
Figure A46: CLSACC and NMI on the pixraw10P dataset (5% to 100% of the features).
Figure A47: CLSACC and NMI on the Prostate_GE dataset (5% to 100% of the features).
Figure A48: CLSACC and NMI on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A49: CLSACC and NMI on the TOX_171 dataset (5% to 100% of the features).
Figure A50: CLSACC and NMI on the warpAR10P dataset (5% to 100% of the features).
Figure A51: CLSACC and NMI on the warpPIE10P dataset (5% to 100% of the features).
Figure A52: CLSACC and NMI on the Yale dataset (5% to 100% of the features).
Figure A53: ACC and AUC of the KNN classifier on the ALLAML dataset (0.5% to 10% of the features).
Figure A54: ACC and AUC of the KNN classifier on the arcene dataset (0.5% to 10% of the features).
Figure A55: ACC and AUC of the KNN classifier on the Carcinom dataset (0.5% to 10% of the features).
Figure A56: ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A57: ACC and AUC of the KNN classifier on the COIL20 dataset (0.5% to 10% of the features).
Figure A58: ACC and AUC of the KNN classifier on the colon dataset (0.5% to 10% of the features).
Figure A59: ACC and AUC of the KNN classifier on the gisette dataset (0.5% to 10% of the features).
Figure A60: ACC and AUC of the KNN classifier on the GLI_85 dataset (0.5% to 10% of the features).
Figure A61: ACC and AUC of the KNN classifier on the GLIOMA dataset (0.5% to 10% of the features).
Figure A62: ACC and AUC of the KNN classifier on the Isolet dataset (0.5% to 10% of the features).
Figure A63: ACC and AUC of the KNN classifier on the leukemia dataset (0.5% to 10% of the features).
Figure A64: ACC and AUC of the KNN classifier on the lung dataset (0.5% to 10% of the features).
Figure A65: ACC and AUC of the KNN classifier on the lung_discrete dataset (0.5% to 10% of the features).
Figure A66: ACC and AUC of the KNN classifier on the madelon dataset (0.5% to 10% of the features).
Figure A67: ACC and AUC of the KNN classifier on the ORL dataset (0.5% to 10% of the features).
Figure A68: ACC and AUC of the KNN classifier on the orlraws10P dataset (0.5% to 10% of the features).
Figure A69: ACC and AUC of the KNN classifier on the pixraw10P dataset (0.5% to 10% of the features).
Figure A70: ACC and AUC of the KNN classifier on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A71: ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A72: ACC and AUC of the KNN classifier on the TOX_171 dataset (0.5% to 10% of the features).
Figure A73: ACC and AUC of the KNN classifier on the warpAR10P dataset (0.5% to 10% of the features).
Figure A74: ACC and AUC of the KNN classifier on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A75: ACC and AUC of the KNN classifier on the Yale dataset (0.5% to 10% of the features).
Figure A76: ACC and AUC of the KNN classifier on the ALLAML dataset (5% to 100% of the features).
Figure A77: ACC and AUC of the KNN classifier on the arcene dataset (5% to 100% of the features).
Figure A78: ACC and AUC of the KNN classifier on the Carcinom dataset (5% to 100% of the features).
Figure A79: ACC and AUC of the KNN classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A80: ACC and AUC of the KNN classifier on the COIL20 dataset (5% to 100% of the features).
Figure A81: ACC and AUC of the KNN classifier on the colon dataset (5% to 100% of the features).
Figure A82: ACC and AUC of the KNN classifier on the gisette dataset (5% to 100% of the features).
Figure A83: ACC and AUC of the KNN classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A84: ACC and AUC of the KNN classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A85: ACC and AUC of the KNN classifier on the Isolet dataset (5% to 100% of the features).
Figure A86: ACC and AUC of the KNN classifier on the leukemia dataset (5% to 100% of the features).
Figure A87: ACC and AUC of the KNN classifier on the lung dataset (5% to 100% of the features).
Figure A88: ACC and AUC of the KNN classifier on the lung_discrete dataset (5% to 100% of the features).
Figure A89: ACC and AUC of the KNN classifier on the madelon dataset (5% to 100% of the features).
Figure A90: ACC and AUC of the KNN classifier on the ORL dataset (5% to 100% of the features).
Figure A91: ACC and AUC of the KNN classifier on the orlraws10P dataset (5% to 100% of the features).
Figure A92: ACC and AUC of the KNN classifier on the pixraw10P dataset (5% to 100% of the features).
Figure A93: ACC and AUC of the KNN classifier on the Prostate_GE dataset (5% to 100% of the features).
Figure A94: ACC and AUC of the KNN classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A95: ACC and AUC of the KNN classifier on the TOX_171 dataset (5% to 100% of the features).
Figure A96: ACC and AUC of the KNN classifier on the warpAR10P dataset (5% to 100% of the features).
Figure A97: ACC and AUC of the KNN classifier on the warpPIE10P dataset (5% to 100% of the features).
Figure A98: ACC and AUC of the KNN classifier on the Yale dataset (5% to 100% of the features).
Figure A99: ACC and AUC of the decision tree classifier on the ALLAML dataset (0.5% to 10% of the features).
Figure A100: ACC and AUC of the decision tree classifier on the arcene dataset (0.5% to 10% of the features).
Figure A101: ACC and AUC of the decision tree classifier on the Carcinom dataset (0.5% to 10% of the features).
Figure A102: ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A103: ACC and AUC of the decision tree classifier on the COIL20 dataset (0.5% to 10% of the features).
Figure A104: ACC and AUC of the decision tree classifier on the colon dataset (0.5% to 10% of the features).
Figure A105: ACC and AUC of the decision tree classifier on the gisette dataset (0.5% to 10% of the features).
Figure A106: ACC and AUC of the decision tree classifier on the GLI_85 dataset (0.5% to 10% of the features).
Figure A107: ACC and AUC of the decision tree classifier on the GLIOMA dataset (0.5% to 10% of the features).
Figure A108: ACC and AUC of the decision tree classifier on the Isolet dataset (0.5% to 10% of the features).
Figure A109: ACC and AUC of the decision tree classifier on the leukemia dataset (0.5% to 10% of the features).
Figure A110: ACC and AUC of the decision tree classifier on the lung dataset (0.5% to 10% of the features).
Figure A111: ACC and AUC of the decision tree classifier on the lung_discrete dataset (0.5% to 10% of the features).
Figure A112: ACC and AUC of the decision tree classifier on the madelon dataset (0.5% to 10% of the features).
Figure A113: ACC and AUC of the decision tree classifier on the ORL dataset (0.5% to 10% of the features).
Figure A114: ACC and AUC of the decision tree classifier on the orlraws10P dataset (0.5% to 10% of the features).
Figure A115: ACC and AUC of the decision tree classifier on the pixraw10P dataset (0.5% to 10% of the features).
Figure A116: ACC and AUC of the decision tree classifier on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A117: ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A118: ACC and AUC of the decision tree classifier on the TOX_171 dataset (0.5% to 10% of the features).
Figure A119: ACC and AUC of the decision tree classifier on the warpAR10P dataset (0.5% to 10% of the features).
Figure A120: ACC and AUC of the decision tree classifier on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A121: ACC and AUC of the decision tree classifier on the Yale dataset (0.5% to 10% of the features).
Figure A122: ACC and AUC of the decision tree classifier on the ALLAML dataset (5% to 100% of the features).
Figure A123: ACC and AUC of the decision tree classifier on the arcene dataset (5% to 100% of the features).
Figure A124: ACC and AUC of the decision tree classifier on the Carcinom dataset (5% to 100% of the features).
Figure A125: ACC and AUC of the decision tree classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A126: ACC and AUC of the decision tree classifier on the COIL20 dataset (5% to 100% of the features).
Figure A127: ACC and AUC of the decision tree classifier on the colon dataset (5% to 100% of the features).
Figure A128: ACC and AUC of the decision tree classifier on the gisette dataset (5% to 100% of the features).
Figure A129: ACC and AUC of the decision tree classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A130: ACC and AUC of the decision tree classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A131: ACC and AUC of the decision tree classifier on the Isolet dataset (5% to 100% of the features).
Figure A132: ACC and AUC of the decision tree classifier on the leukemia dataset (5% to 100% of the features).
Figure A133: ACC and AUC of the decision tree classifier on the lung dataset (5% to 100% of the features).
Figure A134: ACC and AUC of the decision tree classifier on the lung_discrete dataset (5% to 100% of the features).
Figure A135: ACC and AUC of the decision tree classifier on the madelon dataset (5% to 100% of the features).
Figure A136: ACC and AUC of the decision tree classifier on the ORL dataset (5% to 100% of the features).
Figure A137: ACC and AUC of the decision tree classifier on the orlraws10P dataset (5% to 100% of the features).
Figure A138: ACC and AUC of the decision tree classifier on the pixraw10P dataset (5% to 100% of the features).
Figure A139: ACC and AUC of the decision tree classifier on the Prostate_GE dataset (5% to 100% of the features).
Figure A140: ACC and AUC of the decision tree classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A141: ACC and AUC of the decision tree classifier on the TOX_171 dataset (5% to 100% of the features).
Figure A142: ACC and AUC of the decision tree classifier on the warpAR10P dataset (5% to 100% of the features).
Figure A143: ACC and AUC of the decision tree classifier on the warpPIE10P dataset (5% to 100% of the features).
Figure A144: ACC and AUC of the decision tree classifier on the Yale dataset (5% to 100% of the features).
Figure A145: ACC and AUC of the SVM classifier on the ALLAML dataset (0.5% to 10% of the features).
Figure A146: ACC and AUC of the SVM classifier on the arcene dataset (0.5% to 10% of the features).
Figure A147: ACC and AUC of the SVM classifier on the Carcinom dataset (0.5% to 10% of the features).
Figure A148: ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A149: ACC and AUC of the SVM classifier on the COIL20 dataset (0.5% to 10% of the features).
Figure A150: ACC and AUC of the SVM classifier on the colon dataset (0.5% to 10% of the features).
Figure A151: ACC and AUC of the SVM classifier on the gisette dataset (0.5% to 10% of the features).
Figure A152: ACC and AUC of the SVM classifier on the GLI_85 dataset (0.5% to 10% of the features).
Figure A153: ACC and AUC of the SVM classifier on the GLIOMA dataset (0.5% to 10% of the features).
Figure A154: ACC and AUC of the SVM classifier on the Isolet dataset (0.5% to 10% of the features).
Figure A155: ACC and AUC of the SVM classifier on the leukemia dataset (0.5% to 10% of the features).
Figure A156: ACC and AUC of the SVM classifier on the lung dataset (0.5% to 10% of the features).
Figure A157: ACC and AUC of the SVM classifier on the lung_discrete dataset (0.5% to 10% of the features).
Figure A158: ACC and AUC of the SVM classifier on the madelon dataset (0.5% to 10% of the features).
Figure A159: ACC and AUC of the SVM classifier on the ORL dataset (0.5% to 10% of the features).
Figure A160: ACC and AUC of the SVM classifier on the orlraws10P dataset (0.5% to 10% of the features).
Figure A161: ACC and AUC of the SVM classifier on the pixraw10P dataset (0.5% to 10% of the features).
Figure A162: ACC and AUC of the SVM classifier on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A163: ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A164: ACC and AUC of the SVM classifier on the TOX_171 dataset (0.5% to 10% of the features).
Figure A165: ACC and AUC of the SVM classifier on the warpAR10P dataset (0.5% to 10% of the features).
Figure A166: ACC and AUC of the SVM classifier on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A167: ACC and AUC of the SVM classifier on the Yale dataset (0.5% to 10% of the features).
Figure A168: ACC and AUC of the SVM classifier on the ALLAML dataset (5% to 100% of the features).
Figure A169: ACC and AUC of the SVM classifier on the arcene dataset (5% to 100% of the features).
Figure A170: ACC and AUC of the SVM classifier on the Carcinom dataset (5% to 100% of the features).
Figure A171: ACC and AUC of the SVM classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A172: ACC and AUC of the SVM classifier on the COIL20 dataset (5% to 100% of the features).
Figure A173: ACC and AUC of the SVM classifier on the colon dataset (5% to 100% of the features).
Figure A174: ACC and AUC of the SVM classifier on the gisette dataset (5% to 100% of the features).
Figure A175: ACC and AUC of the SVM classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A176: ACC and AUC of the SVM classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A177: ACC and AUC of the SVM classifier on the Isolet dataset (5% to 100% of the features).
Figure A178: ACC and AUC of the SVM classifier on the leukemia dataset (5% to 100% of the features).
Figure A179: ACC and AUC of the SVM classifier on the lung dataset (5% to 100% of the features).
Figure A180: ACC and AUC of the SVM classifier on the lung_discrete dataset (5% to 100% of the features).
Figure A181: ACC and AUC of the SVM classifier on the madelon dataset (5% to 100% of the features).
Figure A182: ACC and AUC of the SVM classifier on the ORL dataset (5% to 100% of the features).
Figure A183: ACC and AUC of the SVM classifier on the orlraws10P dataset (5% to 100% of the features).
Figure A184: ACC and AUC of the SVM classifier on the pixraw10P dataset (5% to 100% of the features).
Figure A185: ACC and AUC of the SVM classifier on the Prostate_GE dataset (5% to 100% of the features).
Figure A186: ACC and AUC of the SVM classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A187: ACC and AUC of the SVM classifier on the TOX_171 dataset (5% to 100% of the features).
Figure A188: ACC and AUC of the SVM classifier on the warpAR10P dataset (5% to 100% of the features).
Figure A189: ACC and AUC of the SVM classifier on the warpPIE10P dataset (5% to 100% of the features).
Figure A190: ACC and AUC of the SVM classifier on the Yale dataset (5% to 100% of the features).
Figure A191: ACC and AUC of the random forest classifier on the ALLAML dataset (0.5% to 10% of the features).
Figure A192: ACC and AUC of the random forest classifier on the arcene dataset (0.5% to 10% of the features).
Figure A193: ACC and AUC of the random forest classifier on the Carcinom dataset (0.5% to 10% of the features).
Figure A194: ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A195: ACC and AUC of the random forest classifier on the COIL20 dataset (0.5% to 10% of the features).
Figure A196: ACC and AUC of the random forest classifier on the colon dataset (0.5% to 10% of the features).
Figure A197: ACC and AUC of the random forest classifier on the gisette dataset (0.5% to 10% of the features).
Figure A198: ACC and AUC of the random forest classifier on the GLI_85 dataset (0.5% to 10% of the features).
Figure A199: ACC and AUC of the random forest classifier on the GLIOMA dataset (0.5% to 10% of the features).
Figure A200: ACC and AUC of the random forest classifier on the leukemia dataset (0.5% to 10% of the features).
Figure A201: ACC and AUC of the random forest classifier on the lung dataset (0.5% to 10% of the features).
Figure A202: ACC and AUC of the random forest classifier on the lung_discrete dataset (0.5% to 10% of the features).
Figure A203: ACC and AUC of the random forest classifier on the madelon dataset (0.5% to 10% of the features).
Figure A204: ACC and AUC of the random forest classifier on the ORL dataset (0.5% to 10% of the features).
Figure A205: ACC and AUC of the random forest classifier on the orlraws10P dataset (0.5% to 10% of the features).
Figure A206: ACC and AUC of the random forest classifier on the pixraw10P dataset (0.5% to 10% of the features).
Figure A207: ACC and AUC of the random forest classifier on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A208: ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A209: ACC and AUC of the random forest classifier on the TOX_171 dataset (0.5% to 10% of the features).
Figure A210: ACC and AUC of the random forest classifier on the warpAR10P dataset (0.5% to 10% of the features).
Figure A211: ACC and AUC of the random forest classifier on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A212: ACC and AUC of the random forest classifier on the Yale dataset (0.5% to 10% of the features).
Figure A213: ACC and AUC of the random forest classifier on the ALLAML dataset (5% to 100% of the features).
Figure A214: ACC and AUC of the random forest classifier on the arcene dataset (5% to 100% of the features).
Figure A215: ACC and AUC of the random forest classifier on the Carcinom dataset (5% to 100% of the features).
Figure A216: ACC and AUC of the random forest classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A217: ACC and AUC of the random forest classifier on the COIL20 dataset (5% to 100% of the features).
Figure A218: ACC and AUC of the random forest classifier on the colon dataset (5% to 100% of the features).
Figure A219: ACC and AUC of the random forest classifier on the gisette dataset (5% to 100% of the features).
Figure A220: ACC and AUC of the random forest classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A221: ACC and AUC of the random forest classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A222: ACC and AUC of the random forest classifier on the leukemia dataset (5% to 100% of the features).
Figure A223: ACC and AUC of the random forest classifier on the lung dataset (5% to 100% of the features).
Figure A224: ACC and AUC of the random forest classifier on the lung_discrete dataset (5% to 100% of the features).
Figure A225: ACC and AUC of the random forest classifier on the madelon dataset (5% to 100% of the features).
Figure A226: ACC and AUC of the random forest classifier on the ORL dataset (5% to 100% of the features).
Figure A227: ACC and AUC of the random forest classifier on the orlraws10P dataset (5% to 100% of the features).
Figure A228: ACC and AUC of the random forest classifier on the pixraw10P dataset (5% to 100% of the features).
Figure A229: ACC and AUC of the random forest classifier on the Prostate_GE dataset (5% to 100% of the features).
Figure A230: ACC and AUC of the random forest classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A231: ACC and AUC of the random forest classifier on the TOX_171 dataset (5% to 100% of the features).
Figure A232: ACC and AUC of the random forest classifier on the warpAR10P dataset (5% to 100% of the features).
Figure A233: ACC and AUC of the random forest classifier on the warpPIE10P dataset (5% to 100% of the features).
Figure A234: ACC and AUC of the random forest classifier on the Yale dataset (5% to 100% of the features).
Figure A235: ACC and AUC of the XGBoost classifier on the ALLAML dataset (0.5% to 10% of the features).
Figure A236: ACC and AUC of the XGBoost classifier on the arcene dataset (0.5% to 10% of the features).
Figure A237: ACC and AUC of the XGBoost classifier on the Carcinom dataset (0.5% to 10% of the features).
Figure A238: ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (0.5% to 10% of the features).
Figure A239: ACC and AUC of the XGBoost classifier on the COIL20 dataset (0.5% to 10% of the features).
Figure A240: ACC and AUC of the XGBoost classifier on the colon dataset (0.5% to 10% of the features).
Figure A241: ACC and AUC of the XGBoost classifier on the gisette dataset (0.5% to 10% of the features).
Figure A242: ACC and AUC of the XGBoost classifier on the GLI_85 dataset (0.5% to 10% of the features).
Figure A243: ACC and AUC of the XGBoost classifier on the GLIOMA dataset (0.5% to 10% of the features).
Figure A244: ACC and AUC of the XGBoost classifier on the Isolet dataset (0.5% to 10% of the features).
Figure A245: ACC and AUC of the XGBoost classifier on the leukemia dataset (0.5% to 10% of the features).
Figure A246: ACC and AUC of the XGBoost classifier on the lung dataset (0.5% to 10% of the features).
Figure A247: ACC and AUC of the XGBoost classifier on the lung_discrete dataset (0.5% to 10% of the features).
Figure A248: ACC and AUC of the XGBoost classifier on the madelon dataset (0.5% to 10% of the features).
Figure A249: ACC and AUC of the XGBoost classifier on the ORL dataset (0.5% to 10% of the features).
Figure A250: ACC and AUC of the XGBoost classifier on the orlraws10P dataset (0.5% to 10% of the features).
Figure A251: ACC and AUC of the XGBoost classifier on the pixraw10P dataset (0.5% to 10% of the features).
Figure A252: ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (0.5% to 10% of the features).
Figure A253: ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (0.5% to 10% of the features).
Figure A254: ACC and AUC of the XGBoost classifier on the TOX_171 dataset (0.5% to 10% of the features).
Figure A255: ACC and AUC of the XGBoost classifier on the warpAR10P dataset (0.5% to 10% of the features).
Figure A256: ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (0.5% to 10% of the features).
Figure A257: ACC and AUC of the XGBoost classifier on the Yale dataset (0.5% to 10% of the features).
Figure A258: ACC and AUC of the XGBoost classifier on the ALLAML dataset (5% to 100% of the features).
Figure A259: ACC and AUC of the XGBoost classifier on the arcene dataset (5% to 100% of the features).
Figure A260: ACC and AUC of the XGBoost classifier on the Carcinom dataset (5% to 100% of the features).
Figure A261: ACC and AUC of the XGBoost classifier on the CLL_SUB_111 dataset (5% to 100% of the features).
Figure A262: ACC and AUC of the XGBoost classifier on the COIL20 dataset (5% to 100% of the features).
Figure A263: ACC and AUC of the XGBoost classifier on the colon dataset (5% to 100% of the features).
Figure A264: ACC and AUC of the XGBoost classifier on the gisette dataset (5% to 100% of the features).
Figure A265: ACC and AUC of the XGBoost classifier on the GLI_85 dataset (5% to 100% of the features).
Figure A266: ACC and AUC of the XGBoost classifier on the GLIOMA dataset (5% to 100% of the features).
Figure A267: ACC and AUC of the XGBoost classifier on the Isolet dataset (5% to 100% of the features).
Figure A268: ACC and AUC of the XGBoost classifier on the leukemia dataset (5% to 100% of the features).
Figure A269: ACC and AUC of the XGBoost classifier on the lung dataset (5% to 100% of the features).
Figure A270: ACC and AUC of the XGBoost classifier on the lung_discrete dataset (5% to 100% of the features).
Figure A271: ACC and AUC of the XGBoost classifier on the madelon dataset (5% to 100% of the features).
Figure A272: ACC and AUC of the XGBoost classifier on the ORL dataset (5% to 100% of the features).
Figure A273: ACC and AUC of the XGBoost classifier on the orlraws10P dataset (5% to 100% of the features).
Figure A274: ACC and AUC of the XGBoost classifier on the pixraw10P dataset (5% to 100% of the features).
Figure A275: ACC and AUC of the XGBoost classifier on the Prostate_GE dataset (5% to 100% of the features).
Figure A276: ACC and AUC of the XGBoost classifier on the SMK_CAN_187 dataset (5% to 100% of the features).
Figure A277: ACC and AUC of the XGBoost classifier on the TOX_171 dataset (5% to 100% of the features).
Figure A278: ACC and AUC of the XGBoost classifier on the warpAR10P dataset (5% to 100% of the features).
Figure A279: ACC and AUC of the XGBoost classifier on the warpPIE10P dataset (5% to 100% of the features).
Figure A280: ACC and AUC of the XGBoost classifier on the Yale dataset (5% to 100% of the features).
Unsupervised feature selection seeks a compact subset of informative features without access to class labels, making feature utility difficult to define. Existing UFS methods therefore rely on indirect structural criteria, such as similarity preservation, locality, sparsity, cluster geometry, or reconstruction quality. In this paper, we instead study UFS through representation consistency and propose Inverted Contrastive Learning for Unsupervised Feature Selection (ICLFS), a feature-wise contrastive framework that reformulates UFS as a representation learning problem over features rather than samples. ICLFS first inverts the data matrix so that each feature is represented by its sample-profile vector, then constructs multiple masked positive views together with a shuffled negative view, and learns projector-space representations that remain consistent across these structured perturbations under an InfoNCE-based objective. Motivated by recent findings that cosine-based and InfoNCE-based training affect embedding norms, we use projector-space embedding magnitude as the saliency signal for ranking features. The resulting norm-based ranking is subsequently refined through Laplacian-Gated Ranking Correction, which suppresses locally redundant candidates while preserving salient ones. Extensive experiments on 12 benchmark datasets show that ICLFS achieves the best clustering accuracy on 10 datasets against both classical and neural baselines under the standard clustering-based UFS evaluation protocol, while remaining competitive on the other two. These results show that feature-wise contrastive representation consistency provides a strong and effective alternative to neighborhood, cluster, and reconstruction-based UFS formulations.
Utsab Ghosh, Roshni Chakraborty
ABV-Indian Institute of Information Technology and Management, Gwalior, India.
Many novel unsupervised feature selection methods are proposed each year, yet their empirical evaluation is limited to supervised and unsupervised evaluation metrics computed on selected datasets, along with comparisons to existing methods. However, in the absence of an established evaluation baseline, it is difficult to determine the value added to the existing literature by each of these methods, and how effective their underlying approaches are. In this evaluation study, we are using random feature selection as a baseline for evaluating unsupervised feature selection methods. While a random selection of features might seem to be a rather weak baseline, we show in this evaluation study that many of the state-of-the-art methods in unsupervised feature selection are outperformed by random feature selection in both performance and efficiency. Accordingly, we emphasize the strict requirement of considering random feature selection as a baseline in the development process of novel unsupervised feature selection methods, and we encourage the community to develop better evaluation measures for feature selection. Both would help to ensure consistent improvement and realistic assessment.
Muhammad Rajabinasab, Michael E. Houle, Oussama Chelly +1
University of Southern Denmark, Odense, Denmark · New Jersey Institute of Technology, Newark, NJ, United States · Oratio Technologies, Tunis, Tunisia
Feature selection is one of the most important and fundamental tasks in data mining, tackled by a family of methods with an established set of evaluation techniques to measure the quality of a specific method. Most of the methods commonly used for the unsupervised evaluation of feature selection algorithms suffer from critical design flaws which question their unsupervised nature. In this paper, we provide a critical discussion on the established allegedly unsupervised evaluation techniques, and shed light on the reasons why they are not truly unsupervised but, at best, supervised evaluation under an unsupervised downstream task. We also propose a novel, truly unsupervised evaluation framework to measure the quality of the feature selection algorithms without any form of information about the labels. The proposed framework utilizes unsupervised Principal Component Analysis, and optimal transport to measure the quality of the feature selection methods in a truly unsupervised manner.
Hafiz Saud Arshad, Muhammad Rajabinasab, Arthur Zimek