Feature Selection for Latent Factor Models

Rittwika Kansabanik; Adrian Barbu

cs.LGDec 13, 2024

Feature Selection for Latent Factor Models

Authors: Rittwika Kansabanik, Adrian Barbu

Organizations: Department of Statistics, Florida State University

Abstract

Feature selection is crucial for pinpointing relevant features in high-dimensional datasets, mitigating the 'curse of dimensionality,' and enhancing machine learning performance. Traditional feature selection methods for classification use data from all classes to select features for each class. This paper explores feature selection methods that select features for each class separately, using class models based on low-rank generative methods and introducing a signal-to-noise ratio (SNR) feature selection criterion. This novel approach offers a theoretically sound guarantee of feature recovery under specific assumptions and has been demonstrated to outperform some existing feature selection methods on standard classification datasets. Additionally, it can be easily applied in a class-incremental learning setup, yielding performance comparable to that of some current methods.

Figures & tables

Panel of Figure 1 — Fig. 1 : Estimation error vs. several iterations for (a) the signal variance $\hat{{\boldsymbol{sig}}}$ , and (b) the noise variance, $\hat{{\boldsymbol{\psi}}}$ , when $n=1000$ and $d=110$ . (c) The true SNR ( ${\boldsymbol{SNR}}^{\ast}$ ) and the estimated SNRs obtained by the four methods.

Panel of Figure 2 — Fig. 2 : Average estimation error vs. number $n$ of observations for (a) the signal variance, $\hat{{\boldsymbol{sig}}}$ , (b) the noise variance, $\hat{{\boldsymbol{\psi}}}$ , and (c) the SNRs, $\hat{{\boldsymbol{SNR}}}$ , when $d=110$ .

Fig. 3 : Estimation error and theoretical bound vs. number of observations ( $n$ ) for the PPCA signal variance.

Panel of Table 4 — TABLE I : Feature selection accuracy (%) for simulated data.

Dataset	Method	# Selected Features
		1024	1000	800	600	400	200
CIFAR-10	FSA	24	23	23	22	22	21
CIFAR-10	TISP	26	26	26	25	25	24
CIFAR-100	FSA	27	26	26	26	25	25
CIFAR-100	TISP	30	29	28	27	26	26
		1024	1000	800	600	400	200

TABLE II : Training time (seconds) for FSA and TISP on the datasets evaluated.

Dataset	# Features	Methods
		PPCA	LFA	ELF	HeteroPCA
CIFAR-10	1024	5	18	9	57
CIFAR-100	1024	9	24	12	59
ImageNet	1024	133	476	916	611

TABLE III : Training time (seconds) for low-rank generative methods on the datasets evaluated.

Method	# Selected Features
CIFAR-10, $n=60,000,d=1024$
	1024	1000	800	600	400	200
FSA	98.87	98.93	98.84	98.79	97.67	94.24
TISP	98.93	98.93	98.78	98.7	97.78	94.53
ELF	98.88	98.9	98.92	98.84	98.74	98.08
HeteroPCA	98.88	98.93	98.91	98.86	98.83	98.36

TABLE IV : Classification accuracy (

\%

) for different methods on real datasets.

Fig. 4 : Accuracy vs. number of selected features for CIFAR-10 (left), CIFAR-100(middle) and ImageNet (right).

Method	CIFAR-100				ImageNet-1K
Method	$\bar{a}_{10}$	$a_{10}$	$\bar{a}_{5}$	$a_{5}$	$\bar{a}_{10}$	$a_{10}$	$\bar{a}_{5}$	$a_{5}$
iCaRL(2017) [ 42 ]	52.57	50	57.17	45.5	46.72	45.6	51.36	39.89
BiC(2019) [ 46 ]	53.21	-	56.86	40.21	84.02	73.2	-	-
UCIR (NME)(2019) [ 45 ]	60.12	-	63.12	-	59.92	-	61.56	-
UCIR (CNN)(2019) [ 45 ]	60.18	43.39	63.42	-	61.28	-	64.34	-
PODNet(2020) [ 44 ]	63.19	41.05	64.83	-	64.13	-	66.95	-

TABLE V : Comparison of average incremental accuracy (%) and final accuracy on CIFAR-100 and ImageNet-1K in the class-incremental learning setting (exemplar-free).

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 194 papers

Random Feature Pruning for Multi-Class Classification
Oct 10, 2026Mohammad Amin Omidi, Babak Seyfe, Shahrokh ValaeeFeature Selection Random Feature Methods
Information Theoretic Learning Systems Laboratory (ITLSL), Department of Electrical Engineering, Shahed University, Tehran, Iran · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada
When to Truncate a Feature Ranking: A Residual-Overlap Stopping Rule for Subset Selection
Jun 30, 2026Jesus S. Aguilar-RuizFeature Selection
School of Engineering Pablo de Olavide University ES-41013 Seville, Spain
MOELIGA: a multi-objective evolutionary approach for feature selection with local improvement
Mar 21, 2026Leandro Vignolo, Matias GerardFeature Selection Genetic Algorithms
Research Institute for Signals, Systems and Computational Intelligence (sınc(i)), FICH–UNL/CONICET, Ciudad Universitaria UNL, Santa Fe, Argentina.

Oct 10, 2026J/K move · Enter open · S save

Random Feature Pruning for Multi-Class Classification

Mohammad Amin Omidi, Babak Seyfe, Shahrokh Valaee

Information Theoretic Learning Systems Laboratory (ITLSL), Department of Electrical Engineering, Shahed University, Tehran, Iran · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Canada

Feature Selection Random Feature Methods Multiclass Classification Statistical Learning Theory

Open paper AskPDF arXiv

Weighted sums of random kitchen sinks (W-RKS) is a commonly used feature extraction technique for supervised learning tasks. A major drawback of this method is the redundancy and ineffectiveness of many extracted random features. This paper investigates efficient methods for reducing such redundancy in multi-class classification problems. We specifically formulate feature selection as a variant of the sequentially nested model selection problem and introduce the Successive Empirical Excess Risk Difference Maximization (SEED-MAX) algorithm. It recursively computes empirical risks to identify the most parsimonious yet informative set of features. We derive an analytical lower bound for the probability of correct feature selection by the SEED-MAX algorithm. This lower bound on the probability of correct feature selection can be made arbitrarily close to one by increasing the training dataset size. To further enhance the rate of pruning, we evaluate the features retained by SEED-MAX using the Fisher test (F-Test). Empirical evaluations on UCR time-series classification datasets show that the sequential application of SEED-MAX with the Fisher test achieves approximately 92 percent rate of pruning with negligible degradation in classification accuracy. Implementations of the SEED-MAX and SEED-MAX-F-Test methods are available at https://github.com/OmidiAmin/seed-max-ftest.

Abstract

Feature Selection

Explore similar work

Random Feature Pruning for Multi-Class Classification

When to Truncate a Feature Ranking: A Residual-Overlap Stopping Rule for Subset Selection

MOELIGA: a multi-objective evolutionary approach for feature selection with local improvement