Feature Selection

Follow this topic Ask about this topic

Feature selection studies which variables, channels, bands, or learned dimensions carry useful information, aiming to simplify models and improve prediction, interpretation, or measurement efficiency. Common approaches include Lasso and elastic net regularization, filters such as mRMR and Boruta, Bayesian spike-and-slab models, and wrapper or evolutionary searches; newer methods use neural gates, attribution scores, and sequential policies. Unsupervised methods select features for clustering or representation learning, while applications span tabular data, omics, EEG, hyperspectral imagery, and time series. A central challenge is that selected features can change with correlated inputs, evaluation design, or distribution shifts, so predictive accuracy must be considered alongside stability, error control, computational cost, and strong baselines.

Latest papers

196 papers27 papers in the last four weeks, up 59% on the four weeks before. 0.2% of all new papers.

Feb 26, 2026

LUMOS: Democratizing SciML Workflows with L0-Regularized Learning for Unified Feature and Parameter Adaptation

The rapid growth of scientific machine learning (SciML) has accelerated discovery across diverse domains, yet designing effective SciML models remains a challenging task. In practice, building such models often requires substantial prior knowledge and manual expertise, particularly in determining which input features to use and how large the model should be. We introduce LUMOS, an end-to-end framework based on L0-regularized learning that unifies feature selection and model pruning to democratize SciML model design. By employing semi-stochastic gating and reparameterization techniques, LUMOS dynamically selects informative features and prunes redundant parameters during training, reducing the reliance on manual tuning while maintaining predictive accuracy. We evaluate LUMOS across 13 diverse SciML workloads, including cosmology and molecular sciences, and demonstrate its effectiveness and generalizability. Experiments on 13 SciML models show that LUMOS achieves 71.45% parameter reduction and a 6.4x inference speedup on average. Furthermore, Distributed Data Parallel (DDP) training on up to eight GPUs confirms the scalability of

Shouwei Gao, Xu Zheng, Dongsheng Luo +2

Oregon State University · Florida International University · Argonne National Laboratory

Scientific ML Automated Machine Learning Feature Selection Neural Network Pruning

Feb 25, 2026

X-REFINE: XAI-based RElevance input-Filtering and archItecture fiNe-tuning for channel Estimation

AI-native architectures are vital for 6G wireless communications. The black-box nature and high complexity of deep learning models employed in critical applications, such as channel estimation, limit their practical deployment. While perturbation-based eXplainable Artificial Intelligence (XAI) solutions offer input filtering, they often neglect internal structural optimization. We propose X-REFINE, an XAI-based framework for joint input-filtering and architecture fine-tuning. By utilizing a decomposition-based, sign-stabilized LRP epsilon rule, X-REFINE backpropagates predictions to derive high-resolution relevance scores for both subcarriers and hidden neurons. This enables a reliable optimization that identifies the most reliable model components. Simulation results demonstrate that X-REFINE achieves a superior performance-complexity-interpretability trade-off compared to the external perturbation-based XAI frameworks, significantly reducing computational complexity while maintaining robust bit error rate (BER) performance.

Abdul Karim Gizzini, Yahia Medjahdi

University of Paris-Est Créteil (UPEC), LISSI/TincNET, F-94400, Vitry-sur-Seine, France. · IMT Nord Europe, Institut Mines Télécom, Centre for Digital Systems, F-59653 Villeneuve d’Ascq, France.

Channel Estimation Efficient Neural Network Inference Neural Network Pruning Feature Selection Layer-Wise Relevance Propagation

Feb 24, 2026

Sparse Bayesian Deep Functional Learning with Structured Region Selection

In modern applications such as ECG monitoring, neuroimaging, wearable sensing, and industrial equipment diagnostics, complex and continuously structured data are ubiquitous, presenting both challenges and opportunities for functional data analysis. However, existing methods face a critical trade-off: conventional functional models are limited by linearity, whereas deep learning approaches lack interpretable region selection for sparse effects. To bridge these gaps, we propose a sparse Bayesian functional deep neural network (sBayFDNN). It learns adaptive functional embeddings through a deep Bayesian architecture to capture complex nonlinear relationships, while a structured prior enables interpretable, region-wise selection of influential domains with quantified uncertainty. Theoretically, we establish rigorous approximation error bounds, posterior consistency, and region selection consistency. These results provide the first theoretical guarantees for a Bayesian deep functional model, ensuring its reliability and statistical rigor. Empirically, comprehensive simulations and real-world studies confirm the effectiveness and superiority of sBayFDNN. Crucially, sBayFDNN excels in recognizing intricate dependencies for accurate predictions and more precisely identifies functionally meaningful regions, capabilities fundamentally beyond existing approaches.

Xiaoxian Zhu, Yingmeng Li, Shuangge Ma +1

School of Statistics and Data Science, Shanghai University of Finance and Economics · Department of Biostatistics, Yale School of Public Health

Functional Data Analysis Bayesian Neural Networks Feature Selection Sparse Neural Networks

Feb 23, 2026

FairFS: Addressing Deep Feature Selection Biases for Recommender System

Large-scale online marketplaces and recommender systems serve as critical technological support for e-commerce development. In industrial recommender systems, features play vital roles as they carry information for downstream models. Accurate feature importance estimation is critical because it helps identify the most useful feature subsets from thousands of feature candidates for online services. Such selection enables improved online performance while reducing computational cost. To address feature selection problems in deep learning, trainable gate-based and sensitivity-based methods have been proposed and proven effective in industrial practice. However, through the analysis of real-world cases, we identified three bias issues that cause feature importance estimation to rely on partial model layers, samples, or gradients, ultimately leading to inaccurate importance estimation. We refer to these as layer bias, baseline bias, and approximation bias. To mitigate these issues, we propose FairFS, a fair and accurate feature selection algorithm. FairFS regularizes feature importance estimated across all nonlinear transformation layers to address layer bias. It also introduces a smooth baseline feature close to the classifier decision boundary and adopts an aggregated approximation method to alleviate baseline and approximation biases. Extensive experiments demonstrate that FairFS effectively mitigates these biases and achieves state-of-the-art feature selection performance.

Xianquan Wang, Zhaocheng Du, Jieming Zhu +3

Huawei Noah’s Ark Lab, Beijing, China

Feature Selection Recommender Systems

Feb 19, 2026

Valid Feature-Level Inference for Tabular Foundation Models via the Conditional Randomization Test

Modern machine learning models are highly expressive but notoriously difficult to analyze statistically. In particular, while black-box predictors can achieve strong empirical performance, they rarely provide valid hypothesis tests or p-values for assessing whether individual features contain information about a target variable. This article presents a practical approach to feature-level hypothesis testing that combines the Conditional Randomization Test (CRT) with TabPFN, a probabilistic foundation model for tabular data. The resulting procedure yields finite-sample valid p-values for conditional feature relevance, even in nonlinear and correlated settings, without requiring model retraining or parametric assumptions.

Mohamed Salem

Statistical Hypothesis Testing Feature Selection Conditional Independence Testing

Feb 18, 2026

ML-driven detection and reduction of ballast information in multi-modal datasets

Modern datasets often contain ballast as redundant or low-utility information that increases dimensionality, storage requirements, and computational cost without contributing meaningful analytical value. This study introduces a generalized, multimodal framework for ballast detection and reduction across structured, semi-structured, unstructured, and sparse data types. Using diverse datasets, entropy, mutual information, Lasso, SHAP, PCA, topic modelling, and embedding analysis are applied to identify and eliminate ballast features. A novel Ballast Score is proposed to integrate these signals into a unified, cross-modal pruning strategy. Experimental results demonstrate that significant portions of the feature space as often exceeding 70% in sparse or semi-structured data, can be pruned with minimal or even improved classification performance, along with substantial reductions in training time and memory footprint. The framework reveals distinct ballast typologies (e.g. statistical, semantic, infrastructural), and offers practical guidance for leaner, more efficient machine learning pipelines.

Yaroslav Solovko

Independent Researcher, Dublin, Ireland

Feature Selection Dimensionality Reduction

Feb 18, 2026

Separating Oblivious and Adaptive Models of Variable Selection

Sparse recovery is among the most well-studied problems in learning theory and high-dimensional statistics. In this work, we investigate the statistical and computational landscapes of sparse recovery with

\ell_\infty

error guarantees. This variant of the problem is motivated by variable selection tasks, where the goal is to estimate the support of a

k

-sparse signal in

\mathbb{R}^d

. Our main contribution is a provable separation between the oblivious (for each'') and *adaptive* (for all'') models of

\ell_\infty

sparse recovery. We show that under an oblivious model, the optimal

\ell_\infty

error is attainable in near-linear time with

\approx k\log d

samples, whereas in an adaptive model,

\gtrsim k^2

samples are necessary for any algorithm to achieve this bound. This establishes a surprising contrast with the standard

\ell_2

setting, where

\approx k \log d

samples suffice even for adaptive sparse recovery. We conclude with a preliminary examination of a partially-adaptive model, where we show nontrivial variable selection guarantees are possible with

\approx k\log d

measurements.

Ziyun Chen, Jerry Li, Kevin Tian +1

University of Washington · University of Texas at Austin

Sparse Recovery Feature Selection Statistical Learning Theory Inverse Problems

Jan 8, 2026

ROOFS: RObust biOmarker Feature Selection

Feature selection (FS) is essential for biomarker discovery and clinical predictive modeling. Over the past decades, methodological literature on FS has become rich and mature, offering a wide spectrum of algorithmic approaches. However, much of this methodological progress has not fully translated into applied biomedical research. Moreover, challenges inherent in biomedical data, such as high-dimensional feature space, low sample size, multicollinearity, and missing values, make FS non-trivial. To help bridge this gap between methodological development and practical application, we propose ROOFS (RObust biOmarker Feature Selection), a Python package available at https://gitlab.inria.fr/compo/roofs, designed to help researchers in the choice of FS method adapted to their problem. ROOFS benchmarks multiple FS methods on the user's data and generates reports summarizing a comprehensive set of evaluation metrics, including downstream predictive performance estimated using optimism correction, stability, robustness of individual features, and true positive and false positive rates assessed on semi-synthetic data with a simulated outcome. We demonstrate the utility of ROOFS on data from the PIONeeR clinical trial, aimed at identifying predictors of resistance to anti-PD-(L)1 immunotherapy in lung cancer. Of the 34 FS methods gathered in ROOFS, we evaluated 23 in combination with 11 classifiers (253 models) and identified a filter based on the union of Benjamini-Hochberg false discovery rate-adjusted p-values from t-test and logistic regression as the optimal approach, outperforming other methods including widely used LASSO. We conclude that comprehensive benchmarking with ROOFS has the potential to improve the reproducibility of FS discoveries and increase the translational value of clinical models.

Anastasiia Bakhmach, Paul Dufossé, Simon Charpigny +4

1*COMPO Team (COMPutational Pharmacology and clinical Oncology), Inria Sophia Antipolis – Méditerranée, Inserm U1068, CNRS UMR7258, Institut Paoli-Calmettes, Aix-Marseille University, Marseille, France. · 2Veracyte SAS, Marseille, France. · 3Assistance Publique-Hôpitaux de Marseille (APHM), Marseille, France. +1

Scientific Discovery Clinical Prediction Feature Selection

Dec 4, 2025

Provable FDR Control for Deep Feature Selection: Deep MLPs and Beyond

We develop a flexible feature selection framework based on deep neural networks that approximately controls the false discovery rate (FDR), a measure of Type-I error. The method applies to architectures whose first layer is fully connected. From the second layer onward, it accommodates multilayer perceptrons (MLPs) of arbitrary width and depth, convolutional and recurrent networks, attention mechanisms, residual connections, and dropout. The procedure also accommodates stochastic gradient descent with data-independent initializations and learning rates. To the best of our knowledge, this is the first work to provide a theoretical guarantee of FDR control for feature selection within such a general deep learning setting. Our analysis is built upon a multi-index data-generating model and an asymptotic regime in which the feature dimension

n

diverges faster than the latent dimension

q^{*}

, while the sample size, the number of training iterations, the network depth, and hidden layer widths are left unrestricted. Under this setting, we show that each coordinate of the gradient-based feature-importance vector admits a marginal normal approximation, thereby supporting the validity of asymptotic FDR control. As a theoretical limitation, we assume

\mathbf{B}

-right orthogonal invariance of the design matrix, and we discuss broader generalizations. We also present numerical experiments that underscore the theoretical findings.

Kazuma Sawaya

The University of Tokyo, Bunkyo, Tokyo, Japan

FDR Control Feature Selection Statistical Learning Theory Multiple Hypothesis Testing

Oct 22, 2025

Survival of the fittest Cox model: Pivotal variable selection for time-to-event data

We revisit Cox's proportional hazards model to improve variable selection in survival analysis. A square-root transformation of the partial likelihood renders the selection of the regularization parameter pivotal, free of the unknown baseline hazard and censoring mechanism. The resulting criterion borrows from information criteria such as BIC and from penalized regression methods such as the lasso, taking the best of both. On simulated and real data, our method substantially improves upon state-of-the-art approaches used daily in support recovery.

Maxime van Cutsem, Sylvain Sardy

Department of Mathematics, University of Geneva

Model Selection Survival Analysis Feature Selection Hyperparameter Optimization

Sep 17, 2025

A Compositional Kernel Model for Feature Learning

We study a compositional variant of kernel ridge regression in which the predictor is applied to a coordinate-wise reweighting of the inputs. Formulated as a variational problem, this model provides a tractable setting for studying feature learning in compositional architectures. From the perspective of variable selection, we show how relevant variables are recovered while noise variables are eliminated. We prove that both global minimizers and stationary points discard noise coordinates when the noise variables are Gaussian distributed. A central finding is that

\ell_1

-type kernels, such as the Laplace kernel, succeed in recovering features contributing to nonlinear effects at stationary points, whereas Gaussian kernels recover only linear ones.

Feng Ruan, Keli Liu, Michael Jordan

*Department of Statistics and Data Science, Northwestern University. · *Company E · *Department of Statistics and Computer Science, University of California, Berkeley.

Kernel Ridge Regression Kernel Regression Kernel Methods Feature Selection

Jun 25, 2025

Counterfactual Operator Relevance for PDE Discovery: Screening, Pruning, and Identifiability

We study operator relevance in data-driven partial differential equation (PDE) discovery. Sparse residual methods can select terms that improve residual fit, but residual contribution is not the same as functional necessity. We formalize this distinction through counterfactual operator interventions, where a candidate term is deleted or perturbed and the factual and intervened trajectories, or observables, are compared. The resulting theory gives six reusable results. A residual--counterfactual gap theorem shows that deletion effects are governed by the inverse linearized PDE map, not by residual magnitude alone. A certified decision theorem gives error margins for relevance, irrelevance, and abstention under neural or numerical surrogate error. An aliasing theorem characterizes experiment-dependent non-identifiability through the null space of the operator-evaluation design. A constraint-manifold theorem shows that operators vanishing on invariant constraint classes cannot be identified from trajectories restricted to those classes. A pruning-consistency theorem proves that sparse screening followed by counterfactual deletion recovers the functionally relevant support under a recall and margin condition. An observable-level adjoint theorem extends relevance testing from full-state deviations to scientific quantities of interest. Validation experiments test these mechanisms on synthetic PDEs with known support and on public geophysical fields from atmospheric reanalysis and NOAA OISST. The real-data results are reported as operator-surrogate diagnostics, not as unconditional recovery of physical laws. The framework provides a rigorous diagnostic layer for distinguishing residual usefulness from counterfactual operator relevance within a specified library, experiment class, norm, and tolerance.

Ronald Katende

Department of Mathematics Kabale University Kikungiri Hill, Katuna Road, 317, Kabale, Uganda

Counterfactual Evaluation PDE Discovery Feature Selection PDE Inverse Problems

Jun 6, 2025

Scalable unsupervised feature selection via weight stability

Unsupervised feature selection is critical for improving clustering performance in high-dimensional data, where irrelevant features can obscure meaningful structure. In this work, we propose the Minkowski weighted

k

-means++, a novel initialisation strategy for the Minkowski Weighted

k

-means. Our initialisation selects centroids probabilistically using feature relevance estimates derived from the data itself. Building on this, we propose two new feature selection algorithms, FS-MWK++, which aggregates feature weights across a range of Minkowski exponents identifying stable and informative features, and SFS-MWK++, a scalable variant based on subsampling. We support our approach with a theoretical analysis, demonstrating that, under explicit assumptions on noise features and cluster structure, relevant features are assigned consistently higher weights than noise features across a range of Minkowski exponents. Our software can be found at https://github.com/xzhang4-ops1/FSMWK.

Xudong Zhang, Renato Cordeiro de Amorim

School of Computer Science and Electronic Engineering, University of Essex, Wivenhoe, UK.

Unsupervised Learning Clustering Unsupervised Clustering Feature Selection

Apr 1, 2025

Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection

To develop a trustworthy AI system, it is essential to understand its behavior using attribution methods, which aim to identify the input regions that most influence the model's decisions. For discrete inputs such as images, however, accurately identifying input-prediction interactions is particularly challenging because of combinatorial explosion. We formulate black-box attribution as an ordered interpretable subset-selection problem. Inspired by the marginal-gain principle of submodular optimization, the formulation permits redundancy, positive complementarity, and negative interference among input sub-regions. We propose LiMA (Less input is More faithful for Attribution), which ranks sub-regions by their contributions along a sequence of selected subsets. LiMA employs a composite objective that evaluates subsets through semantic consistency, collaboration, prediction confidence, and semantic effectiveness. We further design a bidirectional greedy search to efficiently identify important and non-critical sub-regions. Experiments on eight foundation models (CLIP, ImageBind, LanguageBind, QuiltNet, etc.) and six datasets (ImageNet, VGG-Sound, CUB-200-2011, etc.) demonstrate strong generalization and faithful interpretations with fewer sub-regions. LiMA achieves state-of-the-art Insertion, Deletion, and average highest confidence, with average improvements of 36.3% in Insertion and 39.6% in Deletion; it is also 1.6 times faster than naive greedy search. For failure attribution on incorrect predictions, LiMA improves average highest confidence by 86.1% over state-of-the-art methods. The code is available at https://github.com/RuoyuChen10/LIMA.

Ruoyu Chen, Siyuan Liang, Jingzhi Li +4

Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093, China · University of Chinese Academy of Sciences, Beijing 100049, China · College of Computing and Data Science, Nanyang Technological University, 639798, Singapore +4

Feature Attribution VLM Interpretability Explainability Evaluation Feature Selection

Dec 13, 2024

Feature Selection for Latent Factor Models

Feature selection is crucial for pinpointing relevant features in high-dimensional datasets, mitigating the 'curse of dimensionality,' and enhancing machine learning performance. Traditional feature selection methods for classification use data from all classes to select features for each class. This paper explores feature selection methods that select features for each class separately, using class models based on low-rank generative methods and introducing a signal-to-noise ratio (SNR) feature selection criterion. This novel approach offers a theoretically sound guarantee of feature recovery under specific assumptions and has been demonstrated to outperform some existing feature selection methods on standard classification datasets. Additionally, it can be easily applied in a class-incremental learning setup, yielding performance comparable to that of some current methods.

Rittwika Kansabanik, Adrian Barbu

Department of Statistics, Florida State University

Feature Selection

Jul 14, 2019

Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts

Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predictors, and are widely used for regression, classification and model-based clustering of heterogeneous data. Fitting MoE by maximum likelihood becomes unstable, and sometimes infeasible, when the predictors are numerous or correlated. We propose a regularized maximum likelihood framework for simultaneous parameter estimation and feature selection in MoE whose experts belong to the generalized linear model family, covering Gaussian, Poisson and multinomial responses within a single formulation. Sparsity is induced in both the gating network and the experts through

\ell_1

penalties, and the penalized log-likelihood is maximized by a proximal Newton-EM algorithm whose M-step reduces to weighted Lasso problems with closed-form coordinate-ascent updates. Unlike existing penalized MoE procedures, the algorithm requires neither a local quadratic approximation of the penalty nor any matrix inversion, it returns exactly sparse estimates without thresholding, and a proximal Newton-type variant guarantees a monotone increase of the penalized objective at every iteration. On simulated data and five real data sets, the method recovers the actual sparsity support and delivers prediction and clustering accuracy that is competitive with, and often better than, state-of-the-art regularized MoE. The source codes of our developed algorithms and their documentation are publicly available on Github at https://github.com/nv-thin/GLM-RMoE.

Thin Nguyen-Van, Faicel Chamroukhi, Ha Hoang Van +1

Faculty of Mathematics and Computer Science, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · IRT SystemX, 91120 Palaiseau, France +1

Mixture of Experts Sparse Mixture-of-Experts Expectation-Maximization Feature Selection Generalized Linear Models