Safe Distributionally Robust Feature Selection under Covariate Shift

Hiroyuki Hanada; Satoshi Akahane; Noriaki Hashimoto; Shion Takeno; Ichiro Takeuchi

stat.MLMar 17, 2026

Safe Distributionally Robust Feature Selection under Covariate Shift

Authors: Hiroyuki Hanada, Satoshi Akahane, Noriaki Hashimoto, Shion Takeno, Ichiro Takeuchi

Organizations: Graduate School of Engineering, Nagoya University, Nagoya, Japan · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan

Abstract

In practical machine learning, the environments encountered during the model development and deployment phases often differ, especially when a model is used by many users in diverse settings. Learning models that maintain reliable performance across plausible deployment environments is known as distributionally robust (DR) learning. In this work, we study the problem of distributionally robust feature selection (DRFS), with a particular focus on sparse sensing applications motivated by industrial needs. In practical multi-sensor systems, a shared subset of sensors is typically selected prior to deployment based on performance evaluations using many available sensors. At deployment, individual users may further adapt or fine-tune models to their specific environments. When deployment environments differ from those anticipated during development, this strategy can result in systems lacking sensors required for optimal performance. To address this issue, we propose safe-DRFS, a novel approach that extends safe screening from conventional sparse modeling settings to a DR setting under covariate shift. Our method identifies a feature subset that encompasses all subsets that may become optimal across a specified range of input distribution shifts, with finite-sample theoretical guarantees of no false feature elimination.

Figures & tables

Figure 1: This figure illustrates the sparse sensing problem studied in this work, which consists of two phases: a development phase (top) and a deployment phase (bottom). In the development phase, a system developer selects a subset of sensors from a large pool of candidates and designs a system equipped with these sensors, after which the sensor set is fixed. In the deployment phase, the system is used by many end users operating in diverse environments with uncertainty, modeled as covariate shift, where the input distribution varies across users within a specified range. Each user adapts or fine-tunes a sparse model, such as a regression or classification model, using only the sensors available in the system. Under covariate shift, the optimal support of sparse models may differ across users, leading to different sensor requirements. The key challenge for the developer is therefore to select, during development, all sensors that may be required under plausible deployment environments while safely eliminating sensors that will never be used.

Dataset	$n$	$d$	Source
For regressions
cpusmall	8192	12	LIBSVM
housing	506	13	LIBSVM
facebook_comment	40949	52	UCI
triazines	186	58	LIBSVM
online_news	39644	58	UCI

Table 1: Datasets for the experiments. “LIBSVM”: LIBSVM Data ( Chang and Lin, 2011 ) , “UCI”: UCI Machine Learning Repository ( Dheeru and Karra Taniskidou, 2017 ) . See Appendix B.1 for the preprocesses conducted on them. (Values of

n

and

d

are the ones after the preprocesses.)

Panel of Figure 2 — Figure 2: Ratios of removed features to the original features for four regression datasets. Horizontal axis: The uncertainty levels of distributional change $V$ , Vertical axis: Ratio of reduced features.

Panel of Figure 3 — Figure 3: Ratio of removed features for eight binary classification datasets to the original features. Horizontal axis: The uncertainty levels of distributional change $V$ , Vertical axis: Ratio of reduced features.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 198 papers

Causally-Guided Diffusion for Stable Feature Selection
Mar 21, 2026Arun Vignesh Malarkkan, Xinyuan Wang, Kunpeng Liu +2Feature Selection Distribution Shift Robustness
Arizona State University Tempe, Arizona, USA · Clemson University Clemson, South Carolina, USA · Stevens Institute of Technology Hoboken, New Jersey, USA
Statistical Properties of Robust Learning under Distributional Shifts
Aug 13, 2026Zhiyi Li, Xiaojie Mao, Yunbei Xu +1Distribution Shift Robustness Distributionally Robust Optimization
National University of Singapore · Tsinghua University · University College London
Scalable and Adaptive Distribution Regression via Random Features
Oct 17, 2026Yinsong Wang, Xiao Liu, Yu DingDistributional Regression Random Feature Methods
School of Mechanical, Industrial and Material Engineering, University of Nevada Reno, Reno 89557, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta 30332, USA.

Mar 21, 2026J/K move · Enter open · S save

Causally-Guided Diffusion for Stable Feature Selection

Arun Vignesh Malarkkan, Xinyuan Wang, Kunpeng Liu, Denghui Zhang+1

Arizona State University Tempe, Arizona, USA · Clemson University Clemson, South Carolina, USA · Stevens Institute of Technology Hoboken, New Jersey, USA

Feature Selection Distribution Shift Robustness OOD Generalization

Open paper AskPDF arXiv

Feature selection is fundamental to robust data-centric AI, but most existing methods optimize predictive performance under a single data distribution. This often selects spurious features that fail under distribution shifts. Motivated by principles from causal invariance, we study feature selection from a stability perspective and introduce Causally-Guided Diffusion for Stable Feature Selection (CGDFS). In CGDFS, we formalized feature selection as approximate posterior inference over feature subsets, whose posterior mass favors low prediction error and low cross-environment variance. Our framework combines three key insights: First, we formulate feature selection as stability-aware posterior sampling. Here, causal invariance serves as a soft inductive bias rather than explicit causal discovery. Second, we train a diffusion model as a learned prior over plausible continuous selection masks, combined with a stability-aware likelihood that rewards invariance across environments. This diffusion prior captures structural dependencies among features and enables scalable exploration of the combinatorially large selection space. Third, we perform guided annealed Langevin sampling that combines the diffusion prior with the stability objective, which yields a tractable, uncertainty-aware posterior inference that avoids discrete optimization and produces robust feature selections. We evaluate CGDFS on open-source real-world datasets exhibiting distribution shifts. Across both classification and regression tasks, CGDFS consistently selects more stable and transferable feature subsets, which leads to improved out-of-distribution performance and greater selection robustness compared to sparsity-based, tree-based, and stability-selection baselines.

Abstract

Feature Selection

Explore similar work

Causally-Guided Diffusion for Stable Feature Selection

Statistical Properties of Robust Learning under Distributional Shifts

Scalable and Adaptive Distribution Regression via Random Features