Causally-Guided Diffusion for Stable Feature Selection

Arun Vignesh Malarkkan; Xinyuan Wang; Kunpeng Liu; Denghui Zhang; Yanjie Fu

cs.LGMar 21, 2026

Causally-Guided Diffusion for Stable Feature Selection

Authors: Arun Vignesh Malarkkan, Xinyuan Wang, Kunpeng Liu, Denghui Zhang, Yanjie Fu

Organizations: Arizona State University Tempe, Arizona, USA · Clemson University Clemson, South Carolina, USA · Stevens Institute of Technology Hoboken, New Jersey, USA

Abstract

Feature selection is fundamental to robust data-centric AI, but most existing methods optimize predictive performance under a single data distribution. This often selects spurious features that fail under distribution shifts. Motivated by principles from causal invariance, we study feature selection from a stability perspective and introduce Causally-Guided Diffusion for Stable Feature Selection (CGDFS). In CGDFS, we formalized feature selection as approximate posterior inference over feature subsets, whose posterior mass favors low prediction error and low cross-environment variance. Our framework combines three key insights: First, we formulate feature selection as stability-aware posterior sampling. Here, causal invariance serves as a soft inductive bias rather than explicit causal discovery. Second, we train a diffusion model as a learned prior over plausible continuous selection masks, combined with a stability-aware likelihood that rewards invariance across environments. This diffusion prior captures structural dependencies among features and enables scalable exploration of the combinatorially large selection space. Third, we perform guided annealed Langevin sampling that combines the diffusion prior with the stability objective, which yields a tractable, uncertainty-aware posterior inference that avoids discrete optimization and produces robust feature selections. We evaluate CGDFS on open-source real-world datasets exhibiting distribution shifts. Across both classification and regression tasks, CGDFS consistently selects more stable and transferable feature subsets, which leads to improved out-of-distribution performance and greater selection robustness compared to sparsity-based, tree-based, and stability-selection baselines.

Figures & tables

Figure 1. Framework overview of CGDFS. Feature selection is formulated as approximate posterior inference over feature subsets. A diffusion model learns a structured prior over continuous feature masks from a diverse mask pool. A stability-based likelihood evaluates predictive performance and cross-environment variability. Guided Langevin dynamics combines prior and likelihood gradients to sample from the posterior, and the samples are aggregated to produce stable, interpretable feature selections, with invariance-based causal signals incorporated as soft guidance through the prior.

Dataset	#Samples	#Features	Feature Selection Method
Dataset	#Samples	#Features	LASSO	MI	RF	StabSel	Greedy	Grad-Stab	ICP	CGDFS
Classification (F1-Macro $\uparrow$ )
German Credit	1001	24	0.680	0.665	0.695	0.692	0.705	0.688	0.712	0.725
Ionosphere	351	34	0.825	0.810	0.838	0.835	0.842	0.830	0.848	0.985
Pima Indian	768	9	0.655	0.642	0.668	0.665	0.672	0.660	0.680	0.712
Spam Base	4601	58	0.885	0.878	0.892	0.890	0.896	0.888	0.902	0.915

Table 1. Overall predictive performance across 12 datasets under distribution shift. For classification tasks, we report F1-macro (higher is better). For regression tasks, we report MSE (lower is better). All results are averaged over multiple random seeds.

Panel of Figure 2 — Figure 2. Cross-Environment Stability Analysis

Panel of Figure 3 — Figure 3. CGDFS - Ablation Study

Uncertainty Metric	Activity	Spam Base	Boston Housing
Central Tendency
Mean Uncertainty	0.1847	0.0832	0.2761
Median Uncertainty	0.1693	0.0654	0.2453
Std. Dev. Uncertainty	0.1214	0.1456	0.2422
Extreme Values
Minimum Uncertainty	0.0008	0.0001	0.0001

Table 2. Uncertainty statistics of feature selection for CGDFS. Uncertainty is computed from posterior inclusion frequencies, where higher values indicate greater feature uncertainty.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Default	Sweep range
$\lambda_{\mathrm{var}}$	0.1	{0.0, 0.01, 0.05, 0.1, 0.2, 0.5}
Guidance $\beta$	0.5	{0.0, 0.25, 0.5, 0.75, 1.0}
Langevin steps $K$	100	{20, 50, 100, 200}
Chains $R$	10	{1, 3, 5, 10}
Mask pool size $M$	500	{100, 300, 500}
Diffusion steps $T$	100	{50, 100}

Appendix

Table 3. Hyperparameter defaults and sweep ranges used in our experiments.

Method	UCI Credit (C)		Pima Indian (C)		Boston Housing (R)		OpenML-618 (R)
Method	F1 $\uparrow$	Std $\downarrow$	F1 $\uparrow$	Std $\downarrow$	MSE $\downarrow$	Std $\downarrow$	MSE $\downarrow$	Std $\downarrow$
Full CGDFSS	0.725	0.051	0.712	0.034	15.1	1.60	22.9	1.13
W/O ICP	0.713	0.088	0.709	0.055	17.6	2.10	20.3	1.73
W/O Diffusion	0.643	0.058	0.680	0.039	36.3	1.67	24.1	1.42
W/O Posterior Sampling	0.691	0.079	0.695	0.061	29.8	3.14	24.5	1.45

Appendix

Table 4. Ablation and stability analysis of CGDFS For classification, we report F1-macro (higher is better). For regression, we report MSE (lower is better). Stability is measured as the standard deviation of the primary metric across test environments (lower is better).

Dataset	Stability Sel.	Grad-Stab	ICP	CGDFS
UCI Credit (C)	0.735 $\pm$ 0.058	0.742 $\pm$ 0.072	0.748 $\pm$ 0.045	0.763 $\pm$ 0.051
Activity (C)	0.820 $\pm$ 0.081	0.815 $\pm$ 0.090	0.835 $\pm$ 0.065	0.984 $\pm$ 0.063
OpenML-618 (R)	25.82 $\pm$ 6.22	26.45 $\pm$ 4.01	24.22 $\pm$ 4.62	22.85 $\pm$ 4.13
OpenML-637 (R)	17.23 $\pm$ 3.05	17.64 $\pm$ 2.07	15.84 $\pm$ 2.06	14.82 $\pm$ 2.08

Appendix

Table 5. Cross-environment stability (mean

\pm

standard deviation across test environments). For classification datasets (UCI Credit, Activity), we report F1-macro (higher is better). For regression datasets (OpenML-618, OpenML-637), we report MSE (lower is better).

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 196 papers

Mar 17, 2026

Safe Distributionally Robust Feature Selection under Covariate Shift

In practical machine learning, the environments encountered during the model development and deployment phases often differ, especially when a model is used by many users in diverse settings. Learning models that maintain reliable performance across plausible deployment environments is known as distributionally robust (DR) learning. In this work, we study the problem of distributionally robust feature selection (DRFS), with a particular focus on sparse sensing applications motivated by industrial needs. In practical multi-sensor systems, a shared subset of sensors is typically selected prior to deployment based on performance evaluations using many available sensors. At deployment, individual users may further adapt or fine-tune models to their specific environments. When deployment environments differ from those anticipated during development, this strategy can result in systems lacking sensors required for optimal performance. To address this issue, we propose safe-DRFS, a novel approach that extends safe screening from conventional sparse modeling settings to a DR setting under covariate shift. Our method identifies a feature subset that encompasses all subsets that may become optimal across a specified range of input distribution shifts, with finite-sample theoretical guarantees of no false feature elimination.

Hiroyuki Hanada, Satoshi Akahane, Noriaki Hashimoto +2

Graduate School of Engineering, Nagoya University, Nagoya, Japan · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan

Feature Selection Distributionally Robust Optimization

Ask PDF

Feb 18, 2026

CAFE: Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning

Automated feature engineering (AFE) enables AI systems to autonomously construct high-utility representations from raw tabular data. However, existing AFE methods rely on statistical heuristics, yielding brittle features that fail under distribution shift. We introduce CAFE, a framework that reformulates AFE as a causally-guided sequential decision process, bridging causal discovery with reinforcement learning-driven feature construction. Phase I learns a sparse directed acyclic graph over features and the target to obtain soft causal priors, grouping features as direct, indirect, or other based on their causal influence with respect to the target. Phase II uses a cascading multi-agent deep Q-learning architecture to select causal groups and transformation operators, with hierarchical reward shaping and causal group-level exploration strategies that favor causally plausible transformations while controlling feature complexity. Across 15 public benchmarks (classification with macro-F1; regression with inverse relative absolute error), CAFE achieves statistically significant gains over the strongest AFE baseline (Wilcoxon p<0.001, large effect), reduces episodes-to-convergence, and delivers competitive time-to-target. Under both controlled covariate shifts and naturally occurring distribution shifts (Folktables), CAFE reduces out-of-distribution performance drop by up to 4x relative to a non-causal multi-agent baseline, and produces more compact feature sets with more stable post-hoc attributions. These findings underscore that causal structure, used as a soft inductive prior rather than a rigid constraint, can substantially improve the robustness and efficiency of automated feature engineering.

Arun Vignesh Malarkkan, Wangyang Ying, Yanjie Fu

School of Computing, Arizona State University

Automated Feature Engineering Distribution Shift Robustness Multi-Agent Reinforcement Learning

Ask PDF

Mar 26, 2026

Learning domain-invariant features through channel-level sparsification for Out-Of Distribution Generalization

Out-of-Distribution (OOD) generalization has become a primary metric for evaluating image analysis systems. Since deep learning models tend to capture domain-specific context, they often develop shortcut dependencies on these non-causal features, leading to inconsistent performance across different data sources. Current techniques, such as invariance learning, attempt to mitigate this. However, they struggle to isolate highly mixed features within deep latent spaces. This limitation prevents them from fully resolving the shortcut learning problem.In this paper, we propose Hierarchical Causal Dropout (HCD), a method that uses channel-level causal masks to enforce feature sparsity. This approach allows the model to separate causal features from spurious ones, effectively performing a causal intervention at the representation level. The training is guided by a Matrix-based Mutual Information (MMI) objective to minimize the mutual information between latent features and domain labels, while simultaneously maximizing the information shared with class labels.To ensure stability, we incorporate a StyleMix-driven VICReg module, which prevents the masks from accidentally filtering out essential causal data. Experimental results on OOD benchmarks show that HCD performs better than existing top-tier methods.

Haoran Pei, Yuguang Yang, Kexin Liu +2

School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, P. R. China · School of Electronic Information Engineering, Beihang University, Beijing 100191, P. R. China · School of Artificial Intelligence, Beihang University, Beijing 100191, P. R. China

OOD Generalization Invariant Representation Learning Shortcut Learning Causal Representation Learning

Ask PDF

Abstract

Feature Selection

Explore similar work

Safe Distributionally Robust Feature Selection under Covariate Shift

CAFE: Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning

Learning domain-invariant features through channel-level sparsification for Out-Of Distribution Generalization