Partial Invariance for Classification under Structural Causal Models
Authors: Austin Goddard, Yu Xiang
Organizations: Department of Electrical and Computer Engineering, University of Utah · Department of Mathematics and Statistics, Florida Atlantic University
Making predictions in an unseen environment from data collected across multiple training environments is challenging. This challenge is particularly pronounced when the target-generating mechanism varies across environments. In this work, we introduce environment-label separation, a form of partial environmental invariance that decomposes the target conditional distribution into an environment-invariant component and a label-invariant component. We show that this partial invariance enables estimation of the target conditional distribution in an unseen environment when using labeled training data together with unlabeled test features. Within the structural causal model framework, we provide sufficient conditions for environment-label separation. Building on these conditions, we develop an algorithm, dIMP-Classifier, which jointly minimizes an estimate of the test risk and encourages invariance across environments. We further provide finite-sample guarantees that characterize the test risk in terms of deviations from invariance, estimation error, and sample complexity. Experiments on simulated data and the real-world light tunnel Causal Chamber data demonstrate that the proposed approach can perform under interventions on the target mechanism and consistently outperforms the considered standard and invariance-based baselines.
Figures & tables
Figure 1: The causal diagram detailing the motivating example. Variables inside the dashed box are used to identify EY-separable functions. Solid edges represent invariant mechanisms while dashed edges represent mechanisms that may vary over environment. We consider two settings with (a) and without (b) hidden confounders.
Figure 2: Simulation results for the binary setting when ∣CH∣=1 .
Figure 3: Test accuracy across simulated datasets with a multinomial target.
Figure 4: Light tunnel Causal Chamber setup used in the real-data experiments.
Figure 5: Test accuracy results from the light tunnel causal chambers dataset.
We consider learning from labeled data collected across multiple environments, where the data distribution may vary across these environments. This problem is commonly approached from a causal perspective, seeking invariant representations that retain causal factors while discarding spurious ones. However, this framework assumes that the environment has no direct effect on the target. In contrast, we consider settings in which this assumption fails, but still aim to learn representations that support robust prediction on average across previously unseen environments. To this end, we study representations learned by explicitly modeling variation across environments and then marginalizing that variation out. We analyze the resulting representations and characterize when they are preferable to those learned by causal invariant-representation methods. We propose a concrete method based on generalized random-intercept models, a class of predictors in which such marginalization is possible, and study their generalization properties. Empirically, we show that these models outperform invariant-learning methods across a range of challenging settings.
Yuli Slavutsky, David M. Blei
Department of Statistics Columbia University · Departments of Statistics and Computer Science Columbia University
Invariant prediction [Peters et al., 2016] analyzes feature/outcome data from multiple environments to identify invariant features - those with a stable predictive relationship to the outcome. Such features support generalization to new environments and help reveal causal mechanisms. Previous methods have primarily tackled this problem through hypothesis testing or regularized optimization. Here we develop Bayesian Invariant Prediction (BIP), a probabilistic model for invariant prediction. BIP encodes the indices of invariant features as a latent variable and recover them by posterior inference. Under the assumptions of Peters et al. [2016], the BIP posterior targets the true invariant features. We prove that the posterior is consistent and that greater environment heterogeneity leads to faster posterior contraction. To handle many features, we design an efficient variational approximation called VI-BIP. In simulations and real data, we find that BIP and VI-BIP are more accurate and scalable than existing methods for invariant prediction.
Luhuan Wu, Mingzhang Yin, Yixin Wang +2
Department of Applied Mathematics and Statistics, Johns Hopkins University. · Warrington College of Business, University of Florida. · Department of Statistics, University of Michigan. +2
Causal discovery, the problem of inferring the direction of causality, is generally ill-posed. We use the language of structural causal models (SCM) to show that assuming that the causal relations are acyclic and invariant across multiple environments (e.g., the way minimum wage affects employment rate is stable across different geographical regions), only two auxiliary environments are sufficient to infer the causal graph for arbitrary nonlinear mechanisms. Moreover, we demonstrate that this implies identifiability of the SCM functional mechanisms: as a corollary, we show that two auxiliary environments are sufficient to guarantee correct counterfactual inference. We empirically support our theoretical results on synthetic data.