Authors: Saketh Vishnubhatla, Pingchuan Ma, Shu Wan, Alimohammad Beigi, Andre Harrison, Adrienne Raglin, Huan Liu
Organizations: School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · DEVCOM Army Research Laboratory, Adelphi, MD 20783, USA
Counterfactual inference is the task of answering hypothetical "what-if" queries about a unit, given an observational dataset, modeling assumptions, and a causal graph. This enables a modeler to reason about hypothetical scenarios that cannot otherwise be answered without assumptions or a causal graph. The simpler case involves a causal graph with scalar 1-D variables that denote different attributes in the data-generation process. While the literature is clear in the 1-D case, extending this to settings with high-dimensional nodes (e.g., images) remains challenging. Conventionally, in the literature, methods recover the latent state from the factual image (inversion), generate counterfactual attributes from the graph, and render the counterfactual image from the latent state using the updated attributes. In our work, we investigate the necessity of the inversion step in computing high-dimensional counterfactuals. We first discuss moving from the scalar 1-D case to the higher-dimensional case. Secondly, we discuss the relevance of the inversion step and evaluate an inversion-free counterpart repurposed from the image-editing literature. Finally, on image datasets such as CelebA and ADNI, we benchmark our approach using two measures: namely, effectiveness and identity preservation. While effectiveness is a measure of how faithfully the counterfactual attribute is rendered, identity preservation measures whether the same unit (image) is being rendered, both of which are crucial for high-dimensional counterfactual inference. We present the tradeoffs of using inversion-based and inversion-free approaches in our work.
Figures & tables
Fig. 1: The two transports, both given the same counterfactual attributes obtained by propagating the antecedent on age through the structural equations. Left: inversion carries the factual to noise under the source condition and regenerates under the target. Right: the inversion-free map moves the factual directly, by accumulated velocity differences between the two conditions, always staying on the image manifold.
Fig. 2: Attribute-level diagram and structural equations used as a running example, following Melistas et al. [17] : ApoE- ε4 status ap , sex s , age a , brain volume b and ventricular volume v . The image node is introduced in Section IV-B .
Fig. 3: Abduction at fixed parents. Solid curves are fv(a,b,⋅) at the observed brain volume b , dashed curves the same mechanism under do(B=b∗) . In (a) the observed v has a single preimage u^v , and the counterfactual is read off the dashed curve at that point. In (b) the mechanism attains v at two exogenous values, and the two preimages give different counterfactuals.
Fig. 4: The two-stage model. The attribute mechanisms and their exogenous variables are those of Fig. 2 . All five attributes pa=(ap,s,a,b,v) are rendered into the image by g (grey edges), together with a high-dimensional exogenous state ux carrying everything the diagram does not name. The labelling map ℓ recovers the attributes from the image.
Method
Image mechanism
Exogenous state
Pawlowski et al. [21]
VAE, trained
latent + residual
De Sousa Ribeiro et al. [3]
HVAE, trained
latents + residual
Melistas et al. [17]
GAN, trained
single latent
Sanchez and Tsaftaris [24]
diffusion, trained
uncond. noise map
Rasal et al. [23]
diff. AE, trained
noise map, semantic code, guidance tokens
De Sousa Ribeiro et al. [2]
OT-coupled flow
flow noise map
TABLE I: Image mechanisms and the exogenous states they recover.
Fig. 5: The attribute-level diagram assumed on CelebA.
Effectiveness F1 ↑
Realism FID ↓
Identity LPIPS ↓
Method
Backbone
a
g
br
bl
a
g
br
bl
a
g
br
bl
Explicit inversion — encoder; DDIM for Causal-Adapter, flow-ODE for SD3-inv
VAE
deep-SCM
0.390
0.990
0.284
0.004
60.0
63.4
59.9
57.4
0.283
0.295
0.283
0.284 †
HVAE
deep-SCM
0.246
0.742
0.476
0.122
2.2
5.9
25.6
31.7
0.026 †
0.076
0.138
0.229 †
GAN
deep-SCM
0.471
0.989
0.273
0.340
22.1
26.6
23.0
24.3
0.256
0.363
0.262
0.285
Causal-Adapter
SD1.5 + adapter
0.530
0.997
0.424
0.497
8.2
9.0
8.0
17.7
0.166
0.246
0.119
0.240
TABLE II: CelebA counterfactuals, grouped by whether the exogenous state is recovered at all. Best per column in bold, runner-up underlined. Cells marked † are excluded from the ranking.
Fig. 6: CelebA counterfactuals with four antecedents, one method per column.
Effectiveness F1 ↑
Identity LPIPS ↓
Inference map
Fwd./query
a
g
br
bl
a
g
br
bl
copy input
0
0.142
0.349
0.050
0.007
0.000
0.000
0.000
0.000
sample at pa∗
32
0.569
0.992
0.312
0.700
0.495
0.517
0.508
0.517
inversion-free, M=8
512
0.674
0.994
0.361
0.724
0.070
0.325
0.128
0.430
+ FlowAlign, γ=0.01
512
0.420
0.968
0.247
0.550
0.033
0.239
0.104
0.402
inversion, N=32
64
0.610
0.992
0.314
0.717
0.148
0.343
0.188
0.432
TABLE III: One model, several inference maps. All rows share the same scratch-trained DiT-B/2 (131 M, SD3 latents, independent coupling), so no difference between them is attributable to training. The first two rows are bounds and are excluded from the ranking; best per column in bold, runner-up underlined.
Fig. 7: CelebA counterfactuals from the scratch-trained model, one inference map per column.
Fig. 8: do(vent_vol→p95) on four held-out subjects, taken at even quantiles of the requested change among stable outputs.
Inversion
Inversion-free
Effectiveness
Delivered (mm 3 ) ↑
+41,459±4,671
+23,634±2,732
Realised ↑
82.1%
46.8%
Identity (LPIPS)
Reconstruction ↓
0.0508
0.0223
Edit magnitude, median (max)
0.149 (0.618)
0.062 (0.108)
TABLE IV: Counterfactual editing on ADNI, do(vent_vol→p95) . on The Same Backbone.
Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier's prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as sensitivity to shortcut features and calibration errors. In this paper, we propose a classifier-free approach for visual counterfactual generation based on Contrastive Analysis (CA). Given two datasets corresponding to different classes (e.g., healthy and patients), we disentangle the generative factors that are common across the two datasets from those that are salient to each dataset, and generate counterfactual images by swapping only the salient factors. By operating directly on data distributions rather than decision boundaries, our method provides model-agnostic VCEs that are less sensitive to classifier biases. Our approach leverages the high-quality synthesis and well-structured latent space of StyleGAN2. We use the feature space F, instead than the usual W-space, to improve detail preservation. Unlike conventional CA approaches, which typically assume salient factors in only one dataset, we introduce an adapted framework and loss functions for VCE that allow multiple salient factors in each dataset. We evaluate our method on three medical imaging datasets and demonstrate superior counterfactual generation quality compared to existing approaches.
Yunlong He, Pietro Gori
LTCI, Télécom Paris, Institut Polytechnique de Paris, France
Counterfactual image generation answers questions about how a subject would have looked under retrospective, hypothetical scenarios. Recent methods have improved perceptual quality, identity preservation and faithfulness to an underlying causal model, but their adoption in healthcare is limited by scarce annotated data, distribution shift between datasets, and mismatches between pretrained generative models and those required for counterfactual inference. We propose specialisation, a data and parameter-efficient framework for adapting pretrained, non-causal generative models into causal mechanisms under distribution shift. Based on this framework, we train a radiology counterfactual image generation model, called RadCF, using latent flow matching. We validate our approach on three chest X-ray datasets spanning different dataset shifts, data volumes, and counterfactual questions, associated with challenging, highly-localised interventions. Our results show that RadCF and specialisation improve counterfactual soundness over existing methods while being data and parameter efficient, and that the resulting counterfactuals can detect and mitigate shortcut learning in a downstream medical classifier. Code is available at https://github.com/GSK-AI/RadCF/.
Medical image analysis is often hindered by biased datasets, which can lead to biased models and limited clinical applicability. A promising strategy for mitigating such biases is to augment training data with synthetic images. Counterfactual (CF) generation is one such strategy, though the term is used in two different senses: in some works, CFs are produced through causality-based interventions derived from structural causal models, whereas in others, they are produced by non-causal image edits or conventional conditional generative models, such as altering anatomy or adding pathologies. In this work, we study this distinction and evaluate its practical consequences for medical image augmentation. We compare three conditioning strategies: Deterministic, which changes selected variables while holding the remaining variables fixed; Undirected, which updates variables according to learned statistical associations without assigning causal directions; and Causal, which propagates interventions along a directed causal graph. We analyse how these choices affect the resulting images, and explore when causally grounded methods improve dataset augmentation or bring limited benefit. In particular, we assess downstream performance and fairness, where fairness refers to reduced sensitivity to dataset biases across sensitive subgroups. Our experiments demonstrate that using a causal approach to synthetic training data generation can lead to tangible benefits, with these insights offering valuable guidance to machine learning practitioners for the effective design of data generation protocols.
Yasin Ibrahim, Robin J. Evans, Konstantinos Kamnitsas
Department of Engineering Science, University of Oxford · Department of Statistics, University of Oxford