Authors: Saketh Vishnubhatla, Pingchuan Ma, Shu Wan, Alimohammad Beigi, Andre Harrison, Adrienne Raglin, Huan Liu
Organizations: School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ 85281, USA · DEVCOM Army Research Laboratory, Adelphi, MD 20783, USA
Counterfactual inference is the task of answering hypothetical "what-if" queries about a unit, given an observational dataset, modeling assumptions, and a causal graph. This enables a modeler to reason about hypothetical scenarios that cannot otherwise be answered without assumptions or a causal graph. The simpler case involves a causal graph with scalar 1-D variables that denote different attributes in the data-generation process. While the literature is clear in the 1-D case, extending this to settings with high-dimensional nodes (e.g., images) remains challenging. Conventionally, in the literature, methods recover the latent state from the factual image (inversion), generate counterfactual attributes from the graph, and render the counterfactual image from the latent state using the updated attributes. In our work, we investigate the necessity of the inversion step in computing high-dimensional counterfactuals. We first discuss moving from the scalar 1-D case to the higher-dimensional case. Secondly, we discuss the relevance of the inversion step and evaluate an inversion-free counterpart repurposed from the image-editing literature. Finally, on image datasets such as CelebA and ADNI, we benchmark our approach using two measures: namely, effectiveness and identity preservation. While effectiveness is a measure of how faithfully the counterfactual attribute is rendered, identity preservation measures whether the same unit (image) is being rendered, both of which are crucial for high-dimensional counterfactual inference. We present the tradeoffs of using inversion-based and inversion-free approaches in our work.
Figures & tables
Fig. 1: The two transports, both given the same counterfactual attributes obtained by propagating the antecedent on age through the structural equations. Left: inversion carries the factual to noise under the source condition and regenerates under the target. Right: the inversion-free map moves the factual directly, by accumulated velocity differences between the two conditions, always staying on the image manifold.
Fig. 2: Attribute-level diagram and structural equations used as a running example, following Melistas et al. [17] : ApoE- ε4 status ap , sex s , age a , brain volume b and ventricular volume v . The image node is introduced in Section IV-B .
Fig. 3: Abduction at fixed parents. Solid curves are fv(a,b,⋅) at the observed brain volume b , dashed curves the same mechanism under do(B=b∗) . In (a) the observed v has a single preimage u^v , and the counterfactual is read off the dashed curve at that point. In (b) the mechanism attains v at two exogenous values, and the two preimages give different counterfactuals.
Fig. 4: The two-stage model. The attribute mechanisms and their exogenous variables are those of Fig. 2 . All five attributes pa=(ap,s,a,b,v) are rendered into the image by g (grey edges), together with a high-dimensional exogenous state ux carrying everything the diagram does not name. The labelling map ℓ recovers the attributes from the image.
Method
Image mechanism
Exogenous state
Pawlowski et al. [21]
VAE, trained
latent + residual
De Sousa Ribeiro et al. [3]
HVAE, trained
latents + residual
Melistas et al. [17]
GAN, trained
single latent
Sanchez and Tsaftaris [24]
diffusion, trained
uncond. noise map
Rasal et al. [23]
diff. AE, trained
noise map, semantic code, guidance tokens
De Sousa Ribeiro et al. [2]
OT-coupled flow
flow noise map
TABLE I: Image mechanisms and the exogenous states they recover.
Fig. 5: The attribute-level diagram assumed on CelebA.
Effectiveness F1 ↑
Realism FID ↓
Identity LPIPS ↓
Method
Backbone
a
g
br
bl
a
g
br
bl
a
g
br
bl
Explicit inversion — encoder; DDIM for Causal-Adapter, flow-ODE for SD3-inv
VAE
deep-SCM
0.390
0.990
0.284
0.004
60.0
63.4
59.9
57.4
0.283
0.295
0.283
0.284 †
HVAE
deep-SCM
0.246
0.742
0.476
0.122
2.2
5.9
25.6
31.7
0.026 †
0.076
0.138
0.229 †
GAN
deep-SCM
0.471
0.989
0.273
0.340
22.1
26.6
23.0
24.3
0.256
0.363
0.262
0.285
Causal-Adapter
SD1.5 + adapter
0.530
0.997
0.424
0.497
8.2
9.0
8.0
17.7
0.166
0.246
0.119
0.240
TABLE II: CelebA counterfactuals, grouped by whether the exogenous state is recovered at all. Best per column in bold, runner-up underlined. Cells marked † are excluded from the ranking.
Fig. 6: CelebA counterfactuals with four antecedents, one method per column.
Effectiveness F1 ↑
Identity LPIPS ↓
Inference map
Fwd./query
a
g
br
bl
a
g
br
bl
copy input
0
0.142
0.349
0.050
0.007
0.000
0.000
0.000
0.000
sample at pa∗
32
0.569
0.992
0.312
0.700
0.495
0.517
0.508
0.517
inversion-free, M=8
512
0.674
0.994
0.361
0.724
0.070
0.325
0.128
0.430
+ FlowAlign, γ=0.01
512
0.420
0.968
0.247
0.550
0.033
0.239
0.104
0.402
inversion, N=32
64
0.610
0.992
0.314
0.717
0.148
0.343
0.188
0.432
TABLE III: One model, several inference maps. All rows share the same scratch-trained DiT-B/2 (131 M, SD3 latents, independent coupling), so no difference between them is attributable to training. The first two rows are bounds and are excluded from the ranking; best per column in bold, runner-up underlined.
Fig. 7: CelebA counterfactuals from the scratch-trained model, one inference map per column.
Fig. 8: do(vent_vol→p95) on four held-out subjects, taken at even quantiles of the requested change among stable outputs.
Inversion
Inversion-free
Effectiveness
Delivered (mm 3 ) ↑
+41,459±4,671
+23,634±2,732
Realised ↑
82.1%
46.8%
Identity (LPIPS)
Reconstruction ↓
0.0508
0.0223
Edit magnitude, median (max)
0.149 (0.618)
0.062 (0.108)
TABLE IV: Counterfactual editing on ADNI, do(vent_vol→p95) . on The Same Backbone.