Medical image analysis is often hindered by biased datasets, which can lead to biased models and limited clinical applicability. A promising strategy for mitigating such biases is to augment training data with synthetic images. Counterfactual (CF) generation is one such strategy, though the term is used in two different senses: in some works, CFs are produced through causality-based interventions derived from structural causal models, whereas in others, they are produced by non-causal image edits or conventional conditional generative models, such as altering anatomy or adding pathologies. In this work, we study this distinction and evaluate its practical consequences for medical image augmentation. We compare three conditioning strategies: Deterministic, which changes selected variables while holding the remaining variables fixed; Undirected, which updates variables according to learned statistical associations without assigning causal directions; and Causal, which propagates interventions along a directed causal graph. We analyse how these choices affect the resulting images, and explore when causally grounded methods improve dataset augmentation or bring limited benefit. In particular, we assess downstream performance and fairness, where fairness refers to reduced sensitivity to dataset biases across sensitive subgroups. Our experiments demonstrate that using a causal approach to synthetic training data generation can lead to tangible benefits, with these insights offering valuable guidance to machine learning practitioners for the effective design of data generation protocols.
Figures & tables
Fig. 1 : Effect of intervention with each approach; Yellow: intervened, Green: affected, Red: unaffected.
Fig. 2 : Model outline. Green: observed, Grey: latent, Red: predicted, Blue: causal generative, Yellow: decoding. (left) Training: we use the y predictions for decoding unless they are observed, (right) CF generation.
Fig. 3 : Example coloured MNIST images
Fig. 4 : We implemented four generative processes for synthesising four coloured MNIST datasets, when relations between variables are follow the above DAGs. Variables: Digit d , thickness t , intensity i , background colour b , foreground colour f .
Fig 5: True relations (green) are reversed in the misspecified model (red). TABLE V: Similar to Table II but for misspecified model (Fig 5). “Wrong assumptions” affect Causal .
Fig 6: DAG used for the chest x-ray experiments. Age (a) causes pleural effusion (p) , and both cause lung volume (l) . TABLE VI: Effect on parent of changing child variable for each of the approaches. Age is measured in MAE, Pleural Effusion (PE) is measured by percentage error. Effects followed by * are evaluated via the proxy approach described above and thus, measured using the intervention.
Labels
Bias
No Aug.
Sampling
Counterfactual
Det.
Undir.
Causal
Det.
Undir.
Causal
100%
0.5
91.4
92.0
91.0
91.8
91.2
91.9
91.6
0.8
86.4
87.8
87.4
87.3
88.0
87.2
87.6
0.9
85.1
85.1
85.2
83.5
84.3
86.7
85.0
0.95
81.8
82.2
83.6
83.1
82.6
84.0
84.3
1
50.3
51.5
51.1
51.8
52.8
51.3
52.7
TABLE VII : AUC of pleural effusion predictor for raw CheXpert data (no augmentation) and augmented with synthetic data from each of the three conditioning approaches and each of the two generation methods. Green indicates the best AUC in that row. Bold indicates the best performance within the specific generation approach (sampling/counterfactual).
Metric
Sampling
Counterfactual
Det.
Undir.
Causal
Det.
Undir.
Causal
FID
2.280
2.292
3.307
1.815
2.221
1.730
MMD
0.054
0.049
0.055
0.051
0.048
0.038
SSIM
4.37%
5.36%
4.79%
4.01%
3.85%
4.07%
LPIPS
0.198
0.157
0.158
0.147
0.135
0.150
TABLE VIII : Quality of the data generated by each method.
Fig. 7 : Left: Examples of sampled (top) and counterfactual (bottom) images. Right: PCA of embeddings of images generated via Sampling vs Counterfactual when 1% data is labelled.
Fig. 8 : Diversity of the data generated by each of the methods. Higher indicates greater diversity for each metric.
Fig. 9 : AUC of each method stratified by age group.
Method
Acc. ( ↑ )
AUC ( ↑ )
F1 ( ↑ )
Gap - Sex ( ↓ )
Gap AUC - Age ( ↓ )
No Augmentation
0.7262
0.8729
0.5963
0.0214
0.0867
Sample
Deterministic
0.7490
0.8918
0.6027
0.0192
0.0552
Undirected
0.7638
0.8899
0.6129
0.0187
0.0697
Causal
0.7604
0.8922
0.6183
0.0199
0.0519
CF
Deterministic
0.7518
0.8905
0.6115
0.0196
0.0622
Undirected
0.7652
0.8942
0.6186
0.0202
0.0774
TABLE IX : Performance on MIMIC data of methods trained using non-augmented and augmented Chexpert data.
Fig. 10 : Density estimate of ages from samples produced by each method. Undirected samples age depends on disease status, resulting in a difference between ages of samples with pleural effusion (red) and those without (green). Deterministic and Causal sample age unconditionally, resulting in greater overlap between samples with and without pleural effusion.
Fig. 11 : Example interventions using Causal vs Undirected . On the left (red), we intervene with do(disease) and on the right (green) with do(healthy). For each of these we show the original image, interventions using Causal and Undirected , and difference maps between original and counterfactuals.
Counterfactual image generation answers questions about how a subject would have looked under retrospective, hypothetical scenarios. Recent methods have improved perceptual quality, identity preservation and faithfulness to an underlying causal model, but their adoption in healthcare is limited by scarce annotated data, distribution shift between datasets, and mismatches between pretrained generative models and those required for counterfactual inference. We propose specialisation, a data and parameter-efficient framework for adapting pretrained, non-causal generative models into causal mechanisms under distribution shift. Based on this framework, we train a radiology counterfactual image generation model, called RadCF, using latent flow matching. We validate our approach on three chest X-ray datasets spanning different dataset shifts, data volumes, and counterfactual questions, associated with challenging, highly-localised interventions. Our results show that RadCF and specialisation improve counterfactual soundness over existing methods while being data and parameter efficient, and that the resulting counterfactuals can detect and mitigate shortcut learning in a downstream medical classifier. Code is available at https://github.com/GSK-AI/RadCF/.
Deep learning models for medical diagnosis frequently exhibit substantial performance disparities across sensitive subgroups (e.g., race, sex), even when average accuracy is high. While generative data augmentation offers a route to mitigate this, existing strategies are suboptimal; they typically address only one or two dependency channels between sensitive attributes and image features. We formalize the medical image formation process via a structural causal model, revealing that sensitive attributes actually influence image content through four distinct pathways-a structural complexity neglected by prior works. Based on this insight, we introduce CIPHER (Causal Intervention Pathways for Healthcare Equity and Robustness), a framework designed to systematically intervene on all four causal paths. To achieve this, CIPHER utilizes a diffusion backbone equipped with classifier-free guidance and null-text inversion. This technical design enables the faithful reconstruction of patient-specific anatomy while allowing for the precise, editable synthesis of counterfactuals required to break sensitive dependency chains. We tested CIPHER using chest X-ray and dermoscopy benchmarks across both standard and shifted data distributions. By employing a multi-pathway intervention strategy, our model reduced worst-group disparities by an average of 35.8% compared to disease-conditioned synthesis baselines, while also improving total diagnostic accuracy
Xinyu Jia, Weidong Guo, Wangyuan Zhao +3
College of Biomedical Engineering, Fudan University, Shanghai, China
Counterfactual medical image generation have emerged as a critical tool for enhancing AI-driven systems in medical domain by answering "what-if" questions. However, existing approaches face two fundamental limitations: First, they fail to prevent unintended modifications, resulting collateral changes in demographic attributes when only disease features should be affected. Second, they lack interpretability in their editing process, which significantly limits their utility in real-world medical applications. To address these limitations, we present InstructX2X, a novel interpretable local editing model for counterfactual medical image generation featuring Region-Specific Editing. This approach restricts modifications to specific regions, effectively preventing unintended changes while simultaneously providing a Guidance Map that offers inherently interpretable visual explanations of the editing process. Additionally, we introduce MIMIC-EDIT-INSTRUCTION, a dataset for counterfactual medical image generation derived from expert-verified medical VQA pairs. Through extensive experiments, InstructX2X achieve state-of-the-art performance across all major evaluation metrics. Our model successfully generates high-quality counterfactual chest X-ray images along with interpretable explanations.