Counterfactual medical image generation have emerged as a critical tool for enhancing AI-driven systems in medical domain by answering "what-if" questions. However, existing approaches face two fundamental limitations: First, they fail to prevent unintended modifications, resulting collateral changes in demographic attributes when only disease features should be affected. Second, they lack interpretability in their editing process, which significantly limits their utility in real-world medical applications. To address these limitations, we present InstructX2X, a novel interpretable local editing model for counterfactual medical image generation featuring Region-Specific Editing. This approach restricts modifications to specific regions, effectively preventing unintended changes while simultaneously providing a Guidance Map that offers inherently interpretable visual explanations of the editing process. Additionally, we introduce MIMIC-EDIT-INSTRUCTION, a dataset for counterfactual medical image generation derived from expert-verified medical VQA pairs. Through extensive experiments, InstructX2X achieve state-of-the-art performance across all major evaluation metrics. Our model successfully generates high-quality counterfactual chest X-ray images along with interpretable explanations.
Figures & tables
Figure 1: Comparison of counterfactual medical image generation results between existing methods and our proposed approach. When adding edema features to an input chest X-ray image, existing methods (a–d) demonstrate unintended modifications (red arrows), causing significant variations in age and race (note the demographic predictions below each image). In contrast, InstructX2X preserves the demographic attributes while achieving precise editing and provides a visual explanation via guidance map (red overlay).
Figure 2: Overview of the InstructX2X training framework. The pipeline is adapted from the InstructPix2Pix architecture [ 7 ] , modified to accept longitudinal chest X-ray pairs. The top panel illustrates the dataset construction process converting descriptions pairs into the MIMIC-EDIT-INSTRUCTION data. The bottom panel shows the training pipeline where the model learns to transform Ipast to Icur using the constructed instructions.
Split
Number of Samples
Train
19,204
Holdout
2,248
Test
334
Validation
171
Total
21,957
Operation Type
Count (Percentage)
Table 1: Statistics of the MIMIC-EDIT-INSTRUCTION dataset. The dataset is split into training, holdout, test, and validation sets. The operation distribution shows the count of each editing type (Add, Remove, Change). Note that the total count of operations exceeds the number of samples because some samples contain multiple operations.
Figure 3: Overview of our proposed Region-Specific Editing (RSE) mechanism. We apply RSE on top of the inference pipeline of a pre-trained InstructPix2Pix backbone [ 7 ] . The relevance map computation [1] is adapted from prior work [ 20 ] , while the anatomical pseudo masks and their integration with the relevance map to form the guidance map are proposed in this thesis. By restricting edits to instruction-relevant regions, RSE prevents unintended modifications and provides inherent interpretability via the visual guidance map.
Model
CMIG ( ↑ )
KL ( ↓ )
FID ( ↓ )
Patho ( ↑ )
Race ( ↑ )
Age ( ↑ )
CMIG ( ↑ )
Real Images (GT)
84.66
99.56
88.83
90.99
-
-
LLM-CXR
55.82
48.62
0.10
52.06
60.03
51.42
RoentGen
84.53
53.15
18.60
67.74
48.92
35.19
RadEdit
82.80
80.33
48.24
80.14
38.85
26.31
BiomedJourney
81.27
79.07
64.37
80.31
20.45
12.46
Table 2: Quantitative comparison of InstructX2X with baselines and the ablation study. CMIG, KL divergence, and FID metrics are reported. Bold indicates the best performance among generated methods.
Figure 4: Demonstration of fine-grained controllable editing by InstructX2X. The model generates counterfactual images with varying severities (Small vs. Moderate) and precise anatomical locations (Right, Left, Bilateral) for pleural effusion. The guidance maps (red heatmaps) visualize the model’s focus, confirming that edits are strictly confined to the instruction-specified regions and intensities.