Positional Segmentor-Guided Counterfactual Fine-Tuning for Spatially Localized Image Synthesis
Authors: Tian Xia, Matthew Sinclair, Andreas Schuh, Fabio De Sousa Ribeiro, Raghav Mehta, Rajat Rasal, Esther Puyol-Antón, Samuel Gerber, Kersten Petersen, Michiel Schaap, Ben Glocker
Organizations: Department of Computing, Imperial College London, UK · HeartFlow, Inc., Mountain View, CA, USA
Counterfactual image generation enables controlled data augmentation, bias mitigation, and disease modeling. However, existing methods guided by external classifiers or regressors are limited to subject-level factors (e.g., age) and fail to produce localized structural changes, often resulting in global artifacts. Pixel-level guidance using segmentation masks has been explored, but requires user-defined counterfactual masks, which are tedious and impractical. Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT) addressed this by using segmentation-derived measurements to supervise structure-specific variables, yet it remains restricted to global interventions. We propose Positional Seg-CFT, which subdivides each structure into regional segments and derives independent measurements per region, enabling spatially localized and anatomically coherent counterfactuals. Experiments on coronary CT angiography show that Pos-Seg-CFT generates realistic, region-specific modifications, providing finer spatial control for modeling disease progression.
Figures & tables
Figure 1: Visual results under Reg-CFT and Pos-Seg-CFT for calcified plaque area (CPA) interventions of +20 mm 2 applied to the proximal, mid, and distal regions. Each row shows the counterfactual image (left) and the corresponding difference map (right). Vertical dashed lines indicate the proximal , mid , and distal vessel segments.
Intervention
CFT type
Proximal
Mid
Distal
do(NCPA-proximal)
No-CFT
29.50
8.79
3.25
Reg-CFT
18.71
8.01
2.65
Pos-Seg-CFT
17.85
7.27
2.42
do(NCPA-mid)
No-CFT
13.15
37.22
7.01
Reg-CFT
12.82
26.32
6.15
Pos-Seg-CFT
11.50
23.44
4.91
Table 1: Quantitative effectiveness of causal interventions do(variable–position) across positional regions (Proximal–Mid–Distal). Each block reports results for No-CFT , Reg-CFT , and the proposed Pos-Seg-CFT . The blue and orange values denote the intervened and unintervened regions, respectively. Bold numbers indicate the best (lowest) error.