Visual counterfactual explanations aim to reveal the minimal semantic modifications that can alter a model's prediction, providing causal and interpretable insights into deep neural networks. However, existing diffusion-based counterfactual generation methods are often computationally expensive, slow to sample, and imprecise in localizing the modified regions. To address these limitations, we propose MaskDiME, a simple, fast, yet effective diffusion framework that unifies semantic consistency and spatial precision through localized sampling. Our approach adaptively focuses on decision-relevant regions to achieve localized and semantically consistent counterfactual generation while preserving high image fidelity. Our training-free framework, MaskDiME, performs inference over 30x faster than the baseline and achieves comparable or state-of-the-art performance across five benchmark datasets spanning diverse visual domains, establishing a practical and generalizable solution for efficient counterfactual explanation.
Figures & tables
Figure 1: Previous methods such as DiME [ 20 ] and FastDiME [ 45 ] produce global or scattered edits, whereas our MaskDiME achieves localized, decision-relevant modifications consistent with the classifier’s saliency map, computed via SmoothGrad [ 39 ] .
Figure 2: Overview of MaskDiME. We illustrate a complete counterfactual generation process ( No Smile → Smile ) with τ=60 , meaning that the diffusion starts from z60=z~60 and x60=x , where zt and xt denote the noisy and clean images at step t , respectively , and z~t is obtained from the original forward diffusion process ( Eq. 1 ) . Each step applies Gradient-Guided Denoising, modeled as N(μθ(zt)−Σθ(zt)∇zt,Σθ(zt)) , where the gradient ∇zt is derived from Eq. 7 . The adaptive masks Mtx⊆Mtz are derived from classifier gradients on xt , where top- k% and top- ρk% gradient regions construct Mtz and Mtx , respectively ( ρ∈(0,1] ).
Figure 3: Heatmap visualization of diffusion trajectories with different masking strategies. Each column shows noisy samples zt at different timesteps, and each row corresponds to a masking method. The heatmap represents the per-pixel update magnitude during each reverse step (red indicates stronger updates, blue indicates weaker ones). Our Adaptive Dual-mask yields the most focused and semantically consistent updates across diffusion steps.
Method
Smile
Age
FID↓
sFID↓
FVA↑
FS↑
MNAC↓
CD↓
COUT↑
FR(%)↑
FID↓
sFID↓
FVA↑
FS↑
MNAC↓
CD↓
COUT↑
FR(%)↑
CelebA
DiVE
29.4
-
97.3
-
-
-
-
-
33.8
-
98.2
-
4.58
-
-
-
DiVE 100
36.8
-
73.4
-
4.63
2.34
-
-
39.9
-
52.2
-
4.27
-
-
-
STEEX
10.2
-
96.9
-
4.11
-
-
-
11.8
-
97.5
-
3.44
-
-
-
DiME
3.17
4.89
98.3
0.73
3.72
2.30
0.53
97.2
4.15
5.89
95.3
0.67
3.13
3.27
0.44
99.0
Table 1: CelebA and CelebA-HQ assessment. We report results extracted from DiVE [ 32 ] , STEEX [ 18 ] , DiME [ 20 ] , ACE [ 21 ] , FastDiME [ 45 ] , LDCE [ 9 ] , TiME [ 22 ] , and RCSB [ 40 ] . Best and second-best results are shown in bold and underline , respectively.
Method
FID↓
sFID↓
S3 ↑
COUT↑
FR(%)↑
BDD100K
STEEX
58.80
–
–
–
99.5
DiME †
7.94
11.40
0.95
0.24
90.5
ACE l1
1.02
6.25
1.0
0.75
99.9
ACE l2
1.56
6.53
1.0
0.79
99.9
TiME
51.5
76.18
0.77
0.15
81.8
Table 2: BDD assessment . We extract results from STEEX [ 18 ] , ACE [ 21 ] and TiME [ 22 ] , where results marked with † are directly reported from ACE. Best and second-best results are shown in bold and underline , respectively.
Method
Sorrel – Zebra
Persian Cat – Egyptian Cat
Cougar – Cheetah
FID↓
sFID↓
S3 ↑
COUT↑
FR(%)↑
FID↓
sFID↓
S3 ↑
COUT↑
FR(%)↑
FID↓
sFID↓
S3 ↑
COUT↑
FR(%)↑
DiME †
222.9
243.2
0.19
-0.31
0.0
322.8
352.1
0.44
-0.05
0.0
268.2
292.0
0.11
-0.16
0.0
ACE l1
84.5
122.7
0.92
-0.45
47.0
93.6
156.7
0.85
0.25
85.0
70.2
100.5
0.91
0.02
77.0
ACE l2
67.7
98.4
0.90
-0.25
81.0
107.3
160.4
0.78
0.34
97.0
74.1
102.5
0.88
0.12
95.0
LDCE-cls
84.2
107.2
0.78
-0.06
88.0
102.7
140.7
0.63
0.52
99.0
71.0
91.8
0.62
0.51
100.0
LDCE-txt
82.4
107.2
0.71
-0.21
81.0
121.7
162.4
0.61
0.56
99.0
91.2
117.0
0.59
0.34
98.0
Table 3: ImageNet Assessement . We extract results from ACE [ 21 ] , LDCE [ 9 ] , and RCSB [ 40 ] , where results marked with † (e.g. DVCE [ 1 ] ) are directly reported from RCSB. Best and second-best results are shown in bold and underline , respectively.
Figure 4: Comparison of methods on the CelebA smile attribute by FID (from Tab. 1 ), runtime (batch size = 5). The area of the circles indicates the peak GPU memory allocated during the sampling process. MaskDiME is significantly faster than previous methods, while also achieving the lowest FID, and sustaining low GPU usage—approximately one-tenth of that required by ACE and RCSB. See Supplementary Tab. 7 for quantitative results.
Figure 5: Qualitative results. Compared with ACE l1 , MaskDiME effectively preserves the overall image structure and produces more pronounced counterfactual explanations, with superior performance in semantic consistency, visual realism, and modification precision.
Figure 6: Ablation study on the No Smile → Smile sample of CelebA. Increasing the gradient scaling ( s ) and introducing Adaptive Dual-mask ( M ) progressively improve MaskDiME’s results, localizing edits to decision-relevant regions and yielding realistic, semantically consistent counterfactuals.
Method
FID↓
FS↑
MNAC↓
CD↓
COUT↑
FR(%)↑
DiME
3.17
0.73
3.72
2.30
0.53
97.2
s =1 & w/o mask
95.76
0.63
6.15
2.43
-0.16
55.2
s =8 & w/o mask
15.94
0.77
5.71
4.20
0.96
100.0
Fixed Mask
4.21
0.86
2.98
2.03
0.70
99.7
s =8 & mask ( ρ =1)
0.71
0.90
2.66
2.25
0.81
100.0
MaskDiME ( ρ =0.5)
0.71
0.91
2.78
2.41
0.87
100.0
Table 4: Ablation study of MaskDiME on Smile of CelebA.
Medical counterfactual generation modifies images to change model predictions for interpretability. However, existing diffusion-based approaches are often prohibitively slow and memory-intensive, making them difficult to apply in high-resolution settings. Moreover, existing masking strategies are tightly coupled with pixel-space representations, making them incompatible with latent-space diffusion editing. To address these challenges, we propose MedDiME, a latent-space classifier-guided diffusion framework that reduces computational and memory overhead while introducing a latent-compatible, gradient-driven adaptive masking mechanism for spatially precise medical counterfactual generation. Extensive experiments demonstrate that MedDiME achieves high-quality counterfactual generation with significant efficiency gains compared to prior classifier-guided diffusion baselines, achieving up to 40 times faster inference and 13 times lower peak GPU memory usage.
Yan Zeng, Changlu Guo, Anders Nymark Christensen +2
Department of Applied Mathematics and Computer Science, Technical University of Denmark
Visual counterfactual explanations aim to change classifier decisions through realistic and localized edits while preserving decision-irrelevant content. Existing DDPM-based methods typically perform classifier-guided editing along a long reverse denoising trajectory. The changing noise levels make semantic editability and spatial control difficult to balance, and the editable state is noisy, whereas the target classifier is trained on clean images. As a result, these methods require either costly recursive denoising or low-quality one-step estimates to obtain classifier-facing clean images. We propose FiRe, a Fixed-noise Refinement framework for visual counterfactual explanations. Rather than following a reverse denoising trajectory, FiRe maps the input to a fixed noise level and iteratively refines the noisy state at that level. To provide clean images for classifier guidance, FiRe first adapts Pixel Mean Flow to visual counterfactual explanation, enabling direct clean-image prediction from noisy states. To make fixed-noise refinement produce minimal and localized counterfactual edits, FiRe introduces three FiRe-specific controls: a dynamic dual-mask strategy, adaptive guidance, and early stopping, which determine where edits accumulate, which changes become visible, and when refinement stops. Experiments on five tasks across three datasets show that, compared with the strongest recent baseline, FiRe achieves about 3× faster online inference and 8× fewer FLOPs while obtaining comparable or state-of-the-art counterfactual quality.
Yan Zeng, Changlu Guo, Oskar Kristoffersen +3
Department of Applied Mathematics and Computer Science, Technical University of Denmark
Modern neural networks achieve strong performance but remain difficult to interpret in high-dimensional visual domains. Counterfactual explanations (CFEs) provide a principled approach to interpreting black-box predictions by identifying minimal input changes that alter model outputs. However, existing CFE methods often rely on dataset-specific generative models and incur substantial computational cost, limiting their scalability to high-resolution data. We propose SCE-LITE-HQ, a scalable framework for counterfactual generation that leverages pretrained generative foundation models without task-specific retraining. The method operates in the latent space of the generator, incorporates smoothed gradients to improve optimization stability, and applies mask-based diversification to promote realistic and structurally diverse counterfactuals. We evaluate SCE-LITE-HQ on natural and medical datasets using a desiderata-driven evaluation protocol. Results show that SCE-LITE-HQ produces valid, realistic, and diverse counterfactuals competitive with or outperforming existing baselines, while avoiding the overhead of training dedicated generative models.
Ahmed Zeid, Sidney Bender
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany