MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations

Changlu Guo; Anders Nymark Christensen; Anders Bjorholm Dahl; Morten Rieger Hannemose

cs.CVFeb 21, 2026

MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations

Authors: Changlu Guo, Anders Nymark Christensen, Anders Bjorholm Dahl, Morten Rieger Hannemose

Organizations: Technical University of Denmark, Denmark

Abstract

Visual counterfactual explanations aim to reveal the minimal semantic modifications that can alter a model's prediction, providing causal and interpretable insights into deep neural networks. However, existing diffusion-based counterfactual generation methods are often computationally expensive, slow to sample, and imprecise in localizing the modified regions. To address these limitations, we propose MaskDiME, a simple, fast, yet effective diffusion framework that unifies semantic consistency and spatial precision through localized sampling. Our approach adaptively focuses on decision-relevant regions to achieve localized and semantically consistent counterfactual generation while preserving high image fidelity. Our training-free framework, MaskDiME, performs inference over 30x faster than the baseline and achieves comparable or state-of-the-art performance across five benchmark datasets spanning diverse visual domains, establishing a practical and generalizable solution for efficient counterfactual explanation.

Figures & tables

Figure 1: Previous methods such as DiME [ 20 ] and FastDiME [ 45 ] produce global or scattered edits, whereas our MaskDiME achieves localized, decision-relevant modifications consistent with the classifier’s saliency map, computed via SmoothGrad [ 39 ] .

Figure 2: Overview of MaskDiME. We illustrate a complete counterfactual generation process ( No Smile → Smile ) with $\tau=60$ , meaning that the diffusion starts from $z_{60}=\tilde{z}_{60}$ and $x_{60}=x$ , where $z_{t}$ and $x_{t}$ denote the noisy and clean images at step $t$ , respectively , and $\tilde{z}_{t}$ is obtained from the original forward diffusion process ( Eq. 1 ) . Each step applies Gradient-Guided Denoising, modeled as $\mathcal{N}\!\big(\mu_{\theta}(z_{t})-\Sigma_{\theta}(z_{t})\nabla_{z_{t}},\,\Sigma_{\theta}(z_{t})\big)$ , where the gradient $\nabla_{z_{t}}$ is derived from Eq. 7 . The adaptive masks $M_{t}^{x}\subseteq M_{t}^{z}$ are derived from classifier gradients on $x_{t}$ , where top- $k\%$ and top- $\rho k\%$ gradient regions construct $M_{t}^{z}$ and $M_{t}^{x}$ , respectively ( $\rho\in(0,1]$ ).

Figure 3: Heatmap visualization of diffusion trajectories with different masking strategies. Each column shows noisy samples $z_{t}$ at different timesteps, and each row corresponds to a masking method. The heatmap represents the per-pixel update magnitude during each reverse step (red indicates stronger updates, blue indicates weaker ones). Our Adaptive Dual-mask yields the most focused and semantically consistent updates across diffusion steps.

Method	Smile								Age
Method	FID↓	sFID↓	FVA↑	FS↑	MNAC↓	CD↓	COUT↑	FR(%)↑	FID↓	sFID↓	FVA↑	FS↑	MNAC↓	CD↓	COUT↑	FR(%)↑
	CelebA
DiVE	29.4	-	97.3	-	-	-	-	-	33.8	-	98.2	-	4.58	-	-	-
DiVE 100	36.8	-	73.4	-	4.63	2.34	-	-	39.9	-	52.2	-	4.27	-	-	-
STEEX	10.2	-	96.9	-	4.11	-	-	-	11.8	-	97.5	-	3.44	-	-	-
DiME	3.17	4.89	98.3	0.73	3.72	2.30	0.53	97.2	4.15	5.89	95.3	0.67	3.13	3.27	0.44	99.0

Table 1: CelebA and CelebA-HQ assessment. We report results extracted from DiVE [ 32 ] , STEEX [ 18 ] , DiME [ 20 ] , ACE [ 21 ] , FastDiME [ 45 ] , LDCE [ 9 ] , TiME [ 22 ] , and RCSB [ 40 ] . Best and second-best results are shown in bold and underline , respectively.

BDD100K
Method	FID↓	sFID↓	$S^{3}$ ↑	COUT↑	FR(%)↑
STEEX	58.80	–	–	–	99.5
DiME †	7.94	11.40	0.95	0.24	90.5
ACE $l_{1}$	1.02	6.25	1.0	0.75	99.9
ACE $l_{2}$	1.56	6.53	1.0	0.79	99.9
TiME	51.5	76.18	0.77	0.15	81.8

Table 2: BDD assessment . We extract results from STEEX [ 18 ] , ACE [ 21 ] and TiME [ 22 ] , where results marked with † are directly reported from ACE. Best and second-best results are shown in bold and underline , respectively.

Method	Sorrel – Zebra					Persian Cat – Egyptian Cat					Cougar – Cheetah
Method	FID↓	sFID↓	$S^{3}$ ↑	COUT↑	FR(%)↑	FID↓	sFID↓	$S^{3}$ ↑	COUT↑	FR(%)↑	FID↓	sFID↓	$S^{3}$ ↑	COUT↑	FR(%)↑
DiME †	222.9	243.2	0.19	-0.31	0.0	322.8	352.1	0.44	-0.05	0.0	268.2	292.0	0.11	-0.16	0.0
ACE $l_{1}$	84.5	122.7	0.92	-0.45	47.0	93.6	156.7	0.85	0.25	85.0	70.2	100.5	0.91	0.02	77.0
ACE $l_{2}$	67.7	98.4	0.90	-0.25	81.0	107.3	160.4	0.78	0.34	97.0	74.1	102.5	0.88	0.12	95.0
LDCE-cls	84.2	107.2	0.78	-0.06	88.0	102.7	140.7	0.63	0.52	99.0	71.0	91.8	0.62	0.51	100.0
LDCE-txt	82.4	107.2	0.71	-0.21	81.0	121.7	162.4	0.61	0.56	99.0	91.2	117.0	0.59	0.34	98.0

Table 3: ImageNet Assessement . We extract results from ACE [ 21 ] , LDCE [ 9 ] , and RCSB [ 40 ] , where results marked with † (e.g. DVCE [ 1 ] ) are directly reported from RCSB. Best and second-best results are shown in bold and underline , respectively.

Figure 4: Comparison of methods on the CelebA smile attribute by FID (from Tab. 1 ), runtime (batch size = 5). The area of the circles indicates the peak GPU memory allocated during the sampling process. MaskDiME is significantly faster than previous methods, while also achieving the lowest FID, and sustaining low GPU usage—approximately one-tenth of that required by ACE and RCSB. See Supplementary Tab. 7 for quantitative results.

Figure 5: Qualitative results. Compared with ACE $l_{1}$ , MaskDiME effectively preserves the overall image structure and produces more pronounced counterfactual explanations, with superior performance in semantic consistency, visual realism, and modification precision.

Figure 6: Ablation study on the No Smile $\rightarrow$ Smile sample of CelebA. Increasing the gradient scaling ( $s$ ) and introducing Adaptive Dual-mask ( $M$ ) progressively improve MaskDiME’s results, localizing edits to decision-relevant regions and yielding realistic, semantically consistent counterfactuals.

Method	FID↓	FS↑	MNAC↓	CD↓	COUT↑	FR(%)↑
DiME	3.17	0.73	3.72	2.30	0.53	97.2
$s$ =1 & w/o mask	95.76	0.63	6.15	2.43	-0.16	55.2
$s$ =8 & w/o mask	15.94	0.77	5.71	4.20	0.96	100.0
Fixed Mask	4.21	0.86	2.98	2.03	0.70	99.7
$s$ =8 & mask ( $\rho$ =1)	0.71	0.90	2.66	2.25	0.81	100.0
MaskDiME ( $\rho$ =0.5)	0.71	0.91	2.78	2.41	0.87	100.0

Table 4: Ablation study of MaskDiME on Smile of CelebA.

Counterfactual Image Generation

See all 18 papers

MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation
Sep 14, 2026Yan Zeng, Changlu Guo, Anders Nymark Christensen +2Counterfactual Explanations Latent Diffusion Models
Department of Applied Mathematics and Computer Science, Technical University of Denmark
FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanations
Aug 9, 2026Yan Zeng, Changlu Guo, Oskar Kristoffersen +3Counterfactual Explanations Diffusion Guidance
Department of Applied Mathematics and Computer Science, Technical University of Denmark
SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models
Mar 17, 2026Ahmed Zeid, Sidney BenderCounterfactual Explanations
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany

Sep 14, 2026J/K move · Enter open · S save

MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation

Yan Zeng, Changlu Guo, Anders Nymark Christensen, Morten Rieger Hannemose+1

Department of Applied Mathematics and Computer Science, Technical University of Denmark

Counterfactual Explanations Latent Diffusion Models Causal Counterfactual Generation Explainable Medical Image Analysis Diffusion Model Acceleration Diffusion-Based Image Editing+1

Open paper AskPDF arXiv

Medical counterfactual generation modifies images to change model predictions for interpretability. However, existing diffusion-based approaches are often prohibitively slow and memory-intensive, making them difficult to apply in high-resolution settings. Moreover, existing masking strategies are tightly coupled with pixel-space representations, making them incompatible with latent-space diffusion editing. To address these challenges, we propose MedDiME, a latent-space classifier-guided diffusion framework that reduces computational and memory overhead while introducing a latent-compatible, gradient-driven adaptive masking mechanism for spatially precise medical counterfactual generation. Extensive experiments demonstrate that MedDiME achieves high-quality counterfactual generation with significant efficiency gains compared to prior classifier-guided diffusion baselines, achieving up to 40 times faster inference and 13 times lower peak GPU memory usage.

Abstract

Counterfactual Image Generation

Explore similar work

MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation

FiRe: Fixed-Noise Refinement for Visual Counterfactual Explanations

SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models