SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models

Ahmed Zeid; Sidney Bender

cs.LGMar 17, 2026

SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models

Authors: Ahmed Zeid, Sidney Bender

Organizations: Machine Learning Group, Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany

Abstract

Modern neural networks achieve strong performance but remain difficult to interpret in high-dimensional visual domains. Counterfactual explanations (CFEs) provide a principled approach to interpreting black-box predictions by identifying minimal input changes that alter model outputs. However, existing CFE methods often rely on dataset-specific generative models and incur substantial computational cost, limiting their scalability to high-resolution data. We propose SCE-LITE-HQ, a scalable framework for counterfactual generation that leverages pretrained generative foundation models without task-specific retraining. The method operates in the latent space of the generator, incorporates smoothed gradients to improve optimization stability, and applies mask-based diversification to promote realistic and structurally diverse counterfactuals. We evaluate SCE-LITE-HQ on natural and medical datasets using a desiderata-driven evaluation protocol. Results show that SCE-LITE-HQ produces valid, realistic, and diverse counterfactuals competitive with or outperforming existing baselines, while avoiding the overhead of training dedicated generative models.

Figures & tables

Figure 1: Comparison between SCE-LITE-HQ and traditional counterfactual explanation frameworks. Conventional methods (left) rely on computationally intensive training of dataset-specific generative models, whereas SCE-LITE-HQ (right) leverages pretrained Generative Foundation Models (GFMs). We illustrate counterfactuals generated on the CelebA-HQ dataset for the blond attribute. SCE-LITE-HQ produces counterfactuals at the original image resolution while preserving visual quality. In contrast, FastDIME (left) requires adjusting the image resolution, which leads to lower-quality counterfactuals.

Figure 2: Overview of the SCE-LITE-HQ algorithm for generating counterfactual images via latent-space optimization guided by a foundation model and classifier gradients.

Figure 3: Visualization of counterfactual explanations generated at high resolution on the CelebA-HQ dataset using SCE-LITE-HQ and FastDIME as a baseline. SCE-LITE-HQ successfully identifies the confounding attribute of blue eyes, which is highly correlated with blond hair, while FastDIME fails to capture this relationship due to information loss at lower resolutions.

Dataset / Model	Method	(NA)	(sparsity)	(NAFR)	(diversity)	gain
		desiderata
		fidelity	understandability	sufficiency
Celeb-A Blond	ACE	73.3	71.7	44.0	47.3	-
	DIME	64.4	71.1	58.0	51.7	-
	FastDIME	62.5	71.3	50.0	39.9	-
	SCE	100.0	71.4	90.0	74.6	-

Table 1: Quantitative evaluation of counterfactual explanation methods across desiderata metrics. We evaluate fidelity (NA: Non-Adversarial rate), understandability (sparsity in latent space), and sufficiency (NAFR: Non-Adversarial Flip Rate, diversity). The gain column measures improvement in model accuracy after fine-tuning with CFKD ( Bender et al., 2025 ) algorithm. SCE-LITE-HQ achieves competitive performance on standard benchmarks (CelebA-Blond, CelebA-Smile) while demonstrating superior robustness on the poisoned dataset (CelebA-Blond Male) and substantial gains on medical imaging (Camelyon17), effectively identifying spurious correlations for model debiasing. Bold indicates best performance; underline indicates second-best.

Figure 4: Comparison of counterfactual explanations generated by different methods across multiple tasks and datasets. For each example, we show the original input, its predicted label, and the counterfactual generated by each method. On the blond hair classification task, SCE-LITE-HQ achieves a favorable balance between Sparsity, realism, and robustness, successfully flipping the model’s prediction without excessive perturbations (in contrast to SCE, which overshoots, or other methods, which undershoot). On the Camelyon dataset, SCE-LITE-HQ identifies confounding factors, enabling the generation of counterfactuals that expose spurious correlations; these can be used with the CFKD algorithm to fine-tune the original model and improve accuracy, explaining the substantial performance gains achieved by SCE-LITE-HQ on this dataset.

Gradient Source	NAFR	NA	Sparsity	Diversity
Vanilla Gradient	28.0	44.4	73.2	75.5
SmoothGrad	56.0	80.0	72.8	61.3
SmoothDiff	35.0	52.2	74.1	75.0
Integrated Gradients (IG)	36.0	55.4	73.2	74.6
SmoothGrad $\times$ IG	18.0	29.5	70.8	67.3
SmoothDiff $\times$ IG	23.0	34.3	70.4	70.1

Table 2: Effect of gradient source on counterfactual generation (CelebA-Blond, ResNet-18).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Total Time (s)	Time/Sample (s)	Peak GPU (MB)	Peak CPU (MB)
ACE	1001.99	25.05	40961	2803
DIME	1081.61	27.04	965	2867
FastDIME	70.29	1.76	964	2983
SCE	1295.81	32.40	52999	3046
SCE-LITE-HQ	26.24	0.66	11569	8468

Appendix

Table 3: Runtime and memory consumption of counterfactual generation methods on poisoned CelebA-Blond Male (40 samples, 3 runs average, NVIDIA H100 GPU). Peak GPU Mem. measures maximum memory during computation.

Figure 5: High-resolution counterfactual examples on CelebA-HQ. We compare SCE-LITE-HQ against FastDIME, illustrating qualitative differences in visual fidelity and semantic coherence.

Figure 7: Qualitative Comparison on Poisoned CelebA. SCE-LITE-HQ correctly identifies the underlying bias: it transforms blond female factuals into male-featured counterfactuals to satisfy the classification flip. Conversely, for dark-haired males, the model introduces feminine semantic features while preserving the original hair color. These counterfactuals can later be used to debias the model using the CFKD algorithm ( Bender et al., 2025 ) .

Counterfactual Explanations

Counterfactual explanations show how changing an input could alter a model’s prediction, supporting model understanding, auditing, and actionable recourse.

See all 109 papers

Density-Guided Robust Counterfactual Explanations on Tabular Data under Model Multiplicity
May 29, 2026Jun Tan, Qing Guo, Zicheng Xu +3Counterfactual Explanations
School of Computer Science and Engineering, Central South University, Changsha, China.
MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations
Feb 21, 2026Changlu Guo, Anders Nymark Christensen, Anders Bjorholm Dahl +1Counterfactual Image Generation Counterfactual Explanations
Technical University of Denmark, Denmark
CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations
Feb 19, 2026Oleksii Furman, Patryk Marszałek, Jan Masłowski +3Counterfactual Explanations Constrained Generative Modeling
Wrocław University of Science and Technology, Wrocław, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Kraków, Poland · Doctoral School of Exact and Natural Sciences, Jagiellonian University, Kraków, Poland +1

May 29, 2026J/K move · Enter open · S save

Density-Guided Robust Counterfactual Explanations on Tabular Data under Model Multiplicity

Jun Tan, Qing Guo, Zicheng Xu, Jinglin Li+2

School of Computer Science and Engineering, Central South University, Changsha, China.

Counterfactual Explanations

Open paper AskPDF arXiv

Counterfactual explanations (CEs) are essential for actionable recourse, yet their reliability is often compromised in low-density regions, where classifiers exhibit high variance. Unlike existing methods that rely on expensive ensemble intersections to define stability, we propose DensityFlow, a generative framework that constructs robust CEs by adhering to the high-confidence data manifold. Specifically, we model the counterfactual generation as continuous-time dynamics parameterized by Neural ODE, guided by a differentiable density score to actively avoid uncertain, low-density areas. This density score is learned via Noise Contrastive Estimation, effectively leveraging a

(K{+}1)

-way discriminator to estimate density ratios. For black-box settings, we introduce a local proxy distillation mechanism that aligns a lightweight surrogate with the target model strictly within the trajectory of CE generation, enabling efficient gradient-based optimization with minimal queries. Experiments demonstrate that DensityFlow achieves superior validity under model multiplicity while significantly reducing query costs compared to ensemble-based baselines. Our implementation is available at https://github.com/G-AILab/DensityFlow.

Abstract

Counterfactual Explanations

Explore similar work

Density-Guided Robust Counterfactual Explanations on Tabular Data under Model Multiplicity

MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations

CounterFlowNet: From Minimal Changes to Meaningful Counterfactual Explanations