SCE-LITE-HQ: Smooth visual counterfactual explanations with generative foundation models
Organizations: Machine Learning Group, Technische Universität Berlin, Berlin, Germany · BIFOLD – Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Abstract
Modern neural networks achieve strong performance but remain difficult to interpret in high-dimensional visual domains. Counterfactual explanations (CFEs) provide a principled approach to interpreting black-box predictions by identifying minimal input changes that alter model outputs. However, existing CFE methods often rely on dataset-specific generative models and incur substantial computational cost, limiting their scalability to high-resolution data. We propose SCE-LITE-HQ, a scalable framework for counterfactual generation that leverages pretrained generative foundation models without task-specific retraining. The method operates in the latent space of the generator, incorporates smoothed gradients to improve optimization stability, and applies mask-based diversification to promote realistic and structurally diverse counterfactuals. We evaluate SCE-LITE-HQ on natural and medical datasets using a desiderata-driven evaluation protocol. Results show that SCE-LITE-HQ produces valid, realistic, and diverse counterfactuals competitive with or outperforming existing baselines, while avoiding the overhead of training dedicated generative models.
Figures & tables
| desiderata | |||||||
|---|---|---|---|---|---|---|---|
| fidelity | understandability | sufficiency | |||||
| Dataset / Model | Method | (NA) | (sparsity) | (NAFR) | (diversity) | gain | |
| Celeb-A Blond | ACE | 73.3 | 71.7 | 44.0 | 47.3 | - | |
| DIME | 64.4 | 71.1 | 58.0 | 51.7 | - | ||
| FastDIME | 62.5 | 71.3 | 50.0 | 39.9 | - | ||
| SCE | 100.0 | 71.4 | 90.0 | 74.6 | - | ||
| Gradient Source | NAFR | NA | Sparsity | Diversity |
|---|---|---|---|---|
| Vanilla Gradient | 28.0 | 44.4 | 73.2 | 75.5 |
| SmoothGrad | 56.0 | 80.0 | 72.8 | 61.3 |
| SmoothDiff | 35.0 | 52.2 | 74.1 | 75.0 |
| Integrated Gradients (IG) | 36.0 | 55.4 | 73.2 | 74.6 |
| SmoothGrad IG | 18.0 | 29.5 | 70.8 | 67.3 |
| SmoothDiff IG | 23.0 | 34.3 | 70.4 | 70.1 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Total Time (s) | Time/Sample (s) | Peak GPU (MB) | Peak CPU (MB) |
|---|---|---|---|---|
| ACE | 1001.99 | 25.05 | 40961 | 2803 |
| DIME | 1081.61 | 27.04 | 965 | 2867 |
| FastDIME | 70.29 | 1.76 | 964 | 2983 |
| SCE | 1295.81 | 32.40 | 52999 | 3046 |
| SCE-LITE-HQ | 26.24 | 0.66 | 11569 | 8468 |
Counterfactual Explanations
Counterfactual explanations show how changing an input could alter a model’s prediction, supporting model understanding, auditing, and actionable recourse.
See all 109 papers