Why Does It Look There? Structured Explanations for Image Classification
Authors: Jiarui Li, Zixiang Yin, Samuel J Landry, Zhengming Ding, Ramgopal R. Mettu
Organizations: Department of Computer Science, Tulane University, New Orleans LA 70118, USA · Department of Biochemistry and Molecular Biology, Tulane University School of Medicine, New Orleans LA 70112, USA
Deep learning models achieve remarkable predictive performance, yet their black-box nature limits transparency and trustworthiness. Although numerous explainable artificial intelligence (XAI) methods have been proposed, they primarily provide saliency maps or concepts (i.e., unstructured interpretability). Existing approaches often rely on auxiliary models (\eg, GPT, CLIP) to describe model behavior, thereby compromising faithfulness to the original models. We propose Interpretability to Explainability (I2X), a framework that builds structured explanations directly from unstructured interpretability by quantifying progress at selected checkpoints during training using prototypes extracted from post-hoc XAI methods (e.g., GradCAM). I2X answers the question of "why does it look there" by providing a structured view of both intra- and inter-class decision making during training. Experiments on MNIST and CIFAR10 demonstrate effectiveness of I2X to reveal prototype-based inference process of various image classification models. Moreover, we demonstrate that I2X can be used to improve predictions across different model architectures and datasets: we can identify uncertain prototypes recognized by I2X and then use targeted perturbation of samples that allows fine-tuning to ultimately improve accuracy. Thus, I2X not only faithfully explains model behavior but also provides a practical approach to guide optimization toward desired targets.
Figures & tables
Figure 1: The difference between unstructured interpretability ( i.e . , with saliency maps) and our approach to structured explainability. I2X tracks model evolution across training checkpoints using a prototype-based representation.
Figure 2: Interpretability to Explainability (I2X). I2X is a framework that builds structured explanation from evolutions of prototype intensity from saliency maps obtained by post-hoc method, and model confidences. I2X can systematically explain how a model infers and learns a predictive label and further utilize the resulting structured explanation to guide optimization toward the desired targets then improve performance.
Figure 3: Visualization of Prototype and Confidence Evolution of ResNet-50 on MNIST (shared prototypes of digit 7 as example). The first column shows the abstract prototypes of digit 7. Subsequent columns depict how sharing each prototype affects confidence for digit 7 (increase or decrease) and which competing class drives the change. White blanks denote no confidence change at that checkpoint, while black blanks indicate a change because 7 contains the prototype whereas another digit does not.
Figure 4: Annotated Training Checkpoints for ResNet-50 on MNIST (shared prototypes of digit 7 as example). This figure shows how the raw model evolves across training checkpoints. An edge a→b denotes that the model decreases confidence of class a and increase confidence of class b . For example, 6,2→T0 indicates that the model increases the prediction confidence of digit 7 (training checkpoint T0 ) while decreasing the confidence of digits 6 and 2. The labels on the arrows specify the prototypes responsible for each confidence change.
Figure 5: Annotated training checkpoints for ResNet-50 on MNIST illustrating all prototypes involved in distinguishing digit 7 from digit 2. Prototypes P-20 and P-19 are consistently assigned as discriminative evidence for digit 7. In contrast, prototype P-26 confuses model with its contribution alternating between digits 7 and 2 across training checkpoints, indicating uncertain evidence during training.
Figure 6: Annotated training checkpoints for ResNet-50 on MNIST illustrating how a re-trained model distinguishes digit 7 from digit 2. The figure demonstrates that, due to the randomness in training data order, different training runs result in distinct prototype-based inference strategies.
Figure 7: Visualization of prototype contributions distinguishing digit 7 from digit 2 for the secound round re-trained model. Each column shows how sharing a given prototype affects the prediction confidence for digit 7 and indicates which other digit is responsible for the confidence change at each training checkpoint.
Figure 8: Difference of confusion matrices between the two training rounds evaluated on the test dataset. Due to the distinct training strategies, the second training introduces an uncertain prototype that increases confusion between digits 7 and 2.
Dataset
Acc. (%)
7→2
2→7
2↔7
full
98.52 ± 0.34
11.40 ± 8.69
3.40 ± 4.03
14.80 ± 6.31
curated
98.67 ± 0.18
5.60 ± 3.26
4.20 ± 3.19
9.80 ± 2.93
full → full
98.46 ± 0.31
7.60 ± 3.38
2.00 ± 2.53
9.60 ± 2.87
curated → curated
98.31 ± 0.63
6.40 ± 5.95
2.60 ± 2.42
9.00 ± 4.90
curated → full
98.64 ± 0.12
4.80 ± 2.79
3.60 ± 3.44
8.40 ± 1.85
Table 1: Accuracy (%) and confusion statistics between digits 2 and 7 after fine-tuning on the full explanation dataset (10% of the training data) and the curated dataset derived from structured explanations. Here, 2→7 denotes samples with true label 2 predicted as 7, 7→2 denotes the opposite, and 2↔7 represents their sum.
Figure 9: Identified uncertain prototypes in ResNet-50 on CIFAR10 and InceptionV3 on MNIST. Prototype P-72 (ResNet-50, CIFAR10) captures the edge between black and orange regions, leading to confusion between orange cats and dogs. Prototype P-7 (InceptionV3, MNIST) corresponds to the upper-right arc of digit 9, which resembles certain handwritten forms of digit 4, thereby causing misclassification.
ResNet-50 on CIFAR10
InceptionV3 on MNIST
Dataset
Acc. (%)
cat ↔ dog
Acc. (%)
4↔9
full → full
81.43 ± 2.79
261.20 ± 30.77
99.13 ± 0.29
12.60 ± 3.07
curated → full
84.02 ± 2.70
238.60 ± 21.90
99.11 ± 0.27
10.80 ± 2.71
Table 2: Accuracy (%) and confusion statistics between objects cat and dog or digit 4 and 9 after fine-tuning on the full explanation dataset (10% of the training data) and the curated dataset derived from structured explanations.
The growing demand for transparency in automated decision-making has propelled eXplainable Artificial Intelligence (XAI) to the forefront of machine learning research. In computer vision, however, existing explanation methods often prioritize end-user accessibility at the expense of formal guarantees, leaving a critical gap between practical utility and theoretical rigor. In this paper, we address this gap by introducing OPTIMUS, a novel framework for generating concept-based visual explanations for deep classification models. OPTIMUS explanations take the form of visual heatmaps that not only remain interpretable to end users, but are grounded in the well-established theory of prime implicants, providing formal guarantees that have been largely absent from existing saliency-based methods. Specifically, OPTIMUS explanations satisfy two desirable properties: sufficiency, ensuring that the highlighted concepts provably guarantee the classifier's prediction, and minimality, ensuring that no strict subset of those concepts retains this guarantee. Together, these properties yield explanations that are both logically tight and visually coherent. We validate our approach on a visual classification benchmark, demonstrating that OPTIMUS heatmaps naturally and faithfully surface the decision-relevant concepts underlying model predictions.
Arthur Hoarau, Chenrui Zhu, Vu Linh Nguyen
Université de Lorraine, CentraleSupélec Loria, CNRS, Metz, France · Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc, France
Segment-based explanation methods such as LIME return a single explanation for each prediction, computed from one fixed image segmentation. This hides two important facts: a prediction can be supported by many different sets of image segments, and the segmentation itself shapes which explanations can be found. We introduce MOXIE (Multi-Objective eXplanation Imaging Engine), an evolutionary framework that searches for segment subsets that preserve the classifier's confidence while keeping as little of the image as possible. Instead of one explanation, MOXIE returns a Pareto front of alternative explanations that range from compact to highly faithful. We evaluate MOXIE with NSGA-II and four segmentation methods (SLIC, Felzenszwalb, Watershed and Voronoi) on BloodMNIST and HAM10000 datasets, using the same evaluation budget as LIME. Results show that MOXIE achieves a higher hypervolume than LIME on every image. LIME's explanations often appear convincing, yet the classifier's confidence collapses when only the highlighted segments are shown. MOXIE's fronts reveal how much of the image is needed to preserve the model's confidence and which contextual regions influence it. We also find that segmentation strongly affects evaluation: methods with unequal segment sizes appear most compact when segments are counted. These results show that alternative explanations provide a more complete view of a model's decision than a single explanation.
Abiha Tahsin Chowdhury, Rahul Dubey
Department of Computer Science, Missouri State University, Springfield, MO, USA
Ensuring trustworthiness in open-world visual recognition requires models that are interpretable, fair, and robust to distribution shifts. Yet modern vision systems are increasingly deployed as proprietary black-box APIs, exposing only output probabilities and hiding architecture, parameters, gradients, and training data. This opacity prevents meaningful auditing, bias detection, and failure analysis. Existing explanation methods assume white- or gray-box access or knowledge of the training distribution, making them unusable in these real-world settings. We introduce UNBOX, a framework for class-wise model dissection under fully data-free, gradient-free, and backpropagation-free constraints. UNBOX leverages Large Language Models and text-to-image diffusion models to recast activation maximization as a purely semantic search driven by output probabilities. The method produces human-interpretable text descriptors that maximally activate each class, revealing the concepts a model has implicitly learned, the training distribution it reflects, and potential sources of bias. We evaluate UNBOX on ImageNet-1K, Waterbirds, and CelebA through semantic fidelity tests, visual-feature correlation analyses and slice-discovery auditing. Despite operating under the strictest black-box constraints, UNBOX performs competitively with state-of-the-art white-box interpretability methods. This demonstrates that meaningful insight into a model's internal reasoning can be recovered without any internal access, enabling more trustworthy and accountable visual recognition systems.