Why Does It Look There? Structured Explanations for Image Classification

Jiarui Li; Zixiang Yin; Samuel J Landry; Zhengming Ding; Ramgopal R. Mettu

cs.CVMar 10, 2026

Why Does It Look There? Structured Explanations for Image Classification

Authors: Jiarui Li, Zixiang Yin, Samuel J Landry, Zhengming Ding, Ramgopal R. Mettu

Organizations: Department of Computer Science, Tulane University, New Orleans LA 70118, USA · Department of Biochemistry and Molecular Biology, Tulane University School of Medicine, New Orleans LA 70112, USA

Abstract

Deep learning models achieve remarkable predictive performance, yet their black-box nature limits transparency and trustworthiness. Although numerous explainable artificial intelligence (XAI) methods have been proposed, they primarily provide saliency maps or concepts (i.e., unstructured interpretability). Existing approaches often rely on auxiliary models (\eg, GPT, CLIP) to describe model behavior, thereby compromising faithfulness to the original models. We propose Interpretability to Explainability (I2X), a framework that builds structured explanations directly from unstructured interpretability by quantifying progress at selected checkpoints during training using prototypes extracted from post-hoc XAI methods (e.g., GradCAM). I2X answers the question of "why does it look there" by providing a structured view of both intra- and inter-class decision making during training. Experiments on MNIST and CIFAR10 demonstrate effectiveness of I2X to reveal prototype-based inference process of various image classification models. Moreover, we demonstrate that I2X can be used to improve predictions across different model architectures and datasets: we can identify uncertain prototypes recognized by I2X and then use targeted perturbation of samples that allows fine-tuning to ultimately improve accuracy. Thus, I2X not only faithfully explains model behavior but also provides a practical approach to guide optimization toward desired targets.

Figures & tables

Figure 1: The difference between unstructured interpretability ( i.e . , with saliency maps) and our approach to structured explainability. I2X tracks model evolution across training checkpoints using a prototype-based representation.

Figure 2: Interpretability to Explainability (I2X). I2X is a framework that builds structured explanation from evolutions of prototype intensity from saliency maps obtained by post-hoc method, and model confidences. I2X can systematically explain how a model infers and learns a predictive label and further utilize the resulting structured explanation to guide optimization toward the desired targets then improve performance.

Panel of Figure 3 — Figure 3: Visualization of Prototype and Confidence Evolution of ResNet-50 on MNIST (shared prototypes of digit 7 as example). The first column shows the abstract prototypes of digit 7. Subsequent columns depict how sharing each prototype affects confidence for digit 7 (increase or decrease) and which competing class drives the change. White blanks denote no confidence change at that checkpoint, while black blanks indicate a change because 7 contains the prototype whereas another digit does not.

Figure 4: Annotated Training Checkpoints for ResNet-50 on MNIST (shared prototypes of digit 7 as example). This figure shows how the raw model evolves across training checkpoints. An edge

a\to b

denotes that the model decreases confidence of class

a

and increase confidence of class

b

. For example,

6,2\to T0

indicates that the model increases the prediction confidence of digit 7 (training checkpoint

T0

) while decreasing the confidence of digits 6 and 2. The labels on the arrows specify the prototypes responsible for each confidence change.

Figure 5: Annotated training checkpoints for ResNet-50 on MNIST illustrating all prototypes involved in distinguishing digit 7 from digit 2. Prototypes P-20 and P-19 are consistently assigned as discriminative evidence for digit 7. In contrast, prototype P-26 confuses model with its contribution alternating between digits 7 and 2 across training checkpoints, indicating uncertain evidence during training.

Figure 6: Annotated training checkpoints for ResNet-50 on MNIST illustrating how a re-trained model distinguishes digit 7 from digit 2. The figure demonstrates that, due to the randomness in training data order, different training runs result in distinct prototype-based inference strategies.

Figure 7: Visualization of prototype contributions distinguishing digit 7 from digit 2 for the secound round re-trained model. Each column shows how sharing a given prototype affects the prediction confidence for digit 7 and indicates which other digit is responsible for the confidence change at each training checkpoint.

Figure 8: Difference of confusion matrices between the two training rounds evaluated on the test dataset. Due to the distinct training strategies, the second training introduces an uncertain prototype that increases confusion between digits 7 and 2.

Dataset	Acc. (%)	$7\to 2$	$2\to 7$	$2\leftrightarrow 7$
full	98.52 $\pm$ 0.34	11.40 $\pm$ 8.69	3.40 $\pm$ 4.03	14.80 $\pm$ 6.31
curated	98.67 $\pm$ 0.18	5.60 $\pm$ 3.26	4.20 $\pm$ 3.19	9.80 $\pm$ 2.93
full $\to$ full	98.46 $\pm$ 0.31	7.60 $\pm$ 3.38	2.00 $\pm$ 2.53	9.60 $\pm$ 2.87
curated $\to$ curated	98.31 $\pm$ 0.63	6.40 $\pm$ 5.95	2.60 $\pm$ 2.42	9.00 $\pm$ 4.90
curated $\to$ full	98.64 $\pm$ 0.12	4.80 $\pm$ 2.79	3.60 $\pm$ 3.44	8.40 $\pm$ 1.85

Table 1: Accuracy (%) and confusion statistics between digits 2 and 7 after fine-tuning on the full explanation dataset (10% of the training data) and the curated dataset derived from structured explanations. Here,

2\to 7

denotes samples with true label 2 predicted as 7,

7\to 2

denotes the opposite, and

2\leftrightarrow 7

represents their sum.

Panel of Figure 9 — Figure 9: Identified uncertain prototypes in ResNet-50 on CIFAR10 and InceptionV3 on MNIST. Prototype P-72 (ResNet-50, CIFAR10) captures the edge between black and orange regions, leading to confusion between orange cats and dogs. Prototype P-7 (InceptionV3, MNIST) corresponds to the upper-right arc of digit 9, which resembles certain handwritten forms of digit 4, thereby causing misclassification.

	ResNet-50 on CIFAR10		InceptionV3 on MNIST
Dataset	Acc. (%)	cat $\leftrightarrow$ dog	Acc. (%)	$4\leftrightarrow 9$
full $\to$ full	81.43 $\pm$ 2.79	261.20 $\pm$ 30.77	99.13 $\pm$ 0.29	12.60 $\pm$ 3.07
curated $\to$ full	84.02 $\pm$ 2.70	238.60 $\pm$ 21.90	99.11 $\pm$ 0.27	10.80 $\pm$ 2.71

Table 2: Accuracy (%) and confusion statistics between objects cat and dog or digit 4 and 9 after fine-tuning on the full explanation dataset (10% of the training data) and the curated dataset derived from structured explanations.

Explainable Image Classification

Explainable image classification develops ways to show which pixels, regions, concepts, or model features support an image classifier’s predictions.

See all 90 papers

OPTIMUS-Prime: Minimal and Sufficient Concept Explanations for Deep Vision Models
Jun 5, 2026Arthur Hoarau, Chenrui Zhu, Vu Linh NguyenExplainable Artificial Intelligence Concept-Based Explanations
Université de Lorraine, CentraleSupélec Loria, CNRS, Metz, France · Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc, France
MOXIE: Discovering Alternative Explanations for Biomedical Image Classifiers
Oct 3, 2026Abiha Tahsin Chowdhury, Rahul DubeyExplainable Image Classification Explainable Medical Image Analysis
Department of Computer Science, Missouri State University, Springfield, MO, USA
UNBOX: Unveiling Black-box visual models with Natural-language
Mar 9, 2026Simone Carnemolla, Chiara Russo, Simone Palazzo +5Neural Network Interpretability Explainable Image Classification
University of Catania. · Technical University of Munich. · Helmholtz Munich.

Jun 5, 2026J/K move · Enter open · S save

OPTIMUS-Prime: Minimal and Sufficient Concept Explanations for Deep Vision Models

Arthur Hoarau, Chenrui Zhu, Vu Linh Nguyen

Université de Lorraine, CentraleSupélec Loria, CNRS, Metz, France · Université de technologie de Compiègne UMR CNRS 7253 Heudiasyc, France

Explainable Artificial Intelligence Concept-Based Explanations Neural Network Interpretability

Open paper AskPDF arXiv

The growing demand for transparency in automated decision-making has propelled eXplainable Artificial Intelligence (XAI) to the forefront of machine learning research. In computer vision, however, existing explanation methods often prioritize end-user accessibility at the expense of formal guarantees, leaving a critical gap between practical utility and theoretical rigor. In this paper, we address this gap by introducing OPTIMUS, a novel framework for generating concept-based visual explanations for deep classification models. OPTIMUS explanations take the form of visual heatmaps that not only remain interpretable to end users, but are grounded in the well-established theory of prime implicants, providing formal guarantees that have been largely absent from existing saliency-based methods. Specifically, OPTIMUS explanations satisfy two desirable properties: sufficiency, ensuring that the highlighted concepts provably guarantee the classifier's prediction, and minimality, ensuring that no strict subset of those concepts retains this guarantee. Together, these properties yield explanations that are both logically tight and visually coherent. We validate our approach on a visual classification benchmark, demonstrating that OPTIMUS heatmaps naturally and faithfully surface the decision-relevant concepts underlying model predictions.

Abstract

Explainable Image Classification

Explore similar work

OPTIMUS-Prime: Minimal and Sufficient Concept Explanations for Deep Vision Models

MOXIE: Discovering Alternative Explanations for Biomedical Image Classifiers

UNBOX: Unveiling Black-box visual models with Natural-language