Hierarchical Vision Transformer with Prototypes for Interpretable Medical Image Classification
Authors: Luisa Gallée, Meinrad Beer, Michael Götz
Organizations: Experimental Radiology, Ulm University Medical Center, Germany · XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology, Germany · Department of Diagnostic and Interventional Radiology, Ulm University Medical Center, Germany · i2SouI - Innovative Imaging in Surgical Oncology Ulm, Ulm University Medical Center, Germany · BGZ - Bildgebungszentrum, Ulm University Medical Center, Germany
Explainability is an important requirement for machine learning models in high-risk domains such as medicine. We introduce HierViT, a hierarchical Vision Transformer that combines competitive predictive performance with multiple mechanisms to support the interpretation of its predictions. The model first predicts clinically relevant visual appearance attributes, which are then used to infer the final target prediction through a hierarchical reasoning process. To facilitate understanding of the intermediate representations, attribute-specific prototypes are visualized as representative image examples, while attribute-specific attention maps highlight image regions associated with each visual feature. Together, these attribute-level components provide complementary views of the model's reasoning by linking representative image examples and spatial attention to semantically defined visual features. We evaluate HierViT on two medical benchmark datasets: LIDC-IDRI for lung nodule assessment and derm7pt for skin lesion classification. Across both datasets, HierViT achieves competitive predictive performance while providing attribute-level prototypes and attention visualizations that support the interpretation of its predictions.
Figures & tables
Fig. 1 : HierViT A hierarchical decision process first extracts diagnostic criteria (attributes) and the region of interest. Target prediction is based on the extracted attribute information. The attributes are defined by humans, explained by a rating, an attention heatmap, and a prototype image illustrating a similar sample.
Fig. 2 : Proposed model The patchified image is linearly projected and processed by a transformer encoder, producing a token vector that serves as the input for both a decoder and a hierarchical classifier. The optional decoder segments a region-of-interest mask. The hierarchical classifier processes the token vector through multiple transformer layers, one for each attribute, with individual heads providing attribute ratings. For target prediction, the token vectors from the attribute layers are stacked and further processed by the target branch.
TABLE I : Annotations of the LIDC-IDRI dataset used in experiments.
TABLE II : Annotations of the derm7pt dataset used in experiments.
Fig. 3 : Reasoning process Three sample cases are illustrated, (a) correctly predicted, (b) and (c) incorrectly predicted. For three of the eight attributes (spiculation, sphericity, lobulation), the score, attention heatmap, and prototype image of the respective attribute are displayed.
attributes
target
sub
is
cal
sph
mar
lob
spic
tex
malignancy
CNN-based
3D-CNN+MTL [ 27 ]
-
-
-
-
-
-
-
-
91.3 [89.7,92.9]
TumorNet [ 28 ]
-
-
-
-
-
-
-
-
92.3 [90.8,93.8]
X-Caps [ 17 ]
90.4
-
-
85.4
84.1
70.7
75.2
93.1
86.4
Concept Bottleneck [ 16 ]
94.7
99.0
93.9
97.0
94.6
89.0
92.4
92.6
91.6 [91.3,91.9]
TABLE III : Performance comparison LIDC-IDRI The performance of the experiments on the LIDC-IDRI dataset is reported in the Within-1-Accuracy metric (for ordinal rating 1-5). In contrast, the results marked with an asterisk * represent the accuracy (ACC) of the binary classification between benign and malignant cases. Mean (black) and standard deviation (grey) are calculated from 5-fold experiments and reported if available in the literature. Methods denoted with ”P” provide reasoning with prototypes. The 95% binomial confidence interval is reported for the target if the test dataset is detailed in the literature. The highest value is in bold.
attributes
target
∅
pn
bmv
vs
pig
str
dag
rs
diagnosis
using dermoscopic images only
Inception- xd [ 25 ]
69.4
85.8
80.3
62.8
71.4
60.8
77.5
71.9 [67.5,76.3]
72.5 [68.1,76.9]
AMFAM [ 32 ]
66.1
87.1
80.5
66.6
71.1
60.0
78.5
69.4 [64.9,73.9]
72.4 [68.0,76.8]
FusionM4Net [ 36 ]
69.0
87.2
81.4
68.3
73.7
60.0
80.1
74.7 [70.4,79.0]
74.3 [70.0,78.6]
MTL-standard [ 37 ]
55.4
85.1
63.5
62.5
49.1
48.6
65.1
45.8 [40.9,50.7]
59.4 [54.6,64.2]
TABLE IV : Performance comparison derm7pt The performance results (in accuracy percentage) are listed, differentiated by data modality. The average ∅ is the mean over the attributes and target (diag). The 95% binomial confidence interval is specified for the target and the average. In the HierViT’s category using dermoscopic images only, the highest values are in bold, and the second-highest are underlined.
attributes
target
sub
is
cal
sph
mar
lob
spic
tex
malignancy
HierViT (proposed)
96.3
99.8
95.5
97.4
92.7
94.3
90.8
93.3
94.8 [94.5,95.1]
0.9
0.3
2.1
0.8
1.7
2.9
1.4
1.4
1.4
A. + proto. inference
93.7
99.8
95.1
92.2
87.9
88.7
86.0
93.0
94.4 [94.1,94.7]
2.0
0.3
1.8
7.5
3.4
3.7
1.9
3.2
1.9
B. − proto. learning
95.6
99.8
95.9
97.3
91.5
94.3
91.3
93.9
95.0 [94.7,95.3]
TABLE V : Ablation study Performance of the proposed method when model components were abstracted. The experiments were conducted with the LIDC-IDRI dataset, and the Within-1-Accuracy was measured. Mean (in black) and standard deviation (in grey) are calculated from 5-fold experiments. The 95% binomial confidence interval is reported for the target.
Fig. 4 : Test-time intervention Intervening in incorrect attribute predictions improves overall accuracy, with performance gains plateauing after three interventions. From this point onward, the ablated model using prototype vector-based inference for malignancy prediction (malignancy A.) achieves performance comparable to the original model. Error bars show the standard deviation of 5-fold experiments.
Fig. 5 : Analysis of learned prototypes (a) agreement between prototypes, ground truth, and made predictions, (b) diversity of used prototypes, and (c) counterfactual prototypes.
Fig. 6 : Sparse data study Performance results with reduced availability of attribute annotations using the LIDC-IDRI dataset. Error bars show the standard deviation of 5-fold experiments.
Fig. 7 : Evaluation of the parameter λproto in the combined loss Equation 7 on the LIDC-IDRI dataset. Labels with the suffix ’A.’ (gray background) indicate results where prototypes were used during inference (see Section V-D for details).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
diag
nev
sk
misc
bcc
mel
using dermoscopic images only
AMFAM derm. only [ 32 ]
sen
73.0
0.0
34.8
14.3
71.7
spe
88.5
95.2
91.4
96.1
86.5
pre
94.1
0.0
20.0
6.3
58.4
auc
88.4
73.6
89.1
90.5
85.2
Appendix
TABLE VI : derm7pt: detailed target performance comparison The target (diag) performance of the experiment on the derm7pt dataset is reported in sensitivity (sen), specificity (spe), precision (pre), and Area Under the Curve (auc). In the HierViT’s category using dermoscopic images only, the highest values are in bold.
pn
bmv
vs
pig
str
dag
rs
abs
typ
aty
abs
pre
abs
reg
irg
abs
reg
irg
abs
reg
irg
abs
reg
irg
abs
pre
using dermoscopic images only
AMFAM derm. only [ 32 ]
sen
73.3
74.2
46.4
87.2
85.3
83.1
48.3
0.0
65.1
72.2
73.6
75.5
1.0
52.4
61.7
52.8
64.8
83.1
62.1
spe
83.8
80.1
85.3
85.3
87.3
69.0
89.6
92.4
83.1
90.7
75.1
68.8
90.9
83.9
81.2
83.0
74.3
62.1
83.1
pre
75.6
63.0
54.8
98.4
38.7
97.1
26.9
0.0
94.6
27.1
31.5
88.7
20.5
46.8
37.0
63.6
70.6
88.6
50.9
auc
86.9
81.4
77.1
87.4
87.4
84.2
85.3
71.1
79.2
82.9
79.6
79.7
86.9
77.1
76.7
75.8
77.2
80.0
80.0
Appendix
TABLE VII : derm7pt: detailed attribute performance comparison The attribute performance of the experiment on the derm7pt dataset is reported in sensitivity (sen), specificity (spe), precision (pre), and Area Under the Curve (auc). In the HierViT’s category using dermoscopic images only, the highest values are in bold.