cs.CVApr 8, 2026
SaveDINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification
Organizations: Institute for Information Processing, L3S - Leibniz University Hannover
Abstract
Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work proposes DINO-QPM, which converts these powerful but entangled features into contrastive, class-independent representations that are interpretable by humans. DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification, adapting the Quadratic Programming Enhanced Model (QPM) to operate on strictly frozen DINO backbones. While classification with visual foundation models typically relies on the CLS token, we deliberately diverge from this standard. By leveraging average-pooling, we directly connect the patch embeddings to the model's features and therefore enable spatial localisation of DINO-QPM's globally interpretable features within the input space. Furthermore, we apply a sparsity loss to minimise spatial scatter and background noise, ensuring that explanations are grounded in relevant object parts. With DINO-QPM we make the level of interpretability of QPM available as an adapter while exceeding the accuracy of DINOv2 linear probe. Evaluated through an introduced Plausibility metric and other interpretability metrics, extensive experiments demonstrate that DINO-QPM is superior to other applicable methods for frozen visual foundation models in both classification accuracy and explanation quality.
Figures & tables
Figure 2: Radar Plot demonstrating the quality of DINO-QPM in a set of interpretability metrics and accuracy compared to non-frozen QPM and the corresponding baselines. DINO-QPM outperforms each of them reaching of the maximal score calculated as the fraction of area of the heptagon reached by the respective model. We present rigorous insights in Sec. 5 including a more detailed presentation of the interpretability metrics.
Figure 3: Three-Stage Training Procedure for QPM
Figure 4: Architecture of our proposed DINO-QPM. Patch embeddings extracted using the frozen backbone are first projected via an MLP into the problem-specific feature space. Subsequently, our BLDD layer performs sparse feature assignment to yield a globally interpretable image classification.
Figure 5: Comparison of a Brewer’s Blackbird image with a Rusty Blackbird image. From the selected features , utilised features were selected for both classes using the QP; the corresponding feature maps from are visualised as saliency maps. Both classes share 4 out of the 5 features and can thus be distinguished by the non-shared features. Notably, the model differentiates the Brewer’s Blackbird using feature 24, which localises the beak. This aligns perfectly with established ornithological expertise, where beak morphology is considered a primary diagnostic trait [ 55 , 53 ] .
Figure 6: Visualisation of the Plausibility metric on a sample from the CUB2011 dataset (American Goldfinch). The metric quantifies the fraction of the cumulative weighted feature map activation that falls within the dilated object segmentation mask . The arrows highlight the relevant features at the objects’ edge that are missed by the non-dilated mask.
| Method | Local. Features | Accuracy | Plausibil. | SID@5 | Class-Indep. | Contrast. | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| CUB | CARS | CUB | CARS | CUB | CARS | CUB | CARS | |||
| DINOv2 Linear Probe | ✗ | 99.2 | 99.1 | |||||||
| Dense | ✓ | 91.8 | 93.1 | |||||||
| Resnet50 Baseline [ 41 ] | ✓ | |||||||||
| Resnet50 QPM [ 41 ] | ✓ | |||||||||
| DINO-SLDD | ✓ | |||||||||
Table 1: Comparison with state-of-the-art interpretable models. We report Accuracy, Plausibility, SID@5, Class-Independence, and Contrastiveness (all metrics in ). Features of a model are localised if they have a direct connection to the feature vector used for classification. The Plausibility metric is evaluated only on CUB-2011 due to the availability of segmentation masks. Dense is the dense model of DINO-QPM and DINOv2 Linear Probe is a linear probe [ 10 ] trained on top of the frozen CLS representation. For DINO-SLDD and DINO-QSENN, we employ a pipeline closely resembling the one described in Sec. 4 , with the exception of the feature selection mechanisms, which follow Norrenbrock et al. [42] and Norrenbrock et al. [43] , respectively.
| Method | Features ( ) | Features per class ( ) | Avg. Training Time per Epoch [s] |
|---|---|---|---|
| DINOv2 Linear Probe | |||
| DINOv2 Baseline | |||
| Resnet50 Baseline | |||
| Resnet50 QPM | |||
| DINO-SLDD | |||
| DINO-QSENN |
Table 2: Comparison of model complexity and training efficiency. We report the total number of features ( ), features per class ( ), and the average training time per epoch in seconds on the referenced hardware ( Sec. 9 ).
| Input Source | Loc. Feat. | Registers | Acc. | Plausibil. |
|---|---|---|---|---|
| ✗ | ✗ | |||
| ✓ | ||||
| ✓ | ✗ | |||
| ✓ |
Table 3: Ablation study showing the impact of input source and register tokens on CUB-2011. The local features column indicates whether the input representation preserves spatial information.
| Backbone Architecture | Acc. | Patch Context. |
|---|---|---|
| DINO ViT-B/16 | ||
| DINOv2 ViT-S/14 Reg. | ||
| DINOv2 ViT-B/14 Reg. | ||
| DINOv2 ViT-L/14 Reg. |
Table 4: Comparison of different backbone architectures on CUB-2011. We compare DINO with DINOv2 and different ViT sizes and report accuracy alongside Patch Contextualisation.
Figure 7: Impact of the number of MLP layers ( ) on classification accuracy. The plot compares the accurcy on CUB-2011 of using frozen patch-level feature maps ( ) versus the global feature vector ( ) for both Dense and QPM
Figure 8: Impact of sparsity constraints on classification accuracy. We vary the total number of selected features and the number of features assigned per class . Compared to the QPM [ 41 ] we observe relatively low impact on accuracy.
Figure 9: Effect of varying the weight during finetuning compared to the dense model. Higher penalty weights yield substantial improvements in both accuracy and plausibility metrics.
Figure 10: Qualitative ablation of on the Gray-crowned Rosy Finch. Without (top row), feature activations exhibit background noise and spatial scatter. Adding (bottom row) suppresses this noise, resulting in distinct activations semantically localised to specific object parts.
Figure 11: Accuracy and Feature Diversity (SID@5) on CUB-2011 across variations of the weight during dense and finetuning training.
Figure 12: Impact of the on CUB-2011 accuracy during finetuning.
Figure 13: Mean finetuning accuracy on CUB-2011 for various numbers of features across a range of hidden layer neurons in the MLP. We observe small accuracy gains up until regardless of the number of features .
Figure 14: Faithful global interpretability on CUB-2011: DINO-QPM autonomously discovers the 5 diverse, generalisable features for each class used to represent the Hooded Oriole and Hooded Warbler, completely without external supervision. The probed QPM distinguishes them using their evidently different throat.
Figure 15: Faithful global interpretability on CUB-2011: DINO-QPM autonomously discovers the 5 diverse, generalisable features for each class used to represent the Blue Winged Warbler and Kentucky Warbler, completely without external supervision. The probed QPM distinguishes them using their evidently different eye area.
Figure 16: Faithful global interpretability on Stanford Cars: DINO-QPM autonomously discovers the 5 diverse, generalisable features for each class used to represent the Acura Rl Sedan 2012 and Cadillac Cts-V Sedan 2012, completely without external supervision. The probed QPM distinguishes them using their evidently different headlights.
Figure 17: Faithful global interpretability on Stanford Cars: DINO-QPM autonomously discovers the 5 diverse, generalisable features for each class used to represent the Bentley Continental Gt Coupe 2007 and Bentley Continental Flying Spur Sedan 2007, completely without external supervision. The probed QPM distinguishes them using their evidently different door configurations. The probed QPM distinguishes them using their evidently different door configurations. As the most prominent distinguishing factor is the number of doors, the model’s non-overlapping features for the Flying Spur (Sedan) specifically highlight the rear doors and rear door handles, which the GT (Coupe) lacks
Figure 18: Comparison on the Orchard Oriole (CUB-2011). We show test samples where our DINO-QPM correctly classifies the image while the dense baseline fails. Columns from left to right: original image ( ), GradCAM activation map of the dense model, the most similar training sample from the dense-predicted class alongside its cosine similarity score ( ), and the colour-coded local explanation of DINO-QPM for the true class. Row labels indicate the dense prediction (top) and the DINO-QPM prediction (bottom). The dense model consistently confuses Orchard Orioles with visually similar species by attending to non-discriminative regions such as foliage and branches. In contrast, DINO-QPM correctly localises diverse features strictly on the bird’s body, enabling accurate classification despite the visual similarity to other species.
Figure 19: Comparison on the Louisiana Waterthrush (CUB-2011). We show test samples where our DINO-QPM correctly classifies the image while the dense baseline fails. Columns from left to right: original image ( ), GradCAM activation map of the dense model, the most similar training sample from the dense-predicted class alongside its cosine similarity score ( ), and the colour-coded local explanation of DINO-QPM for the true class. Row labels indicate the dense prediction (top) and the DINO-QPM prediction (bottom). The dense model consistently confuses the Louisiana Waterthrush with extremely similar species (e.g., Northern Waterthrush or Palm Warbler), often attending to less discriminative regions. In contrast, DINO-QPM correctly localises diverse features strictly on the bird’s body, enabling accurate classification despite the extreme visual overlap between these species.
Figure 20: Comparison on the Summer Tanager (CUB-2011). We compare the dense baseline and DINO-QPM on eight test images, correctly classified by both models. Each sample is shown as a triplet: the original image ( ), the GradCAM attribution of the dense model, and the local explanation of DINO-QPM for the true class . The GradCAM attributions of the dense model frequently spread across the background or miss the bird entirely (e.g., samples on the right), demonstrating inconsistent localisation despite correct predictions. In contrast, DINO-QPM’s local explanation consistently focuses on the bird, decomposing it into interpretable parts such as the red body plumage (Feature 42), the upper head (Feature 32), and the eye region (Feature 38). This illustrates that DINO-QPM not only localises more reliably but also provides explicit part-level evidence for its decisions, whereas the dense model relies on diffuse, poorly grounded visual cues.
Figure 21: Indigo Bunting samples from the CUB-2011 test set. We compare the dense baseline and DINO-QPM on eight test images, correctly classified by both models. Each sample is shown as a triplet: the original image ( ), the GradCAM attribution of the dense model, and the local explanation of DINO-QPM for the true class . While both models localise the bird reliably across varying poses and backgrounds, the key difference lies in what each model communicates. The dense model focuses on a single discriminative region, resulting in a non-diverse localisation. In contrast, DINO-QPM decomposes its focus into semantically distinct parts—e.g. the belly (Feature 20), the mantle and back (Feature 7), and the head region (Feature 12)—offering a richer, more diverse, part-level explanation of why the prediction is made.
Figure 22: Failure analysis on the Green Kingfisher (CUB-2011). Comparison of test samples misclassified by both models. Columns (left to right): original image ( ); dense model GradCAM; DINO-QPM local explanations for both the true and predicted classes; and the nearest training sample from the predicted class with its cosine similarity ( ). Although both models struggle to distinguish these highly fine-grained kingfisher species, their failure modes differ significantly. The dense model provides no meaningful insight into its errors, whereas DINO-QPM transparently communicates the source of its confusion through faithful, part-level local explanations. Notably, some of these misclassifications might be due to incorrect ground-truth labels [ 63 ] . For example, the fourth sample appears to be incorrectly annotated, demonstrating how our concept-based explanations can assist in auditing dataset quality.
Figure 23: Failure analysis on the Least Flycatcher (CUB-2011). We show test samples where the dense baseline classifies correctly but DINO-QPM does not. Columns from left to right: original image ( ), GradCAM attribution of the dense model, DINO-QPM local explanations for the true and predicted classes, and the most similar training sample from the predicted class with its cosine similarity score ( ). While we previously demonstrated that DINO-QPM successfully overcomes poorly localised dense representations, this strict feature decomposition can occasionally induce errors. In this failure case, although the dense baseline correctly predicts the target class using ungrounded cues, DINO-QPM’s refusal to exploit these uninterpretable shortcuts leads to confusion among visually similar flycatcher and Sayornis species.
More in this topic
Explainable Image Classification
Explainable image classification develops ways to show which pixels, regions, concepts, or model features support an image classifier’s predictions.
See all 96 papers