Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
Organizations: Kioxia Corporation
Abstract
Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.
Figures & tables
| ImageNet-Seg. | PASCAL VOC | |||||
| Loss | mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) |
| DINOv2 | 63.67 | 80.69 | 85.39 | 58.10 | 75.28 | 84.79 |
| w/ | 63.93 | 80.61 | 86.03 | 57.82 | 74.76 | 85.00 |
| w/ | 63.65 | 80.94 | 84.69 | 58.23 | 75.73 | 84.97 |
| w/ & | 67.29 | 83.20 | 86.43 | 60.68 | 77.36 | 85.27 |
| COCO | PASCAL VOC | ||
| Loss | Retrieval scheme | ||
| DINOv2 | Fixed per image | 54.2 | 72.8 |
| DINOv2 | Visual prompt per object | 55.2 | 74.2 |
| w/ & | Fixed per image | 52.7 | 72.4 |
| w/ & | Visual prompt per object | 55.9 | 75.8 |
| ImageNet-Seg. | PASCAL VOC | ||||||
| Size | Loss | mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) |
| B | DINOv2 | 68.32 | 83.81 | 87.63 | 62.20 | 78.14 | 86.46 |
| w/ & | 69.33 | 84.40 | 87.84 | 62.98 | 78.65 | 86.77 | |
| g | DINOv2 | 63.37 | 80.53 | 85.20 | 58.34 | 75.55 | 84.85 |
| w/ & | 67.61 | 83.31 | 86.86 | 61.31 | 77.65 | 86.09 | |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| ImageNet-Seg. | PASCAL VOC | |||||
| Method | mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) |
| DeiT III ( Touvron et al., 2022 ) | 38.21 | 59.73 | 74.36 | 37.98 | 58.43 | 77.54 |
| CLIP ( Radford et al., 2021 ) | 47.06 | 72.06 | 76.41 | 45.37 | 69.59 | 77.66 |
| DINOv2 w/o reg. ( Oquab et al., 2024 ) | 52.35 | 76.14 | 82.29 | 52.75 | 74.40 | 83.88 |
| DINOv2 w/ reg. ( Darcet et al., 2024 ) | 53.71 | 73.95 | 81.61 | 55.50 | 74.53 | 84.03 |
| ImageNet-Seg. | PASCAL VOC | |||||
| Loss | mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) |
| Pretrained | 53.71 | 73.95 | 81.61 | 55.50 | 74.53 | 84.03 |
| DINOv2 | 54.90 | 74.76 | 82.15 | 53.07 | 72.14 | 83.23 |
| w/ | 57.26 | 75.64 | 83.87 | 52.37 | 70.68 | 82.70 |
| w/ | 60.90 | 78.79 | 84.71 | 55.04 | 73.32 | 84.24 |
| w/ & | 61.63 | 78.86 | 85.40 | 55.70 | 73.30 | 84.73 |
| Loss | VOC07 | VOC12 | COCO |
| Pretrained | 56.2 | 60.2 | 43.0 |
| DINOv2 | 57.0 | 61.4 | 44.6 |
| w/ | 58.1 | 62.9 | 45.1 |
| w/ | 53.5 | 59.1 | 42.5 |
| w/ & | 57.0 | 61.6 | 44.6 |
| Loss weights | ImageNet-Seg. | PASCAL VOC | ||||
| mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) | |
| 51.94 | 72.04 | 81.23 | 49.66 | 68.81 | 82.19 | |
| 60.97 | 78.71 | 84.76 | 56.17 | 73.76 | 83.90 | |
| 60.79 | 78.46 | 83.73 | 56.04 | 73.54 | 83.02 | |
| 64.10 | 81.26 | 85.87 | 60.39 | 77.35 | 85.91 | |
| 62.94 | 80.55 | 85.73 | 58.87 | 76.39 | 85.35 | |
| ImageNet-Seg. | PASCAL VOC | ||||||
| Loss | mIoU (%) | Acc (%) | mAP (%) | mIoU (%) | Acc (%) | mAP (%) | |
| – | DINOv2 | 51.94 | 72.04 | 81.23 | 49.66 | 68.81 | 82.19 |
| 2 | w/ | 64.43 | 81.12 | 85.83 | 58.53 | 75.51 | 84.91 |
| 5 | w/ | 60.79 | 78.46 | 83.73 | 56.04 | 73.54 | 83.02 |
| 8 | w/ | 58.06 | 76.50 | 82.59 | 53.94 | 72.01 | 82.23 |
| 2 | w/ | 62.66 | 80.35 | 85.51 | 56.55 | 74.72 | 84.51 |
| COCO | PASCAL VOC | ||
| Loss | |||
| – | DINOv2 | 55.1 | 73.9 |
| 2 | w/ | 54.9 | 75.2 |
| 5 | w/ | 57.5 | 78.1 |
| 8 | w/ | 56.7 | 77.6 |
| 2 | w/ | 49.4 | 69.5 |
| Loss | Mean Per-Head Entropy | Entropy of Head-Averaged Attention |
| DINOv2 | 0.8841 | 0.9265 |
| w/ & | 0.7492 | 0.9166 |
| Size | Loss | ADE20k | Cityscapes | VOC |
| B | DINOv2 | 30.80 | 54.63 | 66.82 |
| w/ & | 31.14 | 55.30 | 68.66 | |
| L | DINOv2 | 43.15 | 64.25 | 81.13 |
| w/ & | 42.17 | 64.82 | 80.89 |
| Size | mIoU (%) | Acc (%) | mAP (%) |
| B | 65.63 | 82.50 | 85.69 |
| L | 53.71 | 73.95 | 81.61 |
| g | 55.56 | 75.47 | 82.41 |
| Zero-shot segmentation | |||||
| Loss | mIoU (%) | Acc (%) | mAP (%) | k-NN (%) | Linear (%) |
| DINOv2 | 63.67 | 80.69 | 85.39 | 81.7 | 82.6 |
| w/ & , last block PHS | 67.29 | 83.20 | 86.43 | 80.3 | 81.8 |
| w/ & , third-to-last block PHS | 67.85 | 84.09 | 86.35 | 81.0 | 82.2 |
Attention Head Specialization
Attention head specialization studies how transformer attention heads acquire distinct roles in routing and processing information.
See all 43 papers