DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles
Authors: Yiming Ma, Hongkun Yang, Lionel Z. Wang, Bin Chen, Weizhi Xian, Jianzhi Teng
Organizations: Chongqing Research Institute of Harbin Institute of Technology, China · Ocean University of China, China · The Hong Kong Polytechnic University, Hong Kong SAR · Harbin Institute of Technology (Shenzhen), China
Prompt learning is a dominant paradigm for adapting pre-trained Vision-Language Models (VLMs) to downstream tasks. However, existing methods often rely on a simplistic, layer-centric view, assuming shallow layers capture general features while deep layers handle task-specific knowledge. This assumption results in uncontrolled interactions between learnable tokens and original tokens. Task-specific knowledge could degrades the model's core generalization and creates a trade-off between task adaptation and the preservation of zero-shot generalization. To address this, we challenge the layer-centric view and propose DeAR, a framework that achieves fine-grained VLM adaptation by Decomposing Attention head Roles. We posit that the functional specialization within VLMs occurs not between layers, but at the finer-grained level of individual attention heads in the deeper layers. Based on this insight, we introduce a novel metric, Concept Entropy, to systematically classify attention heads into distinct functional roles: Attribute, Generalization, and Mixed. Guided by these roles, we introduce specialized attribute tokens and a Role-Based Attention Mask mechanism to precisely control information flow, ensuring generalization heads remain isolated from task-specific knowledge. We further incorporate a Task-Adaptive Fusion Strategy for inference. Extensive experiments on fifteen datasets show that DeAR achieves a strong balance between task adaptation and generalization, outperforming previous methods across various tasks.
Figures & tables
Figure 1: For a Generalization Head (e.g., Layer 9 Head 0), our Role-Based Mask explicitly blocking the original CLS and Patch tokens from interacting to the inserted Attribute Tokens.
Figure 2: The overall framework of DeAR. At a deep layer J , we insert learnable attribute tokens into the frozen vision encoder and the text encoder. Information flow in these subsequent layers is precisely controlled by our Role-Based Attention Mask mechanism. For inference, the class feature (from the [CLS] token) and attribute feature first compute the logits with the text feature. These logits are then combined via learnable fusion weights to produce the final prediction.
Object Head
Shape Head
Texture Head
Color Head
Location Head
(L12, H7)
(L12, H5)
(L11, H5)
(L12, H10)
(L12, H0)
- A stick
- A regular octagon
- Delicate embroidery
- An image with cold green tones
- Picture taken in Scotland
- A cup
- Rectangular object
- Intricate architectural carving
- Image with a red color
- Photo taken in Barcelona
- A bonnet
- A wavy pattern
- Artwork with kaleidoscopic patterns
- A charcoal gray color
- Photo taken in Tokyo, Japan
- A trampoline
- Herringbone pattern
- Image with woven basket textures
- Image with a yellow color
- Image taken in South Africa
- A shoelace
- A scalene triangle
- Artwork with Mondrian-like grids
- Image with a blue color
- Picture taken in Texas, USA
Table 1: Representative examples of functionally specialized Attribute Heads identified in the later layers of ViT-B-16. Each head demonstrates a strong and unambiguous focus on a single core visual attribute, as evidenced by its top descriptive phrases.
Figure 3: Results of attribute-conditioned image retrieval on ImageNet. For each query image, we retrieval using features from different learned attribute tokens.
Method
Average
ImageNet
Caltech101
OxfordPets
StanfordCars
Flowers102
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP (ICML2021)
69.34
74.22
71.70
72.43
68.14
70.22
96.84
94.00
95.40
91.17
97.96
94.12
63.37
74.89
68.65
72.08
77.80
74.83
CoOp (ICCV2021)
82.69
63.22
71.66
76.47
67.88
71.92
98.01
89.81
93.73
93.67
95.29
94.47
78.12
60.40
68.13
97.60
59.67
74.06
ATPrompt CoOp (ICCV2025)
82.68
68.04
74.65
76.27
70.60
73.33
97.95
93.63
95.74
94.77
96.59
95.67
77.43
66.55
71.58
97.44
67.52
79.77
ProGrad (CVPR2023)
82.48
70.75
76.16
77.02
66.66
71.46
98.02
93.89
95.91
95.07
97.63
96.33
77.68
68.63
72.88
95.54
71.87
82.03
MaPLe (CVPR2023)
82.28
75.14
78.55
76.60
70.54
73.47
97.74
94.36
96.02
95.43
97.76
96.58
72.94
74.00
73.47
95.92
72.46
82.56
Table 2: Comparison of DeAR with previous methods on base-to-novel generalization across 11 datasets. Bold values indicate the best results.
Source
Target
ImageNet
Average
Caltech101
OxfordPets
StanfordCars
Flowers102
Food101
FGVC- Aircraft
SUN397
DTD
EuroSAT
UCF101
CoOp
71.51
63.88
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
MaPLe
70.72
66.30
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69
PromptSRC
71.27
65.81
93.60
90.25
65.70
70.25
86.15
23.90
67.10
46.87
45.50
68.75
MMRL
72.03
67.25
94.67
91.43
66.10
72.77
86.40
26.30
67.57
45.90
53.10
68.27
TAC
72.77
66.53
94.53
90.67
65.30
72.20
85.83
23.53
67.63
47.57
48.07
70.00
Table 3: Comparison of DeAR with previous methods on cross-dataset evaluation. Bold values indicate the best results.
Source
Target
ImageNet
Average
-V2
-S
-A
-R
CoOp
71.51
59.28
64.20
47.99
49.71
75.21
MaPLe
70.72
60.28
64.07
49.15
50.90
76.98
PromptSRC
71.27
60.65
64.35
49.55
50.90
77.80
MMRL
72.03
60.59
64.47
49.17
51.20
77.53
TAC
72.77
61.63
65.97
50.30
51.73
78.50
Table 4: Comparison of DeAR with previous methods on domain generalization. Bold values indicate the best results.
Figure 4: Comparison of DeAR with previous methods on few-shot learning. Detailed results are provided in the Appendix
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China. · School of Intelligence Science and Engineering, Southeast University, China.