Vision-language models such as CLIP exhibit strong semantic generalization, but remain limited in fine-grained visual perception. A recent work named KUEA presents a natural remedy by finetuning the image encoder under the supervision of the vision-centric DINOv2 to align their kernel matrices element-wisely, while regularizing the embeddings to remain close to the pretrained visual encoder for preserving image-text semantics in CLIP. However, we show that diminishing the role of the alignment loss to DINOv2 does not necessarily degrade its fine-grained visual performance, suggesting that the kernel-matrix discrepancy may be insufficient for further visual representation enhancement, motivating us to revisit the alignment formulation. In this work, we present a novel perspective to characterize representation alignment on feature subspaces through Kernel Canonical Correlation Analysis (KCCA), which maximizes the projection correlations. In optimization, we derive an efficient end-to-end training scheme upon KKT conditions, avoiding the eigenvalue problem in KCCA. Further, we extend our method into a 3-view formulation, i.e., 3vKCCA, in which the projections from the pretrained text encoder are also incorporated under a unified optimization framework for joint alignment. With CLIP ViT-L/14 on ImageNet-1K, our 3vKCCA improves the MMVP-VLM accuracy from 17.8 to 25.9, substantially outperforming the existing methods, and meanwhile maintains zero-shot image--text retrieval performance.
Figures & tables
Figure 1: Overall schematics of 3vKCCA. The CLIP vision encoder is jointly aligned with the given DINOv2 and the frozen CLIP text encoder via 3-view KCCA, together with regularization.
Table 4: Complete zero-shot image–text retrieval results on Flickr30K and MSCOCO. All methods use OpenAI CLIP ViT-L/14 and DINOv2.
Model
Zero-shot Datasets
Average Zero-shot
CIFAR10
CIFAR100
Caltech101
FER2013
OxfordPets
DTD
RESISC45
EuroSAT
ImageNet-O
CLIP [ 15 ]
95.20
75.49
83.83
50.00
93.21
55.21
64.90
62.65
32.25
68.08
Ours
93.28
72.94
84.17
54.50
89.21
53.94
60.95
57.48
46.05
68.06
Appendix
Table 5: Zero-shot classification accuracy of the pretrained CLIP model and our post-trained model.
Figure 2: Training dynamics of KUEA and 3vKCCA. The left and right columns show the alignment and regularization losses, respectively, for KUEA and 3vKCCA. In our implementation, the alignment and regularization losses are recorded as loss_align and loss_clean , respectively.
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.