cs.CVApr 1, 2026

Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis

Authors: Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos

Organizations: University of Tennessee, Knoxville, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.

Figures & tables

Fig. 1: CCA cross-model alignment pipeline. (a) A backbone encoder MXM_{X} and a partner encoder MYM_{Y} produce baseline embeddings from the same input image. (b) CCA projects both embeddings into a shared, maximally correlated subspace. (c) The aligned space enables dimensionality reduction while transferring shared structure across models. (d) The resulting representations are evaluated on a downstream classification task.
ModelRepr. Dimensionality
ViT-T192
ViT-S384
ViT-B768
ViT-L1024
TABLE I: Vision transformer sizes evaluated in this work.
Fig. 2: ImageNet-1k classification accuracy of pre-trained vision transformers and classification heads using checkpoints obtained from [ 30 ] . The shaded area around each data point represents the number of parameters in the vision transformer.
Fig. 3: The ImageNet-1k classification accuracy of a linear probe trained using representations of varying amounts of the ImageNet-1k training data produced by a ViT-B and ViT-L.
ModelCCA PartnerBaselinePCACCAOrig. Dim.Proj. Dim.Dim. Δ\Delta
ViT-SViT-T63.3%61.9%65.9%384192-50%
ViT-BViT-T68.6%65.8%69.9%768192-75%
ViT-S68.4%73.1%768384-50%
ViT-LViT-T79.6%78.1%76.3%1024192-81.3%
ViT-S79.4%79.3%1024384-62.5%
ViT-B79.6%79.8%80.9%1024768-25%
TABLE II : Reducing dimensionality of larger ViT representations using smaller ViT representations
ModelCCA PartnerDimBaselineCCA
ViT-TViT-S19234.2%44.7%
ViT-B44.8%
ViT-L44.8%
ViT-SViT-B38463.3%68.4%
ViT-L68.4%
ViT-BViT-L76868.6%74.7%
TABLE III : Knowledge distillation from larger ViT to smaller ViT representations. Note that dimensionality is unchanged.
ModelDimFinetunedBaselineCCA
ViT-T192No29.0%41.6%
Yes34.2%45.0%
ViT-S384No58.8%66.2%
Yes63.3%68.4%
ViT-B768No62.0%71.9%
Yes68.6%74.7%
TABLE IV : Structure transfer from fine-tuned ViT to non-finetuned ViT representations. Note that dimensionality is unchanged.
ModelMethodDim.Dataset
IN-1kCIFAR-100MNISTCaltech-101Oxford-IIIT Pets
ViT-BBaseline51258.89%69.72%94.8%86.29%86.4%
PCA51259.46%70.91%96.4%91.59%87.91%
CCA51262.38%75.8%98.09%55.22%86.81%
ViT-LBaseline76865.9%76.59%95.58%89.33%90.4%
PCA51266.32%77.28%96.88%92.73%91.43%
TABLE V : Linear probing accuracy of representations of various image classification datasets when using vision transformer models trained as CLIP image encoders on LAION-400M.
Fig. 4 : Classification accuracy of a linear probe trained on representations of increasingly imbalanced subsets of the Caltech-101 dataset. These representations are produced by a ViT-B and ViT-L trained using the CLIP objective. The X-axis is the maximum allowed ratio between the number of samples in the least common class and the number of samples in the most common class.
Fig. 5 : Comparing CCA improvements over baseline as a function of the parameter ratio between backbone and partner models.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

ModelParam. Count (M)Training MethodSourcePre-train DatasetFine-tune DatasetRepr. Dim.
ViT-T5.7StandardtimmIN-21kIN-1k192
ViT-S22.1StandardtimmIN-21kIN-1k384
ViT-B86.6StandardtimmIN-21kIN-1k768
ViT-B86.6CLIPOpenCLIPLaion-400MN/A512
ViT-L304.3StandardtimmIN-21kIN-1k1024
ViT-L304.3CLIPOpenCLIPLaion-400MN/A768
Appendix
TABLE VI : Further details about the pre-trained vision transformers used in this work.
More in this topic

Canonical Correlation Analysis

See all 10 papers

Explore similar work

CardsList

  1. Supervised Dimensionality Reduction Revisited: Why LDA on Frozen CNN Features Deserves a Second Look

    Apr 5, 2026Indar Kumar, Girish Karhana, Sai Krishna Jasti +1Dimensionality ReductionTransfer Learning

    independent researchers

  2. Variational Feature Compression for Model-Specific Representations

    Apr 8, 2026Zinan Guo, Zihan Wang, Chuan Yan +3Privacy-Preserving Representation LearningPrivacy-Preserving ML

    The University of Queensland, Brisbane, Australia · University of Wollongong, Wollongong, Australia · City University of Hong Kong, Hong Kong SAR, China

  3. Revisiting Visual Representation Enhancement of VLMs via Kernel Canonical Correlation Analysis

    Oct 2, 2026Peilin Yang, Xiaoyu Liu, Jian Sun +1VLM AdaptationVisual Representation Learning

    Beijing Institute of Technology