Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.
Figures & tables
Figure 1: Layer by layer. Evolution of the linear probing performance across each layer of an 86 M ViT pretrained on ImageNet (the x-axis is the depth percentage of the layer). The solid line denotes the model only pretrained, and the dashed line denotes the model finetuned on the dataset at hand. From left to right , the shift between the pretraining and the downstream data increases. The stronger the shift, the worse the final layers.
Table 1: Module by module. For each module, we report the best linear probing accuracy over the layers. The best performance per dataset is in bold and the module with the highest win rate is in gray .
Figure 3: Layer by layer, module by module. Evolution of the linear probing performance of transformer modules across the layers of an 86 M ViT pretrained on ImageNet. From left to right , the shift between the pretraining and the downstream data increases.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: ViT-Base Implementation.
dataset
optimizer
batch size
training steps
learning rates η
Cifar10
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Cifar100
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Contrast
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Gaussian Noise
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Motion Blur
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Snow
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Appendix
Table 2: Finetuning hyperparameters . We report the choice of optimizer, batch size, training steps, and learning rates.
Dataset
Cifar10
Cifar100
Contrast
Gaussian Noise
Motion Blur
Snow
Speckle Noise
Clipart
Sketch
Flowers102
Pets
full finetuning
99.02±0.02
92.74±0.05
97.23±0.18
87.14±1.16
94.67±0.14
95.42±0.13
89.58±0.43
78.50±0.49
71.30±0.26
99.15±0.05
94.57±0.29
Appendix
Table 3: Full finetuning results. We report the best top- 1 accuracy (%) on the test set over the learning rate grid of each dataset ( ↑ ). Each entry shows the mean and standard deviation over three finetuning runs with different seeds.
Figure 5: Layer by layer. Evolution of the linear probing performance across the layers of an 86 M ViT pretrained on ImageNet. The solid line denotes the model only pretrained, and the dashed line denotes the model finetuned on the dataset at hand. From left to right , the shift between the pretraining and the downstream data increases. The stronger the shift, the worse the final layers.
Figure 6: Layer by layer, module by module. Evolution of the linear probing performance of transformer modules across the layers of an 86 M ViT pretrained on ImageNet. From left to right , the shift between the pretraining and the downstream data increases.
With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a lightweight head, is computationally efficient but often restricted to last-layer representations. We show that task-relevant information is distributed across the network hierarchy rather than encoded solely in the last layers. To leverage this distribution of information, we apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer. This attentive layer fusion (ALF) learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions. Across 20 diverse datasets and multiple pretrained foundation models, ALF achieves consistent, substantial gains over standard linear probes. Attention heatmaps further reveal that tasks different from the pre-training domain benefit most from intermediate representations. Overall, our findings underscore the value of intermediate layers and demonstrate a principled, task-aware approach for unlocking their potential for probing-based adaptation.
Laure Ciernik, Marco Morik, Lukas Thede +4
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany +7
We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.
Qingtao Xia, Jiahua Bao, Siyao Cheng +1
Faculty of Computing Harbin Institute of Technology Harbin, China
Despite exposing rich intermediate representations, Vision Transformers (ViTs) are almost exclusively utilized as black-box feature extractors, where only the last layer is considered for downstream tasks. We challenge this convention by introducing the notion of recoverability: the capacity of intermediate representations to correct last-layer failures. By evaluating independent classification probes at every model depth across 16 datasets, we observe that intermediate probes correctly classify 18% to 76% of samples that the last-layer probe misclassifies. We show that these gains are not primarily driven by predictive diversity, but by a redundancy-correctness correspondence, where the internal hierarchy acts as a series of stable, redundant probes of a shared discriminative signal. While established horizontal ensemble strategies (i.e., across multiple models) can improve performance, they incur high computational cost and ignore this vertical signal within a single model. To bridge this gap, we propose VFusion, a principled vertical aggregation strategy employing a learnable mapping into a low-dimensional latent space that synthesizes features across the internal ViT hierarchy. VFusion substantially outperforms established aggregation baselines in both in-distribution and out-of-distribution settings, notably closing 45% of the accuracy gap between the best individual layer and a theoretical oracle performance. Our gains consistently generalize across model sizes and pre-training regimes, confirming that VFusion offers a robust and efficient alternative to horizontal ensemble methods. The code is available at https://github.com/francescodisalvo05/vit-vertical-fusion.
Francesco Di Salvo, Shyam Nandan Rai, Hamed Damirchi +4
University of Bamberg, Bamberg, Bavaria, Germany · AIML, The University of Adelaide, Adelaide, South Australia, Australia