Layer by layer, module by module: Choose both for optimal OOD probing of ViT

Ambroise Odonnat; Vasilii Feofanov; Laetitia Chapel; Romain Tavenard; Ievgen Redko

cs.CVMar 5, 2026

Layer by layer, module by module: Choose both for optimal OOD probing of ViT

Authors: Ambroise Odonnat, Vasilii Feofanov, Laetitia Chapel, Romain Tavenard, Ievgen Redko

Organizations: Noah’s Ark Lab, Inria · L’Institut Agro, IRISA · Univ. Rennes 2, IRISA · Noah’s Ark Lab

Abstract

Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.

Figures & tables

Figure 1: Layer by layer. Evolution of the linear probing performance across each layer of an

86

M ViT pretrained on ImageNet (the x-axis is the depth percentage of the layer). The solid line denotes the model only pretrained, and the dashed line denotes the model finetuned on the dataset at hand. From left to right , the shift between the pretraining and the downstream data increases. The stronger the shift, the worse the final layers.

Panel of Table 1 — Table 1: Module by module. For each module, we report the best linear probing accuracy over the layers. The best performance per dataset is in bold and the module with the highest win rate is in gray .

Figure 3: Layer by layer, module by module. Evolution of the linear probing performance of transformer modules across the layers of an

86

M ViT pretrained on ImageNet. From left to right , the shift between the pretraining and the downstream data increases.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: ViT-Base Implementation.

dataset	optimizer	batch size	training steps	learning rates $\eta$
Cifar10	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }
Cifar100	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }
Contrast	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }
Gaussian Noise	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }
Motion Blur	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }
Snow	SGD	512	$10000$	{ $1\mathrm{e}{-3}$ , $3\mathrm{e}{-3}$ , $1\mathrm{e}{-2}$ , $3\mathrm{e}{-2}$ }

Appendix

Table 2: Finetuning hyperparameters . We report the choice of optimizer, batch size, training steps, and learning rates.

Dataset	Cifar10	Cifar100	Contrast	Gaussian Noise	Motion Blur	Snow	Speckle Noise	Clipart	Sketch	Flowers102	Pets
full finetuning	$\text{99.02}_{\pm\text{0.02}}$	$\text{92.74}_{\pm\text{0.05}}$	$\text{97.23}_{\pm\text{0.18}}$	$\text{87.14}_{\pm\text{1.16}}$	$\text{94.67}_{\pm\text{0.14}}$	$\text{95.42}_{\pm\text{0.13}}$	$\text{89.58}_{\pm\text{0.43}}$	$\text{78.50}_{\pm\text{0.49}}$	$\text{71.30}_{\pm\text{0.26}}$	$\text{99.15}_{\pm\text{0.05}}$	$\text{94.57}_{\pm\text{0.29}}$

Appendix

Table 3: Full finetuning results. We report the best top-

1

accuracy (%) on the test set over the learning rate grid of each dataset (

\uparrow

). Each entry shows the mean and standard deviation over three finetuning runs with different seeds.

Figure 5: Layer by layer. Evolution of the linear probing performance across the layers of an

86

M ViT pretrained on ImageNet. The solid line denotes the model only pretrained, and the dashed line denotes the model finetuned on the dataset at hand. From left to right , the shift between the pretraining and the downstream data increases. The stronger the shift, the worse the final layers.

Figure 6: Layer by layer, module by module. Evolution of the linear probing performance of transformer modules across the layers of an

86

M ViT pretrained on ImageNet. From left to right , the shift between the pretraining and the downstream data increases.

Explore similar work

Attentive multilayer fusion for vision transformers
Jan 14, 2026Laure Ciernik, Marco Morik, Lukas Thede +4Linear Probing Vision Transformer
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany +7
Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders
Sep 14, 2026Qingtao Xia, Jiahua Bao, Siyao Cheng +1VLM Adaptation Parameter-Efficient Fine-Tuning
Faculty of Computing Harbin Institute of Technology Harbin, China
Vertical Fusion: Condensing Internal Representations for Robust ViT Classification
Jul 11, 2026Francesco Di Salvo, Shyam Nandan Rai, Hamed Damirchi +4Vision Transformer Image Classification
University of Bamberg, Bamberg, Bavaria, Germany · AIML, The University of Adelaide, Adelaide, South Australia, Australia

Jan 14, 2026J/K move · Enter open · S save

Attentive multilayer fusion for vision transformers

Laure Ciernik, Marco Morik, Lukas Thede, Luca Eyring+3

Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany+7

Linear Probing Vision Transformer Vision Foundation Model Adaptation

Open paper AskPDF arXiv

With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a lightweight head, is computationally efficient but often restricted to last-layer representations. We show that task-relevant information is distributed across the network hierarchy rather than encoded solely in the last layers. To leverage this distribution of information, we apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer. This attentive layer fusion (ALF) learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions. Across 20 diverse datasets and multiple pretrained foundation models, ALF achieves consistent, substantial gains over standard linear probes. Attention heatmaps further reveal that tasks different from the pre-training domain benefit most from intermediate representations. Overall, our findings underscore the value of intermediate layers and demonstrate a principled, task-aware approach for unlocking their potential for probing-based adaptation.

Abstract

Explore similar work

Attentive multilayer fusion for vision transformers

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

Vertical Fusion: Condensing Internal Representations for Robust ViT Classification