Which representation should be probed in a transformer-based foundation model? Recent work shows that the standard practice of probing the final layer is often suboptimal, as intermediate layers can provide better features. Yet, representations are still largely studied along depth alone, treating each transformer block as an atomic unit. In this work, we analyze transformer representations layer by layer, module by module. Through more than 144,000 linear probing evaluations of vision foundation models (VFMs) and large language models (LLMs), we show that layer and module choices should be considered jointly: the best layer-module pair yields substantially larger gains over the final-layer residual stream than the best layer with the residual stream fixed. Moreover, module performance can vary substantially across layers, showing that useful representations cannot be characterized by depth alone. Building on these findings, we evaluate simple fixed probing strategies: the final-layer feedforward activation (Act@100) for vision and the second normalization layer at 60% depth (LN2@60) for language. These strategies improve over the probing baseline in a majority of evaluations and recover, on average, 42% and 86% of the gains from probing the best layer-module pair for vision and language, respectively. For VFMs, these gains increase with the transfer difficulty between pretraining and downstream data.
Figures & tables
Figure 1: Layer by layer, module by module. Heatmaps showing the optimality frequency of each layer–module pair, i.e., how often it achieves the highest probing accuracy, with marginal distributions of optimal layers (top) and modules (right). For both vision foundation models ( left ) and large language models ( right) , performance depends jointly on layer and module choices. The probing baseline on the final-layer residual stream ( ∘ ) is largely suboptimal, while the fixed strategies Act@100 and LN2@60 , for vision and language ( {\color[rgb]{1,0,0}\star} ), respectively, often achieve the best performance.
Figure 2: Transformer representations. We probe transformer representations (colored) across layers in VFMs and LLMs, covering models with standard ( left ) and gated FFN ( right ).
Figure 3: Substantial gains from optimal probing locations in VFMs. Complementary cumulative distribution of gains over the probing baseline across 270 model-dataset pairs (each defining a run) when using the best representation across layers ( Glayer ), modules ( Gmodule ), or both ( Gjoint ).
Figure 4: Impact of Act and intermediate layers in VFMs. Probing accuracy across layers for Act and RC2 on Cifar10, Sketch, and Quickdraw. Results are reported for the 307 M-parameter variant, common to each model family. The star indicates the best Act performance over depth.
Strategy
Flowers102
EuroSAT
Cifar10
Pet
Contrast
Food-101
Cifar100
Motion Blur
DTD
Speckle Noise
Clipart
Painting
Aircraft
Sketch
iWildCam
Quickdraw
Infograph
RxRx1
Average
ViT
\arrayrulecolor gray!50 Baseline
89.9
96.5
94.9
85.4
90.2
73.2
82.8
79.5
72.6
74.5
59.9
61.0
49.6
45.1
62.8
43.5
23.9
2.1
66.0
Act@100
+2.0
+0.4
+0.8
+4.5
-0.5
+7.3
+2.1
+1.2
+1.1
+0.2
+0.7
+4.0
+4.6
+3.4
+3.5
+0.8
+4.6
+0.1
+2.3
\RowStyle [color=oraclegray] Oracle
+5.4
+1.9
+1.2
+4.8
+1.4
+8.8
+2.9
+2.7
+2.6
+3.5
+5.0
+6.7
+6.2
+7.3
+4.4
+4.1
+7.2
+2.7
+4.4
\arrayrulecolor black DINOv2
\arrayrulecolor gray!50 Baseline
99.6
97.4
99.0
91.4
96.1
91.7
91.4
87.4
81.6
85.3
82.3
77.9
70.3
73.9
70.1
59.6
44.3
4.8
78.0
Table 1: Performance comparison in VFMs between the probing baseline on the final-layer residual stream ( RC2@100 ), the fixed strategy on the final-layer activation ( Act@100 ), and the oracle strategy that probes the best representation over layers and modules for each model-dataset pair. Results are reported for the 307 M-parameter variant, common to each model family, with datasets ordered from left to right by increasing mean probing error (easy to hard). The Baseline row reports the probing test accuracy ( % ), while Act@100 and Oracle report accuracy differences in percentage points relative to the baseline, shown in green when positive and red otherwise.
Figure 5: Act@100 recovers nearly half of the oracle gain. Recovery of oracle gain Gjoint , averaged across all model-dataset pairs with 95% confidence intervals. The fixed strategy Act@100 outperforms the best layer-wise RC2 , while probing the best layer-wise Act recovers 78.1% of the gain.
Figure 6: Oracle gains increase with transfer difficulty and corruption severity, but decrease with pretraining scale. (a) Oracle gain as a function of empirical transfer difficulty, defined as the mean probing error over evaluated models. Markers show the gain for each dataset, averaged across 15 models, with a 95% confidence interval. (b) Oracle gain as a function of corruption severity on 5 types of corruption from Cifar10-C, averaged across 15 models, with a 95% confidence interval. (c) The optimal depth associated with Gjoint decreases with corruption severity. (d) Oracle gain across model families, averaged across datasets and model sizes, with a 95% confidence interval. Models are ordered by increasing number of tokens seen during pretraining.
Figure 7: Substantial gains from optimal probing locations in LLMs. Gains over the probing baseline on MTEB(Multilingual) from probing the best representation across layers ( Glayer ), modules ( Gmodule ), or both ( Gjoint ) in Qwen3, Llama3, and Gemma3 models.
Figure 8: Impact of LN2 and intermediate layers on performance. Probing accuracy across layers for LN2 and RC2 on MTEB(Multilingual) . The star indicates the best LN2 performance over depth.
Strategy
Qwen3
Llama3
Gemma3
1.7B
4B
8B
1B
3B
8B
1B
4B
12B
Baseline
55.5
56.6
57.6
56.6
58.9
59.9
59.1
60.9
61.5
RC2@60
56.1
+0.6
56.7
+0.1
58.3
+0.7
55.5
-1.1
59.7
+0.8
61.3
+1.3
59.3
+0.2
61.7
+0.8
62.6
+1.0
LN2@60
57.6
+2.1
59.4
+2.9
61.1
+3.5
58.6
+2.0
61.1
+2.2
62.3
+2.4
61.1
+2.0
63.1
+2.3
64.2
+2.6
\RowStyle [color=oraclegray] Oracle
58.1
+2.5
60.1
+3.6
61.6
+4.1
58.6
+2.0
61.3
+2.4
62.8
+2.9
61.6
+2.6
63.8
+3.0
64.3
+2.8
Table 2: Performance comparison in LLMs between the probing baseline on the final-layer residual stream ( RC2 ), residual-stream probing at 60% depth ( RC2@60 ), the fixed strategy LN2@60 , and the oracle strategy that probes the best layer-module pair for each model. The accuracy difference in percentage points relative to the probing baseline is shown in green if positive, and in red otherwise.
Figure 9: LN2@60 recovers substantial gains. Recovery of oracle gain Gjoint , averaged across all models with 95% confidence intervals. The fixed LN2@60 strategy outperforms the best layer-wise RC2 strategy, while probing the best layer-wise LN2 recovers 99.8% of the gain.
Table 3: Characteristics of ViT variants ( dosovitskiy2021vit ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the patch size, the number of parameters, and the link to the HuggingFace checkpoint.
Model
Backbone
Layers
Emb. dim.
FFN dim.
Attn. heads
Patch size
Parameters
Checkpoint
DINOv2-S
ViT-S/14
12
384
1536
6
14
22M
dinov2-small
DINOv2-B
ViT-B/14
12
768
3072
12
14
86M
dinov2-base
DINOv2-L
ViT-L/14
24
1024
4096
16
14
307M
dinov2-large
DINOv2-g
ViT-g/16
40
1536
4096
24
14
1.1B
dinov2-giant
Appendix
Table 4: Characteristics of DINOv2 variants ( oquab2024dinov2 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the patch size, the number of parameters, and the link to the HuggingFace checkpoint.
Model
Backbone
Layers
Emb. dim.
FFN dim.
Attn. heads
Patch size
Parameters
Checkpoint
DINOv3-S
ViT-S/16
12
384
1536
6
16
22M
dinov3-small
DINOv3-B
ViT-B/16
12
768
3072
12
16
86M
dinov3-base
DINOv3-L
ViT-L/16
24
1024
4096
16
16
307M
dinov3-large
DINOv3-7B
ViT-7B/16
40
4096
8192
32
16
7B
dinov3-7B
Appendix
Table 5: Characteristics of DINOv3 variants ( simeoni2025dinov3 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the patch size, the number of parameters, and the link to the HuggingFace checkpoint.
Model
Layers
Emb. dim.
FFN dim.
Attn. heads
Patch size
Parameters
Checkpoint
AIMv2-L
24
1024
1536
6
14
307M
aimv2-large
AIMv2-H
24
1536
3072
12
14
632M
aimv2-huge
AIMv2-1B
24
2048
4096
16
14
1.2B
aimv2-1B
AIMv2-3B
24
3072
8192
32
14
2.7B
aimv2-3B
Appendix
Table 6: Characteristics of AIMv2 variants ( fini2025aimv2 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the patch size, the number of parameters, and the link to the HuggingFace checkpoint.
Dataset
Reference
Train
Test
Classes
Aircraft
maji2013aircraft
6 667
3 333
100
Cifar10
krizhevsky2009cifar
50 000
10 000
10
Cifar100
krizhevsky2009cifar
50 000
10 000
100
Clipart
peng2019domainnet
39 066
9 767
345
Contrast
hendrycks2018imagenetc
8 000
2 000
10
DTD
cimpoi2014dtd
1 880
1 880
47
Appendix
Table 7: Characteristics of our 18 image classification benchmarks. We provide the number of training and test images, as well as the number of classes. Contrast, Motion Blur, and Speckle Noise are corrupted variants of the Cifar10 test set with a corruption severity of 5 , from the Cifar10-C benchmark ( hendrycks2018imagenetc ) , Clipart, Infograph, Painting, Quickdraw, and Sketch are domains from the DomainNet benchmark ( peng2019domainnet ) . Cifar10 → Cifar10-C indicates that the linear probe is trained on the training set of Cifar10 and evaluated on Cifar10-C, i.e., on a corrupted version of the Cifar10 test set.
Figure 12: BaseLanguageModel implementation.
Model
Layers
Emb. dim.
FFN dim.
Attn. heads
Context length
Parameters
Checkpoint
Qwen3 1.7B
28
2048
6144
16
32K
1.72B
Qwen3-1.7B
Qwen3 4B
36
2560
9728
32
128K
4.02B
Qwen3-4B
Qwen3 8B
36
4096
12288
32
128K
8.19B
Qwen3-8B
Appendix
Table 8: Characteristics of Qwen3 variants ( yang2025qwen3 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the context length, the number of parameters, and the link to the HuggingFace checkpoint.
Model
Layers
Emb. dim.
FFN dim.
Attn. heads
Context length
Parameters
Checkpoint
Llama3 1B
16
2048
8192
32
128K
1.24B
Llama3.2 1B
Llama3 3B
28
3072
9728
24
128K
3.21B
Llama3.2 3B
Llama3 8B
32
4096
14336
32
128K
8.03B
Llama3.1 8B
Appendix
Table 9: Characteristics of Llama3 variants ( grattafiori2024llama3 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the context length, the number of parameters, and the link to the HuggingFace checkpoint.
Model
Layers
Emb. dim.
FFN dim.
Attn. heads
Context length
Parameters
Checkpoint
Gemma3 1B
26
1152
6912
4
128K
1.00B
gemma-3-1b-pt
Gemma3 4B
34
2560
10240
8
128K
4.30B
gemma-3-4b-pt
Gemma3 12B
48
3840
15360
16
128K
12.19B
gemma-3-12b-pt
Appendix
Table 10: Characteristics of Gemma3 variants ( kammath2025gemma3 ) . We provide the number of layers, the embedding dimension, the FFN hidden dimension, the number of attention heads, the number of parameters, and the link to the HuggingFace checkpoint.
Task
Reference
Languages
Domains
Sample creation
Samples
BulgarianStoreReviewSentimentClassification
trifonova2018bulgarian
bul
Reviews, Written
found, human-annotated
182
DalajClassification
volodina2021dalaj
swe
Non-fiction, Written
created, expert-annotated
888
DBpediaClassification
zhang2015dbpedia
eng
Encyclopaedic, Written
found, derived
2 048
EstonianValenceClassification
pajupuu2023estonian
est
News, Written
found, human-annotated
818
FinancialPhrasebankClassification
malo2014financial
eng
News, Written, Financial
found, expert-annotated
2 264
GujaratiNewsClassification
-
guj
News, Written
found, derived
1 318
Appendix
Table 11: Characteristics of MTEB(Multilingual) classification tasks. Experiments are conducted on the 24 tasks below, spanning a diverse set of languages and domains. The column names follow enevoldsen2025mmteb , and the list of ISO 639 codes can be found at https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes .
Dataset
Optimizer
Batch size
Training steps
Learning rates η
Cifar10
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Contrast
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Speckle Noise
SGD
512
10000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Flowers102
SGD
512
5000
{ 1e−3 , 3e−3 , 1e−2 , 3e−2 }
Appendix
Table 12: Finetuning setup . We report the optimizer, batch size, training steps, and learning rates.
Figure 13: Intermediate layers under distribution shift. Evolution of the linear probing performance across layers of a ViT-B ( 86 M parameters) pretrained on ImageNet-21K. The solid line is the accuracy on the pretrained model, and the dashed line is the accuracy on the same model after finetuning on the dataset at hand. Datasets are ordered by decreasing finetuning performance.
Model
Reference
Batch size
Iterations
Pretraining images
ViT
dosovitskiy2021vit
4 096
307 617
1 259 999 232
DINOv2
oquab2024dinov2
3 072
625 000
1 920 000 000
DINOv3
simeoni2025dinov3
4 096
1 000 000
4 096 000 000
AIMv2
fini2025aimv2
8 192
1 500 000
12 288 000 000
Appendix
Table 13: Number of images seen during pretraining. We report the batch size, number of training iterations, and total number of images seen during pretraining for each model, along with the official reference for the figures.
Model
Size
Patch size
Image resolution
Global crops (nb. - resolution)
Local crops
Nb. of tokens per image
ViT
B
16
224×224
-
-
197
L
16
224×224
-
-
197
H
14
224×224
-
-
257
DINOv2
S
14
224×224
2 - 224×224
8- 98×98
914
B
14
224×224
2 - 224×224
8- 98×98
914
L
14
224×224
2 - 224×224
8- 98×98
914
Appendix
Table 14: Number of visual tokens per image. We report for each model the patch size, image resolution, global and local crops resolution (when supported) and the corresponding amount of visual tokens per image.
Figure 14: Gain over model families. The three types of gains ( Glayer , Gmodule , and Gjoint ) are displayed on the representative datasets spanning low, medium, and high transfer difficulty (Cifar10, Sketch, Quickdraw). Results are reported on the Large variant ( 307 M parameters) of each family. The magnitude of gains increases with transfer difficulty, consistent with Figure 6(a) .
7 Module
LN1
MHA
RC1
LN2
FC1
Act
FC2
RC2
winning frequency ( % )
6.3
7.4
3.7
9.3
0.4
64.8
2.2
5.9
Appendix
Table 15: Module comparison. Percentage of evaluations, out of a total of 270 , for which a given module belongs to the layer-module pair achieving the oracle gain Gjoint .
Figure 15: Impact of Act and intermediate layers in ViT (first part). Probing accuracy across layers of ViT models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 16: Impact of Act and intermediate layers in ViT (second part). Probing accuracy across layers of ViT models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 17: Impact of Act and intermediate layers in ViT (last part). Probing accuracy across layers of ViT models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 18: Impact of Act and intermediate layers in DINOv2 (first part). Probing accuracy across layers of DINOv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 19: Impact of Act and intermediate layers in DINOv2 (second part). Probing accuracy across layers of DINOv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 20: Impact of Act and intermediate layers in DINOv2 (last part). Probing accuracy across layers of DINOv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 21: Impact of Act and intermediate layers in DINOv3 (first part). Probing accuracy across layers of DINOv3 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 22: Impact of Act and intermediate layers in DINOv3 (second part). Probing accuracy across layers of DINOv3 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 23: Impact of Act and intermediate layers in DINOv3 (last part). Probing accuracy across layers of DINOv3 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 24: Impact of Act and intermediate layers in AIMv2 (first part). Probing accuracy across layers of AIMv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 25: Impact of Act and intermediate layers in AIMv2 (second part). Probing accuracy across layers of AIMv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Figure 26: Impact of Act and intermediate layers in AIMv2 (last part). Probing accuracy across layers of AIMv2 models for Act and RC2 . The star indicates the best Act performance over depth. Datasets are ordered from top to bottom by increasing transfer difficulty.
Family
χ2
p-value
Kendall’s W
ViT
0.778
0.678
0.022
DINOv2
4.773
0.189
0.088
DINOv3
0.627
0.890
0.012
AIMv2
4.600
0.204
0.085
Appendix
Table 16: Friedman test on oracle gain Gjoint . Model sizes play the role of treatments, and the 18 datasets are blocks. No family shows a statistically significant effect at α=0.05 , and Kendall’s W indicates small effects throughout.
Model
Strategy
Flowers102
EuroSAT
Cifar10
Pet
Contrast
Food-101
Cifar100
Motion Blur
DTD
Speckle Noise
Clipart
Painting
Aircraft
Sketch
iWildCam
Quickdraw
Infograph
RxRx1
Average
ViT
\arrayrulecolor gray!50 B
Baseline
98.3
95.0
91.4
90.2
82.2
73.0
75.0
68.9
69.4
60.1
51.2
58.8
44.6
36.2
56.6
36.4
23.1
1.8
61.8
Act@100
95.2
88.1
87.7
83.8
76.4
66.7
69.0
61.2
57.1
49.6
39.8
53.7
35.2
28.8
55.4
26.5
18.1
1.4
55.2
\RowStyle [color=oraclegray] Oracle
98.6
98.6
92.2
90.6
84.2
73.5
77.3
76.2
72.4
70.6
56.2
59.6
47.8
42.4
61.7
42.6
25.3
4.6
65.2
\arrayrulecolor gray!50 L
Baseline
89.9
96.5
94.9
85.4
90.2
73.2
82.8
79.5
72.6
74.5
59.9
61.0
49.6
45.1
62.8
43.5
23.9
2.1
66.0
Act@100
91.9
96.9
95.7
89.9
89.8
80.5
84.9
80.8
73.7
74.8
60.6
65.0
54.2
48.4
66.3
44.2
28.5
2.2
68.2
Appendix
Table 17: Full performance comparison between the probing baseline on the final-layer residual stream ( RC2@100 ), the fixed strategy on the final-layer activation ( Act@100 ), and the oracle strategy that probes the best representation over layers and modules for each model-dataset pair. Probing accuracies are reported for all models, with datasets ordered from left to right by increasing mean probing error (easy to hard). We report for each strategy the probing test accuracy ( % ).
Strategy
Flowers102
EuroSAT
Cifar10
Pet
Contrast
Food-101
Cifar100
Motion Blur
DTD
Speckle Noise
Clipart
Painting
Aircraft
Sketch
iWildCam
Quickdraw
Infograph
RxRx1
Average
ViT
\arrayrulecolor gray!50 Baseline
89.9
96.5
94.9
85.4
90.2
73.2
82.8
79.5
72.6
74.5
59.9
61.0
49.6
45.1
62.8
43.5
23.9
2.1
66.0
FC1@100
91.1
96.0
94.0
86.5
88.7
72.8
80.9
77.8
71.7
69.9
54.8
58.0
50.0
40.3
61.2
40.2
21.1
1.9
64.3
Act@100
91.9
96.9
95.7
89.9
89.8
80.5
84.9
80.8
73.7
74.8
60.6
65.0
54.2
48.4
66.3
44.2
28.5
2.2
68.2
\arrayrulecolor black DINOv2
\arrayrulecolor gray!50 Baseline
99.6
97.4
99.0
91.4
96.1
91.7
91.4
87.4
81.6
85.3
82.3
77.9
70.3
73.9
70.1
59.6
44.3
4.8
78.0
Appendix
Table 18: Additional performance comparison between the probing baseline on the final-layer residual stream ( RC2@100 ), the final-layer FFN first linear layer ( FC1@100 ), and the fixed strategy on the final-layer activation ( Act@100 ). Probing accuracies are reported on the 307 M-parameter variant common to each model family, providing a controlled comparison across families. Datasets are ordered by increasing mean probing error, from easy to hard. We report for each strategy the probing test accuracy ( % ).
Model
Qwen3
Llama3
Gemma3
1.7B
4B
8B
1B
3B
8B
1B
4B
12B
best module
LN2
LN2
LN2
LN2
LN2
LN2
LN2
FC1
LN2
Appendix
Table 19: Module comparison. Module in the layer-module pair that achieves the oracle gain Gjoint .
Figure 27: Impact of LN2 and intermediate layers on performance across model sizes. Probing accuracy across layers for LN2 and RC2 on MTEB(Multilingual) . The star indicates the best performance obtained on LN2 .
Task
Baseline
RC2@60
LN2@60
BulgarianStoreReviewSentimentClassification
50.2
56.3
+6.1
59.0
+8.9
DalajClassification
50.2
50.5
+0.3
50.5
+0.3
DBpediaClassification
91.3
89.2
-2.2
91.8
+0.5
EstonianValenceClassification
37.2
39.6
+2.4
40.5
+3.2
FinancialPhrasebankClassification
64.3
66.6
+2.2
69.1
+4.8
GujaratiNewsClassification
77.2
77.4
+0.2
80.5
+3.3
Appendix
Table 20: Full performance comparison per task between the probing baseline on the final-layer residual stream ( RC2 ), residual-stream probing at 60% depth ( RC2@60 ), and the fixed strategy LN2@60 . The accuracy is averaged over models for each task, and the difference in percentage points relative to the probing baseline is shown in green if improved, and in red otherwise.
Task
Qwen3 1.7B
Qwen3 4B
Qwen3 8B
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
BulgarianStoreReviewSentimentClassification
45.0
53.6
57.0
46.4
53.5
57.3
46.8
56.4
61.8
DalajClassification
49.7
50.5
50.2
49.7
50.4
50.4
50.0
50.3
50.7
DBpediaClassification
91.2
85.8
88.4
91.3
82.4
88.8
91.1
86.1
90.2
EstonianValenceClassification
33.7
36.4
37.3
36.4
39.7
40.3
36.9
40.3
41.3
FinancialPhrasebankClassification
64.3
68.6
70.0
64.2
65.5
70.9
64.3
67.3
72.3
Appendix
Table 21: Full performance comparison on Qwen3 between the probing baseline on the final-layer residual stream ( RC2 ), residual-stream probing at 60% depth ( RC2@60 ), and the fixed strategy LN2@60 . We report for each strategy the probing test accuracy ( % ).
Task
Llama3 1B
Llama3 3B
Llama3 8B
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
BulgarianStoreReviewSentimentClassification
46.3
51.9
54.6
51.4
57.5
58.3
53.1
58.9
60.0
DalajClassification
50.1
50.1
50.3
50.4
50.5
50.5
50.5
50.4
50.5
DBpediaClassification
90.3
91.8
92.9
91.1
93.1
93.6
91.3
93.3
93.8
EstonianValenceClassification
34.3
35.6
36.3
36.5
38.8
39.4
38.8
41.0
41.2
FinancialPhrasebankClassification
63.5
67.7
69.9
62.5
66.7
68.9
60.8
67.3
69.6
Appendix
Table 22: Full performance comparison on Llama3 between the probing baseline on the final-layer residual stream ( RC2 ), residual-stream probing at 60% depth ( RC2@60 ), and the fixed strategy LN2@60 . We report for each strategy the probing test accuracy ( % ).
Task
Gemma3 1B
Gemma3 4B
Gemma3 12B
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
Baseline
RC2@60
LN2@60
BulgarianStoreReviewSentimentClassification
51.1
58.0
59.0
54.9
58.4
61.8
56.4
58.5
61.5
DalajClassification
50.2
50.1
50.2
50.2
50.8
50.8
50.9
51.4
50.9
DBpediaClassification
91.7
87.5
92.6
92.0
90.7
92.9
91.9
91.7
93.3
EstonianValenceClassification
36.4
40.3
40.7
41.0
43.0
44.0
41.1
41.5
43.5
FinancialPhrasebankClassification
65.8
67.6
68.8
67.2
65.4
67.4
66.5
63.1
64.2
Appendix
Table 23: Full performance comparison on Gemma3 between the probing baseline on the final-layer residual stream ( RC2 ), residual-stream probing at 60% depth ( RC2@60 ), and the fixed strategy LN2@60 . We report for each strategy the probing test accuracy ( % ).
Task
Gemini Embedding
mE5large-instruct
Gemma3 12B
Gemma3 4B
EmbeddingGemma
GritLM-7B
mE5large
E5mistral-7b
Gemma3 12B
mE5base
Gemma3 4B
LN2@60
LN2@60
Baseline
Baseline
BulgarianStoreReviewSentimentClassification
78.1
78.6
61.5
61.8
71.3
77.6
63.9
79.3
56.4
69.4
54.9
DalajClassification
50.5
50.0
50.9
50.8
50.3
50.6
50.0
50.9
50.9
49.8
50.2
DBpediaClassification
94.8
95.5
93.3
92.9
94.3
92.9
88.3
90.9
91.9
87.1
92.0
EstonianValenceClassification
53.5
49.1
43.5
44.0
38.3
38.7
42.9
42.3
41.1
41.2
41.0
FinancialPhrasebankClassification
88.6
84.4
64.2
67.4
86.5
84.4
83.9
79.2
66.5
82.1
67.2
Appendix
Table 24: Performance comparison with embedding models. We compare Gemma3 4 B and 12 B, probed on the final-layer residual stream ( Baseline ) or with the fixed strategy LN2@60 , to the top- 5 models of enevoldsen2025mmteb , EmbeddingGemma ( vera2025embeddinggemma ) and Gemini Embedding ( lee2025geminiembedding ) . Results for embedding models are taken from the MTEB(Multilingual) leaderboard at https://huggingface.co/spaces/mteb/leaderboard . We report the test accuracy ( % ) on each task, the average accuracy, and the average rank across tasks (lower is better). Following MMTEB , models are ordered by average rank, which yields the same ordering as the Borda count used by the leaderboard since all models are evaluated on all tasks.
Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.
With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a lightweight head, is computationally efficient but often restricted to last-layer representations. We show that task-relevant information is distributed across the network hierarchy rather than encoded solely in the last layers. To leverage this distribution of information, we apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer. This attentive layer fusion (ALF) learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions. Across 20 diverse datasets and multiple pretrained foundation models, ALF achieves consistent, substantial gains over standard linear probes. Attention heatmaps further reveal that tasks different from the pre-training domain benefit most from intermediate representations. Overall, our findings underscore the value of intermediate layers and demonstrate a principled, task-aware approach for unlocking their potential for probing-based adaptation.
Laure Ciernik, Marco Morik, Lukas Thede +4
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany +7
We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains with fewer trainable parameters when adapting large vision--language models (VLMs). Unlike the common practice of applying LoRA and other adapters to all layers at once---where layer selection often relies on heuristic rules---we focus on the vision encoder and directly evaluate the "adaptability'' of each Transformer layer. Specifically, we characterize each layer from two perspectives: (i) the statistical properties of its Q/K/V projection weights (e.g., norms and condition numbers); (ii) robustness under controlled parameter perturbations. We then systematically compare these indicators with the downstream performance gains brought by applying PEFT to a single layer. Across experiments covering seven benchmarks and five PEFT variants, we observe a consistent correlation: layers (or matrices) with larger weight norms and higher condition numbers are usually more robust to perturbations and are more likely to yield larger fine-tuning gains. These results show that distribution-statistics analysis and perturbation tests before fine-tuning can provide practical signals for adaptation-layer selection, thereby maintaining or improving performance while reducing trainable parameters.
Qingtao Xia, Jiahua Bao, Siyao Cheng +1
Faculty of Computing Harbin Institute of Technology Harbin, China