cs.LGNov 8, 2026

Probing Transformer Representations Layer by Layer, Module by Module

Authors: Ambroise Odonnat, Theo Gnassounou, Laetitia Chapel, Romain Tavenard, Ievgen Redko

Organizations: Noah’s Ark Lab, Inria · EPFL · L’Institut Agro, IRISA · Univ. Rennes 2, IRISA · 

Abstract

Which representation should be probed in a transformer-based foundation model? Recent work shows that the standard practice of probing the final layer is often suboptimal, as intermediate layers can provide better features. Yet, representations are still largely studied along depth alone, treating each transformer block as an atomic unit. In this work, we analyze transformer representations layer by layer, module by module. Through more than 144,000 linear probing evaluations of vision foundation models (VFMs) and large language models (LLMs), we show that layer and module choices should be considered jointly: the best layer-module pair yields substantially larger gains over the final-layer residual stream than the best layer with the residual stream fixed. Moreover, module performance can vary substantially across layers, showing that useful representations cannot be characterized by depth alone. Building on these findings, we evaluate simple fixed probing strategies: the final-layer feedforward activation (Act@100) for vision and the second normalization layer at 60% depth (LN2@60) for language. These strategies improve over the probing baseline in a majority of evaluations and recover, on average, 42% and 86% of the gains from probing the best layer-module pair for vision and language, respectively. For VFMs, these gains increase with the transfer difficulty between pretraining and downstream data.

Figures & tables

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Layer by layer, module by module: Choose both for optimal OOD probing of ViT

    Mar 5, 2026Ambroise Odonnat, Vasilii Feofanov, Laetitia Chapel +2Vision TransformerOOD Generalization

  2. Attentive multilayer fusion for vision transformers

    Jan 14, 2026Laure Ciernik, Marco Morik, Lukas Thede +4Linear ProbingVision Transformer

  3. Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

    Sep 14, 2026Qingtao Xia, Jiahua Bao, Siyao Cheng +1VLM AdaptationParameter-Efficient Fine-Tuning