Linear Probing

Follow this topic Ask about this topic

In machine learning, linear probing means freezing a model’s representations and training a simple classifier or regressor—often logistic regression or ridge regression—to measure which task-relevant features they make accessible. Researchers apply it across layers, modules, and subspaces in vision, language, audio, medicine, and scientific models, using benchmarks for classification, geometry, factual knowledge, and out-of-distribution transfer. Methods now include layer fusion, low-rank and whitened probes, and probe-based steering, while comparisons with fine-tuning help assess label efficiency. A central tension is that easy linear decoding does not by itself show that a model uses that information causally, and probe results can be sensitive to label noise, task design, and distribution shift.

Latest papers

38 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement
Mar 6, 2026Yakov Pyotr ShkolnikovRepresentation Geometry in VLMs Vision-Language Models
Layer by layer, module by module: Choose both for optimal OOD probing of ViT
Mar 5, 2026Ambroise Odonnat, Vasilii Feofanov, Laetitia Chapel +2Vision Transformer OOD Generalization
Noah’s Ark Lab, Inria · L’Institut Agro, IRISA · Univ. Rennes 2, IRISA +1
Information Routing in Atomistic Foundation Models: How Task Alignment and Equivariance Shape Linear Disentanglement
Mar 3, 2026Joshua SteierMolecular Representation Learning Representation Disentanglement
Independent Researcher
Spectral Overfitting in Noisy Linear Probing of Pretrained Representations
Mar 2, 2026Zice Wang, Zhenyu ZhangRepresentation Learning Linear Probing
School of Future Technology, Northeastern University, Shenyang, China.
Multi-Probe Zero Collision Hash (MPZCH): Mitigating Embedding Collisions and Enhancing Model Freshness in Large-Scale Recommenders
Feb 19, 2026Ziliang Zhao, Bi Xue, Emma Lin +16Linear Probing Recommender Systems
Meta Platforms, Inc., Menlo Park, CA, USA. · OpenAI, San Francisco, CA, USA.
The Information Geometry of Softmax: Probing and Steering
Feb 17, 2026Kiho Park, Todd Nief, Yo Joong Choe +1Representation Geometry in Language Models Information Geometry
University of Chicago · INSEAD
Attentive multilayer fusion for vision transformers
Jan 14, 2026Laure Ciernik, Marco Morik, Lukas Thede +4Linear Probing Vision Transformer
Machine Learning Group, Technische Universität Berlin, Berlin, Germany · Hector Fellow Academy, Karlsruhe, Germany · Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany +7
Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings
Jan 13, 2026Hilde I. Hummel, Sandjai Bhulai, Rob D. van der Mei +1Audio Representation Learning Linear Probing
Centre of Mathematics and Computer Science, Stochastics group, Science Park 123, Amsterdam, 1098 XG, Netherlands · Vrije Universiteit, Department of Mathematics, De Boelelaan 1111, Amsterdam, 1081 HV, Netherlands · Naturalis Biodiversity Center, Darwinweg 2, Leiden, 2333 CR, Netherlands

Mar 6, 2026J/K move · Enter open · S save

Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement

Yakov Pyotr Shkolnikov

Representation Geometry in VLMs Vision-Language Models Linear Probing

Open paper AskPDF arXiv

Vision-language models encode continuous geometry that their text pathway fails to express: a 6,000-parameter linear probe extracts hand joint angles at 6.1 degrees MAE from frozen features, while the best text output achieves only 20.0 degrees -- a 3.3x bottleneck. LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees, providing evidence for a pathway-training deficit rather than a representational one. Training objective determines accuracy more than architecture: five encoders spanning self-supervised, contrastive, and hybrid paradigms converge to statistically equivalent accuracy (R^2 approximately 0.55, TOST-equivalent at delta=0.03) despite sharing as little as CKA=0.41 representational similarity -- functional convergence without representational convergence. Autoregressive generation damages geometric fidelity, but the damage originates in the generation process, not in language alignment: Qwen2.5-VL's LLM layers actually improve probe accuracy over its raw vision encoder. Layer-wise analysis reveals a universal mid-network accuracy peak across all architectures, with attention heads in layers 18-22 carrying disproportionate geometric signal. These findings enable a single frozen backbone to function as a multi-task geometric sensor through lightweight probes, without fine-tuning or text generation.