Loss-Invariant Projections as Passive Probes of Learned Representations

Akshay Chandrasekhar; Pavlo Melnyk

cs.LGOct 5, 2026

Loss-Invariant Projections as Passive Probes of Learned Representations

Authors: Akshay Chandrasekhar, Pavlo Melnyk

Organizations: Linköping University

Abstract

Learned feature representations in neural networks often contain structure beyond that directly used by the final task output. We study this structure using $\textit{passive probes}$ that apply fixed, untrained, property-independent projections to representations as they evolve during training. We motivate this approach through the task of prediction on $S^2$ where equivalent vector and Hermitian parameterizations reveal an additional loss-invariant trace coordinate. This motivates a general construction in which fixed random projections serve as observers of learned features. Because the observer is loss-invariant and independent of the property being studied, changes in accessibility reflect changes in the representation relative to the fixed observer rather than adaptation of the observer itself. We show that ensembles of passive probes can directly reflect task-relevant information such as target alignment. Under our constructions, the accessibility of eventual difficulty evolves differently across tasks. It increases during training in the regression tasks of surface-normal estimation and image inpainting but remains near its initial level in image classification. Comparisons with learned linear probes further show that recoverability and passive accessibility can evolve differently during training. Together, these results show how passive probes can separately characterize changes in representation geometry and the accessibility of eventual task difficulty.

Figures & tables

Panel of Figure 1 — Figure 1: Learned and passive observers of a changing representation. At a checkpoint, $f_{\theta}$ maps input $x$ to representation $h(x)$ , and the task readout $W$ produces prediction $Wh(x)$ . (a) A learned probe $({\bm{b}},c)$ is fitted from representation–property pairs $\{(h(x_{i}),e_{i})\}_{i=1}^{N}$ , where $e_{i}$ is the property of interest. (b) A passive probe instead uses a fixed, property-independent direction ${\bm{a}}_{j}$ , sampled once and never optimized, with $q_{j}(x)={\bm{a}}_{j}^{\top}h(x)$ . (c) The strict construction uses the final task readout $W$ to define the row-space projector $P_{W}=W^{\top}(WW^{\top})^{+}W$ , projects both $h(x)$ and ${\bm{a}}_{j}$ with $I-P_{W}$ , normalizes the resulting vectors, and computes their inner product.

Task	Family	Probe metrics ( $\times 10^{-3}$ )					Performance
		$\bm{A_{K}}$	$\bm{A_{G}}$	Pearson 2	$\bm{\|\mathrm{Spearman}\|}$	cal. $\bm{R^{2}}$
Surface normals	Raw	23.60 $\rightarrow$ 30.87 $(+7.27\pm 5.28)$	14.55 $\rightarrow$ 21.57 $(+7.01\pm 1.19)$	18.24 $\rightarrow$ 25.81 $(+7.56\pm 3.10)$	131.26 $\rightarrow$ 191.87 $(+60.61\pm 11.05)$	27.91 $\rightarrow$ 64.28 $(+36.37\pm 3.98)$	75.58 $\rightarrow$ 19.20 $(-56.38\pm 21.70)$ deg
Surface normals	Strict	16.79 $\rightarrow$ 16.00 $(-0.79\pm 2.04)$	9.33 $\rightarrow$ 14.78 $(+5.45\pm 1.33)$	13.79 $\rightarrow$ 15.60 $(+1.81\pm 1.50)$	110.84 $\rightarrow$ 154.95 $(+44.11\pm 8.80)$	17.13 $\rightarrow$ 51.76 $(+34.63\pm 4.61)$	75.58 $\rightarrow$ 19.20 $(-56.38\pm 21.70)$ deg
Classification	Raw	2.47 $\rightarrow$ 2.37 $(-0.11\pm 0.37)$	1.41 $\rightarrow$ 1.39 $(-0.03\pm 0.22)$	1.88 $\rightarrow$ 2.31 $(+0.43\pm 0.38)$	69.70 $\rightarrow$ 46.78 $(-22.92\pm 3.97)$	$-1.07\rightarrow-0.23$ $(+0.84\pm 0.47)$	1.00 $\rightarrow$ 70.62 $(+69.62\pm 0.56)$ %
Classification	Strict	2.28 $\rightarrow$ 2.02 $(-0.25\pm 0.53)$	1.39 $\rightarrow$ 1.26 $(-0.13\pm 0.37)$	1.87 $\rightarrow$ 2.12 $(+0.25\pm 0.59)$	69.27 $\rightarrow$ 44.20 $(-25.06\pm 7.40)$	$-1.23\rightarrow-0.91$ $(+0.32\pm 0.62)$	1.00 $\rightarrow$ 70.62 $(+69.62\pm 0.56)$ %
Inpainting	Raw	0.62 $\rightarrow$ 4.50 $(+3.88\pm 0.87)$	0.61 $\rightarrow$ 2.97 $(+2.36\pm 0.55)$	0.68 $\rightarrow$ 4.84 $(+4.16\pm 0.92)$	26.94 $\rightarrow$ 76.69 $(+49.75\pm 8.05)$	0.81 $\rightarrow$ 5.34 $(+4.54\pm 1.11)$	5.71 $\rightarrow$ 17.31 $(+11.60\pm 0.01)$ dB

Table 1: Passive accessibility during training. Entries report initialization

\rightarrow

final checkpoint, with

(\Delta\pm\mathrm{SD}_{\Delta})

shown below. Initial and final values are across-seed means;

\Delta

is computed within each seed before aggregation over five independently trained models. Probe metrics are reported in units of

10^{-3}

. Calibrated

R^{2}

uses a scalar calibration fitted on CAL and evaluated on TEST. Task performance is evaluated on TEST and is shared by the raw and strict probe families.

Figure 2: Strict passive accessibility across training. Calibrated

R^{2}

(top) and

A_{G}

(bottom) for strict passive probes at each evaluated training checkpoint, averaged over five independent training seeds; shading denotes

\pm 1

standard deviation across seeds. Surface-normal estimation shows a strong early increase in both measures followed by an elevated but non-monotonic trajectory, while inpainting shows a sustained increase across training. Classification exhibits intermediate variation but no sustained initial-to-final increase in either measure. Despite their different constructions, the two measures therefore show consistent task-level patterns. In particular,

A_{G}

requires no fitted mapping from the passive coordinates to eventual difficulty, whereas calibrated

R^{2}

uses calibration-set bin means.

Task	Observer	Metrics ( $\times 10^{-3}$ )
		Pearson 2	$\bm{\|\mathrm{Spearman}\|}$	cal. $\bm{R^{2}}$
Surface normals	Passive	18.24 $\rightarrow$ 25.81 $(+7.56\pm 3.10)$	131.26 $\rightarrow$ 191.87 $(+60.61\pm 11.05)$	27.91 $\rightarrow$ 64.28 $(+36.37\pm 3.98)$
Surface normals	Learned	93.01 $\rightarrow$ 186.94 $(+93.94\pm 13.28)$	329.68 $\rightarrow$ 517.11 $(+187.43\pm 7.68)$	97.17 $\rightarrow$ 189.82 $(+92.65\pm 9.25)$
Classification	Passive	1.88 $\rightarrow$ 2.31 $(+0.43\pm 0.38)$	69.70 $\rightarrow$ 46.78 $(-22.92\pm 3.97)$	$-1.07\rightarrow-0.23$ $(+0.84\pm 0.47)$
Classification	Learned	10.86 $\rightarrow$ 188.56 $(+177.70\pm 6.55)$	173.96 $\rightarrow$ 511.53 $(+337.57\pm 14.12)$	5.29 $\rightarrow$ 188.76 $(+183.48\pm 9.82)$
Inpainting	Passive	0.68 $\rightarrow$ 4.84 $(+4.16\pm 0.92)$	26.94 $\rightarrow$ 76.69 $(+49.75\pm 8.05)$	0.81 $\rightarrow$ 5.34 $(+4.54\pm 1.11)$

Table 2: Passive accessibility versus learned recoverability. Passive probes are fixed, property-independent scalar observers, whereas learned ridge probes are fitted to eventual difficulty separately at each checkpoint. Both use the raw representation and the same CAL/TEST observations. Entries report initialization

\rightarrow

final checkpoint, with

(\Delta\pm\mathrm{SD}_{\Delta})

shown below, across five training seeds. All metrics are reported in units of

10^{-3}

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Probe family	$\Delta A_{K}$	$\Delta A_{G}$	$\Delta R^{2}_{\mathrm{cal}}$
Surface normals	Raw	$+7.27\pm 5.28$	$+7.01\pm 1.19$	$+36.37\pm 3.98$
	Unit	$-0.92\pm 1.60$	$+5.79\pm 1.42$	$+38.12\pm 4.92$
	Orth	$+9.26\pm 4.82$	$+6.39\pm 1.32$	$+31.90\pm 3.77$
	Strict	$-0.79\pm 2.04$	$+5.45\pm 1.33$	$+34.63\pm 4.61$
Classification	Raw	$-0.11\pm 0.37$	$-0.03\pm 0.22$	$+0.84\pm 0.47$
	Unit	$-0.19\pm 0.36$	$-0.08\pm 0.21$	$+0.60\pm 0.37$

Appendix

Table 3: Passive-probe control analysis. Entries are mean within-seed endpoint change

\Delta

with sample standard deviation across five training seeds, in units of

10^{-3}

Task	Probe family	$B=5$	$B=10$	$B=20$	$B=40$
Surface normals	Raw	$+30.52\pm 3.92$	$+34.60\pm 3.85$	$+36.37\pm 3.98$	$+37.11\pm 4.01$
	Strict	$+23.27\pm 3.34$	$+30.59\pm 4.19$	$+34.63\pm 4.61$	$+36.39\pm 4.79$
Classification	Raw	$+0.34\pm 0.40$	$+0.57\pm 0.46$	$+0.84\pm 0.47$	$+0.98\pm 0.45$
	Strict	$+0.09\pm 0.57$	$+0.23\pm 0.60$	$+0.32\pm 0.62$	$+0.29\pm 0.63$
Inpainting	Raw	$+4.25\pm 1.04$	$+4.41\pm 1.09$	$+4.54\pm 1.11$	$+4.62\pm 1.12$
	Strict	$+8.29\pm 1.48$	$+9.02\pm 1.57$	$+9.42\pm 1.61$	$+9.60\pm 1.63$

Appendix

Table 4: Sensitivity of calibrated

R^{2}

to the number of CAL quantile bins. Entries are mean within-seed endpoint change

\Delta R^{2}_{\mathrm{cal}}

with sample standard deviation across five training seeds, in units of

10^{-3}

Explore similar work

Deep Minds and Shallow Probes
May 12, 2026Su Hyeong Lee, Risi KondorNeural Representations Hidden States
Department of Statistics, University of Chicago. · Department of Statistics and Department of Computer Science, University of Chicago.
Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning
Sep 27, 2026Soutrik Sarangi, Yonatan Sverdlov, Adir Dayan +2Neural Network Hidden States
Microsoft · Faculty of Mathematics, Technion – Israel Institute of Technology · Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology +1
Platonic Projection Structures: Operator-Induced Observability in Representation Learning
Jul 6, 2026Kazuo Ishii, Bishnu Prasad Gautam, Jieling Wu +1Partial Observability Representation Space
Department of Applied Information Engineering, Faculty of Engineering, Suwa University of Science, Chino 391-0292, Nagano, Japan · Department of Information Engineering, Kanazawa Gakuin University, Kanazawa 920-1392, Ishikawa, Japan

May 12, 2026, cs.LGJ/K move · Enter open · S save

Deep Minds and Shallow Probes

Su Hyeong Lee, Risi Kondor

Department of Statistics, University of Chicago. · Department of Statistics and Department of Computer Science, University of Chicago.

Neural Representations Hidden States Symmetry

Open paper AskPDF arXiv

Neural representations are not unique objects. Even when two systems realize the same downstream computation, their hidden coordinates may differ by reparameterization. A probe family intended to reveal structure already present in a representation should therefore be stable under the relevant representation symmetries rather than be tied to a particular basis. We prove that, under standard nondegeneracy assumptions, behaviorally equivalent LLMs have hidden representations related by an invertible affine transformation. The resulting symmetry principle singles out a unique hierarchy of shallow coordinate-stable probes, with linear probes as its degree-1 member. We also show that a natural object for cross-model probe transfer is a shared probe-visible quotient--the representation modulo directions invisible to the probe family--rather than the full hidden state. Experiments on synthetic and real-world tasks support both predictions, showing where degree-2 probes help beyond linear ones and how quotient-based transfer enables coverage-aware monitor portability across model families. These results point toward a broader geometric representation theory of neural probing, with coverage-aware monitor transfer as a concrete operational consequence.

Abstract

Explore similar work

Deep Minds and Shallow Probes

Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning

Platonic Projection Structures: Operator-Induced Observability in Representation Learning