Transformation Laws in Neural Representations: Structure, Realisability, and Construction

Yuan Sun

cs.LGSep 16, 2026

Transformation Laws in Neural Representations: Structure, Realisability, and Construction

Authors: Yuan Sun

Abstract

How neural representations preserve the structure of input changes connects representation analysis with internal intervention. We study operable representational content through compatible actions of reference transformations on neural features. We characterise when a transformation descends through an encoder, and give a linear setting in which the defect is governed by the transformation's demand for discarded information, measured in the metric the representation induces. On a rectifier the failure to realise a transformation has two distinguishable sources --- what the source region has already made unrecoverable, and what it costs to satisfy every region the transformation visits with one operator --- and for a \textit{measured} harmonic carrier the same question has a closed answer: a linear realisation exists exactly when the retained harmonic blocks are invariant under the action. Using colour as the in-depth instance, we find that hue orbits in frozen visual features concentrate 84--88% of their energy in the first two harmonics with rotation planes shared across shapes, that this organisation is substantially inherited from input and architecture and is reshaped by training and depth, and that the measured structure supports prediction, transport from new starting states, and composition --- with global and local realisations differing sharply in which they achieve. Guided by the measurements, we construct a compact interface whose rotation action is fixed by the structure and never fitted: it reads hue zero-shot at 3.4 $^\circ$ median error on unseen shapes. Theory, structural measurement, and construction together establish transformation laws as a concrete object connecting the understanding of neural representations to their design.

Explore similar work

Jan 14, 2026cs.LG

Geometric Stability: The Missing Axis of Representations

Representational similarity analysis and related methods compare the internal geometries of neural networks, but they measure only alignment between spaces, leaving a blind spot -- whether a representation's structure is reliably recoverable, not merely similar. We introduce geometric stability, a distinct axis, and \textit{Shesha}, a metric that quantifies it from a single representation by correlating dissimilarity matrices built from complementary random halves of the feature dimensions. Unlike CKA and Procrustes distance, Shesha is provably non-invariant to orthogonal rotations of the feature basis. This is by design: the basis is privileged for learned models, since probes, patching, and steering act on coordinates, and a rotation-invariant metric cannot see whether the targeted structure survives them. A double dissociation isolates the mechanism -- removing the top principal component collapses CKA while Shesha holds, whereas rotating a representation into its eigenbasis, which preserves the spectrum and CKA exactly, collapses Shesha. Across 2,463 encoder configurations in seven domains, the metrics are redundant under geometry-preserving transforms and anti-correlate under compression (

ρ=-0.47

). Across 170 vision models spanning 6 clean and 38 corruption-shifted datasets, DINOv2 ranks first or second in transferability on three of six clean datasets yet bottom-quartile in stability on five, an isolated dissociation rather than a trade-off.

Prashant C. Raju

Semantic-Geometric Representations Reference-Free Metric Missing Data

Ask PDF

May 12, 2026cs.LG

Deep Minds and Shallow Probes

Neural representations are not unique objects. Even when two systems realize the same downstream computation, their hidden coordinates may differ by reparameterization. A probe family intended to reveal structure already present in a representation should therefore be stable under the relevant representation symmetries rather than be tied to a particular basis. We study this group action in the tractable exact setting of the final readout layer, where equivalent realizations induce affine changes of hidden coordinates. The resulting symmetry principle singles out a unique hierarchy of shallow coordinate-stable probes, with linear probes as its degree-1 member. We also show that a natural object for cross-model probe transfer is a shared probe-visible quotient--the representation modulo directions invisible to the probe family--rather than the full hidden state. Experiments on synthetic and real-world tasks support both predictions, showing where degree-2 probes help beyond linear ones and how quotient-based transfer enables coverage-aware monitor portability across model families. These results point toward a broader geometric representation theory of neural probing, with coverage-aware monitor transfer as a concrete operational consequence.

Su Hyeong Lee, Risi Kondor

Implicit Neural Representations Hidden States Symmetry

Ask PDF

Jun 2, 2026cs.LG

Neural Networks Provably Learn Spectral Representations for Group Composition

Understanding how structured internal structure emerges during neural network training is central to the study of deep learning. We investigate this phenomenon through the group composition task, where a two-layer neural network is trained to predict

g_1 \star g_2

for elements of a finite group

G

. By lifting the projected gradient flow to the Fourier domain, we demonstrate that the training dynamics are governed by a Riemannian gradient ascent on a representation-theoretic energy functional. We prove that, under random initialization, this flow drives each neuron to converge almost surely toward a single irreducible representation, while the cross-layer Fourier coefficients achieve a rotational rank-one alignment. This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon for matrix-valued group representations. Moreover, for Abelian groups, we provide a complete population-level description: random initialization promotes uniform diversification across nontrivial representations and induces Haar-uniform phases, jointly approximating the indicator via a majority-vote mechanism. We further prove that both phase alignment and representation competition emerge with exponential convergence rates.

Jianliang He, Leda Wang, Fengzhuo Zhang +2

Implicit Neural Representations Two-Layer Neural Networks Neural Network Gradient Flow Group Value Alignment

Ask PDF