stat.MLAug 24, 2026

A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks

Authors: Kaj Nyström

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We develop a finite-width geometric framework describing how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. Incompatibility among weight-generated covariance, gates, and backward sensitivities is quantified through three families of commutators: between gates and covariance, between sensitivities and covariance, and between average gradient outer products (AGOPs) and neural feature matrices (NFMs). An exact layerwise identity decomposes the sensitivity-covariance commutator into four sources: downstream transport, adjacent-layer imbalance, pointwise sensitivity fluctuations, and nonlinear gate-covariance interactions. The AGOP-NFM commutator is a singular-value-weighted transport of the internal commutator, explaining why observed feature-side alignment alone does not determine the internal geometry from which it emerges. Buffered localized energies resolve mixing between separated covariance subspaces. We establish spectral-gap, projector-evolution, and stabilization estimates, and formulate conditional Lyapunov principles that yield decay under explicit geometric error-bound or intrinsic-damping assumptions. These criteria do not follow from gradient flow alone and clarify why risk reduction need not imply commutator collapse. Analytic examples and numerical experiments exhibit factorization of spectral and activation geometry, transient growth, and cancellation among nonzero sources. In tested finite-time regimes, cancellation dominated by a negative transport-imbalance interaction persists across depths, widths, and two regression benchmarks. Spectral alignment therefore appears as a layer- and scale-dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.

Figures & tables

experimentHHwidthseedsr≤10−2r\leq 10^{-2}r≤10−3r\leq 10^{-3}r≤10−4r\leq 10^{-4}
synthetic depth23220202013
synthetic depth43220202020
synthetic depth63220202019
synthetic capacity4161010104
synthetic capacity46410101010
Concrete43220700
Table 1. Numbers of seeds reaching the indicated relative-risk levels. No architecture comparison is reported at a risk level not attained by every seed in the corresponding ensemble.
Figure 1. The three normalized commutator energies at matched relative risk r=10−3r=10^{-3} , plotted against normalized depth uℓ=ℓ/(H−1)u_{\ell}=\ell/(H-1) . Curves are means over the twenty-seed ensembles, subject to the active-seed convention for an undefined normalized entry, and shaded regions are pointwise 95%95\% bootstrap intervals. The logarithmic vertical scale makes clear that the gate, averaged internal, and aggregated feature geometries remain quantitatively distinct.
 G^D,ℓ\widehat{\mathcal{G}}_{D,\ell}G^Q,ℓ\widehat{\mathcal{G}}_{Q,\ell}G^S,ℓ\widehat{\mathcal{G}}_{\mathcal{S},\ell}
ℓ\ell10−110^{-1}10−310^{-3}10−110^{-1}10−310^{-3}10−110^{-1}10−310^{-3}
00.44860.44860.04460.08050.01340.0199
10.24180.24180.03620.03470.03060.0306
20.23780.23810.03270.03180.03290.0329
30.23680.23620.03320.03310.02780.0275
Table 2. Means for the four-hidden-layer width- 3232 synthetic network at two matched relative-risk levels. All twenty seeds enter except for an undefined normalized entry covered by the active-seed convention. A hundredfold decrease of the risk does not produce global commutator decay.
Figure 2. Source-energy fractions and aggregate cancellation ratio at r=10−3r=10^{-3} . Curves are active-seed means and shaded regions are pointwise 95%95\% bootstrap intervals; all twenty seeds enter whenever the displayed ratio is defined. Imbalance is the largest interior source, transport remains substantial, and the fluctuation fraction decreases toward the output. At the terminal hidden layer, transport and fluctuation vanish by the boundary structure of the recursion, so that layer is not directly comparable with the interior ones.
Figure 3. Width comparison for four hidden layers at r=10−3r=10^{-3} . The width- 3232 curves use twenty seeds and the width- 1616 and width- 6464 curves use ten seeds, subject to the active-seed convention for an undefined normalized entry; bands are pointwise 95%95\% bootstrap intervals. The internal and feature defects decrease over the tested widths, whereas the gate defect does not.
widthG^D,0\widehat{\mathcal{G}}_{D,0}G^Q,1\widehat{\mathcal{G}}_{Q,1}G^S,1\widehat{\mathcal{G}}_{\mathcal{S},1}χ1\chi_{1}χ2\chi_{2}ρ1tr,imb\rho_{1}^{\mathrm{tr},\mathrm{imb}}
160.36100.06350.05550.3520.348-0.663
320.44860.03470.03060.2960.290-0.712
640.46850.01850.01700.3140.270-0.649
Table 3. Seed means at r=10−3r=10^{-3} in the four-hidden-layer synthetic capacity experiment. The separation of the three alignment levels and the negative transport-imbalance interaction both persist across the tested widths. Source ratios are active-seed means whenever a denominator vanishes.
widthmean qQ,0,N(1,0),avq_{Q,0,N}^{(1,0),\mathrm{av}}median paired factormean q^D,0,N(1,0)\widehat{q}_{D,0,N}^{(1,0)}
 r=10−1r=10^{-1}r=10−3r=10^{-3}q(10−3)/q(10−1)q(10^{-3})/q(10^{-1})r=10−1r=10^{-1}r=10−3r=10^{-3}
164.61×10−34.61\times 10^{-3}9.15×10−39.15\times 10^{-3}2.090.19490.1960
321.88×10−31.88\times 10^{-3}3.41×10−33.41\times 10^{-3}2.030.22870.2287
647.63×10−47.63\times 10^{-4}1.24×10−31.24\times 10^{-3}1.790.23950.2395
Table 4. Localized input-layer behavior in the four-hidden-layer synthetic experiment. The raw averaged- QQ defect grows across the matched-risk interval, while the normalized localized gate defect is stationary. Every displayed projector boundary passes the spectral reliability test.
Figure 4. Test-sample alignment profiles at the common final training time for Concrete Compressive Strength and Energy Efficiency with heating load as the response. Curves are means over the twenty-seed ensembles, subject to the active-seed convention for an undefined normalized entry, and bands are pointwise 95%95\% bootstrap intervals. Both datasets reproduce the quantitative separation of the gate geometry from the averaged internal and aggregated feature geometries.
datasetχ1\chi_{1}ρ1tr,imb\rho_{1}^{\mathrm{tr},\mathrm{imb}}χ2\chi_{2}ρ2tr,imb\rho_{2}^{\mathrm{tr},\mathrm{imb}}e1imbe_{1}^{\mathrm{imb}}e2imbe_{2}^{\mathrm{imb}}
Concrete0.327-0.7220.347-0.6890.5170.539
Energy0.310-0.6780.289-0.7130.4800.516
Table 5. Twenty-seed means of the final-time source geometry on the two benchmarks, with undefined ratios omitted as above. The interior cancellation and the negative transport-imbalance interaction reproduce the synthetic pattern.
ccρ1tr,imb\rho_{1}^{\mathrm{tr},\mathrm{imb}}I1tr,imbI_{1}^{\mathrm{tr},\mathrm{imb}}χ1\chi_{1}e1tre_{1}^{\mathrm{tr}}e1imbe_{1}^{\mathrm{imb}}e1fluce_{1}^{\mathrm{fluc}}e1gatee_{1}^{\mathrm{gate}}
1/21/2-0.997-0.9940.0370.49700.50000.00070.0026
11-0.704-0.4920.2960.24100.49200.05800.2090
22-0.091-0.0050.4770.00210.50100.10700.3890
Table 6. Ten-seed means of the middle-layer source geometry at t=100t=100 in the function-preserving rescaling experiment. The intervention redistributes the source budget and changes its directional cancellation.
Figure 5. Evolution of the middle-layer transport-imbalance interaction and aggregate cancellation ratio in the paired rescaling experiment. Curves summarize ten paired seeds. The same realized initial function can lead to nearly exact source cancellation or to a regime in which transport is almost absent.
More in this topic

Neural Representation Geometry

Neural representation geometry studies how learned features are arranged and how that structure relates to model behavior and performance.

See all 256 papers

Explore similar work

CardsList
May 11, 2026

Flag Varieties: A Geometric Framework for Deep Network Alignment

Alignment, the tendency of adjacent weight matrices in deep networks to develop compatible subspace orientations, underlies gradient flow, Neural Collapse, and representation similarity across architectures. Despite extensive empirical documentation, these phenomena have resisted unified theoretical treatment: existing explanations are post-hoc, each fitted to a specific observation with whatever mathematics is at hand. We reverse this direction by deriving the mathematical structure that layerwise alignment inherently demands. Using geometric invariant theory, we prove that alignment geometry has a canonical closed, polystable stratum given by a flag variety, and that subspace intersection dimension is its unique reparameterization-invariant observable, establishing that subspace metrics are not empirical conventions but mathematical necessities. This unified framework yields two dynamical consequences: ridge regularization drives subspace alignment at an exponential rate set by weight decay, whereas nonlinear activations induce a commutator obstruction to exact basis alignment, generically present in nonlinear networks and absent in linear ones. Together these give a geometric explanation of the Level-2/3 hierarchy in Neural Collapse from first principles rather than post-hoc analysis. The commutator magnitude and head subspace overlap further serve as weight-space windows into internal alignment structure, requiring no forward passes. Experiments on multilayer perceptrons, residual networks, and pretrained language models support the proposed diagnostics and delineate their scope.

Jingchuan Xiao, Xinyi Sui, Cihan Ruan

Department of Mathematics and Computer Studies, Mary Immaculate College, Ireland · Department of Computer Science and Engineering, Santa Clara University, USA

Neural Representation GeometryNeural CollapseNeural Network Training Dynamics
AskPDF
May 4, 2026

Geometric and Spectral Alignment for Deep Neural Network II

This paper develops the angular and static-channel component of Geometric and Spectral Alignment for residual Jacobian chains. Starting from Cartan-coordinate rigidity and fitted effective-rank windows, we study how dominant singular subspaces are transported across adjacent layers and how the resulting finite matrices can be displayed in physical channel coordinates. The main results are deterministic, margin-verified results. We bound the error between full interface transport and its dominant-window truncation, add fitted-tail errors so that empirical spectra can be certified against the Gibbs--Cartan tail model, and distinguish source-mode incidence from fully physical input-output channel incidence. Given row groups and active supports, the Physical Alignment Matrix decomposes orthogonally as core plus overlap plus noise. Active-column gaps, pairwise overlap margins, and noise bounds combine into a static certificate radius under which the full transport and the truncated transport induce the same active supports, pairwise incidence graph, SRS sets, hub columns, and core/overlap/noise masks. The finer SC/SA/ST labels of the Invariant Channel Mapping require additional row-energy and profile-correlation margins, stated as explicit perturbation tests. The empirical section reports the matrices and block-energy heatmaps that measure these certificate quantities across CNNs, language models, and vision/diffusion backbones. The figures are interpreted as finite-dimensional measurements; complete membership in the Physical GSA certificate domain requires checking the numerical margin protocol stated in Section 10.

Ziran Liu, Wei Wang, Jinhao Wang +5

Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai 200433, China · Futurewei Technologies, Inc., San Jose, CA 95131 · Dept. of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95050 +1

Neural Representation Geometry
AskPDF
May 7, 2026

The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks

Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models.

Taehun Cha, Daniel Beaglehole, Adityanarayanan Radhakrishnan +1

Department of Mathematics Korea University · Computer Science and Engineering UC San Diego · MIT Mathematics Broad Institute of MIT and Harvard

Neural Network Training DynamicsGradient Descent Dynamics
AskPDF