cs.LGAug 7, 2026

Hidden Gauge Controls Feature Specialization in ReLU Networks

Authors: Tongxi Wang

Organizations: School of Future Technology Southeast University Nanjing, China

Abstract

Training changes a network's predictions while allocating task-relevant structure across its internal units. In an overparameterized ReLU network, several neurons can begin with exactly the same functional role, yet one may acquire a teacher feature while the others become redundant. We call the identity of that neuron feature ownership and ask whether it can be controlled by a parameter choice invisible to the initial predictor. In a tractable Gaussian teacher--student model, we fix the complete initial function and vary only a positive-homogeneous scaling gauge. Opposite gauges produce distinct feature trajectories and a sharp Θ(D2)Θ(D^2) separation in specialization time that no global change of clock can explain. Among any fixed number of initially duplicate students, assigning the favorable gauge to one neuron deterministically selects it as the owner and drives the remaining functional contribution to zero. An exact reaction--transport decomposition attributes the effect to different mobilities for changing a feature's coefficient and direction. We prove global selection and functional pruning, extend finite-time selection to visible perturbations and small-step full-batch gradient descent, and verify the predicted loss, alignment, pruning, and dissipation trajectories in population and finite-sample training. The initial predictor therefore determines neither when the feature is learned nor which neuron learns it.

Explore similar work

CardsList
May 5, 2026cs.LG

Most ReLU Networks Admit Identifiable Parameters

We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.
Moritz Grillo, Guido Montúfar
Rectified Linear Unit NetworksIntrinsic Dimensionality
AskPDF
Jul 3, 2026cs.LG

Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the structure of this stationary measure and the predictor it represents. We show that, despite the network being infinitely overparameterized, the learned predictor admits an effectively finite representation: the input weights and biases align along finitely many directions, leading to an effective width collapse. In particular, the solution function is continuous piecewise affine, with affine regions determined by the cells of a finite hyperplane arrangement. The number of learned directions, and hence hyperplanes, is bounded above by 2P−12\mathcal{P}-1, where P\mathcal{P} denotes the number of linear dichotomies realizable on the training inputs. We further establish a non-redundancy property of the learned representation by proving that each learned direction induces a unique ternary activation pattern on the training data. Consequently, the complexity of the learned predictor is governed by the combinatorial geometry of the training data.
Shuang Liang, Tom Jacobs, Guido Montúfar
Rectified Linear Unit NetworksStochastic Gradient DescentGradient FlowMean-Field LimitRepresentation CollapseImplicit Bias
AskPDF
Apr 28, 2026cs.LG

Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix B=(F~⊤F~+ηI)−1B = (\widetilde{F}^\top \widetilde{F} + ηI)^{-1} during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries BjℓB_{j\ell}, producing an effective repulsive force that drives them apart. However, the theorem does not specify when this mechanism becomes empirically observable, nor whether it leaves a measurable spectral signature in the parameter updates. We test this directly on Tian's modular addition setup (M=71M = 71, K=2048K = 2048, MSE loss) and observe a clear structure-mechanism dissociation. The predicted sign rule holds robustly on the top-200 most-similar feature pairs across activations (empirical sign-match rising from 0.865 to 0.985 on σ=x2σ= x^2 across 5 seeds, and saturating at 1.000 on σ=ReLU⁡σ= \operatorname{ReLU}). However, the spectral signature in the parameter updates is strongly activation-dependent. With σ=x2σ= x^2, a simple slope detector on the rolling eigengap σ2/σ3σ_2 / σ_3 of ΔWΔW fires in 15/15 grokking seeds at epoch 174 (IQR [173,174]) and in 0/15 non-grokking controls, with 229×\times late-stage magnitude separation; the spectrum is rank-2. In contrast, with σ=ReLU⁡σ= \operatorname{ReLU}, the detector never fires and the spectrum remains effectively rank-1. This dissociation aligns with Tian's Theorem 5 distinction between focused (power-law) and spreading (ReLU) memorization: while the sign structure of BB depends only on F~⊤F~\widetilde{F}^\top \widetilde{F}, how feature repulsion translates into weight updates critically depends on the activation derivative σ′σ'.
Yongzhong Xu
Feature LearningGrokkingTwo-Layer Neural NetworksRotational Repulsion MechanismEmpirical Study
AskPDF