Scale redundancy and soft gauge fixing in positively homogeneous neural networks

Rodrigo Carmo Terin

cs.LGFeb 16, 2026

Scale redundancy and soft gauge fixing in positively homogeneous neural networks

Authors: Rodrigo Carmo Terin

Organizations: King Juan Carlos University, Facuty of Experimental Sciences and Technology, Department of Applied Physics, Av. del Alcalde de Móstoles, 28933, Madrid, Spain

Abstract

Neural networks with positively homogeneous activations exhibit an exact continuous reparametrization symmetry: neuron-wise rescalings generate parameter-space orbits along which the input--output function is invariant. We interpret this symmetry as a gauge redundancy and introduce gauge-adapted coordinates that separate invariant and scale-imbalance directions. Inspired by gauge fixing in field theory, we introduce a soft orbit-selection (norm-balancing) functional acting only on redundant scale coordinates. We show analytically that it induces dissipative relaxation of imbalance modes to preserve the realized function. In controlled experiments, this orbit-selection penalty expands the stable learning-rate regime and suppresses scale drift without changing expressivity. These results establish a structural link between gauge-orbit geometry and optimization conditioning, providing a concrete connection between gauge-theoretic concepts and machine learning.

Figures & tables

Figure 1: Gauge redundancy in parameter space. Neuron-wise positive rescalings generate continuous gauge orbits.

Figure 2: Toy regression dataset used in Sec. V . Training samples are drawn from $[-2,2]$ with additive Gaussian noise. The black curve shows the noiseless target function evaluated on the validation grid over $[-3,3]$ .

Figure 3: Validation mean-squared error as a function of the gauge-fixing strength $\lambda$ . Each point corresponds to the mean over independent random seeds, with error bars given by the standard deviation.

$\lambda$	Train MSE	Validation MSE
0.00	$0.5358\pm 0.0017$	$0.5169\pm 0.0038$
0.05	$0.5312\pm 0.0025$	$0.5156\pm 0.0044$
0.10	$0.5281\pm 0.0055$	$0.5218\pm 0.0128$
0.20	$0.5240\pm 0.0083$	$0.5418\pm 0.0413$
0.50	$0.5177\pm 0.0081$	$0.63\pm 0.16$

Table 1: Mean

\pm

standard deviation over independent random seeds.

Method	Learning rate	Validation MSE	Stability
Baseline	$5\times 10^{-3}$	$0.515360$	Stable
Baseline	$1\times 10^{-2}$	$0.516924$	Marginal
Baseline	$2\times 10^{-2}$	$0.521469$	Unstable
Gauge-fixed ( $\lambda=0.2$ )	$5\times 10^{-3}$	$0.524778$	Stable
Gauge-fixed ( $\lambda=0.2$ )	$1\times 10^{-2}$	$0.541801$	Stable
Gauge-fixed ( $\lambda=0.2$ )	$2\times 10^{-2}$	$0.567897$	Stable

Table 2: Learning-rate stress test. Gauge-fixed training remains stable at learning rates where baseline training becomes unstable in this toy setup.

Figure 4: Learning-rate stress test comparing baseline and gauge-fixed training ( $\lambda=0.2$ ). Gauge fixing expands the empirically stable learning-rate regime in this toy setup.

Figure 5: Functional invariance under random gauge transformations. Shown is a boxplot of the invariance error $\Delta_{\mathrm{inv}}$ defined in Eq. ( 11 ) across 200 random neuron-wise rescalings with $s_{i}>0$ . All values remain at double-precision numerical accuracy (min $3.99\times 10^{-18}$ , median $6.68\times 10^{-18}$ , max $1.14\times 10^{-17}$ ), confirming that the gauge transformations leave the network function unchanged up to machine precision.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Sep 8, 2026Hasan Amin, Wei-Kai Chang, Rajiv KhannaLearning Rate Scheduling Weight Decay
Purdue University
Hidden Gauge Controls Feature Specialization in ReLU Networks
Aug 7, 2026Tongxi WangReLU Neural Networks Neural Network Training Dynamics
School of Future Technology Southeast University Nanjing, China
Path-conditioned training: a principled way to rescale ReLU neural networks
Feb 23, 2026Arthur Lebeurrier, Titouan Vayer, Rémi GribonvalReLU Neural Networks Neural Network Training Dynamics
ENS de Lyon, CNRS, Inria, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France · Inria, Rennes, France · Inria, CNRS, ENS de Lyon, Université Claude Bernard Lyon 1, LIP, UMR 5668, 69342, Lyon cedex 07, France

Sep 8, 2026J/K move · Enter open · S save

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Hasan Amin, Wei-Kai Chang, Rajiv Khanna

Purdue University

Learning Rate Scheduling Weight Decay Adaptive Gradient Methods Neural Network Training Dynamics Neural Network Stability

Open paper AskPDF arXiv

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning. Code is available in https://github.com/shasanamin/normalized-optimization-dynamics.

Abstract

Neural Network Optimization

Explore similar work

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Hidden Gauge Controls Feature Specialization in ReLU Networks

Path-conditioned training: a principled way to rescale ReLU neural networks