On Parameter Symmetries and Conservation Laws in Gradient Flow

Khang Nguyen; Guido Montúfar

cs.LGSep 28, 2026

On Parameter Symmetries and Conservation Laws in Gradient Flow

Authors: Khang Nguyen, Guido Montúfar

Organizations: UCLA · UCLA & MPI MiS

Abstract

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Combination Type	Operation	Typical Examples
Additive	$z=g_{\theta}(x)+f_{\omega}(x)$	ResNet [ He et al., 2015 ] , LoRA [ Hu et al., 2022 ]
Concatenation	$z=[g_{\theta}(x);\,f_{\omega}(x,g_{\theta}(x))]$	Inception [ Szegedy et al., 2015 ] , U-Net [ Ronneberger et al., 2015 ]
Composition	$z=f_{\omega}(g_{\theta}(x)),\text{ or }z=g_{\theta}(f_{\omega}(x))$	MLPs [ Rumelhart et al., 1986 ] , CNNs [ Lecun et al., 1998 ]
Hadamard Product / Gating	$z=g_{\theta}(x)\odot f_{\omega}(x)$	LSTM [ Hochreiter and Schmidhuber, 1997 ] and GRU gates [ Cho et al., 2014 ]
Input-dependent Weighted Sum	$z=a_{\omega}(x)\,g_{\theta}(x)+f_{\omega}(x)$	Mixture-of-Experts [ Shazeer et al., 2017 ]
Outer Product	$Z=g_{\theta}(x)\,f_{\omega}(x)^{\top}$	Bilinear CNNs [ Lin et al., 2015 ]

Appendix

Table 1: Examples of function combinations compatible with Proposition 15 . The listed operations occur within the indicated architectures. The categories are not mutually exclusive. The list is not exhaustive. We write

g_{\theta}(u)=g(\theta,u)

, and

\mathcal{N}(i)

is the neighborhood of node

i

Explore similar work

Conservation Laws from Data Symmetry in Neural Networks
Jun 9, 2026Jakob Galley, Vahid Shahverdi, Axel FlinthEquivariant Neural Networks Symmetry
Umeå University, Umeå, Sweden
Conservation Laws for Modern Neural Architectures
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Tan Lai Ngoc +3Overparameterization Gradient
National University of Singapore · Center for AI Research, VinUniversity · Independent Researcher +1
Discovering Symmetries in Neural Network Parameter Spaces
Sep 27, 2026Bo Zhao, Nima Dehmamy, Robin Walters +1Symmetry Equivariant Neural Networks
Harvard University · IBM Research · Northeastern University +1

Jun 9, 2026, cs.LGJ/K move · Enter open · S save

Conservation Laws from Data Symmetry in Neural Networks

Jakob Galley, Vahid Shahverdi, Axel Flinth

Umeå University, Umeå, Sweden

Equivariant Neural Networks Symmetry Neural Network Loss Function

Open paper AskPDF arXiv

We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks. Under the assumption that the loss function is analytic and non-polynomial, we prove that data symmetries generically do not induce any additional integrals of motion. For mean squared error (MSE) loss, on the other hand, there are situations in which data augmentation yields extra conserved quantities. We build a framework, utilizing \emph{tensorizable networks} to describe this phenomenon. Tensorizable networks are a family of architectures whose dependence on parameters and inputs can be separated using an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention.

Abstract

Explore similar work

Conservation Laws from Data Symmetry in Neural Networks

Conservation Laws for Modern Neural Architectures

Discovering Symmetries in Neural Network Parameter Spaces