Parameter-Level Attribution of Symmetry in Trained Networks Though Parameter-Wise Functional Sensitivity

Alan Muriithi; Vedanta Thapar; Torben Berndt

cs.LGAug 25, 2026

Parameter-Level Attribution of Symmetry in Trained Networks Though Parameter-Wise Functional Sensitivity

Authors: Alan Muriithi, Vedanta Thapar, Torben Berndt

Organizations: Mathematical Institute, University of Oxford · Heidelberg Institute for Theoretical Studies

Abstract

When a network has learned a function with a known symmetry, can that symmetry be moved through the parametrisation---is there a motion in parameter space realising the group action in function space? We formulate this as a lifting problem for the realisation map $Φ:θ\mapsto f_θ$ , and show that a smooth parameter-space action exists only if the tangent space to the function's symmetry orbit lies within the image of $\mathrm dΦ_θ$ , whose columns are the functional sensitivities of individual parameters. This condition is also sufficient for pointwise first-order lifting. Relaxing it in least squares yields two local parameter directions: one following the symmetry orbit, one descending towards the equivariant subspace, with residuals measuring what the parametrisation cannot reach. On a rotationally invariant classifier we find these directions induce their predicted function-space motion, but only locally: recomputed directions track the orbit and reduce the equivariance defect, while directions held fixed depart from both after training. The same holds for Hamiltonian neural networks trained on a rotationally symmetric potential, even though the architecture does not explicitly enforce the symmetry.

Figures & tables

Panel of Figure 1 — Figure 1: Resolved parameter-space flows for the wedge-trained classifier. (a) The equivariance-directed flow along $b^{\star}$ moves the realised function towards the equivariant subspace, reducing its non-invariant component. (b) The symmetry-directed flow along $c^{\star}$ moves the realised function along its rotational symmetry orbit while approximately preserving its distance from the equivariant subspace. The local direction is recomputed at every integration step.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Global and infinitesimal lifting through the neural-network realisation map. Left: can a symmetry transformation in function space be realised by a transformation in parameter space? Right: do the infinitesimal symmetry directions lie in the function-space tangent directions accessible through the parametrisation?

Figure 3: Angular deviation profiles for the ASRNN trained to model the Mexican hat potential for various values of

\alpha

unseen during training. Top: symmetry-directed flow along

c^{\star}

. Bottom: equivariance-directed flow along

b^{\star}

. Here the local direction is recomputed at every step, i.e. the Resolved trajectory.

Figure 4: Angular deviation profiles for the ASRNN trained to model the Mexican hat potential for various values of

\alpha

unseen during training. Top: symmetry-directed flow along

c^{\star}

. Bottom: equivariance-directed flow along

b^{\star}

. Here the direction is computed once at

t=0

and kept fixed, i.e. the Fixed trajectory.

Panel of Figure 5 — Figure 5: Fixed parameter-space flows across training conditions. (a) Angular deviation profiles along the symmetry- and equivariance-directed flows. (b) Decision boundaries along the symmetry-directed flow $c^{\star}$ . (c) Decision boundaries along the equivariance-directed flow $b^{\star}$ . The parameter-space directions are computed once at $t=0$ and kept fixed throughout the trajectory.

Panel of Figure 6 — Figure 6: Resolved parameter-space flows across training conditions. (a) Angular deviation profiles along the symmetry- and equivariance-directed flows. (b) Decision boundaries along the symmetry-directed flow $c^{\star}$ . (c) Decision boundaries along the equivariance-directed flow $b^{\star}$ . The Jacobian and corresponding parameter-space directions are recomputed at every step.

Figure 7: Angular deviation profiles for the wedge-trained classifier. Top: symmetry-directed flow along

c^{\star}

; bottom: equivariance-directed flow along

b^{\star}

. Left: the local direction is recomputed at every step; right: the initial direction is kept fixed. Solid curves denote the realised trajectory, dotted curves the ideal function-space evolution.

Panel of Figure 8 — Figure 8: Fixed parameter-space flows for the wedge-trained classifier. (a) The initial equivariance-directed direction $b^{\star}$ does not continue to move the realised function towards the equivariant subspace. (b) The initial symmetry-directed direction $c^{\star}$ deviates from the intended rotational symmetry orbit. In both cases, the direction is computed once at $t=0$ and kept fixed throughout the trajectory.

Explore similar work

Discovering Symmetries in Neural Network Parameter Spaces
Sep 27, 2026Bo Zhao, Nima Dehmamy, Robin Walters +1Scientific Discovery
Harvard University · IBM Research · Northeastern University +1
A Complete Symmetry Classification of Shallow ReLU Networks
Apr 15, 2026Pranavkrishnan RamakrishnanShallow Neural Networks Neural Representation Geometry
On Parameter Symmetries and Conservation Laws in Gradient Flow
Sep 28, 2026Khang Nguyen, Guido MontúfarNeural Network Training Dynamics Gradient Descent Dynamics
UCLA · UCLA & MPI MiS

Sep 27, 2026J/K move · Enter open · S save

Discovering Symmetries in Neural Network Parameter Spaces

Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

Harvard University · IBM Research · Northeastern University+1

Scientific Discovery

Open paper AskPDF arXiv

Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Abstract

Explore similar work

Discovering Symmetries in Neural Network Parameter Spaces

A Complete Symmetry Classification of Shallow ReLU Networks

On Parameter Symmetries and Conservation Laws in Gradient Flow