When a network has learned a function with a known symmetry, can that symmetry be moved through the parametrisation---is there a motion in parameter space realising the group action in function space? We formulate this as a lifting problem for the realisation map Φ:θ↦fθ, and show that a smooth parameter-space action exists only if the tangent space to the function's symmetry orbit lies within the image of dΦθ, whose columns are the functional sensitivities of individual parameters. This condition is also sufficient for pointwise first-order lifting. Relaxing it in least squares yields two local parameter directions: one following the symmetry orbit, one descending towards the equivariant subspace, with residuals measuring what the parametrisation cannot reach. On a rotationally invariant classifier we find these directions induce their predicted function-space motion, but only locally: recomputed directions track the orbit and reduce the equivariance defect, while directions held fixed depart from both after training. The same holds for Hamiltonian neural networks trained on a rotationally symmetric potential, even though the architecture does not explicitly enforce the symmetry.
Figures & tables
Figure 1: Resolved parameter-space flows for the wedge-trained classifier. (a) The equivariance-directed flow along b⋆ moves the realised function towards the equivariant subspace, reducing its non-invariant component. (b) The symmetry-directed flow along c⋆ moves the realised function along its rotational symmetry orbit while approximately preserving its distance from the equivariant subspace. The local direction is recomputed at every integration step.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Global and infinitesimal lifting through the neural-network realisation map. Left: can a symmetry transformation in function space be realised by a transformation in parameter space? Right: do the infinitesimal symmetry directions lie in the function-space tangent directions accessible through the parametrisation?
Figure 3: Angular deviation profiles for the ASRNN trained to model the Mexican hat potential for various values of α unseen during training. Top: symmetry-directed flow along c⋆ . Bottom: equivariance-directed flow along b⋆ . Here the local direction is recomputed at every step, i.e. the Resolved trajectory.
Figure 4: Angular deviation profiles for the ASRNN trained to model the Mexican hat potential for various values of α unseen during training. Top: symmetry-directed flow along c⋆ . Bottom: equivariance-directed flow along b⋆ . Here the direction is computed once at t=0 and kept fixed, i.e. the Fixed trajectory.
Figure 5: Fixed parameter-space flows across training conditions. (a) Angular deviation profiles along the symmetry- and equivariance-directed flows. (b) Decision boundaries along the symmetry-directed flow c⋆ . (c) Decision boundaries along the equivariance-directed flow b⋆ . The parameter-space directions are computed once at t=0 and kept fixed throughout the trajectory.
Figure 6: Resolved parameter-space flows across training conditions. (a) Angular deviation profiles along the symmetry- and equivariance-directed flows. (b) Decision boundaries along the symmetry-directed flow c⋆ . (c) Decision boundaries along the equivariance-directed flow b⋆ . The Jacobian and corresponding parameter-space directions are recomputed at every step.
Figure 7: Angular deviation profiles for the wedge-trained classifier. Top: symmetry-directed flow along c⋆ ; bottom: equivariance-directed flow along b⋆ . Left: the local direction is recomputed at every step; right: the initial direction is kept fixed. Solid curves denote the realised trajectory, dotted curves the ideal function-space evolution.
Figure 8: Fixed parameter-space flows for the wedge-trained classifier. (a) The initial equivariance-directed direction b⋆ does not continue to move the realised function towards the equivariant subspace. (b) The initial symmetry-directed direction c⋆ deviates from the intended rotational symmetry orbit. In both cases, the direction is computed once at t=0 and kept fixed throughout the trajectory.
Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.
Bo Zhao, Nima Dehmamy, Robin Walters +1
Harvard University · IBM Research · Northeastern University +1
Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as reverse engineering," or parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the neuromanifold, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.