cs.LGSep 27, 2026

Discovering Symmetries in Neural Network Parameter Spaces

Authors: Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

Organizations: Harvard University · IBM Research · Northeastern University · University of California, San Diego

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

SettingMotion (%)OutputCompositionInverseFits
Linear5.254.73 ⁣× ⁣10−44.73\!\times\!10^{-4}3.68 ⁣× ⁣10−33.68\!\times\!10^{-3}3.27 ⁣× ⁣10−33.27\!\times\!10^{-3}5/5
ReLU3.748.70 ⁣× ⁣10−48.70\!\times\!10^{-4}3.07 ⁣× ⁣10−33.07\!\times\!10^{-3}6.14 ⁣× ⁣10−36.14\!\times\!10^{-3}3/5
Sigmoid, k=1k=111.955.23 ⁣× ⁣10−55.23\!\times\!10^{-5}3.71 ⁣× ⁣10−43.71\!\times\!10^{-4}5.16 ⁣× ⁣10−45.16\!\times\!10^{-4}5/5
Sigmoid, k=2k=26.351.46 ⁣× ⁣10−41.46\!\times\!10^{-4}1.48 ⁣× ⁣10−31.48\!\times\!10^{-3}2.16 ⁣× ⁣10−32.16\!\times\!10^{-3}4/5
Tanh, separated layers1.509.57 ⁣× ⁣10−49.57\!\times\!10^{-4}3.29 ⁣× ⁣10−33.29\!\times\!10^{-3}3.53 ⁣× ⁣10−33.53\!\times\!10^{-3}2/5
GELU, separated layers1.501.07 ⁣× ⁣10−31.07\!\times\!10^{-3}4.41 ⁣× ⁣10−34.41\!\times\!10^{-3}3.77 ⁣× ⁣10−33.77\!\times\!10^{-3}2/5
Appendix
Table 1: Learned one-generator approximate actions in the small-network experiments. Errors are medians across five seeds of within-seed 9595 th percentiles over 512512 samples at coefficient radius 0.50.5 and one factor. Output error uses a fixed calibration scale; composition and inverse errors are relative to attempted motion. Motion is the median relative change in the transformed parameter block. The final column counts fits satisfying every tolerance in Appendix D , including two-factor tests, transport, subdivision, and nontrivial motion.
SettingObjectiveOutputComp.InverseTransportFits
LinearHybrid4.73 ⁣× ⁣10−44.73\!\times\!10^{-4}3.68 ⁣× ⁣10−33.68\!\times\!10^{-3}3.27 ⁣× ⁣10−33.27\!\times\!10^{-3}3.92 ⁣× ⁣10−33.92\!\times\!10^{-3}5/5
LinearFinite1.31 ⁣× ⁣10−31.31\!\times\!10^{-3}7.33 ⁣× ⁣10−37.33\!\times\!10^{-3}7.84 ⁣× ⁣10−37.84\!\times\!10^{-3}1.27 ⁣× ⁣10−21.27\!\times\!10^{-2}0/5
ReLUHybrid8.70 ⁣× ⁣10−48.70\!\times\!10^{-4}3.07 ⁣× ⁣10−33.07\!\times\!10^{-3}6.14 ⁣× ⁣10−36.14\!\times\!10^{-3}6.55 ⁣× ⁣10−36.55\!\times\!10^{-3}3/5
ReLUFinite2.56 ⁣× ⁣10−32.56\!\times\!10^{-3}1.10 ⁣× ⁣10−21.10\!\times\!10^{-2}1.50 ⁣× ⁣10−21.50\!\times\!10^{-2}1.87 ⁣× ⁣10−21.87\!\times\!10^{-2}0/5
Sigmoid, k=1k=1Hybrid5.23 ⁣× ⁣10−55.23\!\times\!10^{-5}3.71 ⁣× ⁣10−43.71\!\times\!10^{-4}5.16 ⁣× ⁣10−45.16\!\times\!10^{-4}8.88 ⁣× ⁣10−48.88\!\times\!10^{-4}5/5
Sigmoid, k=1k=1Finite1.07 ⁣× ⁣10−41.07\!\times\!10^{-4}3.06 ⁣× ⁣10−33.06\!\times\!10^{-3}3.60 ⁣× ⁣10−33.60\!\times\!10^{-3}4.24 ⁣× ⁣10−34.24\!\times\!10^{-3}5/5
Appendix
Table 2: Objective comparisons for the one-generator small-network settings. Errors are medians across all five seeds of their within-seed 9595 th percentiles at radius 0.50.5 and one factor. The joint numerical criterion additionally includes the two-factor cell, subdivision, and motion. Each objective is tuned independently.
SettingConditioning batchFresh batch, same weights
Sigmoid, k=1k=14.46 ⁣× ⁣10−54.46\!\times\!10^{-5}1.11 ⁣× ⁣10−11.11\!\times\!10^{-1}
Sigmoid, k=2k=21.14 ⁣× ⁣10−41.14\!\times\!10^{-4}1.13 ⁣× ⁣10−21.13\!\times\!10^{-2}
Tanh8.06 ⁣× ⁣10−48.06\!\times\!10^{-4}3.58 ⁣× ⁣10−33.58\!\times\!10^{-3}
GELU9.23 ⁣× ⁣10−49.23\!\times\!10^{-4}5.48 ⁣× ⁣10−35.48\!\times\!10^{-3}
Appendix
Table 3: Preservation on conditioning and fresh batches using the same transformed weights. Entries are medians across five seeds of within-seed 9595 th-percentile errors, using the original calibration scale. Both columns use exactly the same frozen weights.
SettingField dim.Orbit rankReference-span error
Linear111.92 ⁣× ⁣10−31.92\!\times\!10^{-3}
ReLU117.14 ⁣× ⁣10−37.14\!\times\!10^{-3}
Sigmoid, k=1k=1111.35 ⁣× ⁣10−41.35\!\times\!10^{-4}
Sigmoid, k=2k=2119.93 ⁣× ⁣10−49.93\!\times\!10^{-4}
Local ReLU host113.91 ⁣× ⁣10−33.91\!\times\!10^{-3}
Appendix
Table 4: Induced-field agreement for the reported one-generator actions. Containment entries are medians across five seeds of mean normalized reference-span distances. Field dimension and sampled orbit rank equal one in each seed.
NetworkWidthsDimensionCenterDerivedOutput error
Linear(2,2,2)(2,2,2)4131.10×10−151.10\times 10^{-15}
Linear(2,2,2,2)(2,2,2,2)8261.47×10−151.47\times 10^{-15}
ReLU(2,3,2)(2,3,2)3301.13×10−151.13\times 10^{-15}
ReLU(2,3,3,1)(2,3,3,1)6601.16×10−151.16\times 10^{-15}
ReLU(2,2,2,2,2,1)(2,2,2,2,2,1)8801.26×10−151.26\times 10^{-15}
Appendix
Table 5: Higher-dimensional recovery by the affine-field solver. Widths include input and output dimensions. The dimension is inferred from the sampled constraints; all three extraction seeds agree. The center consists of generators commuting with every generator, and the derived algebra is the span of their commutators. Exact reconstruction confirms the reference-span identification and the displayed algebra dimensions. Output errors are medians across seeds of within-seed 9595 th percentiles over 128128 independent parameter/input samples and four-factor transformations.
RadiusLocal motionFull motionLocal outputFull output
0.10.01190.01151.92 ⁣× ⁣10−41.92\!\times\!10^{-4}1.12 ⁣× ⁣10−31.12\!\times\!10^{-3}
0.30.04300.04165.91 ⁣× ⁣10−45.91\!\times\!10^{-4}4.09 ⁣× ⁣10−34.09\!\times\!10^{-3}
0.50.04860.04729.12 ⁣× ⁣10−49.12\!\times\!10^{-4}5.26 ⁣× ⁣10−35.26\!\times\!10^{-3}
0.80.08530.08281.61 ⁣× ⁣10−31.61\!\times\!10^{-3}8.20 ⁣× ⁣10−38.20\!\times\!10^{-3}
Appendix
Table 6: Paired local and full-parameter discovery in the (2,8,8,1)(2,8,8,1) ReLU host. Motion is the median across seeds of within-seed median raw displacement. Output is the median across seeds of within-seed 9595 th-percentile absolute host error.
ObjectiveOutputCompositionInverseTransportCancellationFits
Finite6.00 ⁣× ⁣10−46.00\!\times\!10^{-4}8.00 ⁣× ⁣10−38.00\!\times\!10^{-3}3.16 ⁣× ⁣10−33.16\!\times\!10^{-3}1.40 ⁣× ⁣10−21.40\!\times\!10^{-2}2.43 ⁣× ⁣10−22.43\!\times\!10^{-2}0/3
Hybrid2.18 ⁣× ⁣10−42.18\!\times\!10^{-4}1.93 ⁣× ⁣10−31.93\!\times\!10^{-3}1.12 ⁣× ⁣10−31.12\!\times\!10^{-3}1.72 ⁣× ⁣10−31.72\!\times\!10^{-3}9.98 ⁣× ⁣10−39.98\!\times\!10^{-3}2/3
Appendix
Table 7: Pythia-160M, layer 66 , eight protected tokens. Errors are medians across three seeds of within-seed 9595 th percentiles at radius 0.50.5 and one factor. The final column counts fits satisfying every tolerance, including two factors, subdivision, and visible compensation.
ModelTransformationMotion (%)Relative logitsRMS logitsCancellation
160MLearned0.15418.82 ⁣× ⁣10−78.82\!\times\!10^{-7}7.13 ⁣× ⁣10−47.13\!\times\!10^{-4}4.54 ⁣× ⁣10−34.54\!\times\!10^{-3}
160MAnalytic0.15238.36 ⁣× ⁣10−78.36\!\times\!10^{-7}6.76 ⁣× ⁣10−46.76\!\times\!10^{-4}1.95 ⁣× ⁣10−51.95\!\times\!10^{-5}
160MIncoming only0.02311.84 ⁣× ⁣10−61.84\!\times\!10^{-6}1.48 ⁣× ⁣10−31.48\!\times\!10^{-3}1.00 ⁣× ⁣1001.00\!\times\!10^{0}
160MRandom0.15203.09 ⁣× ⁣10−63.09\!\times\!10^{-6}2.50 ⁣× ⁣10−32.50\!\times\!10^{-3}6.56 ⁣× ⁣10−16.56\!\times\!10^{-1}
1BLearned0.06871.50 ⁣× ⁣10−61.50\!\times\!10^{-6}5.37 ⁣× ⁣10−65.37\!\times\!10^{-6}3.29 ⁣× ⁣10−33.29\!\times\!10^{-3}
1BAnalytic0.06951.41 ⁣× ⁣10−61.41\!\times\!10^{-6}5.02 ⁣× ⁣10−65.02\!\times\!10^{-6}5.22 ⁣× ⁣10−45.22\!\times\!10^{-4}
Appendix
Table 8: Installed transformations at radius 0.50.5 , starting from the original checkpoint. Motion is the median percentage change in the selected parameter block. Error columns are medians across all three seeds of within-seed 9595 th percentiles over 1616 transformations. Analytic and random controls are calibrated to similar physical motion; incoming-only removes the learned compensation.
ModelObjectiveRaw motionCompositionInverse
160MFinite2.94 ⁣× ⁣10−32.94\!\times\!10^{-3}2.51 ⁣× ⁣10−32.51\!\times\!10^{-3}1.91 ⁣× ⁣10−31.91\!\times\!10^{-3}
160MHybrid2.97 ⁣× ⁣10−32.97\!\times\!10^{-3}3.08 ⁣× ⁣10−43.08\!\times\!10^{-4}5.92 ⁣× ⁣10−45.92\!\times\!10^{-4}
1BFinite2.39 ⁣× ⁣10−32.39\!\times\!10^{-3}3.21 ⁣× ⁣10−33.21\!\times\!10^{-3}2.86 ⁣× ⁣10−32.86\!\times\!10^{-3}
1BHybrid2.39 ⁣× ⁣10−32.39\!\times\!10^{-3}5.81 ⁣× ⁣10−45.81\!\times\!10^{-4}9.00 ⁣× ⁣10−49.00\!\times\!10^{-4}
Appendix
Table 9: Stored-weight action consistency at radius 0.50.5 . Both learned objectives use the same selected support and separately tuned recipes. Error entries are medians across three seeds of within-seed 9595 th percentiles. These evaluations start at the original checkpoint.

Explore similar work

CardsList
Sep 28, 2026cs.LG

On Parameter Symmetries and Conservation Laws in Gradient Flow

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

Khang Nguyen, Guido Montúfar

UCLA · UCLA & MPI MiS

SymmetryGradientIdentifiability
AskPDF
Sep 1, 2026cs.LG

Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks

Artificial neural networks are often regarded as powerful yet opaque black boxes. Here, we demonstrate that learning in deep neural networks generates local symmetries known in graph theory as fibrations and coverings. We prove that covering symmetries are stable attractors of stochastic gradient descent. Consistent with this theory, we report the emergence of covering symmetries across major network architectures, including multilayer, convolutional, recurrent, and transformer networks. Exploiting these symmetries enables drastic model compression - reducing networks to 17% of their original size without sacrificing performance. Furthermore, controlled breaking of covering symmetry overcomes the loss of plasticity, achieving state-of-the-art performance in continual learning. The theoretical results provide a new foundation for AI systems based on symmetries that convert black boxes into interpretable colored graphs and enable more efficient inference and lifelong learning.

Osvaldo M Velarde, Lucas C Parra, Alireza Hashemi +1

Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USA. · Biomedical Engineering Department, City College of New York, 160 Convent Ave, NY, 610101, NY, USA.

SymmetryCompressed ModelAttractorsData Compression MethodsEmergence
AskPDF
Jun 9, 2026cs.LG

Conservation Laws from Data Symmetry in Neural Networks

We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks. Under the assumption that the loss function is analytic and non-polynomial, we prove that data symmetries generically do not induce any additional integrals of motion. For mean squared error (MSE) loss, on the other hand, there are situations in which data augmentation yields extra conserved quantities. We build a framework, utilizing \emph{tensorizable networks} to describe this phenomenon. Tensorizable networks are a family of architectures whose dependence on parameters and inputs can be separated using an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention.

Jakob Galley, Vahid Shahverdi, Axel Flinth

Umeå University, Umeå, Sweden

Equivariant Neural NetworksSymmetryNeural NetworkLoss Function
AskPDF