cs.LGSep 27, 2026

Discovering Symmetries in Neural Network Parameter Spaces

Authors: Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

Organizations: Harvard University · IBM Research · Northeastern University · University of California, San Diego

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characterization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

SettingMotion (%)OutputCompositionInverseFits
Linear5.254.73 ⁣× ⁣10−44.73\!\times\!10^{-4}3.68 ⁣× ⁣10−33.68\!\times\!10^{-3}3.27 ⁣× ⁣10−33.27\!\times\!10^{-3}5/5
ReLU3.748.70 ⁣× ⁣10−48.70\!\times\!10^{-4}3.07 ⁣× ⁣10−33.07\!\times\!10^{-3}6.14 ⁣× ⁣10−36.14\!\times\!10^{-3}3/5
Sigmoid, k=1k=111.955.23 ⁣× ⁣10−55.23\!\times\!10^{-5}3.71 ⁣× ⁣10−43.71\!\times\!10^{-4}5.16 ⁣× ⁣10−45.16\!\times\!10^{-4}5/5
Sigmoid, k=2k=26.351.46 ⁣× ⁣10−41.46\!\times\!10^{-4}1.48 ⁣× ⁣10−31.48\!\times\!10^{-3}2.16 ⁣× ⁣10−32.16\!\times\!10^{-3}4/5
Tanh, separated layers1.509.57 ⁣× ⁣10−49.57\!\times\!10^{-4}3.29 ⁣× ⁣10−33.29\!\times\!10^{-3}3.53 ⁣× ⁣10−33.53\!\times\!10^{-3}2/5
GELU, separated layers1.501.07 ⁣× ⁣10−31.07\!\times\!10^{-3}4.41 ⁣× ⁣10−34.41\!\times\!10^{-3}3.77 ⁣× ⁣10−33.77\!\times\!10^{-3}2/5
Appendix
Table 1: Learned one-generator approximate actions in the small-network experiments. Errors are medians across five seeds of within-seed 9595 th percentiles over 512512 samples at coefficient radius 0.50.5 and one factor. Output error uses a fixed calibration scale; composition and inverse errors are relative to attempted motion. Motion is the median relative change in the transformed parameter block. The final column counts fits satisfying every tolerance in Appendix D , including two-factor tests, transport, subdivision, and nontrivial motion.
SettingObjectiveOutputComp.InverseTransportFits
LinearHybrid4.73 ⁣× ⁣10−44.73\!\times\!10^{-4}3.68 ⁣× ⁣10−33.68\!\times\!10^{-3}3.27 ⁣× ⁣10−33.27\!\times\!10^{-3}3.92 ⁣× ⁣10−33.92\!\times\!10^{-3}5/5
LinearFinite1.31 ⁣× ⁣10−31.31\!\times\!10^{-3}7.33 ⁣× ⁣10−37.33\!\times\!10^{-3}7.84 ⁣× ⁣10−37.84\!\times\!10^{-3}1.27 ⁣× ⁣10−21.27\!\times\!10^{-2}0/5
ReLUHybrid8.70 ⁣× ⁣10−48.70\!\times\!10^{-4}3.07 ⁣× ⁣10−33.07\!\times\!10^{-3}6.14 ⁣× ⁣10−36.14\!\times\!10^{-3}6.55 ⁣× ⁣10−36.55\!\times\!10^{-3}3/5
ReLUFinite2.56 ⁣× ⁣10−32.56\!\times\!10^{-3}1.10 ⁣× ⁣10−21.10\!\times\!10^{-2}1.50 ⁣× ⁣10−21.50\!\times\!10^{-2}1.87 ⁣× ⁣10−21.87\!\times\!10^{-2}0/5
Sigmoid, k=1k=1Hybrid5.23 ⁣× ⁣10−55.23\!\times\!10^{-5}3.71 ⁣× ⁣10−43.71\!\times\!10^{-4}5.16 ⁣× ⁣10−45.16\!\times\!10^{-4}8.88 ⁣× ⁣10−48.88\!\times\!10^{-4}5/5
Sigmoid, k=1k=1Finite1.07 ⁣× ⁣10−41.07\!\times\!10^{-4}3.06 ⁣× ⁣10−33.06\!\times\!10^{-3}3.60 ⁣× ⁣10−33.60\!\times\!10^{-3}4.24 ⁣× ⁣10−34.24\!\times\!10^{-3}5/5
Appendix
Table 2: Objective comparisons for the one-generator small-network settings. Errors are medians across all five seeds of their within-seed 9595 th percentiles at radius 0.50.5 and one factor. The joint numerical criterion additionally includes the two-factor cell, subdivision, and motion. Each objective is tuned independently.
SettingConditioning batchFresh batch, same weights
Sigmoid, k=1k=14.46 ⁣× ⁣10−54.46\!\times\!10^{-5}1.11 ⁣× ⁣10−11.11\!\times\!10^{-1}
Sigmoid, k=2k=21.14 ⁣× ⁣10−41.14\!\times\!10^{-4}1.13 ⁣× ⁣10−21.13\!\times\!10^{-2}
Tanh8.06 ⁣× ⁣10−48.06\!\times\!10^{-4}3.58 ⁣× ⁣10−33.58\!\times\!10^{-3}
GELU9.23 ⁣× ⁣10−49.23\!\times\!10^{-4}5.48 ⁣× ⁣10−35.48\!\times\!10^{-3}
Appendix
Table 3: Preservation on conditioning and fresh batches using the same transformed weights. Entries are medians across five seeds of within-seed 9595 th-percentile errors, using the original calibration scale. Both columns use exactly the same frozen weights.
SettingField dim.Orbit rankReference-span error
Linear111.92 ⁣× ⁣10−31.92\!\times\!10^{-3}
ReLU117.14 ⁣× ⁣10−37.14\!\times\!10^{-3}
Sigmoid, k=1k=1111.35 ⁣× ⁣10−41.35\!\times\!10^{-4}
Sigmoid, k=2k=2119.93 ⁣× ⁣10−49.93\!\times\!10^{-4}
Local ReLU host113.91 ⁣× ⁣10−33.91\!\times\!10^{-3}
Appendix
Table 4: Induced-field agreement for the reported one-generator actions. Containment entries are medians across five seeds of mean normalized reference-span distances. Field dimension and sampled orbit rank equal one in each seed.
NetworkWidthsDimensionCenterDerivedOutput error
Linear(2,2,2)(2,2,2)4131.10×10−151.10\times 10^{-15}
Linear(2,2,2,2)(2,2,2,2)8261.47×10−151.47\times 10^{-15}
ReLU(2,3,2)(2,3,2)3301.13×10−151.13\times 10^{-15}
ReLU(2,3,3,1)(2,3,3,1)6601.16×10−151.16\times 10^{-15}
ReLU(2,2,2,2,2,1)(2,2,2,2,2,1)8801.26×10−151.26\times 10^{-15}
Appendix
Table 5: Higher-dimensional recovery by the affine-field solver. Widths include input and output dimensions. The dimension is inferred from the sampled constraints; all three extraction seeds agree. The center consists of generators commuting with every generator, and the derived algebra is the span of their commutators. Exact reconstruction confirms the reference-span identification and the displayed algebra dimensions. Output errors are medians across seeds of within-seed 9595 th percentiles over 128128 independent parameter/input samples and four-factor transformations.
RadiusLocal motionFull motionLocal outputFull output
0.10.01190.01151.92 ⁣× ⁣10−41.92\!\times\!10^{-4}1.12 ⁣× ⁣10−31.12\!\times\!10^{-3}
0.30.04300.04165.91 ⁣× ⁣10−45.91\!\times\!10^{-4}4.09 ⁣× ⁣10−34.09\!\times\!10^{-3}
0.50.04860.04729.12 ⁣× ⁣10−49.12\!\times\!10^{-4}5.26 ⁣× ⁣10−35.26\!\times\!10^{-3}
0.80.08530.08281.61 ⁣× ⁣10−31.61\!\times\!10^{-3}8.20 ⁣× ⁣10−38.20\!\times\!10^{-3}
Appendix
Table 6: Paired local and full-parameter discovery in the (2,8,8,1)(2,8,8,1) ReLU host. Motion is the median across seeds of within-seed median raw displacement. Output is the median across seeds of within-seed 9595 th-percentile absolute host error.
ObjectiveOutputCompositionInverseTransportCancellationFits
Finite6.00 ⁣× ⁣10−46.00\!\times\!10^{-4}8.00 ⁣× ⁣10−38.00\!\times\!10^{-3}3.16 ⁣× ⁣10−33.16\!\times\!10^{-3}1.40 ⁣× ⁣10−21.40\!\times\!10^{-2}2.43 ⁣× ⁣10−22.43\!\times\!10^{-2}0/3
Hybrid2.18 ⁣× ⁣10−42.18\!\times\!10^{-4}1.93 ⁣× ⁣10−31.93\!\times\!10^{-3}1.12 ⁣× ⁣10−31.12\!\times\!10^{-3}1.72 ⁣× ⁣10−31.72\!\times\!10^{-3}9.98 ⁣× ⁣10−39.98\!\times\!10^{-3}2/3
Appendix
Table 7: Pythia-160M, layer 66 , eight protected tokens. Errors are medians across three seeds of within-seed 9595 th percentiles at radius 0.50.5 and one factor. The final column counts fits satisfying every tolerance, including two factors, subdivision, and visible compensation.
ModelTransformationMotion (%)Relative logitsRMS logitsCancellation
160MLearned0.15418.82 ⁣× ⁣10−78.82\!\times\!10^{-7}7.13 ⁣× ⁣10−47.13\!\times\!10^{-4}4.54 ⁣× ⁣10−34.54\!\times\!10^{-3}
160MAnalytic0.15238.36 ⁣× ⁣10−78.36\!\times\!10^{-7}6.76 ⁣× ⁣10−46.76\!\times\!10^{-4}1.95 ⁣× ⁣10−51.95\!\times\!10^{-5}
160MIncoming only0.02311.84 ⁣× ⁣10−61.84\!\times\!10^{-6}1.48 ⁣× ⁣10−31.48\!\times\!10^{-3}1.00 ⁣× ⁣1001.00\!\times\!10^{0}
160MRandom0.15203.09 ⁣× ⁣10−63.09\!\times\!10^{-6}2.50 ⁣× ⁣10−32.50\!\times\!10^{-3}6.56 ⁣× ⁣10−16.56\!\times\!10^{-1}
1BLearned0.06871.50 ⁣× ⁣10−61.50\!\times\!10^{-6}5.37 ⁣× ⁣10−65.37\!\times\!10^{-6}3.29 ⁣× ⁣10−33.29\!\times\!10^{-3}
1BAnalytic0.06951.41 ⁣× ⁣10−61.41\!\times\!10^{-6}5.02 ⁣× ⁣10−65.02\!\times\!10^{-6}5.22 ⁣× ⁣10−45.22\!\times\!10^{-4}
Appendix
Table 8: Installed transformations at radius 0.50.5 , starting from the original checkpoint. Motion is the median percentage change in the selected parameter block. Error columns are medians across all three seeds of within-seed 9595 th percentiles over 1616 transformations. Analytic and random controls are calibrated to similar physical motion; incoming-only removes the learned compensation.
ModelObjectiveRaw motionCompositionInverse
160MFinite2.94 ⁣× ⁣10−32.94\!\times\!10^{-3}2.51 ⁣× ⁣10−32.51\!\times\!10^{-3}1.91 ⁣× ⁣10−31.91\!\times\!10^{-3}
160MHybrid2.97 ⁣× ⁣10−32.97\!\times\!10^{-3}3.08 ⁣× ⁣10−43.08\!\times\!10^{-4}5.92 ⁣× ⁣10−45.92\!\times\!10^{-4}
1BFinite2.39 ⁣× ⁣10−32.39\!\times\!10^{-3}3.21 ⁣× ⁣10−33.21\!\times\!10^{-3}2.86 ⁣× ⁣10−32.86\!\times\!10^{-3}
1BHybrid2.39 ⁣× ⁣10−32.39\!\times\!10^{-3}5.81 ⁣× ⁣10−45.81\!\times\!10^{-4}9.00 ⁣× ⁣10−49.00\!\times\!10^{-4}
Appendix
Table 9: Stored-weight action consistency at radius 0.50.5 . Both learned objectives use the same selected support and separately tuned recipes. Error entries are medians across three seeds of within-seed 9595 th percentiles. These evaluations start at the original checkpoint.

Explore similar work

CardsList

  1. On Parameter Symmetries and Conservation Laws in Gradient Flow

    Sep 28, 2026Khang Nguyen, Guido MontúfarSymmetryGradient

    UCLA · UCLA & MPI MiS

  2. Emergence of Fibrations, Compression, and Symmetry Breaking in Artificial Neural Networks

    Sep 1, 2026Osvaldo M Velarde, Lucas C Parra, Alireza Hashemi +1SymmetryCompressed Model

    Levich Institute and Physics Department, City College of New York, 160 Convent Ave, NY, 10031, NY, USA. · Biomedical Engineering Department, City College of New York, 160 Convent Ave, NY, 610101, NY, USA.

  3. Conservation Laws from Data Symmetry in Neural Networks

    Jun 9, 2026Jakob Galley, Vahid Shahverdi, Axel FlinthEquivariant Neural NetworksSymmetry

    Umeå University, Umeå, Sweden