cs.LGSep 30, 2026
SaveParameter symmetries determine representational geometry in overparameterized nonlinear networks
Organizations: University of Tübingen · International Max Planck Research School for Intelligent Systems · Allen Institute for Neural Dynamics · Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London · University of Alberta · Amii
Abstract
Representations are routinely used across machine learning, psychology, and neuroscience to draw inferences about the computations of biological and artificial systems. Such inferences presume a meaningful link between representational geometry and the computation being performed. For artificial neural networks, however, the extent to which function constrains representation remains unclear. One key obstacle is that these networks admit parameter symmetries: changes in parameterization that preserve function exactly while reshaping representational geometry. Here, we show that a broad class of parameter symmetries acts on representations through just three primitive feature transformations: addition, duplication, and scaling. This feature-level characterization yields a closed-form decomposition of representational geometry into essential and auxiliary components, which makes precise how degeneracy in representational geometry can grow with overparameterization even when function is held fixed. Finally, we show that implementation-level selection rules can resolve this degeneracy, yielding identifiable geometries in which features are weighted according to their contributions to the network's function. Together, our results delineate when representations can support inferences about computation, and when they cannot.
Figures & tables
Figure 1 : Parameter symmetries can doubly dissociate function and representation . (A) Parameter symmetries let networks implement the same function via different parameterizations. They act on individual neurons (e.g., positive scaling for ReLU networks), redistribute or cancel computation across groups (duplicates, zero groups), or couple groups of neurons through algebraic structure in the activation (linear or constant duplicates, linear or constant groups). (B) Gradient descent from random initializations yields six distinct two-neuron ReLU networks solving XOR (center), shown as input-space heatmaps with neuron decision boundaries overlaid. They form two families (top left and bottom right) that solve the same task using different functions, with representational similarity matrices of solutions from opposite families being nearly uncorrelated. Crucially, such representational variability persists even when the function is held fixed: overparameterized networks implementing exactly the same function can have nearly uncorrelated representational similarity matrices (top and bottom). Conversely, networks implementing different functions can have highly correlated representational similarity matrices (left and right), yielding a double dissociation.
Figure 2 : Primitive feature transformations unify parameter symmetries and separate essential from auxiliary geometry . (A) Parameter symmetries decompose into three primitive feature transformations and their inverses: addition , duplication , and scaling . Shown are their effects on hidden activations and representational similarity matrices when adding a zero-readout neuron to a solution in Panel B of Figure 1 , duplicating it, and rescaling both copies. (B) Decomposing the full representational similarity matrix of the resulting network (center) into essential and auxiliary contributions reveals the source of the dissociation in Figure 1 . Whereas the essential component is nearly uncorrelated with the representational similarity matrix of an opposite-family solution, the auxiliary component is nearly perfectly correlated with it, driving the full representational similarity matrix toward the reference geometry without changing the function being computed.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B.1 : Oppositely oriented irreducible parameterizations of the tent function . (Left) A parameterization of the tent function composed of three neurons, all with positive incoming weight. (Center) The same function, realized by an implementation consisting of the same three neurons, but with negated incoming parameters . (Right) The net contributions of the three linear-neuron groups comprising the auxiliary parameterization used to transform into . None vanishes individually, but their contributions sum to zero, allowing to be adjoined without changing the realized function.
| Activation | Even+Lin | Const+Odd | Scaling | |
| celu | – | – | ✗ | |
| elu | – | – | ✗ | |
| gelu | – | ✗ | ||
| hard_sigmoid | – | ✗ | ||
| hard_silu | – | ✗ | ||
| hard_tanh | – | ✗ |
Table C.1 : Symmetries of all scalar activation functions in jax.nn (v0.10.0). The aliases swish and hard_swish for silu and hard_silu, respectively, are omitted. We assume (celu, elu, leaky_relu, selu), (selu), and (squareplus). For leaky_relu, we additionally assume , since gives the identity. Even+Lin and Const+Odd report the linear slope and constant , respectively. Odd functions, which are the ones exhibiting the sign-flip symmetry, have . Scaling denotes the positive-scaling symmetry of positively -homogeneous activations.
Figure C.1 : Activations from jax.nn classified as even-linear, shown with their decomposition into even and odd components. Solid dark lines show the activations, dashed lines their even components, and dotted lines their odd components. In each case, the odd component is a line passing through the origin.
Figure C.2 : Activations from jax.nn classified as constant-odd, shown with their decomposition into even and odd components. Solid dark lines show the activations, dashed lines their even components, and dotted lines their odd components. In each case, the even component is a horizontal line—at for purely odd functions, where the odd component coincides with the activation itself.
Figure C.3 : Activations from jax.nn that are either both even-linear and constant-odd (the identity) or neither (the asymmetric activations), shown with their decomposition into even and odd components. Solid dark lines show the activations, dashed lines their even components, and dotted lines their odd components.
| Input | Transformation | Result |
|---|---|---|
| repeats the th entry of exactly times | ||
| repeats each row of exactly times | ||
| repeats each column of exactly times |
Table D.1 : Effect of multiplying vectors and matrices by the duplication matrix and its transpose.
| Symmetry | Addition | Duplication | Scaling |
|---|---|---|---|
| Permutation | ✗ | ✗ | ✗ |
| Positive scaling | ✗ | ✗ | ✓ |
| Sign flip | ✗ | ✗ | ✓ |
| Zero-neuron group | ✓ | ✓ | ✗ |
| Duplicate-neuron group | ✗ | ✓ | ✗ |
| Constant neuron | ✓ | ✗ | ✗ |
Table D.2 : Decomposition of parameter symmetries into primitive feature transformations. Checkmarks indicate the primitives appearing in the displayed decompositions, possibly acting trivially when duplication counts equal . For width-changing operations, the table describes the addition or replacement direction; reverse operations use the corresponding inverse primitives.
Figure G.1 : Boundary geometry of the hidden neurons across converged networks. Ticks indicate the canonical values of the six solution types listed in Table G.1 . Histogram counts are shown on a logarithmic scale for the signed distances (left) and a linear scale for the angles (right). Of the neurons, have signed distances distinct from both dominant values, while two have angles distinct from all four dominant values.
Figure G.2 : Cluster medoids and maximally dissimilar members. The leftmost panel in each row shows the cluster medoid; moving right, each panel shows the member farthest from those already selected. Because training produces substantially larger readout weights than the closed-form representatives in Table G.1 , the readouts of each network are rescaled for visualization to match the logit scale of Equation 339 .
| # | Family | ||||
|---|---|---|---|---|---|
| 1 | Diagonal | ||||
| 2 | Diagonal | ||||
| 3 | Diagonal | ||||
| 4 | Anti-diagonal | ||||
| 5 | Anti-diagonal | ||||
| 6 | Anti-diagonal |
Table G.1 : Closed-form representatives of the six ReLU solution types for exclusive or identified through the gradient-descent sweep. Each row specifies the angles , signed distances , readout weights , and output bias . All solutions listed here use unit gain.
Figure G.3 : The six closed-form ReLU solutions of Table G.1 at unit gain. Solutions 1–3 (top row) form the diagonal family, while solutions 4–6 (bottom row) form the anti-diagonal family. Within each family, the activation boundaries share a common orientation, while the solutions differ in their signed distances and readout weights . Each panel shows the logit of Equation 333 as a heatmap, the activation boundaries as dashed lines with unit normals , and the four exclusive or data points. All six networks assign every data point a logit of magnitude .