Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Yossi Arjevani

Figure 1: Hessian spectra of a three-layer network on a mixture of

k=15

Gaussians with random centers: at a global minimum of the original setting (orange) and at a reference configuration (blue), close to the original yet highly symmetric. At the symmetric reference. The spectral structure falls into place. Symmetry forces the large kernel and the multiplicities in the table, and identifies invariant subspaces of directions weakly aligned with the data. For the family of global minima considered here, this results in only

2k-1

eigenvalues remaining bounded away from zero. Symmetry breaking. Returning to the original configuration breaks exact symmetry in a controlled manner. The resulting clusters arise as perturbative remnants of the degenerate reference eigenvalues (left). Spectral multiplets. A reference eigenvalue of multiplicity

k-1

splits into

k-1

simple outliers (right), providing a mechanism for the empirically observed scaling of the outlier count with the number of classes Sagun et al. (2016) ; Sagun et al. (2018) . With the parameter-space geometry taken into account, the spectra of a network trained on MNIST reveal the same organizing principle ( Figure 8 ).

Figure 2: Hessian spectra (left and center) for the three-layer networks

N^{3}

and

N^{3,\scalebox{0.53}{

\varnothing

}}

(the latter obtained by removing the final-layer bias), and Hessian structure for the bias-free

N^{3,\circ}

(right), on the two-Gaussian mixture in Learning Problem 1 . At trained models, their Hessian spectra (left) exhibit a large near-zero bulk and a few outliers, consistent with Sagun et al. (2016) . The spectrum at a single random draw, likewise examined in that work, is compared with that of the averaged Hessian (center). At the symmetric reference. Despite weight symmetries being generically trivial, the argument illustrated in ( 1.22 ) yields a

D_{4}\times C_{2}

Hessian invariance. This invariance, richer than its

k=2

counterpart in Figure 1 , forces all nonzero eigenvalues to have multiplicity two (left, blue). The large kernels reflect symmetry-induced parameter degeneracy. Symmetry breaking. Restoring the final-layer bias breaks the symmetry and splits two reference doublets, with the leading one producing two isolated outliers (left, orange), while retaining the large reference kernel. This explains the reported bulk-outlier pattern. The detailed spectral description, including the full Cauchy interlacing argument, is given in Table 3 . Random initialization. We show that the averaged Hessian has a rich invariance that forces a small positive eigenvalue of multiplicity quadratic in

h

(center, blue), which in turn yields, with probability tending to one as

h\to\infty

, a near-zero bulk containing a

1-o(1)

fraction of the spectrum at a random draw (center, orange).

Figure 3: Weights and Hessian spectra of

N^{3}

with

d=h\geq 4

, trained on Learning Problem 2 . Top. At the trained model, the Hessian has a rich invariance structure and nontrivial eigenvalue multiplicities—despite the learned weights having only trivial symmetry under the standard rescaling and permutation symmetries. Our framework instead traces these invariances to symmetries of the differential of

N^{3}

. The same obstruction appears for Learning Problem 1 and for general

L

-layer fully connected networks in Arjevani and Field (2019) . Bottom. Nontrivial multiplicities also occur in the layerwise Hessian spectra. The limitation of weight symmetries is even more pronounced layerwise, since transformations coupling adjacent layers need not survive restriction to a single layer. Nevertheless, the framework captures the layerwise Hessian invariances and the resulting multiplicities recorded in Table 1 .

Parameter set	Dimension of the Hessian kernel	Multiplicity structure
A.e. point with $W_{2},\mathbf{b}_{2}>0$ , critical or not	$d(d-1)$ (all transverse)	$d(d-1)\|_{0},1^{[d^{2}+4d+1]}$
Global minimizers	$d(d+2)$ ( $d^{2}$ transverse)	$d(d+2)\|_{0},\;\frac{(d-1)(d-2)}{2},\;\frac{d(d-3)}{2},(d-1)^{[4]},\;1^{[4]}$
		$W_{1}$ ${\frac{(d-1)(d-2)}{2},\frac{d(d-3)}{2},(d-1)^{[3]},1^{[2]}}$ $\mathbf{b}_{1}$ ${d-1,1}$ $W_{2}$ ${d(d-1)\|_{0},d-1,1}$ $(\mathbf{b}_{2},W_{3})$ ${(d-1)\|_{0},d-1,1^{[2]}}$ $\mathbf{b}_{3}$ ${1}$

Table 1: Lower bounds on Hessian kernel dimensions for

N^{3}

under Learning Problem 2 , together with multiplicity structures for

d=h\geq 4

. Top. Throughout an open region, points are typically noncritical, yet the Hessian kernel is large and generically consists only of directions transverse to the orbits of global symmetries (see Table 13 ). The proof uses generalized functions. Replacing ReLU by GELU removes these transverse directions from the kernel (see Figure 11 ). Bottom. At the global minimizers in ( 4.136 ), the kernel dimension is larger and, under an invariant inner product, the Hessian has large eigenvalue multiplicities, established in Theorem 30 , as do the corresponding layerwise restrictions (see Table 14 ).

Invariance analysis	Isotypic decomposition on $(\ker K_{\infty})^{\perp}$	Positive eigenvalue decay
Output invariance only ${{\text{O}(2)}}$	$\mathrm{std}_{{{\text{O}(2)}}}^{\oplus\infty}$	$\lambda_{j}=\Theta(j^{-2})$
Full composite invariance ${{\text{O}(2)}}\times C_{2}^{2}$	$\displaystyle\bigoplus_{\chi\in\widehat{C_{2}^{2}}}(\mathrm{std}_{{{\text{O}(2)}}}\otimes\chi)^{\oplus\infty}$	$\lambda_{\chi,j}=\Theta(j^{-2})$

Table 2: Spectral analysis of the nonzero bounded operator

K_{\infty}

associated with

d_{\infty}^{\mathrm{NTK}}

. Incorporating distributional symmetries adds the

C_{2}^{2}

input factor, refining the isotypic decomposition into four character sectors. Here

\widehat{C_{2}^{2}}

is the set of four characters of

C_{2}^{2}

, and

\mathrm{std}_{{{\text{O}(2)}}}

is the standard two-dimensional representation. Additional kernel analysis for the two-Gaussian law establishes

\Theta(j^{-2})

decay of the positive eigenvalues in each sector and the multiplicity structure

\{\!\{\aleph_{0}|_{0},\;2|_{\lambda_{\max}},\;2^{[\aleph_{0}]}\}\!\}

.

Figure 4: Schematic illustration of the SB mechanisms developed in this work. Exact symmetry. A symmetric reference configuration is constructed by modifying the setting. There, structural symmetries identified directly from derivatives of the pointwise loss combine with symmetries of the data distribution to produce Hessian invariances, even when weight symmetries are trivial. The large Hessian kernel, the high eigenvalue multiplicities, and the different eigenvalue scales now become transparent through the resulting rich symmetry structure. Local SB. The spectra of the original models unfold as controlled SB splits these degeneracies into the observed hierarchy of clusters and outliers. Nonlocal SB. Depending on the initialization and algorithm, optimization may select critical points of lower symmetry, producing a more refined spectrum.

Table 3: Hessian spectra on

\mathcal{T}_{\star}

along nested subarchitectures of

N^{3}

, viewed as subspaces in its parameter space. In the symmetric cases,

D_{4}\times C_{2}

-invariance forces the nonzero eigenvalues to have multiplicity two. Passing to

N^{3}

splits the reference eigenvalues, with the splitting constrained by Cauchy interlacing. This multiplicity structure is observed in Figure 2 . Successive applications of Cauchy interlacing along the full chain give a sharper localization of the

N^{3}

eigenvalues, now relative to the spectra of the subarchitectures.

Tangency arc	Source			Target
$\left[\begin{smallmatrix}0_{2,d}\\ t\mathbf{1}_{d-2}\mathbf{1}_{d}^{\top}\end{smallmatrix}\right]$	$0$	Saddle	$S_{d}\times S_{d}$	$\left[\begin{smallmatrix}0_{2,d}\\ \Theta\bigl(d^{-1}\bigr)\mathbf{1}_{d-2}\mathbf{1}_{d}^{\top}\end{smallmatrix}\right]$	Saddle	$(S_{2}\times S_{d-2})\times S_{d}$
$tI_{d}$	$0$	Saddle	$S_{d}\times S_{d}$	$I_{d}$	Global minimum	$\Delta S_{d}$
$tI_{d-1}\oplus[0]$	$0$	Saddle	$S_{d}\times S_{d}$	$I_{d-1}\oplus[0]$	Saddle	$\Delta S_{d-1}$
$I_{d-1}\oplus[t]$	$I_{d-1}\oplus[0]$	Saddle	$\Delta S_{d-1}$	$I_{d}$	Global minimum	$\Delta S_{d}$
$I_{d-2}\oplus\begin{bmatrix}a(t)&0\\ t&0\end{bmatrix},$ $~a(t)=\sum_{m=0}^{\infty}\binom{1/3}{m}(-1)^{m}t^{3m}$	$I_{d-1}\oplus[0]$	Saddle	$\Delta S_{d-1}$	$I_{d-2}\oplus\begin{bmatrix}0&0\\ 1&0\end{bmatrix}$	Saddle	$\Delta S_{d-1}$

Table 4: Tangency arcs for the symmetric tensor decomposition problem ( 1.45 ). Isotropy groups are listed up to conjugacy. The examples illustrate recursive application of the construction, leading to critical points with maximal and submaximal isotropy groups for the

S_{d}^{2}

-action. These are related, via Goursat’s lemma, to the corresponding subgroup structures in the

S_{d}

factors, as illustrated in Figure 5 .

Figure 6: Tangency sets of

h

in ( 1.46 ) relative to different critical points. The tangency arcs may break the symmetry of the reference critical point. Any critical point lying on an arc has at least the symmetry of that arc.

Figure 7: Weight matrices illustrating three forms of SB. Top. Weights of a trained

N^{3,\circ}

network with

d=k=7

and

h=5

, learned on the mixture of seven Gaussians in Learning Problem 1 . The observed symmetry breaking is from

S_{7}\times S_{5}^{2}

to approximately

\Delta(S_{4}\times S_{1})\times S_{2}

. Middle. A point on a tangency arc emanating from

W^{\star}

in ( 1.43 ), with approximate isotropy

\Delta(C_{3}\times S_{2})

(see Remark 9 ). Bottom. Weight matrices in the fixed-point space of

D_{5}\leq\operatorname{AGL}(1,5)\leq S_{5}

, illustrating forced SB. Although

D_{5}

is not maximal as a subgroup of

S_{5}

, it is a maximal isotropy group for the diagonal action on

M(5,5)

. Indeed,

\operatorname{AGL}(1,5)

is 2-transitive, so its fixed-point space in

M(5,5)

coincides with the

S_{5}

-fixed-point space. Similar phenomena also occur on parameter manifolds, as illustrated in Section 4.3 for parameters constrained to lie on a sphere.

Figure 8: When Hessian invariances are compatible with the parameter-space geometry, they force large eigenvalue multiplicities. Here we follow the empirical spectra for

N^{3}

trained on MNIST as the parameter-space inner product varies from an

S_{10}

-invariant one to Euclidean. Top. Approximate invariance produces narrow bands at the invariant endpoint. Bottom. The action is incompatible with the Euclidean parameter-space geometry, and the bands refract into spectral clusters ( Remark 15 ). The left and right panels concern the suffixes

(W_{3},\mathbf{b}_{3})

and

(W_{3},\mathbf{b}_{3},W_{2},\mathbf{b}_{2})

, respectively. The approximate invariance arises from a phenomenon we refer to as neural registration , whereby approximate symmetries emerge in hidden feature–label laws. The construction and analysis are given in Section 2.5 .

Spectrum on the parameter quotient
scale of eigenvalues	multiplicities forced by symmetry	total out of $n+8$
$\Theta(1)$	one eigenvalue of multiplicity $n-9$ two eigenvalues of multiplicity $2$ four simple eigenvalues	$n-1$
$\Theta(d^{2})$	two eigenvalues of multiplicity $2$ four simple eigenvalues	$8$
$\Theta(d^{4})$	one simple eigenvalue	$1$
The kernel in parameter space has dimension $p-n-8$ .

Table 5: Asymptotic spectral structure on the parameter quotient.

Figure 9: Distinct eigenvalue branches of the Hessian on the parameter quotient at a global minimum of a thin AlexNet, shown for

d=10

,

n=d^{2}=100

, and

s=1

. The parameter

t\in[0,1]

rescales

\delta K_{1}\mapsto t\,\delta K_{1}

. At each

t

, the spectrum is computed modulo

\operatorname{span}\{(K_{\delta},-t\mathbf{1}_{n}^{\top})\}

. At

t=0

, the positive spectrum is

\lambda_{\mathrm{std}}^{[n-1]}\oplus\lambda_{\mathrm{triv}}

, with

\lambda_{\mathrm{std}}=1-\frac{1}{\pi}

and

\lambda_{\mathrm{triv}}=1-\frac{1}{\pi}+\frac{n}{\pi}

. Eight branches emanate from zero, eight split from

\lambda_{\mathrm{std}}

, and the branch from

\lambda_{\mathrm{triv}}

evolves into the largest eigenvalue, while

n-9=91

eigenvalues remain exactly pinned at

\lambda_{\mathrm{std}}

. The right-hand region shows spectral refraction as the

D_{4}

-compatible quotient metric is deformed to the parameter-induced metric. Hollow circles mark the four

D_{4}

-forced multiplicity-two eigenvalues. Their subsequent splitting is below the scale in the figure.

Depth	Multiplicity pattern	$\mathfrak{s}_{(n)}$	$\mathfrak{s}_{(n-1,1)}$	$\dim\ker d_{W}^{2}{\mathcal{L}}$
$L=1$	$(2n^{2}-1)\|_{0},\ (n-1),\ 1^{[2]}$	$\Theta(n^{5}),\ \Theta(1)$	$\Theta(n)$	$2n^{2}-1$
$L=2$	$(4n^{2}-n)\|_{0},\ (n-1)^{[2]},\ 1^{[2]}$	$\Theta(n^{7}),\ \Theta(n^{2})$	$\Theta(n^{3}),\ \Theta(n^{-1})$	$4n^{2}-n$
$L\geq 3$	$(2Ln^{2}-n)\|_{0},\ (n-1)^{[2]},\ 1^{[2]}$	$\Theta(n^{2L+3}),\ \Theta(n^{2L-2})$	$\Theta(n^{2L-1}),\ \Theta(n^{2L-6})$	$2Ln^{2}-n$

Table 6: Asymptotic spectrum of the Euclidean Hessian at

W^{\star}

for fixed

\rho,\beta\in(0,1)

and fixed depth

L

as

n\to\infty

. Each

\mathfrak{s}_{(n)}

branch is simple, while each

\mathfrak{s}_{(n-1,1)}

branch has multiplicity

n-1

. The reference point

W^{\star}

belongs to a larger local family of global minima, similarly to the AlexNet setting above.

#	Symmetry type	Definition	Isomorphism class	Invariance of ${\mathcal{L}}^{\mathrm{UFM}}$
$\mathsf{a}_{0}^{\mathrm{UFM}}(M_{0})$	Architectural	$(FM_{0}^{-1},M_{0}\mathbf{x})$	${\rm GL}_{m}$	No
$\mathsf{a}_{1}^{\mathrm{UFM}}(M_{1})$	Architectural	$(WM_{1}^{-1},M_{1}F)$	${\rm GL}_{h}$	Yes
$\mathsf{s}^{\mathrm{UFM}}(U)$	Structural	$(UW,U\mathbf{y})$	$O(k)$	No
$\mathsf{d}_{\mathrm{within}}^{\mathrm{UFM}}(\boldsymbol{\tau})$	Distributional	$P_{\boldsymbol{\tau}}\mathbf{x}$	$S_{q}^{k}$	No
$\mathsf{d}_{\mathrm{between}}^{\mathrm{UFM}}(\pi)$	Distributional	$(P_{\pi}^{(m)}\mathbf{x},P_{\pi}\mathbf{y})$	$S_{k}$	No
$\mathsf{c}^{\mathrm{UFM}}$	Composite	$(P_{\pi}WM_{1}^{-1},M_{1}FP_{\omega}^{\top})$	${\rm GL}_{h}\times(S_{q}\wr S_{k})$	Yes

Table 7: Architectural, structural, and distributional symmetries of the UFM and the resulting composite invariance. Here

M_{0}\in{\rm GL}_{m}

,

M_{1}\in{\rm GL}_{h}

, and

U\in O(k)

.

Space	Isotypic decomposition	Multiplicity structure
$T_{(W,F)}\Theta$	$\begin{aligned} &\bigl(h(k+m)-km+k\bigr)\mathfrak{t}\oplus k\mathfrak{s}_{k-1,1}\\ &\oplus k\operatorname{Ind}_{S_{q}^{k}\rtimes S_{k-1}}^{S_{q}\wr S_{k}}\mathfrak{s}_{q-1,1}\end{aligned}$	$\begin{aligned} &\bigl(h(k+m)-km\bigr)\|_{0},\\ &1^{[k]},\hskip 9.24994pt(k-1)^{[k]},\hskip 9.24994pt(m-k)^{[k]}\end{aligned}$
$\dfrac{T_{(W,F)}\Theta}{\ker d^{2}{\mathcal{L}}^{\mathrm{UFM}}}$	$k\mathfrak{t}\oplus k\mathfrak{s}_{k-1,1}\oplus k\operatorname{Ind}_{S_{q}^{k}\rtimes S_{k-1}}^{S_{q}\wr S_{k}}\mathfrak{s}_{q-1,1}$	$1^{[k]},\hskip 9.24994pt(k-1)^{[k]},(m-k)^{[k]}$
Quotient under ${{\text{O}(k-1)}}\times(S_{q}\wr S_{k})$	$\begin{aligned} &(\mathfrak{t}\oplus\mathrm{std}_{O(k-1)})\\ &\hskip 18.49988pt\otimes\Bigl(\mathfrak{t}\oplus\mathfrak{s}_{k-1,1}\oplus\operatorname{Ind}_{S_{q}^{k}\rtimes S_{k-1}}^{S_{q}\wr S_{k}}\mathfrak{s}_{q-1,1}\Bigr)\end{aligned}$	$\begin{aligned} &1,\hskip 9.24994pt(k-1)^{[2]},\hskip 9.24994pt(k-1)^{2},\\ &m-k,\hskip 9.24994pt(k-1)(m-k)\end{aligned}$

Table 8: Isotypic decompositions at global minima under the full composite

S_{q}\wr S_{k}

action and, in the third row where ( 2.103 ) holds, its

O(k-1)

enlargement. Here

\mathrm{std}_{O(k-1)}

is the standard representation of

O(k-1)

, and

\mathfrak{s}_{\lambda}

denotes irreducible representations of symmetric groups as in Section 3.2 . The first row uses a lift acting trivially on the Hessian kernel. In the induced representations,

\mathfrak{s}_{q-1,1}

acts through the first

S_{q}

factor, with the other

S_{q}

factors and

S_{k-1}

acting trivially. The resulting multiplicity structures use the notation of Definition 7 . When the subgroup in ( 2.100 ) equals

S_{q}\wr S_{k}

, the stated multiplicities apply to the Euclidean Hessian; otherwise, spectral refraction may split them.

Figure 10: Registration and neural collapse across hidden layers. Red shows

\overline{\operatorname{NC1}}

from ( 2.108 ), and green the normalized empirical registration diagnostic. The left uses

6{,}313

MNIST images per class for a trained ResNet-

18

(indices denote feature cuts before successive parameter blocks). Both diagnostics decrease toward its final blocks. The right uses

10{,}000

independent samples per class from

\mathbf{x}\mid\mathbf{y}=\mathbf{e}_{c}\sim\mathcal{N}(\mathbf{e}_{c},I_{5})

at the explicit

N^{5}

global minimizer. Registration improves while the NC1 diagnostic remains high.

#	Symmetry type	Action	Invariance of ${\mathcal{L}}$
$\mathsf{a}_{0}(M)$	Architectural	$(W_{1},\mathbf{x})\mapsto(W_{1}M^{-1},M\mathbf{x})$	No
$\mathsf{a}^{\pi}_{i}(P_{\pi})$	Architectural	$(W_{i},W_{i+1})\mapsto(P_{\pi}W_{i},W_{i+1}P_{\pi}^{-1})$	Yes
$\mathsf{a}^{\lambda}_{i}(\boldsymbol{\lambda})$	Architectural	$(W_{i},W_{i+1})\mapsto(\mathrm{diag}_{\boldsymbol{\lambda}}W_{i},W_{i+1}\mathrm{diag}_{\boldsymbol{\lambda}}^{-1})$	Yes
$\mathsf{s}^{\mathtt{ce}}_{1}(\mathbf{v})$	Structural	$W_{3}\mapsto W_{3}+\mathbf{1}\mathbf{v}^{\top}$	Yes
$\mathsf{s}^{\mathtt{ce}}_{2}$	Structural	$(W_{3},\mathbf{y})\mapsto(P_{(1,2)}W_{3},P_{(1,2)}\mathbf{y})$	No
$\mathsf{s}^{\mathtt{ce}}_{3}$	Structural	$W_{3}\mapsto-P_{(1,2)}W_{3}$	Yes

Table 9: Examples of architectural symmetries of

N^{3,\circ}

, structural symmetries under the cross-entropy loss, and distributional symmetries of the mixture of two Gaussians in Learning Problem 1 , together with the resulting composite invariances. The final column indicates whether each symmetry is an invariance of the expected loss

{\mathcal{L}}

, as determined by Proposition 13 . Here

M\in{\rm GL}_{d}

,

P_{\pi}

denotes the permutation matrix associated with

\pi

,

\boldsymbol{\lambda}\in\mathbb{R}_{>0}^{h}

, and

\mathbf{v}\in\mathbb{R}^{h}

.

	$\mathbf{0}$	$\mathbf{e}_{i}$	Generic point
$\operatorname{Stab}_{S_{n}}(\mathbf{x})$	$S_{n}$	$S_{n-1}$	$S_{1}^{n}$
Stabilizer multiplicity structure	$1,\ n-1$	$1^{[2]},\ n-2$	$1^{[n]}$
Hessian multiplicity structure	$1,\ n-1$	$1,\ n-1$	$1,\ n-1$

Table 11: Point stabilizers and multiplicity structure ( Definition 7 ) for the natural action of

S_{n}

on

\mathbb{R}^{n}

. The quadratic example

q

in ( 3.112 ) illustrates one limitation of the point-stabilizer approach: its Hessian is constant, so the same spectral structure persists even at points with trivial stabilizer (see also Section 2.6 ). In the ReLU settings considered here, the Hessian need not be constant, yet weight stabilizers fail to recover the rich invariance structure that forces its nontrivial eigenvalue multiplicities.

	Notation	Definition
Local automorphism group	$\mathrm{Aut}_{W}\Theta$	$\mathrm{Aut}(T_{W}\Theta)$
Weight-component projection	$\pi_{W}$	Projection onto $\mathrm{Aut}_{W}\Theta$
Joint local automorphism group	$\mathrm{Aut}_{W}\mathbb{A}$	$\mathrm{Aut}(Z)\times\mathrm{Aut}_{W}\Theta$
Distribution group	$\mathsf{D}(\mu)$	$\operatorname{Stab}_{\mathrm{Aut}(Z)}{\mu}$
Initialization group	$\mathsf{I}(\nu)$	$\operatorname{Stab}_{\mathrm{Aut}(\Theta)}{\nu}$
Symmetry groups of pointwise quantities

Table 12: Definitions of local automorphism groups and of symmetry and invariance groups for higher derivatives and the GN form, extending Definition 12 and Definition 14 . At fixed

W

, each classical pointwise derivative is viewed as a partially defined map on

Z

with values in symmetric multilinear forms, and its stabilizer is taken under the induced action. For generalized derivatives, the corresponding groups are defined on open sets; see Section 5.2 . Here

\mathrm{Aut}_{W}(\Theta)=\mathrm{Aut}(T_{W}\Theta)

, whereas

\mathrm{Aut}(\Theta)_{W}

is the stabilizer of

W

in

\mathrm{Aut}(\Theta)

. Without a subscript

W

,

\mathsf{L}^{[r]}

,

\mathsf{S}^{[r]}

, and

\mathsf{S}^{[\mathrm{GN}]}

denote the corresponding global stabilizers under the induced action

g\cdot(\mathbf{z},W,\delta W)=(g\mathbf{z},gW,g\delta W)

. In the barred rows,

\pi_{W}

extracts the induced linear action on perturbations, while the intersections impose compatibility with

\nu

and, in the composite case, with

\mu

. These groups preserve the corresponding initialization averages.

Figure 11: Left. For the network

N^{2\nicefrac{{1}}{{2}},\circ}

in ( 1.2 ), consider any point

W

at which all second-layer weights are positive. One finds that the

\delta W_{2}

-component of

\nabla^{2}{\mathcal{L}}(W)\delta W

always has identical rows. This reflects a rich Hessian invariance group ( Lemma 28 ), present despite

W

generically having trivial symmetry. Middle. Under the conditions of Theorem 27 , the Hessian of

N^{3}

( 1.1 ) with ReLU has

\Theta(h^{2})

zero eigenvalues—independently of the data distribution. The result extends directly to the setting shown here, namely

d=k=2

,

h=5

for

\mathcal{N}_{\pm}

in Learning Problem 1 . Right. If ReLU is replaced by GELU, this result no longer holds, but Theorem 20 , which holds regardless of the activation, still applies and guarantees

\Omega(h)

zero eigenvalues. The distinct “knee” locations in spectra evaluated at the same weights are numerically consistent with the corresponding lower bounds.

Type	Space	Dimension	Description
Transverse	$({\mathrm{std}}_{O_{W_{3}}},\mathfrak{t})$ -isotypic component	$h(h-1)$	$\delta W_{2}$ with columns orthogonal to $W_{3}$
Transverse	$(\mathfrak{t},{\mathrm{std}}_{O_{W_{3}}})$ -isotypic component	$h-1$	$\delta\mathbf{b}_{2}$ orthogonal to $W_{3}^{\top}$
Transverse	in the $(\mathfrak{t},\mathfrak{t})$ -isotypic component	$1$	$\delta\mathbf{b}_{2}\in\mathrm{Span}(W_{3}^{\top})$ and $W_{3}\,\delta\mathbf{b}_{2}=-\,\delta\mathbf{b}_{3}$
Orbit	$T_{W}(\Lambda(N^{3})W)$	$2h$	$\delta W=(AW_{1},A\mathbf{b}_{1},BW_{2}-W_{2}A,B\mathbf{b}_{2},-W_{3}B,0)$
Total		$h(h+2)$

Table 13: Subspaces of

\ker d_{W}N^{3}

yielding the GN bound in Theorem 27 , with

A,B

diagonal. Orbit flat directions contribute only

\Theta(h)

zero eigenvalues, compared with

\Theta(h^{2})

from transverse directions. The two nontrivial isotypic components contain all but one transverse direction and, through Proposition 19 , yield the Hessian kernel bound.

	$T_{W_{1}}\Theta$	$T_{\mathbf{b}_{1}}\Theta$	$T_{W_{2}}\Theta$	$T_{\mathbf{b}_{2},W_{3}}\Theta$	$T_{\mathbf{b}_{3}}\Theta$
$\delta\mathsf{s}_{1}$	trivial	trivial	$M_{1}\delta W_{2}$	$\bigl(M_{2}\delta\mathbf{b}_{2},\delta W_{3}\bigr)$	trivial
$\delta\mathsf{c}_{1}$	$B_{\pi}\delta W_{1}P_{\pi}^{\top}$	$B_{\pi}\delta\mathbf{b}_{1}$	$\delta W_{2}C_{\pi}$	$\Phi_{\pi}$	trivial
$\langle\delta\mathsf{s}_{1},\delta\mathsf{c}_{1}\rangle\!\mid_{E}$	$S_{d}$	$S_{d}$	$\operatorname{Stab}_{O(M(1,d))}(W_{3})\times S_{d}$	$\operatorname{Stab}_{O(M(1,d))}(W_{3})\times S_{d}$	trivial
Isotypic dec.	$2\mathfrak{t}\oplus 3\mathfrak{s}_{d-1,1}$ $\oplus\,\mathfrak{s}_{d-2,2}\oplus\,\mathfrak{s}_{d-2,1,1}$	$\mathfrak{t}\oplus\,\mathfrak{s}_{d-1,1}$	$(\mathfrak{t},\mathfrak{t})\oplus(\mathfrak{t},\mathfrak{s}_{d-1,1})$ $\oplus\,({\mathrm{std}}_{O_{W_{3}}},\mathfrak{t})\oplus\,({\mathrm{std}}_{O_{W_{3}}},\mathfrak{s}_{d-1,1})$	$2(\mathfrak{t},\mathfrak{t})\oplus({\mathrm{std}}_{O_{W_{3}}},\mathfrak{t})$ $\oplus\,(\mathfrak{t},\mathfrak{s}_{d-1,1})$	$\mathfrak{t}$
Multi. structure	$\{\!\{1^{[2]},(d-1)^{[3]},$ $\frac{d(d-3)}{2},\frac{(d-1)(d-2)}{2}\}\!\}$	$\{\!\{1,d-1\}\!\}$	$\{\!\{1,(d-1)^{[2]},(d-1)^{2}\}\!\}$	$\{\!\{1^{[2]},(d-1)^{[2]}\}\!\}$	$\{\!\{1\}\!\}$

Table 14: Restrictions of

\delta\mathsf{s}_{1}

and

\delta\mathsf{c}_{1}

to the relevant parameter subspaces, together with the corresponding restricted invariance groups, isotypic decompositions, and multiplicity structures. Here

B_{\pi}\coloneqq W_{1}P_{\pi}W_{1}^{-1}

,

C_{\pi}\coloneqq W_{1}P_{\pi}^{\top}W_{1}^{-1}

,

R_{\pi}\coloneqq W_{2}W_{1}P_{\pi}^{\top}W_{1}^{-1}W_{2}^{-1}

,

A_{\pi}\coloneqq W_{2}W_{1}P_{\pi}W_{1}^{-1}W_{2}^{-1}

, and

\Phi_{\pi}(\delta\mathbf{b}_{2},\delta W_{3})\coloneqq(\delta\mathbf{b}_{2}-\delta W_{3}A_{\pi}^{-1}(I-A_{\pi})\mathbf{b}_{2}\frac{W_{3}^{\top}}{W_{3}W_{3}^{\top}},\delta W_{3}R_{\pi})

. The two

\operatorname{Stab}_{O(M(1,d))}(W_{3})

factors in

\delta\mathsf{s}_{1}(M_{1},M_{2})

correspond, respectively, to

M_{1}

and

M_{2}

. Trivial factors are suppressed in the displayed product groups and isotypic decompositions where no confusion results. Since

\delta\mathsf{c}_{1}

maps a

W_{3}

variation to one with a

\mathbf{b}_{2}

component, the natural invariant block is

T_{\mathbf{b}_{2},W_{3}}\Theta

rather than

T_{W_{3}}\Theta

.

Figure 12: Weights

W_{1}

and corresponding spectra of the Hessian of

{\mathcal{L}}|_{\mathbb{S}(\rho)}

for

N^{1\nicefrac{{1}}{{2}},\circ}

. The top row corresponds to the global minimum

W_{1}=I_{d}

, whose observed eigenvalue multiplicities agree with those predicted by ( 4.148 ), up to the numerical coalescence of nearby eigenvalues. The bottom row shows a trained SB point with stabilizer

\Delta(S_{d-2}\times S_{2})

.

	Points	$\dim\mathrm{Flat}$	$\mathrm{codim}\,\mathrm{Flat}$	$\mathsf{c}^{\mathsf{s}}$	$\mathsf{c}^{\mathsf{r}}$	$\delta\mathsf{s}_{2}$	isotypic decomposition of $T_{W}\Theta/\mathrm{Flat}$
$N^{3,\circ}$	$\mathcal{T}$	$8$	$4$	$T_{W}\Theta$	$C^{(-1)}$	$T_{W}\Theta$	$\bigl(\mathfrak{s}_{(2)}\oplus\mathfrak{s}_{(1,1)}\bigr)_{\mathsf{c}^{\mathsf{s}}}\otimes\bigl(\mathfrak{s}_{(2)}\oplus\mathfrak{s}_{(1,1)}\bigr)_{\delta\mathsf{s}_{2}}$
$N^{3,\circ}$	$\mathcal{T}_{\star}$	$8$	$4$	$T_{W}\Theta$	$T_{W}\Theta$	$T_{W}\Theta$	$(\mathfrak{s}_{(2)}\oplus\mathfrak{s}_{(1,1)})_{\mathsf{c}^{\mathsf{s}}}\otimes\mathrm{Std}_{D_{4}}$
$N^{3,\scalebox{0.53}{$ \varnothing $}}$	$\mathcal{T}_{\star}$	$10$	$6$	$T_{W}\Theta$	$T_{W}\Theta$	$T_{W}\Theta$	$\left(\mathfrak{s}_{(1,1)}\oplus 2\mathfrak{s}_{(2)}\right)_{\mathsf{c}^{\mathsf{s}}}\otimes\mathrm{Std}_{D_{4}}$
$N^{3,\mathord{\vbox{\hbox{\hbox to2.58pt{\vbox to2.58pt{\pgfpicture\makeatletter\hbox{\hskip 1.28802pt\lower-1.28802pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {\lx@inpgf@ignorespaces}\lx@inpgf@ignorespaces{\lx@inpgf@ignorespaces}\lx@inpgf@ignorespaces\lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.14pt} \lx@inpgf@ignorespaces\lxSVG@roundcap\lxSVG@begingroup@{stroke-linecap=round} \lx@inpgf@ignorespaces{ {}{{}}{}{{{}}{\lx@inpgf@ignorespaces}{}{}{}{}{}{}{}}{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 0 0 M 1.45 0 C 1.45 0.8 0.8 1.45 0 1.45 C -0.8 1.45 -1.45 0.8 -1.45 0 C -1.45 -0.8 -0.8 -1.45 0 -1.45 C 0.8 -1.45 1.45 -0.8 1.45 0 Z M 0 0}{fill:none} \lx@inpgf@ignorespaces {}{{}}{} {}{}{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M -1.69 -1.22 L 1.22 1.69}{fill:none} \lx@inpgf@ignorespaces {}{{}}{} {}{}{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M -1.22 -1.69 L 1.69 1.22}{fill:none} \lx@inpgf@ignorespaces } \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}}}}$	$\mathcal{T}_{\star}$	$11$	$6$	$T_{W}\Theta$	$T_{W}\Theta$	$T_{W}\Theta$	$\left(\mathfrak{s}_{(1,1)}\oplus 2\mathfrak{s}_{(2)}\right)_{\mathsf{c}^{\mathsf{s}}}\otimes\mathrm{Std}_{D_{4}}$
$N^{3}$	$\mathcal{T}_{\star}$	$12$	$6$	$T_{W}\Theta$	$T_{W}\Theta$	$\delta\mathbf{b}_{3}\\|\mathbf{1}$	$\left(\mathfrak{s}_{(1,1)}\oplus 2\mathfrak{s}_{(2)}\right)_{\mathsf{c}^{\mathsf{s}}}\otimes\left(\mathfrak{s}_{(2)}\oplus\mathfrak{s}_{(1,1)}\right)_{\mathsf{c}^{\mathsf{r}}}$

Table 15: Isotypic decompositions of

T_{W}\Theta/\mathrm{Flat}

for the indicated models under Learning Problem 1 with

k=2

, where

\mathrm{Flat}

is the space of flat directions. The table also gives the dimension and codimension of

\mathrm{Flat}

and the subspaces on which

\mathsf{c}^{\mathsf{s}}

,

\mathsf{c}^{\mathsf{r}}

, and

\delta\mathsf{s}_{2}

act.

space	isotypic decomposition
$T_{W^{(k)}}\Theta$	$16\,\mathfrak{s}_{(k)}\oplus 24\,\mathfrak{s}_{(k-1,1)}\oplus 8\,\mathfrak{s}_{(k-2,2)}\oplus 8\,\mathfrak{s}_{(k-2,1,1)}$
$T_{W^{(k)}}\Theta/\ker d_{W}^{[\mathrm{GN}]}$	$4\,\mathfrak{s}_{(k)}\oplus 8\,\mathfrak{s}_{(k-1,1)}\oplus 4\,\mathfrak{s}_{(k-2,2)}\oplus 4\,\mathfrak{s}_{(k-2,1,1)}$
$\mathfrak{Z}_{0}/(\mathfrak{Z}_{0}\cap\ker d_{W}^{[\mathrm{GN}]})$	$4\,\mathfrak{s}_{(k)}\oplus 6\,\mathfrak{s}_{(k-1,1)}\oplus 4\,\mathfrak{s}_{(k-2,2)}\oplus 4\,\mathfrak{s}_{(k-2,1,1)}$
$T_{W^{(k)}}\Theta/(\mathfrak{Z}_{0}+\ker d_{W}^{[\mathrm{GN}]})$	$2\,\mathfrak{s}_{(k-1,1)}$

Table 16: Isotypic decompositions for the

k

-class Gaussian mixture with

k\geq 4

.

Name	Notation	Definition
Lebesgue space	$L^{p}_{\mathrm{loc}}(\Omega;Y)$ , $1\leq p\leq\infty$	$\left\{u:\Omega\to Y\text{ measurable}\ \middle\|\ \\|u\\|_{L^{p}(K;Y)}<\infty\text{ for all }K\Subset\Omega\right\}$
Sobolev space	$W^{k,p}_{\mathrm{loc}}(\Omega;Y)$	$\left\{u\in L^{p}_{\mathrm{loc}}(\Omega;Y)\ \middle\|\ D^{\boldsymbol{\alpha}}u\in L^{p}_{\mathrm{loc}}(\Omega;Y),\ \|\boldsymbol{\alpha}\|\leq k\right\}$
Radon measures	$\mathcal{M}_{\mathrm{loc}}(\Omega;Y)$	locally finite $Y$ -valued Radon measures on $\Omega$
Bounded variation	$BV_{\mathrm{loc}}(\Omega;Y)$	$\left\{u\in L^{1}_{\mathrm{loc}}(\Omega;Y)\ \middle\|\ Du\in\mathcal{M}_{\mathrm{loc}}\bigl(\Omega;Y\otimes E^{*}\bigr)\right\}$
Special bounded variation	$SBV_{\mathrm{loc}}(\Omega;Y)$	$\left\{u\in BV_{\mathrm{loc}}(\Omega;Y)\ \middle\|\ D^{c}u=0\right\}$
Bounded Hessian	$BH_{\mathrm{loc}}(\Omega;Y)$	$\{u\in W^{1,1}_{\mathrm{loc}}(\Omega;Y)\ \|\ D^{2}u\in\mathcal{M}_{\mathrm{loc}}\bigl(\Omega;Y\otimes\operatorname{Sym}^{2}(E^{*})\bigr)\}$

Table 17: Definitions from functional analysis, the theory of generalized functions, and geometric measure theory used in this section. Here

E

and

Y

are Euclidean spaces,

\Omega\subseteq E

is open, and the superscript

c

denotes the Cantor part.

quantity	expression
	$\mathbf{z}_{1}=W_{1}\mathbf{x},\hskip 8.50012pt\mathbf{r}_{1}=\sigma(\mathbf{z}_{1}),\hskip 8.50012pt\mathbf{z}_{2}=W_{2}\mathbf{r}_{1},\hskip 8.50012pt\mathbf{r}_{2}=\sigma(\mathbf{z}_{2}),\hskip 8.50012pt\mathbf{u}=W_{3}\mathbf{r}_{2}$
ReLU masks	$D_{1}=\operatorname{diag}(\mathbf{1}_{\mathbf{z}_{1}>0}),\hskip 17.00024ptD_{2}=\operatorname{diag}(\mathbf{1}_{\mathbf{z}_{2}>0})$
preactivation variations	$\delta\mathbf{z}_{1}=\delta W_{1}\mathbf{x},\hskip 17.00024pt\delta\mathbf{z}_{2}=\delta W_{2}\mathbf{r}_{1}+W_{2}D_{1}\delta\mathbf{z}_{1}$
first variation of logits	$\delta\mathbf{u}=\delta W_{3}\mathbf{r}_{2}+W_{3}D_{2}\delta\mathbf{z}_{2}$
absolutely continuous second variation	$\begin{aligned} \delta^{2}\mathbf{u}_{\mathrm{ac}}&=2\delta W_{3}D_{2}\delta\mathbf{z}_{2}+2W_{3}D_{2}\delta W_{2}D_{1}\delta\mathbf{z}_{1}\\ &=2\Big(\delta W_{3}D_{2}\delta W_{2}\mathbf{r}_{1}+\delta W_{3}D_{2}W_{2}D_{1}\delta\mathbf{z}_{1}+W_{3}D_{2}\delta W_{2}D_{1}\delta\mathbf{z}_{1}\Big)\end{aligned}$
layer-one singular second variation	$\delta^{2}\mathbf{u}_{\mathrm{sing},1}=W_{3}D_{2}W_{2}\operatorname{diag}(\delta(\mathbf{z}_{1}))(\delta\mathbf{z}_{1}\odot\delta\mathbf{z}_{1})$

Table 18: Distributional second-variation formulas for

N^{3,\circ}

. The singular terms are interpreted through the interface-jump formula ( 5.181 ).

Learning Problem 1
Phenomenon	Explanation within the framework
Large Hessian kernels at convergence.	• orbit directions account for $O(h)$ • transverse directions (for ReLU, but not GELU, see Figure 11 ) account for $\Omega(h^{2})$
Clusters and outliers.	The rich Hessian invariance group forces eigenvalues with multiplicities of order $1$ , $k$ , and $k^{2}$ , unfolding into clusters and outliers under symmetry breaking.
Outlier count roughly matches the number of classes.	A leading eigenvalue of multiplicity $k-1$ at the symmetric reference splits under distributional SB into $k-1$ simple outliers (see Figure 1 ).
Structure of the Hessian ( $k=2$ ).	Symmetry identifies local degrees of freedom that account for the large kernel. On a codimension-one subspace, the Hessian lies in the associated commutant algebra.
Same qualitative picture across different variances ( $k=2$ ).	The composite Hessian invariance group and the kernel dimension do not depend on the variance.

Table 29

Setting	Explanation within the framework
Trivial weight symmetries ( Figure 3 ).	The point-stabilizer approach generally fails to detect the relevant Hessian invariances. The composite invariance mechanism makes them explicit by combining structural symmetries of the network differential with distributional symmetries.
$L$ -layer fully connected networks.	The general $N^{L}$ case was considered in Arjevani and Field (2019) . There, the lack of weight symmetry was also identified as a major obstruction. The composite invariance mechanism overcomes it, detecting Hessian invariances that force multiplicities of orders $\Theta(1)$ , $\Theta(d)$ , and $\Theta(d^{2})$ .
Residual connections	Residual connections can remove the transverse flat directions, allowing weight symmetries to capture the relevant Hessian invariances Arjevani (2026) .
Diagonal networks	For the diagonal linear networks (see Pesme and Flammarion (2023) ), $N^{3,\odot}(\mathbf{x};\mathbf{a},\mathbf{b},\mathbf{c})=\mathbf{1}^{\top}(\mathbf{a}\odot\mathbf{b}\odot\mathbf{c}\odot\mathbf{x})$ with $y(\mathbf{x})=\mathbf{1}^{\top}\mathbf{x}$ , the global minima are precisely $\mathbf{a}\odot\mathbf{b}\odot\mathbf{c}=\mathbf{1}$ , where $\dim\ker\nabla_{W}^{2}\mathcal{L}(W^{\star})=2d$ and $T_{W^{\star}}(\mathbb{R}^{d})^{3}/\ker\nabla_{W}^{2}\mathcal{L}(W^{\star})\cong\mathfrak{t}\oplus\mathfrak{s}_{d-1,1}$ (and see algorithmic symmetries in Section 6.1 )

Table 30

Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Abstract

Figures & tables

Explore similar work

Explaining Near-Zero Hessian Eigenvalues Through Approximate Symmetries in Neural Networks

How the Hessian-Spectrum of Neural Networks Depends on Data

Hessian Surgery: Class-Targeted Post-Hoc Rebalancing via Hessian Spike Perturbation