Symmetry Breaking Is the Price of Finite Capacity

Maggie Chlon; Leon Chlon

stat.MLOct 26, 2026

Symmetry Breaking Is the Price of Finite Capacity

Authors: Maggie Chlon, Leon Chlon

Organizations: Hassana Labs.

Abstract

If rotating a problem never changes its answer, one might expect its best model to be rotation-invariant; we show that this fails, predictably and by an exact amount, when capacity is limited. A symmetry groups a model's directions into blocks of equally strong directions, a symmetric model must use each block whole, and when the budget cuts a block every optimal model breaks the symmetry. For a Gaussian problem with a $k$ -unit bilinear predictor, the log-loss price of symmetry is exactly $(U_k-I_k)/(2σ^2)$ , the signal of the best $k$ directions minus that of the best whole blocks. Because an attention head compares tokens through a similarity of rank at most its head dimension, whatever computation precedes it, the same price governs heads fitted to a symmetric similarity by squared error, and heads trained this way on three-dimensional directions break symmetry at exactly the predicted head dimensions, with the predicted rotation sensitivity (RMSE 0.004). Under softmax attention, flatter attention restores the predicted symmetric head where tested, while with peaked attention broken heads beat every whole-shell symmetric head at 16 of 20 head dimensions. We give a procedure that separates capacity-driven breaking from other causes.

Figures & tables

Figure 1: Why limited capacity breaks symmetry. Bars are directions, coloured by block (sizes 1, 3, 5, 7). With budget 6, the unrestricted model keeps the 6 strongest directions and cuts the third block; the symmetric model must keep whole blocks and leaves two units unused. The signal it gives up,

3.64-2.92

, divided by

2\sigma^{2}

, is the exact log-loss price of symmetry ( Theorem 3.1 ).

Figure 2: Logit regression reproduces Corollary 4.3 . (a) Rotation sensitivity of trained heads (mean

\pm

s.d., three seeds) against the prediction ( 17 ); heads are symmetric only where

k

completes a shell (dotted). (b) Excess squared error of trained heads, of the best head (

S-U_{k}

, with

S=\sum_{a}d_{a}\lambda_{a}^{2}

) and of the best symmetric head (

S-I_{k}

). The gap between the lines is the price trained heads avoid by breaking.

Figure 3: Softmax attention. Dotted lines mark the predicted symmetric heads (

k=3,8,15

); both panels divide by

m^{2}

. (a) Rotation sensitivity (means over three seeds). At

m=0.1

the head at

k=8

is symmetric, over 300 times less sensitive than its neighbours; at

m=0.3

heads are symmetric at

k=3

, nearly so at

15

, and broken at

8

. (b) KL of the best whole-shell head (

l\leq 5

) minus KL of trained heads (s.d. bars on the 1,500-step series); positive means breaking wins. The negative values at

k=10

m=0.3

are where training ends at a broken head that a symmetric head outperforms ( Section 6 ).

Items, symmetry	Block sizes	Whole-block budgets
Angles, $O(2)$	$1,2,2,\dots$	$1,3,5,\dots$ ( $2,4,\dots$ )
Directions in $\mathbb{R}^{3}$ , $SO(3)$	$1,3,5,\dots$	$1,4,9,\dots$ ( $3,8,\dots$ )
Directions in $\mathbb{R}^{d}$ , $SO(d)$	$1,\,d,\,\tfrac{d(d+1)}{2}{-}1,\dots$	$1,\,d{+}1,\dots$ ( $d,\dots$ )
$n$ symbols, $S_{n}$	$1,\,n{-}1$	$1,\,n$ ( $n{-}1$ )

Table 1: Blocks of common symmetries and the budgets at which whole blocks fit, assuming strengths decrease in the order listed; under softmax the constant block is free (bracketed).

$m$	$k$	Whole shells	Trained	Sens.	Reading
0.3	3	0.566 (1)	0.567	0.009	symmetric
0.3	8	0.259 (1,2)	0.208	14.8	breaking wins
0.3	10	0.138 (1,3)	0.150	13.3	symmetric better
0.3	11	0.138 (1,3)	0.137	9.6	tie
0.3	15	0.091 (1,2,3)	0.091	0.21	nearly symmetric
0.1	8	0.0228 (1,2)	0.0228	0.0024	symmetric

Table 2: Softmax attention: trained heads against the best head built from whole shells

l\leq 5

(KL to the target attention; lower is better; means over three seeds, 6,000 steps for

k=8,10

and 1,500 otherwise). Sensitivity is divided by

m^{2}

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Tokens	independent uniform unit vectors in $\mathbb{R}^{3}$
Target similarity	( 16 ) with $\lambda=(1,0.8,0.6,0.45,0.33,0.25)$ , shells $l=0,\dots,5$
Head	separate query and key MLPs, SiLU, $3\to 128\to 128\to 128\to k$ ; logit $=$ dot product
Initialisation	weights $N(0,2/\text{fan-in})$ , zero biases
Logit regression	squared error on 8,192 fresh query–key pairs per step
Softmax attention	KL to $\mathrm{softmax}(m\kappa)$ over 64 keys, 256 fresh queries per step
Optimiser	Adam, learning rate $2\times 10^{-3}$ , 100 warm-up steps, cosine decay to $10^{-5}$

Appendix

Table 3: Configuration of the GPU runs.

$k$	Head features	KL	Sensitivity $/m^{2}$	Dimensions used, $l=1,\dots,5$	Best whole-shell
8	exact features	0.208–0.212	15.1–15.6	2.3 / 2.9 / 2.3 / 0.4 / 0.1	0.259
10	MLP	0.155	11.3	1.8 / 4.0 / 3.1 / 0.5 / 0.1	0.138
10	exact features	0.150–0.153	12.3–13.4	2.5 / 2.7 / 3.5 / 1.0 / 0.2	0.138
10	exact, symmetric start	0.137–0.138	0.001–0.92	2.7–3.0 / $\leq$ 0.1 / 6.6–7.0 / $\leq$ 0.1 / 0.0	0.138
15	MLP	0.091	0.15	2.9 / 4.8 / 6.4 / 0.0 / 0.0	0.091
15	exact features	0.078–0.081	8.5–8.6	2.7 / 3.2 / 4.1 / 4.1 / 0.8	0.091

Appendix

Table 4: Which shells softmax heads use (

m=0.3

). Dimensions used counts how many of the

3,5,7,9,11

dimensions of shells

l=1,\dots,5

the head’s key features span. Exact features replace the multilayer perceptrons with spherical harmonics up to degree five: three random starts at

k=8,10

and two at

k=15

, with initial weights of standard deviation 0.05, and at

k=10

seven starts near the symmetric head built from shells 1 and 3. The MLP rows are single CPU runs of the network of the GPU sweep; the sweep itself, with three seeds, gives KL 0.156 and 0.091 and scaled sensitivity 11.9 and 0.21 at

k=10

and

15

Check	Theory / reference value	Numerical result
Equal-block gap, $r=3$ , $k=5$ , $\sigma=0.9$	0.79012346	0.79012346
Repeated-isotypic gap, $k=4$ , $\sigma=1.2$	0.31250000	0.31250000
Irreducible gap, $d=5$ , $k=2$ , $\lambda=0.7$	0.49000000	0.49021011
Invariant anisotropy, $\varepsilon=0.12$	$[0.254800,\ 0.725200]$	0.61465600
Haar mean $\\|\widehat{\Pi M}-\Pi M\\|_{F}$	0	0.00463236
Rank before / exact Haar mean	2 / 5	2 / 5

Appendix

Table 5: Checks of the reference implementation.

Symmetry of the items compared	Block sizes	Whole-block budgets	Under softmax
Rotations and reflections of angles	$1,2,2,2,\dots$	$1,3,5,\dots$	$2,4,6,\dots$
Rotations of directions in $\mathbb{R}^{3}$	$1,3,5,7,\dots$	$1,4,9,16,\dots$	$3,8,15,\dots$
Rotations of directions in $\mathbb{R}^{d}$ , $d\geq 3$	$1,\ d,\ \tfrac{d(d+1)}{2}-1,\dots$	$1,\ d+1,\ \tfrac{d(d+3)}{2},\dots$	$d,\ \tfrac{d(d+3)}{2}-1,\dots$
Relabelling $n$ equally likely symbols	$1,\ n-1$	$1,\ n$	$n-1$

Appendix

Table 6: Blocks of some common symmetries. The budgets listed assume that block strengths decrease in the order given. Under softmax attention the constant block is free ( Section 4 ).

Trained loss versus best symmetric loss	Sensitivity	Reading	Action
lower	high	breaking is the price of capacity	complete the blocks, add a cheap invariant path, or accept it
equal	near zero	a symmetric solution was reached; if strong blocks are left unused, the features may be the limit ( Section 6 , $k=15$ )	none, or richer features
equal	high	breaking neither helps nor hurts at this budget ( Section 6 , $k=11$ )	none needed; a symmetric solution is available
higher	high	not the price of capacity: training ended at a worse broken solution ( Section 6 , $k=10$ )	start from, or constrain to, the symmetric solution
any	high, and runs agree on the orientation	suspect the data, architecture or optimiser	check the data and the symmetry action

Appendix

Table 7: Reading the outcome of the procedure. Losses are compared at the same budget.

Symmetry Breaking

Symmetry breaking studies how equivalent states become distinguishable in physical systems, algorithms, and learned representations.

See all 27 papers

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
May 6, 2026Dominik Dahlem, Diego Maniloff, Mac MisiuraTransformer Interpretability Attention Head Analysis
Red Hat AI
Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects
Oct 26, 2026John J. Vastola, Kanaka RajanSymmetry Breaking Diffusion Models
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
Jul 25, 2026Byeong Hoon YoonSoftmax Attention Transformer Expressivity
Independent Researcher

May 6, 2026J/K move · Enter open · S save

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

Dominik Dahlem, Diego Maniloff, Mac Misiura

Red Hat AI

Transformer Interpretability Attention Head Analysis Self-Attention LLM Hallucination Detection Spectral Methods

Open paper AskPDF arXiv

When a language model processes a hallucinated response, its attention routing tends to fail in one of two shapes: over-concentrating on a narrow set of positions, or spreading so diffusely that relevance is diluted, and the shape of the failure carries diagnostic signal. We study these shapes as a diagnostic characterization, computed from attention matrices under forced scoring of benchmark-labeled responses rather than during live generation. A widely used family of spectral methods analyzes the symmetric component of the degree-normalized attention operator, which governs transport capacity; we prove that every transpose-invariant spectral diagnostic of this operator is structurally orientation-blind (it cannot distinguish an operator from its transpose, and therefore cannot detect information-flow direction), with a converse to the blindness theorem bounding any Lipschitz diagnostic's transpose sensitivity by the asymmetry coefficient

G

. Pairing this with a closed-form bipartite-Cheeger landscape for canonical causal architectures, we show that uniform causal attention satisfies an

n

-independent floor

φ\ge 1/5

, while window attention pierces the floor as

O(w/n)

; failure modes are shape-different, not just value-different. This floor is an idealized-architecture benchmark, not an empirical attractor: the fraction of real attention heads that pierce it is itself an architectural signature. The resulting two-axis diagnostic (

φ

for capacity,

G

for direction) yields a falsifiable polarity prediction: bottleneck- and diffuse-dominated benchmarks should exhibit opposite polarity. Under length-controlled evaluation, transport features retain interpretable signal (0.62-0.84 LC-AUROC) across the tested decoder-only, encoder-only, and encoder-decoder models, with polarity reversing as predicted between HaluEval and MedHallu.

Abstract

Symmetry Breaking

Explore similar work

Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics

Quantifying the failure of symmetry learning in linear diffusion models due to initialization and finite-sample effects

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers