Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Liam Storan; Andreas Tolias; Nina Miolane

cs.LGSep 30, 2026

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Authors: Liam Storan, Andreas Tolias, Nina Miolane

Organizations: Stanford University · University of California, Santa Barbara

Abstract

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to $\textit{translation symmetry}$ in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group $G$ , the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of $G$ . Circles and Pringles arise when $G$ is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group $\mathbb{Z}_{12}$ : for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group ( $T/I$ ) acting on chords (music theory), which $\textit{implies}$ that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Figures & tables

Figure 1: Months, musical chords, and celestial objects exhibit distinct latent geometries , dictated by the symmetry of their word co-occurrence statistics in text corpora. Left : month statistics are approximately invariant under cyclic shift (

\mathbb{Z}_{12}

), which places them on a circle. Middle : musical chord statistics are approximately invariant under transposition and inversion (

D_{12}

), which places them on a dodecagonal prism. Right : celestial object statistics are approximately invariant under 3D rotations (

SO(3)

), which places them on a sphere.

Figure 2: Month statistics are

\mathbb{Z}_{12}

-invariant

\rightarrow

embed as circles (known result, re-derived).

M^{\star}

is formed using month co-occurrences in the corpus text8 , using month-sense disambiguation for “may.” The approximate

\mathbb{Z}_{12}

-invariance of

M^{\star}

causes circles to emerge in the months’ word embeddings. See App. E for all details. (a) Spectrum of

PM^{\star}P

(corpus bootstrap error bars,

2.5\%-97.5\%

), wavenumber assigned by dominant Fourier mode. The first 4 eigenvalues split into approximate pairs with the same wavenumber, a hint that

\mathbb{Z}_{12}

-invariance could be present in the data. (b) Averaged kernel

C(d)

of the individual month pair co-occurrences. The data follow the kernel rather than just averaging to it (average predicts raw data with

R^{2}=0.87)

, asserting an approximate invariance to

Z_{12}

. (c) The two circular planes. Left:

k=1

plane with median

31^{\circ}

/month. Right:

k=2

plane with median

66^{\circ}

/month.

Figure 3: Chord statistics are $T/I$ -invariant $\rightarrow$ embed as dihedral irreps. See Appendix F for all details. (a) Left: Empirical $M^{\star}$ over the $24$ triads from $371$ Bach chorales. Majors and minors are listed from $C$ to $B$ in chromatic order. Right: $\Pi_{T/I}M^{\star}$ is the projection of $M^{\star}$ onto matrices that are $T/I$ -invariant. Importantly, $\Pi_{T/I}M^{\star}$ captures much of the structure of $M^{\star}$ ( $R^{2}=0.822$ ), demonstrating an approximate $T/I$ -invariance in $M^{\star}$ . (b) Uppercase and lowercase refer to major and minor chords respectively. An orthogonal transform is used to align theoretical predictions with empirical embeddings in each case. Left: due to the statistical invariance in (a), a dihedral irrep, in the form of interleaved circles of majors and minors, emerges in the Bach chord embeddings. Our predictions obtain a per-chord cosine similarity of $0.98$ with the real embeddings. Right: Our empirical embeddings are the average activation of each triad in the layer 3 (selected for high isotypic energy, Figure 7 ) residual stream of ChordBERT across all 371 chorales. The same dihedral structure emerges. We use the Gram matrix of these centered embeddings (normalized by the root mean square of its diagonal) as a proxy for $M^{\star}$ . This gives us theoretical predictions with a per-chord cosine sim. of $0.91$ .

Figure 4: LLM’s structure is function of true celestial angles $\rightarrow$ spherical harmonics emerge, same structure also in text. Mistral Large 2 ( $123$ B) residual stream for $188$ sky objects ( Berdnikov and Liokumovich, 2026 ) at layer 72 (where positions $\in\mathbb{R}^{3}$ on the celestial sphere can be best linearly decoded). See Appendix G for all details. (a) Weighted Nadaraya–Watson regression kernel predicts Gram matrix of LLM activations over objects from actual angular separation. Standard deviation follows the curves shape, suggesting structure is largely dependent on the true angle, our proxy for statistical invariance of $M^{\star}$ . (b) Spherical harmonic purity of the model’s PCs: blocks of spherical harmonics appear. $\ell=0$ mode is removed by centering. (c) Comparison with $\ell=1$ harmonics (sphere) evaluated at true celestial positions after a fitting rotation (and global scaling for the LLM). Color bar shows right ascension Left: LLM residual steam with a randomly selected 75/188 objects shown. Best-aligned $3$ -dimensional subspace is found within the first four empirical modes (same search repeated for permutation null). Per-object cosine sim. of $0.92$ (median $0.97$ ) against an object permutation null of $0.12$ ( $95\%$ $0.17$ ). Right: Top 3 PCs of book corpus’s centered $M^{\star}$ . Per-object cosine sim. of $0.55$ (median $0.68$ ) against a null $0.22$ ( $95\%$ $0.32$ ).

Figure 5: Weaker harmonic structure in book corpus could be a sample size effect. We take the zonal kernel fit to the books as ground truth and simulate new synthetic

M^{\star}

’s, using Poisson counts for off-diagonal elements at various multiples of the corpus’s size. (full methodology in App. G ). We measure how well the simulated data aligns with dipole spherical harmonics, using the same analysis pipeline as in Figure 4 . Per-object cos. sim. (mean with 5–95% band) captures the observed corpus (star) at

1\times

and crosses the LLM’s performance (

0.92

, dashed) at

{\sim}32

–

64\times

the books’ counts: an effective corpus size for the model.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Irrep content of the top three embedding planes: we consider the first six modes of the centered

M^{\star}

. One-dimensional irreps are

A_{1},A_{2},B_{1},B_{2}

and two-dimensional irreps are

E_{1},\ldots E_{5}

A_{1}

corresponds to the trivial irrep which centering automatically removes. We compute the isotypic energy of each plane for each irrep. One irrep dominates per plane (

E_{5},E_{2},E_{3}

; energies

0.958,0.873,0.662

vs. chance

1/6

). Eigenvalues of centered

M^{\star}

are listed in legend; notice they approximately pair.

Figure 7: Irrep content of ChordBERT layer 3 PCs: one-dimensional irreps are $B_{1},B_{2},A_{2}$ , and two-dimensional irreps are $E_{1},E_{2},E_{3},E_{4},E_{5}$ . We do not show the remaining trivial irrep $A_{1}$ as it is removed automatically from centering. The first 2 PCs demonstrate high isotypic energy in $E_{5}$ .

Figure 8: (a) Held-out embedding prediction per plane for the eight hypotheses of Table 1 : mean

\pm

sample standard deviation over

10

random splits of the chorales into disjoint training/test halves. (b) Data efficiency: Training sets are nested subsets drawn without replacement from one fixed half of the

371

chorales, the other half serving as a fixed test set. Each point averages

15

redraws (shaded:

\pm 1

sample standard deviation), except the rightmost, where the training set is the entire pool and no redraw is possible. The

19

-parameter

T/I

hypothesis converges by

{\sim}12

chorales. The

300

-parameter unconstrained model (RAW) needs

\approx 4\times

more data to match it.

Model	Params	Indep.	Invariance assumed on $M^{\star}$ over the $24$ triads
$T/I$	$24$	$19$	Statistical invariance under $T/I$ .
$\mathbb{Z}_{12}$	$48$	$26$	Statistical invariance under only transposition of $T/I$ .
$\mathbb{Z}_{24}$	$24$	$13$	Invariance under a “transposition” that moves along both majors and minors as in the circular order seen in Figure 3 B. This hypothesis predicts embeddings where the major and minor circles are perfectly inter-spaced.
$PLR$	$24$	$19$	Statistical invariance under the classic $PLR$ group of music theory
$\mathbb{Z}_{12}\times\mathbb{Z}_{12}$	$26$	$15$	Statistical invariance under transposition, but separately for both majors and minors. (let $m,m+1,m+2$ be sequential major chord indices while $n,n+1,n+2$ correspond to minors) E.g., $M^{\star}_{m,m+1}=M^{\star}_{m+1,m+2}$ and $M^{\star}_{n,n+1}=M^{\star}_{n+1,n+2}$ but in general we have $M^{\star}_{m,m+1}\neq M^{\star}_{n+1,n+2}$ ) Major and minor cross terms of $M^{\star}$ are set to one constant. This hypothesis essentially assumes independent circular structures in majors and minors.
latent $\mathbb{Z}_{12}$	$12$	$7$	Majors are identified with their relative minors (music theory term, this identification looked reasonable from Figure 3 B), and invariance under transposition is assumed. This hypothesis corresponds to the embeddings forming a single circle with majors on top of minors.

Appendix

Table 1: The eight statistical hypotheses of the music model comparison. Each hypothesis is applied to

M^{\star}

by setting elements of

M^{\star}

which would be equal under the hypothesis to their average (Except RAW which is

M^{\star}

itself). Params counts the model’s number of equivalence classes without considering the symmetry of

M^{\star}

. Indep. counts the number of free parameters once this symmetry merges equivalence classes together.

Figure 9: We use various window sizes for constructing the centered

M^{\star}

(the original size was 3) which we then use to form 6-dimensional chord embeddings. (a) Rotation angles are computed by fitting a rotation matrix to each pair of dimensions (plane) of the embeddings; we consider the angle between chords in chromatic order separately for both the majors and minors. (b) Isotypic energy of each plane in its respective irrep. Chance level is always

d_{\rho}^{2}/24=1/6

for the considered irreps. (c) eigenvalues of the centered

M^{\star}

Model	layer	$R^{2}_{\mathrm{zonal}}$	$R^{2}_{\mathrm{dec}}$	canon 2 (top-3)	degree sequence
Qwen3-235B	90	0.581	0.928	0.94	$1\,1\,1\,1\,2\,2\,2\,2\,2$
Mistral-Large-123B	72	0.490	0.905	0.91	$1\,1\,1\,1\,2\,2\,2\,2\,3$
GLM-4.5-Air	46	0.484	0.904	0.91	$1\,1\,1\,1\,2\,2\,2\,2\,2$
Qwen3-32B	62	0.440	0.865	0.87	$1\,1\,1\,2\,1\,2\,2\,2\,3$
Llama-3.3-70B	63	0.395	0.898	0.91	$1\,1\,1\,2\,1\,2\,2\,3\,3$
Mixtral-8x22B	56	0.238	0.836	0.84	1 1 1 2 1 2 2 4

Appendix

Table 2: Mistral-Large-123B is the same as Mistral Large 2. Each model is evaluated at the layer where celestial positions on the sphere (

\in\mathbb{R}^{3}

) can be linearly decoded best from the activations (

R^{2}_{\mathrm{dec}}

R^{2}_{\mathrm{zonal}}

: the fraction of the activation Gram matrix structure explained by true angular separation alone. Canon 2 (top-3): average squared canonical cosines between the three leading activation modes and the dipole harmonics (subspace overlap;

1

= identical). Degree sequence: the harmonic degree (

\ell

) of each successive eigenmode—a clean harmonic embedding reads

1\,1\,1\,2\,2\,2\,2\,2\,3\ldots

Five of seven models show strongly harmonic structure (relatively high zonality

R^{2}

with degree-ordered modes). Mixtral-8x22B is borderline, and GPT-OSS-120B is the true outlier, with weak spherical harmonic structure by every measure.

Book	Author, year	ment.	$D$ ( $p$ )	coher. ( $z$ )	$\lambda_{1}$ ( $p$ )	kernel corr
Round the Year with the Stars	Olcott, 1912	361	+0.45 (0.00)	30 (8.8)	+1.98 (0.00)	+0.86
A Field Book of the Stars	Olcott, 1907	345	+0.34 (0.00)	46 (14.3)	+1.69 (0.00)	+0.85
Astronomy for Young Australians	Bonwick, 1866	116	+0.26 (0.01) †	46 (9.6)	+1.16 (0.01)	+0.74 †
Astronomy with an Opera-Glass	Serviss, 1888	364	+0.22 (0.00)	33 (7.4)	+0.86 (0.03)	+0.75
Half-Hours with the Telescope	Proctor, 1868	101	+0.14 (0.15) †	25 (3.5)	+0.34 (0.08)	+0.50 †
Recreations in Astronomy	Warren, 1879	158	+0.11 (0.12) †	18 (3.8)	+0.19 (0.17)	+0.61 †

Appendix

Table 3: Corpus selection (leading 9 candidates of

24

). Selection column

D=\operatorname{corr}(M^{\star},\cos\theta)

with one-sided object celestial sphere location-permutation

p

; boldface: selected at

p<0.05

with 500 shuffles. Coherence is the median angular hop between successive distinct mentions in degrees.

z\;=\;(\bar{m}_{\mathrm{null}}-m_{\mathrm{obs}})/\sigma_{\mathrm{null}}

is the hop’s deficit below the object textual position order-shuffled null (500 shuffles), with

\bar{m}_{\mathrm{null}}

and

\sigma_{\mathrm{null}}

, the mean and standard deviation of the median hop across all shuffles.

\lambda_{1}

is the empirical

c_{1}

from Theorem 4.2 using the book’s kernel; 500 shuffles with object celestial sphere location-permutation are used to compute

p

. Kernel corr is the correlation with the LLM activation kernel. All listed books contain

\geq 15

well-counted (

\geq 5

mentions, † : fall back of 3 mentions) objects.

Explore similar work

Feb 16, 2026cs.LG

Symmetry in language statistics shapes the geometry of model representations

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2

1UC Berkeley · 2EPFL · 3Johns Hopkins +1

Symmetry Manifolds

Ask PDF

May 12, 2026math.OC

Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization

Large language models (LLMs) are pretrained by minimizing the cross-entropy loss for next-token prediction. In this paper, we study whether this optimization strategy can induce geometric structure in the learned model weights and context embeddings. We approach this problem by analyzing a constrained layer-peeled optimization program, which serves as a mathematically tractable surrogate for LLMs by treating the output projection matrix and last-layer context embeddings as optimization variables. Our analysis of this nonconvex optimization program demonstrates that symmetries in the target next-token distributions are transferred to the global minimizers of the layer-peeled model in a precise group-theoretic sense. Specifically, we prove that when the target tokens exhibit a cyclic-shift symmetry (such as the seven days of the week or the twelve months of the year), the optimal logit matrix is exactly circulant, and the Gram matrices of both the output projections and the context embeddings form circulant geometries as well. Next, for exchangeable target distributions invariant under the symmetric group and, more generally, under two-transitive group actions, we show that the global optimal output projection matrix forms a simplex equiangular tight frame, while the optimal logit matrix and context embeddings inherit the permutation symmetries present in the input data. A key technical step is to reduce the constrained nonconvex factorized problem to an explicit logit-level convex characterization for cyclic symmetry and to a symmetry-based lower bound for permutation symmetry, together with a sharp characterization of the optimal factorization. Finally, we empirically demonstrate that open-source LLMs naturally exhibit symmetries consistent with our theoretical predictions, despite being trained without any explicit regularization promoting such geometric structure.

Zhehang Du, Hangfeng He, Weijie Su

The Wharton School, University of Pennsylvania · University of Rochester

Symmetry

Ask PDF

May 22, 2026cs.CL

Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

We propose a distributional theory of how hypernymy -- the ``is-a'' relation between general and specific concepts -- is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a \emph{hierarchical splitting geometry} with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.

Andres Nava, Matthieu Wyart

Johns Hopkins University · EPFL & Johns Hopkins University

Contextual Embeddings Token Co-Occurrence Graphs

Ask PDF