Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Liam Storan; Andreas Tolias; Nina Miolane

cs.LGSep 30, 2026

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Authors: Liam Storan, Andreas Tolias, Nina Miolane

Organizations: Stanford University · University of California, Santa Barbara

Abstract

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to $\textit{translation symmetry}$ in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group $G$ , the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of $G$ . Circles and Pringles arise when $G$ is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group $\mathbb{Z}_{12}$ : for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations -- that transposition and chord inversion form a group ( $T/I$ ) acting on chords (music theory), which $\textit{implies}$ that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Figures & tables

Figure 1: Months, musical chords, and celestial objects exhibit distinct latent geometries , dictated by the symmetry of their word co-occurrence statistics in text corpora. Left : month statistics are approximately invariant under cyclic shift (

\mathbb{Z}_{12}

), which places them on a circle. Middle : musical chord statistics are approximately invariant under transposition and inversion (

D_{12}

), which places them on a dodecagonal prism. Right : celestial object statistics are approximately invariant under 3D rotations (

SO(3)

), which places them on a sphere.

Figure 2: Month statistics are

\mathbb{Z}_{12}

-invariant

\rightarrow

embed as circles (known result, re-derived).

M^{\star}

is formed using month co-occurrences in the corpus text8 , using month-sense disambiguation for “may.” The approximate

\mathbb{Z}_{12}

-invariance of

M^{\star}

causes circles to emerge in the months’ word embeddings. See App. E for all details. (a) Spectrum of

PM^{\star}P

(corpus bootstrap error bars,

2.5\%-97.5\%

), wavenumber assigned by dominant Fourier mode. The first 4 eigenvalues split into approximate pairs with the same wavenumber, a hint that

\mathbb{Z}_{12}

-invariance could be present in the data. (b) Averaged kernel

C(d)

of the individual month pair co-occurrences. The data follow the kernel rather than just averaging to it (average predicts raw data with

R^{2}=0.87)

, asserting an approximate invariance to

Z_{12}

. (c) The two circular planes. Left:

k=1

plane with median

31^{\circ}

/month. Right:

k=2

plane with median

66^{\circ}

/month.

Figure 3: Chord statistics are $T/I$ -invariant $\rightarrow$ embed as dihedral irreps. See Appendix F for all details. (a) Left: Empirical $M^{\star}$ over the $24$ triads from $371$ Bach chorales. Majors and minors are listed from $C$ to $B$ in chromatic order. Right: $\Pi_{T/I}M^{\star}$ is the projection of $M^{\star}$ onto matrices that are $T/I$ -invariant. Importantly, $\Pi_{T/I}M^{\star}$ captures much of the structure of $M^{\star}$ ( $R^{2}=0.822$ ), demonstrating an approximate $T/I$ -invariance in $M^{\star}$ . (b) Uppercase and lowercase refer to major and minor chords respectively. An orthogonal transform is used to align theoretical predictions with empirical embeddings in each case. Left: due to the statistical invariance in (a), a dihedral irrep, in the form of interleaved circles of majors and minors, emerges in the Bach chord embeddings. Our predictions obtain a per-chord cosine similarity of $0.98$ with the real embeddings. Right: Our empirical embeddings are the average activation of each triad in the layer 3 (selected for high isotypic energy, Figure 7 ) residual stream of ChordBERT across all 371 chorales. The same dihedral structure emerges. We use the Gram matrix of these centered embeddings (normalized by the root mean square of its diagonal) as a proxy for $M^{\star}$ . This gives us theoretical predictions with a per-chord cosine sim. of $0.91$ .

Figure 4: LLM’s structure is function of true celestial angles $\rightarrow$ spherical harmonics emerge, same structure also in text. Mistral Large 2 ( $123$ B) residual stream for $188$ sky objects ( Berdnikov and Liokumovich, 2026 ) at layer 72 (where positions $\in\mathbb{R}^{3}$ on the celestial sphere can be best linearly decoded). See Appendix G for all details. (a) Weighted Nadaraya–Watson regression kernel predicts Gram matrix of LLM activations over objects from actual angular separation. Standard deviation follows the curves shape, suggesting structure is largely dependent on the true angle, our proxy for statistical invariance of $M^{\star}$ . (b) Spherical harmonic purity of the model’s PCs: blocks of spherical harmonics appear. $\ell=0$ mode is removed by centering. (c) Comparison with $\ell=1$ harmonics (sphere) evaluated at true celestial positions after a fitting rotation (and global scaling for the LLM). Color bar shows right ascension Left: LLM residual steam with a randomly selected 75/188 objects shown. Best-aligned $3$ -dimensional subspace is found within the first four empirical modes (same search repeated for permutation null). Per-object cosine sim. of $0.92$ (median $0.97$ ) against an object permutation null of $0.12$ ( $95\%$ $0.17$ ). Right: Top 3 PCs of book corpus’s centered $M^{\star}$ . Per-object cosine sim. of $0.55$ (median $0.68$ ) against a null $0.22$ ( $95\%$ $0.32$ ).

Figure 5: Weaker harmonic structure in book corpus could be a sample size effect. We take the zonal kernel fit to the books as ground truth and simulate new synthetic

M^{\star}

’s, using Poisson counts for off-diagonal elements at various multiples of the corpus’s size. (full methodology in App. G ). We measure how well the simulated data aligns with dipole spherical harmonics, using the same analysis pipeline as in Figure 4 . Per-object cos. sim. (mean with 5–95% band) captures the observed corpus (star) at

1\times

and crosses the LLM’s performance (

0.92

, dashed) at

{\sim}32

–

64\times

the books’ counts: an effective corpus size for the model.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Irrep content of the top three embedding planes: we consider the first six modes of the centered

M^{\star}

. One-dimensional irreps are

A_{1},A_{2},B_{1},B_{2}

and two-dimensional irreps are

E_{1},\ldots E_{5}

A_{1}

corresponds to the trivial irrep which centering automatically removes. We compute the isotypic energy of each plane for each irrep. One irrep dominates per plane (

E_{5},E_{2},E_{3}

; energies

0.958,0.873,0.662

vs. chance

1/6

). Eigenvalues of centered

M^{\star}

are listed in legend; notice they approximately pair.

Figure 7: Irrep content of ChordBERT layer 3 PCs: one-dimensional irreps are $B_{1},B_{2},A_{2}$ , and two-dimensional irreps are $E_{1},E_{2},E_{3},E_{4},E_{5}$ . We do not show the remaining trivial irrep $A_{1}$ as it is removed automatically from centering. The first 2 PCs demonstrate high isotypic energy in $E_{5}$ .

Figure 8: (a) Held-out embedding prediction per plane for the eight hypotheses of Table 1 : mean

\pm

sample standard deviation over

10

random splits of the chorales into disjoint training/test halves. (b) Data efficiency: Training sets are nested subsets drawn without replacement from one fixed half of the

371

chorales, the other half serving as a fixed test set. Each point averages

15

redraws (shaded:

\pm 1

sample standard deviation), except the rightmost, where the training set is the entire pool and no redraw is possible. The

19

-parameter

T/I

hypothesis converges by

{\sim}12

chorales. The

300

-parameter unconstrained model (RAW) needs

\approx 4\times

more data to match it.

Model	Params	Indep.	Invariance assumed on $M^{\star}$ over the $24$ triads
$T/I$	$24$	$19$	Statistical invariance under $T/I$ .
$\mathbb{Z}_{12}$	$48$	$26$	Statistical invariance under only transposition of $T/I$ .
$\mathbb{Z}_{24}$	$24$	$13$	Invariance under a “transposition” that moves along both majors and minors as in the circular order seen in Figure 3 B. This hypothesis predicts embeddings where the major and minor circles are perfectly inter-spaced.
$PLR$	$24$	$19$	Statistical invariance under the classic $PLR$ group of music theory
$\mathbb{Z}_{12}\times\mathbb{Z}_{12}$	$26$	$15$	Statistical invariance under transposition, but separately for both majors and minors. (let $m,m+1,m+2$ be sequential major chord indices while $n,n+1,n+2$ correspond to minors) E.g., $M^{\star}_{m,m+1}=M^{\star}_{m+1,m+2}$ and $M^{\star}_{n,n+1}=M^{\star}_{n+1,n+2}$ but in general we have $M^{\star}_{m,m+1}\neq M^{\star}_{n+1,n+2}$ ) Major and minor cross terms of $M^{\star}$ are set to one constant. This hypothesis essentially assumes independent circular structures in majors and minors.
latent $\mathbb{Z}_{12}$	$12$	$7$	Majors are identified with their relative minors (music theory term, this identification looked reasonable from Figure 3 B), and invariance under transposition is assumed. This hypothesis corresponds to the embeddings forming a single circle with majors on top of minors.

Appendix

Table 1: The eight statistical hypotheses of the music model comparison. Each hypothesis is applied to

M^{\star}

by setting elements of

M^{\star}

which would be equal under the hypothesis to their average (Except RAW which is

M^{\star}

itself). Params counts the model’s number of equivalence classes without considering the symmetry of

M^{\star}

. Indep. counts the number of free parameters once this symmetry merges equivalence classes together.

Figure 9: We use various window sizes for constructing the centered

M^{\star}

(the original size was 3) which we then use to form 6-dimensional chord embeddings. (a) Rotation angles are computed by fitting a rotation matrix to each pair of dimensions (plane) of the embeddings; we consider the angle between chords in chromatic order separately for both the majors and minors. (b) Isotypic energy of each plane in its respective irrep. Chance level is always

d_{\rho}^{2}/24=1/6

for the considered irreps. (c) eigenvalues of the centered

M^{\star}

Model	layer	$R^{2}_{\mathrm{zonal}}$	$R^{2}_{\mathrm{dec}}$	canon 2 (top-3)	degree sequence
Qwen3-235B	90	0.581	0.928	0.94	$1\,1\,1\,1\,2\,2\,2\,2\,2$
Mistral-Large-123B	72	0.490	0.905	0.91	$1\,1\,1\,1\,2\,2\,2\,2\,3$
GLM-4.5-Air	46	0.484	0.904	0.91	$1\,1\,1\,1\,2\,2\,2\,2\,2$
Qwen3-32B	62	0.440	0.865	0.87	$1\,1\,1\,2\,1\,2\,2\,2\,3$
Llama-3.3-70B	63	0.395	0.898	0.91	$1\,1\,1\,2\,1\,2\,2\,3\,3$
Mixtral-8x22B	56	0.238	0.836	0.84	1 1 1 2 1 2 2 4

Appendix

Table 2: Mistral-Large-123B is the same as Mistral Large 2. Each model is evaluated at the layer where celestial positions on the sphere (

\in\mathbb{R}^{3}

) can be linearly decoded best from the activations (

R^{2}_{\mathrm{dec}}

R^{2}_{\mathrm{zonal}}

: the fraction of the activation Gram matrix structure explained by true angular separation alone. Canon 2 (top-3): average squared canonical cosines between the three leading activation modes and the dipole harmonics (subspace overlap;

1

= identical). Degree sequence: the harmonic degree (

\ell

) of each successive eigenmode—a clean harmonic embedding reads

1\,1\,1\,2\,2\,2\,2\,2\,3\ldots

Five of seven models show strongly harmonic structure (relatively high zonality

R^{2}

with degree-ordered modes). Mixtral-8x22B is borderline, and GPT-OSS-120B is the true outlier, with weak spherical harmonic structure by every measure.

Book	Author, year	ment.	$D$ ( $p$ )	coher. ( $z$ )	$\lambda_{1}$ ( $p$ )	kernel corr
Round the Year with the Stars	Olcott, 1912	361	+0.45 (0.00)	30 (8.8)	+1.98 (0.00)	+0.86
A Field Book of the Stars	Olcott, 1907	345	+0.34 (0.00)	46 (14.3)	+1.69 (0.00)	+0.85
Astronomy for Young Australians	Bonwick, 1866	116	+0.26 (0.01) †	46 (9.6)	+1.16 (0.01)	+0.74 †
Astronomy with an Opera-Glass	Serviss, 1888	364	+0.22 (0.00)	33 (7.4)	+0.86 (0.03)	+0.75
Half-Hours with the Telescope	Proctor, 1868	101	+0.14 (0.15) †	25 (3.5)	+0.34 (0.08)	+0.50 †
Recreations in Astronomy	Warren, 1879	158	+0.11 (0.12) †	18 (3.8)	+0.19 (0.17)	+0.61 †

Appendix

Table 3: Corpus selection (leading 9 candidates of

24

). Selection column

D=\operatorname{corr}(M^{\star},\cos\theta)

with one-sided object celestial sphere location-permutation

p

; boldface: selected at

p<0.05

with 500 shuffles. Coherence is the median angular hop between successive distinct mentions in degrees.

z\;=\;(\bar{m}_{\mathrm{null}}-m_{\mathrm{obs}})/\sigma_{\mathrm{null}}

is the hop’s deficit below the object textual position order-shuffled null (500 shuffles), with

\bar{m}_{\mathrm{null}}

and

\sigma_{\mathrm{null}}

, the mean and standard deviation of the median hop across all shuffles.

\lambda_{1}

is the empirical

c_{1}

from Theorem 4.2 using the book’s kernel; 500 shuffles with object celestial sphere location-permutation are used to compute

p

. Kernel corr is the correlation with the LLM activation kernel. All listed books contain

\geq 15

well-counted (

\geq 5

mentions, † : fall back of 3 mentions) objects.

Explore similar work

Symmetry in language statistics shapes the geometry of model representations
Feb 16, 2026Dhruva Karkada, Daniel J. Korchinski, Andres Nava +2Symmetry Manifolds
1UC Berkeley · 2EPFL · 3Johns Hopkins +1
Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization
May 12, 2026Zhehang Du, Hangfeng He, Weijie SuSymmetry
The Wharton School, University of Pennsylvania · University of Rochester
Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
May 22, 2026Andres Nava, Matthieu WyartContextual Embeddings Token Co-Occurrence Graphs
Johns Hopkins University · EPFL & Johns Hopkins University

Feb 16, 2026, cs.LGJ/K move · Enter open · S save

Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart+1

1UC Berkeley · 2EPFL · 3Johns Hopkins+1

Symmetry Manifolds

Open paper AskPDF arXiv

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. Our results indicate that these representational manifolds originate in the statistical symmetries of natural language.

Abstract

Explore similar work

Symmetry in language statistics shapes the geometry of model representations

Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization

Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence