A Gauge Theory of Superposition: Toward a Sheaf-Theoretic Atlas of Neural Representations

Hossein Javidnia

cs.LGFeb 28, 2026

A Gauge Theory of Superposition: Toward a Sheaf-Theoretic Atlas of Neural Representations

Authors: Hossein Javidnia

Organizations: School of Computing, Dublin City University, Ireland

Abstract

We develop a discrete gauge-theoretic framework for superposition in large language models (LLMs) that replaces the single-global-dictionary premise with a sheaf-theoretic atlas of local semantic charts. Contexts are clustered into a stratified context complex; each chart carries a local feature space and a local information-geometric metric (Fisher/Gauss-Newton) identifying predictively consequential feature interactions. This yields a Fisher-weighted interference energy and three measurable obstructions to global interpretability: (O1) local jamming (active load exceeds Fisher bandwidth), (O2) proxy shearing (mismatch between geometric transport and a fixed correspondence proxy), and (O3) nontrivial holonomy (path-dependent transport around loops). We prove and instantiate four results on a frozen open LLM (Llama-3.2-3B Instruct) using WikiText-103, a C4-derived English web-text subset, and the-stack-smol. (A) After constructive gauge fixing on a spanning tree, each chord residual equals the holonomy of its fundamental cycle, making holonomy computable and gauge-invariant. (B) Shearing lower-bounds a data-dependent transfer mismatch energy, turning $D_{\mathrm{shear}}$ into an unavoidable failure bound. (C) We obtain non-vacuous certified jamming/interference bounds with high coverage and zero violations across seeds and hyperparameters. (D) Bootstrap and sample-size experiments show stable estimation of $D_{\mathrm{shear}}$ and $D_{\mathrm{hol}}$ , with improved concentration on well-conditioned subsystems.

Figures & tables

Subsystem	Tree residual mean	Tree residual max	Chord/hol mean $\\|h-I\\|_{F}$	Chord/hol max $\\|h-I\\|_{F}$
Baseline ( $s_{\min}=0$ )	$1.6003\times 10^{-5}$	$2.3228\times 10^{-5}$	$4.4330$	$6.7364$
Persistent ( $s_{\min}=0.015$ )	$1.9701\times 10^{-5}$	$3.1521\times 10^{-5}$	$4.0650$	$6.3957$

Table 1: Result A identity checks on the LCC (

k=32

). Tree residuals quantify successful gauge fixing on the spanning tree. Chord residuals coincide with fundamental-cycle holonomy as predicted by Theorem 5.1 .

$s_{\min}$	$E_{\mathrm{LCC}}$	LCC size	#chords	mean $D_{\mathrm{hol}}$	max $D_{\mathrm{hol}}$
0.0000	173	90	84	0.554	0.842
0.0100	129	84	46	0.517	0.799
0.0125	106	73	34	0.517	0.799
0.0150	90	62	29	0.508	0.799
0.0200	75	59	17	0.456	0.789

Table 2: Persistence sweep (

k=32

). Increasing

s_{\min}

improves conditioning but reduces connectivity and cycle availability. The reported

D_{\mathrm{hol}}

values are computed on fundamental cycles induced by chords in the LCC.

Figure 1: Holonomy defect distributions (normalised) comparing baseline (

s_{\min}=0

) and persistent (

s_{\min}=0.015

) subsystems.

Subsystem	$n_{\text{edges}}$	$D_{\min}$	$D_{\text{med}}$	$D_{\text{mean}}$	$D_{\max}$	slack min	slack ${}_{\text{med}}$	slack max
Baseline	173	0.0354	0.2080	0.2015	0.3743	2.314	7.802	399.525
Persistent	90	0.0637	0.1966	0.1910	0.3730	2.565	7.837	399.525

Table 3: Result B summary statistics for shearing and slack (normalised) computed over all retained edges in the subsystem. Slack is

\widehat{\Delta}_{uv}/\widehat{\mathrm{LB}}_{uv}

; values

>1

confirm the bound (up to numerical error), and large values quantify conservativeness due to small

\lambda_{\min}(\widehat{\Sigma}_{u})

Figure 2: Baseline subsystem: (left)

\widehat{\Delta}_{uv}

vs lower bound

\widehat{\mathrm{LB}}_{uv}

(points above diagonal), (middle)

\widehat{\Delta}_{uv}

D_{\mathrm{shear}}(u,v)

, (right) slack distribution.

Figure 3: Persistent subsystem (

s_{\min}=0.015

): the same diagnostics as Figure 2 .

Seed	Cert. charts	Cert. rate	Slack median	Slack min	Corr $(J,\mathcal{E}^{(r)})$	Corr $(J,\mathcal{E}_{A}^{(r)})$
0	35	0.875	7.158	1.204	0.634	0.539
1	34	0.850	7.580	1.205	0.601	0.575
2	34	0.850	7.400	1.185	0.624	0.630
3	35	0.875	7.277	1.177	0.624	0.592
4	35	0.875	8.157	1.181	0.607	0.534

Table 4: Seed stability for Result C (fixed

m=256

\alpha=1.0

; 40 charts). Cert. rate measures non-vacuity. Slack min

>1

indicates zero violations. Correlations quantify the empirical association between jamming and interference.

$(m,\alpha)$	Cert. charts	Cert. rate	Slack median	Slack min	Corr $(J,\mathcal{E}^{(r)})$	Corr $(J,\mathcal{E}_{A}^{(r)})$
(128, 0.5)	35	0.875	6.785	1.198	0.464	0.464
(128, 1.0)	36	0.900	6.801	1.198	0.538	0.550
(256, 0.5)	34	0.850	6.746	1.204	0.497	0.477
(256, 1.0)	35	0.875	7.158	1.204	0.634	0.539
(512, 0.5)	34	0.850	7.054	1.202	0.424	0.292
(512, 1.0)	34	0.850	8.566	1.202	0.662	0.620

Table 5: Hyperparameter robustness for Result C (40 charts). Coverage and correctness are stable across dictionary sizes and sparsity strengths.

Figure 4: Result C diagnostics (PDF). Left: certificate check

\mathcal{E}_{A}^{(r)}\geq\widehat{\mathrm{LB}}

. Middle: projected interference energy vs jamming index

J(c)

. Right: certified subset energy vs

J(c)

Subsystem	Metric	#samples	mean	std	q05	q50	q95
Baseline	$D_{\mathrm{shear}}$	10000	0.2403	0.0689	0.0966	0.2463	0.3334
Baseline	$D_{\mathrm{hol}}$	1200	0.5772	0.0977	0.4277	0.5777	0.7386
Persistent ( $s_{\min}=0.015$ )	$D_{\mathrm{shear}}$	10000	0.2033	0.0629	0.0845	0.2104	0.2951
Persistent ( $s_{\min}=0.015$ )	$D_{\mathrm{hol}}$	4200	0.5454	0.1355	0.3875	0.5038	0.8018

Table 6: Global bootstrap stability summaries (normalised). Each row aggregates many bootstrap replicates across sampled edges (shearing) or cycles (holonomy). Quantiles describe the distributional tails of the measured defects.

$n_{\mathrm{boot}}$	mean $D_{\mathrm{shear}}$	std
256	0.69348	0.00760
512	0.70326	0.00515
1024	0.70058	0.00588
2000	0.70735	0.00537
4000	0.70086	0.00824

Table 7: Within-edge stability for

D_{\mathrm{shear}}

on baseline edge (3–33),

B=200

$n_{\mathrm{boot}}$	mean $D_{\mathrm{shear}}$	std
256	0.07365	0.00606
512	0.05668	0.00394
1024	0.04654	0.00247
2000	0.04146	0.00193
2736	0.03994	0.00169

Table 8: Within-edge stability for

D_{\mathrm{shear}}

on persistent edge (68–83),

B=200

(last row capped by available overlap). The decreasing std indicates concentration with increasing overlap size.

$n_{\mathrm{boot}}$	mean $D_{\mathrm{hol}}$	std
256	0.48596	0.03030
512	0.44759	0.02893
1024	0.44692	0.02899
2000	0.44595	0.02749
4000	0.44489	0.02925

Table 9: Within-loop stability for

D_{\mathrm{hol}}

on a baseline fundamental cycle (chord 40–102),

B=200

$n_{\mathrm{boot}}$	mean $D_{\mathrm{hol}}$	std
256	0.56800	0.02492
512	0.54174	0.02157
1024	0.52724	0.02404
2000	0.52747	0.02169
4000	0.52957	0.02452

Table 10: Within-loop stability for

D_{\mathrm{hol}}

on a persistent fundamental cycle (chord 71–96),

B=200

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Baseline model	meta-llama/Llama-3.2-3B-Instruct
Additional model 1	Qwen/Qwen2.5-3B-Instruct
Additional model 2	google/gemma-2-2b-it
Layer	$16$
Activation budget	$200{,}000$ tokens
Sequence length	$256$

Appendix

Table 11: Configuration used for the model-generalisation experiments. All settings were matched to the baseline pipeline except for the model checkpoint.

Metric	Llama 3.2 3B Instruct	Qwen 2.5 3B Instruct	Gemma 2 2B IT
Connected components	7	10	6
Largest connected component size	90	78	106
Usable edges in LCC	173	138	185
Tree edges	89	77	105
Chord edges	84	61	80
Tree residual mean	$1.60\times 10^{-5}$	$2.05\times 10^{-5}$	$2.00\times 10^{-5}$

Appendix

Table 12: Holonomy comparison across the baseline Llama run and the two additional model-generalisation runs. In all cases tree residuals remain numerically negligible, while chord residuals and holonomy defects agree to numerical precision. Qwen exhibits the largest mean holonomy defect and the largest mean normalised holonomy, while Gemma remains close to the Llama baseline.

Metric	Llama 3.2 3B Instruct	Qwen 2.5 3B Instruct	Gemma 2 2B IT
Clusters analysed	40	40	40
Clusters with positive certified bound	35	38	40
Certification rate	87.5%	95.0%	100.0%
Certified slack min	1.204	2.061	1.980
Certified slack median	7.158	6.834	3.368
Certified slack mean	10.938	9.663	4.038

Appendix

Table 13: Certified jamming comparison across models. All three runs exhibit nontrivial certification with zero violations. Gemma achieves full certification coverage on the analysed subset and the tightest certified slack distribution, while the strong positive correlation between jamming index and projected/certified energy seen in the baseline Llama run does not persist in Qwen or Gemma.

Metric	Llama 3.2 3B Instruct	Qwen 2.5 3B Instruct	Gemma 2 2B IT
Usable edges	173	170	191
Slack min	2.314	2.644	3.290
Slack median	7.802	11.055	9.541
Slack mean	14.634	39652.777	28.476
Slack max	399.525	5368921.276	2073.852
$D_{\mathrm{shear}}$ min	0.0354	0.0669	0.0415

Appendix

Table 14: Shearing comparison across models. All runs satisfy the lower-bound check, but the slack distributions differ substantially. Qwen exhibits the strongest typical shearing together with the heaviest-tailed slack distribution, whereas Gemma shows the weakest typical shearing of the three models.

Setting	Value
Model	meta-llama/Llama-3.2-3B-Instruct
Layers tested	$4,8,12,16,20,24$
Activation budget	$200{,}000$ tokens
Sequence length	$256$
Sampling stride	every $8$ tokens
Number of clusters	$128$

Appendix

Table 15: Configuration used for the layerwise generalisation experiment on the baseline Llama model. All settings were held fixed across runs except for the hidden layer index.

Metric	Layer 4	Layer 8	Layer 12	Layer 16	Layer 20	Layer 24
Connected components	9	6	9	7	11	8
Largest connected component size	40	95	70	90	81	51
Tree residual mean	$1.35\times 10^{-5}$	$1.93\times 10^{-5}$	$1.93\times 10^{-5}$	$1.60\times 10^{-5}$	$1.53\times 10^{-5}$	$1.16\times 10^{-5}$
Chord edges	24	76	101	84	85	80
Normalised holonomy mean $D_{\mathrm{hol}}$	0.668	0.578	0.430	0.554	0.607	0.588
Normalised holonomy max $D_{\mathrm{hol}}$	0.841	0.915	0.780	0.842	0.890	0.794

Appendix

Table 16: Layerwise summary of holonomy, certified jamming, and shearing on meta-llama/Llama-3.2-3B-Instruct . All quantities are computed under a matched atlas configuration; only the hidden layer changes.

Setting	Value
Model	meta-llama/Llama-3.2-3B-Instruct
Layer	$16$
Seeds tested	$0,1,2,3,4$
Activation source	fixed baseline activations
Graph source	fixed baseline cluster graph
Number of clusters	$128$

Appendix

Table 17: Configuration used for the downstream seed-reproducibility experiment. The baseline activations and cluster graph were held fixed, while seed-aware downstream stages were rerun for each seed.

Metric	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4
Largest connected component size	90	90	90	90	90
Chord edges	84	84	84	84	84
Tree residual mean	$1.60\times 10^{-5}$	$1.60\times 10^{-5}$	$1.56\times 10^{-5}$	$1.57\times 10^{-5}$	$1.57\times 10^{-5}$
Normalised holonomy mean $D_{\mathrm{hol}}$	0.5541	0.5445	0.5582	0.5556	0.5453
Normalised holonomy max $D_{\mathrm{hol}}$	0.8420	0.8546	0.8851	0.8925	0.8566
Saved clusters for jamming	124	124	124	124	124

Appendix

Table 18: Seedwise summary of the downstream reproducibility experiment on meta-llama/Llama-3.2-3B-Instruct at layer

16

. The baseline activations and graph were fixed; only seed-aware downstream stages were rerun.

Metric	Mean across seeds	Std. across seeds
Normalised holonomy mean $D_{\mathrm{hol}}$	0.5515	0.0061
Clusters with positive certified bound	34.6	0.5
Certified slack median	7.514	0.365
$\mathrm{Corr}(J,E_{\mathrm{full\_proj}})$	0.618	0.012
$\mathrm{Corr}(J,E_{\mathrm{cert}})$	0.574	0.036
Slack median	8.198	0.170

Appendix

Table 19: Compact mean/std summary across seeds for the main reproducibility metrics. Variability is small for all reported quantities, indicating strong downstream seed stability.

Metric	Baseline Llama	Random-bases null
Largest connected component size	90	90
Chord edges	84	84
Tree residual mean	$1.60\times 10^{-5}$	$1.51\times 10^{-5}$
Normalised holonomy mean $D_{\mathrm{hol}}$	0.554	1.002
Normalised holonomy max $D_{\mathrm{hol}}$	0.842	1.045
Usable edges in shearing analysis	183	183

Appendix

Table 20: Comparison between the baseline Llama run and a null control in which the learned local bases are replaced by random orthonormal bases of the same shape. The numerical gauge construction remains stable, but both holonomy and shearing increase sharply under the null, indicating that the reported geometric structure depends on learned local alignment rather than arbitrary frame choices.

Setting	Value
Model	meta-llama/Llama-3.2-3B-Instruct
Layer	$16$
Atlas sizes tested	$C\in\{64,128,256\}$
Local basis dimensions tested	$k\in\{16,32\}$
Activation budget	$200{,}000$ tokens
Sequence length	$256$

Appendix

Table 21: Configuration used for the atlas hyperparameter-robustness experiment. All settings were fixed except the number of clusters

C

and the local basis dimension

k

Metric	$(64,16)$	$(64,32)$	$(128,16)$	$(128,32)$	$(256,16)$	$(256,32)$
Graph edges	310	310	580	580	1188	1188
Saved edge ops	142	142	192	192	171	171
Connected components	2	2	7	7	28	28
Largest connected component size	55	55	90	90	99	99
Chord edges in LCC	85	85	84	84	23	23
Tree residual mean	$7.24\times 10^{-6}$	$1.26\times 10^{-5}$	$9.25\times 10^{-6}$	$1.60\times 10^{-5}$	$1.60\times 10^{-5}$	$2.44\times 10^{-5}$

Appendix

Table 22: Hyperparameter-robustness sweep over atlas size

C

and local basis dimension

k

on meta-llama/Llama-3.2-3B-Instruct at layer

16

. The baseline setting used in the main text is

(C,k)=(128,32)

Setting	Value
Model	meta-llama/Llama-3.2-3B-Instruct
Layer	$16$
Activation source	fixed baseline activations
Atlas size	$C=128$
Local basis dimension	$k=32$
Graph degrees tested	$\mathrm{knn}\in\{4,6,8\}$

Appendix

Table 23: Configuration used for the graph-construction robustness experiment. All settings were fixed except the neighbourhood parameter

\mathrm{knn}

used in the cluster graph.

Metric	$\mathrm{knn}=4$	$\mathrm{knn}=6$ (baseline)	$\mathrm{knn}=8$
Graph edges	388	580	780
Saved edge ops	177	192	200
Connected components	10	7	7
Largest connected component size	84	90	90
Chord edges in LCC	74	84	92
Tree residual mean	$1.56\times 10^{-5}$	$1.60\times 10^{-5}$	$1.70\times 10^{-5}$

Appendix

Table 24: Graph-construction robustness sweep over the neighbourhood parameter

\mathrm{knn}

on meta-llama/Llama-3.2-3B-Instruct at layer

16

. The baseline setting used in the main text is

\mathrm{knn}=6

Setting	Value
Model	meta-llama/Llama-3.2-3B-Instruct
Layer	$16$
Activation source	fixed baseline activations
Graph source	fixed baseline cluster graph
Local bases source	fixed baseline local bases
Atlas size	$C=128$

Appendix

Table 25: Configuration used for the transport-ablation experiment. All settings were fixed except the ridge parameter

\lambda

used in the edge-transport estimation step.

Metric	$\lambda=10^{-3}$	$\lambda=10^{-2}$ (baseline)	$\lambda=10^{-1}$
Saved edge ops	192	192	192
Connected components	7	7	7
Largest connected component size	90	90	90
Chord edges in LCC	84	84	84
Tree residual mean	$1.59\times 10^{-5}$	$1.60\times 10^{-5}$	$1.60\times 10^{-5}$
Normalised holonomy mean $D_{\mathrm{hol}}$	0.554125	0.554122	0.554091

Appendix

Table 26: Transport-ablation sweep over the ridge parameter

\lambda

on meta-llama/Llama-3.2-3B-Instruct at layer

16

. The baseline setting used in the main text is

\lambda=10^{-2}

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

In-context superposition: human-like working memory interference in large language models
Apr 1, 2026Hua-Dong Xiong, Li Ji-An, Jiaqi Huang +3Working Memory in Language Models LLM Interpretability
School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · Department of Cognitive Science, Indiana University Bloomington +3
Abstract representational geometry supports inference in large language models
Jun 22, 2026Yunan Zeng, Yuwang WangLLM Inference LLM Interpretability
College of Future Information Technology, Fudan University, Shanghai, 200433, China · Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, 100084, China
Laguerre Geometry for Interpreting Large Language Models
Jul 12, 2026Chunwei Ma, Russell WolfingerTransformer Interpretability LLM Interpretability
JMP Statistical Discovery, Cary, NC, USA

Apr 1, 2026J/K move · Enter open · S save

In-context superposition: human-like working memory interference in large language models

Hua-Dong Xiong, Li Ji-An, Jiaqi Huang, Robert C. Wilson+2

School of Psychological and Brain Sciences, Georgia Tech · Department of Psychology, New York University · Department of Cognitive Science, Indiana University Bloomington+3

Working Memory in Language Models LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Intelligent systems must maintain and manipulate task-relevant information online to adapt to dynamic environments. This capacity, known as working memory, is fundamental to human reasoning. Yet, human working memory is strikingly limited, maintaining only three to four items in a brain with billions of neurons. Surprisingly, large language models (LLMs), despite different substrates and direct access to prior context through attention, exhibit similar working memory limitations. Why should such different systems face analogous constraints? We propose that working memory limitations reflect a general trade-off of shared representations: representational compression and reuse support efficient learning and generalization, but also cause simultaneously active representations to interfere. We show a two-layer transformer trained on a working memory task can solve it perfectly, but diverse trained LLMs exhibit human-like limitations: performance declines with memory load, while retrieval is biased by recency and stimulus statistics. Mirroring humans, working memory performance in LLMs is also associated with broader model capability. Mechanistically, we show that LLMs encode multiple memories in entangled representations --- a condition we call in-context superposition --- and progressively suppress competing content while aligning the target with the readout. Moreover, a causal intervention that suppresses interfering information improves performance. Together, these findings suggest that working memory capacity reflects the ability to select task-relevant information under interference, a computational challenge shared by biological and artificial systems.

Abstract

LLM Interpretability

Explore similar work

In-context superposition: human-like working memory interference in large language models

Abstract representational geometry supports inference in large language models

Laguerre Geometry for Interpreting Large Language Models