Finding Belief Geometries with Sparse Autoencoders

Matthew Levinson

cs.LGApr 3, 2026

Finding Belief Geometries with Sparse Autoencoders

Authors: Matthew Levinson

Organizations: Independent Researcher

Abstract

Understanding the geometric structure of internal representations is a central goal of mechanistic interpretability. Prior work has shown that transformers trained on sequences generated by hidden Markov models encode probabilistic belief states as simplex-shaped geometries in their residual stream, with vertices corresponding to latent generative states. Whether large language models trained on naturalistic text develop analogous geometric representations remains an open question. We introduce a pipeline for discovering candidate simplex-structured subspaces in transformer representations, combining sparse autoencoders (SAEs), $k$ -subspace clustering of SAE features, and simplex fitting using AANet. We validate the pipeline on a transformer trained on a multipartite hidden Markov model with known belief-state geometry. Applied to Gemma-2-9B, we identify 13 priority clusters exhibiting candidate simplex geometry ( $K \geq 3$ ). A key challenge is distinguishing genuine belief-state encoding from tiling artifacts: latents can span a simplex-shaped subspace without the mixture coordinates carrying predictive signal beyond any individual feature. We therefore adopt barycentric prediction as our primary discriminating test. Among the 13 priority clusters, 3 exhibit a highly significant advantage on near-vertex samples (Wilcoxon $p < 10^{-14}$ ) and 4 on simplex-interior samples. Together 5 distinct real clusters pass at least one split, while no null cluster passes either. One cluster, 768_596, additionally achieves the highest causal steering score in the dataset. This is the only case where passive prediction and active intervention converge. We present these findings as preliminary evidence that genuine belief-like geometry exists in Gemma-2-9B's representation space, and identify the structured evaluation that would be required to confirm this interpretation.

Figures & tables

Figure 1: Overview of the belief geometry discovery pipeline. Text sequences are processed through Gemma-2-9B, and residual-stream activations at layer 20 are encoded by a GemmaScope JumpReLU SAE. Decoder directions of SAE latents are clustered into candidate latent groups using $k$ -subspace clustering. Each candidate group is then fit with AANet to test for simplex structure and recover barycentric coordinates. The resulting candidate belief geometries are evaluated using barycentric predictive advantage and causal steering as the primary validation analyses.

Cluster	$m$	Semantic Confidence	Vertex interpretation (summary)
512_17	15	HIGH	Text register (instructional / promotional / metadata)
512_229	22	MEDIUM	Syntactic complexity (simple / formal-specific / complex-embedded)
512_261	18	MEDIUM	Grammatical function (content words / function words / technical register)
512_504	22	MEDIUM	Syntactic role (function words / content nouns / predicates)
768_140	26	MEDIUM	Referential specificity (generic / specific / anaphoric)
768_210	4	MEDIUM	Syntactic complexity (complex NP / simple / intermediate)

Table 1: The 13 priority clusters. Clusters are identified by the number of clusters for the

k

-subspace clustering (512 or 768) and their cluster index.

m

denotes the number of SAE latents in the cluster. Vertex consistency is labeled as consistent, mixed, or inconsistent. Confidence is derived from 20-iteration autointerp synthesis: HIGH

=\geq

2 consistent vertices; MEDIUM

=

no inconsistent vertices; LOW

=\geq

1 inconsistent vertex.

		Near-vertex (NV)		Simplex-interior (SI)
Cluster	$n$	Frac. wins	$p$	Frac. wins	$p$
512_17	15	0.00	1.00	0.04	1.00
512_22	6	0.00	1.00	0.04	1.00
512_67	25	0.00	1.00	0.00	1.00
512_181	26	1.00	$<10^{-15}$	1.00	$<10^{-15}$
512_229	22	1.00	$<10^{-15}$	0.58	$0.003$

Table 2: Barycentric vs. best-latent predictive advantage, both evaluation splits. Frac. bary wins : fraction of top-50 highest-variance tokens for which the barycentric coordinate achieves higher

R^{2}

than the best individual latent. Wilcoxon

p

-values test the one-sided hypothesis that barycentric

R^{2}

is stochastically greater. NV = near-vertex samples; SI = simplex-interior samples. No null cluster passes on NV. One null cluster passes SI; see text.

Cluster	Steering score	Best type	Best scale
768_596	0.419	type2	20
512_22	0.356	type1	20
768_210	0.336	type2	1
768_140	0.289	type1	20
512_67	0.250	type1	20
512_261	0.242	type1	1

Table 3: Causal steering scores. Null cluster scores overlap with those of most qualifying real clusters; 768_596 is the only cluster where a significant barycentric advantage and a positive steering score converge.

Figure 2: Cluster 768_596: per-latent centroid positions. Mean barycentric centroid of each of the six latents. The latents partition across the three vertices, consistent with vertex-specialized feature coding.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Cluster	Assigned component	$R^{2}$
0	tom_quantum	0.551
1	mess3_2	0.314
2	tom_quantum_1	0.577
3	mess3_1	0.719
4	mess3	0.893
5	(noise)	—

Appendix

Table 4: Toy model clustering recovery (

R^{2}

), layer 1. Each cluster is assigned to the generative component whose belief state it best predicts (linear regression, held-out data). All five components are recovered with no conflicts.

R^{2}

values are modest but uniformly positive, consistent with genuine but partial recovery under the high-entropy multipartite setting.

Panel of Figure 3 — Figure 3: Per-sub-component PCA projections, toy model layer 1. Each panel shows the PCA subspace that best reveals one sub-component’s geometry, colored by that component’s true discrete output token. Every panel also exhibits clear separation for at least one other sub-component, demonstrating cross-component entanglement.

Panel of Figure 4 — Figure 4: Two representative latents from Cluster 4 (multipartite toy model, layer 1, TopK $K=12$ ). Each panel shows KDE-smoothed activation density over all five component belief geometries. The assigned component is marked $\star$ ( $R^{2}=0.89$ ). Top: Latent 4 fires near the top vertex of each Mess3 simplex and at the centre of the Tom Quantum disks. Bottom: Latent 26 fires near the base vertices and along the annular rim of the Tom Quantum disks. The two latents tile complementary regions of the same geometry, demonstrating geometry-consistent partitioning of belief-state information.

Figure 5: Cluster 512_181: barycentric vs. best-latent $R^{2}$ . Barycentric coordinates (mean $R^{2}=0.612$ ) outperform the best individual latent (mean $R^{2}=0.539$ ) for every one of the 50 tokens (Wilcoxon $p<10^{-15}$ ).

Figure 6: Cluster 512_229: barycentric vs. best-latent $R^{2}$ . Barycentric coordinates (mean $R^{2}=0.378$ ) outperform the best individual latent (mean $R^{2}=0.244$ ) for all 50 tokens (Wilcoxon $p<10^{-15}$ ), with a $\sim$ 55% relative improvement.

Figure 7: Cluster 768_596: barycentric vs. best-latent $R^{2}$ . Barycentric coordinates (mean $R^{2}=0.269$ ) outperform the best individual latent (mean $R^{2}=0.217$ ) for 98% of tokens (Wilcoxon $p<10^{-14}$ ). This cluster also has the highest causal steering score (0.419).

Cluster	$m$	KL ratio	Bary wins	Steering	Interp.
512_17	15	1.054	0.00	0.118	HIGH
512_22	6	1.058	0.00	0.356	LOW
512_67	25	1.229	0.00	0.250	LOW
512_181	26	1.037	1.00 (sig.)	—	LOW
512_229	22	1.166	1.00 (sig.)	—	MED
512_261	18	1.032	0.00	0.242	MED

Appendix

Table 5: Full results across all four validation strands. KL ratio: cross/same symmetric KL divergence ratio. Bary wins: fraction of top-50 tokens where

R^{2}_{\mathrm{bary}}>R^{2}_{\mathrm{best\,latent}}

(“sig.” if Wilcoxon

p<0.001

). Steering: best-combination score (“—” if cluster did not qualify). Interp.: frontier-LLM vertex label consistency (HIGH/MED/LOW).

Latent	Neuronpedia interp (GPT-4o-mini)	Active at (pie%)
L2149	Occurrences of research methodologies and results in scientific texts	V1 (19%)
L2318	Instances of error logging and debugging messages in code	—
L2759	Programming constructs and function calls associated with database management	—
L3510	Mathematical expressions related to derivatives	—
L3954	Mathematical concepts related to derivatives and calculus	—
L4321	References to privilege and inequality in social contexts	V1 (73%)

Appendix

Table 13

Latent	Neuronpedia interp (GPT-4o-mini)	Active at (pie%)
L449	Special characters, particularly underscores and certain punctuation marks	—
L1234	Various forms of punctuation and mathematical symbols	—
L1367	References to academic papers and their citation formats	—
L1462	Quotes or dialogue from characters or people	V0 (5%)
L1575	References to experimental findings and numerical data	—
L1993	Punctuations and metadata references within the text	—

Appendix

Table 14

Latent	Neuronpedia interp (GPT-4o-mini)	Active at (pie%)
L4269	Code-related syntax and structures	—
L7398	Conversational expressions and phrases	V1 (2%), V2 (100%)
L9729	Punctuation marks and their contexts within sentences	—
L12076	Questions related to religious practices and their origins	V1 (2%)
L13939	References to non-profit organizations and charitable donations	V1 (97%)
L14895	Code-related terms and programming concepts	V0 (99%)

Appendix

Table 15

Latent	Neuronpedia interp (GPT-4o-mini)	Active at (pie%)
L1189	Snippets of mathematical or programming expressions and logical statements	—
L1277	References to individuals in positions of authority or influence	—
L1425	References to historical events and their impacts on society	—
L1481	References to educational institutions and their operational details	V1 (16%)
L1540	Words indicative of financial or economic conditions and events	V1 (2%)
L1818	Punctuation and markup elements in text	—

Appendix

Table 16

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

May 9, 2026

SMIXAE: Towards Unsupervised Manifold Discovery in Language Models

Sparse autoencoders (SAEs) have been used widely to decompose and interpret neural network activations, especially those of transformer language models. One key issue with SAEs is their inability to directly model multidimensional features. Instead, SAEs may tile such features by a set of independent directions that must be grouped together after the SAE training phase, impeding discoverability and interpretation of learned feature representations. We begin to address this issue by introducing the Sparse MIXture of Autoencoders (SMIXAE) architecture. Empirically, we provide evidence that SMIXAE models have success both in directly learning previously identified manifold structures, as well as finding novel structures, within the open source Gemma 2 2B and 9B models. Finally, we discuss several limitations and point towards areas for future work.

Collin Francel

Department of Arts and Sciences, University of Alabama, Tuscaloosa, AL, United States.

Autoencoders LLM Interpretability Sparse Autoencoders Manifold Learning

Ask PDF

Jun 8, 2026

Trajectory Geometry of Transformer Representations Across Layers

Understanding how transformer representations evolve across layers, not merely what they encode, remains an open problem in mechanistic interpretability. We recast the transformer forward pass as a discrete population trajectory through a high-dimensional representation manifold, drawing on geometric tools from computational neuroscience. Rather than probing for pre-specified features, we characterize trajectory geometry using five metrics computed directly in the ambient space: trajectory length, curvature, a semantic convergence index, layerwise cosine similarity, and representational stability. Across three model families (GPT-2, TinyLlama, Qwen2.5) and five controlled prompt families, we report four findings. First, semantically related prompts converge significantly in middle-to-late layers (peak CI 0.41--0.58, p<0.001, Mann-Whitney U), consistent with attractor-like dynamics. Second, reasoning tasks produce trajectories of greater curvature than lexical variations (0.71--0.83 rad vs. 0.27--0.31 rad), suggesting curvature encodes computational complexity. Third, ambiguous tokens exhibit trajectory bifurcation with up to 5.6x representational separation by the final layer, absent in unambiguous controls. Fourth, layerwise cosine similarity reveals a universal three-phase structure: encoding, elaboration, and output preparation, consistent across all three architectures. All four effects vanish under shuffled-layer and random-embedding controls. We release a fully open-source, model-agnostic pipeline and argue that trajectory geometry constitutes a principled, probe-free lens for mechanistic interpretability.

Vishal Pandey, Gopal Singh, Yacine Mahdid

London, UK · Athens, GR · Montreal, CA

Transformer Interpretability Neural Representation Geometry Transformer Representation Geometry in Language Models

Ask PDF

Oct 7, 2026

Fully Interpretable Minimal Transformers: From Geometry to Algorithm

We present a framework for building and interpreting minimal transformer models. By constraining a transformer's embedding dimension and head size to 2, we enable full two-dimensional visualization of its internal representations. Embeddings, query/key/value transforms, attention outputs, residual streams, and decision boundaries can all be seen directly. Our central claim is that the learned geometry implies an algorithm; the arrangement of points and boundaries in R^2 can be read as a step-by-step procedure. We train a transformer on a simple task where it must produce the most recently observed even number whenever the '+' operator appears in a sequence of digits. Once trained, we visually walk through every step of the transformer's computation. We show how the model embeds the tokens and their respective positions in the sequence, transforms them via the Q, K, and V matrices, uses the dot product between the Q and K representations to form the attention matrix, and uses the attention matrix to select values that move the representation of each input token to the region of the domain of the output layer that will correctly predict the next token. We introduce a suite of interpretability visualizations that make the algorithmic interpretation of this procedure explicit. Our framework offers a pedagogical and experimental testbed to explore how transformers use informational geometry to implement next-token prediction.

Raneem Mahajne, Toviah Moldwin

Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem

Transformer Interpretability Mechanistic Interpretability Representation Geometry in Language Models Transformer Attention

Ask PDF

Abstract

Representation Geometry in Language Models

Explore similar work

SMIXAE: Towards Unsupervised Manifold Discovery in Language Models

Trajectory Geometry of Transformer Representations Across Layers

Fully Interpretable Minimal Transformers: From Geometry to Algorithm