Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models

Marshall Brett

cs.LGApr 8, 2026

Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models

Authors: Marshall Brett

Organizations: MARS Labs

Abstract

Language models operate on discrete tokens but compute in continuous vector spaces, inducing a Voronoi tessellation over the representation manifold. We study this tessellation empirically on Qwen3.5-4B-Base, making two contributions. First, using float32 margin recomputation to resolve bfloat16 quantization artifacts, we validate Mabrok's (2026) linear scaling law of the expressibility gap with $R^2$ = 0.9997 - the strongest confirmation to date - and identify a mid-layer geometric ambiguity regime where margin geometry is anti-correlated with cross-entropy (layers 24-28, $ρ$ = -0.29) before crystallizing into alignment at the final layer ( $ρ$ = 0.836). Second, we show that the Voronoi tessellation of a converged model is reshapable through margin refinement procedures (MRP): short post-hoc optimization runs that widen token-decision margins without retraining. We compare direct margin maximization against Fisher information distance maximization across a dose-response sweep. Both methods find the same ceiling of ~16,300 correctable positions per 256K evaluated, but differ critically in collateral damage. Margin maximization damage escalates with intervention strength until corrections are overwhelmed. Fisher damage remains constant at ~5,300 positions across the validated range ( $λ$ = 0.15-0.6), achieving +28% median margin improvement at $λ$ = 0.6 with invariant downstream benchmarks - a geometric reorganization that compresses the expressibility gap while preserving its scaling law. However, frequency and token-class audits reveal that gains concentrate in high-frequency structural tokens (84% of net corrections at $λ$ = 0.6), with content and entity-like contributions shrinking at higher $λ$ . Fisher MRP is therefore a viable geometric polishing tool whose practical ceiling is set not by aggregate damage but by the uniformity of token-level benefit.

Figures & tables

Metric	bf16 (original)	fp32 (recomputed)	Mabrok range
$\beta$ (log-log slope)	0.957	0.912	0.873-1.117
R 2	0.927	0.9997	> 0.985
$\alpha$ (gap coefficient)	0.655	0.762	not reported

Table 1

Metric	Qwen3.5-4B	Mabrok range (124M-1.5B)
m 0 . 0 5	0.065	0.041-0.055
m 0 . 2 5	0.401	0.228-0.297
Median	1.030	0.487-0.641
m 0 . 7 5	2.287	0.856-1.055
m 0 . 9 5	6.153	1.484-1.756
Pr(m < 0.5)	0.300	0.395-0.512

Table 2

Model	Median	$\Delta$ Median	Pr(m<0.5)	$\Delta$ Pr	$\alpha$	R 2
Baseline	1.030	—	0.300	—	0.762	0.9997
MM $\lambda$ _MRP=0.1	1.288	+25.0%	0.236	-21.3%	0.535	0.9995
MM $\lambda$ _MRP=0.15	1.366	+32.6%	0.222	-26.0%	0.494	0.9995
MM $\lambda$ _MRP=0.3	1.571	+52.5%	0.180	-39.9%	0.389	0.9995
MM $\lambda$ _MRP=1.0	1.932	+87.5%	0.062	-79.5%	0.130	0.9881
Fish $\lambda$ _MRP=0.15	1.174	+13.9%	0.271	-9.7%	0.670	0.9994

Table 3

Model	Churn%	W $\rightarrow$ R	R $\rightarrow$ W	Flip Ratio	Net Corrected
MM $\lambda$ _MRP=0.1	20.0%	16,358	5,716	2.9x	+10,642
MM $\lambda$ _MRP=0.15	20.5%	16,390	6,065	2.7x	+10,325
MM $\lambda$ _MRP=0.3	22.9%	16,680	7,910	2.1x	+8,770
MM $\lambda$ _MRP=1.0	50.8%	16,719	39,108	0.4x	-22,389
Fish $\lambda$ _MRP=0.15	19.3%	16,195	5,317	3.0x	+10,878
Fish $\lambda$ _MRP=0.3	19.3%	16,256	5,298	3.1x	+10,958

Table 4

Model	Rotated%	Wider Margin%	Mean Margin $\Delta$
MM $\lambda$ _MRP=0.1	25.1%	68.6%	+0.820
MM $\lambda$ _MRP=0.3	26.2%	74.8%	+0.902
Fish $\lambda$ _MRP=0.15	24.7%	64.8%	+0.790
Fish $\lambda$ _MRP=0.3	24.8%	67.5%	+0.840
Fish $\lambda$ _MRP=0.6	24.8%	72.0%	+0.922
Fish $\lambda$ _MRP=1.0	23.0%	75.0%	+0.949

Table 5

Model	m<0.5	0.5-1	1-2	2-5	5+	Overall
Baseline	21.6% (77K)	31.5% (49K)	46.6% (56K)	78.1% (53K)	99.3% (21K)	47.1%
MM $\lambda$ _MRP=0.3	21.5% (46K)	27.4% (38K)	37.5% (69K)	72.1% (71K)	99.3% (33K)	50.5%
Fish $\lambda$ _MRP=0.3	22.6% (68K)	31.6% (46K)	46.6% (54K)	78.3% (56K)	99.4% (33K)	51.4%
Fish $\lambda$ _MRP=0.6	22.1% (64K)	30.9% (44K)	44.6% (54K)	75.9% (60K)	99.3% (35K)	51.4%
Fish $\lambda$ _MRP=1.0	22.0% (63K)	30.3% (43K)	43.3% (54K)	74.1% (61K)	99.1% (35K)	51.0%
Fish $\lambda$ _MRP=2.0	21.2% (55K)	28.7% (38K)	39.3% (52K)	67.2% (72K)	98.4% (40K)	50.8%

Table 6

Model	Positions Wider	Mean $\Delta$	Median $\Delta$
MM $\lambda$ _MRP=0.1	65.4%	+0.507	+0.197
MM $\lambda$ _MRP=0.3	72.8%	+0.653	+0.408
Fish $\lambda$ _MRP=0.15	60.0%	+0.448	+0.112
Fish $\lambda$ _MRP=0.3	63.1%	+0.493	+0.151
Fish $\lambda$ _MRP=0.6	68.2%	+0.574	+0.232

Table 7

Model	ARC-C	HellaSwag	WinoGrande	PIQA	LAMBADA	TQA	Mean
Baseline	.528	.751	.709	.784	.668	.354	.632
MM $\lambda$ _MRP=0.15	.535	.754	.706	.785	.683	.355	.636
MM $\lambda$ _MRP=0.3	.529	.754	.696	.786	.698	.353	.636
Fish $\lambda$ _MRP=0.15	.534	.754	.700	.785	.670	.351	.632
Fish $\lambda$ _MRP=0.3	.535	.754	.706	.784	.671	.349	.633
Fish $\lambda$ _MRP=0.6	.527	.751	.710	.783	.667	.354	.632

Table 8

Bucket	Baseline	Fish 0.6	$\Delta$ 0.6	Fish 1.0	$\Delta$ 1.0	Fish 2.0	$\Delta$ 2.0
1	0.283	0.294	+0.011	0.287	+0.004	0.284	+0.001
2-4	0.310	0.324	+0.014	0.320	+0.011	0.318	+0.008
5-19	0.341	0.356	+0.015	0.351	+0.010	0.347	+0.006
20-99	0.391	0.409	+0.019	0.404	+0.014	0.399	+0.008
100+	0.557	0.620	+0.063	0.616	+0.059	0.617	+0.059

Table 9

Class	Fish 0.6 Net	Share 0.6	Fish 1.0 Net	Share 1.0	Fish 2.0 Net	Share 2.0
Structural	7,231	65.9%	7,001	70.6%	7,135	75.0%
Function	1,208	11.0%	979	9.9%	982	10.3%
Content	1,708	15.6%	1,284	13.0%	890	9.4%
Entity-like	356	3.2%	240	2.4%	82	0.9%
Numeric	435	4.0%	382	3.9%	386	4.1%

Table 10

Metric	Instruct Baseline	Instruct Fisher	$\Delta$ / Note
Token accuracy	45.4%	48.1%	+2.63%
Flip ratio	—	1.74x	materially weaker than base-model Fisher
Head-bucket share	—	85.0%	100+ frequency bucket share of net gain
Structural class share	—	58.8%	dominant source of gains
Entity-like net	—	-219	net negative
6-task mean*	—	0.632	no matched baseline bundle preserved

Table 11

Variant	$\Delta$ Acc	Flip Ratio	Head Share of Net	Entity-like Net
Bulk Fish $\lambda$ _MRP=0.6	+4.28%	3.05x	84.0%	+356
Targeted base fix03	+2.02%	1.94x	94.1%	-61
Targeted base fix06	+1.95%	1.86x	94.9%	-105
Targeted instruct fix03	+2.58%	1.68x	83.4%	-98
Targeted instruct v2 fix03	+2.63%	1.74x	85.0%	-219

Table 12

264M Pilot	Accuracy	Flip Ratio	W $\rightarrow$ R	R $\rightarrow$ W
Mean-protect, 50 steps	0.3938	2.90x	255	88
Mean-protect, 200 steps	0.4013	2.43x	336	138
Mean-protect + EWC $\lambda$ =1	0.4003	2.42x	331	137
Mean-protect + EWC $\lambda$ =10000	0.4013	2.43x	336	138

Table 13

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 404 papers

Mar 17, 2026

Latent Semantic Manifolds in Large Language Models

Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws

Mohamed A. Mabrok

Department of Mathematics and Statistics, Qatar University

Representation Geometry in Language Models Information Geometry

Ask PDF

Jul 5, 2026

Geometry of Ordinal Representations in Language Models

Recent work showed that language models represent character counts on curved 1D manifolds, with attention heads performing geometric transformations to enable computation. We test whether this generalizes across four ordinal tasks (bracket depth, indentation, table position, numeric magnitude) in Gemma-2-2B, Gemma-2-9B, and Qwen3-4B. We find that 1D manifolds with place-cell feature tiling emerge for tasks where the ordinal variable is locally computable from token identity, while tasks requiring cross-position integration or semantic extraction produce higher-dimensional or incoherent representations. Geometric computation is architecture-dependent: Qwen3-4B shows substantially stronger twisting than Gemma models for indentation, and its twisters preserve ordinal order, unlike its numeric twisters. Activation patching confirms that the identified manifold subspaces concentrate task-relevant information, with manifold-direction ablation causing dramatically larger probe accuracy drops than random-direction controls.

Saksham Bassi, Sharvi Tomar

*Equal contribution

Geometric Representation Learning Representation Geometry in Language Models

Ask PDF

Apr 2, 2026

On the Geometric Structure of Layer Updates in Deep Language Models

We study the geometric structure of layer updates in deep language models. Rather than analyzing what information is encoded in intermediate representations, we ask how representations change from one layer to the next. We show that layerwise updates admit a decomposition into a dominant tokenwise component and a residual that is not captured by restricted tokenwise function classes. Across multiple architectures, including Transformers and state-space models, we find that the full layer update is almost perfectly aligned with the tokenwise component, while the residual exhibits substantially weaker alignment, larger angular deviation, and significantly lower projection onto the dominant tokenwise subspace. This indicates that the residual is not merely a small correction, but a geometrically distinct component of the transformation. This geometric separation has functional consequences: approximation error under the restricted tokenwise model is strongly associated with output perturbation, with Spearman correlations often exceeding 0.7 and reaching up to 0.95 in larger models. Together, these results suggest that most layerwise updates behave like structured reparameterizations along a dominant direction, while functionally significant computation is concentrated in a geometrically distinct residual component. Our framework provides a simple, architecture-agnostic method for probing the geometric and functional structure of layer updates in modern language models.

Jun-Sik Yoo

Institute of Basic Science, Korea University, 145 Anam-ro, Seoul, South Korea 02841.

Representation Geometry in Language Models LLM Interpretability

Ask PDF

Abstract

Representation Geometry in Language Models

Explore similar work

Latent Semantic Manifolds in Large Language Models

Geometry of Ordinal Representations in Language Models

On the Geometric Structure of Layer Updates in Deep Language Models