GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models

Tianyu Bell Pan; Damon L. Woodard

cs.LGMar 19, 2026

GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models

Authors: Tianyu Bell Pan, Damon L. Woodard

Organizations: Department of Electrical and Computer Engineering Florida Institute for National Security Applied Artificial Intelligence Group University of Florida

Abstract

Large language models (LLMs) demonstrate strong performance, but they often lack transparency. We introduce GeoLAN, a training framework that treats token representations as geometric trajectories and applies stickiness conditions inspired by recent developments related to the Kakeya Conjecture. We have developed two differentiable regularizers, Katz-Tao Convex Wolff (KT-CW) and Katz-Tao Attention (KT-Attn), that promote isotropy and encourage diverse attention. Our experiments with Gemma-3 (1B, 4B, 12B) and Llama-3-8B show that GeoLAN frequently maintains task accuracy while improving geometric metrics and reducing certain fairness biases. These benefits are most significant in mid-sized models. Our findings reveal scale-dependent trade-offs between geometric precision and performance, suggesting that geometry-aware training is a promising approach to enhance mechanistic interpretability.

Figures & tables

Figure 1: Conceptual comparison of latent space regularization.

Figure 2: Heatmap of effect sizes for geometric regularization across model scales. The values represent the standardized difference between GeoLAN and Control.

Figure 3: Evidence for the capacity preservation hypothesis. Comparison of MMLU clean accuracy between the Control and GeoLAN models.

Figure 4: Scale-dependent geometric effects of GeoLAN (

d_{\text{Cohen}}

). This displays the standardized mean difference between GeoLAN and the Control baseline across three model scales.

Model	Metric	Comparison	Mean Difference	$p$ -value	Effect Size
Gemma3-1B	Crows Stereotype Rate	Geolan vs. Baseline	0.0050	0.0957	0.22 (small)
Gemma3-1B	Crows Stereotype Rate	Control vs. Baseline	0.0065	0.5000	0.27 (small)
Gemma3-4B	Crows Stereotype Rate	Geolan vs. Control	0.0030	0.5534	0.29 (small)
Gemma3-4B	Crows Stereotype Rate	Control vs. Baseline	-0.0030	0.4765	-0.20 (small)
Gemma3-12B	Crows Stereotype Rate	Geolan vs. Baseline	-0.0065	0.0065	-0.38 (small)
Gemma3-12B	Crows Stereotype Rate	Geolan vs. Control	-0.0045	0.2420	-0.26 (small)

Table 1: Statistical comparison of stereotype bias. This table presents the mean difference,

p

-value, and

d_{\text{Cohen}}

effect size for the CrowS-Pairs stereotype rate across models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model, indices, representations
Symbol	Meaning	Notes
$V$	Vocabulary.
$S=(w_{1},\dots,w_{N})$	Input token sequence.
$w_{i}$	$i$ -th token.
$N$	Sequence length.
$M$	Transformer model.

Appendix

Table 2: Notation Summary.

Model	Metric	Comparison	Mean Difference	$p$ -value	Effect Size $d$
Gemma3 1B	Stable Examples	Geolan vs. Baseline	1.2500	0.5930	0.38 (small)
Gemma3 1B	Stable Examples	Geolan vs. Control	1.7500	0.0354	0.64 (medium)
Gemma3 1B	Stable Examples	Control vs. Baseline	-0.5000	0.8130	-0.18 (negligible)
Gemma3 1B	Distribution Kl Mean	Geolan vs. Baseline	-0.0016	0.7703	-0.25 (small)
Gemma3 1B	Distribution Kl Mean	Geolan vs. Control	0.0023	0.4790	0.39 (small)
Gemma3 1B	Distribution Kl Mean	Control vs. Baseline	-0.0039	0.2440	-0.68 (medium)

Appendix

Table 3: Statistical comparison of semantic stability (Experiment 2). This table details the impact of GeoLAN on model reasoning stability under perturbation.

Model	Metric	Comparison	Mean Difference	$p$ -value	Effect Size $d$
Gemma3 1B	Final Layer Cone Top10	Geolan vs.Baseline	0.0079	0.1392	1.29 (large)
Gemma3 1B	Final Layer Cone Top10	Geolan vs.Control	0.0081	0.2297	1.18 (large)
Gemma3 1B	Final Layer Cone Top10	Control vs.Baseline	-0.0002	0.9784	-0.02 (negligible)
Gemma3 1B	Final Layer Cone Top50	Geolan vs.Baseline	0.0059	0.1555	1.21 (large)
Gemma3 1B	Final Layer Cone Top50	Geolan vs.Control	0.0060	0.2362	1.13 (large)
Gemma3 1B	Final Layer Cone Top50	Control vs.Baseline	-0.0002	0.9731	-0.03 (negligible)

Appendix

Table 4: Statistical comparison of representation anisotropy (Experiment 3). This table evaluates the impact of GeoLAN on the geometric structure of the latent space.

Model	Metric	Comparison	Mean Difference	$p$ -value	Effect Size $d$
Gemma3 1B	Clean Acc	Geolan vs.Baseline	0.0040	0.4564	0.34 (small)
Gemma3 1B	Clean Acc	Geolan vs.Control	0.0055	0.3999	0.47 (small)
Gemma3 1B	Clean Acc	Control vs.Baseline	-0.0015	0.6084	-0.14 (negligible)
Gemma3 1B	Corr Acc	Geolan vs.Baseline	-0.0025	0.4309	-0.29 (small)
Gemma3 1B	Corr Acc	Geolan vs.Control	0.0015	0.8006	0.16 (negligible)
Gemma3 1B	Corr Acc	Control vs.Baseline	-0.0040	0.4795	-0.44 (small)

Appendix

Table 5: Statistical comparison of causal faithfulness and capacity (Experiment 4). This table analyzes the impact of GeoLAN on clean and corrupted task performance (MMLU).

Metric	Model	Baseline	Control	GeoLAN	G vs. C ( $d$ )	Pattern
BBQ Accuracy	Gemma3 1B	$0.2735\pm 0.0249$	$0.2755\pm 0.0203$	$0.2755\pm 0.0263$	+0.00	G $\approx$ C
	Gemma3 4B	$0.2930\pm 0.0267$	$0.2960\pm 0.0217$	$0.2945\pm 0.0217$	-0.07	G $\approx$ C
	Llama3 8B	$0.2890\pm 0.0294$	$0.2855\pm 0.0237$	$0.2855\pm 0.0254$	+0.00	G $\approx$ C
	Gemma3 12B	$0.2865\pm 0.0431$	$0.2855\pm 0.0314$	$0.2860\pm 0.0381$	+0.01	G $\approx$ C
Cone Concentration	Gemma3 1B	$0.8455\pm 0.0054$	$0.8453\pm 0.0066$	$0.8534\pm 0.0053$	+1.35*	G < C
	Gemma3 4B	$0.8842\pm 0.0121$	$0.8872\pm 0.0059$	$0.8819\pm 0.0085$	-0.71	G > C

Appendix

Table 6: Three-way comparison of Baseline, Control (weight decay), and GeoLAN across representation quality metrics.

d

= Cohen’s effect size for GeoLAN vs. Control. * indicates

|d|>1

(large effect).

LLM Interpretability

Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.

See all 778 papers

Laguerre Geometry for Interpreting Large Language Models
Jul 12, 2026Chunwei Ma, Russell WolfingerTransformer Interpretability LLM Interpretability
JMP Statistical Discovery, Cary, NC, USA
From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
Jun 3, 2026Chen Chu, Bita Azarijoo, Li Xiong +2Spatial Reasoning Benchmarks Multimodal Large Language Models
University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA
Measuring Representation Robustness in Large Language Models for Geometry
Apr 3, 2026Vedant Jawandhia, Yash Sinha, Murari Mandal +2LLM Evaluation Representation Geometry in Language Models
Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani

Jul 12, 2026J/K move · Enter open · S save

Laguerre Geometry for Interpreting Large Language Models

Chunwei Ma, Russell Wolfinger

JMP Statistical Discovery, Cary, NC, USA

Transformer Interpretability LLM Interpretability Representation Geometry in Language Models

Open paper AskPDF arXiv

Existing hypotheses represent a concept in an LLM as a single point, a linear direction, or a Gaussian cluster, yet it remains unclear how and why such structures emerge. Here, we show that concept geometry can be precisely characterized via Laguerre Geometry, in which a concept is defined as a region--a Laguerre-Voronoi cell or a union of cells--allowing us to strictly define, measure, and separate concepts. Building on this formulation, we show that finer-grained concept structures, such as inclusion and hierarchy, are naturally revealed by the Laguerre weights. We then push this geometry inside the transformer. Decomposing each layer into piecewise-linear operators, we show that a token's hidden trajectory is governed by two coupled mechanisms: a static tree of self-contained piecewise-linear flow, and a dynamic transport that hops the trajectory across trees when cross-token attention fires. This decomposition yields Geometric Lens, a training-free, hyperparameter-free method for reading out the exact concept a hidden vector encodes at any layer. We also develop Laguerre Autoencoder, a 2D visualizer that renders both the decision geometry and a model's full reasoning trajectory in one view. Finally, we move beyond explanatory geometry toward actionable interpretability, showing that Geometric Lens recovers the correct factual token when a model is prompted with in-context interference. The code is available on GitHub.

Abstract

LLM Interpretability

Explore similar work

Laguerre Geometry for Interpreting Large Language Models

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

Measuring Representation Robustness in Large Language Models for Geometry