The Geometry of Language Model Logits: Generic Structure and Learned Departures

Swathi Shree Narashiman; Aravind Ramana Venkatramanan; Nambi Seshadri

cs.CLOct 13, 2026

The Geometry of Language Model Logits: Generic Structure and Learned Departures

Authors: Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri

Organizations: Stanford University · Indian Institute of Technology Madras · University of California San Diego (UCSD)

Abstract

Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically identify that the ordered vocabulary-head logits follow a remarkably consistent log-linear relation

z_{(r)} \approx μ- α\log r.

across prediction contexts, architectures, and scales, emerging progressively during pretraining. To our knowledge, this is the first systematic analysis of the logit geometry underlying the distributional assumptions used in the prior decoding work. While controlled order-statistic baselines spanning bounded, light-tailed, and heavier-tailed distributions can largely explain the observed log-linear geometry, they do not explain the systematic shift in the slope $α$ : trained language models exhibit $α\approx 1.4$ -- $1.7$ , sitting well above the initialization baseline ( $α\approx 0.2$ ) . Moreover, $α$ correlates with predictive performance, suggesting that it captures learned structure beyond generic extreme-value effects.

Figures & tables

Panel of Figure 1 — Figure 1: Ordered logit geometry. (a) Ordered logit spectrum for a representative prediction context from Llama-3.2-1B, with logits sorted in descending order and plotted against $\log r$ . The red line shows the least-squares fit of $z_{(r)}=\mu-\alpha\log r$ over the top $K=2000$ vocabulary ranks. (b) Collapse of normalized ordered logit spectra across 1,000 randomly sampled prediction contexts from ten books for Llama-3.2-1B. Each spectrum is normalized independently as $(z-\mu)/\sigma$ , and the black curve shows the corpus mean. (c) Mean divergence between the true next-token distribution and its spectral reconstruction using $K$ exact logits and a two-parameter log-linear approximation of the remaining ranks, averaged over four language models. Experimental details and additional results are provided in Appendix B

Figure 2: Rank-cutoff robustness of the learned spectral regime. Spectral slope $\alpha$ as a function of rank cutoff $K$ for representative models from four model families. Each language model is compared with Gaussian and Exponential order-statistic baselines matched to its vocabulary size. Error bars denote standard deviations across the 10 books for language models and across synthetic samples for the null distributions. The corresponding dependence of goodness of fit $R^{2}$ on $K$ is reported in Figure 5 (Appendix C ).

Figure 3: Evolution of ordered logit geometry during pretraining. Mean normalized ordered-logit spectra for Pythia-410M (solid) and Pythia-1B (dashed) across representative training checkpoints. Shaded regions denote the interquartile range across prediction contexts.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Form	#	$R^{2}$	RMSE
Logarithmic	2	$0.9922\pm 0.0074$	$0.1299\pm 0.0588$
Power	3	$0.9962\pm 0.0045$	$0.0874\pm 0.0437$
Exponential	2	$0.9460\pm 0.0290$	$0.3678\pm 0.0849$
Square-root	2	$0.9043\pm 0.0391$	$0.4941\pm 0.1298$
Linear	2	$0.7752\pm 0.0534$	$0.7692\pm 0.1464$
Inverse	2	$0.1950\pm 0.0409$	$1.4642\pm 0.2010$

Appendix

Table 1: Comparison of functional forms fitted to ordered logit spectra (Llama-3.2 1B,

K=2000

). # is the number of free parameters.

Panel of Figure 4 — Figure 4: Cross-corpus normalized logit collapse for GPT-2. Left: GPT-2 evaluated on WikiText-2. Right: GPT-2 evaluated on French Wikipedia. In both panels, blue curves show individual prediction contexts and the black curve shows the corpus mean. The normalized ordered-logit spectra $(z-\mu)/\sigma$ exhibit a shared trajectory across contexts in both corpora.

Family	Model	$\alpha$	$R^{2}$
GPT-2	124M	$1.426\pm 0.243$	0.9910
	355M	$1.413\pm 0.213$	0.9914
	774M	$1.488\pm 0.220$	0.9905
	1.5B	$1.494\pm 0.229$	0.9898
Pythia	410M	$1.649\pm 0.260$	0.9889
Pythia	1.0B	$1.695\pm 0.266$	0.9889

Appendix

Table 2: Log-linear fit statistics across autoregressive language model families, averaged over 10 books (

K=2000

). Reported

\alpha

is mean

\pm

standard deviation across books;

R^{2}

is the mean per-context coefficient of determination, averaged across books.

Figure 5: Dependence of goodness of fit on rank cutoff. Coefficient of determination $R^{2}$ as a function of rank cutoff $K$ for the same representative language models and vocabulary-matched Gaussian and Exponential order-statistic baselines as in Figure 2 . Error bars denote standard deviations across the 10 books for language models and across synthetic samples for the null distributions.

		Pythia-410M		Pythia-1B
Checkpoint	Training stage	$\alpha$	$R^{2}$	$\alpha$	$R^{2}$
Step 0	Initialization	0.203	0.992	0.199	0.991
Step 8	Early	0.258	0.881	0.241	0.909
Step 64	Early	0.507	0.680	0.779	0.805
Step 512	Intermediate	1.128	0.990	1.164	0.988
Step 8000	Late	1.485	0.987	1.498	0.987

Appendix

Table 3: Evolution of the ordered-logit spectral regime during pretraining. We report the mean fitted exponent

\alpha

and

R^{2}

K=500

across evaluation contexts for Pythia-410M and Pythia-1B.

Model	$K$	$N$	Mean $R^{2}$	Mean KL	Mean JS	Mean TV
Pythia-1B	100	9880	0.9689	0.0805	0.0238	0.0772
	200	9880	0.9781	0.0536	0.0163	0.0566
	500	9880	0.9860	0.0292	0.0091	0.0348
Qwen2.5-1.5B	100	9880	0.9721	0.0848	0.0249	0.0786
	200	9880	0.9803	0.0548	0.0167	0.0562
	500	9880	0.9874	0.0300	0.0094	0.0347

Appendix

Table 4: Model-wise spectral reconstruction results. Divergences compare the exact next-token distribution with the distribution reconstructed from the top-

K

exact ordered logits and a log-linear approximation of the remaining ranks.

$K$	Mean KL	Mean JS	Mean TV	Mean $R^{2}$
100	0.0731	0.0215	0.0715	0.9661
200	0.0479	0.0145	0.0518	0.9758
500	0.0266	0.0083	0.0325	0.9848

Appendix

Table 5: Aggregate spectral reconstruction results across 39,520 contexts and four language models.

$R^{2}$	Statistic	Pearson		Sign	Spearman
		$r$	95% CI		$\rho$	95% CI
$\geq 0.95$	Entropy	$-0.332$	$[-0.370,-0.293]$	$\mathbf{-100\%}$ $(1.82e\!-\!12)$	$-0.290$	$[-0.331,-0.248]$
	NLL	$-0.206$	$[-0.236,-0.175]$	$\mathbf{-97.5\%}$ $(7.46e\!-\!11)$	$-0.159$	$[-0.193,-0.125]$
	$p_{\mathrm{true}}$	$+0.111$	$[+0.079,+0.143]$	$\mathbf{+87.5\%}$ $(1.38e\!-\!06)$	$+0.159$	$[+0.125,+0.193]$
	Log-rank	$-0.209$	$[-0.236,-0.181]$	$\mathbf{-97.5\%}$ $(7.46e\!-\!11)$	$-0.146$	$[-0.176,-0.116]$
$\geq 0$	Entropy	$-0.290$	$[-0.334,-0.245]$	$\mathbf{-100\%}$ $(1.82e\!-\!12)$	$-0.262$	$[-0.304,-0.218]$

Appendix

Table 6: Predictive-behavior analysis across 40 model–book pairs. Entries are random-effects pooled Pearson and Spearman correlations with 95% confidence intervals. The Sign column reports the percentage of book-level correlations with negative/positive sign. For the Pearson meta-analysis,

p<1e\!-\!12

for all statistics except

p_{\mathrm{true}}

, for which

p=2.02e\!-\!11

R^{2}\geq 0.95

and

p=1.36e\!-\!12

R^{2}\geq 0

. Exact two-sided binomial sign-test

p

-values are given in parentheses.

$R^{2}$ filter	Entropy	NLL	$p_{\mathrm{true}}$	Log-rank
$\geq 0.95$	$66.6\%$	$37.3\%$	$40.3\%$	$24.7\%$
$\geq 0$	$73.7\%$	$69.7\%$	$39.5\%$	$48.9\%$

Appendix

Table 7: Heterogeneity (

I^{2}

) of the pooled Pearson correlations reported in Table 6 , by fit-quality filter.

Statistic	$r$	$p$
Entropy	$0.227$	$0.159$
NLL	$0.210$	$0.194$
$p_{\mathrm{true}}$	$-0.066$	$0.684$
Log-rank	$0.189$	$0.244$

Appendix

Table 8: Pearson correlation between each model–book pair’s

R^{2}\geq 0.95

rejection fraction and its mean predictive statistics (

N=40

pairs). None of the four associations is significant at

\alpha=0.05

Autoregressive Language Modeling

Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.

See all 131 papers

The information geometry of large language models is shared, learned, and controllable
Sep 10, 2026Dario PicozziInformation Geometry Language Model Steering
Department of Physics and Astronomy, University College London (UCL), Gower Street, London WC1E 6BT, United Kingdom · London Centre for Nanotechnology, 19 Gordon St, London WC1H 0AH, United Kingdom
Scale Determines Whether Language Models Organize Representation Geometry for Prediction
May 16, 2026Weilun XuLanguage Model Pretraining Neural Representation Geometry
School of Computer and Communication Sciences ´Ecole Polytechnique F´ed´erale de Lausanne
Vocabulary shapes cross-lingual variation of word-order learnability in language models
Mar 19, 2026Jonas Mayer Martins, Jaap Jumelet, Viola Priesemann +1Linguistic Typology
University of Göttingen, Germany · University of Groningen, Netherlands · MPI for Dynamics and Self-Organization, Germany

Sep 10, 2026J/K move · Enter open · S save

The information geometry of large language models is shared, learned, and controllable

Dario Picozzi

Department of Physics and Astronomy, University College London (UCL), Gower Street, London WC1E 6BT, United Kingdom · London Centre for Nanotechnology, 19 Gordon St, London WC1H 0AH, United Kingdom

Information Geometry Language Model Steering Language Model Editing Causal Interventions in Language Models Representation Geometry in Language Models

Open paper AskPDF arXiv

Large language models learn similar behaviours, yet it remains unclear what structure they share or how to change one behaviour without disturbing others. The Fisher-Rao geometry of next-token probabilities connects these questions: behaviour determines this geometry up to output-preserving symmetries, whereas activation geometry depends on coordinates. Across transformer, state-space and recurrent models, output geometries agree more strongly than activation geometries, and shared geometry supports semantic-category transfer. Agreement with human word choices increases with predictive accuracy, scale and training, and improves further after model-only calibration. Token probabilities and read-out geometry jointly predict the spectrum and its effective dimension. Controlled language assignments show that geometry follows the language law across architectures. Pretraining corpus statistics predict held-out fact acquisition without recalibration, while randomised experiments show that deeper evidence substantially delays acquisition across every tested architecture and evidence construction. Finally, the geometry prescribes minimum-disturbance local interventions, predicts their relative cost, and supports reusable control: updates learned on donor prompts transfer to unseen prompts while better preserving behaviour on reference prompts than Euclidean control. The same geometric correction improves steering, editing, attribution, dictionary learning and fine-tuning.

Abstract

Autoregressive Language Modeling

Explore similar work

The information geometry of large language models is shared, learned, and controllable

Scale Determines Whether Language Models Organize Representation Geometry for Prediction

Vocabulary shapes cross-lingual variation of word-order learnability in language models