Local and Global Regimes of Geometric Complexity in Language Model Representations

Arwa Osman; Marco Baroni; Iuri Macocco

cs.CLAug 14, 2026

Local and Global Regimes of Geometric Complexity in Language Model Representations

Authors: Arwa Osman, Marco Baroni, Iuri Macocco

Organizations: Universitat Pompeu Fabra (UPF) · ICREA

Abstract

Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.

Figures & tables

Figure 1: Apparent NOUN vs. ADP differences in intrinsic dimensionality reverse once lexical diversity is controlled (GRIDE, Qwen3-8B,

k=128

;

N=10{,}000

samples). Left: naive comparison, unmatched vocabulary size (NOUN: 2600+ types; ADP:

\sim

43 types). Right: both classes matched to 25 types, with 400 tokens per type (Section 3 ).

Unique Nouns	Samples per Noun
1	10,000
25	400
50	200
100	100
250	40
500	20

Table 1: Lexical diversity conditions.

Figure 2: ID curves across layers for all considered scales and lexical-diversity conditions on Qwen3-8B. The “transition point” curve is shown with a thicker line.

Figure 3: Schematic illustration of the local and global regimes. Left: at low lexical diversity, a small neighbourhood scale (blue) stays within the same word’s cluster (local regime), while a large scale (orange) already extends into a different word’s cluster (global regime). Right: at high lexical variety, clusters are smaller and more numerous, so even a small scale can extend beyond the target word’s own cluster.

Figure 4: ID as a function of

k/m

across four layers of Qwen3-8B (layers 5, 21, 30, and 36).

Figure 5: Same-word fraction across layers, evaluated at

k=m-1

for each lexical diversity condition.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: ID curves across layers for each lexical diversity level

n

, with one line per scale

k

. The bold line marks the transition scale.

Figure 7: MLE ID curves across layers for all scales on Qwen3-8B. The transition pattern observed with GRIDE (Figure 2 ) is reproduced at every scale

k

Figure 8: ID curves across layers for all scales on Meta-Llama-3-8B. The transition occurs at

n=N/k

nouns at every scale, matching the pattern observed on Qwen3-8B (Figure 2 ).

Figure 9: ID curves across layers for all scales on Qwen3-8B, computed on the POS-diversified dataset (Appendix A ). The transition pattern is preserved at each scale

k

, matching the noun-only results in Figure 2 .

Intrinsic Dimensionality

Intrinsic dimensionality measures the effective degrees of freedom underlying data embedded in high-dimensional spaces.

See all 56 papers

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis
Oct 10, 2026Samhita Bollepally, Takashi Yamauchi, Manju Muralee +1LLM Interpretability Representation Geometry in Language Models
Latent Semantic Manifolds in Large Language Models
Mar 17, 2026Mohamed A. MabrokRepresentation Geometry in Language Models Information Geometry
Department of Mathematics and Statistics, Qatar University
A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings
Aug 30, 2026Alexandre QuemyIntrinsic Dimensionality Neural Representation Geometry
Hother Labs

Oct 10, 2026J/K move · Enter open · S save

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis

Samhita Bollepally, Takashi Yamauchi, Manju Muralee, Motunrayo Adebimpe

LLM Interpretability Representation Geometry in Language Models Language Model Probing

Open paper AskPDF arXiv

Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning. A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input. The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer. The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.

Abstract

Intrinsic Dimensionality

Explore similar work

Do LLMs differentiate linguistic traits internally?: A layer-wise analysis

Latent Semantic Manifolds in Large Language Models

A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings