Intrinsic dimensionality (ID) is widely used to probe the representational complexity of language models, but it remains unclear whether ID differences reflect properties of language itself or artefacts of how the underlying dataset was constructed. In this paper, we focus specifically on how lexical diversity, the number of unique last-token items present in a dataset, affects ID estimates of that dataset. We find a scale-dependent transition between two regimes: at low lexical diversity, conditions with fewer unique final words produce higher ID, while at high lexical diversity, this ordering reverses, and conditions with more unique words produce higher ID. We derive an exact, parameter-free formula for the point at which this reversal occurs, which matches the observed transition point at every scale tested. On the one hand, our results highlight how care must be taken when interpreting the intrinsic dimensionality of a set of representations as a straightforward cue of their complexity. On the other hand, our discovery of the two ID regimes reveals a general principle of organisation of linguistic data in LLMs that sheds new light on their inner manifold structures.
Figures & tables
Figure 1: Apparent NOUN vs. ADP differences in intrinsic dimensionality reverse once lexical diversity is controlled (GRIDE, Qwen3-8B, k=128 ; N=10,000 samples). Left: naive comparison, unmatched vocabulary size (NOUN: 2600+ types; ADP: ∼ 43 types). Right: both classes matched to 25 types, with 400 tokens per type (Section 3 ).
Unique Nouns
Samples per Noun
1
10,000
25
400
50
200
100
100
250
40
500
20
Table 1: Lexical diversity conditions.
Figure 2: ID curves across layers for all considered scales and lexical-diversity conditions on Qwen3-8B. The “transition point” curve is shown with a thicker line.
Figure 3: Schematic illustration of the local and global regimes. Left: at low lexical diversity, a small neighbourhood scale (blue) stays within the same word’s cluster (local regime), while a large scale (orange) already extends into a different word’s cluster (global regime). Right: at high lexical variety, clusters are smaller and more numerous, so even a small scale can extend beyond the target word’s own cluster.
Figure 4: ID as a function of k/m across four layers of Qwen3-8B (layers 5, 21, 30, and 36).
Figure 5: Same-word fraction across layers, evaluated at k=m−1 for each lexical diversity condition.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: ID curves across layers for each lexical diversity level n , with one line per scale k . The bold line marks the transition scale.
Figure 7: MLE ID curves across layers for all scales on Qwen3-8B. The transition pattern observed with GRIDE (Figure 2 ) is reproduced at every scale k .
Figure 8: ID curves across layers for all scales on Meta-Llama-3-8B. The transition occurs at n=N/k nouns at every scale, matching the pattern observed on Qwen3-8B (Figure 2 ).
Figure 9: ID curves across layers for all scales on Qwen3-8B, computed on the POS-diversified dataset (Appendix A ). The transition pattern is preserved at each scale k , matching the noun-only results in Figure 2 .
Large language models often produce judgments that resemble human responses, but it remains unclear whether they internally capture linguistic nuances such as conventional versus idiomatic meaning. A layer-wise representational analysis was conducted on two equal-sized pre-trained models, Llama-3.2-1B and Gemma-3-1B-PT, to examine whether and where these models differentiate linguistic traits from input. The analysis combined layer-wise probing with geometric distance measures, using the top embedding dimensions ranked by principal component loadings for each layer. The results show that even with equal capacity, models organize linguistic information in fundamentally different ways, suggesting the need to analyze internal representational geometry, examining not just behavioral outputs but semantic space alignment when evaluating how closely human and model language interpretation correspond.
Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws
Mohamed A. Mabrok
Department of Mathematics and Statistics, Qatar University
A token-embedding table holds a hub of short rows near its origin, and we show that this cluster biases what nearest-neighbor intrinsic-dimension (ID) estimators report. Because of the concentration of measure, a token is closer to the central cluster than to any other token, so its first two neighbors are both hub rows at nearly the same distance. As a result, the ID estimators such as TwoNN return a dimension far above the real ID. Measured one token at a time, dimension is a heavy-tailed distribution. Measured on the full vocabulary, it grows with the model's parameter count. However, when we remove the hub, the heavy tail disappears and the measured dimension collapses to a narrow range for eleven models, from GPT-2 to models such as K3 and GLM-4.7. The hub acts as a switch: a few hundred rows are enough to fully inflate the estimate. We reproduced an experiment stating that the intrinsic dimension (ID) of Pythia's token-embedding table grows with the parameter count, from 27 to 122 between 160M and 12B parameters. We show that this result disappears when the hub is removed: the table then reads 10 to 17 at every size. The hub contains a subset of the population that under-trained-token detectors flag, but on Pythia the hub that we detected and removed as a whole was updated during training: what seem to characterize these rows is simply their length, not an absence of updates. Finally, we show that normalizing the rows instead of removing them gives the same lower reading.