The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions

Jarosław Hryszko

cs.CLMar 29, 2026

The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions

Authors: Jarosław Hryszko

Organizations: Institute of Computer Science, Faculty of Mathematics and Computer Science, Jagiellonian University, Krak´ow, Poland

Abstract

Language models cannot be random. This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution, and measures it systematically across 31,200 generations spanning seven models, two architectures (transformer and state space), nine prompt categories, three temperatures, and five languages. Under semantically neutral prompts (empty strings, random characters, nonsense syllables) transformers still exhibit ED of approximately 0.30, meaning that 88-93% of the non-randomness observed under semantic prompts is intrinsic to the learned weights rather than induced by context. Three transformer families (Gemma, Llama, Qwen) converge on nearly identical ED values despite different training data and vocabularies. A state space model (Mamba2) reveals a qualitatively different regime: twice the ED, three times lower within-sequence variance, and massive sensitivity to temperature (r = -0.78) where transformers are nearly immune (r < 0.05). Cross-lingual experiments with Qwen-32B show a stable gradient across five languages (English, Japanese, Chinese, Polish, Arabic) that does not correlate with token fertility and persists when two languages sharing an identical tokeniser subset are compared. These findings establish a structural lower bound on randomness in pretrained language models, characterise how this bound differs across architectures, and demonstrate that language itself modulates the bound independently of tokenisation.

Figures & tables

Model	Arch.	Params	Vocab	Phase
Phi-3-mini-4K	TF	3.8B	32K	Pilot
Mistral-7B	TF	7B	32K	Pilot
Llama-3-8B	TF	8B	128K	Pilot
Gemma-2-27B-IT	TF	27B	256K	Main
Qwen-2.5-32B-Instruct	TF	32B	152K	Main
Llama-3.3-70B-Instruct	TF	70B	128K	Main

Table 1: Models used in the study. TF = transformer, SSM = state space model. All models were run locally using llama.cpp ( Gerganov, 2023 ) in GGUF format with Q4_K_M quantization on dual NVIDIA Tesla P40 GPUs (24 GB VRAM each).

Figure 1: Decomposition of ED into intrinsic (neutral prompt) and semantic components. Percentages indicate the intrinsic fraction. Across transformer families, 87–89% of observed ED persists under semantically empty inputs. For Mamba2, the fraction reaches 93%.

Figure 2: ED across all models. Pilot models (left) show lower ED at smaller scales. The three main transformer families (center) converge on nearly identical values despite different training data, architectures, and vocabulary sizes. Mamba2 (right, SSM architecture) exhibits ED approximately twice that of transformers.

Figure 3: Temperature sensitivity of ED. (a) Transformers show near-flat ED across temperatures, with a slight positive trend under semantic prompts. (b) Mamba2 shows a steep decline in ED with increasing temperature (

r=-0.78

). Note the different

y

-axis scales.

Figure 4: Multilingual ED gradient for Qwen-2.5 32B on Wikipedia prompts. All pairwise differences are significant (

p<10^{-6}

). The gradient does not track token fertility or vocabulary allocation.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Test	Description	Result
F1	ED $\neq 0$ (one-sample $t$ -test)	$p<10^{-6}$ in all partitions
F2	Domain differences (Kruskal-Wallis)	Significant in all models
F3	Size effect (regression on $\log$ params)	$R^{2}=0.276$ , $p\approx 0$
F4	Temperature effect (ANOVA)	TF: negligible; SSM: $p\approx 0$
F5	Autocorrelation (Durbin-Watson)	Weak negative (TF neutral)
F6	Intrinsic fraction (semantic vs neutral)	86.8–92.6% intrinsic

Appendix

Table 2: Summary of pre-registered falsification tests. F8 identified a small but statistically significant generation-order drift, likely attributable to hardware state or numerical precision changes during long experimental runs. Shuffling temperature and prompt order across runs is recommended for future work.

Category	TF Mean ED	Mamba2 Mean ED
Random ASCII	0.268	0.540
Nonsense syllables	0.286	0.535
Explicit randomness	0.308	0.665
Neutral stubs	0.312	0.659
Empty string	0.314	0.704

Appendix

Table 3: ED by neutral prompt category. TF Mean is the average across three main transformers. Note the reversed ordering for Mamba2: empty strings produce the highest ED, consistent with the interpretation that absence of context strengthens default distributional preferences in the SSM architecture.

Domain	Gemma-2	Qwen-2.5	Llama-3.3	Mamba2
Code	0.356	0.360	0.361	0.741
News	0.342	0.327	0.338	0.648
Fiction	0.339	0.336	0.337	0.675
Wikipedia	0.337	0.329	0.337	0.662

Appendix

Table 4: ED by semantic prompt domain across models. Code consistently produces the highest ED in all architectures. Mamba2 domains are limited to the four categories included in the SSM experimental runs.

Language	ED	Fertility	Vocab alloc.	Unique gen.	Cohen’s $d$
English	0.329	0.221	104,969	1,996	–
Japanese	0.388	0.747	38,877	1,358	2.4
Chinese	0.395	0.750	37,115	1,808	3.1
Polish	0.401	0.352	104,969	1,815	3.7
Arabic	0.408	0.413	15,429	1,712	4.1

Appendix

Table 5: Multilingual ED with tokenizer properties. Fertility = tokens per source character. Vocab alloc. = number of tokenizer entries allocated to the language’s script. Unique gen. = mean unique tokens per generation. Cohen’s

d

is relative to English. Note that English and Polish share identical vocabulary allocation yet differ by

d=3.7

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 504 papers

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
Jul 29, 2026Poulami Ghosh, Preethi JyothiMultilingual Language Model Evaluation Language Model Robustness
*IIT Bombay, India
Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning
Dec 14, 2024Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9LLM Interpretability Distributional Semantics
Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3
Stochasticity in Tokenisation Improves Robustness
Apr 17, 2026Sophie Steger, Rui Li, Sofiane Ennadir +4Adversarial Attacks on LLMs Neural Network Robustness
Institute of Signal Processing and Speech Communication, Graz University of Technology, Graz, Austria · ELLIS Institute Finland & Aalto University, Espoo, Finland · King AI Labs, Microsoft Gaming +2

Jul 29, 2026J/K move · Enter open · S save

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

Poulami Ghosh, Preethi Jyothi

*IIT Bombay, India

Multilingual Language Model Evaluation Language Model Robustness Tokenization in Language Models

Open paper AskPDF arXiv

Despite the existence of exponentially many valid tokenizations for a given string, language models operate on a single canonical sequence deterministically produced by the tokenizer, leaving the broader tokenization space largely uncharacterized. In this paper, we investigate this overlooked space by studying the behavior of language models under non-canonical tokenizations across diverse languages. For English, prior work shows that models are largely invariant to alternative tokenizations that represent the same underlying string. We ask whether this invariance generalizes to other languages beyond English. We conduct a multilingual study across 27 languages spanning diverse scripts and evaluate LLM behavior under alternative tokenizations across six downstream tasks. We find that tokenization invariance does not generalize: model behavior varies substantially across languages with instruction-tuned models exhibiting an average relative performance drop of 23.7% for Llama-3.1-8B, 11.4% for Qwen3-8B, and 9.9% for Gemma-3-12B. The variation of tokenization invariance is systematic across languages. Languages that exhibit higher token fragmentation show significantly greater sensitivity to non-canonical tokenizations. Our study of tokenization robustness serves as a diagnostic of how tightly a model is coupled to its tokenizer. These results demonstrate that tokenization robustness is not a universal property of language models, but depends strongly on the language and its interaction with the tokenizer. We also show that LoRA fine-tuning with multi-tokenization training data provides an effective mitigation for tokenization sensitivity. Fine-tuning on English alone improves tokenization robustness across languages, while systematically sampling diverse non-canonical tokenizations achieves the strongest overall performance.

Abstract

Language Modeling

Explore similar work

Language Models are not Equally Robust to Non-Canonical Tokenization across Languages

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Stochasticity in Tokenisation Improves Robustness