Entropy in Large Language Models

Marco Scharringhausen

cs.CLFeb 23, 2026

Entropy in Large Language Models

Authors: Marco Scharringhausen

Organizations: Carl von Ossietzky Universität Oldenburg Oldenburg, Germany

Abstract

In this study, the output of large language models (LLM) is considered an information source generating an unlimited sequence of symbols drawn from a finite alphabet. Given the probabilistic nature of modern LLMs, we assume a probabilistic model for these LLMs, following a constant random distribution and the source itself thus being stationary. We compare this source entropy (per word) to that of natural language (written or spoken) as represented by the Open American National Corpus (OANC). Our results indicate that the word entropy of such LLMs is lower than the word entropy of natural speech both in written or spoken form. The long-term goal of such studies is to formalize the intuitions of information and uncertainty in large language training to assess the impact of training an LLM from LLM generated training data. This refers to texts from the world wide web in particular.

Figures & tables

Name	#Parameters / B	#Active parameters / B
Ministral-3-14B-Instruct-2512	14
GPT-OSS-120b	117	5.1
MiniMax-M2	230	10
Apertus-8B-Instruct-2509	8
Qwen3 235B - A22B	235	22
Qwen3-Coder-30B-A3B-Instruct	30	3

Table 1: Models of this study. Number of parameters are listed in billions.

Model	Word count / $10^{6}$
OANC spoken	3.1
OANC written	11.5
LLM accumulated, $T=0.3\dots T=1.5$	101.5
LLM accumulated, $T=0.3$	24.1
LLM accumulated, $T=0.5$	24.7
LLM accumulated, $T=0.7$	26.5

Table 2: Model output sizes of this study. Counts are number of million words in the respective data. Depending on the model and the temperature, the output size varies. Therefore, usually a range of word counts is listed. "Accumulated" sizes are concatenations of all output files for the Mistral or the Blablador model family. "LLM accumulated" contains all models and all temperatures

T=0.3\dots 0.7"

Figure 1: Entropy rates for the OANC and the output of all LLMs (both Mistral and Blablador model family concatenated) for

T=0.3\dots 1.5

as well as for

T=0.3\dots 1.0

. The difference between

T_{\rm{max}}=1.0

and

T_{\rm{max}}=1.5

is small (0.618 and 0.574, respectively), probably since the amount of LLM output with somewhat erroneous entropy (see 2 ) is small. See table 3 for numeric values of the entropy rates.

Figure 2: Entropy rates for the two model families Mistral and Blablador, distinguished by model temperature. The Blablador output for

T=1.5

leads to very little output (140.000 words). This leads to an estimate of the entropy rate that is most likely wrong, see 2.4 and the comment about sample sizes therein. The impact of model temperature on the entropy rate of the output is small. The values for the Mistral model family (

0.450\dots 0.475

) are lower than for the Blablador family (

0.530\dots 0.587

). It is not clear, however, wether this is a significant effect.

Source	Entropy / bit/word
OANC written	0.716
OANC spoken	1.255
LLM accumulated, $T_{max}=1.5$	0.574
LLM accumulated, $T_{max}=1.0$	0.618
Mistral accumulated, $T=0.3$	0.450
Mistral accumulated, $T=0.5$	0.433

Table 3: Source entropy for different sources.

Repeated Sequences Reveal Gaps between Large Language Models and Natural Language
May 24, 2026Kumiko Tanaka-IshiiLLM Evaluation Language Model Generation Evaluation
Waseda University, Japan.
Semantic Chunking and the Entropy of Natural Language
Feb 13, 2026Weishun Zhong, Doron Sivan, Tankut Can +2
School of Natural Sciences, Institute for Advanced Study, Princeton, NJ, 08540, USA · Department of Brain Sciences, Weizmann Institute of Science, Rehovot, 76100, Israel
The Probabilistic Structure of Large Language Models
Sep 20, 2026Adnan AboulalaâLanguage Modeling Autoregressive Language Modeling

May 24, 2026J/K move · Enter open · S save

Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

Kumiko Tanaka-Ishii

Waseda University, Japan.

LLM Evaluation Language Model Generation Evaluation Sequence Modeling

Open paper AskPDF arXiv

Evaluating whether large language models (LLMs) capture the structure of natural language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text. We propose a complementary evaluation framework based on repeated subsequences. By analyzing their distribution across scales and relating it to higher-order Rényi entropies, we probe how texts reuse previously established structure under finite-length conditions. Experiments on human-written texts and length-matched GPT-generated texts show that, while power-law models can describe restricted ranges of block length, the observed entropy growth is often equally or better characterized by logarithmic--power forms. Across datasets, natural language exhibits stable entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts. In contrast, GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size. These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic that reveals systematic differences in long-range organization, distinguishing natural language from state-of-the-art LLM outputs beyond surface-level fluency.

Abstract

Explore similar work

Repeated Sequences Reveal Gaps between Large Language Models and Natural Language

Semantic Chunking and the Entropy of Natural Language

The Probabilistic Structure of Large Language Models