Authors: Ortal Hadad, Edoardo Loru, Jacopo Nudo, Niccolò Di Marco, Matteo Cinelli, Walter Quattrociocchi
Organizations: Department of Computer Science, Sapienza University of Rome. · Department of Computer, Control and Management Engineering, Sapienza University of Rome. · Department of Legal, Social, and Educational Sciences, Tuscia University.
Large language models generate text through probabilistic sampling from high-dimensional distributions, yet how this process reshapes the structural statistical organization of language remains incompletely characterized. Here we show that lossless compression provides a simple, model-agnostic measure of statistical regularity that differentiates generative regimes directly from surface text. We analyze compression behavior across three progressively more complex information ecosystems: controlled human-LLM continuations, generative mediation of a knowledge infrastructure (Wikipedia vs. Grokipedia), and fully synthetic social interaction environments (Moltbook vs. Reddit). Across settings, compression reveals a persistent structural signature of probabilistic generation. In controlled and mediated contexts, LLM-produced language exhibits higher structural regularity and compressibility than human-written text, consistent with a concentration of output within highly recurrent statistical patterns. However, this signature shows scale dependence: in fragmented interaction environments the separation attenuates, suggesting a fundamental limit to surface-level distinguishability at small scales. This compressibility-based separation emerges consistently across models, tasks, and domains and can be observed directly from surface text without relying on model internals or semantic evaluation. Overall, our findings introduce a simple and robust framework for quantifying how generative systems reshape textual production, offering a structural perspective on the evolving complexity of communication.
Figures & tables
Figure 1: (A) Relationship between vocabulary entropy and compression ratio for texts generated from word distributions with fixed entropy. The colored points show the average values for Humans and LLMs. The bars indicate one standard deviation from the mean. The inset displays the density distribution of document length (number of words) for human-written and LLM-generated texts. (B) Compression ratios distribution for human-written texts, LLM-generated texts, and randomly generated texts. Higher compression ratios correspond to lower compressibility. Each group contains 1000 documents. (C) Average compression ratio of human-written and LLM-generated texts according to the number of sentences in the text. The shaded area shows the interquartile range of the distribution of compressions. For LLM-generated text, as the length increases, compressibility also increases, unlike human text.
Figure 2: Distribution of structural and compression-based features across human-written and LLM-generated texts in the Human–AI Parallel Corpus.
Figure 3: Global feature importance based on mean absolute Shapley values for the Gradient Boosting Classifier. Bars indicate the average magnitude of each feature’s contribution to the predicted probability of the Human class across the test set.
Figure 4: (A) Average compression ratio of Wikipedia and Grokipedia page texts as a function of the number of sentences in the text. The shaded area represents the interquartile range of the compression ratio distribution. (B) Distribution of Conditional Compression Ratio, Normalized Word-Level Entropy, Mean Repetition Distance, and Repetition Distance Variability across texts from Wikipedia and Grokipedia pages.
Figure 5: (A) Average compression ratio of Moltbook and Reddit comments as a function of the number of sentences in the text. The shaded area represents the interquartile range of the compression ratio distribution. (B) Distribution of Normalized Compression Distance, Prefix Ratio Trend, Unique Word ratio, and Conditional Compression ratio across Moltbook and Reddit posts.
Evaluating whether large language models (LLMs) capture the structure of natural language beyond local fluency remains an open challenge. Existing evaluation methods, largely based on task performance or short-context behavior, provide limited insight into the long-range statistical organization of generated text. We propose a complementary evaluation framework based on repeated subsequences. By analyzing their distribution across scales and relating it to higher-order Rényi entropies, we probe how texts reuse previously established structure under finite-length conditions. Experiments on human-written texts and length-matched GPT-generated texts show that, while power-law models can describe restricted ranges of block length, the observed entropy growth is often equally or better characterized by logarithmic--power forms. Across datasets, natural language exhibits stable entropy-growth patterns over accessible ranges, with consistent average behavior despite variability across individual texts. In contrast, GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size. These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic that reveals systematic differences in long-range organization, distinguishing natural language from state-of-the-art LLM outputs beyond surface-level fluency.
This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.
While factual correctness and task-performance have been in focus of Large Language Model (LLM) research for a long time, the fundamental question of how human-like generated texts are on a linguistic level has been underexplored. From a corpus-linguistic perspective, language production is inherently context-dependent, with distinct communicative contexts giving rise to differences in frequencies and co-occurrence patterns of linguistic features. A text failing to adhere to these patterns can be content-wise correct, but still be unfavorable to human readers. In this work, we propose a context-aware evaluation framework in which human-likeness is assessed using a two-sample problem between the linguistic feature distribution of a human reference corpus for a given register and a corresponding LLM-generated corpus. We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber, which are commonly applied in corpus linguistics. In our experiments, we compare seven instruction-tuned, open-source models across five English-language datasets spanning distinct registers against a human baseline. While across all tested setups, LLMs deviate from the human baseline, which models are closest to human language depends on the register and is not dictated by model size.
Björn Nieth, Marianna Gracheva, Michaela Mahlberg +2
Department Artificial Intelligence in Biomedical Engineering (AIBE), FAU Erlangen-Nürnberg, Germany · Department of Digital Humanities and Social Studies (DHSS), FAU Erlangen-Nürnberg, Germany · University of Birmingham, United Kingdom +2