Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system
Authors: Zhongxin Yang, Chun Bao, Yuanwei Bin, Xiang I. A. Yang, Shiyi Chen
Organizations: College of Engineering, Peking University, Beijing, 100871, China · Shenzhen Tenfong Technology Co., Ltd., Shenzhen 518000, China · Mechanical Engineering, Pennsylvania State University, State College, PA, 16802, USA
Natural language is a complex system that exhibits robust statistical regularities. Here, we represent text as a trajectory in a high-dimensional embedding space generated by transformer-based language models, and quantify scale-dependent fluctuations along the token sequence using an embedding-step signal. Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to 5/3 over an extended frequency range. This scaling is observed consistently in contextual embeddings from both human-written and AI-generated text, but is absent in static word embeddings and is disrupted by randomization of token order. These results show that the observed scaling reflects multiscale, context-dependent organization rather than lexical statistics alone. By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales, and provide a quantitative, model-agnostic benchmark for studying complex structure in language representations.
Figures & tables
Figure 1: (a) Power spectra of contextual embedding-step sequences. The spectra are normalized by the variance σ2=∫01E(f)d(f/fmax) . The dashed line indicates the +5/3 slope. The four panels are Chinese/BERT (top left); English/GPT-OSS (top right); German/BERT (bottom left); Japanese/BERT (bottom right). (b) Polar heatmaps of normalized per-dimension power spectra. The radial axis represents frequency (log scale), and the angular axis represents embedding dimension index. Top row: AI, bottom row: human; From right to the left: Chinese, English, German, Japanese.
Figure 2: (a) Summary of fitted scaling exponents α . ‘CN’, ‘EN’, ‘DE’, and ‘JP’ denote Chinese, English, German, and Japanese, respectively. Error bars show variability across embedding dimensionalities (one standard deviation). (b) Fitted power-law exponent versus transformer layer index for human-authored text. The inset shows a representative mid-layer spectrum (highlighted by the green box) with a fitted power-law exponent α≈1.47 .