Zipf's Law

Follow this topic Ask about this topic

Latest papers

8 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Exact Scaling Laws for a Random-Features Bigram Model
Oct 13, 2026Julian Enoch, Elvis DohmatobZipf's Law Random Feature Methods
Concordia University · Mila – Quebec AI Institute
A Layered Simplex Architecture for Large Alphabets
Aug 20, 2026Meir Feder, Yaniv Fogel, Ruediger UrbankeZipf's Law Bayesian Inference
School of Electrical and Computer Engineering Tel Aviv University, Tel Aviv, Israel · School of Computer and Communication Sciences EPFL, Lausanne, Switzerland
Three Buddhist Vocabularies: Computational Stylometry of the English Pali Canon across Sutta, Vinaya, and Abhidhamma
Jun 24, 2026Joy BoseComputational Literary Studies Zipf's Law
Independent Researcher, Bengaluru, India
Recovering the Zipfian Distribution in Unsupervised Term Discovery
Jun 9, 2026Danel Slabbert, Simon Malan, Herman KamperZipf's Law Clustering
Electrical and Electronic Engineering, Stellenbosch University, South Africa
Universal statistical laws governing culinary design
Apr 30, 2026Ganesh Bagler, Gopal Krishna Tewari, Aditya Raj Yadav +5Zipf's Law
Department of Computational Biology · 2Infosys Center for Artificial Intelligence · Center of Excellence in Healthcare +3
Non-Zipfian Distribution of Stopwords or Function Words and Subset Selection Models
Mar 5, 2026Wentian Li, Oscar FontanelliZipf's Law Heavy-Tailed Distributions
Department of Applied Mathematics and Statistics, Stony Brook University, Stony Brook, NY, USA · The Robert S. Boas Center for Genomics and Human Genetics, The Feinstein Institutes for Medical Research, Northwell Healths, Manhasset, NY, USA · Methods Lab, Facultad Latinoamericana de Ciencias Sociales, México
From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis
Dec 15, 2025Łukasz DębowskiZipf's Law Neural Scaling Laws
Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland
Multiperiodic Processes: Ergodic Sources with a Sublinear Entropy
Feb 17, 2023Łukasz DębowskiZipf's Law
Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland

Oct 13, 2026J/K move · Enter open · S save

Exact Scaling Laws for a Random-Features Bigram Model

Julian Enoch, Elvis Dohmatob

Concordia University · Mila – Quebec AI Institute

Zipf's Law Random Feature Methods Language Model Scaling Laws Scaling Laws Neural Scaling Laws

Open paper AskPDF arXiv

Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even in simple settings. In this work, we study capacity-limited scaling in a bottlenecked bigram model with power-law (Zipf) token statistics, viewed as an analytically tractable toy model of a large language model. Fixed random embeddings serve as a proxy for representational capacity and induce an explicit bottleneck of width, allowing us to derive scaling laws for prediction regret as a function of model width, vocabulary size, and the Zipf exponent. Our results complement recent theoretical accounts based on optimization dynamics by isolating scaling due purely to representational constraints. In the appropriate regime, the theory recovers the functional form of classical neural scaling laws and suggests that heavy-tailed language statistics may underlie both optimization- and capacity-driven scaling. Experiments natural language datasets (WikiText2, OpenWebText) closely match the predicted curves.