cs.LGOct 13, 2026

Exact Scaling Laws for a Random-Features Bigram Model

Authors: Julian Enoch, Elvis Dohmatob

Organizations: Concordia University · Mila – Quebec AI Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Understanding neural scaling laws is increasingly important for theory and practice, yet first-principles explanations remain limited even in simple settings. In this work, we study capacity-limited scaling in a bottlenecked bigram model with power-law (Zipf) token statistics, viewed as an analytically tractable toy model of a large language model. Fixed random embeddings serve as a proxy for representational capacity and induce an explicit bottleneck of width, allowing us to derive scaling laws for prediction regret as a function of model width, vocabulary size, and the Zipf exponent. Our results complement recent theoretical accounts based on optimization dynamics by isolating scaling due purely to representational constraints. In the appropriate regime, the theory recovers the functional form of classical neural scaling laws and suggests that heavy-tailed language statistics may underlie both optimization- and capacity-driven scaling. Experiments natural language datasets (WikiText2, OpenWebText) closely match the predicted curves.

Figures & tables

DatasetVocabulary size ddSample size nnZipf exponent α\alpha
WikiText-23327720519100.9354
OpenWebText5025721108247210.9443
Table 1: Global parameters and descriptive statistics for real-world natural language datasets. In our experiments, we sample subsets of these datasets, restricting both the vocabulary size dd and the sample size nn .
Figure 1: Capacity scaling on WikiText-2. Converged normalized excess risk (regret) rn,d(N)r_{n,d}(N) ( 7 ) for the random-features bigram model, as the embedding width is scaled as N=γdβN=\gamma d^{\beta} (( 19 )). Left: vary the width multiplier γ\gamma at fixed exponents β\beta . Right: vary the exponent β\beta at fixed multipliers γ\gamma . Solid curves (markers) show empirical means over random projections; shaded regions indicate variability across trials. Dashed curves are the theoretical predictions from our Theorem 2 . Notice the excellent match.
Figure 2: Capacity scaling on OpenWebText2. Converged normalized excess risk (regret) rn,d(N)r_{n,d}(N) ( 7 ) for the random-features bigram model with a rank- NN embedding bottleneck, where N=γdβN=\gamma d^{\beta} (( 19 )). Left: vary γ\gamma at fixed β\beta . Right: vary β\beta at fixed γ\gamma . Solid curves (markers) show empirical means over random projections; shaded regions show trial variability. Dashed curves are the corresponding theoretical predictions from our Theorem 2 . Notice the excellent match.
More in this topic

Zipf's Law

See all 8 papers

Explore similar work

CardsList

  1. Deriving Neural Scaling Laws from the statistics of natural language

    Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Language Model Scaling LawsNeural Scaling Laws

    Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland +1

  2. From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

    Dec 15, 2025Łukasz DębowskiZipf's LawNeural Scaling Laws

    Institute of Computer Science, Polish Academy of Sciences, ul. Jana Kazimierza 5, 01-248 Warszawa, Poland

  3. Smooth Scaling Laws Hide Stepwise Token Learning

    Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Language Model PretrainingLanguage Model Scaling Laws

    Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university