Do Quantum Models Scale Like LLMs?

David S. Berman; Ying-Jer Kao; Roger G. Melko; Alexander G. Stapleton

cs.LGSep 17, 2026

Do Quantum Models Scale Like LLMs?

Authors: David S. Berman, Ying-Jer Kao, Roger G. Melko, Alexander G. Stapleton

Organizations: Centre for Theoretical Physics, Queen Mary University of London, Mile End Road, London E1 4NS, United Kingdom · Department of Physics, National Taiwan University, Taipei 10607, Taiwan · Department of Physics and Astronomy, University of Waterloo, Waterloo, Ontario N2L 3G1, Canada · Perimeter Institute for Theoretical Physics, Waterloo, Ontario N2L 2Y5, Canada

Abstract

In this work, we study the neural scaling laws of RydbergGPT, an autoregressive transformer model trained on qubit projective measurement data gathered from interacting Rydberg atom arrays. The quantum system is known to exhibit a finite-size remnant of a critical point as the laser detuning parameter is varied. We find that near the critical point the transformer loss as a function of training dataset size is well described by a power-law with a loss floor correction. However, away from criticality the quality of the power-law description is substantially reduced. We then compare the statistical structure of both Rydberg measurements and natural-language corpora using an entropy-normalised, finite sample corrected mutual information "two-point" function. We find that near-critical statistics of the two point functions are closest to those observed in natural-language, whilst other qubit configurations far from the critical point have two-point functions that decay more rapidly. This supports the hypothesis that multi-scale dependence contributes to stable neural scaling, and that scaling behaviour should be viewed as a property of the model-data pair.

Figures & tables

Figure 1: Representative randomly selected occupation-basis configurations from an open $6\times 6$ Rydberg array at the four labelled detunings [ 10 ] . Light sites are in the ground state and dark sites represent Rydberg excitations. The samples were selected with a fixed pseudorandom seed from the same data archive used for the data-scaling and mutual-information analyses. They are individual measurements and are shown for visual orientation only.

Figure 2: Fixed-model RydbergGPT data-scaling curves on the two sides of the critical detuning point. The system parameters are

L=6

R_{b}/a=1.15

and

\beta\Omega=8

. Solid and dashed curves denote the seed-averaged training and validation losses, respectively; shaded regions show one standard deviation across seeds. Dotted and dash-dotted curves are fits to Equation ( 11 ). The scientific-notation multiplier is shown once per vertical axis.

$\delta/\Omega$	series	$B_{D}$	$\gamma_{D}$	$\mathcal{L}_{\infty}$	$R^{2}$
1.05	train	0.0409	0.736	0.4844	0.998
1.05	validation	0.00980	0.656	0.4839	0.989
1.17	train	0.0647	0.950	0.4697	0.997
1.17	validation	0.0121	0.791	0.4683	0.983

Table 1: Aggregate fits to Equation ( 11 ). Each fit uses ten data fractions. The validation loss floor is lower at

\delta/\Omega=1.17

than at

1.05

, while the validation exponent remains of order unity on both sides of the critical point.

Figure 3: Normalised aggregate loss curves and fitted forms for the two detunings in Figure 2 . Points are observed seed means and lines are fits to Equation ( 11 ). Each series is normalised by its largest observed loss for display only.

Figure 4: Validation-fit parameters for the six retained detunings. Faint points denote fits to individual random seeds, while circles and error bars indicate the corresponding mean and standard deviation at each detuning. The dashed vertical line marks the critical point at

\delta/\Omega=1.10

Figure 5: Entropy-normalised excess mutual-information profiles for the Rydberg occupations (left) and natural language symbols (right). Markers show the signed permutation-corrected estimates in Equation ( 13 ), and lines join successive separations. The symmetric-logarithmic vertical scale is linear for

|\widehat{\rho}_{\mathrm{MI}}|\leq 10^{-4}

, so small negative finite-sample estimates remain visible. Separations are divided by the largest available cutoff within each system. The left panel contains the six open

6\times 6

Rydberg detunings. The right panel contains the AG News and Yahoo answers profiles in the fixed character-level representation; the dashed line is their pointwise arithmetic mean at each separation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Fit quality of the mutual-information power-law ansatz over detuning. Points give the

R^{2}

of the log–log fit in Equation ( 17 ); the dashed line marks the critical point.

Neural Scaling Laws

Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.

See all 83 papers

Deriving Neural Scaling Laws from the statistics of natural language
Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Language Model Scaling Laws Neural Scaling Laws
Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland +1
Exact Scaling Laws for a Random-Features Bigram Model
Oct 13, 2026Julian Enoch, Elvis DohmatobZipf's Law Random Feature Methods
Concordia University · Mila – Quebec AI Institute
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
May 22, 2026Xu Ouyang, Deyi Liu, Yuhang Cai +5Language Model Scaling Laws
1ByteDance Seed · University of Virginia · University of California, Berkeley

Feb 7, 2026J/K move · Enter open · S save

Deriving Neural Scaling Laws from the statistics of natural language

Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart

Theoretical and Scientific Data Science, International School for Advanced Studies (SISSA), Trieste, Italy · Department of Applied Physics, Stanford University, Stanford, California · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland+1

Language Model Scaling Laws Neural Scaling Laws

Open paper AskPDF arXiv

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

Abstract

Neural Scaling Laws

Explore similar work

Deriving Neural Scaling Laws from the statistics of natural language

Exact Scaling Laws for a Random-Features Bigram Model

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws