cs.LGSep 11, 2026

One Spectrum, Two Resources: Data-Memory Scaling in Autoregressive Prediction

Authors: Chiwun Yang, Xiaoyu Li

Organizations: City University of Hong Kong · University of New South Wales

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

How much learned memory is needed to benefit from more data? We show that the two resources are governed by one predictive-energy spectrum in a positive-entropy autoregressive retrieval source. Each coordinate contributes its query probability times the squared radius of its unknown logit. Writing μμ for the resulting energy spectrum, we prove the minimax law Rvalue∗(n,B)≍RΦμ(n−1)+Φμ(τB),Φμ(t)=∫min⁡{x,t} μ(dx),\mathfrak R^*_{\rm value}(n,B)\asymp_R Φ_μ(n^{-1})+Φ_μ(τ_B), Φ_μ(t)=\int\min\{x,t\}\,μ(\mathrm dx), for nn prediction blocks and a learned state with at most 2B2^B values. Data set the resolution 1/n1/n; memory sets the level τBτ_B reached by optimal bit allocation. The complete curve also recovers the positive spectrum. Energy-dimension pairing is essential: two causal sources with identical block-energy and block-dimension marginals have different data and memory exponents. A masked query-key attention head learns the route and values, realizing the law with explicit routing, format, and arithmetic errors. Further results give exponent-adaptive allocation, finite-precision realization, and compute-precision laws under two-sided arithmetic assumptions. Experiments recover the data-memory collapse and coupling exponents, explain the routing and allocation mechanisms, and examine weight-only quantization across six pretrained-model scales.

Figures & tables

StudyObservations and parametersCode and riskConclusion
Zhu and Lafferty (2014)Gaussian sequence; deterministic means in a Euclidean ballArbitrary codebook, mbmb bits; normalized squared errorSharp asymptotic estimation–storage trade-off as m→∞m\to\infty .
Luschgy and Pagès (2004)Gaussian random element; covariance eigenvaluesNcN_{c} -point quantizer; expected squared Hilbert errorHigh-resolution distortion from eigenvalue decay; B=log⁡2NcB=\log_{2}N_{c} .
Allen-Zhu and Li (2025)Random discrete factual tuples; trained language model weightsFinite weight state; factual generation negative log-likelihoodAchieved loss lower-bounds the information that the weights must retain.
Pan et al. (2025)Hierarchical syntax–knowledge mixture; frequency-ranked knowledgeBayesian/universal code; predictive redundancy under data or model capacityPower-law knowledge frequencies generate data and model scaling laws.
Goel et al. (2026)Gaussian linear regression; softmax attentionExpected empirical-gradient oracle; squared loss; no bit constraintGeometric optimization with a finite-context statistical term.
This workRandom coordinate observations; independently bounded scalar logitsAny 2B2^{B} -state learner; held-out excess Bernoulli CEUniform (n,B)(n,B) risk through one spectrum; pairing changes both exponents.
Table 1: Closest results and the resource question they answer. Here mm is Gaussian sequence dimension, NcN_{c} is codebook size, and bb is bits per Gaussian coordinate, BB is total retained bits, and nn is the number of training blocks in our source. CE denotes cross-entropy. The last row identifies the observation model and finite-resource conclusion needed here.
Figure 1: Retained precision sets resource ceilings and error floors. Left: analytic bit-operation ratio with retained high-precision work. Right: risk–cost laws under the stated arithmetic model; fixed softmax precision creates a numerical floor.
Figure 2: One spectrum organizes data and memory. Four fixed sources, each with 256 energy groups, at K=32K=32 : held-out excess CE versus Gμ(n,B)=Φμ(n−1)+Φμ(τB)G_{\mu}(n,B)=\Phi_{\mu}(n^{-1})+\Phi_{\mu}(\tau_{B}) . Blue circles: weighted-average SGD; orange squares: the oracle-coordinate clipped estimator. Each curve averages 30 budget pairs into six bins chosen only from GμG_{\mu} , within each seed. Bands: 95% BCa over eight paired seeds.
Figure 3: Pairing separates data and memory; routing enables learning. (a,d) Fixed-source data sweeps for χ=1,2\chi=1,2 ; blue/orange denote easy/hard. (b,e) Their complementary storage sweeps, evaluated by population excess CE. (c) Squared leakage versus row visits, K=8/32K=8/32 (blue/green). (f) Finite-source excess CE with learned/known/frozen routes (blue circles/green triangles/orange squares), without an added asymptotic tail. Intervals: 95% BCa over 16 data seeds (a,d), 16 source amplitudes (b,e), or eight paired seeds (c,f).
Figure 4: Data and deployed storage in Pythia. (a–f) 70M, 160M, 410M, 1B, 1.4B, 2.8B test CE after 2262^{26} extra tokens. Blue: 2/3/4/6/8-bit and BF16 (left to right); dashed orange: additive fits to first-three-budget 4/8-bit/BF16 validation. All points hold out data; 6-bit also holds out format. Shared log-y; bands: 95% BCa, eight paired training seeds.
Table 2: Deployment, pairing, and learning mechanisms. (a) HQQ-g32 minus BF16 CE: 96 WikiText-103 windows, 512 LAMBADA items. (b) Storage slopes, 16 seeds; E/H: easy/hard. (c,d) SGD decomposition/interventions: 10−310^{-3} nats, eight seeds; (d) has (χ,γ)=(1,.5)(\chi,\gamma)=(1,.5) , M=64M=64 , K=32K=32 , bit ratio .5 (Table 3 : 1). ±\pm : rounded 95% BCa envelopes (Unif.: constant across seeds). Bold/underline: lowest/second-lowest in (a,d).
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Six-profile exponent agreement. (a) Data-limited and (b) memory-limited fitted exponents for profile indices (0.5,0),(0.5,0.5),(1,0),(1,0.5),(2,0),(2,0.5)(0.5,0),(0.5,0.5),(1,0),(1,0.5),(2,0),(2,0.5) . Green, blue, and orange denote low, middle, and high fixed resource ratios; gray diamonds give the theoretical exponent. Bands are 95% BCa intervals over sixteen paired slopes: eight seeds for each of two frequency–amplitude decompositions (constant and alternating logit radius).
Figure 6: Allocation beyond exact power laws. Left, center, and right show a broken power law, logarithmic correction, and perturbed ordering, respectively, at (χ,γ,M)=(1,0.5,256)(\chi,\gamma,M)=(1,0.5,256) . Each panel compares known-energy, adaptive, and uniform allocation. Known-energy uses the true qiq_{i} ; empirical allocation uses the count-corrected scores in Appendix R . Bands are 95% BCa intervals over eight paired seeds.
Figure 7: Complete additional-data and deployment-format scan. (a–f) Pythia 70M, 160M, 410M, 1B, 1.4B, and 2.8B test CE after four nested additional-token budgets. Curves show complete group-128 2/3/4/6/8-bit deployments and BF16; the shared logarithmic CE axis compares absolute loss and keeps low-bit collapse visible. Bands are 95% BCa intervals over eight paired training seeds evaluated on the same 96 test windows.
Figure 8: Stored precision and retained high-precision work. Left: format distortion versus nominal bits for shared-scale, block-scale, and adaptive-bit representations of the causal source. Right: analytic bit-operation ratios for 8-bit and 4-bit matrix paths with FP32 softmax retained. The source-format curves use paired 95% BCa intervals; the right-hand curves evaluate the analytic operation-count formula.
Figure 9: Finite precision across six Pythia scales. Left and center: paired cross-entropy changes on WikiText-103 (96 paired windows) and LAMBADA (512 paired examples), respectively. Right: parameter-weighted relative Frobenius error for the two HQQ formats, averaging ∥W^−W∥F/∥W∥F\|\widehat{W}-W\|_{F}/\|W\|_{F} over layers with weights proportional to their parameter counts. Blue circles, green triangles, orange squares, gray down-triangles, and magenta diamonds denote INT8-channel, INT8-tensor, INT4-RTN-g128, INT4-HQQ-g128, and INT4-HQQ-g32, respectively.
Table 4: Reference losses and spectral slopes. Left: Pythia BF16 CE over 96 WikiText-103 windows or 512 LAMBADA items; bold/underlined marks the lowest/second-lowest value per dataset. Right: slope against the spectral predictor (reference one), from eight unbinned seed fits. Each ±\pm is the larger deviation to the 95% BCa endpoints.
ModelData onlyStorage onlyAdditiveInteraction
70M2.060 ±0.0612.060\,{\scriptstyle\pm\,0.061}1.293 ±0.0231.293\,{\scriptstyle\pm\,0.023}1.151‾ ±0.080\underline{1.151}\,{\scriptstyle\pm\,0.080}1.149 ±0.087\mathbf{1.149}\,{\scriptstyle\pm\,0.087}
160M0.805 ±0.0160.805\,{\scriptstyle\pm\,0.016}0.619 ±0.0340.619\,{\scriptstyle\pm\,0.034}0.425‾ ±0.037\underline{0.425}\,{\scriptstyle\pm\,0.037}0.392 ±0.036\mathbf{0.392}\,{\scriptstyle\pm\,0.036}
410M0.086 ±0.0040.086\,{\scriptstyle\pm\,0.004}0.156 ±0.0050.156\,{\scriptstyle\pm\,0.005}0.058‾ ±0.006\underline{0.058}\,{\scriptstyle\pm\,0.006}0.054 ±0.006\mathbf{0.054}\,{\scriptstyle\pm\,0.006}
1B0.061‾ ±0.003\underline{0.061}\,{\scriptstyle\pm\,0.003}0.096 ±0.0020.096\,{\scriptstyle\pm\,0.002}0.041 ±0.003\mathbf{0.041}\,{\scriptstyle\pm\,0.003}0.041 ±0.003\mathbf{0.041}\,{\scriptstyle\pm\,0.003}
1.4B0.057‾ ±0.003\underline{0.057}\,{\scriptstyle\pm\,0.003}0.085 ±0.0010.085\,{\scriptstyle\pm\,0.001}0.033 ±0.004\mathbf{0.033}\,{\scriptstyle\pm\,0.004}0.033 ±0.004\mathbf{0.033}\,{\scriptstyle\pm\,0.004}
2.8B0.055‾ ±0.003\underline{0.055}\,{\scriptstyle\pm\,0.003}0.074 ±0.0020.074\,{\scriptstyle\pm\,0.002}0.034 ±0.004\mathbf{0.034}\,{\scriptstyle\pm\,0.004}0.034 ±0.004\mathbf{0.034}\,{\scriptstyle\pm\,0.004}
Appendix
Table 5: Predicting the largest data budget from 4/8-bit and BF16 calibration. Root-mean-square error (RMSE, nats; lower is better) over 4/6/8-bit and BF16 test CE at 2262^{26} additional tokens. Each response function in Appendix R is fitted on validation CE at the first three data budgets. Eight paired training seeds and 96 fixed test windows are used; 5,000 seed-level BCa resamples refit each function. Smaller ±\pm values are rounded-up symmetric envelopes of the 95% intervals. Bold/underline mark the lowest/second-lowest displayed RMSE per row.
ModelData onlyStorage onlyAdditiveInteraction
70M1.304 ±0.015\mathbf{1.304}\,{\scriptstyle\pm\,0.015}1.845 ±0.0211.845\,{\scriptstyle\pm\,0.021}1.842‾ ±0.023\underline{1.842}\,{\scriptstyle\pm\,0.023}1.842‾ ±0.023\underline{1.842}\,{\scriptstyle\pm\,0.023}
160M0.649 ±0.016\mathbf{0.649}\,{\scriptstyle\pm\,0.016}0.732 ±0.0170.732\,{\scriptstyle\pm\,0.017}0.731‾ ±0.017\underline{0.731}\,{\scriptstyle\pm\,0.017}0.731‾ ±0.017\underline{0.731}\,{\scriptstyle\pm\,0.017}
410M0.093‾ ±0.003\underline{0.093}\,{\scriptstyle\pm\,0.003}0.094 ±0.0030.094\,{\scriptstyle\pm\,0.003}0.081 ±0.003\mathbf{0.081}\,{\scriptstyle\pm\,0.003}0.081 ±0.003\mathbf{0.081}\,{\scriptstyle\pm\,0.003}
1B0.054‾ ±0.002\underline{0.054}\,{\scriptstyle\pm\,0.002}0.055 ±0.0020.055\,{\scriptstyle\pm\,0.002}0.046 ±0.002\mathbf{0.046}\,{\scriptstyle\pm\,0.002}0.046 ±0.002\mathbf{0.046}\,{\scriptstyle\pm\,0.002}
1.4B0.053 ±0.0020.053\,{\scriptstyle\pm\,0.002}0.047‾ ±0.001\underline{0.047}\,{\scriptstyle\pm\,0.001}0.041 ±0.002\mathbf{0.041}\,{\scriptstyle\pm\,0.002}0.041 ±0.002\mathbf{0.041}\,{\scriptstyle\pm\,0.002}
2.8B0.051 ±0.0020.051\,{\scriptstyle\pm\,0.002}0.043‾ ±0.002\underline{0.043}\,{\scriptstyle\pm\,0.002}0.039 ±0.002\mathbf{0.039}\,{\scriptstyle\pm\,0.002}0.039 ±0.002\mathbf{0.039}\,{\scriptstyle\pm\,0.002}
Appendix
Table 6: Predicting 6-bit loss from 4/8-bit and BF16 calibration. RMSE (nats; lower is better) over the first three data budgets, holding out 6-bit weights from calibration. Response functions, eight paired training seeds, 96 fixed test windows, refitted 95% BCa envelopes, and ranking conventions match Table 5 .
ModelData onlyStorage onlyAdditiveInteraction
410M3.776 ±0.0443.776\,{\scriptstyle\pm\,0.044}1.354 ±0.0081.354\,{\scriptstyle\pm\,0.008}1.328‾ ±0.030\underline{1.328}\,{\scriptstyle\pm\,0.030}1.327 ±0.036\mathbf{1.327}\,{\scriptstyle\pm\,0.036}
1B3.426 ±0.0323.426\,{\scriptstyle\pm\,0.032}1.088 ±0.0101.088\,{\scriptstyle\pm\,0.010}1.050‾ ±0.019\underline{1.050}\,{\scriptstyle\pm\,0.019}1.045 ±0.022\mathbf{1.045}\,{\scriptstyle\pm\,0.022}
1.4B3.724 ±0.0313.724\,{\scriptstyle\pm\,0.031}1.094 ±0.0111.094\,{\scriptstyle\pm\,0.011}1.063‾ ±0.020\underline{1.063}\,{\scriptstyle\pm\,0.020}1.060 ±0.028\mathbf{1.060}\,{\scriptstyle\pm\,0.028}
2.8B3.255 ±0.0243.255\,{\scriptstyle\pm\,0.024}0.797 ±0.0050.797\,{\scriptstyle\pm\,0.005}0.772‾ ±0.006\underline{0.772}\,{\scriptstyle\pm\,0.006}0.771 ±0.007\mathbf{0.771}\,{\scriptstyle\pm\,0.007}
Appendix
Table 7: Predicting the largest data budget with 2-bit weights included in calibration. RMSE (nats; lower is better) over all six deployment formats at 2262^{26} additional tokens, fitted using 2/4/8-bit and BF16 validation CE at the first three budgets. This bounded-response comparison covers 410M–2.8B. The 70M/160M 2-bit calibration losses exceed its [0,20][0,20] -nat range; Figure 7 shows their measured quantization losses. Eight paired seeds, 96 test windows, 95% BCa envelopes and ranking conventions match Table 5 .
ModelData onlyStorage onlyAdditiveInteraction
410M1.976 ±0.0161.976\,{\scriptstyle\pm\,0.016}1.573 ±0.0321.573\,{\scriptstyle\pm\,0.032}1.568 ±0.033\mathbf{1.568}\,{\scriptstyle\pm\,0.033}1.569‾ ±0.033\underline{1.569}\,{\scriptstyle\pm\,0.033}
1B1.870 ±0.0081.870\,{\scriptstyle\pm\,0.008}1.512‾ ±0.014\underline{1.512}\,{\scriptstyle\pm\,0.014}1.511 ±0.014\mathbf{1.511}\,{\scriptstyle\pm\,0.014}1.511 ±0.014\mathbf{1.511}\,{\scriptstyle\pm\,0.014}
1.4B2.152 ±0.0142.152\,{\scriptstyle\pm\,0.014}1.585‾ ±0.012\underline{1.585}\,{\scriptstyle\pm\,0.012}1.584 ±0.012\mathbf{1.584}\,{\scriptstyle\pm\,0.012}1.584 ±0.012\mathbf{1.584}\,{\scriptstyle\pm\,0.012}
2.8B1.874 ±0.0031.874\,{\scriptstyle\pm\,0.003}1.145‾ ±0.003\underline{1.145}\,{\scriptstyle\pm\,0.003}1.144 ±0.003\mathbf{1.144}\,{\scriptstyle\pm\,0.003}1.144 ±0.003\mathbf{1.144}\,{\scriptstyle\pm\,0.003}
Appendix
Table 8: Predicting 3/6-bit loss with 2-bit weights included in calibration. RMSE (nats; lower is better) at the first three data budgets, holding out 3/6-bit weights from 2/4/8-bit/BF16 calibration. This comparison covers 410M–2.8B, with the calibration range and scale coverage described in Table 7 . Eight paired training seeds, 96 test windows, 95% BCa envelopes and ranking conventions match Table 5 .
Figure 10: Candidate-count replication under route-adjusted loss. (a) Direct finite-sample construction; (b) mini-batch two-stage implementation. Horizontal and vertical axes show E−R+E-R_{+} at K=32K=32 and K=8K=8 , respectively, using identical spectral bins. Legends give (χ,γ)(\chi,\gamma) ; the dashed identity line is deterministic. Intervals on both axes are 95% BCa intervals over eight paired seeds.
Figure 11: Unbinned risk-to-spectrum ratios on fixed sources. Panels (a)–(d) use the four profiles of Figure 2 , at K=32K=32 . The horizontal axis is GμG_{\mu} ; the vertical axis is measured excess CE divided by GμG_{\mu} . Each point represents a distinct sample–state budget pair; lines connect budget pairs ordered by GμG_{\mu} , rather than a fitted response. Blue/orange denote weighted SGD and the oracle-coordinate estimator. Bands retain the raw 95% BCa intervals over eight paired seeds.

Explore similar work

CardsList

  1. Prediction Is Not a Sufficient Memory State

    Oct 9, 2026Robert Jenkinson AlvarezOnline Learning

  2. Prescriptive Scaling Laws for Data Constrained Training

    May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight DecayLanguage Model Scaling Laws

    Department of Computer Science Cornell University Ithaca, NY

  3. Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating

    Jul 27, 2026Maruthi Vemula, Neeraj Praneeth GajulaKV CachingKV-Cache Eviction

    University of North Carolina at Chapel Hill · New York University