The neural scaling law relating longer training to better performance through a power law is central to today's large language models (LLMs), yet its origin remains debated. One recent proposal is that power laws can emerge from the strong non-linearity of a single softmax head learning peaked distributions. What happens with multiple softmax functions, as in LLMs, is unclear. Here, we show through toy models that any softmax learning peaked distributions, regardless of its position in the model, can develop logit magnitudes that grow in a power law with exponent 1/3, becoming a training bottleneck whose loss contribution decays as a power law with the same exponent 1/3. The overall loss therefore obeys 1/3 scaling whenever at least one softmax learns peaked distributions. We confirm that many softmax functions in LLMs learn peaked distributions and that LLM loss scaling matches this 1/3 prediction. Moreover, logit growth dynamics reveal that attention heads, rather than the language modeling head, are the bottleneck likely driving the 1/3 loss scaling in LLMs. Attention trying to concentrate on specific information, which is the heart of Transformers, may therefore also be the heart of the neural scaling law of training.
Figures & tables
Figure 1: In the gate-head (GH) model, any cold softmax, whether the head or the gate, leads to 1/3 time scaling of loss. (a) The GH model has two parts: a gate layer and a head layer. The gate layer uses the key matrix K and softmax to weight the value matrices. The head layer uses a matrix W and softmax to project hidden states to distributions. (b) The GH model’s loss exhibits 1/3 time scaling whenever the target coldness βG∗ , βH∗ , or both are large. Experiment details in Appendix A.1 .
Figure 2: When gate coldness is large but head coldness is low, the gate logit magnitude grows as a 1/3 power law after the head saturates, sustaining 1/3 loss scaling. (a) The late-time loss follows 1/3 time scaling across different βH∗ (represented by colors). Different loss curves enter their 1/3 time scaling regions at different times, with the transitions marked by the vertical bands. (b) When loss enters the 1/3 time scaling region, student head coldness saturates. (c) The 1/3 scaling in loss also corresponds to student gate coldness growing as a 1/3 power law. Details in Appendix A.1 .
Figure 3: Training only the gate with MSE between hidden states yields 1/3 scaling in both loss and logit magnitude. (a) The loss enters the asymptotic 1/3 time scaling regime, with the vertical line marking its onset. (b) The student coldness exhibits positive 1/3 time scaling, beginning at the same time. (c) The value matrix displacement from initialization saturates as the loss and coldness enter their 1/3 scaling regimes. Curves from different learning rates collapse when plotted against τ , supporting τ as the fundamental variable governing dynamics. See Appendix A.2 for details.
Figure 4: In Pythia LLMs, attention heads are more strongly peaked than the LM head, with logit magnitudes growing as 1/3 power laws, identifying attention as the bottleneck for 1/3 loss scaling. (a) Entropy suggests that later-layer attention heads and the LM head are in the low-temperature regime (below the horizontal dashed lines). (b) Probability densities at zero energy gap are non-zero. (c) LLMs have losses following the 1/3 time scaling. (d) Logit coldness of attention heads rather than the LM head grows as the positive 1/3 time scaling at late training. Details in Appendix C.1 .
Figure 5: Using MSE between final logits for training, attention logit magnitude grows like a positive 1/3 scaling while the LM head saturates (panel a), ruling out the LM head as the only bottleneck for 1/3 loss scaling. (b) MSE loss has a tail following the 1/3 time scaling, as suggested by loss universality. Details in Appendix C.2 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Under Adam, the GH model’s loss exhibits approximate 1/3 time scaling whenever the target coldness βG∗ , βH∗ , or both are large. Each panel shows the test loss LKLGH against dynamic time τ on log-log axes, with a dashed reference line of slope −1/3 . The 2×2 layout arranges the four corners of the (βG∗,βH∗) sweep, with gate coldness increasing rightward and head coldness increasing upward. When both coldnesses are small (bottom left), the loss initially descends rapidly as in the SGD case, but Adam becomes unstable at late times. In the remaining three panels, approximate 1/3 loss scaling is observed, consistent with the SGD results in Figure 1 b.
Figure 7: Under Adam, when gate coldness is large but head coldness is small, the head saturates and the gate logit magnitude subsequently grows approximately as τ1/3 , sustaining approximate 1/3 loss scaling. Three representative runs are shown, color-coded by run. (a) Test loss LKLGH against τ ; the dashed line has slope −1/3 . (b) Head logit standard deviation βH against τ ; βH grows initially then saturates. (c) Gate logit standard deviation βG against τ ; the dashed line has slope +1/3 . The sequence of head saturation followed by gate growth and continued loss decay mirrors the SGD behavior in Figure 2 , though the alignment with the reference slopes is less tight.
Figure 8: Under Adam, training only the gate with MSE loss yields approximate 1/3 scaling in both loss and gate logit magnitude, mirroring the SGD results in Figure 3 . We fix βG∗=1000 and show three representative learning rates, color-coded by log10η from dark purple to light green. (a) Test loss LShG against τ ; the dashed line has slope −1/3 . (b) Gate logit standard deviation βG against τ ; the dashed line has slope +1/3 . (c) Value tensor displacement ∥V(t)−V(0)∥ against τ ; ∥V(t)−V(0)∥ rises and then saturates. Curves from different learning rates collapse well when plotted against τ .
Figure 9: The GH-II model reproduces 1/3 time scaling in loss and gate coldness for both sigmoid and tanh gate non-linearities ( βG∗=1000 , βH∗=1 ). Each pair of panels shows LKLGH (left) and βG (right) against τ on log-log axes. (a,b) Sigmoid variant. (c,d) Tanh variant. Dashed reference lines have slope −1/3 (panels a,c) and +1/3 (panels b,d).
Figure 10: The GH-III model with real attention as the gate reproduces 1/3 time scaling in loss and gate coldness ( βG∗=1000 , βH∗=1 ). (a) The loss LKLGH follows τ−1/3 scaling. (b) The gate coldness βG grows as τ1/3 . Dashed reference lines have slope −1/3 (panel a) and +1/3 (panel b).
Figure 11: Later-layer attention heads and the LM head are in the low-temperature regime for Pythia-410M (a) and Pythia-160M (b). Same format as Figure 4 a: black dots show per-head entropy at the final checkpoint, red dots show layer averages, and the red diamond shows the upper bound of the LM head target entropy ( 1.69 nats). Dashed lines mark the entropy thresholds for attention heads (left) and the LM head (right). The pattern is consistent with Pythia-12B in the main text.
Figure 12: Energy gap density distributions for all attention heads across all 36 layers of Pythia-12B (final checkpoint), extending Figure 4 b. Each panel corresponds to one layer; curves are colored by head index. For most heads across all layers, the density is non-zero at Δϵ→0 , confirming the theoretical condition broadly. Heads with density approaching zero at small Δϵ are in the high-temperature regime. Analysis follows Appendix C.1.2 .
Figure 13: Attention logit coldness versus dynamic time τ for all attention heads across all 36 layers of Pythia-12B, extending Figure 4 d. Each panel corresponds to one layer; curves are colored by head index. The dashed red line has slope +1/3 for reference. Low-temperature heads broadly track the +1/3 slope at late training, while high-temperature heads grow more slowly or saturate. Analysis follows Appendix C.1.2 .
Figure 14: Energy gap density (left column) and attention logit coldness versus τ (right column) for Pythia-410M (a–b) and Pythia-160M (c–d), replicating Figure 4 b and d. In each energy gap panel, black shows a representative attention head and gray shows the LM head; both have non-zero density at Δϵ→0 . In each coldness panel, circles show an attention head and diamonds show the LM head; the dashed line has slope +1/3 for reference. Analysis follows Appendix C.1.2 .
Figure 15: The raw cross-entropy loss and the fitting procedure for all eight Pythia model sizes (70M to 12B), extending Figure 4 c. Left: raw loss L versus τ with power-law-plus-constant fits overlaid (solid lines). The loss converges to a strictly positive floor L\τ that decreases with model size, requiring the constant to be included in the fit. Right: training-dependent component L−L\τ after subtracting the fitted irreducible loss. Curves from all model sizes collapse and follow a single ∼τ−1/3 power law (dashed reference line). Analysis follows Appendix C.1.2 .
Figure 16: Replication of the Figure 4 analysis for OLMo-2, extending the results to a different model family. (a) Per-head entropy of OLMo-2-13B at the final checkpoint. The red diamond shows the upper bound of the LM head target entropy. Although the mean entropy decreases in later layers, later-layer heads fall at most slightly below the entropy threshold (horizontal dashed line), and the heads deepest in the low-temperature regime appear in early layers. The layer-1 head highlighted by a blue square is in the low-temperature regime. (b) Energy gap density for the highlighted layer-1 head; the density is non-zero at Δϵ→0 . (c) Loss versus dynamic time τ for OLMo-2-1B, 7B, and 13B, with fitted exponents 0.35 , 0.30 , and 0.36 . (d) Attention logit coldness of the highlighted layer-1 head grows as τ1/3 (dashed reference line), while the LM head saturates. Analysis follows Appendix C.1.2 .
Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power laws are fixed by generic mechanisms: a one-third time scaling due to the strong nonlinearity of Softmax, an inverse width scaling due to representational superposition, and an inverse depth scaling due to ensemble averaging of Transformer layers. These mechanisms are robust to a wide range of data structures and architectural details, placing current large language models in a universality class with fixed exponents. The coefficients, however, are expected to be sensitive to data and architecture details, and directly determine practical quantities such as the optimal model shape and the compute-optimal frontier. We therefore argue that understanding the coefficients is the key to near-term performance improvements, and that a closer examination of the current universality class may reveal pathways to better universality classes.
Yizhou Liu, Jeff Gore
Massachusetts Institute of Technology Cambridge, MA 02139
Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong R2 scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on ≤6.9B Pythia models with ≤180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled R2=0.847, while monotonic baselines collapse.
Xu Ouyang, Deyi Liu, Yuhang Cai +5
1ByteDance Seed · University of Virginia · University of California, Berkeley
Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.
Zirui Peng, Yizhou Liu, Ziming Liu +1
Department of Physics, Peking University · Department of Physics, Massachusetts Institute of Technology, Cambridge, MA · College of AI, Tsinghua University