cs.LGJan 27, 2026

Neural Neural Scaling Laws

Authors: Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri, Nicholas Lourie, Kyunghyun Cho

Organizations: New York University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors: some improve monotonically, others plateau, and some even degrade with scale. We argue that predicting downstream performance from validation loss suffers from two limitations: averaging token-level losses obscures signal, and no simple parametric family can capture the full spectrum of scaling behaviors. To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without the limitations inherent in assuming a specific functional form. Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 1.99% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 44% reduction compared to logistic scaling laws (3.56% MAE). Furthermore, NeuNeu generalizes zero-shot to unseen model families, architectures, parameter counts, and downstream tasks. Our work suggests that predicting downstream scaling directly from data outperforms parametric alternatives.

Figures & tables

Figure 1: Richer signal from token-level losses (center) enables NeuNeu to better forecast accuracies for downstream tasks (right). Average validation loss, used in logistic scaling laws, averages away token-level loss changes.
Figure 2: NeuNeu encodes and processes token-level validation probabilities alongside a sequence of historical downstream accuracies and compute gaps, which are projected into context tokens. The BERT-style transformer ( Devlin et al., 2019 ) backbone uses this information to predict a distribution over the downstream accuracy via quantile regression on the [CLS] token representation.
Figure 3: Generalization results for downstream task accuracy prediction.
Figure 4: Visualizing NeuNeu ’s predictions. Black dots are ground truth accuracies, and the grey line marks the beginning of NeuNeu ’s predictions, after observing the first 20% of the training run. The light-blue band is the 10%-90% interquantile interval predicted by NeuNeu itself. NeuNeu tightly captures the ground truth data across model scales, while logistic scaling laws overpredict performance for the 150M model and underpredict performance for the 1B model.
Figure 5: NeuNeu generalizes beyond the settings seen during meta-training and provides uncertainty estimates that support downstream decision-making. (A) NeuNeu maintains low error on downstream tasks held out during training, outperforming logistic scaling laws fit directly to those tasks. (B) The predicted 10%–90% interquantile interval from NeuNeu contains 77.6% of ground truth accuracies, close to the 80% target. (C) NeuNeu achieves the highest ranking accuracy when predicting which of two model configurations will reach better final downstream performance.
Figure 6: Additional extrapolation results. (A) NeuNeu and the neural ablations maintain low error as the extrapolation horizon increases, while parametric and general-purpose learning-curve baselines remain substantially higher. (B) LC-PFN improves as more of the training trajectory is observed, indicating that it is inferring from context.
Figure 7: Using token probabilities produces better neural predictors than token losses. Another reason to use probabilities is that the function e−xe^{-x} , or the conversion from loss to probability, has larger derivative for smaller loss values, meaning that small changes in loss near convergence for the language model are amplified. This likely matters more than changes when the loss is large and language model performance on downstream tasks is near-chance.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Transformer Encoder Architecture
Hidden dimension512
Transformer layers6
Attention heads8
Feed-forward dimension2048
Max sequence length512
Appendix
Table 1: Model hyperparameters, shared across our neural models. We chose learning rate and weight decay values based on models of similar size in Magnusson et al. [2025] .
Figure 8: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
Figure 9: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
Figure 10: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
More in this topic

Language Model Performance Prediction

See all 30 papers

Explore similar work

CardsList

  1. Unified Neural Scaling Laws

    May 25, 2026Ethan Caballero, Priyank Jaini, David Krueger +1Language Model Scaling LawsScaling Laws

    Mila, University of Montreal · Google DeepMind

  2. On the Invariance and Generality of Neural Scaling Laws

    May 8, 2026Xing Han, Liu Ziyin, Suchi Saria +1Language Model Scaling LawsCross-Domain Generalization

    Johns Hopkins University Baltimore, MD 21218, USA · NTT Research, MIT Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health Baltimore, MD 21218, USA +1

  3. Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

    Aug 7, 2026Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1Language Model Scaling LawsNeural Scaling Laws

    1FAIR at Meta