Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling behaviors: some improve monotonically, others plateau, and some even degrade with scale. We argue that predicting downstream performance from validation loss suffers from two limitations: averaging token-level losses obscures signal, and no simple parametric family can capture the full spectrum of scaling behaviors. To address this, we propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation. NeuNeu combines temporal context from observed accuracy trajectories with token-level validation losses, learning to predict future performance without the limitations inherent in assuming a specific functional form. Trained entirely on open-source model checkpoints from HuggingFace, NeuNeu achieves 1.99% mean absolute error in predicting model accuracy on 66 downstream tasks -- a 44% reduction compared to logistic scaling laws (3.56% MAE). Furthermore, NeuNeu generalizes zero-shot to unseen model families, architectures, parameter counts, and downstream tasks. Our work suggests that predicting downstream scaling directly from data outperforms parametric alternatives.
Figures & tables
Figure 1: Richer signal from token-level losses (center) enables NeuNeu to better forecast accuracies for downstream tasks (right). Average validation loss, used in logistic scaling laws, averages away token-level loss changes.
Figure 2: NeuNeu encodes and processes token-level validation probabilities alongside a sequence of historical downstream accuracies and compute gaps, which are projected into context tokens. The BERT-style transformer ( Devlin et al., 2019 ) backbone uses this information to predict a distribution over the downstream accuracy via quantile regression on the [CLS] token representation.
Figure 3: Generalization results for downstream task accuracy prediction.
Figure 4: Visualizing NeuNeu ’s predictions. Black dots are ground truth accuracies, and the grey line marks the beginning of NeuNeu ’s predictions, after observing the first 20% of the training run. The light-blue band is the 10%-90% interquantile interval predicted by NeuNeu itself. NeuNeu tightly captures the ground truth data across model scales, while logistic scaling laws overpredict performance for the 150M model and underpredict performance for the 1B model.
Figure 5: NeuNeu generalizes beyond the settings seen during meta-training and provides uncertainty estimates that support downstream decision-making. (A) NeuNeu maintains low error on downstream tasks held out during training, outperforming logistic scaling laws fit directly to those tasks. (B) The predicted 10%–90% interquantile interval from NeuNeu contains 77.6% of ground truth accuracies, close to the 80% target. (C) NeuNeu achieves the highest ranking accuracy when predicting which of two model configurations will reach better final downstream performance.
Figure 6: Additional extrapolation results. (A) NeuNeu and the neural ablations maintain low error as the extrapolation horizon increases, while parametric and general-purpose learning-curve baselines remain substantially higher. (B) LC-PFN improves as more of the training trajectory is observed, indicating that it is inferring from context.
Figure 7: Using token probabilities produces better neural predictors than token losses. Another reason to use probabilities is that the function e−x , or the conversion from loss to probability, has larger derivative for smaller loss values, meaning that small changes in loss near convergence for the language model are amplified. This likely matters more than changes when the loss is large and language model performance on downstream tasks is near-chance.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Transformer Encoder Architecture
Hidden dimension
512
Transformer layers
6
Attention heads
8
Feed-forward dimension
2048
Max sequence length
512
Appendix
Table 1: Model hyperparameters, shared across our neural models. We chose learning rate and weight decay values based on models of similar size in Magnusson et al. [2025] .
Figure 8: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
Figure 9: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
Figure 10: Blue: NeuNeu . Dark grey: Logistic scaling law fitted to the task on the training set (§ 2.3 ).
We present a functional form (that we refer to as a Unified Neural Scaling Law (UNSL)) that accurately models and extrapolates the scaling behaviors of deep neural networks as multiple dimensions all vary simultaneously (i.e. how the evaluation metric of interest varies as one simultaneously varies the number of model parameters, training dataset size, number of training steps, number of inference steps, amount of compute, and various hyperparameters) for various architectures and for each of various tasks within a varied set of upstream and downstream tasks. This set includes large-scale vision, language, math, and reinforcement learning. When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set.
Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. This paper poses the research question of how to develop generalizable scaling laws: laws fit once on a well-resourced source domain and reliably transported to new domains where running a full sweep is infeasible, which requires a fundamental understanding of when and why scaling properties change. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution ρ: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within 3% error.
Xing Han, Liu Ziyin, Suchi Saria +1
Johns Hopkins University Baltimore, MD 21218, USA · NTT Research, MIT Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health Baltimore, MD 21218, USA +1
Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.
Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz +1