Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer Vision. We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting. To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture, and we use continual training with constant learning rates and periodic cooldowns as an efficient training strategy. We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules. Cooldown phases can be re-purposed to improve downstream performance, e.g., enabling accurate multi-step rollouts over longer forecast horizons as well as sharper predictions through spectral loss adjustments. We also systematically explore a wide range of model and dataset sizes under various compute budgets to construct IsoFLOP curves, and we identify compute-optimal training regimes. Extrapolating these trends to larger scales highlights potential performance limits, demonstrating that neural scaling can serve as an important diagnostic for efficient resource allocation. We open-source our code for reproducibility.
Figures & tables
Figure 1. Neural scaling for weather emulation. We pre-train several models using continual training (constant learning rates with periodic cooldowns; see § 3 ), and we identify compute-optimal regimes to train the neural emulator so that neither data nor model size saturate at different compute budgets. At each FLOP budget, several model sizes (up to 400M) are trained to different dataset sizes to form IsoFLOPs that demonstrate the tradeoff between model and data size. Unlike NLP models, these systems are trained for multiple epochs (indicated by vertical dotted lines), causing samples to be revisited after the first epoch and effectively be treated as pseudo-samples. We fit parabolas to each IsoFLOP, and we track the compute-optimal model at each budget.
Figure 2. Loss behavior for cosine vs constant LR with cooldown. (left) LR schedules: The cosine schedule follows a half-cosine decay after a fixed warmup, while the constant + cooldown has a constant LR after the same fixed warmup, but then cools down rapidly to 0 at the end. The cooldowns happen at the last 5% of iterations. (right) Loss vs iterations for different Swin models: The validation loss of the model continuously trained with a constant LR and cooled down to different iteration counts (here 5% of the total iteration is used as a cooldown period) shows better losses compared to the Swin trained from scratch with different cosine schedulers that match the total iteration count.
Figure 3. Loss as a function of total iterations used for cooldown. We show the MSE over 36 hours (6 autoregressive steps) of prediction averaged over the validation data (2017). The MSE loss decreases predictably with longer cooldown durations and this is true over multistep predictions. At around 5%, the gains start to diminish. The loss behavior also holds when the cooldown is repurposed with 4-step AR loss that allows for lower errors across the longer horizon.
Figure 4. Cooldowns can be used for alignment. When evaluated on testing year 2020, the Swin cooled down at 24000 iterations is able to surpass the NWP (HRES) and is comparable to the state-of-the-art deterministic deep learning benchmark, Graphcast. When AR is used in cooldown ( Swin - AR 4), the RMSE drops further, consistent with the use of this loss. When AMSE is used ( Swin - AMSE ), the PSD retains high wavenumbers. This is easily seen in q700 where the AMSE spectra matches ERA5 perfectly, but other models blur significantly. AR contributes to more blurring in favor of reduced RMSE (visible in dissipation of power in high wavenumbers). We note that HRES models weather at a 0.1∘ resolution and hence shows higher resolution.
Figure 5. Optimal model sizes as a function of compute. (left) Similar to Fig. 1 , we show the validation loss as a function of model size for different compute budgets. For each budget, the different model sizes are trained to a different number of iterations to create an IsoFLOP. We track the minima for each IsoFLOP (through a fitted parabola). (middle) We fit an empirical scaling law to find the optimal model size for any FLOP budget and project to 2.25E+21 FLOPs to find the optimal model. (right) We also project the loss to the final FLOP value—the measured loss at this FLOP value is saturated at 0.005.
Figure 6. RMSE performance of compute-optimal Swin vs baselines. The 204M Swin (compute-optimal 6E+19 FLOPs) outperforms NWP / HRES and is on-par with GraphCast when evaluated over initial conditions in 2020. The model is cooled down with 4 step AR . With more steps (8 step AR ), the performance gap reduces at longer lead times, as expected (GraphCast finetunes with 2–12 step AR ).
Figure 7. RMSE performance of the closest compute-optimal Swin models at each compute budget. We find that RMSE at extended forecast horizons, up to 240 hours, consistently improves with increased compute. However, at the highest compute levels, the rollout performance begins to exhibit signs of saturation.
Figure 8. PSD performance at lead time 24 hours of the closest compute-optimal Swin models at each compute budget. The PSD is evaluated at a 24-hour lead time and averaged over the test year. All models accurately capture the low wavenumbers, while higher wavenumbers show gradual improvement as compute increases. The bottom row zooms in on the high-wavenumber range, where larger models exhibit spectra that more closely match ERA5, whereas smaller models appear more blurred and display some artificial high-wavenumber features.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Param (M)
Embed
Depth
LR
3.1
192
6
1.5e-3
6.9
256
8
1.5e-3
15.1
384
8
1.5e-3
26.4
512
8
1.2e-3
39.0
512
12
1e-3
60.5
640
12
8e-4
Appendix
Table 1. Model configurations used for the neural scaling: We run each of these models once to different final iteration counts based on the required FLOP budgets with 5% cooldown periods at the end for the LR.
Figure 9. Predictions from the 204M parameter model: We show the ground truth ERA5, the prediction, and the error in physical units for four variables. The predicted forecasts also qualitatively match the ground truth.
Figure 10. PSD of predictions at 1 day and 5 days: While AR produces blurrier forecasts (though with lower RMSE), AMSE helps retain power at high wavenumbers. At longer lead times, such as 120 hours, Swin - AMSE continues to show significant power at these small scales. This allows practitioners to define an effective resolution based on this noise-level as outlined in Subich et al. [2025] .
Figure 11. PSD performance at lead time 120 hours of the closest compute-optimal Swin models at each compute budget: The PSD is evaluated at a 120-hour lead time and averaged over the test year. All models accurately capture the low wavenumbers, while higher wavenumbers show gradual improvement as compute increases. The bottom row zooms in on the high-wavenumber range, where larger models exhibit spectra that more closely match ERA5, whereas smaller models appear more blurred and display some artificial high-wavenumber features. These features are more dominant at this longer lead time compared to 24 hours in Fig. 8 .
Figure 12. Tropical Cyclone (TC) visualization and tracks before Typhoon Hagibis made landfall in Japan : We reproduce the TC analysis from [ Price et al., 2023 ] . We show the ERA5 ground truth for specific humidity at 700hPa on October 12, 2019, 06 UTC in the top left. The following three rows show the Swin predictions at the same time from all the closest compute-optimal models (between 6E17 and 6E19 FLOPs, initialized 1, 3, and 5 days earlier, respectively. The last column shows the PSD for the predictions (zoomed into the high wavenumbers). Forecasts initialized 1 day earlier show that all models capture the TC structure, although the larger models produce systematically sharper features, which is also reflected in the PSD. For initializations 3 and 5 days earlier, the smaller models fail to capture the TC structure, exhibiting patching artifacts and generally inconsistent blending of physical features. This behavior is likewise evident in the PSD as artificial high-wavenumber energy. In contrast, the larger models consistently capture the TC across all lead times. The last row shows the TC track from the different initializations. Although a deterministic model is insufficient for accurately assessing track performance, some variability among the different models is seen.
Figure 13. Training and validation loss as a function of iterations for different models: We show the training curves for all the (closest) compute-optimal models. All losses were logged at a frequency of 2000 iterations except the 1.3B model that was logged at a frequency of 10,000 iterations. The training and validation loss curves track each other closely. However, for the 1.3B model, we observe significant overfitting.
Data-driven models are revolutionizing weather forecasting. To optimize training efficiency and model performance, this paper analyzes empirical scaling laws within this domain. We investigate the relationship between model performance (validation loss) and three key factors: model size (N), dataset size (D), and compute budget (C). Across a range of models, we find that Aurora exhibits the strongest data-scaling behavior: increasing the training dataset by 10x reduces validation loss by up to 3.2x. GraphCast demonstrates the highest parameter efficiency, yet suffers from limited hardware utilization. Our compute-optimal analysis indicates that, under fixed compute budgets, allocating resources to more total training data yields greater performance gains than increasing model size. Furthermore, we analyze model shape and uncover scaling behaviors that differ fundamentally from those observed in language models: weather forecasting models consistently favor increased width over depth. These findings suggest that future weather models should prioritize wider architectures and larger effective training datasets to maximize predictive performance.
Yuejiang Yu, Langwen Huang, Alexandru Calotoiu +1
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
Compute-optimal scaling laws are relatively well studied for NLP and CV, where objectives are typically single-step and targets are comparatively homogeneous. Weather forecasting is harder to characterize in the same framework: autoregressive rollouts compound errors over long horizons, outputs couple many physical channels with disparate scales and predictability, and globally pooled test metrics can disagree sharply with per-channel, late-lead behavior implied by short-horizon training. We extend neural scaling analysis for autoregressive weather forecasting from single-step training loss to long rollouts and per-channel metrics. We quantify (1) how prediction error is distributed across channels and how its growth rate evolves with forecast horizon, (2) if power law scaling holds for test error, relative to rollout length when error is pooled globally, and (3) how that fit varies jointly with horizon and channel for parameter, data, and compute-based scaling axes. We find strong cross-channel and cross-horizon heterogeneity: pooled scaling can look favorable while many channels degrade at late leads. We discuss implications for weighted objectives, horizon-aware curricula, and resource allocation across outputs.
Alexander Kiefer, Prasanna Balaprakash, Xiao Wang
Bredesen Center, University of Tennessee, Knoxville, Tennessee · Oak Ridge National Laboratory, Oak Ridge, Tennessee
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to 10–100×.
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4