AI weather prediction has advanced rapidly, yet no unified mathematical framework explains what determines forecast skill. Existing theory addresses specific architectural choices rather than the learning pipeline as a whole, while operational evidence from 2023-2026 demonstrates that training methodology, loss function design, and data diversity matter at least as much as architecture selection. This paper makes two interleaved contributions. Theoretically, we construct a framework rooted in approximation theory on the sphere, dynamical systems theory, information theory, and statistical learning theory that treats the complete learning pipeline (architecture, loss function, training strategy, data distribution) rather than architecture alone. We establish a Learning Pipeline Error Decomposition showing that estimation error (loss- and data-dependent) dominates approximation error (architecture-dependent) at current scales. We develop a Loss Function Spectral Theory formalizing MSE-induced spectral blurring in spherical harmonic coordinates, and derive Out-of-Distribution Extrapolation Bounds proving that data-driven models systematically underestimate record-breaking extremes with bias growing linearly in record exceedance. Empirically, we validate these predictions via inference across ten architecturally diverse AI weather models using NVIDIA Earth2Studio with ERA5 initial conditions, evaluating six metrics across 30 initialization dates spanning all seasons. Results confirm universal spectral energy loss at high wavenumbers for MSE-trained models, rising Error Consensus Ratios showing that the majority of forecast error is shared across architectures, and linear negative bias during extreme events. A Holistic Model Assessment Score provides unified multi-dimensional evaluation, and a prescriptive framework enables mathematical evaluation of proposed pipelines before training.
Figures & tables
Model
Arch.
Params
Loss
Strategy
Data
Rate
Atlas
DiT
4300M
Score matching
Single-step
ERA5
N−s/2
AIFS
GNN
400M
MSE (weighted)
Curriculum rollout
ERA5 + oper. analyses
N−s/2
AIFS-ENS
GNN
400M
CRPS (afCRPS)
Curriculum rollout
ERA5 + oper. analyses
N−s/2
Aurora
ViT
1300M
MSE (pres. wt.)
Multi-dataset pretrain + LoRA
ERA5 + GFS + CMIP6
N−s/2
FCN3
SFNO
200M
CRPS (sp.+spec.)
Multi-step rollout
ERA5
L−s
FengWu
ViT
450M
MSE
Multi-step rollout
ERA5 + oper. analyses
N−s/2
Table 1: The ten AI weather models evaluated in this study with their complete pipeline components: architecture family, approximate parameter count, loss function, training strategy, primary data source, and theoretical convergence rate. See Supplementary Table 7 for detailed training periods and additional data sources.
Figure 1: Power spectra of U500 (kinetic energy, top) and Z500 (geopotential height, bottom) for ten AI weather models vs. ERA5 analysis, at lead times of 1, 5, 8, 10, and 15 days. All seven MSE-trained models exhibit progressive spectral energy loss at high wavenumbers, confirming the architecture-independent deficit predicted by Theorem 4.1 : ΔE(ℓ,τ)=Varℓ(τ) . The day 8 panel captures the transition regime where the deficit is actively deepening. Atlas (score-matching, red), FCN3 (CRPS), and AIFS-ENS (afCRPS) retain high-wavenumber energy, consistent with non-MSE losses preserving spectral variance. The dual power law ℓ−3/ℓ−5/3 is clearly visible in ERA5 (black) at short lead times.
Figure 2: Spectral energy ratio Efcst(k)/EERA5(k) for Z500 at lead times of 1, 5, 8, 10, and 15 days. Ratio <1 at high k confirms MSE-induced spectral bias; the day 8 panel shows the advancing predictability frontier. See Supplementary Fig. 14 for U500.
Loss
Target
Spectral
Models
MSE (wt.)
Cond. mean
Deficit
AIFS, Aurora, FuXi, GC, Pangu
MSE
Cond. mean
Deficit
FengWu, SFNO
CRPS
Calibrated distr.
Preserved
AIFS-ENS, FCN3
Score match.
Full p(u∣u0)
Preserved
Atlas
Table 2: Loss function taxonomy with spectral properties and corresponding models from our ten-model evaluation.
Figure 3: Global area-weighted RMSE vs. lead time for Z500, T850, and T2M, averaged over 30 initialization dates spanning all four seasons, with inter-initialization-date 90% confidence intervals (shaded bands). The tight clustering of architecturally diverse models confirms εarch≪εest (Proposition 5.1 ). See Supplementary Fig. 36 for all six verification variables.
Figure 4: Anomaly Correlation Coefficient vs. lead time for Z500, T850, and T2M, with inter-initialization-date 90% CIs. Most models sustain useful skill (ACC >0.6 ) for Z500 through ∼ 7 days; the best models extend to ∼ 10 days.
Figure 5: Model scorecard for Z500: RMSE (top) and rank (bottom, 1=best) by lead time. Rank changes confirm that no single architecture dominates all horizons. See Supplementary Figs. 17 – 18 for T2M and T850.
Figure 6: Regional RMSE decomposition for Z500 by latitude band: Tropics ( ∣ϕ∣<20∘ ), Extratropics ( 20∘≤∣ϕ∣<60∘ ), Polar ( ∣ϕ∣≥60∘ ). Note that y-axis ranges differ across panels to resolve intra-band model spread; cross-panel comparison of absolute RMSE values requires reading the axis labels. Polar errors amplify fastest (reaching ∼ 100–140 m by day 10 vs. ∼ 70–100 m in the extratropics); extratropical and polar bands show comparable relative inter-model spread. See Supplementary Figs. 32 – 33 for T2M and T850.
Figure 7: Left : Z500 RMSE on log scale vs. lead time. The approximately linear growth in log-RMSE confirms exponential error growth consistent with Proposition 6.3 . Right : Global kinetic energy stability EKE(τ)/EKE(0) . Deviation from unity indicates energy drift. FCN3 (CRPS-trained) shows energy inflation; most MSE-trained models show dissipation, consistent with MSE-induced smoothing.
Figure 8: Four-component Physical Consistency Score. Left : Composite PCS (mean of four sub-scores) vs. lead time. ERA5 analysis (black) maintains PCS ≈0.55 . Right : Sub-score breakdown at day 5 for all ten models, showing geostrophic balance (geo.), non-divergence (n-div.), thermal wind (t.w.), and hydrostatic (hyd.) components. Hydrostatic consistency is uniformly high; non-divergence shows the greatest inter-model spread. See Supplementary Fig. 37 for the day 15 sub-score breakdown.
Figure 9: Error Consensus Analysis for Z500. Left : ECR vs. lead time; ECR rises from ∼ 0.50 at day 1 to ∼ 0.62 at day 8, then gently declines to ∼ 0.59 at day 15. Right : Mean pairwise error correlation. See Supplementary Figs. 15 – 16 for T2M and T850.
Figure 10: Scale-dependent Model Error Divergence (MED) for Z500, decomposed by architecture family. Solid colored lines show within-family MED for GNN pairs (blue), spectral pairs (green), and ViT pairs (orange); red dashed line shows cross-family MED. The purple shaded region between mean within-family and cross-family MED highlights the “architecture fingerprint” at fine scales ( ℓ>50 ), where models from the same family produce more similar error structures than models from different families. At large scales ( ℓ<20 ), all curves converge, confirming universal planetary-wave handling.
Figure 11: Tail fidelity for the 2021 PNW heatwave. Mean forecast bias (K) vs. standardized exceedance δ ( σ ) at lead times 1–5 days. All models show increasingly negative bias at higher exceedances, confirming Proposition 10.1 . See Supplementary Figs. 19 – 25 for the 2023 European heatwave, 2021 Texas freeze, and Winter Storm Elliott, and Figs. 22 – 29 for α evolution curves across all four events.
Figure 12: Tail fidelity for the February 2021 Texas freeze (cold extreme). Attenuation coefficients α≈0.34 – 0.56 at day 5 are substantially larger than for heat events ( α≲0.28 ), revealing a systematic cold–warm asymmetry in OOD bias. Notably, AIFS, which achieves near-zero α for heat events, shows α≈0.44 here, demonstrating that warm-extreme mitigation does not transfer to cold extremes. See Supplementary Figs. 25 – 29 for Winter Storm Elliott.
Model
SFI
ℓeff
τd
EES
PCS
ASI
HMAS
FCN3 (SFNO)
0.978
1.000
0.774
0.646
0.545
0.913
0.823
Atlas (DiT)
0.936
1.000
0.713
0.643
0.524
0.904
0.800
AIFS-ENS (GNN)
0.946
0.980
0.743
0.642
0.473
0.895
0.794
Aurora (ViT)
0.799
1.000
0.721
0.640
0.586
0.900
0.782
Pangu (3D-ET)
0.667
1.000
0.762
0.676
0.626
0.897
0.772
GraphCast (GNN)
0.632
1.000
0.774
0.646
0.593
0.842
0.747
Table 3: Holistic Model Assessment Scores (HMAS) at Day 3 (short-range). Models ranked by HMAS.
Model
SFI
ℓeff
τd
EES
PCS
ASI
HMAS
FCN3 (SFNO)
0.977
1.000
0.774
0.634
0.540
0.913
0.820
Atlas (DiT)
0.938
1.000
0.713
0.634
0.511
0.904
0.797
AIFS-ENS (GNN)
0.946
0.976
0.743
0.632
0.472
0.895
0.792
Aurora (ViT)
0.798
0.976
0.721
0.630
0.588
0.900
0.777
Pangu (3D-ET)
0.663
0.948
0.762
0.655
0.625
0.897
0.761
GraphCast (GNN)
0.615
0.910
0.774
0.642
0.588
0.842
0.729
Table 4: Holistic Model Assessment Scores (HMAS) at Day 5 (medium-range). Models ranked by HMAS.
Model
SFI
ℓeff
τd
EES
PCS
ASI
HMAS
FCN3 (SFNO)
0.973
1.000
0.774
0.632
0.541
0.913
0.819
AIFS-ENS (GNN)
0.948
0.982
0.743
0.634
0.472
0.895
0.793
Atlas (DiT)
0.953
0.987
0.713
0.624
0.397
0.904
0.780
Aurora (ViT)
0.790
0.863
0.721
0.631
0.590
0.900
0.759
Pangu (3D-ET)
0.654
0.840
0.762
0.641
0.628
0.897
0.741
GraphCast (GNN)
0.599
0.851
0.774
0.629
0.568
0.842
0.711
Table 5: Holistic Model Assessment Scores (HMAS) at Day 15 (extended-range). Models ranked by HMAS. Note: FengWu’s ℓeff=1.0 is a metric artifact caused by spectral energy inflation exceeding the half-power threshold at all wavenumbers despite very low SFI ( =0.078 ); see text following Definition 6.2 .
Figure 13: HMAS parallel coordinates at day 5 (medium-range). Each line represents one model; each vertical axis represents one of the six HMAS metrics. Line crossings between adjacent axes reveal trade-offs: for example, models with high SFI (spectral fidelity) tend to show lower PCS (physical consistency), and vice versa. No single model achieves the highest score on all six dimensions simultaneously. Models are ordered in the legend by composite HMAS (highest first). See Supplementary Figs. 30 – 31 for radar chart representations at day 3 and day 15.
Task
Loss function
Data strategy
Training method
Architecture
Primary determinant
Short-range (1–3 d)
MSH or CRPS
ERA5 + oper. analyses
AR rollout fine-tune
Any hybrid
Training method
Medium-range (3–15 d)
MSH + ensemble
Multi-source
AR rollout + LoRA
Hybrid GNN–transformer
Loss + data
Extremes
Heavy-tailed score match.
+ synthetic extremes
Augmented OOD training
Any + physics constraints
Training data + OOD bounds
Table 6: Holistic pipeline assessment across forecast timescales.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Primary data
Training period
Additional data
Fine-tuning
Test cutoff
AIFS
ERA5
∼ 1979–2018
—
ECMWF oper. analyses (2019–2020)
2021
AIFS-ENS
ERA5
∼ 1979–2018
—
ECMWF oper. analyses (2019–2020)
2021
Atlas
ERA5
1980–2019
—
—
2020
Aurora
ERA5
1979–2020
GFS analyses, CMIP6, IFS-HR, HRES
LoRA rollout fine-tune (HRES-T0, 2016–2021)
2022
FCN3
ERA5
1980–2016
—
—
2020
FengWu
ERA5
1979–2017
—
—
2018
Appendix
Table 7: [Supplementary] Training datasets and time periods for the ten AI weather models evaluated in this study. All models use ERA5 reanalysis as the primary training dataset; several incorporate additional data sources for pretraining or fine-tuning. The “Test cutoff” column indicates the earliest year that can be used for out-of-sample evaluation. Our 30 initialization dates (2021–2024) postdate all training cutoffs.
Figure 14: [Supplementary] Spectral energy ratio for U500 at day 1 through day 15. Same qualitative pattern as Z500: universal deficit at high k for MSE-trained models.
Figure 15: [Supplementary] Error Consensus Analysis for T2M. ECR starts at ≈0.63 at short range and remains above 0.53 through day 15.
Figure 16: [Supplementary] Error Consensus Analysis for T850.
Figure 17: [Supplementary] Model scorecard for T2M.
Figure 18: [Supplementary] Model scorecard for T850.
Figure 19: [Supplementary] Tail fidelity for the 2023 European heatwave.
Figure 20: [Supplementary] Tail attenuation coefficients α for the PNW heatwave.
Figure 21: [Supplementary] Tail attenuation coefficients α for the 2023 European heatwave.
Figure 22: [Supplementary] α coefficient evolution with lead time for the PNW heatwave, with R2 values quantifying the goodness of fit of the linear bias–exceedance relationship.
Figure 23: [Supplementary] α coefficient evolution for the 2023 European heatwave.
Figure 24: [Supplementary] Tail fidelity for the February 2021 Texas freeze. Cold extreme attenuation coefficients α≈0.34 – 0.56 at day 5 are substantially larger than for heat events, indicating stronger OOD bias for cold extremes.
Figure 25: [Supplementary] Tail fidelity for Winter Storm Elliott (December 2022). Similar cold-extreme attenuation pattern to the Texas freeze, with α≈0.34 – 0.56 at day 5.
Figure 26: [Supplementary] Tail attenuation coefficients α for the February 2021 Texas freeze.
Figure 28: [Supplementary] α coefficient evolution with lead time for the February 2021 Texas freeze.
Figure 29: [Supplementary] α coefficient evolution with lead time for Winter Storm Elliott (December 2022).
Figure 30: [Supplementary] HMAS radar chart at day 3 (short-range).
Figure 31: [Supplementary] HMAS radar chart at day 15 (extended-range).
Figure 32: [Supplementary] Regional RMSE decomposition for T2M by latitude band.
Figure 33: [Supplementary] Regional RMSE decomposition for T850 by latitude band.
Figure 34: [Supplementary] HMAS weight sensitivity analysis. Grouped bar chart showing HMAS values under five weighting schemes (default, equal, accuracy-focused, extremes-focused, stability-focused) for all ten models. Kendall’s W=0.97 confirms strong rank concordance, with the top and bottom rankings stable across all schemes.
Figure 35: [Supplementary] HMAS dimension cross-correlation matrix at day 5. SFI and PCS are strongly anti-correlated ( ρ=−0.86 ), confirming the spectral fidelity–physical consistency trade-off. ASI is largely independent of accuracy and balance metrics. Mean absolute off-diagonal correlation ∣ρ∣ˉ=0.38 .
Figure 36: [Supplementary] Global area-weighted RMSE vs. lead time for all six verification variables (Z500, T2M, U500, V500, T850, Q700), averaged over 30 initialization dates with inter-initialization-date 90% CIs.
Figure 37: [Supplementary] Balance sub-scores at day 15 (extended range) for all ten models. Compared with day 5 (Fig. 8 , right panel), Atlas shows marked degradation in geostrophic balance and non-divergence, while Pangu maintains high sub-scores across all four balance components.
Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.
AI weather models are often reported to underestimate extremes, but most evidence concerns deterministic regression models verified against reanalysis. We evaluate twelve physical and AI forecast models against ECMWF IFS using ten months of European station observations. The evaluation covers 10 m wind, 2 m temperature, solar radiation, and precipitation within regimes defined from a fixed ERA5 1991-2020 climatology. We find no uniform AI-specific deficit in the tails. Several AI models remain more accurate than IFS under extreme conditions, while others deteriorate markedly; comparable variation occurs among physical models. Every model nevertheless exhibits a common conditional-error pattern, overpredicting low observations and underpredicting high observations. Attenuation of extreme values therefore does not imply a uniform loss of relative skill: tail performance depends on the model, variable, and evaluation setting rather than on whether the forecast is produced by AI or physical numerical modelling.
Marvin Vincent Gabler, Roberto Molinaro, Niall Siegenheim +5
Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs required for traditional physics-based methods. However, they are primarily (1) data-driven and (2) evaluated using pixel-wide error metrics (e.g., RMSE), so there are no guarantees that their forecasts are consistent with known physical laws. We introduce PhysMetrics.Weather, an evaluation framework that assesses the physical realism of MLWP models across three types of metrics: conservation, spectral, and dynamical. By quantifying physical realism, this tool guides the development of physics-informed architectures and helps evaluate whether MLWP models are reliable for operational use. Our framework is available on Github at https://github.com/Emmakast/PhysMetrics.Weather.
Emma Kasteleyn, Timo Maier, Axel Lauer +3
University of Amsterdam, Amsterdam, The Netherlands · Deutsches Zentrum für Luft- und Raumfahrt (DLR), Institut für Physik der Atmosphäre, Oberpfaffenhofen, Germany · University of Bremen, Institute of Environmental Physics (IUP), Bremen, Germany +1
Can probabilistic artificial intelligence (AI) weather forecasts retain skill beyond their training domain? We compare four state-of-the-art AI models (ArchesWeatherGen, U-Cast, WeatherNext2 and GenCast-mini) with the physics-based IFS-ENS over land in 2022, at 1.5° resolution and lead times of 2--10 days. Monthly ERA5 records from 1979--2021 define common historical boundaries, and we characterize new records set in 2022 by their depth, i.e., the exceedance margin relative to local historical variability. Two proper scores, record-threshold weighted CRPS and Brier score, assess the magnitude and probability of record exceedances across all eligible cases, retaining both missed extremes and false alarms. Verified against ERA5, AI models outperform IFS-ENS for heat and cold records at almost every lead and depth. For heat, the advantage strengthens with depth, partly because IFS-ENS over-forecasts deep records, and largely persists for deeper records under HRES-fc0 verification. For wind, gains erode for deeper records and longer leads. Better tail scores can nevertheless coexist with aggregate probability deficits or surpluses, and some improvements arise chiefly on non-exceedances. Increasing underprediction on progressively more extreme observed subsets can also arise through outcome selection, even for an ideal conditional-mean forecast. These findings establish that native probabilistic AI ensembles retain, and for heat increase, their advantage over IFS-ENS as thresholds move beyond local records, although all advantages narrow with lead time. They caution against generalizing the extrapolation failure reported for deterministic AI forecasts to probabilistic systems, and motivate common verification conventions that separate comparative tail skill, performance on realized extremes, and probabilistic calibration.
Bastiaan Van Velthoven, Cas Decancq, Diego G. Miralles +2