The Recipe Matters More Than the Kitchen:Mathematical Foundations of the AI Weather Prediction Pipeline

Piyush Garg; Diana R. Gergel; Andrew E. Shao; Galen J. Yacalis

cs.LGApr 1, 2026

The Recipe Matters More Than the Kitchen:Mathematical Foundations of the AI Weather Prediction Pipeline

Authors: Piyush Garg, Diana R. Gergel, Andrew E. Shao, Galen J. Yacalis

Organizations: Commercial AI Lab, RWE Trading Americas Inc., Bellevue, WA, USA · AI Research Lab, Hewlett Packard Enterprise, Victoria, BC, Canada

Abstract

AI weather prediction has advanced rapidly, yet no unified mathematical framework explains what determines forecast skill. Existing theory addresses specific architectural choices rather than the learning pipeline as a whole, while operational evidence from 2023-2026 demonstrates that training methodology, loss function design, and data diversity matter at least as much as architecture selection. This paper makes two interleaved contributions. Theoretically, we construct a framework rooted in approximation theory on the sphere, dynamical systems theory, information theory, and statistical learning theory that treats the complete learning pipeline (architecture, loss function, training strategy, data distribution) rather than architecture alone. We establish a Learning Pipeline Error Decomposition showing that estimation error (loss- and data-dependent) dominates approximation error (architecture-dependent) at current scales. We develop a Loss Function Spectral Theory formalizing MSE-induced spectral blurring in spherical harmonic coordinates, and derive Out-of-Distribution Extrapolation Bounds proving that data-driven models systematically underestimate record-breaking extremes with bias growing linearly in record exceedance. Empirically, we validate these predictions via inference across ten architecturally diverse AI weather models using NVIDIA Earth2Studio with ERA5 initial conditions, evaluating six metrics across 30 initialization dates spanning all seasons. Results confirm universal spectral energy loss at high wavenumbers for MSE-trained models, rising Error Consensus Ratios showing that the majority of forecast error is shared across architectures, and linear negative bias during extreme events. A Holistic Model Assessment Score provides unified multi-dimensional evaluation, and a prescriptive framework enables mathematical evaluation of proposed pipelines before training.

Figures & tables

Model	Arch.	Params	Loss	Strategy	Data	Rate
Atlas	DiT	4300M	Score matching	Single-step	ERA5	$N^{-s/2}$
AIFS	GNN	400M	MSE (weighted)	Curriculum rollout	ERA5 + oper. analyses	$N^{-s/2}$
AIFS-ENS	GNN	400M	CRPS (afCRPS)	Curriculum rollout	ERA5 + oper. analyses	$N^{-s/2}$
Aurora	ViT	1300M	MSE (pres. wt.)	Multi-dataset pretrain + LoRA	ERA5 + GFS + CMIP6	$N^{-s/2}$
FCN3	SFNO	200M	CRPS (sp.+spec.)	Multi-step rollout	ERA5	$L^{-s}$
FengWu	ViT	450M	MSE	Multi-step rollout	ERA5 + oper. analyses	$N^{-s/2}$

Table 1: The ten AI weather models evaluated in this study with their complete pipeline components: architecture family, approximate parameter count, loss function, training strategy, primary data source, and theoretical convergence rate. See Supplementary Table 7 for detailed training periods and additional data sources.

Figure 1: Power spectra of U500 (kinetic energy, top) and Z500 (geopotential height, bottom) for ten AI weather models vs. ERA5 analysis, at lead times of 1, 5, 8, 10, and 15 days. All seven MSE-trained models exhibit progressive spectral energy loss at high wavenumbers, confirming the architecture-independent deficit predicted by Theorem 4.1 :

\Delta E(\ell,\tau)=\mathrm{Var}_{\ell}(\tau)

. The day 8 panel captures the transition regime where the deficit is actively deepening. Atlas (score-matching, red), FCN3 (CRPS), and AIFS-ENS (afCRPS) retain high-wavenumber energy, consistent with non-MSE losses preserving spectral variance. The dual power law

\ell^{-3}/\ell^{-5/3}

is clearly visible in ERA5 (black) at short lead times.

Figure 2: Spectral energy ratio

E_{\rm fcst}(k)/E_{\rm ERA5}(k)

for Z500 at lead times of 1, 5, 8, 10, and 15 days. Ratio

<1

at high

k

confirms MSE-induced spectral bias; the day 8 panel shows the advancing predictability frontier. See Supplementary Fig. 14 for U500.

Loss	Target	Spectral	Models
MSE (wt.)	Cond. mean	Deficit	AIFS, Aurora, FuXi, GC, Pangu
MSE	Cond. mean	Deficit	FengWu, SFNO
CRPS	Calibrated distr.	Preserved	AIFS-ENS, FCN3
Score match.	Full $p(\bm{u}\|\bm{u}_{0})$	Preserved	Atlas

Table 2: Loss function taxonomy with spectral properties and corresponding models from our ten-model evaluation.

Figure 3: Global area-weighted RMSE vs. lead time for Z500, T850, and T2M, averaged over 30 initialization dates spanning all four seasons, with inter-initialization-date 90% confidence intervals (shaded bands). The tight clustering of architecturally diverse models confirms $\varepsilon_{\rm arch}\ll\varepsilon_{\rm est}$ (Proposition 5.1 ). See Supplementary Fig. 36 for all six verification variables.

Figure 4: Anomaly Correlation Coefficient vs. lead time for Z500, T850, and T2M, with inter-initialization-date 90% CIs. Most models sustain useful skill (ACC $>0.6$ ) for Z500 through $\sim$ 7 days; the best models extend to $\sim$ 10 days.

Figure 5: Model scorecard for Z500: RMSE (top) and rank (bottom, 1=best) by lead time. Rank changes confirm that no single architecture dominates all horizons. See Supplementary Figs. 17 – 18 for T2M and T850.

Figure 6: Regional RMSE decomposition for Z500 by latitude band: Tropics (

|\phi|<20^{\circ}

), Extratropics (

20^{\circ}\leq|\phi|<60^{\circ}

), Polar (

|\phi|\geq 60^{\circ}

). Note that y-axis ranges differ across panels to resolve intra-band model spread; cross-panel comparison of absolute RMSE values requires reading the axis labels. Polar errors amplify fastest (reaching

\sim

100–140 m by day 10 vs.

\sim

70–100 m in the extratropics); extratropical and polar bands show comparable relative inter-model spread. See Supplementary Figs. 32 – 33 for T2M and T850.

Figure 7: Left : Z500 RMSE on log scale vs. lead time. The approximately linear growth in log-RMSE confirms exponential error growth consistent with Proposition 6.3 . Right : Global kinetic energy stability

E_{\rm KE}(\tau)/E_{\rm KE}(0)

. Deviation from unity indicates energy drift. FCN3 (CRPS-trained) shows energy inflation; most MSE-trained models show dissipation, consistent with MSE-induced smoothing.

Figure 8: Four-component Physical Consistency Score. Left : Composite PCS (mean of four sub-scores) vs. lead time. ERA5 analysis (black) maintains PCS

\approx 0.55

. Right : Sub-score breakdown at day 5 for all ten models, showing geostrophic balance (geo.), non-divergence (n-div.), thermal wind (t.w.), and hydrostatic (hyd.) components. Hydrostatic consistency is uniformly high; non-divergence shows the greatest inter-model spread. See Supplementary Fig. 37 for the day 15 sub-score breakdown.

Figure 9: Error Consensus Analysis for Z500. Left : ECR vs. lead time; ECR rises from $\sim$ 0.50 at day 1 to $\sim$ 0.62 at day 8, then gently declines to $\sim$ 0.59 at day 15. Right : Mean pairwise error correlation. See Supplementary Figs. 15 – 16 for T2M and T850.

Figure 10: Scale-dependent Model Error Divergence (MED) for Z500, decomposed by architecture family. Solid colored lines show within-family MED for GNN pairs (blue), spectral pairs (green), and ViT pairs (orange); red dashed line shows cross-family MED. The purple shaded region between mean within-family and cross-family MED highlights the “architecture fingerprint” at fine scales ( $\ell>50$ ), where models from the same family produce more similar error structures than models from different families. At large scales ( $\ell<20$ ), all curves converge, confirming universal planetary-wave handling.

Figure 11: Tail fidelity for the 2021 PNW heatwave. Mean forecast bias (K) vs. standardized exceedance

\delta

(

\sigma

) at lead times 1–5 days. All models show increasingly negative bias at higher exceedances, confirming Proposition 10.1 . See Supplementary Figs. 19 – 25 for the 2023 European heatwave, 2021 Texas freeze, and Winter Storm Elliott, and Figs. 22 – 29 for

\alpha

evolution curves across all four events.

Figure 12: Tail fidelity for the February 2021 Texas freeze (cold extreme). Attenuation coefficients

\alpha\approx 0.34

–

0.56

at day 5 are substantially larger than for heat events (

\alpha\lesssim 0.28

), revealing a systematic cold–warm asymmetry in OOD bias. Notably, AIFS, which achieves near-zero

\alpha

for heat events, shows

\alpha\approx 0.44

here, demonstrating that warm-extreme mitigation does not transfer to cold extremes. See Supplementary Figs. 25 – 29 for Winter Storm Elliott.

Model	SFI	$\bm{\ell_{\textbf{eff}}}$	$\bm{\tau_{d}}$	EES	PCS	ASI	HMAS
FCN3 (SFNO)	0.978	1.000	0.774	0.646	0.545	0.913	0.823
Atlas (DiT)	0.936	1.000	0.713	0.643	0.524	0.904	0.800
AIFS-ENS (GNN)	0.946	0.980	0.743	0.642	0.473	0.895	0.794
Aurora (ViT)	0.799	1.000	0.721	0.640	0.586	0.900	0.782
Pangu (3D-ET)	0.667	1.000	0.762	0.676	0.626	0.897	0.772
GraphCast (GNN)	0.632	1.000	0.774	0.646	0.593	0.842	0.747

Table 3: Holistic Model Assessment Scores (HMAS) at Day 3 (short-range). Models ranked by HMAS.

Model	SFI	$\bm{\ell_{\textbf{eff}}}$	$\bm{\tau_{d}}$	EES	PCS	ASI	HMAS
FCN3 (SFNO)	0.977	1.000	0.774	0.634	0.540	0.913	0.820
Atlas (DiT)	0.938	1.000	0.713	0.634	0.511	0.904	0.797
AIFS-ENS (GNN)	0.946	0.976	0.743	0.632	0.472	0.895	0.792
Aurora (ViT)	0.798	0.976	0.721	0.630	0.588	0.900	0.777
Pangu (3D-ET)	0.663	0.948	0.762	0.655	0.625	0.897	0.761
GraphCast (GNN)	0.615	0.910	0.774	0.642	0.588	0.842	0.729

Table 4: Holistic Model Assessment Scores (HMAS) at Day 5 (medium-range). Models ranked by HMAS.

Model	SFI	$\bm{\ell_{\textbf{eff}}}$	$\bm{\tau_{d}}$	EES	PCS	ASI	HMAS
FCN3 (SFNO)	0.973	1.000	0.774	0.632	0.541	0.913	0.819
AIFS-ENS (GNN)	0.948	0.982	0.743	0.634	0.472	0.895	0.793
Atlas (DiT)	0.953	0.987	0.713	0.624	0.397	0.904	0.780
Aurora (ViT)	0.790	0.863	0.721	0.631	0.590	0.900	0.759
Pangu (3D-ET)	0.654	0.840	0.762	0.641	0.628	0.897	0.741
GraphCast (GNN)	0.599	0.851	0.774	0.629	0.568	0.842	0.711

Table 5: Holistic Model Assessment Scores (HMAS) at Day 15 (extended-range). Models ranked by HMAS. Note: FengWu’s

\ell_{\rm eff}=1.0

is a metric artifact caused by spectral energy inflation exceeding the half-power threshold at all wavenumbers despite very low SFI (

=0.078

); see text following Definition 6.2 .

Figure 13: HMAS parallel coordinates at day 5 (medium-range). Each line represents one model; each vertical axis represents one of the six HMAS metrics. Line crossings between adjacent axes reveal trade-offs: for example, models with high SFI (spectral fidelity) tend to show lower PCS (physical consistency), and vice versa. No single model achieves the highest score on all six dimensions simultaneously. Models are ordered in the legend by composite HMAS (highest first). See Supplementary Figs. 30 – 31 for radar chart representations at day 3 and day 15.

Task	Loss function	Data strategy	Training method	Architecture	Primary determinant
Short-range (1–3 d)	MSH or CRPS	ERA5 + oper. analyses	AR rollout fine-tune	Any hybrid	Training method
Medium-range (3–15 d)	MSH + ensemble	Multi-source	AR rollout + LoRA	Hybrid GNN–transformer	Loss + data
Extremes	Heavy-tailed score match.	+ synthetic extremes	Augmented OOD training	Any + physics constraints	Training data + OOD bounds

Table 6: Holistic pipeline assessment across forecast timescales.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Primary data	Training period	Additional data	Fine-tuning	Test cutoff
AIFS	ERA5	$\sim$ 1979–2018	—	ECMWF oper. analyses (2019–2020)	2021
AIFS-ENS	ERA5	$\sim$ 1979–2018	—	ECMWF oper. analyses (2019–2020)	2021
Atlas	ERA5	1980–2019	—	—	2020
Aurora	ERA5	1979–2020	GFS analyses, CMIP6, IFS-HR, HRES	LoRA rollout fine-tune (HRES-T0, 2016–2021)	2022
FCN3	ERA5	1980–2016	—	—	2020
FengWu	ERA5	1979–2017	—	—	2018

Appendix

Table 7: [Supplementary] Training datasets and time periods for the ten AI weather models evaluated in this study. All models use ERA5 reanalysis as the primary training dataset; several incorporate additional data sources for pretraining or fine-tuning. The “Test cutoff” column indicates the earliest year that can be used for out-of-sample evaluation. Our 30 initialization dates (2021–2024) postdate all training cutoffs.

Figure 14: [Supplementary] Spectral energy ratio for U500 at day 1 through day 15. Same qualitative pattern as Z500: universal deficit at high

k

for MSE-trained models.

Figure 15: [Supplementary] Error Consensus Analysis for T2M. ECR starts at $\approx 0.63$ at short range and remains above $0.53$ through day 15.

Figure 16: [Supplementary] Error Consensus Analysis for T850.

Figure 17: [Supplementary] Model scorecard for T2M.

Figure 18: [Supplementary] Model scorecard for T850.

Figure 19: [Supplementary] Tail fidelity for the 2023 European heatwave.

Figure 20: [Supplementary] Tail attenuation coefficients

\alpha

for the PNW heatwave.

Figure 21: [Supplementary] Tail attenuation coefficients

\alpha

for the 2023 European heatwave.

Figure 22: [Supplementary]

\alpha

coefficient evolution with lead time for the PNW heatwave, with

R^{2}

values quantifying the goodness of fit of the linear bias–exceedance relationship.

Figure 23: [Supplementary]

\alpha

coefficient evolution for the 2023 European heatwave.

Figure 24: [Supplementary] Tail fidelity for the February 2021 Texas freeze. Cold extreme attenuation coefficients

\alpha\approx 0.34

–

0.56

at day 5 are substantially larger than for heat events, indicating stronger OOD bias for cold extremes.

Figure 25: [Supplementary] Tail fidelity for Winter Storm Elliott (December 2022). Similar cold-extreme attenuation pattern to the Texas freeze, with

\alpha\approx 0.34

–

0.56

at day 5.

Figure 26: [Supplementary] Tail attenuation coefficients

\alpha

for the February 2021 Texas freeze.

Figure 27: [Supplementary] Tail attenuation coefficients

\alpha

for Winter Storm Elliott (December 2022).

Figure 28: [Supplementary]

\alpha

coefficient evolution with lead time for the February 2021 Texas freeze.

Figure 29: [Supplementary]

\alpha

coefficient evolution with lead time for Winter Storm Elliott (December 2022).

Figure 30: [Supplementary] HMAS radar chart at day 3 (short-range).

Figure 31: [Supplementary] HMAS radar chart at day 15 (extended-range).

Figure 32: [Supplementary] Regional RMSE decomposition for T2M by latitude band.

Figure 33: [Supplementary] Regional RMSE decomposition for T850 by latitude band.

Figure 34: [Supplementary] HMAS weight sensitivity analysis. Grouped bar chart showing HMAS values under five weighting schemes (default, equal, accuracy-focused, extremes-focused, stability-focused) for all ten models. Kendall’s $W=0.97$ confirms strong rank concordance, with the top and bottom rankings stable across all schemes.

Figure 35: [Supplementary] HMAS dimension cross-correlation matrix at day 5. SFI and PCS are strongly anti-correlated ( $\rho=-0.86$ ), confirming the spectral fidelity–physical consistency trade-off. ASI is largely independent of accuracy and balance metrics. Mean absolute off-diagonal correlation $\bar{|\rho|}=0.38$ .

Figure 36: [Supplementary] Global area-weighted RMSE vs. lead time for all six verification variables (Z500, T2M, U500, V500, T850, Q700), averaged over 30 initialization dates with inter-initialization-date 90% CIs.

Figure 37: [Supplementary] Balance sub-scores at day 15 (extended range) for all ten models. Compared with day 5 (Fig. 8 , right panel), Atlas shows marked degradation in geostrophic balance and non-divergence, while Pangu maintains high sub-scores across all four balance components.

Weather Forecasting

Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.

See all 176 papers

Do AI weather models miss extremes?
Jul 31, 2026Marvin Vincent Gabler, Roberto Molinaro, Niall Siegenheim +5Extreme Weather Forecasting Weather Forecasting
Jua.ai AG
PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models
Jun 9, 2026Emma Kasteleyn, Timo Maier, Axel Lauer +3Forecasting Benchmarks Physics-Informed ML
University of Amsterdam, Amsterdam, The Netherlands · Deutsches Zentrum für Luft- und Raumfahrt (DLR), Institut für Physik der Atmosphäre, Oberpfaffenhofen, Germany · University of Bremen, Institute of Environmental Physics (IUP), Bremen, Germany +1
Beyond the record: evaluating probabilistic AI weather forecasts deep in the tail
Oct 18, 2026Bastiaan Van Velthoven, Cas Decancq, Diego G. Miralles +2Weather Forecasting Probabilistic Forecasting

Jul 31, 2026J/K move · Enter open · S save

Do AI weather models miss extremes?

Marvin Vincent Gabler, Roberto Molinaro, Niall Siegenheim, Henry Martin+4

Jua.ai AG

Extreme Weather Forecasting Weather Forecasting

Open paper AskPDF arXiv

AI weather models are often reported to underestimate extremes, but most evidence concerns deterministic regression models verified against reanalysis. We evaluate twelve physical and AI forecast models against ECMWF IFS using ten months of European station observations. The evaluation covers 10 m wind, 2 m temperature, solar radiation, and precipitation within regimes defined from a fixed ERA5 1991-2020 climatology. We find no uniform AI-specific deficit in the tails. Several AI models remain more accurate than IFS under extreme conditions, while others deteriorate markedly; comparable variation occurs among physical models. Every model nevertheless exhibits a common conditional-error pattern, overpredicting low observations and underpredicting high observations. Attenuation of extreme values therefore does not imply a uniform loss of relative skill: tail performance depends on the model, variable, and evaluation setting rather than on whether the forecast is produced by AI or physical numerical modelling.

Abstract

Weather Forecasting

Explore similar work

Do AI weather models miss extremes?

PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models

Beyond the record: evaluating probabilistic AI weather forecasts deep in the tail