Beyond the record: evaluating probabilistic AI weather forecasts deep in the tail

Bastiaan Van Velthoven; Cas Decancq; Diego G. Miralles; Stijn Luca; Thomas Mortier

cs.LGOct 18, 2026

Beyond the record: evaluating probabilistic AI weather forecasts deep in the tail

Authors: Bastiaan Van Velthoven, Cas Decancq, Diego G. Miralles, Stijn Luca, Thomas Mortier

Abstract

Can probabilistic artificial intelligence (AI) weather forecasts retain skill beyond their training domain? We compare four state-of-the-art AI models (ArchesWeatherGen, U-Cast, WeatherNext2 and GenCast-mini) with the physics-based IFS-ENS over land in 2022, at 1.5° resolution and lead times of 2--10 days. Monthly ERA5 records from 1979--2021 define common historical boundaries, and we characterize new records set in 2022 by their depth, i.e., the exceedance margin relative to local historical variability. Two proper scores, record-threshold weighted CRPS and Brier score, assess the magnitude and probability of record exceedances across all eligible cases, retaining both missed extremes and false alarms. Verified against ERA5, AI models outperform IFS-ENS for heat and cold records at almost every lead and depth. For heat, the advantage strengthens with depth, partly because IFS-ENS over-forecasts deep records, and largely persists for deeper records under HRES-fc0 verification. For wind, gains erode for deeper records and longer leads. Better tail scores can nevertheless coexist with aggregate probability deficits or surpluses, and some improvements arise chiefly on non-exceedances. Increasing underprediction on progressively more extreme observed subsets can also arise through outcome selection, even for an ideal conditional-mean forecast. These findings establish that native probabilistic AI ensembles retain, and for heat increase, their advantage over IFS-ENS as thresholds move beyond local records, although all advantages narrow with lead time. They caution against generalizing the extrapolation failure reported for deterministic AI forecasts to probabilistic systems, and motivate common verification conventions that separate comparative tail skill, performance on realized extremes, and probabilistic calibration.

Figures & tables

Figure 1: New ERA5 records set in 2022. For heat (top), cold (lower left), and wind speed (lower right), the figure shows how many months of the year in 2022 experienced an extreme that went beyond historical (1979-2021) monthly ERA5 records.

Figure 2: Forecasting and evaluating a record-breaking extreme. The forecast distribution is Gaussian with mean

41\,^{\circ}\mathrm{C}

and standard deviation

1\,\mathrm{K}

; the historical record is

R=40\,^{\circ}\mathrm{C}

. (a) Forecast CDF and the step function corresponding to the verifying value

x=43\,^{\circ}\mathrm{C}

. (b) Clipping forecast and verifying values at

R

collapses values below the record onto the boundary. Here, approximately 16% of forecast probability moves to

R

, while the upper tail remains unchanged. (c) Squared CDF error for the record exceedance. CRPS integrates the combined grey and blue areas; rtwCRPS retains only the blue area beyond

R

, equivalently computing CRPS after clipping. (d) The same forecast evaluated against

x=38\,^{\circ}\mathrm{C}

. Forecast probability beyond the record still incurs a penalty, retaining false alarms in the assessment. For this forecast, rtwCRPS is identical for all verifying values

x\leq R

. Raising the threshold reduces the integration region and therefore cannot increase rtwCRPS for a fixed forecast and verifying value.

Figure 3: rtwCRPS and Brier skill scores as a function of lead time. Fair rtwCRPS skill (top) and Brier skill (bottom) relative to IFS-ENS at the 1979–2021 monthly record boundary. Positive values indicate better scores than IFS-ENS.

Figure 4: Depth-based rtwCRPS skill across lead times. Fair rtwCRPS skill relative to IFS-ENS, verified against ERA5 over land in 2022. Increasing

k

moves the threshold beyond the historical record in units of local historical variability. All eligible cases enter the score at every threshold.

Figure 5: Proper versus record-conditioned score rankings and score contributions. 48-hour forecasts verified against ERA5. (a,b) WN2 heat and AWG wind illustrate different rankings from record-conditioned CRPS and all-case record scores. (c) Mean forecast heat-exceedance probability divided by observed frequency for all five systems; unity indicates aggregate frequency agreement. (d) Observed-exceedance and non-exceedance contributions to the fair heat rtwCRPS reduction relative to IFS-ENS at

k=0

as in ( 10 ). Positive contributions favour AI. WN2’s improvement arises almost entirely on non-exceedances, whereas AWG, U-Cast and GenCast-mini reduce both contributions.

Figure 6: Depth-based skill when verifying against HRES-fc0. Fair rtwCRPS skill against HRES-fc0, using the same 2022 land sample and ERA5 historical records and scales for heat (top), cold (middle) and wind (bottom).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: ERA5 versus HRES-fc0 monthly records in 2022. For each grid cell, the figure shows the number of months in the year 2022 that experienced record-breaking heat (top), cold (middle) and wind (bottom) for both ERA5 (left) and HRES-fc0 (right) against 1979–2021 historical ERA5 conditions.

Figure S2: ERA5 versus HRES-fc0 max record depths in 2022. For each grid cell, the figure shows the maximum record depth in the year 2022 for heat (top), cold (middle) and wind (bottom) for both ERA5 (left) and HRES-fc0 (right) against 1979–2021 historical ERA5 conditions.

Extreme	ERA5	HRES-fc0	Both	ERA5 only	HRES-fc0 only	F1
Heat	7,754	4,102	2,548	5,206	1,554	0.430
Cold	1,084	1,214	565	519	649	0.492
Wind	2,676	5,096	1,902	774	3,194	0.489

Appendix

Table S1: Agreement between the two 2020 references at the ERA5-defined record boundary. F1 is the binary-event overlap,

2N_{\rm both}/(N_{\rm ERA5}+N_{\rm HRES-fc0})

Figure S3: Temperature ranking sensitivity to the verification reference for 48-hour forecasts. Left and centre: area-weighted precision–recall curves under each event reference, obtained by shifting forecast detection thresholds; dots mark the issued, unshifted forecasts. Right: record-conditioned RMSE with event selection fixed to ERA5 and only the error reference changed. Temperature rankings reverse, while HRES retains its wind RMSE advantage. The threshold sweeps are descriptive controls, not fitted postprocessing.

		All cases		Observed records
Extreme	Model	$T=\mathrm{E}$	$T=\mathrm{H}$	$(E,E)$	$(E,H)$	$(H,E)$	$(H,H)$
Heat	HRES	1.602	0.956	2.268	0.873	1.662	1.156
	GraphCast	0.975	1.355	1.420	1.402	1.269	1.803
	Pangu	1.083	1.333	1.481	1.423	1.318	1.846
	FuXi	1.016	1.407	1.422	1.486	1.325	1.895
Cold	HRES	1.602	0.956	2.508	1.250	1.321	1.375

Appendix

Table S2: Area-weighted RMSE at 48 h in 2020, on all eligible cases and on observed record exceedances (

k=0

). E denotes ERA5 and H denotes HRES-fc0. All-case columns specify the error reference

T

; record columns specify

(E,T)

, where

E

selects events and

T

measures errors. Every column compares models on a common sample against a common target. Heat and cold use kelvin; wind uses m s -1 . The all-case temperature values are repeated in the heat and cold blocks because they use the same full temperature sample. Bold indicates the lowest RMSE within each extreme and column.

		ERA5 events			HRES-fc0 events
Extreme	Model	Precision	Recall	F1	Precision	Recall	F1
Heat	HRES	0.347	0.249	0.290	0.439	0.679	0.533
	GraphCast	0.743	0.340	0.466	0.401	0.396	0.398
	Pangu	0.680	0.372	0.481	0.363	0.429	0.393
	FuXi	0.680	0.376	0.484	0.345	0.412	0.376
Cold	HRES	0.406	0.388	0.397	0.608	0.462	0.525

Appendix

Table S3: Area-weighted precision, recall and F1 at 48 h in 2020 for the issued, unshifted record forecasts. Reference events are crossings of the fixed ERA5 historical boundary. Bold identifies the largest F1 within each extreme and reference; precision and recall are shown separately to expose their trade-off.

Model	48 h	120 h	240 h	48 h	120 h	240 h	48 h	120 h	240 h
	Heat			Cold			Wind
HRES	5,489	5,219	4,874	1,066	1,189	914	4,438	4,533	4,681
GraphCast	3,888	2,604	1,966	562	625	1,159	1,210	810	681
Pangu	4,728	3,732	4,100	561	556	460	1,344	1,369	1,452
FuXi	4,616	4,143	1,158	694	705	19	1,096	795	38

Appendix

Table S4: Forecast record-crossing counts (unweighted) in 2020 on the same 4,649,708 land cell–time cases at every lead. Counts depend on the forecast and the fixed ERA5 historical boundary, not on the subsequent choice of verifying reference. Reference crossing counts are given in Table S1 .

Figure S4: Area-weighted precision–recall curves at 48 h for heat, cold and wind under the two references. The finite forecast-shift sweep changes detection thresholds; filled dots show the issued, unshifted classifications summarized in Table S3 .

			ERA5 errors		HRES-fc0 errors
Extreme	AI model	$N$	AI	HRES	AI	HRES
Heat	GraphCast	1,716	1.065	1.313	1.092	0.900
	Pangu	1,885	1.082	1.355	1.116	0.926
	FuXi	1,790	1.065	1.308	1.170	0.908
Cold	GraphCast	370	1.079	1.078	1.105	0.711
	Pangu	355	1.255	1.197	1.122	0.733

Appendix

Table S5: Area-weighted RMSE at 48 h where the named AI model and HRES both forecast a record.

N

is the (unweighted) number of selected cell–time cases. Each row uses the same subset under both error references, but subsets differ across rows. Heat and cold use kelvin; wind uses m s -1 . Bold identifies the lower RMSE within each row and reference.

Figure S5: Conditional RMSE at 48 h as a function of standardized observed depth, for all four

(E,T)

combinations. The scale is the historical standard deviation of annual monthly extrema. Each point uses cases exceeding the plotted cutoff; sparse deep subsets, especially cold, require caution.

Figure S6: Conditional severity bias at 48 h as a function of standardized observed depth, for all four

(E,T)

combinations. Cold errors are reflected, so negative bias denotes insufficient extremeness for every variable. Selection and scaling follow Figure S5 .

Figure S7: Conditional RMSE at 120 h as a function of standardized observed depth. Event selection, error references and historical scales follow Figure S5 , providing a longer-lead comparison on the same eligible sample.

Figure S8: Conditional severity bias at 120 h as a function of standardized observed depth. The four reference combinations, cumulative selection rule and cold sign convention follow Figure S6 .

Reference	Extreme	$k=0$	$0.5$	$1$	$2$	$2.5$	$3$
ERA5	Heat	5,129	1,772	610	28	4	—
	Cold	1,665	764	446	203	157	114
	Wind	2,971	1,437	688	200	115	59
HRES-fc0	Heat	3,514	1,369	509	21	4	1
	Cold	2,275	1,013	561	261	183	130
	Wind	5,838	3,153	1,727	552	324	206

Appendix

Table S6: Observed 2022 land grid-cell–time exceedances. Counts are unweighted and identical across models and leads on the common sample.

Figure S9: Fair record Brier skill against ERA5. All cases contribute at every threshold. Heat remains favourable to all AI systems, while some deep-wind comparisons lose skill.

Figure S10: Fair record Brier skill against HRES-fc0 using ERA5 historical thresholds. Negative near-record heat skill for AWG and U-Cast becomes positive at greater depth; wind is more mixed.

Figure S11: Ordinary fair CRPS skill on all cases under both common references. Heat and cold panels coincide because they use the same untransformed temperature forecasts and verifying values.

Figure S12: Reference switch at 48 h: ordinary fair CRPS (top) and fair rtwCRPS at

k=0

(bottom). Each line connects one model’s skill against the two references; each skill uses IFS-ENS verified against that same reference.

Figure S13: Aggregate forecast-probability ratios

Q

against ERA5. The logarithmic vertical axis distinguishes probability deficits and surpluses relative to one.

Figure S14: The same occurrence-frequency diagnostic against HRES-fc0. Thresholds and scales remain ERA5-based, while the verifying values and observed rates change.

Figure S15: Ensemble-mean severity bias on realized exceedances against ERA5. Cold temperature bias is multiplied by

-1

, so negative values denote underprediction of extreme severity for all variables.

Figure S16: Conditional severity bias against HRES-fc0 with the same cold sign convention. Longer leads are shown for completeness; both predictive uncertainty and outcome selection affect these diagnostics.

Weather Forecasting

Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.

See all 175 papers

Oct 15, 2026

Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones

Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at a fraction of the computational cost. Yet, whether this success extends to extreme events remains contested. Earlier evaluations have exposed important shortcomings, but have largely focused on deterministic models and metrics, leaving the capabilities of the latest probabilistic models less understood. We conduct an extensive benchmark of tropical cyclone forecasting, comparing GenCast, FourCastNet 3, and FGN with ECMWF's operational physics-based ensemble, IFS-ENS. We develop a suite of probabilistic metrics and diagnostic analyses to assess cyclone tracking, intensity biases, uncertainty, and the wind-pressure relationship that characterizes storm structure. We find that, despite remaining limitations in intensity and ensemble spread, leading AI models substantially outperform IFS-ENS in cyclone tracking: FGN achieves better probabilistic location scores for 90% of evaluated storms in 2025. Further analyses reveal model-specific failure modes and suggest a lingering influence of pretraining data, pointing to opportunities for targeted improvement. Nonetheless, these results suggest that AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions, motivating further investigation into how these capabilities emerge and how to strengthen them.

Nathan Chalumeau, Xavier Renard, Remi Meynadier +3

INRIA, Paris, France · AXA, Paris, France · Sorbonne Université, CNRS, LIP6, F-75005 Paris, France +1

Tropical Cyclone Intensity Forecasting Weather Forecasting Extreme Weather Forecasting Probabilistic Forecasting Forecasting Benchmarks

Ask PDF

Jul 31, 2026

Do AI weather models miss extremes?

AI weather models are often reported to underestimate extremes, but most evidence concerns deterministic regression models verified against reanalysis. We evaluate twelve physical and AI forecast models against ECMWF IFS using ten months of European station observations. The evaluation covers 10 m wind, 2 m temperature, solar radiation, and precipitation within regimes defined from a fixed ERA5 1991-2020 climatology. We find no uniform AI-specific deficit in the tails. Several AI models remain more accurate than IFS under extreme conditions, while others deteriorate markedly; comparable variation occurs among physical models. Every model nevertheless exhibits a common conditional-error pattern, overpredicting low observations and underpredicting high observations. Attenuation of extreme values therefore does not imply a uniform loss of relative skill: tail performance depends on the model, variable, and evaluation setting rather than on whether the forecast is produced by AI or physical numerical modelling.

Marvin Vincent Gabler, Roberto Molinaro, Niall Siegenheim +5

Jua.ai AG

Extreme Weather Forecasting Weather Forecasting

Ask PDF

Apr 10, 2026

U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster

AI-based weather forecasting now rivals traditional physics-based ensembles, but state-of-the-art (SOTA) models rely on specialized architectures and massive computational budgets, creating a high barrier to entry. We demonstrate that such complexity is unnecessary for frontier performance. We introduce \ours, a probabilistic forecaster built on a standard U-Net backbone trained with a simple recipe: deterministic pre-training on Mean Absolute Error followed by short probabilistic fine-tuning on the Continuous Ranked Probability Score (CRPS) using Monte Carlo Dropout for stochasticity. As a result, our model matches or exceeds the probabilistic skill of GenCast and IFS ENS at

1.5^\circ

resolution while reducing training compute by over

10\times

compared to leading CRPS-based models and inference latency by over

10\times

compared to diffusion-based models. U-Cast trains in under 12 H200 GPU-days and generates a 15-day ensemble forecast in 3 seconds. These results suggest that scalable, general-purpose architectures paired with efficient training curricula can match complex domain-specific designs at a fraction of the cost, opening the training of frontier probabilistic weather models to the broader community.

Salva Rühling Cachay, Duncan Watson-Parris, Rose Yu

UC San Diego

Monte Carlo Dropout Weather Forecasting Probabilistic Forecasting

Ask PDF

Abstract

Weather Forecasting

Explore similar work

Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones

Do AI weather models miss extremes?

U-Cast: A Surprisingly Simple and Efficient Frontier Probabilistic AI Weather Forecaster