Can AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones
Authors: Nathan Chalumeau, Xavier Renard, Remi Meynadier, Hugo Rakotoarimanga, Marcin Detyniecki, Emmanuel de Bezenac
Organizations: INRIA, Paris, France · AXA, Paris, France · Sorbonne Université, CNRS, LIP6, F-75005 Paris, France · Polish Academy of Science, IBS PAN, Warsaw, Poland
Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at a fraction of the computational cost. Yet, whether this success extends to extreme events remains contested. Earlier evaluations have exposed important shortcomings, but have largely focused on deterministic models and metrics, leaving the capabilities of the latest probabilistic models less understood. We conduct an extensive benchmark of tropical cyclone forecasting, comparing GenCast, FourCastNet 3, and FGN with ECMWF's operational physics-based ensemble, IFS-ENS. We develop a suite of probabilistic metrics and diagnostic analyses to assess cyclone tracking, intensity biases, uncertainty, and the wind-pressure relationship that characterizes storm structure. We find that, despite remaining limitations in intensity and ensemble spread, leading AI models substantially outperform IFS-ENS in cyclone tracking: FGN achieves better probabilistic location scores for 90% of evaluated storms in 2025. Further analyses reveal model-specific failure modes and suggest a lingering influence of pretraining data, pointing to opportunities for targeted improvement. Nonetheless, these results suggest that AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions, motivating further investigation into how these capabilities emerge and how to strengthen them.
Figures & tables
Figure 1: Evaluation windows of the benchmark. To evaluate probabilistic models on tropical cyclones, evaluation years are chosen according to each model’s training and fine-tuning periods.
Figure 2: Typhoon Bualoi (2025). FGN (red) and IFS-ENS (blue) forecasts at four lead times: dots are individual members, contours are their density, and black dots are the observed IBTrACS positions along the track (grey line). IFS-ENS’s members separate into distinct clusters at intermediate leads, which an ensemble mean cannot represent.
Model
Track detection rate
Cyclone track NLL ( ↓ )
GenCast
74.9±30.9 %
13.31±1.44
FCN3
73.5±31.4 %
13.39±1.27
IFS-ENS
72.4±28.2 %
13.49±1.07
ERA5 (baseline)
94.4±23.0 %
—
Table 1: Trajectory NLL and track detection rates (2021–2023). Left: mean ± std across storms; only the lowest forecast NLL is bold. The track detection rate is the percentage of members with a detected track matched to IBTrACS, computed on all 125 storms. ERA5 is a reference-field baseline. Right: share of the 106 retained storms on which each model has the lower conditional NLL, for each pair and for all three at once.
Figure 3: Better cyclone-local CRPS need not mean less intensity bias (2021–2023, ERA5 reference). Left: reduction of GenCast’s summed normalized fair-CRPS relative to FCN3’s, 100(1−CRPSGenCast/CRPSFCN3) , in cyclone regions (solid) and globally (dashed); positive favors GenCast. The shaded area is GenCast’s additional reduction in the cyclone region: 3–5% from 24–220 h, versus about 1% globally. Right: bias of the ensemble-mean maximum 10m wind speed, averaged across storms per lead and normalized by ERA5’s mean maximum wind (16.6 m/s). GenCast underestimates peak winds at short and intermediate leads despite its better local CRPS.
Model
Track detection rate
Cyclone track NLL ( ↓ )
FGN
78.9±34.6 %
12.13±1.21
GenCast Operational
78.0±33.9 %
12.55±2.08
IFS-ENS
66.2±36.4 %
13.06±1.57
HRES-fc0 (baseline)
96.2±19.1 %
—
Table 2: Trajectory NLL and track detection rates (2025). Left: mean ± std across storms; only the lowest forecast NLL is bold. FGN and GenCast have similar track detection rates, computed on all 53 storms. HRES-fc0 is a reference-field baseline. Right: share of the 41 retained storms on which each model has the lower conditional NLL, for each pair and for all three at once. FGN’s NLL advantage over GenCast is smaller than its advantage over IFS-ENS.
Figure 4: Location metrics by lead time (2025, 10m wind, HRES-fc0 reference). Brier score (left), IoU (middle), and Hausdorff distance in kilometres (right), at the 90th percentile. All members of a model share one threshold, the percentile of the pooled member values; the reference uses its own.
Figure 5: Better operational CRPS coexists with persistent intensity bias (2025, HRES-fc0 reference). Left: fair-CRPS reduction of each AI model relative to IFS-ENS, 100(1−CRPSmodel/CRPSIFS-ENS) , in the cyclone region (solid) and globally (dashed); positive favors the AI model. The shaded area is a model’s additional reduction in the cyclone region. FGN leads GenCast in both domains, and both improve more on IFS-ENS in the cyclone region than globally, though GenCast’s global CRPS is above IFS-ENS’s at the first lead. Right: bias of the ensemble-mean maximum 10m wind speed, averaged across storms per lead and normalized by HRES-fc0’s mean maximum wind (19.5 m/s), with ERA5 for comparison. FGN and GenCast have similar negative peak-wind biases at short and intermediate leads despite their different CRPS.
Figure 6: Operational AI forecasts differ from HRES-fc0 in storm intensity, not in how wind follows pressure (2025). Each source contributes minimum pressure and maximum wind speed at timesteps matched to IBTrACS observations, pooled over all lead times. (a) Contours of the pooled joint distributions, with marginal densities. (b) The same points ranked within each source and converted to standard normal scores, so all marginals are identical and the contours show only the dependence. FGN and GenCast have weaker winds and shallower pressures than their fine-tuning reference, HRES-fc0, with marginals close to ERA5’s (a), but every gridded source has nearly the same dependence (b). Dashed and solid contours enclose about 70% and 30% of each source’s points; IBTrACS’s sustained winds differ in definition from the others’ gridded instantaneous winds.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Cycle
Implemented
ENS grid spacing
Levels
47r1
30 June 2020
∼ 18 km
91
47r2
11 May 2021
∼ 18 km
91 → 137
47r3
12 October 2021
∼ 18 km
137
48r1
27 June 2023
18 → 9 km
137
49r1
12 November 2024
9 km
137
Appendix
Table 3: IFS-ENS cycles in operation over the evaluation periods , from ECMWF’s record of changes to the forecasting system. The 2021–2023 storms span four cycles: IFS-ENS is not a fixed baseline within that period, and forecasts after the 48r1 upgrade of June 2023 come from a model with half the grid spacing. All 2025 forecasts come from 49r1.
Figure 7: Stability of the model comparison across subgroups (2021–2023). Track NLL (top row) and cyclone-local CRPS (bottom row) for GenCast, IFS-ENS and FCN3, averaged within subgroups defined by Saffir–Simpson category (left), ocean basin (center) and year (right); n gives the subgroup size. The NLL uses the 106 retained storms and the CRPS all 125, hence the different subgroup sizes. The three models move together across nearly all subgroups.
Figure 8: Stability of the model comparison across subgroups (2025). As in Figure 7 , for FGN, IFS-ENS and operational GenCast, at the same scale; subgroups are defined by Saffir–Simpson category (left) and ocean basin (right). The NLL uses the 41 retained storms and the CRPS all 53. Some basins contain very few storms, down to n=1 , which explains the larger gaps there.
Figure 9: Wind–pressure distributions in the ERA5 setting (2021–2023). Same construction as Figure 6 . All three models have nearly the same dependence as ERA5 (b): ρ is −0.83 for FCN3 and GenCast, −0.84 for IFS-ENS and −0.86 for ERA5, with overlapping 95% intervals. IBTrACS uses sustained rather than gridded instantaneous winds.
Figure 10: Rank histograms of 10m wind footprints. At each grid point of a storm’s region, the reference footprint (maximum wind over the forecast window) is ranked within the members’ footprints; frequencies are averaged across storms. The dashed line is the uniform frequency of a calibrated ensemble.
Figure 11: Wind-field calibration by lead-time range. At each grid point and lead time, the reference 10m wind speed is ranked within the forecast ensemble. Rank frequencies are averaged across storms and within the displayed lead-time ranges; dashed lines indicate uniform ranks. In 2025 (b), the AI ensembles show excesses at both ends at short and intermediate leads, consistent with insufficient spread; the larger upper-rank excess at intermediate leads is also consistent with weak predicted winds. Long-lead histograms are closer to uniform.
Figure 12: Wind-field rank histograms at each lead time (2021–2023, ERA5 reference). As in Figure 11 a, with one panel per 12-hour lead time instead of three ranges.
Figure 13: Wind-field rank histograms at each lead time (2025, HRES-fc0 reference). As in Figure 11 b, with one panel per 12-hour lead time instead of three ranges.
Figure 14: Track error against intensity error, by model (2021–2023, ERA5 reference). Each matched ensemble member contributes its mean track distance to IBTrACS and its mean absolute intensity error, for mean sea-level pressure (left) and 10m wind speed (right). Lines show a LOWESS trend per model with a 95% bootstrap band, over the hexagonal density of the members; the Pearson correlation r and the number of members n are given per model.
Figure 15: Track error against intensity error, by model (2025, HRES-fc0 reference). As in Figure 14 , for FGN, IFS-ENS and operational GenCast.
Figure 16: Location metrics by lead time (2025, HRES-fc0 reference). Columns: Brier score, positive-class Brier score, IoU, Hausdorff distance and G-score; arrows indicate whether higher or lower is better. Rows: sea-level pressure and 10m wind speed, at the 90th percentile. All members of a model share one threshold; the reference uses its own. Lines are means over the 53 storms; dark bands are 95% bootstrap confidence intervals of the mean and light bands ±1 standard deviation across storms.
Figure 17: Location metrics on storm footprints (2025, HRES-fc0 reference). As in Figure 16 , for footprints (maximum wind or minimum pressure over the forecast window) as a function of the percentile threshold.
Figure 18: Energy spectra of global sea-level pressure (top rows) and 10m wind speed (bottom rows) by lead time (2021–2023, ERA5 reference). Lines are the members’ spectra averaged over members and storms, and bands ±1 standard deviation across members; the dashed line is ERA5.
Figure 19: Energy spectra (2025, HRES-fc0 reference). As in Figure 18 , for FGN, IFS-ENS and operational GenCast; the dashed line is HRES-fc0 and the dotted line ERA5.
Can probabilistic artificial intelligence (AI) weather forecasts retain skill beyond their training domain? We compare four state-of-the-art AI models (ArchesWeatherGen, U-Cast, WeatherNext2 and GenCast-mini) with the physics-based IFS-ENS over land in 2022, at 1.5° resolution and lead times of 2--10 days. Monthly ERA5 records from 1979--2021 define common historical boundaries, and we characterize new records set in 2022 by their depth, i.e., the exceedance margin relative to local historical variability. Two proper scores, record-threshold weighted CRPS and Brier score, assess the magnitude and probability of record exceedances across all eligible cases, retaining both missed extremes and false alarms. Verified against ERA5, AI models outperform IFS-ENS for heat and cold records at almost every lead and depth. For heat, the advantage strengthens with depth, partly because IFS-ENS over-forecasts deep records, and largely persists for deeper records under HRES-fc0 verification. For wind, gains erode for deeper records and longer leads. Better tail scores can nevertheless coexist with aggregate probability deficits or surpluses, and some improvements arise chiefly on non-exceedances. Increasing underprediction on progressively more extreme observed subsets can also arise through outcome selection, even for an ideal conditional-mean forecast. These findings establish that native probabilistic AI ensembles retain, and for heat increase, their advantage over IFS-ENS as thresholds move beyond local records, although all advantages narrow with lead time. They caution against generalizing the extrapolation failure reported for deterministic AI forecasts to probabilistic systems, and motivate common verification conventions that separate comparative tail skill, performance on realized extremes, and probabilistic calibration.
Bastiaan Van Velthoven, Cas Decancq, Diego G. Miralles +2
AI weather models are often reported to underestimate extremes, but most evidence concerns deterministic regression models verified against reanalysis. We evaluate twelve physical and AI forecast models against ECMWF IFS using ten months of European station observations. The evaluation covers 10 m wind, 2 m temperature, solar radiation, and precipitation within regimes defined from a fixed ERA5 1991-2020 climatology. We find no uniform AI-specific deficit in the tails. Several AI models remain more accurate than IFS under extreme conditions, while others deteriorate markedly; comparable variation occurs among physical models. Every model nevertheless exhibits a common conditional-error pattern, overpredicting low observations and underpredicting high observations. Attenuation of extreme values therefore does not imply a uniform loss of relative skill: tail performance depends on the model, variable, and evaluation setting rather than on whether the forecast is produced by AI or physical numerical modelling.
Marvin Vincent Gabler, Roberto Molinaro, Niall Siegenheim +5
AI weather models are in the process of revolutionising weather forecasting. While these models have been shown to achieve superior performance to physics-based NWP in forecasting tropical cyclone (TC) tracks, they tend to dramatically underestimate intensity. Here we present AIFS-TC, a simple correction to the AIFS-Single model that is competitive with the operational state-of-the-art for forecasting maximum wind speed and minimum central pressure at lead times of 12 h to seven days. This performance also holds for rapid intensification events. Notably, the entire system was autonomously designed and built by a large language model (Claude Fable 5) in a few hours, directed through a small number of natural-language prompts by a single domain scientist. That the operational frontier can be reached with an open-source AI forecast model (AIFS-Single) and relatively simple, cheap post-processing is significant for TC science, and points to agentic coding as a route to rapid exploration and progress in life-saving early-warning systems in other domains.
Anna Allen, Wessel P. Bruinsma, Michael Maier-Gerber +3
University of Cambridge · 2European Centre for Medium-Range Weather Forecasts