cs.LGOct 15, 2026
SaveCan AI Forecast the Extremes? Evaluating Probabilistic Weather Models on Tropical Cyclones
Organizations: INRIA, Paris, France · AXA, Paris, France · Sorbonne Université, CNRS, LIP6, F-75005 Paris, France · Polish Academy of Science, IBS PAN, Warsaw, Poland
Abstract
Weather prediction is undergoing an important shift, with data-driven AI models outperforming leading numerical weather prediction systems on average at a fraction of the computational cost. Yet, whether this success extends to extreme events remains contested. Earlier evaluations have exposed important shortcomings, but have largely focused on deterministic models and metrics, leaving the capabilities of the latest probabilistic models less understood. We conduct an extensive benchmark of tropical cyclone forecasting, comparing GenCast, FourCastNet 3, and FGN with ECMWF's operational physics-based ensemble, IFS-ENS. We develop a suite of probabilistic metrics and diagnostic analyses to assess cyclone tracking, intensity biases, uncertainty, and the wind-pressure relationship that characterizes storm structure. We find that, despite remaining limitations in intensity and ensemble spread, leading AI models substantially outperform IFS-ENS in cyclone tracking: FGN achieves better probabilistic location scores for 90% of evaluated storms in 2025. Further analyses reveal model-specific failure modes and suggest a lingering influence of pretraining data, pointing to opportunities for targeted improvement. Nonetheless, these results suggest that AI models are capable of capturing physical relationships that remain predictive well beyond typical weather conditions, motivating further investigation into how these capabilities emerge and how to strengthen them.
Figures & tables
Figure 1: Evaluation windows of the benchmark. To evaluate probabilistic models on tropical cyclones, evaluation years are chosen according to each model’s training and fine-tuning periods.
Figure 2: Typhoon Bualoi (2025). FGN (red) and IFS-ENS (blue) forecasts at four lead times: dots are individual members, contours are their density, and black dots are the observed IBTrACS positions along the track (grey line). IFS-ENS’s members separate into distinct clusters at intermediate leads, which an ensemble mean cannot represent.
| Model | Track detection rate | Cyclone track NLL ( ) |
|---|---|---|
| GenCast | % | |
| FCN3 | % | |
| IFS-ENS | % | |
| ERA5 (baseline) | % | — |
Table 1: Trajectory NLL and track detection rates (2021–2023). Left: mean std across storms; only the lowest forecast NLL is bold. The track detection rate is the percentage of members with a detected track matched to IBTrACS, computed on all 125 storms. ERA5 is a reference-field baseline. Right: share of the 106 retained storms on which each model has the lower conditional NLL, for each pair and for all three at once.
Figure 3: Better cyclone-local CRPS need not mean less intensity bias (2021–2023, ERA5 reference). Left: reduction of GenCast’s summed normalized fair-CRPS relative to FCN3’s, , in cyclone regions (solid) and globally (dashed); positive favors GenCast. The shaded area is GenCast’s additional reduction in the cyclone region: 3–5% from 24–220 h, versus about 1% globally. Right: bias of the ensemble-mean maximum 10m wind speed, averaged across storms per lead and normalized by ERA5’s mean maximum wind (16.6 m/s). GenCast underestimates peak winds at short and intermediate leads despite its better local CRPS.
| Model | Track detection rate | Cyclone track NLL ( ) |
|---|---|---|
| FGN | % | |
| GenCast Operational | % | |
| IFS-ENS | % | |
| HRES-fc0 (baseline) | % | — |
Table 2: Trajectory NLL and track detection rates (2025). Left: mean std across storms; only the lowest forecast NLL is bold. FGN and GenCast have similar track detection rates, computed on all 53 storms. HRES-fc0 is a reference-field baseline. Right: share of the 41 retained storms on which each model has the lower conditional NLL, for each pair and for all three at once. FGN’s NLL advantage over GenCast is smaller than its advantage over IFS-ENS.
Figure 4: Location metrics by lead time (2025, 10m wind, HRES-fc0 reference). Brier score (left), IoU (middle), and Hausdorff distance in kilometres (right), at the 90th percentile. All members of a model share one threshold, the percentile of the pooled member values; the reference uses its own.
Figure 5: Better operational CRPS coexists with persistent intensity bias (2025, HRES-fc0 reference). Left: fair-CRPS reduction of each AI model relative to IFS-ENS, , in the cyclone region (solid) and globally (dashed); positive favors the AI model. The shaded area is a model’s additional reduction in the cyclone region. FGN leads GenCast in both domains, and both improve more on IFS-ENS in the cyclone region than globally, though GenCast’s global CRPS is above IFS-ENS’s at the first lead. Right: bias of the ensemble-mean maximum 10m wind speed, averaged across storms per lead and normalized by HRES-fc0’s mean maximum wind (19.5 m/s), with ERA5 for comparison. FGN and GenCast have similar negative peak-wind biases at short and intermediate leads despite their different CRPS.
Figure 6: Operational AI forecasts differ from HRES-fc0 in storm intensity, not in how wind follows pressure (2025). Each source contributes minimum pressure and maximum wind speed at timesteps matched to IBTrACS observations, pooled over all lead times. (a) Contours of the pooled joint distributions, with marginal densities. (b) The same points ranked within each source and converted to standard normal scores, so all marginals are identical and the contours show only the dependence. FGN and GenCast have weaker winds and shallower pressures than their fine-tuning reference, HRES-fc0, with marginals close to ERA5’s (a), but every gridded source has nearly the same dependence (b). Dashed and solid contours enclose about 70% and 30% of each source’s points; IBTrACS’s sustained winds differ in definition from the others’ gridded instantaneous winds.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Cycle | Implemented | ENS grid spacing | Levels |
|---|---|---|---|
| 47r1 | 30 June 2020 | 18 km | 91 |
| 47r2 | 11 May 2021 | 18 km | 91 137 |
| 47r3 | 12 October 2021 | 18 km | 137 |
| 48r1 | 27 June 2023 | 18 9 km | 137 |
| 49r1 | 12 November 2024 | 9 km | 137 |
Table 3: IFS-ENS cycles in operation over the evaluation periods , from ECMWF’s record of changes to the forecasting system. The 2021–2023 storms span four cycles: IFS-ENS is not a fixed baseline within that period, and forecasts after the 48r1 upgrade of June 2023 come from a model with half the grid spacing. All 2025 forecasts come from 49r1.
Figure 7: Stability of the model comparison across subgroups (2021–2023). Track NLL (top row) and cyclone-local CRPS (bottom row) for GenCast, IFS-ENS and FCN3, averaged within subgroups defined by Saffir–Simpson category (left), ocean basin (center) and year (right); gives the subgroup size. The NLL uses the 106 retained storms and the CRPS all 125, hence the different subgroup sizes. The three models move together across nearly all subgroups.
Figure 8: Stability of the model comparison across subgroups (2025). As in Figure 7 , for FGN, IFS-ENS and operational GenCast, at the same scale; subgroups are defined by Saffir–Simpson category (left) and ocean basin (right). The NLL uses the 41 retained storms and the CRPS all 53. Some basins contain very few storms, down to , which explains the larger gaps there.
Figure 9: Wind–pressure distributions in the ERA5 setting (2021–2023). Same construction as Figure 6 . All three models have nearly the same dependence as ERA5 (b): is for FCN3 and GenCast, for IFS-ENS and for ERA5, with overlapping 95% intervals. IBTrACS uses sustained rather than gridded instantaneous winds.
Figure 10: Rank histograms of 10m wind footprints. At each grid point of a storm’s region, the reference footprint (maximum wind over the forecast window) is ranked within the members’ footprints; frequencies are averaged across storms. The dashed line is the uniform frequency of a calibrated ensemble.
Figure 11: Wind-field calibration by lead-time range. At each grid point and lead time, the reference 10m wind speed is ranked within the forecast ensemble. Rank frequencies are averaged across storms and within the displayed lead-time ranges; dashed lines indicate uniform ranks. In 2025 (b), the AI ensembles show excesses at both ends at short and intermediate leads, consistent with insufficient spread; the larger upper-rank excess at intermediate leads is also consistent with weak predicted winds. Long-lead histograms are closer to uniform.
Figure 12: Wind-field rank histograms at each lead time (2021–2023, ERA5 reference). As in Figure 11 a, with one panel per 12-hour lead time instead of three ranges.
Figure 13: Wind-field rank histograms at each lead time (2025, HRES-fc0 reference). As in Figure 11 b, with one panel per 12-hour lead time instead of three ranges.
Figure 14: Track error against intensity error, by model (2021–2023, ERA5 reference). Each matched ensemble member contributes its mean track distance to IBTrACS and its mean absolute intensity error, for mean sea-level pressure (left) and 10m wind speed (right). Lines show a LOWESS trend per model with a 95% bootstrap band, over the hexagonal density of the members; the Pearson correlation and the number of members are given per model.
Figure 15: Track error against intensity error, by model (2025, HRES-fc0 reference). As in Figure 14 , for FGN, IFS-ENS and operational GenCast.
Figure 16: Location metrics by lead time (2025, HRES-fc0 reference). Columns: Brier score, positive-class Brier score, IoU, Hausdorff distance and G-score; arrows indicate whether higher or lower is better. Rows: sea-level pressure and 10m wind speed, at the 90th percentile. All members of a model share one threshold; the reference uses its own. Lines are means over the 53 storms; dark bands are 95% bootstrap confidence intervals of the mean and light bands standard deviation across storms.
Figure 17: Location metrics on storm footprints (2025, HRES-fc0 reference). As in Figure 16 , for footprints (maximum wind or minimum pressure over the forecast window) as a function of the percentile threshold.
Figure 18: Energy spectra of global sea-level pressure (top rows) and 10m wind speed (bottom rows) by lead time (2021–2023, ERA5 reference). Lines are the members’ spectra averaged over members and storms, and bands standard deviation across members; the dashed line is ERA5.
Figure 19: Energy spectra (2025, HRES-fc0 reference). As in Figure 18 , for FGN, IFS-ENS and operational GenCast; the dashed line is HRES-fc0 and the dotted line ERA5.