physics.ao-phSep 26, 2026

Global Evaluation of AI and NWP Precipitation Forecasts During Atmospheric River Events

Authors: Marina Vicens-Miquel, Taylor Mandelbaum, Amy McGovern, Aaron J. Hill, Daniel Rothenberg

Organizations: School of Meteorology and School of Computer Science, University of Oklahoma, Norman, 73072, OK, USA. · NSF AI Institute for Research on Trustworthy AI in Weather, Climate, and Coastal Oceanography, Norman, 73072, OK, USA. · Brightband, San Francisco, CA, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Atmospheric rivers (ARs) produce many of the world's most extreme precipitation events and hydrometeorological hazards. Although artificial intelligence weather prediction (AIWP) models have demonstrated skill comparable to or exceeding numerical weather prediction (NWP) systems for large-scale atmospheric variables, their ability to forecast AR-related precipitation remains insufficiently characterized globally. Here, we evaluate 24-hour precipitation forecasts from the Global Forecast System (GFS), Global Ensemble Forecast System (GEFS), GraphCast, and Artificial Intelligence Forecasting System (AIFS) from Day 1 through Day 10 globally and across North America, Europe, and Australia and New Zealand. Using the Extreme Weather Bench framework, forecasts are evaluated against Integrated Multi-satellitE Retrievals for GPM (IMERG) observations using measures of precipitation magnitude, spatial structure, and localization. GraphCast and AIFS exhibit greater spatial skill than GFS and GEFS, particularly for heavy precipitation and at longer lead times, and better preserve the spatial organization of AR-related precipitation through Day 10. However, this improved spatial skill does not translate into accurate precipitation magnitudes. AIWP models tend to overpredict moderate-to-heavy accumulations while underpredicting the heaviest precipitation at longer lead times, whereas NWP systems develop pronounced dry biases. These results reveal distinct strengths and limitations of AIWP for high-impact precipitation forecasting and provide a reproducible benchmark.

Figures & tables

Figure 1: Global precipitation forecast performance during atmospheric river events from Day 1 through Day 10 for GFS, GEFS Mean, GraphCast, and AIFS. Panels show (a) bias, (b) Pearson correlation, (c) Fractions Skill Score (FSS), and (d) Critical Success Index (CSI). Line styles distinguish overall, rain/no-rain, and 95th-percentile precipitation metrics.
Figure 2: Global distributions of 24-hour precipitation during atmospheric river events at (a) Day 1, (b) Day 5, and (c) Day 10 for IMERG, GFS, GEFS Mean, GraphCast, and AIFS. Curves show the relative frequency of precipitation amounts from 0 to 20 mm and are averaged across all available 24-hour AR evaluation periods with each period weighted equally. The IMERG curve represents the mean of the observational distributions aligned to the native grids of the four forecast systems. Note the logarithmic scale on the vertical axis.
Figure 3: Regional precipitation forecast performance during atmospheric river events from Day 1 through Day 10 for GFS, GEFS Mean, GraphCast, and AIFS across North America (a, d, g, j), Europe (b, e, h, k), and Australia and New Zealand (c, f, i, l). Rows show bias (a–c), Pearson correlation (d–f), Fractions Skill Score (FSS; g–i), and Critical Success Index (CSI; j–l). Bias is shown for the overall precipitation field and the 95th-percentile (p95) threshold, while FSS and CSI are evaluated for rain/no-rain occurrence and p95 precipitation. Metrics are averaged across all 24-hour AR evaluation periods within each region, with each evaluation period weighted equally.
Figure 4: Seasonal variability in regional atmospheric river precipitation forecast performance for GFS, GEFS Mean, GraphCast, and AIFS. Monthly mean rain/no-rain Fractions Skill Score (FSS) and overall precipitation bias are shown for Day 1 (24 h; a–f), Day 5 (120 h; g–l), and Day 10 (240 h; m–r) forecasts across North America (a, d, g, j, m, p), Europe (b, e, h, k, n, q), and Australia and New Zealand (c, f, i, l, o, r). Within each lead time, the upper row shows FSS and the lower row shows bias. Metrics are averaged across all AR evaluation periods occurring within each calendar month.
Figure 5: Representative North American atmospheric river event valid from 1 July 2023 at 06:00 UTC to 2 July 2023 at 05:00 UTC. IMERG 24-hour accumulated precipitation is compared with GFS, GEFS Mean, GraphCast, and AIFS forecasts at 10-, 5-, 3-, and 1-day lead times. The black contour indicates the AR mask used to define the event.
Figure 6: As in Fig. 5 , but for a representative European atmospheric river event valid from 4 November 2023 at 00:00 UTC to 23:00 UTC.
Figure 7: As in Fig. 5 , but for a representative Australia and New Zealand atmospheric river event valid from 1 May 2023 at 00:00 UTC to 23:00 UTC.
Figure 8: Schematic of the AR event construction procedure. a , Hourly AR objects identified by EWB are linked through time to construct continuous AR events. Each AR mask (blue) is expanded by a 250-km tracking buffer (gray), and objects at the subsequent timestep are linked when they overlap the buffered mask. The union of the linked hourly detections defines the continuous AR event footprint. The tracking buffer is distinct from the buffer used for precipitation evaluation. b , Temporal gap-handling procedure. Tracks may remain open for up to 3 consecutive hours without an AR detection. A spatially matched AR reappearing within this period is reconnected to the original track only when the new segment persists for at least 6 consecutive detected hours.
Figure 9: Schematic of the AR precipitation evaluation framework. a , Continuous AR events are divided into 24-h evaluation windows beginning at 00, 06, 12, and 18 UTC. For ARs persisting longer than 24 h, windows are generated at 6-h intervals, resulting in an 18-h overlap. Only windows with AR detections during all 24 consecutive hours are retained. b , Spatial evaluation domains for three consecutive 24-h windows. Hourly AR masks are combined to define the AR footprint for each window (black), and a 250-km buffer is applied to define the precipitation evaluation domain (red).
Evaluation DomainUnique AR Events24-h Evaluation Windows
Global3,56446,140
North America5665,661
Europe175842
Australia and New Zealand3292,652
Table 1: Number of unique atmospheric river (AR) events and 24-hour precipitation evaluation windows included in the global and regional analyses.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Regional precipitation bias with 95% bootstrap confidence intervals. Mean bias for GFS, GEFS Mean, GraphCast, and AIFS at Day 1 ( a–c ), Day 5 ( d–f ), and Day 10 ( g–i ). Columns correspond to North America ( a,d,g ), Europe ( b,e,h ), and Australia and New Zealand ( c,f,i ). Circles show overall bias and squares show bias conditioned on observed precipitation exceeding the local 95th-percentile threshold. Error bars indicate 95% confidence intervals obtained from 5,000 event-level clustered bootstrap replicates.
Figure 11: Regional Pearson correlation with 95% bootstrap confidence intervals. Mean Pearson correlation for GFS, GEFS Mean, GraphCast, and AIFS at Day 1 ( a–c ), Day 5 ( d–f ), and Day 10 ( g–i ). Columns correspond to North America ( a,d,g ), Europe ( b,e,h ), and Australia and New Zealand ( c,f,i ). Error bars indicate 95% confidence intervals obtained from 5,000 event-level clustered bootstrap replicates.
Figure 12: Regional Fractions Skill Score with 95% bootstrap confidence intervals. Mean Fractions Skill Score (FSS) for GFS, GEFS Mean, GraphCast, and AIFS at Day 1 ( a–c ), Day 5 ( d–f ), and Day 10 ( g–i ). Columns correspond to North America ( a,d,g ), Europe ( b,e,h ), and Australia and New Zealand ( c,f,i ). Circles show rain/no-rain FSS and squares show FSS evaluated at the local 95th-percentile precipitation threshold. Error bars indicate 95% confidence intervals obtained from 5,000 event-level clustered bootstrap replicates.
Figure 13: Regional Critical Success Index with 95% bootstrap confidence intervals. Mean Critical Success Index (CSI) for GFS, GEFS Mean, GraphCast, and AIFS at Day 1 ( a–c ), Day 5 ( d–f ), and Day 10 ( g–i ). Columns correspond to North America ( a,d,g ), Europe ( b,e,h ), and Australia and New Zealand ( c,f,i ). Circles show rain/no-rain CSI and squares show CSI evaluated at the local 95th-percentile precipitation threshold. Error bars indicate 95% confidence intervals obtained from 5,000 event-level clustered bootstrap replicates.

Explore similar work

CardsList

  1. Improving global precipitation forecasts with an AI weather model trained on satellite observations

    Sep 2, 2026Julian F. Schmitt, Bertrand Delorme, Robert C. King +4Precipitation ForecastingProbabilistic Forecasting

    X, The Moonshot Factory, Google LLC, Mountain View, CA, USA · Google, Google LLC, Mountain View, CA, USA · Environmental Science and Engineering, California Institute of Technology, Pasadena, CA, USA +1

  2. West-WRF AI 2-km: High-Resolution Prediction of Integrated Vapor Transport and Precipitation

    Sep 22, 2026Nazak Rouzegari, Vesta Afzali Gorooh, Agniv Sengupta +6Precipitation ForecastingWeather Forecasting

    Center for Hydrometeorology and Remote Sensing (CHRS), The Henry Samueli School of Engineering, Department of Civil and Environmental Engineering, University of California, Irvine, USA · Center for Western Weather and Water Extremes (CW3E), Scripps Institution of Oceanography, University of California, San Diego, USA

  3. Suitable Measures for the Potential Operational Utility of AI NWP Rainfall Forecasts Over Africa

    Sep 24, 2026Shruti Nath, Docko Sow, Koomi Toussaint Amoussouvi +5Weather ForecastingPrecipitation Forecasting

    Department of Physics, University of Oxford, United Kingdom · AfriClimate AI, South Africa · ECMWF, Reading, United Kingdom +1