ClimateBench v2.0: Probabilistic Climate Model Benchmarking
Authors: Duncan Watson-Parris, Willa Tobin, Aytaç Paçal, Manuel Schlund, V. Balaji, Kevin Bowman, Chris Bretherton, Peter M. Caldwell, Will Chapman, William D. Collins, Gregory S. Elsaesser, Pierre Gentine, Helene Hewitt, Stephan Hoyer, Ralph Keeling, Nikolay Koldunov, David M. Lawrence, Christian Lessig, Daniel J. Lunt, J. David Neelin, Mike Pritchard, Sarah Purkey, Gavin Schmidt, Tapio Schneider, Michael Schulz, Tiffany Shaw, Isla R. Simpson, Graeme Stephens, Aneesh C. Subramanian, Joao Teixeira, Jessica Tierney, Andrew I. L. Williams, Laure Zanna, Veronika Eyring, Rose Yu
Organizations: Scripps Institution of Oceanography, University of California San Diego · Halicioglu Data Science Institute, University of California San Diego · University of Tübingen, connAIx Research School for Applied AI, Heilbronn, Germany · University of Tübingen, Tübingen AI Center, Tübingen, Germany · Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut für Physik der Atmosphäre, Oberpfaffenhofen, Germany · Schmidt Sciences · Jet Propulsion Laboratory, California Institute of Technology · Allen Institute for Artificial Intelligence · Lawrence Livermore National Laboratory · University of Colorado Boulder · Lawrence Berkeley National Laboratory and the University of East Anglia · Columbia University · NASA Goddard Institute for Space Studies · Met Office, Exeter, UK · Google Research · Alfred Wegener Institute · National Center for Atmospheric Research · Otto von Guericke University Magdeburg · University of Bristol, UK · University of California, Los Angeles · NVIDIA Research · California Institute of Technology · Norwegian Meteorological Institute · University of Chicago · University of Arizona · New York University
We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in projecting mid-century regional temperature and precipitation changes. The protocol is designed to evaluate any physics-based, data-driven, or hybrid climate model on equal footing using a common set of observational and out-of-distribution tests. We define three tiers of evaluation. Tier I establishes physical credibility through entry-ticket tests of energy conservation, coupled (co-)variability, and basic forced responses. Tier II scores models against post-2015 observations of surface temperature, precipitation, radiative fluxes, sea ice, and key modes of variability using fair CRPS as the primary probabilistic score, complemented by distributional and ensemble-consistency diagnostics. Tier III tests out-of-distribution generalization through paleoclimate simulations spanning the Last Interglacial, Last Glacial Maximum, and Mid-Holocene, and through perfect-model experiments in which data-driven models must predict the future climate of existing Earth system models from historical data alone. We reserve all observational data after 2015 for testing, and submissions must include multiple ensemble members to enable probabilistic evaluation. This reservation exploits a new opportunity provided by the decade of observations accumulated since the end of the CMIP6 historical experiment, which constitutes an out-of-sample record of forced climate change (and internal variability) for the current generation of models, and we quantify, in an idealized setting, the information it carries about mid-century warming. We provide the evaluation code, observational reference datasets, and perfect-model training data as an open benchmark to drive measurable progress in climate projection across all modeling approaches.
Figures & tables
Figure 1: Quantifying the information the reserved test period carries about the benchmark target, following Watson-Parris (2025) . (a) Joint posterior over equilibrium climate sensitivity (ECS) and aerosol forcing amplitude, conditioned on the NorESM2-LM Seland et al. (2020) global-mean temperature record to 2014 only (blue) and additionally on the 2015–2025 test window (red); contours enclose 50% and 90% of the posterior mass. (b) The corresponding posteriors for the benchmark target, the 2050 change T(2041 – 2050)−T(1990 – 2020) : adding the test window reduces the posterior variance by ∼50% (42–54% across the three realizations of internal variability) and recenters the distribution on the true forced response of NorESM2-LM (dashed). Full methods are given in C .
Test
Experiment(s)
Pass criterion
Status
Energy balance
piControl
∣μ(N)∣<0.5 W m -2 ; drift ∣δ∣<0.02 W m -2 decade -1
Required
Water budget closure
piControl
∣⟨P⟩−⟨E⟩∣<0.05 mm day -1
Required
Atmospheric energy closure
piControl
∣⟨Qrad⟩−(⟨LP⟩+⟨SHF⟩)∣<2 W m -2
Required
Clear-sky LW feedback
abrupt-4xCO2
forced global-mean Δrlutcs/ΔTs within ±25% of 1.9 W m -2 K -1
Required
Geostrophic balance
historical (daily)
ρ(u,ug) at 850 hPa, 30 – 60∘ , >0.9
Required
Precipitation–buoyancy
historical
P′ –MSE ′ slope (1983–2014) within ±30% of GPCP/ERA5
Required
Table 1: Summary of Tier I entry-ticket tests. All criteria are pass/fail. A required test is recorded as not applicable by architecture for a submission that does not produce the state it diagnoses (Section 3 ). Experiments follow the CMIP6/PMIP4 protocols (Table 4 ); precise definitions and formulae are given in B .
Figure 2: Distribution of each Tier I gate statistic across the 60 CMIP6 models with required Tier I output; shading and dashed lines mark the protocol pass window (Table 1 ). Models inside the pass window are shown in blue and those outside it in red, and each panel gives the number of models passing out of those for which the test could be evaluated. The fast CO 2 forcing test (I.4b), available for only 8 models, is shown as individual points. Panel labels (I.1–I.8b) follow the row order of Table 1 ; extended tests are not shown.
Figure 3: The ClimateBench v2 train/validation/test split. Right: annual global-mean surface temperature anomalies relative to 1850–1900. The blue line and band show the median and 5–95% range across 33 CMIP6 models ( historical to 2014 spliced to SSP2-4.5 thereafter, variant r1i1p1f1 , area-weighted global-mean tas ). The black line and band show the median and range of three observational products (HadCRUT5, GISTEMP v4, Berkeley Earth). Observational products blend land air and sea surface temperatures whereas the model diagnostic is surface air temperature; no blending correction is applied for this illustration. The grey band marks the reserved post-2015 test window and the dashed box the 2041–2050 benchmark target decade (scored relative to 1995–2014; see Section 3 ). Left: assessed global-mean temperature anomalies for the three Tier III paleoclimate periods, also relative to 1850–1900, from IPCC AR6 WG1 Chapter 2; the Last Glacial Maximum lies below the plotted range and is indicated by an arrow.
Variable
Temporal Resolution
Observational Datasets
Status
2m temperature (mean, trends)
Seasonal; annual
NASA GISS Lenssen et al. (2019) , Berkeley Earth Rohde and Hausfather (2020) , HadCRUT Morice et al. (2021) , NOAA GlobalTemp Huang et al. (2020)
Core
2m temperature (extremes, daily PDF)
Daily
Berkeley Earth (daily), HadGHCND Caesar et al. (2006)
Core
Surface temperature
Seasonal; annual
HadSST, HadISST (ocean) Rayner et al. (2003) ; CRU TS (land) Harris et al. (2020)
Core
Precipitation (mean, trends)
Seasonal; annual
GPCP Adler et al. (2018) , IMERG Huffman et al. (2020) , MSWEP Beck et al. (2019)
Core
Precipitation (intensity PDF, extremes)
Daily
IMERG (V07 Final, daily) Huffman et al. (2023) , MSWEP
Core
TOA radiative fluxes (LW, SW; clear-sky and all-sky) and cloud radiative effect
Seasonal; annual
CERES EBAF Loeb et al. (2018)
Core
Table 2: Tier II evaluation variables and observational datasets.
Figure 4: Example Tier II scorecard for the CMIP6 ensemble over the post-2015 test window, computed from their historical and SSP2-4.5 simulations. Each cell gives the relative skill score S=1−Emodel/Eref (Equation 2 ) for one model and one variable, where E is the fair CRPS (Equation 1 ) of the annual global-mean anomaly series over the test window (September Arctic and February Antarctic extent for the sea-ice columns; the land-area mean of the per-gridpoint annual maximum 1-day and 5-day precipitation, Rx1day and Rx5day, for the two extremes columns), and Eref is the median E over the other CMIP6 models (leave-one-out; 14–23 models depending on the variable, as indicated above each column). Every ensemble member, and the reference, is expressed as an anomaly about its own 1985–2014 climatology (2001–2014 for the CERES- and IMERG-referenced columns and 2003–2014 for cloud fraction). Each model’s ensemble comprises its ri1p1f members, capped at the first 10; the 24 CMIP6 models with at least two such historical+SSP2-4.5 members are shown. Cells marked n/a are variables for which that model has only one member, so that the fair CRPS is undefined; blank cells indicate variables not available for that model. Median performance therefore scores S=0 , a perfect match to observations S=1 , and performance worse than the median S<0 ; the score is bounded above but not below, and values below −1 are clipped to the darkest shade. Models are ordered by the mean relative score across variables (rightmost column). The references are HadCRUT5 (2 m temperature), GPCP (precipitation), CERES-EBAF (TOA fluxes, and the cloud radiative effects swcre=rsutcs−rsut and lwcre=rlutcs−rlut ), HadISST (sea-ice extent), MODIS Aqua MYD08_M3 Collection 6.1 (total cloud fraction) and IMERG V07 Final daily precipitation on a 1 ∘ grid (Rx1day, Rx5day); see Table 2 . Where the reference record ends before 2025 the window ends earlier: 2022 for 2 m temperature, 2023 for precipitation, 2024 for the CERES fluxes and the precipitation extremes, and 2021 for sea ice.
Test
Experiment
Reference / target
Metric
Status
Last Interglacial
lig127k
SST Osman (2026) ; Hoffman et al. (2017) ; land T Otto-Bliesner et al. (2021) ; precip. Scussolini et al. (2019) ; Kaushal et al. (2024)
Fair CRPS vs proxies
Extended
Last Glacial Maximum
lgm
SST Tierney et al. (2020) ; Osman et al. (2021) ; land T Bartlein et al. (2011) ; Cleator et al. (2020)
Fair CRPS vs proxies
Required
Mid-Holocene
midHolocene
N. Africa JJAS precip. Bartlein et al. (2011) ; Harrison et al. (2015) ; SST Osman et al. (2021) ; land T Kaufman et al. (2020)
Fair CRPS + monsoon pass/fail ( ≥0.5 mm day -1 )
Extended
Perfect model (CESM2)
historical → SSP2-4.5
Held-out ESM daily tas, pr
Tier II metrics
Required (ML only)
Perfect model (MPI-ESM, GISS ModelE2)
historical → SSP2-4.5
Held-out ESM daily tas, pr
Tier II metrics
Extended (ML only)
Large-ensemble spread
hist + SSP2-4.5 ( > 20 members)
CESM-LE inter-member variability
Variance ratio; spatial correlation
Extended
Table 3: Summary of Tier III out-of-distribution tests. Paleoclimate simulations follow the PMIP4 protocols; perfect-model and large-ensemble tests are most relevant for data-driven models. Scoring uses the seasonal aggregated-diagnostic (fair CRPS) framework of Section 5.1 except where a specific threshold is stated.
Figure 5: Paleoclimate periods as a lever on the Earth-system response to forcing. Global-mean surface temperature anomalies (relative to pre-industrial) against CO 2 radiative forcing, for proxy-based assessments (large circles, with assessed ranges) and individual model simulations (small circles: PMIP4 LGM, PlioMIP mPWP, and DeepMIP EECO ensembles), from the data underlying Figure 7.19 of Forster et al. (2021) , as archived by Lunt (2023) . The LIG assessment is from Fischer et al. (2018) and the Mid-Holocene from Kaufman et al. (2020) . Grey lines indicate constant Earth-system response ratios. Filled circles denote the periods evaluated in Tier III (LGM, LIG, Mid-Holocene); open circles show deeper-time context periods excluded from the protocol because their boundary conditions are less well known. Model simulations are plotted at the central assessed CO 2 forcing of each period, computed as 5.35ln(C/C0) with C0=284 ppm. Note, each period also has significant, and distinct non-CO2 forcing contributions as well.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Simulation
Tier(s)
Ensemble members
Duration (years)
Status
Tier I – Entry Ticket
piControl
I, II
1
≥ 500
Required
amip
I
1
∼ 36
Required
amip-4xCO2
I
1
∼ 36
Required
SST patch (East Pacific)
I
1
∼ 30
Extended
SST patch (West Pacific)
I
1
∼ 30
Extended
Appendix
Table 4: Required and extended simulations across all three tiers, with ensemble requirements and estimated duration.
Variable
Short name(s)
Temporal averaging
Tier(s)
Status
Surface & Atmospheric Temperature
2m air temperature
tas
Daily
I, II, III
Core
Surface temperature (skin)
ts
Monthly
II
Core
Sea surface temperature
tos
Monthly
I, III
Core
Near-surface temp range
dtr
Daily
II
Extended
Precipitation & Water Cycle
Appendix
Table 5: Required output variables by tier, with CMIP short names and temporal resolution.
Figure 6: Tier II scorecard for the aggregated (spatial-pattern) score of Section 5.1 , for the same 24 CMIP6 models and ensemble definition as Figure 4 . For each variable the post-2015 time-mean field of every member is expressed as an anomaly about the reference’s 1985–2014 climatology (2001–2014 for the CERES-referenced columns, 2003–2014 for cloud fraction), projected onto the reference’s EOFs computed over its pre-2015 record (the leading modes reaching 90% of the variance, at most 20), each coefficient standardized by its pre-2015 observed standard deviation, and E is the equal-weight mean over coefficients of the fair CRPS; S and Eref are defined as in Figure 4 . Because the anomaly is taken about the observed rather than the model’s own climatology, the mean-state bias pattern remains in the scored field. Sea-ice extent, a scalar series, has no projection and is omitted.
Climate modeling uses physical simulations and data-driven methods to understand, predict, and project Earth’s climate and its response to external forces.
We evaluate the climate simulation capabilities of ArchesWeather and ArchesWeatherGen, two machine learning models originally trained for weather forecasting and evaluated up to a 10-day lead time. ArchesWeather is a deterministic model, while ArchesWeatherGen is a probabilistic flow-matching model leveraging ArchesWeather's forecasts, enabling ensemble-based uncertainty quantification. In this work, we adapt these models to act as forced atmospheric models by using additional conditioning on the monthly mean sea surface temperature (SST) and sea ice cover (SIC) as boundary conditions. In particular, we follow the AI Model Intercomparison Project (AIMIP) Phase 1 protocol, which, analogous to the Atmospheric Model Intercomparison Project (AMIP), proposes a standardized experimental setup to evaluate the climate skill of ML-based forced atmospheric models. We present a comprehensive evaluation of both models under these conditions, including comparison against numerical climate models, ablation studies that examine key design choices in the extension, and an analysis of forced versus unforced configurations. Despite being originally developed for weather forecasting, we demonstrate that forced configurations of ArchesWeather and ArchesWeatherGen produce stable long-term climate simulations, have a stable annual cycle, and capture the drift of many climate variables. The models faithfully reproduce ERA5's climatology, large-scale circulations and interannual variability, and they capture the tails of the distributions.
Renu Singh, Robert Brunstein, Antonia Jost +5
Google DeepMind, Paris, France · INRIA, Paris, France · Otto von Guericke University Magdeburg, Magdeburg, Germany +3
AI foundation models pretrained on weather and climate data are increasingly fine-tuned to Earth science tasks well beyond weather forecasting. Their development and adoption are outpacing the scientific community's ability to evaluate them. These models are judged almost entirely by benchmark skill metrics, which measure how closely a forecast reproduces a reference product but not whether a model represents the physical processes governing the system it predicts. Forecast skill and physical reliability are therefore distinct properties. The distinction is most consequential under the nonstationary conditions of a changing climate for which these models were never trained. We identify five priorities for the physical evaluation of AI models in Earth science from task-specific emulators to foundation models, spanning training data, fine-tuning, behavioral testing, mechanistic interpretability, and output validation. We recommend three activities for the coming decade: 1) open AI-ready evaluation datasets, 2) a shared reporting standard for physics-based evaluation, and 3) a dedicated research program on the safety of these models.
Comparative studies of predictive models often end by ranking candidate models, yet these rankings depend on evaluation protocols whose influence is rarely treated as a source of uncertainty. We formalize this problem as protocol-induced ranking uncertainty and introduce a framework that compares ranking displacement caused by switching protocols with displacement produced by conventional choices within a fixed protocol. We quantify these effects using the Protocol Sensitivity Score (PSS) and a full-refit intraprotocol reference. We validate the framework in a large-scale sequential prediction study of daily PM10. Static-split and rolling-origin evaluation are compared across 425 European background stations and 365 US EPA monitors. Switching protocols produces mean PSS values of 0.801 and 0.772 and changes the selected model at 35.3% and 31.5% of stations, respectively. In Europe, intraprotocol perturbations with identical scored targets produce PSS values of 0.072 and 0.230, with winner-swap rates of 0.8% and 4.9%. Between-protocol displacement is therefore substantially larger than the selected within-protocol references. Expanding the candidate set from three to nine models increases the between-protocol winner-swap rate to 60.2% in Europe. The pattern also persists under a frozen protocol applied to held-out background and non-background stations. These results show that model-selection conclusions can depend materially on legitimate evaluation choices. We recommend reporting ranking stability under a small set of defensible intraprotocol perturbations alongside claims of model superiority.
Rafael da Silva, Kiersten Monahan
Applied Data Science program, Eastern University 1300 Eagle Road, St. Davids, PA 19087, USA