Assessing the Robustness of Climate Foundation Models under No-Analog Distribution Shifts
Authors: Maria Conchita Agana Navarro, Geng Li, Theo Wolf, Maria Perez-Ortiz
Organizations: Centre for Artificial Intelligence Department of Computer Science University College London, UK · Division of Emerging Interdisciplinary Areas Hong Kong University of Science and Technology Hong Kong · Department of Computer Science University of Oxford, UK · Centre for Artificial Intelligence Department of Computer Science University College London
The accelerating pace of climate change introduces profound non-stationarities that challenge the ability of Machine Learning based climate emulators to generalize beyond their training distributions. While these emulators offer computationally efficient alternatives to traditional Earth System Models, their reliability remains a potential bottleneck under "no-analog" future climate states, which we define here as regimes where external forcing drives the system into conditions outside the empirical range of the historical training data. A fundamental challenge in evaluating this reliability is data contamination; because many models are trained on simulations that already encompass future scenarios, true out-of-distribution (OOD) performance is often masked. To address this, we benchmark the OOD robustness of three state-of-the-art architectures: U-Net, ConvLSTM, and the ClimaX foundation model specifically restricted to a historical-only training regime (1850-2014). We evaluate these models using two complementary strategies: (i) temporal extrapolation to the recent climate (2015-2023) and (ii) cross-scenario forcing shifts across divergent emission pathways. Our analysis within this experimental setup reveals an accuracy vs. stability trade-off: while the ClimaX foundation model achieves the lowest absolute error, it exhibits higher relative performance changes under distribution shifts, with precipitation errors increasing by up to 8.44% under extreme forcing scenarios. These findings suggest that when restricted to historical training dynamics, even high-capacity foundation models are sensitive to external forcing trajectories. Our results underscore the necessity of scenario-aware training and rigorous OOD evaluation protocols to ensure the robustness of climate emulators under a changing climate.
Figures & tables
Dataset Role
Scenario(s)
Year Range
Split Strategy
Training (Train)
Historical
1850–2014
90% random samples
In-Distribution (ID Test)
Historical
1850–2014
10% held-out random samples
Benchmark 1 (OOD Test)
Historical
2015–2023
Time-Domain Shift
Benchmark 2 (OOD Test)
SSP1-2.6, SSP5-8.5
2015–2100
Source-Domain Shift
Table 1: Data Partitioning for ML Training and OOD Benchmarking
Figure 1: Comparison of surface air temperature (TAS) and precipitation (PR) test LL-RMSE for ML models across Baseline and temporal shift test sets.
U-Net
Conv-LSTM
ClimaX
Model
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
-2.90
2.31
-3.55
1.92
4.82
5.18
MPI-ESM1-2-HR
-1.28
1.59
-1.97
0.43
2.61
2.83
Table 2: Percent change (%) for ML Models emulating surface air temperature (TAS) and precipitation (PR) under temporal shift benchmark.
Figure 2: Comparison of surface air temperature (TAS) and precipitation (PR) test LL-RMSE for ML models across Baseline, SSP1-2.6 and SSP5-8.5 test sets.
SSP1-2.6
U-Net
Conv-LSTM
ClimaX
Model
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
-3.58
4.18
-4.23
3.77
4.61
4.76
MPI-ESM1-2-HR
-1.92
3.05
-2.61
1.88
4.18
3.99
SSP5-8.5
U-Net
Conv-LSTM
ClimaX
Model
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
-3.92
7.15
-4.57
6.74
8.18
8.44
Table 3: Percent change (%) for ML Models emulating surface air temperature (TAS) and precipitation (PR) under SSP1-2.6 scenario test set
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Scenario
U-Net
Conv-LSTM
ClimaX
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
Time Shift
0.971
1.075
0.971
1.075
0.829
0.834
Baseline
1.000
1.051
1.007
1.055
0.791
0.793
MPI-ESM1-2-HR
Time Shift
0.985
1.080
0.985
1.080
0.831
0.858
Baseline
0.998
1.063
1.005
1.075
0.810
0.834
Appendix
Table 4: Baseline and Temporal Shift LL-RMSE values for each ML model: surface air temperature (TAS) and precipitation (PR) predictions
Model
Scenario
U-Net
Conv-LSTM
ClimaX
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
SSP1-2.6
0.965
1.095
0.965
1.095
0.828
0.831
Baseline
1.000
1.051
1.007
1.055
0.791
0.793
MPI-ESM1-2-HR
SSP1-2.6
0.979
1.095
0.979
1.095
0.844
0.868
Baseline
0.998
1.063
1.005
1.075
0.810
0.834
Appendix
Table 5: Baseline and SSP1-2.6 LL-RMSE values for each ML model, surface air temperature (TAS) and precipitation (PR) predictions
Model
Scenario
U-Net
Conv-LSTM
ClimaX
TAS
PR
TAS
PR
TAS
PR
EC-Earth3
SSP5-8.5
0.961
1.126
0.961
1.126
0.856
0.860
Baseline
1.000
1.051
1.007
1.055
0.791
0.793
MPI-ESM1-2-HR
SSP5-8.5
0.971
1.113
0.971
1.113
0.861
0.892
Baseline
0.998
1.063
1.005
1.075
0.810
0.834
Appendix
Table 6: Baseline and SSP5-8.5 LL-RMSE values for each ML model: surface air temperature (TAS) and precipitation (PR) predictions
Climate emulation is an out-of-distribution (OOD) projection task. This is precisely the challenge where modern Machine Learning (ML) methods are most prone to failure. Consequently, while current ML emulators trained on present climate achieve high in-distribution performance, their future reliability under the inevitable distribution shifts of a changing climate remains a critical, poorly understood blind spot. Addressing this challenge requires a fundamental shift in how we understand, evaluate, and design climate emulators. In this work, we first confirm that climate change drives a statistically significant and progressively growing shift in atmospheric state distributions, rendering standard evaluation protocols insufficient. We empirically establish that seasonal variation serves as an effective proxy for these long-term climate shifts, providing access to real-world distribution shifts without recourse to heuristics like synthetic perturbations. Motivated by this link, we introduce a novel evaluation framework that leverages seasonal shifts as a rigorous, zero-overhead testbed for emulator robustness. Our systematic characterisation confirms that current state-of-the-art hybrid-ML emulators degrade significantly under these realistic shifts. Finally, we chart a path forward by identifying compositional generalisation, the ability to form novel combinations from observed elementary components, as a principled route towards robust climate emulation. We demonstrate that physically motivated decompositions substantially improve OOD performance with only modest trade-offs against in-distribution performance, providing an avenue towards ML-driven climate emulators robust to an unknown future.
Bradley Stanley-Clamp, Anson Lei, Hannah M. Christensen +1
Applied AI Lab · University of Oxford, UK · Atmospheric, Oceanic and Planetary Physics
Current Earth observation benchmarks focus on measuring performance on diverse tasks and applications, typically measuring generalization in-distribution. But when models are deployed, they must generalize to myriad out-of-distribution scenarios, such as new time periods, geographies, scales, and sensors. We introduce EarthShift: the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing. EarthShift enables users to measure distributional robustness by comparing performance in- and out-of-distribution using datasets from paired datasets from different sources, temporal windows, geographic locations, and sensors. Our experiments on 8 geospatial foundation models (GFMs) and 11 tasks covering 5 shift types show that GFMs consistently perform 15-20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy. We show that GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models. This highlights a need for future research to strive for improvements in distributional robustness, not just performance, which can be benchmarked using EarthShift. We release our code and datasets to provide a testbed to guide future work to create foundation models that are robust and reliable in real-world applications. Code and data for EarthShift are available at: https://earthshift.github.io
Kelsey Doerksen, Hannah Kerner
School of Computing and Augmented Intelligence Arizona State University
Foundation models (FMs) are increasingly deployed in open-world settings where distribution shift is the rule rather than the exception. The out-of-distribution (OOD) phenomena they face -- knowledge boundaries, capability ceilings, compositional shifts, and open-ended task variation -- differ in kind from the settings that have shaped prior OOD research, and are further complicated because the pretraining and post-training distributions of modern FMs are often only partially observed. Our position is that OOD for foundation models is a structurally distinct problem that cannot be solved within the prevailing model-centric paradigm, and that agentic systems constitute the missing paradigm required to address it. We defend this claim through four steps. First, we give a stage-aware formalization of OOD that accommodates partially observed multi-stage training distributions. Second, we prove a parameter coverage ceiling: there exist practically relevant inputs that no model-centric method (training-time or test-time) can handle within tolerance ε, for reasons intrinsic to parameter-based representation. Third, we characterize agentic OOD systems by four structural properties -- perception, strategy selection, external action, and closed-loop verification -- and show that they strictly extend the reachable set beyond the ceiling. Fourth, we respond to seven counterarguments, conceding two, and outline a research agenda. We do not claim that agentic methods subsume model-centric ones; we argue that the two are complementary, and that progress on FM-OOD requires explicit recognition of the agentic paradigm as a first-class research direction.