Organizations: Physical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · Department of Ocean Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia.
Deep-learning models increasingly rival numerical simulation in Earth-system forecasting, yet forecast error alone cannot determine whether their learned input-output dependencies are physically consistent. Here we introduce sensitivity fidelity as an additional criterion: alongside prediction error, model responses to input perturbations should be examined against a trusted reference or established physical knowledge. In an idealized Rayleigh-Bénard convection benchmark, four surrogate architectures achieved low forecast errors yet encoded markedly different sensitivities. Quantitative comparison with a matched adjoint distinguished the models far more sharply than forecast accuracy and showed that predictive skill and sensitivity fidelity need not coincide, while finite-perturbation experiments further linked these differences to the propagation of initial-condition uncertainty. We then applied this strategy to El Niño-Southern Oscillation prediction, where no adjoint reference exists and learned sensitivities showed event-wise robustness, validity under finite perturbations, and consistency with established precursors. These results identify gradient-based evaluation as a discriminating complement to forecast-error metrics, providing a practical test of whether data-driven Earth-system models learn physically credible dependencies.
Figures & tables
Figure 1: Sensitivity-fidelity workflow for differentiable Earth-system models . (a) A trained DL forecast model maps an input state or input sequence X to a forecast Y^ , from which a scalar objective J(Y^) is selected. (b) Backpropagation through the trained model yields the input sensitivity SX=∂J(Y^)/∂X , which characterizes the learned local, first-order response of the objective to input perturbations. In the RB benchmark, neural sensitivities are directly compared with sensitivities from a matched physics-based adjoint for fidelity validation. In the ENSO application, where no adjoint reference is available, sensitivities are assessed through statistical robustness across events, quantitative agreement with the model’s own finite-perturbation responses, and consistency with established physical mechanisms. These diagnostics provide complementary indirect evidence for evaluating learned sensitivities.
Figure 2: Forecast performance of different surrogate models for two-dimensional RB convection . (a) Full-field relative root-mean-square error (RMSE) on the held-out test set as a function of forecast lead step for IFactFormer-RB, FNO, AFNO, and Swin Transformer. (b) Corresponding relative RMSE of the predicted Nusselt number. In (a) and (b), colors distinguish the four models, solid lines show the mean relative RMSE across held-out test trajectories at each forecast lead step, and shaded bands indicate the corresponding 95% bootstrap confidence intervals. (c) Ground-truth states and forecast residuals for a representative held-out trajectory at lead steps 50 (top row) and 100 (bottom row). The first column shows the ground-truth temperature and velocity fields; the remaining columns show predictions minus ground truth for IFactFormer-RB, FNO, AFNO, and Swin Transformer, respectively. Color shading denotes temperature T or temperature residuals ΔT=T^−T , and arrows denote velocity u or velocity-vector residuals Δu=u^−u , with separate color scales and reference arrows for the ground-truth and residual fields.
Figure 3: Comparison of surrogate sensitivity fields with the adjoint reference for the RB forecast metric . The forecast metric is defined as the mean temperature within the target region at forecast lead step 20. (a) RB states and corresponding sensitivity fields for one held-out test trajectory. The top row shows the state at the input time ( t=10.0 ) and at forecast lead steps 6, 13, and 19. Color shading represents temperature, and arrows represent velocity. The remaining rows show the corresponding sensitivity fields computed using the adjoint model, IFactFormer-RB, FNO, AFNO, and Swin Transformer. In these rows, color shading represents sensitivity to temperature, and arrows represent sensitivity to velocity. White boxes identify the target region in the state panels, and black boxes identify the same region in the sensitivity panels. (b) Relative L2 error between each surrogate sensitivity field and the adjoint reference. (c) Averaged cosine similarity between each surrogate sensitivity field and the adjoint reference. In (b) and (c), boxplots show the distributions of the corresponding metrics across the held-out test trajectories. Crosses indicate sample means, and shaded bars indicate the 95% bootstrap confidence intervals of the means.
Figure 4: Sensitivity of the ENSO forecast metric to SST and wind-stress fields during El Niño events . (a–l) Evolution of composite CTEFNet gradient-based sensitivities over the 12 months preceding the target periods of El Niño events. The SST and wind-stress components of the full multivariate sensitivity field are shown, with only the grid points significant at the 5% level retained, based on a two-sided Student’s t test for the SST sensitivity and Hotelling’s T2 test for the zonal and meridional wind-stress sensitivities treated jointly. Shading indicates sensitivity to SST, and vectors represent sensitivities to zonal and meridional wind stress. The green box outlines the target Niño 3.4 region.
Machine-learning models for physical systems are currently evaluated primarily through errors between predicted and reference states and, increasingly, through tests of physical consistency. These metrics assess whether predictions are accurate and satisfy selected physical requirements, but provide limited insight into whether learned trajectories reproduce the underlying dynamics. Domain experts examine such relationships through physical representations that expose relevant processes, interactions, and responses, but these analyses are often separated from typical machine-learning evaluation. We introduce a practical framework for evaluating dynamical fidelity through predictive structure in physical representation spaces. Experts define the representations, while reference trajectories determine which relationships are predictive and retained as evaluation tests. We demonstrate the approach in atmospheric forecasting using ERA5 representations of planetary-wave activity and Northern Annular Mode evolution, and evaluate Pangu-Weather, GraphCast, and FengWu. The models exhibit distinct departures from reference predictive structure that are not reflected by conventional forecast errors. The framework thereby turns domain-expert representations into systematic tests of learned physical dynamics without prescribing the relationships in advance.
Oskar Bohn Lassen, Joao Paulo de Souza Boger, Simon Driscoll +4
Department of Technology, Management, and Economics Technical University of Denmark · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Department of Mathematics and Statistics University of Exeter
AI foundation models pretrained on weather and climate data are increasingly fine-tuned to Earth science tasks well beyond weather forecasting. Their development and adoption are outpacing the scientific community's ability to evaluate them. These models are judged almost entirely by benchmark skill metrics, which measure how closely a forecast reproduces a reference product but not whether a model represents the physical processes governing the system it predicts. Forecast skill and physical reliability are therefore distinct properties. The distinction is most consequential under the nonstationary conditions of a changing climate for which these models were never trained. We identify five priorities for the physical evaluation of AI models in Earth science from task-specific emulators to foundation models, spanning training data, fine-tuning, behavioral testing, mechanistic interpretability, and output validation. We recommend three activities for the coming decade: 1) open AI-ready evaluation datasets, 2) a shared reporting standard for physics-based evaluation, and 3) a dedicated research program on the safety of these models.
Data-driven neural networks are increasingly used as surrogate forward models in geophysics, but it remains unclear whether they recover only the data mapping or also the underlying physical sensitivity structure. Here we test this question using surface-wave dispersion. By comparing automatically differentiated gradients from a neural-network surrogate with theoretical sensitivity kernels, we show that the learned gradients can recover the main depth-dependent structure of physical kernels across a broad range of periods. This indicates that neural surrogate models can learn physically meaningful differential information, rather than acting as purely black-box predictors. At the same time, strong structural priors in the training distribution can introduce systematic artifacts into the inferred sensitivities. Our results show that neural forward surrogates can recover useful physical information for inversion and uncertainty analysis, while clarifying the conditions under which this differential structure remains physically consistent.
Ziye Yu, Yuqi Cai, Xin Liu
Institute of Geophysics, China Earthquake Administration, Beijing 100081, China · Laboratory of Seismology and Physics of Earth’s Interior, School of Earth and Space Sciences, University of Science and Technology of China, Hefei 230026, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230088, China