Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models

Qi Chen; Peng Zhan; Hongxing Cui; Wenqing Cui; Ibrahim Hoteit

cs.LGOct 11, 2026

Sensitivity Fidelity Reveals Physical Dependencies Beyond Forecast Accuracy in Deep-Learning Earth-System Models

Authors: Qi Chen, Peng Zhan, Hongxing Cui, Wenqing Cui, Ibrahim Hoteit

Organizations: Physical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · Department of Ocean Science and Engineering, Southern University of Science and Technology, Shenzhen, China. · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia.

Abstract

Deep-learning models increasingly rival numerical simulation in Earth-system forecasting, yet forecast error alone cannot determine whether their learned input-output dependencies are physically consistent. Here we introduce sensitivity fidelity as an additional criterion: alongside prediction error, model responses to input perturbations should be examined against a trusted reference or established physical knowledge. In an idealized Rayleigh-Bénard convection benchmark, four surrogate architectures achieved low forecast errors yet encoded markedly different sensitivities. Quantitative comparison with a matched adjoint distinguished the models far more sharply than forecast accuracy and showed that predictive skill and sensitivity fidelity need not coincide, while finite-perturbation experiments further linked these differences to the propagation of initial-condition uncertainty. We then applied this strategy to El Niño-Southern Oscillation prediction, where no adjoint reference exists and learned sensitivities showed event-wise robustness, validity under finite perturbations, and consistency with established precursors. These results identify gradient-based evaluation as a discriminating complement to forecast-error metrics, providing a practical test of whether data-driven Earth-system models learn physically credible dependencies.

Figures & tables

Figure 1: Sensitivity-fidelity workflow for differentiable Earth-system models . (a) A trained DL forecast model maps an input state or input sequence $X$ to a forecast $\hat{Y}$ , from which a scalar objective $J(\hat{Y})$ is selected. (b) Backpropagation through the trained model yields the input sensitivity $S_{X}=\partial J(\hat{Y})/\partial X$ , which characterizes the learned local, first-order response of the objective to input perturbations. In the RB benchmark, neural sensitivities are directly compared with sensitivities from a matched physics-based adjoint for fidelity validation. In the ENSO application, where no adjoint reference is available, sensitivities are assessed through statistical robustness across events, quantitative agreement with the model’s own finite-perturbation responses, and consistency with established physical mechanisms. These diagnostics provide complementary indirect evidence for evaluating learned sensitivities.

Figure 2: Forecast performance of different surrogate models for two-dimensional RB convection . (a) Full-field relative root-mean-square error (RMSE) on the held-out test set as a function of forecast lead step for IFactFormer-RB, FNO, AFNO, and Swin Transformer. (b) Corresponding relative RMSE of the predicted Nusselt number. In (a) and (b), colors distinguish the four models, solid lines show the mean relative RMSE across held-out test trajectories at each forecast lead step, and shaded bands indicate the corresponding 95% bootstrap confidence intervals. (c) Ground-truth states and forecast residuals for a representative held-out trajectory at lead steps 50 (top row) and 100 (bottom row). The first column shows the ground-truth temperature and velocity fields; the remaining columns show predictions minus ground truth for IFactFormer-RB, FNO, AFNO, and Swin Transformer, respectively. Color shading denotes temperature $T$ or temperature residuals $\Delta T=\hat{T}-T$ , and arrows denote velocity $\mathbf{u}$ or velocity-vector residuals $\Delta\mathbf{u}=\hat{\mathbf{u}}-\mathbf{u}$ , with separate color scales and reference arrows for the ground-truth and residual fields.

Figure 3: Comparison of surrogate sensitivity fields with the adjoint reference for the RB forecast metric . The forecast metric is defined as the mean temperature within the target region at forecast lead step 20. (a) RB states and corresponding sensitivity fields for one held-out test trajectory. The top row shows the state at the input time ( $t=10.0$ ) and at forecast lead steps 6, 13, and 19. Color shading represents temperature, and arrows represent velocity. The remaining rows show the corresponding sensitivity fields computed using the adjoint model, IFactFormer-RB, FNO, AFNO, and Swin Transformer. In these rows, color shading represents sensitivity to temperature, and arrows represent sensitivity to velocity. White boxes identify the target region in the state panels, and black boxes identify the same region in the sensitivity panels. (b) Relative $L_{2}$ error between each surrogate sensitivity field and the adjoint reference. (c) Averaged cosine similarity between each surrogate sensitivity field and the adjoint reference. In (b) and (c), boxplots show the distributions of the corresponding metrics across the held-out test trajectories. Crosses indicate sample means, and shaded bars indicate the 95% bootstrap confidence intervals of the means.

Figure 4: Sensitivity of the ENSO forecast metric to SST and wind-stress fields during El Niño events . (a–l) Evolution of composite CTEFNet gradient-based sensitivities over the 12 months preceding the target periods of El Niño events. The SST and wind-stress components of the full multivariate sensitivity field are shown, with only the grid points significant at the 5% level retained, based on a two-sided Student’s $t$ test for the SST sensitivity and Hotelling’s $T^{2}$ test for the zonal and meridional wind-stress sensitivities treated jointly. Shading indicates sensitivity to SST, and vectors represent sensitivities to zonal and meridional wind stress. The green box outlines the target Niño 3.4 region.

Sensitivity Analysis

Sensitivity analysis studies how changes in inputs, assumptions, or model parameters affect outputs, predictions, and decisions.

See all 39 papers

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations
Sep 28, 2026Oskar Bohn Lassen, Joao Paulo de Souza Boger, Simon Driscoll +4Spatiotemporal Forecasting Weather Forecasting
Department of Technology, Management, and Economics Technical University of Denmark · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Department of Mathematics and Statistics University of Exeter
Strategic Governance of AI Models in Earth Science
Oct 1, 2026Makoto Kelp, Amirhossein Arzani, Patricia Castellanos +6AI Safety Evaluation Climate Modeling
University of Utah · NASA Goddard Space Flight Center · University of Bristol +5
Physical Sensitivity Kernels Can Emerge in Data-Driven Forward Models: Evidence From Surface-Wave Dispersion
Apr 5, 2026Ziye Yu, Yuqi Cai, Xin LiuNeural Surrogate Modeling Seismology
Institute of Geophysics, China Earthquake Administration, Beijing 100081, China · Laboratory of Seismology and Physics of Earth’s Interior, School of Earth and Space Sciences, University of Science and Technology of China, Hefei 230026, China · Institute of Advanced Technology, University of Science and Technology of China, Hefei 230088, China

Sep 28, 2026J/K move · Enter open · S save

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

Oskar Bohn Lassen, Joao Paulo de Souza Boger, Simon Driscoll, Stephen I. Thomson+3

Department of Technology, Management, and Economics Technical University of Denmark · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Department of Mathematics and Statistics University of Exeter

Spatiotemporal Forecasting Weather Forecasting

Open paper AskPDF arXiv

Machine-learning models for physical systems are currently evaluated primarily through errors between predicted and reference states and, increasingly, through tests of physical consistency. These metrics assess whether predictions are accurate and satisfy selected physical requirements, but provide limited insight into whether learned trajectories reproduce the underlying dynamics. Domain experts examine such relationships through physical representations that expose relevant processes, interactions, and responses, but these analyses are often separated from typical machine-learning evaluation. We introduce a practical framework for evaluating dynamical fidelity through predictive structure in physical representation spaces. Experts define the representations, while reference trajectories determine which relationships are predictive and retained as evaluation tests. We demonstrate the approach in atmospheric forecasting using ERA5 representations of planetary-wave activity and Northern Annular Mode evolution, and evaluate Pangu-Weather, GraphCast, and FengWu. The models exhibit distinct departures from reference predictive structure that are not reflected by conventional forecast errors. The framework thereby turns domain-expert representations into systematic tests of learned physical dynamics without prescribing the relationships in advance.

Abstract

Sensitivity Analysis

Explore similar work

Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations

Strategic Governance of AI Models in Earth Science

Physical Sensitivity Kernels Can Emerge in Data-Driven Forward Models: Evidence From Surface-Wave Dispersion