Distinguishing Initial-Condition and Model Error Using Machine-Learning Weather Models
Authors: P. Trent Vonich, Gregory J. Hakim
Organizations: Department of Atmospheric Sciences, University of Washington, Seattle, WA, USA · Air Force Institute of Technology, Wright-Patterson AFB, OH, USA
Differentiable weather models allow atmospheric initial conditions to be directly optimized to shadow a known trajectory, which we take from a reanalysis dataset. Here, we extend this optimization to include the forecast model, GraphCast, using low-rank adaptation (LoRA). We compare model optimization with initial-condition (IC) optimization, and then joint optimization, where both are tuned simultaneously. When compared against a control forecast initialized with reanalysis data, LoRA and IC optimization individually produce similar improvements. We find similar results when IC and LoRA are jointly optimized across a spectrum of solutions where a tuning coefficient smoothly shifts the weight of the joint optimization from all-LoRA to all-IC; forecast improvement is not unique to the relative contributions of model and IC corrections. To better discriminate the source of error, we introduce a second model, Pangu-Weather, as a supervisor that gates IC updates during joint optimization. This gated optimization is the same as before, except that IC updates at each epoch are accepted only when they improve the forecasts of both models; otherwise, only the LoRA update is accepted. Results show that this cross-model constraint substantially improves agreement between the two models in their response to the same optimized ICs, narrowing the gap in mean forecast error reduction from 64% to 10%. Moreover, when used together, the LoRA and IC corrections from the gated optimization yield GraphCast forecast improvements similar to those from the ungated optimization. These results imply that model error that is encoded in the ungated ICs is shifted to the LoRA corrections via gating, yielding IC corrections that may provide better analysis estimates.
Figures & tables
Figure 1: Error reduction (y-axis) relative to the control forecast initialized with ERA5 as a function of the number of trainable LoRA parameters (x-axis) averaged over 36 initialization times. Each curve represents a single lead time. Trainable matrices progressively increase from 1 to all 64 available in the GraphCast processor at fixed rank r=1 , followed by increasing rank r for all 64 matrices. Error bars show one standard error and are within symbol size. Green symbols show the average improvement for IC optimization using GraphCast for the same cases.
Figure 2: Joint IC–LoRA optimization results as a function of the IC–LoRA weighting term, ρ , averaged over 36 initialization dates. The gray curve shows the total 10-day GraphCast Z500 RMSE reduction when both controls are applied. Orange curves show forecast outcome from only using the IC or LoRA component from the joint optimization. The green dashed curve is the sum of the IC-only and LoRA-only reductions. The purple curve shows the Pangu-Weather Z500 RMSE reduction obtained from the optimized IC alone. Shading indicates ±1 standard error of the 36-date mean.
Figure 3: Performance of the GraphCast joint solution’s IC component from Fig. 2 in Pangu-Weather as a function of forecast lead and ρ . Each cell shows the mean reduction in Z500 RMSE relative to each date’s Pangu-Weather control averaged over 36 cases. Red boxes indicate ρ -values that outperform ρ=20 at that lead under a paired two-sided t -test ( p<0.01 ). Red stars mark the best-performing ρ at each lead time.
Figure 4: Comparison of forecast error improvements for the gated and ungated optimizations for all 36 initialization dates (green denotes gated; red, ungated). (a) Average Z500 forecast error reduction by lead time. Solid curves show GraphCast error reduction from the gated solution with the IC ( δx ) and LoRA ( δW ) corrections applied independently or jointly ( δx,δW ). Dashed curves show the forecast improvement of gated and ungated δx for Pangu-Weather. Shading denotes ±1 standard error. (b) Per-date reduction in Pangu-Weather Z500 error from the gated δx alone, plotted against the reduction produced by the same δx in GraphCast. Markers represent the forecast error reduction of the summed day-1–10 RMSE relative to each model’s own corresponding control. Bold diamonds designate the 36-initialization means.
Figure 5: Mean GraphCast forecast loss by lead time evaluated every 6 h on a logarithmic vertical axis. The control uses unmodified ERA5 initial conditions and GraphCast weights. IC-only optimizes only the initial conditions; LoRA-only optimizes a single set of rank-2 matrices for each initialization; and Joint optimizes ICs and a single-set of rank-2 matrices simultaneously. Multi-LoRA fits a separate set of rank-2 matrices for each 24 h period for all initialization times. Legend percentages show reductions in loss relative to controls over the whole 10-day window. The shaded region beyond day 10 lies outside the optimization window, where multi-LoRA applies no adapters.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: LoRA depth versus rank presented in an alternative form to Fig. 1 . GraphCast objective loss reduction is relative to the control forecast as a function of lead time for LoRA-only optimization out to ten days (averaged over 36 dates). Solid curves represent the matrix depth varied at fixed rank 1, adapting the first 1, 4, 16, 32, or all 64 processor weight matrices. Dashed curves represent all 64 matrices at rank 4 and rank 32. The legend gives trainable parameter counts.
Figure A2: Gated and ungated optimizations evaluated in GraphCast and Pangu-Weather. The first panel (upper-left) shows the mean over the 36 dates; remaining panels show first-of-month 12 UTC optimization descents from January–November 2017. The y-axis metric is the summed day-1–10 Z500 RMSE verified against ERA5, expressed as percent change from each model’s date-matched control. The gated arm optimizes the initial conditions jointly with a LoRA weight correction; its GraphCast curve uses the initial conditions alone (LoRA removed), the same state Pangu receives. Stars and diamonds mark the lowest Pangu errors in the 5- and 10-day windows, respectively. Percentages above each panel show the best Pangu improvement relative to control for gated (green) and ungated (red) optimization. Ungated outperforms gated on half of dates when stopped early. Gating does not guarantee a better Pangu performance, but it does produce more comparable GraphCast and Pangu-Weather improvements on all 36 cases.
Figure A3: As in Fig. 4 , but measuring 850 hPa temperature.
Figure A4: As in Fig. 4 , but measuring 250 hPa wind.
Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.
Unresolved atmospheric processes can introduce history dependence beyond the short input windows used by many weather forecasting models. We introduce GraphCast--Mamba, which augments a frozen, pretrained 1∘ GraphCast backbone with a trainable Mamba correction branch combining graph message passing and Mamba layers. The branch retains a temporal memory state across successive six-hour forecast steps, learning corrections informed by temporal context beyond GraphCast's two-frame input. Training is confined to the correction branch, using ERA5 data from 2015--2021. After 2,500 optimizer updates, including warm-up, the selected model achieves substantial improvements relative to the uncorrected GraphCast backbone. It reduces the GraphCast loss (normalized weighted mean-squared error; MSE) by 14.42% when aggregated over the full ten-day forecast and by 21.66% at day 10 (240 h after forecast initialization). At this final forecast lead time, root-mean-square error (RMSE) decreases by 7.09--14.60% across all 11 evaluated variable groups, spanning temperature, wind, pressure, geopotential, specific humidity, and precipitation. These results demonstrate broad improvements in ten-day weather forecasts through a short correction-training schedule while retaining the pretrained backbone.
Ilya Vilkoviskiy, Liang-Hong Mo, Suozhi Huang +1
Department of Physics, Princeton University, Princeton, NJ 08544, USA · Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA · École Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland +2
Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs required for traditional physics-based methods. However, they are primarily (1) data-driven and (2) evaluated using pixel-wide error metrics (e.g., RMSE), so there are no guarantees that their forecasts are consistent with known physical laws. We introduce PhysMetrics.Weather, an evaluation framework that assesses the physical realism of MLWP models across three types of metrics: conservation, spectral, and dynamical. By quantifying physical realism, this tool guides the development of physics-informed architectures and helps evaluate whether MLWP models are reliable for operational use. Our framework is available on Github at https://github.com/Emmakast/PhysMetrics.Weather.
Emma Kasteleyn, Timo Maier, Axel Lauer +3
University of Amsterdam, Amsterdam, The Netherlands · Deutsches Zentrum für Luft- und Raumfahrt (DLR), Institut für Physik der Atmosphäre, Oberpfaffenhofen, Germany · University of Bremen, Institute of Environmental Physics (IUP), Bremen, Germany +1
ML foundation models are able to emulate atmospheric dynamics accurately and efficiently but operate as opaque ``black boxes''. We investigate the internal representations of the Aurora model using spatially pooled PCA and layer-wise relevance propagation (LRP). We find evidence that Aurora's latent space is primarily organized by seasonal cycles, whereas extreme storm events do not form a linearly separable cluster. LRP indicates that the model attends to features consistent with the 3D vertical structure of the Great Storm of 1987. Perturbation tests show masking relevant regions degrades forecasts 3.31× more than random masking. These findings suggest that Aurora learns meteorological coherence and vertical structure without explicit instruction.