Long-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models
Authors: Scott A. Martin, Noah Brenowitz, Dale Durran, Michael Pritchard
Organizations: NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA
Accurate long-range weather forecasting remains a major challenge for AI models, both because errors accumulate over autoregressive rollouts and because reanalysis training datasets provide limited sampling of the slow modes of variability underpinning predictability. Most AI and all numerical weather prediction models are autoregressive, repeatedly applying short-lead forecasts to reach subseasonal-to-seasonal (S2S) or seasonal lead times, often causing instability and miscalibration. Long-timestep probabilistic models that generate long-range forecasts in a single step offer an alternative, but training on the 40-year reanalysis record leads to overfitting, suggesting orders of magnitude more data are required. We introduce long-range distillation, whereby a long-timestep probabilistic student'' model is trained to forecast directly at long-range using a huge synthetic dataset generated by an autoregressive teacher'' model. Using the Deep Learning Earth System Model (DLESyM) as teacher, we generate over 10,000 years of simulated climate to train student models across a range of timescales. In perfect-model experiments, the distilled models outperform climatology and approach their teacher's skill while replacing hundreds of autoregressive steps with a single timestep. In the real world, both a diffusion model student fine-tuned on ERA5 and a linear inverse model student with mixed DLESyM-ERA5 training achieve S2S forecast skill competitive with the ECMWF IFS ensemble, with synthetic pre-training providing improvements beyond ERA5-only training. This demonstrates the benefit of synthetic pre-training for both deep learning and classical data-driven methods, and represents the first demonstration that real-world skill can be improved by training on AI-generated synthetic data in volumes far exceeding the observational record.
Figures & tables
Figure 1: Schematic of long-range distillation approach. First, an autoregressive teacher model is trained for short lead time prediction using reanalysis datasets like ERA5 — in our case we leverage DLESyM as a pre-trained autoregressive teacher capable of stable long-running simulations. The autoregressive teacher is used to generate a huge simulation with orders of magnitude higher temporal coverage than ERA5. Finally, a long timestep probabilistic model, the ”student”, is trained on the autoregressive teacher to do long-range prediction in a single model timestep.
Figure 2: DLESyM ensemble simulation training data. (a) Nino3.4 index calculated for each of the DLESyM ensemble members throughout the 90-year simulation with the corresponding initialization date denoted by a red cross. (b) Zoomed in view of first 20 years of 2 ensemble members initialized 16 days apart showing how after initial spin-up, they quickly diverge and generate independent training data samples despite being initialized from similar initial conditions. (c) Same as (a) but for the Northern Annular Mode (NAM). (d) Same as (b) but for NAM.
Figure 3: Error growth in DLESyM autoregressive ensemble members for varying initial condition perturbation strengths. Global average Z500 RMSE of DLESyM ensemble members with respect to a withheld ensemble member for varying initial condition perturbation sizes (colored lines) and the real-world global average Z500 RMSE of IFS for reference (black dashed line). The initial condition condition perturbation strengths in the legend are provided in normalized units since we perturb all fields using a multiple of each field’s global standard deviation. For Z500 , the standard deviation is 2605 m2/s2 .
Figure 4: Example distilled student model 2 m air temperature forecasts across a range of lead times initialized on 2017-01-01. Ground truth forecast targets from the withheld DLESyM evaluation simulation or (a) medium-range, (b) S2S, and (c) seasonal lead times. (d-f) Single forecast member predictions for each lead time. Both forecast targets and ground truth targets are visualized as anomalies from climatology (a-f). (g-i) Ensemble mean forecast error for a 32-member ensemble forecast for each lead time. Medium-range forecasts (a, d, g) are daily means at 7-day lead time, S2S forecasts (b, e, h) are weekly means at 4-week lead time, and seasonal forecasts (c, f, i) are 4-week means at 12-week lead time.
Lead Time
Inputs
Outputs
N
M
Medium-Range
Days 0, -1, -2, -3
Day 7
28
4
S2S
Days 0, -1, -2, -3
Average of Days 25-31
112
28
Seasonal
Days 0, -1, -2, -3
Average of Days 70-98
336
112
Table 1: Inputs and outputs for the different distilled student models. N is the lead time in number of 6-hour autoregressive teacher steps and M is the size of the target averaging window in 6-hour teacher steps (Equation 1 ). Inputs beyond Day 0 were added to compensate for the information lost by averaging the 6-hourly DLESyM teacher model data to daily resolution during data generation.
Figure 5: Scaling of distilled student model S2S forecast skill with synthetic training dataset size. (a) Elucidated diffusion model (EDM) loss Karras et al. (2022) throughout training for a 40 year training dataset (blue) and the full 11,000 year dataset (orange). Dashed lines indicate training loss and solid the validation loss. Learning curves have been smoothed for plotting purposes using bin averaging. (b) Distilled model week 4 2 m air temperature global mean CRPS with increasing number of training years (orange) and the corresponding minimum validation EDM loss achieved during training (blue). Dashed lines indicate the length of the ERA5 record and the CRPS skill of climatology for reference. Note all axes are logarithmic.
Figure 6: Separating effect of overfitting on small dataset and added sample diversity from large synthetic dataset. (a) EDM loss throughout training for 40 years sampled in a contiguous block to mimic reanalysis record (blue), 40 years sampled randomly from throughout the 11,000 years of simulation output (orange), and for the full 11,000 year dataset (green). Dashed lines indicate training loss and solid the validation loss. Learning curves have been smoothed for plotting purposes using bin averaging. (b) Minimum validation EDM loss achieved during training by training dataset size for contiguously (blue dots) and randomly (from 11,000 years, green squares) sampled training datasets
Figure 7: Calibration of medium-range distilled student model forecasts using classifier-free guidance. (a) Day 7 global Z500 ensemble spread for varying classifier-free guidance strengths (black solid) with reference ensemble spread for a 20 year climatology (red dashed). (b) Distilled model ensemble spread-skill ratio for varying classifier-free guidance strengths (black solid). Ensemble spread-skill plot uses unbiased estimator for ensemble spread. (c) Day 7 global mean Z500 CRPS for distilled student model with varying classifier-free guidance strengths (black solid), compared to a probabilistic (red dashed) and deterministic (blue dashed) 20 year climatology. Vertical grid lines align with integer guidance strengths, with an extra line at 0.5 between 0 and 1. Each point on the plot is averaged over 400 initialization dates (Section 3.1 ) with ensemble size of 32 (Table 5 ).
Figure 8: Spread-skill plot for day 7 Z500 forecasts comparing distilled student models to initial condition ensembles from the autoregressive teacher model. (a) Spread-skill obtained given an imperfect initial condition (Section 3.1 ) using the distilled student model (solid black) and initial condition ensembles from the autoregressive teacher model (solid blue). The 1-1 calibration line (grey dashed) and climatology RMSE (red dashed) are shown for reference. For the distilled student model, the classifier-free guidance weight used is indicated by the color of the marker whereas for the autoregressive teacher it indicates initial condition perturbation strength used to generate the ensemble. (b) Same as (a) but for the idealized case of perfect initial conditions. Each point on the plot is averaged over 400 initialization dates (Section 3.1 ) with ensemble size of 32 (Table 5 ).
Figure 9: Distilled student model long-range forecast skill in perfect-model experiment. (a) Global mean 2-m air temperature CRPS at 4-week lead time for the autoregressive teacher (blue), distilled student (orange), deterministic climatology (light grey), and probabilistic climatology (dark grey) with the associated percentage change with respect to probabilistic climatology shown in the accompanying bars with error bars showing 95% confidence interval by bootstrapping over evaluation dates. (b) Same as (a) but at 12-week lead time. (c) Global map of time mean difference between autoregressive teacher CRPS and that of probabilistic climatology at 4-week lead time. (d) Same as (c) but for 12-week lead time. (e) Same as (c) but for DLESyM10K and (f) same as (d) but for DLESyM10K. In (c-f) blue indicates an improvement with respect to climatology and stippling indicates a difference that is significantly different from zero at significance level 0.05 ( A ).
Figure 10: Quality of the LIM fit improves with increasing volume of synthetic data. (a) Validation MSE of 4-week 2 m temperature forecasts on withheld DLESyM data, as a function of the number of training years; the dashed line shows climatology. (b) Same as (a) but for the predictive negative log-likelihood, −logp(zτ∣z0) , which scores both the forecast mean and the forecast-error covariance (Appendix D.2 ). Both panels show results for estimating G^z,DLESyM with a fixed EOF truncation of K=500 .
Figure 11: Fine-tuning distilled student models on real-world ERA5 reanalysis data for S2S forecasting. Orange curves: training (dashed) and validation (solid) loss for the DLESyM pre-trained model during fine-tuning on ERA5. Blue curves: same as orange but for model trained from scratch on ERA5. Note, the learning curves were smoothed for plotting purposes.
Figure 12: Mixed DLESyM-ERA5 LIM parameter sweep. (a) Global mean week-4 T2M MSE on 2016-2018 cross-validation years for varying EOF truncations, K , and dataset mixing weights, α . (b) Same as (a) but for the Gaussian posterior negative log likelihood (defined in Appendix D.2 ). In both panels, blue indicates improvement with respect to climatology and the green square highlights the optimal parameter combination.
Land
Method
Global
Tropical
Extratropical
ECMWF
-0.7 (-19.3 ∗ )
2.5 ∗ (-46.4 ∗ )
-1.6 ∗ (-10.8 ∗ )
Long Timestep ERA5
-6.8 ∗ (-37.8 ∗ )
0.7 (-60.0 ∗ )
-9.2 ∗ (-30.9 ∗ )
DLESyM10K
-2.9 (-11.1 ∗ )
1.2 (-4.9 ∗ )
-4.2 (-13.0 ∗ )
Damped Persistence
-9.5 ∗
-10.2 ∗
-9.3 ∗
DLESyM LIM
-0.2
0.1
-0.3
Table 2: Regional-mean 2 m temperature skill, S , as percent improvement in fair CRPS over probabilistic climatology: S=100×(CRPSclim−CRPSfcst)/CRPSclim , with overbars denoting means over initialization dates. Positive values beat climatology. Tropics are ∣lat∣≤30∘ and extratropics ∣lat∣>30∘ . For bias-corrected methods, values in parentheses are without bias correction. An asterisk marks entries whose mean CRPS difference from climatology has a 95% confidence interval that excludes zero, using a paired t -test with lag-1 effective sample size Neff=N(1−r1)/(1+r1) on the time series of differences. For bias-corrected methods, the unbiased improvement is shown with the biased improvement in parentheses.
Figure 13: Comparison of the spatial patterns of real-world 4-week forecast skill between ECMWF and distilled student models for 2 m air temperature. (a) Difference between ECMWF and 20-year probabilistic climatology CRPS. (b) Same as (a) but for distilled student model. (c) Difference between distilled student model and ECMWF CRPS. (d-e) Same as (a) but for the linear inverse models (d) G^ERA5 and (e) G^mixed . (f) Difference between G^mixed and ECMWF CRPS. In all panels blue indicates an improvement in forecast skill and stippling indicates a statistically significant difference between the skill of the two methods under comparison ( A ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Model timestep
6 hours
Output frequency saved
Daily averages
Data volume
7 TB
Number of stable ensemble members
162
Initialization date range
2008-01-01 to 2016-12-31
Simulation duration
90 years
Appendix
Table 3: Details about the huge DLESyM ensemble simulation used to generate training data.
Parameter
DLESyM Training
ERA5 Fine-Tuning
ERA5 From Scratch
Distilled student architecture
Song UNet HPX64
Song UNet HPX64
Song UNet HPX64
Number of parameters
149M
149M
149M
Batch size
64
64
64
Learning rate
10−4
10−5
10−4
Condition dropout fraction
0.1
0.1
0.1
Optimizer
Adam
Adam
Adam
Appendix
Table 4: Training configurations for distilled student models.
Parameter
Value
σmin
0.002
σmax
200
Training σ distribution
Log uniform
Number of sampler steps
18
Schurn
0
Sampler scheme
EDM ODE
Appendix
Table 5: Diffusion sampler settings used for all distilled student forecasts presented in this study except where stated otherwise.
Region
Autoregressive Teacher
DLESyM10K
Clim. (Probabilistic)
Clim. (Deterministic)
S2S lead time
Global land only
1.15
1.18
1.16
1.70
Global ocean only
0.31
0.32
0.34
0.50
Tropical ocean
0.18
0.19
0.22
0.32
Tropical land
0.60
0.62
0.63
0.91
Extratropical ocean
0.45
0.46
0.48
0.70
Appendix
Table 6: Regional mean 2 m temperature CRPS [ K ] in the model world experiment. Scores are averaged over initialization dates, and using area-weighted averaging in space. Tropics are defined here as ∣latitude∣≤30∘ .
Figure 14: The ERA5 EOF basis. (a) Cumulative fraction of variance explained by the first K ERA5 EOFs, for ERA5 training data, withheld ERA5 data, and DLESyM data. At K=500 the EOFs explain 96% of the training variance and 93% of both the withheld ERA5 and the DLESyM variance. (b) Variance explained by each EOF. The dotted line shows the expected variance of a random unit vector, 1/d . (c) ERA5 EOF 500 and (d) a random unit vector.
Figure 15: Effect of the LIM ensemble construction on real-world week-4 2 m temperature CRPS. Each panel shows the CRPS difference from the probabilistic climatology (blue indicates improvement), for the DLESyM-only (left), ERA5-only (centre), and mixed (right) LIMs, all with the same forecast mean. (a–c) Ensemble formed by adding climatological anomalies to the LIM mean, as used in the main text. (d–f) Gaussian forecast distribution using the LIM error covariance (Eq. 18 ). The number at the top right of each panel is the global mean CRPS. Stippling indicates differences that are statistically significant (Appendix A). Scores are computed on the HEALPix grid.
Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.
Diffusion models generate skillful weather ensembles but require costly iterative sampling. We introduce a supervised energy-distance distillation method that compresses a multi-step diffusion teacher into a single-step student by aligning student forecasts with teacher samples and ground-truth observations. Experiments on global forecasting and typhoon-track prediction show that our student outperforms existing distillation methods and preserves skill for extreme events. It matches or surpasses the teacher across key metrics using only one neural function evaluation per autoregressive step.
Yiming Yang, Valentin Brekke, James Briant +1
Department of Statistical Science University College London · MPLS Doctoral Training Centre University of Oxford
Deep learning has made rapid advances in weather forecasting: autoregressive models trained on atmospheric reanalyses now rival dynamical models across nowcasting, medium-range, and subseasonal-to-seasonal lead times, producing well-calibrated ensemble forecasts at reduced cost. We review these advances and consider their extension to climate horizons, where the challenge shifts from initial-condition skill to producing reliable statistical responses under altered forcings. AI-powered climate prediction systems must produce credible forced responses to drivers (e.g., greenhouse gases, land-use change) typically outside the observed record. We propose two minimum requirements for AI in climate modeling: (i) external forcing agents must enter explicitly enough to support interventions in which they vary independently; and (ii) robustness must be stress-tested in out-of-distribution regimes, including extremes and counterfactual trajectories. Using leading AI autoregressive emulators and hybrid physics-AI models, we identify development and coupling challenges. Comparing the reported throughput of these models with that of GPU-ported dynamical models highlights how AI can reduce time-to-solution by advancing only the target variables at the required resolution and using longer time steps, rather than integrating a full high-frequency, multivariate state. Diverse AI downscaling strategies can partially substitute for explicit fine-scale resolution, paving the way toward inexpensive local hazard assessment across prediction horizons.
Tom Beucler, J. David Neelin, Hui Su +10
University of Lausanne, Lausanne, Switzerland. · University of California, Los Angeles, Los Angeles, CA, USA. · Hong Kong University of Science and Technology, Hong Kong SAR, China. +6
We evaluate the climate simulation capabilities of ArchesWeather and ArchesWeatherGen, two machine learning models originally trained for weather forecasting and evaluated up to a 10-day lead time. ArchesWeather is a deterministic model, while ArchesWeatherGen is a probabilistic flow-matching model leveraging ArchesWeather's forecasts, enabling ensemble-based uncertainty quantification. In this work, we adapt these models to act as forced atmospheric models by using additional conditioning on the monthly mean sea surface temperature (SST) and sea ice cover (SIC) as boundary conditions. In particular, we follow the AI Model Intercomparison Project (AIMIP) Phase 1 protocol, which, analogous to the Atmospheric Model Intercomparison Project (AMIP), proposes a standardized experimental setup to evaluate the climate skill of ML-based forced atmospheric models. We present a comprehensive evaluation of both models under these conditions, including comparison against numerical climate models, ablation studies that examine key design choices in the extension, and an analysis of forced versus unforced configurations. Despite being originally developed for weather forecasting, we demonstrate that forced configurations of ArchesWeather and ArchesWeatherGen produce stable long-term climate simulations, have a stable annual cycle, and capture the drift of many climate variables. The models faithfully reproduce ERA5's climatology, large-scale circulations and interannual variability, and they capture the tails of the distributions.
Renu Singh, Robert Brunstein, Antonia Jost +5
Google DeepMind, Paris, France · INRIA, Paris, France · Otto von Guericke University Magdeburg, Magdeburg, Germany +3