cs.LGDec 28, 2025
SaveLong-Range Distillation: Distilling 10,000 Years of Simulated Climate into Long Timestep AI Weather Models
Organizations: NVIDIA Research, Santa Clara, CA, USA · School of Oceanography, University of Washington, Seattle, WA, USA · Department of Atmospheric & Climate Science, University of Washington, Seattle, WA, USA
Abstract
Accurate long-range weather forecasting remains a major challenge for AI models, both because errors accumulate over autoregressive rollouts and because reanalysis training datasets provide limited sampling of the slow modes of variability underpinning predictability. Most AI and all numerical weather prediction models are autoregressive, repeatedly applying short-lead forecasts to reach subseasonal-to-seasonal (S2S) or seasonal lead times, often causing instability and miscalibration. Long-timestep probabilistic models that generate long-range forecasts in a single step offer an alternative, but training on the 40-year reanalysis record leads to overfitting, suggesting orders of magnitude more data are required. We introduce long-range distillation, whereby a long-timestep probabilistic student'' model is trained to forecast directly at long-range using a huge synthetic dataset generated by an autoregressive teacher'' model. Using the Deep Learning Earth System Model (DLESyM) as teacher, we generate over 10,000 years of simulated climate to train student models across a range of timescales. In perfect-model experiments, the distilled models outperform climatology and approach their teacher's skill while replacing hundreds of autoregressive steps with a single timestep. In the real world, both a diffusion model student fine-tuned on ERA5 and a linear inverse model student with mixed DLESyM-ERA5 training achieve S2S forecast skill competitive with the ECMWF IFS ensemble, with synthetic pre-training providing improvements beyond ERA5-only training. This demonstrates the benefit of synthetic pre-training for both deep learning and classical data-driven methods, and represents the first demonstration that real-world skill can be improved by training on AI-generated synthetic data in volumes far exceeding the observational record.
Figures & tables
Figure 1: Schematic of long-range distillation approach. First, an autoregressive teacher model is trained for short lead time prediction using reanalysis datasets like ERA5 — in our case we leverage DLESyM as a pre-trained autoregressive teacher capable of stable long-running simulations. The autoregressive teacher is used to generate a huge simulation with orders of magnitude higher temporal coverage than ERA5. Finally, a long timestep probabilistic model, the ”student”, is trained on the autoregressive teacher to do long-range prediction in a single model timestep.
Figure 2: DLESyM ensemble simulation training data. (a) Nino3.4 index calculated for each of the DLESyM ensemble members throughout the 90-year simulation with the corresponding initialization date denoted by a red cross. (b) Zoomed in view of first 20 years of 2 ensemble members initialized 16 days apart showing how after initial spin-up, they quickly diverge and generate independent training data samples despite being initialized from similar initial conditions. (c) Same as (a) but for the Northern Annular Mode (NAM). (d) Same as (b) but for NAM.
Figure 3: Error growth in DLESyM autoregressive ensemble members for varying initial condition perturbation strengths. Global average RMSE of DLESyM ensemble members with respect to a withheld ensemble member for varying initial condition perturbation sizes (colored lines) and the real-world global average RMSE of IFS for reference (black dashed line). The initial condition condition perturbation strengths in the legend are provided in normalized units since we perturb all fields using a multiple of each field’s global standard deviation. For , the standard deviation is 2605 .
Figure 4: Example distilled student model 2 m air temperature forecasts across a range of lead times initialized on 2017-01-01. Ground truth forecast targets from the withheld DLESyM evaluation simulation or (a) medium-range, (b) S2S, and (c) seasonal lead times. (d-f) Single forecast member predictions for each lead time. Both forecast targets and ground truth targets are visualized as anomalies from climatology (a-f). (g-i) Ensemble mean forecast error for a 32-member ensemble forecast for each lead time. Medium-range forecasts (a, d, g) are daily means at 7-day lead time, S2S forecasts (b, e, h) are weekly means at 4-week lead time, and seasonal forecasts (c, f, i) are 4-week means at 12-week lead time.
| Lead Time | Inputs | Outputs | N | M |
|---|---|---|---|---|
| Medium-Range | Days 0, -1, -2, -3 | Day 7 | 28 | 4 |
| S2S | Days 0, -1, -2, -3 | Average of Days 25-31 | 112 | 28 |
| Seasonal | Days 0, -1, -2, -3 | Average of Days 70-98 | 336 | 112 |
Table 1: Inputs and outputs for the different distilled student models. is the lead time in number of 6-hour autoregressive teacher steps and is the size of the target averaging window in 6-hour teacher steps (Equation 1 ). Inputs beyond Day 0 were added to compensate for the information lost by averaging the 6-hourly DLESyM teacher model data to daily resolution during data generation.
Figure 5: Scaling of distilled student model S2S forecast skill with synthetic training dataset size. (a) Elucidated diffusion model (EDM) loss Karras et al. (2022) throughout training for a 40 year training dataset (blue) and the full 11,000 year dataset (orange). Dashed lines indicate training loss and solid the validation loss. Learning curves have been smoothed for plotting purposes using bin averaging. (b) Distilled model week 4 2 m air temperature global mean CRPS with increasing number of training years (orange) and the corresponding minimum validation EDM loss achieved during training (blue). Dashed lines indicate the length of the ERA5 record and the CRPS skill of climatology for reference. Note all axes are logarithmic.
Figure 6: Separating effect of overfitting on small dataset and added sample diversity from large synthetic dataset. (a) EDM loss throughout training for 40 years sampled in a contiguous block to mimic reanalysis record (blue), 40 years sampled randomly from throughout the 11,000 years of simulation output (orange), and for the full 11,000 year dataset (green). Dashed lines indicate training loss and solid the validation loss. Learning curves have been smoothed for plotting purposes using bin averaging. (b) Minimum validation EDM loss achieved during training by training dataset size for contiguously (blue dots) and randomly (from 11,000 years, green squares) sampled training datasets
Figure 7: Calibration of medium-range distilled student model forecasts using classifier-free guidance. (a) Day 7 global ensemble spread for varying classifier-free guidance strengths (black solid) with reference ensemble spread for a 20 year climatology (red dashed). (b) Distilled model ensemble spread-skill ratio for varying classifier-free guidance strengths (black solid). Ensemble spread-skill plot uses unbiased estimator for ensemble spread. (c) Day 7 global mean CRPS for distilled student model with varying classifier-free guidance strengths (black solid), compared to a probabilistic (red dashed) and deterministic (blue dashed) 20 year climatology. Vertical grid lines align with integer guidance strengths, with an extra line at 0.5 between 0 and 1. Each point on the plot is averaged over 400 initialization dates (Section 3.1 ) with ensemble size of 32 (Table 5 ).
Figure 8: Spread-skill plot for day 7 forecasts comparing distilled student models to initial condition ensembles from the autoregressive teacher model. (a) Spread-skill obtained given an imperfect initial condition (Section 3.1 ) using the distilled student model (solid black) and initial condition ensembles from the autoregressive teacher model (solid blue). The 1-1 calibration line (grey dashed) and climatology RMSE (red dashed) are shown for reference. For the distilled student model, the classifier-free guidance weight used is indicated by the color of the marker whereas for the autoregressive teacher it indicates initial condition perturbation strength used to generate the ensemble. (b) Same as (a) but for the idealized case of perfect initial conditions. Each point on the plot is averaged over 400 initialization dates (Section 3.1 ) with ensemble size of 32 (Table 5 ).
Figure 9: Distilled student model long-range forecast skill in perfect-model experiment. (a) Global mean 2-m air temperature CRPS at 4-week lead time for the autoregressive teacher (blue), distilled student (orange), deterministic climatology (light grey), and probabilistic climatology (dark grey) with the associated percentage change with respect to probabilistic climatology shown in the accompanying bars with error bars showing 95% confidence interval by bootstrapping over evaluation dates. (b) Same as (a) but at 12-week lead time. (c) Global map of time mean difference between autoregressive teacher CRPS and that of probabilistic climatology at 4-week lead time. (d) Same as (c) but for 12-week lead time. (e) Same as (c) but for DLESyM10K and (f) same as (d) but for DLESyM10K. In (c-f) blue indicates an improvement with respect to climatology and stippling indicates a difference that is significantly different from zero at significance level 0.05 ( A ).
Figure 10: Quality of the LIM fit improves with increasing volume of synthetic data. (a) Validation MSE of 4-week 2 m temperature forecasts on withheld DLESyM data, as a function of the number of training years; the dashed line shows climatology. (b) Same as (a) but for the predictive negative log-likelihood, , which scores both the forecast mean and the forecast-error covariance (Appendix D.2 ). Both panels show results for estimating with a fixed EOF truncation of .
Figure 11: Fine-tuning distilled student models on real-world ERA5 reanalysis data for S2S forecasting. Orange curves: training (dashed) and validation (solid) loss for the DLESyM pre-trained model during fine-tuning on ERA5. Blue curves: same as orange but for model trained from scratch on ERA5. Note, the learning curves were smoothed for plotting purposes.
Figure 12: Mixed DLESyM-ERA5 LIM parameter sweep. (a) Global mean week-4 T2M MSE on 2016-2018 cross-validation years for varying EOF truncations, , and dataset mixing weights, . (b) Same as (a) but for the Gaussian posterior negative log likelihood (defined in Appendix D.2 ). In both panels, blue indicates improvement with respect to climatology and the green square highlights the optimal parameter combination.
| Land | |||
|---|---|---|---|
| Method | Global | Tropical | Extratropical |
| ECMWF | -0.7 (-19.3 ∗ ) | 2.5 ∗ (-46.4 ∗ ) | -1.6 ∗ (-10.8 ∗ ) |
| Long Timestep ERA5 | -6.8 ∗ (-37.8 ∗ ) | 0.7 (-60.0 ∗ ) | -9.2 ∗ (-30.9 ∗ ) |
| DLESyM10K | -2.9 (-11.1 ∗ ) | 1.2 (-4.9 ∗ ) | -4.2 (-13.0 ∗ ) |
| Damped Persistence | -9.5 ∗ | -10.2 ∗ | -9.3 ∗ |
| DLESyM LIM | -0.2 | 0.1 | -0.3 |
Table 2: Regional-mean 2 m temperature skill, , as percent improvement in fair CRPS over probabilistic climatology: , with overbars denoting means over initialization dates. Positive values beat climatology. Tropics are and extratropics . For bias-corrected methods, values in parentheses are without bias correction. An asterisk marks entries whose mean CRPS difference from climatology has a 95% confidence interval that excludes zero, using a paired -test with lag-1 effective sample size on the time series of differences. For bias-corrected methods, the unbiased improvement is shown with the biased improvement in parentheses.
Figure 13: Comparison of the spatial patterns of real-world 4-week forecast skill between ECMWF and distilled student models for 2 m air temperature. (a) Difference between ECMWF and 20-year probabilistic climatology CRPS. (b) Same as (a) but for distilled student model. (c) Difference between distilled student model and ECMWF CRPS. (d-e) Same as (a) but for the linear inverse models (d) and (e) . (f) Difference between and ECMWF CRPS. In all panels blue indicates an improvement in forecast skill and stippling indicates a statistically significant difference between the skill of the two methods under comparison ( A ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Model timestep | 6 hours |
| Output frequency saved | Daily averages |
| Data volume | 7 TB |
| Number of stable ensemble members | 162 |
| Initialization date range | 2008-01-01 to 2016-12-31 |
| Simulation duration | 90 years |
Table 3: Details about the huge DLESyM ensemble simulation used to generate training data.
| Parameter | DLESyM Training | ERA5 Fine-Tuning | ERA5 From Scratch |
|---|---|---|---|
| Distilled student architecture | Song UNet HPX64 | Song UNet HPX64 | Song UNet HPX64 |
| Number of parameters | 149M | 149M | 149M |
| Batch size | 64 | 64 | 64 |
| Learning rate | |||
| Condition dropout fraction | 0.1 | 0.1 | 0.1 |
| Optimizer | Adam | Adam | Adam |
Table 4: Training configurations for distilled student models.
| Parameter | Value |
|---|---|
| 0.002 | |
| 200 | |
| Training distribution | Log uniform |
| Number of sampler steps | 18 |
| 0 | |
| Sampler scheme | EDM ODE |
Table 5: Diffusion sampler settings used for all distilled student forecasts presented in this study except where stated otherwise.
| Region | Autoregressive Teacher | DLESyM10K | Clim. (Probabilistic) | Clim. (Deterministic) |
|---|---|---|---|---|
| S2S lead time | ||||
| Global land only | 1.15 | 1.18 | 1.16 | 1.70 |
| Global ocean only | 0.31 | 0.32 | 0.34 | 0.50 |
| Tropical ocean | 0.18 | 0.19 | 0.22 | 0.32 |
| Tropical land | 0.60 | 0.62 | 0.63 | 0.91 |
| Extratropical ocean | 0.45 | 0.46 | 0.48 | 0.70 |
Table 6: Regional mean 2 m temperature CRPS [ ] in the model world experiment. Scores are averaged over initialization dates, and using area-weighted averaging in space. Tropics are defined here as .
Figure 14: The ERA5 EOF basis. (a) Cumulative fraction of variance explained by the first ERA5 EOFs, for ERA5 training data, withheld ERA5 data, and DLESyM data. At the EOFs explain 96% of the training variance and 93% of both the withheld ERA5 and the DLESyM variance. (b) Variance explained by each EOF. The dotted line shows the expected variance of a random unit vector, . (c) ERA5 EOF 500 and (d) a random unit vector.
Figure 15: Effect of the LIM ensemble construction on real-world week-4 2 m temperature CRPS. Each panel shows the CRPS difference from the probabilistic climatology (blue indicates improvement), for the DLESyM-only (left), ERA5-only (centre), and mixed (right) LIMs, all with the same forecast mean. (a–c) Ensemble formed by adding climatological anomalies to the LIM mean, as used in the main text. (d–f) Gaussian forecast distribution using the LIM error covariance (Eq. 18 ). The number at the top right of each panel is the global mean CRPS. Stippling indicates differences that are statistically significant (Appendix A). Scores are computed on the HEALPix grid.
More in this topic
Weather Forecasting
Weather forecasting predicts future atmospheric conditions, from local storms to global patterns, using physical models, observations, and machine learning.
See all 176 papers