Fair scores reward ensemble forecast members that behave like samples from the same distribution as the verifying observations. They are therefore an attractive choice as loss functions to train data-driven ensemble forecasts or post-processing methods when large training ensembles are either unavailable or computationally prohibitive. The adjusted continuous ranked probability score (aCRPS) is a fair and ensemble-size-unbiased scoring rule when applied to ensembles with forecast members that are interpretable as conditionally independent draws from an underlying predictive distribution. However, distribution-aware post-processing methods that introduce structural dependency between members can violate this assumption, such that aCRPS no longer functions as a fair scoring rule. We demonstrate this effect using two approaches designed to minimize the expected aCRPS of a finite ensemble: (1) a linear member-by-member calibration, which couples members through a common dependency on the sample ensemble mean, and (2) a deep-learning method, which couples members via transformer self-attention across the ensemble dimension. In both cases, the results are sensitive to ensemble size and apparent gains in aCRPS can correspond to systematic unreliability characterized by over-dispersion. We introduce trajectory transformers as a proof-of-concept that ensemble-size independence can be achieved. This approach is an adaptation of the Post-processing Ensembles with Transformers (PoET) framework and applies self-attention over lead time while preserving the conditional independence required by aCRPS. When applied to weekly mean T2m forecasts from the ECMWF subseasonal forecasting system, this approach successfully reduces systematic model biases whilst also improving or maintaining forecast reliability regardless of the ensemble size used in training or real-time forecasts.
Figures & tables
Figure 1: (Top) Schematic diagram of the PoET hierarchical ensemble transformer architecture as configured during training ( Ben Bouallègue et al., 2024 ) . (Bottom) Schematic diagram of the hierarchical trajectory transformer described in this study, as configured during training. Figure generated using PlotNeuralNet ( Iqbal, 2018 ) .
Figure 2: Expected values of the unadjusted (left column) and adjusted (right column) CRPS sample estimators plotted against the total variance ratio (VR) of linearly-calibrated idealised Gaussian forecasts of different ensemble sizes as described in section 2 . The expected scores are calculated using the optimal value of b∗ (equation 10 ) and for a range of values of c , with optimal values of c∗ indicated as markers corresponding to the minima of the expected scores. The underlying raw forecasts are either reliable (top row) or unreliable (bottom row) and use model parameters consistent with table 1 . Vertical dotted lines indicate the expected variance ratio of a reliable forecast.
Figure 3: Optimal linear calibration parameters for idealized Gaussian forecasts following equations 10 and 11 for a 10-member perfectly reliable ( α=β ) forecast and different values of σs .
Parameter
Ensemble Transformer
Trajectory Transformer
Architecture
Attention dimension
Ensemble members ( N )
Lead times ( T )
Encoder channels
[16, 32, 64]
[16, 32, 64]
Transformers per level
1
1
Attention heads
8
8
Input channels
22
22
Table 2: Training and architecture hyperparameters for ensemble transformer and trajectory transformer models based on the PoET framework ( Ben Bouallègue et al., 2024 ) .
Category
Description
Forecast fields (18 channels)
2-metre temperature
Temperature at 850, 500, 200, and 50 hPa
Geopotential height at 500 hPa
Mean sea-level pressure
Total precipitation rate
10-metre zonal wind
Table 3: Input features for post-processing weekly mean T 2m forecasts. Forecast fields are standardized to zero mean and unit variance with precipitation log-epsilon transformed prior to scaling.
Figure 4: (a) Weighted global mean aCRPS of raw and post-processed T 2m forecasts in normalized units as a function of training epoch for the independent evaluation period (2021-2023). Each epoch represents a single pass through the entire training dataset (1959-2017). (b-c) As above, but for unbiased estimates of the ensemble spread and root mean square error (RMSE) of the ensemble mean calculated to be unbiased with ensemble size following Leutbecher and Palmer (2008) and Roberts and Leutbecher (2025) . All metrics are averaged over lead times with equal weight.
Figure 5: Impact of PoET hierarchical Ensemble Transformer post-processing method ( Ben Bouallègue et al., 2024 ) on the adjusted CRPS of weekly mean T 2m forecasts as a function of forecast lead time, where negative values are indicative of lower aCRPS in the post-processed reforecasts. The ensemble transformer post-processing method was trained on 9-members (see figure 4 ) for the period 1959-2017 and evaluated on either 9-member (left) or 100-member (right) forecasts for the period 2018-2023. Stippling indicates regions where 95 % confidence intervals for Δ aCRPS do not include zero. Confidence intervals are estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure.
Figure 6: As figure 5 , but for the hierarchical Trajectory Transformer approach described in this study.
Figure 7: (a) Impact of transformer-based post-processing methods on weighted global mean aCRPS calculated from weekly mean T 2m , where negative values are indicative of lower aCRPS in the post-processed reforecasts. (b) As (a), but for weekly mean forecast anomalies calculated following the ‘by-member–other-years’ method of Roberts and Leutbecher (2025) . Error bars correspond to 95 % confidence intervals estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure.
Figure 8: (a) Anomaly-based global spread-error ratios (SER) calculated following equation 34 of Roberts and Leutbecher (2025) combined with their ‘by-member–other-years’ method for climatology calculation such that estimates are unbiased with respect to both ensemble size and climatology sample size. Error bars correspond to 95 % confidence intervals estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure. Anomaly-based global activity ratios defined as the square root of a generalisation of equation 7 such that the numerator and denominator represent the weighted global mean variance of weekly mean anomalies in forecast members and observations, respectively. Confidence intervals were not estimated for this quantity.
Statistical post-processing has proven to be an effective tool in improving ensemble forecast of different weather variables. Case studies show that post-processing can remedy the typically underdispersive and potentially biased behaviour of the ensemble while optimizing a proper scoring rule expressing the forecast skill. The price of these positive effects is generally a deterioration in sharpness; the width of the central prediction intervals and the uncertainty of the predictions are increasing, especially for shorter lead times. This work aims to reduce the extent of the latter phenomenon for neural network-based parametric post-processing methods by extending the network's loss function with a penalty term. We demonstrate the effect of the proposed technique for 2m temperature ensemble forecasts of the European Centre for Medium-Range Weather Forecasts downloaded from the EUPPBench benchmark dataset and verified against synoptic observations. Here, the predictive distribution is Gaussian, and we use the continuous ranked probability score (CRPS) as loss function. The case studies confirm a substantial relative decrease (8.2%−12.5%) in the width of the nominal central prediction interval compared to the width of the predictive distribution computed without the penalty term, while there is no deterioration in the mean CRPS of probabilistic forecasts and in the RMSE of the predictive mean.
Ágnes Baran, Máté Mihalina
Faculty of Informatics, University of Debrecen, Debrecen, Hungary
Statistical post-processing improves ensemble weather forecasts, but generating calibrated predictions at locations without observations remains challenging. This study compares statistical and machine-learning-based methods for post-processing ECMWF 2-m temperature and 10-m wind speed forecasts at observed and unobserved stations in Germany. We consider EMOS-based approaches, distributional regression networks, Transformers, and graph neural networks under both limited and extended predictor settings. For temperature, we also investigate linear forecast combinations and propose an altitude-aware linear pool (ALP). The results show that post-processing improves upon the raw ensemble in most settings, but no single method performs best across all variables, station groups, and evaluation metrics. The proposed ALP provides a small but significant improvement over the standard linear pool at unobserved locations.
Mária Lakatos
Faculty of Informatics, University of Debrecen, Hungary
Recent probabilistic weather forecasters train stochastic predictors with the continuous ranked probability score (CRPS) to generate each ensemble member in a single forward pass. These models learn the predictive distribution from the forecast context alone, which becomes difficult at longer forecast horizons where uncertainty is high. To learn the predictive distribution more effectively, we introduce auxiliary conditional denoising tasks that predict the same future state from the context and its corrupted version, which provides partial future information that can reduce prediction ambiguity. Building on distributional diffusion models, we learn the conditional distributions of these tasks with a single stochastic predictor by minimizing a proper scoring rule across noise levels. At inference, the predictor can still generate each ensemble member in a single forward pass at the fully corrupted endpoint. Standard CRPS training is recovered as the endpoint-only special case of our formulation, so our framework extends existing CRPS-based forecasters with only additional conditioning inputs. Controlled experiments show that the auxiliary tasks improve one-step forecasting across architectures, with larger gains at longer forecast horizons. The gains extend to high-dimensional global weather forecasting under both training from scratch and fine-tuning, along with improved calibration and potential benefits for generalization under distribution shift.