Ensemble-size-dependence of deep-learning post-processing methods that minimize an (un)fair score: motivating examples and a proof-of-concept solution

Christopher David Roberts

physics.ao-phFeb 17, 2026

Ensemble-size-dependence of deep-learning post-processing methods that minimize an (un)fair score: motivating examples and a proof-of-concept solution

Authors: Christopher David Roberts

Organizations: ECMWF, Shinfield Park, Reading, United Kingdom

Abstract

Fair scores reward ensemble forecast members that behave like samples from the same distribution as the verifying observations. They are therefore an attractive choice as loss functions to train data-driven ensemble forecasts or post-processing methods when large training ensembles are either unavailable or computationally prohibitive. The adjusted continuous ranked probability score (aCRPS) is a fair and ensemble-size-unbiased scoring rule when applied to ensembles with forecast members that are interpretable as conditionally independent draws from an underlying predictive distribution. However, distribution-aware post-processing methods that introduce structural dependency between members can violate this assumption, such that aCRPS no longer functions as a fair scoring rule. We demonstrate this effect using two approaches designed to minimize the expected aCRPS of a finite ensemble: (1) a linear member-by-member calibration, which couples members through a common dependency on the sample ensemble mean, and (2) a deep-learning method, which couples members via transformer self-attention across the ensemble dimension. In both cases, the results are sensitive to ensemble size and apparent gains in aCRPS can correspond to systematic unreliability characterized by over-dispersion. We introduce trajectory transformers as a proof-of-concept that ensemble-size independence can be achieved. This approach is an adaptation of the Post-processing Ensembles with Transformers (PoET) framework and applies self-attention over lead time while preserving the conditional independence required by aCRPS. When applied to weekly mean $T_{2m}$ forecasts from the ECMWF subseasonal forecasting system, this approach successfully reduces systematic model biases whilst also improving or maintaining forecast reliability regardless of the ensemble size used in training or real-time forecasts.

Figures & tables

Figure 1: (Top) Schematic diagram of the PoET hierarchical ensemble transformer architecture as configured during training ( Ben Bouallègue et al., 2024 ) . (Bottom) Schematic diagram of the hierarchical trajectory transformer described in this study, as configured during training. Figure generated using PlotNeuralNet ( Iqbal, 2018 ) .

Figure 2: Expected values of the unadjusted (left column) and adjusted (right column) CRPS sample estimators plotted against the total variance ratio (VR) of linearly-calibrated idealised Gaussian forecasts of different ensemble sizes as described in section 2 . The expected scores are calculated using the optimal value of

b^{*}

(equation 10 ) and for a range of values of

c

, with optimal values of

c^{*}

indicated as markers corresponding to the minima of the expected scores. The underlying raw forecasts are either reliable (top row) or unreliable (bottom row) and use model parameters consistent with table 1 . Vertical dotted lines indicate the expected variance ratio of a reliable forecast.

Figure 3: Optimal linear calibration parameters for idealized Gaussian forecasts following equations 10 and 11 for a 10-member perfectly reliable (

\alpha=\beta

) forecast and different values of

\sigma_{s}

Architecture
Parameter	Ensemble Transformer	Trajectory Transformer
Attention dimension	Ensemble members ( $N$ )	Lead times ( $T$ )
Encoder channels	[16, 32, 64]	[16, 32, 64]
Transformers per level	1	1
Attention heads	8	8
Input channels	22	22

Table 2: Training and architecture hyperparameters for ensemble transformer and trajectory transformer models based on the PoET framework ( Ben Bouallègue et al., 2024 ) .

Category	Description
Forecast fields (18 channels)	2-metre temperature
	Temperature at 850, 500, 200, and 50 hPa
	Geopotential height at 500 hPa
	Mean sea-level pressure
	Total precipitation rate
	10-metre zonal wind

Table 3: Input features for post-processing weekly mean T 2m forecasts. Forecast fields are standardized to zero mean and unit variance with precipitation log-epsilon transformed prior to scaling.

Figure 4: (a) Weighted global mean aCRPS of raw and post-processed T 2m forecasts in normalized units as a function of training epoch for the independent evaluation period (2021-2023). Each epoch represents a single pass through the entire training dataset (1959-2017). (b-c) As above, but for unbiased estimates of the ensemble spread and root mean square error (RMSE) of the ensemble mean calculated to be unbiased with ensemble size following Leutbecher and Palmer (2008) and Roberts and Leutbecher (2025) . All metrics are averaged over lead times with equal weight.

Figure 5: Impact of PoET hierarchical Ensemble Transformer post-processing method ( Ben Bouallègue et al., 2024 ) on the adjusted CRPS of weekly mean T 2m forecasts as a function of forecast lead time, where negative values are indicative of lower aCRPS in the post-processed reforecasts. The ensemble transformer post-processing method was trained on 9-members (see figure 4 ) for the period 1959-2017 and evaluated on either 9-member (left) or 100-member (right) forecasts for the period 2018-2023. Stippling indicates regions where 95 % confidence intervals for $\Delta$ aCRPS do not include zero. Confidence intervals are estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure.

Figure 6: As figure 5 , but for the hierarchical Trajectory Transformer approach described in this study.

Figure 7: (a) Impact of transformer-based post-processing methods on weighted global mean aCRPS calculated from weekly mean T 2m , where negative values are indicative of lower aCRPS in the post-processed reforecasts. (b) As (a), but for weekly mean forecast anomalies calculated following the ‘by-member–other-years’ method of Roberts and Leutbecher (2025) . Error bars correspond to 95 % confidence intervals estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure.

Figure 8: (a) Anomaly-based global spread-error ratios (SER) calculated following equation 34 of Roberts and Leutbecher (2025) combined with their ‘by-member–other-years’ method for climatology calculation such that estimates are unbiased with respect to both ensemble size and climatology sample size. Error bars correspond to 95 % confidence intervals estimated as the 2.5 and 97.5 percentiles of an empirical distribution derived by block bootstrap resampling (with replacement) 500 times. Start dates within the same calendar month are treated as a single block during resampling to preserve temporal correlation structure. Anomaly-based global activity ratios defined as the square root of a generalisation of equation 7 such that the numerator and denominator represent the weighted global mean variance of weekly mean anomalies in forecast members and observations, respectively. Confidence intervals were not estimated for this quantity.

Probabilistic Forecasting

Probabilistic forecasting predicts distributions or events with quantified uncertainty to support reliable decisions under uncertainty.

See all 230 papers

Improving the sharpness in neural network-based parametric post-processing of ensemble forecasts
Jun 7, 2026Ágnes Baran, Máté MihalinaProbabilistic Forecasting Regression Prediction Intervals
Faculty of Informatics, University of Debrecen, Debrecen, Hungary
Statistical versus machine learning-based spatial interpolation of post-processed ensemble weather forecasts
Sep 7, 2026Mária LakatosEnsemble Learning Probabilistic Forecasting
Faculty of Informatics, University of Debrecen, Hungary
Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting
Sep 29, 2026Joonhyeong Park, Giung Nam, Hyungi Lee +3Probabilistic Forecasting Proper Scoring Rules
KAIST · Kookmin University · New York University

Jun 7, 2026J/K move · Enter open · S save

Improving the sharpness in neural network-based parametric post-processing of ensemble forecasts

Ágnes Baran, Máté Mihalina

Faculty of Informatics, University of Debrecen, Debrecen, Hungary

Probabilistic Forecasting Regression Prediction Intervals Weather Forecasting

Open paper AskPDF arXiv

Statistical post-processing has proven to be an effective tool in improving ensemble forecast of different weather variables. Case studies show that post-processing can remedy the typically underdispersive and potentially biased behaviour of the ensemble while optimizing a proper scoring rule expressing the forecast skill. The price of these positive effects is generally a deterioration in sharpness; the width of the central prediction intervals and the uncertainty of the predictions are increasing, especially for shorter lead times. This work aims to reduce the extent of the latter phenomenon for neural network-based parametric post-processing methods by extending the network's loss function with a penalty term. We demonstrate the effect of the proposed technique for 2m temperature ensemble forecasts of the European Centre for Medium-Range Weather Forecasts downloaded from the EUPPBench benchmark dataset and verified against synoptic observations. Here, the predictive distribution is Gaussian, and we use the continuous ranked probability score (CRPS) as loss function. The case studies confirm a substantial relative decrease (

8.2\%-12.5\%

) in the width of the nominal central prediction interval compared to the width of the predictive distribution computed without the penalty term, while there is no deterioration in the mean CRPS of probabilistic forecasts and in the RMSE of the predictive mean.

Abstract

Probabilistic Forecasting

Explore similar work

Improving the sharpness in neural network-based parametric post-processing of ensemble forecasts

Statistical versus machine learning-based spatial interpolation of post-processed ensemble weather forecasts

Proper Scoring Rule-based Diffusion for Probabilistic Weather Forecasting