Distribution-Free Uncertainty Quantification for Weather Forecasting: Conformal Prediction Under Spatiotemporal Dependence
Organizations: Stanley College of Engineering and Technology for Women, Hyderabad, India
Abstract
Conformal prediction gives finite-sample, distribution-free marginal coverage and has begun to be applied to gridded weather forecasts. Marginal coverage concerns the long-run fraction of covered points at a location, not how coverage behaves jointly across space and time on a given day. I benchmark eight interval-construction procedures (Gaussian, pooled and per-cell split conformal, per-season, adaptive conformal inference and rolling windows) for gridded 2 m temperature forecasts at eight lead times from 6 hours to 10 days, using ERA5 at 5.625 degree resolution, a deliberately modest convolutional forecaster and ECMWF's operational HRES, with calibration in 2017-2018 and testing in 2019-2021. Alongside coverage, width and interval score, I report dependence diagnostics: a spatial overdispersion index, quantiles of daily spatial coverage and miss persistence relative to a shuffled-days null. Per-cell methods reach near-nominal marginal coverage, yet the day-to-day variance of the fraction of cells covered is roughly 3 to 10 times what independent cells would give, and misses cluster in time; both effects grow with lead time for the CNN, also appear for HRES, and are shared by all methods. An oracle that divides each day's normalised errors by that day's mean removes 81 to 92 percent of the excess dispersion for the CNN and 93 to 98 percent for HRES, so days of large errors across much of the grid account for most of it. A day-scale predicted from seasonal terms and already-verified errors lowers the overdispersion index without widening intervals but improves interval score by at most 0.6 percent; on HRES from 12 hours to 3 days it has the lowest overdispersion index while ACI or season-conditional calibration has the best interval score, so no method is best on both. The study covers one variable, one coarse grid, a reanalysis target and two forecasters.
Figures & tables
| Lead | 6 h | 12 h | 1 d | 2 d | 3 d | 5 d | 7 d | 10 d |
|---|---|---|---|---|---|---|---|---|
| Forecaster (CNN) | 1.99 | 2.32 | 1.28 | 1.81 | 2.13 | 2.44 | 2.57 | 2.67 |
| HRES | 0.56 | 0.67 | 0.69 | 0.77 | 0.88 | 1.24 | 1.75 | 2.42 |
| Persistence | 2.56 | 3.58 | 1.73 | 2.34 | 2.62 | 2.90 | 3.06 | 3.20 |
| Climatology (1979–2015) | 2.48 | 2.49 | 2.45 | 2.45 | 2.45 | 2.45 | 2.45 | 2.45 |
| Method | Coverage | Interval score (K) | |
|---|---|---|---|
| Lead 24 h | |||
| Gaussian (RMSE) | 0.900 [0.897, 0.903] | 4.62 [4.52, 4.71] | 3.85 [3.20, 4.40] |
| Split, pooled | 0.936 [0.932, 0.940] | 6.53 [6.36, 6.68] | 5.74 [4.64, 6.56] |
| Split, per-cell | 0.897 [0.894, 0.900] | 4.62 [4.52, 4.71] | 3.84 [3.20, 4.38] |
| Per-cell + season | 0.898 [0.896, 0.900] | 4.42 [4.31, 4.51] | 3.31 [2.98, 3.62] |
| ACI per-cell (γ = 0.005) | 0.901 [0.898, 0.903] | 4.59 [4.48, 4.67] | 3.59 [3.06, 3.95] |
| Method | 6 h | 12 h | 1 d | 2 d | 3 d | 5 d | 7 d | 10 d |
|---|---|---|---|---|---|---|---|---|
| Gaussian (RMSE) | 3.48 | 3.62 | 3.86 | 4.62 | 5.23 | 6.10 | 6.34 | 6.53 |
| Split, pooled | 8.71 | 3.47 | 9.27 | 11.43 | 11.71 | 12.66 | 12.86 | 13.79 |
| Split, per-cell | 3.70 | 4.40 | 3.85 | 4.58 | 5.23 | 6.03 | 6.24 | 6.47 |
| Per-cell + season | 4.60 | 3.78 | 3.31 | 3.87 | 4.67 | 5.09 | 5.38 | 5.30 |
| ACI per-cell (γ = 0.005) | 3.20 | 3.87 | 3.59 | 4.72 | 5.52 | 6.65 | 7.16 | 7.59 |
| ACI per-cell (γ = 0.02) | 3.45 | 3.48 | 3.74 | 5.23 | 6.06 | 6.97 | 7.98 | 8.43 |
| Contrast | Lead | Δ interval score (K) | Δ coverage | |
|---|---|---|---|---|
| Per-cell + season | 24 h | [−0.25,−0.15] | [−1.04,0.08] | [−0.002,0.004] |
| 120 h | [−0.65,−0.36] | [−2.15,0.06] | [−0.004,0.003] | |
| 240 h | [−0.67,−0.39] | [−2.36,−0.13] | [−0.006,0.002] | |
| ACI (γ = 0.005) | 24 h | [−0.07,−0.01] | [−0.74,0.13] | [0.002,0.005] |
| 120 h | [−0.12,0.02] | [−0.15,1.31] | [0.005,0.009] | |
| 240 h | [−0.06,0.08] | [0.19,1.91] | [0.005,0.009] |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Year | 6 h | 12 h | 1 d | 2 d | 3 d | 5 d | 7 d | 10 d |
|---|---|---|---|---|---|---|---|---|
| 2017 | 0.390 | 0.269 | − 0.074 | − 0.099 | − 0.155 | − 0.086 | 0.021 | 0.151 |
| 2018 | 0.417 | 0.300 | − 0.062 | − 0.080 | − 0.128 | − 0.044 | 0.069 | 0.197 |
| 2019 | 0.413 | 0.295 | − 0.058 | − 0.080 | − 0.139 | − 0.065 | 0.050 | 0.193 |
| 2020 | 0.401 | 0.303 | − 0.064 | − 0.083 | − 0.138 | − 0.064 | 0.047 | 0.179 |
| 2021 | 0.403 | 0.282 | − 0.087 | − 0.133 | − 0.196 | − 0.146 | − 0.058 | 0.047 |
| δ | 6 h | 12 h | 1 d | 2 d | 3 d | 5 d | 7 d | 10 d |
|---|---|---|---|---|---|---|---|---|
| 0.02 | 0.097 | 0.147 | 0.134 | 0.179 | 0.207 | 0.228 | 0.250 | 0.273 |
| 0.05 | 0.001 | 0.001 | 0.001 | 0.003 | 0.008 | 0.020 | 0.019 | 0.008 |
| 0.10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Method | Coverage | Interval score (K) | π − π 0 | 5th pct | |
|---|---|---|---|---|---|
| Gaussian (RMSE) | 0.801 [0.796, 0.807] | 6.99 [6.82, 7.14] | 6.53 | 0.263 | 0.759 |
| Split, pooled | 0.867 [0.861, 0.873] | 9.29 [9.04, 9.53] | 6.69 | 0.211 | 0.834 |
| Split, per-cell | 0.792 [0.786, 0.798] | 6.98 [6.80, 7.15] | 6.82 | 0.262 | 0.749 |
| Per-cell + season | 0.790 [0.787, 0.794] | 6.66 [6.48, 6.83] | 5.47 | 0.235 | 0.751 |
| ACI (γ = 0.005) | 0.800 [0.794, 0.806] | 6.95 [6.77, 7.12] | 7.12 | 0.265 | 0.758 |
| ACI (γ = 0.02) | 0.800 [0.795, 0.806] | 6.79 [6.60, 6.96] | 6.70 | 0.253 | 0.758 |
| Method | Coverage | Interval score (K) | π − π 0 | 5th pct | |
|---|---|---|---|---|---|
| Gaussian (RMSE) | 0.941 [0.938, 0.943] | 9.78 [9.52, 10.04] | 5.82 | 0.276 | 0.915 |
| Split, pooled | 0.971 [0.968, 0.974] | 16.28 [15.84, 16.68] | 7.27 | 0.310 | 0.950 |
| Split, per-cell | 0.946 [0.943, 0.948] | 9.76 [9.54, 9.98] | 5.50 | 0.265 | 0.921 |
| Per-cell + season | 0.943 [0.941, 0.945] | 9.09 [8.84, 9.31] | 4.74 | 0.253 | 0.921 |
| ACI (γ = 0.005) | 0.950 [0.948, 0.953] | 9.72 [9.49, 9.93] | 5.94 | 0.267 | 0.926 |
| ACI (γ = 0.02) | 0.950 [0.948, 0.953] | 9.70 [9.42, 9.97] | 6.36 | 0.273 | 0.926 |
| Lead | Base VIF [95% CI] | Oracle VIF [95% CI] | Oracle floor | Excess removed | Split-half drop [95% CI] |
|---|---|---|---|---|---|
| 6 h | 3.69 [3.23, 4.17] | 1.14 [1.02, 1.23] | 0.55 | 81% | 0.22 [−0.02, 0.41] |
| 12 h | 4.39 [3.65, 4.97] | 1.03 [0.90, 1.16] | 0.61 | 89% | 0.76 [0.45, 1.06] |
| 1 d | 3.84 [3.19, 4.33] | 0.77 [0.70, 0.84] | 0.49 | 92% | 0.68 [0.32, 0.94] |
| 2 d | 4.57 [3.90, 5.14] | 0.86 [0.78, 0.95] | 0.49 | 91% | 0.98 [0.67, 1.25] |
| 3 d | 5.23 [4.50, 5.86] | 0.94 [0.83, 1.03] | 0.48 | 90% | 1.26 [0.94, 1.54] |
| 5 d | 6.02 [4.85, 7.38] | 1.23 [1.08, 1.40] | 0.52 | 87% | 1.48 [0.92, 2.09] |
| Lead | CNN R² | CNN VIF, base to primary | CNN IS ratio | HRES R² | HRES VIF, base to primary | HRES IS ratio |
|---|---|---|---|---|---|---|
| 6 h | 0.31 | 3.69 to 3.01 | 0.999 | 0.67 | 9.01 to 3.37 | 0.996 |
| 12 h | 0.45 | 4.39 to 2.46 | 0.998 | 0.74 | 8.05 to 2.71 | 0.998 |
| 1 d | 0.51 | 3.84 to 2.46 | 0.997 | 0.62 | 3.89 to 1.95 | 0.997 |
| 2 d | 0.46 | 4.57 to 3.19 | 0.996 | 0.38 | 3.25 to 2.16 | 0.998 |
| 3 d | 0.40 | 5.23 to 3.71 | 0.996 | 0.17 | 3.08 to 2.40 | 0.999 |
| 5 d | 0.37 | 6.02 to 4.65 | 0.995 | 0.06 | 4.16 to 3.64 | 0.999 |
| Lead | Base VIF [95% CI] | Oracle VIF [95% CI] | Oracle floor | Excess removed | Split-half drop [95% CI] |
|---|---|---|---|---|---|
| 6 h | 9.01 [2.70, 14.07] | 0.63 [0.56, 0.68] | 0.50 | 98% | 3.26 [0.11, 5.77] |
| 12 h | 8.05 [3.64, 11.67] | 0.74 [0.68, 0.81] | 0.54 | 97% | 2.82 [0.58, 4.66] |
| 1 d | 3.89 [3.00, 4.56] | 0.65 [0.60, 0.71] | 0.52 | 96% | 0.68 [0.25, 1.05] |
| 2 d | 3.25 [2.81, 3.56] | 0.64 [0.58, 0.69] | 0.50 | 95% | 0.31 [0.08, 0.49] |
| 3 d | 3.08 [2.72, 3.34] | 0.68 [0.61, 0.75] | 0.50 | 93% | 0.17 [−0.07, 0.34] |
| 5 d | 4.16 [3.67, 4.56] | 0.64 [0.57, 0.70] | 0.51 | 96% | 0.81 [0.58, 1.07] |
| Lead | Split (per-cell) | Mondrian season | ACI (0.02) | Rolling 90 | Rolling 365 | Day-scale model |
|---|---|---|---|---|---|---|
| 6 h | 9.02 | 9.23 | 3.91 | 4.54 | 6.87 | 3.37 |
| 12 h | 8.05 | 8.49 | 4.91 | 5.64 | 7.11 | 2.71 |
| 1 d | 3.89 | 3.57 | 3.38 | 3.41 | 3.75 | 1.95 |
| 2 d | 3.25 | 2.86 | 3.09 | 3.24 | 3.15 | 2.16 |
| 3 d | 3.08 | 2.86 | 3.10 | 3.10 | 3.02 | 2.40 |
| 5 d | 4.16 | 3.81 | 3.92 | 4.02 | 4.01 | 3.64 |
| Lead | Split (per-cell) | Mondrian season | ACI (0.02) | Rolling 90 | Rolling 365 | Day-scale model |
|---|---|---|---|---|---|---|
| 6 h | 1.866 | 1.805 | 1.733 | 1.774 | 1.814 | 1.859 |
| 12 h | 2.032 | 1.955 | 1.891 | 1.948 | 1.989 | 2.028 |
| 1 d | 2.142 | 2.047 | 1.993 | 2.045 | 2.101 | 2.136 |
| 2 d | 2.506 | 2.401 | 2.389 | 2.428 | 2.474 | 2.500 |
| 3 d | 2.959 | 2.840 | 2.864 | 2.896 | 2.935 | 2.955 |
| 5 d | 4.329 | 4.162 | 4.270 | 4.290 | 4.315 | 4.325 |
| Lead | 2019 | 2020 | 2021 | Base VIF | VIF after removing year means |
|---|---|---|---|---|---|
| 6 h | 0.886 | 0.879 | 0.864 | 9.01 | 7.63 |
| 12 h | 0.890 | 0.892 | 0.881 | 8.05 | 7.67 |
| 1 d | 0.890 | 0.894 | 0.885 | 3.89 | 3.65 |
| 2 d | 0.895 | 0.900 | 0.894 | 3.25 | 3.13 |
| 3 d | 0.899 | 0.905 | 0.899 | 3.08 | 2.91 |
| 5 d | 0.903 | 0.909 | 0.903 | 4.16 | 3.99 |
| Lead | Split (per-cell) | Mondrian season | ACI (0.005) | ACI (0.02) | Rolling 90 | Rolling 365 |
|---|---|---|---|---|---|---|
| 6 h | +5.64 [+0.61, +9.77]* | +5.85 [+0.72, +8.85]* | +2.66 [+0.38, +5.59]* | +0.54 [−0.51, +2.38] | +1.16 [−0.12, +3.68] | +3.49 [+0.34, +6.82]* |
| 12 h | +5.34 [+1.33, +8.48]* | +5.78 [+1.81, +8.70]* | +4.55 [+1.05, +7.95]* | +2.20 [+0.56, +4.80]* | +2.93 [+0.71, +6.10]* | +4.40 [+0.85, +7.53]* |
| 1 d | +1.94 [+1.14, +2.46]* | +1.62 [+0.70, +2.39]* | +2.21 [+1.17, +2.98]* | +1.43 [+0.55, +2.26]* | +1.46 [+0.58, +2.34]* | +1.80 [+1.06, +2.34]* |
| 2 d | +1.09 [+0.69, +1.35]* | +0.70 [+0.29, +1.10]* | +1.27 [+0.66, +1.64]* | +0.92 [+0.33, +1.44]* | +1.08 [+0.43, +1.68]* | +0.99 [+0.60, +1.22]* |
| 3 d | +0.68 [+0.37, +0.91]* | +0.46 [+0.17, +0.83]* | +0.80 [+0.31, +1.11]* | +0.69 [+0.25, +1.06]* | +0.69 [+0.24, +1.10]* | +0.61 [+0.27, +0.91]* |
| 5 d | +0.51 [+0.16, +0.90]* | +0.16 [−0.11, +0.60] | +0.49 [+0.13, +0.80]* | +0.27 [−0.09, +0.62] | +0.37 [+0.02, +0.74]* | +0.36 [+0.00, +0.76]* |
| Lead | Split (per-cell) | Mondrian season | ACI (0.005) | ACI (0.02) | Rolling 90 | Rolling 365 |
|---|---|---|---|---|---|---|
| 6 h | +0.007 [+0.002, +0.011]* | −0.054 [−0.072, −0.037]* | −0.071 [−0.083, −0.060]* | −0.126 [−0.144, −0.112]* | −0.086 [−0.106, −0.066]* | −0.045 [−0.052, −0.039]* |
| 12 h | +0.004 [+0.001, +0.006]* | −0.073 [−0.091, −0.055]* | −0.063 [−0.075, −0.053]* | −0.137 [−0.154, −0.121]* | −0.081 [−0.102, −0.059]* | −0.039 [−0.047, −0.031]* |
| 1 d | +0.006 [+0.004, +0.007]* | −0.089 [−0.109, −0.068]* | −0.064 [−0.080, −0.049]* | −0.143 [−0.166, −0.122]* | −0.091 [−0.122, −0.061]* | −0.035 [−0.045, −0.027]* |
| 2 d | +0.005 [+0.004, +0.007]* | −0.099 [−0.120, −0.076]* | −0.049 [−0.065, −0.035]* | −0.112 [−0.135, −0.089]* | −0.072 [−0.106, −0.038]* | −0.026 [−0.034, −0.017]* |
| 3 d | +0.004 [+0.002, +0.007]* | −0.115 [−0.140, −0.084]* | −0.041 [−0.058, −0.024]* | −0.090 [−0.117, −0.063]* | −0.058 [−0.099, −0.018]* | −0.019 [−0.028, −0.010]* |
| 5 d | +0.004 [+0.000, +0.009]* | −0.163 [−0.203, −0.114]* | −0.027 [−0.051, −0.005]* | −0.055 [−0.093, −0.015]* | −0.035 [−0.099, +0.027] | −0.010 [−0.017, −0.003]* |
Conformal Prediction
Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.
See all 397 papers