Reconciling European Electricity Data with Diffusion Priors Learned from Observations Alone
Organizations: Montefiore Institute, University of Liège, Belgium · CMAP, École Polytechnique, France
Abstract
The ENTSO-E Transparency Platform publishes generation, load, and cross-border power flows for European bidding zones. The values carry undocumented errors and violate energy conservation, and data reconciliation consists in correcting them to satisfy that balance. In this work, we formulate data reconciliation as Bayesian inference. Classical methods implicitly take a uniform prior, which we replace with a diffusion model estimated from the observations alone by empirical-Bayes expectation-maximization. The resulting posterior is sampled with a guided process whose projection step makes every sample satisfy the balance exactly. On synthetic systems of six to thirty-five bidding zones, built from ENTSO-E resource-adequacy data, the diffusion prior gives the most accurate reconciliation, increasingly so as the system grows. Its credible intervals are calibrated or conservative, and it remains the most accurate under a misspecified noise covariance. Finally, when the generation of entire zones goes unreported, the classical methods fail while the learned prior recovers it from cross-zone structure.
Figures & tables
| RMSE, fully observed | CRPS, fully observed | RMSE on masked signals | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | eu6 | euwest | euwide | eufull | eu6 | euwest | euwide | eufull | eu6 | euwest | euwide | eufull |
| Raw observations | 146.4 | 142.4 | 138.2 | 140.8 | – | – | – | – | – | – | – | – |
| Naive WLS | 141.9 | 138.0 | 134.0 | 136.5 | – | – | – | – | – | – | – | – |
| Oracle WLS | 134.9 | 130.1 | 126.9 | 129.3 | 75.4 | 79.3 | 84.8 | 84.7 | – | – | – | – |
| Constrained WLS QP | 115.1 | 110.0 | 104.0 | 103.9 | – | – | – | – | 5 197 | 5 311 | 5 580 | 5 615 |
| De Chalendar [ 20 ] | 287.3 | 232.7 | 197.9 | 194.2 | – | – | – | – | – | – | – | – |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Historical data | |
| Naive WLS | None | |
| Oracle WLS | True | None |
| Constrained WLS QP | True | None |
| De Chalendar (Eq. ( 14 )) | From magnitudes | None |
| Gaussian EB prior | True | CY 1-28 observations |
| Diffusion EB prior [ 11 ] | True | CY 1-28 observations |
| Method | Gen | Load | Flow | Overall |
|---|---|---|---|---|
| Raw observations | 131.7 | 151.5 | 200.0 | 146.4 |
| Naive WLS | 129.5 | 147.5 | 187.9 | 141.9 |
| Oracle WLS | 120.3 | 147.4 | 185.6 | 134.9 |
| Constrained WLS QP | 103.7 | 149.7 | 147.9 | 115.1 |
| De Chalendar | 237.5 | 729.9 | 233.5 | 287.3 |
| Gaussian EB prior | 106.2 | 141.0 | 153.8 | 117.6 |
| System | Prior | Marginal | Sliced- | Std mismatch |
|---|---|---|---|---|
| eu6 | Oracle | 0.05 | 0.06 | 0.07 |
| eu6 | Diffusion EB | 0.08 | 0.05 | 0.08 |
| euwest | Diffusion EB | 0.10 | 0.06 | 0.08 |
| euwide | Diffusion EB | 0.13 | 0.08 | 0.12 |
| eufull | Diffusion EB | 0.13 | 0.08 | 0.12 |
| System | Diffusion EB | Gaussian EB | Oracle WLS |
|---|---|---|---|
| eu6 | 291 | 344 | 419 |
| euwest | 308 | 333 | 442 |
| euwide | 345 | 303 | 472 |
| eufull | 347 | 284 | 471 |
| Constrained WLS QP | 114.1 | 115.1 | 118.6 |
|---|---|---|---|
| Gaussian EB prior | 121.6 | 117.6 | 151.2 |
| Diffusion EB prior | 112.9 | 99.8 | 104.9 |
| Diffusion EB gap to QP | |||
| Diffusion EB gap to Gaussian EB |
Diffusion Models
Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.
See all 1,251 papers