eess.SYJan 21, 2026

Calibrated uncertainty quantification for prosumer flexibility aggregation in ancillary service markets

Authors: Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A. R. Liisberg, Julian Lemos-Vinasco

Organizations: School of Electrical and Mechanical Engineering, The University of Adelaide, Ingkarni Wardli Building, Adelaide, SA 5005, Australia · Watts A/S, Brogade 19D, Køge 4600, Denmark

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reliable forecasting of prosumer flexibility is critical for demand response aggregators in frequency controlled ancillary service markets, where reliability requirements such as the P90 standard are enforced. Limited behavioural data and dependence on exogenous forecasts introduce uncertainty into flexibility estimates, which can cause reserve overestimation and financial penalties. This study systematically evaluates established uncertainty quantification approaches based on Monte Carlo dropout (MCD), conformal prediction (CP), and conformalised quantile regression (CQR) to construct prediction intervals for aggregated prosumer flexibility and support uncertainty-aware bidding. The approaches are evaluated for a behind-the-meter aggregator in the Danish manual frequency restoration reserve capacity market, using a large-scale synthetic dataset generated with an industry-grade home energy management system and publicly available load, solar, price, activation, and device-level data. Multiple multivariate CP strategies are applied to MCD predictions, alongside CQR using a separately trained quantile regression model. Under nominal test conditions, conformal methods substantially improve joint coverage over uncalibrated MCD baselines. More conservative methods satisfy the 90% lower-tail reliability requirement, while CQR and multivariate CP (MCP) yield narrower intervals and higher economic value, retaining approximately 69% and 68% of perfect-information profit, respectively, under the stricter penalty interpretation. A controlled robustness analysis shows that increasing perturbations of load, solar, price, and activation inputs increase overbidding and reduce coverage and economic performance. The results highlight the trade-off between reliability, economic value, and robustness to input uncertainty in aggregator bidding.

Figures & tables

Figure 1: Aggregator business model
 α^=⌈(∣Dcal∣+1)(1−α)⌉∣Dcal∣+1\hat{\alpha}=\frac{\lceil(|\mathcal{D}_{\text{cal}}|+1)(1-\alpha)\rceil}{|\mathcal{D}_{\text{cal}}|+1} 
Algorithm 1 Hybrid MCD and split CP for obtaining multivariate prediction intervals
Figure 2: Example conformal prediction region ξ\xi for y∈R2\boldsymbol{y}\in\mathbb{R}^{2} with bounding rectangle Bξ\mathcal{B}_{\xi}
Figure 3: Illustration of prediction regions ξ\xi and corresponding bounding hyperrectangles for y∈R2\boldsymbol{y}\in\mathbb{R}^{2} , across the evaluated CP methods.
Figure 4: Synthetic data generation framework for aggregated HEMS flexibility
HyperparameterMCD modelQR model
Batch size3216
Number of epochs10001900
OptimiserAdamAdam
Learning rate3×10−43\times 10^{-4}4.3×10−54.3\times 10^{-5}
Loss metricmean square errorPinball loss
Network architecture2 hidden layers2 hidden layers
Table 1: Neural network training hyperparameters
Figure 5: Hourly residuals between true and predicted aggregated up-regulation flexibility for the MCD and QR models
Figure 6: Distribution of per-sample RMSE between true and predicted aggregated up-regulation flexibility for MCD and QR models
UQ MethodExpectedJointJoint MPIWJoint IS
 CoveragePICPkWkW
Individual quantiles0.800.0033.20128.26
Naïve joint quantiles0.800.0086.56122.58
CQR0.800.80144.18147.99
MCP0.800.80163.90167.86
MMCP0.900.90211.37213.54
Table 2: Joint prediction interval quality comparison across various methods
Figure 7: Distribution of daily overbid frequency for the evaluated prediction methods
Figure 8: Distribution of daily average predicted up-regulation flexibility lower bounds as a percentage of aggregated maximum battery discharge power
Figure 9: Hourly heatmap for different parameters across evaluated prediction methods
Figure 10: Mean daily unadjusted and adjusted profit for all UQ methods across all β∈Sβ\beta\in\mathcal{S}_{\beta}
UQ MethodMean percentage of PI profit (%)
 R(xu,β)R(\boldsymbol{x^{\text{u}}},\beta)R1(xu,β)R_{1}(\boldsymbol{x^{\text{u}}},\beta)R2(xu,β)R_{2}(\boldsymbol{x^{\text{u}}},\beta)
Sample mean μ\boldsymbol{\mu}101-56-97
Individual quantiles9410-43
Naïve joint quantiles8746-10
CQR737069
MCP716968
Table 3: Mean percentage of PI profit for each UQ method
Figure 11: Sensitivity of aggregator performance to the revenue-sharing factor β\beta
Figure 12: Sensitivity of interval evaluation metrics to simultaneous perturbations of all input sources.
Figure 13: Sensitivity of mean daily overbid frequency to perturbations in individual input sources.
Figure 14: Sensitivity of adjusted profit to input perturbations for β=0.1\beta=0.1 .
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

BlockSymbolQuantityPerturbation treatment
Purchase priceλP\lambda^{P}Retail purchase price forecastCoupled additive
Selling priceλS\lambda^{S}Retail selling price forecastCoupled additive
Incentive priceλI,u\lambda^{I,u}Aggregator incentive priceNot perturbed
Activation signala↑a^{\uparrow}mFRR up-regulation activation forecastCopula partial resampling
Loadxlx_{l}Aggregated load forecastMultiplicative log-normal
Solarxsx_{s}Aggregated PV forecastMultiplicative log-normal
Appendix
Table 4: Input feature groups and corresponding perturbation treatment.
Input groupPerturbationDistributionLevel parameter
Load, solarMultiplicative xexp⁡(ε)x\exp(\varepsilon)ε∼N(0,s2)\varepsilon\sim\mathcal{N}(0,s^{2})ss = log-SD
Retail pricesShared additive shockϵ∼N(0,s2σS2)\epsilon\sim\mathcal{N}(0,s^{2}\sigma_{S}^{2})sσSs\sigma_{S}
Other price inputsAdditive x+ηx+\etaη∼N(0,s2σ2)\eta\sim\mathcal{N}(0,s^{2}\sigma^{2})sσs\sigma
ActivationCopula partial resamplingGaussian copula (Σ,p)(\boldsymbol{\Sigma},\boldsymbol{p})ss = fraction of hours
Settlement priceAdditive λU,pre+ξ\lambda^{U,\mathrm{pre}}+\xiξ∼N(0,s2σ2)\xi\sim\mathcal{N}(0,s^{2}\sigma^{2})sσs\sigma
Appendix
Table 5: Summary of perturbation models used in the robustness analysis.
More in this topic

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 647 papers

Explore similar work

CardsList
Jan 31, 2026

CoCopula: Conformal Calibration and Copula Aggregation for Reliable Renewable Energy Forecasting

The rapid growth of renewable energy penetration has intensified the need for reliable probabilistic forecasts to support grid operations at aggregated (fleet or system) levels. In practice, however, system operators often lack access to fleet-level probabilistic models and instead rely on site-level forecasts produced by heterogeneous third-party providers. Constructing coherent and calibrated fleet-level probabilistic forecasts from such inputs remains challenging due to complex cross-site dependencies and aggregation-induced miscalibration. Motivated by a real-world operational challenge identified through collaboration with an industry partner, this paper presents a probabilistic aggregation and calibration framework that directly converts site-level probabilistic forecasts into reliable fleet-level forecasts. The paper introduces \method~ (Conformal Calibration and Copula Aggregation), which integrates Copula-based dependence modeling to capture cross-site correlations followed by Context-Aware Conformal Prediction (CACP) to mitigate miscalibration at the aggregated level. This combination enables dependence-aware aggregation while improving empirical coverage and maintaining sharp prediction intervals. Experiments on large-scale solar generation datasets from three system operators in the United States including MISO, ERCOT, and SPP demonstrate that the proposed \method{} approach achieves near-nominal empirical coverage and consistently lower Winkler scores than competing aggregation-based baselines across different target coverage levels. Importantly, \method{} achieves performance comparable to directly calibrated system-level forecasts, demonstrating its practical value when dedicated fleet-level forecasting models are unavailable.

Alireza Moradi, Mathieu Tanneau, Reza Zandehshahvar +1

NSF Artificial Intelligence Institute for Advances in Optimization H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA, USA

Probabilistic ForecastingSolar Power ForecastingConformal Prediction
AskPDF
Jun 8, 2026

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

As renewable energy integration increases market volatility, probabilistic electricity price forecasting has become essential for effective risk management. However, current-proper-scoring rules often prioritize forecast sharpness at the expense of calibration, leading to overconfident and statistically unreliable uncertainty estimates. This work highlights the critical gap between theoretical scoring and practical calibration, demonstrating that models can become mere proxies for deterministic forecasts when reliability is neglected. We conclude that future research must shift toward calibration-aware objectives and architectures to ensure the distributional integrity of energy market forecasts.

Jan Niklas Lettner, Hadeer El Ashhab, Benjamin Schäfer

Institute for Automation and Applied Informatics

Probability CalibrationElectricity Price ForecastingProbabilistic ForecastingProper Scoring RulesProbabilistic Time Series Forecasting
AskPDF
Jun 8, 2026

Decision-Calibrated Conformal Uncertainty for Pacing Decisions in Streaming Advertising

We develop a decision-calibrated conformal framework for pacing decisions in streaming advertising. Pacing depends on uncertain future inventory, demand pressure, incremental response, and member-experience load. Instead of calibrating a generic forecast residual, the framework measures forecast error by its largest impact on the policies that could actually be deployed. The main theorem shows that the proposed score is the smallest valid uncertainty measure that uniformly protects all deployable pacing policies. Geometrically, it is the support function of the signed policy sensitivity set. Split conformal calibration gives finite-sample coverage for this score. A high-dimensional separation theorem shows that traditional residual calibration can be arbitrarily more conservative by paying for nuisance inventory dimensions, and a robust pacing result combines inventory, response, and experience uncertainty. On public-data-calibrated pacing replays built from Criteo Uplift and KuaiRand datasets, traditional conformal pacing remains unresolved with high residual radii of 7236.7 on Criteo and 4629.4 on KuaiRand. With the proposed decision calibration approach, the uncertainty radii are reduced to 18.4 and 278.6 respectively, with separate margins for value, delivery, budget, and member load. On Criteo, the proposed method certifies a less aggressive pacing policy than the point-forecast baseline, and reduces held-out any-violation rate from 16.7% to 3.3%, with zero budget and member-load violations. On KuaiRand, the choice remains unresolved. In a nutshell, the paper establishes that forecasts, response estimates, and member-experience models should be judged by whether they shrink the uncertainty that the pacing decision uses, as this leads to confident decisions that are not overly conservative.

Prashant Shekhar, Caroline Howard

Department of Mathematics Embry-Riddle Aeronautical University Daytona Beach, FL, USA

Sequential Decision MakingOnline AdvertisingConformal PredictionDecision-Making under Uncertainty
AskPDF