Nearest Neighbor Functional Curve Completion for Intraday Probabilistic Revisions of Renewable Generation Forecasts

Guillermo Terrén-Serrano; Michael Ludkovski

cs.LGOct 12, 2026

Nearest Neighbor Functional Curve Completion for Intraday Probabilistic Revisions of Renewable Generation Forecasts

Authors: Guillermo Terrén-Serrano, Michael Ludkovski

Organizations: Environmental Markets Lab (emLab), University of California Santa Barbara, Santa Barbara, CA, USA. · Environmental Studies Program, University of California Santa Barbara, Santa Barbara, CA, USA. · Statistics and Applied Probability, University of California Santa Barbara, Santa Barbara, CA, USA.

Abstract

We propose a nonparametric approach that fuses deterministic initial forecasts and a partial realization into a probabilistic forecast using the framework of functional curve completion. Our motivation is to construct intra-day forecasts for renewable energy generation, which is a key operational task in modern power grids. The aim is to combine the day-ahead (DA) forecast and the morning realized generation profile to create trajectories for the rest of the day, quantifying operational uncertainty. Starting with a large database of complete forecast-actual pairs across many assets, we adopt a functional point of view and construct a weighted neighborhood based on similarity metrics for the observed partial curve, the DA forecast, the day, and the geographical location. The selected complete nearest-neighbor curves provide a full predictive distribution, including functional focal envelopes and predictive bands based on functional depth. The hyperparameters governing neighborhood size, distances, and fusion rates are tuned across forecast update times and directly target probabilistic calibration in terms of the predictive probability integral transform. Three case studies using synthetic wind and solar assets, as well as real-life regional wind generation, showcase that our algorithm is able to achieve excellent calibration.

Figures & tables

Panel of Figure 1 — Figure 1: Intraday functional probabilistic forecasting. Wind generation at the Blanco Canyon Wind asset on June 26. The plots show probabilistic forecasts for the afternoon given the realized capacity factors (CF) from 12am until the forecast update time $\tau=12$ pm. (a) Heatmap for the predictive empirical density function, darker colors correspond to higher probabilities. The DA forecast is in black, the morning observations are in blue, and the afternoon (unseen from forecast update time $\tau$ ) generation is in dashed blue. (b) histograms and KDE estimates of the marginal predictive distributions at 12:30pm, 1pm, 2pm, 4pm, and 8pm (these times are indicated by the dashed lines in the top panel). The vertical black and blue lines indicate the DA forecast and the realized generation at that time.

Panel of Figure 2 — Figure 2: Mapping functional neighbors. Neighboring curves selected for Blanco Canyon Wind (marked with a bright green marker) on Apr 23 at $\tau=2$ pm (left panel). Gray markers represent assets without any similar curves to $(f_{\star},e_{\star})$ . Shaded markers with a white/black edge represent assets with $(f_{\star},e_{\star})$ -neighbors before/after applying the distance filter. The Haversine distance threshold, defined by the most distant functional neighbor, is the dashed black circle. Right panel: Distribution of $(f_{\star},e_{\star})$ -neighbors in the larger $\kappa_{0}$ -neighborhood by month (x-axis) and by distance to Blanco Canyon Wind (y-axis, black dashed line shows the distance cutoff). In both panels, light/dark colors represent low/high frequency.

Panel of Figure 3 — Figure 3: Wind generation day-ahead (DA) forecast update with adaptive uncertainty quantification. Blanco Canyon asset at forecast update time $\tau=12$ pm on Apr 23, when the fused neighborhood is dense (a, c, e, g, i, k), and on Jun 26, when it is sparse (b, d, f, h, j, l). Grey shading marks the observed period preceding $\tau$ . (a, b) Predictive marginal density heatmap with the median curve $\hat{\mu}_{\mathrm{med}}(s)$ (orange), the deepest curve $\hat{\mu}_{\mathrm{deep}}(s)$ (violet), and the focal curve $\hat{\mu}_{\mathrm{focal}}(s)$ (yellow). (c, d) Fused forecasts $\hat{\mu}_{i}(s)$ coloured by Modified Band Depth (MBD) rank. (e–l) Nested predictive regions at $\alpha\in\{0.1,0.2,0.3,0.4\}$ (light to dark): MBD-functional envelope $\mathcal{R}^{\mathrm{fun}}$ (e, f), adjusted MBD-functional envelope $\mathcal{R}^{\mathrm{adj}}$ (g, h), focal envelope $\mathcal{R}^{\mathrm{prj}}$ (i, j), and marginal quantiles $\mathcal{R}^{\mathrm{mar}}$ (k, l). The realized capacity factor $f_{\cdot}(t)$ is shown in blue and the DA forecast $e_{\cdot}(t)$ in black throughout.

Tuning Hyperparameter		6am	12pm	6pm
Trust level	$\nu$	9.7%	20.2%	0%
Trust horizon	$\vartheta$	6 h	10 h	12 h
DA forecast similarity weight	$\lambda_{e}$	21.8	11.4	13.5
Seasonal similarity weight	$\lambda_{d}$	4.38	0	0.98
kNN step I neighborhood size	$\kappa_{0}$	150	1,815	150
Partially observed curve forgetting rate	$\varepsilon_{f}$	0.125 h

Table 1: Tuning hyperparameters for dynamic updating of wind assets’ unbiased DA forecast. Estimated optimal hyperparameter for forecast update times

\tau=6

am, 12pm, and 6pm. The upper block lists the hyperparameters fine-tuned via Bayesian optimization. The lower block lists the hyperparameters held fixed across forecast update times.

Estimator	$\tau=6$ am			$\tau=12$ pm			$\tau=6$ pm
	RMSE	MAE	MBE	RMSE	MAE	MBE	RMSE	MAE	MBE
$\hat{\mu}_{\mathrm{mar}}$	0.132	0.099	0.003	0.144	0.111	-0.004	0.140	0.117	-0.006
$\hat{\mu}_{\mathrm{deep}}$	0.142	0.107	0.002	0.155	0.119	0.0	0.143	0.119	-0.005
$\hat{\mu}_{\mathrm{focal}}$	0.132	0.103	-0.003	0.147	0.116	-0.003	0.139	0.119	-0.004
ARIMA	0.249	0.204	-0.036	0.217	0.171	0.046	0.206	0.177	0.073
BRDA	0.143	0.109	0.001	0.148	0.115	0.001	0.159	0.135	0.005

Table 3: Wind asset unbiased DA forecast probabilistic revision deterministic error metrics. Root mean squared error (RMSE), mean absolute error (MAE), and mean bias error (MBE) of the point forecasts at forecast update times

\tau\in\{72,144,216\}

(i.e., 6am, 12pm, and 6pm). The estimators are the weighted marginal median curve

\hat{\mu}_{\text{med}}(s)

, the deepest curve

\hat{\mu}_{\text{deep}}(s)

ranked by Modified Band Depth, the focal curve

\hat{\mu}_{\text{focal}}(s)

, the ARIMA and Bootstrapped Re-anchored Day-Ahead (BRDA) forecast only. Errors are computed over the forecast horizon

\mathcal{I}_{>\tau}

and expressed in capacity factor units.

Panel of Figure 4 — Figure 4: Wind generation DA forecast functional probabilistic revision. intra-day forecasting for Barranca Wind asset on May 1. We consider three update times: $\tau=$ 6am (a, b, g), 12pm (c, d, h) and 6pm (e, f, i). Empirical density function (EDF) heatmap (a, c, e) with the median curves $\hat{\mu}_{\text{med}}$ Eq. ( 20 ) (orange), focal curves $\hat{\mu}_{\text{focal}}$ Eq. ( 27 ) (yellow), and deepest curves $\hat{\mu}_{\text{deep}}$ Eq. ( 24 ) (violet). The focal curve envelope confidence regions (b, d, f). Bottom row shows the PIT evaluated in test dataset from the update events —6am (g), 12pm (h) and 6pm (i)— to 3h lead time (g, h, i, left), and from 3h lead time to 6h lead time (g, h, i, right).

Panel of Figure 5 — Figure 5: Rolling dynamic updating. Blanco Canyon wind asset on Apr 23 (left) and Jun 26 (right), matching Fig 3 . The focal curve $\hat{\mu}_{\text{focal}}(s),s\in\mathcal{I}_{>\tau}$ is updated every 30 minutes from $\tau=$ 12:30am to $\tau=12$ am + $1$ day as new observations $f_{\star}(s)$ become available, with the colors indicating the forecast update time $\tau$ . The realized trajectory $f(\cdot)$ is in blue and the original DA forecast (given only hourly) $e_{\star}(\cdot)$ is in black.

Panel of Figure 6 — Figure 6: Functional neighborhood for solar generation. Neighboring curves selected for Blue Bell Solar II (indicated with a bright green diamond) asset on Aug 8 (left panel). Gray markers represent assets without any similar curves to Blue Bell Solar II. Shaded markers with a white edge represent $\kappa_{0}$ solar assets with neighboring curves before applying the distance filter, and shaded markers with a black edge those with neighbors after applying the distance filter. The Haversine distance threshold, defined by the most distant functional neighbor, is the dashed black circle. Frequency of similar curves in the $\kappa$ -neighborhood for each month, ranked by the distance to Blue Bell Solar II (right panel), before further filtering out by seasonal distance. In all panels, light colors represent low frequency and darker colors higher frequency.

Panel of Figure 7 — Figure 7: Solar generation DA forecast functional probabilistic revision. Blue Bell Solar II asset on Aug 8 at forecast update time $\tau=11$ am. Marginal empirical density function (EDF) heatmap with the median curve $\hat{\mu}_{\text{med}}$ in Eq. ( 20 ) (orange), focal curve $\hat{\mu}_{\text{focal}}$ in Eq. ( 27 ) (yellow), and deepest curve $\hat{\mu}_{\text{deep}}$ in Eq. ( 24 ) (violet) (a). Marginal quantile regions (b), fused forecasts $\hat{\mu}_{i}$ ranked by Modified Band Depth (MBD) (c), and focal envelope regions (d). Rolling probabilistic revision with the focal curve updated every 30 minutes from $\tau=7$ am to $\tau=6$ pm as new observations become available (e). PIT histograms at forecast update time $\tau=11$ am, evaluated on lead times $s\in$ 12-1pm (f, left) and $s\in$ 1-2pm (f, right).

Panel of Figure 8 — Figure 8: Wind 2DA regional forecast probabilistic revision. ERCOT’s North region on $d_{\star}=$ Feb 19 at forecast update time $\tau=12$ pm with a $\mathcal{N}^{(1)}$ neighborhood. (a) Wind regions in ERCOT (Panhandle, North, West, South, and Coastal), wind asset locations (black dots) and the graph $G$ defining the neighboring zones (thick edges link first-order neighbors and thin edges represent second-order neighbors). (b) Marginal quantile regions $\mathcal{R}^{\mathrm{mar}}$ and $\hat{\mu}_{\text{med}}$ in Eq. ( 20 ). (c) Focal envelopes $\mathcal{R}^{\mathrm{prj}}$ and $\hat{\mu}_{\text{focal}}$ in Eq. ( 27 ). In both panels, we show the envelopes for levels $\alpha=0.1,0.2,0.3,0.4$ . (d) Rolling probabilistic revision with the focal curve updated every hour as new observations become available, from $\tau=12$ pm to $\tau=11$ pm $+\,2$ days. Panel (e) PIT histograms evaluated over 0-18 lead hours (left), and 18-36 lead hours (right) starting at $\tau=12$ pm.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Functional nearest neighbors. Barranca Wind asset on $d_{\star}$ = May 1 at forecast update time $\tau=6$ am. Left column, panels (a,b,c): Day-ahead forecast neighbors $e_{1},\ldots,e_{\kappa}$ , realized capacity factor neighbors $f_{1}(t),\ldots,f_{\kappa}(t)$ , and fused curves $\hat{\mu}_{1}(s),\ldots,\hat{\mu}_{\kappa}(s)$ , all ranked by Modified Band Depth; Panel d: Spatiotemporal distribution of neighbors; $d_{\star}$ marked as the blue vertical line; Panel e: temporal distribution of neighbors by month; $d_{\star}$ is the blue horizontal line.

Panel of Figure 10 — Figure 10: Functional nearest neighbors. Blanco Canyon asset at forecast update time $\tau=12$ pm. Left column (a, c, e) : $d_{\star}$ = Apr 23; Right column (b, d, f): $d_{\star}$ = Jun 26. Top Row: DA forecast neighbors $e_{1},\ldots,e_{\kappa}$ , visualized as enhanced functional boxplot based on MBD (panels a, b); Middle row: realized partial CF curve neighbors $f_{1},\ldots,f_{\kappa}$ (panels c, d); Bottom row: fused curves $\hat{\mu}_{1},\ldots,\hat{\mu}_{\kappa}$ (panels e, f).

Panel of Figure 11 — Figure 11: Functional nearest neighbors for a solar asset. Blue Bell Solar on Aug 8 at forecast update time $\tau=11$ am. Left column: Blue curve is the partial realized $f_{\star}$ and solid black is the original DA forecast $e_{\star}$ . Panel a: Enhanced functional boxplot of the DA-forecast neighbors $e_{1},\ldots,e_{\kappa}$ . Panel b: Enhanced functional boxplot of the CF neighbors $f_{1},\ldots,f_{\kappa}$ . Panel c: Enhanced functional boxplot of the fused neighbors $\hat{\mu}_{1},\ldots,\hat{\mu}_{\kappa}$ . Panel d: Spatiotemporal distribution of the $\kappa_{0}$ neighbors across months of the years and Haversine distances; $d_{\star}$ is marked as the blue vertical line. Panel e: Distribution of neighbors by month $d_{\star}$ is the blue horizontal line.

Panel of Figure 12 — Figure 12: Functional nearest neighbors for ERCOT regional wind 2 Day-Ahead (DA) forecast updating. North ERCOT region on $d_{\star}=$ Feb 19 at forecast update time $\tau=12$ pm with a $\mathcal{N}^{(1)}$ neighborhood. Left column: Blue curve is the realized $f_{\star}$ and solid black is the original 2DA forecast $e_{\star}$ . Panel a: Enhanced functional boxplot for the 2DA-forecast neighbors $e_{1},\ldots,e_{\kappa}$ . Panel b: Realized capacity factor neighbors $f_{1},\ldots,f_{\kappa}$ . Panel c: Fused curves $\hat{\mu}_{1},\ldots,\hat{\mu}_{\kappa}$ ranked by Modified Band Depth. Panel d: Distribution of neighboring curves selected from ERCOT’s North ( $k=0$ ) and West ( $k=1$ ) regions, black edges mark the target region (North). Panel e: Temporal distribution of nearest neighbors by month, the blue line indicates the day $d_{\star}$ .

Panel of Figure 15 — Figure S1 : Average capacity factor (CF), forecasting error, and power capacity for the wind and solar assets in the Texas-7k grid. Relations between: (a) power capacity and RMSE shaded by average CF, (b) average CF and RMSE shaded by power capacity, and (c) average CF and power capacity shaded by RMSE. (d, g) Average hourly CF of all generation curves in each wind farm, (brighter indicates a higher average CF). (f) DA forecast hourly root mean squared error (RMSE) for each wind farm (brighter color, greater RMSE). (e) AC Installed power capacity (i.e., nameplate capacity) in each wind farm, darker represents larger power capacity.

Figure S2 : Sun elevation distance

\|d_{\star}-d\|_{S}

in Eq. ( 7 ) as a function of

d

. We illustrate with

d_{\star}=40

(Feb 9) in red,

d_{\star}=80

(March 21) in green,

d_{\star}=160

(June 9) in blue, and

d_{\star}=320

(Nov 16) in yellow.

Figure S3 : Examples of the Linear inverse exponential (LIE) function at 12pm (

\tau=720

)

\delta_{\vartheta,\nu,\eta}(s)

for

s>720

. We illustrate with 3 parameter configurations

\vartheta=2,\nu=0.4,\eta=2.5

in green,

\vartheta=4,\nu=0.6,\eta=2.5

in blue, and

\vartheta=6,\nu=0.8,\eta=2.5

in red. The dashed horizontal lines indicate the horizontal asymptotes at respective

\nu

levels.

Figure S4 : Wind generation capacity in ERCOT. Monthly operating wind capacity aggregated by commissioning vintage from Sep 1999 to May 2025 in the five regions that comprise ERCOT’s grid.

$\boldsymbol{\tau}$	Estimator	RMSE	MAE	MBE
10am	$\hat{\mu}_{\mathrm{mar}}$	0.143	0.087	$-$ 0.002
	$\hat{\mu}_{\mathrm{deep}}$	0.146	0.089	$-$ 0.003
	$\hat{\mu}_{\mathrm{focal}}$	0.142	0.092	$-$ 0.004
11am	$\hat{\mu}_{\mathrm{mar}}$	0.131	0.078	$-$ 0.002
	$\hat{\mu}_{\mathrm{deep}}$	0.134	0.079	$-$ 0.000
	$\hat{\mu}_{\mathrm{focal}}$	0.130	0.082	$-$ 0.001

Appendix

Table SI3 : Solar asset intra-day point forecast metrics. Root mean squared error (RMSE), mean absolute error (MAE), and mean bias error (MBE) of the point forecasts at forecast update times

\tau\in\{120,132,144,168\}

(i.e.,

\tau=

10am, 11am, 12pm, and 2pm). The estimators are the weighted median curve

\hat{\mu}_{\text{med}}(s)

, the deepest curve

\hat{\mu}_{\text{deep}}(s)

ranked by Modified Band Depth, and the focal curve

\hat{\mu}_{\text{focal}}(s)

. Errors are computed over the forecast horizon

\mathcal{I}_{>\tau}

and expressed in capacity factor units. Updating is based on fusing with de-biased DA forecasts.

Lead Interval		Update Time $\boldsymbol{\tau}$
$T_{0}$	$T_{1}$	10am	12pm	2pm
10am	11pm	0.022	—	—
11pm	12pm	0.033	—	—
12pm	1pm	0.028	0.035	—
1pm	2pm	0.029	0.027	—
2pm	3pm	0.036	0.039	0.042

Appendix

Table SI5 :

D_{KS}

scores for probabilistic revisions of solar asset generation with de-biased DA forecasts. Kolmogorov–Smirnov (KS) statistic of the PIT evaluated across each 1-hour interval of lead times

s\in[T_{0},T_{1}]

increments over the forecast horizon

\mathcal{I}_{>\tau}

at forecast update times

\tau=

10am, 12pm, 2pm. Lower is better.

$\boldsymbol{\tau}$	Estimator	RMSE	MAE	MBE
12pm	$\hat{\mu}_{mar}(s)$	0.131	0.106	0.014
	$\hat{\mu}_{deep}(s)$	0.142	0.115	0.012
	$\hat{\mu}_{focal}(s)$	0.128	0.105	0.004
12am+1D	$\hat{\mu}_{mar}(s)$	0.106	0.088	0.007
	$\hat{\mu}_{deep}(s)$	0.116	0.096	0.008
	$\hat{\mu}_{focal}(s)$	0.105	0.088	0.001

Appendix

Table SI7 : Wind zonal generation point forecast metrics. Root mean squared error (RMSE), mean absolute error (MAE), and mean bias error (MBE) at forecast update times

\tau\in\{12,24,36\}

(i.e., 12pm, 12am+1D, and 12pm+1D), averaged over the forecast horizon

\mathcal{I}_{>\tau}

and expressed in capacity factor units. The point estimators are the weighted median curve

\hat{\mu}_{\text{med}}

, the deepest curve

\hat{\mu}_{\text{deep}}

ranked by Modified Band Depth, and the focal curve

\hat{\mu}_{\text{focal}}

Probabilistic Time Series Forecasting

Probabilistic time series forecasting estimates distributions of future values or trajectories to quantify uncertainty and support risk-aware decisions.

See all 161 papers

Jan 31, 2026

CoCopula: Conformal Calibration and Copula Aggregation for Reliable Renewable Energy Forecasting

The rapid growth of renewable energy penetration has intensified the need for reliable probabilistic forecasts to support grid operations at aggregated (fleet or system) levels. In practice, however, system operators often lack access to fleet-level probabilistic models and instead rely on site-level forecasts produced by heterogeneous third-party providers. Constructing coherent and calibrated fleet-level probabilistic forecasts from such inputs remains challenging due to complex cross-site dependencies and aggregation-induced miscalibration. Motivated by a real-world operational challenge identified through collaboration with an industry partner, this paper presents a probabilistic aggregation and calibration framework that directly converts site-level probabilistic forecasts into reliable fleet-level forecasts. The paper introduces \method~ (Conformal Calibration and Copula Aggregation), which integrates Copula-based dependence modeling to capture cross-site correlations followed by Context-Aware Conformal Prediction (CACP) to mitigate miscalibration at the aggregated level. This combination enables dependence-aware aggregation while improving empirical coverage and maintaining sharp prediction intervals. Experiments on large-scale solar generation datasets from three system operators in the United States including MISO, ERCOT, and SPP demonstrate that the proposed \method{} approach achieves near-nominal empirical coverage and consistently lower Winkler scores than competing aggregation-based baselines across different target coverage levels. Importantly, \method{} achieves performance comparable to directly calibrated system-level forecasts, demonstrating its practical value when dedicated fleet-level forecasting models are unavailable.

Alireza Moradi, Mathieu Tanneau, Reza Zandehshahvar +1

NSF Artificial Intelligence Institute for Advances in Optimization H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA, USA

Probabilistic Forecasting Solar Power Forecasting Conformal Prediction

Ask PDF

Jul 13, 2026

Climate-Invariant Conformal Prediction Intervals for Multi-Horizon Solar and Wind Forecasting

Reliable uncertainty quantification is essential for integrating solar and wind generation into modern power systems, where operators must weigh risk rather than act on point forecasts alone. Existing probabilistic methods, however, often either lack finite-sample validity or require per-site recalibration, so a single model rarely transfers across the diverse climates of a dispersed generation fleet. This paper proposes a heteroscedastic, asymmetric, group-conditional split-conformal framework built on a bootstrap-diverse XGBoost ensemble, producing prediction intervals that adapt in width to local difficulty while retaining distribution-free coverage guarantees. A single fixed specification, with no per-site or per-horizon tuning, is evaluated across four climatologically distinct sites spanning both hemispheres, at horizons of 1 to 12 hours, for both solar irradiance and wind speed. The framework holds near-nominal coverage on both targets and reduces the Interval Score by up to 35% relative to competitive baselines, with the calibration and sharpness of its intervals shown to be properties of the method rather than of site-specific tuning.

Shreedhar Gangwar, Abhinav Bains, Banalaxmi Brahma

B. R. Ambedkar National Institute of Technology, Jalandhar, India · Department of Computer Science and Engineering, Dr. B.R. Ambedkar National Institute of Technology Jalandhar, India

Regression Prediction Intervals Group-Conditional Conformal Prediction Time Series Forecasting Conformal Prediction Solar Power Forecasting Wind Power Forecasting+2

Ask PDF

Jun 8, 2026

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

As renewable energy integration increases market volatility, probabilistic electricity price forecasting has become essential for effective risk management. However, current-proper-scoring rules often prioritize forecast sharpness at the expense of calibration, leading to overconfident and statistically unreliable uncertainty estimates. This work highlights the critical gap between theoretical scoring and practical calibration, demonstrating that models can become mere proxies for deterministic forecasts when reliability is neglected. We conclude that future research must shift toward calibration-aware objectives and architectures to ensure the distributional integrity of energy market forecasts.

Jan Niklas Lettner, Hadeer El Ashhab, Benjamin Schäfer

Institute for Automation and Applied Informatics

Probability Calibration Electricity Price Forecasting Probabilistic Forecasting Proper Scoring Rules Probabilistic Time Series Forecasting

Ask PDF

Abstract

Probabilistic Time Series Forecasting

Explore similar work

CoCopula: Conformal Calibration and Copula Aggregation for Reliable Renewable Energy Forecasting

Climate-Invariant Conformal Prediction Intervals for Multi-Horizon Solar and Wind Forecasting

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting