Data Leakage Inflates Generalizability of Power Outage Prediction Models

Yamil Essus; Ranga Raju Vatsavai; Benjamin Rachunok

cs.LGAug 25, 2026

Data Leakage Inflates Generalizability of Power Outage Prediction Models

Authors: Yamil Essus, Ranga Raju Vatsavai, Benjamin Rachunok

Organizations: Department of Civil and Mineral Engineering, University of Toronto, Toronto, Canada · Computer Science Department, North Carolina State University, Raleigh, NC · Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC · Operations Research Program, North Carolina State University, Raleigh, NC

Abstract

Power outage prediction models are increasingly used in assessments of climate-driven infrastructure risk, yet current evaluation practices obscure whether these models generalize to the novel conditions such applications require. We identify three common methodological choices in power outage prediction models that influence their ability to generalize across spatial, temporal, and event-based settings. We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023 and feature sets derived from weather reanalysis and land-cover data, and embeddings from a GeoAI foundation model (Prithvi WxC). Specifically, we assess model performance under multiple test selection strategies, including unfiltered random splits, leave-one-state-out, and leave-one-event-out designs, which increasingly approximate real-world deployment conditions. While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation. Under spatial and temporal holdout experiments, predictive accuracy degrades substantially, with models often failing to outperform a simple null baseline. Incorporating GeoAI foundation model embeddings yields limited and inconsistent improvements, primarily for spatial generalization, and does not resolve poor event-level transferability. These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value. Progress will likely require improved data coverage, more realistic evaluation protocols, and a shift in focus from marginal modeling advances toward addressing structural data constraints.

Figures & tables

Feature	Abbreviation	Source
Temperature Dew Point	d2m	ERA5 - Single Levels
Temperature	t2m	ERA5 - Single Levels
Wind Gust	i10fg	ERA5 - Single Levels
Mean Large-Scale Precipitation Rate	mlspr	ERA5 - Land
Wind Speed	u10 and v10	ERA5 - Land
Volumetric Soil Water	swvl1-4	ERA5 - Land

Table 1: Features used to train the predictive models. All features were aggregated to a county-day resolution using an average spatial join operation between point features and county polygons. The max function was used for temporal aggregation.

Figure 1: Example of interpolation procedure. (a) Prithvi WxC embedding dimension 4 at original resolution, (b) bilinear interpolation of embedding that covers the extent of the study area, and (c) county-day average for this sample computed as the mean of the interpolated values.

Figure 2: Strategies for selecting a test set. In each subfigure, an example for each strategy is provided. Counties in orange would be chosen as the test set. (a) Example of random split strategy using 80-20 ratio, (b) Leave-state-out strategy, North Carolina in this case, and (c) Leave-event-out, which selects the entire study region as test set, for a specific date.

Figure 3: Performance of random split experiment for each feature set and target variable as percentage of variance in the dependent variable that is accounted for in the model ( $R^{2}$ ).

	$R^{2}$		$MAE$
	pCoP	CoP	pCoP	CoP
ERA5	0.44 ± 0.01	0.32 ± 0.07	0.06 ± 0.00	1,807.20 ± 48.13
ERA5 + Tree Density	0.45 ± 0.01	0.36 ± 0.04	0.06 ± 0.00	1,796.08 ± 47.62
Prithvi	0.49 ± 0.01	0.33 ± 0.04	0.05 ± 0.00	1,720.44 ± 32.31

Table 2: 95-percent confidence interval for

R^{2}

and mean absolute error (

MAE

) performance metrics for each experiment in the 80-20 split category.

Figure 4: Mean absolute error for the leave-state-out experiment for each target variable ( $pCoP$ and $CoP$ ), feature set and state. Lower is better. In each case, $MAE$ of using a null model is labeled, and the positive or negative difference between the weather model and the null model is presented in grey or red, respectively.

Figure 5: Results of the leave-event-out experiment as the difference in Mean Absolute Error between the trained model and null model. Higher is better.

Figure 6: Results of the leave-event-out experiment as difference in Mean Absolute Error between null model and the best trained model among all feature sets in each case, colored by event type. Higher is better.

Target Variable	Feature Set	Test Type	# of test sets	Avg. Test MAE
CoP (absolute)	ERA5 (No static features)	80-20 Split	20	1,807.202
		Spatial LOO	13	2,185.835
		Temporal LOO	28	2,766.400
	ERA5 + Tree Density (Static features)	80-20 Split	20	1,796.083
		Spatial LOO	13	2,235.405
		Temporal LOO	28	2,674.593

Table 3: Summary of experimental design and average Mean Absolute Error. Different test types produce different number of test sets. Spatial Leave-One-Out experiments are conducted by leaving all the data from a single State out of training and used for testing. Temporal Leave-One-Out experiments leave data for a given date out of training and used for testing.

Type	Date	Name	CoP		pCoP
Type	Date	Name	Best Features	MAE	Best Features	MAE
Hurricane	2018-09-14	Florence	Prithvi	3,060.08	ERA5	0.09
Hurricane	2018-10-11	Michael	Prithvi	4,028.98	ERA5 + Tree Density	0.17
Hurricane	2020-08-04	Isaias	ERA5	3,452.35	ERA5 + Tree Density	0.11
Hurricane	2020-10-29	Zeta	ERA5	4,388.31	ERA5	0.08
Hurricane	2022-09-29	Ian	ERA5 + Tree Density	6,448.92	ERA5	0.05

Table 4: Detailed results of the leave-event-out experiment for the best feature set in each run.

Explore similar work

Sep 2, 2026

Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning

This study examines the ability of large language models (LLMs) to predict the risk of weather-related forced outages in the distribution grid in a zero-shot framework, without labeled training data. The problem is formulated as a binary severity classification task across three forecast horizons (3h, 6h, 12h), using six years of outage records and high-resolution weather data for a utility service area in central Texas. Four zero-shot LLMs are benchmarked against two supervised classifiers across two input configurations: one using current weather observations and the other using weather forecast data. Results show that supervised models outperform LLMs on macro-F1 and precision, while newer LLM generations achieve competitive scores. Beyond accuracy, LLMs offer complementary strengths in actionable reasoning and geographic scalability, suggesting that combining them with supervised models may be the best practice.

Christos Petridis, Zoran Obradovic, Mladen Kezunovic

Temple University · Texas A&M University

LLM Evaluation Zero-Shot Learning

Ask PDF

Apr 6, 2026

Empowering Power Outage Prediction with Spatially Aware Hybrid Graph Neural Networks and Contrastive Learning

Extreme weather events, such as severe storms, hurricanes, snowstorms, and ice storms, which are exacerbated by climate change, frequently cause widespread power outages. These outages halt industrial operations, impact communities, damage critical infrastructure, profoundly disrupt economies, and have far-reaching effects across various sectors. To mitigate these effects, the University of Connecticut and Eversource Energy Center have developed an outage prediction modeling (OPM) system to provide pre-emptive forecasts for electric distribution networks before such weather events occur. However, existing predictive models in the system do not incorporate the spatial effect of extreme weather events. To this end, we develop Spatially Aware Hybrid Graph Neural Networks (SA-HGNN) with contrastive learning to enhance the OPM predictions for extreme weather-induced power outages. Specifically, we first encode spatial relationships of both static features (e.g., land cover, infrastructure) and event-specific dynamic features (e.g., wind speed, precipitation) via Spatially Aware Hybrid Graph Neural Networks (SA-HGNN). Next, we leverage contrastive learning to handle the imbalance problem associated with different types of extreme weather events and generate location-specific embeddings by minimizing intra-event distances between similar locations while maximizing inter-event distances across all locations. Thorough empirical studies in four utility service territories, i.e., Connecticut, Western Massachusetts, Eastern Massachusetts, and New Hampshire, demonstrate that SA-HGNN can achieve state-of-the-art performance for power outage prediction.

Xuyang Shen, Zijie Pan, Diego Cerrai +4

University of Connecticut · Western Connecticut State University · Eversource Energy

Power Systems Graph Neural Networks

Ask PDF

Sep 1, 2026

OutageDiT: A Generative Foundation Model for Power Outage Forecasting and Scenario Simulation

Power-outage planning requires scenarios before an event occurs. These scenarios must represent uncertainty in magnitude, timing, and duration while preserving temporal dependence. However, severe events are rare, and data from any single region contain few examples of extreme outage and restoration patterns. To address this challenge, we introduce OutageDiT, a foundation model for generating seven-day outage trajectories at quarter-hour resolution, trained on outage and weather records across the United States. Specifically, a condition encoder processes the historical context and known future covariates once per forecast, and a shallow flow decoder reuses the resulting horizon-aligned states to generate complete trajectories. The resulting samples support point forecasting, uncertainty quantification, and conditional event simulation within one deep generative model. Across outage forecasting benchmarks, OutageDiT improves forecast accuracy and scenario quality over strong baselines and supports zero-shot transfer to held-out regions. Together, these results position conditional outage simulation as a bridge from outage forecasting to operational planning under uncertainty.

Yunqin Zhu, Feng Qiu, Yao Xie

H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology · Argonne National Laboratory

Conditional Generative Modeling Time-Series Foundation Models Probabilistic Time Series Forecasting Power Systems

Ask PDF