Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning

Christos Petridis; Zoran Obradovic; Mladen Kezunovic

cs.LGSep 2, 2026

Evaluating Large Language Models for Forced Outage Risk Prediction: Benefits and Comparison to Machine Learning

Authors: Christos Petridis, Zoran Obradovic, Mladen Kezunovic

Organizations: Temple University · Texas A&M University

Abstract

This study examines the ability of large language models (LLMs) to predict the risk of weather-related forced outages in the distribution grid in a zero-shot framework, without labeled training data. The problem is formulated as a binary severity classification task across three forecast horizons (3h, 6h, 12h), using six years of outage records and high-resolution weather data for a utility service area in central Texas. Four zero-shot LLMs are benchmarked against two supervised classifiers across two input configurations: one using current weather observations and the other using weather forecast data. Results show that supervised models outperform LLMs on macro-F1 and precision, while newer LLM generations achieve competitive scores. Beyond accuracy, LLMs offer complementary strengths in actionable reasoning and geographic scalability, suggesting that combining them with supervised models may be the best practice.

Figures & tables

Figure 1: Daily average outages (blue) with NOAA-annotated severe weather days (red) for the central Texas service area in 2018-2023.

Figure 2: Hourly outage counts (log scale).

Year	High	High%	High	High%	High	High%
	3-hour		6-hour		12-hour
2018	139	4.8%	69	4.7%	33	4.5%
2019	136	4.7%	78	5.3%	41	5.6%
2020	104	3.6%	55	3.8%	28	3.8%
2021	171	6.3%	94	7.0%	50	7.4%
2022	102	3.5%	55	3.8%	24	3.3%

Table 1: Class distribution per forecast horizon and year (high risk / total timesteps). February 2021 (Winter Storm Uri) is excluded from all splits.

With weather forecast
	3h			6h			12h
Model	Macro-F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Macro-F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Macro-F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
GPT-4o	0.57	0.15	0.56	0.57	0.15	0.65	0.55	0.14	0.82
GPT-4o-mini	0.37	0.07	0.78	0.33	0.07	0.88	0.29	0.07	0.93
GPT-4.1	0.60	0.29	0.20	0.63	0.29	0.31	0.63	0.25	0.41
GPT-4.1-mini	0.61	0.27	0.26	0.63	0.24	0.40	0.61	0.20	0.60

Table 2: Model performance across forecast horizons and weather configurations. Best value overall per column is shown in bold .

	Forecast gain $\Delta$ W (with $-$ without)
Model	3h	6h	12h
GPT-4o	+0.09 (+18.7%)	+0.08 (+16.3%)	+0.05 (+10.0%)
GPT-4o-mini	+0.22 (+146.7%)	+0.17 (+106.2%)	+0.16 (+123.1%)
GPT-4.1	+0.02 (+3.4%)	+0.04 (+6.8%)	+0.06 (+10.5%)
GPT-4.1-mini	+0.00 (+0.0%)	+0.02 (+3.3%)	+0.03 (+5.2%)
LightGBM	+0.03 (+4.2%)	+0.01 (+1.4%)	+0.06 (+9.8%)

Table 3: Macro-F1 gain with weather forecast (

\Delta

W, absolute) across forecast horizons.

Explore similar work

Data Leakage Inflates Generalizability of Power Outage Prediction Models
Aug 25, 2026Yamil Essus, Ranga Raju Vatsavai, Benjamin RachunokOOD Generalization Power Systems
Department of Civil and Mineral Engineering, University of Toronto, Toronto, Canada · Computer Science Department, North Carolina State University, Raleigh, NC · Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC +1
Forecasting Supply Chain Disruptions with Foresight Learning
Apr 1, 2026Benjamin Turtel, Paul Wilczewski, Kris SkotheimProbabilistic Forecasting Supply Chain Management
Lightning Rod Labs
LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
Aug 24, 2026Xiaogang Xu, Jiaqi Tang, Jianmin Chen +12LLM Agents LLM-Based Time Series Forecasting
Zhejiang University, China · The Hong Kong University of Science and Technology, Hong Kong · Northwestern Polytechnical University, China +4

Aug 25, 2026J/K move · Enter open · S save

Data Leakage Inflates Generalizability of Power Outage Prediction Models

Yamil Essus, Ranga Raju Vatsavai, Benjamin Rachunok

Department of Civil and Mineral Engineering, University of Toronto, Toronto, Canada · Computer Science Department, North Carolina State University, Raleigh, NC · Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC+1

OOD Generalization Power Systems Temporal Data Leakage Spatiotemporal Forecasting

Open paper AskPDF arXiv

Power outage prediction models are increasingly used in assessments of climate-driven infrastructure risk, yet current evaluation practices obscure whether these models generalize to the novel conditions such applications require. We identify three common methodological choices in power outage prediction models that influence their ability to generalize across spatial, temporal, and event-based settings. We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023 and feature sets derived from weather reanalysis and land-cover data, and embeddings from a GeoAI foundation model (Prithvi WxC). Specifically, we assess model performance under multiple test selection strategies, including unfiltered random splits, leave-one-state-out, and leave-one-event-out designs, which increasingly approximate real-world deployment conditions. While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation. Under spatial and temporal holdout experiments, predictive accuracy degrades substantially, with models often failing to outperform a simple null baseline. Incorporating GeoAI foundation model embeddings yields limited and inconsistent improvements, primarily for spatial generalization, and does not resolve poor event-level transferability. These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value. Progress will likely require improved data coverage, more realistic evaluation protocols, and a shift in focus from marginal modeling advances toward addressing structural data constraints.

Abstract

Explore similar work

Data Leakage Inflates Generalizability of Power Outage Prediction Models

Forecasting Supply Chain Disruptions with Foresight Learning

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications