Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids

Sajib Debnath; Md. Uzzal Mia

cs.LGMar 9, 2026

Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids

Authors: Sajib Debnath, Md. Uzzal Mia

Organizations: O&M Analytics, AES Clean Energ; The AES Corporation; Louisville, CO 80027, USA · Department of Information and Communication Engineering; Pabna University of Science and Technology; Rajapur, Pabna, Bangladesh

Abstract

The reliable operation of modern power grids requires probabilistic load forecasts with well-calibrated uncertainty estimates. However, existing deep learning models produce overconfident point predictions that fail catastrophically under extreme weather distributional shifts. This study proposes a Bayesian Transformer (BT) framework that integrates three complementary uncertainty mechanisms into a PatchTST backbone: Monte Carlo Dropout for epistemic parameter uncertainty, variational feed-forward layers with log-uniform weight priors, and stochastic attention with learnable Gaussian noise perturbations on pre-softmax logits, representing, to the best of our knowledge, the first application of Bayesian attention to probabilistic load forecasting. A seven-level multi-quantile pinball-loss prediction head and post-training isotonic regression calibration produce sharp, near-nominally covered prediction intervals. Evaluation of five grid datasets (PJM, ERCOT, ENTSO-E Germany, France, and Great Britain) augmented with NOAA covariates across 24, 48, and 168-hour horizons demonstrates state-of-the-art performance. On the primary benchmark (PJM, H=24h), BT achieves a CRPS of 0.0289, improving 7.4% over Deep Ensembles and 29.9% over the deterministic LSTM, with 90.4% PICP at the 90% nominal level and the narrowest prediction intervals (4,960 MW) among all probabilistic baselines. During heat-wave and cold snap events, BT maintained 89.6% and 90.1% PICP respectively, versus 64.7% and 67.2% for the deterministic LSTM, confirming that Bayesian epistemic uncertainty naturally widens intervals for out-of-distribution inputs. Calibration remained stable across all horizons (89.8-90.4% PICP), while ablation confirmed that each component contributed a distinct value. The calibrated outputs directly support risk-based reserve sizing, stochastic unit commitment, and demand response activation.

Figures & tables

Figure 1: Overview of the proposed Bayesian Transformer framework for probabilistic load forecasting

Dataset	Territory	Period	Obs. (hrs)	Peak Demand	Renew. Share
PJM	Mid-Atlantic & Midwest US	2015–2023	78,840	$\sim$ 165 GW	$\sim$ 12%
ERCOT	Texas, US	2015–2023	78,840	$\sim$ 85 GW	$\sim$ 32%
ENTSO-E (DE)	Germany	2015–2023	78,840	$\sim$ 83 GW	$\sim$ 52%
ENTSO-E (FR)	France	2015–2023	78,840	$\sim$ 97 GW	$\sim$ 28%
ENTSO-E (GB)	Great Britain	2015–2023	78,840	$\sim$ 62 GW	$\sim$ 40%

Table 1: Summary of load demand datasets used in the experimental evaluation.

Figure 2: Mean monthly demand profiles with inter-annual variability across five grid datasets (2015–2023)

Figure 3: Normalised hourly demand distributions across five grid datasets

Figure 4: Annual renewable penetration share and demand coefficient of variation (2015–2023)

Figure 5: Mean diurnal load profiles for weekday and weekend by grid

Figure 6: Nonlinear temperature–demand relationship across five grid datasets

Partition	Period	Purpose	Proportion
Training	Jan 2015 – Dec 2020	Model parameter optimization	$\sim$ 67%
Validation (calibration fit)	Jan 2021 – Jun 2021	Calibration pipeline fitting, early stopping	$\sim$ 8%
Validation (calibration eval)	Jul 2021 – Dec 2021	Post-calibration performance verification	$\sim$ 8%
Test	Jan 2022 – Dec 2023	Final held-out evaluation	$\sim$ 22%

Table 2: Chronological data partitioning protocol applied uniformly across all datasets.

Figure 7: Autocorrelation function of hourly load demand (PJM, 2015–2023)

Hyperparameter	Value	Selection Criterion
Transformer layers ( $N$ )	6	Validation CRPS grid search
Attention heads	8	Standard for $d=512$
Model dimension ( $d$ )	512	Validation CRPS grid search
Feed-forward dimension	2048	$4\times$ model dimension convention
Patch size ( $P$ )	16 hrs	Captures intraday ramp patterns
Lookback window ( $L$ )	168 hrs	Full weekly periodicity coverage

Table 3: Complete hyperparameter configuration of the proposed Bayesian Transformer.

Model	MAE (MW)	RMSE (MW)	CRPS	PICP 90%	MPIW (MW)	Winkler
Det. LSTM	1,842	2,531	0.0412	81.3%	7,210	0.0891
Standard Transformer	1,621	2,218	0.0387	83.1%	6,870	0.0823
Quantile LSTM	1,698	2,344	0.0351	87.6%	5,940	0.0714
Deep Ensemble (5 $\times$ )	1,589	2,176	0.0312	89.2%	5,480	0.0643
CQR (PatchTST)	1,634	2,257	0.0328	90.1%	6,120	0.0671
BT (Ours)	1,573	2,143	0.0289	90.4%	4,960	0.0598

Table 4: Overall performance on PJM dataset,

H=24

h. BT = Bayesian Transformer (proposed).

Dataset	Det. LSTM	Std. Transformer	Quant. LSTM	Deep Ensemble	CQR	BT (Ours)
PJM-CRPS	0.0412	0.0387	0.0351	0.0312	0.0328	0.0289
PJM-PICP 90%	81.3%	83.1%	87.6%	89.2%	90.1%	90.4%
ERCOT-CRPS	0.0448	0.0421	0.0382	0.0334	0.0357	0.0311
ERCOT-PICP 90%	79.8%	81.4%	86.1%	88.4%	90.0%	90.2%
ENTSO-E DE-CRPS	0.0391	0.0363	0.0328	0.0297	0.0311	0.0274
ENTSO-E DE-PICP 90%	82.1%	83.8%	88.2%	89.6%	90.3%	90.5%

Table 5: Cross-dataset CRPS and PICP (90%) for all models,

H=24

h. Best values in bold.

Figure 8: Representative 72-hour probabilistic forecast during a heatwave event: Bayesian Transformer multi-quantile prediction bands versus deterministic LSTM point forecast (ERCOT)

Model	HW MAE (MW)	HW PICP 90%	CS MAE (MW)	CS PICP 90%
Det. LSTM	3,241	64.7%	3,089	67.2%
Standard Transformer	2,987	66.8%	2,874	68.9%
Quantile LSTM	2,634	76.3%	2,512	78.1%
Deep Ensemble (5 $\times$ )	2,318	85.1%	2,201	86.4%
CQR (PatchTST)	2,541	88.7%	2,389	89.1%
BT (Ours)	2,104	89.6%	1,987	90.1%

Table 6: Extreme weather robustness. HW: Heatwave, CS = Cold Snap. Results averaged across all five datasets.

Figure 9: Extreme weather robustness: PICP at 90% nominal level under normal, heatwave, and cold snap conditions

Nominal Level	Quantile LSTM	Deep Ensemble	CQR	BT (Ours)	Target
5th percentile	3.1%	4.2%	5.1%	5.0%	5.0%
10th percentile	7.4%	8.9%	10.0%	9.9%	10.0%
25th percentile	20.1%	23.4%	24.8%	24.9%	25.0%
50th percentile	47.8%	49.1%	50.0%	50.0%	50.0%
75th percentile	70.3%	73.2%	74.9%	75.1%	75.0%
90th percentile	85.9%	88.6%	90.1%	90.4%	90.0%

Table 7: Empirical coverage at all seven nominal quantile levels on PJM test set (

H=24

h).

Figure 10: Reliability diagram: empirical coverage probability versus nominal quantile level on the PJM test set (H = 24 h)

Figure 11: Forecast horizon sensitivity: CRPS and PICP degradation across 24 h, 48 h, and 168 h horizons (PJM dataset)

Configuration	MAE (MW)	RMSE (MW)	CRPS	PICP 90%	MPIW (MW)
(A) Base PatchTST (deterministic)	1,621	2,218	0.0387	83.1%	—
(B) A + MC Dropout	1,598	2,189	0.0334	87.4%	5,820
(C) B + Variational FF layers	1,584	2,162	0.0311	88.9%	5,490
(D) C + Stochastic Attention	1,577	2,153	0.0296	89.8%	5,180
(E) D + Quantile head (pinball)	1,574	2,148	0.0293	89.9%	5,040
(F) E + Post-training Calibration (Full)	1,573	2,143	0.0289	90.4%	4,960

Table 8: Ablation study results on PJM test set,

H=24

h. Each row adds one component cumulatively.

Figure 12: Ablation study: incremental contribution of each Bayesian component to CRPS and PICP (PJM, H = 24 h)

Model	Params (M)	Train Time / Epoch	Inference Latency	MC Passes
Det. LSTM	18.4	1.2 min	0.018 s	1
Standard Transformer	42.1	2.1 min	0.031 s	1
Quantile LSTM	18.4	1.4 min	0.019 s	1
Deep Ensemble (5 $\times$ )	210.5	10.5 min	0.155 s	5
CQR (PatchTST)	42.1	2.1 min	0.033 s	1
BT (Ours)	44.3	2.8 min	1.84 s	100

Table 9: Computational cost comparison. Training time measured per epoch on NVIDIA A100 GPU.

Figure 13: Predictive uncertainty decomposition: aleatoric and epistemic contributions over a 48-hour forecast window spanning heatwave onset, and epistemic uncertainty scaling with temperature deviation from the training distribution

Explore similar work

A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods
Jul 17, 2026Matthias Hertel, Sebastian Pütz, Jonathan Kolar +3Time Series Forecasting Forecasting Benchmarks
Karlsruhe Institute of Technology, Germany · Helmholtz AI, Germany
From Shared Demand Patterns to Local Uncertainty: Probabilistic Load Forecasting by Mixing Compact Adaptations
Oct 6, 2026Haoran Li, Zhe Cheng, Yang WengElectricity Load Forecasting Probabilistic Time Series Forecasting
Learning-based Probabilistic Load Forecasting with Post-hoc and In-model Uncertainty
Jul 14, 2026Sarah Al-Shareeda, Gulcihan Ozdemir, Heung Seok JeonQuantile Regression Time Series Forecasting
aComputer System and Technology Department, Faculty of Computer Science and Information Technology, University of Malaya, Malaysia · bInformatics Institute, Istanbul Technical University, Turkey · cComputer Engineering Department Konkuk University South Korea

Jul 17, 2026J/K move · Enter open · S save

A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods

Matthias Hertel, Sebastian Pütz, Jonathan Kolar, Benjamin Schäfer+2

Karlsruhe Institute of Technology, Germany · Helmholtz AI, Germany

Time Series Forecasting Forecasting Benchmarks Electricity Load Forecasting Time Series Transformers Time-Series Foundation Models Power Systems

Open paper AskPDF arXiv

Accurate load forecasting at multiple grid levels is essential for future smart grids, ranging from aggregated control area forecasts for balancing supply and demand to forecasts of individual end-consumer loads for demand-side management and energy management systems. We present a comprehensive benchmark for load forecasting across grid levels, comprising three datasets that represent a transmission system operator control area, low-voltage grid feeders, and individual end consumers. We evaluate ten methods for short-term load forecasting and find that Transformer-based approaches consistently outperform established methods, reducing forecast error by 6.6-10.7 %. To analyze the impact of architectural design, we introduce YAformer, a flexible Transformer architecture that integrates modifications from prior work and is optimized via hyperparameter optimization. However, the standard Transformer achieves superior performance, suggesting that these architectural modifications are not required for accurate load forecasting. We further evaluate the Transformer-based time-series foundation model Chronos-2, which demonstrates competitive zero-shot performance on two datasets but fails to accurately capture special events in the TSO data. Detailed analyses reveal model-specific strengths and weaknesses, and ablation studies highlight the importance of long input contexts, covariates and continuous retraining - aspects that are often overlooked in the time-series forecasting literature.

Abstract

Explore similar work

A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods

From Shared Demand Patterns to Local Uncertainty: Probabilistic Load Forecasting by Mixing Compact Adaptations

Learning-based Probabilistic Load Forecasting with Post-hoc and In-model Uncertainty