Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids

Sajib Debnath; Md. Uzzal Mia

cs.LGMar 9, 2026

Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids

Authors: Sajib Debnath, Md. Uzzal Mia

Organizations: O&M Analytics, AES Clean Energ; The AES Corporation; Louisville, CO 80027, USA · Department of Information and Communication Engineering; Pabna University of Science and Technology; Rajapur, Pabna, Bangladesh

Abstract

The reliable operation of modern power grids requires probabilistic load forecasts with well-calibrated uncertainty estimates. However, existing deep learning models produce overconfident point predictions that fail catastrophically under extreme weather distributional shifts. This study proposes a Bayesian Transformer (BT) framework that integrates three complementary uncertainty mechanisms into a PatchTST backbone: Monte Carlo Dropout for epistemic parameter uncertainty, variational feed-forward layers with log-uniform weight priors, and stochastic attention with learnable Gaussian noise perturbations on pre-softmax logits, representing, to the best of our knowledge, the first application of Bayesian attention to probabilistic load forecasting. A seven-level multi-quantile pinball-loss prediction head and post-training isotonic regression calibration produce sharp, near-nominally covered prediction intervals. Evaluation of five grid datasets (PJM, ERCOT, ENTSO-E Germany, France, and Great Britain) augmented with NOAA covariates across 24, 48, and 168-hour horizons demonstrates state-of-the-art performance. On the primary benchmark (PJM, H=24h), BT achieves a CRPS of 0.0289, improving 7.4% over Deep Ensembles and 29.9% over the deterministic LSTM, with 90.4% PICP at the 90% nominal level and the narrowest prediction intervals (4,960 MW) among all probabilistic baselines. During heat-wave and cold snap events, BT maintained 89.6% and 90.1% PICP respectively, versus 64.7% and 67.2% for the deterministic LSTM, confirming that Bayesian epistemic uncertainty naturally widens intervals for out-of-distribution inputs. Calibration remained stable across all horizons (89.8-90.4% PICP), while ablation confirmed that each component contributed a distinct value. The calibrated outputs directly support risk-based reserve sizing, stochastic unit commitment, and demand response activation.

Figures & tables

Figure 1: Overview of the proposed Bayesian Transformer framework for probabilistic load forecasting

Dataset	Territory	Period	Obs. (hrs)	Peak Demand	Renew. Share
PJM	Mid-Atlantic & Midwest US	2015–2023	78,840	$\sim$ 165 GW	$\sim$ 12%
ERCOT	Texas, US	2015–2023	78,840	$\sim$ 85 GW	$\sim$ 32%
ENTSO-E (DE)	Germany	2015–2023	78,840	$\sim$ 83 GW	$\sim$ 52%
ENTSO-E (FR)	France	2015–2023	78,840	$\sim$ 97 GW	$\sim$ 28%
ENTSO-E (GB)	Great Britain	2015–2023	78,840	$\sim$ 62 GW	$\sim$ 40%

Table 1: Summary of load demand datasets used in the experimental evaluation.

Figure 2: Mean monthly demand profiles with inter-annual variability across five grid datasets (2015–2023)

Figure 3: Normalised hourly demand distributions across five grid datasets

Figure 4: Annual renewable penetration share and demand coefficient of variation (2015–2023)

Figure 5: Mean diurnal load profiles for weekday and weekend by grid

Figure 6: Nonlinear temperature–demand relationship across five grid datasets

Partition	Period	Purpose	Proportion
Training	Jan 2015 – Dec 2020	Model parameter optimization	$\sim$ 67%
Validation (calibration fit)	Jan 2021 – Jun 2021	Calibration pipeline fitting, early stopping	$\sim$ 8%
Validation (calibration eval)	Jul 2021 – Dec 2021	Post-calibration performance verification	$\sim$ 8%
Test	Jan 2022 – Dec 2023	Final held-out evaluation	$\sim$ 22%

Table 2: Chronological data partitioning protocol applied uniformly across all datasets.

Figure 7: Autocorrelation function of hourly load demand (PJM, 2015–2023)

Hyperparameter	Value	Selection Criterion
Transformer layers ( $N$ )	6	Validation CRPS grid search
Attention heads	8	Standard for $d=512$
Model dimension ( $d$ )	512	Validation CRPS grid search
Feed-forward dimension	2048	$4\times$ model dimension convention
Patch size ( $P$ )	16 hrs	Captures intraday ramp patterns
Lookback window ( $L$ )	168 hrs	Full weekly periodicity coverage

Table 3: Complete hyperparameter configuration of the proposed Bayesian Transformer.

Model	MAE (MW)	RMSE (MW)	CRPS	PICP 90%	MPIW (MW)	Winkler
Det. LSTM	1,842	2,531	0.0412	81.3%	7,210	0.0891
Standard Transformer	1,621	2,218	0.0387	83.1%	6,870	0.0823
Quantile LSTM	1,698	2,344	0.0351	87.6%	5,940	0.0714
Deep Ensemble (5 $\times$ )	1,589	2,176	0.0312	89.2%	5,480	0.0643
CQR (PatchTST)	1,634	2,257	0.0328	90.1%	6,120	0.0671
BT (Ours)	1,573	2,143	0.0289	90.4%	4,960	0.0598

Table 4: Overall performance on PJM dataset,

H=24

h. BT = Bayesian Transformer (proposed).

Dataset	Det. LSTM	Std. Transformer	Quant. LSTM	Deep Ensemble	CQR	BT (Ours)
PJM-CRPS	0.0412	0.0387	0.0351	0.0312	0.0328	0.0289
PJM-PICP 90%	81.3%	83.1%	87.6%	89.2%	90.1%	90.4%
ERCOT-CRPS	0.0448	0.0421	0.0382	0.0334	0.0357	0.0311
ERCOT-PICP 90%	79.8%	81.4%	86.1%	88.4%	90.0%	90.2%
ENTSO-E DE-CRPS	0.0391	0.0363	0.0328	0.0297	0.0311	0.0274
ENTSO-E DE-PICP 90%	82.1%	83.8%	88.2%	89.6%	90.3%	90.5%

Table 5: Cross-dataset CRPS and PICP (90%) for all models,

H=24

h. Best values in bold.

Figure 8: Representative 72-hour probabilistic forecast during a heatwave event: Bayesian Transformer multi-quantile prediction bands versus deterministic LSTM point forecast (ERCOT)

Model	HW MAE (MW)	HW PICP 90%	CS MAE (MW)	CS PICP 90%
Det. LSTM	3,241	64.7%	3,089	67.2%
Standard Transformer	2,987	66.8%	2,874	68.9%
Quantile LSTM	2,634	76.3%	2,512	78.1%
Deep Ensemble (5 $\times$ )	2,318	85.1%	2,201	86.4%
CQR (PatchTST)	2,541	88.7%	2,389	89.1%
BT (Ours)	2,104	89.6%	1,987	90.1%

Table 6: Extreme weather robustness. HW: Heatwave, CS = Cold Snap. Results averaged across all five datasets.

Figure 9: Extreme weather robustness: PICP at 90% nominal level under normal, heatwave, and cold snap conditions

Nominal Level	Quantile LSTM	Deep Ensemble	CQR	BT (Ours)	Target
5th percentile	3.1%	4.2%	5.1%	5.0%	5.0%
10th percentile	7.4%	8.9%	10.0%	9.9%	10.0%
25th percentile	20.1%	23.4%	24.8%	24.9%	25.0%
50th percentile	47.8%	49.1%	50.0%	50.0%	50.0%
75th percentile	70.3%	73.2%	74.9%	75.1%	75.0%
90th percentile	85.9%	88.6%	90.1%	90.4%	90.0%

Table 7: Empirical coverage at all seven nominal quantile levels on PJM test set (

H=24

h).

Figure 10: Reliability diagram: empirical coverage probability versus nominal quantile level on the PJM test set (H = 24 h)

Figure 11: Forecast horizon sensitivity: CRPS and PICP degradation across 24 h, 48 h, and 168 h horizons (PJM dataset)

Configuration	MAE (MW)	RMSE (MW)	CRPS	PICP 90%	MPIW (MW)
(A) Base PatchTST (deterministic)	1,621	2,218	0.0387	83.1%	—
(B) A + MC Dropout	1,598	2,189	0.0334	87.4%	5,820
(C) B + Variational FF layers	1,584	2,162	0.0311	88.9%	5,490
(D) C + Stochastic Attention	1,577	2,153	0.0296	89.8%	5,180
(E) D + Quantile head (pinball)	1,574	2,148	0.0293	89.9%	5,040
(F) E + Post-training Calibration (Full)	1,573	2,143	0.0289	90.4%	4,960

Table 8: Ablation study results on PJM test set,

H=24

h. Each row adds one component cumulatively.

Figure 12: Ablation study: incremental contribution of each Bayesian component to CRPS and PICP (PJM, H = 24 h)

Model	Params (M)	Train Time / Epoch	Inference Latency	MC Passes
Det. LSTM	18.4	1.2 min	0.018 s	1
Standard Transformer	42.1	2.1 min	0.031 s	1
Quantile LSTM	18.4	1.4 min	0.019 s	1
Deep Ensemble (5 $\times$ )	210.5	10.5 min	0.155 s	5
CQR (PatchTST)	42.1	2.1 min	0.033 s	1
BT (Ours)	44.3	2.8 min	1.84 s	100

Table 9: Computational cost comparison. Training time measured per epoch on NVIDIA A100 GPU.

Figure 13: Predictive uncertainty decomposition: aleatoric and epistemic contributions over a 48-hour forecast window spanning heatwave onset, and epistemic uncertainty scaling with temperature deviation from the training distribution

Explore similar work

Jul 17, 2026

A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods

Accurate load forecasting at multiple grid levels is essential for future smart grids, ranging from aggregated control area forecasts for balancing supply and demand to forecasts of individual end-consumer loads for demand-side management and energy management systems. We present a comprehensive benchmark for load forecasting across grid levels, comprising three datasets that represent a transmission system operator control area, low-voltage grid feeders, and individual end consumers. We evaluate ten methods for short-term load forecasting and find that Transformer-based approaches consistently outperform established methods, reducing forecast error by 6.6-10.7 %. To analyze the impact of architectural design, we introduce YAformer, a flexible Transformer architecture that integrates modifications from prior work and is optimized via hyperparameter optimization. However, the standard Transformer achieves superior performance, suggesting that these architectural modifications are not required for accurate load forecasting. We further evaluate the Transformer-based time-series foundation model Chronos-2, which demonstrates competitive zero-shot performance on two datasets but fails to accurately capture special events in the TSO data. Detailed analyses reveal model-specific strengths and weaknesses, and ablation studies highlight the importance of long input contexts, covariates and continuous retraining - aspects that are often overlooked in the time-series forecasting literature.

Matthias Hertel, Sebastian Pütz, Jonathan Kolar +3

Karlsruhe Institute of Technology, Germany · Helmholtz AI, Germany

Time Series Forecasting Forecasting Benchmarks Electricity Load Forecasting Time Series Transformers Time-Series Foundation Models Power Systems

Ask PDF

Oct 6, 2026

From Shared Demand Patterns to Local Uncertainty: Probabilistic Load Forecasting by Mixing Compact Adaptations

Probabilistic load forecasting has been widely studied for power-system operation and planning, but customer- and transformer-level forecasting introduces a distinct scalability challenge. At these levels, load uncertainty is strongly affected by customer behavior, weather, and mixed load composition, making it difficult for a single shared model to capture heterogeneous patterns. Using separate probabilistic models can improve local accuracy, but becomes costly to train, store, update, and validate at scale. To address this challenge, we develop a scalable customer-aware forecasting framework that learns common demand behavior through a shared model while adapting only a compact subset of parameters. Rather than using an independent model for each load or assigning each load to a specialized model, the proposed design learns a small bank of low-dimensional adaptation components and allows each load to combine them according to its forecasting characteristics. This preserves shared knowledge across customers while providing sufficient flexibility for heterogeneous and mixed load compositions. Experiments on 590 load profiles from the SMART-DS dataset show consistent improvements in deterministic accuracy and probabilistic quality over statistical, neural-network, Transformer-based, and pretrained time-series baselines, while retaining low storage and inference costs.

Haoran Li, Zhe Cheng, Yang Weng

Electricity Load Forecasting Probabilistic Time Series Forecasting Parameter-Efficient Fine-Tuning

Ask PDF

Jul 14, 2026

Learning-based Probabilistic Load Forecasting with Post-hoc and In-model Uncertainty

Smart-building load forecasters are often trained offline on dense, multivariate, high-frequency data, but deployment may provide only hourly, feature-limited inputs. Missing features must then be reconstructed, and their errors can propagate through the model. If this input uncertainty is not reflected, prediction intervals may become miscalibrated, affecting demand-response scheduling. Our work examines where uncertainty should be placed once inference inputs are reconstructed. We develop a unified one-day-ahead probabilistic forecasting framework that aligns temporal resolution, reconstructs the unavailable inputs, and derives causal features, and we compare a modular post-hoc residual-quantile scheme with an integrated in-model quantile-learning scheme. The comparison uses three mid-scale Deep Learning (DL) backbones: recurrent, hybrid recurrent, and attention-based Temporal Fusion Transformer (TFT) models, under identical inputs, forecasting horizon, preprocessing rules, and training budgets. Results show that uncertainty placement is backbone-dependent. Integrated quantile learning is most reliable with the TFT, yielding 2.2-3.6% MAPE and 28-83W RMSE on the labeled test window, while producing intervals about 5x narrower than the modular intervals at the closest-to-nominal coverage level. Diebold-Mariano tests support the TFT ranking and the mixed behavior of the recurrent backbones. A reconstruction-sensitivity test shows that reconstructed inputs increase the Quantile Score (QS) by 106% while interval width remains nearly unchanged, indicating that the model does not automatically absorb reconstruction-induced uncertainty. Robustness checks against non-DL baselines and seasonal hold-out weeks support this ranking. Our results expose the limits of post-hoc residual quantiles when inference depends on reconstructed inputs.

Sarah Al-Shareeda, Gulcihan Ozdemir, Heung Seok Jeon

aComputer System and Technology Department, Faculty of Computer Science and Information Technology, University of Malaya, Malaysia · bInformatics Institute, Istanbul Technical University, Turkey · cComputer Engineering Department Konkuk University South Korea

Quantile Regression Time Series Forecasting Probabilistic Forecasting Electricity Load Forecasting Uncertainty Calibration Probabilistic Time Series Forecasting

Ask PDF