Extreme Low-State Quantized Transformer for Electricity-Load Forecasting
Authors: Muhammad Sabbir Alam, Harindra Sandun Mavikumbure, Victor Cobilean, Milos Manic, Jayasimha Atulasimha
Organizations: Department of Mechanical and Nuclear Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USA · Department of Computer Science, Virginia Commonwealth University, Richmond, VA, 23284, USA · Department of Electrical and Computer Engineering, Virginia Commonwealth University, Richmond, VA, 23284, USA
Aggressive low-state weight quantization can substantially reduce neural-network representational flexibility, making predictive performance difficult to preserve when only a few discrete weight states are available. We investigate this regime for Transformer-based electricity-load forecasting using 3-, 5-, 7-, and 16-state weight quantization across feed-forward-only and attention-plus-feed-forward quantization. Two tasks predict either the next hour or the next 24 hours from the preceding 48 hours across two co-primary electricity-consumption datasets. Direct post-training quantization (PTQ) increased mean normalized root-mean-square error (NRMSE) relative to full precision by 6.25 and 7.04 percentage points in Datasets A and B, whereas quantization-aware fine-tuning (QAT-FT) recovered approximately 90.2% and 94.5% of these penalties. Quantization-aware training from random initialization (Scratch-QAT) also produced competitive low-state solutions, with particularly favorable behavior at three states. We further introduce accumulated-update thresholding, which accumulates optimizer-proposed weight changes before commitment. Relative to paired Scratch-QAT controls, this reduced committed shadow-weight updates by approximately 99.7% and discrete state transitions by approximately 77--80%, while mean NRMSE changed by only +0.042 and +0.197 percentage points. These results show that useful Transformer forecasting can be retained under highly discrete weight representations while strongly suppressing target-state activity, supporting a prospective route toward compact, low-activity edge forecasting using multilevel synaptic devices.
Figures & tables
Dataset A
Dataset B
Source
UCI ElectricityLoadDiagrams20112014
UCI Individual Household Electric Power Consumption
Role
Co-primary
Co-primary dataset
Series
MT_004, MT_022, MT_324
Household_Power
Distinct series
3
1
Modeling resolution
Hourly mean from 15-minute measurements
Hourly
Study period
2012-01-01 through 2014-12-31
2007-01-01 through 2009-12-31
Table 1: Co-primary datasets and forecasting protocol.
Dataset
Stage
Training period
Validation / test period
Dataset A
Fold 1
Jan 2012–Dec 2012
Jan 2013–Jun 2013
Fold 2
Jan 2012–Jun 2013
Jul 2013–Dec 2013
Fold 3
Jan 2012–Dec 2013
Jan 2014–Jun 2014
Final
Jan 2012–30 Jun 2014
1 Jul–31 Dec 2014, final test
Dataset B
Fold 1
Jan 2007–Dec 2007
Jan 2008–Jun 2008
Fold 2
Jan 2007–Jun 2008
Jul 2008–Dec 2008
Table 2: Chronological development and final evaluation protocol
ki(0)←index[QL(wi(0))]
Algorithm 1 Accumulated-update thresholding for Scratch-QAT
Metric
Dataset A
Dataset B
FP NRMSE (%)
11.3299
57.7067
PTQ NRMSE (%)
17.5770
64.7475
PTQ penalty vs. FP (pp)
+6.2471
+7.0408
QAT-FT NRMSE (%)
11.9424
58.0938
PTQ penalty recovered by QAT-FT (%)
90.1957
94.5019
Scratch α=0 E40 NRMSE (%)
11.8001
57.6379
Table 3: Headline forecasting and accumulated-update results across the two co-primary datasets.
Dataset
Series
PTQ − FP (pp)
QAT-FT − PTQ (pp)
Commit reduction (%)
Transition reduction (%)
Dataset A
MT_004
+7.4708
-7.0838
99.7391
80.5192
Dataset A
MT_022
+7.2828
-5.9642
99.6379
76.6080
Dataset A
MT_324
+3.9878
-3.8559
99.8044
82.7968
Dataset B
Household_Power
+7.0408
-6.6537
99.6741
77.0203
Table 4: Series-level mean effects across the four load series.
Factor
Levels
PTQ degradation
QAT-FT improvement over PTQ
Commit reduction
Transition reduction
State count
4
4/4
4/4
4/4
4/4
Forecast horizon
2
2/2
2/2
2/2
2/2
Quantization location
2
2/2
2/2
2/2
2/2
Table 5: Cross-dataset directional concordance across the principal experimental factors.
Primary effect
Expected direction
Four-series mean
LOO mean range
All four LOO analyses retain direction
PTQ degradation vs FP (pp)
Positive
+6.4455
+6.1038 to +7.2648
Yes
QAT-FT improvement vs PTQ (pp)
Negative
-5.8894
-6.5672 to -5.4913
Yes
High- α commit reduction (%)
Positive
99.7139
99.6837 to 99.7392
Yes
High- α transition reduction (%)
Positive
79.2361
78.0492 to 80.1121
Yes
Table 6: Leave-one-series-out sensitivity of the four primary effects.
Accurate load forecasting at multiple grid levels is essential for future smart grids, ranging from aggregated control area forecasts for balancing supply and demand to forecasts of individual end-consumer loads for demand-side management and energy management systems. We present a comprehensive benchmark for load forecasting across grid levels, comprising three datasets that represent a transmission system operator control area, low-voltage grid feeders, and individual end consumers. We evaluate ten methods for short-term load forecasting and find that Transformer-based approaches consistently outperform established methods, reducing forecast error by 6.6-10.7 %. To analyze the impact of architectural design, we introduce YAformer, a flexible Transformer architecture that integrates modifications from prior work and is optimized via hyperparameter optimization. However, the standard Transformer achieves superior performance, suggesting that these architectural modifications are not required for accurate load forecasting. We further evaluate the Transformer-based time-series foundation model Chronos-2, which demonstrates competitive zero-shot performance on two datasets but fails to accurately capture special events in the TSO data. Detailed analyses reveal model-specific strengths and weaknesses, and ablation studies highlight the importance of long input contexts, covariates and continuous retraining - aspects that are often overlooked in the time-series forecasting literature.
Matthias Hertel, Sebastian Pütz, Jonathan Kolar +3
Karlsruhe Institute of Technology, Germany · Helmholtz AI, Germany
Large-scale transformer training and deployment are increasingly constrained by the transfer of activations, gradients, and optimizer states across accelerators. Low-bit quantization offers a natural remedy, but transformer activations are often heavy-tailed and outlier-dominated, making simple quantization highly lossy. We show that this difficulty is not only a property of the quantizer, but also of the architecture. Specifically, residual connections can drive transformer activations away from Gaussianity during training. Using controlled comparisons between residual and residual-free transformers, we demonstrate that this effect leads to substantially higher quantization error and accuracy degradation at low precision in residual models. We explain the phenomenon through an excess kurtosis analysis, showing that residual mixing can amplify non-Gaussianity, whereas dense mixing in residual-free contracts non-Gaussianity. We then show that residual-free transformers can be made trainable using orthogonal initialization, spectral or second-order optimization, and depth-aware scaling of attention temperature. In language tasks, while there is a small drop in full precision performance, these models retain near-Gaussian activations and exhibit significantly improved robustness to low-bit quantization. Our results identify an accuracy--compressibility trade-off in transformer design and motivate architecture-level approaches to quantization-friendly foundation models.
Yiping Ji, Mahalakshmi Sabanayagam, Peyman Moghadam +2
Australian Institute for Machine Learning, Adelaide University · DATA61, CSIRO
Short-term load forecasting is essential for reliable energy management, but practical deployment on edge devices requires models that remain accurate under limited memory, finite measurement budgets, and hardware noise. This work proposes a hardware-efficient Quantum Reservoir Computing (QRC) framework for energy load forecasting, where a fixed quantum reservoir transforms temporal input windows into high-dimensional features and only a classical Elastic Net readout is trained. To reduce deployment cost, the trained readout is compressed using post-training fixed-point quantization at bit widths from 8 to 2 bits. The framework is evaluated on the Tetouan and Spain energy load datasets under exact statevector simulation, 512-shot finite sampling, and realistic hardware-noise models from IBM FakeTorino and IBM FakeMarrakesh. Results show that 6-bit readout precision preserves full-precision forecasting performance while reducing readout memory by 81.2%. Below this point, degradation becomes dataset dependent, with Tetouan showing stronger sensitivity and Spain degrading more gradually. Hardware-noise validation further shows that the trained readout transfers to noisy reservoir states without retraining. These findings support quantized QRC as a resource-aware forecasting approach for near-term quantum time-series applications.
Mansi Od, Param Pathak, Nouhaila Innan +1
University of Greenwich, London, United Kingdom · Fractal Analytics, QuantumAI Lab, Mumbai, Maharashtra, India · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE +1