Capacity-constrained demand response in smart grids using deep reinforcement learning

Shafagh Abband Pashaki; Sepehr Maleki; Amir Badiee

cs.LGFeb 18, 2026

Capacity-constrained demand response in smart grids using deep reinforcement learning

Authors: Shafagh Abband Pashaki, Sepehr Maleki, Amir Badiee

Organizations: Lincoln AI Lab, University of Lincoln, Lincoln, United Kingdom

Abstract

This paper presents a capacity-constrained incentive-based demand response approach for residential smart grids. It aims to maintain electricity grid capacity limits and prevent congestion by financially incentivising end users to reduce or shift their energy consumption. The proposed framework adopts a hierarchical architecture in which a service provider adjusts hourly incentive rates based on wholesale electricity prices and aggregated residential load. The financial interests of both the service provider and end users are explicitly considered. A deep reinforcement learning approach is employed to learn optimal real-time incentive rates under explicit capacity constraints. Heterogeneous user preferences are modelled through appliance-level home energy management systems and dissatisfaction costs. Using real-world residential electricity consumption and price data from three households, simulation results show that the proposed approach effectively reduces peak demand and smooths the aggregated load profile. This leads to an approximately 22.82% reduction in the peak-to-average ratio compared to the no-demand-response case.

Figures & tables

Figure 1: Hierarchical electricity market model.

Figure 2: Schematic of RL to find the optimal incentive rates.

Inputs: policy network $Q(s,a;\theta)$ , target network $Q(s,a;\theta^{-})$ , replay buffer $\mathcal{B}$ , weighting factor $\rho$ , DDQN hyperparameters $(\gamma,\varepsilon,\tau,\alpha)$ , incentive rate bounds $\lambda_{\min},\lambda_{\max}$ .
1:	for episode $e=1,\dots,N_{\mathrm{ep}}$ do
2:	% LSTM-based price forecasting
3:	for hour $h=1,\dots,H$ do
4:	DofWeek[ $h$ ] $\leftarrow$ updateDayOfWeek( $h$ ); HofDay[ $h$ ] $\leftarrow$ updateHourOfDay( $h$ ); MofYear[ $h$ ] $\leftarrow$ updateMonthOfYear( $h$ ); IsWeekend[ $h$ ] $\leftarrow$ updateWeekendFlag( $h$ ); IsHoliday[ $h$ ] $\leftarrow$ updateHolidayFlag( $h$ )
5:	PrSeq[ $h$ ] $\leftarrow$ updateHistoricalPriceSequence()
6:	FuPrice[ $h$ ] $\leftarrow$ PriceLSTM(DofWeek[ $h$ ], HofDay[ $h$ ], MofYear[ $h$ ], IsWeekend[ $h$ ], IsHoliday[ $h$ ], PrSeq[ $h$ ])

Table 1: Algorithm for real-time IBDR with LSTM and DDQN

Inputs for price forecasting		Inputs for load forecasting
Index	Description	Index	Description
1	Month of year (1–12)	1	Month of year (1–12)
2	Day of week (1–7)	2	Day of week (1–7)
3	Hour of day (1–24)	3	Hour of day (1–24)
4	Is holiday (0 or 1)	4	Is holiday (0 or 1)
5	Is weekend (0 or 1)	5	Is weekend (0 or 1)

Table 2: Inputs to the LSTM-based forecasting models

Hyper-parameter	Value
Number of LSTM layers	2
Hidden units	64
Dropout	0.2
Window size	24
Forecast horizon	1
Optimiser	Adam

Table 3: Hyper-parameters of the LSTM-based forecasting models

Figure 3: Wholesale price forecasting for July 23–29, 2018.

Panel of Figure 4 — Figure 4: Load forecasting of the three end users for July 23–29, 2018.

	MAE	MAPE
Wholesale price forecasting model	1.87	0.53
Load forecasting model of EU1	0.90	0.93
Load forecasting model of EU2	0.72	0.78
Load forecasting model of EU3	0.41	0.24

Table 4: Accuracy results of the forecasting model

Appliance	Type	Dissatisfaction coeff.
		mean	std
Dryer (DR)	TS, NI	0.10	0.10
Washing Machine (WM)	TS, NI	0.40	0.10
Dish Washer (DW)	TS, NI	0.20	0.10
Electric Vehicle (EV)	TS, I	0.05	0.10
Air Conditioner (AC)	PC	3.50	2.00

Table 5: Household appliances and their dissatisfaction parameters (TS: time-shiftable, PC: power-controllable, NI: non-interruptible, I: interruptible).

Figure 5: Training and validation episode rewards of the DDQN agent over $N_{\text{ep}}=2500$ episodes.

Panel of Figure 6 — Figure 6: Results of different end users for July 27, 2018.

Metric	No DR	CCRL-DR
Peak load (kW)	11.89	7.60
Mean load (kW)	6.46	5.34
PAR	1.84	1.42

Table 6: Results averaged per day in July

Metric	$\rho=0.1$			$\rho=0.3$			$\rho=0.5$			$\rho=0.7$			$\rho=0.9$
	EU1	EU2	EU3	EU1	EU2	EU3	EU1	EU2	EU3	EU1	EU2	EU3	EU1	EU2	EU3
Total energy reduction (kWh)	5.81	14.21	8.7	11.59	17.29	4.99	11.21	17.71	7.26	4.40	6.90	11.08	3.38	21.86	9.66
EU incentive income (¢)	0.47	0.77	0.72	5.64	4.3	1.37	8.59	7.77	2.15	5.64	4.82	9.49	2.54	30.58	9.14
EU discomfort cost (¢)	2.14	91.81	22.37	6.63	72.90	16.35	4.39	52.83	11.92	1.19	0.87	8.31	0.14	10.83	2.65
EU profit (¢)	-1.67	-91.04	-21.65	-0.99	-68.8	-14.97	4.20	-45.05	-9.76	4.45	3.95	1.18	2.40	19.74	6.48
Sum of EU profit (¢)	-114.36			-84.56			-50.61			9.58			28.62

Table 7: Financial analyses of the service provider and three end users

Figure 9: Sensitivity of the service provider profit and the aggregated end-users profit with respect to the weighting factor $\rho$ .

	Off-peak	Mid-peak	On-peak
	(1–6 am, 22–24 pm)	(7–16 pm)	(17–21 pm)
Elasticity $\xi_{h}$	0.5	0.3	0.1

Table 8: Elasticity in different hours

EU	$\mu_{n}$	$\omega_{n}$	$K_{\text{min}}$	$K_{\text{max}}$	$\lambda_{\text{min}}$	$\lambda_{\text{max}}$
EU1	0.3	0.1	0	$0.3E_{n,h}$	$0.3\,p_{\min}$	$p_{\min}$
EU2	0.6
EU3	0.9

Table 9: Three end users’ related parameters

Panel of Figure 10 — Figure 10: Comparison of energy profiles for three residential end users under the CCRL-DR (top row) and EBLR (bottom row) approaches on July 27, 2018.

Panel of Figure 11 — Figure 11: Comparison of aggregated load curves for July 27, 2018 under the CCRL-DR (left) and EBLR (right) methods. The figure illustrates differences in peak load behaviour between the two approaches.

Figure 12: Comparison of average peak load, mean load, and PAR for No DR, EBLR, and CCRL-DR approaches on July 27, 2018.

Demand Response

Demand response shifts or reduces electricity use to help balance the grid while accounting for prices, incentives, comfort, and service needs.

See all 23 papers

Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs
May 12, 2026Jose E. Aguilar Escamilla, Lingdong Zhou, Xiangqi Zhu +1RL Benchmarks Reinforcement Learning
Department of Electrical Engineering and Computer Science, Oregon State University, USA
Learning to Price Electricity for Optimal Demand Response
Sep 30, 2026Jing Shang, Mohammad Mehrabi, Xinyang Zhou +3Demand Response Dynamic Pricing
Stanford University · National Laboratory of the Rockies · Prime Coalition +1
Addressing Terminal Constraints in Data-Driven Demand Response Scheduling
May 14, 2026Maximilian Bloor, Martha White, Ehecatl Antonio del Rio Chanona +1Hierarchical RL Credit Assignment in RL
Sargent Centre for Process Systems Engineering, Imperial College London, London, SW7 2AZ, UK · Department of Computer Science, University of Alberta, Edmonton, AB, Canada · Department of Computing, Imperial College London, London, SW7 2AZ, UK

May 12, 2026J/K move · Enter open · S save

Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs

Jose E. Aguilar Escamilla, Lingdong Zhou, Xiangqi Zhu, Huazheng Wang

Department of Electrical Engineering and Computer Science, Oregon State University, USA

RL Benchmarks Reinforcement Learning Demand Response Power Systems

Open paper AskPDF arXiv

Extreme weather and volatile wholesale electricity markets expose residential consumers to catastrophic financial risks, yet demand response at the distribution level remains an underutilized tool for grid flexibility and energy affordability. While a demand-response program can shield consumers by issuing financial credits during high-price periods, optimizing this sequential decision-making process presents a unique challenge for reinforcement learning despite the plentiful offline historical smart meter and wholesale pricing data available publicly. Offline historical data fails to capture the dynamic, interactive feedback loop between an electric utility's pricing signals and customer acceptance and adaptation to a demand-response program. To address this, we introduce DR-Gym, an open-source, online Gymnasium-compatible environment designed to train and evaluate demand-response from the electric utility's perspective. Unlike existing device-level energy simulators, our environment focuses on the market-level electric utility setting and provides a rich observational space relevant to the electric utility. The simulator additionally features a regime-switching wholesale price model calibrated to real-world extreme events, alongside physics-based building demand profiles. For our learning signal, we use a configurable, multi-objective reward function for specifying diverse learning objectives. We demonstrate through baseline strategies and data snapshots the capability of our simulator to create realistic and learnable environments.

Abstract

Demand Response

Explore similar work

Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs

Learning to Price Electricity for Optimal Demand Response

Addressing Terminal Constraints in Data-Driven Demand Response Scheduling