Organizations: Graduate School of Bio-Applications and Systems Engineering, Tokyo University of Agriculture and Technology, Koganei, Tokyo, Japan · Institute of Science Tokyo, Yokohama 226-8501, Japan · Norwegian University of Science and Technology, 7491 Trondheim, Norway · School of Technology and Innovations, University of Vaasa, Wolffintie 32, 65200 Vaasa, Finland
In power setpoint tracking (PST), an operator follows a purchased charging profile without overloading shared transformers or undercharging vehicles. PST is cast as a constrained Markov decision process that minimizes tracking error subject to overload and mean user-satisfaction constraints. Transformer-Constrained Charging Policy Learning (TC-CPL) extends soft actor-critic with proportional-integral Lagrange-multiplier updates from measured violations, a residual on the incumbent dispatcher that is warm-started from its demonstrations, and a transformer-aware encoder shared by all ports. It observes no departure times or states of charge. Simulations (five seeds, 100 paired days) compare it with the incumbent, Round-Robin. On a 600-port, 122-substation district, raw overload falls from 3.59 to 0.0017 kWh/day (overloaded seed-days from 77% to 0.4%), and absolute tracking error falls by 27%. Delivered energy rises by 16%, and satisfaction rises from 0.959 to 0.989. On a 42-kW transformer with household load, overload falls from 1.86 to 0.0014 kWh/day and satisfaction rises from 0.974 to 0.985, but absolute tracking error is 22% higher. Against hand-set fixed multipliers, self-tuning is statistically indistinguishable on the district. On the single transformer, it raises satisfaction by 0.011 at similar overload, but a short sweep found a better-serving fixed pair. The overload multipliers stay below their cap.
Figures & tables
Reference
Algorithm
Real-time
Constraint handling
Dep.
SoC
Transf. limit
Warm start
V2G
Open-source
[ 23 ]
CPO
✓
full charge at departure
×
✓
×
n/r
✓
×
[ 24 ]
constrained SAC
✓
Lagrangian + safety filter
n/r
n/r
n/r
n/r
✓
×
[ 25 ]
SAC
✓
aug. Lagrangian, dual step per update
×
✓
×
n/r
✓
×
[ 26 ]
SAC
✓
bus-voltage Lagrangian
✓
✓
×
n/r
✓
×
[ 27 ]
multi-agent SAC
✓
overload penalty
n/r
n/r
✓ penalty
n/r
n/r
×
[ 2 ]
multi-agent TD3
✓
fixed-weight aging + user costs
✓
✓
✓ aging
curriculum
✓
×
TABLE I: RL charging methods vs. TC-CPL (DDPG-PST as run here). Real-time = online per interval; Dep., SoC = departure time, state of charge as input; ✓/ × = yes/no; n/r = not reported.
Fig. 1: Transformer-constrained PST. The observation box shows part of ot . “CPO” is the charge point operator, not the baseline.
Fig. 2: (a) Conventional SAC with fixed penalties. (b) TC-CPL adds PI-controlled multipliers (M1), incumbent demonstrations (M2, labeled “safe” but not overload-free) and grid-aware states (M3).
(a) Per scenario
Transformers
Daily demand
Day
Learner
Scenario
Ports N (stations)
∣W∣ (with ports)
Ports each
Limit outside DR, Pwnom (kW)
Lowest in DR (kW) b
Non-EV peak / limit
EV sessions per day
Setpoint / EV energy
Start
Days
Pref (kW)
Eref (kWh)
no
PublicPST-S
20 ( 20 )
1 ( 1 )
20
42
20.4 – 38.0
0.29 – 0.79
6 – 21
1.8
05:00
weekdays
3,221.7
75,321
2,268
cityNL
600 ( 300 )
122 ( 93 )
2 – 28
95 – 760 a
56.6 – 656.7
0.20 – 0.78
981 – 1,112
1.1
00:00
all days
3,325.5
79,813
2,292
TABLE II: Experimental settings (a) per scenario and (b) shared by both scenarios and TC-CPL variants in all main runs. Ranges span transformers and evaluation days.
Controller
Information
Action; network
Constraint handling
Warm start; replay
Budget; tuning
CAFAP
occupancy
aj,t=1 on all ports
none
–
rule
Round-Robin
Ptset , port status
⌈Ptset/Pˉport⌉ EVs in turn
none
M2 incumbent
rule
DDPG-PST
PST state, normalized
direct; MLP 128 - 128 / 64 - 64
none (tracking reward only)
uniform, 106
106×5 ; as [ 14 ]
PPO-PST
PST state, normalized
direct; MLP 128 - 128
none (tracking reward only)
on-policy
1.016×106×5
CPO
ot (no at0 ), normalized
direct; MLP 256 - 128 - 256
trust region; raw overload ≤0
on-policy
1.008×106×5
Self-tuning TC-CPL
ot=(ot,at0)
residual, ρ=1 ; M3, load-aware
ctov above Pw,ttr , ctdef at 0.98 ; PI law (M1) from (10,5)
105 Round-Robin steps (M2); PER
106×5 ; Table II
TABLE III: Compared controllers and evaluation protocol, the same in both scenarios.
Tracking
Service
Overload
Controller
∣ϵtr∣
ϵusr
Edel
Unmet
EVs sk<0.95
ϵov
Seed-days
Max excess
kWh/d
kWh/d
%
%
kWh/d
%
kW
CAFAP
5,727
0.997
11,919
0.14
1.9
3.9807
76.0
41.54
Round-Robin
3,411
0.959
9,891
3.94
23.9
3.5932
77.0
34.20
DDPG-PST
7,756 ± 240
0.823 ± 0.012
6,370 ± 402
17.61 ± 1.23
47.6 ± 3.1
1.3342 ± 0.3913
54.8 ± 8.6
23.92 ± 4.44
PPO-PST
7,502 ± 73
0.845 ± 0.002
5,846 ± 78
15.41 ± 0.22
58.5 ± 0.8
0.9819 ± 0.1133
59.0 ± 1.2
17.44 ± 3.93
TABLE IV: Evaluation on 100 shared days, as mean ± standard deviation over five per-seed means. Max excess is the worst interval per seed. Each Δ is Self-tuning TC-CPL minus the named controller, paired over the same days and seeds (percentage points for %), with its 95% bootstrap interval below ( ∗ if it excludes zero).
Fig. 3: Tracking error ∣ϵtr∣ against (a) raw overload, (b) mean satisfaction and (c) delivered energy on cityNL (top) and PublicPST-S (bottom). Squares, boxes and whiskers give the median, interquartile range (IQR) and 1.5 IQR of daily values, seed-averaged for learners. Hatched desired regions pair tracking no worse than Round-Robin’s median with (a) zero overload (drawn 0.5 kWh/day wide), (b) ϵusr≥0.95 or (c) Edel at least Round-Robin’s median. On cityNL , only the TC-CPL variants (coinciding) and “Optimal (offline)” have medians in all three regions. On PublicPST-S , Round-Robin lies on the boundary by construction (zero median overload, yet overload on 49% of days), and Self-tuning TC-CPL has near-zero overload and the highest satisfaction among limit-respecting online controllers but a median tracking error 19% above Round-Robin’s.
Fig. 4: Overload on cityNL . (a) Mean daily raw overload ( 95% bootstrap intervals) and share of overloaded seed-days (red). (b) Overload per MWh delivered, mean ±1 standard deviation over days (hence bars below zero). (c) Overload beyond a band of b kW, at most 0.004 kWh/day for both TC-CPL variants at every b . The 0.04 share of “Optimal (offline)” is solver residue.
Fig. 5: Setpoint Ptset (blue), site power Pttot (green) and excess over the DR-aware limits summed over substations (red) on a cityNL day outside the evaluation set (seed 27 ; seed- 42 models). Panel titles give the daily overload and delivered energy. Self-tuning TC-CPL tracks without overload and delivers 96.6% of the offline optimum’s energy, 65 kWh less than Fixed-price TC-CPL.
Fig. 6: Self-tuning TC-CPL on cityNL (left) and PublicPST-S (right). (a), (b) Lagrange multipliers. (c), (d) Training overload above the tightened limit and satisfaction per dual window. Seed means with min–max bands (Fixed-price TC-CPL dashed, without bands). Overload nears zero, and λov stays below the cap.
Fig. 7: cityNL evaluation return (daily sum of the tracking reward ( 7 ) in kW 2 , without constraint costs) on five simulator-drawn days every 5×104 steps, with deterministic policies (seed means, min–max bands). CPO logs no such return. The TC-CPL variants overlap and end about ten times closer to zero than DDPG-PST and PPO-PST.
Variant
n
∣ϵtr∣
ϵusr
Edel
ϵov
OV
λov
kWh/d
kWh/d
kWh/d
days %
final
Self-tuning TC-CPL
3
123.6
0.970
114.7
0.003
0.7
54
Round-Robin (incumbent)
–
100.0 ∗
0.974
122.7 ∗
1.858 ∗
49.0 ∗
–
M1 off, prices frozen at (λov,λdef)
(10,5)
2
116.9 ∗
0.978 ∗
119.9 ∗
0.011
4.0
10
(10,50)
2
116.5 ∗
0.974 ∗
118.3 ∗
1.051
27.0
10
TABLE V: Ablations, sensitivity and fixed prices on PublicPST-S .
(a) Deployment checks
Self-tuning TC-CPL
Fixed-price
Observation error
Δϵov
Seed- days
Max excess
Δϵusr
Δϵov
Seed- days
( 106 steps, five seeds)
kWh/d
%
kW
kWh/d
%
None (absolute values)
0.0014
0.6
0.48
0.985
0.0004
0.4
Status flipped, 10%
+0.111
13.4
8.46
+0.002
+0.106
12.8
Status hidden, 10%
+0.149
13.8
9.64
+0.002
+0.129
13.2
Port data 15 min late
+0.163
22.6
7.15
+0.002
+0.134
22.0
TABLE VI: Deployment checks on PublicPST-S and computational cost.
anasc Research, School of Computer Science & IT, University College Cork, IE · School of Computing, Engineering and Digital Technologies, Teesside University, UK · International Energy Research Centre, Tyndall National Institute, Cork, IE +1