As the integration of volatile renewable energy sources increases the strain on modern power grids, the use of Reinforcement Learning (RL) for autonomous topological reconfiguration has emerged as a promising research field to keep strained grids stable and operational. Compared to traditional redispatching measures, topological actions offer a cheaper and more cost-effective way to manage grid congestion. However, their implementation is hindered by a vast combinatorial action space and strict operational constraints. This paper investigates the effectiveness of model-based AlphaZero-inspired approaches that utilize Monte Carlo Tree Search (MCTS) for proactive grid management. We systematically evaluate how reward functions, observation density, and search guidance influence an agent's survivability. Our results demonstrate that the optimized AlphaZero approach achieves a peak survivability of 98.43%, significantly outperforming the proximal policy optimization (PPO) variant. We find that conducting the MCTS without guidance from a prior learned policy or value function can enhance training efficiency, and that a straightforward binary survival reward provides more effective search guidance than complex, multi-objective functions. Our findings demonstrate that while AlphaZero is a powerful framework for topological control, pure reinforcement learning is not sufficient; rather, an effective and reliable system requires a 'minimalist' integration of domain-specific heuristics, binary rewards, and a restricted observation space of line loads.
Figures & tables
Year
Challenge Track
Approach / Agent Name
Key Methodology
2019
IJCNN (Sandbox)
DDQN-2019
Dueling DQN + Imitation Learning ( Lan et al., 2020 )
2020
WCCI (Robustness)
Semi-Markov Afterstate Actor-Critic (SMAAC)
SAC-based ( Yoon et al., 2021 )
2020
NeurIPS (Robustness)
Search with Action Set (SAS)
Evolutionary strategies (ES) ( Zhou et al., 2021 )
2021
ICAPS (Trust)
Different modes
Mix of SAS ( Marot et al., 2022 )
2022
WCCI (Adaptability)
AlphaZero
MCTS ( Dorfer et al., 2022 )
2023
L2RPN Delft (Remake of 2022)
PPO Rainbow-Based
PPO-Rainbow + Heuristics ( Sintes and Dang, 2024 ; Lehna et al., 2023 )
Table 1. Historical evolution of top-performing agents and methodologies in the Learning to Run a Power Network (L2RPN) challenges. The progression shows a shift from early model-free RL toward specialized architectures involving Curriculum Learning and model-based Tree Search.
Reduction Method
Total Actions (+ Do Nothing)
Complete
405
SYM
203
N-0
142
N-1
82
Table 2. Impact of Reduction Strategies on Action Space Size for IEEE-14
Configuration
Included Features
Minimal
Line loads ( ρ )
Custom
Line loads, line status, cooldowns, topology
Reduced
Line load, production, load, overflow, topology
Complete
All available Grid2op features including voltage and time
Table 3. Overview of Observation Space Configurations
Model
Avg. Steps survived
Std. dev.
DQN
89 (1.1%)
Do Nothing
847 (10.5%)
PPO Redispatch
1670 (20.7%)
PPO Topological
3250 (40.3%)
PPO Curriculum
4130 (51.22%)
PPO Rainbow
7403 (91.80%)
1372
Table 4. Peak performances of the different RL approaches compared to the Do Nothing agent. For the PPO Rainbow, AlphaZero Baseline and Alphzero D3QN-2022 we averaged the peaks over 4 runs. Additionally, we can also see the average standard deviation over the 101 training chronics for these.
Figure 1. Comparison of AlphaZero with the different PPO variants. The results for the AlphaZero baseline described in subsection B.2 , the AlphaZero with the D3QN-2022 reward function as well as the PPO rainbow variant are evaluated over 4 runs and plotted with their standard deviation using a sliding window for smoothing.
Figure 2. Training progression of the AlphaZero agent using the different reward functions presented in Appendix A.3 . The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation.
Figure 3. Training progression of the AlphaZero agent using the different action spaces presented in Section 3.2 . The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation.
Figure 4. Training progression of the AlphaZero agent using the different observation spaces presented in Section 3.2 . The average performance is evaluated over 4 runs and plotted with the standard deviation using interpolation.
Figure 5. Comparison of the three MCTS guidance methods of using learned prior and value function, learned prior and heuristic value function or no guidance at all.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Evaluation of different horizon sizes for the heuristic based AlphaZero baseline approach.
Figure 7. Evaluation of different thresholds for the application of the heuristic that leads the agent back to the reference topology.
Figure 8. Evaluation of values for the tstopping and tskipped hyperparameters which define the early stopping mechanism for the MCTS tree.
Figure 9. Evaluation of the effect caused by different choices for the batch size which determines the number of chronics we collect data from between training iterations.