Hierarchical RL-MPC Control for Dynamic Wake Steering in Wind Farms

Marcus Binder Nilsen; Teodor Olof Benedict Åstrand; Tuhfe Göçmen; Pierre-Elouan Réthoré

eess.SYApr 13, 2026

Hierarchical RL-MPC Control for Dynamic Wake Steering in Wind Farms

Authors: Marcus Binder Nilsen, Teodor Olof Benedict Åstrand, Tuhfe Göçmen, Pierre-Elouan Réthoré

Organizations: Technical University of Denmark, Department of Wind and Energy Systems. Roskilde, Frederiksborgvej 399, Denmark

Abstract

Wind farm wake steering optimization is challenging due to complex flow physics and changing conditions. This paper presents a hierarchical framework that combines reinforcement learning with model predictive control, where an RL agent learns compensatory state estimates for an MPC controller, rather than directly controlling turbines. Evaluated on a three-turbine case, the approach achieves a 23% power gain over the baseline control and surpasses the idealized MPC with perfect state knowledge. Compared to direct RL control, the hybrid architecture maintains superior safety characteristics during training while achieving comparable performance with more stable control actions.

Figures & tables

Figure 1: Visualization of the wind farm (environment) with a westerly inflow state

Panel of Figure 2 — Figure 2: Comparison of wind farm control architectures: (a) Pure RL directly controlling yaw angles, (b) Pure MPC relying on accurate state estimates, (c) Hierarchical RL-MPC combining adaptive state estimation with physics-based optimization.

Figure 3: Plot depicting the mean farm power produced by the SAC-MPC approach throughout the model training steps, compared to the Idealized MPC and greedy baseline controller.

Figure 4: Yaw offset control actions, $\gamma$ , for the RL and Idealized MPC controllers for a fixed wind direction of $270^{\circ}$ .

Figure 5: Flow field visualization of wake propagation under the final yaw configurations from Figure 4 . Compared to the Idealized MPC (top plot), the SAC-MPC (bottom plot) achieves stronger wake deflection.

Figure 6: Plots depicting the estimated wind direction (WD), wind speed (WS), and turbulence intensity (TI) from the RL agent and the corresponding true value from WindGym fed to the MPC during the control actions in Figure 4 .

Figure 7: Percentage of time with turbines having a yaw offset of above 30 degrees

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 274 papers

Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations
Apr 13, 2026Marcus Binder Nilsen, Julian Quick, Tuhfe Göçmen +2RL Control Offline-To-Online RL
Department of Wind and Energy Systems, Technical University of Denmark, Roskilde, Denmark
Noise Steering: Exploiting Wake Steering and Acoustic Directivity for Noise-Constrained Wind Farm Control via Reinforcement Learning
Oct 27, 2026Martín de Frutos, Baptiste Corban, Claire Bizon Monroc +4Optimal Control RL Control
ETSIAE-UPM - School of Aeronautics, Universidad Politécnica de Madrid, Plaza Cardenal Cisneros 3, Madrid, E-28040, Spain · Inria, 48 rue Barrault, Paris, 75013, France · DI ENS, Ecole Normale Supérieure, PSL Research University, Paris, France +3
Load constrained wind farm flow control through multi-objective multi-agent reinforcement learning
Apr 13, 2026Teodor Åstrand, Marcus Binder Nilsen, Iasonas Tsaklis +3Multi-Agent Reinforcement Learning Constrained RL
Department of Wind and Energy Systems, Technical University of Denmark, Roskilde, Denmark

Apr 13, 2026J/K move · Enter open · S save

Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations

Marcus Binder Nilsen, Julian Quick, Tuhfe Göçmen, Nikolay Dimitrov+1

Department of Wind and Energy Systems, Technical University of Denmark, Roskilde, Denmark

RL Control Offline-To-Online RL Power Systems

Open paper AskPDF arXiv

Reinforcement learning (RL) offers a promising approach for adaptive wind farm flow control, yet its practical deployment is hindered by slow training convergence and poor initial performance, factors that could translate to years of reduced power output if an untrained agent were deployed directly. This work investigates whether domain knowledge from steady-state wake models can accelerate RL training and improve initial controller performance. We propose a pretraining methodology in which expert demonstrations are generated by deploying a PyWake-based steady-state optimizer within a dynamic wake simulation (WindGym), then used to initialize both the actor and critic networks of a Soft Actor-Critic agent via behavior cloning. Experiments on a 2x2 wind farm show that pretraining eliminates the costly initial learning phase: while an untrained agent underperforms the greedy zero-yaw baseline by approximately 12%, pretraining raises initial performance to near-baseline levels. During online fine-tuning, all configurations converge within 250,000 environment steps to achieve similar performance, ultimately exceeding that of a lookup-table controller, which reaches approximately 7% power gain after 500,000 steps.

Abstract

RL Control

Explore similar work

Accelerating Reinforcement Learning for Wind Farm Control via Expert Demonstrations

Noise Steering: Exploiting Wake Steering and Acoustic Directivity for Noise-Constrained Wind Farm Control via Reinforcement Learning

Load constrained wind farm flow control through multi-objective multi-agent reinforcement learning