Reinforcement learning (RL) offers a promising approach for adaptive wind farm flow control, yet its practical deployment is hindered by slow training convergence and poor initial performance, factors that could translate to years of reduced power output if an untrained agent were deployed directly. This work investigates whether domain knowledge from steady-state wake models can accelerate RL training and improve initial controller performance. We propose a pretraining methodology in which expert demonstrations are generated by deploying a PyWake-based steady-state optimizer within a dynamic wake simulation (WindGym), then used to initialize both the actor and critic networks of a Soft Actor-Critic agent via behavior cloning. Experiments on a 2x2 wind farm show that pretraining eliminates the costly initial learning phase: while an untrained agent underperforms the greedy zero-yaw baseline by approximately 12%, pretraining raises initial performance to near-baseline levels. During online fine-tuning, all configurations converge within 250,000 environment steps to achieve similar performance, ultimately exceeding that of a lookup-table controller, which reaches approximately 7% power gain after 500,000 steps.
Figures & tables
Dataset name
None
Small
Medium
Large
Episodes in dataset
0
10
50
200
Table 1: Dataset name and size
Figure 1: Single-case evaluation at 12 m/s and 275∘ . Top: wind farm power gain relative to Greedy for SAC and PyWake. Bottom: yaw control actions per turbine; the PyWake yaw for the two upstream and two downstream turbines coincide. Right: snapshot of the flow field at t=3000 s with the applied yaw angles.
Figure 2: Comparison of SAC and PyWake controllers for θ=270∘ and U∞=10 m/s. Left: mean wind farm power gain (%) relative to the Greedy baseline; shaded regions indicate 1 standard deviation across turbulence boxes (and seeds for SAC). Right: flow snapshot for one turbulence realization. Green and background indicate higher mean power for SAC and PyWake, respectively.
Figure 3: Box plots of wind farm power gain (%) relative to Greedy across wind speeds (left) and wind directions (right) after 0, 2.5×105 , and 5×105 training steps. Pretraining most strongly improves the untrained policy and low-wind regimes; little gain is observed at edge directions where wake interactions are weak.
Figure 4: Heatmap of the mean percentage increase in power for all pre-training sizes and training steps. Note that the PyWake agent has a mean increase of 5.96% . This value is indicated by the black line in the colorbar on the right
Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.
Wind farm wake steering optimization is challenging due to complex flow physics and changing conditions. This paper presents a hierarchical framework that combines reinforcement learning with model predictive control, where an RL agent learns compensatory state estimates for an MPC controller, rather than directly controlling turbines. Evaluated on a three-turbine case, the approach achieves a 23% power gain over the baseline control and surpasses the idealized MPC with perfect state knowledge. Compared to direct RL control, the hybrid architecture maintains superior safety characteristics during training while achieving comparable performance with more stable control actions.
Marcus Binder Nilsen, Teodor Olof Benedict Åstrand, Tuhfe Göçmen +1
Technical University of Denmark, Department of Wind and Energy Systems. Roskilde, Frederiksborgvej 399, Denmark
This study presents a multi-agent reinforcement learning (MARL) framework for load-constrained wind farm flow control (WFFC). While wake steering can enhance total wind farm power, it often introduces increased structural loads on downstream turbines. To address this, we integrate an Independent Soft Actor-Critic (I-SAC) architecture with a data-driven, local inflow sector-averaged surrogate model to provide real-time estimates of Damage Equivalent Loads (DELs). By incorporating these estimates into a shaped reward function, turbine-specific agents are trained to maximize power production while adhering to specific load-increase thresholds (Δmax) of 10%, 20%, and 30% relative to a baseline controller. The framework is implemented within the WindGym environment using the DYNAMIKS flow solver with Dynamic Wake Meandering (DWM) model to capture non-stationary wake physics. Results indicate that the MARL agents successfully learn collaborative policies that prioritise power gain while actively retreating from high-DEL control strategies.
Teodor Åstrand, Marcus Binder Nilsen, Iasonas Tsaklis +3
Department of Wind and Energy Systems, Technical University of Denmark, Roskilde, Denmark
This paper addresses the wind farm power maximization problem in the presence of wind direction changes. Specifically, a model-free Modified Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning (MTD3-BC) algorithm is proposed to tackle this task through yaw control under varying wind direction conditions. MTD3-BC is an offline reinforcement learning (RL) algorithm that aims to infer good behavior from only a precollected offline dataset. Additionally, to ensure smooth and moderate yaw adjustments, a new action consistency term is introduced into the policy optimization objective. Unlike online RL methods, MTD3-BC does not require extensive interactions with a wind farm simulator during training, significantly reducing computational costs and training time. A wind tunnel experiment is conducted to validate the effectiveness of the algorithm under varying wind directions. The results demonstrate that MTD3-BC successfully mitigates wake effects, delivering farm-level power gains of approximately 10% over the baseline greedy strategy and performance on par with a data-calibrated model-based wake-steering benchmark, while requiring no wake model and only a small fraction of the training cost of online RL. To our knowledge, this is the first time an offline RL wind farm control policy has been validated and demonstrated experimentally.
Yuhan Su, Hongyang Dong, Simone Tamaro +3
Intelligent Control and Smart Energy (ICSE) Research Group, School of Engineering, University of Warwick, Coventry CV4 7AL, U.K. · Wind Energy Institute, Technical University of Munich, 85748 Garching bei München, Germany