Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model

Zesheng Yao; Zhen-Hua Wan; Canjun Yang; Qingchao Xia; Mengqi Zhang

cs.LGApr 5, 2026

Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model

Authors: Zesheng Yao, Zhen-Hua Wan, Canjun Yang, Qingchao Xia, Mengqi Zhang

Organizations: School of Mechanical Engineering, Zhejiang University, Hangzhou, PR China · Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, 117575 Singapore · Department of Modern Mechanics, University of Science and Technology of China, Hefei 230027, PR China

Abstract

Model-free deep reinforcement learning (DRL) methods suffer from poor sample efficiency. To overcome this limitation, this work introduces an adaptive reduced-order-model (ROM)-based reinforcement learning framework for active flow control. In contrast to conventional actor--critic architectures, the proposed approach leverages a ROM to estimate the gradient information required for controller optimization. The design of the ROM structure incorporates physical insights. The ROM integrates a linear dynamical system and a neural ordinary differential equation (NODE) for estimating the nonlinearity in the flow. The parameters of the linear component are identified via operator inference, while the NODE is trained in a data-driven manner using gradient-based optimization. During controller--environment interactions, the ROM is continuously updated with newly collected data, enabling adaptive refinement of the model. The controller is then optimized through differentiable simulation of the ROM. The proposed ROM-based DRL framework is validated on two canonical flow control problems: Blasius boundary layer flow and flow past a square cylinder. For the Blasius boundary layer, the proposed method effectively reduces to a single-episode system identification and controller optimization process, yet it yields controllers that outperform traditional linear designs and achieve performance comparable to DRL approaches with minimal data. For the flow past a square cylinder, the proposed method achieves superior drag reduction with significantly fewer exploration data compared with DRL approaches. The work addresses a key component of model-free DRL control algorithms and lays the foundation for designing more sample-efficient DRL-based active flow controllers.

Figures & tables

Figure 1: The framework of (a) the classical model-free off-policy reinforcement learning and (b) reduced-order model-based reinforcement learning proposed in the current work. The main differences between the two frameworks are highlighted in yellow.

Figure 2: Comparison of model complexity and training data requirements among active flow-control methods reported in the literature.

Figure 3: Schematic diagram of the proposed framework.

Figure 5: Frequency-response and linear stability analysis of Blasius boundary layer : (a) The transfer function from the noise source to sensors $y$ and $z$ ; (b) Neutral stability curve, and the spatial region between sensors $y$ and $z$ is indicated by the shaded area.

Figure 6: POD decomposition of the TS wave with a nondimensional frequency of $4.5\times 10^{-5}$ : (a) the first two modes obtained from POD of the uncontrolled flow; (b) the first four control-induced POD modes.

Figure 7: Accuracy of the ROM during two episodes of training: (a) normalized cost function $J_{1}/J_{\mathrm{ref}}$ and (b) cost function $J_{2}$ , both predicted by the ROM, as functions of the proportional gain $K_{P}$ , where $J_{\mathrm{ref}}$ denotes the value of $J_{1}$ for the uncontrolled case. (c) Time evolution of the sensor signal $u_{z}$ at $K_{P}=-400$ , where the controller becomes unstable; results from the ROM and full-order CFD simulation are compared. (d) Dependence of the normalized sensor amplitude $A/A_{\mathrm{ref}}$ on $K_{P}$ obtained from the ERA, OpInf, and full-order models, where $A_{\mathrm{ref}}$ denotes the amplitude in the uncontrolled case.

Controller type	ERA + systune	Current method
Proportional controller	0.4616	0.3578
First-order controller	Unstable	0.2560
Second-order controller	Unstable	0.1956

Table 1: Normalized

\mathcal{H}_{2}

norms (

\mathcal{H}_{2}/\mathcal{H}_{2,0}

) of the closed-loop transfer function

G_{zw}

for controllers designed using the ERA-based method and the proposed framework, where,

\mathcal{H}_{2,0}

denotes the

\mathcal{H}_{2}

norm of

G_{zw}

without control.

Figure 8: Bode magnitude plots of the transfer function $G_{zw}$ under different controllers: (a) comparison among the uncontrolled case, the ERA-based proportional controller, and the proportional controller designed by the current method; (b) comparison of the proportional, first-order, and second-order controllers designed by the current method.

Figure 9: Disturbance-attenuation performance of the designed second-order controller: (a) time-averaged perturbation-energy field without control; (b) time-averaged perturbation-energy field with the designed second-order controller; (c) comparison of r.m.s. perturbation velocity between the present controller and the DRL-based controller reported in Xu and Zhang (2023) , where the r.m.s. streamwise velocity at $y=1$ is plotted as a function of $x$ .

Figure 10: Computation domain of the flow past a square cylinder: the blue dots represent the sparsely distributed velocity sensors used for construct ROMs, the red dots represent the velocity sensors used for closed-loop control, and the white dots represent performance sensors for constructing the cost function for controller optimization.

Figure 11: Proportional controller design based on the ROM. (a) ROM-predicted suppression of vortex shedding for proportional gains between 70 and 80. The plot shows the streamwise velocity measured at the probe located at $(2.5,0.75)$ , with closed-loop control activated after $t=30$ . (b) ROM prediction for a proportional gain of 85, indicating that the controller becomes unstable. (c) Time histories of drag from CFD simulations with proportional gains of 80 and 85. The controller with $K=80$ effectively reduces drag, whereas the controller with $K=85$ diverges. The inset contour plots illustrate the divergence process.

Figure 12: The influence of NODE correction to the accuracy of the ROM.To assess the accuracy of the reduced-order model (ROM), two probe points are placed in the flow field at coordinates $(2.5,\,0.75)$ and $(2.5,\,-0.75)$ . The time histories of the ROM predictions at these locations are then compared against those obtained from the full-order model (FOM).

Figure 13: Mean drag coefficient of the square cylinder in each optimization episode under different training strategies. In the mixed OL–CL mode, the first two episodes are trained in OL mode, followed by CL mode in the subsequent episodes.

Figure 14: Evolution of ROM predictive accuracy and controller loss during mixed open-loop/closed-loop training. The ROM update panel shows the reduction of prediction error on newly added cases within each epoch, while the policy update panel illustrates the evolution of the controller’s loss function value across epochs during gradient-based optimization on the ROM.

Figure 15: The controller obtained at episode 3 under the mixed OL and CL training mode: (a) the temporal evolution of the drag; (b) visualization of the learned nonlinear controller $u=\pi_{\theta^{*}}(u_{1}-u_{2},u_{3}-u_{4})$ with closed-loop data trajectory.

Reference	Geometry	Number of sensors	Controller design method	Episodes	Total physical time	Drag reduction
Lasagna et al. (2016)	Circular	Full-field data	POD–Galerkin projection	1	$150T_{s}$	3.6%
Rabault et al. (2019)	Circular	151	DRL	150	$975T_{s}$
Paris et al. (2021)	Circular	15	DRL	100	$2180T_{s}$	14.9%
Li and Deng (2024)	Circular	42	Online DMD + LQR	–	–	7.4%
Xia et al. (2024)	Square	64	DRL	300	$8760T_{s}$	8.6%
Current method	Square	4	Adaptive ROM-based RL	4	$43T_{s}$	7.2%

Table 2: Comparison of active flow control strategies for drag reduction in the wake of the cylinder at Re = 100. The total physical time is defined as episodes number

\times

episode length, and the

T_{s}

denotes the vortex shedding period.

Dataset Index	1	2	3	4	5	6
Description	No Control	Proportional	Episode 1	Episode 2	Episode 3	Episode 4
Normalized Loss	1.22	1.35	0.92	1.01	1.02	0.98

Table 3: Normalized loss of the linear dynamical system identified using the full aggregate dataset across different training stages.

Figure 16: POD decomposition of the square cylinder wake flow: (a) the first four modes obtained from POD of the uncontrolled flow; (b) the first eight control-induced POD modes; (c) temporal evolution of the POD modal coefficients in the proportional-control $u=80(u_{1}-u_{2})$ case.

Figure 17: Mean drag coefficient of the square cylinder in each optimization episode under different training strategies. In the mixed OL–CL mode, the first six episodes 0-5 are trained in OL mode, followed by CL mode in the subsequent episodes. The yellow dashed line represents the drag-reduction performance of the optimal controller obtained from SS-ROM optimization.

Figure 18: Training evolution of normalized cumulative reward per episode with two reward definitions: (a) original reward $r=-(z_{1}^{2}+z_{2}^{2})$ , (b) modified reward $r=-(u_{1}^{2}+u_{2}^{2}+u_{3}^{2}+u_{4}^{2})$ ; (c) drag evolution corresponding to the best SAC policy (episode 79) and the controller optimized by the proposed adaptive ROM-based RL method. The cumulative reward $\tilde{R}$ is normalized by the absolute value of the cumulative reward $|R_{\text{ref}}|$ obtained in a reference no-control episode, i.e., $\tilde{R}=\frac{R}{|R_{\text{ref}}|},$ and the blue-shaded region ( $\tilde{R}<-1$ ) indicates cases where the control performance is worse than the no-control case.

Figure 19: Comparison of impulse responses calculated through DNS and ERA-ROM in both inputs to both outputs: (a) From noise to sensors y and z; (b) From actuator to sensors y and z.

Figure 20: Evolution of the normalized loss $J/J_{0}$ (where $J_{0}$ is the initial loss) over epochs during the co-optimization of sensor placement and controller parameters.

Figure 21: Comparison of the Bode magnitude plots of $G_{zw}$ for the initial and optimized sensor placements, corresponding to the (a) proportional, (b) first-order, and (c) second-order controllers.

Controller type	Initial sensor location	Optimized sensor location	Sensor displacement $(\delta x,\delta y)$
Proportional control	0.3578	0.3477	(0.4011, -0.0305)
First-order controller	0.2560	0.1762	(0.2426, 0.1271)
Second-order controller	0.1956	0.1821	(-0.0509, -0.0078)

Table 4: Normalized

\mathcal{H}_{2}

norms (

\mathcal{H}_{2}/\mathcal{H}_{2,0}

) of the closed-loop transfer function

G_{zw}

for controllers designed using the initial and optimized sensor locations, where,

\mathcal{H}_{2,0}

denotes the

\mathcal{H}_{2}

norm of

G_{zw}

without control. The displacement vector (

\delta x

\delta y

) represents the coordinate differences between the optimized and initial sensor positions.

Hyperparameter	MLP (NODE)	LSTM	Transformer
Hidden Layers / Blocks	3	2 (LSTM Cells)	2 (Encoder Layers)
Hidden Sizes	128	128	N/A
Model Dimension $d_{model}$	N/A	N/A	128
Feed-Forward Dimension	N/A	N/A	256
Sequence Length ( $n$ )	N/A (Continuous)	10	10
Hidden Layer Activation	ReLU	Tanh	ReLU

Table 5: Hyperparameters for the architectural comparison study.

Figure 22: Evolution of the training loss with respect to epochs for different ROM architectures: (a) training in the OL mode; (b) training in the CL mode.

Figure 23: Evolution of the training loss with respect to epochs for different ROM architectures: (a) training in the OL mode; (b) training in the CL mode.

Figure 24: Evolution of the training loss with respect to epochs for different ROM architectures: (a) training in the OL mode; (b) training in the CL mode.

Figure 25: Mean drag coefficient of the square cylinder in each optimization episode under different training strategies using pressure-sensor-based closed-loop control. In the mixed OL–CL mode, the first six episodes are trained in OL mode, followed by CL mode in the subsequent episodes.

Figure 26: Comparison of the drag coefficient evolution over training episodes between the original and stabilized training processes: (a) CL training of the POD-ROM, (b) CL training of the SS-ROM.

Hyperparameter	TD3	SAC
Optimiser	Adam	Adam
Learning rate	$1\times 10^{-4}$	$3\times 10^{-4}$
Discount factor ( $\gamma$ )	0.99	0.99
Replay buffer size	$10^{6}$	$10^{6}$
Number of hidden layers (actor/critic)	2	2
Hidden units per layer	128	128

Table 6: Hyperparameter settings for TD3 and SAC algorithms.

Sample-Efficient RL

Sample-efficient reinforcement learning studies how agents learn effective policies from fewer environment interactions, demonstrations, or costly rollouts.

See all 249 papers

Oct 19, 2026

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

Deep reinforcement learning (DRL) is challenging long-held assumptions in active flow control. By learning directly from interaction with a flow, DRL agents can find actuation strategies for nonlinear, unsteady flows without explicit models, often exploiting mechanisms that classical control would not anticipate. High-speed flows are a demanding test of this promise: shocks, shock/boundary-layer interactions, transition, aerothermal loading and microsecond-scale dynamics strain both algorithms and hardware. Yet most demonstrations remain idealized, low-speed simulations with limited evidence of robustness, interpretability, transferability or experimental deployment. In this perspective, we examine what constrains DRL-based high-speed flow control, including compressible simulation cost and fidelity, data efficiency, sensing in extreme environments, reward design, explainability, and generalization across Mach and Reynolds number. We argue that these limits reflect a tension between data-driven optimization and the structured physics, constraints and uncertainties of real flows. We then highlight where DRL may go beyond conventional approaches: adaptive control of shock-induced separation and turbulent boundary layers, coordination of distributed actuators, and discovery of unconventional flow-manipulation mechanisms.The central challenge, however, is translating these capabilities into reliable technology, which will require validated compressible simulation environments, physics- and constraint-aware learning, robustness to off-design conditions and sensor degradation, real-time operation, hardware-in-the-loop and wind-tunnel validation, and credible pathways to safety and certification.

Ameya D. Jagtap

Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA

RL Control Feedback Control Robust RL

Ask PDF

Jul 13, 2026

Heuristic Learning for Active Flow Control Using Coding Agents

Active flow control involves nonlinear dynamics, partial observations, and computationally expensive simulations, making controller design particularly challenging. Deep reinforcement learning (DRL) has emerged as a powerful framework for such problems, but its success typically relies on large numbers of simulator interactions and produces neural-network policies whose decision process often remains difficult to interpret. In this work, we investigate a different paradigm: instead of optimizing neural-network parameters, we use modern coding agents to search directly for explicit executable feedback laws. We introduce a constrained heuristic-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface. The proposed framework is evaluated on 13 active flow-control benchmarks spanning one, two, and three-dimensional problems and compared against the strongest available DRL baselines under identical simulation budgets. The discovered heuristic controllers match or outperform the best DRL policy in 10 of the 13 environments while remaining compact, interpretable, and directly inspectable. Beyond aggregate performance, the resulting controllers reveal physically meaningful feedback mechanisms, transfer successfully across more challenging configurations, and remain competitive under varying Reynolds and Rayleigh numbers, actuator counts, and observation sparsity. These results suggest that heuristic learning through coding agents constitutes a credible and complementary alternative to conventional reinforcement learning, combining competitive performance with physically interpretable controller representations. Prompts and source code are available at https://github.com/DonsetPG/fluid-heuristic-learning.

Paul Garnier, Jonathan Viquerat, Elie Hachem

Mines Paris - PSL University · Centre for Material Forming (CEMEF) · CNRS +1

RL Control Language Model-Based Control Program Synthesis Feedback Control

Ask PDF

Jun 30, 2026

Offline Reinforcement Learning for Fluid Controls: Data-based Multi-observational Policy Extraction

Active flow control is a fundamental application in engineering. Recent advances in deep reinforcement learning have made progress in this field. However, the classical online RL approaches require extensive real-time interactions with the high fidelity environment, while each sensor configuration change necessitates whole policy retraining. All these factors result in prohibitive computational costs for real-world applications. In this work, we propose a novel offline RL framework that addresses both challenges through data-driven policy extraction. We develop a sensor position-conditioned architecture that enables a single policy network to adapt seamlessly to multiple sensor arrangements. The position-conditioned approach incorporated spatial relationship modeling through Point Attention layers to ensure the generalizability to varying sensor placements. We demonstrate the framework on two representative problems, mitigating chaoticity in the Kuramoto-Sivashinsky equation and flow control over airfoils governed by the Navier-Stokes equation. The result demonstrates that the policy extraction from the dataset provides unprecedented flexibility for sensor placement optimization. This approach represents a significant step towards adaptive, intelligent flow control systems.

Deepak Akhare, Luning Sun, Xin-Yang Liu +4

Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN · Lawrence Livermore National Laboratory, University of Notre Dame, Livermore, CA · Sibley School of Mechanical and Aerospace Engineering, Cornell University, Ithaca, NY, USA

RL Control Reinforcement Learning Offline RL Feedback Control

Ask PDF

Abstract

Sample-Efficient RL

Explore similar work

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

Heuristic Learning for Active Flow Control Using Coding Agents

Offline Reinforcement Learning for Fluid Controls: Data-based Multi-observational Policy Extraction