Audited Gradients, Anchored Policies: Model-Based Reinforcement Learning for Efficient Flow Control
Organizations: Department of Mechanical Engineering, University of Washington, Seattle, WA 98195, United States · NSF AI Institute in Dynamic Systems, University of Washington, Seattle, WA 98195, United States
Abstract
Feedback controllers for flows are expensive to design, since each candidate must be evaluated with a high-fidelity solver. Differentiable surrogates reduce that cost, but they also introduce a failure mode that prediction error cannot detect. Policy optimization uses the derivative of the objective with respect to the action, which we call the action gradient and which can be computed either through the surrogate or through the high-fidelity solver. Optimization through the surrogate uses only its action gradient, and a surrogate can follow the flow trajectory closely while misestimating this derivative. The surrogate action gradient is compared against a central finite-difference estimate from the high-fidelity solver on held-out states, under acceptance criteria fixed before the comparison, and we call that comparison an audit. A low-dimensional feedback law is then fitted to the surrogate action gradient when the surrogate passes the audit, and to the solver estimate when it does not. This law initializes a neural policy, and a quadratic penalty holds the policy near it during optimization. The proposed approach is validated on four test cases. The resulting controllers outperform or match a model-free baseline, while using 3 to 156 times fewer high-fidelity solver interactions across the four test cases. One surrogate predicts the objective at correlation 0.9996 but its action gradient at only 0.459, so prediction accuracy does not identify a usable surrogate. Another surrogate, whose action gradient agrees with the solver estimate at correlation 0.996, makes the objective 0.52% worse when the policy is optimized through it without the anchor penalty, so an accurate gradient alone does not provide a working controller. Finally, a surrogate that passes the audit at one fixed action fails it around the actions the deployed controller applies, so a pass is local.
Figures & tables
| Input: solver transitions for fitting, states held out from the fit, reference action , horizon , step , and thresholds fixed in advance | |
|---|---|
| Output: controller and audited gradient from which it was constructed | |
| Fit the surrogate | |
| 1 | fit on the fitting transitions, minimizing in Equation ( 10 ) |
| Audit the action gradient | |
| 2 | for every held-out state and every action coordinate |
| 3 | run the solver twice from , at , to obtain in Equation ( 5 ) |
| Flow and configuration | Dynamical setting | Actuation | Evaluation purpose |
|---|---|---|---|
| Rotating cylinder wake, , one action bounded by , unstructured mesh with nodes | A circular cylinder sheds vortices from alternating sides. The resulting forces oscillate at one dominant frequency. | The action sets the cylinder rotation rate and changes the phase of vortex shedding. | Single action audit, comparison of anchored and unregularized policy optimization. |
| Three cylinder cluster, , three actions bounded by , unstructured mesh with nodes | Three cylinders form a triangular cluster. Their wakes interact and can deflect to either side of the centerline but at the simulated forces show no periodic oscillation. | Each cylinder rotates independently. The three actions reshape the combined wake through the rotation rates. | Comparison of prediction accuracy and action gradient accuracy, evaluation of anchor release. |
| Forced two-dimensional turbulence, , four actions, grid | A periodic two-dimensional flow is sustained by a fixed sinusoidal body force and remains turbulent. | The four actions set the amplitudes of four stationary forcing modes. | Controller built from the measured gradient after a failed audit, and its repeatability across training seeds. |
| Parametric chaotic equation, , grid points, eight actions bounded by | A one dimensional periodic field remains chaotic through growth, damping, and nonlinear transfer across scales. The forcing amplitude breaks the translational symmetry and sets the dynamical regime. | Eight fixed Gaussian actuator profiles apply local forcing to the field. | Transfer of the controller to a second forcing amplitude. |
| Rotating cylinder | Three cylinder cluster | Forced turbulence | Chaotic | Chaotic , transferred | |
|---|---|---|---|---|---|
| Surrogate | latent field model | force model, then latent field model | Fourier neural operator | gradient model | gradient model from , not refitted |
| Audit result | pass, , , one actuator | force model fails at , , , latent field model passes at , , , then , , | fail, , , | pass, , , | inherited from |
| Gradient building the controller | surrogate gradient | gradient of the latent field model, after the force model is rejected | measured gradient | gradient model | none at , the controller is transferred from |
| Reference action, horizon | zero, | zero, | zero, | action of the base controller, | as at |
| Audited states | eight held-out states spread evenly over one vortex shedding cycle | 55 held-out states, 38 of them also used for fitting, then 16 new held-out states | 20 uncontrolled states | eight held-out states from one trajectory | as at |
| Controller | cosine of the shedding phase minus a fitted phase (Equation ( 12 )) | linear map from the nine measured scalars to rotation rates | neural state feedback within of the measured descent direction | neural policy plus a correction from the gradient model | controller from , with sensor noise filtered by a linear observer |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Audit statistic | Threshold | Interval preserving every decision |
|---|---|---|
| Pooled correlation | ||
| Sign agreement | ||
| Median per-actuator correlation |
| Surrogate | Median actuator correlation | Minimum actuator correlation | Median per-state cosine |
|---|---|---|---|
| three cylinder force model, rejected | |||
| three cylinder latent field model, new held-out states | |||
| forced turbulence neural operator | |||
| chaotic equation gradient model, |
| flow and sampled action coordinates | states | stability interval | |
|---|---|---|---|
| rotating cylinder, one rotation | 8 | fixed phase bins | |
| three cylinder cluster, three rotations | 16 | ||
| forced turbulence, four forcing modes | 20 | ||
| parametric chaotic equation, , fixed reference action | 24 | ||
| parametric chaotic equation, , actions of the first controller built at | 24 | ||
| parametric chaotic equation, , actions of the second controller built at | 24 |
| Result | Re | Gain (%) | Statistical unit and status |
|---|---|---|---|
| Phase polynomial pathwise | 100 | 24 paired rollouts, three training seeds, best high-fidelity iteration, exploratory | |
| Model-free PPO, fixed checkpoint | 100 | 12 paired rollouts, three seeds four initial conditions, reference | |
| Model-free PPO, peak checkpoint | 100 | three seeds, best of three checkpoints each, selection biased reference | |
| Extracted law, | 100 | eight paired initial conditions, confirmed extraction | |
| Phase line search, | 100 | eight paired initial conditions, exploratory | |
| Released pathwise | 100 | four seeds eight initial conditions |
| Deployed controller | Audit triple | |
|---|---|---|
| First controller built at , | ||
| Second controller built at , | ||
| Controller transferred from |
| Controller | Accumulated state cost | Paired comparison, interval |
|---|---|---|
| Forcing amplitude | ||
| surrogate-based controller | from to against matched TD3, no detected difference | |
| matched TD3 | baseline | |
| matched polynomial TD3 | from to against the surrogate-based controller | |
| Forcing amplitude | ||
| surrogate-based controller with observer | from to against matched TD3, noninferior under the margin fixed in advance | |
| Finite-difference campaign | Rows | Distinct | Actions | Steps | Role | |
|---|---|---|---|---|---|---|
| paired actions, horizon 25 | 12 | 12 | 5 | 25 | 1,500 | supporting |
| lift resolution, horizon 25 | 16 | 4 | 9 | 25 | 900 | supporting |
| converged response, horizon 25 | 4 | 2 | 3 | 25 | 150 | supporting |
| lift resolution, horizon 5 | 16 | 4 | 9 | 5 | 180 | resolution check |
| lift resolution, horizon 1 | 16 | 4 | 9 | 1 | 36 | resolution check |
| Route or baseline | Rotating cylinder | Forced turbulence | Three cylinder cluster | Chaotic | Chaotic |
|---|---|---|---|---|---|
| finite differences alone | 2,766 to 6,264 | 12,800 | 1,330 | 44,800 | 0 |
| route through the extracted law | 5,766 to 9,264 | 14,400 | 4,930 | 92,752 | 0 |
| complete route | 7,366 to 10,864 | 14,400 | 4,930 | 92,752 | 63,984 |
| one model-free training run | 32,768 | 100,000 | 120,320 | 9,990,000 | 9,990,000 |
| model-free baseline as reported | 98,304 | 400,000 | 360,960 | 9,990,000 | 9,990,000 |
| advantage over one training run | to times | times | times | times | times |
| Test case and algorithm | Settings | Training seeds and interactions |
|---|---|---|
| rotating cylinder, PPO | multilayer perceptron policy, learning rate , 512 steps per update, batch size 128, discount , normalized observations | three seeds, 32,768 each (16,384 at ) |
| three cylinder cluster, PPO | multilayer perceptron policy, learning rate , 512 steps per update, batch size 128, discount , normalized observations and rewards | three seeds, 120,320 each |
| forced turbulence, PPO | convolutional policy on the vorticity field, learning rate , 2048 steps per update, batch size 64, 10 epochs, discount , generalized advantage parameter , clip range , entropy coefficient | four seeds, about 100,000 each |
| parametric chaotic equation, TD3 | actor and critic with hidden width 256, learning rate , batch size 256, target update rate , target policy noise clipped at , policy update every two critic updates, exploration noise , random actions in the first 500 episodes | one seed, 9,990,000 |
| Controller | Scored over | Reduction | Note |
|---|---|---|---|
| Rotating cylinder wake, , objective | |||
| extracted law | eight initial conditions | the audited direction, no policy training | |
| anchored and released pathwise | 32 paired rollouts | retains of the reduction of the extracted law | |
| pathwise without the anchor | three training seeds | raises the objective despite an accurate gradient, with no anchor penalty and an initialization not derived from the law, best of eight training iterations per seed, each scored by one rollout against the mean uncontrolled objective | |
| model-free baseline | 12 paired rollouts | 32,768 solver interactions per training seed | |
| extracted law frozen, at , and | eight initial conditions each | , , | transferred with no refitting of any kind |
Model-Based RL
Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.
See all 266 papers