Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions

Kristian Holme; Jean Rabault; Ricardo Vinuesa; Mikael Mortensen

physics.flu-dynApr 15, 2026

Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions

Authors: Kristian Holme, Jean Rabault, Ricardo Vinuesa, Mikael Mortensen

Organizations: University of Oslo · Independent Researcher, Oslo · University of Michigan

Abstract

Rotating detonation engines (RDEs) are a promising propulsion concept that may offer higher thermodynamic efficiency and specific impulse than conventional systems, but nonlinear phenomena, including transitions to oscillatory or chaotic propagation modes, can hinder practical operation. Deep Reinforcement Learning (DRL) has emerged as a promising method for controlling complex nonlinear dynamics such as those observed in RDEs. However, the multi-timescale nature of the RDE system makes direct application of DRL challenging. We address this challenge by reformulating the DRL problem in a moving reference frame that follows the detonation-wave pattern, making the wave structure appear quasi-steady to the agent. This reformulation enables scale separation between fast detonation propagation and slower operating-mode dynamics. We train DRL controllers to modulate spatially segmented injection pressure in a one-dimensional reduced-order RDE model and induce rapid transitions between different mode-locked states. Across a range of actuation periods, initial states, and target modes, controllers trained in the moving frame learn more reliably than those trained in a stationary frame and remain effective over a broader range of actuation periods. These results suggest that symmetry-aware moving reference frame formulations may be useful for related multiscale flow-control problems and that scale separation should be exploited whenever possible to enable DRL control of multi-timescale systems.

Figures & tables

Name	Agent type	Injection pressure	Reference frame
SU	Single	Uniform	N/A (rotation invariant)
SSS	Single	Segmented	Stationary
SSM	Single	Segmented	Moving
MSS	Multi	Segmented	Stationary
MSM	Multi	Segmented	Moving

Table 1: Learned controller configurations. The reference-frame distinction applies to segmented observations and actions. The SU observation is invariant to rotational phase, so a reference-frame label is not applicable. The letter scheme is detailed in Section 2.4 .

$u_{c}$	$\alpha$	$q_{0}$	$u_{0}$	$n$
1.1	0.3	1.0	0.0	1
$\nu_{1}$	$\nu_{2}$	$\epsilon$	$r$	$s$
0.0075	0.0075	0.15	5.0	3.5

Table 2: Parameter values used in the RDE model equations.

Figure 1: Simulation of ( 1 ) with a predefined step-controller. Starting from an initial bump in $u$ , defined by $u(x)=\frac{3}{2}\sech^{20}(x-1)$ , and spatially uniform combustion progress $\lambda(x)=0.5$ , the system transitions to a single detonation wave stably propagating. As we increase $u_{p}$ , the system bifurcates into a state of two detonation waves with oscillatory phase difference. Before the phase difference stabilizes at a value of $\pi$ , we increase the pressure again, at $t=200$ . At this point, we see a similar bifurcation to three waves, and the phase differences between them almost stabilize at a value of $2\pi/3$ . As we decrease $u_{p}$ at $t=360$ the system enters a time-periodic oscillatory state with two galloping detonation waves. Here, the stronger wave is moving faster than the weaker wave, traveling around the domain until it catches up to the weaker wave, and wave strength is exchanged. The moving reference frame follows the fastest and strongest wave. From this point of view, we see that the weaker wave is traveling backwards across the domain until it is in front of the stronger wave. At this point the strengths of the waves shift, and the previously stronger wave weakens and repeats the pattern. After a series of increases of $u_{p}$ from this state, the system reaches a mode-locked state with four stably propagating waves. (a): $u(x,t)$ shown in the stationary reference frame where $t\in[380,440]$ ; the rapid movement of the detonation waves across the domain is visible. (b): $u(x,t)$ in a moving reference frame with position $\psi$ (see Section 2.3.4 for details on the moving reference frame). (c): Injection pressure $u_{p}(x,t)$ as a function of the non-dimensional time units.

Figure 2: Wave speed as a function of injection pressure $u_{p}$ , during upwards and downwards sweeps of $u_{p}$ . The response is qualitatively similar to Figure 12 in Bykovskii et al. [59] . We begin each simulation by initializing $u(x,t)$ with a localized bump. The injection pressure $u_{p}$ is then uniformly stepped up or down to one of thirteen evenly spaced setpoint values in the interval $[0.4,1.03]$ . From each setpoint value, we sweep $u_{p}$ continuously through a prescribed cycle; up to $1.03$ , down to $0.4$ , and back to the setpoint value. The injection pressure is kept steady for $300$ time units after each step, to allow the system to stabilize. At the end of each stabilization phase, we record the average wave speed and the injection pressure as a single data point. In overlap regions where multiple regimes coexist, the system exhibits hysteresis: the wave count depends on the direction of the pressure sweep. When $u_{p}$ is decreased from a four-wave state, the solution passes through a band of non-mode-locked behavior before settling into a stable three-wave state. In this transitional band, waves form and extinguish rapidly, making speed detection unreliable. A small number of affected data points have been removed as outliers. Simulations were run with $N=2048$ grid points to maximize the accuracy of speed detection.

Controller	Action period $\Delta t$	Policy update rate	Injector modulation rate
	simulation time	physical	physical
SU	$10$	$\approx 2\,\mathrm{kHz}$	$\approx 2\,\mathrm{kHz}$
SSS, MSS	$0.25$ – $8.0$	$\approx 80$ – $2.5\,\mathrm{kHz}$	$\approx 80$ – $2.5\,\mathrm{kHz}$
SSM, MSM	$0.25$ – $8.0$	$\approx 80$ – $2.5\,\mathrm{kHz}$	$\approx 45\,\mathrm{kHz}$ – $3\,\mathrm{MHz}$

Table 3: Approximate physical update and actuation rates under a representative laboratory scaling (circumference

0.224\,\mathrm{m}

, wave speed

\approx 1250\,\mathrm{m\,s^{-1}}

[ 64 ] ), with one simulation time unit

\approx 49\,\mu\mathrm{s}

at the mean model speed

D=1.71

. Rates are order-of-magnitude estimates. All configurations apply each action over a smoothing ramp of

\tau_{\mathrm{smooth}}=0.1

(

\approx 5\,\mu\mathrm{s}

). For the moving-frame controllers, the lower value corresponds to eight injection switches per revolution for the four-lobed bang–bang template in Figure 14 , and the upper value to one update per grid-cell passage (

N=512

Figure 3: Illustration showing one-dimensional RDE simulation data mapped onto a three-dimensional annular cylinder. The red band shows the $u$ variable, with red arrows indicating counterclockwise motion. The blue bars illustrate sections with individually controlled injection pressure, with blue arrows indicating that the sections are attached to the traveling reference frame. Yellow arrows illustrate gas exhaust/expansion.

Figure 4: Observation encodings used by the learned controllers. (a) The SU observation contains the magnitudes of the lowest eight real Fourier coefficients of

u

and

\lambda

. The magnitudes are invariant to circular translations, and truncation removes higher spatial frequencies. (b) The segmented observation contains the maxima of scaled

u

and

\lambda

in 32 bins, followed by the detected and target wave counts. Both counts are divided by four. The dotted curves show the fields from which the observation is constructed. In this example, the detector identifies two of the waves as detonations, and the target is three. Single-agent segmented controllers receive the ordered whole-domain vector. Multi-agent controllers receive a separate circularly recentered copy for each control segment, and moving-frame controllers shift the fields before constructing the displayed encoding.

Figure 5: Reward function used during training. Each rectangle denotes a sub-reward with a range of

(0,1]

[0,1]

. The reward combines a stability term, which favors temporally and spatially regular wave patterns, with a target term, which rewards matching the desired detonation-wave count. The flowchart highlights that successful controllers must both reach the target mode and keep the resulting wave pattern stable.

Figure 6: Example simulation using the best two-step controller for targeting two detonation waves. The data in the top plot is shown with a moving reference frame with position $\psi$ , as detailed in Section 2.3.4 . After an initial drop and subsequent increase in injection pressure, the system eventually bifurcates from the three-detonation-wave mode-locked state into a galloping two-detonation-wave state, which ultimately settles into a mode-locked state with two waves. For a visual comparison of the stationary and moving reference frame, we refer to Figure 1 .

Common simulation and PPO settings
Domain length $L$	$2\pi$
Grid cells $N$	$512$
Spatial scheme	FV + MUSCL/MC + Rusanov
Time integration	SSPRK33
CFL safety factor	$0.62$
Action range	$[-1,1]\to u_{p}\in[0,1.2]$

Table 4: Summary of the common and setup-specific simulation, environment, and PPO parameters. The segmented column applies to SSS, SSM, MSS, and MSM.

Figure 7: Matched uniform-control benchmark: conditional mean transition time over successful trials (five trials per pair). Rows indicate the initial wave count and columns indicate the target wave count. The diagonal entries are unused. All three controllers are assessed with a 0.5-time-unit resolution after a 100-time-unit uncontrolled initialization. SU actions are updated every 10 time units and held between updates. The displayed SU policy is selected separately for each target from six seeds by highest mean success and then lowest conditional mean transition time. SU has a 100% success rate for all transitions, whereas the two-step and PID baselines struggle on $3\to 4$ , where they reach only 80% and 20% success, respectively. The medians over the initial-to-target pairs with finite transition times are 92.05 for SU, 141.55 for two-step, and 128.05 for PID.

Figure 8: Example simulation using the selected SU controller for targeting two detonation waves, starting from the same three-wave initial state as Figure 6 . The data in the top plot is shown with a moving reference frame with position $\psi$ , as detailed in Section 2.3.4 . The controller is activated at $t=100$ and reaches the two-wave mode-locked state after a transition time of $121.5$ time units, compared with $218.2$ for the two-step controller mean in Figure 7 .

Figure 9: Performance of segmented DRL controllers. (a) Conditional mean transition times for the best model selected separately for each method and target by highest mean success and then lowest conditional mean time. Each group corresponds to one target and contains the three off-target initial wave counts. A full-height bar with a red cross denotes no successful trial, not a transition measured at the plotted upper limit. (b) Mean success rate versus action period, averaged over five runs, four targets, and the three off-target initial states. Each policy is evaluated at its trained action period using five trials per initial-to-target pair. Lower transition time and higher success rate are preferable.

Figure 10: Marginal distributions for the two segmented-controller design factors: reference frame (left) and agent structure (right). The top row shows success rate and includes all trained-model and initial-to-target pairs. The bottom row shows transition time and includes successful pairs only. Each raincloud combines a histogram, boxplot, and jittered observations. Success rates are discrete percentages from five trials (

0

20

, …,

100\%

). Each panel pools over the other design factor and over targets, seeds, and action periods.

Figure 11: Segmented-controller success rates resolved by method (figure rows), action period $\Delta t$ (figure columns), initial wave count (rows within each panel), and target wave count (columns within each panel). Each entry is the mean percentage over five trained runs, with five evaluation trials per run and initial-to-target pair. The diagonal entries are unused because the initial and target modes coincide.

Figure 12: Evaluation episode reward during training for the four segmented controllers. The top row contains stationary-frame SSS and MSS, and the bottom row contains moving-frame SSM and MSM. Curves show the mean over five runs and four target wave counts. The shaded bands show one standard deviation, and color identifies the action period

\Delta t

. For MSS and MSM, the horizontal axis counts total pseudo-environment steps.

Figure 13: Internal energy

u

, combustion progress

\lambda

, and injection pressure

u_{p}

during a selected SSM transition from three to two waves, shown in the moving reference frame defined by ( 4 ). The policy applies an asymmetric pressure distribution that weakens one wave and guides the system toward the target two-wave mode.

Figure 14: Internal energy

u

, combustion progress

\lambda

, and injection pressure

u_{p}

t=105

, 85 time units after controller activation, for two selected segmented controllers targeting the

3\to 4

transition. SSS with

\Delta t=0.25

is shown above, and SSM with

\Delta t=4

is shown below. Both rows use the laboratory-frame coordinates. SSS remains irregular at this instant, whereas SSM has reached a four-wave state with the waves near the centers of four high-

u_{p}

regions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 15: Conditional mean transition times for the four segmented controllers, resolved by method (figure rows), action period $\Delta t$ (figure columns), initial wave count (rows within each panel), and target wave count (columns within each panel). Each value averages the per-run conditional means over the trained runs with at least one successful trial in that cell. Blank entries have no successful transition, and diagonal entries are unused.

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 272 papers

Oct 19, 2026

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

Deep reinforcement learning (DRL) is challenging long-held assumptions in active flow control. By learning directly from interaction with a flow, DRL agents can find actuation strategies for nonlinear, unsteady flows without explicit models, often exploiting mechanisms that classical control would not anticipate. High-speed flows are a demanding test of this promise: shocks, shock/boundary-layer interactions, transition, aerothermal loading and microsecond-scale dynamics strain both algorithms and hardware. Yet most demonstrations remain idealized, low-speed simulations with limited evidence of robustness, interpretability, transferability or experimental deployment. In this perspective, we examine what constrains DRL-based high-speed flow control, including compressible simulation cost and fidelity, data efficiency, sensing in extreme environments, reward design, explainability, and generalization across Mach and Reynolds number. We argue that these limits reflect a tension between data-driven optimization and the structured physics, constraints and uncertainties of real flows. We then highlight where DRL may go beyond conventional approaches: adaptive control of shock-induced separation and turbulent boundary layers, coordination of distributed actuators, and discovery of unconventional flow-manipulation mechanisms.The central challenge, however, is translating these capabilities into reliable technology, which will require validated compressible simulation environments, physics- and constraint-aware learning, robustness to off-design conditions and sensor degradation, real-time operation, hardware-in-the-loop and wind-tunnel validation, and credible pathways to safety and certification.

Ameya D. Jagtap

Aerospace Engineering Department, Worcester Polytechnic Institute, Worcester, MA 01609, USA

RL Control Feedback Control Robust RL

Ask PDF

Apr 5, 2026

Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model

Model-free deep reinforcement learning (DRL) methods suffer from poor sample efficiency. To overcome this limitation, this work introduces an adaptive reduced-order-model (ROM)-based reinforcement learning framework for active flow control. In contrast to conventional actor--critic architectures, the proposed approach leverages a ROM to estimate the gradient information required for controller optimization. The design of the ROM structure incorporates physical insights. The ROM integrates a linear dynamical system and a neural ordinary differential equation (NODE) for estimating the nonlinearity in the flow. The parameters of the linear component are identified via operator inference, while the NODE is trained in a data-driven manner using gradient-based optimization. During controller--environment interactions, the ROM is continuously updated with newly collected data, enabling adaptive refinement of the model. The controller is then optimized through differentiable simulation of the ROM. The proposed ROM-based DRL framework is validated on two canonical flow control problems: Blasius boundary layer flow and flow past a square cylinder. For the Blasius boundary layer, the proposed method effectively reduces to a single-episode system identification and controller optimization process, yet it yields controllers that outperform traditional linear designs and achieve performance comparable to DRL approaches with minimal data. For the flow past a square cylinder, the proposed method achieves superior drag reduction with significantly fewer exploration data compared with DRL approaches. The work addresses a key component of model-free DRL control algorithms and lays the foundation for designing more sample-efficient DRL-based active flow controllers.

Zesheng Yao, Zhen-Hua Wan, Canjun Yang +2

School of Mechanical Engineering, Zhejiang University, Hangzhou, PR China · Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, 117575 Singapore · Department of Modern Mechanics, University of Science and Technology of China, Hefei 230027, PR China

Sample-Efficient RL Model-Based RL Feedback Control Reduced-Order Modeling

Ask PDF

Sep 14, 2026

Improving Reduced-Order Rotating Detonation Engine Models with Data Assimilation and Machine Learning

Rotating detonation engines (RDEs) exhibit strongly nonlinear, multiscale wave dynamics that set the observed thermal field. High-fidelity simulations (DNS/LES) resolve these structures but remain computationally prohibitive, while low-order models such as the one-dimensional Koch-Kutz model capture circumferential wave motion yet lack the expressivity for high-frequency content. We use continuous data assimilation (nudging) to synchronize the Koch-Kutz solver with processed high-fidelity temperature data, introducing the prediction-observation mismatch as a relaxation source in the conserved energy equation; where observations are temporally sparse, interpolation supplies a target at every source update. As the nudging strength increases, the reduced model is progressively drawn onto the high-fidelity trajectory, and the forcing recorded along it provides an explicit, state-dependent estimate of the correction the model requires. We then train a Jacobian-regularized closure a priori on this recorded source. With the observation term removed, the corrected model advances autonomously, remains bounded, and recovers the temperature spectrum and the marginal statistics of the conserved variables relative to the baseline.

Ashwin Suriyanarayanan, Romit Maulik

School of Mechanical Engineering, Purdue University, West Lafayette, IN-47906

Data Assimilation Reduced-Order Modeling Computational Fluid Dynamics

Ask PDF

Abstract

RL Control

Explore similar work

Glass-Box Deep Reinforcement Learning for High-Speed Flow Control

Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model

Improving Reduced-Order Rotating Detonation Engine Models with Data Assimilation and Machine Learning