Learning step-level dynamic soaring in shear flow

Lunbing Chen; Jixin Lu; Yufei Yin; Jinpeng Huang; Yang Xiang; Hong Liu

Figure 1: Problem formulation and deep reinforcement learning framework for autonomous dynamic soaring. (A) Three-dimensional trajectory of the navigation task. (B) The point-mass glider model [ 14 ] . The egocentric frame

(x_{e},y_{e},z)

denotes heading, left-wing, and up directions.

\mathbf{u}

,

\mathbf{v}

, and

\mathbf{w}

represent airspeed, ground velocity, and wind velocity. The aerodynamic states are defined by pitch

\theta

, heading

\psi

, and bank angle

\phi

[ 41 ] .

L

,

D

, and

W

denote lift, drag, and weight.

\psi_{w}

and

\psi_{t}

denote the wind direction and the target direction relative to the wind in the inertial (Earth-fixed) frame. (C) Horizontal projections of representative tailwind, crosswind, and headwind trajectories, illustrating the characteristic zig-zag motion [ 11 ] . The circular regions indicate the randomized initial positions and the target success regions. (D) Deep reinforcement learning (DRL) framework. The agent interacts with the wind environment through observations and outputs continuous control actions

(\phi,C_{L})

. (E) Parameter space of wind conditions. Grey shading indicates available energy (

\sigma_{\mathrm{max}}=w_{\mathrm{ref}}/(4\delta)

, subsection 4.1 ). Shaded regions denote prior studies: blue [ 16 ] , yellow [ 38 ] , and green this work. The red line marks the minimum wind-speed boundary [ 18 ] (left region infeasible), and the magenta dotted line indicates the theoretical perceptual resolution limit

l_{\mathrm{decision}}=v_{c}t_{\mathrm{decision}}

. (F) Logistic wind profiles. (G, H) Training success-rate curves under different observation settings ( Table 1 ) and reward formulations ( Table 2 ). Curves are averaged over five random seeds, with shaded regions indicating standard deviation. (I, J) Success rate as a function of

(\psi_{t},\,w_{\mathrm{ref}})

, and

(w_{\mathrm{ref}},\,\delta)

, where white dashed contours denote the

90\%

SR boundary. Statistical definitions of success are provided in subsection 4.3 .

No.	Setting	Obs.	Training SR	Test SR
E1	full	$\Delta x_{e},\Delta y_{e},z,u_{x,e},u_{z},w_{x,e},w_{y,e},\sigma_{w}$	$95.5\%\pm 0.7\%$	$97.3\%\pm 0.8\%$
E2	No shear	$\Delta x_{e},\Delta y_{e},z,u_{x,e},u_{z},w_{x,e},w_{y,e}$	$93.8\%\pm 1.1\%$	$92.3\%\pm 1.8\%$
E3	ground speed	$\Delta x_{e},\Delta y_{e},z,v_{x,e},v_{y,e},u_{z},w_{x,e},w_{y,e},\sigma_{w}$	$93.1\%\pm 2.4\%$	$97.9\%\pm 0.7\%$
E4	polar wind	$\Delta x_{e},\Delta y_{e},z,u_{x,e},u_{z},w,\psi_{w,e},\sigma_{w}$	$53.7\%\pm 4.3\%$	$62.4\%\pm 4.9\%$
G1	geocentric	$\Delta x_{g},\Delta y_{g},z,u_{x,g},u_{y,g},u_{z},w_{x,g},w_{y,g},\sigma_{w}$	$74.8\%\pm 8.0\%$	$88.0\%\pm 4.4\%$
E0	No wind	$\Delta x_{e},\Delta y_{e},z,u_{x,e},u_{z}$	$30.8\%\pm 5.6\%$	$23.0\%\pm 9.0\%$

Table 1: Observation ablation. This table summarizes the observation configurations used in the sensory ablation study. All policies are trained with the same reward formulation (Rwd.1 in Table 2 ). Success-rate definitions are provided in subsection 4.3 . E1 defines the full egocentric observation set. Removing shear information (E2) or replacing airspeed with groundspeed (E3) leads to moderate performance changes, while the polar representation (E4) significantly degrades learning. The geocentric formulation (G1) also shows reduced performance compared to the egocentric baseline. Configurations without wind (E0) or without airspeed information (E0’) fail to learn meaningful policies. For some settings, curriculum learning does not cover the full

0^{\circ}

–

180^{\circ}

range ( subsection 4.2 ): E0 converges within

40^{\circ}

–

140^{\circ}

, E0’ fails beyond

80^{\circ}

–

100^{\circ}

, and E4 converges within

10^{\circ}

–

170^{\circ}

. Reported success rates are computed over these respective ranges.

No.	$r_{\mathrm{process}}$	Training SR	Test SR
1	$\xi_{\dot{e}}\dot{e}/e_{\mathrm{norm}}\sigma_{\mathrm{norm}}+\xi_{v}\,v_{\mathrm{net}}/u$	$95.5\%\pm 0.7\%$	$97.3\%\pm 0.8\%$
2	$\xi_{v}\,v_{\mathrm{net}}/{u}$	$92.6\%\pm 1.1\%$	$94.5\%\pm 1.3\%$
3	$\xi_{\dot{e}}\dot{e}/e_{\mathrm{norm}}\sigma_{\mathrm{norm}}$	$58.0\%\pm 3.3\%$	$62.4\%\pm 4.9\%$
4	$\xi_{e}\Delta e/e_{\mathrm{norm}}+\xi_{d}\Delta d/d_{\mathrm{norm}}$	$93.4\%\pm 0.3\%$	$91.8\%\pm 1.9\%$

Table 2: Process reward ablation. Training and test success rates under different reward formulations. All policies use the same observation space (Obs.E1 in Table 1 ). The combined reward (Rwd.1) achieves the best performance, while the directional-progress term alone (Rwd.3) yields comparable results. In contrast, the energy-based term alone (Rwd.2) fails to produce a robust policy. State-based rewards (Rwd.4) achieve moderate performance but remain inferior to process-based formulations.

Figure 2: Emergence of a two-phase dynamic-soaring navigation strategy governed by kinetic-energy management. (A–D) Time evolution of key variables along a representative crosswind trajectory ( Figure 1 A): (A) airspeed

u

, ground-directed velocity

v_{\mathrm{net}}

, and altitude

z

; (B) total energy

e

, kinetic energy

e_{k}

, and potential energy

e_{p}

; (C) pitch angle

\theta

and heading angle

\psi

; (D) control actions

C_{L}

and

\phi

. The grey line indicates the transition time

t^{*}

( subsection 4.3 ), separating the dynamic-soaring (DS) and targeted-gliding (TG) phases. (E–H) Statistical characterization across

(\psi_{t},w_{\mathrm{ref}})

: (E) two-phase significance ratio (SiR, subsection 4.3 ); (F) transition time

t^{*}

( subsection 4.3 ); (G) airspeed at transition

u(t^{*})

; (H) altitude at transition

z(t^{*})

. (I–P) Energy and navigation performance in the two phases. (I–J) net change in total energy

\Delta e=\Delta e_{k}+\Delta e_{p}

; (K–L) net change in kinetic energy

\Delta e_{k}

; (M–N) net change in potential energy

\Delta e_{p}

; (O–P) mean directional velocity

\bar{v}_{\mathrm{net}}

. Panels (I, K, M, O) correspond to the DS phase, and (J, L, N, P) to the TG phase. Energy changes are computed relative to the transition time

t^{*}

:

\Delta e_{\mathrm{DS}}=e(t^{*})-e(0)

and

\Delta e_{\mathrm{TG}}=e(t_{N})-e(t^{*})

, with analogous definitions for

\Delta e_{k}

and

\Delta e_{p}

.

\bar{v}_{\mathrm{net}}

is averaged over each phase. These results show that kinetic energy is accumulated during the DS phase and expended during the TG phase to enable goal-directed navigation.

Figure 3: Structured policy representation in observation space under a fixed condition (

\psi_{t}=90^{\circ}

,

w_{\mathrm{ref}}=10\,\mathrm{m/s}

,

\delta=0.55\,\mathrm{m}

). Columns 1–2 show a representative successful trajectory colored by

\phi

and

C_{L}

, with the start, target, and DS–TG transition marked by a green circle, red circle, and red cross. Columns 3–4 show occupancy-filtered heatmaps from

1,000

successful DS-phase trajectories (TG phase in Figure S3 ), retaining states with more than

100

samples. Rows 1–3 correspond to relative position

(\Delta x_{e},\Delta y_{e})

, velocity

(u_{x,e},u_{z})

, and relative wind

(w_{x,e},w_{y,e})

. Trajectories show temporal evolution, while heatmaps reveal consistent observation–action mappings. In

(\Delta x_{e},\Delta y_{e})

, the state encodes distance

d_{e}

and target direction

\psi_{t,e}

, organizing navigation and the DS–TG transition. In

(u_{x,e},u_{z})

, the auxiliary curve indicates the

3g

load-factor limit for

u=u_{c}

. In

(w_{x,e},w_{y,e})

, concentric circles at

5

and

10\,\mathrm{m/s}

indicate characteristic shear-layer and free-stream wind magnitudes. The final row shows coordinate mappings: (M, N)

(x_{e},y_{e})\rightarrow(d_{e},\psi_{t,e})

and (O, P)

(w_{x,e},w_{y,e})\rightarrow(w,\psi_{w,e})

.

Figure 4: Robustness and generalization under out-of-distribution conditions. (A, C) Representative trajectory in a spatially varying wind field with coupled speed and shear variations. (B) Normalized spatial distribution of the harmonic disturbance field

H(\mathbf{p})

(defined in subsection 4.4 ). (D–F) Success-rate heatmaps under perturbed wind conditions, showing robust performance across variations in wind-direction scale

\psi_{w}

, reference wind speed

w_{\mathrm{ref}}

, and shear thickness

\delta

. G Representative trajectories for navigation tasks. (H–I) Success-rate heatmaps for static targets

(\psi_{t},d_{t})

and dynamic targets

(v_{t,m},\psi_{t,m})

. (J) Robustness to observation noise with increasing intensity

\sigma

. The policy maintains high performance across environmental variability, task complexity, and sensing uncertainty.

Figure 5: Comparison of ground-speed envelopes and energy-direction trade-offs across learned, biological, and optimal strategies. (A–C) Ground-speed envelopes under different wind conditions. (A) RL policy predictions for

w_{\mathrm{ref}}=6,10,18\,\mathrm{m/s}

in polar coordinates. (B) Experimental envelopes derived from biological flight data [ 10 ] , fitted using a generalized additive model [ 11 ] , with background shading indicating data density. (C) Numerically optimal envelopes obtained via IPOPT-based trajectory optimization [ 14 ] . (D–F) Joint distributions of energy-harvesting coefficient

\eta

and directional-flight coefficient

\epsilon

(defined in [ 14 ] ) for

\psi_{t}\in[60^{\circ},120^{\circ}]

. Panels show (D) RL policy samples, (E) filtered experimental data, and (F) IPOPT solutions. Color maps indicate log-scaled density with overlaid contours. All three approaches exhibit a consistent trade-off structure between energy acquisition and directional progress, despite variability in experimental data.

No.	$\textit{NN}_{\text{Actor}}$	$\textit{NN}_{\text{Critic}}$	Training SR	Test SR
1	$\left[512,512,512\right]$	$\left[512,512,512\right]$	$95.5\%\pm 0.7\%$	$97.3\%\pm 0.8\%$
2	$\left[512,512\right]$	$\left[512,512\right]$	$82.6\%\pm 6.4\%$	$82.6\%\pm 2.5\%$
3	$\left[256,256,256\right]$	$\left[256,256,256\right]$	$68.2\%\pm 10.4\%$	$62.4\%\pm 4.9\%$
4	$\left[512,512,512\right]$	$\left[1024,1024,1024\right]$	$91.8\%\pm 3.3\%$	$95.8\%\pm 2.7\%$

Table S1: Training and test success rates for different neural network architectures. All configurations use the same observation space (Obs.E1, Table 1 ) and reward formulation (Rwd.1, Table 2 ). Each architecture is defined by the layer widths of the actor and critic networks.

Figure S1: Flight trajectories and state variables under representative wind conditions (corresponding to Figure 1 C, Figure 2 A–D). The figure presents agent performance across five scenarios: (row 1) tailwind target (

\psi_{t}=0^{\circ},\,w_{\mathrm{ref}}=10\penalty\ \mathrm{m/s},\,\delta=0.55\penalty\ \mathrm{m}

); (row 2) headwind target (

\psi_{t}=180^{\circ},\,w_{\mathrm{ref}}=10\penalty\ \mathrm{m/s},\,\delta=0.55\penalty\ \mathrm{m}

); (row 3) low wind speed (

\psi_{t}=90^{\circ},\,w_{\mathrm{ref}}=6\penalty\ \mathrm{m/s},\,\delta=0.55\penalty\ \mathrm{m}

); (row 4) high wind speed (

\psi_{t}=90^{\circ},\,w_{\mathrm{ref}}=20\penalty\ \mathrm{m/s},\,\delta=0.55\penalty\ \mathrm{m}

); and (row 5) thick shear layer (

\psi_{t}=90^{\circ},\,w_{\mathrm{ref}}=10\penalty\ \mathrm{m/s},\,\delta=1.33\penalty\ \mathrm{m}

). Column 1 shows top-down trajectories in the

x

–

y

plane, color-coded by altitude

z

. Column 2 presents kinematic variables. Column 3 shows flight angles. Column 4 displays control inputs.

Figure S2: Phase-wise performance metrics across wind-shear conditions (

w_{\mathrm{ref}}

and

\delta

; corresponding to Figure 2 E–P). (A) Two-phase significance ratio (SiR); (B) transition time

t^{*}

; (C) airspeed at transition

u(t^{*})

; (D) altitude at transition

z(t^{*})

. (E–L) Energy and navigation metrics for the two phases. (E, F) net change in total energy

\Delta e

; (G, H) net change in kinetic energy

\Delta e_{k}

; (I, J) net change in potential energy

\Delta e_{p}

; (K, L) mean directional velocity

\bar{v}_{\mathrm{net}}

. Panels (E, G, I, K) correspond to the DS phase, and (F, H, J, L) to the TG phase.

Figure S3: Policy distribution during the targeted gliding (TG) phase (corresponding to Figure 3 ). The left column ( A , C , E ) shows the bank angle

\phi

, while the right column ( B , D , F ) shows the lift coefficient

C_{L}

.

Figure S4: Evolution of learned state–action mappings across training. Heatmaps illustrate the policy-induced state–action distributions in multiple kinematic domains at four training checkpoints (

1.6

,

1.7

,

1.8

, and

1.9\times 10^{8}

timesteps), arranged from left to right.

Figure S5: Trajectory tracking and policy heatmaps under tailwind and headwind conditions. Results are shown for the environmental setting

w_{\mathrm{ref}}=10\penalty\ \mathrm{m/s}

and

\delta=0.55\penalty\ \mathrm{m}

. The left two columns (A, B, E, F, I, J) correspond to a target direction

\psi_{t}=0^{\circ}

, while the right two columns (C, D, G, H, K, L) correspond to

\psi_{t}=180^{\circ}

. Columns 1 and 3 show representative successful trajectories, whereas columns 2 and 4 present aggregated state–action visitation heatmaps over successful episodes.

Figure S6: Trajectory tracking and policy heatmaps under high-wind and thick-shear conditions. Results are shown for a fixed target direction

\psi_{t}=90^{\circ}

. The left two columns (A, B, E, F, I, J) correspond to

w_{\mathrm{ref}}=20.0\penalty\ \mathrm{m/s}

with

\delta=0.55\penalty\ \mathrm{m}

, while the right two columns (C, D, G, H, K, L) correspond to

w_{\mathrm{ref}}=10.0\penalty\ \mathrm{m/s}

with

\delta=1.33\penalty\ \mathrm{m}

. Columns 1 and 3 show representative successful trajectories, whereas columns 2 and 4 present aggregated state–action visitation heatmaps over successful episodes.

Figure S7: Training dynamics and observation-space-dependent performance across task and environmental conditions. (A–D) Training curves over timesteps (corresponding to Figure 1 G), including cumulative reward (A), timeout rate (B), crash rate (C), and steps per episode (D). Curves correspond to different observation-space designs ( Table 1 ); solid lines denote means and shaded regions indicate standard deviations. (E–L) Success-rate distributions evaluated within the training domain (corresponding to Figure 1 I, J). Columns correspond to Obs.E2 (E,I), Obs.E3 (F,J), Obs.E4 (G,K), and Obs.G1 (H,L). Colormaps represent success rate (SR, %), with dashed contours indicating regions where SR exceeds 90%.

Figure S8: Success-rate distributions and trajectory patterns under varying wind directions. (A, B) Contour maps of navigation success rate as a function of target direction (

\psi_{t}

) and wind direction (

\psi_{w}

) for egocentric (relative) observations (Obs.E1) (A) and geocentric (absolute) observations (Obs.G1) (B). Dashed contours denote the 90% success-rate boundary. (C–F) Representative trajectories projected onto the horizontal (East–North) plane under different wind directions with

\psi_{w}=-90^{\circ}

( C ),

0^{\circ}

( D ),

90^{\circ}

( E ), and

180^{\circ}

( F ), all shown for Obs.E1.

Figure S9: Training dynamics and reward-dependent performance across task and environmental conditions. (A–D) Training curves over timesteps (corresponding to Figure 1 H), including cumulative reward (A), timeout rate (B), crash rate (C), and steps per episode (D). Curves correspond to different reward designs ( Table 2 ); solid lines denote means and shaded regions indicate standard deviations. (E–L) Success-rate distributions evaluated within the training domain. Columns correspond to Rwd.1 (E,I), Rwd.2 (F,J), Rwd.3 (G,K), and Rwd.4 (H,L). Colormaps represent success rate (SR, %), with dashed contours indicating regions where SR exceeds 90%.

Figure S10: Evolution and decomposition of the process reward during training. Results are shown for Obs.E1 with reward formulation Rwd.1 ( Table 2 ). A detailed definition of the process reward and its components is provided in subsection 4.2 .

Figure S11: Mission feasibility boundaries and limiting behaviors in static target-navigation tasks. (A–D) Performance landscapes over target direction and distance (corresponding to Figure 4 H), including success rate (A) , timeout rate (B) , terminal distance to the target at episode end (C) , and average episode length (D) . (E–H) Flight analysis for a successful boundary case (

\psi_{t}=180^{\circ},\,d_{t}=760\penalty\ \mathrm{m}

). (I–L) Analysis of a timeout case beyond the feasibility limit (

\psi_{t}=180^{\circ},\,d_{t}=800\penalty\ \mathrm{m}

). In (I–L), although sustained flight is maintained through repeated energy-harvesting cycles, the agent fails to progress toward the target due to limited exposure to such relative configurations during training (K), resulting in a quasi-stationary loitering pattern until termination.

Figure S12: Performances of dynamic target interception across velocity and heading regimes. (A–D) Performance landscapes over target velocity and heading (corresponding to Figure 4 I), including success rate (A) , timeout rate (B) , terminal distance to the target at episode end (C) , and average episode length (D) . (E–H) Successful interception of a fast downwind-moving target (

v_{t,m}=16\penalty\ \mathrm{m/s}

,

\psi_{t,m}=-90^{\circ}

). (I–L) Interception failure for an identical target moving upwind (

v_{t,m}=16\penalty\ \mathrm{m/s}

,

\psi_{t,m}=90^{\circ}

). Although the agent is unable to close the distance, resulting in a timeout-driven chase, it continuously pursues the target. In (E,I) , the red arrow indicates the direction of target motion, with endpoints marking the initial and final positions.

Figure S13: Flight trajectories, wind estimation, and energy/direction-related coefficients from empirical seabird data. Representative samples are from the BiP_WA_CROZET2019_F0957_N18035_G dataset [ 10 ] . (A, D) Spatial trajectories of the flights. (B, E) Relationship between heading angle (

\psi

) and horizontal ground speed (

v

). Gray markers denote empirical data, while the solid red curve shows the fitted trigonometric model used to estimate wind speed (

w

) and wind direction (

\psi_{w}

, indicated by vertical dashed blue lines), following the method of [ 50 ] . (C, F) Temporal evolution of the energy-harvesting rate (

\eta

, green) and the directional efficiency (

\epsilon

, purple) over the selected flight segment. Horizontal dashed lines indicate the corresponding time-averaged values, computed following [ 14 ] .

Learning step-level dynamic soaring in shear flow

Abstract

Figures & tables

Feedback Control

Shaping Wind-Tunnel Airflow for Unmanned Aerial Vehicles using Online Learning

Physics-guided surrogate learning enables zero-shot control of turbulent wings

High-lift Wing Separation Control via Bayesian Optimization and Deep Reinforcement Learning

Abstract

Feedback Control

Explore similar work

Shaping Wind-Tunnel Airflow for Unmanned Aerial Vehicles using Online Learning

Physics-guided surrogate learning enables zero-shot control of turbulent wings

High-lift Wing Separation Control via Bayesian Optimization and Deep Reinforcement Learning