Organizations: Institute of Cyber-Systems and Control, College of Control Science and Engineering, Zhejiang University, Hangzhou, China. · Huzhou Institute of Zhejiang University, Huzhou, China. · Differential Robotics Technology Co., Ltd., Hangzhou, China.
Precise aggressive maneuvers with lightweight onboard sensors remains a key bottleneck in fully exploiting the maneuverability of drones. Such maneuvers are critical for expanding the systems' accessible area by navigating through narrow openings in the environment. Among the most relevant problems, a representative one is aggressive traversal through narrow gaps with quadrotors under SE(3) constraints, which require the quadrotors to leverage a momentary tilted attitude and the asymmetry of the airframe to navigate through gaps. In this paper, we achieve such maneuvers by developing sensorimotor policies directly mapping onboard vision and proprioception into low-level control commands. The policies are trained using reinforcement learning (RL) with end-to-end policy distillation in simulation. We mitigate the fundamental hardness of model-free RL's exploration on the restricted solution space with an initialization strategy leveraging trajectories generated by a model-based planner. Careful sim-to-real design allows the policy to control a quadrotor through narrow gaps with low clearances and high repeatability. For instance, the proposed method enables a quadrotor to navigate a rectangular gap at a 5 cm clearance, tilted at up to 90-degree orientation, without knowledge of the gap's position or orientation. Without training on dynamic gaps, the policy can reactively servo the quadrotor to traverse through a moving gap. The proposed method is also validated by training and deploying policies on challenging tracks of narrow gaps placed closely. The flexibility of the policy learning method is demonstrated by developing policies for geometrically diverse gaps, without relying on manually defined traversal poses and visual features.
Figures & tables
Fig 1 : Working principle and system implementation. (A) Illustration of a goshawk precisely executing aggressive flight through a narrow gap between two tree trunks, guided entirely by its embodied perception system. The bird’s cerebellum integrates visual input from its eyes with proprioceptive signals from its vestibular system to generate precise motor commands for gap navigation. (B) Illustration of the quadrotor platform equipped with a sensorimotor policy for precise aggressive flight using only onboard sensing. The policy integrates the visual sensing and proprioception that directly generate low-level control commands without explicit intermediate interfaces, such as state estimates and reference trajectory. (C) Key capabilities demonstrated by the proposed method through the quadrotor system: single gap traversal, dynamic gap traversal, consecutive gap traversal, and traversal through geometrically diverse gaps.
Fig 2 : Traversal through a rectangular gap with low clearances. (A) Autonomous rollouts from a large range of initial positions through a gap with an unknown but constant tilted pose with a ground-truth of 60°. (B) Autonomous rollouts through a gap with different orientations. (C) Control commands output by the policy and measured responses in two trials from (B) : 90° rolled gap (top row) and 45° pitched gap (bottom row). In the figures, the reference control commands (policy output) are denoted as “Ref”, and the measured responses of the corresponding quantities are denoted as “Resp”. The third column shows the measured roll and pitch angles. We plot the traversal periods as indicated by the legend, defined as starting when the vehicle first enters the gap plane and ending when it exits the gap plane. This period is determined by analyzing the recorded video. We also plot an “invisible period” defined as the duration when the entire landmark is out of the FoV, if it occurs, prior to the traversal period.
Fig 3 : Traversal through a dynamic gap. (A) Snapshots of dynamic gap traversal during flight. Each row represents temporal progression from left to right. The top row shows snapshots during flight through a rotating gap. The bottom shows snapshots during flight through an upward-moving gap. The transparent frames indicate the historical positions of the gap. (B) Three simulated trials through differently moving gaps. Case 1 to Case 3 represent experiments involving a horizontal unidirectional moving gap, a horizontal bidirectional moving gap, a downward moving gap, respectively. The figures in the first row are the spatial trajectories during the flight, where we highlight
Fig 5 : Traversal through consecutive gaps with low clearances. (A) Autonomous rollouts of policies across 6 different tracks containing two or three closely positioned gaps. (B) Comparison of real-world and simulated trajectories (position and velocity magnitude) for Track 4 and Track 5, with real-world data recorded using the HIL setup. Trajectory color represents velocity magnitude. (C) Policy outputs and measured system responses during experiments. The three rows correspond to results from Track 1, Track 2, and Track 4, respectively. Legend abbreviations follow the same conventions as Fig. 2 C, with additional color blocks defined as follows: “Approach Gap 1” denotes the pre-traversal period before the first gap, “Gap 1-2” represents the inter-gap period between the first and second gaps, and “Gap 2-3” indicates the period between the second and third gaps.
Fig 6 : Traversal through narrow openings with various geometries. (A) Autonomous rollouts through physical triangular and parallelogram gaps. We highlight the state of the quadrotor when it is traversing the gap. (B) Simulated rollouts through elliptical, diamond-shaped, and arch-shaped gaps. Trajectories are visualized using Blender [ 40 ] , with the quadrotor’s traversal frame highlighted and other trajectory points shown with white masks. The traversal frame is defined as the point on the trajectory closest to the gap plane. (C) Distribution of traversal roll angles relative to the gaps in simulation and HIL (real-world) experiments. The reference angles used for statistical analysis are marked in the figures for each type of gap.
Fig 7 : Narrow gap traversal with learned segmentation in different visual backgrounds. (A) Autonomous rollouts with 3 different visual backgrounds with different gap orientations. (B) Flawed segmentation results during flight, where the labeled numbers in the panels correspond to the snapshots of the quadrotor marked in (A). (C) Records of experimental success, with each point indicating the approximately measured horizontal (X-Y) starting position. Among these trials, the gap orientation is greater than or equal to 60°and the gap is positioned at (0,0) while its lies in x=0 plane. Red points denote successful trials, while white indicate unsuccessful ones.
Fig 8 : Key ingredient ablation for policy learning and sim-to-real transfer. (A) Ablation of the informed reset (IR) strategy. The left picture shows the success rate evolution for RL training on a rectangular gap, where we compute the mean (solid line) and standard deviation (shadow region) across three trials, and the right shows success rate evolution on Track 5 including three gaps. In the right picture, the “1 gap”, “2 gaps”, and “3 gaps” legends represent the rates of successful traversal of the first gap, second gap, and third gap, respectively. (B) Ablation study of the sim-to-real techniques. We visualize several failure cases when ablating some components of the randomization. The three columns of the figures from left to right correspond to using all the randomization components,ablation of the perturbation force (PF), ablation of response randomization (RR), and ablation of response parameter randomization ( RPR ). The experiment setups corresponding to the two rows of figures are test point 4 of a 60° tilted gap for the top row, and test point 8 of an 80° tilted gap for the bottom row of figures. The test point configuration is
Fig 10 : Baseline performance evaluation. (A) Trajectories tracking visualization of Falanga’s planner, Wang’s planner, and the rollout trajectory of the distilled (vision-based) policy. (B) Traversal position (left) and tilted angle difference (middle) relative to the gap, and success rate in 9 trials for each case. Here, Falanga’s and Wang’s indicate trajectory generation and tracking methods using privileged states, “Ours” in the left and middle panels of the figure indicates the distilled (vision-based) policy, and “Ours (RL)” indicates the RL policy with privileged information. (C) Traversal position (left) and tilted angle difference (middle) relative to the gap, and success trials distribution on initial distances to the gap d0 (right), comparing ours method and Falanga’s system in an onboard-sensing setup.
Fig 11 : Success rates of different approaches across gaps of varying size and tilt angle. Each cell reports the number of successful traversals out of 6 trials for a given gap dimension (height × width) and tilt angle combination, corresponding to test points 1–6 defined in Fig. S3.
Fig 12 : Policy training framework. We use a two-stage sim-to-real policy learning method where the agent learn policies by interacting with the simulated environment. First, an RL policy learns solutions for an oracle MDP, utilizing an informed reset strategy to guide exploration. Subsequently, the RL policy’s behavior is distilled into a recurrent neural policy (denoted as SL policy in the figure) using SL. The SL policy can be directly deployed on the quadrotor without fine-tuning. Upon completion of gap traversal, as determined by a traversal prediction head, a recovery program is triggered to stabilize the quadrotor.
Fig S1 : Simulation trajectories of traversal through gaps with various geometries. Results are shown for gaps with ellipse shape (A) , diamond shape (B) , and arch shape (C) .
Fig S2 : The detailed neural network architecture of the lightweight segmentation model. “C”, “K”, “S”, and “P” are the input channel, kernel size, stride length, the padding, respectively, of the (2D) CNN. “BN” is batch normalization. “Avg Pooling” is global average pooling. “Bilinear Up” is bilinear upsampling. “feature[0]” represents the first feature in the feature stack (in a form of the Sequential data structure of PyTorch) output by the MobileNetv3 encoder.
Fig S3 : Test points and selected results of the ablation study in the Key Ingredients for Policy Learning and Sim-to-Real Transfer section. (A) The test point layout for a pose of gap. (B) Results of the ablation on a gap with a 60°orientation. (C) Results of the ablation on a gap with a 60° orientation. Sectors of different colors represent whether different configurations can successfully traverse the narrow gap at the corresponding test point; if successful, the color sector is present at that point, and vice versa if unsuccessful.
Fig S4 : Detailed neural network architectures in for the policies. (A) The neural network architecture of the actor and critic used in RL training. “D” is the input dimension of the corresponding linear layer. (B) The neural network architecture of the deployed policy in distillation. “C”, “K”, “S”, and “P” are the input channel, kernel size, stride length, the padding, respectively, of the (2D) CNN. D and L are the input dimension and layer number of the GRU, respectively. “D” labeled after the hidden state is its dimension.
Fig S5 : Technical comparison between this work and vision-based drone racing [ 25 ] . The two ellipses represent the technical components of the two research. The overlapping area indicates commonalities or similarities between the two works, while the non-overlapping area represents their differences.
Fig S6 : Illustration of the geometry relationship of the problem studied in this research. A coordinate system is established with its axis parallel to the normal vector of the gap plane Pg to facilitate a clear description of the reward functions.
Fig S7 : Throttle-thrust mapping parameters calibration. The black points represent recorded calibration data, while the colorful surface illustrates the final fitting results. We list the relevant information about the calibration in a table.
Fig S8 : Visualization of the SFC layout and planned SE(3) trajectory. The numbers labeled on the picture represent the indices of the convex polyhedrons that make up the SFC. (A) The SFC and a planned trajectory for traversal through a planar rectangular passable region. (B) The SFC and a planned trajectory for traversal through a planar triangular passable region. (C) The SFC and a planned trajectory for traversal through three consecutive passable regions on planes.
Fig S9 : Trajectory generation performance of the planner in [ 14 ] under simulated state estimation noise. Figures (A), (B), and (C) visualize the replanned trajectories under different state estimation errors, where the initial distance to the gap plane X0 is 2.5 m. Trajectories replanned at different times during flight are represented by distinct colors, while the yellow trajectory at the end of the final successfully planned trajectory denotes the traverse trajectory described in [ 14 ] . Here, the yaw estimation noise is simulated as nψ(d)=U(−εd2,εd2) (°), and position error is simulated as np(d)=[U(−δd2,δd2),U(−δd2,δd2),U(−δd2,δd2)] (m) where d is the distance from the quadrotor to the gap center (unit: meters), ε and δ are noise-level coefficients, and U(⋅) represents uniform distribution. Figure (D) shows success rates under different state estimation error levels. The corresponding value X0 at the start point, the tilted angle ϕgap of the gap, and the noise coefficients are labeled in the figures.
Fig S10 : Ablation study on SE(3) constraints in policy learning and track configuration. (A) Success rates of policies trained with and without full SE(3) constraints, evaluated on Track 4 and Track 5 under both SE(3) and position-only success criteria (see Section S10 for details). “Succ n Gap(s)” denotes the success rate of traversing n consecutive gap(s) within the track. (B) Trajectory visualization of a position-only constrained policy on Track 5. Although the policy is trained without awareness of full SE(3) constraints, the drone successfully traverses the first gap by coincidence, while subsequent gaps result in collision.
Fig S11 : Autonomous rollout trajectories with abrupt halting moving gap. The top row of images illustrate the 3D trajectories with dmin being 1 m (left), 2m (middle) and 0.5 m (right), respectively. The bottom row of images are the corresponding X-Y view of the trajectories.
Fig S12 : Failure modes of consecutive gap traversal. (A) Policy learning with Informed Reset on a more challenging variant of Track 5 (see Table S1 , where the second and third gaps are positioned at x-ranges of 1.1 ∼ 1.2 m and 2.3 ∼ 2.4 m, respectively. Success rate evolution curves compare training from scratch (donated as “from scratch”) versus curriculum-based initialization using the policy trained on the original Track 5 (donated as “resume”). (B) Sim-to-real performance degradation on Track 5. Left and right panels show representative failure cases where the quadrotor collides with the short edge of the second and third gaps, respectively.
Fig S13 : Policy performance with 60° tilted gap under different speed constraints. The numbers labeled on the picture represent the indices of the convex polyhedrons that make up the SFC. (A) The SFC and a planned trajectory for traversal through a planar rectangular passable region. (B) The SFC and a planned trajectory for traversal through a planar triangular passable region. (C) The SFC and a planned trajectory for traversal through three consecutive passable regions on planes.
Track
Gap
Position ( m )
Gap roll ( rad )
Track 1
Gap 1
(0,0,1.5)
π/4.3∼π/3.7
Gap 2
(0.80∼0.90,−0.05∼0.05,1.45∼1.55)
π/7∼π/6
Track 2
Gap 1
(0,0,1.5)
π/3.3∼π/2.7
Gap 2
(0.85∼0.95,−0.10∼0.00,1.35∼1.45)
π/6.5∼π/5.5
Track 3
Gap 1
(0,0,1.5)
π/4.3∼π/3.7
Gap 2
(1.30∼1.40,−0.75∼−0.65,1.45∼1.55)
−π/5.8∼−π/6.2
Table S1 : Training configurations for different tracks in the Traversal through Consecutive Narrow Gaps with Low Clearances section. The x-direction of the following coordinate information is defined as the normal direction of the gap plane, with the positive direction pointing from one side of the starting position to the other. The positive direction of the z-axis is opposite to the direction of gravity and, together with the positive directions of the x-axis and y-axis, forms a right-handed coordinate system. Assume that the ground on which the frame is placed is perpendicular to the direction of gravity and that each gap plane is parallel to each other. The following coordinate information contains the distribution range of the x, y, and z axis coordinates.
Table S2 : Geometry of the gaps in the Traversal through Narrow Passable Regions with Various Geometries section. The 2D coordinates in the table describe the geometric parameters of the passable region corresponding to each gap.
PPO
Hyperparameter
Value
Hyperparameter
Value
# of envs
2048
batch size
100000 (steps)
λGAE
0.95
Reward discount
0.99
Actor learning rate
0.0003
Critic learning rate
0.0005
State normalization trick [ 75 ]
True
Gradient clip norm [ 75 ]
1.5
# of rollout steps of each env between networks updates
200
Table S3 : Hyperparameter setup of policy optimization algorithms. The annotation (single/consecutive) indicates that the parameters for the single gap traversal task and the consecutive gap traversal are different.
Policy
Roll 30°
Roll 60°
Roll 80°
Pitch 20°
Pitch 45°
Pitch 60°
Unified
100%
95.7%
88.6%
90.8%
76.8%
74.4%
Separate
100%
97.5%
92.3%
100%
81.8%
76.8%
Table S4 : Success rates of unified and separate student policies at various tilt angles. Starting from the second column, each column represents the success rates at the specified tilt angle.
Traversing narrow gaps is challenging for autonomous quadrotors, especially when control commands come directly from high-dimensional visual observations. Existing end-to-end methods often rely on behavior cloning or full-rollout backpropagation through time (BPTT) via differentiable simulation, which can limit policy performance or incur high training costs. We propose a two-stage reinforcement learning framework for more efficient ego-centric visuomotor gap-traversal policy training, leveraging quasi-analytical policy gradients (QPG) via differentiable simulation and critic warm-starting. The framework utilizes QPG to avoid backpropagation through visual rendering, reducing computation and memory costs while improving sample efficiency. In the first stage, an expert actor and critic are trained using privileged observations, including gap geometry. Unlike prior gap-traversal approaches, our training utilizing QPG does not require resetting the agent along optimized reference trajectories. In the second stage, a visual policy is trained using binary gap masks from two ego-centric cameras and low-dimensional observations, while its privileged critic is warm-started from the first stage. This substantially improves training efficiency and traversal success compared with cold-starting the critic or using full-rollout BPTT. Our framework does not require retraining the expert actor when system parameters change, enabling more efficient generalization across drone platforms than state-of-the-art visual gap-traversal methods based on action supervision. The learned visual policy also generalizes to gaps with unseen shapes. Extensive real-world experiments further demonstrate robust gap traversal using binary masks rendered online. Beyond gap traversal, the proposed framework is generic and can be extended to other visuomotor robot learning tasks.
Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications. When planning the trajectory, existing approaches both on optimization and learning typically regard quadrotor as a point-mass model, giving path or velocity commands then tracking the commands by outer-loop controller. However, at high speeds, planned trajectories sometimes become dynamically infeasible in actual flight, which beyond the capacity of controller. In this paper, we propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning via differentiable simulation. The high-fidelity simulation in training after parameter identification significantly reduces all the gaps between training, simulation and real world. Analytical process by differentiable simulation provides accurate gradient to ensure efficiently training the low-level policy without expert guidance. The policy employs a lightweight and the most simple inference pipeline that runs without explicit mapping, backbone networks, primitives, recurrent structures, or backend controllers, nor curriculum or privileged guidance. By inferring low-level command directly to the hardware controller, the method enables full flight envelope control and avoids the dynamic-infeasible issue.Experimental results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines across multiple benchmarks. The policy also exhibits strong generalization, successfully deploying zero-shot in unseen, outdoor environments while reaching speeds of up to 7.5m/s as well as stably flying in the super-dense forest. This work is released at https://github.com/Fanxing-LI/avoidance.
Fanxing Li, Shengyang Wang, Yuxiang Huang +5
Shanghai Jiao Tong University · Kunshan Duke University
Although quadcopters boast impressive traversal capabilities enabled by their omnidirectional maneuverability, the need for continuous pilot control in complex environments impedes their application in GNSS and telemetry-denied scenarios. To this end, we propose a novel sensorimotor policy that uses stereo-vision depth and visual-inertial odometry (VIO) to autonomously navigate through obstacles in an unknown environment to reach a goal point. The policy is comprised of a pre-trained autoencoder as the perception head followed by a planning and control LSTM network which outputs velocity commands that can be followed by an off-the-shelf commercial drone. We leverage reinforcement and privileged learning paradigms to train the policy in simulation through a two-stage process: 1) initial training with optimal trajectories generated by a global motion planner acting as a supervisory backbone, 2) further fine-tuning in a curriculum environment. To bridge the sim-to-real gap, we employ domain randomization and reward shaping to create a policy that is both robust to noise and domain shift. In outdoor experiments, our approach achieves successful zero-shot transfer to both obstacle environments and a drone platform that were never encountered during training.
Shiladitya Dutta, Aayush Gupta, Varun Saran +1
College of Engineering, Department of Electrical Engineering and Computer Science, University of California Berkeley