Organizations: The authors are with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 · The author is with the School of Electrical and Information Engineering, Tianjin University, Tianjin, China 300072.
Existing navigation systems commonly rely on dedicated depth sensors or explicit depth reconstruction, while learning to generate trajectories directly from RGB images remains challenging due to the sim-to-real domain gap. In this work, we propose Stereo-Planner, an end-to-end vision-to-trajectory framework that enables zero-shot sim-to-real planning from stereo RGB observations. The key idea is to formulate trajectory generation as state- and scale-conditioned querying of latent stereo geometry. A Scene Representor incorporates a pretrained vision grounding foundation model to extract geometric cues shared across simulated and real-world environments, supporting generalization across visual domains. A lightweight Trajectory Decoder then translates these representations into trajectories through interleaved attention, separating trajectory-state interaction from scene-feature retrieval and incorporating the stereo baseline as a metric-scale cue. With these designs, Stereo-Planner is trained entirely on simulated data by minimizing differentiable trajectory costs and deployed directly in real-world environments without fine-tuning. Benchmark results demonstrate strong cross-scene generalization, with a 95% success rate in simulation environments at a commanded speed of 8 m/s, and competitive performance against depth-based planners using only stereo RGB observations for scene perception. Furthermore, deployment on a compact quadrotor equipped with an NVIDIA Orin NX enables autonomous flight at speeds of up to 7 m/s, with a mean onboard vision-to-action inference time of 24.27 ms. The software, simulation environments, and hardware designs will be open-sourced at https://github.com/Stereo-Planner/Stereo-Planner.
Figures & tables
Fig. 1: Stereo-Planner: an end-to-end vision-to-action framework that directly generates agile, collision-free trajectories from onboard stereo RGB observations and enables zero-shot sim-to-real transfer without explicit depth reconstruction or dedicated depth sensors.
TABLE I: Comparison of autonomous quadrotor navigation methods and platforms.
Fig. 2: t-SNE [ 61 ] visualization of the synthetic and real-world RGB images shown in Fig. 1 and their corresponding depth images, illustrating the sim-to-real domain gap in each modality.
Fig. 3: Network architecture of Stereo-Planner. The encoder-decoder design in (c) bridges the visual-domain gap between the simulated and real images shown in (a) and (b). Conditioned on the learned scene representation, the decoder refines the learnable trajectory (d) to infer the collision-free trajectory (e).
Fig. 6: Illustration of the proposed VBA. Without recomputing the ESDF, virtual scene scaling is achieved by modifying only the ESDF indexing and queried distances, allowing each stereo image pair to be associated with arbitrary virtual scene scales and corresponding stereo baselines for efficient data augmentation.
Fig. 7: t-SNE visualization of scene representations extracted from the random training environment and the unseen forest and industrial test environments.
Fig. 8: Trajectory predictions under different virtual stereo baselines in forest (top) and industrial (bottom) scenarios. Within each scenario, the same stereo image pair is associated with baselines of 0.10 , 0.15 , and 0.20m , with the voxel grid size scaled accordingly to represent different metric scene scales. The visualized map covers a 10×10m area for all baselines. The trajectories illustrate the comparison between Stereo-Planner (red trajectories) and its variant without VBA (black trajectories).
Fig. 9: Time consumption of Stereo-Planner and Stereo-Planner w Joint-Attn .
Fig. 10: Planning success rates with and without VBA across different stereo baselines in the forest and industrial environments.
Fig. 11: Executed flight trajectories of Stereo-Planner in the previously unseen forest and industrial environments.
Fig. 12: Computation time comparison of Stereo-Planner, YOPO V2, and Ego-Planner V2. GT denotes simulator-provided ground-truth depth; DA3-S and DA3-L denote depth predicted by DA3-Small and DA3-Large.
Fig. 13: Success rates at different commanded speeds in the forest and industrial environments.
Fig. 14: Depth estimation comparison in the forest environment. Top row: DA3-Small. Bottom row: DA3-Large. Left column: absolute depth estimation errors relative to ground truth. Right column: predicted depth maps. Depth and error values are shown in meters.
Fig. 15: Overview of Stereo-MAV. Top left: the assembled quadrotor with an 8cm spacing between the front motor axes and the onboard software pipeline integrating visual-inertial state estimation, Stereo-Planner, and differential flatness mapping. Top right: an exploded view of the hardware components. Bottom: CPU and GPU utilization of the onboard modules on the NVIDIA Orin NX.
Fig. 16: Autonomous flight in a previously unseen carpark using Stereo-Planner. (A, B) Overview of the executed flight paths, with red circles marking the locations of the corresponding snapshots (A1-A6 and B1-B6). Each snapshot pair shows the onboard camera view with overlaid planned trajectories and an external view of Stereo-MAV. The colors of the overlaid trajectories encode the desired speed, ranging from blue at 0m/s to red at 7m/s .
Fig. 17: Speed profiles along each axis during the real-world flight test, with a maximum speed exceeding 7m/s .
Fig. 18: Onboard processing time of Stereo-Planner on the NVIDIA Orin NX during real-world flight.
Although quadcopters boast impressive traversal capabilities enabled by their omnidirectional maneuverability, the need for continuous pilot control in complex environments impedes their application in GNSS and telemetry-denied scenarios. To this end, we propose a novel sensorimotor policy that uses stereo-vision depth and visual-inertial odometry (VIO) to autonomously navigate through obstacles in an unknown environment to reach a goal point. The policy is comprised of a pre-trained autoencoder as the perception head followed by a planning and control LSTM network which outputs velocity commands that can be followed by an off-the-shelf commercial drone. We leverage reinforcement and privileged learning paradigms to train the policy in simulation through a two-stage process: 1) initial training with optimal trajectories generated by a global motion planner acting as a supervisory backbone, 2) further fine-tuning in a curriculum environment. To bridge the sim-to-real gap, we employ domain randomization and reward shaping to create a policy that is both robust to noise and domain shift. In outdoor experiments, our approach achieves successful zero-shot transfer to both obstacle environments and a drone platform that were never encountered during training.
Shiladitya Dutta, Aayush Gupta, Varun Saran +1
College of Engineering, Department of Electrical Engineering and Computer Science, University of California Berkeley
Existing visual navigation policies are inherently bound to fixed camera configurations, creating a fundamental barrier to zero-shot deployment across heterogeneous robot sensor layouts. To overcome this limitation, we present an embodiment-informed navigation policy capable of generalizing across diverse depth sensor configurations on a specific aerial platform. Instead of implicitly learning spatial alignments, our approach explicitly unprojects depth measurements from arbitrary depth sensor payloads, varying in sensor count, mounting extrinsics, and intrinsics, into a shared robot-centric frame, stitching them into a unified spherical range image and a binary validity mask. This mask allows the downstream policy to explicitly distinguish covered space from unobserved blind spots. Trained via reinforcement learning with aggressive camera randomization, our policy generalizes zero-shot to unseen layouts featuring up to seven cameras, scaling success rates from 78% to 95% as total spatial sensing coverage increases. Finally, real-world flight trials on a physical quadrotor, conducted in an obstacle-filled corridor and an outdoor forest, validate the policy's zero-shot transfer across camera configurations and its resilience to sudden online sensor dropouts.
Welf Rehberg, Kostas Alexis
Norwegian Institute of Science and Technology, Norway
Existing high-speed navigation systems for micro aerial vehicles (MAVs) commonly rely on depth cameras, stereo vision, or LiDAR to obtain explicit geometric information, while monocular cameras offer a lighter and lower-cost alternative. However, monocular flight remains challenging because RGB images offer no direct metric cues and visual appearance varies considerably across environments. We propose a geometry-aware monocular visual planning framework that predicts trajectories from two image frames, their relative pose, and the MAV state. Without explicitly reconstructing depth, the proposed method complements the downstream planning objective with feature-level geometric supervision by transferring the planning-relevant representation learned by a privileged depth expert to the RGB planner through cross-modal distillation. The framework first trains a depth-based geometric expert, then distills its planning features into a monocular branch using paired RGB-D observations, and finally adapts the planner to the downstream trajectory planning objective. This design encourages the policy to learn planning-relevant geometric features, enabling collision avoidance. We further use a 3D Gaussian Splatting (3DGS) simulator to render visual observations for data collection, paired RGB-D supervision, and simulation evaluation. The planner trained in simulation is then deployed in real-world environments without real-world fine-tuning, achieving obstacle avoidance at speeds of up to 6.5 m/s.
Wencan Feng, Bailing Tian, Yulin Hui +2
School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.