Organizations: School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore · College of Design and Engineering, National University of Singapore, Singapore 119077, Singapore · College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China
Learning visuomotor policies for Autonomous Aerial Vehicles (AAVs) relying solely on monocular vision is an attractive yet highly challenging paradigm. Existing end-to-end learning approaches directly map high-dimensional RGB observations to action commands, which frequently suffer from low sample efficiency and severe sim-to-real gaps due to the visual discrepancy between simulation and physical domains. To address these long-standing challenges, we propose GaussFly, a novel framework that explicitly decouples representation learning from policy optimization through a cohesive real-to-sim-to-real paradigm. First, to achieve a high-fidelity real-to-sim transition, we reconstruct training scenes using 3D Gaussian Splatting (3DGS) augmented with explicit geometric constraints. Second, to ensure robust sim-to-real transfer, we leverage these photorealistic simulated environments and employ contrastive representation learning to extract compact, noise-resilient latent features from the rendered RGB images. By utilizing this pre-trained encoder to provide low-dimensional feature inputs, the computational burden on the visuomotor policy is significantly reduced while its resistance against visual noise is inherently enhanced. Extensive experiments in simulated and real-world environments demonstrate that GaussFly achieves superior sample efficiency and asymptotic performance compared to baselines. Crucially, it enables robust and zero-shot policy transfer to unseen real-world environments with complex textures, effectively bridging the sim-to-real gap.
Figures & tables
Fig. 1 : The framework of GaussFly. (A) 3DGS-Based Scene Reconstruction. Background environments and foreground assets are reconstructed independently. By utilizing a frozen SAM2 model [ 30 ] for asset masking, the framework enables flexible object composition and extensive domain randomization to generate diverse simulation environments. (B) Contrastive Representation Learning. Visual observations are rendered on-the-fly based on camera intrinsics and extrinsics. Two differently augmented views of the same rendered image are processed by CNN encoders, and an InfoNCE loss ( LInfoNCE ) is applied to align their latent representations, thereby extracting noise-resilient visual features. The optimized CNN encoder is subsequently frozen for policy learning. (C) Sim-to-Real Transfer. Benefiting from the high-fidelity 3DGS environments and robust latent representations, the trained visuomotor policy achieves seamless zero-shot transfer, demonstrating high navigation success rates across both simulation and real-world environments.
Fig. 2 : Illustration of the simulation environments.
Fig. 3 : Training reward curves. GaussFly achieves comparable asymptotic performance to NPE but exhibits significantly higher sample efficiency. Although NPE benefits from prior guidance, its direct reliance on raw RGB inputs still bottlenecks its learning speed. Furthermore, the standard PPO and D3QN baselines show inferior sample efficiency and asymptotic returns due to the lack of specialized representation mechanisms in vanilla RL algorithms.
Method
NE ↓
OS ↑
SR ↑
SPL ↑
CR ↓
TTS ↓
D3QN [ 38 ]
2.06 ± 0.55
46.0 ± 6.0
42.5 ± 5.0
0.34 ± 0.06
53.0 ± 5.5
398 ± 13
PPO [ 36 ]
1.72 ± 0.47
57.5 ± 5.0
51.5 ± 4.0
0.36 ± 0.06
40.5 ± 4.5
373 ± 11
NPE [ 9 ]
1.04 ± 0.31
64.0 ± 4.0
60.5 ± 3.5
0.48 ± 0.04
36.0 ± 4.0
329 ± 9
DAgger
1.27 ± 0.33
60.5 ± 4.5
56.5 ± 4.0
0.46 ± 0.04
42.0 ± 4.5
341 ± 11
Hybrid-APF [ 40 ]
0.84 ± 0.54
74.5 ± 6.5
70.0 ± 6.0
0.68 ± 0.06
12.0 ± 10.0
353 ± 17
GaussFly (Ours)
0.52 ± 0.24
88.5 ± 3.0
84.0 ± 2.0
0.81 ± 0.03
9.5 ± 2.0
306 ± 7
TABLE I : Performance of Baselines in the Unseen Evaluation Environment
Fig. 4 : Visualization of representative flight trajectories. GaussFly (red) generates the smoothest and most direct path in the unseen environment, whereas baselines suffer from collisions or highly curved detours.
Fig. 5 : Representative real-world flight trajectories in two indoor environments. Top: Third-person views of the AAV navigating towards the goal. Middle: Consecutive monocular FPV observations captured onboard. Bottom: Executed 2D flight trajectories with velocity profiles. Despite varying obstacle layouts and visual textures, GaussFly achieves robust and collision-free zero-shot navigation across environments.
Fig. 6 : t-SNE visualization under visual interference. Red and green dots denote features from the original and perturbed images, respectively. Our contrastive-based encoder yields tightly aligned feature clusters regardless of visual noise, indicating highly invariant representations. Conversely, the features generated by PPO are highly scattered and disorganized, demonstrating a strong vulnerability to domain shifts.
Fig. 7 : Attention visualization in simulation. Our contrastive-based encoder consistently maintains a precise focus on essential geometric features, such as obstacles and targets, regardless of brightness, hue, or artificial color pad perturbations. Conversely, PPO demonstrates high vulnerability to domain shifts, often misdirecting its attention toward irrelevant textures or scattering across the background when image properties are altered.
Fig. 8 : Attention visualization in real-world environments. By combining 3DGS-based training environments with contrastive representation learning, GaussFly robustly anchors its attention on task-critical physical objects. Conversely, PPO is more easily distracted by visual interference as it lacks this noise-resilient feature extraction.
Fig. 9 : Ablation study on contrastive pre-training. We evaluate navigation performance by comparing our contrastively learned encoder against a baseline using frozen ImageNet weights (w/ IN). Our framework achieves superior overall performance. In contrast, the w/ IN variant leads to significant performance degradation, as standard classification pre-training fails to distill the task-relevant and noise-robust features required for complex visuomotor policies.
Fig. 10 : Ablation study on geometric constraints. We evaluate navigation performance by comparing our geometrically consistent 3DGS framework against a vanilla 3DGS baseline (w/o geo). Our framework achieves superior overall performance. In contrast, the w/o geo variant leads to increased collision rates and general performance degradation, as unconstrained 3DGS generates structural artifacts and depth inconsistencies that impair precise perception.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Traversing narrow gaps is challenging for autonomous quadrotors, especially when control commands come directly from high-dimensional visual observations. Existing end-to-end methods often rely on behavior cloning or full-rollout backpropagation through time (BPTT) via differentiable simulation, which can limit policy performance or incur high training costs. We propose a two-stage reinforcement learning framework for more efficient ego-centric visuomotor gap-traversal policy training, leveraging quasi-analytical policy gradients (QPG) via differentiable simulation and critic warm-starting. The framework utilizes QPG to avoid backpropagation through visual rendering, reducing computation and memory costs while improving sample efficiency. In the first stage, an expert actor and critic are trained using privileged observations, including gap geometry. Unlike prior gap-traversal approaches, our training utilizing QPG does not require resetting the agent along optimized reference trajectories. In the second stage, a visual policy is trained using binary gap masks from two ego-centric cameras and low-dimensional observations, while its privileged critic is warm-started from the first stage. This substantially improves training efficiency and traversal success compared with cold-starting the critic or using full-rollout BPTT. Our framework does not require retraining the expert actor when system parameters change, enabling more efficient generalization across drone platforms than state-of-the-art visual gap-traversal methods based on action supervision. The learned visual policy also generalizes to gaps with unseen shapes. Extensive real-world experiments further demonstrate robust gap traversal using binary masks rendered online. Beyond gap traversal, the proposed framework is generic and can be extended to other visuomotor robot learning tasks.
Embodied AI research is undergoing a shift toward vision-centric perceptual paradigms. While massively parallel simulators have catalyzed breakthroughs in proprioception-based locomotion, their potential remains largely untapped for vision-informed tasks due to the prohibitive computational overhead of large-scale photorealistic rendering. Furthermore, the creation of simulation-ready 3D assets heavily relies on labor-intensive manual modeling, while the significant sim-to-real physical gap hinders the transfer of contact-rich manipulation policies. To address these bottlenecks, we propose GS-Playground, a multi-modal simulation framework designed to accelerate end-to-end perceptual learning. We develop a novel high-performance parallel physics engine, specifically designed to integrate with a batch 3D Gaussian Splatting (3DGS) rendering pipeline to ensure high-fidelity synchronization. Our system achieves a breakthrough throughput of 10^4 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL. Additionally, we introduce an automated Real2Sim workflow that reconstructs photorealistic, physically consistent, and memory-efficient environments, streamlining the generation of complex simulation-ready scenes. Extensive experiments on locomotion, navigation, and manipulation demonstrate that GS-Playground effectively bridges the perceptual and physical gaps across diverse embodied tasks. Project homepage: https://gsplayground.github.io.
Prevailing 2D-centric visuomotor policies exhibit a pronounced deficiency in novel view generalization, as their reliance on static observations hinders consistent action mapping across unseen views. In response, we introduce GenSplat, a feed-forward 3D Gaussian Splatting framework that facilitates view-generalized policy learning through novel view rendering. GenSplat employs a permutation-equivariant architecture to reconstruct high-fidelity 3D scenes from sparse, uncalibrated inputs in a single forward pass. To ensure structural integrity, we design a 3D-prior distillation strategy that regularizes the 3DGS optimization, preventing the geometric collapse typical of purely photometric supervision. By rendering diverse synthetic views from these stable 3D representations, we systematically augment the observational manifold during training. This augmentation forces the policy to ground its decisions in underlying 3D structures, thereby ensuring robust execution under severe spatial perturbations where baselines severely degrade.