Organizations: Shanghai Artificial Intelligence Laboratory · Northwestern Polytechnical University · Beihang University · Shanghai Jiao Tong University · University of Science and Technology of China · Zhejiang University · Institute of Artificial Intelligence of China Telecom (TeleAI)
Human cognition flexibly draws on a shared perceptual foundation to understand environments, attend to referenced entities, reason about spatial relationships, and act. Endowing aerial robots with an analogous functional organization remains challenging, as existing systems are often specialized for individual capabilities, dependent on offboard computation, or unable to sustain closed-loop operation on resource-constrained platforms. We present OpenFlyV2, a unified onboard vision-language-action architecture that uses a shared vision-language representation to support semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control within a single model. A compact action expert converts the shared multimodal context into short-horizon flight waypoints, while motion-aware temporal memory and action priors reduce the computational cost of closed-loop execution. We construct \dataset{}, a simulation-based dataset comprising approximately 70,000 navigation trajectories and 60,000 vision-language instruction--response pairs. OpenFlyV2 generates action chunks, each comprising 12 waypoints, entirely onboard at 10 Hz using an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to physical quadrotor flight, achieving a 48% navigation success rate across four real-world environments without real-world aerial training data. Extensive results demonstrate that broad vision-language competence and continuous closed-loop flight can coexist within a single model on a resource-constrained aerial platform. Code will be made publicly available at https://github.com/Ideal-ljl/AeroBrain.
Figures & tables
Figure 2: Shared environments and data composition of OpenFlyV2-130K. (A) Thirteen environments supporting both navigation and VQA: four AirSim environments, two Unreal Engine city environments, and seven 3DGS reconstructions of real campuses and post-disaster ruins. (B) Distribution of 72,352 navigation trajectories across the environments. (C) Distribution of 60,006 VQA pairs across color recognition, object description, distance estimation, spatial relationship reasoning, and visual grounding.
Figure 3: Aerial vision-language understanding and spatial reasoning with OpenFlyV2. (A) Distance estimation for the building marked in green: 179 m compared with a reference distance of approximately 180 m. (B) Description of the building within the yellow box. (C) Relative-distance reasoning identifies the red-box building as closer to the camera than the blue-box building. (D) Localization of a language-described target. The predicted bounding box (dashed orange) and reference box (solid cyan) have an intersection over union (IoU) of 0.753.
Figure 4: Quantitative evaluation of aerial understanding and navigation. (A) Comparison with general-purpose vision-language models on color recognition, description generation, visual grounding, distance estimation, and spatial relationship reasoning. Visual grounding is reported as accuracy at an IoU threshold of 0.5 ([email protected]), and distance estimation is shown as relative error. (B) Navigation performance in seen and unseen simulated environments, measured by success rate (SR), oracle success rate (OSR), success weighted by path length (SPL), and navigation error (NE). The navigation baselines were trained on our aerial navigation dataset and evaluated on the same 800 seen and 300 unseen episodes as OpenFlyV2. Success requires an explicit stop within 25 m of the goal. Blue and orange bars in (B) indicate seen and unseen environments, respectively. Arrows indicate the direction of better performance.
Figure 5: Onboard aerial vision-language navigation. Three physical flight trials performed by OpenFlyV2 on a DJI M30T, with all navigation inference executed on an onboard NVIDIA Jetson Orin NX.
Figure 6: Architecture of OpenFlyV2. (A) A shared vision-language backbone encodes the current image, instruction, and memory tokens to condition action generation. (B) Motion-Adaptive Memory (MAM) compresses historical action segments and their visual anchors into compact event-level tokens. (C) The Motion-Structured Prior (MSP) selects and adjusts a data-derived motion prototype to construct the action source distribution. The action expert transforms a sample from this distribution into the final action chunk through conditional flow matching.
School of Electronic Information Engineering, Xi’an Technological University, Xi’an, China · School of Computer Science and Engineering, Xi’an Technological University, Xi’an, China