Organizations: Shanghai Artificial Intelligence Laboratory · Northwestern Polytechnical University · Beihang University · Shanghai Jiao Tong University · University of Science and Technology of China · Zhejiang University · Institute of Artificial Intelligence of China Telecom (TeleAI)
Human cognition flexibly draws on a shared perceptual foundation to understand environments, attend to referenced entities, reason about spatial relationships, and act. Endowing aerial robots with an analogous functional organization remains challenging, as existing systems are often specialized for individual capabilities, dependent on offboard computation, or unable to sustain closed-loop operation on resource-constrained platforms. We present OpenFlyV2, a unified onboard vision-language-action architecture that uses a shared vision-language representation to support semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control within a single model. A compact action expert converts the shared multimodal context into short-horizon flight waypoints, while motion-aware temporal memory and action priors reduce the computational cost of closed-loop execution. We construct \dataset{}, a simulation-based dataset comprising approximately 70,000 navigation trajectories and 60,000 vision-language instruction--response pairs. OpenFlyV2 generates action chunks, each comprising 12 waypoints, entirely onboard at 10 Hz using an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to physical quadrotor flight, achieving a 48% navigation success rate across four real-world environments without real-world aerial training data. Extensive results demonstrate that broad vision-language competence and continuous closed-loop flight can coexist within a single model on a resource-constrained aerial platform. Code will be made publicly available at https://github.com/Ideal-ljl/AeroBrain.
Figures & tables
Figure 2: Shared environments and data composition of OpenFlyV2-130K. (A) Thirteen environments supporting both navigation and VQA: four AirSim environments, two Unreal Engine city environments, and seven 3DGS reconstructions of real campuses and post-disaster ruins. (B) Distribution of 72,352 navigation trajectories across the environments. (C) Distribution of 60,006 VQA pairs across color recognition, object description, distance estimation, spatial relationship reasoning, and visual grounding.
Figure 3: Aerial vision-language understanding and spatial reasoning with OpenFlyV2. (A) Distance estimation for the building marked in green: 179 m compared with a reference distance of approximately 180 m. (B) Description of the building within the yellow box. (C) Relative-distance reasoning identifies the red-box building as closer to the camera than the blue-box building. (D) Localization of a language-described target. The predicted bounding box (dashed orange) and reference box (solid cyan) have an intersection over union (IoU) of 0.753.
Figure 4: Quantitative evaluation of aerial understanding and navigation. (A) Comparison with general-purpose vision-language models on color recognition, description generation, visual grounding, distance estimation, and spatial relationship reasoning. Visual grounding is reported as accuracy at an IoU threshold of 0.5 ([email protected]), and distance estimation is shown as relative error. (B) Navigation performance in seen and unseen simulated environments, measured by success rate (SR), oracle success rate (OSR), success weighted by path length (SPL), and navigation error (NE). The navigation baselines were trained on our aerial navigation dataset and evaluated on the same 800 seen and 300 unseen episodes as OpenFlyV2. Success requires an explicit stop within 25 m of the goal. Blue and orange bars in (B) indicate seen and unseen environments, respectively. Arrows indicate the direction of better performance.
Figure 5: Onboard aerial vision-language navigation. Three physical flight trials performed by OpenFlyV2 on a DJI M30T, with all navigation inference executed on an onboard NVIDIA Jetson Orin NX.
Figure 6: Architecture of OpenFlyV2. (A) A shared vision-language backbone encodes the current image, instruction, and memory tokens to condition action generation. (B) Motion-Adaptive Memory (MAM) compresses historical action segments and their visual anchors into compact event-level tokens. (C) The Motion-Structured Prior (MSP) selects and adjusts a data-derived motion prototype to construct the action source distribution. The action expert transforms a sample from this distribution into the final action chunk through conditional flow matching.
Aerial vision-language navigation (VLN) requires an embodied agent to integrate visual evidence over time, plan future actions, and determine when it has reached a navigation goal under partial observability. Although recent VLA models offer a promising perception-to-action paradigm, adapting them to aerial navigation remains challenging due to limited historical context, short planning horizons, and unreliable implicit termination. To address these challenges, we propose DreamFly, a diffusion-based aerial VLN framework built on Dream-VLA. DreamFly introduces a causally aligned historical memory that augments the current visual representation using only observations preceding the current decision step, enabling temporal reasoning without future information leakage. We further formulate navigation as receding-horizon diffusion planning, where the policy predicts a K-step action chunk but executes only the first action before replanning. This plan-K, execute-one strategy uses future actions as auxiliary planning targets while preserving closed-loop visual feedback. Finally, LiteStop estimates the stop probability directly from action logits at the initial all-mask state, decoupling explicit termination from action generation. Experiments on the OpenFly benchmark demonstrate consistent improvements in seen and unseen environments. DreamFly achieves 32.04%/29.46% SR and 28.22%/23.54% SPL on the test-seen/test-unseen splits, respectively, outperforming all compared methods on both metrics while attaining the lowest navigation error. These results demonstrate the effectiveness of jointly modeling historical context, future action structure, and explicit termination for aerial VLN.
Yan Deng, Fei Xu
School of Electronic Information Engineering, Xi’an Technological University, Xi’an, China · School of Computer Science and Engineering, Xi’an Technological University, Xi’an, China
Running vision-language navigation fully onboard an aerial robot is hard, since grounding, planning, and control must share limited compute and a single-stage error is difficult to isolate in flight. End-to-end aerial policies fuse these stages into one network, giving up the observability and safety checks a modular stack keeps available. We propose VLN on the Fly, an onboard stack that keeps grounding, planning, and control as separate, inspectable stages. A quantized VLM grounds an instruction to a coarse image cell, depth lifts it to a 3D goal, a fast B-spline planner returns a feasible trajectory, and a pretrained reinforcement learning policy tracks it to motor commands across quadrotors. Across 15 onboard flights over three everyday referents in a controlled indoor volume, the stack reaches the target in 13 of 15 trials with 5.72 cm mean goal error and 39.3% average GPU utilization. In 6 additional cluttered-environment trials, the stack tracks collision-free trajectories under onboard perception gating.
Marco S. Tayar, Felipe Tommaselli, Gianluca Capezzuto +6
We introduce ActiveFly-Bench, the first benchmark to bridge cyberspace reasoning and physical-world interaction for UAV embodied perception. The benchmark decomposes active perception into three hierarchical tasks: Aerial Embodied Question Answering (Air-EQA), Observation Behavior Planning (OBP), and Fine-grained Language-guided UAV Control (FLUC), explicitly connecting high-level task understanding, behavior planning, and low-level control. The datasets are collected from both real-world and simulated outdoor environments for training and evaluation. We further develop ActiveFly, a closed-loop UAV agent that integrates visual-language reasoning with fine-grained control, and deploy it on a physical UAV platform. Experiments with representative VLMs and VLA models show that current UAV agents still struggle with behavior planning, viewpoint adjustment, and robust task completion in active perception. These results establish ActiveFly-Bench as a new testbed for embodied aerial intelligence.