QuadAgent: A Responsive Agent System for Vision-Language Guided Quadrotor Agile Flight

Ao Zhuang; Feng Yu; Tianbao Zhang; Linzuo Zhang; Danping Zou

cs.ROApr 3, 2026

QuadAgent: A Responsive Agent System for Vision-Language Guided Quadrotor Agile Flight

Authors: Ao Zhuang, Feng Yu, Tianbao Zhang, Linzuo Zhang, Danping Zou

Abstract

We present QuadAgent, a training-free agent system for agile quadrotor flight guided by vision-language inputs. Unlike prior end-to-end or serial agent approaches, QuadAgent decouples high-level reasoning from low-level control using an asynchronous multi-agent architecture: Foreground Workflow Agents handle active tasks and user commands, while Background Agents perform look-ahead reasoning. The system maintains scene memory via the Impression Graph, a lightweight topological map built from sparse keyframes, and ensures safe flight with a vision-based obstacle avoidance network. Simulation results show that QuadAgent outperforms baseline methods in efficiency and responsiveness. Real-world experiments demonstrate that it can interpret complex instructions, reason about its surroundings, and navigate cluttered indoor spaces at speeds up to 5 m/s.

Figures & tables

Fig. 1: Complex reasoning using our agent system. Given the identical conditional task, the left and right panels illustrate the agent’s behavior depending on observations: the agent navigates to the badminton net through random obstacles when "yellow sign number 7" is observed (left), and proceeds to the white table in front of the yellow frame when it is not (right).

Fig. 2: System Overview. In Foreground Workflow Agents, the orchestrator monitors events ( $\epsilon_{usr},\epsilon_{phy}$ ) in the idle state and routes tasks to the planner or executor. Both the executor and pre-executor autonomously call mnemonic, navigation, or perceptual skills from the skill library. Notably, the navigation skill triggers $\epsilon_{path}$ to drive the onboard physical layer state machine for actuation, transitioning among the following states: $q_{hover}$ (hovering), $q_{align\_s}$ (in-place rotation to align with the initial path heading), $q_{avoid}$ (path tracking via collision avoidance policy), and $q_{align\_e}$ (in-place rotation to the final target orientation). The Impression Graph at the top right underpins mnemonic and navigation skills as the scene prior representation. Images in the Impression Graph and the physical layer show the arena and UAV used in real-world experiments.

Fig. 3: Timelines of Typical Cases. In each sub-image, the upper rows depict the Foreground Workflow Agents’ lifecycle, where $q_{idle}$ marks the idle state, $q_{orc}$ indicates the orchestrator’s active routing phase, and $q_{plan}/q_{exec}$ denote the engagement of the planner and executor, respectively. The lower rows track the physical layer states. (a) Our suspend-and-resume protocol yields the agent to the idle state ( $q_{idle}$ ) immediately after triggering $\epsilon_{path}$ , enabling “Chatting-while-Flying”, unlike the Blocking Baseline (b) where the agent is blocked until physical completion. (c) Informational Queries are resolved in parallel with the ongoing flight state ( $q_{avoid}$ ). (d) Conflicting Commands trigger preemptive replanning, issuing a new $\epsilon_{path}$ to immediately redirect the UAV.

Fig. 4: Data Flow of Background Pre-execution. The pipeline branches from the task registry (center left). While the Foreground Workflow Agents execute the task from the first one, upcoming sub-tasks are assigned across Background Agents (bottom) for pre-execution using mnemonic skills. The retrieved context is cached in a shared buffer (center right) and dynamically injected into the prompt of the Executor (top right).

Fig. 5: Impression Graph Construction Pipeline. (a) Topological Connectivity: The depth map is tessellated into patches and projected into geometric pyramidal frustums. Edges $(n_{i},n_{j})$ are established solely if the volumetric intersection of their frustums exceeds $\sigma_{vol}$ . (b) Semantic Generation: $I_{rgb}$ is segmented into depth-stratified views (Near, Far, Full) and concatenated into a composite input for the VLM.

Method	SR (%) $\uparrow$	OSR (%) $\uparrow$	Prog. (%) $\uparrow$	SPL $\uparrow$	TET (s) $\downarrow$	HT (s) $\downarrow$	IA (%) $\uparrow$
Serial Plan-ReAct [ 7 ]	78.3	83.3	92.0	57.2	470.2	249.9	0.0
Scene Graph Agent [ 24 ]	15.0	18.3	40.7	11.9	338.6	170.8	24.9
Traj NavGraph [ 28 ]	76.7	85.0	92.2	49.8	379.2	125.8	57.5
w/o Look-ahead	76.7	81.7	90.3	55.5	445.2	226.0	38.2
w/o On-demand Retrieval	60.0	68.3	78.8	42.5	335.0	113.7	52.6
QuadAgent (Ours)	80.0	86.7	93.3	58.6	353.9	118.0	58.3

TABLE I: Quantitative Analysis in Simulation.

\uparrow

: higher is better,

\downarrow

: lower is better. The best results are highlighted in bold .

Fig. 6: Qualitative evaluation of the QuadAgent in real-world environments. (a) Experiment 1: The UAV executes a long-horizon conditional task. The trajectory highlights an agile avoidance of unmapped obstacles (red box) and the timeline illustrates the autonomous sequential completion of sub-tasks. (b) Experiment 2: The agent utilizes the topological connectivity of the Impression Graph to identify and execute a spatial shortcut (solid blue line) to Goal 1 compared with the longer one (dashed blue line). (c) Experiment 3: During the navigation to Goal 1, a conflicting user input triggers an immediate trajectory update towards Goal 2 (blue line), preempting the original planned path to Goal 1 (dark green dashed line). While navigating to Goal 3, the agent processes an informational user input, answering the user in parallel without interrupting the physical flight. Note: Text color-coding and goal annotations are for visualization only. In practice, the agent receives a single unified instruction and infers targets autonomously.

Explore similar work

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
Sep 1, 2026Jaewoo Park, Minyoung Lee, Sukmin Seo +11Aerial Robotics Language Model-Based Control
Drone AI Team, NAVER Cloud
AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight
Jul 16, 2026Xinhong Zhang, Qiyuan Zhu, Yubo Huang +8Robotic Control Aerial Robotics
School of Automation, Beijing Institute of Technology · School of Mechanical Engineering, Beijing Institute of Technology · Independent researcher
Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation
Jun 5, 2026Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6UAV Navigation Vision-Language Navigation
Colab, Beihang University · National University of Singapore · Meituan

Sep 1, 2026J/K move · Enter open · S save

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim+10

Drone AI Team, NAVER Cloud

Aerial Robotics Language Model-Based Control Multimodal Large Language Models Vision-Language-Action Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.

Abstract

Explore similar work

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation