AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

Kautuk Astu; Naina Rabha; Yogesh Simmhan

cs.ROOct 7, 2026

AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

Authors: Kautuk Astu, Naina Rabha, Yogesh Simmhan

Organizations: Department of Computational and Data Sciences, Indian Institute of Science, Bangalore 560012, India

Abstract

Large Language Models (LLMs) can generate drone programs from natural-language mission descriptions, but syntactically valid programs may still violate user intent, environmental constraints, and mission-level behavior. This problem is pronounced in cyber-physical applications, where correctness depends on the interaction among generated code, mobile sensing, environmental geometry, event-driven analytics, and physical execution. Existing drone code-generation systems primarily use prompt guardrails or simulator outcomes and provide limited failure localization. We present AeroEval, an agent-assisted middleware for staged validation of AI-generated drone missions. AeroEval combines deterministic program analysis with context-grounded LLM agents. It first validates program syntax, platform API usage, and mission intent, and then evaluates the realized behavior using execution trajectories, mission requirements, and environmental context. Each stage returns structured failure information for iterative regeneration. In our evaluation using 20 navigation tasks and five analytical mission types over AirSim and Gazebo simulators, AeroEval improves navigation success from 55% to 95%. In a stagewise ablation study, our Code and Trajectory Validators by themselves achieve mean run-level success rates of 44% and 56%, respectively, while the full AeroEval pipeline achieves 88%; the stages detect complementary failures in program structure, API usage, mission intent, obstacle avoidance, altitude, coverage, and event-driven transitions and the guided regeneration corrects for them. Across the main analytics missions, AeroEval increases aggregate run-level success from 34% for one-shot AeroGen to 88% within the regeneration budget. These results demonstrate the benefit of combining program-level and execution-grounded agentic validation for AI-generated drone applications in the evaluated environment.

Figures & tables

Panel of Figure 1 — Figure 1 : Impact and composition of staged mission validation. (a) Run-level success for one-shot AeroGen and AeroGen +AeroEval over 10 independent runs per analytical mission using o3-mini; aggregate error bars show Wilson score $95\%$ confidence intervals. (b) Mean run-level success of the Code Validator-only, Trajectory Validator-only, and full AeroEval pipeline with a budget of up to three regenerations.

Figure 2 : AeroEval architecture and validation lifecycle. Program Validation combines deterministic syntax and API checks with an agent-based intent check. Programs that pass execute in an external simulator, and the resulting trace is evaluated using either a reference-path comparator or a context-grounded validation agent. Failures return stage-specific diagnoses to the generator. Acceptance is scoped to the evaluated mission and environment. Dashed components and links are external or optional.

Property	Evidence	Validator
Syntax	Program	Deterministic
API availability	Program, API metadata	Deterministic
Mission phases	Program, specification	Intent Agent
Reference path	Waypoints	Comparator
Obstacle/corridor	Trace, world context	Context Agent
Coverage/orientation	Trace, mission region	Context Agent

Table 1 : Evidence and validation mechanisms used for program-level and execution-grounded mission properties.

Study	Generator	Environment	Missions	Budget
Navigation	CLG	AirSim	20 tasks	15 iterations
Analytical	AeroGen	Gazebo	5 types $\times$ 10 runs	15 iterations
Ablation	AeroGen	Gazebo	5 types $\times$ 10 runs	3 regenerations
Physical	AeroGen	Tello	3 types	supervised

Table 2 : Summary of the experimental configurations.

Panel of Figure 3 — Figure 3 : Contribution and dependability of the validation stages. Panel (a) reports mean $SR_{\mathrm{run}}$ across five analytical mission types, with 10 independent runs per mission and a common limit of three regeneration cycles for each configuration. Panel (b) reports expert agreement, repeated-trace consistency, and cross-model agreement for the Intent and Trajectory Validator; false-accept and false-reject counts are shown below the panel.

Panel of Figure 4 — Figure 4 : End-to-end mission reliability using o3-mini. Panel (a) compares native CLG with CLG +AeroEval (AE) over 20 navigation tasks. Panel (b) reports $SR_{\mathrm{run}}$ for five analytical mission types over 10 independent runs per mission; error bars show 95% confidence intervals.

Panel of Figure 5 — Figure 5 : Failure localization and physical applicability of AeroEval. Panel (a) shows measured transitions of generated analytical candidates through the validation lifecycle using o3-mini; backward flows indicate regeneration and DNF denotes runs that do not converge within the refinement budget. Panel (b) shows representative simulated and physical trajectories for missions accepted by AeroEval and executed on a DJI Ryze Tello under expert supervision.

Model	$SR_{\mathrm{run}}$	Validation latency	Cached input	Cost/iteration
o3-mini	$44/50$ ( $88\%$ )	$49$ – $585$ s	$\approx 83\%$	$\approx\$ 0.03$
Gemini-3.1-Pro	$47/50$ ( $94\%$ )	$7$ – $1499$ s	$\approx 85\%$	$\approx\$ 0.07 $–$ 0.14$

Table 3 : Practicality and model sensitivity for analytical missions. Values summarize the five mission types over converged runs.

Figure 6 : Median end-to-end latency composition for one refinement iteration across the analytical missions using o3-mini. Labels report the total median duration, while segments show the contributions of code generation, Code Validation, simulated execution, and Trajectory Validation.

Explore similar work

Aug 17, 2026cs.AI

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit

Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2

School of Aeronautic Science and Engineering, Beihang University

Pilot Large Language Model Agents Task Success Rate Trajectory-Level Credit

Ask PDF

May 5, 2026cs.AI

Say the Mission, Execute the Swarm: Agent-Enhanced LLM Reasoning in the Web-of-Drones

Large Language Models (LLMs) are increasingly explored as high-level reasoning engines for cyber-physical systems, yet their application to real-time UAV swarm management remains challenging due to heterogeneous interfaces, limited grounding, and the need for long-running closed-loop execution. This paper presents a mission-agnostic, agent-enhanced LLM framework for UAV swarm control, where users express mission objectives in natural language and the system autonomously executes them through grounded, real-time interactions. The proposed architecture combines an LLM-based Agent Core with a Model Context Protocol (MCP) gateway and a Web-of-Drones abstraction based on W3C Web of Things (WoT) standards. By exposing drones, sensors, and services as standardized WoT Things, the framework enables structured tool-based interaction, continuous state observation, and safe actuation without relying on code generation. We evaluate the framework using ArduPilot-based simulation across four swarm missions and six state-of-the-art LLMs. Results show that, despite strong reasoning abilities, current general-purpose LLMs still struggle to achieve reliable execution - even for simple swarm tasks - when operating without explicit grounding and execution support. Task-specific planning tools and runtime guardrails substantially improve robustness, while token consumption alone is not indicative of execution quality or reliability.

Andrea Iannoli, Lorenzo Gigli, Luca Sciullo +2

Department of Computer Science and Engineering University of Bologna Bologna, Italy

Unmanned Aerial Vehicle Swarms Swarms Autonomous Drones LLM Reasoning Strategies Model Context Protocol

Ask PDF

Sep 1, 2026cs.RO

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.

Jaewoo Park, Minyoung Lee, Sukmin Seo +11

Drone AI Team, NAVER Cloud

Autonomous Drones Multimodal Large Language Models Small Reasoning Model War

Ask PDF