AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

Kautuk Astu; Naina Rabha; Yogesh Simmhan

cs.ROOct 7, 2026

AeroEval: Staged Program and Execution Validation for AI-Generated Drone Missions

Authors: Kautuk Astu, Naina Rabha, Yogesh Simmhan

Organizations: Department of Computational and Data Sciences, Indian Institute of Science, Bangalore 560012, India

Abstract

Large Language Models (LLMs) can generate drone programs from natural-language mission descriptions, but syntactically valid programs may still violate user intent, environmental constraints, and mission-level behavior. This problem is pronounced in cyber-physical applications, where correctness depends on the interaction among generated code, mobile sensing, environmental geometry, event-driven analytics, and physical execution. Existing drone code-generation systems primarily use prompt guardrails or simulator outcomes and provide limited failure localization. We present AeroEval, an agent-assisted middleware for staged validation of AI-generated drone missions. AeroEval combines deterministic program analysis with context-grounded LLM agents. It first validates program syntax, platform API usage, and mission intent, and then evaluates the realized behavior using execution trajectories, mission requirements, and environmental context. Each stage returns structured failure information for iterative regeneration. In our evaluation using 20 navigation tasks and five analytical mission types over AirSim and Gazebo simulators, AeroEval improves navigation success from 55% to 95%. In a stagewise ablation study, our Code and Trajectory Validators by themselves achieve mean run-level success rates of 44% and 56%, respectively, while the full AeroEval pipeline achieves 88%; the stages detect complementary failures in program structure, API usage, mission intent, obstacle avoidance, altitude, coverage, and event-driven transitions and the guided regeneration corrects for them. Across the main analytics missions, AeroEval increases aggregate run-level success from 34% for one-shot AeroGen to 88% within the regeneration budget. These results demonstrate the benefit of combining program-level and execution-grounded agentic validation for AI-generated drone applications in the evaluated environment.

Figures & tables

Panel of Figure 1 — Figure 1 : Impact and composition of staged mission validation. (a) Run-level success for one-shot AeroGen and AeroGen +AeroEval over 10 independent runs per analytical mission using o3-mini; aggregate error bars show Wilson score $95\%$ confidence intervals. (b) Mean run-level success of the Code Validator-only, Trajectory Validator-only, and full AeroEval pipeline with a budget of up to three regenerations.

Figure 2 : AeroEval architecture and validation lifecycle. Program Validation combines deterministic syntax and API checks with an agent-based intent check. Programs that pass execute in an external simulator, and the resulting trace is evaluated using either a reference-path comparator or a context-grounded validation agent. Failures return stage-specific diagnoses to the generator. Acceptance is scoped to the evaluated mission and environment. Dashed components and links are external or optional.

Property	Evidence	Validator
Syntax	Program	Deterministic
API availability	Program, API metadata	Deterministic
Mission phases	Program, specification	Intent Agent
Reference path	Waypoints	Comparator
Obstacle/corridor	Trace, world context	Context Agent
Coverage/orientation	Trace, mission region	Context Agent

Table 1 : Evidence and validation mechanisms used for program-level and execution-grounded mission properties.

Study	Generator	Environment	Missions	Budget
Navigation	CLG	AirSim	20 tasks	15 iterations
Analytical	AeroGen	Gazebo	5 types $\times$ 10 runs	15 iterations
Ablation	AeroGen	Gazebo	5 types $\times$ 10 runs	3 regenerations
Physical	AeroGen	Tello	3 types	supervised

Table 2 : Summary of the experimental configurations.

Panel of Figure 3 — Figure 3 : Contribution and dependability of the validation stages. Panel (a) reports mean $SR_{\mathrm{run}}$ across five analytical mission types, with 10 independent runs per mission and a common limit of three regeneration cycles for each configuration. Panel (b) reports expert agreement, repeated-trace consistency, and cross-model agreement for the Intent and Trajectory Validator; false-accept and false-reject counts are shown below the panel.

Panel of Figure 4 — Figure 4 : End-to-end mission reliability using o3-mini. Panel (a) compares native CLG with CLG +AeroEval (AE) over 20 navigation tasks. Panel (b) reports $SR_{\mathrm{run}}$ for five analytical mission types over 10 independent runs per mission; error bars show 95% confidence intervals.

Panel of Figure 5 — Figure 5 : Failure localization and physical applicability of AeroEval. Panel (a) shows measured transitions of generated analytical candidates through the validation lifecycle using o3-mini; backward flows indicate regeneration and DNF denotes runs that do not converge within the refinement budget. Panel (b) shows representative simulated and physical trajectories for missions accepted by AeroEval and executed on a DJI Ryze Tello under expert supervision.

Model	$SR_{\mathrm{run}}$	Validation latency	Cached input	Cost/iteration
o3-mini	$44/50$ ( $88\%$ )	$49$ – $585$ s	$\approx 83\%$	$\approx\$ 0.03$
Gemini-3.1-Pro	$47/50$ ( $94\%$ )	$7$ – $1499$ s	$\approx 85\%$	$\approx\$ 0.07 $–$ 0.14$

Table 3 : Practicality and model sensitivity for analytical missions. Values summarize the five mission types over converged runs.

Figure 6 : Median end-to-end latency composition for one refinement iteration across the analytical missions using o3-mini. Labels report the total median duration, while segments show the contributions of code generation, Code Validation, simulated execution, and Trajectory Validation.

Explore similar work

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit
Aug 17, 2026Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2LLM Safety Benchmarks AI Agent Safety Benchmarks
School of Aeronautic Science and Engineering, Beihang University
Say the Mission, Execute the Swarm: Agent-Enhanced LLM Reasoning in the Web-of-Drones
May 5, 2026Andrea Iannoli, Lorenzo Gigli, Luca Sciullo +2Aerial Robotics Multi-Robot Systems
Department of Computer Science and Engineering University of Bologna Bologna, Italy
Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
Sep 1, 2026Jaewoo Park, Minyoung Lee, Sukmin Seo +11Aerial Robotics Language Model-Based Control
Drone AI Team, NAVER Cloud

Aug 17, 2026, cs.AIJ/K move · Enter open · S save

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit

Yuchen Yuan, Zhenghuang Wu, Yuangan Li, Liang Ma+1

School of Aeronautic Science and Engineering, Beihang University

LLM Safety Benchmarks AI Agent Safety Benchmarks LLM Agent Reliability

Open paper AskPDF arXiv

Aviation knowledge question answering cannot directly assess the operational effectiveness and safety compliance of large language models throughout aircraft emergency procedures. We introduce AeroCopilotBench and its executable cockpit environment, ACOE, which define state-transition rules, task goals, and trajectory-level safety constraints based on aircraft-specific Pilot's Operating Handbooks (POHs). The benchmark comprises 12 scenario templates and 73 tasks across two aircraft types, evaluating task completion, safety compliance, execution discipline, and repeatability. Across repeated evaluations of 12 models, the highest safety-gated success rate is 72.6%. Most failed episodes achieve all critical goals but do not satisfy all remaining terminal goals. Across repeated runs, models still omit steps that they execute in other runs of the same task. Analysis of failed trajectories further reveals that some reasons for actions that conflict with the aircraft's POH recur across models. These results expose shortcomings in complete procedure execution, consistency across runs, and aircraft-specific emergency response.

Abstract

Explore similar work

AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit

Say the Mission, Execute the Swarm: Agent-Enhanced LLM Reasoning in the Web-of-Drones

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching