A Model for Mediating Multi-Modal Human Intent into Safe Maneuvers for UAVs

In IoT-enabled smart-city settings, Uncrewed Aerial Vehicles (UAVs) are evolving from passive sensing platforms into cyber-physical decision makers that must respect operational rules under degraded observations and ambiguous language. Existing UAV and multimodal benchmarks cover aerial perception, navigation, collaboration, and task reasoning, but rarely test whether physical evidence, protocol constraints, and action risk stay coupled at critical decisions. We introduce MulRobBench, an offline, protocol-conditioned benchmark for Vision-Language-Action (VLA) UAV agents that links real UAV multimodal observations, protocol-level security-policy constraints, and action-level cyber-physical safety within an auditable decision contract. The evaluation set contains 3,024 samples spanning 17 task-taxonomy nodes and 12 metric scoring dimensions, organized around context understanding, multimodal evidence arbitration, degradation-aware reasoning, and risk-aware action planning. MulRobBench reports controlled semantic scores alongside strict structural diagnostics for policy compliance, formatting, unsafe actions, parsing, and dimension-level validity. Across 17 uniformly audited models, the best semantic protocol-decision score reaches 0.5141 and the best strict mean scoring-dimension accuracy reaches 0.1599. A matched 20-anchor modality-removal study changes 4-15 action selections per model, showing both visual and textual inputs influence decisions while the strongest input condition varies across metrics. Per-dimension and conditional analyses identify modality-trust selection, constraint extraction, strong glare, missing data, and high-entropy operator shorthand as principal sources of action instability. The central challenge is thus stable coupling of degraded evidence, security-policy constraints, and risk-bearing action, not isolated scene recognition.

Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed agent iterations are often acceptable. UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic reasoning from onboard execution and security/safety enforcement. Slow cloud or edge reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated UAV actions.

Unmanned aerial vehicles (UAVs) extend embodied intelligence into continuous three-dimensional space, where perception, reasoning, physical embodiment, and action are tightly coupled through flight and environmental interaction. Recent advances in foundation models, world models, and AI agents are shifting UAV autonomy from task-specific perception and control toward systems that can interpret human intent, understand open environments, reason about physical consequences, and organize complex behaviors under embodiment and flight-dynamic constraints. We characterize this emerging paradigm as UAV embodied intelligence (UAV EI) and distinguish it from its system realization, the embodied-intelligent UAV (EI UAV). To provide a unified view of the field, we introduce a 5+5 framework that describes UAV EI through five capability dimensions and EI UAVs through five architectural layers spanning physical embodiment, general cognition, embodied skills, external interaction, and system harnessing. Based on this framework, we systematically review recent progress in embodied morphology, embodied perception, world models, embodied planning, vision-language navigation, embodied manipulation, and embodied collaboration. We further identify long-horizon autonomy, predictive physical reasoning, test-time skill acquisition, and autonomous capability evolution as key challenges toward more general aerial embodied intelligence. Finally, we argue that harnessing physical-digital AI agents, through persistent coupling of digital intelligence with physical sensing, dynamics, action, and feedback, provides a system-level pathway toward adaptive and continuously evolving UAV autonomy. Project resources are available at our project website and GitHub repository.