CAVEAT: Recurrent Multimodal Diffusion Planning for Mapless Aerial Exploration
Authors: Steven Visch, Nicolò Botteghi, Antonio Franchi, Barbara Bazzana
Organizations: Robotics and Mechatronics group, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, 7500AE Enschede, The Netherlands. · Modelling and Scientific Computing Laboratory (MOX), Politecnico di Milano. · Department of Computer, Control and Management Engineering, Sapienza University of Rome, 00185 Rome, Italy.
Can exploratory UAV waypoint sequences be generated from multimodal onboard observations and a fixed-dimensional recurrent internal state without maintaining a persistent global map in the deployed policy? We investigate this question through CAVEAT, a diffusion policy conditioned on a recurrent internal state updated from fused LiDAR, visual, and pose features and trained from trajectories generated by the map-based FUELv2 expert. Rolling inference partially warm-starts consecutive predictions, while a temporary local signed distance field provides heuristic obstacle guidance. Simulation results evaluate both inference mechanisms and compare CAVEAT with its demonstration-generating expert. Proof-of-concept experiments on a Flyability Elios 3 demonstrate partial exploration of a previously unseen indoor environment and target-directed visual servoing using a separately trained policy.
Figures & tables
Fig. 1 : Overview of CAVEAT. Multimodal observations update a recurrent internal state that conditions diffusion-based waypoint generation. Rolling inference partially warm-starts consecutive predictions, while a temporary local SDF provides heuristic obstacle guidance.
Fig. 2 : Architecture of the CAVEAT exploration policy. Modality-specific encoders process the current LiDAR, visual, and pose observations. Their fused features update a recurrent internal state that conditions diffusion-based waypoint generation through feature-wise linear modulation.
Fig. 3 : Gazebo environment families used to collect the expert exploration demonstrations. Environments Atrain – Dtrain correspond to panels (a)–(d), respectively.
Fig. 4 : Gazebo test environments. Environments Atest – Btest correspond to panels (a)–(b), respectively.
Trajectory statistic
Steps
Environment
Trajectories
Mean
74.65
Atrain
1000
Median
79.00
Btrain
700
Standard deviation
22.64
Ctrain
400
Minimum
1
Dtrain
574
Maximum
114
Total
2674
TABLE I : Exploration-demonstration dataset, comprising 2,674 trajectories and 199,603 recorded time steps.
Fig. 5 : Occupancy coverage versus distance traveled in simulation. Panels (a)–(b) compare independent and rolling initialization; panels (c)–(d) compare SDF-guidance thresholds dmin=0.5m and dmin=1.0m ; panels (e)–(f) compare CAVEAT and FUELv2. Curves represent the average exploration trajectory across three to four runs; shaded regions represent ±1 standard deviation across the evaluation trajectories.
Fig. 6 : Comparison of CAVEAT and FUELv2, over the training environment Btrain .
Fig. 7 : Two views of the previously unseen real-world test environment.
Fig. 8 : Real-world exploration trajectories obtained with DDPM and DDIM sampling. Black lines indicate the principal environment boundaries, colored curves show the UAV trajectories, triangles indicate heading, and green and red markers denote the start and end positions.
Fig. 9 : Target-directed visual-servoing proof of concept. Three trajectories approach the wall-mounted marker from different initial positions. Triangles indicate UAV heading, black points show the LiDAR-observed wall, and the inset images show representative camera observations.
LiDAR-based unmanned aerial vehicle (UAV) exploration builds maps by continually selecting where to observe next. However, decisions based on the measured map provide limited foresight into spatial continuations behind occlusions, leaving potentially informative directions unrecognized. We present WOLF, a world-model-guided framework that predicts future observations to enhance autonomous exploration. In the training stage, a recurrent world model learns observation dynamics from exploration trajectories, with recurrent memory retaining the spatial context needed to interpret partial observations across successive views. Building on this context, the model combines observation history with candidate motions during exploration to predict local occupancy and visibility. To guide further sensing, a predictive frontier generation mechanism then aligns and fuses these predictions using confidence, branch agreement, and observation quality to identify promising regions. The resulting predictive frontiers join measured ones to guide geometric viewpoint selection and trajectory generation, while new scans update subsequent predictions. In simulations, our method reduces mean terminal time by 10.9% relative to EPIC in Garage at comparable coverage and increases mean coverage from 42.12% to 98.35% in Tunnel. Real-world experiments further demonstrate onboard deployment of the learned model for online inference during physical flight.
Yuyang Tian, Penghui Yang, Pengyuan Wu +7
University of Science and Technology of China, Hefei, China. · Shanghai Artificial Intelligence Laboratory, Shanghai, China. · East China University of Science and Technology, Shanghai, China. +3
Autonomous UAV navigation is conventionally solved by pipelines that separate perception, mapping, and planning into distinct stages, which propagates errors, accumulates latency, and requires environment-specific retuning. End-to-end generative models remove these interfaces by mapping raw observations directly to trajectories, but inherit a subtle failure mode: trained on clean data, they cannot recognise when an observation is unreliable, and treat degraded regions such as glass, mirrors, and overexposed surfaces as valid evidence for planning. We present a reliability-aware diffusion planner for 3D UAV navigation. It conditions trajectory generation on the observation together with a scene-level reliability heatmap that marks where perception cannot be trusted, produced by a lightweight network that distils the open-vocabulary reasoning of a vision-language model within the real-time planning budget. To generalise to unseen environments without retraining, we steer the denoising process with a differentiable two-stage ESDF cost that treats physical obstacles from depth and virtual obstacles from highly unreliable regions on equal footing. In simulation and on a real quadrotor, our planner produces markedly safer trajectories than a state-of-the-art diffusion baseline, reducing the obstacle-violation rate from 40.3% to 9.6% and raising the mean reliability of traversed regions from 0.588 to 0.925. Ablating the reliability term alone drops mean reliability from 0.898 to 0.783, confirming it as the decisive component, while distillation runs the framework up to 2 times faster than the full vision-language model.
Vision-based UAV navigation becomes challenging when navigation targets are distributed across complementary camera views and cannot be reliably observed from a single viewpoint. We propose AgenticDiffusion, an agentic multi-view UAV navigation framework that semantically coordinates first-person-view (FPV) and top-view observations for mission-level navigation. Given a natural-language instruction, AgenticDiffusion identifies the requested targets, selects the most appropriate camera view for each navigation task, determines the corresponding navigation goal, and invokes the appropriate view-conditioned diffusion planner for trajectory generation. The resulting trajectories are executed using Nonlinear Model Predictive Control (NMPC). AgenticDiffusion was evaluated in four real-world indoor scenarios, achieving an overall mission success rate of 80% across 40 physical-flight trials. In mixed-visibility scenarios, where the requested targets were distributed across FPV and top-view observations, coordinated multi-view navigation reduced average mission time by 50.8% relative to FPV-only navigation and by 26.8% relative to Top-only navigation. The semantic view-selection mechanism was also robust to lexical variation in target descriptions, achieving 100% accuracy across 66 test cases, compared with 63.64% for a confidence-based view-selection baseline. In a substantially larger Gazebo environment, AgenticDiffusion achieved a 90% mission success rate and completed the multi-stage mission, whereas the FPV-only and Top-only variants were unable to complete all requested navigation tasks.
Faryal Batool, Muhammad Ahsan Mustafa, Fawad Mehboob +2