Spatial-Semantic Reasoning using Large Language Models for Efficient UAV Search Operations
Authors: Marin Maletic, Marijana Peti, Tamara Petrovic, Stjepan Bogdan
Organizations: University of Zagreb Faculty of Electrical Engineering and Computing, LARICS (Laboratory for Robotics and Intelligent Control Systems), Unska 3, 10000 Zagreb, Croatia
We present a real-time semantic navigation framework for Unmanned Aerial Vehicles (UAVs) focused on improving time efficiency in the Object Goal Navigation (ObjectNav) task. Central to our approach is a Large Language Model (LLM) that interprets user-provided natural language instructions and performs semantic reasoning over detected objects and spatial context to prioritize high-probability search regions. The system combines real-time object detection, 3D spatial mapping, and polynomial spline interpolation for smooth and feasible UAV trajectory planning. Unlike prior methods that rely on offline reasoning or simulator-constrained action spaces, our framework can operate in real time, continuously updating semantic relevance based on new observations. Experiments in both simulated and real-world settings demonstrate reductions in mission duration while maintaining high search accuracy, underscoring the effectiveness of LLM-guided reasoning for time- efficient UAV-based ObjectNav.
Figures & tables
Fig. 1: Illustration of the proposed method consisting of four interconnected modules: (1) Visual Perception, employing a camera and object detection model to identify semantic objects; (2) Volumetric Space Representation, which utilizes a depth sensor to build a spatial occupancy map; (3) Position Estimation, projecting detected objects into 3D space and associating them with depth information from the sensor data; and (4) Relevance Evaluation, which integrates all acquired semantic and spatial data with user instructions to determine the most relevant object around which the UAV performs a thorough search.
Fig. 2: Illustration of the 3D position estimator PE(⋅) , which maps detected objects from an image to their corresponding world coordinates in two stages: (a) Computation of the 3D viewing direction from the object’s pixel position ( u,v) in the image, based on camera intrinsics; (b) Ray casting from the UAV along the computed direction to identify the first occupied voxel in the OctoMap, belonging to the object.
Fig. 3: Overall schematic diagram of the developed system, illustrating data collection through the Visual Perception Module (VPM), Volumetric Space Representation Module (VSRM) and 3D Position Estimation Module (PEM), with emphasis on the Probabilistic Relevance Module (PRM) and the internal LLM black box. The Large Language Model determines the next search action based on accumulated data D , including user instructions U , previously searched areas Visited , and the current set of detected objects O along with their spatial relationships. The LLM selects the most relevant object O∗ ; if its relevance exceeds a defined threshold, a thorough search is conducted around it. Otherwise, the system reverts to a fallback strategy (e.g., wall-following). If the relevance score for an object is sufficiently high, the LLM concludes that O∗ is the target T and terminates the search successfully. If the UAV exhaustively searches the space S without locating T , the Bnone hypothesis becomes dominant, and the search ends in failure.
Fig. 4: Top-down 2D visualization of generated trajectories: (a) Baseline Lawn Mower scanning trajectory; (b) Thorough search extending from the UAV’s current position (1.9,8.1,2.0) to, and around, the relevant object located at (9.0,2.5,1.1) . Since the object is positioned near walls along x=10 and y=0 , the circular inspection path is adapted to a semicircular trajectory, maintaining a safe distance from obstacles to ensure the UAV remains within the free navigable space.
Fig. 5: Illustrated simulation test scenarios: the process begins with the user providing natural-language instructions, after which the UAV performs a 360-degree scan to survey and detect surrounding objects. The large language model (LLM) then reasons over the gathered information, determines the next action, and generates an explanation for the bolded relevant object. Finally, the UAV executes a thorough-search routine around the relevant object until the specified target object is successfully located.
Scenario
Method
Success Rate (%)
Search Time Improvement (%)
Path Length Reduction (%)
Banana
LLM-guided
100
36.02 ± 30.52
52.26 ± 38.16
Lawn Mower
65
Computer Mouse
LLM-guided
100
24.83 ± 25.13
43.50 ± 33.79
Lawn Mower
75
Toy Zebra
LLM-guided
100
29.24 ± 28.71
51.93 ± 27.86
Lawn Mower
95
TABLE I: Performance Comparison Between LLM-Guided Search and Lawn Mower Baseline Across Three Scenarios
Fig. 6: Real-world experiment with the target object set as a computer mouse: (a) Initial camera view of the scene, where the target object is occluded but the computer monitor is visible; (b) OctoMap representation, with occupied space visualized by blue voxels and the UAV’s executed trajectory shown in green; (c) Camera view during the thorough search process, where the computer mouse becomes detectable, leading to successful completion of the search.
Aerial Object Goal Navigation (ObjectNav) requires an unmanned aerial vehicle (UAV) to locate a described target in an unknown outdoor environment using onboard visual observations. Vision-language models (VLMs) can interpret open-ended target descriptions and visual observations, but their frame-level outputs are often noisy, sparse, and spatially transient. We propose AeroBelief, a dual-layer semantic-spatial belief mapping framework that transforms transient VLM observations into persistent spatial guidance. It separates broad contextual plausibility from target-specific evidence: an intuition layer accumulates scene-level semantic cues for exploration, while an evidence layer preserves qualified target-specific observations for approach and confirmation. Evidence-gated fusion combines the two layers into spatial belief hotspots. We further introduce object-conditioned visual reasoning with conservative evidence qualification to improve observation reliability before spatial accumulation. In parallel, egocentric regional guidance converts quadtree coverage into UAV-centered, yaw-aligned directional proposals and stabilizes them through temporal commitment. Its regional scoring is independent of semantic belief values, maintaining exploration pressure and reducing repeated low-gain search. Experiments on the UAV-ON benchmark show that AeroBelief achieves the best reported overall SR, OSR, and SPL among the compared methods, reaching 21.61%, 35.57%, and 10.62, respectively. These results support the effectiveness of persistent semantic-spatial belief, conservative evidence qualification, and temporally stable regional guidance for aerial ObjectNav.
UAV vision-language navigation (UAV-VLN) focuses on enabling an aerial agent to follow natural-language instructions in open 3D environments from egocentric visual observations. Current approaches suffer from three coupled issues: weak grounding of instruction-relevant landmarks in visual observations, insufficient exploitation of long-horizon history, and unstable decisions under local traps or repeated exploration. To address these issues, we propose a unified semantic-to-decision framework. First, we present an instruction-grounded semantic enhancement module that injects object-level semantics and relative spatial cues into the current observation state. Subsequently, we develop a relevance-aware dynamic temporal aggregation strategy that reweights the full history buffer while converting a few high-relevance frames into structured landmark prompts for the decoder. Finally, we devise a topology-aware decision method that combines local-optimum cognition with group-relative policy optimization under progress, goal, semantic, and path-compliance rewards. Experiments on the widely used AerialVLN and OpenFly benchmarks clearly demonstrate that our method achieves state-of-the-art performance.
Zeyuan Ma, Jiaxin Chen, Di Huang
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China
Traditional autonomous UAV search missions rely on geometric coverage patterns that ignore the semantic context of the target, leading to significant time waste in large-scale environments. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors for Unmanned Aerial Vehicle (UAV) search missions that leverages semantics. Given a basic geofence and an object of interest prompt, LMPath uses generative language models to determine what regions of the environment should contain that object and a foundation vision model ran over satellite imagery to segment sub-regions that form the exploration prior. This prior can then be used to generate UAV paths with various objectives, such as minimizing the expected time to locate the object of interest, maximizing the probability that the object is found given a limited travel distance, or narrowing down the search space to sub-regions that are most likely to contain the object. To demonstrate it's capabilities, we used LMPath to generate various UAV paths and ran them using a real UAV over large-scale environments. We also ran simulations to demonstrate how paths generated using LMPath outperform traditional path planning approaches for search missions.
Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor +1