Spatial-Semantic Reasoning using Large Language Models for Efficient UAV Search Operations
Authors: Marin Maletic, Marijana Peti, Tamara Petrovic, Stjepan Bogdan
Organizations: University of Zagreb Faculty of Electrical Engineering and Computing, LARICS (Laboratory for Robotics and Intelligent Control Systems), Unska 3, 10000 Zagreb, Croatia
We present a real-time semantic navigation framework for Unmanned Aerial Vehicles (UAVs) focused on improving time efficiency in the Object Goal Navigation (ObjectNav) task. Central to our approach is a Large Language Model (LLM) that interprets user-provided natural language instructions and performs semantic reasoning over detected objects and spatial context to prioritize high-probability search regions. The system combines real-time object detection, 3D spatial mapping, and polynomial spline interpolation for smooth and feasible UAV trajectory planning. Unlike prior methods that rely on offline reasoning or simulator-constrained action spaces, our framework can operate in real time, continuously updating semantic relevance based on new observations. Experiments in both simulated and real-world settings demonstrate reductions in mission duration while maintaining high search accuracy, underscoring the effectiveness of LLM-guided reasoning for time- efficient UAV-based ObjectNav.
Figures & tables
Fig. 1: Illustration of the proposed method consisting of four interconnected modules: (1) Visual Perception, employing a camera and object detection model to identify semantic objects; (2) Volumetric Space Representation, which utilizes a depth sensor to build a spatial occupancy map; (3) Position Estimation, projecting detected objects into 3D space and associating them with depth information from the sensor data; and (4) Relevance Evaluation, which integrates all acquired semantic and spatial data with user instructions to determine the most relevant object around which the UAV performs a thorough search.
Fig. 2: Illustration of the 3D position estimator PE(⋅) , which maps detected objects from an image to their corresponding world coordinates in two stages: (a) Computation of the 3D viewing direction from the object’s pixel position ( u,v) in the image, based on camera intrinsics; (b) Ray casting from the UAV along the computed direction to identify the first occupied voxel in the OctoMap, belonging to the object.
Fig. 3: Overall schematic diagram of the developed system, illustrating data collection through the Visual Perception Module (VPM), Volumetric Space Representation Module (VSRM) and 3D Position Estimation Module (PEM), with emphasis on the Probabilistic Relevance Module (PRM) and the internal LLM black box. The Large Language Model determines the next search action based on accumulated data D , including user instructions U , previously searched areas Visited , and the current set of detected objects O along with their spatial relationships. The LLM selects the most relevant object O∗ ; if its relevance exceeds a defined threshold, a thorough search is conducted around it. Otherwise, the system reverts to a fallback strategy (e.g., wall-following). If the relevance score for an object is sufficiently high, the LLM concludes that O∗ is the target T and terminates the search successfully. If the UAV exhaustively searches the space S without locating T , the Bnone hypothesis becomes dominant, and the search ends in failure.
Fig. 4: Top-down 2D visualization of generated trajectories: (a) Baseline Lawn Mower scanning trajectory; (b) Thorough search extending from the UAV’s current position (1.9,8.1,2.0) to, and around, the relevant object located at (9.0,2.5,1.1) . Since the object is positioned near walls along x=10 and y=0 , the circular inspection path is adapted to a semicircular trajectory, maintaining a safe distance from obstacles to ensure the UAV remains within the free navigable space.
Fig. 5: Illustrated simulation test scenarios: the process begins with the user providing natural-language instructions, after which the UAV performs a 360-degree scan to survey and detect surrounding objects. The large language model (LLM) then reasons over the gathered information, determines the next action, and generates an explanation for the bolded relevant object. Finally, the UAV executes a thorough-search routine around the relevant object until the specified target object is successfully located.
Scenario
Method
Success Rate (%)
Search Time Improvement (%)
Path Length Reduction (%)
Banana
LLM-guided
100
36.02 ± 30.52
52.26 ± 38.16
Lawn Mower
65
Computer Mouse
LLM-guided
100
24.83 ± 25.13
43.50 ± 33.79
Lawn Mower
75
Toy Zebra
LLM-guided
100
29.24 ± 28.71
51.93 ± 27.86
Lawn Mower
95
TABLE I: Performance Comparison Between LLM-Guided Search and Lawn Mower Baseline Across Three Scenarios
Fig. 6: Real-world experiment with the target object set as a computer mouse: (a) Initial camera view of the scene, where the target object is occluded but the computer monitor is visible; (b) OctoMap representation, with occupied space visualized by blue voxels and the UAV’s executed trajectory shown in green; (c) Camera view during the thorough search process, where the computer mouse becomes detectable, leading to successful completion of the search.
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China