Organizations: University at Albany, Department of Computer Science, NY, USA. · GE Aerospace Research, NY, USA. · Missouri State University, Department of Computer Science, Missouri, USA
This paper proposes a command-aligned sampling distribution that incorporates navigation commands directly into the exploration process to improve the efficiency of sampling-based motion planning for quadcopters in three-dimensional environments. Inspired by the Language-Aided Subset Sampling Motion Planner, this paper extends command-aware subset sampling to aerial systems. Subset sampling is generalized to command-aligned regions, enabling structured 3D exploration while preserving probabilistic completeness. Natural language instructions are grounded into yaw-conditioned and altitude-aware geometric constraints using a prompt-based large language model for entity extraction and instruction validation. Extensive simulations in complex environments demonstrate that the proposed method generates collision-free, consistent trajectories while significantly reducing sampling effort compared to uniform RRT-based planners.
Figures & tables
Fig. 1 : Extend LASMP: A drone navigating a multi-room environment based on natural language input while generating obstacle free paths on the fly. The red line shows the planned path while the yellow dotted lines show smoothened path executed on the robot.
Fig. 2 : (a) Architectural overview of the Extended LASMP pipeline: The Qwen [ 22 ] model process the natural language instructions to extract a sequence of directional primitives (e.g., UP , LEFT ). This turn list serves as a heuristic constraint for the Extend LASMP framework, which optimizes the search space toward a task-relevant subset of the planning domain.
Fig. 3 : An illustration of Chat Markup Language (ChatML)few-shot prompt to extract the motion entities. Messages are structured by roles such as system, user, and assistant. The system message defines the model’s behavior, the user message defines the task, and the assistant message can include an example output.
Fig. 4 : Diverse 3D point cloud environments for experimental validation. The environments vary in obstacle density, and corridor width to evaluate the robustness of command-aware subset sampling under different structural complexities.
Fig. 5 : A comparative evaluation of the planners Extended LASMP, RRT, and RRT* in terms of the number of nodes added to the search tree. The red line denotes the collision-free path extracted from the search tree. Extended LASMP needed significantly less number of nodes as compared to RRT and RRT*.
Fig. 6 : Performance comparison of Extended LASMP, RRT, RRT* and RRTConnect w.r.t number of nodes added to the search tree, the number of random-state queries, the computation time, and PathLength.
Parameter
Extended LASMP
RRT
RRT*
RRT-Connect
Step size (m)
0.1
0.5
0.4
0.5
Maximum iterations
200
5000
4000
5000
Goal tolerance(m)
2
N/A
N/A
N/A
Goal bias
N/A
0.1
0.1
0.1
Rewiring (m)
N/A
N/A
1
N/A
TABLE III : Planning parameters used for each algorithm.
Fig. 7 : Instruction sequence Go up, then turn back, take a left followed by another left, then go down and turn back to reach your destination . The snapshots illustrate yaw-conditioned subset sampling and ray-based switching between commands, resulting in a collision-free trajectory. The green blocks indicate the quadcopter poses at various time sequences.
We present a hierarchical UAV navigation framework that aligns natural-language intent with dynamically feasible flight behaviors in cluttered indoor environments. To bridge the gap between abstract semantics and low-level control, we employ a parallelized ensemble of six behavior-conditioned Model Predictive Path Integral (MPPI) planners. Crucially, by designing mode-specific guiding costs and sampling biases, we induce distinct trajectory modes that converge to unique behavioral means, yielding a compact set of intentionally diverse candidates rather than mere stochastic variations. We project these 3D candidates onto the onboard first-person-view RGB stream, turning language grounding into a visual action selection problem. A pretrained vision--language model (VLM) asynchronously selects the candidate index given the overlaid FPV image and a natural-language prompt, while MPPI replans at 20Hz and a PID-based low-level controller tracks the selected trajectory. We implement the full pipeline in NVIDIA Isaac Sim and on a real-world quadrotor platform equipped with LiDAR and RGB sensing. Experiments in both simulation and real-world flights show semantically meaningful behavior diversity, robust language alignment despite VLM latency, and safe, repeatable flight across all modes, achieving 100% task success in our evaluated scenarios.
Hanbing Zhang, Fangguo Zhao, Zerui Li +3
College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China · Australian Institute for Machine Learning, Adelaide University
Traditional autonomous UAV search missions rely on geometric coverage patterns that ignore the semantic context of the target, leading to significant time waste in large-scale environments. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors for Unmanned Aerial Vehicle (UAV) search missions that leverages semantics. Given a basic geofence and an object of interest prompt, LMPath uses generative language models to determine what regions of the environment should contain that object and a foundation vision model ran over satellite imagery to segment sub-regions that form the exploration prior. This prior can then be used to generate UAV paths with various objectives, such as minimizing the expected time to locate the object of interest, maximizing the probability that the object is found given a limited travel distance, or narrowing down the search space to sub-regions that are most likely to contain the object. To demonstrate it's capabilities, we used LMPath to generate various UAV paths and ran them using a real UAV over large-scale environments. We also ran simulations to demonstrate how paths generated using LMPath outperform traditional path planning approaches for search missions.
Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor +1
Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability. While Vision-Language-Action (VLA) models excel at semantic reasoning, they suffer from brittleness due to geometric inconsistency and dynamics mismatch. To address this, we propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling. Instead of direct regression, ImagineUAV employs a latent video diffusion model to generate instruction-conditioned future observations, explicitly imagining environmental evolution, from which 6-DoF motions are inferred via an action extractor. A kinodynamic planner then refines these estimates into collision-free trajectories. Additionally, a step-distilled inference pipeline ensures real-time execution. With only 1.3B parameters, ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights, validating the practicality of imagination-driven aerial navigation.
Xuchen Liu, Jiawei Huang, Shihao Xia +3
Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China