LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset

Yousef Emami; Mohammadhossein Homaei; Hao Zhou; Miguel Gutiérrez Gaitán; Atefeh Hajijamali Arani; Rui Zhang

cs.ROAug 15, 2026

LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset

Authors: Yousef Emami, Mohammadhossein Homaei, Hao Zhou, Miguel Gutiérrez Gaitán, Atefeh Hajijamali Arani, Rui Zhang

Abstract

Uncrewed aerial vehicles (UAVs) are increasingly deployed for autonomous navigation in complex outdoor environments, where dynamic conditions and mission requirements require intelligent adaptive decision-making. Existing optimization-based, Machine Learning (ML), and Reinforcement Learning (RL) approaches often rely on predefined models or task-specific training, limiting their generalization and adaptability in uncertain scenarios. Recent Large Language Model (LLM)-assisted approaches offer promising reasoning capabilities but remain constrained by limited agentic functionality, including insufficient memory, planning, and tool interaction mechanisms.This paper proposes an LLM-Agent-Based Path Finder (LAPF) framework for autonomous UAV navigation in town-scale outdoor environments. LAPF extends LLM-assisted navigation by integrating perception, memory, planning, and action modules into a closed-loop cognitive architecture. The proposed agent leverages prior navigation experiences, performs Chain-of-Thought (CoT) reasoning, couples each detected hazard to a bounded corrective action, and dynamically refines waypoint decisions based on environmental feedback.The three independent trials per method demonstrate that LAPF achieves mean path lengths of 512.83 m and 506.37 m, compared to the straight-line optimum of 497.33 m, corresponding to path length reductions of 17.2% and 15.6% relative to CoT prompting and absolute path efficiencies of 97.1% and 98.1% in open-field and obstacle-injected scenarios, respectively. Furthermore, LAPF is the only evaluated approach that couples every detected hazard to a bounded, metric-neutral corrective action while maintaining near-goal stability, with zero clamp events in both scenarios, whereas CoT prompting increases from 9.7 to 14.0 events.

Figures & tables

Figure 1: Architectural comparison between standalone LLM prompting and LLM agents. Conventional prompting (top) executes a single prompt-to-answer pass without autonomous correction capabilities. In contrast, the LLM agent (bottom) establishes a closed-loop system via an internal Plan

\rightarrow

Action cycle and an Update plan feedback mechanism. As illustrated on the right, these paradigms map directly onto a control-autonomy scale, highlighting the trade-off between direct human control and system autonomy.

Framework	Domain	Multimodal Input	Explicit Memory	Tool Use	Fine-tuning Required	UAV- specific	Real Sensor Fusion
[ 19 ]	Urban UAV safety path planning	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 20 ]	Urban UAV obstacle avoidance	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 21 ]	Urban UAV reconstruction	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
[ 22 ]	Risk-aware UAV path planning	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 23 ]	UAV optimization-based path planning (LGPSO)	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 24 ]	GNSS-aware urban UAV navigation	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$

Table I : Comparative Analysis of LAPF Against Related UAV and LLM-Agent Frameworks

Figure 2: A typical UAV path planning pipeline with LLM agents. The agent receives multimodal input from the UAVScenes dataset and, through an iterative Reason

\rightarrow

Plan

\rightarrow

Refine reasoning cycle, leverages external tools and memory in a bidirectional exchange to generate the UAV’s navigation decisions, one waypoint per decision step.

Symbol	System model	Instantiation
$\mathcal{I}_{t}$	RGB image	Passed natively to the VLM backbone at 1024 $\times$ 768; registered and time-matched
$\mathcal{P}_{t}$	Raw point cloud	Entered via the summary $\sigma_{t}$ , ( 3 )
$\mathcal{L}_{t}$	6-DoF pose	Pose $(R_{t},p_{t})$ with the time-matched RTK fix
$\mathcal{D}_{t}$	Dense depth map	Not fused (future work)
$\mathcal{S}_{t}$	Semantic mask	Not fused (future work)
$M_{t}$	Episodic memory	$k{=}3$ most recent episodes, recency-based

Table II : Instantiation of the system model in the present implementation

Figure 3: Overview of the proposed LAPF framework. The observation reaches the backbone through two distinct pathways: the RGB frame is passed natively as an image block and is never textualized, whereas the LiDAR return and the pose are reduced to a symbolic state; fusion of the two pathways occurs inside the backbone at inference time, within a single multimodal call. The two-tier Action module evaluates the clearance test with guaranteed invocation at every decision step, so that a registered violation is always coupled to a bounded corrective replan, which is metric-neutral by construction; the discretionary tier remains under agent authority.

Module	Input	Output	Main Functions
Perception	RGB image, LiDAR point cloud, and 6-DoF pose	RGB image and symbolic state	Registers and synchronizes RGB images for native VLM inference, converts LiDAR and localization data into a compact symbolic representation, and summarizes traversability, obstacle proximity, environmental semantics, and navigation constraints to provide situational awareness for downstream reasoning.
Memory	Current observation, executed waypoint, and execution outcome	Retrieved navigation experience	Stores one navigation episode per decision step together with an LLM-generated reflection, retrieves relevant historical experiences, and maintains persistent memory to support temporal continuity, long-term spatial awareness, and adaptive reasoning.
Planning	Current observation and retrieved memory	Candidate waypoint	Performs high-level multimodal reasoning using CoT to generate one candidate waypoint per decision step, together with any required corrective tool calls, according to the current scene and historical context.
Action	Candidate waypoint and agent tool calls	Executable waypoint	Coordinates external tool invocation, executes discretionary waypoint-adjustment operations when required, and translates high-level planning decisions into executable navigation commands.

Table III : Summary of the Proposed LAPF Framework Modules

Panel of Figure 4 — Figure 4: Multimodal prompt supplied to the VLM backbone at each planning cycle. The RGB frame is passed natively as an image block; all remaining fields are text, instantiated at runtime from $\sigma_{t}$ , $\mathcal{L}_{t}$ , the mission goal, the retrieved episodic memory $M_{t}$ , and the feedback signal $\phi_{t}$ . The returned JSON is parsed into Thought , Waypoint , and ToolCalls by the Planning module.

Parameter	Value
LLM inference
Backbone	Qwen2-VL-7B-Instruct (open-weight VLM)
Serving backend	mlx-vlm, locally hosted
Image input resolution	1024 $\times$ 768 (resampled from $2448\times 2048$ )
Temperature	0.7
Max tokens (planning)	8192

Table IV : LAPF implementation hyperparameters

Figure 5: Open-field scenario (AMtown03, no injected obstacles). Panels (a) and (b) show path length and efficiency (mean

\pm

SD) across three trials per method, relative to the

497.33

m straight-line optimum shown as a dashed red line. Panel (c) shows mean clamp events (near-goal step truncations). LAPF converges to near-optimal path length and efficiency with zero clamp events, while CoT LLM shows the largest deviation from optimal and the only nonzero clamp-event rate even in the absence of obstacles.

Figure 6: Obstacle-injected scenario (AMtown03, five injected clearance violations, four below

d_{\mathrm{safe}}=5

m). Panels (a) and (b) show path length and efficiency (mean

\pm

SD) across three trials per method, relative to the

497.33

m straight-line optimum shown as a dashed red line. Panel (c) shows mean unsafe-event and clamp-event counts. LAPF converges to near-optimal path length and efficiency while incurring zero clamp events, whereas CoT LLM shows the largest deviation and the highest clamp-event frequency, consistent with less regulated step proposals under obstacle constraints.

Explore similar work

May 13, 2026

LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration

Traditional autonomous UAV search missions rely on geometric coverage patterns that ignore the semantic context of the target, leading to significant time waste in large-scale environments. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors for Unmanned Aerial Vehicle (UAV) search missions that leverages semantics. Given a basic geofence and an object of interest prompt, LMPath uses generative language models to determine what regions of the environment should contain that object and a foundation vision model ran over satellite imagery to segment sub-regions that form the exploration prior. This prior can then be used to generate UAV paths with various objectives, such as minimizing the expected time to locate the object of interest, maximizing the probability that the object is found given a limited travel distance, or narrowing down the search space to sub-regions that are most likely to contain the object. To demonstrate it's capabilities, we used LMPath to generate various UAV paths and ran them using a real UAV over large-scale environments. We also ran simulations to demonstrate how paths generated using LMPath outperform traditional path planning approaches for search missions.

Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor +1

GRASP Laboratory, University of Pennsylvania

Trajectory Generation Large Language Model-Based Robot Planning UAV Navigation Robot Motion Planning

Ask PDF

Jul 21, 2026

No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation

Test-time scaling offers a promising method to improve the inference performance of Vision-Language Models (VLMs) without additional training. Existing approaches to vision-language navigation (VLN) for Unmanned Aerial Vehicle (UAV) typically relies on a single inference pass, which can falter in complex environments by producing suboptimal or unsafe trajectories. In this paper, we explore a simple and effective approach to apply test-time scaling to VLN for UAV. We enhance navigation reasoning through an iterative refinement process that requires no extra model training, guiding the model to re-evaluate its initial navigation plan for better accuracy and safety. Our method first prompts the model to generate multiple parallel candidates and then performs a self-correction step, achieving deeper and more robust planning without changing the underlying model. To further strengthen decision-making, we design a multi-criteria scoring function to evaluate the refined candidates based on safety, goal alignment, and forward-progress. This simple yet powerful combination enables a frozen UAV navigation VLMs to self-correct and generate more accurate and reliable flight plans, achieving SOTA performance in this task.

Feinan Cheng, Dongliang Xu, Wenli Nong +4

Shandong University Weihai, China · China University of Petroleum (East China) Qingdao, China · MBZUAI Abu Dhabi, United Arab Emirates

Vision-Language Models UAV Navigation Test-Time Scaling for VLMs Test-Time Scaling Vision-Language Navigation VLM Reasoning+1

Ask PDF

Jun 5, 2026

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers. To address this gap in UAV task settings, we introduce FLIGHT, a Fine-grained Long-horizon Instruction-Guided benchmark for Hybrid UAV navigation and reasoning Tasks, which combines multi-stage instructions with dense 6-DoF trajectory annotations across two dataset splits: Fine-grained VLN and Long-horizon Flow. To endow the UAV agent with the capability of real-time in-flight reasoning over task execution status and mission planning, while simultaneously accommodating high-frequency, real-time precise control, we further propose FLIGHT VLA, an asynchronous architecture that decouples a low-frequency Streaming Pilot Vision-Language Model (VLM) for task-state reasoning from a high-frequency diffusion action model for continuous control, supervised by explicit Pilot Reasoning texts that summarize the current flight state and anticipate the next subgoal. In closed-loop evaluation, FLIGHT VLA consistently surpasses representative VLN and VLA baselines on our FLIGHT benchmarks, achieving stronger multi-stage completion, subgoal adherence, and terminal control. Its trained Streaming Pilot Reasoning VLM further improves UAV video reasoning, validating the effectiveness of our design.

Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6

Colab, Beihang University · National University of Singapore · Meituan

UAV Navigation Vision-Language Navigation VLM Reasoning Vision-Language-Action Models Aerial VLN

Ask PDF