LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset

Yousef Emami; Mohammadhossein Homaei; Hao Zhou; Miguel Gutiérrez Gaitán; Atefeh Hajijamali Arani; Rui Zhang

cs.ROAug 15, 2026

LAPF: LLM-Agent-Based Path Finder Using the UAVScenes Dataset

Authors: Yousef Emami, Mohammadhossein Homaei, Hao Zhou, Miguel Gutiérrez Gaitán, Atefeh Hajijamali Arani, Rui Zhang

Abstract

Uncrewed aerial vehicles (UAVs) are increasingly deployed for autonomous navigation in complex outdoor environments, where dynamic conditions and mission requirements require intelligent adaptive decision-making. Existing optimization-based, Machine Learning (ML), and Reinforcement Learning (RL) approaches often rely on predefined models or task-specific training, limiting their generalization and adaptability in uncertain scenarios. Recent Large Language Model (LLM)-assisted approaches offer promising reasoning capabilities but remain constrained by limited agentic functionality, including insufficient memory, planning, and tool interaction mechanisms.This paper proposes an LLM-Agent-Based Path Finder (LAPF) framework for autonomous UAV navigation in town-scale outdoor environments. LAPF extends LLM-assisted navigation by integrating perception, memory, planning, and action modules into a closed-loop cognitive architecture. The proposed agent leverages prior navigation experiences, performs Chain-of-Thought (CoT) reasoning, couples each detected hazard to a bounded corrective action, and dynamically refines waypoint decisions based on environmental feedback.The three independent trials per method demonstrate that LAPF achieves mean path lengths of 512.83 m and 506.37 m, compared to the straight-line optimum of 497.33 m, corresponding to path length reductions of 17.2% and 15.6% relative to CoT prompting and absolute path efficiencies of 97.1% and 98.1% in open-field and obstacle-injected scenarios, respectively. Furthermore, LAPF is the only evaluated approach that couples every detected hazard to a bounded, metric-neutral corrective action while maintaining near-goal stability, with zero clamp events in both scenarios, whereas CoT prompting increases from 9.7 to 14.0 events.

Figures & tables

Figure 1: Architectural comparison between standalone LLM prompting and LLM agents. Conventional prompting (top) executes a single prompt-to-answer pass without autonomous correction capabilities. In contrast, the LLM agent (bottom) establishes a closed-loop system via an internal Plan

\rightarrow

Action cycle and an Update plan feedback mechanism. As illustrated on the right, these paradigms map directly onto a control-autonomy scale, highlighting the trade-off between direct human control and system autonomy.

Framework	Domain	Multimodal Input	Explicit Memory	Tool Use	Fine-tuning Required	UAV- specific	Real Sensor Fusion
[ 19 ]	Urban UAV safety path planning	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 20 ]	Urban UAV obstacle avoidance	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 21 ]	Urban UAV reconstruction	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
[ 22 ]	Risk-aware UAV path planning	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 23 ]	UAV optimization-based path planning (LGPSO)	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\times$
[ 24 ]	GNSS-aware urban UAV navigation	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$

Table I : Comparative Analysis of LAPF Against Related UAV and LLM-Agent Frameworks

Figure 2: A typical UAV path planning pipeline with LLM agents. The agent receives multimodal input from the UAVScenes dataset and, through an iterative Reason

\rightarrow

Plan

\rightarrow

Refine reasoning cycle, leverages external tools and memory in a bidirectional exchange to generate the UAV’s navigation decisions, one waypoint per decision step.

Symbol	System model	Instantiation
$\mathcal{I}_{t}$	RGB image	Passed natively to the VLM backbone at 1024 $\times$ 768; registered and time-matched
$\mathcal{P}_{t}$	Raw point cloud	Entered via the summary $\sigma_{t}$ , ( 3 )
$\mathcal{L}_{t}$	6-DoF pose	Pose $(R_{t},p_{t})$ with the time-matched RTK fix
$\mathcal{D}_{t}$	Dense depth map	Not fused (future work)
$\mathcal{S}_{t}$	Semantic mask	Not fused (future work)
$M_{t}$	Episodic memory	$k{=}3$ most recent episodes, recency-based

Table II : Instantiation of the system model in the present implementation

Figure 3: Overview of the proposed LAPF framework. The observation reaches the backbone through two distinct pathways: the RGB frame is passed natively as an image block and is never textualized, whereas the LiDAR return and the pose are reduced to a symbolic state; fusion of the two pathways occurs inside the backbone at inference time, within a single multimodal call. The two-tier Action module evaluates the clearance test with guaranteed invocation at every decision step, so that a registered violation is always coupled to a bounded corrective replan, which is metric-neutral by construction; the discretionary tier remains under agent authority.

Module	Input	Output	Main Functions
Perception	RGB image, LiDAR point cloud, and 6-DoF pose	RGB image and symbolic state	Registers and synchronizes RGB images for native VLM inference, converts LiDAR and localization data into a compact symbolic representation, and summarizes traversability, obstacle proximity, environmental semantics, and navigation constraints to provide situational awareness for downstream reasoning.
Memory	Current observation, executed waypoint, and execution outcome	Retrieved navigation experience	Stores one navigation episode per decision step together with an LLM-generated reflection, retrieves relevant historical experiences, and maintains persistent memory to support temporal continuity, long-term spatial awareness, and adaptive reasoning.
Planning	Current observation and retrieved memory	Candidate waypoint	Performs high-level multimodal reasoning using CoT to generate one candidate waypoint per decision step, together with any required corrective tool calls, according to the current scene and historical context.
Action	Candidate waypoint and agent tool calls	Executable waypoint	Coordinates external tool invocation, executes discretionary waypoint-adjustment operations when required, and translates high-level planning decisions into executable navigation commands.

Table III : Summary of the Proposed LAPF Framework Modules

Panel of Figure 4 — Figure 4: Multimodal prompt supplied to the VLM backbone at each planning cycle. The RGB frame is passed natively as an image block; all remaining fields are text, instantiated at runtime from $\sigma_{t}$ , $\mathcal{L}_{t}$ , the mission goal, the retrieved episodic memory $M_{t}$ , and the feedback signal $\phi_{t}$ . The returned JSON is parsed into Thought , Waypoint , and ToolCalls by the Planning module.

Parameter	Value
LLM inference
Backbone	Qwen2-VL-7B-Instruct (open-weight VLM)
Serving backend	mlx-vlm, locally hosted
Image input resolution	1024 $\times$ 768 (resampled from $2448\times 2048$ )
Temperature	0.7
Max tokens (planning)	8192

Table IV : LAPF implementation hyperparameters

Figure 5: Open-field scenario (AMtown03, no injected obstacles). Panels (a) and (b) show path length and efficiency (mean

\pm

SD) across three trials per method, relative to the

497.33

m straight-line optimum shown as a dashed red line. Panel (c) shows mean clamp events (near-goal step truncations). LAPF converges to near-optimal path length and efficiency with zero clamp events, while CoT LLM shows the largest deviation from optimal and the only nonzero clamp-event rate even in the absence of obstacles.

Figure 6: Obstacle-injected scenario (AMtown03, five injected clearance violations, four below

d_{\mathrm{safe}}=5

m). Panels (a) and (b) show path length and efficiency (mean

\pm

SD) across three trials per method, relative to the

497.33

m straight-line optimum shown as a dashed red line. Panel (c) shows mean unsafe-event and clamp-event counts. LAPF converges to near-optimal path length and efficiency while incurring zero clamp events, whereas CoT LLM shows the largest deviation and the highest clamp-event frequency, consistent with less regulated step proposals under obstacle constraints.

Explore similar work

LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration
May 13, 2026Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor +1Trajectory Generation Large Language Model-Based Robot Planning
GRASP Laboratory, University of Pennsylvania
No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation
Jul 21, 2026Feinan Cheng, Dongliang Xu, Wenli Nong +4Vision-Language Models UAV Navigation
Shandong University Weihai, China · China University of Petroleum (East China) Qingdao, China · MBZUAI Abu Dhabi, United Arab Emirates
Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation
Jun 5, 2026Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6UAV Navigation Vision-Language Navigation
Colab, Beihang University · National University of Singapore · Meituan

May 13, 2026J/K move · Enter open · S save

LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration

Jonathan A. Diller, Fernando Cladera, Camillo J. Taylor, Vijay Kumar

GRASP Laboratory, University of Pennsylvania

Trajectory Generation Large Language Model-Based Robot Planning UAV Navigation Robot Motion Planning

Open paper AskPDF arXiv

Traditional autonomous UAV search missions rely on geometric coverage patterns that ignore the semantic context of the target, leading to significant time waste in large-scale environments. In this paper we present LMPath, a pipeline for generating language-mediated exploration priors for Unmanned Aerial Vehicle (UAV) search missions that leverages semantics. Given a basic geofence and an object of interest prompt, LMPath uses generative language models to determine what regions of the environment should contain that object and a foundation vision model ran over satellite imagery to segment sub-regions that form the exploration prior. This prior can then be used to generate UAV paths with various objectives, such as minimizing the expected time to locate the object of interest, maximizing the probability that the object is found given a limited travel distance, or narrowing down the search space to sub-regions that are most likely to contain the object. To demonstrate it's capabilities, we used LMPath to generate various UAV paths and ran them using a real UAV over large-scale environments. We also ran simulations to demonstrate how paths generated using LMPath outperform traditional path planning approaches for search missions.

Abstract

Explore similar work

LMPath: Language-Mediated Priors and Path Generation for Aerial Exploration

No Training, Better Flights: Test-Time Scaled VLMs for UAV Navigation

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation