Multi-Task Visual Perception Network with LLM Conditioning for Autonomous Navigation

Praveen Kumar; K. R. Guruprasad; Tushar Sandhan

cs.ROSep 13, 2026

Multi-Task Visual Perception Network with LLM Conditioning for Autonomous Navigation

Authors: Praveen Kumar, K. R. Guruprasad, Tushar Sandhan

Organizations: Department of Electrical Engineering, Indian Institute of Technology Kanpur, India · Department of Mechanical Engineering, Indian Institute of Technology Kanpur, India

Abstract

Long-term navigation for service robots faces crit- ical challenges like the accumulation of odometry drift and sensor error, which progressively degrade 2D maps and renders traditional path planning algorithms (e.g., A*, RRT*, DiPPer, ViT-A*) ineffective over time. To address this, we propose a user-friendly, interactive framework that eliminates the reliance on globally consistent maps. Our approach integrates visual perception with Large Language Models (LLM) to interpret user commands via text or voice. Instead of relying on a drift- prone global map, the system generates a sequential action plan based on local visual cues and egocentric geometric instructions. These action plans are executed sequentially, allowing the robot to navigate known and unknown environments safely. By reset- ting localization relative to immediate targets, our framework effectively works with a minimum accumulation drift strategy, ensuring accurate, efficient, and collision-free navigation without the maintenance overhead of traditional mapping. Experiments on real-world and simulated data have shown significant improve- ments over other methods. Our source code is publicly accessible at https://github.com/PraveenSingh24/VL-Navigation.

Figures & tables

Fig. 1: Overview of the proposed scenario. The user leverages prior spatial knowledge of the environment to provide natural language task instructions to the service robot. The robot interprets these instructions and interacts with the environment through continuous observation and action, executing navigation tasks sequentially without relying on a global map.

Fig. 2: Overview of the proposed vision-language navigation framework, which is distributed across two nodes: the onboard client (Husky A200) and an offboard GPU server for heavy computation. The workflow begins with Oracle Task Instructions (Natural Language), which are processed by LLM to generate an action plan. The Action Classifier parses this plan into geometric or visual sub-goals. To enable safe navigation, the Scene Parsing module generates a Segmented (Seg.) Point Cloud—representing specific semantic regions—which is concatenated with the filtered terrain point cloud. Finally, the Motion Primitive Local Planner computes optimal velocity commands $(\upsilon,\omega)$ to drive the robot toward the Proximal Target.

Fig. 3: The multi-layered architecture for converting user task instructions into robot action plan. The system resolves lexical ambiguity and distinguishes between geometric goal and semantic target constraints before grounding them into a unified robot system for execution.

Fig. 4: Structured prompt is created via injecting static system constraints into the context window alongside the user task instruction. LLM translates the natural language request into a machine-readable format.

Fig. 5: Visual perception to point cloud generation pipeline. The RGB–D stream acquired from the Intel RealSense D455 camera is processed by an RGB-D semantic segmentation network (ESANet) to extract task-relevant regions. The resulting segmented mask is projected into 3D space using depth information to generate a semantic point cloud, which is subsequently utilized for terrain analysis and context-aware navigation.

Fig. 6: (a) Set of candidate motion trajectories produced using the motion primitive framework. These trajectories represent feasible forward motions under kinematic constraints and are evaluated against environmental and semantic constraints to select an optimal navigation path. (b) Data flow of the local planner. The process begins by synchronizing the pre-computed path library and sensor data to the robot’s current state. Feasible paths are filtered through a voxel-grid collision check, and the final trajectory is chosen by minimizing a cost function to ensure safe and efficient navigation.

Fig. 7: Illustration of odometry drift estimation during navigation. The robot moves through intermediate subgoals

G_{1}

–

G_{3}

toward the final destination

D

, and the drift is computed based on the deviation between the robot pose and the corresponding target.

Fig. 8: Experimental validation in the Gazebo simulation environment. (Left) The 3D test workspace containing the Husky A200 robot and various semantic obstacles labeled as: 2) Statue, 3) Chair, 4) Bicycle, 5) Color-marked restricted zone, and 6) Moving pedestrian. (Right) Top-down view of the generated point cloud map illustrating the autonomous navigation performance, showing the Start Point, Goal Point, and the resulting collision-free Traversed Path

Fig. 9: Explored environment and traversed paths for two distinct navigation tasks. The left and right panels display the top-down occupancy map, where black regions represent physical barriers and context-aware constraints. The robot successfully navigates from the Start Point to the Goal Point by executing a sequence of semantic actions. Key landmarks are labeled as S (Statue/Human), C (Visitor Chair), and B (Bicycle). The planner also accounts for dynamic and static constraints, such as the Moving Pedestrian (MP) and the designated Restricted Zone.

Method	Global Map	Drift Sensitivity	Long-Term Nav.	Unknown Env.	Semantic Understanding	Context Awareness	User Interaction	Replanning
A* / RRT*	✓	✓	✗	✗	✗	✗	✗	Limited
DiPPer	✓	✓	$\triangle$	✗	✗	✗	✗	Limited
ViT-A*	✓	✓	$\triangle$	$\triangle$	$\triangle$	✗	✗	Limited
Vision-Aided A* [ 20 ]	✓	✓	✗	✓	✗	✓	✗	Online
SemExp (Semantic Exploration) [ 21 ]	✗	$\triangle$	$\triangle$	✓	✓	✓	✗	Moderate
CARe (Context-Aware Replanning) [ 22 ]	✗	$\triangle$	✓	✓	✓	✓	✗	Advanced

TABLE I: Comparison of Modern Navigation Paradigms (Extended). ✓ Supported, ✗ Not supported,

\triangle

Partial.

Case 1: Complex Navigation
User Input: “Buddy go to the statue (human), take a right turn and move a little ahead, then find the visitor chair to reach close to it, and on the right you may see a bicycle.”
LLM Output: [{"action": "FIND", "target": "statue (human)", "id": 0}, {"action": "MOVE", "desc": "turn right...", "f_dist": 2.5, "yaw": -1.57}, {"action": "FIND", "target": "visitor chair", "id": 56}, {"action": "FIND", "target": "bicycle", "id": 1}]
Status: Success
Case 2: Metric Constraints
User Input: “Buddy take left turn, move 4 m ahead, then find the standing person to reach close to it, In your right you may see the visitor chair follow it. Finally go at spot where bicycle is standing.”
LLM Output: [{"action": "MOVE", "desc": "left turn 4m", "f_dist": 4.0, "yaw": 1.57}, {"action": "FIND", "target": "standing person", "id": 0}, {"action": "FIND", "target": "visitor chair", "id": 56}, {"action": "FIND", "target": "bicycle", "id": 1}]

TABLE II: Qualitative Evaluation of LLM-Guided Instruction Parsing.

Fig. 10: Indoor experimental environment with the Husky A200 robot and onboard Intel RealSense D455 camera. The scene includes multiple visual targets and restricted regions in a confined workspace used for real-world navigation evaluation.

Fig. 11: ESANet model input and output: The frame (left) shows the settings: the marked objects are placed on the surface, the red-marked box is drawn on the ground, and the remaining items are at a significant height. The segmentation model output is shown (right), where we show the segmented region of the labeled items, which the user defines.

Fig. 12: Real-world navigation results across multiple trials. The robot starts from S and first moves toward an egocentric geometric goal (G1), followed by visually detected semantic targets (G2–G4). White regions indicate free space, black regions represent obstacles, and the colored path shows the executed trajectory.

Fig. 13: Experimental results of the visual navigation system. (a) The wheeled robot successfully tracking and following a dynamic human target. (b) Estimation of the fixed visual target point; note that despite the spatial spread of the estimated points, the robot successfully navigated to the visual destination. (c) The view of the red-marked box target, showing lateral shifts caused by the robot’s to-and-fro motion during navigation.

Module	Task	Latency (ms)	Execution Mode
Server Platform (NVIDIA RTX A4000)
RedNet	RGB-D Segmentation	$29.2\pm 1.8$	Continuous
ESANet	RGB-D Segmentation	$34.8\pm 0.7$	Continuous
OSNet	Person Re-ID	$50\pm 5$	On-demand
YOLOv8	Object Detection	$12.3\pm 1.4$	On-demand
LLM (Gemini Pro)	Instruction Parsing	$150\pm 30$	Per command

TABLE III: Inference Time Performance Across Computing Platforms

Scenario	Constraint	Geometric	Visual	Hybrid (Ours)
Featureless Corridor	No visual landmarks	92.0	15.0	95.0
Symmetric Room	Geometric ambiguity	40.0	94.0	96.0
Occluded Target	Target not visible	85.0	30.0	91.0
Precise Positioning	Exact stopping required	96.0	65.0	98.0
Complex Mission	Sequential task execution	10.0	25.0	88.0

TABLE IV: Performance Comparison of Navigation Modalities Under Different Environmental Constraints

Environment	Success Rate (%)	Soft Collision (%)	Hard Collision (%)	Avg. Path Length (m)	Drift Error (m)
Simulation
Indoor	100	10.0	0.0	12.4	$0.03\pm 0.01$
Garage	100	8.5	0.0	20.8	$0.05\pm 0.03$
Campus	98	10.0	0.0	30.5	$0.05\pm 0.02$
Forest	95	15.2	0.3	15.5	$0.08\pm 0.01$
Tunnel	98	14.0	0.0	25.2	$0.05\pm 0.02$

TABLE V: Performance Evaluation Results Over 50 Trials per Scenario. The simulation scenarios (Indoor, Garage, Campus, Forest, Tunnel) are given in detail [ 17 , 25 ] . These setups are modified to include visual objects and semantic context-aware items (soft collision) for our system performance evaluation. We consider the environmental structure (hard collision) that obstructs the navigation path.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 532 papers

"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control
Sep 19, 2026Sebastian Berger, Katharina Winter, Fabian B. FlohrVision-Language Navigation Vision-Based Robot Control
Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany
SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation
Aug 24, 2026Shibo Zhao, Guofei Chen, Honghao Zhu +7Open-Vocabulary Semantic Mapping Semantic Mapping
The Robotics Institute, Carnegie Mellon University
AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning
Jun 9, 2026Hang Yin, Yinan Liang, Jiazhao Zhang +4RL for Robotics Continual Robot Learning
Tsinghua University, Beijing 100084, China · Galbot Robotics, Beijing, China · Peking University, Beijing 100871, China +1

Sep 19, 2026J/K move · Enter open · S save

"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

Sebastian Berger, Katharina Winter, Fabian B. Flohr

Munich University of Applied Sciences, Intelligent Vehicles Lab (IVL), 80335 Munich, Germany

Vision-Language Navigation Vision-Based Robot Control

Open paper AskPDF arXiv

Vision-language models (VLMs) provide a compelling foundation for reasoning-driven mobile navigation, offering rich contextual understanding and strong generalization from large-scale pretraining. Most existing navigation frameworks rely on imitation learning and therefore require substantial labeled trajectory data, limiting their scalability and robustness. In this work, we propose a parameter-efficient approach to fine-tune a pretrained VLM for autonomous navigation using an Imperative Learning paradigm. By optimizing against differentiable geometric cost fields rather than labeled trajectories, our model learns to generate collision-free paths exclusively from stereoscopic depth observations. We introduce a unified end-to-end navigation pipeline for natural-language-driven robotic control. This system leverages a shared VLM backbone with task-specific Low-Rank Adaptation (LoRA) modules, effectively bridging the gap from semantic target selection to low-level trajectory planning. Our approach achieves competitive Success weighted by Path Length (SPL) in unseen environments while updating less than 1% of the model's total parameters. Qualitative real-world experiments validate sim-to-real generalization and stable path planning without fine-tuning on real-world data. These results highlight a practical approach for deploying VLM-based agents on mobile robots, enabling high-level semantic navigation without the prohibitive requirement for large-scale, labeled trajectory data.

Abstract

Robot Navigation

Explore similar work

"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning