Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

Yuqing Lin; Youngroung Kim

cs.ROSep 13, 2026

Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

Authors: Yuqing Lin, Youngroung Kim

Organizations: School of Civil and Environmental Engineering, Nanyang Technological University, Singapore

Abstract

Autonomous Surface Vehicles (ASVs) are increasingly expected to operate in ports and harbour environments, where operators may specify navigation targets through language-based descriptions rather than predefined coordinates or fixed target identifiers. However, existing ASV navigation methods mainly execute predefined geometric goals or task-specific objectives and give limited attention to language-grounded target specification. This study proposes Semantically Grounded Navigation (SGNav), a framework that enables an ASV to identify and approach a maritime target from an operator-provided description. SGNav integrates text-guided semantic grounding, harbour-aware candidate filtering, CLIP-based semantic verification, grounded target control-state construction, and Proximal Policy Optimisation-based closed-loop control. It grounds the target description in onboard RGB observations, suppresses visually or semantically irrelevant distractors, and converts the selected target into a compact control-oriented representation for policy execution. Experiments in simulated port environments show that SGNav achieves success rates of $97.0\pm1.2\%$ , $92.0\pm1.5\%$ , and $90.0\pm1.8\%$ across three representative target-reaching tasks, with semantic target accuracy above $97\%$ and wrong-target rates below $3\%$ . SGNav also maintains $97.7$ -- $98.7\%$ success rates across held-out port layouts. In the Task~3 ablation study, removing harbour-aware filtering or semantic consistency reduces the success rate to $40.4\pm2.6\%$ and $50.4\pm3.1\%$ , respectively. These findings demonstrate the importance of semantic grounding, harbour-aware filtering, and semantic verification for reliable language-grounded ASV navigation. These results indicate that the proposed perception-to-control framework can support language-grounded target approach manoeuvres of ASV under the complex port environments.

Figures & tables

Figure 1: Overall framework of SGNav for language-grounded semantic target navigation. The framework consists of a language-grounded target resolution front-end and a downstream policy execution back-end.

Target description $g$	Target prompts $\mathcal{P}^{+}(g)$	Harbour distractor prompts $\mathcal{P}^{-}(g)$
Magenta Floating Marker	magenta floating marker, magenta floating ball, floating magenta object	vessel, ship, pier, dock, wave region, land structure, irrelevant buoy
green navigation buoy	green navigation buoy, green buoy, green navigation marker	vessel, ship, pier, dock, wave region, land structure, irrelevant marker, incompatible buoy class
yellow patrol boat	yellow patrol boat, yellow boat, patrol vessel	buoy, pier, dock, wave region, land structure, irrelevant marker, non-target vessel

Table 1: Examples of rule-based prompt construction in SGNav. Target prompts are generated from the operator-provided description and predefined target-expression templates, while harbour distractor prompts are selected from a predefined vocabulary after removing target-compatible terms.

Figure 2: Simulation environment and target-reaching task configurations used in the benchmark. The left panel shows a Unity top-down overview of the simulated port environment, while the right panel provides a simplified schematic of the evaluated task layout. The schematic is not a one-to-one geometric reproduction of the Unity scene, but highlights the ASV start zone, representative distractors, navigation aids, and the three target-reaching tasks: Task 1 (magenta floating marker), Task 2 (large green buoy), and Task 3 (yellow patrol boat).

Simulation factor	Representative elements	Evaluation role
Semantic targets and distractors	Navigation buoys, floating markers, patrol boats, safe-water marks	Test target grounding and distractor rejection.
Port structures	Terminal, dock, pier, berth area, quay crane, turning basin, inbound channel	Introduce occlusion, clutter, and spatial constraints.
Vessels and traffic	Container ships, cargo vessels, tugs, tankers, docked and moving vessels	Increase target ambiguity and collision risk.
Visual appearance variation	Wave motion, water reflection, sunlight reflection, shadows	Affect visibility, colour consistency, and grounding reliability.
Scene ambiguity	Background clutter, object similarity, partial occlusion	Test semantic discrimination and target verification.

Table 2: Key elements of the simulated port environment and their evaluation roles.

Component	Per refresh (ms)	Amortised (ms/step)
GroundingDINO	$310.0\pm 18.0$	$62.0$
CLIP verification	$22.0\pm 4.0$	$4.4$
Filtering and state construction	$3.0\pm 1.0$	$0.6$
PPO inference	$0.5\pm 0.1$	$0.5\pm 0.1$
Total	$335.5\pm 18.5$	$67.5$

Table 3: Runtime of SGNav components measured over 2,000 evaluation frames.

Type	Task 1: Magenta floating marker	Task 2: Large green buoy	Task 3: Yellow patrol boat
Seen	Go to the magenta target point.	Go to the larger green buoy.	Navigate to the yellow patrol boat.
Synonym	Head to the pink-purple target point.	Go to the emerald buoy.	Head to the yellow maritime patrol boat.
Attribute	Go to the bright magenta point marker.	Go to the larger green buoy floating on the water.	Go to the bright yellow patrol boat.
Context	Ignore the boats and move to the magenta point.	Go to the largest green buoy instead of the small one.	Move toward the yellow vessel, not the magenta point.
Long	Please move forward through the scene and stop when you reach the magenta target point.	Please navigate across the water and stop at the largest green buoy.	I want you to approach the yellow patrol boat as the final destination in this scene.

Table 4: Instruction inputs used in Experiment 2.

Panel of Table 5 — Table 5: Controlled distractor settings used in Experiment 3. Each task includes one target and three distractors corresponding to colour-level, shape-level, and semantic-level ambiguity.

Figure 3: Controlled port layout variations used in Experiment 4. Panel (a) shows the training layout, corresponding to Layout A, while Panels (b)–(d) show the held-out evaluation layouts, corresponding to Layouts B–D. The four panels show the ASV starting point, target goal zone, sample route, and channel boundary. The layouts differ in corridor clearance, dock and pier configuration, obstacle distribution, and target placement.

Layout	Corridor clearance	Dock/pier topology	Obstacle complexity	Target placement
A	12 m	Single-side straight dock	Low	Open channel, fully visible
B	10 m	Right-side modified pier	Medium	Offset target, changed approach route
C	7 m	Bilateral narrow passage	High	Near berth, constrained approach
D	15 m	Multi-region open basin	Medium	Open basin, long-range approach

Table 6: Controlled layout factors used in Experiment 4.

Task	Method	SR (%)	STA (%)	WTR (%)	CT (sim. s)	PE
Task 1	Vision-Only PPO	$10.3\pm 1.5$	N/A	N/A	$489.9\pm 18.6$	$0.745\pm 0.031$
	No-Target PPO	$5.2\pm 1.0$	N/A	N/A	$669.1\pm 24.7$	$0.423\pm 0.028$
	Oracle PPO	$97.0\pm 1.5$	N/A	N/A	$157.6\pm 7.9$	$0.996\pm 0.012$
	Proposed SGNav	$\mathbf{97.0\pm 1.2}$	$\mathbf{98.59\pm 0.64}$	$\mathbf{1.41\pm 0.64}$	$\mathbf{283.0\pm 12.4}$	$\mathbf{0.821\pm 0.027}$
Task 2	Vision-Only PPO	$4.1\pm 0.9$	N/A	N/A	$709.6\pm 31.5$	$0.435\pm 0.024$
	No-Target PPO	$1.0\pm 0.6$	N/A	N/A	$869.1\pm 36.8$	$0.223\pm 0.019$

Table 7: Performance comparison of different methods in Experiment 1. Results are reported as mean

\pm

standard deviation over three random seeds.

Figure 4: Representative navigation trajectories for the three semantic navigation tasks in Experiment 1. From top to bottom, the rows correspond to magenta floating marker, large green buoy, and yellow patrol boat navigation. Each row shows the simulator view and the corresponding simplified top-down layout, where the dashed trajectory illustrates the ASV motion from the shared initial position toward the instructed target.

Panel of Figure 5 — Figure 5: Representative success and failure episodes for Task 1: Magenta Floating Marker navigation. The top row shows representative successful cases, where the target is detected from different viewpoints and distances. The bottom row shows failure cases, including out-of-bounds and timeout cases, which indicate that navigation can still fail when the ASV leaves the valid area or cannot complete the approach within the step limit.

Panel of Figure 6 — Figure 6: Representative success and failure episodes for Task 2: Green Buoy navigation. Successful cases demonstrate that SGNav can ground the buoy, maintain stable tracking, and guide the ASV toward the target. Failure cases include out-of-bounds, obstacle-related failure, and timeout, showing the influence of navigation constraints and surrounding distractors on closed-loop execution.

Panel of Figure 7 — Figure 7: Representative success and failure episodes for Task 3: Yellow Patrol Boat navigation. Successful cases show that the target Patrol Boat can be localized when it is visible, closely grounded during approach, and repeatedly detected across frames. Failure cases include obstacle collision, out-of-bounds, and timeout, suggesting that large maritime targets can be grounded reliably but still introduce navigation challenges in cluttered scenes.

Figure 8: Instruction generalisation results in Experiment 2. The left panel reports the success rate under five instruction types, including seen instructions, synonym substitution, attribute-enriched descriptions, context-aware descriptions, and long natural-language instructions. The right panel reports the corresponding grounding accuracy. Results are shown for Tasks 1–3. Error bars indicate the standard deviation over repeated evaluation runs, and the dashed horizontal line indicates the mean performance under seen instructions.

Panel of Figure 9 — Figure 9: Qualitative results of Task 1 under different semantic instruction types. The target is the magenta point object, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Panel of Figure 10 — Figure 10: Qualitative results of Task 2 under different semantic instruction types. The target is the large green buoy, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Panel of Figure 11 — Figure 11: Qualitative results of Task 3 under different semantic instruction types. The target is the yellow patrol boat, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Task	CTR (%)	DRR colour (%)	DRR shape (%)	DRR semantic (%)	WTR (%)	ColR (%)
Task 1	$97.7\pm 1.5$	$99.0\pm 1.0$	$96.3\pm 2.1$	$91.7\pm 3.2$	$2.3\pm 1.5$	$0.3\pm 0.6$
Task 2	$96.0\pm 2.0$	$96.7\pm 1.5$	$98.7\pm 1.5$	$95.3\pm 2.1$	$3.0\pm 1.7$	$1.0\pm 1.0$
Task 3	$98.0\pm 1.7$	$99.3\pm 1.2$	$98.0\pm 1.7$	$97.3\pm 1.5$	$1.7\pm 1.5$	$0.7\pm 0.6$

Table 8: Performance of SGNav under controlled distractor settings in Experiment 3. Results are reported as mean

\pm

standard deviation over three random seeds.

Panel of Figure 12 — Figure 12: Qualitative examples of controlled distractor settings in Experiment 3. Rows correspond to Tasks 1, 2, and 3, while columns correspond to colour, shape, and semantic distractors, respectively. Green boxes indicate instructed targets, and white boxes indicate distractors.

Method	Eval. layout	SR (%)	STA (%)	ColR (%)	CT (sim. s)	PE
Vision-Only PPO	Layout B	$0.3\pm 0.6$	N/A	$2.0\pm 1.0$	N/A	N/A
	Layout C	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
	Layout D	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
No-Target PPO	Layout B	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
	Layout C	$0.0\pm 0.0$	N/A	$0.0\pm 0.0$	N/A	N/A
	Layout D	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A

Table 9: Cross-layout generalisation results for Task 1. All methods are trained on Layout A and evaluated on held-out layouts. Results are reported as mean

\pm

standard deviation over three random seeds.

Figure 13: Representative trajectories and onboard observatios in the training and held-out layouts. Panels (a)–(d) show SGNav trajectories and onboard camera grounding views in Layouts A, B, C, and D, respectively. Layout A is the training layout, while Layouts B, C, and D are held-out evaluation layouts. In each layout, the ASV follows a target-directed trajectory towards the instructed Magenta Floating Marker under different port configurations.

Task	Variant	SR (%)	STA (%)	WTR (%)	DRR (%)
Task 1	Full SGNav	$\mathbf{97.0\pm 1.2}$	$\mathbf{98.59\pm 0.64}$	$\mathbf{1.41\pm 0.64}$	$\mathbf{98.6\pm 0.8}$
	w/o Harbour-Aware Filtering	$93.6\pm 1.7$	$93.6\pm 1.5$	$6.4\pm 1.5$	$93.8\pm 1.8$
	w/o Semantic Consistency	$95.6\pm 1.4$	$99.6\pm 0.4$	$0.4\pm 0.4$	$99.2\pm 0.7$
Task 3	Full SGNav	$\mathbf{90.0\pm 1.8}$	$\mathbf{97.65\pm 0.72}$	$\mathbf{2.35\pm 0.72}$	$\mathbf{97.6\pm 0.9}$
	w/o Harbour-Aware Filtering	$40.4\pm 2.6$	$85.2\pm 2.4$	$14.8\pm 2.4$	$85.0\pm 2.7$
	w/o Semantic Consistency	$50.4\pm 3.1$	$52.4\pm 3.0$	$47.6\pm 3.0$	$52.8\pm 3.2$

Table 10: Ablation study results on Tasks 1 and 3. Results are reported as mean

\pm

standard deviation over three random seeds.

Explore similar work

Sep 20, 2026

RiverVLN: Phase-Grounded Temporal Vision--Language Navigation for Unmanned Surface Vehicles

Vision-language navigation (VLN) has largely been developed for indoor and terrestrial robots, where language can often be treated as a static goal and motion is approximated by discrete or near-instantaneous actions. These assumptions break down for unmanned surface vehicles (USVs): river navigation requires continuous motion under inertia and limited maneuverability, while long-horizon instructions must be executed through sparse and visually ambiguous maritime landmarks. We introduce RiverVLN, to our knowledge the first benchmark designed for long-horizon USV VLN under continuous riverine motion, and PGT-NAV, a phase-grounded temporal navigation framework for USVs. Rather than directly mapping an entire instruction to motion, PGT-NAV converts it into an ordered sequence of visually verifiable semantic phases and maintains the active phase online through grounded visual and motion evidence. This explicit semantic progress state is fused with visual-motion history and phase-specific grounding to predict six local SE(2) pose increments. The resulting trajectory is executed in a predict-execute-re-observe loop, where the vessel executes toward W3, updates phase and grounding, and replans through a map-based safety layer. Experiments show that PGT-NAV substantially reduces recursive position and heading drift relative to GNM-style and ViNT-style baselines and achieves an average success rate of 0.79 in Unity-ROS closed-loop navigation. Unseen bridge-opening trials and real-world USV experiments further demonstrate that the phase-grounded representation transfers from controlled evaluation to physical USV deployment.

Jieling Wu, Yuehao Huang, Jiajun Lv +3

Robot Navigation Vision-Language Navigation

Ask PDF

Feb 26, 2026

WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

While autonomous navigation has achieved remarkable success in passive perception (e.g., object detection and segmentation), it remains fundamentally constrained by a void in knowledge-driven, interactive environmental cognition. In the high-stakes domain of maritime navigation, the ability to bridge the gap between raw visual perception and complex cognitive reasoning is not merely an enhancement but a critical prerequisite for Autonomous Surface Vessels to execute safe and precise maneuvers. To this end, we present WaterVideoQA, the first large-scale, comprehensive Video Question Answering benchmark specifically engineered for all-waterway environments. This benchmark encompasses 3,029 video clips across six distinct waterway categories, integrating multifaceted variables such as volatile lighting and dynamic weather to rigorously stress-test ASV capabilities across a five-tier hierarchical cognitive framework. Furthermore, we introduce NaviMind, a pioneering multi-agent neuro-symbolic system designed for open-ended maritime reasoning. By synergizing Adaptive Semantic Routing, Situation-Aware Hierarchical Reasoning, and Autonomous Self-Reflective Verification, NaviMind transitions ASVs from superficial pattern matching to regulation-compliant, interpretable decision-making. Experimental results demonstrate that our framework significantly transcends existing baselines, establishing a new paradigm for intelligent, trustworthy interaction in dynamic maritime environments.

Runwei Guan, Shaofeng Liang, Ningwei Ouyang +9

Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Hubei Key Laboratory of Inland Shipping Technology (Wuhan University of Technology), Wuhan, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China. +5

Video QA Video Reasoning Unmanned Surface Vehicle Multimodal Agents Neuro-Symbolic Reasoning

Ask PDF

Aug 20, 2026

World-Model-Grounded LLM Planning for AUV and ASV Navigation Near Offshore Wind Farms

Large language models can turn a natural-language mission into a sequence of robot actions, but they do not have a sense of physics: they cannot judge how long a command should run, or whether it will make the robot drift into an obstacle. We proposed the use of a world model to expand the capabilities of Large Language model-based planners. Our method has three components: a physics-grounded neural world model, a three-phase gradient-based trajectory optimizer, and a Model Predictive Controller (MPC)-style closed-loop replanner with a trust-region guard. The language model decides what to do, and the world model decides how long, whether that means driving eight thrusters through 6 DOF or two differential thrusters through 3 DOF. We evaluate two marine vehicle classes operating near offshore wind infrastructure: a 6-DOF Autonomous Underwater Vehicle (AUV) and a 3-DOF differential-drive Autonomous Surface Vehicle (ASV). In five benchmark missions per platform, both vehicles reach every goal with zero predicted collisions, and both transfer to GazeboSim under ocean current, waves, and thruster dynamics, remaining collision-free and cutting GazeboSim goal-distance error versus the ungrounded baseline by 70-82% (ASV) and roughly 93% (AUV), after a residual fine-tuning pass that separately reduces surrogate rollout Root Mean Square Error (RMSE) by 60% (AUV) and 69% (ASV). For the ASV we further demonstrate a Vision language model (VLM)-assisted semantic-mapping pipeline that extracts obstacles and environmental context from satellite imagery, nautical charts, and forecast Application Programming Interface (API) instead of onboard sensors, reaching 96% navigability accuracy as a drop-in replacement for hand-specified obstacle geometry.

Markus Buchholz, Ignacio Carlucho, Yvan R. Petillot

Norwegian Defence Research Establishment (FFI), Kjeller, Norway · School of Engineering & Physical Sciences, Heriot-Watt University, Edinburgh, UK

Large Language Model-Based Robot Planning Robot Navigation World Model-Based Planning Autonomous Underwater Navigation

Ask PDF