Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

Yuqing Lin; Youngroung Kim

Figure 1: Overall framework of SGNav for language-grounded semantic target navigation. The framework consists of a language-grounded target resolution front-end and a downstream policy execution back-end.

Target description $g$	Target prompts $\mathcal{P}^{+}(g)$	Harbour distractor prompts $\mathcal{P}^{-}(g)$
Magenta Floating Marker	magenta floating marker, magenta floating ball, floating magenta object	vessel, ship, pier, dock, wave region, land structure, irrelevant buoy
green navigation buoy	green navigation buoy, green buoy, green navigation marker	vessel, ship, pier, dock, wave region, land structure, irrelevant marker, incompatible buoy class
yellow patrol boat	yellow patrol boat, yellow boat, patrol vessel	buoy, pier, dock, wave region, land structure, irrelevant marker, non-target vessel

Table 1: Examples of rule-based prompt construction in SGNav. Target prompts are generated from the operator-provided description and predefined target-expression templates, while harbour distractor prompts are selected from a predefined vocabulary after removing target-compatible terms.

Figure 2: Simulation environment and target-reaching task configurations used in the benchmark. The left panel shows a Unity top-down overview of the simulated port environment, while the right panel provides a simplified schematic of the evaluated task layout. The schematic is not a one-to-one geometric reproduction of the Unity scene, but highlights the ASV start zone, representative distractors, navigation aids, and the three target-reaching tasks: Task 1 (magenta floating marker), Task 2 (large green buoy), and Task 3 (yellow patrol boat).

Simulation factor	Representative elements	Evaluation role
Semantic targets and distractors	Navigation buoys, floating markers, patrol boats, safe-water marks	Test target grounding and distractor rejection.
Port structures	Terminal, dock, pier, berth area, quay crane, turning basin, inbound channel	Introduce occlusion, clutter, and spatial constraints.
Vessels and traffic	Container ships, cargo vessels, tugs, tankers, docked and moving vessels	Increase target ambiguity and collision risk.
Visual appearance variation	Wave motion, water reflection, sunlight reflection, shadows	Affect visibility, colour consistency, and grounding reliability.
Scene ambiguity	Background clutter, object similarity, partial occlusion	Test semantic discrimination and target verification.

Table 2: Key elements of the simulated port environment and their evaluation roles.

Component	Per refresh (ms)	Amortised (ms/step)
GroundingDINO	$310.0\pm 18.0$	$62.0$
CLIP verification	$22.0\pm 4.0$	$4.4$
Filtering and state construction	$3.0\pm 1.0$	$0.6$
PPO inference	$0.5\pm 0.1$	$0.5\pm 0.1$
Total	$335.5\pm 18.5$	$67.5$

Table 3: Runtime of SGNav components measured over 2,000 evaluation frames.

Type	Task 1: Magenta floating marker	Task 2: Large green buoy	Task 3: Yellow patrol boat
Seen	Go to the magenta target point.	Go to the larger green buoy.	Navigate to the yellow patrol boat.
Synonym	Head to the pink-purple target point.	Go to the emerald buoy.	Head to the yellow maritime patrol boat.
Attribute	Go to the bright magenta point marker.	Go to the larger green buoy floating on the water.	Go to the bright yellow patrol boat.
Context	Ignore the boats and move to the magenta point.	Go to the largest green buoy instead of the small one.	Move toward the yellow vessel, not the magenta point.
Long	Please move forward through the scene and stop when you reach the magenta target point.	Please navigate across the water and stop at the largest green buoy.	I want you to approach the yellow patrol boat as the final destination in this scene.

Table 4: Instruction inputs used in Experiment 2.

Table 5: Controlled distractor settings used in Experiment 3. Each task includes one target and three distractors corresponding to colour-level, shape-level, and semantic-level ambiguity.

Figure 3: Controlled port layout variations used in Experiment 4. Panel (a) shows the training layout, corresponding to Layout A, while Panels (b)–(d) show the held-out evaluation layouts, corresponding to Layouts B–D. The four panels show the ASV starting point, target goal zone, sample route, and channel boundary. The layouts differ in corridor clearance, dock and pier configuration, obstacle distribution, and target placement.

Layout	Corridor clearance	Dock/pier topology	Obstacle complexity	Target placement
A	12 m	Single-side straight dock	Low	Open channel, fully visible
B	10 m	Right-side modified pier	Medium	Offset target, changed approach route
C	7 m	Bilateral narrow passage	High	Near berth, constrained approach
D	15 m	Multi-region open basin	Medium	Open basin, long-range approach

Table 6: Controlled layout factors used in Experiment 4.

Task	Method	SR (%)	STA (%)	WTR (%)	CT (sim. s)	PE
Task 1	Vision-Only PPO	$10.3\pm 1.5$	N/A	N/A	$489.9\pm 18.6$	$0.745\pm 0.031$
	No-Target PPO	$5.2\pm 1.0$	N/A	N/A	$669.1\pm 24.7$	$0.423\pm 0.028$
	Oracle PPO	$97.0\pm 1.5$	N/A	N/A	$157.6\pm 7.9$	$0.996\pm 0.012$
	Proposed SGNav	$\mathbf{97.0\pm 1.2}$	$\mathbf{98.59\pm 0.64}$	$\mathbf{1.41\pm 0.64}$	$\mathbf{283.0\pm 12.4}$	$\mathbf{0.821\pm 0.027}$
Task 2	Vision-Only PPO	$4.1\pm 0.9$	N/A	N/A	$709.6\pm 31.5$	$0.435\pm 0.024$
	No-Target PPO	$1.0\pm 0.6$	N/A	N/A	$869.1\pm 36.8$	$0.223\pm 0.019$

Table 7: Performance comparison of different methods in Experiment 1. Results are reported as mean

\pm

standard deviation over three random seeds.

Figure 4: Representative navigation trajectories for the three semantic navigation tasks in Experiment 1. From top to bottom, the rows correspond to magenta floating marker, large green buoy, and yellow patrol boat navigation. Each row shows the simulator view and the corresponding simplified top-down layout, where the dashed trajectory illustrates the ASV motion from the shared initial position toward the instructed target.

Figure 5: Representative success and failure episodes for Task 1: Magenta Floating Marker navigation. The top row shows representative successful cases, where the target is detected from different viewpoints and distances. The bottom row shows failure cases, including out-of-bounds and timeout cases, which indicate that navigation can still fail when the ASV leaves the valid area or cannot complete the approach within the step limit.

Figure 6: Representative success and failure episodes for Task 2: Green Buoy navigation. Successful cases demonstrate that SGNav can ground the buoy, maintain stable tracking, and guide the ASV toward the target. Failure cases include out-of-bounds, obstacle-related failure, and timeout, showing the influence of navigation constraints and surrounding distractors on closed-loop execution.

Figure 7: Representative success and failure episodes for Task 3: Yellow Patrol Boat navigation. Successful cases show that the target Patrol Boat can be localized when it is visible, closely grounded during approach, and repeatedly detected across frames. Failure cases include obstacle collision, out-of-bounds, and timeout, suggesting that large maritime targets can be grounded reliably but still introduce navigation challenges in cluttered scenes.

Figure 8: Instruction generalisation results in Experiment 2. The left panel reports the success rate under five instruction types, including seen instructions, synonym substitution, attribute-enriched descriptions, context-aware descriptions, and long natural-language instructions. The right panel reports the corresponding grounding accuracy. Results are shown for Tasks 1–3. Error bars indicate the standard deviation over repeated evaluation runs, and the dashed horizontal line indicates the mean performance under seen instructions.

Figure 9: Qualitative results of Task 1 under different semantic instruction types. The target is the magenta point object, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Figure 10: Qualitative results of Task 2 under different semantic instruction types. The target is the large green buoy, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Figure 11: Qualitative results of Task 3 under different semantic instruction types. The target is the yellow patrol boat, and the five cases correspond to seen instruction, synonym substitution, attribute-enriched description, context-aware description, and long natural-language instruction.

Task	CTR (%)	DRR colour (%)	DRR shape (%)	DRR semantic (%)	WTR (%)	ColR (%)
Task 1	$97.7\pm 1.5$	$99.0\pm 1.0$	$96.3\pm 2.1$	$91.7\pm 3.2$	$2.3\pm 1.5$	$0.3\pm 0.6$
Task 2	$96.0\pm 2.0$	$96.7\pm 1.5$	$98.7\pm 1.5$	$95.3\pm 2.1$	$3.0\pm 1.7$	$1.0\pm 1.0$
Task 3	$98.0\pm 1.7$	$99.3\pm 1.2$	$98.0\pm 1.7$	$97.3\pm 1.5$	$1.7\pm 1.5$	$0.7\pm 0.6$

Table 8: Performance of SGNav under controlled distractor settings in Experiment 3. Results are reported as mean

\pm

standard deviation over three random seeds.

Figure 12: Qualitative examples of controlled distractor settings in Experiment 3. Rows correspond to Tasks 1, 2, and 3, while columns correspond to colour, shape, and semantic distractors, respectively. Green boxes indicate instructed targets, and white boxes indicate distractors.

Method	Eval. layout	SR (%)	STA (%)	ColR (%)	CT (sim. s)	PE
Vision-Only PPO	Layout B	$0.3\pm 0.6$	N/A	$2.0\pm 1.0$	N/A	N/A
	Layout C	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
	Layout D	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
No-Target PPO	Layout B	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A
	Layout C	$0.0\pm 0.0$	N/A	$0.0\pm 0.0$	N/A	N/A
	Layout D	$0.0\pm 0.0$	N/A	$0.3\pm 0.6$	N/A	N/A

Table 9: Cross-layout generalisation results for Task 1. All methods are trained on Layout A and evaluated on held-out layouts. Results are reported as mean

\pm

standard deviation over three random seeds.

Figure 13: Representative trajectories and onboard observatios in the training and held-out layouts. Panels (a)–(d) show SGNav trajectories and onboard camera grounding views in Layouts A, B, C, and D, respectively. Layout A is the training layout, while Layouts B, C, and D are held-out evaluation layouts. In each layout, the ASV follows a target-directed trajectory towards the instructed Magenta Floating Marker under different port configurations.

Task	Variant	SR (%)	STA (%)	WTR (%)	DRR (%)
Task 1	Full SGNav	$\mathbf{97.0\pm 1.2}$	$\mathbf{98.59\pm 0.64}$	$\mathbf{1.41\pm 0.64}$	$\mathbf{98.6\pm 0.8}$
	w/o Harbour-Aware Filtering	$93.6\pm 1.7$	$93.6\pm 1.5$	$6.4\pm 1.5$	$93.8\pm 1.8$
	w/o Semantic Consistency	$95.6\pm 1.4$	$99.6\pm 0.4$	$0.4\pm 0.4$	$99.2\pm 0.7$
Task 3	Full SGNav	$\mathbf{90.0\pm 1.8}$	$\mathbf{97.65\pm 0.72}$	$\mathbf{2.35\pm 0.72}$	$\mathbf{97.6\pm 0.9}$
	w/o Harbour-Aware Filtering	$40.4\pm 2.6$	$85.2\pm 2.4$	$14.8\pm 2.4$	$85.0\pm 2.7$
	w/o Semantic Consistency	$50.4\pm 3.1$	$52.4\pm 3.0$	$47.6\pm 3.0$	$52.8\pm 3.2$

Table 10: Ablation study results on Tasks 1 and 3. Results are reported as mean

\pm

standard deviation over three random seeds.

Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

Abstract

Figures & tables

Explore similar work

RiverVLN: Phase-Grounded Temporal Vision--Language Navigation for Unmanned Surface Vehicles

WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

World-Model-Grounded LLM Planning for AUV and ASV Navigation Near Offshore Wind Farms