WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

Runwei Guan; Shaofeng Liang; Ningwei Ouyang; Weichen Fei; Shanliang Yao; Wei Dai; Chenhao Ge; Penglei Sun

cs.CVFeb 26, 2026

WaterVideoQA: ASV-Centric Perception and Rule-Compliant Reasoning via Multi-Modal Agents

Authors: Runwei Guan, Shaofeng Liang, Ningwei Ouyang, Weichen Fei, Shanliang Yao, Wei Dai, Chenhao Ge, Penglei Sun, +4 more

Organizations: Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Hubei Key Laboratory of Inland Shipping Technology (Wuhan University of Technology), Wuhan, China · School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Suzhou, China. · School of Artificial Intelligence, Nanjing University, Nanjing, China. · School of Information Engineering, Yancheng Institute of Technology, Yancheng, China. · School of Engineering, Stanford University, Palo Alto, USA. · the Centre for AI and Data Science Innovation and the School of Science and Engineering, James Cook University, Smithfield, Australia. · School of Navigation, Wuhan University of Technology, Wuhan, China

Abstract

While autonomous navigation has achieved remarkable success in passive perception (e.g., object detection and segmentation), it remains fundamentally constrained by a void in knowledge-driven, interactive environmental cognition. In the high-stakes domain of maritime navigation, the ability to bridge the gap between raw visual perception and complex cognitive reasoning is not merely an enhancement but a critical prerequisite for Autonomous Surface Vessels to execute safe and precise maneuvers. To this end, we present WaterVideoQA, the first large-scale, comprehensive Video Question Answering benchmark specifically engineered for all-waterway environments. This benchmark encompasses 3,029 video clips across six distinct waterway categories, integrating multifaceted variables such as volatile lighting and dynamic weather to rigorously stress-test ASV capabilities across a five-tier hierarchical cognitive framework. Furthermore, we introduce NaviMind, a pioneering multi-agent neuro-symbolic system designed for open-ended maritime reasoning. By synergizing Adaptive Semantic Routing, Situation-Aware Hierarchical Reasoning, and Autonomous Self-Reflective Verification, NaviMind transitions ASVs from superficial pattern matching to regulation-compliant, interpretable decision-making. Experimental results demonstrate that our framework significantly transcends existing baselines, establishing a new paradigm for intelligent, trustworthy interaction in dynamic maritime environments.

Figures & tables

Fig. 1: The overview of pipeline, including (a) the proposed WaterVideoQA dataset; (b) the proposed multi-agent neuro-symbolic reasoning system: NaviMind; (c) the real-word application scenarios for trustworthy navigation guidance.

Datasets	Venues	Tasks	Temporal	Reasoning	EKR	Scenarios	Videos / Frames	Vocab Size
SMD [ 20 ]	CVPRW ${}_{\text{2019}}$	P	✓	✗	✗	Sea	63 clips / 31.6k frames	N/A
Flow [ 21 ]	ICCV ${}_{\text{2021}}$	P	✓	✗	✗	River	200 clips / 54.6k frames	N/A
LaRS-Seq [ 22 ]	ICCV ${}_{\text{2023}}$	P	✓	✗	✗	Sea, River, Lake	- / 53.3k frames	N/A
USVTrack	IROS ${}_{\text{2025}}$	P	✓	✗	✗	River, Lake	60 clips / 68.8k frames	N/A
USVTrack	IROS ${}_{\text{2025}}$	P	✓	✗	✗	Canal, Moat, Dock	60 clips / 68.8k frames	N/A
WaterVG [ 18 ]	TITS ${}_{\text{2025}}$	P, S, R	✗	✓	✗	River, Lake	N/A	133.9K

TABLE I: A comprehensive comparison of waterway datasets. The name of T-Levels is Task-Levels, including perception (P), scene understanding (S), causal prediction (C), action or interaction (A) and reasoning (R). EKR denotes external knowledge retrieval. The upper and lower row in Vocab Size column denote the sizes of total and unique vocabulary.

Fig. 2: The statistics of our proposed WaterVideoQA dataset, including (a) Sample Type Distribution, (b) Average Q/A Length by Category, (c) Video Duration by Question Category, (d) Question Category Distribution, (e) Answer Type Distribution and (f) Word Cloud of Q&A.

Fig. 3: The overview of annotation process for WaterVideoQA.

Fig. 4: The architecture of NaviMind, a multi-agent reasoning system for waterway navigation. NaviMind has two inputs (a user query and a video clip) and an output answer. It has 5 agents for routing, captioning, reasoning, grading and summary.

Fig. 5: The workflow of Situation-Aware Hierarchical Reasoning.

Methods	Params	ROUGE-1	ROUGE-2	ROUGE-L	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	CIDEr	SPICE	GPT-Score
InternVL3 (f)	1B	0.308	0.111	0.259	0.265	0.143	0.083	0.051	0.281	0.612	0.079	0.288
InternVL3 (f)	2B	0.316	0.113	0.262	0.269	0.170	0.082	0.043	0.303	0.328	0.098	0.307
InternVL3 (nf)	8B	0.178	0.042	0.154	0.092	0.046	0.029	0.020	0.114	0.201	0.081	0.424
InternVideo 2.5 (f)	7B	0.351	0.125	0.274	0.273	0.179	0.090	0.074	0.308	0.369	0.139	0.443
QwenVL-3 (f)	8B	0.340	0.120	0.270	0.287	0.174	0.102	0.090	0.310	0.388	0.151	0.450
MiniCPM-V 4.5 (f)	8B	0.349	0.127	0.279	0.289	0.175	0.107	0.086	0.319	0.399	0.150	0.459

TABLE II: Overall performances on the WaterVideoQA dataset. ”nf” denotes no-finetuned while ”f” denotes finetuned. The Params of Navimind indicates ”(parameters of other agents + Reasoner Agent)”.

Methods	P	S	C	A	R	Time(s)
OmAgent-17B	0.460	0.388	0.451	0.352	0.438	19.78
VideoAgent-24B	0.499	0.421	0.468	0.360	0.465	24.55
NaviMind-11B	0.468	0.416	0.447	0.362	0.454	9.74
NaviMind-15B	0.501	0.447	0.489	0.398	0.477	14.27
NaviMind-21B	0.549	0.481	0.534	0.434	0.531	19.54

TABLE III: Comparison of GPT-Score upon WaterVideoQA characteristics. P : perception; S : scene understanding; C : causal prediction; A : : action/interaction; R : knowledge-based reasoning.

Methods	River	Lake	Canal	Moat	Harbor	Sea
NaviMind-11B	0.452	0.475	0.415	0.422	0.458	0.490
NaviMind-15B	0.485	0.508	0.445	0.455	0.492	0.512
NaviMind-21B	0.538	0.565	0.495	0.505	0.542	0.568

TABLE IV: Performances of various waterways.

Fig. 6: Performance visualization of settings in NaviMind.

Methods	Categories	No. Frames	Lingo-J	BLEU-4	METEOR	CIDEr
LingoQA	zero-	5	33.6	8.33	14.33	39.16
VideoAgent	shot	5	54.18	11.79	18.77	42.64
NaviMind	models	5	60.42	13.76	21.88	46.63
LingoQA	fine-	5	60.80	15.00	18.56	65.62
VideoAgent	tuned	5	68.39	20.07	26.65	85.92
NaviMind	models	5	72.55	23.67	31.41	93.77

TABLE V: Generalization experiments on the LingoQA dataset.

Fig. 7: Visualization of prediction by InternVL3 and our proposed NaviMind.

Fig. 8: Visualization of the limitations.

Explore similar work

Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles
Sep 13, 2026Yuqing Lin, Youngroung KimRobot Navigation Language-Conditioned Robot Control
School of Civil and Environmental Engineering, Nanyang Technological University, Singapore
MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments
Apr 9, 2026Xingming Liao, Ning Chen, Muying Shu +5VLM Evaluation Visual Question Answering
Guangdong University of Technology, Guangzhou, China
AquaCap: A Training-Free Underwater Embodied Agent with Code-as-Policy
Sep 19, 2026Xiaoshi Li, Yule Xu, Chunghiu Kong +5Underwater Robotics Autonomous Underwater Navigation
Sun Yat-sen University · Dalian University of Technology · Southern Marine Science and Engineering Guangdong Laboratory

Sep 13, 2026J/K move · Enter open · S save

Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

Yuqing Lin, Youngroung Kim

School of Civil and Environmental Engineering, Nanyang Technological University, Singapore

Robot Navigation Language-Conditioned Robot Control Vision-Language Navigation Vision-Language Grounding

Open paper AskPDF arXiv

Autonomous Surface Vehicles (ASVs) are increasingly expected to operate in ports and harbour environments, where operators may specify navigation targets through language-based descriptions rather than predefined coordinates or fixed target identifiers. However, existing ASV navigation methods mainly execute predefined geometric goals or task-specific objectives and give limited attention to language-grounded target specification. This study proposes Semantically Grounded Navigation (SGNav), a framework that enables an ASV to identify and approach a maritime target from an operator-provided description. SGNav integrates text-guided semantic grounding, harbour-aware candidate filtering, CLIP-based semantic verification, grounded target control-state construction, and Proximal Policy Optimisation-based closed-loop control. It grounds the target description in onboard RGB observations, suppresses visually or semantically irrelevant distractors, and converts the selected target into a compact control-oriented representation for policy execution. Experiments in simulated port environments show that SGNav achieves success rates of

97.0\pm1.2\%

92.0\pm1.5\%

, and

90.0\pm1.8\%

across three representative target-reaching tasks, with semantic target accuracy above

97\%

and wrong-target rates below

3\%

. SGNav also maintains

97.7

98.7\%

success rates across held-out port layouts. In the Task~3 ablation study, removing harbour-aware filtering or semantic consistency reduces the success rate to

40.4\pm2.6\%

and

50.4\pm3.1\%

, respectively. These findings demonstrate the importance of semantic grounding, harbour-aware filtering, and semantic verification for reliable language-grounded ASV navigation. These results indicate that the proposed perception-to-control framework can support language-grounded target approach manoeuvres of ASV under the complex port environments.

Abstract

Explore similar work

Language-Grounded Semantic Target Navigation for Autonomous Surface Vehicles

MARINER: A 3E-Driven Benchmark for Fine-Grained Perception and Complex Reasoning in Open-Water Environments

AquaCap: A Training-Free Underwater Embodied Agent with Code-as-Policy