Organizations: Zhejiang University · Harbin Engineering University · Nanyang Technological University · Harbin Institute of Technology · University of Auckland
Underwater navigation requires translating visual spatial cues into motion, yet marine vision-language datasets rarely include paired robot trajectories. We present NereidVLA, a vision-language-action model that transfers marine spatial knowledge to language-conditioned 3D navigation. We first train Sea2Know on approximately two million marine examples to learn target grounding, spatial relations, and scene structure without paired robot-motion supervision. We then adapt it using expert trajectories executed under simulated underwater vehicle dynamics. Given RGB histories and a language instruction, NereidVLA predicts discrete motion tokens that decode into local 3D trajectories for closed-loop tracking. Evaluation combines an independent six-dimensional underwater navigation understanding benchmark with offline prediction and closed-loop navigation. Under matched navigation training and execution settings, marine spatial pretraining increases simulation success from 72.4% to 84.4%. Without fine-tuning on the real test environment, NereidVLA achieves 78.75% success in laboratory-pool robot trials, compared with 47.50% for InternVLA-N1. These results show that marine spatial supervision can support downstream motion learning and physical underwater navigation.
Figures & tables
Figure 1: Overview of the two-stage learning framework for NereidVLA. Marine spatial learning produces Sea2Know, which initializes the navigation model. Expert demonstrations executed under simulated vehicle dynamics supervise local 3D trajectory prediction, followed by closed-loop tracking with a low-level feedback controller.
Figure 2: Navigation architecture and trajectory supervision. A Sea2Know-initialized navigation VLA predicts discrete motion tokens from four historical RGB observations and a task instruction. Executed expert trajectories provide token-level supervision, and codebook decoding reconstructs local 3D waypoints for closed-loop tracking.
Figure 3: Underwater Navigation Understanding Benchmark. The six panels show representative examples and question proportions for target grounding, spatial relations, passage structure, traversability, temporal understanding, and local action decisions. Each example pairs visual observations with a question, answer options, and the reference answer.
Method
TGA ↑
SRA ↑
TSA ↑
TPA ↑
TUA ↑
LADA ↑
API-based General Models
DeepSeek V4.1 Flash [ 8 ]
87.06
87.66
87.26
86.82
75.48
34.88
Doubao Seed 2.1 Pro [ 5 ]
87.06
84.34
84.91
84.58
72.36
32.43
GPT-6-Sol [ 17 ]
91.54
82.44
80.90
77.73
69.35
33.24
Claude Opus 5 [ 3 ]
85.07
82.92
84.20
83.15
71.15
26.70
Open-source General VLMs
Table 1: Task-oriented underwater VQA evaluation (%). Bold marks the column best; dark/light blue marks the best/second-best Sea2Know results. Parentheses show point changes from size-matched Qwen3-VL backbones (red/green: gains/drops).
Model
ADE ↓
FDE ↓
Z MAE ↓
ADE P95 ↓
FDE P95 ↓
InternVLA-N1
0.20008
0.39498
0.05735
0.67412
1.26874
Qwen3-VL-4B
0.23568
0.45514
0.06190
0.74359
1.37255
Qwen3-VL-2B
0.28983
0.53348
0.06178
0.91908
1.65784
Qwen3-VL-8B
0.29121
0.55396
0.06181
0.91134
1.66065
Qwen3-VL-32B
0.19766
0.38996
0.05219
0.64042
1.20181
NAUTILUS (7B)
0.21748
0.42234
0.05230
0.74137
1.37125
Table 2: Offline trajectory prediction results. All metrics are lower-is-better. Boldface marks the best score in each column.
Method
Task success rate (%) ↑
Overall performance
Obstacle avoidance
Target search / approach
Pipeline navigation
Cave navigation
SR (%) ↑
Jerk RMS ↓
Grid-SPL (%) ↑
U0 †
5.60
19.20
10.40
13.60
12.20
4.1906
10.71
π0.5 -USIM † [ 20 , 9 ]
0.80
8.00
4.80
0.80
3.60
1.7826
3.09
Qwen3-VL-8B
84.00
52.00
71.20
82.40
72.40
1.1247
71.52
Qwen3-VL-32B
76.00
44.00
59.20
65.60
61.20
1.4338
59.47
SimpleVLN (8B)
74.40
38.40
52.00
41.60
51.60
1.9478
49.82
Table 3: Closed-loop navigation in simulation. Each model is evaluated on 500 trials (125 per task). Overall SR averages the success rates across the four equally sized tasks. Bold marks the best score in each column. † : original checkpoints without navigation adaptation.
Figure 4: Closed-loop navigation examples. (a) Target search and in-pipe navigation in simulation, showing successive frames, onboard views, and predicted and executed trajectories. (b) Real-robot goal approach with obstacles and narrow-passage avoidance, shown through external and onboard views. Time labels indicate stages of each execution.
Method
TA
OA
GA
NP
Overall
Qwen3-VL-8B
35.00
80.00
15.00
10.00
35.00
InternVLA-N1
60.00
85.00
30.00
15.00
47.50
Ours
90.00
95.00
70.00
60.00
78.75
Table 4: Real-robot navigation success rates (%, ↑ ). TA: target approach; OA: obstacle avoidance; GA: goal approach with obstacles; NP: narrow-passage avoidance. Overall is the four-task mean.
Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.
Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao +7
Tsinghua University · Pengcheng Laboratory · Xiaomi EV +2
Underwater spatial reasoning depends on object boundaries and geometric cues that are obscured by scattering, color distortion, and low visibility. Existing underwater resources capture these cues through images, annotations, and depth maps, but provide fragmented supervision for vision-language models. We present a spatial data engine that connects visual appearance with object-level geometry through enhancement, 2D and geometry-assisted question generation, and expert filtering. It produces 4.27M QA pairs across five capability groups. We use this corpus to train USI-Brain and introduce an Underwater Prior-guided Rectifier (UPR) that applies gated corrections to visual features. USI-Bench evaluates the resulting capabilities with 6,525 held-out QA pairs. On its 15 task metrics, USI-Brain and USI-Brain + UPR improve over Qwen3-VL-8B-Instruct by 5.76 and 7.63 points on average, respectively, using signed differences on 100-point scales. Ablations show that geometry-assisted supervision improves depth and ordering by 4.8 and 6.1 points with UPR disabled, while UPR adds 18.0 points in scene description under matched training data. Both variants use raw images at evaluation. Across NautData, UWBench, and MarineEval, USI-Brain achieves the best or joint-best score on 28 of 32 metrics. Model weights and USI-Bench will be released at https://github.com/SKY-Zhu-91/USI-Brain.
Vision-and-Language Navigation (VLN) aims to enable an embodied agent to follow natural-language instructions and navigate to a target location in unseen 3D environments. We argue that adapting VLMs to VLN requires endowing them with two complementary capabilities for acquiring such awareness, namely backward action reasoning (why) and forward transition prediction~(how). Based on this insight, we propose SpaAct, a simple yet effective training framework that activates the dynamic spatial awareness in VLMs. Specifically, SpaAct introduces two spatial activation tasks: Action Retrospection, which asks the model to infer the executed action sequence from visual transitions, and Future Frame Selection, which forces the model to predict the visual transitions conditioned on history and action. These two objectives provide lightweight supervision on both backward action reasoning and forward transition prediction, encouraging the model to build dynamic spatial awareness in a VLM-friendly way. To further stabilize adaptation, we design TriPA, a Tri-factor Progressive Adaptive curriculum learning method that organizes training samples from easy to hard, allowing the model to gradually acquire navigation skills from basic locomotion to long-horizon reasoning. Experiments on standard VLN-CE benchmarks show that SpaAct consistently improves VLM-based navigation and achieves state-of-the-art performance. We will release the code and models to support future research.
Pengna Li, Kangyi Wu, Shaoqing Xu +7
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · The State Key Laboratory of Internet of Things for Smart City, Centre for Artificial Intelligence and Robotics, University of Macau · Xiaomi EV +2