Onboard Vision and MPC Navigation for Underwater Robots: An Open BlueROV2 Platform for Multi-Robot Experiments & Docking

Victor Nan Fernandez-Ayala; Wiktor Kowalczyk; Cezary Banaszek; Dimos V. Dimarogonas

cs.ROSep 29, 2026

Onboard Vision and MPC Navigation for Underwater Robots: An Open BlueROV2 Platform for Multi-Robot Experiments & Docking

Authors: Victor Nan Fernandez-Ayala, Wiktor Kowalczyk, Cezary Banaszek, Dimos V. Dimarogonas

Organizations: Department of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden.

Abstract

Autonomous underwater robots require robust perception, estimation and control to operate in confined environments. This paper presents an open-source BlueROV2 platform combining onboard vision with nonlinear Model Predictive Control (NMPC) for autonomous navigation and docking. The platform integrates an NVIDIA Jetson Orin NX and an Intel RealSense D435i stereo camera in a modular pressure housing. Underwater-calibrated stereo depth and realtime object detection provide relative position measurements of nearby BlueROV2 vehicles in the camera and body frames. A quaternion-based estimator fuses external pose and inertial measurements, while an NMPC controller based on a nonlinear six-degree-of-freedom model tracks planned navigation and docking trajectories. To support reproducible development, we also provide open-source physics-based PX4 SITL and Gazebo environments, multi-robot simulation tools and a lowcost physical docking station. Experiments evaluate underwater perception, onboard computational performance, state estimation, trajectory tracking and autonomous docking.

Figures & tables

Fig. 1 : BlueROV2 Heavy with the modular autonomy tube.

Parameter	Value in SI units
$m$ , $V$	$15.6\text{\,}\mathrm{k}\mathrm{g}$ , $0.015\,65\text{\,}\mathrm{m}^{3}$
$\rho$ , $g$	$1000\text{\,}\mathrm{k}\mathrm{g}\mathrm{/}\mathrm{m}^{3}$ , $9.82\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$
$\bm{r}_{b}$	$[0,0,-0.01]^{\top}\,$ \mathrm{m}$$
$[I_{x},I_{y},I_{z}]$	$[0.3000,0.2652,0.4272]\,$ \mathrm{k}\mathrm{g},\mathrm{m}^{2}$$
$\operatorname{diag}(\bm{M}_{A})$	$[6.36,7.12,18.68,0.189,0.135,0.222]$
$\bm{d}_{l}$	$[13.7,0,33.0,0,0.8,0]$

TABLE I : Parameters of the modified BlueROV2 Heavy.

Fig. 2 : Underwater BlueROV2 detection and aligned stereo-depth output produced by the onboard perception module.

Panel of Figure 4 — Fig. 3 : Open experimental infrastructure. The physical docking station and the simulator from the multi-robot launch file.

Parameter	Value
Camera profiles	RGB 1280x720x15fps; D 848x480x30fps
YOLO input resolution	1280x720 pixels
YOLO confidence threshold	0.9
Jetson power mode	15W
IMU sampling rate	200 $\mathrm{H}\mathrm{z}$
Motion-capture rate	100 $\mathrm{H}\mathrm{z}$

TABLE II : Experimental and algorithmic parameters.

Panel of Figure 6 — Fig. 4 : Perception evaluation: (a) stereo range versus motion-capture ground truth; (b) held-out YOLO precision, recall and mAP; and (c) Jetson Orin NX utilization and inference time for the selected profile.

Panel of Figure 7 — Fig. 5 : Docking evaluation: (a) raw motion-capture and EKF surge velocity during the final hold; (b) measured horizontal trajectory, commanded dock pose and final position; and (c) overlaid frames of the physical docking sequence.

Explore similar work

Underwater Visual Target Tracking with Target-Specific Depth Estimation and Adaptive Model-Fusion Predictive Control
Sep 17, 2026Yuheng Zhou, Haiyang Cheng, Yanqi Feng +5Autonomous Underwater Vehicles Underwater Acoustic Target Recognition
Department of Automation, Shanghai Jiao Tong University, Shanghai, China · University of Toronto, Toronto, Canada · University of Calgary, Calgary, Canada
A Vision-based Control Framework for Real-time Autonomous UUV Operations
Aug 5, 2026Erik Tjærand Frøland, Marco Job, Md Ether Deowan +1Autonomous Underwater Vehicles Autonomous Navigation
Dept. of Mechanical and Industrial Engineering, NTNU, Norway · Oceaneering AS, Norway
AquaJEPA: An Action-Conditioned Multimodal JEPA Family for Underwater Robot Dynamics
Jul 31, 2026Alan-Barsag Gazzaev, Alexey Gavrilov, Sergey MuravyovUnderwater Robots Multi-Modal Sensing
ITMO University

Sep 17, 2026, cs.ROJ/K move · Enter open · S save

Underwater Visual Target Tracking with Target-Specific Depth Estimation and Adaptive Model-Fusion Predictive Control

Yuheng Zhou, Haiyang Cheng, Yanqi Feng, Pangkit Fong+4

Department of Automation, Shanghai Jiao Tong University, Shanghai, China · University of Toronto, Toronto, Canada · University of Calgary, Calgary, Canada

Autonomous Underwater Vehicles Underwater Acoustic Target Recognition Visual Servoing Depth Estimation Multi-Object Tracking Model Predictive Control

Open paper AskPDF arXiv

Vision-based underwater target tracking is challenged by unreliable depth measurements and unknown target motion. This paper proposes a stereo visual-servoing framework for an autonomous underwater vehicle (AUV). For perception, the framework derives a stable 3D relative state from stereo images through target-specific depth extraction and Kalman filtering. It constructs a target-depth mask from color, disparity, and temporal cues to select reliable target pixels, and then filters the resulting depth measurement and detected image center separately. For control, the framework decouples yaw regulation from translational control, avoiding computationally expensive coupled multi-DOF optimization and enabling real-time translational MPC. The translational controller employs adaptive model-fusion predictive control, combining constant-velocity and zero-velocity target models to accommodate different target-motion patterns. It updates the model weights using historical prediction errors and computes translational commands subject to actuation, following-distance, and field-of-view constraints. Through simulations and real-world experiments, we validate the effectiveness of the proposed framework and show it has better performance than existing frameworks.

Abstract

Explore similar work

Underwater Visual Target Tracking with Target-Specific Depth Estimation and Adaptive Model-Fusion Predictive Control

A Vision-based Control Framework for Real-time Autonomous UUV Operations

AquaJEPA: An Action-Conditioned Multimodal JEPA Family for Underwater Robot Dynamics