Underwater robotic grasping is difficult due to degraded, highly variable imagery and the expense of collecting diverse underwater demonstrations. We introduce a system that (i) autonomously collects successful underwater grasp demonstrations via a self-supervised data collection pipeline and (ii) transfers grasp knowledge from on-land human demonstrations through a depth-based affordance representation that bridges the on-land-to-underwater domain gap and is robust to lighting and color shift. An affordance model trained on on-land handheld demonstrations is deployed underwater zero-shot via geometric alignment, and an affordance-conditioned diffusion policy is then trained on underwater demonstrations to generate control actions. In pool experiments, our approach improves grasping performance and robustness to background shifts, and enables generalization to objects seen only in on-land data, outperforming RGB-only baselines. Code, videos, and additional results are available at https://umi-under-water.github.io.
Figures & tables
Fig. 1: UMI-Underwater. We address data-collection and generalization bottlenecks in underwater manipulation by pairing autonomous, self-supervised data collection with a zero-shot, depth-based affordance predictor that transfers directly from land to water.
Fig. 2: ROV Manipulation Setup for Self-supervised Underwater Data Collection. The setup includes a swimming pool, with objects scattered across the workspace for repeated grasp attempts. Four fixed external cameras are mounted at the pool corners to provide real-time 3D localization used for safety functions (but not provided as inputs to the learned policy). The tethered ROV operates in this environment to execute the staged grasping routine.
Fig. 3: Autonomous Data Collection Pipeline. Our heuristic controller autonomously collects grasping episodes by using a segmentation model to select a target, servoing the object centroid to stage-specific pixel setpoints using PD control, closing the gripper when a depth threshold is met, and labeling success via drag validation.
Fig. 4: Autonomous Recovery Strategies include (a) regrasp after failed grasps and (b) backup when the robot overshoots. These strategies improve the success rate of demonstration collection as well as improve policy robustness by demonstrating recovery behavior.
Fig. 5: UMI-Aquatic on-land demonstration setup. Our handheld gripper with an iPhone camera system and AprilTags enables portable data collection and reliable gripper-state tracking for automatic demonstration labeling. Cropping and geometric warping via reprojection align the iPhone view to match the underwater robot camera.
Fig. 6: Automatic Affordance Supervision. AprilTag tracking identifies gripper closure timing using a sliding window, which seeds point tracking to backtrack grasp contact keypoints on the object for affordance supervision.
Fig. 7: Model Architecture. (a) Observation preprocessing takes monocular RGB current and goal observations from robot cameras and converts them to depth maps using Depth Anything V2. (b) Affordance prediction takes top camera depth and goal depth observations and uses a convolution U-Net to predict a dense grasp affordance heatmap for target localization. The online affordance model for underwater deployment is trained on exclusively on-land UMI-Aquatic data and is deployed underwater zero-shot. We supervise affordance training using automatically labeled keypoint maps that indicate the ground truth grasp points. (c) Visuomotor policy conditions on the predicted affordance map, depth observations, and robot proprioception to generate a short-horizon action trajectory.
Method
Success (%) ↑
DP + Aff + Depth (ours)
85
DP + RGB
65
TABLE I: ID performance (seen objects + training pool background).
Central University, Moscow, Russia · Center for Engineering Systems and Sciences, Moscow, Russia · Sirius University of Science and Technology, Sirius Federal Territory, Russia +1
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden. · Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China.