AQUAACT: A Joystick-Level Underwater Teleoperation Dataset and Closed-Loop Benchmark
Authors: Zhenqi Wu, Trung Tien Dong, Aditya Penumarti, Mohamed A. Fouad, Menna A. Fouad, Mingyang Mao, Jane Jaejeong Shin, Xiaomin Lin
Organizations: University of South Florida, Tampa, FL, USA. · University of Florida, Gainesville, FL, USA. · Seoul National University, Seoul, Republic of Korea.
Underwater robots support a growing range of work, from inspecting offshore structures and pipelines to surveying reefs and accompanying divers. Most of these tasks, however, are still flown by hand by a pilot, which raises the question of whether they can be carried out autonomously instead. On land, vision-language-action (VLA) models already let robots carry out tasks autonomously from language instructions, which makes them a natural candidate for taking the pilot's place. Underwater, however, there is little data to train such models. We take inspiration from how these tasks are flown today: the pilot watches the camera image and steers with a joystick, whose commands the flight controller turns into motion. We therefore record the pilots and introduce AQUAACT, to our knowledge the first publicly available language-annotated underwater dataset pairing camera images with human joystick commands. It comprises 1.9 hours of real-world and 17.9 hours of simulated demonstrations, both flown by human pilots, together with the vehicle's inertial measurements. Experiments with AQUAACT show that its simulated demonstrations transfer to real data and greatly reduce the real demonstrations a policy needs. A closed-loop benchmark of seven policies in simulation, from imitation-learning baselines to VLA models, each trained with its default recipe, shows that these tasks remain far from solved: no policy completes more than a fifth of the episodes. Pretraining on land data gives large VLA models no advantage over small policies trained from scratch, a gap that land-trained models have yet to close underwater.
Figures & tables
Fig. 1: The BlueROV2 Heavy at Blue Grotto, Florida: scanning the submerged deck (left), scanning the concrete block (top right) and following divers (middle and bottom right).
Fig. 2: Firmware-in-the-loop simulation pipeline: the pilot’s joystick commands pass through QGroundControl and the ArduSub firmware to the simulated vehicle, and the recorder stores the camera, inertial and joystick streams (IMU traces shown with gravity removed). Traj.: trajectories.
Dataset
Underwater
Real
Sim
Action space
Lang.
Open X-Emb. [ 21 ]
–
✓
–
EE / joint
✓
USIM [ 24 ]
✓
–
✓
Thruster PWM, arm
✓
AquaBot [ 25 ]
✓
✓
–
Gamepad, arm (pool)
–
Khanmeh [ 38 ]
✓
✓
–
RC, no video
–
AquaAct (ours)
✓
✓
✓
Joystick (RC)
✓
TABLE I: Comparison with related datasets.
Fig. 3: Steady response to held joystick inputs: real vehicle (black) versus simulator (red); surge and sway speed from the Doppler log, yaw rate from the IMU. Pale: input held for less than 2.5 s.
Fig. 4: Fine-tuning from public weights versus fine-tuning after pretraining on simulated deck episodes, as a function of the real episodes used, for SmolVLA (left) and π0.5 (right). Rows: correlation with the pilot (pooled over test episodes), direction accuracy, error; columns: surge, sway, heave, yaw.
Part
Task family
Count
Frames
Hours
Real
Diver following
10
23,511
0.65
Deck scan
18
21,663
0.60
Other (7 tasks)
25
26,567
0.74
Sim
Diver following
137
202,721
5.63
Shipwreck scan
79
169,609
4.71
Oyster-reef insp.
78
120,408
3.34
TABLE II: AquaAct statistics. Count: episodes (real) or recordings (simulated).
Model
Full
Coll.-free
≥ 75
≥ 50
≥ 25
BC
0/50
0/50
3/50
8/50
21/50
U0
0/50
0/50
4/50
11/50
24/50
SmolVLA
2/50
0/50
3/50
9/50
21/50
MemoryVLA
2/50
2/50
5/50
8/50
14/50
ACT
9/50
3/50
10/50
15/50
26/50
Diff. Policy
9/50
6/50
14/50
22/50
24/50
TABLE III: Closed-loop results in simulation, fixed start: number of episodes out of 50 per model. Full: all criteria met; Coll.-free: full and without any collision; ≥ 75/50/25: completion at least 75/50/25%.
Underwater robotics develops autonomous and remotely operated machines for navigation, inspection, manipulation, and exploration in aquatic environments.
Recent advances in vision-language-action models have stimulated growing interest in underwater embodied intelligence. However, their reliance on large-scale interaction data limits their applicability underwater, where data collection is costly and scarce. To address this challenge, we present AquaCap, a training-free Code-as-Policy framework for autonomous underwater navigation and manipulation. AquaCap employs a dual-layer agent that translates task instructions and environmental observations into condition-aware plans and executable control programs. Structured perception then provides the agent with semantic, geometric, and reliability-aware observations under degraded underwater conditions. A failure-aware memory diagnoses unsuccessful actions and supports closed-loop replanning and code revision. This design enables online adaptation without task-specific training or parameter updates. AquaCap achieves a 66.43% success rate in simulation. Real-world experiments further demonstrate autonomous grasping and object transport with an ROV, including the manipulation of targets displaced by hydrodynamic disturbances.
Xiaoshi Li, Yule Xu, Chunghiu Kong +5
Sun Yat-sen University · Dalian University of Technology · Southern Marine Science and Engineering Guangdong Laboratory
Underwater robots rely on complementary sensors whose reliability changes abruptly with water visibility and vehicle motion. We introduce AquaJEPA, a sensor-configurable family of action-conditioned joint-embedding predictive models spanning full multimodal, camera-only, sonar-only, and sensor-dropout configurations. Its members share a latent objective and receding-horizon control interface that predict future representations and physical dynamics from camera, forward-looking sonar, proprioception, and thruster commands. Trained from scratch on one hour of action-labelled data, the family is evaluated in Stonefish on 120 fresh paired scenarios spanning unseen layouts, visibility changes, dynamics shifts, and scheduled DVL loss. AquaJEPA-base achieves the strongest aggregate closed-loop performance, improving success over state-only by 12.5 percentage points and reducing final error by 0.189 m; both paired 95% intervals exclude zero. In a separate three-seed evaluation, it reduces paired final error relative to AquaJEPA-S by 0.118 m, with the same direction for every seed. AquaJEPA-robust more than halves prediction error during camera and camera-DVL blackouts. These results show that full multimodal prediction improves over state-only control and the sonar-only family member in this benchmark, while sensor-dropout training provides robustness under sensor loss.
Underwater manipulation couples visual decisions, contact forces and a thruster-controlled floating base. We introduce WasserMan, to our knowledge the first multi-task simulation benchmark for visuomotor learning of floating-base underwater contact manipulation. It provides ten expert-solvable tasks, two vehicle-arm platforms, a bimanual configuration and underwater dynamics. Nine tasks have learned-policy evaluations. We compare ACT, diffusion policies (DP) and behavioral cloning on six tasks, with three training runs and equal sampled-window budgets. Pretrained SmolVLA adds evaluations on three tasks. In the tested settings, removing integral action can prevent completion, while changing action interfaces can reduce learned-policy success despite successful expert replay. Currents produce task-dependent changes in completion and actuation effort. Versioned tasks, demonstrations and per-episode evidence support a protocol separating expert feasibility, learned completion and execution effort.
Danil Belov, Artem Erkhov, Sergei Parsegov +1
Central University, Moscow, Russia · Center for Engineering Systems and Sciences, Moscow, Russia · Sirius University of Science and Technology, Sirius Federal Territory, Russia +1