Human-Machine Interfaces for Subsea Telerobotics: From Soda-straw to Natural Language Interactions
Organizations: RoboPI Laboratory. University of Florida, Gainesville, FL, USA · ICIC Laboratory. University of Florida, Gainesville, FL, USA
Abstract
This review explores the evolution of human-machine interfaces (HMIs) in subsea telerobotics, charting the progression from traditional first-person "soda-straw" consoles-- characterized by narrow field-of-view camera feeds-- to contemporary interfaces leveraging gesture recognition, virtual reality, and natural language processing. The review makes three key contributions. First, we introduce a structured framework that analyzes subsea HMIs through three interrelated aspects: operator experience, robotic autonomy, and bidirectional human-machine communication. Second, using these aspects, we compare visual, haptic, extended-reality, gesture, and natural-language interfaces; examine the progression from direct teleoperation to shared autonomy; and assess the distinct roles of simulators and digital twins in training, prototyping, and mission support. Third, we synthesize the reported evidence to distinguish established industrial tools, selectively deployed technologies, and research-stage capabilities, while identifying limitations in interoperability, communication reliability, immersive feedback, and field validation. From this synthesis, we outline standardization needs and research directions for hydrodynamic feedback, latency-aware shared autonomy, grounded language interaction, and real-to-simulation transfer. The report concludes by identifying open research questions and proposing a roadmap toward more intelligent and intuitive subsea HMI systems.
Figures & tables
| Open | Video | Waypoint | Scene | 3D Visual | Fault | Mission | Max. # | Supported | |
| Source † | Recorder † | Navigation † | Mapper ‡ | Supported ‡ | Diagnosis ‡ | Replay † | Vehicles † | OS: W,M,L † | |
| QGroundControl [ 158 ] | ✓ | ✓ | ✓ | ✕ | ✕ | ✓ | ✕ | ✓✓✓ | |
| Mission Planner [ 141 ] | ✓ | ✕ | ✓ | ✕ | ✕ | ✓ | ✕ | ✓✓✕ | |
| APM Planner2 [ 13 ] | ✓ | ✓ | ✓ | ✓ | ✕ | ✓ | ✕ | ✓✓✓ | |
| EOD Workspace [ 60 ] | ✕ | ✓ | ✓ | ✓ | ✕ | ✓ | ✕ | ✓✕✕ | |
| VideoRay Cockpit [ 206 ] | ✕ | ✓ | ✕ | ✕ | ✓ | ✓ | ✓ | ✓✕✕ |
| Feedback | Feedback | Control | Ideal Use Cases ⧫ | User Comfort | Selected |
| Location | Type † | Mechanism † | & Ergonomics ⊛ ⧫ | References | |
| Fingertips | Vibrotactile | Wearable gloves, | Grasping, collecting sample, | High | [ 38 , 167 ] |
| stylus | controlling valve/knob | [ 115 ] | |||
| Wrist, | Force | Wearable gloves, | Sensing drag and tension, | Medium | [ 209 , 189 , 177 ] |
| forearm | exoskeleton | picking object | [ 57 , 56 ] | ||
| Eye | Real/virtual | HMD | Inspecting infrastructure, | Medium | [ 74 , 47 , 37 ] |
| (a) Comparison based on sensor integration capability, scenario libraries, and available robotic agents. | ||||
| Sensor Plugin † ‡ | Built-in Scenarios † | Robot Models † ‡ | Optimal Usage ⧫ | |
| PaleBlue [ 151 ] | Laser | Reefs, oil and gas rigs | Triton XLX ⋄ | VR-supported rig inspection |
| Abyssal [ 6 ] | - | Rigs, sand cloud, pipelines | Abyssal ROV | Manipulator control |
| M8-R Sim [ 130 ] | - | 140+ coral reefs with starfish | M8-R ROV | Education & novice training |
| Era VideoRay [ 48 ] | Sonar, lights | 6 scenarios with shipwrecks | VideoRay Pro5 ⋄ , Defender ⋄ | Training for VideoRay ROVs |
| GRi VROV [ 61 ] | Sonar | Pipelines and rigs | LittleGeek, BigGeek | Maintenance, rescue, mine detection |
| Type of | Human (Teleoperator) Side | Machine (Underwater ROV) Side | Selected | ||
| Intelligence | Responsibility † ⧫ | Cognitive Load ⧫ | Intelligent Capabilities † | Major Applications ⧫ | References |
| Command | Issue high-level | Medium | Translate to low-level action | Surveying, grasping, | [ 39 , 174 ] |
| interpretation | commands | and check feasibility | inspections, interventions | ||
| Online course | Set new goal point | Low | Update local planner and | Dynamic path planning | [ 117 , 224 ] |
| adjustment | or trajectory | mission parameters | in uncertain environment | ||
| Error | Supervise and act | Medium | Diagnose issues, report, | Fault tolerant system for | [ 97 , 216 ] |
Human-Robot Interaction
Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.
See all 475 papers