cs.ROAug 29, 2026

GHOST in the Robots: Real-Time Exocentric Dual-Robot VR Teleoperation from Onboard Cameras

Authors: Yichen Wei, Faisal Zaghloul, Soujanya C Aryal, Aanya K. Agrawal, Chengfan Li, Jason Xinyu Liu, James Tompkin, Stefanie Tellex

Organizations: Department of Computer Science, Brown University, Providence, RI 02912, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands. We present GHOST: an open-source VR teleoperation system that enables single operator control of two mobile manipulators via direct lowlevel commands using only onboard sensing. GHOST creates an exocentric 3D workspace by aligning real-time point clouds from the robots' RGB-D cameras, where scene coverage is improved through learning-based completion to aid operator spatial awareness. For control, the operator uses a mode-switching architecture to command either robot individually or both robots simultaneously. Experiments with 15 novice participants demonstrate 1.6-4x the success rate of an off-the-shelf tablet interface. For experts across nine challenging dual-robot tasks, our system enabled completion of two tasks that were infeasible with the tablet, and was 1.47x faster on average than the tablet. Website and code: https://h2r.github.io/GHOST/.

Figures & tables

Fig. 1 : GHOST enables single-operator dual-robot control. Our VR teleoperation system enables individual or simultaneous dual-robot control to perform complex mobile manipulation tasks, with a 3D virtual scene reconstructed and rendered in real time from onboard cameras.
WorkNovel environments?Single operator?Low-level actions?Multi-robot coordinationReal-time scene?3D scene?Onboard cameras?
Lewis et al. [ 20 ]✓✓✓Mirror✓✗✓
Roldan et al. [ 34 ]✗✓✗High-level action only✗✓✗
CollabTeleop [ 40 ]✓✗✓N/A✓✗✗
Body Extension [ 14 ]✓✓✓Individual control✓✗✓
Laghi et al. [ 19 ]✗✓✓Pivot✗✗✗
Ozdamar et al. [ 30 ]✗✓✓Pivot✗✗✗
TABLE I : GHOST is a new point in the multi-robot teleoperation design space. Existing multi-robot teleoperation works did not produce real-time 3D scene depictions with low-level multi-robot control.
Fig. 2 : A diagram of the GHOST system. The system integrates multi-robot ROS 2 communication, a GPU-accelerated RGB-D processing pipeline, and a Unity-based VR interface. Low-bandwidth state/command data are bridged through ROS 2 and drive the virtual robot models, while high-bandwidth RGB-D streams are processed on the GPU and rendered as interactive point clouds for low-latency teleoperation.
Fig. 3 : Shared 3D scene before (above) and after (below) real-time depth completion. Completion improves scene coverage and makes the point clouds from both robots’ cameras more coherent.
Fig. 4 : Nine evaluation tasks for dual-robot teleoperation. Tasks require bimanual manipulation (1, 2, 5, 9), address perceptual blind spots (3, 4), or demand spatial context awareness (6, 7, 8). All require coordinated control of two mobile manipulators.
Sensor-frame deliveryDepth completionRender
230 ms230\text{\,}\mathrm{ms}131 ms131\text{\,}\mathrm{ms}14 ms14\text{\,}\mathrm{ms}
TABLE II : Average SpotObserver latency per stage.
Fig. 5 : Tablet interface. Teleoperating two robots requires using two tablets, typically by switching between them.
Fig. 6 : Novice task completion times and success rates.
Fig. 7 : Ablation study: Operator view in RGB-only GHOST.
Fig. 8 : Expert task completion times across interfaces. Full GHOST is faster and more reliable than the tablet baseline on most tasks, while the RGB-only GHOST ablation isolates the effect of direct 6-DoF control without the point-cloud-based exocentric scene. Two tasks were infeasible with the tablet. RGB-only GHOST generally falls between the tablet and full GHOST; when point-cloud quality is poor, it can match or exceed full GHOST.
 GHOST (Ours)GHOST (RGB-only)Tablet
1 - Plush Toy Handoff✓✓✓
2 - Box Pick & Place✓2/3✓
3 - Cube Stacking✓✓✓
4 - Cube Collection✓✓✓
5 - Bag Filling✓✓✓
6 - Door Passthrough✓2/32/3
TABLE III : Expert task feasibility.
#NameDescriptionInitial SceneFinal Scene
1Plush Toy HandoffInitially, a white plush toy is placed in front of the robots. The operator needs to pick up the toy with one hand and hand it over to the other robot.  
2Large Box Picking and PlacingInitially, a large box is placed in front of the two robots. The box is too large that it has to be carried by two robots. The operators need to pick up the box, carry it near the platform covered with orange cloth, and place it on top of the platform.  
33-Cube StackingInitially, three cubes are placed in a line in front of the robot. The operator needs to stack the three cubes in one stack in whatever order they deem best.  
4Collect 3 Scattered Cubes into the BasketInitially, a basket and three cubes are scattered around in the scene in front of the robot. The operator is required to navigate and find all three cubes and place them into the basket in the center.  
5Opening and Holding Bags While Filling Them with 2 CubesInitially, one white plastic bag and two cubes are placed in front of the robots. The plastic bag is almost flattened, so it requires one robot to hold up the bag in order to have the opening big enough to fit the gripper and the cube. Thus, the operator will have to use one robot to hold up the bag, while the other robot looks around and searches for the cubes, and then places them into the held-up bag.  
6Weighted Door Opening and Pass ThroughThe robots need to open a weighted door and make one robot go through the door. The door is too heavy for the Boston Dynamics Spot’s automatic door-opening function, which will not properly open the door and let the robot through.  
TABLE IV: Task Descriptions for each task (Part 1)
#NameDescriptionInitial SceneFinal Scene
7Dust Pan SweepInitially, two cubes, one broom, and one dust pan are placed in front of the robot. The operator needs to control the robot to use the broom to sweep the two cubes into the dust pan.  
8Bedsheet FoldingInitially, a very large, long sheet is placed in front of the robots. The operator needs to fold it in half. Note that since the sheet is so large, the robot will have to move the base in order to properly fold the sheet.  
9Rope Tying 1 KnotInitially, a thick rope is placed in front of the two robots. The operator needs to control the robot and tie a knot on the rope.  
TABLE V: Task Descriptions for each task (Part 2)

Explore similar work

CardsList

  1. A Multi-View 3D Telepresence System for XR Robot Teleoperation

    Apr 4, 2026Enes Ulas Dincer, Manuel Zaremski, Alexandra Nick +3Robot TeleoperationVirtual Reality

    Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76137 Karlsruhe, Germany · Institute of Human and Industrial Engineering, Karlsruhe Institute of Technology, 76137 Karlsruhe, Germany

  2. Design and Evaluation of a Collaborative Teleoperation Interface for Multi-Operator Multi-Robot Systems in Challenging Environments

    Oct 10, 2026Alperen Kenan, Juan José García Cárdenas, Adriana Tapus +2Robot TeleoperationMulti-Robot Systems

    Bristol Robotics Laboratory, University of the West of England, Bristol, United Kingdom · Computer Science and System Engineering Department (U2IS), Autonomous Systems and Robotics Lab, ENSTA, Institut Polytechnique de Paris, Paris, France · Kempten University of Applied Sciences, Kempten, Germany

  3. A Multi-Operator Mixed-Reality Interface for Multi-Robot Control and Coordination: Co-Located and Private Workspace Collaboration

    Jun 5, 2026Omotoye Shamsudeen Adekoya, Antonio Sgorbissa, Carmine Tommaso RecchiutoRobot TeleoperationMulti-Robot Coordination

    DIBRIS Department, RICE Laboratory, University of Genoa, Italy