A Multi-View 3D Telepresence System for XR Robot Teleoperation
Authors: Enes Ulas Dincer, Manuel Zaremski, Alexandra Nick, Elias Wucher, Barbara Deml, Gerhard Neumann
Organizations: Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, 76137 Karlsruhe, Germany · Institute of Human and Industrial Engineering, Karlsruhe Institute of Technology, 76137 Karlsruhe, Germany
Robot teleoperation is critical for applications such as remote maintenance, fleet robotics, search and rescue, and data collection for robot learning. Effective teleoperation requires intuitive 3D visualization with reliable depth cues, which conventional screen-based interfaces often fail to provide. We introduce a multi-view VR telepresence system that (1) fuses geometry from three cameras to produce GPU-accelerated point-cloud rendering on standalone VR hardware, and (2) integrates a wrist-mounted RGB stream to provide high-resolution local detail where point-cloud accuracy is limited. Our pipeline supports real-time rendering of approximately 75k points on the Meta Quest 3. A within-subject study was conducted with 31 participants to compare our system to other visualisation modalities, such as RGB streams, a projection of stereo-vision directly in the VR device and point clouds without providing additional RGB information. Across three different teleoperated manipulation tasks, we measured task success, completion time, perceived workload, and usability. Our system achieved the best overall performance, while the Point Cloud modality without RGB also outperforming the RGB streams and OpenTeleVision. These results show that combining global 3D structure with localized high-resolution detail substantially improves telepresence for manipulation and provides a strong foundation for next-generation robot teleoperation systems.
Figures & tables
Fig. 1: Leader–follower teleoperation system. One Panda arm acts as the leader, physically guided in gravity-compensation mode, while the second acts as the follower in the manipulation workspace. The workspace is observed by three static RGB-D cameras (left, right, upper) and a wrist-mounted RGB-D camera. The participant wears a Meta Quest 3 headset and experiences one of the four visualization conditions (RGBs, PC, PC+RGB, OT).
Fig. 2: Visualization modalities evaluated in our VR teleoperation study, illustrated here for the cup–insertion task. From left to right: (a) RGBs: four virtual screens showing left, right, top, and wrist RGB streams; (b) PC: semantically filtered fused point cloud rendered together with the leader arm mesh; (c) PC+RGB: the same fused point cloud augmented with a wrist–mounted RGB view rigidly attached to the end–effector. The magenta inset shows a zoomed crop of this wrist RGB stream, highlighting the available high–resolution local detail; (d) OT: ego-centric stereo view from the OpenTeleVision setup.
Fig. 3: An overview of the study procedure, highlighting the elements of data collection and randomization of both the displayed visualization modalities and the manipulation tasks.
Fig. 4: The three manipulation tasks used in the study, shown as start (top), intermediate (middle), and goal (bottom) states: (a) two sequential cup insertions with occluding obstacles, (b) T-shape stacking/assembly, and (c) wire-loop placement around an L-shaped stand.
Fig. 5: Boxplots of NASA–TLX ratings across the four visualization conditions (RGBs, PC, PC+RGB, OT). Higher values indicate higher subjective workload. Each subplot represents one workload dimension. Asterisks indicate the level of statistical significance (* p < .05, ** p < .01, *** p < .001).
Fig. 6: Boxplots of VR usability ratings (effectiveness, efficiency, satisfaction) across the four visualization conditions (RGBs, PC, PC+RGB, OT). Asterisks denote significant pairwise differences based on Holm-corrected Wilcoxon tests (* p < .05, ** p < .01, *** p < .001).
Fig. 7: Participants’ subjective preference rankings for the four visualization methods across all preference levels.
Fig. 8: Task-wise performance across visualization conditions (RGBs, PC, PC+RGB, OT). Left: mean completion time for each task. Right: success rate for each task. Asterisks denote significant pairwise differences based on Holm-corrected Wilcoxon tests (* p<.05 , ** p<.01 , *** p<.001 ).
Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomy, or restrict operators to high-level commands. We present GHOST: an open-source VR teleoperation system that enables single operator control of two mobile manipulators via direct lowlevel commands using only onboard sensing. GHOST creates an exocentric 3D workspace by aligning real-time point clouds from the robots' RGB-D cameras, where scene coverage is improved through learning-based completion to aid operator spatial awareness. For control, the operator uses a mode-switching architecture to command either robot individually or both robots simultaneously. Experiments with 15 novice participants demonstrate 1.6-4x the success rate of an off-the-shelf tablet interface. For experts across nine challenging dual-robot tasks, our system enabled completion of two tasks that were infeasible with the tablet, and was 1.47x faster on average than the tablet. Website and code: https://h2r.github.io/GHOST/.
Yichen Wei, Faisal Zaghloul, Soujanya C Aryal +5
Department of Computer Science, Brown University, Providence, RI 02912, USA
Current robot-assisted minimally-invasive surgery (RMIS) platforms provide a fixed console for the surgeon to view stereo endoscopic images and teleoperate instruments inside the patient. Several researchers have proposed the use of a head-mounted display (HMD) as a portable console, with video pass-through rendering of the endoscope images which, like the fixed console, restricts the operator to a single endoscopic viewpoint and limits depth perception. We present a digital twin-driven virtual reality (VR) teleoperation platform, where the digital twin is created from markerless perception of the surgical environment and streamed for display on the HMD. This overcomes the limitations of video pass-through by providing multi-view rendering and natural motion-parallax cues, enabling decoupling of the user's hand posture from strict instrument alignment. The system utilizes VR hand controllers to increase the teleoperation workspace and to improve the robustness and stability of instrument control compared to the hand tracking approach adopted by most prior systems. A 15-participant user study on the da Vinci Research Kit (dVRK) shows that our VR platform significantly outperforms a state-of-the-art HoloLens 2 mixed reality baseline, reducing path length by 86% and jerk by 95%, while achieving depth perception confidence comparable to or exceeding the traditional console across all conditions.
Chang Liu, Chenhao Yu, Honghao Zhao +5
Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD, USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
Executing contact-rich tasks efficiently requires the seamless integration of whole-body coordination and physical compliance regulation. However, existing teleoperation and data-collection frameworks often overlook the joint consideration of multimodal perception and coordinated whole-body operation. This limitation can reduce the efficiency and quality of demonstration collection, thereby affecting the effectiveness of downstream policy learning. In this work, we present M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation, enabling stable physical interaction and capturing aligned visual, tactile, force, and proprioceptive observations during task execution. Extensive experiments demonstrate that the proposed framework significantly improves contact-rich teleoperation performance. The proposed controller reduces the force tracking error from 4.132N to 0.346N, the contact loss from 2.46 to 0.02 events per trial and the tactile deformation error by 65%. User studies across four mobile manipulation tasks also verify the reliability and usability of the proposed system. Furthermore, Diffusion Policy experiments highlight the value of joint tactile and force sensing.
Hengxiang Chen, Shenwen Deng, Yujian Ma +3
Arbeit Gruppe Dexterous Robotics Lab, Shenzhen Technology University, China