Learning from demonstration is a promising approach for dexterous manipulation, but collecting high-quality contact-critical demonstrations remains difficult with low-cost teleoperation hardware. We present ViHaTeleop, a lightweight (0.7 kg), low-cost ($550) visual-haptic teleoperation system with SLAM-based wrist tracking, camera-based hand tracking, and finger-wise vibrotactile feedback through Linear Resonant Actuators (LRA). The system includes several design choices (LED illumination, fisheye hand camera, and tactile-aware retargeting constraints) and is deployed on Franka + LEAP Hand + 9DTact in both real and simulated environments. Under matched with/without-haptic conditions with nine participants across six contact-critical tasks, haptics improved success rates across all tasks (+2.2 to +15.6 percentage points), while completion-time effects were task-dependent. Subjective ratings showed significant gains in contact clarity and grasp confidence in both simulation and real-world settings (Wilcoxon signed-rank, p<0.05). We also integrate a lightweight depth-camera-based tactile proxy in Isaac Sim, enabling a full pipeline from multi-modal demonstration collection to visual-tactile policy training. Preliminary downstream validation by training visual-tactile policies from collected demonstrations shows tactile cues benefit contact-critical subtasks (peg-in-hole: +17 percentage points over vision-only).
Figures & tables
Fig. 1: Hardware overview and control architecture of ViHaTeleop. (a) The system consists of two bimanual wrist and hand configuration tracking modules. Each module uses a VSLAM+IMU tracker (HTC Vive Ultimate) for wrist pose tracking and a fisheye camera for hand configuration detection in real-time. LRA rings on the operator’s thumb, index, and middle fingers provide vibrational haptic feedback. (b) The captured wrist and hand configurations are mapped and retargeted to control a robotic arm (Franka) with a dexterous hand (LEAP Hand). Tactile sensors (9DTact) on the robot fingertips provide touch sensory feedback to the operator.
System
Cost
Wearable
Haptic
Occlusion
(USD)
Burden (kg)
Feedback
Robust
NuExo [ 6 ]
> 10k
Heavy (5.2)
✓
✓
BunnyVP [ 7 ]
4k
Light (0.65) ∗
✓
✗
DexCap [ 15 ]
3.6k
Light (1.8) †
✗
✓
AnyTeleop [ 12 ]
50
None
✗
✗
ACE [ 9 ]
600
Medium
✗
✓
TABLE I: Comparison of Teleoperation Systems
Fig. 2: Hardware Components.
Fig. 3: Comparison of standard DexPilot retargeting vs. our tactile-aware extension on two manipulation tasks. (a) Soft tube grasping, (b) Egg grasping. Standard retargeting (left column) produces edge contact with negligible sensor signals, while our constraint (right column) enforces parallel fingertip alignment, enabling effective tactile sensing. Blue insets show tactile sensor heatmaps from thumb and index fingertips.
Fig. 4: Hand skeleton detection in challenging lighting: (a) direct glare toward camera; (b) low-light scene. LED illumination stabilizes detection.
Fig. 5: Experimental tasks for user study. Numbers indicate key stages in task progression. Real-world: (a) egg transfer to bowl, (b) fragile chip grasping. Simulation: (c) peg-in-hole insertion, (d) three-cube stacking, (e) fruit pick-and-place, (f) bimanual bottle handover.
Fig. 6: User study experimental setup and subjective ratings. (a) Experimental snapshots showing teleoperation in real-world (top) and simulation (bottom) environments. (b) Subjective ratings comparing without and with haptic feedback across nine participants. Error bars indicate standard deviation. Statistical comparisons use two-tailed Wilcoxon signed-rank tests; detailed values are reported in Sec. IV-D .
Fig. 7: Dual-purpose tactile proxy pipeline in simulation. A depth camera on each fingertip approximates 9DTact deformation. Top path (blue): the signal drives real-time haptic feedback to the operator via max-deformation extraction and PWM mapping. Bottom path (beige): the same depth data is converted to tactile point clouds and fused with visual observations into a synchronized multi-modal demonstration dataset for downstream policy learning.