Many manipulation tasks require carefully controlled contact forces for successful execution, yet large-scale, force-aware datasets remain scarce. Handheld interfaces can help by transmitting interaction forces directly through the operator's arm, but most existing designs are tailored to simple linear grippers and do not match the nonlinear mechanics of widely used robotic grippers. We present RUMI, a handheld universal manipulation interface matched to the Robotiq 2F-85 with exchangeable fingertip sensors that preserves the target gripper geometry while enabling natural input before contact and force measurement after contact. By integrating the tracking system into a virtual reality headset, our method provides real-time augmented reality feedback about the kinematic feasibility of motion on the target robot platform. We further introduce a haptic feedback mechanism that supports force-aware teleoperation for simulated and real robots. A user study shows fewer episodes containing joint-limit violations with live robot visualization and pose initialization (3/50) than without augmented reality (16/50). We additionally train imitation policies from demonstrations, collected using two different fingertip modalities enabled by the gripper-matched hardware, and assess the haptic interface's capability for collecting force-aware teleoperation datasets in simulation.
Figures & tables
Fig. 1: RUMI supports three demonstration modes: virtual reality (VR), teleoperation, and mixed reality (MR). The hardware features scissor input, selective blocking, Robotiq-matched kinematics, and interchangeable fingers, including CoinFT, standard, and DIGIT.
Interface
Contact sensing
Grasp force
Live robot visualization
Gripper haptics
Tracking / capture setup
Robot counterpart
Two-finger interfaces without fingertip contact sensors
UMI [ 5 ]
–
–
–
–
Visuoinertial SLAM
WSG-50
FastUMI [ 6 ]
–
–
–
–
Realsense T265
Parallel gripper
ForceMimic [ 11 ]
Wrist force
–
–
–
Realsense T265
Parallel gripper
ManipForce [ 12 ]
Wrist force
–
–
–
ArUco tracking
Parallel gripper
ActiveUMI [ 13 ]
–
–
✓ AR Projection
–
Quest VR + Backpack
Custom bimanual system
TABLE I: Reported capabilities of handheld demonstration interfaces. ✓ : reported; –: not reported. Grasp force excludes wrist-only loads; visual guidance is operator-facing; added haptics excludes natural contact.
Fig. 2: Overview of RUMI, a gripper-matched handheld interface for collecting force-aware manipulation demonstrations for robot learning. (a) The hardware combines Robotiq 2F-85-matched kinematics with a D405 wrist camera, a rotary encoder, a Quest controller, and fingertip force/torque (F/T) sensing. (b) The device rests on the back of the hand, while the thumb and index finger operate the rear scissor pads. A human-side CoinFT sensor measures force at the index-side input. (c) The nonlinear transmission couples finger motion, while a selectively engaged blocker constrains further closure. (d) The same interface supports three contact domains: virtual contact in VR, remote robot contact during teleoperation, and direct contact with a local object in MR. In VR and teleoperation, contact engages the blocker, providing resistance at the input pads; subsequent squeeze is measured by the human-side CoinFT sensor. In MR, the local object supplies the physical reaction force, and fingertip F/T sensors measure the interaction while the blocker and human-side force-sensing path remain inactive. Mode scenes are schematic.
Fig. 3: Real-world tasks used to evaluate our data acquisition approach: (a) whiteboard wiping, (b) gear insertion, (c) screw tightening, and (d) box sorting. Evaluation also includes one simulated task (not shown).
Fig. 4: Experimental environments used to evaluate RUMI. (a) Robot-free data collection with QR-code markers and an augmented-reality robot overlay for recording-time validation. (b) Simulated packing task involving a fragile light bulb and a heavy transformer block. (c) Real-world setup for policy deployment using a Franka FR3 robot equipped with a Robotiq 2F-85 gripper.
Fig. 5: Top: Initial poses for all episodes under the two user study conditions, split across translation (left) and rotation (right). Bottom: Average Cartesian space velocity under the two conditions.
Metric
AR
w/o AR
Joint Limit Vio. ↓
3/50
16/50
Joint Velocity Vio. ↓
11/50
29/50
Avg. Manip. ↑
0.055±0.022
0.046±0.0057
TABLE II: Ablation of data quality with and without AR projection. Joint limit- and velocity violations report affected episodes/total episodes, and average manipulability indicates the average value across the entire dataset.
Task
Grasp
Attempt
Success
Gear Assembly (CoinFT)
20/20
20/20
20/20
Gear Assembly (None)
20/20
20/20
7/20
Whiteboard Wiping (CoinFT)
20/20
18/20
16/20
Whiteboard Wiping (None)
20/20
17/20
7/20
Screw Tightening (DIGIT)
17/20
17/20
9/20
Screw Tightening (None)
17/20
17/20
3/20
TABLE III: Task success rates across individual stages and overall execution. Bracket indicates the sensor used to train the model.
Packing Task
Avg. F DS [N]
Avg. F R [N]
Success ( ↑ )
Bulb
5.66
5.42
94%
Block
22.82
19.86
97%
Bulb (no haptic)
18.46
–
0%
Block (no haptic)
15.83
13.67
99%
TABLE IV: Simulated packing results. Haptic feedback produces object-dependent grasp forces in demonstrations (F DS ) that persist in policy rollouts (F R ).
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
Beijing Academy of Artificial Intelligence · MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
Department of Robotics and Mechatronics Engineering, DGIST, Daegu, Korea · Department of Mechanical Engineering, Seoul National University, Seoul, Korea