Organizations: Mu Robotics · McGill University · Mila - Quebec AI Institute · Pontificia Universidad Católica del Perú (PUCP) · NONHUMAN · Technical University of Munich (TUM) · Universidad Peruana de Ciencias Aplicadas (UPC)
HandUMI captures robot-free bimanual demonstrations with a hand-worn parallel-jaw interface. Swappable fingertips match the robot's installed gripper; Quest/PICO controllers track hands and encoders measure aperture. Robot-agnostic recordings are retargeted by inverse kinematics to seven simulated bimanual arms (median position error ≤1.2 mm; ≤4.7% of episodes rejected). On bimanual towel folding, 562 robot-free demonstrations train policies on real PiPER arms: ACT succeeds in 12/20 attempts and fine-tuned π0.5 in 7/20; ACT on the first 101 and base π0.5 each succeed in 0/20. Hardware and software are open source (Apache-2.0), and a pair costs USD 557 with Quest 3S, excluding the optional ZED Mini and host computer. Project website: https://murobotics.ai/research/handumi.
Figures & tables
Figure 1: HandUMI. Left: robot-free capture with the hand-worn device (fisheye camera, servo encoder, swappable fingertips) tracked by PICO 4 Ultra or Meta Quest controllers. Right: recorded trajectories retargeted in simulation to seven bimanual arms.
Device
Pose source
Enc.
Bimanual
Open HW
Open SW
Tips
Audio
USD
UMI [ 7 ]
mono-inertial SLAM
×
✓
✓
✓
–
–
742 a
UMI-3D [ 28 ]
LiDAR-inertial odom.
×
×
✓
✓
–
–
1,400 b
FastUMI [ 37 ]
T265 VIO
×
✓
–
✓ c
✓
–
≈ 2,042 d
DexUMI [ 30 ]
iPhone ARKit
✓
– e
✓
✓
n/a
–
≈ 2,448 f
iPhUMI [ 22 ]
iPhone ARKit
×
✓
✓
✓
–
✓ g
≈ 1,504 h
RDT2 [ 17 ]
Vive Tracker (IR)
–
✓
–
✓ i
–
–
≈ 1,750 j
Table 1: Robot-free interfaces. Enc.: encoder-measured gripper state; Open HW/SW: public files/code; Tips: swappable tips. ✓ / × : yes/no; ∘ : announced; –: not established; n/a: not applicable. USD: named parts for a pair with tracking, excluding host, tax and shipping; > : only the headset is priced.
Table 3
Figure 2: Towel folding: views, a rollout, and successes by training episodes (Wilson 95% CI).
Training robot policies for multi-task pick and place requires demonstrations on many objects, which handheld Universal Manipulation Interfaces (UMIs) can collect quickly and cheaply. However, existing UMIs depend on external base stations, use rigid fingers that give no signal about contact forces, or use parallel-jaw fingers that limit grip strength. We present a battery-powered handheld interface, built on YUBI, whose soft, angled fingers wrap around objects, deform visibly under load as a visual proxy for force, and match the grippers of the robot. With it, we collect a 99.7-hour dataset over 53 objects, train π0.5 policies on 49 of them, and hold out the other 4 as unseen objects. We find that a top camera raises success by 13 points even when only about half of the data includes it, and that mid-training on our multi-task data raises success after fine-tuning on unseen objects from 35% to 57%. Finger design, dataset, and data processing code: https://lute-corp.github.io/soft-yubi-site/
Kobi Kelemen, Tomasz Niedziałkowski, Marcin Paszczak +1
Universal Manipulation Interface (UMI) devices allow human experts to demonstrate tasks for robot learning without teleoperating physical robots. Using these demonstrations as training data still requires a consistent hardware setup, calibration, and data processing. We present UMI-Live, a full-stack open-source bimanual UMI system for in-the-wild data collection. We design our custom UMI-Live hardware to integrate camera and LiDAR--inertial sensing, onboard compute, and power, without external tracking equipment or calibration within the collection environment. We implement a simple user interface for recording data while an asynchronous pipeline performs parallel SLAM reconstruction and publishes training-ready datasets. We also release software that supports dual-YAM teleoperation and VLA policy training and inference. We validate UMI-Live data quality by fine-tuning π0.5 and MolmoAct2, achieving success rates of 10/10 and 8/10, respectively on a bimanual pick-and-place task. Furthermore, we release 40 hours of UMI-Live data across three tasks, and complete hardware designs, software, and reproduction guides at our project website.
Govind Velamoor, Adrian Lin, Sam Jung +3
Yale University · Carnegie Mellon University · AfterQuery +2
Real-robot evaluation is essential for understanding whether learned manipulation policies can operate reliably outside curated demonstrations. This need is particularly pressing for Universal Manipulation Interface (UMI)-style policies, whose performance depends on the coupling between wrist-view observations, action representation, data collection, and physical deployment. Existing real-world benchmarks have made important progress, but they are not designed around this UMI data-to-deployment setting. We present UMI-Bench 1.0, a local-first real-robot benchmark for standardized evaluation of UMI-style manipulation policies. To the best of our knowledge, this is the first benchmark dedicated to real-world evaluation of UMI-based manipulation models. UMI-Bench aligns data collection, scene reset, policy execution, result logging, and task-factor analysis within a unified protocol. By making the full evaluation process reproducible and auditable, UMI-Bench provides a practical testbed for measuring how UMI-trained policies generalize to real physical manipulation.
Shi Jin, Yuntian Wang, Yuhui Duan +16
Soochow University · Lumos Robotics · Fudan University +5