Universal Manipulation Interface (UMI) devices allow human experts to demonstrate tasks for robot learning without teleoperating physical robots. Using these demonstrations as training data still requires a consistent hardware setup, calibration, and data processing. We present UMI-Live, a full-stack open-source bimanual UMI system for in-the-wild data collection. We design our custom UMI-Live hardware to integrate camera and LiDAR--inertial sensing, onboard compute, and power, without external tracking equipment or calibration within the collection environment. We implement a simple user interface for recording data while an asynchronous pipeline performs parallel SLAM reconstruction and publishes training-ready datasets. We also release software that supports dual-YAM teleoperation and VLA policy training and inference. We validate UMI-Live data quality by fine-tuning π0.5 and MolmoAct2, achieving success rates of 10/10 and 8/10, respectively on a bimanual pick-and-place task. Furthermore, we release 40 hours of UMI-Live data across three tasks, and complete hardware designs, software, and reproduction guides at our project website.
Figures & tables
Figure 1: UMI-Live connects in-the-wild bimanual collection and automated processing to LeRobot datasets, policy training, and physical evaluation on dual i2RT YAM arms.
Figure 2: Assembled UMI-Live hardware. (a) Front view showing the wrist camera and ArUco tags. (b) Side view showing the operator-facing buttons and battery module. (c) Rear view showing the LCD and cable routing.
Figure 3: Exploded CAD view of one UMI-Live unit with its main components labeled.
Figure 4: Redesigned gripper fingers and their installation on a YAM arm.
Specification
Value
Mass per unit with battery
1.42 kg
Gripper width
80 mm
Runtime per charge
1.5 h
Hardware cost per pair
$1,930
Table 1: Hardware specifications of the evaluated revision.
Figure 5: LCD interface with example values.
Figure 6: Data collection and processing in UMI-Live. (a) Each gripper i∈{L,R} records wrist images Ii , LiDAR data Pi , and IMU data Ui . (b) Raw episodes R(e) are processed in parallel to estimate jaw widths wti , reconstruct poses Tti , and align the streams. (c) We assemble demonstration trajectories T(e) with wrist images, 14-dimensional bimanual states, and task instructions into dataset D . Training samples use observations ot=(ItL,ItR,gtL,gtR,ℓ) and query-relative action chunks At∈RH×20 , as defined in Appendices E and F .
Figure 7: Handheld collection and physical robot evaluation. (a) Wrist-camera views at two moments in a UMI demonstration collected at a different location and under different lighting from the robot evaluation, brightened for display. (b) Dual i2RT YAM arms with UMI-Live gripper fingers, six oranges, and a bowl. (c) Selected frames from one autonomous π0.5 rollout that ends with all six oranges in the bowl. Frames are ordered from left to right. Appendix C , Figure 13 , shows the full sequence.
Policy
Training condition
UMI / teleoperation
Successful trials
MolmoAct2
Released checkpoint
—
0/10
π0.5
Fine-tuned
95% / 5%
9/10
π0.5
Fine-tuned
90% / 10%
10/10
MolmoAct2
Fine-tuned
75% / 25%
8/10
Table 2: Physical orange-to-bowl evaluation on dual i2RT YAM arms. Every fine-tuned condition starts from its released checkpoint and receives 12,000 updates with a global batch size of 32. Success requires placing all six oranges in the bowl.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Components used to assemble a UMI-Live gripper, including the LiDAR, camera, Raspberry Pi, LCD, printed parts, linear guide, gears, fasteners, and battery and power electronics.
Figure 9: Mechanical and electronic components of UMI-Live. (a) Gripper mechanism with sector gears, bearing-supported pivots, and a linear guide. (b) Trigger and return spring within the handle. (c) Removable power module with the battery, DC-DC converter, power switch, and wiring. (d) Onboard electronics, showing the Raspberry Pi and LCD before LiDAR installation (left), operator-facing buttons and the rear-mounted LCD (center), and the front camera beneath the LiDAR (right).
Figure 10: Small-object grasping and insertion with UMI-Live. Each row begins at tool pickup and follows the handoff between the grippers, insertion into the tool case, and release. Six uncropped frames per row are selected at key moments and shown in chronological order from left to right.
Figure 11: Bimanual match lighting with UMI-Live. A human operator coordinates the two grippers to open the matchbox, retrieve a match, and strike it. The final frame shows the lit match. Frames are shown in chronological order from left to right and top to bottom.
Figure 13: One autonomous π0.5 rollout on dual i2RT YAM arms, transferring six oranges from the table into a bowl. Frames are shown in chronological order from left to right and top to bottom.
Figure 14: Teleoperation setup. (a) A VIVE tracker replaces the LiDAR on each handheld gripper. (b) External base-station tracking supports teleoperation of the dual YAM arms.
Figure 15: Wrist images and reconstructed TCP position trajectories during bimanual folding. Each column shows the left wrist view (top), right wrist view (middle), and the two TCP position traces in a common plotting frame (bottom).
Hyperparameter
Value
Global batch size
32
Parameters updated
Full model
Peak learning rate
2.5×10−5
AdamW β
(0.9,0.95)
AdamW ϵ
10−8
Weight decay
0.01
Appendix
Table 3: π0.5 fine-tuning hyperparameters for the fresh-base runs.
Hyperparameter
Value
Global batch size
32
Parameters updated
VLM LoRA adapters and full action expert
Peak adapter learning rate
5×10−5
Peak action expert learning rate
10−4
AdamW β
(0.9,0.95)
AdamW ϵ
10−6
Appendix
Table 4: MolmoAct2 fine-tuning hyperparameters for the fresh-base run.
Joint j
θjoffset
dj
aj
αj
1
0
0.1135
0.02
−π/2
2
−π
0
0.264
−π
3
−2.901418
0
0.252239
0
4
−0.240173
0
0.073999
π/2
5
π/2
0
0
π/2
6
π/2
0.0356
0
−π
Appendix
Table 5: Classical Denavit–Hartenberg parameters for the YAM arm.
Real-robot evaluation is essential for understanding whether learned manipulation policies can operate reliably outside curated demonstrations. This need is particularly pressing for Universal Manipulation Interface (UMI)-style policies, whose performance depends on the coupling between wrist-view observations, action representation, data collection, and physical deployment. Existing real-world benchmarks have made important progress, but they are not designed around this UMI data-to-deployment setting. We present UMI-Bench 1.0, a local-first real-robot benchmark for standardized evaluation of UMI-style manipulation policies. To the best of our knowledge, this is the first benchmark dedicated to real-world evaluation of UMI-based manipulation models. UMI-Bench aligns data collection, scene reset, policy execution, result logging, and task-factor analysis within a unified protocol. By making the full evaluation process reproducible and auditable, UMI-Bench provides a practical testbed for measuring how UMI-trained policies generalize to real physical manipulation.
Shi Jin, Yuntian Wang, Yuhui Duan +16
Soochow University · Lumos Robotics · Fudan University +5
We introduce Yielding Universal Bidigital Interface (YUBI), a finger-aligned gripper designed to enable intuitive, ergonomic, and scalable data collection for bimanual dexterous manipulation. While handheld data collection systems such as Universal Manipulation Interface (UMI) enable affordable data collection, their bulky pistol-grip designs can pose ergonomic and usability challenges for fine-grained, dexterous manipulation tasks. To address this, YUBI presents a distinct design principle: yielding, finger-driven actuation that directly maps human finger movements to gripper jaw motion. Using the YUBI devices, we set up a data collection system with integrated VR-based 6 DoF tracking of the gripper, ensuring high-fidelity trajectory data acquisition. We curate a UMI-based dataset of unprecedented scale: 8,434 hours across 1.20M episodes and 119 tasks. Experiments show that YUBI offers advantages over the UMI gripper in versatility for complex bimanual tasks, dexterity, and operational efficiency. A single policy trained on the YUBI dataset transfers across multiple bimanual robots (UR, Franka, and ELEY) simply by mounting the gripper on each platform, confirming that the collected data are directly executable as policy supervision. We release the gripper hardware, data-collection software, and dataset as one integrated stack, offering the open community a reproducible path to large-scale data acquisition for advancing robotic foundation models.
Takehiko Ohkawa, Jumpei Arima, Yuki Noguchi +16
1AI Robot Association (AIRoA) · 2Toyota Motor Corporation · 4The University of Tokyo +5
We present UMI-3D, a multimodal extension of the Universal Manipulation Interface (UMI) for robust and scalable data collection in embodied manipulation. While UMI enables portable, wrist-mounted data acquisition, its reliance on monocular visual SLAM makes it vulnerable to occlusions, dynamic scenes, and tracking failures, limiting its applicability in real-world environments. UMI-3D addresses these limitations by introducing a lightweight and low-cost LiDAR sensor tightly integrated into the wrist-mounted interface, enabling LiDAR-centric SLAM with accurate metric-scale pose estimation under challenging conditions. We further develop a hardware-synchronized multimodal sensing pipeline and a unified spatiotemporal calibration framework that aligns visual observations with LiDAR point clouds, producing consistent 3D representations of demonstrations. Despite maintaining the original 2D visuomotor policy formulation, UMI-3D significantly improves the quality and reliability of collected data, which directly translates into enhanced policy performance. Extensive real-world experiments demonstrate that UMI-3D not only achieves high success rates on standard manipulation tasks, but also enables learning of tasks that are challenging or infeasible for the original vision-only UMI setup, including large deformable object manipulation and articulated object operation. The system supports an end-to-end pipeline for data acquisition, alignment, training, and deployment, while preserving the portability and accessibility of the original UMI. All hardware and software components are open-sourced to facilitate large-scale data collection and accelerate research in embodied intelligence: https://umi-3d.github.io.