A wrist-mounted camera for UMI-style data collection must do two jobs: record the manipulation and localize in the scene. Most handheld devices localize online from workspace-facing views crowded by hands and objects, or add dedicated tracking hardware. Room-scale bimanual capture therefore still tends to instrument the operator or the scene for accurate localization. We present KIWI (Kinematic Interface for the Wild), a capture kit whose only electronics are off-the-shelf cameras. Our core system splits the two jobs across the two lenses of a 360-degree camera. The rear lens faces the room and builds a shared metric map that registers both hands, and an optional head camera, in one frame without workspace co-visibility; the front lens records the manipulation, and offline IMU fusion bridges front-lens tracking loss. Through our quick-release plate, the camera module attaches to chopstick grippers, parallel-jaw grippers, hand-wrist mounts, or robot flanges. Across six bimanual recordings, combining the rear and front lenses failed to localize only 0.1% of query frames, whereas front-only bimanual feature alignment failed on 24.8% of frames and lost one recording entirely; against evaluation fiducials, localization error stayed within 4.5 mm. KIWI's recovered poses were sufficiently consistent for the four wrist streams alone to reconstruct the scene as a 3D Gaussian splat.
Figures & tables
Feature
KIWI (ours)
UMI [ 1 ]
UMI-3D [ 2 ]
iPhUMI [ 3 ]
FastUMI [ 4 ]
RDT2 [ 5 ]
YUBI [ 6 ]
EgoMI [ 7 ]
HiFi-UMI [ 8 ]
XRZero-G0 [ 9 ]
DAS Fingers [ 10 ]
Sensor suite
Insta360 X5, GO 3
GoPro Hero9
Livox Mid-360, global-shutter cam
iPhone 15 Pro
GoPro Hero9, RealSense T265
Global-shutter cam, Vive Tracker
ELP fisheye, Quest 3S, rotary encoder
Quest 3S, ZED 2i, wrist cams
Proprietary
PICO 4, ego + wrist cams
Proprietary
Extra compute free
✓
✓
✗
✓
✗
✗
✗
✗
✗
✗
✗
Wrist 6-DoF trajectory
Offline SLAM
Online SLAM
Online LiDAR + IMU
Online ARKit
Online RealSense T265
Online Vive IR
Online VR
Online VR
Offline SLAM + marker
Online VR
Online VIO
Ego view
✓
✗
✗
✓
✗
✗
✓
✓
✓
✓
✓
Inter-hand pose
✓ map
✓ map
N/A
✓ ARKit
NR
✓ Vive
✓ VR
✓ VR
✓ head
✓ VR
✓ Ego
Scene coverage
360∘ wrist + ego
155∘ + mirrors
185∘ + LiDAR
Head + wrist
169∘ wrist
Wrist
180∘ + chest
Head + wrist
∼200∘ + head
Head + wrist
150∘ + Ego
Table 1: Demonstration interfaces: collection hardware and recovered state.
Figure 2: Modular hardware overview. Top: exploded view of the three-part design: an interchangeable gripper module, the shared camera module, and a robot mounting plate. Bottom: (A) chopstick and (B) parallel-jaw handheld grippers and (C) the bare-hand wrist cuff; (D, E, F) the robot-mounted twins.
Figure 3: Chopstick gripper geometry and hinge mechanism. Left: side view of the assembled gripper and camera, showing the angled grip that provides clearance for the camera body. Right: section view of the bearing-supported jaw pivots and meshing gears.
Figure 4: KIWI offline pipeline. (A) One shared rear map and two front maps recover metric scale and gravity from IMU measurements and are aligned using calibrated front–rear geometry. (B) Each episode reuses these maps for independent per-wrist visual–inertial pose estimation, complemented by tool-opening, head-marker, and ground estimation from native videos and recovered poses. (C) Outputs include synchronized demonstration datasets with tool-tip poses and quality records, alongside a 3D Gaussian scene reconstructed from native fisheye videos and aligned camera poses for visual real-to-sim.
Figure 5: Shared-scene registration from fisheye views. (a) Rear-view correspondences illustrate the common scene reference S . (b) Temporal correspondences in each front stream support separate map frames FL and FR . Fixed front–rear calibration Ea and synchronized localizations determine the rigid placements Ga:Fa→S in Eq. ( 2 ). Colored lines indicate computed image matches.
Figure 6: End-effector state for (a) the chopstick and (b) the parallel-jaw gripper. Top: fronto-parallel views of the gripper opening with detected dots and per-jaw line fits at uniform opening steps. The end-effector module estimates state independently on every frame of the 120 fps recordings. Bottom: single-cycle excerpts of the vision estimate (blue) versus the servo-encoder ground truth (orange). RMSE against the encoder is 0.17∘ in (a) and 1.9 px in (b).
Figure 7: Ground-height estimation and pose retargeting. (a) Candidate horizontal planes from known camera poses. (b) Agreement across 20 refocused frames selects the floor ( H=1.25 m) within the plausible camera-height range; the table lies at H=0.46 m. (c) Height-aligned poses retargeted to two embodiments. H denotes camera-to-plane distance.
Figure 8: Task and workspace diversity across six configurations. Images are auxiliary egocentric views and the wrist-front view of box folding.
Recording
Duration
Frames
Front-only (%)
KIWI (%)
Kitchen 1
317.0
38,006
95.509
99.997
Kitchen 2
294.8
35,340
99.997
99.997
Workbench
331.1
39,692
93.734
99.997
Bolt organization
414.5
49,687
0.000
99.998
Tabletop transfers
338.4
40,571
91.765
99.998
Box folding
357.2
42,820
91.240
99.998
Table 2: Visual tracking availability.
Figure 9: Held-out 3DGS renders. Recorded images (left) and held-out 3DGS renders (right) from Kitchen 1 and Kitchen 2. Camera poses are fixed; rear-view people are blurred.
Handheld data-collection interfaces such as the Universal Manipulation Interface (UMI) allow manipulation demonstrations to be gathered in the wild without a robot in the loop, but the quality of the resulting action labels is bounded by the accuracy of vision-based localization. Existing systems recover end-effector pose from monocular visual-inertial odometry, which is subject to scale ambiguity and drift, or delegate tracking to closed-source modules and external infrastructure that reintroduce cost and setup burden. We present OpenJaws, a fully open hardware and software framework for sensorized UMI that improves localization and simplifies deployment across robot platforms. OpenJaws pairs a modular, 3D-printed gripper exposing a standardized ISO adapter with a low-cost stereo camera to recover metric camera trajectories. We further introduce UMI-Loc-Bench-v1, a benchmark quantifying real-world localization accuracy for handheld data-collection devices, and show that stereo localization substantially outperforms monocular alternatives. Finally, we demonstrate that policies trained on data collected with OpenJaws can be deployed on a real robot across multiple tasks. Hardware, software, and data are released under a commercially permissive license. Code, data, and videos are available at https://open-jaws.github.io/
Multi-view perception is often treated as a safe design choice for robot manipulation: if one camera is useful, more cameras should be better. We revisit that assumption in tabletop bimanual manipulation by comparing policies trained with two large-field-of-view (FoV) wrist cameras against policies that additionally receive a head view, using controlled-variable experiments on a real bimanual platform. Across three tabletop tasks, adding a head view does not consistently improve task completion or execution quality: when the wrist cameras already observe the objects, grippers, and goal regions, wrist-only policies form a strong baseline. Restricting the wrist-camera FoV increases the potential value of a head view, but performance improves most reliably when the added view supplies concentrated, task-relevant information: region-of-interest (ROI) cropping yields more consistent head-view gains than the uncropped image, which is dominated by background and task-irrelevant regions. The data-scaling results further suggest an interaction between FoV and demonstration count: wide observations improve coverage but may require more data to stabilize, and they may reach a higher performance ceiling once enough demonstrations are available. These results suggest that camera design should be framed as a task-relevance and information-density problem rather than as a simple count of input views.
Yuhui Duan, Shi Jin, Yuntian Wang +5
Soochow University · Independent Researcher · Shanghai Jiao Tong University +2
We present UMI-3D, a multimodal extension of the Universal Manipulation Interface (UMI) for robust and scalable data collection in embodied manipulation. While UMI enables portable, wrist-mounted data acquisition, its reliance on monocular visual SLAM makes it vulnerable to occlusions, dynamic scenes, and tracking failures, limiting its applicability in real-world environments. UMI-3D addresses these limitations by introducing a lightweight and low-cost LiDAR sensor tightly integrated into the wrist-mounted interface, enabling LiDAR-centric SLAM with accurate metric-scale pose estimation under challenging conditions. We further develop a hardware-synchronized multimodal sensing pipeline and a unified spatiotemporal calibration framework that aligns visual observations with LiDAR point clouds, producing consistent 3D representations of demonstrations. Despite maintaining the original 2D visuomotor policy formulation, UMI-3D significantly improves the quality and reliability of collected data, which directly translates into enhanced policy performance. Extensive real-world experiments demonstrate that UMI-3D not only achieves high success rates on standard manipulation tasks, but also enables learning of tasks that are challenging or infeasible for the original vision-only UMI setup, including large deformable object manipulation and articulated object operation. The system supports an end-to-end pipeline for data acquisition, alignment, training, and deployment, while preserving the portability and accessibility of the original UMI. All hardware and software components are open-sourced to facilitate large-scale data collection and accelerate research in embodied intelligence: https://umi-3d.github.io.