cs.ROSep 19, 2026

Kinematic Interface for the Wild: Modular Bimanual Loco-Manipulation Capture from 360∘^{\circ} Cameras Alone

Authors: Benjamin Yang, Weiying Wang, Shenggao Li, Keming Yan, Sasha Wilkinson, Zelin Wang, Yip Fun Yeung, Lingfeng Sun

Organizations: Autel US

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A wrist-mounted camera for UMI-style data collection must do two jobs: record the manipulation and localize in the scene. Most handheld devices localize online from workspace-facing views crowded by hands and objects, or add dedicated tracking hardware. Room-scale bimanual capture therefore still tends to instrument the operator or the scene for accurate localization. We present KIWI (Kinematic Interface for the Wild), a capture kit whose only electronics are off-the-shelf cameras. Our core system splits the two jobs across the two lenses of a 360-degree camera. The rear lens faces the room and builds a shared metric map that registers both hands, and an optional head camera, in one frame without workspace co-visibility; the front lens records the manipulation, and offline IMU fusion bridges front-lens tracking loss. Through our quick-release plate, the camera module attaches to chopstick grippers, parallel-jaw grippers, hand-wrist mounts, or robot flanges. Across six bimanual recordings, combining the rear and front lenses failed to localize only 0.1% of query frames, whereas front-only bimanual feature alignment failed on 24.8% of frames and lost one recording entirely; against evaluation fiducials, localization error stayed within 4.5 mm. KIWI's recovered poses were sufficiently consistent for the four wrist streams alone to reconstruct the scene as a 3D Gaussian splat.

Figures & tables

FeatureKIWI (ours)UMI [ 1 ]UMI-3D [ 2 ]iPhUMI [ 3 ]FastUMI [ 4 ]RDT2 [ 5 ]YUBI [ 6 ]EgoMI [ 7 ]HiFi-UMI [ 8 ]XRZero-G0 [ 9 ]DAS Fingers [ 10 ]
Sensor suiteInsta360 X5, GO 3GoPro Hero9Livox Mid-360, global-shutter camiPhone 15 ProGoPro Hero9, RealSense T265Global-shutter cam, Vive TrackerELP fisheye, Quest 3S, rotary encoderQuest 3S, ZED 2i, wrist camsProprietaryPICO 4, ego + wrist camsProprietary
Extra compute free✓✓✗✓✗✗✗✗✗✗✗
Wrist 6-DoF trajectoryOffline SLAMOnline SLAMOnline LiDAR + IMUOnline ARKitOnline RealSense T265Online Vive IROnline VROnline VROffline SLAM + markerOnline VROnline VIO
Ego view✓✗✗✓✗✗✓✓✓✓✓
Inter-hand pose✓ map✓ mapN/A✓ ARKitNR✓ Vive✓ VR✓ VR✓ head✓ VR✓ Ego
Scene coverage360∘360^{\circ} wrist + ego155∘155^{\circ} + mirrors185∘185^{\circ} + LiDARHead + wrist169∘169^{\circ} wristWrist180∘180^{\circ} + chestHead + wrist∼200∘\sim 200^{\circ} + headHead + wrist150∘150^{\circ} + Ego
Table 1: Demonstration interfaces: collection hardware and recovered state.
Figure 2: Modular hardware overview. Top: exploded view of the three-part design: an interchangeable gripper module, the shared camera module, and a robot mounting plate. Bottom: (A) chopstick and (B) parallel-jaw handheld grippers and (C) the bare-hand wrist cuff; (D, E, F) the robot-mounted twins.
Figure 3: Chopstick gripper geometry and hinge mechanism. Left: side view of the assembled gripper and camera, showing the angled grip that provides clearance for the camera body. Right: section view of the bearing-supported jaw pivots and meshing gears.
Figure 4: KIWI offline pipeline. (A) One shared rear map and two front maps recover metric scale and gravity from IMU measurements and are aligned using calibrated front–rear geometry. (B) Each episode reuses these maps for independent per-wrist visual–inertial pose estimation, complemented by tool-opening, head-marker, and ground estimation from native videos and recovered poses. (C) Outputs include synchronized demonstration datasets with tool-tip poses and quality records, alongside a 3D Gaussian scene reconstructed from native fisheye videos and aligned camera poses for visual real-to-sim.
Figure 5: Shared-scene registration from fisheye views. (a) Rear-view correspondences illustrate the common scene reference SS . (b) Temporal correspondences in each front stream support separate map frames FLF_{L} and FRF_{R} . Fixed front–rear calibration EaE_{a} and synchronized localizations determine the rigid placements Ga:Fa→SG_{a}:F_{a}\rightarrow S in Eq. ( 2 ). Colored lines indicate computed image matches.
Figure 6: End-effector state for (a) the chopstick and (b) the parallel-jaw gripper. Top: fronto-parallel views of the gripper opening with detected dots and per-jaw line fits at uniform opening steps. The end-effector module estimates state independently on every frame of the 120 fps recordings. Bottom: single-cycle excerpts of the vision estimate (blue) versus the servo-encoder ground truth (orange). RMSE against the encoder is 0.17∘0.17^{\circ} in (a) and 1.91.9 px in (b).
Figure 7: Ground-height estimation and pose retargeting. (a) Candidate horizontal planes from known camera poses. (b) Agreement across 20 refocused frames selects the floor ( H=1.25H=1.25 m) within the plausible camera-height range; the table lies at H=0.46H=0.46 m. (c) Height-aligned poses retargeted to two embodiments. HH denotes camera-to-plane distance.
Figure 8: Task and workspace diversity across six configurations. Images are auxiliary egocentric views and the wrist-front view of box folding.
RecordingDurationFramesFront-only (%)KIWI (%)
Kitchen 1317.038,00695.50999.997
Kitchen 2294.835,34099.99799.997
Workbench331.139,69293.73499.997
Bolt organization414.549,6870.00099.998
Tabletop transfers338.440,57191.76599.998
Box folding357.242,82091.24099.998
Table 2: Visual tracking availability.
Figure 9: Held-out 3DGS renders. Recorded images (left) and held-out 3DGS renders (right) from Kitchen 1 and Kitchen 2. Camera poses are fixed; rear-view people are blurred.
More in this topic

Robotic Data Collection

Robotic data collection develops ways to gather, label, and validate robot and human-generated experience for learning, evaluation, and sensing.

See all 178 papers

Explore similar work

CardsList

  1. OpenJaws: An Open Hardware UMI Framework with Stereo Localization for Generalist Manipulation

    Oct 27, 2026Adithyavairavan Murali, Yash Oza, Karanbir Chahal +6Robot LocalizationRobotic Manipulation

  2. Assessing the Marginal Utility of Exocentric Head Views in UMI-Style Large-FoV Wrist-Camera Manipulation

    Oct 8, 2026Yuhui Duan, Shi Jin, Yuntian Wang +5Vision-Based Robot ControlMulti-View Learning

    Soochow University · Independent Researcher · Shanghai Jiao Tong University +2

  3. UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception

    Apr 15, 2026Ziming WangRobotic Data CollectionRobotic Manipulation

    HKU · USTC