Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation
Organizations: USC Physical Superintelligence Lab · CMU · Texas A&M University
Abstract
The Universal Manipulation Interface (UMI) enables scalable demonstration collection independently of a specific robot. However, its moving, close-range gripper cameras entangle camera and object motion without a stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB. We introduce Spatial-UMI-Dream, a 5B spatial World Action Model (WAM) pretrained on 605 hours of RGB-D UMI data through two-stage training -- world-only adaptation followed by joint world-action learning -- to jointly predict future RGB, fisheye-aligned metric depth, and actions. Stereo-derived depth makes scene scale and spatial relationships explicit within a unified dual-gripper RGB-D representation. Furthermore, evaluating UMI policies requires testing whether UMI's robot-independent demonstration collection translates into policy generalization across robot embodiments, alongside robustness to environmental changes. We introduce RoboTwin-UMI, a twenty-task simulation benchmark built on RoboTwin with UMI wrist-mounted fisheye observations, enabling controlled, repeatable evaluation of UMI policies under environment and embodiment changes separately and jointly without additional fine-tuning. On RoboTwin-UMI, Spatial-UMI-Dream achieves 79.10% in-domain success and 72.15% under combined embodiment and environment shifts, exceeding the strongest evaluated baseline in each setting by 4.65 and 10.30 percentage points, respectively. It also exhibits the smallest success-rate drop under embodiment transfer among the evaluated policies. In a matched pretraining ablation, RGB-only pretraining leads to a 5.75-percentage-point decrease in in-domain success compared with RGB-D pretraining. Across six real-world tasks, Spatial-UMI-Dream achieves 85.0% average success, exceeding the strongest baseline by 24.2 percentage points. Project page: https://weiduoyuan.com/spatial-umi-dream/
Figures & tables
| Method | Pick-and-Place | Handover | Scan-and-Place | Pack-and-Close | Handover-and-Pour | Table Wiping | Avg. |
|---|---|---|---|---|---|---|---|
| 70.0 / 86.7 | 45.0 / 61.3 | 50.0 / 82.0 | 40.0 / 40.0 | 60.0 / 70.0 | 100.0 / 100.0 | 60.8 / 73.3 | |
| RDT2-FM | 70.0 / 76.7 | 5.0 / 35.0 | 5.0 / 42.0 | 30.0 / 47.5 | 45.0 / 63.0 | 85.0 / 92.5 | 40.0 / 59.4 |
| Fast-WAM | 50.0 / 81.7 | 40.0 / 71.3 | 45.0 / 67.0 | 30.0 / 42.5 | 25.0 / 45.0 | 100.0 / 100.0 | 48.3 / 67.9 |
| LingBot-VA | 60.0 / 86.7 | 5.0 / 35.0 | 55.0 / 91.0 | 40.0 / 55.0 | 25.0 / 67.0 | 75.0 / 95.8 | 43.3 / 71.8 |
| Spatial-UMI-Dream | 95.0 / 98.3 | 80.0 / 93.8 | 75.0 / 87.0 | 60.0 / 80.0 | 100.0 / 100.0 | 100.0 / 100.0 | 85.0 / 93.2 |
| Method | Aloha | Aloha | ARX X5 |
|---|---|---|---|
| Clean | Rand. Light | Rand. Light | |
| 74.45 | 66.15 | 61.85 | |
| RDT2 | 38.55 | 16.55 | 12.40 |
| Fast-WAM | 61.20 | 18.85 | 16.90 |
| LingBot-VA | 35.55 | 34.85 | 28.60 |
| Spatial-UMI-Dream | 79.10 | 73.35 | 72.15 |
| Stage | Updates | Batch | GPUs |
|---|---|---|---|
| World-only adaptation | 30k | 192 | 24 |
| Joint world-action | 30k | 192 | 24 |
| Real-world fine-tuning | 22.5k | 64 | 8 |
| RoboTwin-UMI fine-tuning | 10k | 128 | 8 |
| Task | Goal | Limit |
|---|---|---|
| Adjust Bottle | Pick up the bottle upright with the appropriate arm | 400 |
| Blocks Ranking RGB | Arrange red, green and blue blocks left to right | 1200 |
| Click Alarmclock | Press the alarm clock’s top button | 400 |
| Click Bell | Press the top of the bell | 400 |
| Grab Roller | Grasp and lift the roller with both arms | 400 |
| Lift Pot | Lift the pot with both arms | 400 |
| Task | Demos | Frames | Scored stages |
|---|---|---|---|
| Pick-and-Place | 20 | 3,350 | reach, grasp, place |
| Bimanual Handover | 20 | 11,565 | grasp, hand over, right-gripper release, place in box |
| Scan-and-Place | 40 | 18,278 | grasp product, grasp scanner, scan, put down scanner, place product |
| Pack-and-Close | 40 | 13,805 | grasp, place, close lid, press |
| Handover-and-Pour | 30 | 11,103 | reach, grasp, hand over, pour, release |
| Table Wiping | 30 | 10,931 | grasp, wipe, release, grasp, wipe, release |
World Models for Robotics
World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.
See all 426 papers