The Universal Manipulation Interface (UMI) enables scalable demonstration collection independently of a specific robot. However, its moving, close-range gripper cameras entangle camera and object motion without a stable external view, leaving the metric geometry needed for precise manipulation implicit in RGB. We introduce Spatial-UMI-Dream, a 5B spatial World Action Model (WAM) pretrained on 605 hours of RGB-D UMI data through two-stage training -- world-only adaptation followed by joint world-action learning -- to jointly predict future RGB, fisheye-aligned metric depth, and actions. Stereo-derived depth makes scene scale and spatial relationships explicit within a unified dual-gripper RGB-D representation. Furthermore, evaluating UMI policies requires testing whether UMI's robot-independent demonstration collection translates into policy generalization across robot embodiments, alongside robustness to environmental changes. We introduce RoboTwin-UMI, a twenty-task simulation benchmark built on RoboTwin with UMI wrist-mounted fisheye observations, enabling controlled, repeatable evaluation of UMI policies under environment and embodiment changes separately and jointly without additional fine-tuning. On RoboTwin-UMI, Spatial-UMI-Dream achieves 79.10% in-domain success and 72.15% under combined embodiment and environment shifts, exceeding the strongest evaluated baseline in each setting by 4.65 and 10.30 percentage points, respectively. It also exhibits the smallest success-rate drop under embodiment transfer among the evaluated policies. In a matched pretraining ablation, RGB-only pretraining leads to a 5.75-percentage-point decrease in in-domain success compared with RGB-D pretraining. Across six real-world tasks, Spatial-UMI-Dream achieves 85.0% average success, exceeding the strongest baseline by 24.2 percentage points. Project page: https://weiduoyuan.com/spatial-umi-dream/
Figures & tables
Fig. 2 : Model architecture and two-stage training of Spatial-UMI-Dream . In both stages, dual-gripper RGB and metric depth (Fig. 3 ) form a unified RGB-D canvas. The Wan VAE encodes observation history and current RGB-D inputs into visual tokens, while a text encoder supplies language tokens through cross-attention. The shared world-action transformer uses block-causal attention to condition prediction on preceding visual history. Stage 1: world-only adaptation predicts future RGB-D observations, decoded through the Wan VAE and supervised by Lworld . Stage 2: the transformer backbone is initialized from the Stage 1 world-only adaptation checkpoint. Joint world-action training additionally encodes UMI state and noisy actions, and jointly predicts future RGB-D observations and actions through the Wan VAE and action decoder, optimized with Lworld+λaLaction .
Fig. 3 : Metric depth construction for UMI observations. Each UMI gripper has three cameras. We undistort and rectify the middle and right camera views, then use Fast FoundationStereo to recover metric depth. We map the depth back to the original middle-camera fisheye view to align it with the RGB observation.
Fig. 4 : Real-world task suite. We show egocentric keyframes from six real-world tasks, including unimanual manipulation and bimanual coordination. These views are for visualization only and are not used as policy inputs.
Method
Pick-and-Place
Handover
Scan-and-Place
Pack-and-Close
Handover-and-Pour
Table Wiping
Avg.
π0.5
70.0 / 86.7
45.0 / 61.3
50.0 / 82.0
40.0 / 40.0
60.0 / 70.0
100.0 / 100.0
60.8 / 73.3
RDT2-FM
70.0 / 76.7
5.0 / 35.0
5.0 / 42.0
30.0 / 47.5
45.0 / 63.0
85.0 / 92.5
40.0 / 59.4
Fast-WAM
50.0 / 81.7
40.0 / 71.3
45.0 / 67.0
30.0 / 42.5
25.0 / 45.0
100.0 / 100.0
48.3 / 67.9
LingBot-VA
60.0 / 86.7
5.0 / 35.0
55.0 / 91.0
40.0 / 55.0
25.0 / 67.0
75.0 / 95.8
43.3 / 71.8
Spatial-UMI-Dream
95.0 / 98.3
80.0 / 93.8
75.0 / 87.0
60.0 / 80.0
100.0 / 100.0
100.0 / 100.0
85.0 / 93.2
TABLE I : Real-world bimanual manipulation. We report full-episode success rate (SR) and average progress score (Score), both in percent, over 20 evaluation episodes per task. Entries show SR / Score; Avg. is the unweighted mean over the six tasks.
Fig. 5 : RoboTwin-UMI evaluation across twenty tasks. (a) Mean success rates on Aloha and ARX X5. (b) Per-task cross-embodiment success rates on ARX X5. Spatial-UMI-Dream achieves the highest mean success rate on both embodiments and the smallest drop.
Fig. 6 : Ablation of depth and UMI pretraining. Simulation: I, Aloha clean; II, Aloha with randomized lighting; III, ARX X5 clean; IV, ARX X5 with randomized lighting. Bars report mean success over all twenty tasks. Real-world: mean success over six tasks for Spatial-UMI-Dream (RGB-D) and UMI-Dream (RGB).
Method
Aloha
Aloha
ARX X5
Clean ↑
Rand. Light ↑
Rand. Light ↑
π0.5
74.45
66.15
61.85
RDT2
38.55
16.55
12.40
Fast-WAM
61.20
18.85
16.90
LingBot-VA
35.55
34.85
28.60
Spatial-UMI-Dream
79.10
73.35
72.15
TABLE II : Environment and joint-shift generalization on RoboTwin-UMI . Mean success rates (%) over twenty tasks: Aloha under clean and randomized lighting, and ARX X5 under randomized lighting.
Stage
Updates
Batch
GPUs
World-only adaptation
30k
192
24
Joint world-action
30k
192
24
Real-world fine-tuning
22.5k
64
8
RoboTwin-UMI fine-tuning
10k
128
8
TABLE III : Stage-specific training settings.
Task
Goal
Limit
Adjust Bottle
Pick up the bottle upright with the appropriate arm
400
Blocks Ranking RGB
Arrange red, green and blue blocks left to right
1200
Click Alarmclock
Press the alarm clock’s top button
400
Click Bell
Press the top of the bell
400
Grab Roller
Grasp and lift the roller with both arms
400
Lift Pot
Lift the pot with both arms
400
TABLE IV : RoboTwin-UMI tasks and step limits (policy actions).
Task
Demos
Frames
Scored stages
Pick-and-Place
20
3,350
reach, grasp, place
Bimanual Handover
20
11,565
grasp, hand over, right-gripper release, place in box
Scan-and-Place
40
18,278
grasp product, grasp scanner, scan, put down scanner, place product
Pack-and-Close
40
13,805
grasp, place, close lid, press
Handover-and-Pour
30
11,103
reach, grasp, hand over, pour, release
Table Wiping
30
10,931
grasp, wipe, release, grasp, wipe, release
TABLE V : Real-world demonstrations and scored stages.