We present SpatialDex, a framework that learns 3D-grounded bimanual dexterity from RGB-only human videos through reconstruction-augmented visuomotor pre-training. The central idea is to learn human behavior together with its spatial context. From each video, we reconstruct the scene geometry and estimate the hand motion, align them at a shared video-level scale using the hand-size prior of 3D hand estimates. We retain the reconstructed geometry as both a policy input and a prediction target. The policy grounds the token grid of a pre-trained vision-language model in this geometry through gated residual fusion, is trained to predict future scene geometry alongside wrist and finger actions, and generates finger motion conditioned on the predicted trajectories of both wrists. Pre-trained on 1M human-video clips and fine-tuned with a small number of robot demonstrations, SpatialDex outperforms the state-of-the-art methods by 16.5 points on nine simulated dexterous tasks across two embodiments and by 38.3 points on six real-world bimanual dexterous tasks. Under a matched comparison, pre-training with reconstructed geometry improves success by 14.2 points over pre-training on hand motion alone.
Figures & tables
Fig. 1: SpatialDex reconstructs scene geometry and estimates hand motion from RGB-only human videos and aligns them at a shared video-level scale to jointly supervise policy pre-training. The policy maps RGB-D observations to bimanual dexterous actions and is adapted to specific tasks through fine-tuning on robot demonstrations. The lower panel shows the six bimanual dexterous manipulation tasks for real-world evaluation.
Fig. 2: Aligned scene and hand reconstruction. From an RGB-only video, SpatialDex recovers hand meshes and wrist trajectories, point clouds and camera poses up to an unknown scale. Aligning the reconstructed hand points with the estimated hand meshes across frames yields one scale shared by the whole video and a rigid transformation per frame, which places the scene geometry and the hand motion in a common frame for pre-training. The aligned reconstruction supplies the depth observations, the wrist and finger action targets, and the future geometry targets for policy pre-training.
Fig. 3: 3D-grounded visuomotor policy. Reconstructed geometry enters the policy in two roles. As an input, depth samples from each image patch are encoded jointly in 3D and pooled back to their patches, so that the geometry grid g arrives on the same token grid as the visual features x and is fused into f through two gates and a residual whose weight warms up from zero, leaving the priors of the pre-trained VLM intact. As a target, a training-only predictor estimates future scene geometry from the shared context, which requires the representation to retain spatial structure that the action losses alone do not demand. From the same context, a wrist expert predicts both wrist trajectories and a finger expert predicts finger actions conditioned on them.
Fig. 4: Wuji–Tianji simulation tasks. Three Wuji–Tianji tasks are used in addition to the six DexJoCo [ 43 ] tasks for evaluation in simulation.
Method
Allegro + Franka
Wuji + Tianji
Avg.
Fold Glasses
Hammer Nail
Water Plant
Pick Bucket
Bimanual Assembly
Bimanual Microwave
Relocate Ball
Pass Toy
Sweep Trash
Unimanual Allegro
Bimanual Allegro
Bimanual Wuji
All Tasks
ACT [ 50 ]
47.3
50.0
47.3
64.0
0.0
66.0
36.7
34.0
24.7
52.2
33.0
31.8
41.1
iDP3 [ 1 ]
10.0
18.0
48.7
54.0
0.0
38.0
20.0
14.7
10.7
32.7
19.0
15.1
23.8
π0.5 [ 6 ]
72.0
84.7
88.7
84.0
5.3
70.0
64.7
36.0
32.7
82.4
37.7
44.5
59.8
GR00T N1.5 [ 51 ]
27.3
67.3
72.7
72.0
0.7
50.7
38.0
28.7
24.0
59.8
25.7
30.2
42.4
ManiFlow [ 52 ]
12.7
20.0
37.3
56.7
0.0
55.3
30.7
24.0
19.3
31.7
27.7
24.7
28.4
TABLE I: Simulation success rates (%). Avg. columns report unweighted means over the four single-arm Allegro tasks, the two bimanual Allegro tasks, the three bimanual Wuji tasks, and all nine tasks. Values are means over three seeds ( 50 episodes/seed). Best results are shown in bold .
Fig. 5: Real-world rollouts. SpatialDex on Spray Plant, Unzip Bag, and Sweep Trash with Wuji hands and Tianji arms, which involve tool actuation, a deformable object, and coordinated two-hand sweeping. Each row runs from the initial frame on the left to task completion on the right.
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.
Zihan Gao, Yuhong Zhang, Mengzhe Wang +7
Tsinghua University · TsingTec · Shenzhen University +2
We introduce P3-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/