Organizations: Tsinghua University · TsingTec · Fudan University · Shenzhen University · Nanyang Technological University · Georgia Institute of Technology
Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.
Figures & tables
Fig. 2 : Overview of DexCast . (a) Demonstration acquisition. Given monocular videos, we leverage 4D HOI reconstruction and reinforcement learning to generate training data for dexterous policy learning. (b) We pretrain our VAE-based DexHand Motion Perception Module (DMPM) on large-scale motion capture data. (c) The pretrained DMPM encoder produces a compact latent representation of robotic hand motion, which conditions the diffusion process.
Fig. 3 : Visualization of data flow. We recover the full hand–object geometry from raw RGB-D videos, transfer the interaction to a robotic hand, and synthesize complete interaction point clouds for downstream policy training.
Fig. 4 : Details of the DMPM Encoder. (a) Our encoder leverages the articulated kinematic chains of the robotic hand to capture structured inter-joint couplings. (b) SkeletonBlock performs topology-aware residual aggregation along the kinematic graph. (c) Visualization of the skeleton. The skeleton represents the robotic hand as a structured articulated graph, with nodes representing joints and directed edges tracing the kinematic hierarchy.
Model
Grasp
Pour
Flip
Push
Wipe
Stack
Avg
State-Based DP
35.0 | 40.0
50.0 | 40.0
60.0 | 50.0
40.0 | 35.0
20.0 | 30.0
10.0 | 25.0
35.8 | 36.7
ACT-3D
40.0 | 50.0
40.0 | 35.0
65.0 | 70.0
60.0 | 70.0
25.0 | 40.0
20.0 | 30.0
41.7 | 49.2
DP-3D
35.0 | 45.0
60.0 | 50.0
60.0 | 65.0
65.0 | 70.0
60.0 | 55.0
30.0 | 40.0
51.7 | 54.2
CordViP
75.0 | 80.0
70.0 | 75.0
75.0 | 70.0
80.0 | 80.0
75.0 | 80.0
60.0 | 65.0
72.5 | 75.0
Ours
90.0 | 80.0
85.0 | 90.0
80.0 | 85.0
90.0 | 95.0
95.0 | 90.0
85.0 | 80.0
87.5 | 86.7
TABLE I : Quantitative results on six real-world dexterous manipulation tasks. Each entry reports the success rate (%) under Viewpoint-1 | Viewpoint-2, with the final column reporting the average success rates.
Fig. 5 : Real-world performance on six dexterous manipulation tasks.
Model
Pour
Stack
Avg.
Seen
Unseen
Seen
Unseen
DP-3D
60.0
55.0
30.0
30.0
43.8
CordViP
70.0
65.0
60.0
55.0
62.5
Ours
85.0
85.0
85.0
85.0
85.0
TABLE II : Success rates (%) on seen and unseen objects.
Model
Pour
Stack
Grasp
Push
Avg.
Env-1
Env-2
Env-3
Cluttered
Env-1
Env-2
Env-3
Cluttered
Env-1
Env-2
Env-3
Cluttered
Env-1
Env-2
Env-3
Cluttered
DP-3D
60.0
50.0
45.0
30.0
30.0
25.0
40.0
20.0
35.0
25.0
25.0
20.0
65.0
60.0
55.0
45.0
39.4
CordViP
70.0
65.0
65.0
55.0
60.0
55.0
50.0
40.0
75.0
70.0
75.0
60.0
80.0
70.0
75.0
65.0
64.4
Ours
85.0
85.0
80.0
70.0
85.0
85.0
90.0
70.0
90.0
85.0
85.0
75.0
90.0
90.0
85.0
80.0
83.1
TABLE III : Generalization of DexCast to different environments (Env.) (Success Rate %).
Variant
Grasp
Pour
Flip
Push
Wipe
Stack
Avg.
DexCast w/o augmented data
30.0
40.0
20.0
25.0
25.0
15.0
25.8
DexCast w/o SIM(3) & DMPM
60.0
70.0
60.0
75.0
75.0
60.0
66.7
DexCast w/o DMPM
80.0
80.0
70.0
85.0
85.0
70.0
78.3
DexCast w/o SIM(3)
80.0
85.0
75.0
80.0
85.0
80.0
80.8
DexCast (from scratch)
80.0
85.0
60.0
90.0
85.0
80.0
80.0
DexCast (Full)
90.0
85.0
80.0
90.0
95.0
85.0
87.5
TABLE IV : Ablation studies of DexCast. (Success Rate %).
Fig. 6 : UMAP visualization of hand-state representations encoded by DMPM and an MLP baseline.
Metric
w/o Skeleton
Skeleton
16
32
64
128
16
32
64
128
TMPJPE (mm) ↓
11.497
9.328
8.445
13.141
4.005
3.691
4.540
12.345
MPJPE (mm) ↓
3.355
2.722
2.364
3.835
1.150
1.073
1.316
3.653
Pose Consistency (deg) ↓
2.125
1.811
1.505
2.096
0.776
0.699
0.863
1.941
Hand Jitter (deg/s 2 ) ↓
0.133
0.215
0.215
0.248
0.320
0.290
0.255
0.288
TABLE V : Ablation studies on skeleton architecture and sequence length. Clip length T varies from 16 to 128 frames.
Fig. 7 : Visualization of DexHand motion recovery for two DMPM architectures. The DMPM with Skeleton produces higher-fidelity reconstructions.
Metric
32
64
128
256
512
1024
2048
TMPJPE (mm) ↓
6.623
3.370
3.075
3.024
3.402
3.691
4.211
MPJPE (mm) ↓
1.877
0.967
0.893
0.881
0.983
1.073
1.221
Pose Consistency (deg) ↓
1.237
0.639
0.622
0.581
0.656
0.699
0.794
Hand Jitter (deg/s 2 ) ↓
0.366
0.395
0.388
0.346
0.312
0.290
0.298
TABLE VI : Latent Dimension Ablation. Performance of DMPM under different latent dimensions.