Robot demonstrations are expensive to collect and often provide limited distributional coverage of task variations. Human videos offer a low-cost source of complementary manipulation experience, but learning from them requires bridging embodiment gaps in visual appearance and action spaces. We introduce Skel-WAM, a world action model that bridges these differences through a unified hand-skeleton motion interface. The key insight is to align human and robot motion through a common hand topology, combining skeleton overlays that ground motion in the scene with structured 2.5-D keypoints that encode explicit hand kinematics. Video and Keypoint Experts jointly learn visual and skeletal dynamics through a Mixture-of-Transformers, while a separate robot-trained Action Expert maps these predictions to executable controls. This separation enables human and robot demonstrations to directly supervise shared dynamics without requiring robot action labels for human videos. Across four real-world bimanual tasks and seven simulated tasks, Skel-WAM achieves average success rates of 79.86% and 63.29%, surpassing the strongest baseline by 22.22 and 8.28 percentage points, respectively. Human-robot cotraining more than doubles real-world success on task variations absent from robot training data, from 38.89% to 86.11%. These results demonstrate that a shared skeletal interface enables joint learning across human and robot data and expands robot task coverage through complementary human demonstrations.
Figures & tables
Fig. 2: Skel-WAM architecture and cotraining. (a) The Video–Keypoint MoT jointly predicts future overlay-video latents and structured 2.5-D keypoint trajectories from human and robot demonstrations. (b) The causal Action Expert maps these predictions to robot actions. Human and robot videos cotrain the MoT, while only robot data train the Action Expert, enabling learning from human videos without robot action labels.
Fig. 3: Skeleton overlays across embodiments. (a) Inspire dexterous hands in simulation. (b) Human hands and (c) RobotEra XHand1 dexterous hands in the real world.
Fig. 4: Real-world benchmark. (a) Real-world experimental platform with two Franka Research 3 arms, RobotEra XHand1 dexterous hands, and an Intel RealSense D435 camera. (b) Execution sequences for Stack Cups, Pour Water, Place Fruit into Box, and Put a Wrist Guard into a Drawer.
Method
Avg. ↑
Stack Cups
Pour Water
Place Fruit
Put into Drawer
ACT
12.32 / 44.39
11.76 / 58.82
16.67 / 60.42
0.00 / 37.50
20.83 / 20.83
Fast-WAM
38.89 / 50.69
38.89 / 48.61
41.67 / 60.42
37.50 / 52.08
37.50 / 41.67
LDA
42.01 / 53.30
38.89 / 56.94
41.67 / 54.17
45.83 / 58.33
41.67 / 43.75
EgoVLA
57.64 / 69.10
55.56 / 76.39
70.83 / 85.42
45.83 / 56.25
58.33 / 58.33
Skel-WAM
79.86 / 89.24
77.78 / 90.28
87.50 / 95.83
70.83 / 83.33
83.33 / 87.50
TABLE I: Real-world results with robot-only training data. Each entry is SR / PSR (%).
Method
Avg. ↑
Position
Rotation
Disturbance
Height
LDA-cotrain
30.56 / 49.31
33.33 / 41.67
22.22 / 44.44
44.44 / 61.11
22.22 / 50.00
Skel-WAM, no cotrain
38.89 / 61.11
11.11 / 50.00
77.78 / 83.33
66.67 / 66.67
0.00 / 44.44
Skel-WAM, cotrain
86.11 / 95.14
66.67 / 91.67
100.00 / 100.00
77.78 / 88.89
100.00 / 100.00
Cotrain w/o both
16.67 / 31.25
11.11 / 19.44
33.33 / 61.11
22.22 / 33.33
0.00 / 11.11
Cotrain w/o keypoints
13.89 / 34.03
11.11 / 25.00
22.22 / 55.56
22.22 / 38.89
0.00 / 16.67
Cotrain w/o overlay
50.00 / 68.75
44.44 / 86.11
77.78 / 83.33
44.44 / 55.56
33.33 / 50.00
TABLE II: Results on robot-missing task variations. Cotrained models use robot and human demonstrations, whereas the no-cotrain model uses robot demonstrations only. Each entry is SR / PSR (%).
Fig. 5: Robot and human demonstration settings across the four tasks. Red and green denote robot and human settings, respectively. Human demonstrations introduce variations in object position, wrist rotation, distractors, and manipulation height.
Fig. 6: Future-video predictions with and without skeleton overlays. The model without overlays stalls while transporting the wrist guard after grasping it and loses the hand structure needed to grasp the mango under four-finger occlusion in Place Fruit into Box. Full Skel-WAM maintains coherent motion and explicit hand structure in both cases.
Fig. 7: Hand-specific keypoint-to-video attention. Left and right columns show attention from the respective hand keypoint streams, concentrated on corresponding hand regions, indicating spatial alignment with video features.
Method
Avg. ↑
Stack Can
Push Box
Open Drawer
Close Drawer
Flip Mug
Pour Balls
Open Laptop
ACT
24.88 / 55.93
12.90 / 13.98
16.13 / 87.73
22.58 / 70.43
59.14 / 96.77
2.15 / 2.15
5.38 / 64.52
55.91 / 55.91
LDA
34.87 / 61.52
22.58 / 23.66
22.58 / 93.55
21.51 / 76.34
66.67 / 78.49
50.54 / 50.54
2.15 / 43.55
58.06 / 64.52
Fast-WAM
37.33 / 56.22
29.03 / 29.03
19.35 / 89.25
38.71 / 83.33
97.85 / 98.92
24.73 / 26.88
17.20 / 29.57
34.41 / 36.56
EgoVLA
55.01 / 65.31
56.99 / 56.99
64.52 / 80.65
49.46 / 73.16
90.32 / 92.83
4.41 / 4.41
56.99 / 83.56
62.37 / 65.59
Skel-WAM
63.29 / 78.42
58.06 / 59.14
66.67 / 97.85
49.46 / 86.02
100.00 / 100.00
45.16 / 45.16
27.96 / 61.83
95.70 / 98.92
TABLE III: Simulation results. Each entry is SR / PSR (%).
Fig. 8: Background generalization with robot-only training. Average SR and PSR across four tasks under an unseen background.
Fig. 9: Future-video predictions under an unseen background. For Stack Cups and Pour Water, Skel-WAM maintains coherent task-relevant motion and preserves the test-time background throughout the predicted horizon rather than reverting to the training background shown in the inset.
Method
SR
PSR
Skel-WAM
63.29
78.42
w/o keypoints
38.40
58.14
w/o overlay
61.90
77.88
w/o both
34.56
55.53
TABLE IV: Simulation ablation. Average SR and PSR (%) over seven tasks are reported.
Fig. 10: Modality residual contribution to action prediction. The gated residual RMS share is averaged across action frames at three denoising noise levels. Both conditions contribute throughout denoising, with a consistently larger keypoint contribution.
Reusing manipulation experience across robot embodiments is important for scaling robot learning and reducing repeated task-specific data collection. However, changes in embodiment alter visual appearance, action dimensionality and semantics, and the whole-body configurations that can realize the same tool pose. We present SkelWAM, a skeleton-guided world-action model that couples perception and control through one explicit geometric representation for single-source cross-embodiment manipulation. Arm centerline geometry, tool-center-point (TCP) pose, and parallel-jaw commands form a shared 25-D state. The same definition underlies canonical third-person and wrist observations and future whole-body action targets. Trained with predictive visual supervision, a video-action mixture of transformers predicts canonical skeleton action chunks, which embodiment-specific constrained decoders convert into joint or continuum-robot controls. This formulation requires no one-to-one joint correspondence and uses no target-task demonstrations or target policy updates. We introduce LIBERO-Cross10, a source-only cross-embodiment transfer benchmark covering ten tasks and ten target embodiments across four morphological groups. On this benchmark, Franka-trained SkelWAM achieves 43.3% success over 1,000 episodes, exceeding the best-performing evaluated baseline by 36.2 percentage points. We further deploy a JAKA mini2-trained policy on the Feagine A03 continuum robot for three tabletop manipulation tasks, illustrating the approach's potential for real-world cross-embodiment manipulation. Project page: http://www.liukepku.com/skelwam/index.html
Pengjun Niu, Yujia Xie, Rui Peng +2
School of Advanced Manufacturing and Robotics, Peking University, Beijing, 100871, China · Feagine · IIIS, Tsinghua University +1
World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.
Egocentric human data offers scalable supervision for robot manipulation. However, behavior cloning entangles transferable content like objects, scenes, and task semantics, with non-transferable factors like human morphology, head motion, and behavioral style. We study whether World Action Models (WAMs) provide a better training signal by requiring policies to predict not only actions, but also how the scene evolves. The central question is what world representation best enables human-to-robot transfer. We hypothesize that an effective world target should abstract appearance, capture agent-invariant physical effects, and separate camera motion from environment change. We introduce EgoWAM, a controlled human-robot co-training framework that fixes the policy backbone, action head, and data mixture while varying only the world prediction target, comparing Pixel, DINO, and 3D motion flow. Across three real-world bimanual tasks, WAM co-training scales more effectively with in-the-wild egocentric human data than behavior cloning. Pixel-based prediction transfers weakly, while DINO and 3D flow yield substantial gains: DINO improves out-of-distribution object and scene generalization by up to 4x, and 3D flow improves in-domain performance by 20-30%. More details: https://gatech-rl2.github.io/egowam.github.io