SkelWAM: A Skeleton-Guided World-Action Model for Zero-Shot Cross-Embodiment Manipulation
Authors: Pengjun Niu, Yujia Xie, Rui Peng, Hang Zhao, Ke Liu
Organizations: School of Advanced Manufacturing and Robotics, Peking University, Beijing, 100871, China · Feagine · IIIS, Tsinghua University · Research Center for Robotics, Peking University, Beijing, 100871, China
Reusing manipulation experience across robot embodiments is important for scaling robot learning and reducing repeated task-specific data collection. However, changes in embodiment alter visual appearance, action dimensionality and semantics, and the whole-body configurations that can realize the same tool pose. We present SkelWAM, a skeleton-guided world-action model that couples perception and control through one explicit geometric representation for single-source cross-embodiment manipulation. Arm centerline geometry, tool-center-point (TCP) pose, and parallel-jaw commands form a shared 25-D state. The same definition underlies canonical third-person and wrist observations and future whole-body action targets. Trained with predictive visual supervision, a video-action mixture of transformers predicts canonical skeleton action chunks, which embodiment-specific constrained decoders convert into joint or continuum-robot controls. This formulation requires no one-to-one joint correspondence and uses no target-task demonstrations or target policy updates. We introduce LIBERO-Cross10, a source-only cross-embodiment transfer benchmark covering ten tasks and ten target embodiments across four morphological groups. On this benchmark, Franka-trained SkelWAM achieves 43.3% success over 1,000 episodes, exceeding the best-performing evaluated baseline by 36.2 percentage points. We further deploy a JAKA mini2-trained policy on the Feagine A03 continuum robot for three tabletop manipulation tasks, illustrating the approach's potential for real-world cross-embodiment manipulation. Project page: http://www.liukepku.com/skelwam/index.html
Figures & tables
Fig. 2: SkelWAM world–action model. (a) The shared 25-D body/TCP/jaw state defines external and wrist proxies. (b) Thirty paired Video/Action DiT blocks predict a 32×25 skeleton chunk. Starred future-video paths provide supervision only during Stage A. Action attention accesses current-frame keys and values (K/V), which are cached at inference. The deployment heads combine 24-D continuous flow with a jaw event. (c) Native joint IK or PCC decoding prioritizes TCP pose and uses the centerline as posture guidance.
Fig. 3: BBQ-sauce-to-basket on Cross-Object (task 3). The first three columns show one Franka source demonstration and its front/wrist skeleton views. Target columns follow the robot order in Table I and show SkelWAM control replays from recorded states. Rows align initialization, approach, grasp, transport, and the LIBERO goal state across independently selected trials.
Family
Method
Industrial (6-DoF)
Redundant (7-DoF)
Lightweight
Continuum
Mean
JAKA mini2
UR5e
UR10e
Kinova Gen3
xArm7
Rizon4
IIWA
ARX-L5
ViperX300
Feagine A03
IL
Diffusion Policy [ 29 ]
0.0
2.0
3.0
0.0
3.0
0.0
5.0
0.0
0.0
0.0
1.3
VLA
π0.5 [ 30 ]
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.1
OpenVLA-OFT [ 31 ]
0.0
0.0
0.0
0.0
1.0
0.0
0.0
1.0
0.0
0.0
0.2
WAM
FastWAM [ 20 ]
0.0
0.0
0.0
0.0
4.0
0.0
0.0
0.0
0.0
0.0
0.4
LaWAM [ 21 ]
0.0
0.0
1.0
4.0
2.0
0.0
0.0
0.0
2.0
0.0
0.9
TABLE I: Zero-shot success on the ten target embodiments of LIBERO-Cross10 (%). Columns follow the four target reporting groups, and rows are grouped by method category. Each target has 100 episodes (ten tasks, ten trials per task).
Fig. 4: Success on the four selected task subsets, averaged over ten target embodiments. Each baseline bar is the maximum complete-method average within its family. Error bars show descriptive, episode-level Wilson 95% confidence intervals from binary outcomes (300, 300, 200, and 200 episodes per displayed method in the four panels), without adjustment for maximum selection. Task subsets are defined in the benchmark section.
Variant
V
A
W
D
SR (%)
Δ (pp)
Full
✓
✓
✓
✓
43.3
–
w/o VR
×
✓
✓
✓
0.0
-43.3
w/o AR
✓
×
✓
×
0.1
-43.2
w/o WR
✓
✓
×
✓
43.0
-0.3
w/o DI
✓
✓
✓
×
40.9
-2.4
TABLE II: Component interventions on the same 1,000 episodes per variant. Δ is ablation minus full, in percentage points.
Fig. 5: Task-dependent effects of the interventions in Table II . Cells show ablation minus full in percentage points on the four Cross-* subsets. Red indicates a decrease and blue an increase in success.
Fig. 6: Real-robot task setup. Columns show Toy Cleanup, Ring Placement, and Peg Insertion, while rows show source JAKA mini2 and target Feagine A03. Arrows and translucent poses illustrate the task motions.
Fig. 7: Physical visual–action interface. JAKA mini2 is shown on the left and Feagine A03 on the right. For Toy Cleanup, the five rows show external RGB, mask overlays, skeleton composites, wrist RGB, and wrist illustrations. Lower bands: Peg Insertion and Ring Placement. Each side shows four selected times.
Robot demonstrations are expensive to collect and often provide limited distributional coverage of task variations. Human videos offer a low-cost source of complementary manipulation experience, but learning from them requires bridging embodiment gaps in visual appearance and action spaces. We introduce Skel-WAM, a world action model that bridges these differences through a unified hand-skeleton motion interface. The key insight is to align human and robot motion through a common hand topology, combining skeleton overlays that ground motion in the scene with structured 2.5-D keypoints that encode explicit hand kinematics. Video and Keypoint Experts jointly learn visual and skeletal dynamics through a Mixture-of-Transformers, while a separate robot-trained Action Expert maps these predictions to executable controls. This separation enables human and robot demonstrations to directly supervise shared dynamics without requiring robot action labels for human videos. Across four real-world bimanual tasks and seven simulated tasks, Skel-WAM achieves average success rates of 79.86% and 63.29%, surpassing the strongest baseline by 22.22 and 8.28 percentage points, respectively. Human-robot cotraining more than doubles real-world success on task variations absent from robot training data, from 38.89% to 86.11%. These results demonstrate that a shared skeletal interface enables joint learning across human and robot data and expands robot task coverage through complementary human demonstrations.
Zetao Cai, Yaping Li, Yiqun Wang +8
The Chinese University of Hong Kong · Shanghai AI Laboratory · Joy Future Academy +1
Generalizing manipulation policies across robot embodiments remains difficult because standard policies entangle task reasoning with embodiment-specific motor control. We study zero-shot cross-embodiment manipulation, where a policy trained on source embodiments must be deployed on a structurally different target embodiment without additional task demonstrations. We introduce Kinematic Interaction Transfer across Embodiments (KITE), which decouples manipulation into embodiment-agnostic task reasoning and embodiment-specific motor control, connected through a learned latent representation of interaction intent based on contact patterns. Task reasoning is performed by a shared policy that predicts latent intents from source demonstrations, while motor control is performed by an intent-conditioned action decoder learned from each embodiment's kinematic model. With KITE, adaptation to a new embodiment requires only training a new action decoder using its kinematic model, without recollecting demonstration data. We evaluate KITE on three manipulation tasks spanning transfer between parallel grippers, dexterous hands, and composite embodiments. KITE consistently achieves zero-shot transfer to structurally different target embodiments, outperforming state-of-the-art baselines in transfer success and task-embodiment scope.
World action models (WAMs) combine robot action generation with future state prediction. Existing WAMs typically predict videos or learned visual latents, which represent interaction geometry only implicitly and may retain appearance information unrelated to control. We introduce SkeleWAM, a compact WAM that represents a manipulation scene as a sparse 3D skeleton composed of robot joints, object centers, and interaction points. Constructed online from current RGB-D observations and robot proprioception, the skeleton provides a unified geometric state for action generation and future skeleton prediction. Future skeleton prediction provides additional geometric supervision for action learning without requiring visual reconstruction. At inference, SkeleWAM generates actions directly from the current skeleton and language instruction, while Medoid Action Consensus (MAC) serves as an auxiliary consensus strategy for stochastic action samples. On LIBERO-Plus, SkeleWAM achieves an overall success rate of 85.9% with 57.1M parameters, outperforming Cosmos-Policy by 3.7 percentage points. These results demonstrate that sparse 3D robot--object structure provides an effective state space for robust and parameter-efficient world action learning. The project is available at https://skelewam-project.github.io/.