SkelWAM: A Skeleton-Guided World-Action Model for Zero-Shot Cross-Embodiment Manipulation

Pengjun Niu; Yujia Xie; Rui Peng; Hang Zhao; Ke Liu

cs.ROSep 18, 2026

SkelWAM: A Skeleton-Guided World-Action Model for Zero-Shot Cross-Embodiment Manipulation

Authors: Pengjun Niu, Yujia Xie, Rui Peng, Hang Zhao, Ke Liu

Organizations: School of Advanced Manufacturing and Robotics, Peking University, Beijing, 100871, China · Feagine · IIIS, Tsinghua University · Research Center for Robotics, Peking University, Beijing, 100871, China

Abstract

Reusing manipulation experience across robot embodiments is important for scaling robot learning and reducing repeated task-specific data collection. However, changes in embodiment alter visual appearance, action dimensionality and semantics, and the whole-body configurations that can realize the same tool pose. We present SkelWAM, a skeleton-guided world-action model that couples perception and control through one explicit geometric representation for single-source cross-embodiment manipulation. Arm centerline geometry, tool-center-point (TCP) pose, and parallel-jaw commands form a shared 25-D state. The same definition underlies canonical third-person and wrist observations and future whole-body action targets. Trained with predictive visual supervision, a video-action mixture of transformers predicts canonical skeleton action chunks, which embodiment-specific constrained decoders convert into joint or continuum-robot controls. This formulation requires no one-to-one joint correspondence and uses no target-task demonstrations or target policy updates. We introduce LIBERO-Cross10, a source-only cross-embodiment transfer benchmark covering ten tasks and ten target embodiments across four morphological groups. On this benchmark, Franka-trained SkelWAM achieves 43.3% success over 1,000 episodes, exceeding the best-performing evaluated baseline by 36.2 percentage points. We further deploy a JAKA mini2-trained policy on the Feagine A03 continuum robot for three tabletop manipulation tasks, illustrating the approach's potential for real-world cross-embodiment manipulation. Project page: http://www.liukepku.com/skelwam/index.html

Figures & tables

Fig. 2: SkelWAM world–action model. (a) The shared 25-D body/TCP/jaw state defines external and wrist proxies. (b) Thirty paired Video/Action DiT blocks predict a $32\times 25$ skeleton chunk. Starred future-video paths provide supervision only during Stage A. Action attention accesses current-frame keys and values (K/V), which are cached at inference. The deployment heads combine 24-D continuous flow with a jaw event. (c) Native joint IK or PCC decoding prioritizes TCP pose and uses the centerline as posture guidance.

Fig. 3: BBQ-sauce-to-basket on Cross-Object (task 3). The first three columns show one Franka source demonstration and its front/wrist skeleton views. Target columns follow the robot order in Table I and show SkelWAM control replays from recorded states. Rows align initialization, approach, grasp, transport, and the LIBERO goal state across independently selected trials.

Family	Method	Industrial (6-DoF)			Redundant (7-DoF)				Lightweight		Continuum	Mean
Family	Method	JAKA mini2	UR5e	UR10e	Kinova Gen3	xArm7	Rizon4	IIWA	ARX-L5	ViperX300	Feagine A03	Mean
IL	Diffusion Policy [ 29 ]	0.0	2.0	3.0	0.0	3.0	0.0	5.0	0.0	0.0	0.0	1.3
VLA	$\pi_{0.5}$ [ 30 ]	0.0	0.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	0.1
VLA	OpenVLA-OFT [ 31 ]	0.0	0.0	0.0	0.0	1.0	0.0	0.0	1.0	0.0	0.0	0.2
WAM	FastWAM [ 20 ]	0.0	0.0	0.0	0.0	4.0	0.0	0.0	0.0	0.0	0.0	0.4
WAM	LaWAM [ 21 ]	0.0	0.0	1.0	4.0	2.0	0.0	0.0	0.0	2.0	0.0	0.9

TABLE I: Zero-shot success on the ten target embodiments of LIBERO-Cross10 (%). Columns follow the four target reporting groups, and rows are grouped by method category. Each target has 100 episodes (ten tasks, ten trials per task).

Fig. 4: Success on the four selected task subsets, averaged over ten target embodiments. Each baseline bar is the maximum complete-method average within its family. Error bars show descriptive, episode-level Wilson 95% confidence intervals from binary outcomes (300, 300, 200, and 200 episodes per displayed method in the four panels), without adjustment for maximum selection. Task subsets are defined in the benchmark section.

Variant	$V$	$A$	$W$	$D$	SR (%)	$\Delta$ (pp)
Full	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	43.3	–
w/o $V^{R}$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	0.0	-43.3
w/o $A^{R}$	$\checkmark$	$\times$	$\checkmark$	$\times$	0.1	-43.2
w/o $W^{R}$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	43.0	-0.3
w/o $D^{I}$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	40.9	-2.4

TABLE II: Component interventions on the same 1,000 episodes per variant.

\Delta

is ablation minus full, in percentage points.

Fig. 5: Task-dependent effects of the interventions in Table II . Cells show ablation minus full in percentage points on the four Cross-* subsets. Red indicates a decrease and blue an increase in success.

Fig. 6: Real-robot task setup. Columns show Toy Cleanup, Ring Placement, and Peg Insertion, while rows show source JAKA mini2 and target Feagine A03. Arrows and translucent poses illustrate the task motions.

Fig. 7: Physical visual–action interface. JAKA mini2 is shown on the left and Feagine A03 on the right. For Toy Cleanup, the five rows show external RGB, mask overlays, skeleton composites, wrist RGB, and wrist illustrations. Lower bands: Peg Insertion and Ring Placement. Each side shows four selected times.

Explore similar work

Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer
Sep 18, 2026Zetao Cai, Yaping Li, Yiqun Wang +8Cross-Embodiment Robot Learning World Models for Robotics
The Chinese University of Hong Kong · Shanghai AI Laboratory · Joy Future Academy +1
KITE: Decoupling Kinematics and Interaction for Zero-Shot Cross-Embodiment Manipulation
Jun 20, 2026Qianxu Wang, Kuan FangZero-Shot Robotic Manipulation Generalization in Robotic Manipulation
Cornell University
SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation
Oct 1, 2026Juyi Sheng, Hua Wang, Mengyuan LiuWorld Models for Robotics Robotic Manipulation
Peking University

Sep 18, 2026J/K move · Enter open · S save

Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer

Zetao Cai, Yaping Li, Yiqun Wang, Xinyu Zhan+7

The Chinese University of Hong Kong · Shanghai AI Laboratory · Joy Future Academy+1

Cross-Embodiment Robot Learning World Models for Robotics Robot Imitation Learning Bimanual Robotic Manipulation Motion Representation Learning

Open paper AskPDF arXiv

Robot demonstrations are expensive to collect and often provide limited distributional coverage of task variations. Human videos offer a low-cost source of complementary manipulation experience, but learning from them requires bridging embodiment gaps in visual appearance and action spaces. We introduce Skel-WAM, a world action model that bridges these differences through a unified hand-skeleton motion interface. The key insight is to align human and robot motion through a common hand topology, combining skeleton overlays that ground motion in the scene with structured 2.5-D keypoints that encode explicit hand kinematics. Video and Keypoint Experts jointly learn visual and skeletal dynamics through a Mixture-of-Transformers, while a separate robot-trained Action Expert maps these predictions to executable controls. This separation enables human and robot demonstrations to directly supervise shared dynamics without requiring robot action labels for human videos. Across four real-world bimanual tasks and seven simulated tasks, Skel-WAM achieves average success rates of 79.86% and 63.29%, surpassing the strongest baseline by 22.22 and 8.28 percentage points, respectively. Human-robot cotraining more than doubles real-world success on task variations absent from robot training data, from 38.89% to 86.11%. These results demonstrate that a shared skeletal interface enables joint learning across human and robot data and expands robot task coverage through complementary human demonstrations.

Abstract

Explore similar work

Skel-WAM: A Hand-Skeleton-Conditioned World Action Model for Human-to-Robot Manipulation Transfer

KITE: Decoupling Kinematics and Interaction for Zero-Shot Cross-Embodiment Manipulation

SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation