Skeleton-Grounded Latent Action Model for Robot World Models

Qi Lv; Jiayi Lin; Yuxin Jiang; Jianzong Wang; Xiang Deng; Mike Shou Zheng

cs.ROOct 22, 2026

Skeleton-Grounded Latent Action Model for Robot World Models

Authors: Qi Lv, Jiayi Lin, Yuxin Jiang, Jianzong Wang, Xiang Deng, Mike Shou Zheng

Organizations: HIT (Shenzhen) · Show Lab, NUS · Ping An Tech

Abstract

Robot world models simulate how environments evolve in response to robot actions. Latent action models enable learning dynamics from videos by encoding visual transitions as proxy actions. However, reconstruction-driven latent actions mix robot motion with object responses and other scene changes, complicating their correspondence with robot controls. We propose SkeLAM, a skeleton-guided latent action model for controllable robot world modeling. SkeLAM decomposes latent actions into a kinematic component grounded by robot skeleton supervision and a residual component retaining complementary visual dynamics. Cross-modal swap reconstruction encourages RGB-derived and skeleton-derived kinematic representations to be interchangeable during video reconstruction. The world model incorporates both components through separate conditioning pathways, while branch dropout reduces its dependence on future-derived residual information. A lightweight adapter maps robot actions to the kinematic space, and subsequent world-model fine-tuning adapts generation to these predicted conditions. At inference, the model generates future observations from the current observation and robot actions. On DROID, SkeLAM reduces end-effector ADE by 19.5% and LPIPS by 11.1% compared with DreamDojo. Motion probing supports kinematic specialization, while qualitative results demonstrate zero-shot human-to-robot motion transfer in generated rollouts. Policy evaluation shows agreement between predicted success rates and those measured in real robot executions.

Figures & tables

Figure 1: Overview of SkeLAM. (A) Skeleton supervision and cross-modal swap reconstruction structure RGB transition latents into kinematic and residual components. (B) Separate projections condition the world model: keep retains both branches, partial drops one, and null drops both. (C) An action adapter predicts kinematic latents, with the null residual at inference.

Figure 2: Skeleton-grounded latent action learning. (A) RGB and skeleton transition encoding. (B) Grounding through a frozen skeleton decoder. (C) Swap reconstruction using the paired skeleton mean and the same RGB residual sample.

		Visual fidelity				Motion accuracy
Method	Control	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	ADE $\downarrow$	$\mathrm{F1}_{\mathrm{dir}}\uparrow$	$L_{1}^{\mathrm{mag}}\downarrow$	$r_{\mathrm{mag}}\uparrow$
Text-conditioned references
TesserAct	Text	15.08	0.7400	0.2561	0.0349	83.76	0.2950	0.0797	0.0823
GE-Base	Text	12.51	0.4530	0.5864	0.0596	55.30	0.3006	0.0956	0.0504
Action- and geometry-conditioned models
IRASim	Action emb.	11.55	0.4137	0.5993	0.0738	67.11	0.2616	0.0843	0.1193

Table 1: World modeling results. Control denotes the inference-time conditioning interface. Text-conditioned models serve as visual references. Bold and underlined values indicate the best and second-best action-conditioned results. † denotes large-scale interaction pretraining.

Figure 3: Qualitative comparison of action-conditioned world modeling. Ground-truth and generated observations are compared at selected time steps, highlighting differences in end-effector motion, manipulated-object geometry, and scene appearance.

	Visual fidelity				Motion accuracy
Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	ADE $\downarrow$	$\mathrm{F1}_{\mathrm{dir}}\uparrow$	$L_{1}^{\mathrm{mag}}\downarrow$	$r_{\mathrm{mag}}\uparrow$
Full SkeLAM	19.35	0.8340	0.1310	0.0151	41.15	0.5465	0.0510	0.5277
w/o skeleton supervision	18.18	0.8206	0.1538	0.0188	64.59	0.3268	0.0723	0.2009
w/o cross-modal swap	19.30	0.8330	0.1436	0.0192	66.74	0.3132	0.0752	0.1897
w/o residual latent	19.38	0.8348	0.1305	0.0150	43.37	0.5263	0.0547	0.4737
w/o action adaptation	18.14	0.8196	0.1545	0.0189	65.84	0.3259	0.0739	0.1603

Table 2: Component ablations. Each variant removes one component from the full model under the same training budget. Bold and underlined values indicate the best and second-best results.

Figure 5: Zero-shot human-to-robot motion transfer. Human videos from UniSkill ( Kim et al., 2025 ) provide latent conditions for robot rollout generation. Markers indicate end-effector positions.

	Episode-level agreement			Policy-level agreement
Model	BA $\uparrow$	Recall $\uparrow$	MCC $\uparrow$	$\rho\uparrow$	$r\uparrow$	$\Delta_{\mathrm{SR}}\downarrow$
Cosmos-Predict2.5	48.44	0.00	-0.0525	-0.0756	-0.1936	7.62
SkeLAM	63.02	33.33	0.2483	0.5347	0.6091	4.76

Table 4: Open-loop policy evaluation on RoboArena. Seven policies are evaluated across 15 sessions (105 episodes). Episode-level metrics include balanced accuracy (BA, %), success recall (%), and Matthews correlation coefficient (MCC). Policy-level metrics include Spearman rank correlation (

\rho

), Pearson correlation (

r

), and success-rate MAE (

\Delta_{\mathrm{SR}}

Figure 6: Policy-level agreement. The dashed line indicates perfect agreement.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Training workflow. Solid arrows transfer model weights; dashed arrows indicate cached data. Stage 3B starts from the Stage 2B world model and Stage 3A adapter and updates all DiT backbone weights. Cache construction does not add optimization stages.

Setting	Stage 2B	Stage 3B
Trainable	Full DiT; both conditioning branches	Full DiT; adapter; $P_{\mathrm{kin}}$
Learning rates	DiT: $2.5\times 10^{-5}$ ; conditioning: $5\times 10^{-5}$	DiT: $2.5\times 10^{-5}$ ; adapter: $3\times 10^{-5}$ ; $P_{\mathrm{kin}}$ : $10^{-5}$
Residual projection	Trainable	Frozen

Appendix

Table 5: World-model optimization. Learning rates are module-specific.

Model	$N_{m}$	MAE $\downarrow$	tLPIPS $\downarrow$	$\rho_{\mathrm{mag}}\uparrow$
TesserAct	315/400	0.0928	0.0993	0.0431
GE-Base	330/400	0.1587	0.0549	0.0340
IRASim	328/400	0.1917	0.0912	0.1318
Ctrl-World	326/400	0.0859	0.0220	0.4980
Cosmos-Predict2.5	326/400	0.1873	0.0596	0.0351
Cosmos3	328/400	0.0753	0.0570	0.2080

Appendix

Table 6: DROID: additional metrics and coverage. All methods generate 400 valid videos. Primary metrics appear in Table 1 ;

N_{m}

counts valid arm trajectories out of 400.

Model	$N_{v}$	MSE $\downarrow$	MAE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	tLPIPS $\downarrow$
TesserAct	376	0.0223	0.0684	17.19	0.7781	0.2043	0.0673
GE-Base	400	0.0077	0.0392	22.15	0.7732	0.3004	0.0268
IRASim	400	0.1889	0.3568	7.25	0.1489	0.8636	0.5998
Ctrl-World	376	0.0364	0.1146	14.69	0.5471	0.5313	0.0243
Cosmos-Predict2.5	400	0.0576	0.1627	12.67	0.4610	0.5950	0.0408
Cosmos3	400	0.0151	0.0523	19.74	0.7982	0.1740	0.0315

Appendix

Table 7: AgiBot World: video fidelity. All methods generate 400 videos;

N_{v}

is the valid-video count. MSE/MAE use normalized RGB; PSNR is in dB.

Model	$N_{m}$	ADE $\downarrow$	$\mathrm{F1}_{\mathrm{dir}}\uparrow$	$L_{1}^{\mathrm{mag}}\downarrow$	$r_{\mathrm{mag}}\uparrow$	$\rho_{\mathrm{mag}}\uparrow$
TesserAct	684/800	45.74	0.2348	0.0850	-0.0489	0.0548
GE-Base	754/800	16.71	0.4830	0.0336	0.1224	0.5104
IRASim	750/800	78.81	0.2628	0.1182	-0.0335	0.0027
Ctrl-World	707/800	18.52	0.2652	0.0382	0.1127	0.0317
Cosmos-Predict2.5	754/800	26.61	0.2802	0.0481	0.0491	0.0816
Cosmos3	752/800	28.61	0.2853	0.0477	0.0972	0.4452

Appendix

Table 8: AgiBot World: motion and coverage.

N_{m}

counts valid arm trajectories out of 800. ADE is in pixels and magnitude error is normalized by the image diagonal.

Data	Steps	$N_{m}$	ADE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	tLPIPS $\downarrow$	$\mathrm{F1}_{\mathrm{dir}}\uparrow$	$r_{\mathrm{mag}}\uparrow$
DROID	5K	325/400	45.25	0.8299	0.1374	0.0241	0.5257	0.4930
	10K	326/400	45.37	0.8326	0.1337	0.0232	0.5248	0.4986
	15K	327/400	42.48	0.8345	0.1314	0.0228	0.5439	0.5408
	30K	326/400	41.15	0.8340	0.1310	0.0222	0.5465	0.5277
AgiBot	5K	756/800	20.79	0.8452	0.1073	0.0185	0.2968	0.1742
	10K	754/800	21.29	0.8477	0.1056	0.0182	0.2888	0.1332

Appendix

Table 9: Stage 3B adaptation budgets. Each row uses 400 videos;

N_{m}

lists valid/total arm trajectories. Budgets exclude pretraining and the common 2K adapter fit; inference uses zero residual. DROID LPIPS decreases with updates, but motion metrics on both datasets are nonmonotonic.

Panel of Figure 8 — Figure 8: Tabletop scene. Columns: ground truth, SkeLAM, DreamDojo, Cosmos-Predict2.5, OSCAR, and Ctrl-World. Rows: $T$ , $T+4$ , $T+8$ , and $T+12$ . Colored markers and arrows highlight end-effector motion in the final row.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
Oct 27, 2026Haoyu Zhao, Yinzhou Tang, Zixuan Wang +11Embodied World Models World Models
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation
Oct 1, 2026Juyi Sheng, Hua Wang, Mengyuan LiuWorld Models for Robotics Robotic Manipulation
Peking University
WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics
Sep 14, 2026Peidong Liu, Zhiyuan Xiang, Mingyang Li +4Latent Action Learning World Models for Robotics
Joy Future Academy, JD Group

Oct 27, 2026J/K move · Enter open · S save

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

Haoyu Zhao, Yinzhou Tang, Zixuan Wang, Minghao Zhu+10

DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK+1

Embodied World Models World Models Cross-Embodiment Robot Learning Action-Conditioned Video Generation Synthetic Data Generation for Robotics

Open paper AskPDF arXiv

Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.

Abstract

World Models for Robotics

Explore similar work

RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

SkeleWAM: Skeleton World-Action Modeling for Efficient Robotic Manipulation

WLA3^33: World Latent Action Modeling for Semantics, Dynamics, and Kinematics

WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics