LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models

Zhenhao Shen; Jiaqi Liang; Jasper Lu; Feng Jiang; Yuran Wang; Chuanbo Wei; Jiayi Liu; Jianchun Yang

cs.ROAug 23, 2026

LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models

Authors: Zhenhao Shen, Jiaqi Liang, Jasper Lu, Feng Jiang, Yuran Wang, Chuanbo Wei, Jiayi Liu, Jianchun Yang,

Organizations: Peking University · WUJI · Wuhan University · The University of Hong Kong ·

Abstract

Human video is playing an increasingly central role in training World Action Models (WAMs), owing to its diversity and low collection cost relative to teleoperated robot data. However, most WAMs learn from such video only by predicting pixel-level future frames, giving dynamics that are not directly actionable, whereas motion retargeting recovers directly actionable actions but leaves a large visual gap across embodiments. We therefore propose motion-aligned latent dynamics as an embodiment-agnostic representation to bridge video priors and low-level actions. We further present LD4WAM, which pairs a Latent Dynamics Model trained with semantic reconstruction and real motion alignment with a World Dynamics Action Model built as a mixture-of-transformers (MoT), which preserves full future-video generation and uses learnable queries to distill these latent dynamics from generated futures for action conditioning. Pretrained on our curated unified dataset of over 5{,}000 hours of human and robot data, LD4WAM performs strongly in RoboTwin simulation and on real robots equipped with both grippers and dexterous hands, while generalizing well to unseen objects and backgrounds.

Figures & tables

Figure 1: Overview of LD4WAM. Left: a unified pretraining dataset of $5{,}000+$ hours human and robot videos. Middle: a Latent Dynamics Model distills motion-aligned latent dynamics via semantic and motion (Delta EE) supervision; in the World Dynamics Action Model, learnable queries carry these latent dynamics to bridge the video and action experts. Right: qualitative and quantitative results on RoboTwin and real robots demonstrate the strong manipulation capability of LD4WAM.

Figure 2: Human Data Process. Long videos are segmented into subtask clips, downsampled to 15 FPS, and filtered. Action-annotated clips are converted into unified camera and wrist coordinate frames, while video-only clips go directly to the final dataset.

Figure 3: Pipeline of LD4WAM. Latent Dynamics Model (left): Frames sampled with random temporal skips are encoded by a frozen DINOv3; their features are aggregated by a spatio-temporal transformer and mapped via soft-VQ to latent dynamics $\{z_{i}\}$ , supervised by semantic reconstruction and camera-frame end-effector deltas (Delta EE). World Dynamics Action Model (right): A Mixture-of-Transformers in which the video, latent dynamics, and action experts share a self-attention module. Learnable queries transfer the LDM-supervised latent dynamics from the video expert to the action expert; text conditions all three experts through cross-attention and is additionally concatenated with the state as input to the action expert. Training Procedure (bottom): Stage I pretrains the video and latent dynamics experts on all data while freezing the action expert; Stage II aligns all experts on robot data; Stage III post-trains the model on task-specific data.

Figure 4: Generalization Setting. We evaluate generalization along two axes: object and background. Object Generalization (top) replaces the training objects with unseen ones for three tasks. Background Generalization (bottom) perturbs the visual scene while keeping the task fixed, via lighting, table texture, and distractor objects.

Method	Clean	Random	Average
$\pi_{0.5}$ ( 12 )	82.74	76.76	79.8
ACE-Ego-0 ( 16 )	91.12	90.62	90.9
Motus ( 1 )	88.66	87.02	87.8
Lingbot-VA ( 17 )	92.90	91.50	92.2
Fast-WAM ( 37 )	91.88	91.78	91.8
Being-H0.7 ( 22 )	90.20	89.60	89.9

Table 1: Performance comparison on RoboTwin.

Figure 5: Cross-domain latent dynamics retrieval. Nearest neighbors of each query clip (left) by cosine similarity in the frozen LDM latent space, pooled across human and robot data; the source dataset is labeled under each panel.

Method	Sorting	Shift Test Tube	Tidy Desk	Fold Shirt	Handover Mug	Place Rubik’s Cube	Spray Water	Average
	Gripper					Dexterous Hand
$\pi_{0.5}$	63.3	43.3	73.3	76.7	70.0	76.7	40.0	63.3
Fast-WAM	50.0	10.0	53.3	46.7	63.3	76.7	30.0	47.1
Lingbot-VA	80.0	40.0	63.3	73.3	76.7	80.0	36.7	64.3
Ours	80.0	50.0	70.0	80.0	83.3	83.3	46.7	70.5

Table 2: Success rate (%) on real-world gripper and dexterous-hand manipulation tasks.

Rate band	LDM	w/o motion align.	w/o multi-stride	w/ pixel recon.	w/ hard VQ
High-rate	0.21	0.78	0.24	0.25	0.37
Low-rate	2.13	10.59	2.54	2.69	4.00

Table 3: Ablation of LDM under latent dynamics regression (scale-normalized MSE

\downarrow

, averaged within each band).

Figure 6: Real-world Execution. Qualitative real-world rollouts of our policy across all evaluation tasks, spanning both dexterous-hand and gripper manipulation.

Ablation of our architecture
Method	Tidy Desk			Fold Shirt			Handover Mug			Average
Method	Base	Obj	Bg	Base	Obj	Bg	Base	Obj	Bg	Average
$\pi_{0.5}$	73.3	53.3	56.7	76.7	60.0	50.0	70.0	66.7	56.7	62.6
Fast-WAM	53.3	30.0	13.3	46.7	23.3	6.67	63.3	33.3	10.0	31.1
Lingbot-VA	63.3	43.3	30.0	73.3	63.3	33.3	76.7	70.0	40.0	54.8
Video-Action Dual-Expert WAM	56.7	30.0	13.3	63.3	56.7	10.0	66.7	50.0	13.3	40.0

Table 4: Generalization and architecture ablations (success rate, %). “Obj”/“Bg” denote object/background generalization. Bold marks the best result per column. “ + ” cumulatively adds a component that is retained thereafter; “

\hookrightarrow

” replaces the component added immediately above with an alternative that is neither stacked nor carried forward. The last row is the full model.

Category	Dataset	Frames (M)	Hours
Human	Ego4D ( 8 )	99.88	1849.7
	EgoVerse ( 25 )	50.96	943.6
	EgoDex ( 10 )	35.67	660.6
	EgoSuite ( 19 )	16.45	304.7
	Xperience ( 26 )	6.93	128.2
	Subtotal	209.89	3886.9

Table 5: Pre-training Dataset Statistics (FPS = 15).

Hyperparameter	Value
Optimizer	AdamW $(\beta_{1},\beta_{2}){=}(0.9,0.95)$
Weight decay	$0.01$
Gradient clip	$1.0$ (global norm)
LR schedule	cosine, $5\%$ warmup
Peak learning rate	$1\times 10^{-4}$
Min learning rate	$1\%$ of peak

Table 6: Training hyperparameters of the World Dynamics Action Model (WDAM).

Figure 7: Real-robot platforms. Top: the gripper platform, a dual-arm AgileX PiPER with Pika Grippers. Bottom: the dexterous platform, dual Tianji arms with Wuji hands.

	High-rate			Low-rate
Method	50 Hz	25 Hz	12.5 Hz	10 Hz	5 Hz	2.5 Hz
LDM	0.056	0.158	0.426	0.607	1.815	3.956
w/o motion align.	0.131	0.487	1.718	2.581	8.497	20.686
w/o multi-stride	0.057	0.170	0.482	0.712	2.205	4.700
w/ pixel recon.	0.063	0.188	0.503	0.742	2.260	5.070
w/ hard VQ	0.093	0.268	0.738	1.064	3.236	7.692

Table 7: Latent-dynamics regression, full per-rate breakdown (scale-normalized MSE

\downarrow

on translation

+

rotation). Columns are sampling rates; best per column in bold. Bottom rows list the train/validation clip counts per rate.

Figure 8: Real-world Generalization Rollouts. Qualitative rollouts of our policy on three real-world tasks, Tidy Desk, Fold Shirt, and Handover Mug. Time proceeds from left to right along the arrow ( $\mathbf{T}$ ). For each task we test robustness under object generalization (Object A / Object B, unseen target instances) and background generalization (Background A / Background B, unseen tabletop textures, clutter, and lighting).

Task	Ours		Fast-WAM		Lingbot-VA		$\pi_{0.5}$		Motus		LaWAM		ACE-Ego-0
Task	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.
Adjust Bottle	100	100	100	100	90	94	100	99	89	93	100	100	100	100
Beat Block Hammer	95	97	99	97	96	98	96	93	95	88	90	93	98	92
Blocks Ranking RGB	100	100	100	100	99	98	92	85	99	97	97	100	98	97
Blocks Ranking Size	79	69	94	98	94	96	49	26	75	63	93	89	89	91
Click Alarmclock	98	99	100	100	99	100	98	89	100	100	100	100	52	38

Table 8: Per-task success rates on RoboTwin under clean and randomized evaluation settings.

World Action Models

World Action Models connect predictions of how scenes change under action to executable policies for robots, vehicles, and other embodied agents.

See all 223 papers

LeWAM: Latent evolving World Action Model
Sep 23, 2026Xueji Fang, Boqiang Duan, Hua Wu +2Joint-Embedding Predictive Architecture Action-Conditioned World Models
Zhejiang University · Westlake University · Baidu Inc.
Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models
Aug 6, 2026Haodong Yan, Junfeng Li, Junjie He +12Domain Generalization World Action Models
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Beihang University, Beijing, China · Huawei Foundation Model Department
WholeBodyWAM: Learning Whole-Body World Action Models with Scalable Motion Priors
Sep 16, 2026Bowei Zhang, Qiyao Zhang, Shuanghao Bai +8Cross-Embodiment Robot Learning World Action Models
Nankai University · Beijing Innovation Center of Humanoid Robotics · Beijing Institute of Technology +1

Sep 23, 2026J/K move · Enter open · S save

LeWAM: Latent evolving World Action Model

Xueji Fang, Boqiang Duan, Hua Wu, Jingdong Wang+1

Zhejiang University · Westlake University · Baidu Inc.

Joint-Embedding Predictive Architecture Action-Conditioned World Models Robotic Manipulation World Action Models Latent World Models Offline Preference Optimization

Open paper AskPDF arXiv

World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs). In VDM-based WAMs, observations are first encoded by a VAE, and the resulting compressed latents are then processed by large video diffusion backbones to extract effective features for action generation. However, this paradigm ties WAM performance and training cost to large-scale video generation pretraining, limiting WAM efficiency and scalability. In this paper, we theoretically and empirically investigate how visual representations affect action generation in WAMs. Our results show that predictive embeddings from Joint-Embedding Predictive Architecture (JEPA) encoders better support action generation than compressed VAE latents, with I-JEPA performing best in our encoder comparison. Based on these findings, we propose LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone. We further find that imitation learning matches demonstrated actions but does not distinguish better actions from worse ones, even though small action deviations can greatly affect task success. To address this limitation without additional environment interaction or the human oversight required for resets and safety, we introduce Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations. With only 0.4B trainable parameters, LeWAM achieves an average success rate of 92.28% on RoboTwin 2.0, comparable to that of state-of-the-art VLAs and WAMs, and maintains practical effectiveness on real-world manipulation tasks.

Abstract

World Action Models

Explore similar work

LeWAM: Latent evolving World Action Model

Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models

WholeBodyWAM: Learning Whole-Body World Action Models with Scalable Motion Priors