Native Action-Prior Learning from Videos for World Action Models

Zhaochong An; Fei Zhang; Menglin Jia; Duncan Frost; Zijian Zhou; Yikai Wang; Xudong Wang; Aditya Patel

cs.CVOct 2, 2026

Native Action-Prior Learning from Videos for World Action Models

Authors: Zhaochong An, Fei Zhang, Menglin Jia, Duncan Frost, Zijian Zhou, Yikai Wang, Xudong Wang, Aditya Patel,

Organizations: Meta AI · University of Copenhagen · Physical Intelligence · Imperial College London

Abstract

World action models integrate future visual dynamics with robot action prediction, but their scalability remains limited by the need for action-annotated robot trajectories. Observation-only videos contain rich evidence about interaction dynamics, but existing approaches typically use them either to pretrain visual representations that must later be adapted for control, or to infer latent actions that are subsequently grounded to robot commands. We present NAVA-WAM, which introduces native action-prior learning by directly pretraining the action policy from observation-only videos, avoiding indirect representation-to-control transfer or a separate latent-action model. Our training consists of two stages. First, we pretrain on observation-only videos, where future-video flow-matching supervision over visual transitions is propagated through transition-structured joint attention to optimize the Action-DiT and learn action-relevant priors. Second, we use action-labeled demonstrations to post-train the Action-DiT for robot control through joint video--action flow matching, while asymmetric attention decouples the visual branch from iterative action denoising and enables efficient action-only inference. Extensive experiments show that NAVA-WAM consistently outperforms prior approaches under both in-distribution and out-of-distribution settings, while demonstrating strong action-label efficiency and effective real-robot generalization. These results establish native action-prior learning as an effective approach to directly pretrain action policies from observation-only videos, providing a scalable path beyond action-labeled robot data.

Figures & tables

Figure 1: Paradigms of leveraging observation-only videos for robot policy learning. (1) Vision representation learning first learns visual representations from videos within a vision encoder and then trains an action policy on top, introducing an indirect representation-to-control transfer. (2) Latent-action learning extracts latent actions from visual transitions as intermediate supervision for policy learning, introducing an additional modeling stage and a potential representational bottleneck. (3) Native action-prior learning directly uses video dynamics to optimize the action policy, avoiding additional intermediate interfaces and representational bottlenecks.

Figure 2: Overview of NAVA-WAM . During native action-prior pre-training, the Action-DiT is optimized solely through future-video supervision, learning action-relevant priors directly from observation-only videos. During post-training, the Video-DiT and Action-DiT are jointly optimized on action-labeled robot demonstrations using video and action flow-matching objectives, adapting the pretrained Action-DiT to continuous robot control. At inference, NAVA-WAM denoises only robot actions without explicitly generating future videos, enabling efficient closed-loop control.

Method	LIBERO	LIBERO-Plus
Direct action policies
$\pi_{0}$ ( 2024a )	94.1	53.6
$\pi_{0}$ -FAST ( 2025 )	85.5	61.6
StarVLA- $\alpha$ ( 2026b )	96.5	77.0
$\pi_{0.5}$ ( 2025 )	96.9	77.4
ABot-M0 ( 2026c )	98.6	80.5

Table 1: Results on LIBERO/LIBERO-Plus ( left ) and RoboTwin 2.0 ( right ). We report average success rate (%). LIBERO and RoboTwin Clean measure in-domain performance, while LIBERO-Plus and RoboTwin Random evaluate out-of-domain generalization. Best results are shown in bold .

Figure 3: Real-robot evaluation on a Franka FR3. Left : success rate over five trials per task and average success over all $15$ trials. Right : NAVA-WAM rollouts for the three evaluation tasks, shown from third-person and wrist-camera views. NAVA-WAM achieves $93.3\%$ average success and correctly follows the spatial relation in T1, where both baselines fail across all trials.

Table 2: Ablation studies on RoboTwin 2.0. (a) Comparison of observation-only pre-training methods under varying action-label budgets. (b) Action relevance of learned representations, measured by ridge-regression

R^{2}

to ground-truth actions under different probe-data budgets. (c) Inference compute per action chunk and Random-domain performance. Best results are shown in bold .

Figure 4: Attention of learned action representations. We visualize attention from action representations to visual tokens on held-out data, overlaid on the current and next input observations. Our pretrained Action-DiT attends to interaction-relevant regions and aligns with observed motion, while the CoMo latent-action baseline shows more diffuse attention over static regions.

Figure 5: Qualitative visualization of motion transfer across domains. Each example transfers the transition encoded by a source pair ( Source: current $\rightarrow$ Source: next ) to a different target observation. The source transition specifies the underlying motion cue (orange arrows), which is applied to Target: current to obtain Target: transfer . The Overlay visualizes the transferred state relative to the current target observation, while Target: real shows the corresponding real transition (blue arrows). Across simulation, real-world, and sim-to-real examples, the transferred states follow the source motion despite substantial changes in visual appearance and scene configuration.

Figure 6: Scaling observation-only pre-training data. Success across data scales at different action-label budgets.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Cold Start	Joint Post-training
Trainable	Video-DiT	Video-DiT + Action-DiT
Action horizon	—	16
Learning rate	$4\times 10^{-5}$ (video)	$1\times 10^{-4}$ (video), $2\times 10^{-4}$ (action)
Epochs	5	10
Scheduler	cosine, $5\%$ warm-up
Global batch size	256

Appendix

Table 3: Downstream training hyperparameters on LIBERO. We first perform action-free target-domain cold start and subsequently jointly post-train the Video-DiT and Action-DiT using action-labeled demonstrations.

	Cold Start	Joint Post-training
Trainable	Video-DiT	Video-DiT + Action-DiT
Action horizon	—	16
Learning rate	$2\times 10^{-4}$ (video)	$1\times 10^{-4}$ (video), $3\times 10^{-4}$ (action)
Epochs	1	5
Scheduler	cosine, $5\%$ warm-up
Global batch size	1024

Appendix

Table 4: Downstream training hyperparameters on RoboTwin 2.0. We first perform action-free target-domain cold start and subsequently jointly post-train the Video-DiT and Action-DiT using Clean-domain demonstrations.

Figure 7: Transition-structured attention masks. The pre-training mask ( left ) and post-training mask ( right ) are shown for three consecutive transition segments

S_{1}

–

S_{3}

. Each segment contains a clean preceding visual state, a noised future state, and the corresponding Action-DiT tokens, with attention isolated across segments. Pre-training allows noised-future and action-token queries to attend to all tokens within their segment, whereas during post-training, noised-future video queries attend only to visual tokens.

Panel of Figure 8 — Figure 8: Native action-prior pre-training dynamics. Training and held-out future-video flow-matching losses over pre-training. The training curve shows the raw per-step loss (faint) and its exponential moving average (solid), while the held-out loss is evaluated periodically without smoothing.

Figure 10: Additional qualitative visualizations of motion transfer across domains. Each example transfers the transition encoded by a source pair ( Source: current $\rightarrow$ Source: next ) to a different target observation. The source transition specifies the underlying motion cue (orange arrows), which is applied to Target: current to obtain Target: transfer . The Overlay visualizes the transferred state relative to the current target observation, while Target: real shows the corresponding real transition (blue arrows). Across simulation, real-world, and sim-to-real examples, the transferred states follow the source motion despite substantial changes in visual appearance and scene configuration.

Method	Spatial	Object	Goal	Long	Avg.
Direct action policies
$\pi_{0}$ ( 2024a )	96.8	98.8	95.8	85.2	94.1
$\pi_{0.5}$ ( 2025 )	98.8	98.2	98.0	92.4	96.9
World action models
Fast-WAM ( 2026 )	98.2	100.0	97.0	95.2	97.6
Image-WAM ( 2026e )	97.2	99.2	98.8	98.4	98.4

Appendix

Table 5: Per-suite success rates (%) on LIBERO. All methods are trained on 50 demonstrations per task and evaluated over 50 rollouts per task. Avg. denotes the macro-average across the four task suites. Baselines with available per-suite results are reported. The best result in each column is shown in bold .

Method	Camera	Robot	Language	Light	Background	Noise	Layout	Avg.
$\pi_{0}$ ( 2024a )	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6
$\pi_{0}$ -FAST ( 2025 )	65.1	21.6	61.0	73.2	73.2	74.4	68.8	61.6
Fast-WAM ( 2026 )	16.4	44.5	68.9	78.2	53.7	37.7	60.7	51.5
Image-WAM ( 2026e )	80.8	50.3	91.4	98.1	85.5	93.8	80.5	83.1
NAVA-WAM	72.8	65.6	93.8	97.9	82.8	84.8	87.1	83.5

Appendix

Table 6: Per-perturbation success rates (%) on LIBERO-Plus. Each column averages over the four LIBERO task suites, and Avg. denotes the macro-average across the seven perturbation types. No LIBERO-Plus trajectories are used for training. The best result in each column is shown in bold .

Task	DP		RDT		$\pi_{0}$		UP-VLA		BagelVLA		HALO		Fast-WAM		MV-WAM		Ours
	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.	Clean	Rand.
Adjust Bottle	97.0	0.0	81.0	75.0	90.0	56.0	100.0	17.0	100.0	14.0	97.0	9.0	95.0	0.0	83.0	65.0	100.0	97.5
Beat Block Hammer	42.0	0.0	77.0	37.0	43.0	21.0	66.0	16.0	87.0	16.0	96.0	11.0	74.0	2.0	75.0	53.0	100.0	42.5
Blocks Ranking RGB	0.0	0.0	3.0	0.0	19.0	5.0	38.0	0.0	84.0	4.0	94.0	7.0	0.0	1.0	99.0	88.0	100.0	92.5
Blocks Ranking Size	1.0	0.0	0.0	0.0	7.0	1.0	21.0	0.0	45.0	2.0	58.0	2.0	38.0	0.0	63.0	43.0	70.0	60.0
Click Alarmclock	61.0	5.0	61.0	12.0	63.0	11.0	69.0	41.0	85.0	20.0	83.0	14.0	100.0	38.0	90.0	24.0	100.0	100.0

Appendix

Table 7: Per-task success rates (%) on RoboTwin 2.0. All methods are trained exclusively on Clean-domain demonstrations and evaluated in both the Clean and Random ( Rand. ) domains. Baseline per-task results are taken from MV-WAM ( Chen et al., 2026c ) . The final row reports the average across all 50 tasks. The best result for each task and domain is shown in bold .

Figure 11: Qualitative NAVA-WAM rollouts on RoboTwin 2.0. We show seven successful rollouts (top) spanning diverse manipulation skills and two representative failures (bottom). Each row contains four temporal keyframes, with the head-camera view on the left and the two wrist-camera views stacked on the right. The row header gives the RoboTwin task description, and the final keyframe is outlined in blue for success and red for failure. All examples are from the Random (OOD) domain.

		Successful trials $\uparrow$
Model	ms / call $\downarrow$	T1	T2	T3	Avg (%) $\uparrow$
$\pi_{0.5}$ ( Physical Intelligence et al., 2025 )	195.7	0/5	5/5	3/5	53.3
DreamZero ( Ye et al., 2026c )	3523.9	0/5	5/5	5/5	66.7
NAVA-WAM	379.2	4/5	5/5	5/5	93.3

Appendix

Table 8: Real-robot success rates and inference cost. We report successful trials out of five runs per task on a Franka FR3, with Avg. denoting the success rate across all

15

trials. T1: move the cube to the left side of the bowl ; T2: put the banana in the box ; T3: put the blue cube on the red cube . ms / call denotes the server-side wall-clock latency per policy call on a single H100. The best result in each column is shown in bold .

Robot Policy Learning

Robot policy learning develops controllers that map sensory observations, task goals, and interaction history to actions for embodied robots.

See all 377 papers

Vid2WAM: Distilling Video Diffusion Priors into World Action Models
Aug 9, 2026Chenhao Qiu, Ruixiang Wang, Runyi Zhao +7Cross-Modal Knowledge Distillation World Model Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute
Foresight Without Seeing: Latent Futures for World Action Models
Aug 12, 2026Jiakai Huang, Zhongbo Wu, Siyu Xu +5World Models for Robotics Visuomotor Policy Learning
Shanghai Jiao Tong University · ACE Robotics · Nanyang Technological University
WAM-OPD: Joint Video-Action Supervision for World Action Model Post-Training with On-Policy Distillation
Aug 23, 2026Liuhaichen Yang, Zhengyang Zhong, Hanshang Zhu +13World Action Models Action-Conditioned World Models
University College London · Shanghai Jiao Tong University · The University of Manchester +3

Aug 9, 2026J/K move · Enter open · S save

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Chenhao Qiu, Ruixiang Wang, Runyi Zhao, Sixu Lin+6

Fudan University · The Chinese University of Hong Kong, Shenzhen · Shanghai Innovation Institute

Cross-Modal Knowledge Distillation World Model Learning Visuomotor Policy Learning World Action Models

Open paper AskPDF arXiv

World Action Models (WAMs) improve robot policy learning by jointly modeling future visual dynamics and actions. However, their scalability and generalization remain constrained by their reliance on costly expert demonstrations. We challenge this by asking whether future supervision for WAMs must originate from target-task expert trajectories. In this paper, we propose Vid2WAM, an offline distillation framework that transfers visual diffusion priors from a large video foundation model into a compact WAM student. Given an observation and language instruction, Vid2WAM distills supervision through two complementary channels: task-conditioned future rollouts directly supervise the student's future prediction branch, while an inverse dynamics model recovers embodiment-specific pseudo-actions for action learning. To robustly integrate synthetic and real supervision, we introduce source-aware residual action adaptation that learns source-specific corrections around a shared action backbone and mitigates interference from noisy pseudo-actions. During inference, both the video teacher and inverse dynamics model are discarded, leaving only the WAM student for efficient deployment. Simulation and real-world experiments demonstrate that Vid2WAM improves novel-task generalization and data efficiency under limited expert demonstrations while preserving low-latency inference.

Abstract

Robot Policy Learning

Explore similar work

Vid2WAM: Distilling Video Diffusion Priors into World Action Models

Foresight Without Seeing: Latent Futures for World Action Models

WAM-OPD: Joint Video-Action Supervision for World Action Model Post-Training with On-Policy Distillation