PRIME W0: Physical Priors from World Models for Embodied Manipulation

Huabin Peng; Fagang Jin; Yancong Wei; Tengfei Ma; Chuhang Liu; Chao Yu; Liqing Chen

cs.ROOct 29, 2026

PRIME W0: Physical Priors from World Models for Embodied Manipulation

Authors: Huabin Peng, Fagang Jin, Yancong Wei, Tengfei Ma, Chuhang Liu, Chao Yu, Liqing Chen

Organizations: Lumos SZ Lab Team · CUHK · THU · Lumos Robotics ·

Abstract

Vision-Language-Action (VLA) models excel at instruction following, yet most remain feed-forward: they condition actions only on the current observation and discard both long-horizon temporal context and the hierarchical structure of VLM representations. We present PRIME W0, a hierarchical conditional flow-matching framework that closes this gap through three synergistic designs: (1)per-layer KV injection from a pretrained VLM into an Adaptive MoE action expert; (2)dynamics-aware conditioning via asynchronous world-model subgoal-image prediction and a DINOv2 historical-frame semantic bridge; and (3)Action Dynamic Manifold (ADM) flow matching with covariance-structured noise that respects robot kinematic coupling. PRIME W0 is trained in four stages---VLM pre-training, DAT/NTP visuomotor pre-training, DiT flow-matching decoder assembly with WM injection, and end-to-end SFT---and achieves state-of-the-art results on LIBERO and real-robot benchmarks, with systematic ablations validating each architectural component.

Figures & tables

Figure 2: Four-stage training pipeline. (1) VLM Pretrain: the vision–language backbone is trained on general multimodal supervision (e.g. captioning / VQA). (2) NTP / task-specific pre-training: with DAT action tokens, the VLM learns visuomotor next-token prediction over instruction–observation–action sequences. (3) DiT Pretrain: the VLM is frozen while the MoT / Adaptive MoE flow-matching action expert is trained with world-model conditioning to denoise continuous actions (and associated state tokens). (4) SFT: both the VLM and the action expert are jointly fine-tuned end-to-end, optionally with action-prefix tokens that shorten the denoising trajectory at inference. Flame / snowflake icons indicate trainable vs. frozen modules.

Figure 3: Training data mixture. Robot demonstration data used by PRIME W0 totals approximately 10,000 hours , comprising Teleop (60%, ${\sim}$ 6,000 h; laboratory setups, data-collection factories, and public datasets), UMI (30%, ${\sim}$ 3,000 h; portable UMI fisheye egocentric collection), and Simulation (10%, ${\sim}$ 1,000 h; reconstructed scenes with procedural objects and physics randomisation).

Task	OpenVLA	GR00T-N1.6	$\pi_{0.5}$	PRIME W0
Pick-and-place	39.1	75.3	81.4	93.7
Stacking	25.2	61.8	67.9	70.1
Cloth folding	4.6	21.2	85.4	89.1
Small obj. grasp	8.3	26.7	59.2	61.8
Assembly	1.4	22.1	51.8	75.9
Average	15.7	41.4	69.1	78.1

Table 2: Real-robot in-house task comparison. All methods are evaluated on the same physical hardware under identical conditions. Numbers = task success rate (%). Best in bold , second underlined .

Method	D1: subgoal-success $r$	D2: JR (%)	D3: PE (%)
OpenVLA	0.09	3.2	4.1
$\pi_{0}$	0.14	8.7	9.3
Seer	0.22	11.4	12.6
PRIME W0	0.79	34.0	47.0

Table 3: Temporal and dynamic behaviour quantification. Higher is better for all metrics.

r

: Pearson correlation between subgoal alignment and success. JR: jerk reduction (%) from historical encoding. PE: phase routing-entropy drop at contact (%).

Panel of Figure 4 — Figure 4: ADM cross-attention to WM subgoal-image tokens across manipulation phases. Attention to WM-predicted subgoal tokens peaks at contact-establishment and placement phases, confirming that forward scene predictions actively condition manifold-constrained action generation during critical moments.

Variant	LIBERO-Long	Real-robot
Base VLA (OpenVLA)	53.7	15.7
+ DAT & NTP (Stage 2)	61.2	22.4
+ ADM w/ isotropic noise	72.4	45.3
+ Adaptive MoE	81.6	58.2
+ Historical-frame encoder	88.3	64.7
+ WM subgoal injection (async)	93.5	71.4

Table 4: Component ablation. Each row adds one module on top of the previous variant. Metrics: LIBERO-Long success (%) and real-robot average success (%).

Panel of Figure 5 — Figure 5: Scaling with world-model capacity. LIBERO-Long success scales monotonically with WM size; the largest gain is between 5M and 50M parameters (+5.9%). Better subgoal-image prediction directly benefits the asynchronous WM $\to$ VLM $\to$ ADM pipeline.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 323 papers

Learning Action Priors for Cross-embodiment Robot Manipulation
Jun 24, 2026Dong Jing, Tianqi Zhang, Jiaqi Liu +5Cross-Embodiment Robot Learning Robotic Manipulation
Renmin University of China · University of North Carolina at Chapel Hill · University of Toronto +1
TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation
Aug 27, 2026Jiarui Yang, Yehao Lu, Yuning Su +9Robot Learning Long-Horizon Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Zhejiang University, Hangzhou, China. · Simon Fraser University, Burnaby, BC, Canada. +1
VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment
Jul 2, 2026Guoyang Xia, Fengfa Li, Hongjin Ji +4VLA Model Training Vision-Language-Action Models
Li Auto Inc. · School of Artificial Intelligence, Beijing University of Posts and Telecommunications · The Chinese University of Hong Kong, Shenzhen

Jun 24, 2026J/K move · Enter open · S save

Learning Action Priors for Cross-embodiment Robot Manipulation

Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao+4

Renmin University of China · University of North Carolina at Chapel Hill · University of Toronto+1

Cross-Embodiment Robot Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings. In this work, we propose to pretrain the action module with motion priors before cross-modal VLA alignment. Specifically, we introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins. In Stage1, a lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure solely from unconditioned action trajectories, without processing visual or language tokens. In Stage2, this learned prior is transferred to VLA training through decoder reuse and early-stage latent distillation, aligning visual-language features with the action embedding space while still allowing end-to-end policy refinement. In addition, the trained encoder serves as a compact history compressor, summarizing state-action histories into a single temporal context token for history-aware modeling at negligible cost. Extensive experiments across 13 diverse cross-embodiment tasks on both simulated and real-world platforms validate the effectiveness of our approach. Compared with VLA training without action priors, our model achieves faster convergence, higher success rates, and substantially stronger performance on data-scarce real-world tasks. Moreover, scaling up the action data in Stage~1 yields a more generalizable action prior that directly improves downstream VLA performance.

Abstract

Robot Learning

Explore similar work

Learning Action Priors for Cross-embodiment Robot Manipulation

TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation

VLAFlow: A Unified Training Framework for Vision-Language-Action Models via Co-training and Future Latent Alignment