Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to 20.573/0.8004, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS, and correct-versus- shuffled language changes trajectory error by only 2.8--3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.
Figures & tables
Figure 1: AcrossVAM1.0 overview. Four context frames and an instruction are encoded into semantic robot particles and a background latent. Identity-aligned, text-modulated particle dynamics predict five future states. A frozen renderer produces motion structure, while a context-only dense stream supplies appearance; the residual delivery module fuses both predictions. Snowflakes and flames denote frozen and trainable modules. Dashed gray paths are training-only.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Motion PSNR ↑
Persistence
19.97
0.796
0.122
11.89
Particle-conditioned generation
19.93
0.763
0.138
13.23
+ fixed particle-alpha delivery
20.17
0.785
0.134
12.93
AcrossVAM1.0 DiffPrior (3 seeds)
20.573±0.009
0.8004±0.0002
0.1304±0.0004
12.870±0.032
Per-pixel selection oracle
22.12
0.833
0.114
14.09
Table 1: Future prediction on 92 held-out clips (460 future frames). AcrossVAM1.0 reports mean ± sample standard deviation over three independently initialized delivery masks; upstream dynamics, appearance, and diffusion-prior checkpoints are shared. Only results evaluated on the identical VRS protocol are included; scores reported on other datasets are not transplanted. Bold marks the best evaluated deployable result.
Priority
Method
Published
Conditioning
Structured state / status
1
LPWM-Language ( Daniel et al., 2026 )
2026
text
particles / partial: epochs 0–3 saved
2
TextOCVP ( Villar-Corrales et al., 2026 )
2026
text
slots / queued
3
FitVid ( Babaeizadeh et al., 2022 )
2022
frames
dense latent / queued
4
DDLP ( Daniel and Tamar, 2024 )
2024
frames
particles / queued
5
SlotFormer ( Wu et al., 2023 )
2023
frames
slots / queued
Table 2: Public-baseline reproduction protocol. This table tracks comparability and execution status, not quantitative results. Every method will be retrained on the fixed VRS split with four context and five future frames and evaluated on the same 92 clips and 460 future frames as Table 1 .
Figure 2: Representative held-out Franka predictions. Each panel shows five future steps; rows are ground truth, residual delivery, raw generation, persistence, and the learned blend mask visualized with a blue-to-yellow RGB colormap. The left example has a compact interaction region, whereas the right example contains heavier clutter and occlusion. These cases are selected from the fixed seed-0 demo set after checkpoint selection; they are not used for quantitative model selection.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Motion PSNR ↑
Persistence
19.966
0.796
0.122
11.894
Raw DensePrior generation
19.540±0.274
0.718±0.018
0.160±0.011
13.057±0.077
AcrossVAM1.0 joint residual
20.464±0.112
0.795±0.005
0.129±0.001
12.911±0.082
Pixel-selection oracle
21.940±0.071
0.830±0.001
0.112±0.001
13.907±0.074
Table 3: Earlier DensePrior delivery-stage robustness over three appearance/refinement seeds on the same 460-frame test set. The shared particle dynamics checkpoint is frozen, so this is not a full end-to-end seed sweep. Values are mean ± sample standard deviation.
Seed
Selected step
PSNR ↑
SSIM ↑
LPIPS ↓
Motion PSNR ↑
0
1,500
20.563
0.8006
0.1305
12.833
1
3,000
20.578
0.8004
0.1299
12.886
2
1,500
20.578
0.8002
0.1307
12.890
Mean ± s.d.
–
20.573±0.009
0.8004±0.0002
0.1304±0.0004
12.870±0.032
Table 4: Exact DiffPrior delivery sweep. Each learned mask is independently initialized and selected by validation SSIM; the test set is evaluated once after selection. Particle rollouts, appearance features, and DiffPrior candidates are shared across seeds.
Representation
Floats
PSNR ↑
SSIM ↑
LPIPS ↓
Base particle codec
160
20.94
0.637
0.362
Generic dense code
1,024
23.62
0.747
0.310
Particle + residual code
4,096
27.95
0.877
0.148
Table 5: Representation and dynamics diagnostics. Float counts measure the extra dense code per frame; improvement is trajectory-error reduction over persistence.
Text source
Correct–shuffled gap ↑
Seed range
Verb recall
Source annotation
2.8%
[1.5, 4.9]
92.9%
Rule-corrected annotation
3.1%
[2.3, 4.2]
92.9%
Vision-language caption
2.9%
[1.2, 4.0]
47.7%
Kinematic pseudo-text
0.3%
–
–
Random label control
0.2%
–
–
Table 6: Language controls. Gap is the relative trajectory-error increase under shuffled rather than correct text. Values are means over three seeds.
Configuration
Improvement ↑
Residualized boxes + appearance in dynamics
3.9%
Clean boxes, stride 1
4.2%
Clean boxes, stride 3
15.1%
Clean boxes, no appearance shortcut
18.5%
Final width-64, 2-layer model
21.0%
Table 7: Component ablations. Top: held-out particle trajectory-error reduction over persistence. Bottom: delivery alternatives on the same 92-clip/460-frame future-video test as Table 1 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Robot
Windows
Improvement
Franka
136
+21.1%
WidowX
68
+19.0%
Mobile ALOHA
60
−16.1%
Google Robot
32
−5.2%
Fanuc Mate
28
−9.0%
Sawyer
20
−12.9%
Appendix
Table 8: Per-robot trajectory diagnostics. Positive is better than persistence.
World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform event-level video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.
Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for world modeling. However, existing unified models that directly fuse them through shared attention at multiple layers can make it harder to draw on their specialized capabilities. We introduce PINT, a unified model that coordinates semantic reasoning, world modeling, and action generation through a shared chunk-level Predictive Intent while preserving expert-specific pathways. PINT inserts learnable Intent Tokens into the VLM sequence to form this representation, conditioning both the VGM and an Action Expert (AE). It replaces only the VGM's prompt conditioning and retains the VGM at inference, where the AE reads its representations through one-way attention. Rather than prescribing what Predictive Intent should encode, PINT learns it jointly from the native prediction objectives of the VLM, VGM, and AE, which provide complementary views of the same upcoming task execution. Training uses two stages: Stage 1 freezes the VGM so that future video prediction shapes Predictive Intent with task-relevant future dynamics, while Stage 2 fixes the learned Intent and adapts the VGM and AE. Without embodied pretraining, PINT achieves strong in-distribution (ID) performance and substantially improves out-of-distribution (OOD) generalization, reaching 15.5% on RoboTwin 2.0-Hard compared with 1.8% for Fast-WAM and 3.2% for StarVLA, and 79.5% on LIBERO-Plus compared with 51.5% and 75.0%, respectively. Across five real-world bimanual tasks, PINT reaches 68% ID and 24% OOD success, compared with 35%/9% for pretrained Motus and 67%/35% for pretrained π0.5.
Xinyu Guo, Chao Jing, Li Yi +2
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy +2
Vision-language-action (VLA) policies excel at generalist robotic manipulation but generate actions without explicitly modeling their environmental consequences. World action models (WAMs) incorporate future prediction, yet most rely on costly video generation, while latent alternatives either use predicted futures only as auxiliary targets or infer latent actions from suboptimal visual transitions through costly large-scale video pretraining. We introduce WAVE-0 (World-model-Augmented VLA Execution), a framework that bridges VLAs and WAMs through a lightweight latent world model. By connecting the semantic understanding of pretrained VLAs with explicit future prediction, WAVE-0 allows anticipated action consequences to directly guide action generation. Built on a pretrained VLA, its 257M world model translates VLM-generated latent action plans into predicted future representations that condition the action expert, unifying latent planning, future prediction, and action generation. The world model is trained directly on ground-truth robot actions with a counterfactual ranking objective, requiring neither pixel reconstruction nor large-scale video pretraining. WAVE-0 achieves success rates of 91.47% on RoboTwin 2.0 and 98.7% on LIBERO, along with 88.0%, 61.4%, and 75.0% on LIBERO-Plus, LIBERO-Pro, and LIBERO-Para, respectively. On real-world tasks, it improves the success rate of π0.5 from 38.3% to 73.3%. These results demonstrate strong manipulation performance and generalization across diverse distribution shifts.
Haomin Wang, Anqiong Ji, Ke Wang +4
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1