AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

Yafei Zhang; Nan Wu

cs.AIAug 28, 2026

AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction

Authors: Yafei Zhang, Nan Wu

Organizations: Across Physical AI, Beijing, China

Abstract

Predicting robot videos requires both precise motion reasoning and preservation of high-frequency appearance, yet monolithic pixel models entangle these objectives and often conceal their progress behind a strong last-frame baseline. We present AcrossVAM1.0, a lightweight, text-assisted video action model that factorizes future prediction into object-centric motion and dense appearance. A frozen SAM3-DLP codec decomposes four context frames into semantic particles for the robot, arm, and gripper, together with a background latent. A 0.28M-parameter spatio-temporal Transformer aligns particle identities, rolls their states forward, and is modulated by a frozen OpenCLIP instruction embedding through FiLM. A causal dual-stream decoder combines particle-rendered motion with appearance encoded exclusively from the last observed frame; a residual refiner and learned delivery mask produce five future frames without access to future appearance. On our VRS benchmark constructed from diverse real-robot trajectories, particle dynamics reduce trajectory error by 21.0% over persistence. Across three delivery-mask seeds, AcrossVAM1.0 improves future-frame PSNR/SSIM from 19.97/0.796 to 20.573/0.8004, while raw particle generation improves motion-region PSNR from 11.89 to 13.23. The delivered model does not yet beat persistence in LPIPS, and correct-versus- shuffled language changes trajectory error by only 2.8--3.1%. We report these limitations alongside oracle, negative-control, multi-seed, and per-robot analyses. The results show that explicit particle dynamics are a promising low-dimensional interface for robot video prediction, while robust language grounding and appearance delivery remain the principal open challenges.

Figures & tables

Figure 1: AcrossVAM1.0 overview. Four context frames and an instruction are encoded into semantic robot particles and a background latent. Identity-aligned, text-modulated particle dynamics predict five future states. A frozen renderer produces motion structure, while a context-only dense stream supplies appearance; the residual delivery module fuses both predictions. Snowflakes and flames denote frozen and trainable modules. Dashed gray paths are training-only.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion PSNR $\uparrow$
Persistence	19.97	0.796	0.122	11.89
Particle-conditioned generation	19.93	0.763	0.138	13.23
$+$ fixed particle-alpha delivery	20.17	0.785	0.134	12.93
AcrossVAM1.0 DiffPrior (3 seeds)	$\mathbf{20.573{\pm}0.009}$	$\mathbf{0.8004{\pm}0.0002}$	$0.1304{\pm}0.0004$	$12.870{\pm}0.032$
Per-pixel selection oracle	22.12	0.833	0.114	14.09

Table 1: Future prediction on 92 held-out clips (460 future frames). AcrossVAM1.0 reports mean

\pm

sample standard deviation over three independently initialized delivery masks; upstream dynamics, appearance, and diffusion-prior checkpoints are shared. Only results evaluated on the identical VRS protocol are included; scores reported on other datasets are not transplanted. Bold marks the best evaluated deployable result.

Priority	Method	Published	Conditioning	Structured state / status
1	LPWM-Language ( Daniel et al., 2026 )	2026	text	particles / partial: epochs 0–3 saved
2	TextOCVP ( Villar-Corrales et al., 2026 )	2026	text	slots / queued
3	FitVid ( Babaeizadeh et al., 2022 )	2022	frames	dense latent / queued
4	DDLP ( Daniel and Tamar, 2024 )	2024	frames	particles / queued
5	SlotFormer ( Wu et al., 2023 )	2023	frames	slots / queued

Table 2: Public-baseline reproduction protocol. This table tracks comparability and execution status, not quantitative results. Every method will be retrained on the fixed VRS split with four context and five future frames and evaluated on the same 92 clips and 460 future frames as Table 1 .

Panel of Figure 2 — Figure 2: Representative held-out Franka predictions. Each panel shows five future steps; rows are ground truth, residual delivery, raw generation, persistence, and the learned blend mask visualized with a blue-to-yellow RGB colormap. The left example has a compact interaction region, whereas the right example contains heavier clutter and occlusion. These cases are selected from the fixed seed-0 demo set after checkpoint selection; they are not used for quantitative model selection.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion PSNR $\uparrow$
Persistence	19.966	0.796	0.122	11.894
Raw DensePrior generation	$19.540\pm 0.274$	$0.718\pm 0.018$	$0.160\pm 0.011$	$\textbf{13.057}\pm 0.077$
AcrossVAM1.0 joint residual	$\textbf{20.464}\pm 0.112$	$0.795\pm 0.005$	$0.129\pm 0.001$	$12.911\pm 0.082$
Pixel-selection oracle	$21.940\pm 0.071$	$0.830\pm 0.001$	$0.112\pm 0.001$	$13.907\pm 0.074$

Table 3: Earlier DensePrior delivery-stage robustness over three appearance/refinement seeds on the same 460-frame test set. The shared particle dynamics checkpoint is frozen, so this is not a full end-to-end seed sweep. Values are mean

\pm

sample standard deviation.

Seed	Selected step	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Motion PSNR $\uparrow$
0	1,500	20.563	0.8006	0.1305	12.833
1	3,000	20.578	0.8004	0.1299	12.886
2	1,500	20.578	0.8002	0.1307	12.890
Mean $\pm$ s.d.	–	$20.573{\pm}0.009$	$0.8004{\pm}0.0002$	$0.1304{\pm}0.0004$	$12.870{\pm}0.032$

Table 4: Exact DiffPrior delivery sweep. Each learned mask is independently initialized and selected by validation SSIM; the test set is evaluated once after selection. Particle rollouts, appearance features, and DiffPrior candidates are shared across seeds.

Representation	Floats	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Base particle codec	160	20.94	0.637	0.362
Generic dense code	1,024	23.62	0.747	0.310
Particle + residual code	4,096	27.95	0.877	0.148

Table 5: Representation and dynamics diagnostics. Float counts measure the extra dense code per frame; improvement is trajectory-error reduction over persistence.

Text source	Correct–shuffled gap $\uparrow$	Seed range	Verb recall
Source annotation	2.8%	[1.5, 4.9]	92.9%
Rule-corrected annotation	3.1%	[2.3, 4.2]	92.9%
Vision-language caption	2.9%	[1.2, 4.0]	47.7%
Kinematic pseudo-text	0.3%	–	–
Random label control	0.2%	–	–

Table 6: Language controls. Gap is the relative trajectory-error increase under shuffled rather than correct text. Values are means over three seeds.

Configuration	Improvement $\uparrow$
Residualized boxes + appearance in dynamics	3.9%
Clean boxes, stride 1	4.2%
Clean boxes, stride 3	15.1%
Clean boxes, no appearance shortcut	18.5%
Final width-64, 2-layer model	21.0%

Table 7: Component ablations. Top: held-out particle trajectory-error reduction over persistence. Bottom: delivery alternatives on the same 92-clip/460-frame future-video test as Table 1 .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Robot	Windows	Improvement
Franka	136	+21.1%
WidowX	68	+19.0%
Mobile ALOHA	60	$-16.1\%$
Google Robot	32	$-5.2\%$
Fanuc Mate	28	$-9.0\%$
Sawyer	20	$-12.9\%$

Appendix

Table 8: Per-robot trajectory diagnostics. Positive is better than persistence.

Video Prediction

Video prediction forecasts future frames or scene states from observed video for applications in perception, planning, and simulation.

See all 69 papers

Video Prediction Policy 2: Predict Better, Act Better
Oct 7, 2026Yanjiang Guo, Haodong Yan, Zhide Zhong +15Visuomotor Policy Learning World Action Models
Robotera · Tsinghua University · HKUST (GZ) +2
PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation
Oct 12, 2026Xinyu Guo, Chao Jing, Li Yi +2Robotic Manipulation Vision-Language-Action Models
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhongguancun Academy +2
WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
Oct 13, 2026Haomin Wang, Anqiong Ji, Ke Wang +4World Models for Robotics World Action Models
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1

Oct 7, 2026J/K move · Enter open · S save

Video Prediction Policy 2: Predict Better, Act Better

Yanjiang Guo, Haodong Yan, Zhide Zhong, Zhongru Zhang+14

Robotera · Tsinghua University · HKUST (GZ)+2

Visuomotor Policy Learning World Action Models Robot Learning from Video Zero-Shot Robotic Manipulation Video Prediction

Open paper AskPDF arXiv

World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform event-level video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.

Abstract

Video Prediction

Explore similar work

Video Prediction Policy 2: Predict Better, Act Better

PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation

WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction