MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

Jiaxu Wang; Yicheng Jiang; Tianlun He; Jingkai Sun; Qiang Zhang; Jiahang Cao; Zesen Gan; Mingyuan Sun

cs.CVFeb 10, 2026

MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

Authors: Jiaxu Wang, Yicheng Jiang, Tianlun He, Jingkai Sun, Qiang Zhang, Jiahang Cao, Zesen Gan, Mingyuan Sun,

Organizations: MMLab, The Chinese University of Hong Kong, Hong Kong SAR · The Hong Kong University of Science and Technology, Hong Kong · The University of Hong Kong · X-Humanoid Robots ·

Abstract

World-model-based imagine-then-act becomes a promising paradigm for robotic manipulation, yet existing approaches typically support either purely image-based forecasting or reasoning over partial 3D geometry, limiting their ability to predict complete 4D scene dynamics. This work proposes a novel embodied 4D world model that enables geometrically consistent, arbitrary-view RGBD generation: given only a single-view RGBD observation as input, the model imagines the remaining viewpoints, which can then be back-projected and fused to assemble a more complete 3D structure across time. To efficiently learn the multi-view, cross-modality generation, we explicitly design cross-view and cross-modality feature fusion that jointly encourage consistency between RGB and depth and enforce geometric alignment across views. Beyond prediction, converting generated futures into actions is often handled by inverse dynamics, which is ill-posed because multiple actions can explain the same transition. We address this with a test-time action optimization strategy that backpropagates through the generative model to infer a trajectory-level latent best matching the predicted future, and a residual inverse dynamics model that turns this trajectory prior into accurate executable actions. Experiments on three datasets demonstrate strong performance on both 4D scene generation and downstream manipulation, and ablations provide practical insights into the key design choices.

Figures & tables

Figure 1: The overview of our main pipeline.

Figure 2: Qualitative Results of 4D Generation on RoboTwin dataset. Red, green, and blue boxes represent different viewpoints.

Figure 3: Generated Geometries on Real-World Robot dataset

Figure 4: Effect of geometry-aware cross view modeling

	Appearance			Depth			Point Cloud
Method	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	AbRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	EMD $\downarrow$
RLBench
UniPi*	23.88	91.7	19.94	117.9	43.2	91.6	15.0	20.6
4DGen	22.25	87.1	20.51	91.8	29.3	94.1	10.9	16.0
TesserAct	23.86	92.8	27.77	91.8	29.4	96.8	11.0	16.3
Ours	23.31	90.8	18.57	90.5	29.1	97.1	9.6	15.3

Table 1: 4D generation results on the RLBench, RoboTwin, and our real-world datasets. The SSIM, AbRel, RMSE, CD, and EMD are scaled by

10^{2}

	Appearance			Depth			Point Cloud
Method	PSNR↑	SSIM↑	FVD↓	AbRel↓	RMSE↓	$\delta_{1}$ ↑	CD↓	EMD↓
w/o view	21.47	88.3	24.85	3.34	14.30	95.7	8.34	17.50
EA	22.43	89.2	23.36	3.03	12.70	96.9	7.33	12.50
w/o mod	20.16	83.8	25.25	4.03	16.77	95.2	7.51	16.80
Ours	22.91	90.2	21.93	2.60	12.30	97.4	6.51	9.90

Table 2: Ablation studies on cross- view and modality modules

Figure 5: Effect of explicit cross-modality modeling

	Baselines				Ablations
Dataset	P-ACT	UniPi *	4DGen	TesserAct	Act Head	full IDM	w/o R-IDM	full model
RLBench	60.4	34.6	47.0	67.3	72.5	68.8	69.0	72.6
RoboTwin	20.5	16.3	40.2	33.9	42.5	41.7	42.8	43.0

Table 3: Manipulation results on RLBench and RoboTwin.

Method	Arrange Boxes	Cap Bottle	Open Drawer	Place Fruits	Put Orange	Stack Cubes
TesserAct	7	27	37	17	66	45
Ours	15	33	56	23	63	50

Table 4: Success rates for some tasks on the real robot platform.

Dataset	w/o view	w/o mod	w/o view+mod	cat depth	w/o TCN-VAE	Full
RLBench	68.8	67.2	66.5	66.1	68.1	72.6
RoboTwin	38.0	38.6	36.2	39.4	41.5	43.0

Table 5: Manipulation ablations on fusion modules and trajectory representation. All variants use the same 3-view RGB-D input.

Benchmark	4DGen	TesserAct	Ours	+ActionInit
RLBench SR	47.0%	67.3%	72.6%	76.5%
RLBench Time	$51+22=73$ s	$37+16=53$ s	$42+16=58$ s	$42+5=47$ s
RoboTwin SR	40.2%	33.9%	43.0%	46.6%
RoboTwin Time	$83+25=108$ s	$68+23=91$ s	$75+23=98$ s	$75+9=84$ s

Table 6: Efficiency comparison under the same video backbone on a single NVIDIA RTX H200. Time is reported as generation + post-processing.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Simulator camera layout

Panel of Figure 7 — Figure 7: Real robot setup

Figure 8: Illustrations of Manipulation Tasks in our real-world dataset

	Appearance			Depth			Point Cloud
Method	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	EMD $\downarrow$
4DGen	22.25	87.1	20.51	91.8	29.3	94.1	10.9	16.0
TesserAct	23.86	92.8	27.29	91.8	29.4	96.8	11.0	16.3
Ours-mode1	22.55	89.3	21.67	90.8	29.2	96.9	10.2	15.2
Ours-mode2	23.31	90.8	18.57	90.5	29.1	97.1	9.6	15.3

Appendix

Table 7: Comparisons of 4D generation about our two modes strategies for supporting various multiview generations. SSIM, AbsRel, RMSE,

\delta_{1}

, CD, and EMD are scaled by

10^{2}

Method	Unplug Charger	Close Drawer	Close Grill	Close Microwave	Lamp On	Open Drawer	Pick Up Cup	Play Jenga	Push Button	Take Umbrella
P-ACT	6	88	96	64	8	80	12	96	72	52
UniPi*	9	62	70	45	0	68	0	78	12	2
4DGen	46	18	91	65	25	15	26	96	77	10
TesserAct	50	83	83	68	30	92	61	87	83	36
Ours	55	91	96	75	37	98	62	97	89	43

Appendix

Table 8: Detailed Comparisons of per manipulation task success rate on RLBench platform

Method	Adjust Bottle	Beat Hammer	Click Alarm clock	Click Bell	Grab Roller	Lift Pot	Move Pill bottle	Place Container	Object Stand	Phone Stand
P-ACT	56	3	37	22	26	3	0	43	15	0
UniPi*	28	15	10	5	51	18	0	26	10	0
4DGen	65	37	29	35	61	40	49	59	22	5
TesserAct	62	41	38	25	35	26	28	45	37	2
Ours	69	42	31	38	68	42	46	72	15	7

Appendix

Table 9: Detailed comparisons of per manipulation task success rate on RoboTwin platform.

	Appearance			Depth			Point Cloud
Method	PSNR $\uparrow$	SSIM $\uparrow$	FVD $\downarrow$	AbRel $\downarrow$	RMSE $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	EMD $\downarrow$
RoboTwin
Flatten Cam	22.57	89.6	21.26	3.27	15.62	93.9	11.36	12.45
Ours	22.91	90.2	21.93	2.60	12.30	97.4	6.51	9.90
Real-world Dataset
Flatten Cam	22.19	89.5	21.36	21.43	25.65	80.60	17.75	16.82

Appendix

Table 10: Ablation on camera embedding. We compare our proposed camera embedding against a baseline that flattens camera extrinsics and projects them with an MLP ( Flatten Cam ). Results are reported on RoboTwin and our real-world dataset. SSIM, AbsRel, RMSE,

\delta_{1}

, CD, and EMD are scaled by

10^{2}

Num Views	1	2	3	4	5
RLBench (%)	68.6	71.5	72.6	72.9	73.1
Time cost	0.78	0.85	1.00	1.20	1.35

Appendix

Table 11: Effect of the number of views on RLBench average success rate and time cost.

Figure 9: Failure case in the open-drawer task: our method successfully localizes the drawer handle but pulls in an incorrect direction, likely due to ambiguity in the optimal pulling direction under occlusion and limited viewpoints.

Figure 10: Failure case on RoboTwin (contact-sensitive task): the policy roughly reaches the target area but the end-effector is spatially misaligned and misses the red target block.

Figure 11: Multiview generation results by Mode-2 strategy on RoboTwin Dataset and corresponding results from 4DGen

Figure 12: Multiview generation results by Mode-1 strategy on RLBench Dataset and corresponding results from 4DGen

Figure 13: Qualitative Generation Results on RLBench Dataset.

Figure 14: Qualitative Generation Results on Real-World place cubes into cup task.

Figure 15: Qualitative Generation Results on Real-World open drawer task.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 369 papers

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
Jul 7, 2026Haoyu Zhao, Xingyue Zhao, Siteng Huang +3Language-Conditioned Robot Manipulation Bimanual Robotic Manipulation
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +1
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
May 20, 2026Kaichen Zhou, Yuzhen Chen, Fangneng Zhan +8Physical Consistency in Video Generation Robot Motion Generation
Harvard AI and Robotics Lab, Harvard University · Media Lab and EECS, MIT · MIT-IBM Watson AI Lab +1
Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
May 3, 2026Peiyan Tu, Hanxin Zhu, Jingwen Sun +6Camera-Conditioned Video Generation Synthetic Data Generation for Robotics
Zhejiang University · University of Science and Technology of China · Beijing Zhongguancun Academy +3

Jul 7, 2026J/K move · Enter open · S save

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Haoyu Zhao, Xingyue Zhao, Siteng Huang, Xin Li+2

DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK+1

Language-Conditioned Robot Manipulation Bimanual Robotic Manipulation World Models 4D Scene Understanding Embodied World Models Diffusion Models+1

Open paper AskPDF arXiv

Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow (RGB-DF) provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to low-level end-effector actions demanded by robotic systems, narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.

Abstract

World Models for Robotics

Explore similar work

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling