Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling

Haitang Feng; Zhuoman Liu; Zhennan Wang; Sheng Wang; Juelin Li; Chengzhi Zhao; Hexiang Wei; Lin Shao

cs.CVOct 29, 2026

Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling

Authors: Haitang Feng, Zhuoman Liu, Zhennan Wang, Sheng Wang, Juelin Li, Chengzhi Zhao, Hexiang Wei, Lin Shao

Organizations: RoboScience · Nanjing University · The Hong Kong Polytechnic University · National University of Singapore

Abstract

Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.

Figures & tables

Figure 1: GenSight unifies seeing and imagining for embodied world modeling. Given an initial image and a text prompt, GenSight generates future video together with aligned embodiment segmentation, depth, and first-frame-relative 2D flow. Given an observed video, the same model predicts each dense modality in a single step. Together, these outputs characterize appearance, semantics, geometry, and motion. The bottom panel illustrates 4D scene understanding through scene geometry and robot motion trajectories over time.

Figure 2: Overview of GenSight . Top: A shared Video DiT processes noisy video latents and predicts velocities for RGB video ( $V$ ), embodiment segmentation ( $S$ ), depth ( $D$ ), and first-frame-relative 2D flow ( $F$ ) in a unified prediction space. The first $k$ blocks compute task-agnostic features, while the remaining blocks apply task-adaptive RMSNorm with shared Transformer weights. The frozen video autoencoder is shared across modalities. Bottom left: Given an initial image and text prompt, video generation follows a multi-step denoising trajectory, with dense predictions read at modality-specific noise levels. Bottom right: Given an observed video, each dense modality is predicted in one step from a noisy state at its selected noise level. Both regimes share the dense readout and decoding procedure, without independent denoising trajectories for dense modalities.

Figure 3: Comparison of prediction target spaces.

RoboMIND 2.0
	Video				Depth		2D Flow
Method	FVD $\downarrow$	FID $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	nEPE $\downarrow$	$\mathrm{AAE}_{\mathrm{top}\,1\%}\downarrow$	EPE $\downarrow$
Wan2.2-I2V-A14B	69.37	50.01	17.13	0.3083	–	–	–	–	–
RynnWorld4D	73.45	47.90	18.07	0.2661	8.53	92.55	1.133	45.14	4.936
Cosmos3-Nano	44.97	42.09	16.90	0.3002	–	–	–	–	–
GenSight	14.55	16.12	18.01	0.2296	7.88	93.17	0.427	23.10	7.089

Table 1: Generation mode on RoboMIND task-OOD and OOD RoboTwin set. Depth and flow are scored with VGGT-

\Omega

and AllTracker, respectively. AbsRel and

\delta_{1}

are in %;

\mathrm{AAE}_{\mathrm{top}\,1\%}

is in degrees; EPE is in pixels.

Figure 4: Qualitative comparison in generation and perception modes. Left (generation): Given an initial frame and action prompt, the top panels show video examples from Wan 2.2 and Cosmos3-Nano; below, two time frames of video, depth, and 2D flow compare the reference, RynnWorld4D, and GenSight . Right (perception): Given observed RGB frames, we compare GenSight with reference maps and task-specific baselines for embodiment segmentation, depth, and 2D flow. Blue boxes mark zoomed-in regions.

Table 2: Perception mode on RoboMIND task-OOD set. RAFT † and RAFT ‡ share the 2D EPE, and their 3D scores use ground-truth and our predicted depth, respectively. Ranking is per metric: 1st , 2nd .

Figure 5: Embodiment-conditioned generation on RoboTwin 2.0. Given the same initial frame (left), GenSight generates a human-hand continuation (middle) or a robot-arm continuation (right) under the corresponding embodiment condition.

Figure 6: Ablation studies on UR5. Top two rows: Video generation and dense perception metrics across task-agnostic prefix lengths $k$ ; the dashed line marks our selected configuration. Bottom row: Evaluation of a fixed checkpoint across readout noise levels $\sigma$ , where markers indicate optima.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Sub-dataset	Clips	Hours
RoboMIND 2.0	UR5	48,459	359.42
	Tianyi	7,095	20.62
	Tienkung	6,831	42.42
	Franka	112,188	589.07
EgoTable	In-house Tasks	46,471	38.03
Total		221,044	1049.58

Appendix

Table A : Dataset statistics across different sub-datasets.

Figure A : Qualitative 3D tracking results. We obtain 3D trajectories by combining estimated depth and 2D optical flow. The visualizations compare trajectory alignment in image and 3D views. Tracking4World Lu et al. (2026) aligns with the ground truth in the image plane but exhibits a visible depth offset in the side view. Temporal fluctuations in estimated depth can also affect displacement-based trajectory metrics. Trajectory color indicates the magnitude of the absolute error.

Figure B : Qualitative Results on RoboMIND 2.0 (Franka Embodiment) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Figure C : Qualitative Results on RoboMIND 2.0 (UR5 Embodiment) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Figure D : Qualitative Results on RoboMIND 2.0 (Tianyi Embodiment) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Figure E : Qualitative Results on RoboMIND 2.0 (Tienkung Embodiment) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Figure F : Qualitative Results on EgoTable (Human Hand) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Figure G : Qualitative Results on RoboTwin 2.0 (Simulated AgileX Embodiment) Top: generative mode; middle: ground truth (GT); bottom: perception mode.

Embodied World Models

Embodied world models predict how environments change under an agent’s actions to support planning, learning, simulation, and control.

See all 165 papers

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
May 3, 2026Peiyan Tu, Hanxin Zhu, Jingwen Sun +6Camera-Conditioned Video Generation Synthetic Data Generation for Robotics
Zhejiang University · University of Science and Technology of China · Beijing Zhongguancun Academy +3
GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation
May 20, 2026Kaichen Zhou, Yuzhen Chen, Fangneng Zhan +8Physical Consistency in Video Generation Robot Motion Generation
Harvard AI and Robotics Lab, Harvard University · Media Lab and EECS, MIT · MIT-IBM Watson AI Lab +1
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
Feb 10, 2026Jiaxu Wang, Yicheng Jiang, Tianlun He +7World Models for Robotics 4D Scene Generation
MMLab, The Chinese University of Hong Kong, Hong Kong SAR · The Hong Kong University of Science and Technology, Hong Kong · The University of Hong Kong +2

May 3, 2026J/K move · Enter open · S save

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

Peiyan Tu, Hanxin Zhu, Jingwen Sun, Shaojie Ren+5

Zhejiang University · University of Science and Technology of China · Beijing Zhongguancun Academy+3

Camera-Conditioned Video Generation Synthetic Data Generation for Robotics Dynamic Novel View Synthesis Temporal Consistency in Video Generation Embodied World Models

Open paper AskPDF arXiv

Embodied agents require robust and comprehensive 3D spatiotemporal representations to support spatial reasoning, manipulation understanding, and downstream decision making. However, existing robot data are typically captured from fixed or sparse viewpoints, providing only partial and view-dependent observations, which limits multi-view perception and generalization across viewpoints. Given the difficulty of collecting additional viewpoints in real-world settings, we propose Embody4D, a dedicated video-to-video world model for embodied scenarios to bridge this observation gap by transforming a monocular robot video into novel-view videos from flexible target camera viewpoints. First, to tackle training data scarcity, we introduce a 3D-aware compositional synthesis pipeline to curate a heterogeneous dataset compositing cross-embodiment robotic arms with diverse backgrounds, promoting broad generalization. Second, to enforce geometric stability, we devise a latent confidence-aware expert modulation strategy, which estimates the reliability of warped latent priors and adaptively routes regions to copy, repair, or inpaint experts for spatiotemporally consistent 4D generation. Finally, to enhance the fidelity of the manipulation, we incorporate an interaction-aware attention mechanism that explicitly attends to the robotic interaction regions. Extensive experiments show that Embody4D achieves state-of-the-art performance on visual evaluation benchmarks, while both simulated and real-world robotic experiments further demonstrate its effectiveness as a robust data engine for synthesizing high-fidelity, view-consistent videos that empower downstream robotic planning and learning.

Abstract

Embodied World Models

Explore similar work

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

GEM-4D: Geometry-Enhanced Video World Models for Robot Manipulation

MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation