Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation

Mutian Xu; Tianbao Zhang; Tianqi Liu; Zhaoxi Chen; Xiaoguang Han; Ziwei Liu

cs.ROMar 17, 2026

Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulation

Authors: Mutian Xu, Tianbao Zhang, Tianqi Liu, Zhaoxi Chen, Xiaoguang Han, Ziwei Liu

Organizations: S-Lab, Nanyang Technological University · SSE, CUHKSZ

Abstract

Simulating robot-world interactions is a cornerstone of Embodied AI. Recently, a few works have shown promise in leveraging video generations to transcend the rigid visual/physical constraints of traditional simulators. However, they primarily operate in 2D space or are guided by static environmental cues, ignoring the fundamental reality that robot-world interactions are inherently 4D spatiotemporal events that require precise interactive modeling. To restore this 4D essence while ensuring the precise robot control, we introduce Kinema4D, a new action-conditioned 4D generative robotic simulator that disentangles the robot-world interaction into: i) Precise 4D representation of robot controls: we drive a URDF-based 3D robot via kinematics, producing a precise 4D robot control trajectory. ii) Generative 4D modeling of environmental reactions: we project the 4D robot trajectory into a pointmap as a spatiotemporal visual signal, controlling the generative model to synthesize complex environments' reactive dynamics into synchronized RGB/pointmap sequences. To facilitate training, we curated a large-scale dataset called Robo4D-200k, comprising 201,426 robot interaction episodes with high-quality 4D annotations. Extensive experiments demonstrate that our method effectively simulates physically-plausible, geometry-consistent, and embodiment-agnostic interactions that faithfully mirror diverse real-world dynamics. For the first time, it shows potential zero-shot transfer capability, providing a high-fidelity foundation for advancing next-generation embodied simulation.

Figures & tables

Figure 1: We propose Kinema4D , a new action-conditioned 4D generative embodied simulator . Given an initial world image with a robot at a canonical setup space, and an action sequence, our method generates future robot-world interactions in 4D space. It simulates physically plausible and geometrically consistent interactions between complex robot actions and diverse objects across various spatial constraints, providing a new foundation for advancing next-generation embodied simulation.

Figure 2: Overview of our Kinema4D . 1) Kinematics Control : Given a 3D robot with its URDF at initial canonical setup space, and an action sequence, we drive the 3D robot via kinematics to produce a 4D robot trajectory, which is then projected into a pointmap sequence. This process re-represents raw actions as a spatiotemporal visual signal. 2) 4D Generative Modeling : This signal and the initial main-view world image are sent to a shared VAE encoder, then fused with an occupancy-aligned robot mask and noise, which are denoised by a Diffusion Transformer [ 56 ] to generate a full future 4D (pointmap+RGB) world sequence.

Figure 3: Samples from our Robo4D-200k dataset . Our dataset provides a comprehensive data foundation by aggregating diverse real-world demonstrations, including DROID [ 38 ] , Bridge [ 68 ] , and RT-1 [ 6 ] . We further incorporate the LIBERO [ 45 ] to synthesize a vast array of successful/failure cases. Each episode captures a complete robot-world interaction ( e.g . , pick-and-place)—providing the continuous information necessary for robust reasoning. The 4D point clouds viewed from various camera frustums are shown here, demonstrating the spatial precision of our pseudo-annotations.

Method	Action	Output	PSNR $\uparrow$	SSIM $\uparrow$	L2 latent $\downarrow$	FID $\downarrow$	FVD $\downarrow$	LPIPS $\downarrow$
UniSim [ICLR’24] [ 78 ]	Text	RGB	19.32	0.681	0.2120	32.3	153.2	0.175
IRA-Sim [ICCV’25] [ 91 ]	Emb.	RGB	20.21	0.813	0.1722	25.2	126.0	0.135
Cosmos [arXiv’25] [ 2 ]	Emb.	RGB	20.39	0.787	0.1935	27.1	113.4	0.110
EVAC [arXiv’25] [ 37 ]	Emb.+2D	RGB	20.88	0.832	0.1896	29.3	122.0	0.150
ORV [CVPR’26] [ 80 ]	Emb.+3D	RGB	19.45	0.790	0.2002	30.1	130.1	0.143
Ctrl-World [ICLR’26] [ 23 ]	Emb.	RGB	21.03	0.803	0.1533	24.9	112.8	0.122

Table 1: Quantitative comparison of video generation metrics .

\uparrow

and

\downarrow

respectively indicates higher and lower is better. xxx denotes the best and xxx indicates the 2nd-best. “Action” means the representation of robot controls, where “Emb.” is token embedding. “Output” represents the modality of generation output.

Method	CD- $L_{1}$ $\downarrow$	CD- $L_{1}$ (temp) $\downarrow$	CD- $L_{2}$ $\downarrow$	CD- $L_{2}$ (temp) $\downarrow$	F-Score $\uparrow$	F-Score (temp) $\uparrow$
TesserAct [ICCV’25] [ 89 ]	0.0836	0.0067	0.0130	0.0008	0.2896	0.9523
Ours	0.0479	0.0074	0.0077	0.0002	0.4733	0.9686

Table 2: Quantitative comparison of geometric metrics . “(temp)” denotes the self-temporal consistency across frames. The threshold of F-Score is set to @0.01.

Figure 4: Qualitative comparison of 2D video synthesis between our Kinema4D and Ctrl-World [ 23 ] . Our method achieves superior fidelity. In contrast, Ctrl-World exhibits distorted actions and unrealistic environmental transitions.

Figure 5: 4D qualitative comparison between our Kinema4D and TesserAct [ 89 ] . Unlike TesserAct that hallucinate outcomes, our Kinema4D precisely reflects Ground-Truth executions, including “near-miss” failure cases. For example, in the bottom left corner example, our model correctly interprets the spatial gap between the gripper and the plant, even when their RGB textures overlap in 2D views .

Evaluator	Simulation			Real-world (OOD)
	1	2	3	1	2	3
Ground Truth	0.48	0.38	0.80	0.34	0.46	0.78
Ours	0.56	0.46	0.84	0.60	0.76	0.90
Diff	0.08	0.08	0.04	0.26	0.30	0.12

Table 3: Policy evaluation of actual pick&place under 3 different setups of both simulation platform and real-world (zero-shot OOD) environment. “Ground Truth” denotes the actual execution, “Ours” indicates our simulation, and “Diff” is their difference.

Figure 6: Qualitative comparison between real-world Ground Truth (GT) and our method for zero-shot OOD evaluation . To intuitively compare with GT, we also reconstruct the GT using [ 75 ] and present both 4D and 2D results. Our results strongly align with real outcomes, accurately synthesizing both successful rollouts and “near-miss” failures. For instance, in the right figure, our model correctly interprets the spatial gap between the gripper and the cube, even when their RGB textures overlap in 2D views . Besides, the noisy robot pointmap produced by our reconstruction pipeline is shown, proving the robustness of our framework.

Metrics	Ours	text	binary	emb.	RGB	RGB+pm	single	2D-out	w/o mask	0%	20%	50%	70%	remove	gaus.	trans.	rot.
PSNR $\uparrow$	22.50	19.89	21.47	20.89	21.53	22.98	21.26	20.07	21.03	21.10	22.04	21.75	21.83	22.48	21.98	21.87	22.34
FID $\downarrow$	25.2	28.8	27.5	26.3	25.8	25.7	26.7	27.4	26.8	26.1	25.2	25.0	26.0	26.0	25.3	25.6	26.3
CD- $L_{1}$ $\downarrow$	0.0479	0.0750	0.0639	0.0528	0.0677	0.0495	0.0581	0.0712	0.0510	0.0528	0.0433	0.0455	0.0463	0.0499	0.0501	0.0513	0.0483

Table 4: Quantitative results of ablation studies.

Figure VII : Qualitative comparison of 2D video synthesis between our Kinema4D and Ctrl-World [ 23 ] . Our method achieves superior fidelity. In contrast, Ctrl-World exhibits distorted actions and unrealistic environmental transitions.

Figure VIII : 4D qualitative comparison between our Kinema4D and TesserAct [ 89 ] . Unlike TesserAct that hallucinate outcomes, our Kinema4D precisely reflects Ground-Truth executions, including “near-miss” failure cases. For example, in the two bottom examples, our model correctly interprets the spatial gap between the gripper and the objects, even when their RGB textures overlap in 2D views .

Figure IX : 4D qualitative results of our Kinema4D for policy evaluation in simulation platform . Our simulated results strongly align with actual executions, accurately synthesizing both successful rollouts and “near-miss” failures. For instance, in the bottom figures, our model correctly interprets the spatial gap between the gripper and the object, even when their RGB textures overlap in 2D views.

Figure X : Extensive 4D qualitative results of our Kinema4D , across diverse domains, environments, robot actions, and the manipulated objects. These results are non-overlapping with other figures. More animated results are provided in demo.mp4 .

Scenario	Difficulty	Distractors ( $\mathcal{D}$ )	Constraints / Challenges
I. Canonical	Easy	$\emptyset$	Basic OOD texture grasping
II. Cluttered	Medium	$n\in[3,5],r\leq 5\text{cm}$	Spatial proximity & collision avoidance
III. Dynamic	Hard	$n>5$ , randomized	Partial occlusion & narrow passages

Table V : Specifications of Real-World Evaluation Scenarios.

Metrics	Ours	add env. pm	env. text	null text	depth
PSNR $\uparrow$	22.50	21.93	20.97	22.31	21.04
FID $\downarrow$	25.9	26.2	27.0	25.6	26.3
CD- $L_{1}$ $\downarrow$	0.0479	0.0420	0.0592	0.0501	0.0583

Table VI : Quantitative results of more ablation studies.

Method	Time (min)	Memory (GB)	PSNR $\uparrow$	PSNR (temp) $\uparrow$	FID $\downarrow$	FID (temp) $\downarrow$	CD- $L_{1}$ $\downarrow$	CD- $L_{1}$ (temp) $\downarrow$
Ours	15	56	19.23	36.95	29.3	15.4	0.0621	0.0068
Ctrl-World [ 23 ]	2.5	15	12.14	19.32	38.5	49.5	-	-
TesserAct [ 89 ]	10	31	10.80	25.87	41.8	28.7	0.0928	0.0125

Table VII : Comparison of the efficiency and cross-domain transfer. While the baseline methods offer better efficiency, their generalization abilities are extremely worse than our method, considering both the actual quality and the self-temporal consistency.

Robotics Simulation

Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.

See all 320 papers

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
May 3, 2026Peiyan Tu, Hanxin Zhu, Jingwen Sun +6Camera-Conditioned Video Generation Synthetic Data Generation for Robotics
Zhejiang University · University of Science and Technology of China · Beijing Zhongguancun Academy +3
MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation
Feb 10, 2026Jiaxu Wang, Yicheng Jiang, Tianlun He +7World Models for Robotics 4D Scene Generation
MMLab, The Chinese University of Hong Kong, Hong Kong SAR · The Hong Kong University of Science and Technology, Hong Kong · The University of Hong Kong +2
NeuROK: Generative 4D Neural Object Kinematics
May 28, 2026Chen Geng, Guangzhao He, Yue Gao +3Generative Modeling 3D Representation Learning
1Stanford University · 3Cornell University · University of Cambridge

May 3, 2026J/K move · Enter open · S save

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

Peiyan Tu, Hanxin Zhu, Jingwen Sun, Shaojie Ren+5

Zhejiang University · University of Science and Technology of China · Beijing Zhongguancun Academy+3

Camera-Conditioned Video Generation Synthetic Data Generation for Robotics Dynamic Novel View Synthesis Temporal Consistency in Video Generation Embodied World Models

Open paper AskPDF arXiv

Embodied agents require robust and comprehensive 3D spatiotemporal representations to support spatial reasoning, manipulation understanding, and downstream decision making. However, existing robot data are typically captured from fixed or sparse viewpoints, providing only partial and view-dependent observations, which limits multi-view perception and generalization across viewpoints. Given the difficulty of collecting additional viewpoints in real-world settings, we propose Embody4D, a dedicated video-to-video world model for embodied scenarios to bridge this observation gap by transforming a monocular robot video into novel-view videos from flexible target camera viewpoints. First, to tackle training data scarcity, we introduce a 3D-aware compositional synthesis pipeline to curate a heterogeneous dataset compositing cross-embodiment robotic arms with diverse backgrounds, promoting broad generalization. Second, to enforce geometric stability, we devise a latent confidence-aware expert modulation strategy, which estimates the reliability of warped latent priors and adaptively routes regions to copy, repair, or inpaint experts for spatiotemporally consistent 4D generation. Finally, to enhance the fidelity of the manipulation, we incorporate an interaction-aware attention mechanism that explicitly attends to the robotic interaction regions. Extensive experiments show that Embody4D achieves state-of-the-art performance on visual evaluation benchmarks, while both simulated and real-world robotic experiments further demonstrate its effectiveness as a robust data engine for synthesizing high-fidelity, view-consistent videos that empower downstream robotic planning and learning.

Abstract

Robotics Simulation

Explore similar work

Embody4D: A Generalist Data Engine for Embodied 4D World Modeling

MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation

NeuROK: Generative 4D Neural Object Kinematics