AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation

Hao Xue; Peng-Fei Zhang; Ying Cheng; Lanyun Zhu; Lei Zhu; Heng Tao Shen

cs.AIOct 30, 2026

AuroraWorld: Structured 4D World Modeling for Exo-to-Ego Video Generation in Robot Manipulation

Authors: Hao Xue, Peng-Fei Zhang, Ying Cheng, Lanyun Zhu, Lei Zhu, Heng Tao Shen

Abstract

Egocentric observations provide a robot-centric view of manipulation processes, capturing contact interactions and object-state changes that are important for visuomotor policy learning but difficult to observe from external cameras. However, collecting such data at scale remains expensive due to additional hardware requirements and platform-specific deployment costs. Synthesizing egocentric observations from large-scale exocentric manipulation videos provides a scalable alternative. However, existing methods struggle to synthesize physically consistent and policy-relevant egocentric videos under substantial viewpoint transformations, severe occlusions, complex object interactions, and inherently sparse observations from available exocentric viewpoints. Code will be made publicly available at https://github.com/xueha0/AuroraWorld

Figures & tables

Figure 1: AuroraWorld proposes to models the underlying spatial, temporal and motion dynamics, enabling physically plausible exo-to-ego video generation.

Figure 2: An overview of AuroraWorld. The proposed framework consists of two stages. In the first stage, AuroraWorld infers a structured 4D world representation from sparse exocentric observations. Specifically, it identifies informative temporal keyframes, captures spatial structures, and models motion dynamics from exocentric views. The selected keyframes are then transferred to the egocentric viewpoint and, together with spatial and motion representations, serve as structural priors for video generation. In the second stage, these priors condition the diffusion model to synthesize coherent and physically consistent egocentric videos.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
VGGT	8.93	0.206	0.784	208.98	834.02
LagerNVS	15.05	0.5130	0.4332	40.991	245.97
SVD	12.84	0.4971	0.5294	31.874	363.84
BWM	15.77	0.5012	0.3309	25.632	70.073
Exo2Ego-V	15.43	0.4909	0.4020	22.731	130.11
WristWorld	13.68	0.4204	0.4527	30.267	141.98

Table 1: Quantitative comparison of egocentric video generation on DROID. Best results are highlighted in bold .

Figure 3: Qualitative comparison on DROID. Rows correspond to exocentric left/right ground-truth views, egocentric ground-truth views, and generated egocentric views from AuroraWorld and competing methods, including WristWorld, BWM, and Exo-Ego-V. The results show that AuroraWorld generates egocentric videos with improved geometric consistency, appearance fidelity, and temporal coherence compared with existing approaches.

Policy	WristWorld	BWM	AuroraWorld
RVT-2	55.60	61.10	64.00
$\pi_{0.5}$	36.85	37.70	40.15

Table 2: RLBench policy success rates (%) trained with different egocentric observations.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
WristWorld	18.24	0.654	0.414	22.71	306.34
BWM	19.88	0.709	0.422	39.99	373.72
AuroraWorld	20.01	0.681	0.375	16.91	149.93

Table 3: Quantitative comparison of egocentric video generation on ARX X5. Best results are highlighted in bold .

Task	w/o Ego	WW	BWM	AuroraWorld	Real Ego
Fold Towel	2/25	11/25	16/25	15/25	17/25
Pull Drawer	0/25	9/25	12/25	15/25	15/25
Stack Cups	0/25	0/25	1/25	0/25	12/25
Push Ball	8/25	8/25	9/25	11/25	14/25
Push Cube	7/25	13/25	13/25	15/25	18/25
Erase Whiteboard	7/25	7/25	9/25	11/25	12/25

Table 4: Real-robot policy evaluation on the ARX X5. WW denotes Wristworld.

$\mathbf{W}^{\mathrm{sp}}$	$\mathbf{W}^{\mathrm{tp}}$	$\mathbf{W}^{\mathrm{mo}}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
–	–	–	12.96	0.43	0.50	30.78	192.93
✓	–	–	13.09	0.50	0.49	24.69	168.45
–	✓	–	16.69	0.55	0.31	16.95	76.12
–	–	✓	14.52	0.49	0.42	21.99	129.28
✓	–	✓	14.38	0.49	0.42	20.47	110.43
–	✓	✓	17.45	0.57	0.28	15.79	64.66

Table 5: Ablation study of different components.

Interior anchors	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
0	14.38	0.49	0.42	20.47	110.43
1	15.60	0.52	0.35	18.04	78.35
3	17.87	0.58	0.26	15.44	60.17
8	17.85	0.58	0.26	17.86	63.44
All	18.09	0.59	0.26	17.37	73.09

Table 6: Ablation study on the number of interior anchors.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Setting
Backbone	GR00T-Dreams IDM
Batch size	64
Optimizer	AdamW, cosine schedule
Learning rate	$1\times 10^{-4}$
Training steps	150k steps
Chunk horizon	16

Appendix

Table 7: Trajectory predictor configuration on DROID.

Parameter	Setting
Batch size	64
Height	180
Width	320
Optimizer	AdamW, cosine schedule
Grad clip	3
Learning rate	$1\times 10^{-4}$

Appendix

Table 8: LagerNVS configuration on DROID.

Method	$\mathrm{EWMScore}_{\mathrm{ego}}$	Visual Quality	Motion Quality	Content Consistency	Physics Adherence	3D Accuracy	Controllability
WristWorld	62.4	57.7	57.8	57.3	55.1	79.9	69.8
Exo2Ego-V	57.6	47.5	51.4	55.9	52.7	78.8	66.1
BWM	63.3	57.3	59.2	56.8	54.3	82.7	73.3
SVD	58.0	51.8	44.7	56.3	58.2	78.0	69.8
LagerNVS	53.7	49.3	39.1	54.2	44.4	75.7	63.9
VGGT	59.4	51.3	84.8	53.8	25.9	67.8	50.5

Appendix

Table 10: WorldArena scores for DROID egocentric videos.

\mathrm{EWMScore}_{\mathrm{ego}}

averages the 14 applicable metrics defined in Appendix B.2 .

Method	Subj.	Aes.	Img.	BG	Dyn.	Inter.	Persp.	Instr.	Sem.	Flow	Depth	Photo.	Smooth.	JEPA
WristWorld	79.5	29.1	52.8	90.5	58.7	55.1	72.6	53.3	86.3	32.1	87.2	1.8	82.5	91.4
Exo2Ego-V	77.7	27.9	57.4	89.1	49.7	52.7	69.0	45.2	87.0	24.5	88.6	0.7	80.0	57.2
BWM	79.2	29.4	49.9	88.4	59.9	54.3	73.4	57.7	88.9	34.6	91.9	2.8	82.9	92.6
SVD	79.0	31.1	47.7	88.7	30.1	58.2	69.2	53.5	86.1	31.4	86.7	1.1	72.8	76.5
LagerNVS	75.1	25.1	41.7	85.7	31.8	44.4	63.4	41.4	86.3	17.9	87.9	1.8	67.6	81.0
VGGT	73.6	29.5	60.7	87.7	98.1	25.9	53.7	21.1	79.9	100.0	81.9	0.1	56.3	63.8

Appendix

Table 11: Results for the 14 retained normalized WorldArena metrics on the same DROID videos and input conditions as Table 10 . Subj.: subject consistency; Aes.: aesthetic quality; Img.: image quality; BG: background consistency; Dyn.: dynamic degree; Inter.: interaction quality; Persp.: perspectivity; Instr.: instruction following; Sem.: semantic alignment; Photo.: photometric consistency; Smooth.: motion smoothness.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
VGGT	10.74	0.33	0.60	184.46	1451.10
LagerNVS	18.77	0.56	0.25	53.04	401.24
SVD	15.88	0.51	0.45	70.54	450.37
BWM	20.40	0.60	0.24	47.38	146.05
Exo2Ego-V	20.37	0.58	0.19	31.59	269.39
WristWorld	19.55	0.56	0.19	22.75	59.68

Appendix

Table 12: Standard visual evaluation results on the RLBench Ego-Withheld Split.

Figure 4: Qualitative comparison between uniform sampling and motion-stabilized temporal-anchor selection. Under the same five-anchor budget, motion-stabilized selection better aligns the generated sequence with task-relevant manipulation phases.

Anchor selection
Setting	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Uniform	17.39	0.572	0.280	15.754	69.634
Motion-stab.	17.87	0.580	0.265	15.437	60.171
Number of interior anchors
0	14.38	0.492	0.425	20.466	110.430
1	15.60	0.524	0.354	18.039	78.348

Appendix

Table 13: Additional ablations on DROID. The two blocks compare temporal-anchor selection and the number of interior anchors.

Figure 5: Qualitative comparison on RLBench.

Figure 6: DROID examples: exocentric left/right and egocentric-view ground truth, Exo2Ego-V, WristWorld, BWM, SVD, LagerNVS, VGGT, and AuroraWorld. AuroraWorld better preserves target geometry, appearance, and temporal layout.

DROID
Trajectory Source	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Inferred	17.87	0.580	0.265	15.44	60.17
Ground-truth	19.17	0.6218	0.2185	14.16	52.11
RLBench
Inferred	20.89	0.5708	0.1401	28.32	59.67
Ground-truth	21.84	0.5845	0.1216	28.85	72.11

Appendix

Table 14: Video-generation metrics of AuroraWorld under inferred and synchronized ground-truth end-effector state trajectories.

WristWorld
Target Condition	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$
Inferred	13.68	0.4204	0.4527	30.27	141.98
Ground-truth	15.22	0.5161	0.3981	25.90	103.93
BWM
Inferred	15.77	0.5012	0.3309	25.63	70.07
Ground-truth	16.48	0.5319	0.2933	24.28	58.89

Appendix

Table 15: DROID egocentric video generation under inferred and ground-truth target conditions. Bold marks the best result overall; underlining marks the best result with inferred target conditions.

Figure 7: Qualitative comparison of egocentric video generation on ARX X5.

Abbr.	Task	Abbr.	Task
PID	Put item in drawer	RAD	Reach and drag
SBCT	Slide block to color target	TT	Turn tap
PMS	Put money in safe	OD	Open drawer
MOG	Meat off grill	PB	Push buttons

Appendix

Table 16: Abbreviations and names of the eight RLBench evaluation tasks.

Figure 8: ARX X5 performing six real-world manipulation tasks.

	RVT-2			$\pi_{0.5}$
Task	WW	BWM	Ours	WW	BWM	Ours
PID	69.2	79.2	73.6	5.2	8.8	15.6
RAD	69.6	55.6	68.0	77.6	63.2	65.6
TT	56.8	67.6	76.8	24.8	27.2	28.0
SBCT	33.6	37.2	49.6	75.6	65.2	70.4
OD	50.4	47.6	55.6	13.6	10.8	17.2

Appendix

Table 17: Per-task RLBench policy success (%) for the eight-task inferred-action Replacement experiment (250 rollouts per task). WW: WristWorld; BWM: Boundless World Model.

Task	Avail. only	+WW	+BWM	+Ours	Real Ego
Fold Towel	16/25	17/25	23/25	21/25	22/25
Pull Drawer	14/25	15/25	16/25	20/25	20/25
Stack Cups	13/25	6/25	10/25	8/25	16/25
Push Ball	14/25	17/25	17/25	18/25	20/25
Push Cube	16/25	22/25	19/25	22/25	23/25
Erase Whiteboard	10/25	10/25	13/25	17/25	16/25

Appendix

Table 18: ARX X5 policy success under data augmentation. Each entry reports successful trials out of 25 physical trials. WW denotes WristWorld.

Exocentric-to-Egocentric Video Generation

See all 9 papers

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization
Aug 29, 2026Cheng Chen, Jerry Bai, Jiacheng Wei +11Robot Skill Learning World Models for Robotics
Nanyang Technological University · XPENG Robotics · Zhejiang University +2
EgoExo-WM: Unlocking Exo Video for Ego World Models
May 14, 2026Danny Tran, Roberto Martín-Martín, Kristen GraumanAction-Conditioned Video Generation Exocentric-to-Egocentric Video Generation
The University of Texas at Austin
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Jul 30, 2026Zexuan Yan, Yuzhou Wu, Yue Ma +9Action-Conditioned Video Generation Memory-Augmented Video Generation
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4

Aug 29, 2026J/K move · Enter open · S save

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

Cheng Chen, Jerry Bai, Jiacheng Wei, Boyu Chen+10

Nanyang Technological University · XPENG Robotics · Zhejiang University+2

Robot Skill Learning World Models for Robotics World Models Cross-Embodiment Robot Learning Synthetic Data Generation for Robotics Robotic Manipulation

Open paper AskPDF arXiv

Collecting contact-rich robot experiences at scale remains a major bottleneck for generalizable manipulation. Beyond data quantity, robot learning also requires diverse experiences across embodiments, viewpoints, and scenes. Human egocentric videos provide abundant physical interactions, but each video captures only a narrow slice of experience under a single body, camera trajectory, and environment. We propose AnyWorld, a cross-embodiment world modeling framework that expands a single human interaction into diverse robot-native rollouts without paired human-robot demonstrations. Our model factorizes an interaction into action, camera, and embodiment: action controls capture the motion structure, camera controls specify viewpoint evolution, and the target embodiment context defines the acting body and its interaction geometry. This formulation enables independent recomposition of embodiment, viewpoint, and scene factors, allowing a single model to generate many robot-domain experiences while preserving the underlying dynamics and object interactions. We train the model with large-scale human interaction pretraining followed by mixed-embodiment fine-tuning. Experiments show that our model supports controllable recomposition across embodiments, viewpoints, and scenes, and we further demonstrate that the generated data can improve manipulation performance on the RoboCasa GR1 tabletop benchmark and a real IRON humanoid robot. Beyond aggregate gains, we test whether unpaired human experience can be recomposed into robot-native video-action pairs that target a policy gap. Controlled IRON interventions correct a spurious completion prior and establish language-grounded spatial target selection; an action-only counterfactual intervention fails to learn the latter reliably, showing that both action calibration and visual recomposition are necessary.

Abstract

Exocentric-to-Egocentric Video Generation

Explore similar work

AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization

EgoExo-WM: Unlocking Exo Video for Ego World Models

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE