Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control
Organizations: Stanford University · NYU Shanghai · UNC Chapel Hill
Abstract
Extended reality (XR) demands generative models that respond to users' tracked real-world motion, yet current video world models accept only coarse control signals such as text or keyboard input, limiting their utility for embodied interaction. We introduce a human-centric video world model that is conditioned on both tracked head pose and joint-level hand poses. For this purpose, we evaluate existing diffusion transformer conditioning strategies and propose an effective mechanism for 3D head and hand control, enabling dexterous hand--object interactions. We train a bidirectional video diffusion model teacher using this strategy and distill it into a causal, interactive system that generates egocentric virtual environments. We evaluate this generated reality system with human subjects and demonstrate improved task performance as well as a significantly higher level of perceived amount of control over the performed actions compared with relevant baselines.
Figures & tables
| Method | Video Quality | Hand Pose Accuracy | ||||||
|---|---|---|---|---|---|---|---|---|
| PSNR↑ | LPIPS↓ | SSIM↑ | FVD↓ | MPJPE↓ | MPVPE↓ | L2Err↓ | ||
| No Cond. | Baseline (Wan 2.2 Video 14B) | 14.59 | 0.4872 | 0.4855 | 601.55 | 17.86 | 12.29 | 67.50 |
| HPP Cond. | TokenConcat (PlayerOne Tu et al. 2025 ) | 15.09 | 0.4633 | 0.4983 | 560.34 | 18.02 | 12.34 | 65.43 |
| AdaLN (PEVA Bai et al. 2025b ) | 15.02 | 0.4591 | 0.4906 | 677.26 | 18.49 | 12.53 | 65.97 | |
| CrossAttention | 14.71 | 0.4686 | 0.4840 | 662.22 | 17.56 | 12.04 | 63.23 | |
| TokenAddition (ReCamMaster Bai et al. 2025a ) | 15.19 | 0.4520 | 0.4975 | 601.15 | 17.84 | 12.14 | 56.66 | |
| Method | Video Quality | Hand Pose Accuracy | Camera Pose Accuracy | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PSNR↑ | LPIPS↓ | SSIM↑ | FVD↓ | MPJPE↓ | MPVPE↓ | L2Err↓ | TransErr↓ | RotErr↓ | ||
| CamCtrl | CameraCtrl He et al. 2025a | 18.58 | 0.2943 | 0.6099 | 558.94 | 18.37 | 12.72 | 50.33 | 0.23 | 2.77 |
| HandCtrl | Best in Tab. 1 | 16.85 | 0.3874 | 0.5574 | 383.69 | 12.23 | 9.10 | 11.50 | 2.27 | 13.40 |
| JointCtrl | Ours | 18.60 | 0.2800 | 0.6173 | 396.93 | 12.81 | 9.66 | 13.42 | 0.25 | 2.79 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| MPJPE↓ | MPVPE↓ | L2Err↓ | TransErr↓ | RotErr↓ |
| 9.42 | 7.74 | 9.08 | 0.0191 | 0.44 ∘ |
| Method | MPJPE | MPVPE | L2Err |
|---|---|---|---|
| Ground-truth | 16.41 | 11.03 | 59.38 |
| Baseline | 20.86 | 15.08 | 268.49 |
| 3D Cond. | 20.63 | 14.90 | 250.79 |
| 2D Cond. | 19.67 | 14.03 | 134.77 |
| Hybrid Cond. | 17.78 | 12.48 | 89.59 |
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 257 papers