cs.CVMar 13, 2026
SaveEgocentric World Model for Photorealistic Hand-Object Interaction Synthesis
Organizations: Texas A&M University · University of Minnesota · University of Wisconsin–Madison · University of California, Los Angeles ·
Abstract
To serve as a scalable data source for embodied AI, world models should act as true simulators that infer interaction dynamics strictly from user actions, rather than mere conditional video generators relying on privileged future object states. In this context, egocentric Human-Object Interaction (HOI) world models are critical for predicting physically grounded first-person rollouts. However, building such models is profoundly challenging due to rapid head motions, severe occlusions, and high-DoF hand articulations that abruptly alter contact topologies. Consequently, existing approaches often circumvent these physics challenges by resorting to conditional video generation with access to known future object trajectories. We introduce EgoHOI, an egocentric HOI world model that breaks away from this shortcut to simulate photorealistic, contact-consistent interactions from action signals alone. To ensure physical accuracy without future-state inputs, EgoHOI distills geometric and kinematic priors from 3D estimates into physics-informed embeddings. These embeddings regularize the egocentric rollouts toward physically valid dynamics. Experiments on the HOT3D dataset demonstrate consistent gains over strong baselines, and ablations validate the effectiveness of our physics-informed design.
Figures & tables
Figure 1: Qualitative comparison between a baseline world model and EgoHOI. Starting from the same first frame, EgoHOI integrates physics-informed embeddings to model hand–object interaction dynamics, improving ego-motion consistency, kinematic fidelity, and object integrity over time. Zoom-ins highlight clearer contact details and reduced drift.
Figure 2: Overview of EgoHOI pipeline. We formulate EgoHOI as an egocentric world model that represents frames with a latent internal state and predicts action-driven transitions with a DiT backbone. Physics-informed embeddings distilled from reconstruction-based 3D priors, together with the first-frame object appearance, are integrated into the latent dynamics via lightweight adapters, enabling realistic hand–object interactions, geometry-consistent ego-motion, and stable object identity under viewpoint changes.
Figure 3: Qualitative comparison with baselines. Columns from left to right show the ground truth (GT), Wan, Cosmos-2B, Cosmos-14B, Uni3C, and our EgoHOI model. All methods receive the same first frame as input; compared with the baselines, our model better preserves hand and object geometry, maintains object identity, and produces more stable interaction dynamics over time. See the zooming boxes for comparison of the fine-grained details.
| Method | Frame Prediction | Ego-Motion Consistency | Kinematic Fidelity | |||||||
| PSNR | SSIM | LPIPS | Object-CLIP | ATE | RRE | RPE | MR | MPJPE | RMSE | |
| Wan [ 36 ] | 14.78 | 0.46 | 0.52 | 0.86 | 0.124 | 10.687 | 0.036 | 19.67% | 0.049 | 0.083 |
| Cosmos 2B [ 1 ] | 15.49 | 0.56 | 0.41 | 0.87 | 0.116 | 9.515 | 0.026 | 16.23% | 0.044 | 0.078 |
| Cosmos 14B [ 1 ] | 15.89 | 0.59 | 0.38 | 0.88 | 0.112 | 9.046 | 0.022 | 14.61% | 0.041 | 0.074 |
| Uni3C [ 5 ] | 14.21 | 0.50 | 0.55 | 0.82 | 0.168 | 14.209 | 0.026 | 20.50% | 0.073 | 0.089 |
| Ours (EgoHOI) | 21.05 | 0.65 | 0.27 | 0.92 | 0.084 | 5.192 | 0.021 | 5.84% | 0.014 | 0.044 |
Table 1: Comparison with baselines. We compare EgoHOI against a Wan backbone, Cosmos world models (2B, 14B) and Uni3C on frame prediction (PSNR/SSIM/LPIPS/Object-CLIP), ego-motion consistency (ATE/RRE/RPE), and kinematic fidelity (MR/MPJPE/RMSE). higher is better, lower is better.
| Method | Subject Consistency | Background Consistency | Motion Smoothness | Dynamic Degree | Aesthetic Quality | Imaging Quality |
| Wan [ 36 ] | 91.40% | 93.14% | 99.19% | 64.75% | 46.57% | 58.58% |
| Cosmos 2B [ 1 ] | 89.01% | 93.00% | 99.27% | 87.50% | 46.88% | 62.98% |
| Cosmos 14B [ 1 ] | 93.21% | 93.56% | 99.32% | 89.50% | 49.83% | 63.78% |
| Uni3C [ 5 ] | 91.91% | 94.29% | 99.33% | 49.00% | 47.03% | 61.47% |
| Ours (EgoHOI) | 95.51% | 94.91% | 99.40% | 53.29% | 52.03% | 64.48% |
Table 2: VBench Evaluation of the Baselines. Reported scores cover subject consistency, background consistency, motion smoothness, dynamic degree, aesthetic quality, and imaging quality. For the consistency and perceptual quality metrics, higher values correspond to better results, whereas dynamic degree characterizes how dynamic the generated content is rather than its realism.
Figure 4: Qualitative results for ablation on kinematic fidelity. Rows: GT, Base, HKE Injection, Ours (EgoHOI). HKE improves hand articulation stability and reduces local shape distortion. Zoom-in boxes highlight clearer hand–object contact details. The full model is the most stable, matching MR/MPJPE/RMSE.
| Method | MR | MPJPE | RMSE |
| Ours (base) | 28.77% | 0.576 | 0.089 |
| HKE Injection | 6.47% | 0.015 | 0.044 |
| Ours (EgoHOI) | 5.84% | 0.014 | 0.044 |
Table 3: Ablation on kinematic fidelity. We ablate HKE Injection and report hand motion fidelity in generated videos using MR, MPJPE, and RMSE; indicates lower is better.
| Method | ATE | RRE | RPE |
| Ours (base) | 0.133 | 15.249 | 0.039 |
| EME Injection | 0.096 | 6.525 | 0.023 |
| Ours (EgoHOI) | 0.084 | 5.192 | 0.021 |
Table 4: Ablation on ego-motion consistency. We evaluate the EME injection variant on long-horizon egocentric rollouts using ATE, RRE, and RPE, and compare it with the base backbone and the full EgoHOI model; indicates more consistent ego-motion trajectories and reduced accumulated drift.
Figure 5: Qualitative results for ablation of ego-motion consistency Rows: GT, Base, EME Injection, Ours (EgoHOI). EME mitigates long-horizon ego-motion drift and improves viewpoint stability. Zoom-ins highlight cleaner details under viewpoint changes. The full model best matches ATE/RRE/RPE gains.
| Method | Object CLIP | Subject Consistency | Imaging Quality | OPE | OOE |
| Ours (base) | 0.81 | 85.26% | 52.55% | 0.141 | 27.739 |
| OEE Injection | 0.83 | 87.90% | 54.75% | 0.131 | 24.124 |
| Ours (EgoHOI) | 0.92 | 95.51% | 64.48% | 0.015 | 9.412 |
Table 5: Ablation on object integrity. We evaluate the OEE injection variant on egocentric rollouts, where object integrity refers to preserving object identity and maintaining stable object pose under interaction. We report Object CLIP, Subject Consistency, and Imaging Quality ( is better), together with object position error (OPE) and 2D orientation error (OOE) computed from object masks ( is better).
Figure 6: Qualitative results for ablation of object integrity. Rows: GT, Base, OEE Injection, Ours (EgoHOI). OEE reduces texture flicker and appearance drift, preserving object-specific details through interaction and occlusion. Zoom-ins highlight more stable patterns and edges. The full model yields the most consistent results.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | Kernel | Stride | Channels | Padding |
| Conv3d | (3,3,3) | (1,1,1) | 3 16 | (1,1,1) |
| Conv3d | (3,3,3) | (1,1,1) | 16 16 | (1,1,1) |
| Conv3d | (3,3,3) | (1,1,1) | 16 16 | (1,1,1) |
| Conv3d | (3,3,3) | (1,2,2) | 16 16 | (1,1,1) |
| Conv3d | (3,3,3) | (2,2,2) | 16 16 | (1,1,1) |
| Conv3d | (3,3,3) | (2,2,2) | 16 16 | (1,1,1) |
Table 6: Implementation details of the temporal 3D convolutional stack used in HKE. For each Conv3d layer we list the kernel size, stride, input output channels, and padding.
| Layer | Kernel | Stride | Channels | Padding |
| Conv2d | 3 | 1 | 3 16 | 1 |
| Conv2d | 3 | 1 | 16 16 | 1 |
| Conv2d | 3 | 1 | 16 16 | 1 |
| Conv2d | 3 | 2 | 16 16 | 1 |
| Conv2d | 3 | 2 | 16 16 | 1 |
| Conv2d | 3 | 2 | 16 20 | 1 |
Table 7: Implementation details of the reference-hand branch used in HKE. This branch processes the first-frame hand render; for each Conv2d layer we list the kernel size, stride, input output channels, and padding.
Figure 7: Visualization of hand kinematics. For each frame, we use the egocentric RGB image (left) together with a rendered hand-kinematics map (right) that visualizes the articulated meshes of both hands in the camera coordinate frame.
| Method | OPE | OOE |
| Wan | 0.078 | 19.295 |
| Cosmos 2B | 0.112 | 23.823 |
| Cosmos 14B | 0.108 | 23.312 |
| Uni3C | 0.083 | 24.751 |
| Ours (EgoHOI) | 0.015 | 9.412 |
Table 8: Ablation on object integrity. In addition to the object integrity analysis in the main paper, we report object position error (OPE) and 2D orientation error (OOE) to further quantify how well each method preserves object location and in-plane orientation under interaction. is better for both metrics.
Figure 8: Hand Segmentation for RMSE Evaluation. For each frame, we extract the ground-truth hand region (left) and the corresponding inferred hand region produced by EgoHOI (right). The RMSE is computed as the per-pixel root-mean-square error between these two hand masks, averaged over all frames.
Figure 9: Qualitative comparison with baselines (part I). Rows from top to bottom show the ground truth (GT), Cosmos-2B, Cosmos-14B, Wan, Uni3C, and our EgoHOI model. Each column block shows consecutive frames from one egocentric test sequence in temporal order. All methods receive the same first frame as input. Compared with the baselines, our model better preserves hand and object geometry, maintains object identity, and produces more stable interaction dynamics over time. See the zooming boxes for comparison of the fine-grained details.
Figure 10: Qualitative comparison with baselines (part II). Rows from top to bottom show the ground truth (GT), Cosmos-2B, Cosmos-14B, Wan, Uni3C, and our EgoHOI model. Each column block shows consecutive frames from one egocentric test sequence in temporal order. All methods receive the same first frame as input. Compared with the baselines, our model better preserves hand and object geometry, maintains object identity, and produces more stable interaction dynamics over time. See the zooming boxes for comparison of the fine-grained details.
Figure 11: Qualitative analysis under different user inputs. For each example, both rows start from the same first frame but use different user inputs. EgoHOI produces distinct rollout outcomes with different viewpoint evolution and interaction trajectories, while preserving coherent scene structure and plausible hand-object interaction.
More in this topic
HOI Generation
Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.
See all 84 papers