LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
Authors: Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue Wang
Organizations: University of Southern California, Los Angeles, USA · Waymo, Mountain View, USA · Max Planck Institute for Intelligent Syetems, Tübingen, Germany
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.
Figures & tables
Figure 1. Overview . Given a reference image and a text instruction describing the manipulations shown in (a), LOME generates a temporally consistent egocentric hand–object interaction video, as shown in (c), conditioned on the corresponding per-frame human actions as in (b). Beyond accurate action adherence, LOME synthesizes realistic physical consequences of hand–object interactions, such as liquid dynamics when pouring from a bottle into a mug.
Figure 2. Training pipeline of LOME. A pretrained VAE encoder E maps the reference image I , input video V , and rasterized 2D action maps A^ to latent representations. A camera adapter encodes per-frame ray maps into camera features, which are added to the video latents. A Diffusion Transformer (DiT), conditioned on a text prompt, denoises the concatenated noisy action and video latents, and a pretrained decoder D reconstructs the generated video.
Figure 3. Action conditioning at frame i . (a) 3D human pose during video capture. (b) Projected 2D human pose Ai after filtering out keypoints and skeleton segments outside the camera frustum. (c) Background-masked rasterized 2D action map A^i used as the conditioning signal.
Figure 4. Qualitative action-following comparison across tasks. We compare LOME (ours) with CoSHAND, Wan-I2V and Go-with-Flow (GwtF) on diverse human-object manipulations. “Action” denotes our 2D action maps; CoSHAND uses its own hand masks; Wan-I2V uses no action condition; GwtF uses GT optical flow as action condition. Text prompts are overlaid on the ground-truth (GT) frames.
Figure 5. Pouring example. We compare LOME (ours), CoSHAND, Wan-I2V and Go-with-Flow (GwtF) on a “pouring liquid” task. Only LOME produces coherent liquid dynamics with a steadily increasing liquid level consistent with the text instruction. The prompt is overlaid on the GT frames.
Figure 6. Temporal resampling to align text and motion. We propose to resample clips with the varying number of frames to a fixed length ( i.e. 6 frames). (a) Longer clips are uniformly downsampled while preserving the first and last frames. (b) Shorter clips are upsampled by back-and-forth resampling to reach the target length.
Method
FVD ↓
CLIP-I ↑
CLIP-S ↑
tLPIPS ↓
PCK@20 ↑
CoSHAND
59.83
0.846
0.267
0.171
51.33
Wan-I2V
68.77
0.884
0.294
0.057
10.71
GwtF
40.33
0.878
0.256
0.071
47.06
LOME (ours)
39.58
0.903
0.284
0.051
66.85
Table 1. Quantitative comparisons on motion consistency and action-following. We report FVD, CLIP-I, CLIP-S, tLPIPS, and PCK@20 across methods (arrows indicate whether higher or lower is better).
Method
TF ↑
AF ↑
MC ↑
VQ ↑
CoSHAND
0.67
0.67
0.00
2.01
Wan-I2V
1.34
0.00
2.01
4.02
GwtF
0.67
0.67
0.67
0.00
LOME (ours)
97.32
98.66
97.32
93.97
Table 2. User study results. We collect votes from 30 participants over 10 test samples. LOME receives the highest percentage of votes for best text-following (TF), action-following (AF), motion consistency (MC), and visual quality (VQ).
Method
PSNR ↑
SSIM ↑
LPIPS ↓
SA ↑
PC ↑
CoSHAND
19.35
0.731
0.366
1.4
1.1
Wan-I2V
16.53
0.624
0.489
3.3
3.6
GwtF
18.93
0.704
0.435
2.6
2.1
LOME (ours)
18.88
0.719
0.325
4.1
4.0
Table 3. Quantitative comparison on visual quality and physics realism. For visual quality, we report PSNR, SSIM, and LPIPS between generated and ground-truth videos (arrows indicate whether higher or lower is better). For physics realism, we report Semantic Adherence (SA) and Physical Commonsense (PC).
Method
FVD ↓
CLIP-I ↑
tLPIPS ↓
PCK@20 ↑
Ours (w/o joint modeling)
48.01
0.881
0.063
62.81
Ours (channel concatenation)
41.23
0.900
0.055
65.77
Ours (w/o camera adapter)
39.51
0.903
0.049
66.84
Ours
39.78
0.903
0.051
66.85
Table 4. Ablation on motion consistency and action following. We evaluate the effects of removing joint modeling, removing camera adapter, and replacing temporal concatenation of action and video latents with channel concatenation. We report FVD, CLIP-I, tLPIPS, and PCK@20.
Figure 7. Examples of Noisy Dataset . We show an example of projection misalignment due to 3D human pose and camera estimation errors. (a) Input frame. (b) Overlay of the rasterized action map and the true hand positions, highlighting spatial offsets between the conditioning signal and the observed hands.
Figure 8. Failure case with multi-object interaction. In this example, LOME struggles to coordinate simultaneous interactions: it fails to correctly grasp the cup, causing ice cubes to fall into the tray rather than the cup (highlighted in red). The prompt is overlaid on the GT frames.
Figure 9. Ablation on joint action-environment modeling. (a) LOME without joint action–environment modeling, where no Gaussian noise is added to the action maps during training and the original CFG is applied. (b) LOME with joint action–environment modeling.
Figure 10. Occluded-object manipulation and output diversity. We compare LOME (ours), CoSHAND, Wan-I2V and GwtF on a challenging task where some of the objects to be manipulated are not visible in the input image (e.g., behind the fridge door). Among the three methods, only LOME produces plausible human–object interactions in this setting. LOME 1-3 denote three stochastic inference runs under identical conditions, illustrating output diversity. Text prompt is overlaid on the GT video frames.
Figure 11. In-the-wild lab captures. We showcase LOME on real-world egocentric scenes recorded in our lab with novel objects and environments, demonstrating generalization beyond the training data.
Figure 12. Optical flow condition of GwtF. We visualize the optical flow condition used by GwtF to warp the Gaussian noise corresponding to the input image. Optical flow is extracted from the “Input Video” i.e. GT video.
Figure 13. Per-frame PCK@20 visualization. We illustrate the PCK@20 evaluation by comparing detected hand locations in the generated videos against those in the ground-truth videos on a per-frame basis.
Figure 14. More qualitative comparisons across tasks. We compare LOME (ours) with CoSHAND, Wan-I2V and GwtF on diverse human-object manipulations. “Action” denotes our 2D action maps; CoSHAND uses its own hand masks; Wan-I2V uses no action condition; GwtF uses GT optical flow as action condition. Text prompts are overlaid on GT video frames.
Figure 15. Hand mask visualization of CoSHAND. We visualize the hand mask of CoSHAND on the same examples as in Fig. 4 of the main paper.