LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model
Authors: Quankai Gao, Jiawei Yang, Qiangeng Xu, Le Chen, Yue Wang
Organizations: University of Southern California, Los Angeles, USA · Waymo, Mountain View, USA · Max Planck Institute for Intelligent Syetems, Tübingen, Germany
Learning human-object manipulation presents significant challenges due to its fine-grained and contact-rich nature of the motions involved. Traditional physics-based animation requires extensive modeling and manual setup, and more importantly, it neither generalizes well across diverse object morphologies nor scales effectively to real-world environment. To address these limitations, we introduce LOME, an egocentric world model that can generate realistic human-object interactions as videos conditioned on an input image, a text prompt, and per-frame human actions, including both body poses and hand gestures. LOME injects strong and precise action guidance into object manipulation by jointly estimating spatial human actions and the environment contexts during training. After finetuning a pretrained video generative model on videos of diverse egocentric human-object interactions, LOME demonstrates not only high action-following accuracy and strong generalization to unseen scenarios, but also realistic physical consequences of hand-object interactions, e.g., liquid flowing from a bottle into a mug after executing a ``pouring'' action. Extensive experiments demonstrate that our video-based framework significantly outperforms state-of-the-art image based and video-based action-conditioned methods and Image/Text-to-Video (I/T2V) generative model in terms of both temporal consistency and motion control. LOME paves the way for photorealistic AR/VR experiences and scalable robotic training, without being limited to simulated environments or relying on explicit 3D/4D modeling.
Figures & tables
Figure 1. Overview . Given a reference image and a text instruction describing the manipulations shown in (a), LOME generates a temporally consistent egocentric hand–object interaction video, as shown in (c), conditioned on the corresponding per-frame human actions as in (b). Beyond accurate action adherence, LOME synthesizes realistic physical consequences of hand–object interactions, such as liquid dynamics when pouring from a bottle into a mug.
Figure 2. Training pipeline of LOME. A pretrained VAE encoder E maps the reference image I , input video V , and rasterized 2D action maps A^ to latent representations. A camera adapter encodes per-frame ray maps into camera features, which are added to the video latents. A Diffusion Transformer (DiT), conditioned on a text prompt, denoises the concatenated noisy action and video latents, and a pretrained decoder D reconstructs the generated video.
Figure 3. Action conditioning at frame i . (a) 3D human pose during video capture. (b) Projected 2D human pose Ai after filtering out keypoints and skeleton segments outside the camera frustum. (c) Background-masked rasterized 2D action map A^i used as the conditioning signal.
Figure 4. Qualitative action-following comparison across tasks. We compare LOME (ours) with CoSHAND, Wan-I2V and Go-with-Flow (GwtF) on diverse human-object manipulations. “Action” denotes our 2D action maps; CoSHAND uses its own hand masks; Wan-I2V uses no action condition; GwtF uses GT optical flow as action condition. Text prompts are overlaid on the ground-truth (GT) frames.
Figure 5. Pouring example. We compare LOME (ours), CoSHAND, Wan-I2V and Go-with-Flow (GwtF) on a “pouring liquid” task. Only LOME produces coherent liquid dynamics with a steadily increasing liquid level consistent with the text instruction. The prompt is overlaid on the GT frames.
Figure 6. Temporal resampling to align text and motion. We propose to resample clips with the varying number of frames to a fixed length ( i.e. 6 frames). (a) Longer clips are uniformly downsampled while preserving the first and last frames. (b) Shorter clips are upsampled by back-and-forth resampling to reach the target length.
Method
FVD ↓
CLIP-I ↑
CLIP-S ↑
tLPIPS ↓
PCK@20 ↑
CoSHAND
59.83
0.846
0.267
0.171
51.33
Wan-I2V
68.77
0.884
0.294
0.057
10.71
GwtF
40.33
0.878
0.256
0.071
47.06
LOME (ours)
39.58
0.903
0.284
0.051
66.85
Table 1. Quantitative comparisons on motion consistency and action-following. We report FVD, CLIP-I, CLIP-S, tLPIPS, and PCK@20 across methods (arrows indicate whether higher or lower is better).
Method
TF ↑
AF ↑
MC ↑
VQ ↑
CoSHAND
0.67
0.67
0.00
2.01
Wan-I2V
1.34
0.00
2.01
4.02
GwtF
0.67
0.67
0.67
0.00
LOME (ours)
97.32
98.66
97.32
93.97
Table 2. User study results. We collect votes from 30 participants over 10 test samples. LOME receives the highest percentage of votes for best text-following (TF), action-following (AF), motion consistency (MC), and visual quality (VQ).
Method
PSNR ↑
SSIM ↑
LPIPS ↓
SA ↑
PC ↑
CoSHAND
19.35
0.731
0.366
1.4
1.1
Wan-I2V
16.53
0.624
0.489
3.3
3.6
GwtF
18.93
0.704
0.435
2.6
2.1
LOME (ours)
18.88
0.719
0.325
4.1
4.0
Table 3. Quantitative comparison on visual quality and physics realism. For visual quality, we report PSNR, SSIM, and LPIPS between generated and ground-truth videos (arrows indicate whether higher or lower is better). For physics realism, we report Semantic Adherence (SA) and Physical Commonsense (PC).
Method
FVD ↓
CLIP-I ↑
tLPIPS ↓
PCK@20 ↑
Ours (w/o joint modeling)
48.01
0.881
0.063
62.81
Ours (channel concatenation)
41.23
0.900
0.055
65.77
Ours (w/o camera adapter)
39.51
0.903
0.049
66.84
Ours
39.78
0.903
0.051
66.85
Table 4. Ablation on motion consistency and action following. We evaluate the effects of removing joint modeling, removing camera adapter, and replacing temporal concatenation of action and video latents with channel concatenation. We report FVD, CLIP-I, tLPIPS, and PCK@20.
Figure 7. Examples of Noisy Dataset . We show an example of projection misalignment due to 3D human pose and camera estimation errors. (a) Input frame. (b) Overlay of the rasterized action map and the true hand positions, highlighting spatial offsets between the conditioning signal and the observed hands.
Figure 8. Failure case with multi-object interaction. In this example, LOME struggles to coordinate simultaneous interactions: it fails to correctly grasp the cup, causing ice cubes to fall into the tray rather than the cup (highlighted in red). The prompt is overlaid on the GT frames.
Figure 9. Ablation on joint action-environment modeling. (a) LOME without joint action–environment modeling, where no Gaussian noise is added to the action maps during training and the original CFG is applied. (b) LOME with joint action–environment modeling.
Figure 10. Occluded-object manipulation and output diversity. We compare LOME (ours), CoSHAND, Wan-I2V and GwtF on a challenging task where some of the objects to be manipulated are not visible in the input image (e.g., behind the fridge door). Among the three methods, only LOME produces plausible human–object interactions in this setting. LOME 1-3 denote three stochastic inference runs under identical conditions, illustrating output diversity. Text prompt is overlaid on the GT video frames.
Figure 11. In-the-wild lab captures. We showcase LOME on real-world egocentric scenes recorded in our lab with novel objects and environments, demonstrating generalization beyond the training data.
Figure 12. Optical flow condition of GwtF. We visualize the optical flow condition used by GwtF to warp the Gaussian noise corresponding to the input image. Optical flow is extracted from the “Input Video” i.e. GT video.
Figure 13. Per-frame PCK@20 visualization. We illustrate the PCK@20 evaluation by comparing detected hand locations in the generated videos against those in the ground-truth videos on a per-frame basis.
Figure 14. More qualitative comparisons across tasks. We compare LOME (ours) with CoSHAND, Wan-I2V and GwtF on diverse human-object manipulations. “Action” denotes our 2D action maps; CoSHAND uses its own hand masks; Wan-I2V uses no action condition; GwtF uses GT optical flow as action condition. Text prompts are overlaid on GT video frames.
Figure 15. Hand mask visualization of CoSHAND. We visualize the hand mask of CoSHAND on the same examples as in Fig. 4 of the main paper.
Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.
Zexuan Yan, Yuzhou Wu, Yue Ma +9
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4
Scaling dexterous manipulation requires generalization across objects, scenes, and tasks, yet existing data sources face a trade-off between scale and scene/embodiment alignment: teleoperation data is well aligned with robot deployment but expensive to collect; simulation is scalable but limited by the sim-to-real gap; and real egocentric videos scale effectively but remain misaligned with robot deployment. We propose Wh0, a framework that uses generative video world models as scalable and controllable sources of egocentric human-hand manipulation data to unlock the manipulation capabilities of pretrained dexterous VLA models. Conditioned on language, objects, and scenes, Wh0 uses a generative world model to produce WM-H, a 50k-episode dataset of egocentric human-object interaction videos. Wh0 then converts the generated videos into robot-trainable supervision through hand motion reconstruction and visual editing. Co-trained with a limited amount of real robot data, WM-H adapts pretrained VLA models to dexterous manipulation deployment. Across 18 real-world dexterous manipulation tasks, compared with a model post-trained only on robot data, Wh0 improves zero-shot success on unseen tasks from 8.3% to 38.9%. Ablation studies further show that scalable generation and scene/embodiment alignment are key drivers of performance gains. Videos and open-source code can be found on our project website: https://chenyt31.github.io/wh0.github.io/.
Yangtao Chen, Zixuan Chen, Peiyang Wang +4
Shanghai Innovation Institute · Nanjing University · Shanghai Jiaotong University
Egocentric manipulation videos are highly challenging due to severe occlusions during interactions and frequent object entries and exits from the camera view as the person moves. Current methods typically focus on recovering either hand or object pose in isolation, but both struggle during interactions and fail to handle out-of-sight cases. Moreover, their independent predictions often lead to inconsistent hand-object relations. We introduce WHOLE, a method that holistically reconstructs hand and object motion in world space from egocentric videos given object templates. Our key insight is to learn a generative prior over hand-object motion to jointly reason about their interactions. At test time, the pretrained prior is guided to generate trajectories that conform to the video observations. This joint generative reconstruction substantially outperforms approaches that process hands and objects separately followed by post-processing. WHOLE achieves state-of-the-art performance on hand motion estimation, 6D object pose estimation, and their relative interaction reconstruction. Project website: https://judyye.github.io/whole-www
Yufei Ye, Jiaman Li, Ryan Rong +1
Stanford University · Amazon FAR (Frontier AI & Robotics)