Egocentric action transfer aims to decouple the semantic action from egocentric observations and reproduce them in new visual contexts, enabling scalable world simulation, embodied policy learning, and robotic data generation. Existing approaches to transferring actions rely on condition-guided video generation, which converts the observation video into explicit geometric controls (e.g., hand poses or meshes) to drive synthesis. However, these methods introduce geometric estimation errors, making it difficult to preserve physically plausible interactions when transferred to new scenes. Alternatively, motion transfer methods directly extract motion patterns from the observation video, yet motion-level features alone cannot encode rich contact dynamics, frequently leading to severe hand structural collapse and implausible contact layout. To address both limitations, we present EgoACT, a test-time framework for egocentric action transfer that operates directly on the denoising process of video diffusion models without requiring additional geometric estimators. EgoACT uses Velocity-guided Structure Anchoring to stabilize reference-consistent hand-object structure in the early denoising stage, and Sparse Correspondence Calibration to refine reliable local correspondences in the mid-to-late denoising stages. Together, these two components preserve transferable action semantics while improving temporal coherence and interaction realism. We further establish EgoActionBench, a benchmark for evaluating action preservation, visual quality, and hand-object plausibility across diverse egocentric manipulation scenarios. Experiments show that EgoACT generates more coherent and physically plausible interaction videos than strong baselines.
Figures & tables
Figure 2: Motivation. Motion transfer suffers from hand structural collapse and lacks physical interaction. Condition-guided generation preserves hand shape but produces physically implausible contact. Our method maintains both hand structure and realistic interaction.
Figure 3: Overview of the proposed framework. Given an egocentric manipulation video, our framework uses a reference branch and an optimized generation branch for test-time action transfer. We apply Velocity-guided Structure Anchoring (VSA) in the early denoising steps to preserve hand-object structure, and Sparse Correspondence Calibration (SCC) in the mid-to-late steps to refine local correspondences and improve interaction realism.
Figure 4: Illustration of the proposed SCC. (a) Direct transfer injects the reference action without correction, which may lead to errors in weak hand-object interactions. (b) Applying correction throughout the whole denoising process over-constrains early generation, making it difficult to decouple transferable action from source observation. (c) Our SCC performs sparse correspondence calibration only in the mid-to-late denoising stages, where the action layout has been established and local interaction details can be refined, improving temporal consistency and hand-object realism.
Figure 5: Gallery of proposed method. EgoACT can transfer a wide range of egocentric manipulation videos, covering diverse interaction categories, object types, and motion patterns. The generated videos preserve plausible hand-object structure, contact dynamics, and temporal coherence while adapting the reference action to new visual contexts.
Method
Quantitative Metrics
Vbench Metrics
Text Sim. ↑
Motion Fid. ↑
Temp. Cons. ↑
App. Cons. ↑
Sub. Cons. ↑
Back. Cons. ↑
Aes. Qual. ↑
Motion Smooth. ↑
InterDyn [ 2 ]
0.2645
0.2434
0.9673
0.9265
0.9105
0.9069
0.4273
0.9865
Mask2IV [ 30 ]
0.2604
0.1910
0.9749
0.8565
0.8929
0.9495
0.4673
0.9820
CosHand [ 61 ]
0.2602
0.2227
0.9277
0.9133
0.8381
0.9201
0.4226
0.8784
DiTFlow [ 45 ]
0.2579
0.2167
0.9750
0.8839
0.8856
0.9325
0.5065
0.9868
VACE [ 27 ]
0.2624
0.2389
0.9571
0.9149
0.9439
0.9507
0.4647
0.9881
Table 1: Comparison with state-of-the-art methods . Red and Blue denote the best and second best results, respectively.
Figure 6: The overview of EgoActionBench. EgoActionBench contains 350 reference manipulation videos and 1,400 target prompts, covering indoor/outdoor scenes, single-/two-hand interactions, and diverse manipulation categories. The benchmark evaluates whether generated videos preserve the reference action semantics while adapting to new visual contexts.
Figure 7: User Study. B1–B5 denote VACE, Mask2IV, CosHand, InterDyn, and DiTFlow, respectively. Our method is consistently preferred by human evaluators, achieving the highest excellent/good ratio across all evaluated aspects.
Figure 8: Qualitative comparison. Compare with previous methods, our method preserves stable hand-object layout and object response.
Method
Text Sim. ↑
Motion Fid. ↑
Temp. Cons. ↑
App. Cons. ↑
W/o VSA
0.2689
0.2442
0.9683
0.9312
W/o SCC
0.2641
0.2398
0.9714
0.9487
Ours
0.2746
0.2750
0.9866
0.9617
Table 2: Quantitative ablation. Red and blue denote the best and second-best results, respectively.
Figure 9: Visual ablation. VSA suppresses structural collapse, while SCC improves local correspondence.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: More visual results.
Figure 11: More visual results.
Figure 12: More visual results.
Figure 13: More details about EgoActionBench. EgoActionBench covers diverse egocentric manipulation scenarios and is annotated by scene type, and object category.
Figure 14: Failure case. Our method may struggle with actions involving very fast hand or object motion, which can lead to motion blur, temporal aliasing, or loss of hand-object correspondence. This limitation is primarily due to the base video diffusion model, which was not specifically trained to handle rapid egocentric dynamics and tends to produce temporally smoothed outputs when the inter-frame displacement is large.
Figure 15: Attention Visualization. We visualize the cross-frame attention maps before and after enabling VSA during the early denoising stage. Without VSA, attention is dispersed across irrelevant regions, failing to capture coherent hand-object structure. With VSA enabled, the attention concentrates on interaction-relevant areas such as hand, object, and contact regions, indicating that the generation trajectory is better aligned with the reference structural layout. This confirms that VSA effectively anchors the hand-object skeleton through velocity-guided alignment, resulting in more structurally consistent attention patterns.
Figure 16: More Comparison with Baseline. Our method consistently produces more coherent action transfer and more plausible hand-object interactions than baselines, which often suffer from structural collapse, temporal inconsistency, or physically unrealistic interactions.
Egocentric video offers rich manipulation experience for embodied AI, yet collecting diverse egocentric data across scenes, objects, motions, and embodiments remains costly. We present \method, an egocentric world-action simulator that synthesizes controllable, high-quality manipulation videos to expand scarce real-world training data. \method{} builds on a pretrained video generation prior and introduces two geometry-aware conditioning mechanisms. Online Anchored Projective Memory (OAPM) preserves a first-frame 3D scene anchor while periodically refreshing a recent state during autoregressive generation. Action-3D Rotary Position Embedding (A3D-RoPE) encodes end-effector motion with camera-aware 3D rotary coordinates, injecting action geometry into skeleton-to-video cross-attention for precise control. Together, these components improve visual fidelity, geometric stability, and action alignment in long egocentric rollouts. Moreover, augmenting 400 real trajectories with 400 \method-generated trajectories improves out-of-distribution real-robot success from 77% to 84% on single-arm tasks and from 53% to 70% on dual-arm tasks, demonstrating that the synthesized data substantially improve downstream WAM generalization.
Zexuan Yan, Yuzhou Wu, Yue Ma +9
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4
We introduce EgoSim, a closed-loop egocentric world simulator that generates spatially consistent interaction videos and persistently updates the underlying 3D scene state for continuous simulation. Existing egocentric simulators either lack explicit 3D grounding, causing structural drift under viewpoint changes, or treat the scene as static, failing to update world states across multi-stage interactions. EgoSim addresses both limitations by modeling 3D scenes as updatable world states. We generate embodiment interactions via a Geometry-action-aware Observation Simulation model, with spatial consistency from an Interaction-aware State Updating module. To overcome the critical data bottleneck posed by the difficulty in acquiring densely aligned scene-interaction training pairs, we design a scalable pipeline that extracts static point clouds, camera trajectories, and embodiment actions from in-the-wild large-scale monocular egocentric videos. We further introduce EgoCap, a capture system that enables low-cost real-world data collection with uncalibrated smartphones. Extensive experiments demonstrate that EgoSim significantly outperforms existing methods in terms of visual quality, spatial consistency, and generalization to complex scenes and in-the-wild dexterous interactions, while supporting cross-embodiment transfer to robotic manipulation. Codes and datasets will be open soon. The project page is at egosimulator.github.io.
Jinkun Hao, Mingda Jia, Ruiyan Wang +7
Shanghai Jiaotong University · Shanghai AI Laboratory · The University of Hong Kong
Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.
Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana +10
Mohamed bin Zayed University of Artificial Intelligence · Independent Researcher · Nanyang Technological University +1