Human-Object Interaction (HOI) video reenactment aims to transfer the interaction dynamics of a source video to a novel target object while preserving realistic hand-object coordination. Existing methods typically rely on sparse 2D motion controls and monocular references, which are insufficient for complex out-of-plane motion and large viewpoint changes. We present MVHOI, a two-stage framework combining implicit motion extraction, 3D-aware multi-view reasoning, and video generation. In the first stage, a motion extractor encodes object dynamics into implicit motion descriptors. Conditioned on these descriptors, our Motion-Driven Object Prior (MDOP) module queries a 3D foundation model over multi-view references of the target object and autoregressively predicts coarse object anchors, a sequence of images that track the object's evolving orientation and appearance under the source motion without any explicit pose estimation. In the second stage, a DiT-based video generation model uses these anchors as structural guidance and the multi-view references as appearance guidance. We further reuse cross-view attention from MDOP as a soft attention bias to reduce reference-view confusion. For long videos, a cross-iterative inference strategy refreshes subsequent object priors using refined video outputs. Experiments demonstrate consistent improvements over state-of-the-art methods in object fidelity, motion consistency, visual quality, and interaction realism.
Figures & tables
Figure 1: Given a source interaction video and multi-view images of a target object, MVHOI replaces the interacted object while faithfully following the source motion, preserving the target’s identity and structure even under large rotations and viewpoint changes.
Figure 2: Overview of MVHOI. Given a source interaction video and multi-view images of a target object, the Motion-Driven Object Prior (MDOP) module converts implicit source-object dynamics into a sparse sequence of target-specific object anchors within a pretrained 3D-aware multi-view feature space. These anchors provide motion-aligned structural guidance to a DiT-based video generator, while the original multi-view references supply complementary appearance details. For long videos, the refined object state from one clip initializes the prior construction of the next clip.
Figure 3: Qualitative comparison with state-of-the-art methods. Compared to MimicMotion ( Zhang et al. 2025a ) , HunyuanCustom ( Hu et al. 2025 ) , VACE ( Jiang et al. 2025 ) , and GenHOI ( Huang et al. 2026 ) , our method generates object motion that more faithfully follows the driving video. Moreover, MVHOI better preserves the identity and structural consistency of the target object during complex manipulation. Detailed experimental settings are provided in Section 4 .
Method
Self-Reenactment
Cross-Reenactment
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
FVD ↓
O-CLIP ↑
FID ↓
FVD ↓
O-CLIP ↑
MimicMotion
19.34
0.806
0.233
56.13
632.9
0.685
84.52
774.2
0.470
VACE
27.25
0.954
0.036
47.43
204.6
0.706
62.54
308.1
0.556
HuMo
8.71
0.497
0.716
231.03
2120.8
0.449
225.81
2282.5
0.322
HunyuanCustom
25.56
0.925
0.090
48.88
247.4
0.791
62.68
349.7
0.640
GenHOI
29.60
0.949
0.035
19.24
76.8
0.866
48.01
280.4
0.680
Table 1: Quantitative comparison with baseline methods on Self-Reenactment and Cross-Reenactment. Our method achieves the best results on most metrics, with clear advantages in reconstruction fidelity, perceptual quality, and temporal consistency. Best and second-best results are highlighted in bold and underline , respectively.
Figure 4: Qualitative comparison of cross-object motion transfer. The source and target are different objects. While DisMo ( Ressler-Antal et al. 2025 ) transfers motion from a single target-object initialization, MDOP additionally reasons over multi-view target-object observations, leading to improved structural and appearance consistency under viewpoint changes.
Method
PSNR ↑
SSIM ↑
FID ↓
KID ( 10−3 ) ↓
DisMo
16.56
0.8081
111.96
24.81
Ours
20.75
0.8304
84.58
9.67
Table 2: Quantitative comparison with DisMo on held-out cross-object pairs from Objaverse. MDOP improves both frame-aligned reconstruction fidelity and perceptual distribution quality.
Method
Short
Long
MC ↑
VQ ↑
HR ↑
MC ↑
VQ ↑
HR ↑
VACE
3.66
3.20
3.32
3.73
2.14
3.27
HunyuanCustom
3.59
3.29
3.11
3.49
3.56
3.17
GenHOI
4.20
4.10
4.15
4.07
3.94
3.89
Ours
4.47
4.34
4.34
4.50
4.32
4.32
Table 3: User study on Cross-Reenactment for short (single-clip) and long (10-second) video generation. MC, VQ, and HR denote motion consistency, visual quality, and HOI realism, rated on a 1–5 scale (higher is better).
Method
FID ↓
FVD ↓
O-CLIP ↑
Baseline
46.55
296.7
0.611
+MDOP
45.12
282.1
0.642
+MDOP+AE (full)
41.14
274.0
0.645
Table 4: Ablation study on Cross-Reenactment. MDOP denotes conditioning the video generator on the coarse object anchors from our Motion-Driven Object Prior module, and AE denotes the inference-time attention enhancement.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Cross-view association weights between motion-conditioned query states and target-object reference views.
Figure 6: Qualitative comparison of the baseline, MDOP conditioning, and the full model with attention enhancement (AE).
Table 11
Figure 7: Qualitative comparison between vanilla autoregressive (AR) inference and cross-iterative inference, shown at an interval of 80 frames. Frames with human context in the bottom row denote refined outputs used at segment boundaries.
Generating realistic and physically plausible 3D Human-Object Interactions (HOI) remains a key challenge in motion generation. One primary reason is that describing these physical constraints with words alone is difficult. To address this limitation, we propose a new paradigm: extracting rich interaction priors from easily accessible 2D images. Specifically, we introduce ViHOI, a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images to enhance generation quality. We utilize a large Vision-Language Model (VLM) as a powerful prior-extraction engine and adopt a layer-decoupled strategy to obtain visual and textual priors. Concurrently, we design a Q-Former-based adapter that compresses the VLM's high-dimensional features into compact prior tokens, which significantly facilitates the conditional training of our diffusion model. Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences. During inference, it leverages reference images synthesized by a text-to-image generation model to improve generalization to unseen objects and interaction categories. Experimental results demonstrate that ViHOI achieves state-of-the-art performance, outperforming existing methods across multiple benchmarks and demonstrating superior generalization.
Hand-Object Interaction (HOI) remains a core challenge in digital human video synthesis, where models must generate physically plausible contact and preserve object identity across frames. Although recent HOI reenactment approaches have achieved progress, they are typically trained and evaluated in-domain and fail to generalize to complex, in-the-wild scenarios. In contrast, all-in-one video editing models exhibit broader robustness but still struggle with HOI-specific issues such as inconsistent object appearance. In this paper, we present GenHOI, a lightweight augmentation to pretrained video generation models that injects reference-object information in a temporally balanced and spatially selective manner. For temporal balancing, we propose Head-Sliding RoPE, which assigns head-specific temporal offsets to reference tokens, distributing their influence evenly across frames and mitigating the temporal decay of 3D RoPE to improve long-range object consistency. For spatial selectivity, we design a two-level spatial attention gate that concentrates object-conditioned attention on HOI regions and adaptively scales its strength, preserving background realism while enhancing interaction fidelity. Extensive qualitative and quantitative evaluations on unseen, in-the-wild scenes demonstrate that GenHOI significantly outperforms state-of-the-art HOI reenactment and all-in-one video editing methods. Project page: https://xuanhuang0.github.io/GenHOI/
Xuan Huang, Mochu Xiang, Zhelun Shen +9
Department of Computer Vision Technology(VIS), Baidu Inc., China · Shenzhen Campus of Sun Yat-Sen University, China · Northwestern Polytechnical University, China
Recent advances in video diffusion models have spurred interest in human-object interaction (HOI) video generation, which demands fine-grained control over interaction logic beyond single-subject animation. However, existing HOI methods rely heavily on explicit motion control, limiting scalability and generalization across diverse objects and interactions. In this study, we propose AgentHOI, a text-driven HOI video generation following a thinking-before-generation framework that bridges the gap between high-level textual intent and physical execution through multi-agent reasoning over perception, interaction, and motion planning. Building upon the generated interaction plans, we further strengthen text-driven motion understanding. We introduce an implicit text-motion alignment strategy that distills text-to-motion priors into the video diffusion model, enabling robust HOI synthesis without explicit motion inputs at inference. Experiments show that AgentHOI significantly improves interaction naturalness, object appearance preservation, and adherence to complex textual instructions across challenging object-centric scenarios such as wearing and riding. The code is available at https://github.com/bone-11/agenthoi.
Ziyao Huang, Shunkai Li, Juan Cao +7
University of Chinese Academy of Sciences · Tencent HunYuan