MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model

Jinguang Tong; Jinbo Wu; Kaisiyuan Wang; Zhelun Shen; Xuan Huang; Mochu Xiang; Xuesong Li; Yingying Li

cs.CVMar 16, 2026

MVHOI: Bridge Multi-view Condition to Complex Human-Object Interaction Video Reenactment via 3D Foundation Model

Authors: Jinguang Tong, Jinbo Wu, Kaisiyuan Wang, Zhelun Shen, Xuan Huang, Mochu Xiang, Xuesong Li, Yingying Li,

Organizations: The Australian National University · CSIRO · Baidu Inc. · Sun Yat-sen University

Abstract

Human-Object Interaction (HOI) video reenactment aims to transfer the interaction dynamics of a source video to a novel target object while preserving realistic hand-object coordination. Existing methods typically rely on sparse 2D motion controls and monocular references, which are insufficient for complex out-of-plane motion and large viewpoint changes. We present MVHOI, a two-stage framework combining implicit motion extraction, 3D-aware multi-view reasoning, and video generation. In the first stage, a motion extractor encodes object dynamics into implicit motion descriptors. Conditioned on these descriptors, our Motion-Driven Object Prior (MDOP) module queries a 3D foundation model over multi-view references of the target object and autoregressively predicts coarse object anchors, a sequence of images that track the object's evolving orientation and appearance under the source motion without any explicit pose estimation. In the second stage, a DiT-based video generation model uses these anchors as structural guidance and the multi-view references as appearance guidance. We further reuse cross-view attention from MDOP as a soft attention bias to reduce reference-view confusion. For long videos, a cross-iterative inference strategy refreshes subsequent object priors using refined video outputs. Experiments demonstrate consistent improvements over state-of-the-art methods in object fidelity, motion consistency, visual quality, and interaction realism.

Figures & tables

Figure 1: Given a source interaction video and multi-view images of a target object, MVHOI replaces the interacted object while faithfully following the source motion, preserving the target’s identity and structure even under large rotations and viewpoint changes.

Figure 2: Overview of MVHOI. Given a source interaction video and multi-view images of a target object, the Motion-Driven Object Prior (MDOP) module converts implicit source-object dynamics into a sparse sequence of target-specific object anchors within a pretrained 3D-aware multi-view feature space. These anchors provide motion-aligned structural guidance to a DiT-based video generator, while the original multi-view references supply complementary appearance details. For long videos, the refined object state from one clip initializes the prior construction of the next clip.

Figure 3: Qualitative comparison with state-of-the-art methods. Compared to MimicMotion ( Zhang et al. 2025a ) , HunyuanCustom ( Hu et al. 2025 ) , VACE ( Jiang et al. 2025 ) , and GenHOI ( Huang et al. 2026 ) , our method generates object motion that more faithfully follows the driving video. Moreover, MVHOI better preserves the identity and structural consistency of the target object during complex manipulation. Detailed experimental settings are provided in Section 4 .

Method	Self-Reenactment						Cross-Reenactment
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FVD $\downarrow$	O-CLIP $\uparrow$	FID $\downarrow$	FVD $\downarrow$	O-CLIP $\uparrow$
MimicMotion	19.34	0.806	0.233	56.13	632.9	0.685	84.52	774.2	0.470
VACE	27.25	0.954	0.036	47.43	204.6	0.706	62.54	308.1	0.556
HuMo	8.71	0.497	0.716	231.03	2120.8	0.449	225.81	2282.5	0.322
HunyuanCustom	25.56	0.925	0.090	48.88	247.4	0.791	62.68	349.7	0.640
GenHOI	29.60	0.949	0.035	19.24	76.8	0.866	48.01	280.4	0.680

Table 1: Quantitative comparison with baseline methods on Self-Reenactment and Cross-Reenactment. Our method achieves the best results on most metrics, with clear advantages in reconstruction fidelity, perceptual quality, and temporal consistency. Best and second-best results are highlighted in bold and underline , respectively.

Figure 4: Qualitative comparison of cross-object motion transfer. The source and target are different objects. While DisMo ( Ressler-Antal et al. 2025 ) transfers motion from a single target-object initialization, MDOP additionally reasons over multi-view target-object observations, leading to improved structural and appearance consistency under viewpoint changes.

Method	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	KID ( $10^{-3}$ ) $\downarrow$
DisMo	16.56	0.8081	111.96	24.81
Ours	20.75	0.8304	84.58	9.67

Table 2: Quantitative comparison with DisMo on held-out cross-object pairs from Objaverse. MDOP improves both frame-aligned reconstruction fidelity and perceptual distribution quality.

Method	Short			Long
Method	MC $\uparrow$	VQ $\uparrow$	HR $\uparrow$	MC $\uparrow$	VQ $\uparrow$	HR $\uparrow$
VACE	3.66	3.20	3.32	3.73	2.14	3.27
HunyuanCustom	3.59	3.29	3.11	3.49	3.56	3.17
GenHOI	4.20	4.10	4.15	4.07	3.94	3.89
Ours	4.47	4.34	4.34	4.50	4.32	4.32

Table 3: User study on Cross-Reenactment for short (single-clip) and long (10-second) video generation. MC, VQ, and HR denote motion consistency, visual quality, and HOI realism, rated on a 1–5 scale (higher is better).

Method	FID $\downarrow$	FVD $\downarrow$	O-CLIP $\uparrow$
Baseline	46.55	296.7	0.611
+MDOP	45.12	282.1	0.642
+MDOP+AE (full)	41.14	274.0	0.645

Table 4: Ablation study on Cross-Reenactment. MDOP denotes conditioning the video generator on the coarse object anchors from our Motion-Driven Object Prior module, and AE denotes the inference-time attention enhancement.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Cross-view association weights between motion-conditioned query states and target-object reference views.

Figure 6: Qualitative comparison of the baseline, MDOP conditioning, and the full model with attention enhancement (AE).

Table 11

Figure 7: Qualitative comparison between vanilla autoregressive (AR) inference and cross-iterative inference, shown at an interval of 80 frames. Frames with human context in the bottom row denote refined outputs used at segment boundaries.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 80 papers

ViHOI: Human-Object Interaction Synthesis with Visual Priors
Mar 25, 2026Songjin Cai, Linjie Zhong, Ling Guo +1HOI Generation Human Motion Generation
South China University of Technology
GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection
Mar 6, 2026Xuan Huang, Mochu Xiang, Zhelun Shen +9HOI Generation Physical Consistency in Video Generation
Department of Computer Vision Technology(VIS), Baidu Inc., China · Shenzhen Campus of Sun Yat-Sen University, China · Northwestern Polytechnical University, China
AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment
Jul 24, 2026Ziyao Huang, Shunkai Li, Juan Cao +7HOI Generation Video Diffusion Models
University of Chinese Academy of Sciences · Tencent HunYuan

Mar 25, 2026J/K move · Enter open · S save

ViHOI: Human-Object Interaction Synthesis with Visual Priors

Songjin Cai, Linjie Zhong, Ling Guo, Changxing Ding

South China University of Technology

HOI Generation Human Motion Generation Diffusion Models for Motion Generation

Open paper AskPDF arXiv

Generating realistic and physically plausible 3D Human-Object Interactions (HOI) remains a key challenge in motion generation. One primary reason is that describing these physical constraints with words alone is difficult. To address this limitation, we propose a new paradigm: extracting rich interaction priors from easily accessible 2D images. Specifically, we introduce ViHOI, a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images to enhance generation quality. We utilize a large Vision-Language Model (VLM) as a powerful prior-extraction engine and adopt a layer-decoupled strategy to obtain visual and textual priors. Concurrently, we design a Q-Former-based adapter that compresses the VLM's high-dimensional features into compact prior tokens, which significantly facilitates the conditional training of our diffusion model. Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences. During inference, it leverages reference images synthesized by a text-to-image generation model to improve generalization to unseen objects and interaction categories. Experimental results demonstrate that ViHOI achieves state-of-the-art performance, outperforming existing methods across multiple benchmarks and demonstrating superior generalization.

Abstract

HOI Generation

Explore similar work

ViHOI: Human-Object Interaction Synthesis with Visual Priors

GenHOI: Towards Object-Consistent Hand-Object Interaction with Temporally Balanced and Spatially Selective Object Injection

AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment