Generating realistic and physically plausible 3D Human-Object Interactions (HOI) remains a key challenge in motion generation. One primary reason is that describing these physical constraints with words alone is difficult. To address this limitation, we propose a new paradigm: extracting rich interaction priors from easily accessible 2D images. Specifically, we introduce ViHOI, a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images to enhance generation quality. We utilize a large Vision-Language Model (VLM) as a powerful prior-extraction engine and adopt a layer-decoupled strategy to obtain visual and textual priors. Concurrently, we design a Q-Former-based adapter that compresses the VLM's high-dimensional features into compact prior tokens, which significantly facilitates the conditional training of our diffusion model. Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences. During inference, it leverages reference images synthesized by a text-to-image generation model to improve generalization to unseen objects and interaction categories. Experimental results demonstrate that ViHOI achieves state-of-the-art performance, outperforming existing methods across multiple benchmarks and demonstrating superior generalization.
Figures & tables
Figure 2: Overall architecture of ViHOI . We extract visual priors from a set of reference images and textual priors from the input prompt using a VLM. This allows for natural alignment between priors of the two modalities. Subsequently, two Q-Former-based prior adapters distill these high-dimensional priors into a single compact token, respectively, providing the diffusion model with semantically consistent conditioning signals. At each denoising step, a selected HOI generator uses these compact visual and textual prior tokens to guide the synthesis of realistic, semantically coherent human-object interactions.
Figure 3: Illustration of strategies to obtain the set of reference images during the training and inference phases, respectively.
Figure 4: Qualitative comparisons on the FullBodyManipulation dataset [ 26 ] . Compared to state-of-the-art methods, our approach generates more realistic and physically plausible human-object interactions.
R-precision ↑
FID ↓
Diversity ↑
FS ↓
MPJPE ↓
Method
Top-1
Top-2
Top-3
MDM [ 41 ]
0.18
0.35
0.44
8.52
10.31
0.36
22.49
ROG [ 48 ]
0.17
0.33
0.49
6.16
9.62
0.31
19.79
CHOIS [ 25 ]
0.23
0.38
0.48
4.26
8.51
0.29
18.51
CHOIS+ViHOI
0.23
0.41
0.51
3.90
9.94
0.26
17.09
Table 3: Quantitative comparisons on the BEHAVE dataset [ 3 ] .
Method
R-score ↑
FID ↓
Cprec↑
C%
Phand↓
MPJPE ↓
V3-T24
0.67
2.35
0.81
0.67
0.56
18.05
V3-T36
0.71
1.19
0.80
0.61
0.57
17.30
V12-T12
0.75
0.87
0.81
0.62
0.59
15.90
V12-T36
0.67
2.31
0.80
0.66
0.62
17.97
V24-T24
0.61
3.15
0.81
0.64
0.54
16.94
V36-T36
0.74
1.08
0.80
0.63
0.58
16.65
Table 5: Ablation study on the FullBodyManipulation dataset [ 26 ] . We compare different layer combinations to extract Ev and Et . Vn-Tn indicates the LLM layer indices for the visual and textual modalities, respectively.
Figure 5: Qualitative comparisons on the 3D-Future dataset [ 11 ] . Our approach generates more realistic human-object interactions on unseen objects.
Method
R-score ↑
FID ↓
Cprec↑
C%
Phand↓
MPJPE ↓
k = 2
0.72
1.30
0.71
0.61
0.60
17.52
k = 4
0.63
3.07
0.78
0.58
0.59
18.38
k = 8
0.69
2.81
0.77
0.57
0.61
17.94
k = 1
0.79
0.68
0.83
0.64
0.58
14.97
Table 1: Impact of Adaptor Query Number (k) on Generation Performance. We vary only the number of visual prior queries, while keeping the number of text queries fixed at one.
Figure 1: Qualitative results on the FullBodyManipulation dataset [ 26 ] . The three images on the left side are the reference inputs, while the right side shows the motion sequences generated from them. Despite imperfections in these reference images, the generated HOI motions remain plausible and well aligned with the textual semantics.
Method
R-score ↑
FID ↓
Cprec↑
C%↑
Phand↓
MPJPE ↓
ViHOI-GT
0.79
0.29
0.82
0.64
0.59
12.94
ViHOI
0.79
0.68
0.83
0.64
0.58
14.97
Table 2: Impact of reference images. ViHOI-GT uses images rendered from GT motion, while ViHOI uses images from the T2I generation model.
Figure 2: User study on the FullBodyManipulation dataset [ 26 ] .
Figure 3: User study on the 3D-Future dataset [ 11 ] .
Figure 4: Additional visualization result on the FullBodyManipulation dataset [ 26 ] .
Figure 5: Additional visualization result on the 3D-FUTURE dataset [ 11 ] .
Figure 6: Qualitative analysis of VLM understanding. The text annotation is “Pull the smalltable, and set it back down.”
Figure 7: Qualitative analysis of VLM understanding. The text annotation is “Kick the base of the floorlamp, and set it back down.”
Recent advances in video diffusion models have spurred interest in human-object interaction (HOI) video generation, which demands fine-grained control over interaction logic beyond single-subject animation. However, existing HOI methods rely heavily on explicit motion control, limiting scalability and generalization across diverse objects and interactions. In this study, we propose AgentHOI, a text-driven HOI video generation following a thinking-before-generation framework that bridges the gap between high-level textual intent and physical execution through multi-agent reasoning over perception, interaction, and motion planning. Building upon the generated interaction plans, we further strengthen text-driven motion understanding. We introduce an implicit text-motion alignment strategy that distills text-to-motion priors into the video diffusion model, enabling robust HOI synthesis without explicit motion inputs at inference. Experiments show that AgentHOI significantly improves interaction naturalness, object appearance preservation, and adherence to complex textual instructions across challenging object-centric scenarios such as wearing and riding. The code is available at https://github.com/bone-11/agenthoi.
Ziyao Huang, Shunkai Li, Juan Cao +7
University of Chinese Academy of Sciences · Tencent HunYuan
Modeling 4D human-object interaction (HOI) is a compelling challenge in computer vision and an essential technology powering virtual and mixed-reality applications. While existing works have achieved promising results on specific HOI tasks-such as text-conditioned HOI generation and human motion generation from object motion, they typically rely on task-specific architectures and lack a unified framework capable of handling diverse conditional inputs. Building on this, we propose Uni-HOI, a unified framework that learns the joint distribution among text, human motion, and object motion. By leveraging large language models (LLMs) and two motion-specific vector quantized variational autoencoders (VQ-VAEs), we convert heterogeneous motion data into token sequences compatible with LLM inputs, enabling seamless integration and joint modeling of all three modalities. We introduce a two-stage training strategy: the first stage performs multi-task learning on a large-scale HOI dataset to capture the underlying correlations among the three modalities, while the second stage fine-tunes the model on specific tasks to further enhance performance. Extensive experiments demonstrate that Uni-HOI achieves remarkable performances on multiple HOI-related tasks including text-driven HOI generation, object motion-driven human motion generation (optionally with text) and human motion-driven object motion prediction within a unified framework.
Mengfei Zhang, Jinlu Zhang, Zhigang Tu
Wuhan University Wuhan, China · Peking University Beijing, China
Human-Object Interaction (HOI) video reenactment aims to transfer the interaction dynamics of a source video to a novel target object while preserving realistic hand-object coordination. Existing methods typically rely on sparse 2D motion controls and monocular references, which are insufficient for complex out-of-plane motion and large viewpoint changes. We present MVHOI, a two-stage framework combining implicit motion extraction, 3D-aware multi-view reasoning, and video generation. In the first stage, a motion extractor encodes object dynamics into implicit motion descriptors. Conditioned on these descriptors, our Motion-Driven Object Prior (MDOP) module queries a 3D foundation model over multi-view references of the target object and autoregressively predicts coarse object anchors, a sequence of images that track the object's evolving orientation and appearance under the source motion without any explicit pose estimation. In the second stage, a DiT-based video generation model uses these anchors as structural guidance and the multi-view references as appearance guidance. We further reuse cross-view attention from MDOP as a soft attention bias to reduce reference-view confusion. For long videos, a cross-iterative inference strategy refreshes subsequent object priors using refined video outputs. Experiments demonstrate consistent improvements over state-of-the-art methods in object fidelity, motion consistency, visual quality, and interaction realism.
Jinguang Tong, Jinbo Wu, Kaisiyuan Wang +12
The Australian National University · CSIRO · Baidu Inc. +1