Generating realistic and physically plausible 3D Human-Object Interactions (HOI) remains a key challenge in motion generation. One primary reason is that describing these physical constraints with words alone is difficult. To address this limitation, we propose a new paradigm: extracting rich interaction priors from easily accessible 2D images. Specifically, we introduce ViHOI, a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images to enhance generation quality. We utilize a large Vision-Language Model (VLM) as a powerful prior-extraction engine and adopt a layer-decoupled strategy to obtain visual and textual priors. Concurrently, we design a Q-Former-based adapter that compresses the VLM's high-dimensional features into compact prior tokens, which significantly facilitates the conditional training of our diffusion model. Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences. During inference, it leverages reference images synthesized by a text-to-image generation model to improve generalization to unseen objects and interaction categories. Experimental results demonstrate that ViHOI achieves state-of-the-art performance, outperforming existing methods across multiple benchmarks and demonstrating superior generalization.
Figures & tables
Figure 2: Overall architecture of ViHOI . We extract visual priors from a set of reference images and textual priors from the input prompt using a VLM. This allows for natural alignment between priors of the two modalities. Subsequently, two Q-Former-based prior adapters distill these high-dimensional priors into a single compact token, respectively, providing the diffusion model with semantically consistent conditioning signals. At each denoising step, a selected HOI generator uses these compact visual and textual prior tokens to guide the synthesis of realistic, semantically coherent human-object interactions.
Figure 3: Illustration of strategies to obtain the set of reference images during the training and inference phases, respectively.
Figure 4: Qualitative comparisons on the FullBodyManipulation dataset [ 26 ] . Compared to state-of-the-art methods, our approach generates more realistic and physically plausible human-object interactions.
R-precision ↑
FID ↓
Diversity ↑
FS ↓
MPJPE ↓
Method
Top-1
Top-2
Top-3
MDM [ 41 ]
0.18
0.35
0.44
8.52
10.31
0.36
22.49
ROG [ 48 ]
0.17
0.33
0.49
6.16
9.62
0.31
19.79
CHOIS [ 25 ]
0.23
0.38
0.48
4.26
8.51
0.29
18.51
CHOIS+ViHOI
0.23
0.41
0.51
3.90
9.94
0.26
17.09
Table 3: Quantitative comparisons on the BEHAVE dataset [ 3 ] .
Method
R-score ↑
FID ↓
Cprec↑
C%
Phand↓
MPJPE ↓
V3-T24
0.67
2.35
0.81
0.67
0.56
18.05
V3-T36
0.71
1.19
0.80
0.61
0.57
17.30
V12-T12
0.75
0.87
0.81
0.62
0.59
15.90
V12-T36
0.67
2.31
0.80
0.66
0.62
17.97
V24-T24
0.61
3.15
0.81
0.64
0.54
16.94
V36-T36
0.74
1.08
0.80
0.63
0.58
16.65
Table 5: Ablation study on the FullBodyManipulation dataset [ 26 ] . We compare different layer combinations to extract Ev and Et . Vn-Tn indicates the LLM layer indices for the visual and textual modalities, respectively.
Figure 5: Qualitative comparisons on the 3D-Future dataset [ 11 ] . Our approach generates more realistic human-object interactions on unseen objects.
Method
R-score ↑
FID ↓
Cprec↑
C%
Phand↓
MPJPE ↓
k = 2
0.72
1.30
0.71
0.61
0.60
17.52
k = 4
0.63
3.07
0.78
0.58
0.59
18.38
k = 8
0.69
2.81
0.77
0.57
0.61
17.94
k = 1
0.79
0.68
0.83
0.64
0.58
14.97
Table 1: Impact of Adaptor Query Number (k) on Generation Performance. We vary only the number of visual prior queries, while keeping the number of text queries fixed at one.
Figure 1: Qualitative results on the FullBodyManipulation dataset [ 26 ] . The three images on the left side are the reference inputs, while the right side shows the motion sequences generated from them. Despite imperfections in these reference images, the generated HOI motions remain plausible and well aligned with the textual semantics.
Method
R-score ↑
FID ↓
Cprec↑
C%↑
Phand↓
MPJPE ↓
ViHOI-GT
0.79
0.29
0.82
0.64
0.59
12.94
ViHOI
0.79
0.68
0.83
0.64
0.58
14.97
Table 2: Impact of reference images. ViHOI-GT uses images rendered from GT motion, while ViHOI uses images from the T2I generation model.
Figure 2: User study on the FullBodyManipulation dataset [ 26 ] .
Figure 3: User study on the 3D-Future dataset [ 11 ] .
Figure 4: Additional visualization result on the FullBodyManipulation dataset [ 26 ] .
Figure 5: Additional visualization result on the 3D-FUTURE dataset [ 11 ] .
Figure 6: Qualitative analysis of VLM understanding. The text annotation is “Pull the smalltable, and set it back down.”
Figure 7: Qualitative analysis of VLM understanding. The text annotation is “Kick the base of the floorlamp, and set it back down.”