Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
Figures & tables
Figure 2: HOI3D framework overview. Top: We first leverage the existing multimodal foundation model InternVL [ 8 ] to perform decomposed annotation of human, object, and human-object-interaction of samples from the ProciGen [ 57 ] dataset. We then use LLaMa [ 16 ] to create a final detailed caption for the sample. Bottom: We leverage our data consisting of high-quality and diverse human-object-interactions to fine-tune an existing text-to-image model. Subsequently, we establish a pipeline to reconstruct high-fidelity textured 3D meshes. The output of our final text-to-3D inference pipeline consists of segmented meshes for the human and object, as well as an animatable SMPL model.
Figure 3: Analysis of the CLIP score. While our model clearly generates images that follow input interaction descriptions more precisely than SANA [ 56 ] , the CLIP score indicates the opposite, rendering it unusable as a metric for our task.
Figure 4: Qualitative comparison for text to 3D generation . InterFusion [ 9 ] is based on Score Distillation Sampling and hence is slow and produces low-quality 3D due to the well-known Janus problem. TRELLIS [ 55 ] is a learning based native 3D generation method, hence it can produce better 3D but is not interaction-aware. Our method faithfully follows the text prompts, especially the detailed body contact specifications. Our contacts are highlighted with spheres coloured based on contacting body parts.
Method
Text Consistency
3D Quality
GPT ↑
CLIP ↑
Contact ↑
User ↑
GPT ↑
User ↑
TRELLIS
0.04
0.32
N/A
3.44%
0.21
10.16%
InterFusion
0.15
0.35
N/A
5.47%
0.00
3.28%
Ours
0.81
0.42
90%
91.09%
0.79
85.56%
Table 1: Quantitative comparison with text-to-3D models. We compare our method against TRELLIS [ 55 ] , a general text-to-3D object generation model, and InterFusion [ 9 ] , a text-to-3D interaction model. Our method outperforms all prior arts in both consistency to input text and quality of the generated 3D interactions by a very large margin.
Method
Text Consistency
Generation Quality
GPT ↑
CLIP ↑
Contact ↑
GPT ↑
SANA
0.31
0.42
45.76%
0.24
Ours
0.69
0.40
90%
0.76
Table 2: Quantitative comparison with 2D baselines. We compare our method against pretrained text-to-2D generation SANA [ 56 ] model. Our method achieves higher text consistency and 3D interaction quality. The CLIP score is however not very informative due to its limited sensitivity to fine-grained text [ 23 , 73 ] as we also show in Fig. 3 .
Method
Text Consistency
GPT ↑
CLIP ↑
Contact accuracy ↑
a. Ours w/o data filter
0.20
0.413
0.80
b. Ours w/o retexturing
0.05
0.412
0.85
c. Ours
0.75
0.417
0.90
Table 3: Ablation studies. Our proposed data filtering improves contact accuracy and retexturing improves consistency to text input (GPT score). Combining both achieves the best result.
Method
3 views
2 view
1 view
ProciGen [ 57 ] images
94.7%
89.5%
79.6%
Our generated images
90.0%
86.7%
78.3%
Table 4: Contact accuracy of the lifted 3D given different number of 2D images. Our view-conditioned sampling allows generating three views conditioned on the view description, leading to more stable 3D results.
Figure 5: Animation results. Our fitted SMPL and segmented objects allow reanimation of the generated human object interaction mesh
Figure 6: Advantage of view conditioned sampling. Given same interaction prompt, our method generates three views that all correctly follow the contacts. Yet Hunyuan3D stuggle to reason the compositional shape under occlusion such as the leg occluded by the table in front view. By sampling side views as input to Hunyuan3D, we are able to generate at least one plausible 3D human-object interaction for each text prompt.
Figure 7: Prompts used for calculating GPT scores. We instruct GPT-4V to evaluate the text alignment and quality for generated 3D models (rendered as video) or 2D images.
Figure 8: User study instruction and example questions. We guide participants to evaluate the quality or consistency to input text prompt. We randomly select 20 examples for each aspect and users are asked to choose the best one from three options.
Body Part
SANA
Ours
Left Hand ( ↑ )
40%
100%
Right Hand ( ↑ )
90%
100%
Both Hands ( ↑ )
10%
100%
Left Leg ( ↑ )
11.11%
80%
Right Leg ( ↑ )
30%
70%
Both Legs ( ↑ )
90%
90%
Table 5: Per-part contact accuracy. Our model generates correct contacts for various contact scenarios whereas base model SANA [ 56 ] can follow mainly ‘right hand’ and ‘both legs’ but fails in other body parts.
Figure 9: Controllable interaction generation via text. We can control the contact, object, action or human description in our text prompt with minimum changes to the other parts. This makes it possible to use our model as a data generator.
Figure 10: OOD generalization. Fine tuned only on interaction data with 100 humans and 15 object categories, our method generalizes well to different objects, human descriptions and new actions.
Figure 11: More qualitative comparison. Our method consistently produces high quality results with correct contact and details.
Figure 12: More examples from our annotated data. Our annotation pipeline generates detailed descriptions about human, object, interaction action and contacts.
The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.
Generating photorealistic 3D hand-object interactions (HOIs) from text is important for applications like robotic grasping and AR/VR content creation. In practice, however, achieving both visual fidelity and physical plausibility remains difficult, as mesh extraction from text-generated Gaussians is inherently ill-posed and the resulting meshes are often unreliable for physics-based optimization. We present THOM, a training-free framework that generates physically plausible 3D HOI meshes directly from text prompts, without requiring template object meshes. THOM follows a two-stage pipeline: it first generates hand and object Gaussians guided by text, and then refines their interaction using physics-based optimization. To enable reliable interaction modeling, we introduce a mesh extraction method with an explicit vertex-to-Gaussian mapping, which enables topology-aware regularization. We further improve physical plausibility through contact-aware optimization and vision-language model (VLM)-guided translation refinement. Extensive experiments show that THOM produces high-quality HOIs with strong text alignment, visual realism, and interaction plausibility.
Uyoung Jeong, Yihalem Yimolal Tiruneh, Hyung Jin Chang +2
Understanding and generating 3D objects as compositions of meaningful parts is fundamental to human perception and reasoning. However, most text-to-3D methods overlook the semantic and functional structure of parts. While recent part-aware approaches introduce decomposition, they remain largely geometry-focused, lacking semantic grounding and failing to model how parts align with textual descriptions or their inter-part relations. We propose DreamPartGen, a framework for semantically grounded, part-aware text-to-3D generation. DreamPartGen introduces Duplex Part Latents (DPLs) that jointly model each part's geometry and appearance, and Relational Semantic Latents (RSLs) that capture inter-part dependencies derived from language. A synchronized co-denoising process enforces mutual geometric and semantic consistency, enabling coherent, interpretable, and text-aligned 3D synthesis. Across multiple benchmarks, DreamPartGen delivers state-of-the-art performance in geometric fidelity and text-shape alignment.