cs.CVMar 12, 2026
SaveHoi3DGen: Generating High-Quality Human-Object-Interactions in 3D
Organizations: University of Tübingen · Technische Universit4t N4rnberg · Tübingen AI Center · Max Planck Institute for Informatics
Abstract
Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.
Figures & tables
Figure 2: HOI3D framework overview. Top: We first leverage the existing multimodal foundation model InternVL [ 8 ] to perform decomposed annotation of human, object, and human-object-interaction of samples from the ProciGen [ 57 ] dataset. We then use LLaMa [ 16 ] to create a final detailed caption for the sample. Bottom: We leverage our data consisting of high-quality and diverse human-object-interactions to fine-tune an existing text-to-image model. Subsequently, we establish a pipeline to reconstruct high-fidelity textured 3D meshes. The output of our final text-to-3D inference pipeline consists of segmented meshes for the human and object, as well as an animatable SMPL model.
Figure 3: Analysis of the CLIP score. While our model clearly generates images that follow input interaction descriptions more precisely than SANA [ 56 ] , the CLIP score indicates the opposite, rendering it unusable as a metric for our task.
Figure 4: Qualitative comparison for text to 3D generation . InterFusion [ 9 ] is based on Score Distillation Sampling and hence is slow and produces low-quality 3D due to the well-known Janus problem. TRELLIS [ 55 ] is a learning based native 3D generation method, hence it can produce better 3D but is not interaction-aware. Our method faithfully follows the text prompts, especially the detailed body contact specifications. Our contacts are highlighted with spheres coloured based on contacting body parts.
| Method | Text Consistency | 3D Quality | ||||
|---|---|---|---|---|---|---|
| GPT | CLIP | Contact | User | GPT | User | |
| TRELLIS | 0.04 | 0.32 | N/A | 3.44% | 0.21 | 10.16% |
| InterFusion | 0.15 | 0.35 | N/A | 5.47% | 0.00 | 3.28% |
| Ours | 0.81 | 0.42 | 90% | 91.09% | 0.79 | 85.56% |
Table 1: Quantitative comparison with text-to-3D models. We compare our method against TRELLIS [ 55 ] , a general text-to-3D object generation model, and InterFusion [ 9 ] , a text-to-3D interaction model. Our method outperforms all prior arts in both consistency to input text and quality of the generated 3D interactions by a very large margin.
| Method | Text Consistency | Generation Quality | ||
|---|---|---|---|---|
| GPT | CLIP | Contact | GPT | |
| SANA | 0.31 | 0.42 | 45.76% | 0.24 |
| Ours | 0.69 | 0.40 | 90% | 0.76 |
Table 2: Quantitative comparison with 2D baselines. We compare our method against pretrained text-to-2D generation SANA [ 56 ] model. Our method achieves higher text consistency and 3D interaction quality. The CLIP score is however not very informative due to its limited sensitivity to fine-grained text [ 23 , 73 ] as we also show in Fig. 3 .
| Method | Text Consistency | ||
|---|---|---|---|
| GPT | CLIP | Contact accuracy | |
| a. Ours w/o data filter | 0.20 | 0.413 | 0.80 |
| b. Ours w/o retexturing | 0.05 | 0.412 | 0.85 |
| c. Ours | 0.75 | 0.417 | 0.90 |
Table 3: Ablation studies. Our proposed data filtering improves contact accuracy and retexturing improves consistency to text input (GPT score). Combining both achieves the best result.
| Method | 3 views | 2 view | 1 view |
|---|---|---|---|
| ProciGen [ 57 ] images | 94.7% | 89.5% | 79.6% |
| Our generated images | 90.0% | 86.7% | 78.3% |
Table 4: Contact accuracy of the lifted 3D given different number of 2D images. Our view-conditioned sampling allows generating three views conditioned on the view description, leading to more stable 3D results.
Figure 5: Animation results. Our fitted SMPL and segmented objects allow reanimation of the generated human object interaction mesh
Figure 6: Advantage of view conditioned sampling. Given same interaction prompt, our method generates three views that all correctly follow the contacts. Yet Hunyuan3D stuggle to reason the compositional shape under occlusion such as the leg occluded by the table in front view. By sampling side views as input to Hunyuan3D, we are able to generate at least one plausible 3D human-object interaction for each text prompt.
Figure 7: Prompts used for calculating GPT scores. We instruct GPT-4V to evaluate the text alignment and quality for generated 3D models (rendered as video) or 2D images.
Figure 8: User study instruction and example questions. We guide participants to evaluate the quality or consistency to input text prompt. We randomly select 20 examples for each aspect and users are asked to choose the best one from three options.
| Body Part | SANA | Ours |
|---|---|---|
| Left Hand ( ) | 40% | 100% |
| Right Hand ( ) | 90% | 100% |
| Both Hands ( ) | 10% | 100% |
| Left Leg ( ) | 11.11% | 80% |
| Right Leg ( ) | 30% | 70% |
| Both Legs ( ) | 90% | 90% |
Table 5: Per-part contact accuracy. Our model generates correct contacts for various contact scenarios whereas base model SANA [ 56 ] can follow mainly ‘right hand’ and ‘both legs’ but fails in other body parts.
Figure 9: Controllable interaction generation via text. We can control the contact, object, action or human description in our text prompt with minimum changes to the other parts. This makes it possible to use our model as a data generator.
Figure 10: OOD generalization. Fine tuned only on interaction data with 100 humans and 15 object categories, our method generalizes well to different objects, human descriptions and new actions.
Figure 11: More qualitative comparison. Our method consistently produces high quality results with correct contact and details.
Figure 12: More examples from our annotated data. Our annotation pipeline generates detailed descriptions about human, object, interaction action and contacts.