Joint reconstruction of 3D human and object from a single image is an active research area, with pivotal applications in robotics and digital content creation. Despite recent advances, existing approaches suffer from two fundamental limitations. First, their reconstructions rely heavily on physical contact information, which inherently cannot capture non-contact human-object interactions, such as gazing at or pointing toward an object. Second, the reconstruction process is primarily driven by local geometric proximity, neglecting the human and object appearances that provide global context crucial for understanding holistic interactions. To address these issues, we introduce TeHOR, a framework built upon two core designs. First, beyond contact information, our framework leverages text descriptions of human-object interactions to enforce semantic alignment between the 3D reconstruction and its textual cues, enabling reasoning over a wider spectrum of interactions, including non-contact cases. Second, we incorporate appearance cues of the 3D human and object into the alignment process to capture holistic contextual information, thereby ensuring visually plausible reconstructions. As a result, our framework produces accurate and semantically coherent reconstructions, achieving state-of-the-art performance.
Figures & tables
Figure 2: Limitations of existing reconstruction methods. Previous methods suffer from over-reliance on contact information and neglect the global interaction context, leading to implausible reconstructions. In contrast, TeHOR produces accurate and plausible 3D reconstructions by leveraging holistic and semantic guidance from text descriptions.
Figure 3: Overall pipeline of TeHOR. Given an input image, our framework initially reconstructs a 3D human, a 3D object, and a 2D background. Then, the initially reconstructed 3D human and object are jointly optimized using three core loss functions: reconstruction loss, appearance loss, and contact loss, to ensure accurate and semantically plausible human-object interaction.
Figure 4: Gaussians-to-mesh conversion process.
Figure 5: Optimization results of TeHOR. Our text-guided optimization accurately refines the 3D human and object by utilizing their corresponding text descriptions.
Figure 6: Effectiveness of text descriptions in optimization.
CD human↓
CD object↓
Contact ↑
Collision ↓
Before optim.
5.252
31.268
0.305
0.040
After optim. w/o text prompts
5.028
20.348
0.374
0.052
After optim. (Ours)
4.941
16.701
0.412
0.047
Table 1: Effectiveness of text-guided optimization.
Figure 7: Effectiveness of each loss function in our framework.
Lappr
Lcontact
CD human↓
CD object↓
Contact ↑
Collision ↓
✗
✓
5.191
22.094
0.330
0.049
✓
✗
5.311
19.849
0.374
0.054
LCLIP
✓
5.018
18.504
0.366
0.047
✓ (Ours)
✓
4.941
16.701
0.412
0.047
Table 2: Ablation studies for loss configurations.
CD human↓
CD object↓
Contact ↑
Collision ↓
3D Gaussians → Mesh
5.153
25.162
0.308
0.054
w/o 2D background
5.002
18.196
0.389
0.049
TeHOR (Ours)
4.941
16.701
0.412
0.047
Table 3: Ablation studies for appearance rendering.
Figure 8: Qualitative comparison with state-of-the-art methods. We highlight their representative failure cases with red circles.
Open3DHOI
BEHAVE
Methods
CD human↓
CD object↓
Contact ↑
Collision ↓
CD human↓
CD object↓
Contact ↑
Collision ↓
PHOSA [ 85 ]
5.342
49.180
0.243
0.044
5.758
46.003
0.257
0.010
LEMON [ 82 ] + PICO [ 9 ]
5.948
25.889
0.335
0.078
6.159
22.585
0.082
0.045
InteractVLM [ 12 ]
5.252
24.238
0.392
0.054
5.770
19.197
0.379
0.021
HOI-Gaussian [ 74 ]
5.111
19.363
0.348
0.070
5.748
21.774
0.371
0.019
TeHOR (Ours)
4.941
16.701
0.412
0.047
5.615
17.339
0.412
0.016
Table 4: Quantitative comparison with state-of-the-art methods on Open3DHOI [ 74 ] and BEHAVE [ 5 ] . Bold and underlined values indicate the best and second-best scores, respectively. † denotes the evaluation results directly computed based on the centers of reconstructed 3D Gaussians instead of 3D mesh vertices.
Methods
CD human↓
CD object↓
Collision ↓
PHOSA [ 85 ]
5.401
65.537
0.028
LEMON [ 82 ] + PICO [ 9 ]
5.635
33.073
0.029
InteractVLM [ 12 ]
5.390
46.819
0.011
HOI-Gaussian [ 74 ]
5.244
25.374
0.037
TeHOR (Ours)
4.958
17.546
0.005
Table 5: Quantitative comparison with state-of-the-art methods for non-contact scenarios on Open3DHOI [ 74 ] .
Figure S1 : Process of evaluating text alignment for state-of-the-art reconstruction methods.
Open3DHOI
Methods
CLIPScore ↑
VQAScore ↑
PHOSA [ 85 ]
0.689
0.631
LEMON [ 82 ] + PICO [ 9 ]
0.696
0.642
InteractVLM [ 12 ]
0.694
0.647
HOI-Gaussian [ 74 ]
0.698
0.648
TeHOR (Ours)
0.706
0.652
Table S1 : Quantitative evaluation of appearance-text alignment.
Seen object categories
Whole object categories
CD human↓
CD object↓
Collision ↓
CD human↓
CD object↓
Collision ↓
HDM [ 76 ]
4.977
16.349
0.040
6.084
30.422
0.038
TeHOR (Ours)
2.511
14.905
0.035
2.582
17.938
0.047
Table S2 : Quantitative comparison with HDM [ 76 ] on Open3DHOI [ 74 ] .
Contact estimation
3D reconstruction
Contact estimation methods
Contact p↑
Contact r↑
Contact f1↑
CD human↓
CD object↓
Contact ↑
Collision ↓
w/o contact
–
–
–
5.311
19.849
0.374
0.054
Pcontact
0.282
0.342
0.309
4.941
16.701
0.412
0.047
DECO [ 70 ]
0.200
0.264
0.228
5.115
17.229
0.353
0.051
LEMON [ 82 ]
0.426
0.225
0.295
5.084
17.060
0.389
0.050
InteractVLM [ 12 ]
0.422
0.458
0.439
4.988
16.009
0.408
0.052
Table S3 : Impact of contact estimation accuracy on TeHOR’s reconstruction, evaluated on Open3DHOI [ 74 ] .
Figure S2 : Enhancement of Gaussian via optimization process.
Open3DHOI
CD human↓
CD object↓
Contact ↑
Collision ↓
Before conversion
5.020
16.987
0.394
0.052
After conversion (Ours)
4.941
16.701
0.412
0.047
Table S4 : Impact of Gaussians-to-mesh conversion
Figure S5 : Text captioning results on Open3DHOI [ 74 ] . Our text captioning produces accurate and rich text descriptions for a wide range of interaction scenarios.
Figure S6 : Failure case of reconstructing local details.
Figure S7 : More qualitative comparison of 3D human and object reconstruction with PHOSA [ 85 ] , InteractVLM [ 12 ] , and HOI-Gaussian [ 74 ] , on Open3DHOI [ 74 ] . We highlight their representative failure cases with red circles.
Figure S8 : More qualitative comparison of 3D human and object reconstruction with PHOSA [ 85 ] , InteractVLM [ 12 ] , and HOI-Gaussian [ 74 ] , on Open3DHOI [ 74 ] . We highlight their representative failure cases with red circles.
Figure S9 : More qualitative comparison of 3D human and object reconstruction with PHOSA [ 85 ] , InteractVLM [ 12 ] , and HOI-Gaussian [ 74 ] , on Open3DHOI [ 74 ] . We highlight their representative failure cases with red circles.
Figure S10 : More qualitative comparison of 3D human and object reconstruction with PHOSA [ 85 ] , InteractVLM [ 12 ] , and HOI-Gaussian [ 74 ] , on Open3DHOI [ 74 ] . We highlight their representative failure cases with red circles.
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
University of Amsterdam, The Netherlands · Max Planck Institute for Intelligent Systems, Tübingen, Germany · Aristotle University of Thessaloniki, Greece