Human Interaction-Aware 3D Reconstruction from a Single Image
Authors: Gwanghyun Kim, Junghun James Kim, Suh Yoon Jeon, Jason Park, Se Young Chun
Organizations: Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Reconstructing textured 3D human models from a single image is fundamental for AR/VR and digital human applications. However, existing methods mostly focus on single individuals and thus fail in multi-human scenes, where naive composition of individual reconstructions often leads to artifacts such as unrealistic overlaps, missing geometry in occluded regions, and distorted interactions. These limitations highlight the need for approaches that incorporate group-level context and interaction priors. We introduce a holistic method that explicitly models both group- and instance-level information. To mitigate perspective-induced geometric distortions, we first transform the input into a canonical orthographic space. Our primary component, Human Group-Instance Multi-View Diffusion (HUG-MVD), then generates complete multi-view normals and images by jointly modeling individuals and group context to resolve occlusions and proximity. Subsequently, the Human Group-Instance Geometric Reconstruction (HUG-GR) module optimizes the geometry by leveraging explicit, physics-based interaction priors to enforce physical plausibility and accurately model inter-human contact. Finally, the multi-view images are fused into a high-fidelity texture. Together, these components form our complete framework, HUG3D. Extensive experiments show that HUG3D significantly outperforms both single-human and existing multi-human methods, producing physically plausible, high-fidelity 3D reconstructions of interacting people from a single image. Project page: https://jongheean11.github.io/HUG3D_project
Figures & tables
Figure 1: Core challenges in monocular multi-human 3D reconstruction: (a) geometric complexity and perspective distortion, (b) lack of interaction-aware geometric modeling, and (c) missing texture and geometry in occluded regions. HUG3D addresses all three challenges.
Figure 2: Overview of our HUG3D framework. Given a single perspective image, (1) the Canonical Perspective-to-Orthographic View Transform (Pers2Ortho) converts it into a canonical multi-view orthographic representation to resolve scale ambiguity and enable consistent multi-view reasoning. (2) The Human Group-Instance Multi-View Diffusion (HUG-MVD) model completes occluded geometry and texture while maintaining plausible interactions. (3) The Textured Mesh Reconstruction stage refines the mesh and generates high-fidelity textures with our physics-based Human Group-Instance Geometry Reconstruction (HUG-GR) , which enforces physical consistency via optimization with multi-view normal cues and interaction constraints.
Figure 3: Comparison of results from multi-view diffusion trained on perspective vs. orthographic images.
Figure 4: Qualitative comparison on multi-human 3D reconstruction from a single image. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure 5: Qualitative results on in-the-wild images. Our method demonstrates robust multi-human reconstructions, outperforming baseline approaches and highlighting its practical applicability.
Method
CD ↓
P2S ↓
NC ↑
F-score ↑
bbox-IoU ↑
Norm L2 ↓
CP ↑
SIFU
5.644
2.284
0.754
29.244
0.778
0.028
0.089
SiTH
9.251
3.185
0.709
21.037
0.708
0.040
0.135
PSHuman
15.579
6.088
0.617
9.749
0.659
0.069
0.027
DeepMultiCap
13.719
2.555
0.749
18.125
0.513
0.049
0.083
Ours
3.631
1.752
0.811
41.504
0.847
0.019
0.240
Table 1: Quantitative comparison of geometric metrics for multi-human 3D reconstruction. HUG3D achieves the best overall scores in all metrics including CD, P2S, and NC, and also outperforms other baselines in CP, indicating better interaction-aware reconstruction.
Table 7
Module
Method
CD ↓
P2S ↓
Occ.Norm L2 ↓
PSNR ↑
LPIPS ↓
Occ.PSNR ↑
HUG-MVD
Trained on group-only data
4.564
2.203
0.157
15.641
0.191
7.423
Trained on instance-only data
4.645
2.245
0.156
15.840
0.185
7.726
w/o Instance-to-group latent composition
4.646
2.249
0.159
16.198
0.183
7.916
HUG-GR
Instance-only normal supervision
4.642
2.250
0.156
16.180
0.183
7.902
Group-only normal supervision
4.620
2.230
0.159
16.169
0.183
7.678
Ours (Full)
4.316
2.122
0.153
16.454
0.179
8.082
Table 4: Ablation study of key components in HUG3D. We report geometry (CD, P2S, Norm L2 ) and texture (PSNR, LPIPS) metrics under various configurations.
Figure 6: Ablation study on key components. (a) Pers2Ortho improves projection sharpness.(b) Each HUG-GR loss boosts geometric plausibility.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S7 : Depth edge-aware filtering removes uncertain boundary regions to improve orthographic projection stability.
Figure S9 : Illustration of masking strategy for occlusion simulation. (a) For multi-subject datasets, SMPL-X instance masks are used to isolate each subject and specify regions to inpaint. (b) For single-subject datasets, we simulate oclusion through silhouette-shaped masks.
Figure S10 : Illustration of the training procedure of HUG-MVD. The model takes occluded RGB images and SMPL-X normal maps from six views and learns to reconstruct complete RGB and normal views via a multi-view diffusion model. SMPL-X guidance is injected via ControlNet, with attention applied across views and modalities for coherent multi-human reconstruction.
Figure S11 : Illustration of the inference procedure of HUG-MVD. (a) Instance-to-group latent composition and (b) partial RGB latent injection for RGB synthesis.
Figure S12 : Illustration of the interpenetration loss and visibility loss. The interpenetration loss penalizes body-part collisions (blue arrows), while the visibility loss enforces alignment between rendered masks and ground-truth visibility (red region).
Figure S13 : View-aware face restoration enhances frontal views using landmark-guided inpainting.
Figure S14 : Illustration of the occlusion-aware blending strategy. Edge-aware confidence masks, computed from depth discontinuities, suppress artifacts near occlusion boundaries, resulting in cleaner silhouettes and improved cross-view consistency.
Figure S15 : Additional qualitative comparison on multi-human 3D reconstruction from a single in-the-wild image. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure S16 : Additional qualitative comparison on multi-human 3D reconstruction from a single image in the MultiHuman dataset. Yellow boxes highlight broken geometry, missing texture, and incorrect inter-human interactions. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure S17 : Qualitative results on scenes with three or more interacting humans. HUG3D reconstructs plausible multi-person interactions while preserving consistent geometry and contact relationships, demonstrating its scalability to larger human groups.
Figure S18 : Qualitative results demonstrating HUG3D’s generalization capability to novel human types, including stylized 3D characters and children. Despite domain differences, our method produces structurally plausible and semantically consistent outputs.
Figure S19 : Robustness to errors in intermediate predictions. Despite inaccuracies in segmentation, depth estimation, or diffusion outputs, HUG3D maintains plausible multi-human reconstructions with consistent geometry and interaction relationships.
Figure S20 : Robustness to SMPL-X estimation errors. The top two rows demonstrate that our method reduces interpenetration artifacts even with inaccurate SMPL-X estimates. The last row shows a failure case arising from severely corrupted SMPL-X initialization.
Figure S21 : Normal maps rendered from multiple viewpoints of our reconstructed textured 3D mesh, including both training views (gray background) and novel views (white background).
Figure S22 : RGB renderings from multiple viewpoints of our reconstructed textured 3D mesh, including both training views (gray background) and novel views (white background).
Method
CD
P2S
NC
F-score
bbox-IoU
Norm L2
CP
PSNR
SSIM
LPIPS
Occ.Norm L2
Occ.PSNR
Occ.SSIM
SIFU
1.8e-09
3.3e-08
3.9e-14
3.9e-10
1.6e-07
2.0e-39
1.1e-05
3.8e-27
1.7e-38
5.1e-37
5.7e-10
5.9e-10
1.9e-11
SiTH
3.6e-14
3.8e-14
3.6e-14
5.8e-14
4.6e-13
1.4e-51
8.2e-04
1.1e-48
2.2e-38
1.5e-51
1.3e-14
4.2e-14
1.4e-13
PSHuman
3.6e-14
3.6e-14
3.6e-14
5.2e-14
2.7e-13
1.4e-51
1.2e-08
1.4e-51
1.7e-51
1.4e-51
1.9e-18
1.4e-18
5.2e-17
DeepMultiCap
7.8e-13
1.1e-08
2.6e-12
1.3e-13
1.4e-13
1.4e-46
5.3e-06
3.5e-50
1.2e-27
5.2e-47
5.9e-11
1.1e-14
1.5e-14
Appendix
Table S14 : Wilcoxon signed-rank test results (p-values) across all evaluation metrics, confirming statistically significant improvements of our method over baselines.
Figure S23 : Example of results from each component of our HUG3D.
Figure S24 : Qualitative comparison of our SMPL-X fitting (RoBUDDI) against BUDDI [ 30 ] . BUDDI exhibits visible interpenetrations between interacting subjects (yellow arrows), whereas our RoBUDDI produces more physically plausible results.
Figure S25 : Comparison of results from multi-view diffusion models trained directly on perspective images vs. models trained on orthographic images.
Figure S26 : Comparison between state-of-the-art diffusion-based inpainting methods and our HUG-MVD framework for multi-view consistent inpainting and generation.
Figure S27 : Ablation on mask types for training HUG-MVD. Our occlusion-simulated masks enhance inpainting compared to freeform masks.
Multi-view human reconstruction has been extensively studied under simplified settings, yet robust and efficient multi-person reconstruction in unconstrained environments remains challenging. Existing bottom-up methods often rely on accurate camera calibration and explicit cross-view matching, and therefore struggle with severe occlusions and ambiguities. We propose a new top-down paradigm that maintains a unified, instance-centric human-aware 3D space, enabling simultaneous camera calibration, cross-view association, and human reconstruction via cross-modal contrastive learning. Observations from multiple views are lifted and fused into this shared 3D space, where geometric structure, visual appearance, and human-centric semantic cues are jointly encoded at the instance level. We further introduce a spatial contrastive learning strategy that aligns 3D features corresponding to the same human instance across different views and modalities while separating different instances. This enables correspondence reasoning, semantic aggregation, and instance discrimination to be performed natively in 3D, improving cross-view consistency and robustness under severe occlusions. Finally, structured human body models are recovered in a feed-forward manner by regressing SMPL parameters from instance-level 3D human tokens. Extensive experiments demonstrate robust, accurate, and efficient multi-view human reconstruction in challenging real-world scenarios.
Yuanwang Yang, Buzhen Huang, Zongxuan Ren +2
College of Intelligence and Computing, Tianjin University, 135 Yaguan Road, Tianjin, 300350, Tianjin, China.
Estimation of Human-Object Interactions in 3D (3D HOI) is a fundamental problem in 3D computer vision with applications in AR/VR, robotics, and embodied AI. However, reconstructing these interactions in 3D remains challenging due to depth ambiguities, occlusions, and object shape variability. Existing approaches are primarily concerned with reprojection and contact constraints, fitting parametric human models and object templates to 2D images. In this paper, we explore a different avenue. We present MILO, a framework that leverages the visual capabilities of Large Reconstruction Models (LRMs) to recover detailed 3D human-object interactions from a single image. Our key observation is that LRMs provide a powerful geometric scaffold that preserves relative human-object arrangement and proximity cues. This significantly simplifies the reconstruction procedure, reframing the problem as interpreting the LRM mesh: we segment it into human and object components, fit a parametric body model to the human part, and optionally align an object template to the object part (if such a template is available). MILO achieves strong reconstruction accuracy and outperforms existing baselines across multiple benchmarks and interaction scenarios. Our code is available at https://ac5113.github.io/MILO.
Single-view 3D human reconstruction has garnered significant attention in recent years. Despite numerous advancements, prior research has concentrated on reconstructing 3D models from clear, close-up images of individual subjects, often yielding subpar results in the more prevalent multi-person scenarios. Reconstructing 3D human crowd models is a highly intricate task, laden with challenges such as: 1) extensive occlusions, 2) low clarity, and 3) numerous and various appearances. To address this task, we propose CrowdGaussian, a unified framework that directly reconstructs multi-person 3D Gaussian Splatting (3DGS) representations from single-image inputs. To handle occlusions, we devise a self-supervised adaptation pipeline that enables the pretrained large human model to reconstruct complete 3D humans with plausible geometry and appearance from heavily occluded inputs. Furthermore, we introduce Self-Calibrated Learning (SCL). This training strategy enables single-step diffusion models to adaptively refine coarse renderings to optimal quality by blending identity-preserving samples with clean/corrupted image pairs. The outputs can be distilled back to enhance the quality of multi-person 3DGS representations. Extensive experiments demonstrate that CrowdGaussian generates photorealistic, geometrically coherent reconstructions of multi-person scenes.
Yizheng Song, Yiyu Zhuang, Qipeng Xu +5
Nanjing University, China · State Key Laboratory of Novel Software Technology, China · Fudan University, China