cs.CVApr 7, 2026
SaveHuman Interaction-Aware 3D Reconstruction from a Single Image
Organizations: Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea
Abstract
Reconstructing textured 3D human models from a single image is fundamental for AR/VR and digital human applications. However, existing methods mostly focus on single individuals and thus fail in multi-human scenes, where naive composition of individual reconstructions often leads to artifacts such as unrealistic overlaps, missing geometry in occluded regions, and distorted interactions. These limitations highlight the need for approaches that incorporate group-level context and interaction priors. We introduce a holistic method that explicitly models both group- and instance-level information. To mitigate perspective-induced geometric distortions, we first transform the input into a canonical orthographic space. Our primary component, Human Group-Instance Multi-View Diffusion (HUG-MVD), then generates complete multi-view normals and images by jointly modeling individuals and group context to resolve occlusions and proximity. Subsequently, the Human Group-Instance Geometric Reconstruction (HUG-GR) module optimizes the geometry by leveraging explicit, physics-based interaction priors to enforce physical plausibility and accurately model inter-human contact. Finally, the multi-view images are fused into a high-fidelity texture. Together, these components form our complete framework, HUG3D. Extensive experiments show that HUG3D significantly outperforms both single-human and existing multi-human methods, producing physically plausible, high-fidelity 3D reconstructions of interacting people from a single image. Project page: https://jongheean11.github.io/HUG3D_project
Figures & tables
Figure 1: Core challenges in monocular multi-human 3D reconstruction: (a) geometric complexity and perspective distortion, (b) lack of interaction-aware geometric modeling, and (c) missing texture and geometry in occluded regions. HUG3D addresses all three challenges.
Figure 2: Overview of our HUG3D framework. Given a single perspective image, (1) the Canonical Perspective-to-Orthographic View Transform (Pers2Ortho) converts it into a canonical multi-view orthographic representation to resolve scale ambiguity and enable consistent multi-view reasoning. (2) The Human Group-Instance Multi-View Diffusion (HUG-MVD) model completes occluded geometry and texture while maintaining plausible interactions. (3) The Textured Mesh Reconstruction stage refines the mesh and generates high-fidelity textures with our physics-based Human Group-Instance Geometry Reconstruction (HUG-GR) , which enforces physical consistency via optimization with multi-view normal cues and interaction constraints.
Figure 3: Comparison of results from multi-view diffusion trained on perspective vs. orthographic images.
Figure 4: Qualitative comparison on multi-human 3D reconstruction from a single image. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure 5: Qualitative results on in-the-wild images. Our method demonstrates robust multi-human reconstructions, outperforming baseline approaches and highlighting its practical applicability.
| Method | CD ↓ | P2S ↓ | NC ↑ | F-score ↑ | bbox-IoU ↑ | Norm ↓ | CP ↑ |
|---|---|---|---|---|---|---|---|
| SIFU | 5.644 | 2.284 | 0.754 | 29.244 | 0.778 | 0.028 | 0.089 |
| SiTH | 9.251 | 3.185 | 0.709 | 21.037 | 0.708 | 0.040 | 0.135 |
| PSHuman | 15.579 | 6.088 | 0.617 | 9.749 | 0.659 | 0.069 | 0.027 |
| DeepMultiCap | 13.719 | 2.555 | 0.749 | 18.125 | 0.513 | 0.049 | 0.083 |
| Ours | 3.631 | 1.752 | 0.811 | 41.504 | 0.847 | 0.019 | 0.240 |
Table 1: Quantitative comparison of geometric metrics for multi-human 3D reconstruction. HUG3D achieves the best overall scores in all metrics including CD, P2S, and NC, and also outperforms other baselines in CP, indicating better interaction-aware reconstruction.
| Module | Method | CD ↓ | P2S ↓ | Occ.Norm ↓ | PSNR ↑ | LPIPS ↓ | Occ.PSNR ↑ |
| HUG-MVD | Trained on group-only data | 4.564 | 2.203 | 0.157 | 15.641 | 0.191 | 7.423 |
| Trained on instance-only data | 4.645 | 2.245 | 0.156 | 15.840 | 0.185 | 7.726 | |
| w/o Instance-to-group latent composition | 4.646 | 2.249 | 0.159 | 16.198 | 0.183 | 7.916 | |
| HUG-GR | Instance-only normal supervision | 4.642 | 2.250 | 0.156 | 16.180 | 0.183 | 7.902 |
| Group-only normal supervision | 4.620 | 2.230 | 0.159 | 16.169 | 0.183 | 7.678 | |
| Ours (Full) | 4.316 | 2.122 | 0.153 | 16.454 | 0.179 | 8.082 | |
Table 4: Ablation study of key components in HUG3D. We report geometry (CD, P2S, Norm ) and texture (PSNR, LPIPS) metrics under various configurations.
Figure 6: Ablation study on key components. (a) Pers2Ortho improves projection sharpness.(b) Each HUG-GR loss boosts geometric plausibility.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S7 : Depth edge-aware filtering removes uncertain boundary regions to improve orthographic projection stability.
Figure S8 : Depth-aware filtering selects front-visible points for clean orthographic projections.
Figure S9 : Illustration of masking strategy for occlusion simulation. (a) For multi-subject datasets, SMPL-X instance masks are used to isolate each subject and specify regions to inpaint. (b) For single-subject datasets, we simulate oclusion through silhouette-shaped masks.
Figure S10 : Illustration of the training procedure of HUG-MVD. The model takes occluded RGB images and SMPL-X normal maps from six views and learns to reconstruct complete RGB and normal views via a multi-view diffusion model. SMPL-X guidance is injected via ControlNet, with attention applied across views and modalities for coherent multi-human reconstruction.
Figure S11 : Illustration of the inference procedure of HUG-MVD. (a) Instance-to-group latent composition and (b) partial RGB latent injection for RGB synthesis.
Figure S12 : Illustration of the interpenetration loss and visibility loss. The interpenetration loss penalizes body-part collisions (blue arrows), while the visibility loss enforces alignment between rendered masks and ground-truth visibility (red region).
Figure S13 : View-aware face restoration enhances frontal views using landmark-guided inpainting.
Figure S14 : Illustration of the occlusion-aware blending strategy. Edge-aware confidence masks, computed from depth discontinuities, suppress artifacts near occlusion boundaries, resulting in cleaner silhouettes and improved cross-view consistency.
Figure S15 : Additional qualitative comparison on multi-human 3D reconstruction from a single in-the-wild image. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure S16 : Additional qualitative comparison on multi-human 3D reconstruction from a single image in the MultiHuman dataset. Yellow boxes highlight broken geometry, missing texture, and incorrect inter-human interactions. HUG3D outperforms baselines by correcting perspective distortion, preserving inter-human contact, and hallucinating plausible textures under heavy occlusion.
Figure S17 : Qualitative results on scenes with three or more interacting humans. HUG3D reconstructs plausible multi-person interactions while preserving consistent geometry and contact relationships, demonstrating its scalability to larger human groups.
Figure S18 : Qualitative results demonstrating HUG3D’s generalization capability to novel human types, including stylized 3D characters and children. Despite domain differences, our method produces structurally plausible and semantically consistent outputs.
Figure S19 : Robustness to errors in intermediate predictions. Despite inaccuracies in segmentation, depth estimation, or diffusion outputs, HUG3D maintains plausible multi-human reconstructions with consistent geometry and interaction relationships.
Figure S20 : Robustness to SMPL-X estimation errors. The top two rows demonstrate that our method reduces interpenetration artifacts even with inaccurate SMPL-X estimates. The last row shows a failure case arising from severely corrupted SMPL-X initialization.
Figure S21 : Normal maps rendered from multiple viewpoints of our reconstructed textured 3D mesh, including both training views (gray background) and novel views (white background).
Figure S22 : RGB renderings from multiple viewpoints of our reconstructed textured 3D mesh, including both training views (gray background) and novel views (white background).
| Method | CD | P2S | NC | F-score | bbox-IoU | Norm | CP | PSNR | SSIM | LPIPS | Occ.Norm | Occ.PSNR | Occ.SSIM |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SIFU | 1.8e-09 | 3.3e-08 | 3.9e-14 | 3.9e-10 | 1.6e-07 | 2.0e-39 | 1.1e-05 | 3.8e-27 | 1.7e-38 | 5.1e-37 | 5.7e-10 | 5.9e-10 | 1.9e-11 |
| SiTH | 3.6e-14 | 3.8e-14 | 3.6e-14 | 5.8e-14 | 4.6e-13 | 1.4e-51 | 8.2e-04 | 1.1e-48 | 2.2e-38 | 1.5e-51 | 1.3e-14 | 4.2e-14 | 1.4e-13 |
| PSHuman | 3.6e-14 | 3.6e-14 | 3.6e-14 | 5.2e-14 | 2.7e-13 | 1.4e-51 | 1.2e-08 | 1.4e-51 | 1.7e-51 | 1.4e-51 | 1.9e-18 | 1.4e-18 | 5.2e-17 |
| DeepMultiCap | 7.8e-13 | 1.1e-08 | 2.6e-12 | 1.3e-13 | 1.4e-13 | 1.4e-46 | 5.3e-06 | 3.5e-50 | 1.2e-27 | 5.2e-47 | 5.9e-11 | 1.1e-14 | 1.5e-14 |
Table S14 : Wilcoxon signed-rank test results (p-values) across all evaluation metrics, confirming statistically significant improvements of our method over baselines.
Figure S23 : Example of results from each component of our HUG3D.
Figure S24 : Qualitative comparison of our SMPL-X fitting (RoBUDDI) against BUDDI [ 30 ] . BUDDI exhibits visible interpenetrations between interacting subjects (yellow arrows), whereas our RoBUDDI produces more physically plausible results.
Figure S25 : Comparison of results from multi-view diffusion models trained directly on perspective images vs. models trained on orthographic images.
Figure S26 : Comparison between state-of-the-art diffusion-based inpainting methods and our HUG-MVD framework for multi-view consistent inpainting and generation.
Figure S27 : Ablation on mask types for training HUG-MVD. Our occlusion-simulated masks enhance inpainting compared to freeform masks.
More in this topic
3D Human Reconstruction
3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.
See all 103 papers