cs.CVMay 25, 2022

MUG: Multi-human Graph Network for 3D Mesh Reconstruction from 2D Pose

Authors: Chenyan Wu, Yandong Li, Xianfeng Tang, Zhe Huang, James Wang

Organizations: The Pennsylvania State University · Google · Amazon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reconstructing multi-human body meshes from a single monocular image is a challenging problem, requiring both individual body meshes and relative 3D positions to produce a coherent scene representation. We introduce MUG (Multi-hUman Graph network), which uses a single graph neural network to construct coherent multi-human meshes from only 2D poses. Our approach demonstrates that pose-based methods can jointly perform depth reasoning and multi-human mesh generation. Existing image-based methods often rely on lab-collected datasets with accurate 3D labels, introducing domain gaps when applied to in-the-wild or art images due to differences in appearance and context. In contrast, MUG leverages the geometric consistency of 2D poses across diverse datasets to mitigate such discrepancies. MUG operates in three phases. First, it constructs a novel heterogeneous graph from multi-human 2D poses, connecting nodes across people and within individuals to capture inter-human interactions and body geometry, including skeletal and mesh structures. Second, a dual-branch graph neural network predicts inter-human depth relations and root-joint-relative mesh coordinates. Finally, these outputs are combined to construct complete multi-human 3D meshes. Despite using only 2D poses and a streamlined architecture, MUG outperforms existing multi-human mesh estimation methods across datasets from diverse domains, including RH, MuPoTS-3D, and 3DPW. Qualitative results show that MUG can handle art images, through-wall scenarios, and poor lighting conditions when paired with advanced 2D pose networks. Both qualitative and quantitative evaluations highlight its generalization ability in open-world scenarios.

Figures & tables

Figure 1: Previous multi-human mesh networks commonly adopt a detection-style pipeline using images as input. In our pipeline, the proposed Multi-hUman Graph network (MUG) takes the 2D pose as input. The 2D pose can be obtained by 2D human pose estimation networks.
Figure 2: Illustration of the framework. 2D multi-human pose is the input. To explicitly model the inter- and intra-human relation, we build a heterogeneous graph containing joint nodes (blue circles) and mesh nodes (green circles). On top of the graph, we construct node features using 2D pose. The graph network adopts a dual-branch structure. Joint nodes with joint features go through the upper branch (blue blocks), meanwhile mesh nodes with mesh features go through the lower branch (green blocks). One GCN block (orange block) connects these two branches. Finally, the upper branch outputs root-camera depth and the lower branch outputs root-relative mesh coordinates. By combining the outcomes, we can recover the full multi-human meshes. The upper branch also outputs joint coordinates, which plays a supporting role in learning.
Figure 3: Structure of the graph. The graph consists of two kinds of nodes, joint nodes VJoint\mathcal{V}^{\text{Joint}} (blue circles) and mesh nodes VMesh\mathcal{V}^{\text{Mesh}} . Each human corresponds to one subgraph of the whole graph. Within one human, joint nodes are connected to get edge EJoint - Joint\mathcal{E}^{{\text{Joint - Joint}}} (green lines) according to body skeleton structure; following body mesh topology, mesh nodes are connected to get edge EMesh - Mesh\mathcal{E}^{{\text{Mesh - Mesh}}} (orange lines); each mesh node connects to its two nearest joint nodes, generating edge EJoint - Mesh\mathcal{E}^{{\text{Joint - Mesh}}} (red lines). To represent inter-human relation, we connect joint nodes across humans to get edge EInter\mathcal{E}^{\text{Inter}} (purple lines). (b-c) shows the graph structure for image (a). For better display, we do not draw mesh nodes and only draw part of edge EJoint - Mesh\mathcal{E}^{{\text{Joint - Mesh}}} .
MethodMulti-humanMPJPEPA MPJPE
GraphCMR [ 20 ]-88.056.8
Pose2Mesh [ 4 ]-64.946.3
Jiang [ 13 ]✓-52.7
BMP [ 56 ]✓-51.3
MUG✓61.948.5
MUG ∗✓50.338.5
Table 1: Performance comparison for Human3.6M . The evaluation metrics are MPJPE and PA MPJPE (mean per joint position error without and with Procrustes alignment). The unit is mm. ∗ means using ground truth 2D pose as input.
MethodHagglingMafiaUltim.PizzaMean
Zanfir [ 53 ]140.0165.9150.7156.0153.4
Zanfir [ 54 ]141.1152.3145.0162.5150.3
Jiang [ 13 ]129.6133.5153.0156.7143.2
BMP [ 56 ]120.4132.7140.9147.5135.4
ROMP [ 40 ]111.8129.0148.5149.1134.6
MUG113.7120.8137.5139.2127.8
Table 2: Performance comparison for Panoptic . The evaluation metric is MPJPE.
Table 6
Figure 4: Illustration of inter-human edges with ϵ=0+,0,100,200,300\epsilon=0^{+},0,100,200,300 pixels from left to right, where + means no root-to-root edges. Green points represent body joints. Orange and green lines represent root-to-root edges and ϵ\epsilon -controlled edges separately.
Table 8
Figure 5: Qualitative results. We show the multi-human mesh reconstruction of MUG on MuPoTS and 3DPW dataset. The second to the forth column are the front, top, side view respectively.
Figure 1: Detailed network structure. The number inside each graph convolution layer represents the feature dimension of the layer output.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: More qualitative results on MuPoTS. The second to the forth column are the front, top, side view respectively.
Figure 3: More qualitative results on 3DPW. The second to the forth column are the front, top, side view respectively.
More in this topic

3D Human Reconstruction

3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.

See all 109 papers

Explore similar work

CardsList
May 6, 2026

InterMesh: Explicit Interaction-Aware End-to-End Multi-Person Human Mesh Recovery

Humans constantly interact with their surroundings. Existing end-to-end multi-person human mesh recovery methods, typically based on the DETR framework, capture inter-human relationships through self-attention across all human queries. However, these approaches model interactions only implicitly and lack explicit reasoning about how humans interact with objects and with each other. In this paper, we propose InterMesh, a simple yet effective framework that explicitly incorporates human-environment interaction information into human mesh recovery pipeline. By leveraging a human-object interaction detector, InterMesh enriches query representations with structured interaction semantics, enabling more accurate pose and shape estimation. We design lightweight modules, Contextual Interaction Encoder and Interaction-Guided Refiner, to integrate these features into existing HMR architectures with minimal overhead. We validate our approach through extensive experiments on 3DPW, MuPoTS, CMU Panoptic, Hi4D, and CHI3D datasets, demonstrating remarkable improvements over state-of-the-art methods. Notably, InterMesh reduces MPJPE by 9.9% on CMU Panoptic and 8.2% on Hi4D, highlighting its effectiveness in scenarios with complex human-object and inter-human interactions. Code and models are released at https://github.com/Kelly510/InterMesh.

Kaili Zheng, Kaiwen Wang, Xun Zhu +2

Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · College of AI, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology

3D ReconstructionHuman Mesh Recovery3D Human Pose Estimation3D Mesh Reconstruction3D Human Reconstruction
AskPDF
May 25, 2026

MuNet: A Mutualistic Network for Joint 3D Human Mesh Recovery and 3D Clothed Human Reconstruction from Single Images

3D human mesh recovery and 3D clothed human reconstruction are inherently related, yet they have long been studied in isolation, thereby overlooking the potential gains of joint optimization. To overcome this limitation, we propose to address these two tasks within a unified framework, which allows their mutual dependencies to be effectively exploited. Building on this idea, we propose MuNet, a mutualistic network for joint 3D human mesh recovery and 3D clothed human reconstruction from single images. First, we adopt 2-manifold graphs as a unified representation for all 3D models, enabling consistent modeling across 3D human mesh recovery and clothed human reconstruction. Second, we design an end-to-end graph convolutional network that progressively deforms an initial graph into a 3D human mesh and refines it into a detailed 3D clothed human model. Third, we introduce a mutualistic mechanism that allows reciprocal interaction between the two tasks {during training}, where 3D human mesh recovery provides guidance for 3D clothed human reconstruction, and reconstruction feedback refines the 3D human mesh recovery. We extensively evaluate MuNet on six benchmark datasets for 3D human mesh recovery and 3D clothed human reconstruction, including Human3.6M, 3DPW, MPI-INF-3DHP, THuman2.0, CAPE, and RenderPeople. Experimental results demonstrate that MuNet achieves state-of-the-art performance on both tasks across all datasets. The code of MuNet is released for research purposes at https://github.com/starVisionTeam/MuNet.

Yunqi Gao, Leyuan Liu, Yuhan Li +2

National Engineering Research Center for E-Learning Central China Normal University, Wuhan, 430079, Hubei, China · National Engineering Research Center of Educational Big Data Central China Normal University, Wuhan, 430079, Hubei, China · School of Electronic Information and Communications Huazhong University of Science and Technology, Wuhan, 430073, Hubei, China +1

3D ReconstructionHuman Mesh Recovery3D Mesh Reconstruction3D Human ReconstructionSingle-View 3D Reconstruction3D Garment Reconstruction
AskPDF
Apr 7, 2026

Human Interaction-Aware 3D Reconstruction from a Single Image

Reconstructing textured 3D human models from a single image is fundamental for AR/VR and digital human applications. However, existing methods mostly focus on single individuals and thus fail in multi-human scenes, where naive composition of individual reconstructions often leads to artifacts such as unrealistic overlaps, missing geometry in occluded regions, and distorted interactions. These limitations highlight the need for approaches that incorporate group-level context and interaction priors. We introduce a holistic method that explicitly models both group- and instance-level information. To mitigate perspective-induced geometric distortions, we first transform the input into a canonical orthographic space. Our primary component, Human Group-Instance Multi-View Diffusion (HUG-MVD), then generates complete multi-view normals and images by jointly modeling individuals and group context to resolve occlusions and proximity. Subsequently, the Human Group-Instance Geometric Reconstruction (HUG-GR) module optimizes the geometry by leveraging explicit, physics-based interaction priors to enforce physical plausibility and accurately model inter-human contact. Finally, the multi-view images are fused into a high-fidelity texture. Together, these components form our complete framework, HUG3D. Extensive experiments show that HUG3D significantly outperforms both single-human and existing multi-human methods, producing physically plausible, high-fidelity 3D reconstructions of interacting people from a single image. Project page: https://jongheean11.github.io/HUG3D_project

Gwanghyun Kim, Junghun James Kim, Suh Yoon Jeon +2

Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

3D Human ReconstructionSingle-View 3D ReconstructionDiffusion Models
AskPDF