MUG: Multi-human Graph Network for 3D Mesh Reconstruction from 2D Pose

Chenyan Wu; Yandong Li; Xianfeng Tang; Zhe Huang; James Wang

cs.CVMay 25, 2022

MUG: Multi-human Graph Network for 3D Mesh Reconstruction from 2D Pose

Authors: Chenyan Wu, Yandong Li, Xianfeng Tang, Zhe Huang, James Wang

Organizations: The Pennsylvania State University · Google · Amazon

Abstract

Reconstructing multi-human body meshes from a single monocular image is a challenging problem, requiring both individual body meshes and relative 3D positions to produce a coherent scene representation. We introduce MUG (Multi-hUman Graph network), which uses a single graph neural network to construct coherent multi-human meshes from only 2D poses. Our approach demonstrates that pose-based methods can jointly perform depth reasoning and multi-human mesh generation. Existing image-based methods often rely on lab-collected datasets with accurate 3D labels, introducing domain gaps when applied to in-the-wild or art images due to differences in appearance and context. In contrast, MUG leverages the geometric consistency of 2D poses across diverse datasets to mitigate such discrepancies. MUG operates in three phases. First, it constructs a novel heterogeneous graph from multi-human 2D poses, connecting nodes across people and within individuals to capture inter-human interactions and body geometry, including skeletal and mesh structures. Second, a dual-branch graph neural network predicts inter-human depth relations and root-joint-relative mesh coordinates. Finally, these outputs are combined to construct complete multi-human 3D meshes. Despite using only 2D poses and a streamlined architecture, MUG outperforms existing multi-human mesh estimation methods across datasets from diverse domains, including RH, MuPoTS-3D, and 3DPW. Qualitative results show that MUG can handle art images, through-wall scenarios, and poor lighting conditions when paired with advanced 2D pose networks. Both qualitative and quantitative evaluations highlight its generalization ability in open-world scenarios.

Figures & tables

Figure 1: Previous multi-human mesh networks commonly adopt a detection-style pipeline using images as input. In our pipeline, the proposed Multi-hUman Graph network (MUG) takes the 2D pose as input. The 2D pose can be obtained by 2D human pose estimation networks.

Figure 2: Illustration of the framework. 2D multi-human pose is the input. To explicitly model the inter- and intra-human relation, we build a heterogeneous graph containing joint nodes (blue circles) and mesh nodes (green circles). On top of the graph, we construct node features using 2D pose. The graph network adopts a dual-branch structure. Joint nodes with joint features go through the upper branch (blue blocks), meanwhile mesh nodes with mesh features go through the lower branch (green blocks). One GCN block (orange block) connects these two branches. Finally, the upper branch outputs root-camera depth and the lower branch outputs root-relative mesh coordinates. By combining the outcomes, we can recover the full multi-human meshes. The upper branch also outputs joint coordinates, which plays a supporting role in learning.

Figure 3: Structure of the graph. The graph consists of two kinds of nodes, joint nodes $\mathcal{V}^{\text{Joint}}$ (blue circles) and mesh nodes $\mathcal{V}^{\text{Mesh}}$ . Each human corresponds to one subgraph of the whole graph. Within one human, joint nodes are connected to get edge $\mathcal{E}^{{\text{Joint - Joint}}}$ (green lines) according to body skeleton structure; following body mesh topology, mesh nodes are connected to get edge $\mathcal{E}^{{\text{Mesh - Mesh}}}$ (orange lines); each mesh node connects to its two nearest joint nodes, generating edge $\mathcal{E}^{{\text{Joint - Mesh}}}$ (red lines). To represent inter-human relation, we connect joint nodes across humans to get edge $\mathcal{E}^{\text{Inter}}$ (purple lines). (b-c) shows the graph structure for image (a). For better display, we do not draw mesh nodes and only draw part of edge $\mathcal{E}^{{\text{Joint - Mesh}}}$ .

Method	Multi-human	MPJPE	PA MPJPE
GraphCMR [ 20 ]	-	88.0	56.8
Pose2Mesh [ 4 ]	-	64.9	46.3
Jiang [ 13 ]	✓	-	52.7
BMP [ 56 ]	✓	-	51.3
MUG	✓	61.9	48.5
MUG ∗	✓	50.3	38.5

Table 1: Performance comparison for Human3.6M . The evaluation metrics are MPJPE and PA MPJPE (mean per joint position error without and with Procrustes alignment). The unit is mm. ∗ means using ground truth 2D pose as input.

Method	Haggling	Mafia	Ultim.	Pizza	Mean
Zanfir [ 53 ]	140.0	165.9	150.7	156.0	153.4
Zanfir [ 54 ]	141.1	152.3	145.0	162.5	150.3
Jiang [ 13 ]	129.6	133.5	153.0	156.7	143.2
BMP [ 56 ]	120.4	132.7	140.9	147.5	135.4
ROMP [ 40 ]	111.8	129.0	148.5	149.1	134.6
MUG	113.7	120.8	137.5	139.2	127.8

Table 2: Performance comparison for Panoptic . The evaluation metric is MPJPE.

Table 6

Figure 4: Illustration of inter-human edges with $\epsilon=0^{+},0,100,200,300$ pixels from left to right, where + means no root-to-root edges. Green points represent body joints. Orange and green lines represent root-to-root edges and $\epsilon$ -controlled edges separately.

Table 8

Figure 5: Qualitative results. We show the multi-human mesh reconstruction of MUG on MuPoTS and 3DPW dataset. The second to the forth column are the front, top, side view respectively.

Figure 1: Detailed network structure. The number inside each graph convolution layer represents the feature dimension of the layer output.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: More qualitative results on MuPoTS. The second to the forth column are the front, top, side view respectively.

Figure 3: More qualitative results on 3DPW. The second to the forth column are the front, top, side view respectively.

3D Human Reconstruction

3D Human Reconstruction recovers human shape, pose, appearance, and motion from images, video, or other sensor data.

See all 109 papers

InterMesh: Explicit Interaction-Aware End-to-End Multi-Person Human Mesh Recovery
May 6, 2026Kaili Zheng, Kaiwen Wang, Xun Zhu +23D Reconstruction Human Mesh Recovery
Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · College of AI, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology
MuNet: A Mutualistic Network for Joint 3D Human Mesh Recovery and 3D Clothed Human Reconstruction from Single Images
May 25, 2026Yunqi Gao, Leyuan Liu, Yuhan Li +23D Reconstruction Human Mesh Recovery
National Engineering Research Center for E-Learning Central China Normal University, Wuhan, 430079, Hubei, China · National Engineering Research Center of Educational Big Data Central China Normal University, Wuhan, 430079, Hubei, China · School of Electronic Information and Communications Huazhong University of Science and Technology, Wuhan, 430073, Hubei, China +1
Human Interaction-Aware 3D Reconstruction from a Single Image
Apr 7, 2026Gwanghyun Kim, Junghun James Kim, Suh Yoon Jeon +23D Human Reconstruction Single-View 3D Reconstruction
Dept. of Electrical and Computer Engineering, Seoul National University, Republic of Korea · IPAI, Seoul National University, Republic of Korea · INMC, Seoul National University, Republic of Korea

May 6, 2026J/K move · Enter open · S save

InterMesh: Explicit Interaction-Aware End-to-End Multi-Person Human Mesh Recovery

Kaili Zheng, Kaiwen Wang, Xun Zhu, Chenyi Guo+1

Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · College of AI, Tsinghua University, Beijing 100084, China · Beijing National Research Center for Information Science and Technology

3D Reconstruction Human Mesh Recovery 3D Human Pose Estimation 3D Mesh Reconstruction 3D Human Reconstruction

Open paper AskPDF arXiv

Humans constantly interact with their surroundings. Existing end-to-end multi-person human mesh recovery methods, typically based on the DETR framework, capture inter-human relationships through self-attention across all human queries. However, these approaches model interactions only implicitly and lack explicit reasoning about how humans interact with objects and with each other. In this paper, we propose InterMesh, a simple yet effective framework that explicitly incorporates human-environment interaction information into human mesh recovery pipeline. By leveraging a human-object interaction detector, InterMesh enriches query representations with structured interaction semantics, enabling more accurate pose and shape estimation. We design lightweight modules, Contextual Interaction Encoder and Interaction-Guided Refiner, to integrate these features into existing HMR architectures with minimal overhead. We validate our approach through extensive experiments on 3DPW, MuPoTS, CMU Panoptic, Hi4D, and CHI3D datasets, demonstrating remarkable improvements over state-of-the-art methods. Notably, InterMesh reduces MPJPE by 9.9% on CMU Panoptic and 8.2% on Hi4D, highlighting its effectiveness in scenarios with complex human-object and inter-human interactions. Code and models are released at https://github.com/Kelly510/InterMesh.

Abstract

3D Human Reconstruction

Explore similar work

InterMesh: Explicit Interaction-Aware End-to-End Multi-Person Human Mesh Recovery

MuNet: A Mutualistic Network for Joint 3D Human Mesh Recovery and 3D Clothed Human Reconstruction from Single Images

Human Interaction-Aware 3D Reconstruction from a Single Image