MuPPet: Multi-person 2D-to-3D Pose Lifting

Thomas Markhorst; Zhi-Yi Lin; Jouh Yeong Chew; Jan van Gemert; Xucong Zhang

cs.CVApr 8, 2026

MuPPet: Multi-person 2D-to-3D Pose Lifting

Authors: Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew, Jan van Gemert, Xucong Zhang

Organizations: Delft University of Technology · Honda Research Institute Japan

Abstract

Multi-person social interactions are inherently built on coherence and relationships among all individuals within the group, making multi-person localization and body pose estimation essential to understanding these social dynamics. One promising approach is 2D-to-3D pose lifting which provides a 3D human pose consisting of rich spatial details by building on the significant advances in 2D pose estimation. However, the existing 2D-to-3D pose lifting methods often neglect inter-person relationships or cannot handle varying group sizes, limiting their effectiveness in multi-person settings. We propose MuPPet, a novel multi-person 2D-to-3D pose lifting framework that explicitly models inter-person correlations. To leverage these inter-person dependencies, our approach introduces Person Encoding to structure individual representations, Permutation Augmentation to enhance training diversity, and Dynamic Multi-Person Attention to adaptively model correlations between individuals. Extensive experiments on group interaction datasets demonstrate MuPPet significantly outperforms state-of-the-art single- and multi-person 2D-to-3D pose lifting methods, and improves robustness in occlusion scenarios. Our findings highlight the importance of modeling inter-person correlations, paving the way for accurate and socially-aware 3D pose estimation. Our code is available at: https://github.com/Thomas-Markhorst/MuPPet

Figures & tables

Figure 1: We exploit social inter-person correlations for 3D pose estimation to help infer occluded poses and spatial relations between individuals in a group interaction. Our method takes a sequence of detected 2D body poses to predict the sequence of 3D poses, as shown in the figure.

Figure 2: Overview of our MuPPet pipeline. Given a sequence of detected 2D human pose joints from multiple persons $X$ , we use the diffusion process $N$ times to denoise the 3D random poses $Y_{N}$ to the output absolute 3D pose $Y$ . Inside the denoiser, the spatial transformer and person encoding are applied to capture intra- and inter-person relationships, and a temporal transformer is used to capture the joint relationship across frames.

Figure 3: Example of permutation learning for a five person scene. We illustrate the people with color coding on the left and the permutation in the feature space on the right. From the original feature $\hat{X}\in\mathbb{R}^{T\times(P\cdot J)\times C}$ , where $C$ is the number of channels, we show a possible superset permutation of $P$ persons with all $J$ joints, and subset permutation of $(P-1)$ persons with all $J$ joints. During training, $n_{\text{sub}}$ and $n_{\text{sup}}$ permutations are used.

Method	MPJPE ${}_{\text{rel}}$ $\downarrow$	MPJPE ${}_{\text{abs}}$ $\downarrow$	MPJPE ${}_{\text{root}}$ $\downarrow$
D3DP [ 49 ]	58.2	-	-
D3DP ${}_{\text{absolute}}$ [ 49 ]	59.1	144.4	135.9
MuPPet	55.3	119.5	108.9

Table 1: Comparison of MuPPet with single-person lifting method D3DP on the Haggling dataset, in absolute (MPJPE

{}_{\text{abs}}

), relative (MPJPE

{}_{\text{rel}}

), and absolute root MPJPE

{}_{\text{root}}

pose estimation in mm. Our MuPPet achieves better performance than the SOTA single-person pose lifting method D3DP.

Method	Haggling	Mafia	Ultimatum	Mean
MubyNet [ 66 ]	72.4	78.8	66.8	72.7
SMAP [ 73 ]	63.1	60.3	56.6	60.0
VirtualPose [ 54 ]	54.1	61.6	54.6	56.8
POTR-3D [ 42 ]	60.0	57.0	55.5	57.5
MuPPet	56.1	54.3	57.1	55.8

Table 2: Comparing our method with SOTA Multi-Person pose lifting (VirtualPose and POTR-3D) and direct estimation methods (MubyNet and SMAP) on the on the CMU Panoptic dataset. All numbers are averaged joint error in mm. We only list the relative joint error

\text{MPJPE}_{\text{rel}}

since the absolute joint error

\text{MPJPE}_{\text{abs}}

is not reported by previous pose lifting methods. Our MuPPet achieves the best performance in the multi-person pose estimation setting.

Figure 4: We show the performance of different levels of occlusion on the Haggling dataset, in comparison with the adapted D3DP singleperson model. The X-axis is the maximum number of occluded joints, and the Y-axis is the joint error in mm. Our method performs better in relative joint error MPJPE

{}_{\text{rel}}

and signficantly better in absolute joint error MPJPE

{}_{\text{abs}}

over all levels of occlusion.

Multi	PE	Sup.Perm.	Sub.Perm.	MPJPE ${}_{\text{rel}}\downarrow$	MPJPE ${}_{\text{abs}}\downarrow$
-	-	-	-	59.1	144.4
✓	-	-	-	68.4	142.1
✓	✓	-	-	69.1	131.5
✓	✓	✓	-	60.0	126.9
✓	✓	✓	✓	55.3	119.5

Table 3: Ablation showing that each module contributes to the performance. Dynamic multi-person attention (Multi), Person Encoding (PE), super set permutation (Sup.Perm, and subset permutation learning (Sub.Perm). We show both relative joint error MPJPE rel and absolute joint error MPJPE abs .

# Supersets	# Subsets	MPJPE ${}_{\text{rel}}\downarrow$	MPJPE ${}_{\text{abs}}\downarrow$
12	0	58.4	151.5
6	6	56.5	150.0
2	10	56.4	143.0
1	7	56.3	148.6
2	10	56.4	143.0
3	13	55.8	146.8

Table 4: Ablation study on the permutation numbers in superset

n_{\text{sup}}

and subset

n_{\text{sub}}

. Upper: we fix the total number of permutations to be 12 and alter the distribution of

n_{\text{sup}}

and

n_{\text{sub}}

. Lower: we change the total number of permutations while keep the rough ratio of

n_{\text{sup}}

and

n_{\text{sub}}

. All numbers are joint errors in mm. We observe that adding subset permutations and increasing total permutations increases performance significantly.

Figure 5: Qualitative results on an in-the-wild setting predicted by MuPPet. We show one frame from the video input and corresponding three views of the predicted 3D pose from multiple persons in the scene. Our MuPPet demonstrates effective performance in absolute 3D joint prediction, even on highly occluded persons.

Explore similar work

SnapPose3D: Diffusion-Based Single-Frame 2D-to-3D Lifting of Human Poses
Apr 29, 2026Alessandro Simoni, Riccardo Catalini, Davide Di Nucci +63D Human Pose Estimation Diffusion Models
University of Modena and Reggio Emilia (UniMoRe), Italy · Toyota Motor Europe (TME), Brussels, Belgium · Woven by Toyota, Japan
2D Pre-Training for 3D Pose Estimation
Apr 19, 2026Liyao Jiang, Ruichen Chen, Keith G. Mills3D Human Pose Estimation Human Pose Estimation
University of Alberta
Sound-based Multi-Person 3D Pose Estimation
Sep 4, 2026Yusuke Oumi, Yuto Shibata, Go Irie +3Human Pose Estimation 3D Human Pose Estimation
Keio University, Yokohama, Kanagawa 223-8522, Japan · Tokyo University of Science, Katsushika, Tokyo 125-8585, Japan · NTT, Inc., Keihanna Science City, Kyoto 619-0237, Japan

Apr 29, 2026J/K move · Enter open · S save

SnapPose3D: Diffusion-Based Single-Frame 2D-to-3D Lifting of Human Poses

Alessandro Simoni, Riccardo Catalini, Davide Di Nucci, Guido Borghi+5

University of Modena and Reggio Emilia (UniMoRe), Italy · Toyota Motor Europe (TME), Brussels, Belgium · Woven by Toyota, Japan

3D Human Pose Estimation Diffusion Models

Open paper AskPDF arXiv

Depth ambiguity and joint uncertainty are the two main obstacles in obtaining accurate human pose predictions by 2D-to-3D lifting methods proposed in the literature. In particular, these issues are caused by 2D joint locations that can be mapped to multiple 3D positions, inducing multiple possible final poses. Following these considerations, we propose leveraging diffusion-based models generation capability to predict multiple hypotheses and aggregate them in a final accurate pose. Therefore, we introduce SnapPose3D, a pose-lifting framework trained deterministically to denoise 3D poses conditioned on both visual context and 2D pose features. SnapPose3D adopts a probabilistic approach during inference, generating multiple hypotheses through random sampling from a unit Gaussian distribution. Unlike most previous methods that address pose ambiguity by processing temporal sequences, SnapPose3D uses single frames as input, avoiding tracking and limiting computational cost, data acquisition complexity, and the need for online, real-time applications. We extensively evaluate SnapPose3D on well-known benchmarks for the 3D human pose estimation task showing its ability to generate and aggregate accurate hypotheses that lead to state-of-the-art results.

Abstract

Explore similar work

SnapPose3D: Diffusion-Based Single-Frame 2D-to-3D Lifting of Human Poses

2D Pre-Training for 3D Pose Estimation

Sound-based Multi-Person 3D Pose Estimation