Multi-view human mesh recovery (HMR) is broadly deployed in diverse domains where high accuracy and strong generalization are essential. Existing approaches can be broadly grouped into geometry-based and learning-based methods. However, geometry-based methods (e.g., triangulation) rely on cumbersome camera calibration, while learning-based approaches often generalize poorly to unseen camera configurations due to the lack of multi-view training data, limiting their performance in real-world scenarios. To enable calibration-free reconstruction that generalizes to arbitrary camera setups, we propose a training-free framework that leverages pretrained single-view HMR models as strong priors, eliminating the need for multi-view training data. Our method first constructs a robust and consistent multi-view initialization from single-view predictions, and then refines it via test-time optimization guided by multi-view consistency and anatomical constraints. Extensive experiments demonstrate state-of-the-art performance on standard benchmarks, surpassing multi-view models trained with explicit multi-view supervision.
Figures & tables
Figure 1: The model architectures of (a) HMR2.0, (b) CameraHMR, (c) TokenHMR, and (d) a summary of these generic single-view transformer-based HMR models. The optimizable components can be selected from three representative groups: tokens (pink/orange), model parameters (blue), or explicit SMPL parameters (green).
Figure 2: Test-time adaptation (TTA) framework for multi-view human mesh recovery. The pipeline is composed of two stages: (a) aggregating single-view SMPL estimates into a set of SMPL parameters initialization termed virtual view ; (b) Test-time adaptation refines single-view optimizable token and virtual view SMPL parameters using 2D Loss, multi-view consistency Loss, and regularization loss.
Method
MPJPE ↓
PA-MPJPE ↓
HMR2.0(a) [ 9 ]
44.8
33.6
HMR2.0(b) [ 9 ]
50.0
32.4
Single Camera
HMR2.0+scoreHMR-a [ 47 ]
47.9
28.4
HMR2.0+scoreHMR-b [ 47 ]
44.7
29.0
PostureHMR [ 46 ]
44.5
31.0
Shape Aware [ 25 ]
79.9
45.1
Table 1: Comparison on the Human3.6M dataset. Methods are grouped into single-view, multi-view calibration-free, and multi-view calibration-requiring approaches. Bold and underline indicate the best and second-best results respectively.
Method
MPJPE ↓
PCK ↑
AUC ↑
PaFF [ 16 ]
48.4
98.6
67.3
U-HMR [ 23 ]
39.7
74.0
99.4
HeatFormer [ 30 ]
39.8
99.5
72.8
Ours (TokenHMR)
45.2
99.9
79.6
Ours (HMR2.0)
40.3
99.6
80.7
Ours (CameraHMR)
39.0
99.9
83.8
Table 2: Comparison results on MPI-INF-3DHP dataset with multi-view methods. Bold indicates the best result, Underline indicates the second-best result.
Method
MPJPE ↓
PCK ↑
AUC ↑
HeatFormer
45.74
99.15
68.93
Ours(CameraHMR)
43.71
99.44
80.62
Table 3: Comparison results on MPI-INF-3DHP dataset with cross camera settings.
Components
Camerahmr
HMR2.0
TokenHMR
Virtual View
TTA
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
✗
✓
27.8
22.7
35.1
24.1
43.9
27.5
✓
✗
44.2
29.9
44.4
24.0
37.1
25.7
✓
✓
26.9
20.6
31.6
21.5
32.6
23.2
Table 4: Ablation study on the effects of Virtual View design and TTA.
Camerahmr
HMR2.0
TokenHMR
Strategy
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
No Virtual View
27.8
22.7
35.1
24.1
43.9
27.5
T-pose
27.2
21.2
32.5
22.4
33.8
23.6
Averaged
27.1
20.7
32.1
21.7
33.5
23.3
Weighted
26.9
20.6
31.6
21.5
32.6
23.2
Table 5: Ablation study on virtual view initialization strategy.
Camerahmr
HMR2.0
TokenHMR
Guidance
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
anatomical (ana)
39.9
29.4
35.2
24.6
33.5
24.5
keypoint (kp)
39.6
28.5
35.0
24.0
33.2
23.7
kp+ana
36.8
27.2
32.3
22.5
33.0
23.6
kp+ana+consistency (con)
27.3
20.9
32.2
21.6
32.9
23.4
kp+ana+con+regularizer
26.9
20.6
31.6
21.5
32.6
23.2
Table 6: Ablation study on TTA loss components.
Camerahmr
HMR2.0
TokenHMR
Component
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
SMPL Parameters
58.2
32.2
53.0
31.9
48.4
31.1
Decoder
30.8
22.8
37.2
23.3
36.9
27.5
Learnable Tokens
29.7
20.9
36.0
23.0
35.2
25.7
Learned Tokens
26.9
20.6
31.6
21.5
32.6
23.2
Table 7: Ablation study on single-view backbone optimizable component.
Camerahmr
HMR2.0
TokenHMR
Component
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
Pose
31.6
21.9
35.2
24.6
34.6
23.3
Pose+Orientation
29.0
21.4
33.4
22.0
34.4
23.3
Pose+Orientation+Shape
26.9
20.6
31.6
21.5
32.6
23.2
Table 8: Ablation study on Virtual-view Optimizable Component.
Figure 3: Mesh alignment on the Human3.6M [ 14 ] (left) and MPI-INF-3DHP [ 31 ] (right). From left to right: input image, mesh at Step 0, and mesh at step 200.
Figure 4: TTA 2D clues alignment. Green markers denote the pseudo ground-truth from 2D detector, while the red markers represent the model’s predictions.
Hyper-parameter
Camerahmr
HMR2.0
TokenHMR
η
6e−2
1e−1
6e−2
λkp
3e−1
3e−1
3e−1
λana
3e−1
3e−1
3e−1
λreg_orient
3e1
1e1
1e1
λreg_pose
1e−1
1e−1
1e−1
λreg_betas
2e−2
2e−2
2e−2
Table 9: Hyper-parameters for each model.
Steps
MPJPE ↓
PA-MPJPE ↓
AITI(s) ↓
0
44.23
29.86
0.03
50
36.54
26.64
0.63
100
30.36
23.26
1.26
150
28.52
22.17
1.88
200
27.56
21.61
2.50
Table 10: Inference time with star-structured consisntecy graph.
# of view
MPJPE ↓
PA-MPJPE ↓
2
31.2
23.0
3
29.5
22.0
4
26.9
20.6
Table 11: Ablation study on the number of views.
learning rate
MPJPE ↓
PA-MPJPE ↓
1e−2
29.8
21.1
3e−2
27.4
21.0
6e−2
26.9
20.6
1e−1
29.5
22.0
Table 12: Ablation study on learning rate.
Camerahmr
HMR2.0
TokenHMR
Steps
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
50
35.6
25.9
35.4
22.0
35.1
24.3
100
29.9
22.3
32.4
21.5
33.5
23.6
150
28.0
21.2
31.6
21.5
32.8
23.3
200
26.9
20.6
31.6
21.5
32.6
23.2
250
27.4
21.0
31.7
21.5
32.7
23.3
Table 13: Ablation study on TTA steps.
CameraHmr
HMR2.0
TokenHMR
2D detector
EPE
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
w/o evaluation data(ours)
5.94
26.9
20.6
31.6
21.5
32.6
23.2
w/ 1k evaluation data
5.05
23.8
18.0
30.1
20.3
31.7
22.2
w/ 2k evaluation data
4.74
21.8
16.1
29.9
20.1
31.5
21.6
w/ whole evaluation data
2.41
19.0
13.2
27.6
18.3
31.0
21.4
ground truth
0
15.8
10.4
26.7
17.5
30.0
20.7
Table 14: Ablation study on quality of 2D detector.
Camerahmr
HMR2.0
TokenHMR
TTA Strategy
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
MPJPE ↓
PA-MPJPE ↓
w/o gradient clip norm
30.5
22.5
37.0
24.6
35.2
24.1
w/ gradient clip norm
26.9
20.6
31.6
21.5
32.6
23.2
Table 15: Ablation study on gradient clip norm during TTA
Figure 5: TTA 2D clues alignment. Green markers denote the pseudo ground-truth from 2D detector, while the red markers represent the model’s predictions.
Figure 6: Qualitative comparison between HeatFormer and our method on the Human3.6M dataset. Pink meshes denote our predictions, while white meshes correspond to HeatFormer (iteration 4).
Figure 7: Mesh recovery and TTA alignment on real-world scenarios using the CVLab–EPFL Basketball Sequence. Red boxes highlight the misalignment in the initial predictions (Step 0).
Human mesh recovery (HMR) aims to recover 3D human meshes from images. Most existing HMR benchmarks and methods focus on either multi-person reconstruction from a single view or single-person reconstruction from multiple views, where the number of subjects and the scene scale are relatively limited. Such settings are insufficient for real-world applications with large scenes and severe inter-person occlusions. To address this limitation, we introduce a large-scale synthetic benchmark for multiview multi-person HMR, termed MVMP-HMR. The proposed dataset contains 15 complex scenes with up to 50 camera views and 30 interacting persons, featuring large spatial coverage and severe occlusions, which significantly increases the difficulty of human mesh recovery. Based on this benchmark, we further propose a multiview multi-person whole-body human mesh recovery model, referred to as MVMP-HMR model. The model first fuses multiview features into a scene-level 3D feature volume, and then leverages pelvis joints predicted by a 3D pose estimation network to extract person-specific queries from the 3D feature volume. These human queries are cross-attended with the 3D feature volume and integrated to decode each person's 3D mesh. Moreover, we introduce two novel losses--the orientation loss and the 3D joint density loss--to alleviate orientation and pose ambiguities under severe occlusions. Experiments demonstrate that existing state-of-the-art HMR methods struggle on the proposed MVMP-HMR benchmark, while our method consistently outperforms prior SOTAs in large-scale scenes with severe occlusions.
Qi Zhang, Tao Yu, Jiechao He +2
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China
Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.
Patrick Kwon, Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817
We introduce MetricHMSR, a novel framework for recovering metric human meshes and 3D scenes from a single monocular image. Existing methods struggle to recover metric scale due to monocular scale ambiguity and weak-perspective camera assumptions. Moreover, their fully coupled feature representations make it difficult to disentangle local pose from global translation, often requiring multi-stage pipelines that introduce accumulated errors. To address these challenges, we propose MetricHMR (Metric Human Mesh Recovery), which incorporates a bounding camera ray map representation to provide explicit metric cues for human reconstruction,together with a Human Mixture-of-Experts (HumanMoE) that dynamically routes image features to specialized experts, enabling the disentangled perception of local human pose and global metric position. Leveraging the recovered metric human as a geometric anchor, we further refine monocular metric depth estimation to achieve more accurate 3D alignment between humans and scenes.Comprehensive experiments demonstrate that our method achieves state-of-the-art performance on both human mesh recovery and metric human-scene reconstruction. Project Page: https://Metaverse-AI-Lab-THU.github.io/MetricHMSR.
Chentao Song, He Zhang, Haolei Yuan +4
Tsinghua University · Beihang University · Beijing Normal University