Monocular 3D human mesh recovery (HMR) remains unreliable under weak observations caused by long camera-to-person distances, occlusion, and motion blur. In challenging scenarios such as sports, these factors make the problem highly ambiguous, as different 3D poses may yield similar 2D projections. This ambiguity is especially severe for one-stage HMR, which recovers all meshes directly from the full image without person crops or external 2D cues. To handle this ambiguity, we present UT-HMR ( Uncertainty-guided Temporal HMR), to our knowledge the first one-stage HMR framework to capture correlated pose uncertainty across joints. Each person query predicts a Gaussian distribution over the full SMPL pose with a compact low-rank-plus-diagonal covariance, which is converted into joint-wise precision weights for heteroscedastic supervision while keeping inference deterministic. Temporal early fusion further injects neighboring-frame evidence before pose decoding, supplying the visual evidence needed to resolve the ambiguity that the uncertainty captures. With a lightweight ViT-B backbone, UT-HMR achieves state-of-the-art results on VIP-HockeyPose-v2 (ice hockey) and WorldPose (soccer), reducing MPJPE, PA-MPJPE, and PVE by up to 4.7%, 3.8%, and 5.6%, and remains competitive on 3DPW against methods with larger backbones and higher input resolutions.
Figures & tables
Figure 1: Empirical covariance spectrum of per-person 6D pose residuals ( 24 joints ×6=144 dimensions) on the ice hockey dataset. The top two directions explain 39% of the total variance.
Figure 2: Resolving projection ambiguity in one-stage HMR. A distant or occluded player admits multiple plausible 3D poses from a single 2D observation: (a) the target player (red box) is small and partially occluded; (b) single-frame SAT-HMR [ 30 ] predicts an incorrect pose, while (c) UT-HMR reconstructs it correctly. (d) Temporal early fusion gathers multi-frame evidence, and the pose is modeled as a Gaussian: the mean μ yields the deterministic mesh, and the covariance Σ provides joint-wise uncertainty weights during training.
Figure 3: Overall pipeline of UT-HMR. UT-HMR first fuses a short input clip through temporal early fusion and obtains per-person query features with a DETR-style decoder. The pose head predicts the deterministic pose mean μ for final mesh inference, while the low-rank and diagonal heads estimate a Gaussian covariance Σ for uncertainty-aware training. The predicted covariance is supervised by the distribution loss and converted into joint-wise precision weights for the heteroscedastic joint loss.
Method
Res.
Time (ms)
Backbone
Frames
VIP-HockeyPose-v2 [ 36 ]
WorldPose [ 12 ]
3DPW [ 34 ]
MPJPE ↓
PA-MPJPE ↓
PVE ↓
MPJPE ↓
PA-MPJPE ↓
PVE ↓
MPJPE ↓
PA-MPJPE ↓
PVE ↓
ROMP [ 32 ]
512
25.7
HRNet-32
1
58.6
35.5
77.4
74.0
40.7
104.8
76.7
47.3
93.4
BEV [ 33 ]
512
38.1
HRNet-32
1
52.7
31.0
70.4
67.9
38.6
94.0
78.5
46.9
92.3
PSVT (single frame) [ 24 ]
512
–
HRNet-32
1
–
–
–
–
–
–
75.5
45.7
84.9
PSVT [ 24 ]
512
–
HRNet-32
9
–
–
–
–
–
–
73.1
43.5
84.0
Multi-HMR [ 2 ]
896
64.8
ViT-L
1
49.6
30.0
68.0
59.0
36.3
85.7
61.4
41.7
75.9
Table 1: Quantitative comparison on two sports benchmarks (VIP-HockeyPose-v2, WorldPose) and the general-scene 3DPW. Best and second best are marked per column. “Frames” is the number of input frames; for PSVT we list both its single-frame and its best 9-frame setting. Time denotes per-frame inference time in milliseconds. † : 3DPW fine-tuning protocol not specified in [ 26 ] .
Figure 4: Qualitative results on different benchmarks. UT-HMR reconstructions on (a) VIP-HockeyPose-v2 [ 36 ] , (b) WorldPose [ 12 ] , and (c) 3DPW [ 34 ] are overlaid on input frames. Note that referees are excluded from evaluation on VIP-HockeyPose-v2 because ground-truth meshes are not available for them.
Figure 5: Qualitative comparison with SAT-HMR. Meshes predicted by SAT-HMR [ 30 ] and UT-HMR are overlaid on representative challenging crops. Lighter colors indicate higher predicted uncertainty. Please zoom in for details.
Temporal
Uncertainty
MPJPE ↓
PA-MPJPE ↓
PVE ↓
–
–
44.5
26.5
62.9
✓
–
43.5
26.1
61.0
–
✓
43.9
26.2
61.7
✓
✓
42.4
25.5
59.4
Table 2: Ablation of temporal early fusion and pose uncertainty modeling on VIP-HockeyPose-v2. The SAT-HMR baseline (first row) is fine-tuned on the same data as UT-HMR.
Figure 6: Reconstruction error versus predicted joint-uncertainty percentile on VIP-HockeyPose-v2.
Param.
Value
MPJPE ↓
PA-MPJPE ↓
PVE ↓
Range
±0
43.9
26.2
61.7
Range
±1
42.4
25.5
59.4
Range
±2
42.4
25.5
59.7
Rank r
0
43.1
25.6
60.6
Rank r
2
42.4
25.5
59.4
Rank r
4
42.7
25.6
60.0
Table 3: Hyperparameter sensitivity on VIP-HockeyPose-v2. Note that r=0 denotes a diagonal covariance, which still models uncertainty and thus differs from the temporal-only baseline.
Weighting wj
MPJPE ↓
PA-MPJPE ↓
PVE ↓
Uniform
43.5
26.1
61.0
Error-based
43.4
26.0
60.9
High-unc. weighted
42.8
25.6
59.9
Low-unc. weighted (ours)
42.4
25.5
59.4
Table 4: Effect of the joint-weighting source on VIP-HockeyPose-v2. “High-unc.” assigns larger weights to uncertain joints, while “Low-unc.” emphasizes reliable joints.
Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.
Patrick Kwon, Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817
Most advances in human mesh recovery (HMR) have focused on pelvis-centered recovery, overlooking metric 3D localization and detection accuracy in the camera coordinate system - two key factors for real-world applications such as human-robot interaction and social scene understanding. Current evaluation protocols often ignore these aspects, emphasizing per-person, root-centered recovery rather than camera-space perception. As a result, existing approaches rely on fixed camera assumptions or handcrafted post-processing, limiting their robustness and practical deployment. We introduce Multi-HMR 2, a simple yet robust DETR-based framework for Multi-person Camera-centric Human detection, mesh Recovery, and tracking. Multi-HMR 2 predicts a scene-consistent camera together with human meshes, enabling metric 3D localization without ground-truth intrinsics. Moreover, by distilling image-based memory features from SAM2, Multi-HMR 2 extends to tracking, achieving consistent identity association without video supervision. Despite its conceptual simplicity - no handcrafted components, no video input, and no ground-truth cameras - Multi-HMR 2 achieves state-of-the-art pelvis-centered performance while substantially improving detection accuracy and metric 3D localization.
Guénolé Fiche, Philippe Weinzaepfel, Romain Brégier +1
Human mesh recovery (HMR) aims to recover 3D human meshes from images. Most existing HMR benchmarks and methods focus on either multi-person reconstruction from a single view or single-person reconstruction from multiple views, where the number of subjects and the scene scale are relatively limited. Such settings are insufficient for real-world applications with large scenes and severe inter-person occlusions. To address this limitation, we introduce a large-scale synthetic benchmark for multiview multi-person HMR, termed MVMP-HMR. The proposed dataset contains 15 complex scenes with up to 50 camera views and 30 interacting persons, featuring large spatial coverage and severe occlusions, which significantly increases the difficulty of human mesh recovery. Based on this benchmark, we further propose a multiview multi-person whole-body human mesh recovery model, referred to as MVMP-HMR model. The model first fuses multiview features into a scene-level 3D feature volume, and then leverages pelvis joints predicted by a 3D pose estimation network to extract person-specific queries from the 3D feature volume. These human queries are cross-attended with the 3D feature volume and integrated to decode each person's 3D mesh. Moreover, we introduce two novel losses--the orientation loss and the 3D joint density loss--to alleviate orientation and pose ambiguities under severe occlusions. Experiments demonstrate that existing state-of-the-art HMR methods struggle on the proposed MVMP-HMR benchmark, while our method consistently outperforms prior SOTAs in large-scale scenes with severe occlusions.
Qi Zhang, Tao Yu, Jiechao He +2
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China