Monocular 3D human mesh recovery (HMR) remains unreliable under weak observations caused by long camera-to-person distances, occlusion, and motion blur. In challenging scenarios such as sports, these factors make the problem highly ambiguous, as different 3D poses may yield similar 2D projections. This ambiguity is especially severe for one-stage HMR, which recovers all meshes directly from the full image without person crops or external 2D cues. To handle this ambiguity, we present UT-HMR ( Uncertainty-guided Temporal HMR), to our knowledge the first one-stage HMR framework to capture correlated pose uncertainty across joints. Each person query predicts a Gaussian distribution over the full SMPL pose with a compact low-rank-plus-diagonal covariance, which is converted into joint-wise precision weights for heteroscedastic supervision while keeping inference deterministic. Temporal early fusion further injects neighboring-frame evidence before pose decoding, supplying the visual evidence needed to resolve the ambiguity that the uncertainty captures. With a lightweight ViT-B backbone, UT-HMR achieves state-of-the-art results on VIP-HockeyPose-v2 (ice hockey) and WorldPose (soccer), reducing MPJPE, PA-MPJPE, and PVE by up to 4.7%, 3.8%, and 5.6%, and remains competitive on 3DPW against methods with larger backbones and higher input resolutions.
Figures & tables
Figure 1: Empirical covariance spectrum of per-person 6D pose residuals ( 24 joints ×6=144 dimensions) on the ice hockey dataset. The top two directions explain 39% of the total variance.
Figure 2: Resolving projection ambiguity in one-stage HMR. A distant or occluded player admits multiple plausible 3D poses from a single 2D observation: (a) the target player (red box) is small and partially occluded; (b) single-frame SAT-HMR [ 30 ] predicts an incorrect pose, while (c) UT-HMR reconstructs it correctly. (d) Temporal early fusion gathers multi-frame evidence, and the pose is modeled as a Gaussian: the mean μ yields the deterministic mesh, and the covariance Σ provides joint-wise uncertainty weights during training.
Figure 3: Overall pipeline of UT-HMR. UT-HMR first fuses a short input clip through temporal early fusion and obtains per-person query features with a DETR-style decoder. The pose head predicts the deterministic pose mean μ for final mesh inference, while the low-rank and diagonal heads estimate a Gaussian covariance Σ for uncertainty-aware training. The predicted covariance is supervised by the distribution loss and converted into joint-wise precision weights for the heteroscedastic joint loss.
Method
Res.
Time (ms)
Backbone
Frames
VIP-HockeyPose-v2 [ 36 ]
WorldPose [ 12 ]
3DPW [ 34 ]
MPJPE ↓
PA-MPJPE ↓
PVE ↓
MPJPE ↓
PA-MPJPE ↓
PVE ↓
MPJPE ↓
PA-MPJPE ↓
PVE ↓
ROMP [ 32 ]
512
25.7
HRNet-32
1
58.6
35.5
77.4
74.0
40.7
104.8
76.7
47.3
93.4
BEV [ 33 ]
512
38.1
HRNet-32
1
52.7
31.0
70.4
67.9
38.6
94.0
78.5
46.9
92.3
PSVT (single frame) [ 24 ]
512
–
HRNet-32
1
–
–
–
–
–
–
75.5
45.7
84.9
PSVT [ 24 ]
512
–
HRNet-32
9
–
–
–
–
–
–
73.1
43.5
84.0
Multi-HMR [ 2 ]
896
64.8
ViT-L
1
49.6
30.0
68.0
59.0
36.3
85.7
61.4
41.7
75.9
Table 1: Quantitative comparison on two sports benchmarks (VIP-HockeyPose-v2, WorldPose) and the general-scene 3DPW. Best and second best are marked per column. “Frames” is the number of input frames; for PSVT we list both its single-frame and its best 9-frame setting. Time denotes per-frame inference time in milliseconds. † : 3DPW fine-tuning protocol not specified in [ 26 ] .
Figure 4: Qualitative results on different benchmarks. UT-HMR reconstructions on (a) VIP-HockeyPose-v2 [ 36 ] , (b) WorldPose [ 12 ] , and (c) 3DPW [ 34 ] are overlaid on input frames. Note that referees are excluded from evaluation on VIP-HockeyPose-v2 because ground-truth meshes are not available for them.
Figure 5: Qualitative comparison with SAT-HMR. Meshes predicted by SAT-HMR [ 30 ] and UT-HMR are overlaid on representative challenging crops. Lighter colors indicate higher predicted uncertainty. Please zoom in for details.
Temporal
Uncertainty
MPJPE ↓
PA-MPJPE ↓
PVE ↓
–
–
44.5
26.5
62.9
✓
–
43.5
26.1
61.0
–
✓
43.9
26.2
61.7
✓
✓
42.4
25.5
59.4
Table 2: Ablation of temporal early fusion and pose uncertainty modeling on VIP-HockeyPose-v2. The SAT-HMR baseline (first row) is fine-tuned on the same data as UT-HMR.
Figure 6: Reconstruction error versus predicted joint-uncertainty percentile on VIP-HockeyPose-v2.
Param.
Value
MPJPE ↓
PA-MPJPE ↓
PVE ↓
Range
±0
43.9
26.2
61.7
Range
±1
42.4
25.5
59.4
Range
±2
42.4
25.5
59.7
Rank r
0
43.1
25.6
60.6
Rank r
2
42.4
25.5
59.4
Rank r
4
42.7
25.6
60.0
Table 3: Hyperparameter sensitivity on VIP-HockeyPose-v2. Note that r=0 denotes a diagonal covariance, which still models uncertainty and thus differs from the temporal-only baseline.
Weighting wj
MPJPE ↓
PA-MPJPE ↓
PVE ↓
Uniform
43.5
26.1
61.0
Error-based
43.4
26.0
60.9
High-unc. weighted
42.8
25.6
59.9
Low-unc. weighted (ours)
42.4
25.5
59.4
Table 4: Effect of the joint-weighting source on VIP-HockeyPose-v2. “High-unc.” assigns larger weights to uncertain joints, while “Low-unc.” emphasizes reliable joints.
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China