Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.
Figures & tables
Figure 1: DreamHMR architecture. Global Body Representation Transfer adapts a video DiT with complete SMPL-X renderings. Local Joint Feature Enhancement samples joint-local features, weights their corrections using predicted evidence scores, and fuses them into body-and-hand part states.
Figure 2: From SMPL-X to rendered video. Fixed template coordinates provide canonical colors; posed normals determine shading. Their product is rasterized and composited with source RGB across frames. Vertex colors persist across poses; a neutral shape and enlarged pixels aid illustration.
Method
3DPW
EMDB-1
RICH
PA ↓
MPJPE ↓
PVE ↓
PA ↓
MPJPE ↓
PVE ↓
PA ↓
MPJPE ↓
PVE ↓
PromptHMR-Vid
39.7
59.8
67.8
39.8
68.5
78.7
37.8
57.8
65.3
GENMO
34.4
54.2
65.5
42.8
65.5
75.3
36.4
60.2
64.9
SMPLest-X
33.8
54.5
64.1
41.1
63.0
81.0
28.9
52.5
58.7
3DB-DINOv3
33.5
55.1
63.2
38.5
61.2
72.9
30.6
54.0
59.8
Hand4Whole++
37.5
57.0
69.6
45.6
70.2
81.4
35.3
62.0
67.2
Table 1: HMR benchmark comparison. Errors in mm; PA denotes PA-MPJPE. Methods share RGB, targets, joint regressors, alignment, and sequence averaging. Blue bold, underlined, and black bold values mark the best, second-best, and best external result per column, respectively.
Figure 3: Real-scene occlusion and hand detail. Wan-1.3B on a RICH eating sequence with foreground railings. The box and arrow mark the hand-and-container region enlarged below. Methods share the crop and scale; GT denotes ground-truth meshes.
Method
Overall
BEDLAM
EgoBody
Body ↓
Hands ↓
Body ↓
Hands ↓
Body ↓
Hands ↓
PromptHMR-Vid
121.2
–
120.6
–
122.8
–
GENMO
118.8
–
116.8
–
124.6
–
SMPLest-X
115.6
61.6
113.9
60.7
120.5
64.3
3DB-DINOv3
115.2
61.2
114.4
59.4
117.6
66.6
Hand4Whole++
118.3
63.1
117.0
62.4
122.2
65.2
Table 2: Body-and-hand reconstruction under occlusion. MPJPE (mm); Overall weights BEDLAM/EgoBody at 75%/25% and static/dynamic masks equally. Blue bold and underlined values mark the best and second-best results per column; – denotes no articulated-hand output.
Supervision target
Full
Occlusion
Body ↓
Hands ↓
PVE ↓
Body ↓
Hands ↓
PVE ↓
RGB
62.3
39.2
73.4
120.0
68.4
140.6
Shading
55.9
32.4
64.2
109.3
64.8
126.7
Canonical coloring
56.1
32.8
66.8
111.6
60.7
129.1
Shading + skeleton
54.7
31.3
62.5
110.5
64.2
127.4
Shading + canonical coloring
53.8
30.7
61.3
106.3
57.8
121.8
Table 3: Rendering supervision. Wan-1.3B errors (mm) over all 81 frames on the ablation holdout. Geometry targets share complete projected coverage. Body/Hands report MPJPE; body PVE uses shared SMPL-X vertices and pelvis alignment. RGB supervision is the reference.
Figure 4: Rendering targets in Table 3 . The same pose with RGB, shading, canonical coloring, a body-skeleton overlay, or blended shading and canonical colors.
Decoder
Local update
Evidence supervision
Full
Occlusion
Body ↓
Hands ↓
Body ↓
Hands ↓
Flat Query
None
No
55.8
32.2
124.4
65.0
Structured Query
None
No
55.1
31.8
119.3
63.9
Structured Query
Ungated
No
54.0
31.0
116.9
60.5
Structured Query
Ungated
Yes
54.1
30.9
113.1
59.6
Structured Query
Evidence-Gated
No
54.3
31.2
110.7
59.0
Table 4: Local Joint Feature Enhancement ablation. Wan-1.3B full-window body/hand MPJPE (mm) on the ablation holdout. Local updates are absent (None), weighted by one (Ungated), or by predicted evidence scores (Evidence-Gated). Evidence supervision uses binary evidence labels (Equation 8 ).
Body MPJPE ↓
Hands MPJPE ↓
Temporal Context
✓
✗
✓
✗
Full observation
56.44
59.70 (+3.26)
29.57
33.15 (+3.58)
Occlusion
108.92
121.63 (+12.71)
50.26
59.53 (+9.27)
Table 5: Temporal context under occlusion. Center-frame body/hand MPJPE (mm) on the ablation holdout with full-budget Wan-1.3B. ✓: original video; ✗: center RGB and mask duplicated over the window, retaining the original tracks and timestamps.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Feature dimension
Native endpoints
Decoder endpoints
Wan-1.3B
1536
21
21
Wan-5B
3072
21
21
LTX-2B
2048
11
21
Cosmos-2B
2048
21
21
Appendix
Table 6: Backbone interfaces for an 81-frame input. Endpoints refer to temporal positions before and after alignment to the shared decoder.
Setting
Geometry adaptation
Parameter recovery
Optimizer steps
10,000
47,500
Optimizer
AdamW
AdamW
Backbone / task learning rate
10−6 / 10−4
10−6 / 10−4
Weight decay
0.01
0.01
Learning-rate schedule
Constant
Constant
Global batch size
24
24
Appendix
Table 7: Training settings. Task modules comprise the trainable conditioning projections and, during recovery, the structured decoder.
Dataset
Person frames
Sampling proportion
BEDLAM
0.82M
18%
BEDLAM2
1.07M
22%
EgoBody
342K
25%
ARCTIC
287K
15%
InterCap
121K
10%
IDEA400
89K
10%
Appendix
Table 8: Training data and sampling proportions. Available annotations supervise the corresponding output fields.
Figure 5: Observation construction. Two RICH frames (top to bottom) under full, static, and dynamic observations. Static regions stay fixed; dynamic regions move horizontally. Pixels outside the region are neutral gray. All panels share the display crop and scale.
Full
Occlusion
Backbone
Initialization
Body ↓
Hands ↓
Body ↓
Hands ↓
DINOv3-H+
Image SSL
53.92
33.40
116.47
65.36
Wan-1.3B
Random
56.44
34.20
126.34
70.13
Wan-1.3B
Video pretrained
52.96
31.21
112.81
61.85
Wan-5B
Random
55.78
33.65
124.22
68.46
Wan-5B
Video pretrained
52.11
30.68
108.87
55.49
Appendix
Table 9: Backbone initialization. Body/hand MPJPE (mm) over all 81 frames on the ablation holdout after 10,000 task-training steps. Paired Wan runs vary DiT initialization; DINOv3-H+ is the image self-supervised reference.
Body
Hands
Observation
Ungated
Gated
Δ
Ungated
Gated
Δ
Static
115.7
103.5
−12.2
57.7
56.1
−1.6
Dynamic
118.1
109.1
−9.0
63.3
59.5
−3.8
Appendix
Table 10: Evidence gating across observation patterns. Wan-1.3B body/hand MPJPE (mm) on the ablation holdout. Ungated uses unit local weights; Gated uses supervised evidence scores. Δ is Gated minus Ungated, computed from unrounded means.
Figure 6: Reconstruction over time. Wan-1.3B on a RICH bench-dip sequence: seven frames at 0.33-second intervals span two seconds. All methods share a fixed crop and scale throughout the sequence. GT denotes ground-truth meshes.
Figure 7: Qualitative comparisons across scenes. DreamHMR with four video backbones, compared with Hand4Whole++ and SMPLest-X. RGB and ground truth are shown for reference.
Monocular video human mesh recovery is essential for digital humans, avatar animation, and embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world videos.
Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.
Patrick Kwon, Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817
Human mesh recovery (HMR) models 3D human body from monocular videos, with recent works extending it to world-coordinate human trajectory and motion reconstruction. However, most existing methods remain offline, relying on future frames or global optimization, which limits their applicability in interactive feedback and perception-action loop scenarios such as AR/VR and telepresence. To address this, we propose OnlineHMR, a fully online framework that jointly satisfies four essential criteria of online processing, including system-level causality, faithfulness, temporal consistency, and efficiency. Built upon a two-branch architecture, OnlineHMR enables streaming inference via a causal key-value cache design and a curated sliding-window learning strategy. Meanwhile, a human-centric incremental SLAM provides online world-grounded alignment under physically plausible trajectory correction. Experimental results show that our method achieves performance comparable to existing chunk-based approaches on the standard EMDB benchmark and highly dynamic custom videos, while uniquely supporting online processing. Page and code are available at https://tsukasane.github.io/Video-OnlineHMR/.
Yiwen Zhao, Ce Zheng, Yufu Wang +3
Carnegie Mellon University · University of Pennsylvania