Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.
Figures & tables
Figure 1: DreamHMR architecture. Global Body Representation Transfer adapts a video DiT with complete SMPL-X renderings. Local Joint Feature Enhancement samples joint-local features, weights their corrections using predicted evidence scores, and fuses them into body-and-hand part states.
Figure 2: From SMPL-X to rendered video. Fixed template coordinates provide canonical colors; posed normals determine shading. Their product is rasterized and composited with source RGB across frames. Vertex colors persist across poses; a neutral shape and enlarged pixels aid illustration.
Method
3DPW
EMDB-1
RICH
PA ↓
MPJPE ↓
PVE ↓
PA ↓
MPJPE ↓
PVE ↓
PA ↓
MPJPE ↓
PVE ↓
PromptHMR-Vid
39.7
59.8
67.8
39.8
68.5
78.7
37.8
57.8
65.3
GENMO
34.4
54.2
65.5
42.8
65.5
75.3
36.4
60.2
64.9
SMPLest-X
33.8
54.5
64.1
41.1
63.0
81.0
28.9
52.5
58.7
3DB-DINOv3
33.5
55.1
63.2
38.5
61.2
72.9
30.6
54.0
59.8
Hand4Whole++
37.5
57.0
69.6
45.6
70.2
81.4
35.3
62.0
67.2
Table 1: HMR benchmark comparison. Errors in mm; PA denotes PA-MPJPE. Methods share RGB, targets, joint regressors, alignment, and sequence averaging. Blue bold, underlined, and black bold values mark the best, second-best, and best external result per column, respectively.
Figure 3: Real-scene occlusion and hand detail. Wan-1.3B on a RICH eating sequence with foreground railings. The box and arrow mark the hand-and-container region enlarged below. Methods share the crop and scale; GT denotes ground-truth meshes.
Method
Overall
BEDLAM
EgoBody
Body ↓
Hands ↓
Body ↓
Hands ↓
Body ↓
Hands ↓
PromptHMR-Vid
121.2
–
120.6
–
122.8
–
GENMO
118.8
–
116.8
–
124.6
–
SMPLest-X
115.6
61.6
113.9
60.7
120.5
64.3
3DB-DINOv3
115.2
61.2
114.4
59.4
117.6
66.6
Hand4Whole++
118.3
63.1
117.0
62.4
122.2
65.2
Table 2: Body-and-hand reconstruction under occlusion. MPJPE (mm); Overall weights BEDLAM/EgoBody at 75%/25% and static/dynamic masks equally. Blue bold and underlined values mark the best and second-best results per column; – denotes no articulated-hand output.
Supervision target
Full
Occlusion
Body ↓
Hands ↓
PVE ↓
Body ↓
Hands ↓
PVE ↓
RGB
62.3
39.2
73.4
120.0
68.4
140.6
Shading
55.9
32.4
64.2
109.3
64.8
126.7
Canonical coloring
56.1
32.8
66.8
111.6
60.7
129.1
Shading + skeleton
54.7
31.3
62.5
110.5
64.2
127.4
Shading + canonical coloring
53.8
30.7
61.3
106.3
57.8
121.8
Table 3: Rendering supervision. Wan-1.3B errors (mm) over all 81 frames on the ablation holdout. Geometry targets share complete projected coverage. Body/Hands report MPJPE; body PVE uses shared SMPL-X vertices and pelvis alignment. RGB supervision is the reference.
Figure 4: Rendering targets in Table 3 . The same pose with RGB, shading, canonical coloring, a body-skeleton overlay, or blended shading and canonical colors.
Decoder
Local update
Evidence supervision
Full
Occlusion
Body ↓
Hands ↓
Body ↓
Hands ↓
Flat Query
None
No
55.8
32.2
124.4
65.0
Structured Query
None
No
55.1
31.8
119.3
63.9
Structured Query
Ungated
No
54.0
31.0
116.9
60.5
Structured Query
Ungated
Yes
54.1
30.9
113.1
59.6
Structured Query
Evidence-Gated
No
54.3
31.2
110.7
59.0
Table 4: Local Joint Feature Enhancement ablation. Wan-1.3B full-window body/hand MPJPE (mm) on the ablation holdout. Local updates are absent (None), weighted by one (Ungated), or by predicted evidence scores (Evidence-Gated). Evidence supervision uses binary evidence labels (Equation 8 ).
Body MPJPE ↓
Hands MPJPE ↓
Temporal Context
✓
✗
✓
✗
Full observation
56.44
59.70 (+3.26)
29.57
33.15 (+3.58)
Occlusion
108.92
121.63 (+12.71)
50.26
59.53 (+9.27)
Table 5: Temporal context under occlusion. Center-frame body/hand MPJPE (mm) on the ablation holdout with full-budget Wan-1.3B. ✓: original video; ✗: center RGB and mask duplicated over the window, retaining the original tracks and timestamps.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Feature dimension
Native endpoints
Decoder endpoints
Wan-1.3B
1536
21
21
Wan-5B
3072
21
21
LTX-2B
2048
11
21
Cosmos-2B
2048
21
21
Appendix
Table 6: Backbone interfaces for an 81-frame input. Endpoints refer to temporal positions before and after alignment to the shared decoder.
Setting
Geometry adaptation
Parameter recovery
Optimizer steps
10,000
47,500
Optimizer
AdamW
AdamW
Backbone / task learning rate
10−6 / 10−4
10−6 / 10−4
Weight decay
0.01
0.01
Learning-rate schedule
Constant
Constant
Global batch size
24
24
Appendix
Table 7: Training settings. Task modules comprise the trainable conditioning projections and, during recovery, the structured decoder.
Dataset
Person frames
Sampling proportion
BEDLAM
0.82M
18%
BEDLAM2
1.07M
22%
EgoBody
342K
25%
ARCTIC
287K
15%
InterCap
121K
10%
IDEA400
89K
10%
Appendix
Table 8: Training data and sampling proportions. Available annotations supervise the corresponding output fields.
Figure 5: Observation construction. Two RICH frames (top to bottom) under full, static, and dynamic observations. Static regions stay fixed; dynamic regions move horizontally. Pixels outside the region are neutral gray. All panels share the display crop and scale.
Full
Occlusion
Backbone
Initialization
Body ↓
Hands ↓
Body ↓
Hands ↓
DINOv3-H+
Image SSL
53.92
33.40
116.47
65.36
Wan-1.3B
Random
56.44
34.20
126.34
70.13
Wan-1.3B
Video pretrained
52.96
31.21
112.81
61.85
Wan-5B
Random
55.78
33.65
124.22
68.46
Wan-5B
Video pretrained
52.11
30.68
108.87
55.49
Appendix
Table 9: Backbone initialization. Body/hand MPJPE (mm) over all 81 frames on the ablation holdout after 10,000 task-training steps. Paired Wan runs vary DiT initialization; DINOv3-H+ is the image self-supervised reference.
Body
Hands
Observation
Ungated
Gated
Δ
Ungated
Gated
Δ
Static
115.7
103.5
−12.2
57.7
56.1
−1.6
Dynamic
118.1
109.1
−9.0
63.3
59.5
−3.8
Appendix
Table 10: Evidence gating across observation patterns. Wan-1.3B body/hand MPJPE (mm) on the ablation holdout. Ungated uses unit local weights; Gated uses supervised evidence scores. Δ is Gated minus Ungated, computed from unrounded means.
Figure 6: Reconstruction over time. Wan-1.3B on a RICH bench-dip sequence: seven frames at 0.33-second intervals span two seconds. All methods share a fixed crop and scale throughout the sequence. GT denotes ground-truth meshes.
Figure 7: Qualitative comparisons across scenes. DreamHMR with four video backbones, compared with Hand4Whole++ and SMPLest-X. RGB and ground truth are shown for reference.