Organizations: Peking University · The Hong Kong University of Science and Technology · Tencent · East China Normal University · Sun Yat-sen University · Zhejiang University
We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular video. Previous learning-based methods typically regress human motion directly from video and train the network with geometric supervision. However, recovering human motion from monocular video is inherently ambiguous in depth, and direct regression tends to collapse toward an averaged solution. Moreover, the recovered motions are not guaranteed to be physically plausible, so physics-based tracking of them often fails. To address these challenges, we formulate video motion capture as a video-conditioned motion generation problem and first pretrain a flow matching model for this task. Given an input video, the pretrained model generates diverse motion candidates, but not all of them are faithful to the video or physically trackable. We therefore post-train the model using Group Relative Policy Optimization (GRPO) with two rewards. A fidelity reward encourages consistency with the input video. A tracking reward favors motions that a physics-based controller can track successfully. Together, these rewards shift the model's output preference, so the post-trained model stays faithful to the input video while producing more physically plausible motion. We further introduce Wild-4K, a large and diverse dataset of about 4K internet videos, for evaluating human motion recovery in the wild. Qualitative and quantitative experiments on Wild-4K show that our method outperforms state-of-the-art methods in overall motion fidelity and achieves a physical tracking success rate of 82.47%, compared with 62.82% for the strongest baseline, GVHMR.
Figures & tables
Figure 1: Physically plausible motion recovery. Given a monocular video (rows 1 and 3), FlowHMR recovers global 3D human motion (pink) that a physics-based controller can successfully track in simulation (blue).
Figure 2: Training data scale and diversity. (a) Synthetic examples featuring occlusions, challenging poses, and partial visibility. (b) Total durations of motion data and paired motion-video data, compared with BEDLAM, the largest prior synthetic dataset. (c) Proportions of sampled camera trajectory types. We use a diverse set of camera trajectories to simulate the camera motion found in in-the-wild internet videos.
Figure 3: Overview of the FlowHMR training pipeline. To recover faithful and physically plausible global human motion from monocular video, FlowHMR is trained in two stages. Stage 1 performs generative pretraining, in which a video-conditioned flow-matching model is trained on paired video–motion data with a flow-matching loss and a forward kinematics loss. Stage 2 performs physics-based post-training, in which the model, initialized from the Stage 1 checkpoint, is further fine-tuned with GRPO using two rewards: a fidelity reward that measures agreement with the ground-truth motion, and a physical tracking reward that measures how accurately a frozen, pretrained controller tracks the generated motion in a physics simulator.
Table 1: Quantitative Results on Wild-4K. Left: our method outperforms all baselines in blinded human evaluation, with more wins than losses in every comparison. Bars show win/tie/loss percentages over valid judgments; evaluated videos differ across comparisons. Right: simulation Success Rate (SR, %) under PHC+ and physical errors on 4,182 clips. Our method achieves the highest SR, the lowest penetration (Pen.) and floating (Float) errors, and the second-lowest skating (Skate) error.
Figure 4: Qualitative comparison of reconstruction and simulation tracking. The top row shows the input video, followed by GVHMR ( Shen et al., 2024 ) , GENMO ( Li et al., 2025a ) , DuoMo ( Wang et al., 2026b ) , and FlowHMR. Each method panel pairs the reconstructed output (pink, left) with its tracking result in simulation (blue, right). Our method enables successful simulation tracking of this sequence, whereas all compared baselines fail.
Method
Reconstruction
Motion quality
PHC+ execution
WA ↓
W ↓
PA ↓
RTE ↓
Jitter ↓
FS ↓
Success ↑
MPJPE ↓
GVHMR
82.83
133.08
40.05
2.49
20.69
3.64
28.11
154.3
PromptHMR
72.25
115.55
41.92
1.71
19.56
4.27
35.14
150.5
GENMO
88.12
140.51
42.20
2.49
16.44
4.02
35.68
148.2
DuoMo
52.86
78.28
37.43
1.33
7.53
3.29
24.86
179.1
Ours w/o RL
55.70
88.37
30.78
1.92
7.07
3.63
48.65
133.5
Table 2: Quantitative reconstruction and physical-execution results on RICH. WA/W/PA denote WA-/W-/PA-MPJPE; these and foot sliding (FS) use mm. RTE is in %; Jitter in m/s3 . PHC+ Success (%) and tracking MPJPE (mm) use all 185 sequences, including failures; the latter compares execution with the generated reference motion. Ours achieves the highest PHC+ Success Rate and the lowest tracking MPJPE, and competitive reconstruction results.
Method
Success rate (%) ↑
PromptHMR
44.07
GENMO
48.78
DuoMo
51.06
GVHMR
54.15
FlowHMR (ours)
77.72
Table 3: SONIC success rate on Wild-4K. Bold and underlining mark the best and second-best values.
Figure 5: SONIC execution on Unitree G1. Rows show the input video, the motion reconstructed by FlowHMR (pink), and SONIC tracking in Isaac Lab (blue).
Factor
Setting
WA-MPJPE (mm) ↓
MPJPE (mm) ↓
RTE (mm) ↓
MPJRE hand ( ∘ ) ↓
Jitter (mm/s 2 ) ↓
Foot-sliding (mm) ↓
Data scale
200 h
149.44
90.92
4.91
25.17
16.34
3.53
3,000 h
67.82
43.49
1.86
16.08
11.75
2.55
Prediction
Regression
73.28
43.44
2.39
14.55
18.01
3.94
Flow Matching
67.82
43.49
1.86
16.08
11.75
2.55
Model size
0.2B
67.82
43.49
1.86
16.08
11.75
2.55
0.46B
63.39
42.31
1.81
16.48
11.03
2.27
Table 4: Ablations of the pretrained model.
RICH
Wild-4K (PHC+)
Method
Target / reward
WA-MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Success (%) ↑
Score ↑
Pretrained
—
55.70
30.78
78.79
0.692
SFT
Filtered originals
54.80
30.88
78.38
0.690
Projected targets
70.80
42.13
80.25
0.706
GRPO
Fid. only
57.29
31.62
73.79
0.658
Track. only ∗
255.67
139.20
99.62 ∗
0.942 ∗
Table 5: Ablation of post-training. We compare different post-training methods and reinforcement-learning reward settings. Our GRPO with fidelity (Fid.) and physical tracking (Track.) rewards achieves higher simulation success than both SFT variants while keeping reconstruction errors close to the pretrained model. Optimizing tracking alone yields high success at the cost of severe reconstruction errors. Adding a reward based on Phys-Err (penetration, floating, and skating errors) ( Yuan et al., 2023 ) yields comparable tracking success but worse reconstruction accuracy than our joint-reward design. ∗ marks reward hacking, excluded from ranking.
Figure 6: Qualitative comparison of post-training strategies. Columns show the input video, pretrained FlowHMR, SFT with projected targets (Proj.), joint-reward GRPO (Ours), and tracking-only GRPO (Sim.). Rows show three synchronized time points. Each method panel pairs the reconstructed output (pink, left) with its tracking result in simulation (blue, right).
Model
Pairwise seed MPJPE (mm) ↓
Tracking MPJPE (cm) ↓
Ours w/o RL
57.25
8.35
Ours
22.86
6.96
Table 6: Sampling before and after post-training. Ours w/o RL and Ours are each evaluated with eight seeds on a 300-video subset of Wild-4K. (a) Pairwise seed MPJPE averages the 28 pairs per video, then takes the median across videos. Tracking MPJPE averages the eight seed-level errors between execution and generated references over all motions. (b) Mean ± standard deviation of the eight seed-level PHC+ results.
Figure 7: Qualitative comparison. The top row shows the input video, followed by GVHMR, PromptHMR (PHMR), GENMO, DuoMo, and Ours.
Figure 8: Additional qualitative examples. We present two challenging cases: a fitness exercise that demands precise balance control, and a fast-paced boxing sequence. In both cases, our generated motions are successfully tracked in the physics simulator, demonstrating that our results remain physically balanced while faithfully following highly dynamic movements.
Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.
Human motion recovered from monocular videos often appears overly smooth or dynamically inconsistent, even when joint positions are numerically accurate. We observe that this limitation stems from the absence of reliable high-order temporal cues -- velocity and acceleration -- which are essential for reconstructing motion that exhibits realistic momentum, timing, and high-frequency detail. We introduce HTD-Refine, a post-processing framework that augments existing Human Motion Recovery (HMR) pipelines using explicitly estimated high-order temporal dynamics. At the core of our system is PVA-Net, a temporal transformer that infers per-joint 2D positions, 3D velocities, and 3D accelerations directly from a monocular video. These predicted dynamics serve as soft yet informative constraints in a global optimization procedure that refines world-space trajectories, significantly reducing jitter, suppressing over-smoothing, and restoring physically plausible motion. Extensive experiments on challenging in-the-wild benchmarks show that HTD-Refine consistently improves state-of-the-art HMR methods, yielding more accurate global trajectories and substantially more natural motion dynamics. Our results highlight the critical role of high-order temporal modeling in advancing monocular human motion recovery.
Dingkun Wei, Zehong Shen, Yan Xia +3
Zhejiang University · Ant Group · The University of Texas at Austin
Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.
Patrick Kwon, Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817