Organizations: Peking University · The Hong Kong University of Science and Technology · Tencent · East China Normal University · Sun Yat-sen University · Zhejiang University
We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular video. Previous learning-based methods typically regress human motion directly from video and train the network with geometric supervision. However, recovering human motion from monocular video is inherently ambiguous in depth, and direct regression tends to collapse toward an averaged solution. Moreover, the recovered motions are not guaranteed to be physically plausible, so physics-based tracking of them often fails. To address these challenges, we formulate video motion capture as a video-conditioned motion generation problem and first pretrain a flow matching model for this task. Given an input video, the pretrained model generates diverse motion candidates, but not all of them are faithful to the video or physically trackable. We therefore post-train the model using Group Relative Policy Optimization (GRPO) with two rewards. A fidelity reward encourages consistency with the input video. A tracking reward favors motions that a physics-based controller can track successfully. Together, these rewards shift the model's output preference, so the post-trained model stays faithful to the input video while producing more physically plausible motion. We further introduce Wild-4K, a large and diverse dataset of about 4K internet videos, for evaluating human motion recovery in the wild. Qualitative and quantitative experiments on Wild-4K show that our method outperforms state-of-the-art methods in overall motion fidelity and achieves a physical tracking success rate of 82.47%, compared with 62.82% for the strongest baseline, GVHMR.
Figures & tables
Figure 1: Physically plausible motion recovery. Given a monocular video (rows 1 and 3), FlowHMR recovers global 3D human motion (pink) that a physics-based controller can successfully track in simulation (blue).
Figure 2: Training data scale and diversity. (a) Synthetic examples featuring occlusions, challenging poses, and partial visibility. (b) Total durations of motion data and paired motion-video data, compared with BEDLAM, the largest prior synthetic dataset. (c) Proportions of sampled camera trajectory types. We use a diverse set of camera trajectories to simulate the camera motion found in in-the-wild internet videos.
Figure 3: Overview of the FlowHMR training pipeline. To recover faithful and physically plausible global human motion from monocular video, FlowHMR is trained in two stages. Stage 1 performs generative pretraining, in which a video-conditioned flow-matching model is trained on paired video–motion data with a flow-matching loss and a forward kinematics loss. Stage 2 performs physics-based post-training, in which the model, initialized from the Stage 1 checkpoint, is further fine-tuned with GRPO using two rewards: a fidelity reward that measures agreement with the ground-truth motion, and a physical tracking reward that measures how accurately a frozen, pretrained controller tracks the generated motion in a physics simulator.
Table 1: Quantitative Results on Wild-4K. Left: our method outperforms all baselines in blinded human evaluation, with more wins than losses in every comparison. Bars show win/tie/loss percentages over valid judgments; evaluated videos differ across comparisons. Right: simulation Success Rate (SR, %) under PHC+ and physical errors on 4,182 clips. Our method achieves the highest SR, the lowest penetration (Pen.) and floating (Float) errors, and the second-lowest skating (Skate) error.
Figure 4: Qualitative comparison of reconstruction and simulation tracking. The top row shows the input video, followed by GVHMR ( Shen et al., 2024 ) , GENMO ( Li et al., 2025a ) , DuoMo ( Wang et al., 2026b ) , and FlowHMR. Each method panel pairs the reconstructed output (pink, left) with its tracking result in simulation (blue, right). Our method enables successful simulation tracking of this sequence, whereas all compared baselines fail.
Method
Reconstruction
Motion quality
PHC+ execution
WA ↓
W ↓
PA ↓
RTE ↓
Jitter ↓
FS ↓
Success ↑
MPJPE ↓
GVHMR
82.83
133.08
40.05
2.49
20.69
3.64
28.11
154.3
PromptHMR
72.25
115.55
41.92
1.71
19.56
4.27
35.14
150.5
GENMO
88.12
140.51
42.20
2.49
16.44
4.02
35.68
148.2
DuoMo
52.86
78.28
37.43
1.33
7.53
3.29
24.86
179.1
Ours w/o RL
55.70
88.37
30.78
1.92
7.07
3.63
48.65
133.5
Table 2: Quantitative reconstruction and physical-execution results on RICH. WA/W/PA denote WA-/W-/PA-MPJPE; these and foot sliding (FS) use mm. RTE is in %; Jitter in m/s3 . PHC+ Success (%) and tracking MPJPE (mm) use all 185 sequences, including failures; the latter compares execution with the generated reference motion. Ours achieves the highest PHC+ Success Rate and the lowest tracking MPJPE, and competitive reconstruction results.
Method
Success rate (%) ↑
PromptHMR
44.07
GENMO
48.78
DuoMo
51.06
GVHMR
54.15
FlowHMR (ours)
77.72
Table 3: SONIC success rate on Wild-4K. Bold and underlining mark the best and second-best values.
Figure 5: SONIC execution on Unitree G1. Rows show the input video, the motion reconstructed by FlowHMR (pink), and SONIC tracking in Isaac Lab (blue).
Factor
Setting
WA-MPJPE (mm) ↓
MPJPE (mm) ↓
RTE (mm) ↓
MPJRE hand ( ∘ ) ↓
Jitter (mm/s 2 ) ↓
Foot-sliding (mm) ↓
Data scale
200 h
149.44
90.92
4.91
25.17
16.34
3.53
3,000 h
67.82
43.49
1.86
16.08
11.75
2.55
Prediction
Regression
73.28
43.44
2.39
14.55
18.01
3.94
Flow Matching
67.82
43.49
1.86
16.08
11.75
2.55
Model size
0.2B
67.82
43.49
1.86
16.08
11.75
2.55
0.46B
63.39
42.31
1.81
16.48
11.03
2.27
Table 4: Ablations of the pretrained model.
RICH
Wild-4K (PHC+)
Method
Target / reward
WA-MPJPE (mm) ↓
PA-MPJPE (mm) ↓
Success (%) ↑
Score ↑
Pretrained
—
55.70
30.78
78.79
0.692
SFT
Filtered originals
54.80
30.88
78.38
0.690
Projected targets
70.80
42.13
80.25
0.706
GRPO
Fid. only
57.29
31.62
73.79
0.658
Track. only ∗
255.67
139.20
99.62 ∗
0.942 ∗
Table 5: Ablation of post-training. We compare different post-training methods and reinforcement-learning reward settings. Our GRPO with fidelity (Fid.) and physical tracking (Track.) rewards achieves higher simulation success than both SFT variants while keeping reconstruction errors close to the pretrained model. Optimizing tracking alone yields high success at the cost of severe reconstruction errors. Adding a reward based on Phys-Err (penetration, floating, and skating errors) ( Yuan et al., 2023 ) yields comparable tracking success but worse reconstruction accuracy than our joint-reward design. ∗ marks reward hacking, excluded from ranking.
Figure 6: Qualitative comparison of post-training strategies. Columns show the input video, pretrained FlowHMR, SFT with projected targets (Proj.), joint-reward GRPO (Ours), and tracking-only GRPO (Sim.). Rows show three synchronized time points. Each method panel pairs the reconstructed output (pink, left) with its tracking result in simulation (blue, right).
Model
Pairwise seed MPJPE (mm) ↓
Tracking MPJPE (cm) ↓
Ours w/o RL
57.25
8.35
Ours
22.86
6.96
Table 6: Sampling before and after post-training. Ours w/o RL and Ours are each evaluated with eight seeds on a 300-video subset of Wild-4K. (a) Pairwise seed MPJPE averages the 28 pairs per video, then takes the median across videos. Tracking MPJPE averages the eight seed-level errors between execution and generated references over all motions. (b) Mean ± standard deviation of the eight seed-level PHC+ results.
Figure 7: Qualitative comparison. The top row shows the input video, followed by GVHMR, PromptHMR (PHMR), GENMO, DuoMo, and Ours.
Figure 8: Additional qualitative examples. We present two challenging cases: a fitness exercise that demands precise balance control, and a fast-paced boxing sequence. In both cases, our generated motions are successfully tracked in the physics simulator, demonstrating that our results remain physically balanced while faithfully following highly dynamic movements.