Human mesh recovery (HMR) models 3D human body from monocular videos, with recent works extending it to world-coordinate human trajectory and motion reconstruction. However, most existing methods remain offline, relying on future frames or global optimization, which limits their applicability in interactive feedback and perception-action loop scenarios such as AR/VR and telepresence. To address this, we propose OnlineHMR, a fully online framework that jointly satisfies four essential criteria of online processing, including system-level causality, faithfulness, temporal consistency, and efficiency. Built upon a two-branch architecture, OnlineHMR enables streaming inference via a causal key-value cache design and a curated sliding-window learning strategy. Meanwhile, a human-centric incremental SLAM provides online world-grounded alignment under physically plausible trajectory correction. Experimental results show that our method achieves performance comparable to existing chunk-based approaches on the standard EMDB benchmark and highly dynamic custom videos, while uniquely supporting online processing. Page and code are available at https://tsukasane.github.io/Video-OnlineHMR/.
Figures & tables
Figure 2: The online processing workflow of our method. Given a streaming video input, we estimate the world coordinate 3D human body of the most recent frame. Ti here is the homogeneous transformation matirx composed by qic and tic in Sec. 3.1 .
Figure 3: Sliding window learning pipeline. The input sequence is sliced to overlapping windows, learning spatial and temporal information fusion inside each window, and alleviate jitter effect through velocity regularization.
Table 3
Figure 4: Quantitative comparison of OnlineHMR and Human3R on the same EMDB-2 video with ground truth after world coordinate alignments.
Figure 5: Quantitative comparison of OnlineHMR and Human3R on the same EMDB-2 video with ground truth after world coordinate alignments.
Models
Online
FPS ↑
Avg. Delay (s) ↓
WA-MPJPE 100 ↓
TRACE [ 37 ]
✓
24.6
0.04
529.0
GLAMR [ 53 ]
✗
17.2
14.16
280.8
SLAHMR [ 51 ]
✗
0.1
2435
326.9
WHAM (w/ DPVO) [ 34 ]
✗
9.3
26.18
135.6
TRAM [ 47 ]
✗
2.1
115.95
76.4
GVHMR (w/ DPVO) [ 33 ]
✗
12.1
20.12
111.0
Table 3: Efficiency comparison of world coordinate HMR methods. The average delay time is presented in seconds.
Models
3DPW
EMDB-1
Accel ↓
Jitter ↓
Accel ↓
Jitter ↓
w/o Velocity Regularization
8.9
32.3
15.7
70.1
w/ Velocity Regularization
6.4
19.5
9.0
33.7
Table 4: The comparison of jittering effect w/ or w/o the velocity regularization.
Figure 6: Visualization of multi-individual, diverse scene cases. More examples are in Suppl.
Method
Vanilla
Hard Mask
Soft Mask
DROID-SLAM
2.52
1.55
1.07
MAST3R-SLAM
1.22
0.96
0.83
Table 5: Comparison of ATE using different masking strategies for DROID-SLAM and MAST3R-SLAM. Lower is better.
Figure 7: Visualization of estimated camera trajectory compared with GT.
Figure 8: Difference spectrograms computed as GT-Pred, visualizing discrepancies in the time–frequency domain. The left figure corresponds to a model trained for 1K iterations, and the right for 52K iterations. Lower values (darker/closer to zero) indicate better alignment with the ground truth. The converged model (52K) exhibits substantially reduced differences.
Dataset
GVHMR
TRAM
Ours
3DPW
3.59 / 0.02
17.01 / 0.64
19.91 / 0.82
EMDB-1
75.52 / 0.12
25.82 / 1.24
24.98 / 1.41
Table 6: RMSE ↓ / Corr ↓ results.
3DPW
EMDB-1
SWS
PA-MPJPE ↓
MPJPE ↓
PVE ↓
ACCEL ↓
PA-MPJPE ↓
MPJPE ↓
PVE ↓
ACCEL ↓
3
43.7
69.9
83.7
6.4
46.0
74.0
86.1
9.0
4
41.7
64.9
78.6
6.4
46.8
73.8
85.3
9.1
5
43.2
70.8
85.7
6.6
46.7
74.5
86.5
9.4
6
42.5
65.5
80.1
6.6
46.9
74.3
84.4
9.6
Table 7: Ablation study on sliding window size (SWS).
Strategy
WA-MPJPE
W-MPJPE
RTE
ERVE
Vanilla
119.6
412.9
4.1
14.4
Hard Mask
112.6
386.8
3.2
13.3
Soft Mask
93.5
310.4
2.2
12.4
Table 8: Comparison of different masking strategies on world coordinate human reconstruction upon MASt3R-SLAM [ 25 ]
Figure 9: Using the same video input with Fig. 12 , we show the global human mesh, camera trajectory, and the final updated scene.
Figure 10: Qualitative results w/ and w/o EMA correction on custom videos.
Figure 11: Example of failure cases.
Figure 12: Quantitative comparison of OnlineHMR and Human3R on a custom video of a famous dancer. OnlineHMR produces a faithful reconstruction of the human trajectory, whereas Human3R yields trajectories that appear compressed and crowded together.
Figure 13: More examples with multiple individuals and a diverse scene. Dynamic results can be found on the demo page.
Figure 14: Quantitative comparison of OnlineHMR and Human3R on a custom video of running up stairs.
Human Mesh Recovery (HMR) is fundamentally ambiguous: under occlusion or weak depth cues, multiple 3D bodies can explain the same image evidence. This ambiguity is not uniform across the body, as torso pose and root structure are often relatively well constrained, whereas distal articulations such as the arms and legs are more uncertain. Building on this observation, we propose FactorizedHMR, a two-stage framework that treats these two regimes differently. A deterministic regression module first recovers a stable torso-root anchor, and a probabilistic flow-matching module then completes the remaining non-torso articulation. To make this completion reliable, we combine a composite target representation with geometry-aware supervision and feature-aware classifier-free guidance, preserving the torso-root anchor while improving single-reference recovery of ambiguity-prone articulation. We also introduce a synthetic data pipeline that provides the paired image-camera-motion supervision under diverse viewpoints. Across camera-space and world-space benchmarks, FactorizedHMR remains competitive with strong baselines, with the clearest gains in occlusion-heavy recovery and drift-sensitive world-space metrics.
Patrick Kwon, Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817
Monocular video human mesh recovery is essential for digital humans, avatar animation, and embodied simulation, where both temporal stability and expressive whole-body motion are required. Existing video HMR methods produce coherent body motion but often overlook detailed hand articulation, while image-based whole-body methods recover SMPL-X meshes independently per frame, often leading to jittery and inaccurate hand motion. We present a temporally coherent whole-body HMR framework for challenging in-the-wild monocular videos. Our model unifies body context and part-specific hand observations through residual body-hand fusion, enabling stable body motion and detailed hand recovery within a single temporal architecture. We further introduce close-up-aware augmentation to improve robustness under upper-body framing. Experiments on whole-body and body-only benchmarks demonstrate improved hand reconstruction and competitive body accuracy. Our method also produces temporally stable and 2D-consistent SMPL-X motion in challenging real-world videos.
Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.