Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.
Figures & tables
Figure 1: From dedicated specialist models to one unified framework. Left: Existing approaches require a dedicated architecture for each sensor configuration (e.g. partial-body keypoints (RoHM), head-mounted 6-DoF (EgoAllo), sparse IMUs (SIP/RobustCap)), making every new modality a re-engineering effort. OmniPoser incorporates all of them with one flow-matching backbone and training recipe, recovering full-body SMPL motion from different sensor configurations via the msaked design. Right: Accuracy–FPS performance across three settings. Each baseline (circle) is plotted; OmniPoser (star) matches or surpasses baseline methods at > 700 FPS with lower Mean Per Joint Position Error (MPJPE).
Figure 2: Overview of OmniPoser. Heterogeneous observations (multi-modal 3D keypoints and rotations) are unified through a cross-modal masking mechanism and processed by Masked Conditioning Encoder (MCE) ( Sec. 3.2 ). A decoupled dual-stream architecture( Sec. 3.3 ) (a temporal convolutional UNet for global trajectory and a cross-attention Diffusion Transformer (DiT) for local body pose) generates SMPL parameters via Conditional Flow Matching( Sec. 3.3 ), achieving real-time inference in a single ODE step. Best viewed in color.
Table 2: Quantitative results of Setting 1: Sparse Observation (Random joint masking) on AMASS.
AMASS
RICH (Cross-domain)
Nymeria (Wild)
Method
JPE ↓
PA ↓
Con ↑
Th↓
JPE ↓
PA ↓
Con ↑
Th↓
JPE ↓
PA ↓
Con ↑
EgoEgo
167.4
145.8
0.92
54.9
207.8
187.8
0.88
66.5
135.8
117.2
0.57
EgoAllo
119.7
101.1
1.00
6.2
176.2
160.1
0.96
8.9
103.0
89.3
0.63
Ours-S
83.2
73.5
0.92
17.1
131.6
111.5
0.71
25.7
40.7
36.2
0.94
RoHM-U
124.3
101.5
0.94
36.7
195.0
115.4
0.62
56.8
-
-
-
RoHM++-U
94.5
92.5
0.96
19.5
126.3
112.3
0.72
44.8
-
-
-
Table 3: Quantitative results of Setting 2: Egocentric Head 6D Pose.
AIST++
TotalCapture
Nymeria (In-the-wild)
Method
PA ↓
JPE ↓
PVE ↓
TE ↓
HS ↓
Jrk ↓
FS ↓
PA ↓
JPE ↓
PVE ↓
TE ↓
HS ↓
Jrk ↓
FS ↓
PA ↓
JPE ↓
PVE ↓
TE ↓
HS ↓
Jrk ↓
FS ↓
SIP
23.1
29.8
39.0
11.31
8.91
1.50
0.57
28.6
46.5
54.3
5.42
10.6
0.94
0.69
15.2
17.0
16.8
4.13
5.22
0.059
0.25
RobCap
24.0
33.1
43.2
19.92
9.34
2.49
0.61
33.5
48.7
63.4
23.5
13.4
2.09
1.40
16.5
19.5
19.4
4.52
7.51
1.56
0.54
RobCap3D
21.8
28.5
37.6
15.82
8.95
2.31
0.59
27.9
42.3
52.8
9.74
12.8
1.87
1.22
15.0
17.8
17.5
4.25
7.18
1.41
0.48
Ours-S
19.4
26.0
32.9
16.54
7.40
0.29
0.22
26.5
36.2
51.2
15.9
13.3
0.16
0.81
14.5
16.6
16.6
6.83
7.50
0.048
0.39
RoHM-U
123.5
157.5
185.9
76.3
38.1
0.21
9.49
78.6
95.3
100.4
22.8
19.5
0.19
0.95
-
Table 4: Quantitative results of Setting 3: Mocap Data from 6 IMUs with Keypoint Positions.
Figure 3: Qualitative results. Row 1: Five masking configurations applied to the same speed-vault sequence from AMASS dataset. Meshes in gold are the predictions. The mesh on the rightmost is the ground truth. Row 2–3: Visualizations on different datasets. For more qualitative results, please refer to the supplementary material.
RAM incorporates a motion-aware semantic tracker with adaptive Kalman filtering to achieve robust identity association under severe occlusions and dynamic interactions. A memory-augmented Temporal HMR module further enhances human motion reconstruction by injecting spatio-temporal priors for consistent and smooth motion estimation. Moreover, a lightweight Predictor module forecasts future poses to maintain reconstruction continuity, while a gated combiner adaptively fuses reconstructed and predicted features to ensure coherence and robustness. Experiments on in-the-wild multi-person benchmarks such as PoseTrack and 3DPW, demonstrate that RAM substantially outperforms previous state-of-the-art in both Zero-shot tracking stability and 3D accuracy, offering a generalizable paradigm for markerless 3D human motion capture in-the-wild.
Sen Jia, Ning Zhu, Jinqin Zhong +4
University of Washington · Anhui University · East China University of Science and Technology +1
Although recent studies have made remarkable progress in human mesh recovery, they still exhibit limited robustness to occlusions and often produce inaccurate poses and severe motion jitter due to the insufficient spatial features for occluded body parts. Inspired by the rapid advancements in human motion prediction, we discover that compared to occluded image features, pose sequence inherently contains reliable motion prior for estimating occluded body parts. In this paper, we incorporate Motion Prior for Occluded human mesh recovery, called MoPO. Our MoPO mainly consists of two components: 1) The motion de-occlusion module, where we propose a spatial-temporal occlusion detector to detect joint visibility, and then we propose a lightweight motion predictor to complete the occluded body parts by predicting the most plausible joint positions based on history poses. 2) The motion-aware fusion and refinement module, which fuses the completed joint sequence with image features to estimate human shape and initial human pose. Moreover, the completed joint sequence is further used to refine the final human pose through inverse kinematics, which provides the occlusion-free motion prior for regressing human poses. Extensive experiments demonstrate that MoPO achieves state-of-the-art performance on both occlusion-specific and standard benchmarks, significantly enhancing the accuracy and temporal consistency of occluded human mesh recovery. Our code and demo can be found in the supplementary material.
Tao Tang, Hong Liu, Xinshun Wang +1
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China
Human motion follows a temporal hierarchical structure, transitioning from low-frequency global trajectories to high-frequency details. Inspired by the success of multi-level autoregressive models in computer vision, we propose MotionMAR, a coarse-to-fine framework for motion reconstruction from sparse observations. It first estimates the global trajectory of human motion and then gradually refines the temporal details. This architecture consists of four integrated components. The Temporal Multi-scale Tokenization (TMT) VQ-VAE encodes the data at multiple temporal resolutions, separating semantic motion from minor jitters. The Motion Autoregressive Network (MAN) operates in this latent space, predicting motion across scales. It first establishes the global structure through coarse indices and then generates finer indices to recover specific details. Meanwhile, the Scale-Aware Control (SAC) module integrates sparse tracking data to ensure the generated output aligns with actual observations. The Motion Refinement Network (MRN) subsequently smooths consecutive poses and eliminates quantization artifacts. Experiments show that MotionMAR achieves state-of-the-art accuracy on the AMASS dataset, providing a reliable and structure-aware approach for motion reconstruction. The source code is publicly available at http://www.lidarhumanmotion.net/motionmar/.
Yuhua Luo, Junsheng Zhang, Mengyin Liu +7
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University · National Institute for Data Science in Health and Medicine, Xiamen University +1