OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

Minghao Liu; Tutian Tang

cs.CVOct 14, 2026

OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

Authors: Minghao Liu, Tutian Tang

Organizations: Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China

Abstract

Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.

Figures & tables

Figure 1: From dedicated specialist models to one unified framework. Left: Existing approaches require a dedicated architecture for each sensor configuration (e.g. partial-body keypoints (RoHM), head-mounted 6-DoF (EgoAllo), sparse IMUs (SIP/RobustCap)), making every new modality a re-engineering effort. OmniPoser incorporates all of them with one flow-matching backbone and training recipe, recovering full-body SMPL motion from different sensor configurations via the msaked design. Right: Accuracy–FPS performance across three settings. Each baseline (circle) is plotted; OmniPoser (star) matches or surpasses baseline methods at $>$ 700 FPS with lower Mean Per Joint Position Error (MPJPE).

Figure 2: Overview of OmniPoser. Heterogeneous observations (multi-modal 3D keypoints and rotations) are unified through a cross-modal masking mechanism and processed by Masked Conditioning Encoder (MCE) ( Sec. 3.2 ). A decoupled dual-stream architecture( Sec. 3.3 ) (a temporal convolutional UNet for global trajectory and a cross-attention Diffusion Transformer (DiT) for local body pose) generates SMPL parameters via Conditional Flow Matching( Sec. 3.3 ), achieving real-time inference in a single ODE step. Best viewed in color.

	AMASS					Nymeria (In-the-wild)
Method	JPE $\downarrow$	Acc $\downarrow$	Con $\uparrow$	Ska $\downarrow$	Pen $\downarrow$	JPE $\downarrow$	Acc $\downarrow$	Con $\uparrow$	Ska $\downarrow$
HuMoR	88.0	3.3	0.68	0.230	2.59	88.5	8.26	0.78	0.685
RoHM	34.8	2.3	0.95	0.078	0.69	59.3	7.2	0.91	0.610
Ours-S	32.1	2.7	0.94	0.031	1.84	21.1	1.39	0.95	0.033
RoHM-U	158.7	2.9	0.75	0.190	1.48	-	-	-	-
RoHM++-U	39.9	2.3	0.95	0.081	0.56	-	-	-	-

Table 1: Quantitative results of Setting 1: Sparse Observation (Upper-body joint positions).

Metric	HuMoR	RoHM	Ours-S	Ours-U
JPE (mm) $\downarrow$	80.5	38.2	35.7	49.3
Accel. (m/s 2 ) $\downarrow$	3.3	1.9	2.2	2.6
Contact $\uparrow$	0.78	0.95	0.97	0.93
Skating $\downarrow$	0.184	0.024	0.018	0.019

Table 2: Quantitative results of Setting 1: Sparse Observation (Random joint masking) on AMASS.

	AMASS				RICH (Cross-domain)				Nymeria (Wild)
Method	JPE $\downarrow$	PA $\downarrow$	Con $\uparrow$	$T_{h}\downarrow$	JPE $\downarrow$	PA $\downarrow$	Con $\uparrow$	$T_{h}\downarrow$	JPE $\downarrow$	PA $\downarrow$	Con $\uparrow$
EgoEgo	167.4	145.8	0.92	54.9	207.8	187.8	0.88	66.5	135.8	117.2	0.57
EgoAllo	119.7	101.1	1.00	6.2	176.2	160.1	0.96	8.9	103.0	89.3	0.63
Ours-S	83.2	73.5	0.92	17.1	131.6	111.5	0.71	25.7	40.7	36.2	0.94
RoHM-U	124.3	101.5	0.94	36.7	195.0	115.4	0.62	56.8	-	-	-
RoHM++-U	94.5	92.5	0.96	19.5	126.3	112.3	0.72	44.8	-	-	-

Table 3: Quantitative results of Setting 2: Egocentric Head 6D Pose.

	AIST++							TotalCapture							Nymeria (In-the-wild)
Method	PA $\downarrow$	JPE $\downarrow$	PVE $\downarrow$	TE $\downarrow$	HS $\downarrow$	Jrk $\downarrow$	FS $\downarrow$	PA $\downarrow$	JPE $\downarrow$	PVE $\downarrow$	TE $\downarrow$	HS $\downarrow$	Jrk $\downarrow$	FS $\downarrow$	PA $\downarrow$	JPE $\downarrow$	PVE $\downarrow$	TE $\downarrow$	HS $\downarrow$	Jrk $\downarrow$	FS $\downarrow$
SIP	23.1	29.8	39.0	11.31	8.91	1.50	0.57	28.6	46.5	54.3	5.42	10.6	0.94	0.69	15.2	17.0	16.8	4.13	5.22	0.059	0.25
RobCap	24.0	33.1	43.2	19.92	9.34	2.49	0.61	33.5	48.7	63.4	23.5	13.4	2.09	1.40	16.5	19.5	19.4	4.52	7.51	1.56	0.54
RobCap3D	21.8	28.5	37.6	15.82	8.95	2.31	0.59	27.9	42.3	52.8	9.74	12.8	1.87	1.22	15.0	17.8	17.5	4.25	7.18	1.41	0.48
Ours-S	19.4	26.0	32.9	16.54	7.40	0.29	0.22	26.5	36.2	51.2	15.9	13.3	0.16	0.81	14.5	16.6	16.6	6.83	7.50	0.048	0.39
RoHM-U	123.5	157.5	185.9	76.3	38.1	0.21	9.49	78.6	95.3	100.4	22.8	19.5	0.19	0.95	-

Table 4: Quantitative results of Setting 3: Mocap Data from 6 IMUs with Keypoint Positions.

Figure 3: Qualitative results. Row 1: Five masking configurations applied to the same speed-vault sequence from AMASS dataset. Meshes in gold are the predictions. The mesh on the rightmost is the ground truth. Row 2–3: Visualizations on different datasets. For more qualitative results, please refer to the supplementary material.

Variant	JPE $\downarrow$	Acc $\downarrow$	Ska $\downarrow$
OmniPoser (full)	32.1	2.7	0.031
MCE $\to$ zero-padding	88.1	4.0	0.053

Table 5: Ablation studies (AMASS, upper-body).

Explore similar work

RAM: Recover Any 3D Human Motion in-the-Wild
Mar 20, 2026Sen Jia, Ning Zhu, Jinqin Zhong +4Markerless Motion Capture 3D Human Pose Estimation
University of Washington · Anhui University · East China University of Science and Technology +1
MoPO: Incorporating Motion Prior for Occluded Human Mesh Recovery
May 11, 2026Tao Tang, Hong Liu, Xinshun Wang +13D Reconstruction Human Mesh Recovery
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China
MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations
Jun 22, 2026Yuhua Luo, Junsheng Zhang, Mengyin Liu +73D Reconstruction Vector-Quantized Variational Autoencoder
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University · National Institute for Data Science in Health and Medicine, Xiamen University +1

Mar 20, 2026J/K move · Enter open · S save

RAM: Recover Any 3D Human Motion in-the-Wild

Sen Jia, Ning Zhu, Jinqin Zhong, Jiale Zhou+3

University of Washington · Anhui University · East China University of Science and Technology+1

Markerless Motion Capture 3D Human Pose Estimation 3D MOT

Open paper AskPDF arXiv

RAM incorporates a motion-aware semantic tracker with adaptive Kalman filtering to achieve robust identity association under severe occlusions and dynamic interactions. A memory-augmented Temporal HMR module further enhances human motion reconstruction by injecting spatio-temporal priors for consistent and smooth motion estimation. Moreover, a lightweight Predictor module forecasts future poses to maintain reconstruction continuity, while a gated combiner adaptively fuses reconstructed and predicted features to ensure coherence and robustness. Experiments on in-the-wild multi-person benchmarks such as PoseTrack and 3DPW, demonstrate that RAM substantially outperforms previous state-of-the-art in both Zero-shot tracking stability and 3D accuracy, offering a generalizable paradigm for markerless 3D human motion capture in-the-wild.

Abstract

Explore similar work

RAM: Recover Any 3D Human Motion in-the-Wild

MoPO: Incorporating Motion Prior for Occluded Human Mesh Recovery

MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations