RAM: Recover Any 3D Human Motion in-the-Wild

Sen Jia; Ning Zhu; Jinqin Zhong; Jiale Zhou; Huaping Zhang; Jenq-Neng Hwang; Lei Li

cs.CVMar 20, 2026

RAM: Recover Any 3D Human Motion in-the-Wild

Authors: Sen Jia, Ning Zhu, Jinqin Zhong, Jiale Zhou, Huaping Zhang, Jenq-Neng Hwang, Lei Li

Organizations: University of Washington · Anhui University · East China University of Science and Technology · Beijing Institute of Technology

Abstract

RAM incorporates a motion-aware semantic tracker with adaptive Kalman filtering to achieve robust identity association under severe occlusions and dynamic interactions. A memory-augmented Temporal HMR module further enhances human motion reconstruction by injecting spatio-temporal priors for consistent and smooth motion estimation. Moreover, a lightweight Predictor module forecasts future poses to maintain reconstruction continuity, while a gated combiner adaptively fuses reconstructed and predicted features to ensure coherence and robustness. Experiments on in-the-wild multi-person benchmarks such as PoseTrack and 3DPW, demonstrate that RAM substantially outperforms previous state-of-the-art in both Zero-shot tracking stability and 3D accuracy, offering a generalizable paradigm for markerless 3D human motion capture in-the-wild.

Figures & tables

Figure 1: RAM performs online 3D motion reconstruction from monocular video via semantic, motion-aware tracking and occlusion-robust prediction. Unlike prior methods that rely on frame-wise regression and queue-based identity matching, which are often unstable under fast motion and occlusions, RAM maintains consistent identity and accurate reconstruction, while achieving real-time performance that is 2–3 $\times$ faster than previous approaches. Arrows highlight key differences.

Figure 2: Overview of RAM. The framework integrates four components: SegFollow for motion-guided temporal tracking, Temporal HMR for memory-based 3D reconstruction, a Predictor for motion forecasting under occlusion, and a gated Combiner for robust recovery.

Method	PoseTrack18			PoseTrack21
Method	HOTA $\uparrow$	IDs $\downarrow$	MOTA $\uparrow$	MOTA $\uparrow$	IDF1 $\uparrow$	IDP $\uparrow$	IDR $\uparrow$	FP $\downarrow$	FN $\downarrow$	FPS $\uparrow$
4DHumans [ 18 ]	57.8	382	61.4	–	–	–	–	–	–	-
4DHumans (reproduced) ∗	58.0	349	61.8	56.7	70.9	87.1	59.7	7817	50652	0.51
CoMotion [ 66 ] strict	58.2	232	59.9	61.8	74.0	89.1	63.3	6086	45664	-
CoMotion [ 66 ]	54.9	344	51.3	71.4	79.5	87.1	73.0	8115	30394	5.68
RAM ( Zero-shot )	66.4	15	57.7	74.4	85.9	93.8	79.2	5864	24044	10.32

Table 1: Comparison on PoseTrack18 and PoseTrack21: while 4DHumans and CoMotion are trained on these datasets, RAM is evaluated zero-shot without retraining, and achieves the best results across all metrics.

Method	TrackID3x3 (Indoor)			TrackID3x3 (Outdoor)
Method	TI-HOTA $\uparrow$	TI-DetA $\uparrow$	TI-AssA $\uparrow$	TI-HOTA $\uparrow$	TI-DetA $\uparrow$	TI-AssA $\uparrow$
4DHumans [ 18 ]	5.17	4.89	5.47	2.66	2.19	3.22
CoMotion [ 66 ]	42.20	32.93	54.07	30.87	23.06	41.33
RAM (SAM2)	54.23	48.39	60.78	38.60	36.92	40.37
RAM	75.07	62.87	89.66	66.68	51.39	86.63

Table 2: Comparison on two challenging real-world sports scenarios featuring frequent occlusions, and fast motion. We evaluate zero-shot generalization of prior methods and RAM, and additionally report an ablation using only SAM2-based tracking.

	Method	COCO		PoseTrack		3DPW
		[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
2D/3D HPE	PyMAF [ 91 ]	0.68	0.86	0.77	0.92	92.8	58.9
	CLIFF [ 52 ]	0.64	0.88	0.75	0.92	69.0	43.0
	PARE [ 38 ]	0.72	0.91	0.79	0.93	82.0	50.9
	PyMAF-X [ 90 ]	0.79	0.93	0.85	0.95	78.0	47.1
	HMR 2.0a [ 18 ]	0.79	0.95	0.86	0.97	70.0	44.5

Table 3: Pose estimation. Normalized PCK accuracy on projected 2D keypoints at varying thresholds on the COCO and PoseTrack datasets, alongside MPJPE of 3D keypoints on the 3DPW dataset. We highlight that our model performs similarly when provided the full image as input rather than an oracle-resized crop around a target person. See text for analysis.

Figure 3: Qualitative comparison on the Olympics Boxing dataset. Both 4DHumans and CoMotion suffer from identity switches and tracking failures under fast motion and heavy occlusion, resulting in fragmented 3D reconstructions and repeated identity reinitialization. This not only degrades reconstruction quality but also leads to high inference overhead. In contrast, RAM robustly tracks boxers and the referee throughout the sequence with consistent identity association and real-time, accurate 3D motion recovery.

Explore similar work

OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching
Oct 14, 2026Minghao Liu, Tutian Tang3D Human Reconstruction 3D Human Pose Estimation
Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China
DirtyMoCap: Robust Motion Capture from Unconstrained Markers
Sep 17, 2026Long Wang, Shuting Zhao, Shen Yan +5Gauss-Newton Optimization 3D Human Pose Estimation
Zhejiang University, Hangzhou, China · Westlake University, Hangzhou, China · Fudan University, Shanghai, China +3
MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations
Jun 22, 2026Yuhua Luo, Junsheng Zhang, Mengyin Liu +73D Reconstruction Vector-Quantized Variational Autoencoder
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University · National Institute for Data Science in Health and Medicine, Xiamen University +1

Oct 14, 2026J/K move · Enter open · S save

OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

Minghao Liu, Tutian Tang

Shanghai Jiao Tong University, China · Shanghai Artificial Intelligence Laboratory, China

3D Human Reconstruction 3D Human Pose Estimation Human Motion Generation

Open paper AskPDF arXiv

Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.

Abstract

Explore similar work

OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

DirtyMoCap: Robust Motion Capture from Unconstrained Markers

MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations