Accurate full-body tracking in Virtual Reality (VR) is heavily underconstrained when relying solely on consumer-grade sparse sensors, such as Head-Mounted Displays (HMDs) and handheld controllers. Indeed, the self-avatar full-body pose must be inferred based on the 3 available upper-body joints tracked by these devices. This leads to significant tracking error, especially in lower-body pose. To overcome this issue, we propose to extend the tracking setup with external multiview RGB cameras. We show that integrating RGB-based pose estimation this framework is able to significantly decrease lower-body tracking errors, even under common artifacts in RGB-based estimates. Additionally, we evaluate a only-once HMD-camera calibration procedure that does not require any physical markers. We finally propose an implementation of the overall framework, achieving an end-to-end animation at 60 FPS on a Nvidia RTX3080 GPU. Codes are available at https://github.com/antmaio/MuCHMDPoser.
Figures & tables
Figure 1: Virtual configuration that imitates hardware setup. We define 3 virtual cameras, whose the intrinsic and extrinsic parameters are known. Off-the-shelf 2D pose estimation is applied in every rendered view to capture lower-body motion cues.
Figure 2: Root global position distribution on the ground surface under a standard evaluation protocol using AMASS dataset. This delimitates an approximative 60m2 interaction area.
Figure 3: Full-body Pose Estimation model. We extend the architecture from HMD-Poser [ 11 ] to feed the filtered positions pJextt of body joints Jext acquired by the multiview pose estimation setup. Extending the channel-wise embedding, we are able to provide embeddings ejt from body joints Jext outside standard head and hands tracking.
Figure 4: Overview of the calibration setup used to register the multi-camera system, the HMD, and the handheld controllers into a common reference axis system. Left : Schematic of the full setup, showing the subject wearing an HMD inside the capture volume observed by the external RGB cameras. Right : (a) Standard multiview camera calibration protocol [ 21 ] . (b) The Meta Mixed Reality Utility Kit (MRUK) provides a room scan from passthrough, allowing a physically-anchored virtual tracking space Hf . (c) Snapshot from the HMD camera k≡C with corner detection xik and pose registration THf→C . This setup closes the loop between the HMD tracking space, the floor-anchored frame within the virtual application, and the external camera system.
Figure 5: Reference frames involved in HMD pose estimation. The device-centric frame H , re-initialized arbitrarily at each VR application startup, is mapped to the session-invariant frame Hf . Both H and Hf live within the VR software domain, while Hf additionally serves as the bridge to the physical world domain: the rigid-body transform THf→W , obtained from the multi-camera calibration, relates Hf to the shared physical world frame W .
Figure 6: Protocol to assess the consistency and stability of Hf tracking space. We take several snapshots from left camera of the HMD, register its pose relatively to Hf . We use this transform to reproject the control points into the image plane of each snapshot and measure the discrepancy between these 2D points and the corners detected by OpenCV.
ϵ2Da (px)
ϵ3Da (cm)
Ref. snapshot a
H
Hf
Δ
H
Hf
Δ
S1
2.37
2.37
0.00
0.62
0.62
0.00
S2
3.55
3.55
0.00
0.82
0.82
0.00
S3
3.58
3.58
0.00
1.03
1.03
0.00
S4
2.82
2.82
0.00
0.77
0.77
0.00
S5
3.86
3.86
0.00
0.91
0.91
0.00
Table 1: Cross-reprojection errors for each reference snapshot, comparing the standard tracking space H with the MRUK-based frame Hf . The reprojection procedure results in sub-centimer precision, in average. Δ=ϵ(Hf)−ϵ(H) indicates that both tracking space axis systems share similar stability properties. Lower is better.
Protocol
Occlusion
MPJPE ↓
MPJVE ↓
Jitter ↓
rate [%]
[cm]
[cm/s]
ratio
Protocol 1
0.95
12.94
155.78
51.12
Protocol 2
0.58
21.31
206.80
72.53
Table 2: Multiview 2D-to-3D pose lifting with direct linear triangulation (DLT), evaluated on the two protocols. ↓ : lower is better.
Position error
Rotation
Velocity
Smoothness
Configuration
MPJPE ↓
U-PE ↓
L-PE ↓
MPJRE ↓
MPJVE ↓
Jitter ↓
[cm]
[cm]
[cm]
[ ∘ ]
[cm/s]
ratio
Protocol 1
3 trackers
3.67
1.57
6.71
5.43
17.79
1.74
3 trackers + 3D pose (ours)
2.35
1.23
3.97
4.62
16.20
2.40
Protocol 2
Table 3: Full-body pose estimation accuracy on both protocols, comparing the 3-tracker baseline (HMD and two controllers) with our method, which adds external 3D pose observations. Position errors are reported for the full body (MPJPE), upper body (U-PE) and lower body (L-PE), together with the joint rotation error (MPJRE), joint velocity error (MPJVE) and jitter ratio. ↓ : lower is better; best results in bold.
Figure 7: Samples of full-body pose estimated from ( top to bottom ): 3 trackers; 3 trackers + Pose estimation; ground truth. Incorporating 3D positions from external tracking improves the of the full-body pose estimation, especially at lower-body parts.
Figure 8: Poses estimated by our proposed framework ( top ) and from baseline, relying on HMD and controllers solely ( bottom ). The green spheres represent the positions pJextt of body joints estimated from RGB videos (except for the axis system attached to the green sphere, which corresponds to Hf ). Our model is able to follow the user’s motion and improves lower-body tracking compared to full-body pose estimation from sparse tracking alone.
Prior avatar embodiment research has studied tracking completeness (how faithfully the body, hands, and face are sensed and reproduced) and personalized appearance in isolation, each requiring specialized lab equipment. How these factors interact on consumer hardware remains an open question, because no existing system unifies body pose, face, hands, and personalized appearance on a standalone headset. We present FlowAvatar, a system enabling real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs on consumer XR devices (Meta Quest Pro). Our pose model achieves competitive spatial accuracy and state-of-the-art temporal consistency on the AMASS benchmark against prior works with comprehensive spatial and temporal losses validated through architecture and loss ablations, while running in real-time. Two user studies (N=12, N=10) provide converging evidence that tracking completeness and personalized appearance serve complementary roles: tracking completeness broadly improves perceived motion quality, while only their combination achieves positive total embodiment.
Chenfeng Gao, Taeyoung Yeon, Sungheon Park +4
Northwestern University · University of Chicago · Google
We address the problem of 3D body pose estimation of multiple interacting people from their egocentric views with centralized coordination. Each individual wears a camera recording egocentric video and IMU data. Processing this video with VIO SLAM provides high-quality tracking of each egocentric camera through space. The first-person view from one individual provides third-person observations of other people, although these exocentric observations are sparse, intermittent, and of highly variable reliability as both cameras and subjects move. To integrate these synchronized data streams, we propose a diffusion-based approach that fuses estimates of pose based on head motion derived from egocentric camera motion with exocentric pose observations, conditioning on both observation content and reliability. Our model is trained on a mixture of single-person motion-capture data and multi-person video in order to learn rich priors for body motion trajectories and video observation reliability. Evaluation on challenging multi-person datasets suggests our fusion approach improves over motion-only and vision-only baselines in terms of both absolute and relative pose accuracy.
Daeyun Shin, Yunhan Zhao, Shu Kong +2
University of California, Irvine · University of Macau
3D human pose estimation is a key enabling technology for applications such as healthcare monitoring, human-robot collaboration, and immersive gaming, but real-world deployment remains challenged by viewpoint variations. Existing methods struggle to generalize to unseen camera viewpoints, require large amounts of training data, and suffer from high inference latency. We propose MoViD, a viewpoint-invariant 3D human pose estimation framework that disentangles viewpoint information from motion features. The key idea is to extract viewpoint information from intermediate pose features and leverage it to enhance both the robustness and efficiency of pose estimation. MoViD introduces a view estimator that models key joint relationships to predict viewpoint information, and an orthogonal projection module to disentangle motion and view features, further enhanced through physics-grounded contrastive alignment across views. For real-time edge deployment, MoViD employs a frame-by-frame inference pipeline with a view-aware strategy that adaptively activates flip refinement based on the estimated viewpoint. Evaluations on nine public datasets and newly collected multiview UAV and gait analysis datasets show that MoViD reduces pose estimation error by over 24.2% compared to state-of-the-art methods, maintains robust performance under severe occlusions with 60% less training data, and achieves real-time inference at 15 FPS on NVIDIA edge devices.
Yejia Liu, Hengle Jiang, Haoxian Liu +2
Department of Computer Science and Engineering, Hong Kong University of Science and Technology