MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality

Antoine Maiorca; George Fletcher; Thierry Ravet; Thierry Dutoit

cs.CVNov 2, 2026

MuCHMDPoser: Multiview RGB-Camera Full-Body Pose Tracking for Virtual Reality

Authors: Antoine Maiorca, George Fletcher, Thierry Ravet, Thierry Dutoit

Organizations: ISIA Lab UMONS Mons, Belgium · Department of Computer Science University of Bath Bath, UK

Abstract

Accurate full-body tracking in Virtual Reality (VR) is heavily underconstrained when relying solely on consumer-grade sparse sensors, such as Head-Mounted Displays (HMDs) and handheld controllers. Indeed, the self-avatar full-body pose must be inferred based on the 3 available upper-body joints tracked by these devices. This leads to significant tracking error, especially in lower-body pose. To overcome this issue, we propose to extend the tracking setup with external multiview RGB cameras. We show that integrating RGB-based pose estimation this framework is able to significantly decrease lower-body tracking errors, even under common artifacts in RGB-based estimates. Additionally, we evaluate a only-once HMD-camera calibration procedure that does not require any physical markers. We finally propose an implementation of the overall framework, achieving an end-to-end animation at 60 FPS on a Nvidia RTX3080 GPU. Codes are available at https://github.com/antmaio/MuCHMDPoser.

Figures & tables

Figure 1: Virtual configuration that imitates hardware setup. We define 3 virtual cameras, whose the intrinsic and extrinsic parameters are known. Off-the-shelf 2D pose estimation is applied in every rendered view to capture lower-body motion cues.

Figure 2: Root global position distribution on the ground surface under a standard evaluation protocol using AMASS dataset. This delimitates an approximative $60m^{2}$ interaction area.

Figure 3: Full-body Pose Estimation model. We extend the architecture from HMD-Poser [ 11 ] to feed the filtered positions

p^{t}_{J_{ext}}

of body joints

J_{ext}

acquired by the multiview pose estimation setup. Extending the channel-wise embedding, we are able to provide embeddings

e_{j}^{t}

from body joints

J_{ext}

outside standard head and hands tracking.

Figure 4: Overview of the calibration setup used to register the multi-camera system, the HMD, and the handheld controllers into a common reference axis system. Left : Schematic of the full setup, showing the subject wearing an HMD inside the capture volume observed by the external RGB cameras. Right : (a) Standard multiview camera calibration protocol [ 21 ] . (b) The Meta Mixed Reality Utility Kit (MRUK) provides a room scan from passthrough, allowing a physically-anchored virtual tracking space $\mathcal{H}_{f}$ . (c) Snapshot from the HMD camera $k\equiv\mathcal{C}$ with corner detection $\mathbf{x}_{i}^{k}$ and pose registration $\mathbf{T}_{\mathcal{H}_{f}\to\mathcal{C}}$ . This setup closes the loop between the HMD tracking space, the floor-anchored frame within the virtual application, and the external camera system.

Panel of Figure 5 — Figure 5: Reference frames involved in HMD pose estimation. The device-centric frame $\mathcal{H}$ , re-initialized arbitrarily at each VR application startup, is mapped to the session-invariant frame $\mathcal{H}_{f}$ . Both $\mathcal{H}$ and $\mathcal{H}_{f}$ live within the VR software domain, while $\mathcal{H}_{f}$ additionally serves as the bridge to the physical world domain: the rigid-body transform $\mathbf{T}_{\mathcal{H}_{f}\to\mathcal{W}}$ , obtained from the multi-camera calibration, relates $\mathcal{H}_{f}$ to the shared physical world frame $\mathcal{W}$ .

Figure 6: Protocol to assess the consistency and stability of $\mathcal{H}_{f}$ tracking space. We take several snapshots from left camera of the HMD, register its pose relatively to $\mathcal{H}_{f}$ . We use this transform to reproject the control points into the image plane of each snapshot and measure the discrepancy between these 2D points and the corners detected by OpenCV.

	$\epsilon_{\text{2D}}^{a}$ (px)			$\epsilon_{\text{3D}}^{a}$ (cm)
Ref. snapshot $a$	$\mathcal{H}$	$\mathcal{H}_{f}$	$\Delta$	$\mathcal{H}$	$\mathcal{H}_{f}$	$\Delta$
$\mathbf{S}_{1}$	2.37	2.37	0.00	0.62	0.62	0.00
$\mathbf{S}_{2}$	3.55	3.55	0.00	0.82	0.82	0.00
$\mathbf{S}_{3}$	3.58	3.58	0.00	1.03	1.03	0.00
$\mathbf{S}_{4}$	2.82	2.82	0.00	0.77	0.77	0.00
$\mathbf{S}_{5}$	3.86	3.86	0.00	0.91	0.91	0.00

Table 1: Cross-reprojection errors for each reference snapshot, comparing the standard tracking space

\mathcal{H}

with the MRUK-based frame

\mathcal{H}_{f}

. The reprojection procedure results in sub-centimer precision, in average.

\Delta=\epsilon(\mathcal{H}_{f})-\epsilon(\mathcal{H})

indicates that both tracking space axis systems share similar stability properties. Lower is better.

Protocol	Occlusion	MPJPE $\downarrow$	MPJVE $\downarrow$	Jitter $\downarrow$
	rate [%]	[cm]	[cm/s]	ratio
Protocol 1	0.95	12.94	155.78	51.12
Protocol 2	0.58	21.31	206.80	72.53

Table 2: Multiview 2D-to-3D pose lifting with direct linear triangulation (DLT), evaluated on the two protocols.

\downarrow

: lower is better.

Protocol 1
	Position error			Rotation	Velocity	Smoothness
Configuration	MPJPE $\downarrow$	U-PE $\downarrow$	L-PE $\downarrow$	MPJRE $\downarrow$	MPJVE $\downarrow$	Jitter $\downarrow$
	[cm]	[cm]	[cm]	[ ∘ ]	[cm/s]	ratio
3 trackers	3.67	1.57	6.71	5.43	17.79	1.74
3 trackers + 3D pose (ours)	2.35	1.23	3.97	4.62	16.20	2.40
Protocol 2

Table 3: Full-body pose estimation accuracy on both protocols, comparing the 3-tracker baseline (HMD and two controllers) with our method, which adds external 3D pose observations. Position errors are reported for the full body (MPJPE), upper body (U-PE) and lower body (L-PE), together with the joint rotation error (MPJRE), joint velocity error (MPJVE) and jitter ratio.

\downarrow

: lower is better; best results in bold.

Figure 7: Samples of full-body pose estimated from ( top to bottom ): 3 trackers; 3 trackers + Pose estimation; ground truth. Incorporating 3D positions from external tracking improves the of the full-body pose estimation, especially at lower-body parts.

Panel of Figure 8 — Figure 8: Poses estimated by our proposed framework ( top ) and from baseline, relying on HMD and controllers solely ( bottom ). The green spheres represent the positions $p^{t}_{J_{ext}}$ of body joints estimated from RGB videos (except for the axis system attached to the green sphere, which corresponds to $\mathcal{H}_{f}$ ). Our model is able to follow the user’s motion and improves lower-body tracking compared to full-body pose estimation from sparse tracking alone.

Virtual Reality

Virtual reality studies immersive, interactive environments and the technologies and human experiences that make them useful.

See all 45 papers

FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices
Oct 12, 2026Chenfeng Gao, Taeyoung Yeon, Sungheon Park +4Human Avatar Animation Human Pose Estimation
Northwestern University · University of Chicago · Google
Everybody Tracking Every Body
Aug 30, 2026Daeyun Shin, Yunhan Zhao, Shu Kong +23D Human Pose Estimation Diffusion Models
University of California, Irvine · University of Macau
MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement
Mar 29, 2026Yejia Liu, Hengle Jiang, Haoxian Liu +23D Human Pose Estimation Disentangled Representation Learning
Department of Computer Science and Engineering, Hong Kong University of Science and Technology

Oct 12, 2026J/K move · Enter open · S save

FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices

Chenfeng Gao, Taeyoung Yeon, Sungheon Park, Vasco Xu+3

Northwestern University · University of Chicago · Google

Human Avatar Animation Human Pose Estimation

Open paper AskPDF arXiv

Prior avatar embodiment research has studied tracking completeness (how faithfully the body, hands, and face are sensed and reproduced) and personalized appearance in isolation, each requiring specialized lab equipment. How these factors interact on consumer hardware remains an open question, because no existing system unifies body pose, face, hands, and personalized appearance on a standalone headset. We present FlowAvatar, a system enabling real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs on consumer XR devices (Meta Quest Pro). Our pose model achieves competitive spatial accuracy and state-of-the-art temporal consistency on the AMASS benchmark against prior works with comprehensive spatial and temporal losses validated through architecture and loss ablations, while running in real-time. Two user studies (N=12, N=10) provide converging evidence that tracking completeness and personalized appearance serve complementary roles: tracking completeness broadly improves perceived motion quality, while only their combination achieves positive total embodiment.

Abstract

Virtual Reality

Explore similar work

FlowAvatar: Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices

Everybody Tracking Every Body

MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement