EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations

Yaoqing Hu; Xuguo He; Daohan Cui; Yukun Shi; Kai Chen

cs.CVOct 12, 2026

EgoWholeView: Camera-Only Head-Mounted Capture of Body and Bimanual Hand Motion for Embodied Demonstrations

Authors: Yaoqing Hu, Xuguo He, Daohan Cui, Yukun Shi, Kai Chen

Organizations: Zhongguancun Academy · DeepCybo · Beijing University of Chemical Technology · Zhongguancun Institute of Artificial Intelligence

Abstract

Head-mounted body and bimanual motion capture supports self-avatars, teleoperation, and embodied demonstrations. However, a single egocentric camera is vulnerable to a limited field of view, scale ambiguity, and hand-object occlusion. We present EgoWholeView, a wearable capture system of six synchronized wide-angle RGB cameras with a role-specialized pipeline. The outward views localize the rig in a world frame, while an overlapping downward pair independently predicts SMPL-X body and MANO hand parameters. Both predictors are adapted to egocentric fisheye data. To combine the two, reliability cues measured from the predictions themselves weight the fusion of body and hand joint rotations, while shape is aggregated across views and frames. The pair's baseline fixes metric scale, and triangulation recovers root translation. Relative to monocular prediction, fusion reduces PA-MPJPE by 3.7% for the body and 11.2% for the hands, and temporal refinement lowers angular-acceleration magnitude by 32.2% and 31.0%, respectively. On our six-camera recording, the pipeline produces body and hand states in a shared world frame using benchmark-fitted gates without retuning, and temporal refinement lowers hand acceleration magnitude by 41.08%. Using only RGB cameras and no third-person view, EgoWholeView offers a low-cost, portable route to egocentric interaction data.

Figures & tables

Figure 1: Overview of EgoWholeView. Six synchronized cameras on a wearable rig provide observations for continuous outward-view multi-camera trajectory estimation and downward-view body and bimanual pose estimation. Paired downward-view predictions undergo reliability-aware fusion in the canonical space, followed by temporal refinement of local rotations. The system exports world-frame body and bimanual motion.

Output	Representation / frame	Cam.	Step
Rig pose $\bm{T}^{w}_{\mathrm{rig},t}$	$SE(3)$ , rig to world	1/2, 5/6	multi-view SLAM
Body $X_{t}^{B}$	SMPL-X, root relative	3/4	gated fusion + temporal
Hand $X_{t}^{h}$	MANO, root relative	3/4	gated fusion + temporal
Placement $\mathbf{P}_{t}$	root $SE(3)$ , Cam3 frame	3/4	$SO(3)$ orientation fusion + stereo triangulation

Table 1: System outputs, representations, coordinate frames, and processing.

Panel of Figure 2 — Figure 2: Six-camera wearable prototype and task-specialized layout. (a) Physical prototype. (b) Side view: the forward pair (Cam1/2) and the overlapping downward pair (Cam3/4). (c) Front view: the lateral pair (Cam5/6) and the downward-pair baseline $b$ . (d) Downward-view evidence coverage; full-body prediction extends beyond the directly observed regions.

Figure 3: Processing overview. The outward branch recovers the rig trajectory; the downward branch predicts body and hand parameters per view. Paired predictions are fused in the canonical space: local rotations are weighted per joint, shape is aggregated across views and frames. The fused rotations are temporally refined and the roots placed metrically by triangulation.

Figure 4: Per-view prediction and egocentric-domain adaptation. Body and hand crops from each downward view are processed by separate SMPL-X and MANO regressors, with branch-specific weights shared between Cam3 and Cam4. The loss boxes apply during adaptation only. At inference, paired Cam3/Cam4 hypotheses enter the independently parameterized body or hand fusion module in Section 4.3 .

Figure 5: Reliability-aware parametric fusion. (a) Gate structure, shared by the body and hand branches but fitted with independent parameters; every normalized cue and aggregate score follows the higher-is-more-reliable convention. (b)–(c) Per-joint view preference and the resulting weights on a MANO hand.

Cue	Reliable when	Body	Hand
Visibility $V$	Higher	0.3228	0.3716
Perturbation dispersion $U$	Lower	0.0912	0.3243
Crop-boundary support $B$	Higher	0.1109	0.0029
Temporal inconsistency $T$	Lower	0.4551	0.2950
Feature dispersion $F$	Lower	0.0201	0.0062
Gate temperature $\tau_{\mathrm{gate}}$	—	0.1245	0.2285

Table 2: Learned normalized cue weights, raw-cue reliability directions, and gate temperatures for the body and hand gates.

Figure 6: Qualitative body local-pose fusion on UnrealEgo-RW. From left to right: stereo inputs with projected monocular SMPL-X predictions, left- and right-view predictions, equal-weight fusion, and VUBTF soft fusion. Predictions are independently Procrustes-aligned to GT15 (gray dashed), and values report PA-MPJPE in mm ( $\downarrow$ ). Yellow joints and white wireframes show the projected predictions; dotted ankle–foot links indicate the heel–toe correspondences used for GT15 evaluation. The examples have monocular errors close to the test-set mean.

Method	Mean	Median	P90	P95	Gain vs. mono. avg.
Left-view prediction	79.6188	74.1331	102.4118	115.0452	—
Right-view prediction	80.1623	74.5614	101.6197	116.6663	—
Monocular average	79.8905	75.1588	101.2589	112.1287	0.0000
Equal-weight fusion	77.3151	72.9845	96.8666	106.8710	+2.5754
VUBTF soft fusion	76.9409	72.6824	96.0671	106.3893	+2.9496

Table 3: SMPL-X fusion on the UnrealEgo-RW test split. PA-MPJPE uses GT15 and is given in mm.

PA-MPJPE (21 joints)
Method	Mean	Median	P90	P95	Gain vs. mono. avg.
Mono A	4.9092	4.3793	7.7093	9.1141	—
Mono B	4.7478	4.2562	7.4034	8.8242	—
Monocular average	4.8285	4.4214	7.2355	8.4284	0.0000
Equal-weight fusion	4.3642	3.9486	6.7328	7.9130	+0.4643
VUBTF soft fusion	4.2872	3.8732	6.6027	7.7682	+0.5413

Table 4: Dual-view MANO fusion on DexYCB S0 test and frozen transfer to HO3Dv3 (mm). PA-MPJPE uses 21 joints and PA-MPVPE all 778 MANO vertices.

Figure 7: Qualitative dual-view MANO fusion on DexYCB. (a)–(b) Synchronized views with monocular predictions and enlarged hand crops. (c) Ground-truth joints (GT), equal-weight fusion, and VUBTF soft fusion. (d) Per-joint view weights for proximal (Prox.), middle (Mid.), and distal (Dist.) finger joints.

Method	PA- MPJPE	Angular- velocity error	Angular- acceleration error	Acceleration magnitude
VUBTF observation	4.2872	6.1053	8.1901	2.2269
Fixed gain, $\kappa=0.75$	4.2687	5.9673	7.8929	1.4968
Adaptive gain	4.2621	5.9601	7.8861	1.5356

Table 5: Temporal-refinement results on DexYCB S0 test. PA-MPJPE is in mm; angular metrics use ∘ /frame for velocity and ∘ /frame 2 for acceleration.

Figure 8: World-frame rig trajectory and sparse map estimated from the four outward views, colored by time over the 35.7-s sequence. Sampled axes show rig orientation.

Figure 9: Downward-pair output: no occlusion, hand–object interaction, hand–object occlusion, and one hand out of view. SMPL-X body (white) and MANO hands (blue left, red right) are reprojected into Cam3 and Cam4. The right hand in case (3) and the left hand in case (4) are seen in one view only, so they are single-view predictions.

Explore similar work

MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild
Sep 15, 2026Jiangong Xiao, Zhihao Zhang, Yifei Dong +7Markerless Motion Capture 3D Hand Pose Estimation
Northwestern Polytechnical University · Xi'an Jiaotong University · Xi’an Jiaotong University +1
WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos
Feb 25, 2026Yufei Ye, Jiaman Li, Ryan Rong +13D Reconstruction Egocentric Video Understanding
Stanford University · Amazon FAR (Frontier AI & Robotics)
Egocentric Whole-Body Human Mesh Recovery with Prior-Guided Learning
May 9, 2026Soyeon Na, Seung Young Noh, Ju Yong Chang3D Reconstruction Human Mesh Recovery
Dept of ECE, Kwangwoon University, Seoul, Korea

Sep 15, 2026J/K move · Enter open · S save

MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild

Jiangong Xiao, Zhihao Zhang, Yifei Dong, Chao Ma+6

Northwestern Polytechnical University · Xi'an Jiaotong University · Xi’an Jiaotong University+1

Markerless Motion Capture 3D Hand Pose Estimation Multi-View 3D Reconstruction

Open paper AskPDF arXiv

Learning manipulation from human video requires high-fidelity hand-motion reconstruction in metric units. Today's metric hand labels come from studio rigs and instrumented headsets, and both are confined in the same two ways: neither leaves a prepared setting, and neither is checked against an independent reference. Unconstrained head-worn recording promises the opposite trade-off, scaling with the number of people wearing a device. We therefore introduce MEgoVista, an offline pipeline that turns a single unprepared MEgo View recording into metric two-hand and head motion in one gravity-aligned world frame. Three properties set it apart from existing egocentric reconstruction systems: first, it reconstructs in settings studio volumes and tabletop rigs cannot reach, settling hand ownership at detection so bystander hands stay out of the wearer's trajectory; second, it takes its metric gauge from calibrated stereo rather than a monocular prior, installing scale at initialisation so policies receive physical units, not arbitrary coordinates; third, both outputs are scored inside a motion-capture volume against independent Chingmu optical capture, under a protocol that audits its own reference and charges what a method declines to predict. MEgoVista is offered as a measured route from egocentric video to metric hand supervision, one that widens where such labels can be gathered.

Abstract

Explore similar work

MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild

WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos

Egocentric Whole-Body Human Mesh Recovery with Prior-Guided Learning