Egocentric Visibility-Aware Human Pose Estimation

Peng Dai; Yu Zhang; Yiqiang Feng; Zhen Fan; Yang Zhang

cs.CVFeb 27, 2026

Egocentric Visibility-Aware Human Pose Estimation

Authors: Peng Dai, Yu Zhang, Yiqiang Feng, Zhen Fan, Yang Zhang

Organizations: PICO

Abstract

Egocentric human pose estimation (HPE) using a head-mounted device is crucial for various VR and AR applications, but it faces significant challenges due to keypoint invisibility. Nevertheless, none of the existing egocentric HPE datasets provide keypoint visibility annotations, and the existing methods often overlook the invisibility problem, treating visible and invisible keypoints indiscriminately during estimation. As a result, their capacity to accurately predict visible keypoints is compromised. In this paper, we first present Eva-3M, a large-scale egocentric visibility-aware HPE dataset comprising over 3.0M frames, with 435K of them annotated with keypoint visibility labels. Additionally, we augment the existing EMHI dataset with keypoint visibility annotations to further facilitate the research in this direction. Furthermore, we propose EvaPose, a novel egocentric visibility-aware HPE method that explicitly incorporates visibility information to enhance pose estimation accuracy. Extensive experiments validate the significant value of ground-truth visibility labels in egocentric HPE settings, and demonstrate that our EvaPose achieves state-of-the-art performance in both Eva-3M and EMHI datasets.

Figures & tables

Figure 1: We introduce Eva-3M, a large-scale egocentric visibility-aware dataset comprising over 3.0M frames from (a) 31 subjects in daily outfits (b) performing 24 types of daily actions, and of which (c) 435K are annotated with keypoint visibility labels. (d) shows the normalized spatial distribution comparison between Eva-3M and EMHI, indicating that Eva-3M has a wider range of motion diversity than EMHI. (e) illustrates a few representative examples of keypoint invisibility due to self-occlusion and out-of-FoV in Eva-3M dataset.

Dataset	R/S	Cams	Frames	Vis	SMPL	Subj	Act
Mo 2 Cap 2 [ 46 ]	Syn.	1	530K	$\times$	$\times$	700	3K
xR-EgoPose [ 39 ]	Syn.	1	383K	$\times$	$\times$	46	-
UnrealEgo [ 4 ]	Syn.	2	450K	$\times$	$\times$	17	30
EgoGTA [ 44 ]	Syn.	1	320K	$\times$	$\times$	5	-
SynthEgo [ 10 ]	Syn.	2	60K	$\times$	✓	6K	-
EgoCap [ 35 ]	Real	2	75K	$\times$	$\times$	8	-

Table 1: Comparison of egocentric motion datasets. R/S denotes whether the dataset is collected in real or synthetic setting. Cams indicates the number of egocentric cameras. Vis denotes the availability of keypoint visibility labels. Subj is the number of subjects, and Act is the number of action categories.

Figure 2: Overview of the proposed EvaPose. Given a sequence of egocentric observations, we first propose a visibility-aware 3D pose estimation network to extract stereo image features, and predict per-frame 3D keypoints in the camera coordinate system (defined as the left camera coordinate system in this paper) and their corresponding visibility confidence scores. Then, the predicted 3D keypoints are transformed to the canonical coordinate system with the help of camera poses from SLAM system. Next, an iterative intra-and inter-frame attention network is used for temporal feature fusion. Finally, we estimate the 3D poses with the pre-trained VQ-VAE decoder.

Dataset	Methods	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	U-PE $\downarrow$	L-PE $\downarrow$	FootPE $\downarrow$	HandPE $\downarrow$	Jitter $\downarrow$	FPS $\uparrow$
	UnrealEgo [ 4 ]	60.7	38.8	46.5	81.2	104.9	86.6	7.6	58.9
Eva-3M	EgoPoseFormer [ 48 ]	52.9	33.7	40.6	70.5	93.4	79.2	15.4	67.0
	FRAME [ 7 ]	49.8	35.1	42.4	60.5	77.4	80.2	3.1	108.7
	EvaPose-ResNet50 (Ours)	35.6	24.7	28.3	46.1	58.4	49.6	3.0	48.0
	EvaPose-ViT-L (Ours)	34.2	24.0	27.2	44.5	56.4	49.2	3.2	9.4
	UnrealEgo [ 4 ]	55.2	38.6	39.9	77.3	100.2	85.2	5.9	58.9

Table 2: Comparison with state-of-the-art methods on the Eva-3M and EMHI datasets. The best and second best results are highlighted in boldface and underlined , respectively.

Methods	Ankle		Foot		Elbow		Wrist		Mean
Methods	Visible	Invisible	Visible	Invisible	Visible	Invisible	Visible	Invisible	Visible	Invisible
UnrealEgo [ 4 ]	83.0	87.1	91.3	106.8	52.0	63.0	67.0	85.8	73.3	85.7
EgoPoseFormer [ 48 ]	78.2	94.2	83.9	96.4	49.1	63.2	57.9	101.0	67.3	88.7
FRAME [ 7 ]	77.8	79.1	81.2	92.5	53.0	59.5	67.9	86.5	70.0	79.4
EvaPose-ResNet50	52.2	66.4	60.8	82.5	32.2	48.0	38.0	65.7	45.8	65.6
EvaPose-ViT-L	49.2	63.8	57.2	79.4	29.5	45.6	34.2	63.3	42.5	63.0

Table 3: Performance comparisons of visible and invisible keypoints on the Eva-3M dataset.

Figure 3: Qualitative comparisons on both EMHI (the first two rows) and Eva-3M (the last two rows) datasets. For better illustration, the predicted (red) and ground-truth (green) 3D poses are re-projected onto external reference views which are not used for pose estimation.

Method	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	VLK-PE $\downarrow$	ILK-PE $\downarrow$
w/o visibility	40.6	27.9	53.5	65.3
with visibility	35.6	24.7	45.8	65.6

Table 4: Impact of the visibility-aware model and loss design. VLK-PE is the MPJPE for visible limb keypoints, including ankle, foot, elbow and wrist. ILK-PE denotes the MPJPE for invisible limb keypoints.

TTE	STD	VQ-VAE	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Jitter $\downarrow$
✗	✗	✗	46.0	33.3	4.7
✓	✗	✗	40.3	27.0	2.5
✓	✓	✗	39.1	25.9	2.7
✓	✓	✓	35.6	24.7	3.0

Table 5: Ablation study for key components in our model.

Figure 4: Visualization of some representative fitted SMPL meshes.

Figure 5: Representative samples of the keypoint visibility labels in our Eva-3M dataset.

Dataset	Methods	Backbone	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Jitter $\downarrow$
	UnrealEgo [ 4 ]	ResNet18	289.7	171.0	60.4
EMHI: P1	EgoPoseFormer [ 48 ]	ResNet18	180.5	112.5	70.3
	FRAME [ 7 ]	ResNet50	135.6	126.7	12.9
	EvaPose-ResNet50 (Ours)	ResNet50	93.3	59.5	5.0
	EvaPose-ViT-L (Ours)	ViT-Large	49.5	38.4	7.6
	UnrealEgo [ 4 ]	ResNet18	244.9	148.5	40.4

Table 6: Cross-dataset evaluation results. In this experiment, all methods are trained on the Eva-3M dataset and tested on the EMHI dataset.

Figure 6: Overview of the coordinate systems.

Explore similar work

EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR
Mar 4, 2026Zhenyu Li, Sai Kumar Dwivedi, Filip Maric +11Human Pose Estimation 3D Human Pose Estimation
Meta · KAUST · Max Planck Institute for Intelligent Systems
Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands
Aug 12, 2026Ryosei Hara, Masashi Hatano, Rintaro Yanagi +3
Keio University · National Institute of Advanced Industrial Science and Technology (AIST) · The University of Tokyo +1
MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement
Mar 29, 2026Yejia Liu, Hengle Jiang, Haoxian Liu +23D Human Pose Estimation Disentangled Representation Learning
Department of Computer Science and Engineering, Hong Kong University of Science and Technology

Mar 4, 2026J/K move · Enter open · S save

EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR

Zhenyu Li, Sai Kumar Dwivedi, Filip Maric, Carlos Chacon+10

Meta · KAUST · Max Planck Institute for Intelligent Systems

Human Pose Estimation 3D Human Pose Estimation Markerless Motion Capture Transformer Semi-Supervised Learning

Open paper AskPDF arXiv

Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%.

Abstract

Explore similar work

EgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR

Hand Visibility Detector: Per-Keypoint Visibility Estimation for Hands

MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement