cs.CVMar 4, 2026
SaveEgoPoseFormer v2: Accurate Egocentric Human Motion Estimation for AR/VR
Organizations: Meta · KAUST · Max Planck Institute for Intelligent Systems
Abstract
Egocentric human motion estimation is essential for AR/VR experiences, yet remains challenging due to limited body coverage from the egocentric viewpoint, frequent occlusions, and scarce labeled data. We present EgoPoseFormer v2, a method that addresses these challenges through two key contributions: (1) a transformer-based model for temporally consistent and spatially grounded body pose estimation, and (2) an auto-labeling system that enables the use of large unlabeled datasets for training. Our model is fully differentiable, introduces identity-conditioned queries, multi-view spatial refinement, causal temporal attention, and supports both keypoints and parametric body representations under a constant compute budget. The auto-labeling system scales learning to tens of millions of unlabeled frames via uncertainty-aware semi-supervised training. The system follows a teacher-student schema to generate pseudo-labels and guide training with uncertainty distillation, enabling the model to generalize to different environments. On the EgoBody3M benchmark, with a 0.8 ms latency on GPU, our model outperforms two state-of-the-art methods by 12.2% and 19.4% in accuracy, and reduces temporal jitter by 22.2% and 51.7%. Furthermore, our auto-labeling system further improves the wrist MPJPE by 13.1%.
Figures & tables
Figure 2: Architecture overview (left). We stack two transformer decoders for coarse-to-fine pose estimation. A single holistic query, initialized from auxiliary metadata, attends to multi-view features and historic information to estimate 3D keypoints, pose parameters, and per-joint uncertainty in an end-to-end differentiable architecture. Illustration of the two core attention modules (right) . Causal temporal attention enables each frame to attend to its temporal history. Conditioned multi-view cross attention incorporates both view identity and optional 2D keypoint projections of pose proposal to guide spatial feature aggregation across views.
Figure 3: Per-keypoint uncertainty predicted by EPFv2. Larger ellipse extent and higher transparency indicate higher predicted uncertainty. Prediction is in green whereas GT is in red .
Figure 4: Overview of the mixture training in auto-labeling system. We adopt a stronger teacher model for pesudo labeling and apply an uncertainty distillation loss to facilitate the knowledge transfer. The teacher model is pre-trained with the labeled dataset before this stage.
| Method | Reference | Overall | Overall Wrists | Overall Shoulders | Overall Legs | Overall Feet | |
| MPJPE | MPJVE | MPJPE | MPJPE | MPJPE | MPJPE | ||
| UnrealEgo ‡ [ 5 ] | ECCV2022 | 7.41 ({\color[rgb]{0.72,0,0}+84.3\%}) | 1.27 ({\color[rgb]{0.72,0,0}+202.4\%}) | ||||
| Egobody3M † [ 72 ] | ECCV2024 | 5.18 ({\color[rgb]{0.72,0,0}+28.9\%}) | 0.54 ({\color[rgb]{0.72,0,0}+28.6\%}) | 6.14 ({\color[rgb]{0.72,0,0}+23.0\%}) | 2.80 ({\color[rgb]{0.72,0,0}+19.9\%}) | 8.40 ({\color[rgb]{0.72,0,0}+26.1\%}) | 10.25 ({\color[rgb]{0.72,0,0}+17.9\%}) |
| EgoPoseFormer † [ 65 ] | ECCV2024 | 4.75 ({\color[rgb]{0.72,0,0}+18.1\%}) | 0.87 ({\color[rgb]{0.72,0,0}+107.1\%}) | 6.01 ({\color[rgb]{0.72,0,0}+20.4\%}) | 2.72 ({\color[rgb]{0.72,0,0}+16.5\%}) | 7.95 ({\color[rgb]{0.72,0,0}+19.4\%}) | 10.16 ({\color[rgb]{0.72,0,0}+17.0\%}) |
| EPFv2 w/o ALS | - | 4.17 ({\color[rgb]{0.72,0,0}+3.2\%}) | 0.42 ({\color[rgb]{0.5,0.5,0.5}+0.0\%}) | 5.74 ({\color[rgb]{0.72,0,0}+15.3\%}) | 2.38 ({\color[rgb]{0.72,0,0}+2.1\%}) | 6.91 ({\color[rgb]{0.72,0,0}+3.8\%}) | 9.11 ({\color[rgb]{0.72,0,0}+4.8\%}) |
| EPFv2 with ALS | - | 4.02 | 0.42 | 4.99 | 2.33 | 6.66 | 8.69 |
Table 1: Quantitative comparison on Egobody3M benchmark. The percentage in parentheses denotes how much worse each baseline is compared to our method. † : indicates our reproduced results. ‡ : presents results reported in the Egobody3M paper with missing metric numbers filled in with . ALS is short for our Auto-Labeling System.
Figure 5: Qualitative results on Egobody3M. Predictions are colored in green and ground-truths are colored in orange .
| # | Variants | Overall | Wrists | Legs |
| ① | use kpt head | 4.35 | 6.02 | 7.17 |
| ② | w/o temporal attention | 4.35 | 6.04 | 7.21 |
| ③ | w/o projection condition | 4.30 | 5.96 | 7.15 |
| ④ | w/o auxiliary information | 4.39 | 5.98 | 7.34 |
| ⑤ | w/o uncertainty | 4.25 | 5.83 | 7.00 |
| ⑥ | EPFv2 w/o ALS | 4.17 | 5.74 | 6.91 |
Table 2: Ablation study. We remove or modify each module to assess its contribution. These results demonstrate that each component yields a measurable improvement in accuracy and our auto-labeling system further boosts the model performance.
Figure 6: ALS effectiveness in in-domain scaling. As more unlabeled data is used, both students achieve improved accuracy. Notably, the MobileNetv4-S-based model benefits more proportionally from ALS despite having lower model capacity, indicating the pipeline’s suitability for lightweight deployment models.
Figure 8: Our simplified multi-view cross-attention module built on standard attention. A single holistic query attends to all view features using conditioned positional encoding in a batch manner, enabling more efficient and scalable spatial fusion. V denotes the number of views. Learnable layers are highlighted in red , with naming aligned to Tab. 3 .
| Method | Layer | # of params | FLOPs |
| (1) EPFv1 Deformable Attention w/ 16 Queries w/ 16 reference points | sampling offsets | 16512 | 32768 |
| attention weights | 8256 | 16384 | |
| value projection | 16512 | 32768 | |
| output projection | 16512 | 32768 | |
| fuse mlp | 32896 | 32768 | |
| norm | 256 | 0 |
Table 3: Layer-wise comparison of parameter count and FLOPs for different spatial attention modules. (1) EPFv1 uses deformable attention with 16 joint queries and reference points, introducing moderate compute and high development complexity. (2) A baseline standard attention setup with 16 queries shows 4.75 higher FLOPs due to dense spatial interactions. (3) Our final design adopts a single holistic query and standard attention, achieving the lowest computation (131K FLOPs) while maintaining similar parameter count to other variants.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | # of params | FLOPs |
| EgoBody3M [ 72 ] | 14.96M | 39.76G |
| EPFv2 | 0.83M | 10.52G |
Table 4: Efficiency Comparison with EgoBody3M. We compare only the pose estimation heads, assuming a shared backbone. EPFv2 significantly reduces parameter count and FLOPs, demonstrating the efficiency of its streamlined transformer design.
| Backbone | FP16 | Plain PyTorch | ONNX CPU | TensorRT lvl. 0 | TensorRT lvl. 5 |
| ResNet18 | 9.8 ms | 59.2 ms | 3.3ms | 1.3ms | |
| ✓ | 8.3 ms | – | 1.6ms | 1.0ms | |
| MobileNetV4S | 14.9 ms | 23.8 ms | 3.0ms | 1.1ms | |
| ✓ | 13.3 ms | – | 1.3ms | 0.8ms |
Table 5: Latency measurement of EPFv2.
| Method | MPJPE | PA-MPJPE |
| EgoPoseFormer [ 65 ] | 27.36 | 23.31 |
| EgoRear [ 7 ] | 27.04 | 23.18 |
| EPFv2 | 27.94 | 22.53 |
Table 6: Comparison on Ego4View-Syn [ 7 ] . We evaluate a single-frame variant of EPFv2 on the Ego4View-Syn dataset. Despite removing temporal modeling, our method achieves strong results, outperforming prior works in PA-MPJPE and remaining competitive in MPJPE.
| Length | Overall MPJPE |
| 2 | 4.30 |
| 16 (default) | 4.17 |
Table 7: Impact of temporal sequence length. Using only two frames leads to significant performance degradation, while longer sequences offer stable and improved accuracy. We use 16 as the default setting for fair comparison with prior work [ 72 ] .
| Variants | # of params. | Overall MPJPE |
| ResNet-18 [ 24 ] | 12.5M | 4.17 |
| ResNet-50 [ 24 ] | 43.3M | 4.03 |
| DINOv3-ViT-B [ 50 ] | 119.2M | 3.96 |
Table 8: Comparison across different image backbones. Larger backbones, such as DINOv3-B, significantly improve pose estimation accuracy, demonstrating the benefit of strong visual features in egocentric settings.