E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

Mayur Deshmukh; Hiroyasu Akada; Helge Rhodin; Christian Theobalt; Vladislav Golyanik

cs.CVApr 9, 2026

E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

Authors: Mayur Deshmukh, Hiroyasu Akada, Helge Rhodin, Christian Theobalt, Vladislav Golyanik

Organizations: MPI for Informatics, SIC · Saarland University, SIC · Bielefeld University

Abstract

Event cameras offer multiple advantages in monocular egocentric 3D human pose estimation from head-mounted devices, such as millisecond temporal resolution, high dynamic range, and negligible motion blur. Existing methods effectively leverage these properties, but suffer from low 3D estimation accuracy, insufficient in many applications (e.g., immersive VR/AR). This is due to the design not being fully tailored towards event streams (e.g., their asynchronous and continuous nature), leading to high sensitivity to self-occlusions and temporal jitter in the estimates. This paper rethinks the setting and introduces E-3DPSM, an event-driven continuous pose state machine for event-based egocentric 3D human pose estimation. E-3DPSM aligns continuous human motion with fine-grained event dynamics; it evolves latent states and predicts continuous changes in 3D joint positions associated with observed events, which are fused with direct 3D human pose predictions, leading to stable and drift-free final 3D pose reconstructions. E-3DPSM runs in real-time at 80 Hz on a single workstation and sets a new state of the art in experiments on two benchmarks, improving accuracy by up to 19% (MPJPE) and temporal stability by up to 2.7x. See our project page for the source code and trained models.

Figures & tables

Figure 1: Rethinking event-based egocentric 3D human pose estimation. (a) Previous methods [ 26 , 25 ] capture temporal information only through a single previous event frame stored in the frame buffer leading to jitter and drift. (b) Our E-3DPSM approach models motion as a continuous event-driven state evolution, fusing delta and direct 3D human pose updates, thereby achieving real-time and temporally stable 3D reconstruction and significantly outperforming prior approaches in the 3D accuracy.

Figure 2: Overview of the proposed E-3DPSM approach for monocular egocentric 3D human pose estimation. Incoming raw events $e$ are converted into LNES frames $\mathbf{L}_{t}$ and processed by the Spatiotemporal Pose Encoder Module (SPEM, Sec. 4.1 ), as depicted in Fig. 3 . The output of SPEM $\mathbf{F}_{t}$ is passed to the Pose Regression Module (PRM, Sec. 4.2 ), which estimates both the direct 3D poses $\mathbf{P}^{D}_{t}$ and the 3D delta poses $\mathbf{P}^{\Delta}_{t}$ between consecutive LNES frames. These delta poses naturally correspond to the observed events (i.e., changes in the 2D LNES space). Finally, the 3D pose deltas are fused with the direct 3D pose using a learned pose fusion mechanism (a neural Kalman-style filter) to obtain the final 3D poses $\mathbf{P}_{t}$ . The predicted 3D pose $\mathbf{P}_{t}$ is supervised using absolute ground-truth 3D poses ( $\mathbf{\mathcal{L}_{\text{3D}}}$ ) and ground-truth 3D pose differences ( $\mathcal{L}_{\Delta}$ ), respectively. At each timestep $t$ , SPEM updates and propagates its latent state $\mathbf{Z}_{t}$ bidirectionally (the “ $\longleftrightarrow$ ” symbol), whereas PRM propagates its fused pose state $\mathbf{X}_{t}$ causally (the “ $\longrightarrow$ ” symbol) to the next timestep. During training, SPEM operates bidirectionally (non-causal) to exploit the full temporal context, while at inference, it can run in either causal (real-time) or non-causal mode. For visualisation purposes, the differences between $\textbf{P}_{1}$ and $\textbf{P}_{2}$ and the corresponding $\textbf{L}_{1}$ and $\textbf{L}_{2}$ are exaggerated.

Figure 3: Architecture of SPEM , combining multi-stage convolutional encoding, SSM blocks, deformable attention, and a joint-query decoder for temporally-aware pose features.

Figure 4: Qualitative comparison of our method with prior approaches. We compare against EgoPoseFormer [ 44 ] , EventEgo3D [ 26 ] , and EventEgo3D++ [ 25 ] . Left: EE3D-R (real dataset). Right: EE3D-W (in-the-wild). Red: Predicted pose. Green: Ground truth.

EE3D-R	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}$ $\downarrow$
EgoPoseFormer [ 44 ]	151.66	96.99	66.50
EventEgo3D [ 26 ]	110.39	84.52	27.06
EventEgo3D++ [ 25 ]	103.28	77.06	22.93
Ours (Causal)	84.45	62.64	8.40
Ours (Non-Causal)	81.32	60.21	6.65
EE3D-W	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}$ $\downarrow$

Table 1: Quantitative comparisons on EE3D-R and EE3D-W .

Figure 5: We plot the per-frame all-joint average displacement (Eq. ( 24 )) for EE3D-R (top) and EE3D-W (bottom).

EE3D-R	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
EgoPoseFormer [ 44 ]	110.18	55.64
EventEgo3D [ 26 ]	86.90	50.93
EventEgo3D++ [ 25 ]	88.43	49.53
Ours (Causal)	67.49	41.85
Ours (Non-Causal)	64.69	40.10
EE3D-W	MPJPE $\downarrow$	PA-MPJPE $\downarrow$

Table 2: Occlusion-only quantitative comparison on EE3D-R and EE3D-W . Evaluation is performed only on occluded joints.

SPEM (Sec. 4.1 )	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}$ $\downarrow$
w/o SSM Blocks	118.53	87.48	16.94
Single SSM Block (Stage 4)	90.18	66.06	7.84
w/o Deformable Attention	88.27	64.98	7.71
PRM (Sec. 4.2 )	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}$ $\downarrow$
w/o Fusion Module	141.22	84.17	10.14
Direct Pose Only	91.26	65.43	17.22

Table 3: Ablation study on the EE3D-R dataset evaluating the impact of each component of our E-3DPSM approach.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Pose drift over time. Comparison of learned fusion (Eq. ( 15 )), direct pose only (Eq. ( 8 )), and naive fusion (Eq. ( 11 )) across temporal sequence length. Naive fusion leads to rapidly increasing drift, whereas our learned fusion effectively mitigates this drift, maintaining stable accuracy over time.

Method	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}$ $\downarrow$
EgoPoseFormer [ 44 ] with KF	144.27	92.32	37.11
EventEgo3D [ 26 ] with KF	107.23	82.54	15.78
EventEgo3D++ [ 25 ] with KF	100.98	75.57	13.98
Ours (Causal)	84.45	62.64	8.40
Ours (Non-Causal)	81.32	60.21	6.65

Appendix

Table 4: Comparison with Kalman-smoothed baselines on the EE3D-R dataset. We apply inference-time Kalman filtering (KF) to prior methods to rule out post-hoc smoothing as the main reason for improvements. Our method achieves substantially lower MPJPE and

e_{\text{smooth}}

, demonstrating that the performance is due to the proposed architecture and not filtering in post-processing.

Method	Params $\downarrow$	FLOPs $\downarrow$	GPU Memory $\downarrow$	Pose Update Rate $\uparrow$
EgoPoseFormer [ 44 ]	14.1 M	5.5 G	82 MB	130
EventEgo3D [ 26 ]	1.25 M	416.84 M	25 MB	139
EventEgo3D++ [ 25 ]	1.25 M	416.84 M	25 MB	139
Ours	6.64 M	8.16 G	74 MB	80

Appendix

Table 5: Model efficiency comparison in terms of parameters, FLOPs, GPU memory, and 3D pose update rate in Hz (measured on a single NVIDIA A6000 GPU).

Modules	FLOPs
CNN Layers	7.05 G
Deformable Attention	0.89 G
SSM	0.06 G
Query Decoder	0.15 G
Pose Heads	$10^{-3}$ G
Fusion	$10^{-4}$ G

Appendix

Table 6: Detailed module-wise FLOPs breakdown.

Training Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}\downarrow$
Training Directionality
Causal (Forward Only)	89.88	66.74	10.14
Non-Causal (Ours)	84.45	62.64	8.40
Pose Sequence Length ( $\boldsymbol{\mathbf{N}}$ )
20 poses	86.25	65.62	9.76
30 poses	86.03	64.87	8.95

Appendix

Table 7: Training strategy ablation on the EE3D-R dataset. We compare causal (forward) vs. non-causal (bidirectional) training and different sequence lengths used during training.

State Reset Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}\downarrow$
SSM Reset (40 Frames)	104.56	81.24	15.20
Fusion Reset (40 Frames)	95.40	70.90	21.50
No Reset (Ours)	84.45	62.64	8.40

Appendix

Table 8: Inference-time ablation on the EE3D-R dataset comparing different strategies for resetting internal states. We evaluate resetting the SSM block states, resetting the Kalman fusion states, and using continuous state evolution without resets (ours).

Event Representation	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}\downarrow$
Learned Voxel-Based [ 7 ]	100.25	78.54	10.78
Learned LNES	93.78	71.12	9.14
LNES (Ours)	84.45	62.64	8.40

Appendix

Table 9: Design choice study for event stream representation for learning on the EE3D-R dataset. We experiment with learned voxel-based representation, learned LNES and fixed LNES.

Event Frequencies	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	$e_{\textbf{smooth}}\downarrow$
20 Hz (50 ms)	87.45	63.74	11.90
25 Hz (40 ms)	84.57	62.80	9.49
50 Hz (20 ms) (Ours)	84.45	62.64	8.40

Appendix

Table 10: Inference-time ablation on the EE3D-R dataset comparing the use of different 3D pose update rates frequencies.

Learning Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
Input/State Dependent	91.15	69.74
Global Learned (Ours)	84.45	62.64

Appendix

Table 11: Ablation of global and input/state-dependent covariance learning for Q and R on EE3D-R dataset.

Figure 7: Our head-mounted device setup. The device uses a single fisheye egocentric event camera for input, NVIDIA Jetson Orin Nano for onboard processing, and a portable powerbank for standalone operation.

Figure 8: We plot the improvement in MPJPE obtained by increasing the duration of temporal history k, showing how a longer past context yields larger gains for occluded lower body joints.

Figure 9: Failure cases for different scenarios. (A) Strong self-occlusion crawl action, (B) interaction with objects, (C) other humans in the FOV. External views are only for reference. Red : Predicted pose. Green : Ground truth. C visualises our prediction only (no ground truth available). Inputs to E-3DPSM are egocentric LNES frames.

Figure 10: Our real-time viewer. Screenshot of our iPad-viewer showing the live event stream, reference RGB view, and the predicted 3D skeleton rendered in real time. Note that there is a transmission delay of 3–5 poses.

Dataset	Method	Metric $\downarrow$	Elbow	Wrist	Knee	Ankle	Foot	Avg.
EE3D-R	EgoPoseFormer [ 44 ]	MPJPE	47.85	28.34	100.11	204.64	169.94	110.18
	EgoPoseFormer [ 44 ]	PA-MPJPE	47.06	30.50	51.82	83.61	65.20	55.64
	EventEgo3D [ 26 ]	MPJPE	37.46	21.97	79.20	156.93	138.95	86.90
	EventEgo3D [ 26 ]	PA-MPJPE	40.41	25.78	48.58	78.27	61.63	50.93
	EventEgo3D++ [ 25 ]	MPJPE	34.41	20.87	81.73	158.51	146.62	88.43
	EventEgo3D++ [ 25 ]	PA-MPJPE	37.89	24.20	48.48	77.18	59.90	49.53

Appendix

Table 12: Quantitative results for occlusion-only end-effector joints for the EE3D-R and EE3D-W datasets .

Dataset	Method	Metric $\downarrow$	Walk	Crouch	Pushup	Boxing	Kick	Dance	Inter. w/ env.	Crawl	Sports	Jump	Avg.
EE3D-R	EgoPoseFormer [ 44 ]	MPJPE	123.65	175.60	184.12	150.45	125.99	119.71	153.75	226.87	143.84	136.98	154.09
	EgoPoseFormer [ 44 ]	PA-MPJPE	78.70	107.19	109.20	102.66	91.67	86.74	87.95	111.85	96.71	99.60	97.22
	EventEgo3D [ 26 ]	MPJPE	74.75	144.26	109.58	141.19	104.33	89.98	103.15	118.30	108.23	105.77	109.95
	EventEgo3D [ 26 ]	PA-MPJPE	55.92	97.73	87.40	109.70	85.13	72.10	72.63	85.87	84.91	87.31	83.86
	EventEgo3D++ [ 25 ]	MPJPE	71.52	150.57	93.54	125.75	98.46	85.45	96.10	116.76	98.71	97.10	103.39
	EventEgo3D++ [ 25 ]	PA-MPJPE	52.35	99.96	71.12	94.82	79.93	65.97	68.60	81.29	76.16	76.56	76.67

Appendix

Table 13: Per-action quantitative results for the EE3D-R and EE3D-W datasets .

Dataset	Method	Metric $\downarrow$	Head	Neck	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	Foot	Avg.
EE3D-R	EgoPoseFormer [ 44 ]	MPJPE	140.3	173.9	157.3	177.1	181.1	212.6	169.8	144.8	207.6	173.00
		PA-MPJPE	104.3	119.9	102.4	124.3	132.9	111.9	111.9	88.9	120.2	113.70
	EventEgo3D [ 26 ]	MPJPE	23.49	29.87	40.20	91.67	153.17	69.99	120.96	179.02	201.42	110.39
		PA-MPJPE	49.27	41.63	44.48	79.58	133.53	55.66	83.45	108.78	125.26	84.52
	EventEgo3D++ [ 25 ]	MPJPE	22.66	29.36	40.03	77.86	125.41	63.02	119.86	177.03	196.58	103.29
		PA-MPJPE	43.01	36.52	40.54	71.09	112.89	48.17	79.38	105.13	119.58	77.07

Appendix

Table 14: Per-joint quantitative comparison for EE3D-R and EE3D-W datasets.

Figure 13: Per-action qualitative comparison of our method with prior approaches on EE3D-W (challenging sequences). We compare against EgoPoseFormer [ 44 ] , EventEgo3D [ 26 ] , and EventEgo3D++ [ 25 ] . Red: Predicted pose. Green: Ground truth.

Figure 14: Per-action qualitative comparison of our method with prior approaches on EE3D-R (walk and further challenging sequences). We compare against EgoPoseFormer [ 44 ] , EventEgo3D [ 26 ] , and EventEgo3D++ [ 25 ] . Red: Predicted pose. Green: Ground truth.

3D Human Pose Estimation

3D human pose estimation recovers people’s body-joint positions or full-body shape and motion from images, video, and other sensors.

See all 113 papers

Jul 10, 2026

TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation

Egocentric 3D human pose estimation from head-mounted stereo cameras is challenging due to fisheye distortion, severe self-occlusion, and frequent truncation of body joints outside the camera field of view. Recent stereo egocentric methods have improved performance through heatmap lifting, stereo correspondence, and transformer-based refinement, but they often rely heavily on frame-local evidence or use temporal information only as auxiliary pose-level context. This limits robustness when current-frame stereo cues are weak, occluded, or ambiguous. We propose TSR-Ego, a temporally guided stereo framework that couples short-term motion evidence with projection-guided feature sampling. The model first enriches dense stereo feature maps using a causal depthwise-separable temporal convolution, allowing past visual evidence to influence the feature space before deformable cross-attention. A single-stage causal stereo decoder then refines learned 3D joint queries through temporal self-attention, joint self-attention, and fisheye deformable stereo cross-attention, using the evolving pose estimate to generate 2D sampling references. Unlike methods that apply temporal reasoning mainly after pose prediction, TSR-Ego uses motion context to shape both the sampled stereo features and the joint representations while preserving online inference without future frames. Experiments on UnrealEgo2 and UnrealEgo-RW show state-of-the-art performance, with especially strong gains on real-world sequences.

Md Mushfiqur Azam, John Quarles, Kevin Desai

The University of Texas at San Antonio

3D Human Pose Estimation Egocentric Vision

Ask PDF

May 12, 2026

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

Luming Wang, Hao Shi, Jiajun Zhai +2

National Research Center for Optical Instrumentation, Zhejiang University, Hangzhou 310027, China · Ant Group Company Ltd., Hangzhou 310013, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China +1

Event-Based Vision 3D Hand Pose Estimation Gesture Recognition Stereo Vision Egocentric Vision

Ask PDF

Mar 29, 2026

MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement

3D human pose estimation is a key enabling technology for applications such as healthcare monitoring, human-robot collaboration, and immersive gaming, but real-world deployment remains challenged by viewpoint variations. Existing methods struggle to generalize to unseen camera viewpoints, require large amounts of training data, and suffer from high inference latency. We propose MoViD, a viewpoint-invariant 3D human pose estimation framework that disentangles viewpoint information from motion features. The key idea is to extract viewpoint information from intermediate pose features and leverage it to enhance both the robustness and efficiency of pose estimation. MoViD introduces a view estimator that models key joint relationships to predict viewpoint information, and an orthogonal projection module to disentangle motion and view features, further enhanced through physics-grounded contrastive alignment across views. For real-time edge deployment, MoViD employs a frame-by-frame inference pipeline with a view-aware strategy that adaptively activates flip refinement based on the estimated viewpoint. Evaluations on nine public datasets and newly collected multiview UAV and gait analysis datasets show that MoViD reduces pose estimation error by over 24.2% compared to state-of-the-art methods, maintains robust performance under severe occlusions with 60% less training data, and achieves real-time inference at 15 FPS on NVIDIA edge devices.

Yejia Liu, Hengle Jiang, Haoxian Liu +2

Department of Computer Science and Engineering, Hong Kong University of Science and Technology

3D Human Pose Estimation Disentangled Representation Learning

Ask PDF

Abstract

3D Human Pose Estimation

Explore similar work

TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

MoViD: View-Invariant 3D Human Pose Estimation via Motion-View Disentanglement