ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild

Hongyu Ma; Hairong Qu; Shiqi Zhao; Yongsong Yang; Peng Yin

cs.CVSep 28, 2026

ESTHER: Egocentric Stereo Hand Estimation and Reconstruction in the Wild

Authors: Hongyu Ma, Hairong Qu, Shiqi Zhao, Yongsong Yang, Peng Yin

Organizations: City University of Hong Kong

Abstract

Human dexterity is guided by two eyes watching two hands: binocular vision supplies the metric 3D structure that fine-grained manipulation consumes. Egocentric stereo is therefore the natural perceptual interface for robots, AR, and VR-yet metric 3D hand reconstruction from this very signal still has neither an end-to-end model nor an in-the-wild benchmark. We propose ESTHER, a model whose stereo geometry, temporal reasoning, and output representation are designed for wearable egocentric stereo. It is trained on pseudo-labels from a calibrated labeling pipeline and in turn assembles our benchmark ESTHER3D, an egocentric stereo hand dataset pairing a large in-the-wild training set of model-generated labels with a motion capture test set of true metric ground truth. Experiments show state-of-the-art accu?racy, superior external generalization, and robustness to the missing views, dropped frames, and lighting and motion blur extremes of real egocentric capture that break existing meth?ods. This robustness runs deeper than graceful degradation: stereo guidance teaches the model to bind apparent hand scale to metric depth, so it not only adapts to different stereo rigs and modalities with minimal fine-tuning, but more strikingly preserves true metric scale even after collapsing to a single monocular view.

Figures & tables

Figure 1 : ESTHER at a glance. Letters: egocentric stereo frames from the six ESTHER3D industries. Bottom: ESTHER outputs over time — stereo 2D projections and metric 3D meshes.

Figure 2 : Supervision pipeline and data engine. (a) The teacher turns stereo 2D keypoints and per-view WiLoR into metric pseudo-labels via a two-stage optimization and a reprojection/human filter. (b) These train ESTHER, which labels in-the-wild stereo to build ESTHER3D.

Panel of Figure 3 — Figure 3 : ESTHER. (a) The stereo image encoder and masked temporal motion encoder feed a query-gated fusion decoder that outputs MANO with a metric wrist. (b, c) Stereo and temporal transformer blocks.

Figure 4 : Teacher pseudo-labels vs. motion-capture ground truth as MANO meshes projected into both stereo views (two ESTHER3D test scenes; the GT mesh is MANO fitted to the mocap joints).

Figure 5 : ESTHER3D across the six industries. Left: in-the-wild training hours (194 h). Right: motion-capture test tasks (50 tasks,

\approx

10 h).

	Method	P-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Wrist $\downarrow$	Jitter $\downarrow$
	Hiera-FTL [ 33 , 50 ]	43.4	10.0	277	79.5
	POEM-v2 [ 46 ]	19.4	8.9	22.5	8.5
ESTHER3D	MVGFormer [ 19 ]	19.2	8.9	21.0	8.7
	Epipolar Trans. [ 14 ]	19.0	9.0	20.9	13.3
	ESTHER (ours)	17.7	8.6	21.3	3.3
	Hiera-FTL [ 33 , 50 ]	103.2	13.6	324.3	99.6

Table 1 : Main comparison. Errors in mm, jitter in m/s 2 . ESTHER3D test = real mocap GT, zero-shot (z.s.) for all methods; HOT3D reported zero-shot and after fine-tuning (f.t.) on 10% of HOT3D. Best per block in bold.

Figure 6 : Qualitative results on ESTHER3D and HOT3D: MANO meshes projected into both stereo views (pink right, blue left hand). Per block: top row full stereo, bottom row monocular.

	Method	P-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Wrist $\downarrow$	Jitter $\downarrow$
	Hiera-FTL [ 33 , 50 ]	65.2	9.9	217	55.2
	HaWoR [ 48 ] (mono, temporal)	43.8	11.2	97.3	4.8
	HaMeR [ 29 ] (mono)	36.3	11.8	100	10.5
ESTHER3D	WiLoR [ 30 ] (mono)	30.2	11.1	101.6	9.5
	POEM-v2 [ 46 ]	29.5	9.7	96.8	8.8
	MVGFormer [ 19 ]	28.9	9.8	97.1	9.3

Table 2 : Missing-view robustness. One stereo view is dropped for the stereo-trained methods (ESTHER falls back via its view-mask token); monocular methods take their usual single view. Metrics as in Table 1 .

$N$ dropped / 16	ESTHER3D	HOT3D
0	17.7 / 8.6 / 21.3 / 3.3	29.5 / 17.3 / 20.8 / 17.1
2	17.8 / 8.6 / 21.3 / 3.9	29.7 / 17.5 / 20.9 / 18.2
4	17.8 / 8.6 / 21.4 / 4.7	29.8 / 17.4 / 20.9 / 19.5
6	17.8 / 8.6 / 21.5 / 7.7	29.8 / 17.4 / 21.2 / 23.8
8	17.8 / 8.6 / 21.5 / 5.4	30.0 / 17.5 / 21.0 / 20.6

Table 3 : Frame-dropping robustness as

N

of 16 frames are blanked and masked at inference, on the ESTHER3D test set and HOT3D. Each cell: P-MPJPE / PA-MPJPE / wrist (mm) / jitter (m/s 2 ).

Variant	MPJPE $\downarrow$	$\Delta$ vs full
Full ESTHER	19.60	–
w/o Plucker ray embedding	33.60	+14.0
w/o cross-view attention	28.70	+9.1
w/o all guidance, masking, augmentation	23.10	+3.5
w/o flow token	22.10	+2.5
w/o 2D / heatmap guidance	20.99	+1.4

Table 4 : Component ablation on a subject-disjoint validation split of the pseudo-labeled data (absolute MPJPE, mm).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Setup	Resolution	Focal (px)	Baseline (mm)	Modality
Collection rig (ours)	$1920\times 1200$	623	62	RGB
Test-set rig (mocap)	$1920\times 1200$	623	62	RGB
HOT3D Project Aria	$640\times 480$	241	138	monochrome
HOT3D Quest 3	$1280\times 1024$	504	64	monochrome

Appendix

Table 5 : Camera setups used in this work (per-eye resolution and focal length; stereo baseline). Fisheye rigs are rectified to pinhole before use.

Figure 9 : ESTHER on in-the-wild ESTHER3D scenes across six industries. Each block shows five timesteps of one scene (columns, spaced $0.2$ s apart); per timestep ESTHER’s MANO meshes are projected into both stereo eyes side by side — marked L (left) and R (right); pink right, blue left hand — with the same metric meshes in 3D below.

Figure 10 : Motion-capture ground truth of the ESTHER3D test set. Six sequences (Seq 1–6), each shown as two rows — the left eye (L, cam0) and right eye (R, cam1) of the head-mounted stereo rig — across seven timesteps (columns, labeled by time). The ground-truth hands (MANO fitted to the motion-capture hand joints) are projected into each view; they are the true metric supervision against which all methods are evaluated zero-shot.

Method	P-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Wrist $\downarrow$	Jitter $\downarrow$
Hiera-FTL	$43.4\pm 1.2$	$10.0\pm 0.2$	$277\pm 5$	$79.5\pm 3.6$
Epipolar Trans.	$19.0\pm 0.5$	$9.0\pm 0.1$	$\mathbf{20.9\pm 0.5}$	$13.3\pm 0.4$
MVGFormer	$19.2\pm 0.6$	$8.9\pm 0.1$	$21.0\pm 0.4$	$8.7\pm 0.3$
POEM-v2	$19.4\pm 0.6$	$8.9\pm 0.1$	$22.5\pm 0.4$	$8.5\pm 0.3$
ESTHER (ours)	$\mathbf{17.7\pm 0.4}$	$\mathbf{8.6\pm 0.1}$	$21.3\pm 0.5$	$\mathbf{3.3\pm 0.1}$

Appendix

Table 6 : Bootstrap statistics on the ESTHER3D test set (

120

clips,

2000

resamples): mean

\pm

std. Errors in mm, jitter in m/s 2 . ESTHER’s P-MPJPE and jitter

95\%

intervals do not overlap any baseline’s.

Figure 11 : ESTHER under input corruptions on an ESTHER3D motion-capture clip: MANO meshes projected on the degraded left view (top) and shown in 3D (bottom), for the clean input (a) and seven corruptions (b–h) spanning lighting, blur, noise, and color shifts. The mesh stays stable throughout.

Corruption	P-MPJPE	PA-MPJPE	Wrist	Jitter
clean	$17.7$	$8.60$	$21.3$	$3.31$
over-exposure ( $\times 1.6$ )	$17.6$	$8.66$	$20.1$	$3.71$
low light ( $\times 0.5$ )	$17.7$	$8.57$	$21.5$	$3.34$
gamma $2.2$	$17.7$	$8.49$	$21.9$	$3.33$
motion blur ( $k{=}15$ )	$18.0$	$8.66$	$19.5$	$3.24$
sensor noise ( $\sigma{=}25$ )	$18.2$	$8.68$	$21.6$	$3.73$

Appendix

Table 7 : ESTHER under input corruption on the ESTHER3D test set. Errors in mm, jitter in m/s 2 ; every row is within noise of the clean condition.

Method	offbb120	offbb90	offbb150	hc
ESTHER (ours)	32.4 / 20.8 / 16.3	32.5 / 22.8 / 16.6	32.3 / 23.6 / 16.8	28.9 / 48.0 / 15.6
POEM-v2	101.3 / 656 / 635	100.9 / 450 / 395	103.6 / 928 / 1044	46.3 / 65.7 / 48
MVGFormer	95.9 / 429 / 449	95.9 / 291 / 265	97.6 / 633 / 767	43.1 / 69.9 / 52
Epipolar Trans.	69.4 / 414 / 318	70.6 / 303 / 191	69.9 / 571 / 541	43.2 / 66.9 / 66
Hiera-FTL	80.9 / 401 / 105	78.7 / 390 / 100	82.8 / 435 / 118	78.1 / 377 / 91

Appendix

Table 8 : HOT3D undistortion-strategy robustness on a held-out split. Each cell: P-MPJPE / wrist (mm) / jitter (m/s 2 ). The comparison is each method’s stability across strategies, not head-to-head accuracy: ESTHER barely moves (including on the 90/150 px focals it was not trained on) while the multi-view baselines swing widely.

Quantity	Median	Mean	p90
Reprojection residual $r$ (px)	5.3	5.8	8.6
Hand depth $z$ (m)	0.32	0.33	0.46
In-plane error $rz/f$ (mm)	2.8	3.0	4.6
Depth per pixel of disparity $z^{2}/(fb)$ (mm)	2.8	3.2	5.6
Admissible depth error, independent views (mm)	21.3	25.1	44.2
Admissible depth error, worst case (mm)	30.1	35.5	62.5

Appendix

Table 9 : In-the-wild label quality from 2D (hand-frames passing the 15 px check). Depth

z

is the label wrist depth.

Explore similar work

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
May 12, 2026Luming Wang, Hao Shi, Jiajun Zhai +23D Hand Egocentric Motion Generation
National Research Center for Optical Instrumentation, Zhejiang University, Hangzhou 310027, China · Ant Group Company Ltd., Hangzhou 310013, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China +1
EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes
Oct 1, 2026Hongming Fu, Jingcheng Shi, Wenjia Wang +23D Hand Hand
School of AI, Shanghai Jiao Tong University · Rutgers University · The University of Hong Kong +1
MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild
Sep 15, 2026Jiangong Xiao, Zhihao Zhang, Yifei Dong +73D Hand Egocentric Motion Generation
Northwestern Polytechnical University · Maniformer · Xi'an Jiaotong University +1

May 12, 2026, cs.CVJ/K move · Enter open · S save

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

Luming Wang, Hao Shi, Jiajun Zhai, Kailun Yang+1

National Research Center for Optical Instrumentation, Zhejiang University, Hangzhou 310027, China · Ant Group Company Ltd., Hangzhou 310013, China · School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410082, China+1

3D Hand Egocentric Motion Generation Hand Gesture Recognition Egocentric Vision Hand

Open paper AskPDF arXiv

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

Abstract

Explore similar work

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes

MEgoVista: Multi-view Ego-aware Motion Estimation for Metric 4D Hands and Head in the Wild