EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation

Haozhe Wang; Songyuan Wu; Jihai Xu; Jie Zhao

cs.ROOct 19, 2026

EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation

Authors: Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao

Organizations: Boden AI

Abstract

Head-mounted stereo fisheye cameras provide a compact view of the wearer's body, but strong distortion, partial visibility and a short baseline make 3D pose estimation difficult. Existing refinement decoders sample image features independently in each view, without requiring the samples to correspond to a common 3D hypothesis. We introduce EgoRayFormer, a causal pose estimator that couples stereo feature sampling and geometric refinement in ray space. For each joint, the decoder projects shared 3D hypotheses along and around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects the resulting rays; a learned gate blends this geometric estimate with a learned pose update. This formulation introduces an explicit stereo constraint while retaining flexibility when the rays provide weak depth constraints. In local evaluations on the official device-relative test splits, EgoRayFormer achieves three-seed mean MPJPEs of 17.64 mm on UnrealEgo2 and 49.55 mm on UnrealEgo-RW, improving to 47.57 mm with UnrealEgo2 pre-training. In paired validation ablations, removing the geometric branch increases MPJPE in all three seeds at both input resolutions. With a common ResNet-18 encoder and input resolution, our system reduces RW test MPJPE by 9.09 mm relative to the device-relative EgoPoseFormer baseline. A separate protocol analysis quantifies the difference between device-relative and oracle-root evaluation.

Figures & tables

Figure 2: Overview of EgoRayFormer. A stereo encoder provides image features for a coarse pose proposal and three refinement layers. Each refinement layer projects shared 3D hypotheses into both views, aggregates their sampled features, and combines a learned pose update with a regularised ray-intersection update. Causal attention and cached image features provide temporal context for frame-by-frame inference.

				UnrealEgo2		RW, trained on RW		RW, trained on UE2 + RW
Method	Backbone	Frames	Extra input	MPJPE	PA	MPJPE	PA	MPJPE	PA
EgoGlass † [ 37 ]	R18	1	–	79.64	58.22	–	–	97.86	69.92
UnrealEgo [ 1 ]	R18	1	–	72.82	52.90	–	–	92.48	67.15
Akada et al . [ 2 ]	R18	5	scene depth	30.53	26.72	104.14	82.18	72.89	56.19
EgoRayFormer (ours)	ViT-S+	stream	calibration	17.64 $\pm$ 0.12	16.86 $\pm$ 0.08	49.55 $\pm$ 0.40	40.59 $\pm$ 0.30	47.57 $\pm$ 0.34	37.21 $\pm$ 0.46

Table 1: Comparison with published leaderboard values (device-relative protocol, test sets, mm, lower is better). Ours: local evaluations on the current official splits, mean

\pm

s.d. over three seeds; results have not been submitted to the leaderboard. The split version underlying the published RW values is unverified ( Sec. 4.2 ). Backbone: R18 is ResNet-18 at

256^{2}

, ViT-S+ is DINOv3 ViT-S+/16 at

384^{2}

. Frames: input frames per prediction; stream denotes causal streaming over the whole sequence. Extra input: inputs used at inference besides the stereo images; scene depth is rendered from a structure-from-motion reconstruction of the scene [ 2 ] , calibration denotes the fisheye camera model and the stereo rig ( Sec. 4.1 ). † Re-implemented by the benchmark authors. Full comparison conditions are given in Appendix B .

UnrealEgo-RW, trained on RW
		Val	Test
Method	Frames	MPJPE	MPJPE	PA
EPF-device [ 36 ]	1	60.17 $\pm$ 1.48	77.85 $\pm$ 3.20	62.02 $\pm$ 1.22
TSR-like	8, causal	77.97 $\pm$ 3.23	98.92 $\pm$ 7.27	77.32 $\pm$ 4.81
EgoRayFormer-R18	stream	54.73 $\pm$ 0.68	68.76 $\pm$ 1.36	52.78 $\pm$ 1.22
UnrealEgo2, trained on UnrealEgo2

Table 2: Same-backbone comparison on RW (trained on RW only) and on UnrealEgo2 (trained on UnrealEgo2 only); ResNet-18 + FPN at

256^{2}

, device-relative protocol, mm, mean

\pm

s.d. over three seeds. Val: validation MPJPE of the selected checkpoints. Test: one evaluation per frozen checkpoint. EPF-device: the official EgoPoseFormer code with the device origin instead of the ground-truth pelvis as projection origin; it has no UnrealEgo2 entry. TSR-like: our implementation following the description of TSR-Ego; not a reproduction. On UnrealEgo2, TSR-like is trained for 12 epochs as in its paper and EgoRayFormer-R18 for 3 as in our UnrealEgo2 recipe. Frames as in Tab. 1 .

Variant	MPJPE / $\Delta$	worse	PA / $\Delta$	MPJVE / $\Delta$
Full model, $384^{2}$ input	43.79 $\pm$ 0.64	–	31.74	12.02
input $384^{2}\to 256^{2}$	+2.83 $\pm$ 0.78	3/3	+4.15	+0.94
w/o geometric branch	+0.92 $\pm$ 0.51	3/3	+0.65	+0.19
w/o trajectory sampling	$-$ 0.04 $\pm$ 0.94	2/3	$-$ 0.25	$-$ 0.07
w/o track prior	$-$ 0.54 $\pm$ 1.59	2/3	$-$ 0.34	+0.22
plain encoder	+0.05 $\pm$ 1.84	1/3	$-$ 0.47	$-$ 0.24

Table 3: Ablations on RW validation (trained on RW only, three paired seeds, mm). The rows of a full model give mean

\pm

s.d. MPJPE and mean PA-MPJPE and MPJVE; the indented rows give the change caused by the modification. Plain encoder: DINOv3 without the fisheye-specific components of Sec. 3.5 . Worse: seed pairs in which MPJPE increases.

Figure 3: Error and gate on RW validation (final RW-only model, three seeds). (a) RMS error per axis of the final prediction along and across the unit vector from the device centre to the ground-truth joint, by joint distance (seed mean; band: range over seeds); isotropic errors would give two equal curves. (b, c) Mean gate of the last refinement layer by joint distance and by the angle between the two corrected viewing rays, for each seed; joints whose predicted anchor projects into fewer than two images are excluded.

Protocol	MPJPE (own)	RA-MPJPE	PA-MPJPE
Device	77.97 $\pm$ 3.23	79.45 $\pm$ 4.42	61.45 $\pm$ 3.71
Oracle root	57.35 $\pm$ 0.74	57.41 $\pm$ 0.79	48.57 $\pm$ 0.57
Difference	$-$ 20.62 $\pm$ 3.12	$-$ 22.04 $\pm$ 4.23	$-$ 12.88 $\pm$ 3.71
Pairs lower	3/3	3/3	3/3

Table 4: Protocol gap for one architecture (TSR-like, ResNet-18, RW validation, three paired seeds, mm). Device: device-relative target and device origin, no ground-truth input. Oracle root: target relative to the ground-truth mid-hip, which is also the projection origin in training and at inference. Own: device-relative or root-relative MPJPE, respectively; RA: root-aligned. Difference: oracle root minus device, paired by seed; pairs lower: seed pairs in which the oracle-root run has the lower error.

		Params	GFLOPs		Latency
Model	Input	(M)	total	encoder	(ms)
EgoRayFormer	384	35.1	106.9	97.4	42.8–43.7
EgoRayFormer-R18	256	13.7	17.9	13.6	35.4
TSR-like	256	12.8	17.4	13.6	11.2
EPF-device ‡	256	14.2	14.7	13.6	7.5

Table 5: Size, computation and latency of one streaming step (one stereo frame after an 8-frame warm-up; EPF-device: one single-frame forward pass). Input: image size; EgoRayFormer uses DINOv3 ViT-S+/16, the other models ResNet-18 + FPN. GFLOPs from torch.utils.flop_counter , without bilinear sampling and fisheye projections. Latency: median of 200 runs, batch 1, FP32, PyTorch eager, one NVIDIA H200, model forward only; the range covers our three final models. ‡ Deformable attention runs in the pure-PyTorch implementation of mmcv, since the precompiled CUDA kernel does not support this GPU.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Encoder, input, pre-training	3D training data	Temporal context	Extra input at inference	Numbers
EgoRayFormer (ours)	DINOv3 ViT-S+/16, $384^{2}$ ; LVD-1689M, then heatmap stage on the target training split	UE2: 3 epochs on UE2. RW: 12 epochs on RW. UE2 + RW: UE2 model of the same seed, then 12 epochs on RW	causal stream over the whole sequence; temporal attention over 8 frames, trajectory sampling over 4 past frames, previous-frame prior	calibration: synthetic UE2 rig; RW intrinsics of the dataset and per-session extrinsics fitted to training annotations	device-relative; local, frozen, 3 seeds
Akada et al . [ 2 ] (T=5)	2D module ResNet-18, $256^{2}$ , ImageNet; backbones of the depth and heatmap feature extractors not stated	UE2: UE2. RW, UE2 + RW: UE2 pre-training, RW fine-tuning. RW only: as labelled on the leaderboard	stereo video, 5 frames with skip 3, last time step is predicted	depth rendered from a structure-from-motion reconstruction of a 4 s window; whether the window extends beyond the current frame is not stated	leaderboard
UnrealEgo [ 1 ]	ResNet-18, $256^{2}$ ; pre-training of the leaderboard entry not stated	UE2: UE2. RW: UE2, then RW	single stereo frame	none	leaderboard
EgoGlass † [ 37 ]	ResNet-18, $256^{2}$ ; re-implemented without the body-part segmentation branch	as UnrealEgo	single stereo frame	none	leaderboard
TSR-Ego [ 4 ]	ResNet-18 + FPN, $256^{2}$ ; dataset-specific heatmap pre-training	official training splits; no UE2 pre-training reported for RW	causal windows of 8 frames; test windows with stride 8	fisheye calibration; the root in each camera as projection origin (UE2 pelvis, RW mid-hip); ground-truth or predicted root at inference not stated	root-relative, preprint
EgoPoseFormer ∗ [ 36 ]	official code: ResNet-18 + FPN, $256^{2}$ ; ImageNet, then heatmap stage	not stated	single frame	the official data loader uses the ground-truth pelvis of each frame as projection origin; whether the retraining kept it is not stated	root-relative, reported by [ 4 ]

Appendix

Table 6: Comparison conditions. UE2: UnrealEgo2; RW: UnrealEgo-RW. † Re-implemented by the benchmark authors. ∗ Retrained by TSR-Ego with the official code. Local: evaluated by us on the official test split; frozen: under the protocol of Appendix A .

Session	Frames	RW	UE2 + RW	R18	EPF-device
		EgoRayFormer
2023-01-17 §	9,023	47.99	43.01	64.91	75.28
2023-08-23	7,306	44.72	43.34	58.27	59.34
2023-08-25	7,683	55.97	56.94	83.25	98.46
Upper body		31.43	32.21	42.67	50.66
Lower body		67.67	62.92	94.85	105.04

Appendix

Table 7: RW test MPJPE per capture session and for the upper and lower body (mm, three-seed means). RW and UE2 + RW: EgoRayFormer trained on RW only and on UnrealEgo2 and RW; R18: EgoRayFormer-R18; EPF-device as in Tab. 2 . § Session without training or validation data; it uses the rig of the chronologically nearest session.

cm	%	MPJPE	along	lateral	$X^{*}$	learned	gate
		final output			last layer
0–20	6.4	13.5	9.0	9.6	13.5	13.6	0.46
20–40	20.9	21.6	14.4	15.6	21.3	21.4	0.39
40–60	14.6	28.7	22.6	19.0	28.1	29.0	0.35
60–80	18.3	33.2	24.4	21.7	32.5	33.7	0.34
80–100	5.6	41.4	32.6	30.3	40.6	41.9	0.34

Appendix

Table 8: RW validation errors and gate by joint distance (final RW-only model, means over three seeds, mm). Along and lateral: RMS error per axis along and across the unit vector from the device centre to the ground-truth joint.

X^{*}

and learned: MPJPE of the two candidates of the last refinement layer, and Gate: its mean gate, over the joints whose predicted anchor projects into both images.

root-relative, as reported
	UnrealEgo2		RW, trained on RW
Method	MPJPE	PA	MPJPE	PA
EgoPoseFormer [ 36 ] ∗	23.12	21.69	74.31	61.79
TSR-Ego [ 4 ] , 2 layers	22.36	21.23	65.58	56.03
TSR-Ego [ 4 ] , 4 layers	–	–	62.8	55.3
root-aligned, three-seed mean

Appendix

Table 9: Root-relative values reported by TSR-Ego and our root-aligned scores (test sets, mm). Root-relative values are quoted from [ 4 ] ; the root used at inference is not stated, and on UnrealEgo2 the root is the pelvis. Root-aligned: prediction and ground truth are each centred at their own mid-hip, with no ground truth at inference. PA-MPJPE (PA) does not depend on the root. ∗ Retrained by TSR-Ego with the official code. A reference, not a ranking ( Sec. 4.6 ).

Figure 4: Qualitative results on UnrealEgo-RW validation (final RW-only model, seed 42). Rows: the frames whose per-frame error is closest to the 10th, 50th, 90th and 99th percentile, each from a different sequence; the label gives the frame’s MPJPE. Skeletons are projected with the rig of the frame’s capture session; joints that project outside an image are not drawn there. The lines between the neck and the hips are drawn for readability only.

3D Human Pose Estimation

3D human pose estimation recovers people’s body-joint positions or full-body shape and motion from images, video, and other sensors.

See all 113 papers

Jul 10, 2026

TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation

Egocentric 3D human pose estimation from head-mounted stereo cameras is challenging due to fisheye distortion, severe self-occlusion, and frequent truncation of body joints outside the camera field of view. Recent stereo egocentric methods have improved performance through heatmap lifting, stereo correspondence, and transformer-based refinement, but they often rely heavily on frame-local evidence or use temporal information only as auxiliary pose-level context. This limits robustness when current-frame stereo cues are weak, occluded, or ambiguous. We propose TSR-Ego, a temporally guided stereo framework that couples short-term motion evidence with projection-guided feature sampling. The model first enriches dense stereo feature maps using a causal depthwise-separable temporal convolution, allowing past visual evidence to influence the feature space before deformable cross-attention. A single-stage causal stereo decoder then refines learned 3D joint queries through temporal self-attention, joint self-attention, and fisheye deformable stereo cross-attention, using the evolving pose estimate to generate 2D sampling references. Unlike methods that apply temporal reasoning mainly after pose prediction, TSR-Ego uses motion context to shape both the sampled stereo features and the joint representations while preserving online inference without future frames. Experiments on UnrealEgo2 and UnrealEgo-RW show state-of-the-art performance, with especially strong gains on real-world sequences.

Md Mushfiqur Azam, John Quarles, Kevin Desai

The University of Texas at San Antonio

3D Human Pose Estimation Egocentric Vision

Ask PDF

Apr 9, 2026

E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

Event cameras offer multiple advantages in monocular egocentric 3D human pose estimation from head-mounted devices, such as millisecond temporal resolution, high dynamic range, and negligible motion blur. Existing methods effectively leverage these properties, but suffer from low 3D estimation accuracy, insufficient in many applications (e.g., immersive VR/AR). This is due to the design not being fully tailored towards event streams (e.g., their asynchronous and continuous nature), leading to high sensitivity to self-occlusions and temporal jitter in the estimates. This paper rethinks the setting and introduces E-3DPSM, an event-driven continuous pose state machine for event-based egocentric 3D human pose estimation. E-3DPSM aligns continuous human motion with fine-grained event dynamics; it evolves latent states and predicts continuous changes in 3D joint positions associated with observed events, which are fused with direct 3D human pose predictions, leading to stable and drift-free final 3D pose reconstructions. E-3DPSM runs in real-time at 80 Hz on a single workstation and sets a new state of the art in experiments on two benchmarks, improving accuracy by up to 19% (MPJPE) and temporal stability by up to 2.7x. See our project page for the source code and trained models.

Mayur Deshmukh, Hiroyasu Akada, Helge Rhodin +2

MPI for Informatics, SIC · Saarland University, SIC · Bielefeld University

3D Human Pose Estimation Egocentric Vision Event-Based Vision

Ask PDF

Mar 26, 2026

AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimation

Monocular egocentric 3D pose estimation is difficult because severe foreshortening, self-occlusion, and a restricted field of view often remove the image evidence needed to recover the camera wearer's body. Temporal context can resolve these ambiguities, but unconstrained fusion may also corrupt joints that are already localized reliably from the current frame. We present AG-EgoPose, an action-guided framework that treats temporal information as a bounded-gate residual correction to a spatial pose estimate. A frozen heatmap network produces joint heatmaps and feature-pyramid features, from which a spatial transformer estimates an independently supervised 3D pose. The heatmaps further guide joint-local pooling of frozen DINOv2 patch tokens, while DINOv2 frame tokens are processed by an Ego4D-pretrained ActionFormer to obtain action-scale temporal context. Each spatial joint token, augmented with its local appearance feature, attends to the temporal sequence and predicts a zero-initialized 3D residual. A per-joint gate, derived from heatmap location and confidence statistics, bounds the residual weight. AG-EgoPose improves over the strongest evaluated baselines by 10.1% PA-MPJPE on EgoPW, and by 9.0% MPJPE and 6.3% PA-MPJPE on SceneEgo after fine-tuning. Code will be made publicly available at https://github.com/

Md Mushfiqur Azam, John Quarles, Kevin Desai

The University of Texas at San Antonio

3D Human Pose Estimation Egocentric Video Understanding

Ask PDF

Abstract

3D Human Pose Estimation

Explore similar work

TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation

E-3DPSM: A State Machine for Event-Based Egocentric 3D Human Pose Estimation

AG-EgoPose: Spatially Anchored Residual Correction with Action Context for Monocular Egocentric 3D Pose Estimation