OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

Kyungwon Cho; Jeonghyeon Na; Hanbyul Joo

cs.CVDec 22, 2025

OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

Authors: Kyungwon Cho, Jeonghyeon Na, Hanbyul Joo

Organizations: Seoul National University

Abstract

Commercial egocentric devices capture human behavior in everyday settings, yet full-body motion reconstruction must generalize across diverse cameras and mountings. This is challenging because head trajectories under-determine body motion, while hand observations are intermittent and camera-dependent: a hand may disappear simply because it has left the camera's field of view. Existing methods treat such absences as missing data, relying on fixed visibility regimes or post-hoc optimization, and fail to generalize across devices. We present OmniEgoCap, a sequence-level diffusion framework for device-agnostic egocentric full-body motion reconstruction. Our key insight is that egocentric videos contain sequence-level evidence about both the wearer and the camera-induced visibility structure, enabling OmniEgoCap to infer consistent body shape and coherent motion from sparse head and hand cues. To prevent overfitting to a single camera setup, we further introduce geometry-aware visibility augmentation, which synthesizes visibility patterns from realistic variations in camera geometry. We also present OmniEgoDB, the first mocap benchmark that pairs ground-truth motion with recordings from multiple consumer egocentric devices. Experiments on synthetic, real-device, and in-the-wild settings demonstrate superior motion reconstruction, robust cross-device generalization, and coherent in-the-wild results.

Figures & tables

Figure 1: OmniEgoCap is a sequence-level diffusion framework that combines long-range body and camera-observation invariants with geometry-aware visibility augmentation to reconstruct consistent 3D full-body motion across diverse egocentric devices. (L) GoPro and (R) Aria.

Figure 2: Overview of our model. OmniEgoCap takes the head trajectory $\mathbf{T}^{\text{head}}_{0:T}$ and intermittent hand observations $\mathbf{H}_{0:T}$ , which are first converted into spatio-temporally invariant conditioning features $\bm{\Omega}_{1:T}$ . The encoder $\mathcal{E}$ processes $\bm{\Omega}_{1:T}$ to predict a single body shape $\hat{\beta}$ , a visibility context $\bm{\mathcal{G}}$ , and per-frame summary features $\bm{\mathcal{S}}_{1:T}$ . A diffusion decoder $\mathcal{D}$ then conditions on these features to denoise a noisy canonicalized pose $\mathbf{X}^{n}_{1:T}$ and, via global alignment, reconstruct the full-body motion $\mathbf{M}_{1:T}$ .

Figure 3: Real and simulated visibility boundaries. The top row shows calibrated FoVs from real-world devices ( e.g. , Aria, GoPro). The bottom row illustrates visibility patterns sampled by our geometry-aware augmentation. Red regions indicate the visible field of view.

3,6,8 Method		MPJPE ${\downarrow}$	MPJVE ${\downarrow}$	Jerk ${\downarrow}$	Hand PE ${\downarrow}$	Vis Hand PE ${\downarrow}$	Height ${\downarrow}$	Span ${\downarrow}$	Height Std ${\downarrow}$	Span Std ${\downarrow}$	Runtime ${\downarrow}$
EgoAllo (w/o opt)	122.99	44.02	0.350	318.69	298.01	3.80	6.24	1.87	2.43	2.49
EgoAllo (w/ opt)	99.32	31.66	0.152	164.80	41.49	3.86	6.19	1.84	2.35	56.09
EgoPoser †	117.04	58.24	0.972	244.39	117.45	4.65	7.52	2.97	3.52	0.09
EgoAllo † (w/o opt)	103.55	35.57	0.290	212.95	122.89	3.62	6.10	1.70	2.06	2.55
EgoAllo † (w/ opt)	94.00	26.13	0.120	142.24	25.74	3.78	6.25	1.71	2.08	59.26
Ours (w/o opt)	81.35	25.81	0.117	146.71	56.98	2.40	4.61	0.00	0.00	2.07

Table 1: Quantitative results under diverse simulated settings. w/o opt denotes evaluation without optimization; w/ opt denotes evaluation with optimization.

Figure 4: Qualitative results under diverse simulated settings. (Left) EgoAllo produces unnatural, unstable motion during dancing, while ours remains plausible. (Right) During a kick, EgoAllo hallucinates the invisible hand back into the FoV, whereas ours predicts it outside the view.

5,9 Method		$90^{\circ}$ FoV			$120^{\circ}$ FoV			$180^{\circ}$ FoV			$120^{\circ}$ + $30^{\circ}$ Tilt			$120^{\circ}\times 60^{\circ}$
	PE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$
EgoPoser (120)	100.80	0.519	164.19	95.62	0.505	136.04	88.95	0.427	100.93	98.02	0.526	150.97	104.63	0.525	183.79
EgoAllo †† (120)	110.23	0.298	212.93	107.10	0.293	178.94	113.14	0.284	156.06	115.86	0.287	183.02	113.94	0.299	237.91
EgoAllo (w/ opt)	112.06	0.162	239.87	104.80	0.165	192.48	86.70	0.159	75.56	91.36	0.156	108.28	115.13	0.161	259.06
Ours (120)	101.56	0.129	207.62	91.11	0.126	152.14	92.07	0.115	130.77	95.64	0.117	160.50	111.58	0.131	266.18
Ours (Random)	151.73	0.158	278.73	131.97	0.154	246.58	94.78	0.139	137.02	103.51	0.142	172.89	161.32	0.159	295.27

Table 2: Quantitative results across varying camera setups. PE and H-PE denote MPJPE and Hand PE.

Method	Aria (Stereo)				Aria (RGB)				GoPro				Quest
	21 seqs, 67.1 min				21 seqs, 67.1 min				20 seqs, 68.1 min				9 seqs, 42.0 min
	PE $\downarrow$	VE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	VE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	VE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$	PE $\downarrow$	VE $\downarrow$	Jerk $\downarrow$	H-PE $\downarrow$
EgoAllo # (w/o opt)	138.22	24.70	0.146	261.10	138.22	24.70	0.146	261.10	136.86	24.11	0.143	260.80	152.17	22.02	0.140	282.93
EgoAllo # (w/ opt)	115.48	25.06	0.122	142.00	128.71	32.26	0.259	182.86	125.65	28.65	0.189	186.80	122.87	22.97	0.121	135.15
EgoPoser #	126.54	43.56	0.659	172.75	135.99	51.65	0.906	197.39	134.89	47.30	0.776	209.21	130.04	35.17	0.521	153.07
EgoPoser (120)	124.95	58.15	1.265	185.71	132.22	65.00	1.551	198.19	130.40	65.67	1.580	213.15	125.46	46.07	0.979	161.43

Table 3: Quantitative results on OmniEgoDB. PE, VE, and H-PE denote MPJPE, MPJVE, and Hand PE. # denotes methods evaluated with their official checkpoints. – denotes entries not applicable to UniEgoMotion, which needs RGB frames and predicts at 10 FPS.

Figure 5: Qualitative results on in-the-wild data. Reconstructions from (Left) Aria RGB (monocular) and (Right) GoPro captures.

Table 10

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Module	Hidden Dim	Out Dim	Layers	Heads	Dropout
Shape Encoder	384	384	2	6	0.1
Shape Pooler	384	16	1	6	0.1
Global Encoder	384	384	2	6	0.1
Global Pooler	384	512	1	6	0.1
Local Encoder	512	512	6	8	0.1
Decoder	512	132	8	8	0.1

Appendix

Table 6: Architectural hyperparameters. Configurations of the transformer modules in OmniEgoCap.

Table 12

Figure 8: Qualitative results across varying camera setups. (Left) Sudden arm movements in EgoPoser (120) and arms pushed out of bounds in EgoAllo †† at $90^{\circ}$ FoV. (Right) Misaligned tracking of EgoAllo †† at $180^{\circ}$ FoV.

Figure 11: Overview of OmniEgoDB. The left side shows participants equipped with three different egocentric devices and Xsens mocap suits. The right side shows representative egocentric frames from OmniEgoDB, illustrating diverse viewpoints, scenes, and activities.

Figure 12: Qualitative Analysis on OmniEgoDB. From left to right, we show a GoPro depth-direction wrist error, a Quest false-positive wrist observation, temporarily missing Aria wrist observations, and an Aria case with inconsistent body shape estimation. Blue, purple, and yellow denote OmniEgoCap, EgoAllo with optimization, and ground truth, respectively.

Method	ADT (Stereo)				ADT (RGB)
	34 seqs, 52.5 min				34 seqs, 52.5 min
	MPJPE $\downarrow$	MPJVE $\downarrow$	Jerk $\downarrow$	Hand PE $\downarrow$	MPJPE $\downarrow$	MPJVE $\downarrow$	Jerk $\downarrow$	Hand PE $\downarrow$
EgoAllo # (w/o opt)	159.06	18.36	0.117	293.74	159.06	18.36	0.117	293.74
EgoAllo # (w/ opt)	145.78	19.61	0.075	212.97	150.04	26.08	0.256	204.78
EgoPoser #	127.50	32.22	0.429	236.36	126.86	46.84	0.950	206.69
EgoPoser (120)	131.06	40.20	0.857	243.56	130.56	61.49	1.511	213.93

Appendix

Table 9: Quantitative results on Aria Digital Twin. # denotes methods evaluated with their official checkpoints. – denotes entries not applicable to UniEgoMotion, which needs RGB frames and predicts at 10 FPS.

Figure 13: Additional qualitative results on in-the-wild data. Reconstructions from (Upper Left) iPhone 15, (Upper Right) Quest, (Lower Left) Aria Stereo (SLAM), and (Lower Right) Aria Monocular (RGB).

Figure 14: Representative failure cases. Gray corresponds to the ground truth, blue to Ours (w/ opt), and purple to EgoAllo (w/ opt).

3D Human Motion Reconstruction

See all 26 papers

Seeing the World and the Self from Egocentric Video
Sep 1, 2026Kai Guan, Minchao Jiang, Ruichen WangLi +2Markerless Motion Capture 3D Reconstruction
The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo
EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcing
May 13, 2026Inwoo Hwang, Donggeun Lim, Hojun Jang +1Markerless Motion Capture Egocentric Video Understanding
Seoul National University
EgoExoMoCap: Distributed Ego-Exo Human Motion Capture
Jul 17, 2026Jiaxi Jiang, Bharat Lal Bhatnagar, Nan Yang +6Markerless Motion Capture 3D Human Pose Estimation
1Meta Reality Labs · *This work was done during an internship at Meta. · 2ETH Zürich

Sep 1, 2026J/K move · Enter open · S save

Seeing the World and the Self from Egocentric Video

Kai Guan, Minchao Jiang, Ruichen WangLi, Wentao Zhu+1

The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo

Markerless Motion Capture 3D Reconstruction Camera Pose Estimation Egocentric Video Understanding

Open paper AskPDF arXiv

Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Existing methods typically address scene reconstruction and motion estimation separately: scene reconstruction methods ignore the wearer, whereas motion estimation methods lack explicit scene geometry and often depend on external trajectories. Joint recovery is challenging because the two tasks exhibit asymmetric visibility and require different prediction paradigms. The largely visible scene supports deterministic geometric regression, whereas the severely occluded body requires generative motion inference. We therefore propose RESELF (REconstructing the Scene and the sELF), a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation. RESELF adapts a geometry foundation model pre-trained on large-scale exocentric data to egocentric video using frame-wise scale and relative-pose consistency objectives. The resulting camera trajectory and latent geometric features condition a diffusion model that recovers the wearer's motion. A subsequent closed-loop kinematic feedback stage further refines the camera head while preserving the reconstructed scene geometry. To support training and evaluation, we curate EE4D-JSM from EgoExo4D by aligning egocentric video, sparse metric scene geometry, camera trajectories, and full-body motion annotations. Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation. Code, models, and datasets will be available at https://ka1guan.github.io/RESELF/.

Abstract

3D Human Motion Reconstruction

Explore similar work

Seeing the World and the Self from Egocentric Video

EgoForce: Robust Online Egocentric Motion Reconstruction via Diffusion Forcing

EgoExoMoCap: Distributed Ego-Exo Human Motion Capture