EgoMo3R: Joint Egocentric Motion and Scene Reconstruction

Chaitanya Patel; Yusuke Kato; Rin Takahira; Kazuki Kozuka; Juan Carlos Niebles; Ehsan Adeli

cs.CVNov 1, 2026

EgoMo3R: Joint Egocentric Motion and Scene Reconstruction

Authors: Chaitanya Patel, Yusuke Kato, Rin Takahira, Kazuki Kozuka, Juan Carlos Niebles, Ehsan Adeli

Organizations: Stanford University · Panasonic R&D Company of America · Panasonic Holdings Corporation

Abstract

Understanding holistic 3D human motion and the surrounding environment from egocentric video is fundamental for applications in AR/VR and robotics. Although these tasks are inherently coupled, existing methods treat them separately. In particular, egocentric motion reconstruction methods typically assume access to metric, gravity-aligned camera trajectories from specialized inertial sensors. Recovering such trajectories from monocular egocentric video remains a significant challenge due to rapid head movements and motion blur. We present EgoMo3R, a joint optimization framework that simultaneously reconstructs 3D human motion and scene geometry from casual egocentric video. It leverages the robustness of low-level visuo-geometric primitives, including monocular depth, optical flow, and point tracks, integrated with a diffusion-based human motion prior via first-order optimization. We propose an alternating optimization scheme that establishes a synergistic feedback loop: scene reconstruction provides physical grounding and trajectory conditioning for the motion prior, while the motion prior regularizes the camera trajectory and anchors the scene in a metric, gravity-aligned world frame. We evaluate EgoMo3R on the highly dynamic sequences of the EgoExo4D and Nymeria datasets and show that it outperforms specialized baselines in both 3D scene reconstruction and egocentric motion estimation.

Figures & tables

Figure 1: Given an egocentric video, EgoMo3R optimizes for camera poses , 3D pointmaps, and human motion in a common gravity-aligned metric reference frame. We also show two additional views of our scene and motion reconstruction on the side.

Figure 2: Given an input video $\boldsymbol{I}$ , we leverage a monocular depth estimator to obtain relative depth $\boldsymbol{D}^{r}$ as a geometric base, optimizing only the per-frame scales $\boldsymbol{S}$ to derive metric depth $\boldsymbol{D}$ . Our framework jointly optimizes for camera poses $\boldsymbol{C}$ , which transform metric depth into a global world-frame pointmaps $\boldsymbol{P}^{W}$ , and 3D human motion $\boldsymbol{M}$ . We employ an alternating optimization strategy: (1) Scene Refinement updates $\boldsymbol{C}$ and $\boldsymbol{S}$ using optical flow, point tracks, and a scene-alignment loss to anchor the reconstruction in a canonical frame provided by motion; (2) Motion Alignment refines $\boldsymbol{M}$ by leveraging a diffusion motion prior conditioned on $\boldsymbol{C}$ . This synergistic loop prevents optimization collapse and ensures physically grounded joint 3D scene and motion reconstruction.

		Metric Scale				Scale Invariant
		Acc		Comp		Acc		Comp
	Method	Mean	Median	Mean	Median	Mean	Median	Mean	Median
EgoExo4D	VGGT [ 71 ]	$-$	$-$	$-$	$-$	$0.470$	$0.418$	$0.440$	$0.226$
	CUT3R [ 74 ]	$\underline{0.433}$	$\underline{0.397}$	$\underline{0.665}$	$\underline{0.506}$	$\underline{0.410}$	$\underline{0.350}$	$\underline{0.341}$	${0.226}$
	ViPE [ 23 ]	$0.905$	$0.657$	$0.819$	$0.573$	$0.425$	$0.366$	$0.352$	$\underline{0.194}$
	EgoMono4D [ 82 ]	$0.619$	$0.544$	$0.707$	$0.546$	$0.469$	$0.423$	$0.593$	$0.413$

Table 1: 3D Reconstruction : Comparison across metric and scale-invariant 3D reconstruction. Lower values (

\downarrow

) indicate better performance. Our method outperforms all baselines by a significant margin.

Figure 3: 3D Reconstruction : We visualize the predicted pointmaps and camera trajectories (green) against the groundtruth trajectories (grey). The inset shows our reconstructed scene overlaid with sparse groundtruth points (grey), demonstrating the metric accuracy and alignment of EgoMo3R. Please refer to our supplementary video for a better qualitative comparison.

		Metric Scale			Scale Invariant
	Method	ATE	RPE Rot	RPE Trans	ATE	RPE Rot	RPE Trans
EgoExo4D	EgoEgo [ 32 ]	$0.421$	$5.023$	$\underline{0.089}$	$0.275$	$5.023$	$0.067$
	VGGT [ 71 ]	-	-	-	$0.251$	$3.744$	$0.078$
	CUT3R [ 74 ]	$\underline{0.415}$	$4.101$	$0.118$	$0.294$	$4.101$	$0.076$
	ViPE [ 23 ]	$0.426$	$\mathbf{2.568}$	$0.137$	$\mathbf{0.112}$	$\mathbf{2.568}$	$\mathbf{0.032}$
	EgoMono4D [ 82 ]	$1.351$	$4.445$	$0.347$	$0.345$	$4.445$	$0.077$

Table 2: Camera Pose Estimation : Comparison across metric and scale-invariant camera pose estimation. Lower values (

\downarrow

) indicate better performance.

Figure 4: Qualitative comparison of 3D human motion reconstruction. EgoMo3R effectively reconstructs physically grounded human motion that globally aligns well with the groundtruth. In contrast, errors in EgoEgo’s head trajectory estimation lead to significant global pose errors. While UniEgoMotion captures plausible local body poses, it similarly lacks the precise head tracking required for global accuracy. Please refer to our suppl. video for a better qualitative comparison.

	Method	Head Rot	Head Trans	MPJPE	MPJPE-PA	FC
EgoExo4D	EgoEgo [ 32 ]	$0.574$	$0.449$	$0.441$	$0.143$	${0.046}$
	CUT3R + EgoAllo [ 80 ]	$0.278$	$0.431$	$0.427$	$0.167$	$\mathbf{0.035}$
	CUT3R + UEM [ 49 ]	$0.526$	$0.456$	$0.407$	$0.151$	$0.039$
	ViPE + EgoAllo [ 80 ]	$0.203$	$0.411$	$0.348$	$0.155$	$0.092$
	ViPE + UEM [ 49 ]	$0.531$	$0.396$	$0.337$	$0.145$	$0.088$
	UEM-Img-only [ 49 ]	$0.519$	$0.334$	$0.309$	$\mathbf{0.073}$	$0.037$

Table 3: Human Motion Reconstruction: Lower values (

\downarrow

) indicate better performance. Our method significantly outperforms baselines on head tracking and global motion metrics, while baselines retain local pose accuracy at the cost of significantly worse alignment between the motion and the scene.

	ATE	RPE Trans	Acc		Comp		MPJPE
	ATE	RPE Trans	Mean	Median	Mean	Median	MPJPE
$\;\;\mathcal{L}_{\text{flow}}$	$0.604$	$0.222$	$0.426$	$0.352$	$0.207$	$0.122$	–
$+\,\mathcal{L}_{\text{track}}$	$0.361$	$0.111$	$0.418$	$0.342$	$0.213$	$0.139$	–
$+\,\mathcal{L}_{\text{depth}}$	$0.353$	$0.103$	$0.414$	$0.337$	$0.210$	$0.135$	–
$+\,\mathcal{L}_{\text{scene}}$	$0.371$	$0.120$	$0.398$	$0.319$	$0.211$	$0.131$	–
$+\,\mathcal{L}_{\text{motion}}+\mathcal{L}_{\text{egocam}}$ (direct)	$0.350$	$0.096$	$0.403$	$0.328$	$0.214$	$0.139$	$0.456$

Table 4: Ablation Study : Metrics are reported on EgoExo4D locomotion subset with metric scale alignment. Lower values (

\downarrow

) are better. Rows 1–4 optimize only

(\boldsymbol{C},\boldsymbol{S})

. Rows 5–6 add the motion prior, optimized directly or in an alternating manner (ours, gray). Row 7 replaces the motion prior with a camera-smoothness prior.

			Acc		Comp
	ATE	RPE Trans	Mean	Median	Mean	Median	MPJPE
Direct optimization	$0.350$	$0.096$	$0.403$	$0.328$	$0.214$	$0.139$	$0.456$
Alternating optimization
Initialization	$0.604$	$0.222$	$0.426$	$0.352$	$0.207$	$0.122$	–
R1. Optimize $(\boldsymbol{C},\boldsymbol{S})$	$0.353$	$0.102$	$0.410$	$0.333$	$0.212$	$0.137$	–
R1. Optimize $(\boldsymbol{M})$	–	–	–	–	–	–	$0.347$

Table 5: Progression of Alternating Optimization : Metrics are reported on EgoExo4D locomotion subset with metric scale alignment. Lower values (

\downarrow

) are better. Compared to direct optimization, alternation prevents the motion

\boldsymbol{M}

from collapsing to a suboptimal solution while also improving camera pose and scene reconstruction. We report results after two rounds throughout the paper (gray rows). A third round further improves pose, accuracy, and MPJPE.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Point-tracking Failure : Inaccurate long-range correspondences from the point-tracking model cause EgoMo3R to predict stationary camera trajectory (green) and stationary motion (blue), whereas the groundtruth camera and motion trajectories (grey) move in a straight line.

			Metric Scale				Scale Invariant
			Acc		Comp		Acc		Comp
		Method	Mean	Median	Mean	Median	Mean	Median	Mean	Median
EgoExo4D	Random	VGGT [ 71 ]	$-$	$-$	$-$	$-$	$0.432$	$0.382$	$0.425$	$0.231$
		CUT3R [ 74 ]	$0.402$	$0.372$	$0.613$	$0.480$	$0.385$	$0.332$	$0.314$	$0.214$
		ViPE [ 23 ]	$0.682$	$0.545$	$0.829$	$0.607$	$0.384$	$0.329$	$0.320$	$0.173$
		EgoMono4D [ 82 ]	$0.638$	$0.565$	$0.637$	$0.510$	$0.431$	$0.390$	$0.533$	$0.384$

Appendix

Table 6: 3D Reconstruction : Comparison across metric and scale-invariant 3D reconstruction on three subsets (random, locomotion-heavy, rotation-heavy) of each dataset. Lower values (

\downarrow

) indicate better performance.

			Metric Scale			Scale Invariant
		Method	ATE	RPE Rot	RPE Trans	ATE	RPE Rot	RPE Trans
EgoExo4D	Random	EgoEgo [ 32 ]	$0.246$	$2.652$	$0.051$	$0.171$	$2.652$	$0.053$
		VGGT [ 71 ]	$-$	$-$	$-$	$0.156$	$2.510$	$0.049$
		CUT3R [ 74 ]	$0.269$	$2.341$	$0.075$	$0.174$	$2.341$	$0.045$
		ViPE [ 23 ]	$0.308$	$1.378$	$0.082$	$\mathbf{0.053}$	$1.378$	$\mathbf{0.020}$
		EgoMono4D [ 82 ]	$0.971$	$2.829$	$0.252$	$0.204$	$2.829$	$0.049$

Appendix

Table 7: Camera Pose Estimation : Comparison across metric and scale-invariant camera pose estimation on three subsets (random, locomotion-heavy, rotation-heavy) of each dataset. Lower values (

\downarrow

) indicate better performance.

	Full frame				Center crop
	Metric		Scale Invariant		Metric		Scale Invariant
Method	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT [ 71 ]	$-$	$-$	$0.543$	$0.503$	$-$	$-$	$0.353$	$0.576$
CUT3R [ 74 ]	$0.456$	$0.372$	$0.175$	$0.772$	$0.445$	$0.418$	$0.292$	$0.658$
MoGe-2-L [ 76 ] ( Used in EgoMo3R )	$0.313$	$0.551$	$0.158$	$0.809$	$0.251$	${0.651}$	${0.130}$	${0.853}$
UniDepth-v2-L [ 53 ] ( Used in ViPE )	$0.262$	${0.656}$	$0.153$	$0.824$	$0.285$	$0.619$	$0.138$	$0.843$

Appendix

Table 8: Per-Frame Depth Accuracy : Depth predicted by multi-view feedforward geometry models (top) and monocular depth models (bottom) on all frames of the EgoExo4D locomotion subset, using the full rectified frames (

512\times 512

) and their central

384\times 384

crop. Monocular depth models are more accurate than feedforward geometry models on every metric in both settings, which motivates their use as our geometry basis.

	ATE	RPE Trans	Acc		Comp
Tracker + Depth	ATE	RPE Trans	Mean	Median	Mean	Median
CoTracker3 + MoGe-2 (ours)	$\mathbf{0.353}$	$\mathbf{0.103}$	$\mathbf{0.414}$	$\mathbf{0.337}$	$\mathbf{0.210}$	$\mathbf{0.135}$
CoTracker3 + MoGe	$0.389$	$0.123$	$0.415$	$0.345$	$0.213$	$0.141$
CoTracker + MoGe-2	$0.505$	$0.171$	$0.454$	$0.376$	$0.243$	$0.147$

Appendix

Table 9: Sensitivity to Off-the-Shelf Priors : We replace the point tracker or the depth model in the scene-only setting (row 3 of Tab. 4 ). Lower values (

\downarrow

) indicate better performance. Weaker priors degrade the results gradually, without catastrophic failure. The weaker tracker has the larger effect, mainly on ATE and RPE Trans, since less reliable long-range correspondences lead to more drift.

	Acc		Comp		ATE	RPE Rot	RPE Trans
Method	Mean	Median	Mean	Median	ATE	RPE Rot	RPE Trans
VGGT [ 71 ]	out of memory
CUT3R [ 74 ]	$0.614$	$0.506$	$0.621$	$0.401$	$1.226$	$10.879$	$\mathbf{0.251}$
ViPE [ 23 ]	$0.735$	$0.633$	$0.732$	$0.474$	$1.668$	$\mathbf{8.308}$	$0.416$
EgoMono4D [ 82 ]	$0.786$	$0.699$	$0.836$	$0.554$	$2.411$	$8.851$	$0.415$
Ours	$\mathbf{0.505}$	$\mathbf{0.398}$	$\mathbf{0.212}$	$\mathbf{0.105}$	$\mathbf{0.913}$	$9.605$	$0.343$

Appendix

Table 10: Long Sequences : Results on 24 s EgoExo4D locomotion sequences,

3\times

the length of the main evaluation clips, with metric scale alignment. Lower values (

\downarrow

) indicate better performance. Our motion prior, trained on 8 s windows, is applied over overlapping windows with MultiDiffusion [ 2 ] ; the rest of the optimization is unchanged. VGGT runs out of memory on a 48 GB NVIDIA L40S GPU. Even without explicit loop closure, EgoMo3R remains best on all reconstruction metrics and on ATE, which we attribute to the long-range tracks from offline CoTracker3 [ 25 ] .

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 978 papers

Seeing the World and the Self from Egocentric Video
Sep 1, 2026Kai Guan, Minchao Jiang, Ruichen WangLi +2Markerless Motion Capture 3D Reconstruction
The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo
OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction
Dec 22, 2025Kyungwon Cho, Jeonghyeon Na, Hanbyul Joo3D Human Motion Reconstruction Markerless Motion Capture
Seoul National University
EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
Oct 19, 2026Xinhao Ji, Hui En Pang, Hao Li +123D Reconstruction Hand-Object Interaction Reconstruction
Nanyang Technological University · Peking University · Ropedia +1

Sep 1, 2026J/K move · Enter open · S save

Seeing the World and the Self from Egocentric Video

Kai Guan, Minchao Jiang, Ruichen WangLi, Wentao Zhu+1

The Hong Kong Polytechnic University · Eastern Institute of Technology, Ningbo

Markerless Motion Capture 3D Reconstruction Camera Pose Estimation Egocentric Video Understanding

Open paper AskPDF arXiv

Complete 3D perception from egocentric video requires recovering the surrounding scene and the wearer's full-body motion in a shared metric frame. Existing methods typically address scene reconstruction and motion estimation separately: scene reconstruction methods ignore the wearer, whereas motion estimation methods lack explicit scene geometry and often depend on external trajectories. Joint recovery is challenging because the two tasks exhibit asymmetric visibility and require different prediction paradigms. The largely visible scene supports deterministic geometric regression, whereas the severely occluded body requires generative motion inference. We therefore propose RESELF (REconstructing the Scene and the sELF), a unified framework that couples deterministic metric geometry reconstruction with geometry-conditioned motion generation. RESELF adapts a geometry foundation model pre-trained on large-scale exocentric data to egocentric video using frame-wise scale and relative-pose consistency objectives. The resulting camera trajectory and latent geometric features condition a diffusion model that recovers the wearer's motion. A subsequent closed-loop kinematic feedback stage further refines the camera head while preserving the reconstructed scene geometry. To support training and evaluation, we curate EE4D-JSM from EgoExo4D by aligning egocentric video, sparse metric scene geometry, camera trajectories, and full-body motion annotations. Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation. Code, models, and datasets will be available at https://ka1guan.github.io/RESELF/.

Abstract

3D Reconstruction

Explore similar work

Seeing the World and the Self from Egocentric Video

OmniEgoCap: Device-Agnostic Sequence-Level Egocentric Motion Reconstruction

EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience