Static Scene Reconstruction from Dynamic Egocentric Videos

Qifei Cui; Patrick Chen

cs.CVMar 23, 2026

Static Scene Reconstruction from Dynamic Egocentric Videos

Authors: Qifei Cui, Patrick Chen

Organizations: Department of Computer and Information Science, University of Pennsylvania

Abstract

Egocentric videos present unique challenges for 3D reconstruction due to rapid camera motion and frequent dynamic interactions. State-of-the-art static reconstruction systems, such as MapAnything, often degrade in these settings, suffering from catastrophic trajectory drift and "ghost" geometry caused by moving hands. We bridge this gap by proposing a robust pipeline that adapts static reconstruction backbones to long-form egocentric video. Our approach introduces a mask-aware reconstruction mechanism that explicitly suppresses dynamic foreground in the attention layers, preventing hand artifacts from contaminating the static map. Furthermore, we employ a chunked reconstruction strategy with pose-graph stitching to ensure global consistency and eliminate long-term drift. Experiments on HD-EPIC and indoor drone datasets demonstrate that our pipeline significantly improves absolute trajectory error and yields visually clean static geometry compared to naive baselines, effectively extending the capability of foundation models to dynamic first-person scenes.

Figures & tables

Figure 1: Overall pipeline. Module A constructs a binary dynamic activity prior $D_{t}$ from VLM (Gemini) prompts and SAM3 video segmentation/tracking, optionally gated by interaction time windows. Module B runs chunk-wise reconstruction and fusion (MapAnything) while ignoring dynamic pixels indicated by $D_{t}$ , then stitches chunks using overlap-based Sim(3) alignment to obtain globally consistent poses and a fused static map.

Figure 2: Module A: interaction-aware dynamic prior pipeline. Given egocentric video frames (or reconstruction chunks), a VLM selects scene-relevant semantic categories and text prompts (e.g., hand , plate , cup ). SAM3 then performs concept-driven video segmentation and tracking to produce per-category instance masks with consistent IDs. We aggregate these masks into a per-frame dynamic prior $D_{t}$ (union over categories, optionally filtered by proximity to the hand to isolate manipulated objects). Finally, MapAnything consumes $D_{t}$ to suppress dynamic regions and reconstruct the static scene with reduced ghosting artifacts.

Figure 3: Interaction-activated dynamic prior. We show a SAM3 tracking frame with instance IDs together with the corresponding interaction-activated binary prior $D_{t}$ (hands always masked; objects masked cumulatively after onset), which is used to construct attention masks in the reconstruction module.

Figure 4: Dynamic prior injection for robust 3D reconstruction. We augment the standard transformer attention between 3D query tokens (cameras / 3D points) and image tokens (pixel patches) by gating cross-attention with a dynamic mask $D_{t}$ . Tokens marked as dynamic (red) are suppressed, so 3D queries attend primarily to static regions (green), reducing motion-induced contamination and ghosting artifacts in the reconstructed scene.

Variant	Suppression	Mask strategy	Stitching
Baseline	none	-	Sim(3)
Blackout	pixel	dynamic-only / cumulative	Sim(3)
Attn gating (dynamic-only)	attention	per-frame	Sim(3)
Attn gating (cumulative)	attention	after-onset, persistent	Sim(3)

Table 1: Summary of evaluated variants.

Variant	$e^{\text{cen}}{\downarrow}$	$B_{\text{all}}{\downarrow}$	$B_{\text{static}}{\downarrow}$	$D_{\text{all}}{\uparrow}$	$D_{\text{static}}{\uparrow}$	$\bar{s}$
Baseline	0.019497	0.035132	0.034525	0.945419	0.952275	1.004606
Blackout	0.024591	0.043171	0.042285	0.951403	0.960077	0.998633
Attn gating (dynamic-only)	0.051678	0.083331	0.083431	0.800966	0.828438	0.997084
Attn gating (cumulative) (ours)	0.072496	0.100158	0.096840	0.780635	0.809198	1.003003

Table 2: Overlap consistency and completeness (6 Hz, 8 chunks). We condition static-region metrics (

B_{\text{static}}

D_{\text{static}}

) on the footprint evaluation mask union_mask_fulltime to avoid counting any ever-dynamic pixels as static. Dynamic suppression increases overlap residuals and geometry discrepancy, and is accompanied by a large drop in depth completeness (

D_{\text{all}}

from 0.945419 to 0.800966 / 0.780635). Scale remains stable (mean scale

\bar{s}\approx 1

Figure 6: Static reconstruction with/without dynamic suppression. Columns (left $\rightarrow$ right): baseline MapAnything (no suppression), naive pixel-level blackout, and our token-level attention gating. Rows (top $\rightarrow$ bottom): full-scene view and a zoom-in on an interaction-heavy region. Our method removes dynamic ghosting while better preserving clean static geometry than pixel blackout.

	Instantaneous mask (cov=0.077415)			Footprint mask (cov=0.168951)
Variant	$C^{\text{den}}_{\text{ratio}}{\downarrow}$	$C^{\text{occ}}_{\text{ratio}}{\downarrow}$	$C^{\text{od}}_{\text{ratio}}{\downarrow}$	$C^{\text{den}}_{\text{ratio}}{\downarrow}$	$C^{\text{occ}}_{\text{ratio}}{\downarrow}$	$C^{\text{od}}_{\text{ratio}}{\downarrow}$
Baseline	0.998740	1.003221	0.988921	1.087414	1.088884	0.997985
Blackout	0.992527	1.010007	0.974469	1.072562	1.085382	0.986884
Attn gating (dynamic-only)	0.823669	0.848722	0.961242	0.976011	0.988040	0.990571
Attn gating (cumulative) (ours)	0.807543	0.835834	0.955473	0.963738	0.978473	0.988066

Table 3: Density-style contamination (6 Hz, 8 chunks). Lower indicates fewer projected points supported in dynamic regions relative to static regions. Instantaneous evaluation shows strong suppression:

C^{\text{den}}_{\text{ratio}}

drops from 0.998740 (baseline) to 0.823669 (dynamic-only) and 0.807543 (cumulative). Footprint evaluation (harder, less diluted by “currently static” pixels) still shows consistent reductions from 1.087414 to 0.976011 and 0.963738. Naive blackout yields only minor changes compared to attention gating.

Panel of Figure 7 — Figure 7: Overlap alignment for stitching. Camera-center trajectories over the overlap between two adjacent chunks. Left: raw (unstitched). Right: after Umeyama Sim(3) alignment with scale.

Panel of Figure 8 — Figure 8: EPIC-KITCHENS Sense point-cloud reconstructions comparing four masking strategies across three scenes (rows). Baseline (no masking) exhibits severe hand ghosting. Blackout removes dynamic regions at the image level, resulting in large black voids. Dynamin (ours) suppresses most ghosting while preserving sufficient static context to maintain spatial consistency. Overall-mask removes all moving objects from the beginning, often leading to unstable or incomplete structure due to missing spatial cues.

Panel of Figure 9 — Figure 9: Semantic grounding vs. ID fragmentation (Grounded-SAM2). Grounded-SAM2 provides semantic relevance but exhibits severe ID fragmentation in long egocentric videos; our post-hoc matching reduces the number of fragmented IDs but does not fully resolve failures under heavy occlusion and re-entry.

Figure 10: Temporal stability vs. over-segmentation (SAM-Track + DeAOT). While long-term memory improves ID retention, the method progressively segments and tracks many irrelevant regions in long videos, yielding an explosion in tracked objects without semantic filtering.

Panel of Figure 11 — Figure 11: SAM3D single-frame reconstruction (exploratory). For each object category, we show the input image (with mask) and the generated textured mesh. While outside our main scope, these results suggest a future direction for object-level reconstruction on top of our static mapping pipeline.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 955 papers

Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction
Apr 26, 2026Jan Warchocki, Xi Wang, Jonas Kulhanek +13D Gaussian Splatting Monocular 3D Reconstruction
Delft University of Technology · ETH Zurich · TUM/MCML +1
DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction
Apr 9, 2026Tingxi Chen, Zhengxue Cheng, Houqiang Zhong +3Dynamic Scene Reconstruction 3D Gaussian Splatting
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University · Nvidia.
Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos
Jun 17, 2026Jeongmin Bae, Seoha Kim, Marc Pollefeys +33D Gaussian Splatting 4D Reconstruction
Yonsei University · Electronics and Telecommunications Research Institute · ETH Zurich +2

Apr 26, 2026J/K move · Enter open · S save

Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction

Jan Warchocki, Xi Wang, Jonas Kulhanek, Jan van Gemert

Delft University of Technology · ETH Zurich · TUM/MCML+1

3D Gaussian Splatting Monocular 3D Reconstruction Dynamic Scene Reconstruction Egocentric Vision

Open paper AskPDF arXiv

Egocentric video provides a unique view into human perception and interaction, with growing relevance for augmented reality, robotics, and assistive technologies. However, rapid camera motion and complex scene dynamics pose major challenges for 3D reconstruction from this perspective. While 3D Gaussian Splatting (3DGS) has become a state-of-the-art method for efficient, high-quality novel view synthesis, variants, that focus on reconstructing dynamic scenes from monocular video are rarely evaluated on egocentric video. It remains unclear whether existing models generalize to this setting or if egocentric-specific solutions are needed. In this work, we evaluate dynamic monocular 3DGS models on egocentric and exocentric video using paired ego-exo recordings from the EgoExo4D dataset. We find that reconstruction quality is consistently lower in egocentric views. Analysis reveals that the difference in reconstruction quality, measured in peak signal-to-noise ratio, stems from the reconstruction of static, not dynamic, content. Our findings underscore current limitations and motivate the development of egocentric-specific approaches, while also highlighting the value of separately evaluating static and dynamic regions of a video.

Abstract

3D Reconstruction

Explore similar work

Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction

DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstruction

Hand-4DGS: Feed-Forward 3D Gaussian Splatting for 4D Hand Reconstruction from Egocentric Videos