FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos

Alexandros Delitzas; Chenyangguang Zhang; Alexey Gavryushin; Tommaso Di Mario; Boyang Sun; Rishabh Dabral; Leonidas Guibas; Christian Theobalt

cs.CVApr 7, 2026

FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos

Authors: Alexandros Delitzas, Chenyangguang Zhang, Alexey Gavryushin, Tommaso Di Mario, Boyang Sun, Rishabh Dabral, Leonidas Guibas, Christian Theobalt,

Organizations: ETH Zurich · Max Planck Institute for Informatics · Stanford University · Microsoft ·

Abstract

We present FunRec, a method for reconstructing functional 3D digital twins of indoor scenes directly from egocentric RGB-D interaction videos. Unlike existing methods on articulated reconstruction, which rely on controlled setups, multi-state captures, or CAD priors, FunRec operates directly on in-the-wild human interaction sequences to recover interactable 3D scenes. It automatically discovers articulated parts, estimates their kinematic parameters, tracks their 3D motion, and reconstructs static and moving geometry in canonical space, yielding simulation-compatible meshes. Across new real and simulated benchmarks, FunRec surpasses prior work by a large margin, achieving up to +50 mIoU improvement in part segmentation, 5-10 times lower articulation and pose errors, and significantly higher reconstruction accuracy. We further demonstrate applications on URDF/USD export for simulation, hand-guided affordance mapping and robot-scene interaction.

Figures & tables

Figure 2: Method overview . Given an egocentric RGB-D interaction video, FunREC first divides it into static and dynamic fragments. For each dynamic fragment, it estimates camera poses, computes sparse 3D point trajectories, and clusters them into articulated components via articulation-aware motion modeling. The interacting part is then segmented to obtain dense masks and reconstructed together with the static scene. Part pose and articulation parameters are optimized jointly to yield consistent motion across frames. Finally, all reconstructed fragments are globally aligned to produce a coherent, functional 3D digital twin where articulated parts can be interactively manipulated.

Figure 3: Proposed Datasets. We introduce two datasets for functional 3D scene reconstruction and evaluation: RealFun4D (left) , capturing egocentric interactions in real scenes, and OmniFun4D (right) , providing photorealistic simulated interactions in synthetic scenes.

Figure 4: Qualitative comparisons . We show qualitative comparisons between baselines and our method. For each method, we accumulate the reconstructed point clouds of both the articulated part and the static scene across all timesteps, and visualize them under two selected scene states. Green indicates the articulated part, and red lines denote the estimated articulation axes.

Methods	OmniFun4D				HOI4D				RealFun4D
Methods	Axis ( ∘ )	Pos (m)	State ( ∘ /m)	Fail (%)	Axis ( ∘ )	Pos (m)	State ( ∘ /m)	Fail (%)	Axis ( ∘ )	Pos (m)	State ( ∘ /m)	Fail (%)
MonST3R [ 82 ] (ICP)	$52.7/61.5$	$1.14$	$66.2/0.27$	$11.7$	$67.3/66.1$	$0.45$	$60.8/0.06$	$\mathbf{{0.0}}$	$54.6/59.1$	$0.68$	$65.9/0.21$	$\mathbf{{0.0}}$
MonST3R (CoTr3)	$46.8/58.9$	$1.20$	$45.3/0.18$	$11.7$	$54.7/51.5$	$0.44$	$44.8/0.07$	$\mathbf{{0.0}}$	$56.8/52.4$	$0.70$	$57.4/0.22$	$\mathbf{{0.0}}$
MonST3R (GT depth+CoTr3)	$57.6/63.5$	$1.10$	$46.5/0.18$	$11.7$	$48.6/30.1$	$0.41$	$37.6/0.06$	$\mathbf{{0.0}}$	$51.4/56.2$	$0.55$	$54.3/0.21$	$\mathbf{{0.0}}$
SpatialTrackerV2 [ 75 ]	$59.8/69.1$	$1.22$	$48.6/0.19$	$30.0$	$57.9/44.6$	$0.46$	$41.9/0.07$	$\mathbf{{0.0}}$	$59.4/49.8$	$0.49$	$57.5/0.22$	$20.0$
SpatialTrackerV2 (GT depth)	$48.7/69.3$	$1.10$	$46.8/0.19$	$30.0$	$54.3/38.8$	$0.43$	$38.7/0.07$	$\mathbf{{0.0}}$	$60.8/42.6$	$0.58$	$55.6/0.22$	$20.0$

Table 1: Articulated motion estimation. We report articulation axis direction error ( ∘ ), axis position error (meters; applicable only to revolute joints), joint state error ( ∘ for revolute and meters for prismatic joints), and failure rate (%). Values are shown as “XX / YY” for revolute (XX) and prismatic (YY) joints. Best , second-best , and third-best results are highlighted. All metrics are lower-is-better.

Methods	OmniFun4D			HOI4D			RealFun4D
	6D part pose		Reconstruction	6D part pose		Reconstruction	6D part pose		Reconstruction
	ADD-S (%) $\uparrow$	ADD (%) $\uparrow$	CD (cm) $\downarrow$	ADD-S (%) $\uparrow$	ADD (%) $\uparrow$	CD (cm) $\downarrow$	ADD-S (%) $\uparrow$	ADD (%) $\uparrow$	CD (cm) $\downarrow$
MonST3R [ 82 ] (ICP)	$14.36$	$\phantom{1}9.62$	$29.8$	$29.41$	$19.12$	$6.4$	$13.21$	$\phantom{1}9.11$	$13.4$
MonST3R [ 82 ] (CoTr3)	$33.58$	$21.94$	$23.6$	$44.72$	$25.98$	$2.1$	$18.64$	$11.35$	$17.6$
MonST3R [ 82 ] (GT depth+CoTr3)	$37.12$	$30.48$	$13.9$	$54.83$	$37.11$	$1.3$	$20.11$	$14.62$	$13.2$
SpatialTrackerV2 [ 75 ]	$26.94$	$13.11$	$34.7$	$61.25$	$41.02$	$0.9$	$19.77$	13.92	$18.5$

Table 2: 6D part pose estimation and reconstruction. We report 6D part pose accuracy using ADD-S and ADD (higher is better) and surface reconstruction quality using Chamfer Distance (CD, lower is better) across the OmniFun4D, HOI4D, and RealFun4D datasets.

Methods	OmniFun4D	HOI4D	RealFun4D
Methods	mIoU $\uparrow$	mIoU $\uparrow$	mIoU $\uparrow$
MonST3R [ 82 ]	$23.6$	$26.8$	$23.7$
SpatialTrackerV2 [ 75 ] (SAM2)	1 $6.2$	1 $5.8$	$13.4$
FunREC (Ours)	$\mathbf{{77.9}}$	$\mathbf{{76.4}}$	$\mathbf{{74.8}}$

Table 3: Moving part segmentation. Mean Intersection-over-Union (mIoU) for moving-part segmentation.

Figure 5: Isaac Sim deployment. A mobile manipulator interacts with a drawer reconstructed from a real-world scan.

Panel of Figure 6 — Figure 6: Hand-scene interaction. Estimated 3D hand mesh ( left ) and inferred affordance map ( right ). Integrating the hand pose into our functional reconstruction enables finding contact regions and consistent reasoning over the associated scene-part motion.

Figure 7: Robot-scene interaction . Left: Human demonstration of opening a cabinet. Center: The articulation trajectory derived from the functional scene model. Right: The robot leverages the functional information to reliably reproduce the same interaction.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 955 papers

EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates
Apr 13, 2026Weikun Peng, Denys Iliash, Manolis SavvaArticulated Object Reconstruction 3D Part Segmentation
Simon Fraser University
Articulated Object Reconstruction from Rest-State Observation
Jul 30, 2026Daeun Lee, Jaeah Lee, Woosung Kim +2Articulated Object Reconstruction 3D Mesh Reconstruction
Seoul National University, Republic of Korea
Track, Articulate, Act: Generating Articulation from Casual Human Videos
Sep 16, 2026Jiaming Zhang, Homanga BharadhwajArticulated Object Reconstruction Articulated Object Manipulation
Department of Computer Science, Johns Hopkins University

Apr 13, 2026J/K move · Enter open · S save

EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

Weikun Peng, Denys Iliash, Manolis Savva

Simon Fraser University

Articulated Object Reconstruction 3D Part Segmentation 3D Reconstruction Egocentric Video Understanding

Open paper AskPDF arXiv

We present EgoFun3D, a coordinated task formulation, dataset, and benchmark for modeling interactive 3D objects from egocentric videos. Interactive objects are of high interest for embodied AI but scarce, making modeling from readily available real-world videos valuable. Our task focuses on obtaining simulation-ready interactive 3D objects from egocentric video input. While prior work largely focuses on articulations, we capture general cross-part functional mappings (e.g., rotation of stove knob controls stove burner temperature) through function templates, a structured computational representation. Function templates enable precise evaluation and direct compilation into executable code across simulation platforms. To enable comprehensive benchmarking, we introduce a dataset of 517 egocentric videos featuring challenging real-world interactions with paired 3D geometry, segmentation over 2D and 3D, articulation and function template annotations. To tackle the task, we propose a 4-stage pipeline consisting of: 2D part segmentation, reconstruction, articulation estimation, and function template inference. Comprehensive benchmarking shows that the task is challenging for off-the-shelf methods, highlighting avenues for future work.

Abstract

3D Reconstruction

Explore similar work

EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates

Articulated Object Reconstruction from Rest-State Observation

Track, Articulate, Act: Generating Articulation from Casual Human Videos