Articulated 3D Scene Graphs for Open-World Mobile Manipulation

Martin Büchner; Adrian Röfer; Tim Engelbracht; Tim Welschehold; Zuria Bauer; Hermann Blum; Marc Pollefeys; Abhinav Valada

cs.ROFeb 18, 2026

Articulated 3D Scene Graphs for Open-World Mobile Manipulation

Authors: Martin Büchner, Adrian Röfer, Tim Engelbracht, Tim Welschehold, Zuria Bauer, Hermann Blum, Marc Pollefeys, Abhinav Valada

Organizations: University of Freiburg · ETH Zürich · University of Bonn

Abstract

Semantic scene understanding has enabled robots to reason and perform affordance-driven object interaction. Despite this, real-world manipulation remains fundamentally limited by the inability to predict how objects move and are articulated. Long-horizon mobile manipulation requires closing the gap between semantics, geometry, and kinematics. In this work, we present MoMa-SG, a novel framework for building semantic-kinematic 3D scene graphs of articulated scenes containing a myriad of interactable objects. Given RGB-D sequences containing multiple object articulations, we temporally segment object interactions and infer object motion using occlusion-robust point tracking. We lift point trajectories to 3D and estimate articulation models using a novel unified twist estimation formulation that robustly estimates revolute and prismatic joint parameters in a single optimization pass. Next, we associate objects with articulations and detect contained objects by reasoning over parent-child relations at identified opening states. We also introduce the Arti4D-Semantic dataset, which provides parent-child relation annotations between container and contained objects together with object axis annotations across 62 in-the-wild RGB-D sequences containing 600 object interactions and three distinct observation paradigms. We extensively evaluate the performance of MoMa-SG on two datasets and ablate key design choices of our approach. In addition, real-world experiments on both a quadruped and a mobile manipulator demonstrate robust manipulation of articulated objects in everyday home environments. We provide code and data at: https://momasg.cs.uni-freiburg.de.

Figures & tables

Figure 1: MoMa-SG enables the construction of accurate semantic-kinematic 3D scene graphs over articulated scenes and serves as a backbone for long-horizon and language-enabled interactive mobile manipulation.

Figure 2: Overview of MoMa-SG: We first discover interaction segments ( Sec. 3.1 ), then attain object articulation models $\mathcal{A}$ by estimating twists from point trajectories ( Sec. 3.2 ). We match mapped objects $\mathcal{O}$ against articulations and discover objects contained in respective articulated parents ( Sec. 3.3 ).

	Arti4D						DROID
Method	Prismatic	Revolute		Joint Type [%]		Mode	Prismatic	Revolute		Type
Method	$\theta_{err}$ [ ∘ ]	$\theta_{err}$ [ ∘ ]	$d_{L2}$ [m]	Acc.	Recall Pris./Rev.	Acc. [%]	$\theta_{err}$ [ ∘ ]	$\theta_{err}$ [ ∘ ]	$d_{L2}$ [m]	Acc.[%]
Ours w/ CoTracker3	13.190	22.982	0.091	88.4	91.7 / 86.6	98.3
Ours w/ 32x32 queries	14.982	18.424	0.110	86.1	82.5 / 92.9
Ours w/ 96x96 queries	09.263	16.704	0.066	93.8	91.5 / 100.0
Ours w/o $\xi$ -regul. + w/o RANSAC	34.089	23.175	0.103	90.9	87.7 / 97.0

Table 1: Articulated object estimation results on Arti4D. We report axis-angle errors

\theta_{err}

, pos. errors

d_{L2}

of rev. joints, type accuracy, and mode recognition quality. We conduct the ablation study on Arti4D, see top 6 rows.

Figure 3: Qualitative results of MoMa-SG on Arti4D-Semantic: Estimated axis positions and corresponding object masks. As demonstrated, we observe minimal errors for a large number of prismatic objects and small errors on revolute objects. Revolute axes are depicted with a blue torus, prismatic axes without.

Panel of Figure 4 — Figure 4: Articulated 3D scene graphs: The top node represents the root node, the cube-shaped nodes represent articulated objects (parents), while the smaller, lower nodes represent discovered child nodes with respect to their articulated parents.

	3D Part Mapping					Contained Object Discovery
Method	Object	IoU		Recall		IoU	Recall [%]	Relation Acc. [%]
Method	Set	IoU	@0.25	@0.50	@0.75	IoU	@IoU=0.25	STATIC / ARTIC.
OpenMask3D [ 56 ]	$\mathcal{O}$	0.082	0.087	0.019	0.000
ConceptGraphs [ 4 ]		0.438	0.332	0.127	0.018
Ours w/o bound.-resp. merging		0.481	0.807	0.534	0.086
MoMa-SG (ours)		0.533	0.824	0.646	0.163

Table 2: Articulated 3D part segmentation (left) and contained object discovery (right) evaluation on Arti4D-Semantic: The upper four rows refer to free articul.-indep. objects

\mathcal{O}

, the lower section contains objects

\mathcal{O}^{\mathcal{A}}

matched against articulations. The IoU denotes mutual overlap under a linear assignment. The recall indicates the GT overlap at various IoUs. For object discovery (right), we only evaluate objects paired with GT objects on the, the IoU denotes the highest-achieved overlap between a predicted child node and all GT children. The relation accuracy measures correctly identified parent-child edges. Evaluation on RH201 split of Arti4D-Semantic.

Figure 5: We portray our real-world mobile manipulation environments of the HSR as well as the Spot. We demonstrate that MoMa-SG allows for robust mobile manipulation even under various articulation states of objects. The sample size for each robot-task combination is 50.

Figure A.1: We demonstrate the ability of MoMa-SG to map natural language commands (e.g., “get the milk from the fridge”) to an appropriate sequence of high-level actions, such as opening the fridge, inspecting its contents, and retrieving the target object. All of this is done from open-vocabulary language features and object articulation models estimated from ego-centric video in a one-shot fashion while not relying on any fiducial markers.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.2: Estimated twist components given an observed prismatic motion. We depict observed 3D point trajectories (left), sampled point pairs used for articulation estimation (middle), and the estimated articulation model (right).

Figure A.3: Estimated twist components given an observed revolute motion of small angular range using our proposed regularized twist estimation scheme. We visualize point tracks (left), sampled point pairs (middle), and rotational and translation twist components (right).

Figure A.4: Articulation mode prompt consisting of four concatenated keyframes along the interaction.

Figure A.5: Axis and position prediction errors of a regularized twist estimator (ours), under a privileged purely prismatic/revolute model (prismatic/revolute twist), and a non-regularized twist at various articulation configurations sampled synthetically and subject to point-level Gaussian noise.

Figure A.6: We depict dense 2D segmentation masks across two distinct keyframes and denote masks considered as isolated and non-isolated, respectively. The scene stems from the HSR split of Arti4D-Semantic.

Figure A.7: Dense object segmentation for object reconstruction using SAM2 based on repeated TapNext++ initialization along each interaction.

Figure A.8: Qualitative results of articulation-to-object association scheme introduced in Sec. 3.3 . Across all depicted frames, the shaded mask represents the most appropriate identified free object when articulated under the estimated articulation model that is projected into the camera frame as a red axis. Solid green image boundaries denote successful pairings, dashed yellow intermediate cases represent either wrong segmentations or misidentified articulation magnitudes under accurate articulation models, and dotted red boundaries denote wrongly associated object masks under permissible estimated axes.

Panel of Figure 16 — Figure A.9: We depict the object discovery component of MoMa-SG. Each of the 10 frames displays identified maximum-opening object states. Instance masks represent identified child objects wrt. the articulated parent.

Figure A.10: Selection of grasps constructed using our handle detection pipeline employing SAM3 [ 55 ] and predominant axis estimation. The blue axis of each grasp constitutes its approach axis.

Figure A.11: Qualitative full-scene results on articulated object estimation and object association on splits of Arti4D-Semantic.

Figure A.12: Estimated articulated 3D objects: The colored arrow represents the articulation axis and the distinctly colored point cloud represents the object. A torus around an axis denotes a revolute axis.

Panel of Figure 20 — Figure A.13: We depict the ability of MoMa-SG to perform long-horizon mobile manipulation of articulated scenes using the Toyota HSR. Throughout this episode (left to right, top to bottom), the robot successfully opened 7 different articulated objects, 3 revolute objects, and 4 prismatic objects. Throughout our experiments, we found that grasping failures are much more dominant than errors due to mediocre articulation axes.

Figure A.14: We depict three consecutive opening actions conducted using the Boston Dynamics Spot.

Split	Paradigm	GT Poses	Prismatic joints		Revolute joints		Type Quality [%]			Mode
Split	Paradigm	GT Poses	$\theta_{err}$ [deg]	$d_{L2}$ [m]	$\theta_{err}$ [deg]	$d_{L2}$ [m]	Accuracy	Prismatic Recall	Revolute Recall	Acc.[%]
RH078	EGO	✓	05.43	–	15.88	0.063	0.862	0.840	0.900	0.980
RR080	EGO	✓	19.36	–	14.80	0.035	0.808	0.792	1.000	0.952
DR080	EGO	✓	07.99	–	15.67	0.094	0.940	0.922	0.970	1.000
RH201	EGO	✓	03.52	–	13.97	0.099	0.979	0.973	0.979	0.993
MHZH	EGO	$\times$	42.01	–	21.34	0.271	0.917	0.727	1.000	0.972

Appendix

Table A.1: Split-wise articulated object estimation on Arti4D-Semantic using ground truth interaction segmentation: We report the axis-angle errors

\theta_{err}

, the positional errors

d_{L2}

of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1 .

Method	Split	Paradigm	GT Poses	IoU	Recall
Method	Split	Paradigm	GT Poses	IoU	@0.25	@0.50	@0.75
w/ CoTracker3	RH078	EGO	✓	0.184	0.327	0.181	0.000
	RR080	EGO	✓	0.196	0.347	0.154	0.035
	DR080	EGO	✓	0.243	0.387	0.241	0.019
	RH201	EGO	✓	0.262	0.416	0.287	0.049
	MHZH	EGO	$\times$	0.208	0.317	0.108	0.025

Appendix

Table A.2: Split-wise articulated 3D part segmentation on Arti4D-Semantic: We report the same set of metrics as in Tab. 2 . Below, we list the performance using either CoTracker3 [ 53 ] or TapNext++ [ 46 ] , which is employed in default MoMa-SG.

Object Set	Split	$top_{1}$	$top_{2}$	$top_{5}$	$top_{10}$
$\mathcal{O}$	RH078	0.387	0.506	0.660	0.830
	RR080	0.069	0.174	0.452	0.532
	DR080	0.489	0.663	0.774	0.888
	RH201	0.410	0.605	0.781	0.915
	MHZH	0.357	0.523	0.725	0.980
	MHZH-EXO	0.333	0.562	0.896	1.000

Appendix

Table A.3: Articulated 3D part semantics evaluation on Arti4D-Semantic: We report open-vocabulary top-k retrieval scores as outlined in Sec. A.2-A .

Method	1D-IoU	P	R	Segment-IoU	$I_{\text{on}}$ [ $\mathrm{s}$ ]	$I_{\text{off}}$ [ $\mathrm{s}$ ]
GPT-5-mini	0.55	0.862	0.6111	0.753	8.179	10.12
Pandora [ 44 ]	0.359	0.690	0.400	0.689	0.7442	0.725
HMM	0.471	0.871	0.532	0.683	0.770	0.762
ArtiPoint [ 15 ]	0.575	0.678	0.714	0.698	1.05	0.558
MoMa-SG	0.649	0.786	0.800	0.718	0.816	0.742

Appendix

Table A.4: Temporal interaction segmentation evaluation on Arti4D-Semantic: The 1D-IoU metric denotes the intersection over union of the dense binary timeseries signals, P and R, as well as the Segment-IoU denote the precision and recall, as well as the segment-wise IoU, following a linear sum assignment between predicted and ground truth segments under an IoU

>

0.5.

I_{\text{on}}

and

I_{\text{off}}

measure the absolute start and end errors of matched segments.

Object	Trans. Error		Rot. Error
Type	$\Delta d$ [ $\mathrm{m}$ ]	$\sigma$ [ $\mathrm{m}$ ]	$\Delta\theta$ [ $\mathrm{\SIUnitSymbolDegree}$ ]	$\sigma$ [ $\mathrm{\SIUnitSymbolDegree}$ ]
Prismatic	0.0174	0.006	-	-
Revolute	-	-	6.477	3.604

Appendix

Table A.5: Real-world articulation state estimation: Experiments are conducted across 54 various object articulation states.

Method	Prismatic joints		Revolute joints		Failure	Joint Type [%]		Mode
Method	$\theta_{err}$ [deg]	$d_{L2}$ [m]	$\theta_{err}$ [deg]	$d_{L2}$ [m]	Rate [%]	Accuracy	Pris./Rev. Recall	Acc.[%]
Pandora [ 44 ]	46.31	–	50.23	0.182	00.0	55.8	45.0 / 75.9	-
ArtiPoint [ 15 ]	24.88	–	30.24	0.293	23.2	77.4	88.2 / 72.2	-
MoMa-SG (ours)	20.66	–	17.31	0.071	10.5	82.0	73.8 / 98.4	80.7

Appendix

Table A.6: Articulated object estimation on Arti4D across methods under predicted interaction segments: Our evaluation matched predicted articulations against ground truth articulations if a 1D-IoU of at least 0.25 is exceeded. We report the axis-angle errors

\theta_{err}

, the positional errors

d_{L2}

of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1 .

Figure A.15: Contained object discovery given the articulated 3D parent objects.

Split	Paradigm	GT Poses	Prismatic joints		Revolute joints		Type Quality [%]			Mode
Split	Paradigm	GT Poses	$\theta_{err}$ [deg]	$d_{L2}$ [m]	$\theta_{err}$ [deg]	$d_{L2}$ [m]	Accuracy	Prismatic Recall	Revolute Recall	Acc.[%]
RH078	EGO	✓	17.58	–	06.42	0.035	0.794	0.667	1.000	0.825
RR080	EGO	✓	29.84	–	32.76	0.118	0.657	0.626	1.000	0.879
DR080	EGO	✓	16.32	–	21.57	0.062	0.889	1.000	0.816	0.815
RH201	EGO	✓	12.94	–	17.34	0.085	0.919	0.877	0.966	0.734
	Overall		20.66	–	17.31	0.071	0.820	0.738	0.984	0.807

Appendix

Table A.7: Articulated object estimation on Arti4D across dataset splits: We report the axis-angle errors

\theta_{err}

, the positional errors

d_{L2}

of revolute joints, and the accuracy of predicted joint types, including their respective recalls as in Tab. 1 .

Method	Prismatic joints	Revolute joints
Method	$\theta_{err}$ [deg] $\pm\sigma$	$\theta_{err}$ [deg] $\pm\sigma$	$d_{L2}$ [m] $\pm\sigma$
Pandora	46.81 $\pm$ 37.49	50.62 $\pm$ 28.58	0.20 $\pm$ 0.18
	{ 47.31, 85.58, 88.43 }	{ 51.72, 79.69, 86.34 }	{ 0.11, 0.31, 0.42 }
ArtiPoint	23.27 $\pm$ 31.67	26.36 $\pm$ 25.53	0.25 $\pm$ 0.26
	{ 04.85, 45.19, 81.80 }	{ 12.86, 38.38, 62.22 }	{ 0.05, 0.10, 0.31 }
MoMa-SG	10.37 $\pm$ 24.10	14.80 $\pm$ 22.99	0.09 $\pm$ 0.15

Appendix

Table A.8: Error variance and percentiles of articulated axes on Arti4D

Figure A.16: Tracked keypoints of CoTracker3 [ 53 ] throughout interactions of texture-less objects on the MHZH split (ego-centric) of Arti4D-Semantic.

Figure A.17: Tracked keypoints of TapNext++ [ 46 ] on the MHZH-EXO split (exo-centric) of Arti4D-Semantic along an interaction (left to right, top to bottom).

Split	Paradigm	GT Poses	IoU	Recall
Split	Paradigm	GT Poses	IoU	@0.25	@0.50	@0.75
RH078	EGO	✓	0.508	0.692	0.415	0.134
RR080	EGO	✓	0.411	0.605	0.429	0.146
DR080	EGO	✓	0.548	0.889	0.556	0.111
RH201	EGO	✓	0.408	0.674	0.501	0.124
	Overall		0.448	0.675	0.460	0.132

Appendix

Table A.9: Articulated 3D part segmentation performance of MoMa-SG on Arti4D-Semantic using predicted interaction segments. MoMa-SG employs TapNext++ point tracks. We report the same set of metrics as in Tab. 2 .

Figure A.18: Robustness study across the stages of MoMa-SG on the RH201 split of the Arti4D-Semantic dataset.

Panel of Figure 35 — Figure A.19: Sensitivity of the proposed regularized twist estimation to various $\eta$ on the RH201 split of Arti4D-Semantic.

Panel of Figure 36 — Figure A.20: We visualize the three new dataset splits of Arti4D-Semantic: The ego-centric MHZH / exo-centric MHZH-EXO splits (left) and the robot-centric HSR split (right).

Panel of Figure 37 — Figure A.21: We visualize articulated 3D parent instance labels for two scenes contained in RH078 (top) and DIN080 (bottom), respectively. Each distinctly colored mask represents a closed articulated object.

Split	GT	Paradigm	Sequence	Recording	#	#	#	#	Parent	Child
Split	Poses	Paradigm	ID	Recording	Objects	Interactions	PRISM / REV	EASY / HARD	Labels	Labels
RH078	✓	EGO	RH078-00	scene_2025-04-04-19-14-38	7	7	3 / 4	7 / 0	✓	✓
	✓	EGO	RH078-01	scene_2025-04-04-19-18-54	6	7	2 / 4	4 / 2	✓	✓
	✓	EGO	RH078-02	scene_2025-04-07-11-39-17	7	7	3 / 4	7 / 0	✓	✓
	✓	EGO	RH078-03	scene_2025-04-07-11-41-52	8	9	3 / 5	6 / 2	✓	✓
	✓	EGO	RH078-04	scene_2025-04-07-11-48-40	7	7	3 / 4	7 / 0	✓	✓

Appendix

Table A.10: Overview of all demonstration sequences of Arti4D-Semantic. The RH078 , RR080 , DR080 , RH201 , and MHZH splits are ego-centric. The MHZH-EXO split is exo-centric and the HSR split is robot-centric. We list the distribution of articulation types ( PRISM / REV ), difficulties, and the number of distinct interactions vs. the number of articulated objects.

3D SGG

3D scene graph generation (3D SGG) represents objects, parts, and their spatial or functional relations as structured models of 3D environments.

See all 63 papers

Pandora: Articulated 3D Scene Graphs from Egocentric Vision
Mar 30, 2026Alan Yu, Yun Chang, Christopher Xie +13D Scene Representation 3D Scene Understanding
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology · Meta Reality Labs
FunArt: Decoding Functional Structure and Articulation from Generative 3D Latents
Sep 17, 2026Dennis Rotondi, Abdelrhman Werby, Kai O. ArrasArticulated Object Reconstruction 3D Part Segmentation
University of Stuttgart, Germany · International Max Planck Research School for Intelligent Systems
PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning
Jun 7, 2026Haoyu Li, Aaron Thomas, Shuyan Zhou +13D SGG 3D Scene Representation
Department of Mechanical Engineering and Materials Science, Duke University, Durham, NC 27708, USA. · Department of Computer Science, Duke University, Durham, NC 27708, USA.

Mar 30, 2026J/K move · Enter open · S save

Pandora: Articulated 3D Scene Graphs from Egocentric Vision

Alan Yu, Yun Chang, Christopher Xie, Luca Carlone

Laboratory for Information and Decision Systems, Massachusetts Institute of Technology · Meta Reality Labs

3D Scene Representation 3D Scene Understanding 3D SGG Articulated Object Reconstruction

Open paper AskPDF arXiv

Robotic mapping systems typically approach building metric-semantic scene representations from the robot's own sensors and cameras. However, these "first person" maps inherit the robot's own limitations due to its embodiment or skillset, which may leave many aspects of the environment unexplored. For example, the robot might not be able to open drawers or access wall cabinets. In this sense, the map representation is not as complete, and requires a more capable robot to fill in the gaps. We narrow these blind spots in current methods by leveraging egocentric data captured as a human naturally explores a scene wearing Project Aria glasses, giving a way to directly transfer knowledge about articulation from the human to any deployable robot. We demonstrate that, by using simple heuristics, we can leverage egocentric data to recover models of articulate object parts, with quality comparable to those of state-of-the-art methods based on other input modalities. We also show how to integrate these models into 3D scene graph representations, leading to a better understanding of object dynamics and object-container relationships. We finally demonstrate that these articulated 3D scene graphs enhance a robot's ability to perform mobile manipulation tasks, showcasing an application where a Boston Dynamics Spot is tasked with retrieving concealed target items, given only the 3D scene graph as input.

Abstract

3D SGG

Explore similar work

Pandora: Articulated 3D Scene Graphs from Egocentric Vision

FunArt: Decoding Functional Structure and Articulation from Generative 3D Latents

PhysGraph: A Physics-aware 3D Scene Graph for Perception and Reasoning