FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

Zhengyu Fu; René Zurbrügg; Kaixian Qu; Marc Pollefeys; Marco Hutter; Hermann Blum; Zuria Bauer

cs.CVApr 4, 2026

FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning

Authors: Zhengyu Fu, René Zurbrügg, Kaixian Qu, Marc Pollefeys, Marco Hutter, Hermann Blum, Zuria Bauer

Organizations: ETH Zürich · Microsoft · University of Bonn & Lamarr Institute

Abstract

Recent work in 3D scene understanding is moving beyond purely spatial analysis toward functional scene understanding. However, existing methods often consider functional relationships between object pairs in isolation, failing to capture the scene-wide interdependence that humans use to resolve ambiguity. We introduce FunFact, a framework for constructing probabilistic open-vocabulary functional 3D scene graphs from posed RGB-D images. FunFact first builds an object- and part-centric 3D map and uses foundation models to propose semantically plausible functional relations. These candidates are converted into factor graph variables and constrained by both LLM-derived common-sense priors and geometric priors. This formulation enables joint probabilistic inference over all functional edges and their marginals, yielding substantially better calibrated confidence scores. To benchmark this setting, we introduce FunThor, a synthetic dataset based on AI2-THOR with part-level geometry and rule-based functional annotations. Experiments on SceneFun3D, FunGraph3D, and FunThor show that FunFact improves node and relation discovery recall and significantly reduces calibration error for ambiguous relations, highlighting the benefits of holistic probabilistic modeling for functional scene understanding. See our project page at https://funfact-scenegraph.github.io/

Figures & tables

Figure 1: FunFact for functional scene understanding. Given posed RGB-D inputs, FunFact reconstructs an object- and part-centric 3D map and builds a functional scene graph (top). Candidate relations (e.g., remote controls TV , switch toggles lamp ) are encoded as binary variables in a dual factor graph (bottom), where cardinality and proximity factors jointly resolve ambiguities via belief propagation, yielding calibrated per-edge confidence scores.

Figure 2: Overview of FunFact : Given Posed RGB-D images FunFact generates scene reconstructions and functional 3D scene graphs in two stages: i) Scene Reconstruction. Given a set of RGB-D images and respective poses, we extract bounding boxes, scene description, object list, and candidate part names using GPT-4.1 . These textual cues are used to obtain open-vocabulary object detections with GroundingDINO, which are filtered for consistency and turned into region proposals and SAM-based instance masks. A second GroundingDINO + SAM branch segments functional parts conditioned on the predicted object and part names. Finally, we lift all object and part instances to 3D and fuse them across views yielding a coherent, part-aware 3D reconstruction that forms the basis for the subsequent functional scene graph inference. ii) Functional Scene-Graph Creation. Given the semantic 3D reconstruction and the part / object nodes, GPT-4.1 proposes object–object and object–part relations to form an initial functional scene graph. We convert this graph into a dual factor graph with different priors and perform belief propagation to obtain calibrated edge probabilities. This yields the posterior functional scene graph grounded in the reconstructed scene.

Figure 3: Functional Scene Graph and its Dual Factor Graph. Left: Candidate functional scene graph with edges

e_{1},\dots,e_{4}

representing knob–burner relations. Right: The dual factor graph, where each binary variable

x_{i}

is the dual of edge

e_{i}

, encoding whether that relation is present.

p_{i}

: unary proximity prior on

x_{i}

;

K_{i}

: cardinality factor enforcing one-to-one association per knob;

B_{i}

: cardinality factor enforcing one-to-one association per burner.

Figure 4: Examples of two of our newly annotated AI2-THOR environments. For two scenes (bathroom, top; kitchen, bottom) we show a top-down view with mapped functional objects (left) and the corresponding instance-, part-segmentation and functional edges (right), illustrating our functional annotations.

Methods	SceneFun3D						FunGraph3D
	Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$		Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$
	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10
Open3DSG	61.2	70.7	54.4	61.8	56.7	64.7	50.9	58.1	21.8	33.9	33.4	43.6
ConceptGraph	71.3	77.1	6.6	8.6	28.3	31.4	58.0	66.3	2.5	4.1	20.1	25.2
ConceptGraph + IED	71.3	77.1	53.1	59.5	60.1	66.0	58.0	66.3	20.5	33.4	38.9	45.0
OpenFunGraph	81.8	87.8	71.0	79.5	73.0	82.8	70.7	79.1	44.4	57.6	55.5	65.8

Table 1: Scene Reconstruction Comparison. We report Recall@K ( R@3 / R@10 ; higher is better) for three node categories— Objects , Interactive Elements , and Overall Nodes —on SceneFun3D and FunGraph3D . Bold numbers mark the best result per column. Compared to Open3DSG [ 22 ] , ConceptGraph [ 12 ] , ConceptGraph variant, and OpenFunGraph [ 46 ] , FunFact attains the highest recall across nearly all categories and both datasets, indicating more reliable recovery of functional entities and relations.

Methods	SceneFun3D						FunGraph3D						FunThor
	Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )		Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )		Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )
	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@3	R@5	R@3	R@5	R@3	R@5
OpenFunGraph	68.3	73.0	88.1	96.2	60.4	70.3	45.8	49.3	65.1	91.4	29.8	45.0	26.4	26.4	59.0	66.7	15.1	17.6
FunFact (Ours)	62.1	71.8	66.1	80.7	41.0	57.9	71.1	80.0	67.9	79.9	48.7	63.9	60.8	62.8	88.9	87.1	54.1	54.7

Table 2: Triplet Evaluation. We report node association, edge prediction and overall triplet recall as Recall@K (R@5/R@10) on SceneFun3D and FunGraph3D and (R@3/R@5) on FunThor .

Figure 5: Qualitative Results. Top: Input images with detected functional objects. Bottom-left: Reconstructed object and part point clouds with predicted functional relations (red: confidence $<$ 0.5; yellow: confidence $\geq$ 0.5). Bottom-right: Final functional 3D scene graph after confidence thresholding; red edges indicate object-part hierarchy and gray edges indicate functional relations.

Methods	Mapping			Functional Graph
	Recall @ 3 $(\uparrow)$			Prec. [%]	Recall [%]	F1 [%]	ECE
	Objects	Inter. Elem.	Overall Nodes	$(\uparrow)$	$(\uparrow)$	$(\uparrow)$	All $(\downarrow)$	Ambiguous $(\downarrow)$
OpenFunGraph	54.6	41.1	51.2	23.4	12.2	16.0	0.43	0.51
FunFact (Ours)	68.2	69.5	68.5	31.9	49.3	38.7	0.36	0.07
w/o FactorGraph	68.2	69.5	68.5	21.9	53.4	31.1	0.70	0.45
w/o Hierarchical Object and Part proposal	68.9	41.8	62.1	21.6	18.2	19.8	0.36	0.14

Table 3: Ablation study and calibration scores on the FunThor Dataset. We report mapping quality (Recall@3), functional edge scores (Precision, Recall, F1; higher is better), and calibration metrics (ECE, ECE-ambiguous; lower is better).

Figure 6: Examples of our dual factor graph and interactive verification. We highlight how our dual graph can be used to calibrate functional predictions over time. Thicker and darker blue edges indicate higher confidence, while thinner and lighter blue edges represent less probable associations. In the beginning, each knob-burner edge receives confidence solely from the proximity and cardinality priors. After introducing evidence (i.e., the left knob controls the left burner plate), our model propagates this information to update the remaining three knob–burner associations.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Examples of scene filtering for FunThor . Scenes are excluded if switch-light associations are ambiguous (left) or if pans are present on stove burners (right).

VLM Backbone	SceneFun3D						FunGraph3D
	Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$		Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$
	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10
gemini-3-flash-preview	94.3	97.1	63.2	74.5	73.5	82.0	93.2	95.9	68.8	78.8	79.0	85.9
qwen3-vl-32b-instruct	92.4	98.1	64.2	72.2	73.5	80.8	88.4	94.5	59.4	73.3	71.6	82.2

Appendix

Table 4: Scene Reconstruction with Alternative VLM Backbones. We report Recall@K ( R@3 / R@10 ; higher is better) for three node categories— Objects , Interactive Elements , and Overall Nodes —on SceneFun3D and FunGraph3D , using gemini-3-flash-preview and qwen3-vl-32b-instruct as drop-in replacements for GPT-4.1. Bold numbers mark the best result per column.

VLM Backbone	SceneFun3D						FunGraph3D						FunThor
	Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )		Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )		Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )
	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10	R@3	R@5	R@3	R@5	R@3	R@5
gemini-3-flash-preview	67.2	72.3	51.1	68.1	34.4	49.2	74.3	79.6	61.5	72.1	45.2	57.4	70.9	72.3	76.2	78.5	54.1	56.8
qwen3-vl-32b-instruct	66.2	69.7	46.5	67.7	30.8	47.2	69.6	74.8	60.6	73.8	42.2	55.2	68.9	70.9	72.5	78.1	50.0	55.4

Appendix

Table 5: Triplet Evaluation with Alternative VLM Backbones. We report node association, edge prediction and overall triplet recall as Recall@K (R@5/R@10) on SceneFun3D and FunGraph3D , and (R@3/R@5) on FunThor , using gemini-3-flash-preview and qwen3-vl-32b-instruct as drop-in replacements for GPT-4.1. Bold numbers mark the best result per column.

Methods	SceneFun3D						FunGraph3D						FunThor
	Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$		Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$		Objects $(\uparrow)$		Inter. Elements $(\uparrow)$		Overall Nodes $(\uparrow)$
	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10	R@3	R@10
OpenFunGraph	78.1	79.0	64.4	78.3	69.1	78.5	61.1	68.5	42.1	52.5	50.3	59.2	43.7	52.3	34.8	40.4	41.5	49.3
FunFact (Ours)	90.5	90.5	64.4	75.5	73.2	80.4	84.9	89.0	62.9	75.2	72.1	81.0	57.2	61.0	64.5	68.8	59.1	63.0

Appendix

Table 6: Mapping Quality under Exclusive Matching. We report Recall@K ( R@3 / R@10 ; higher is better) for three node categories— Objects , Interactive Elements , and Overall Nodes —on SceneFun3D , FunGraph3D , and FunThor under the exclusive matching constraint, where each predicted node maps to at most one ground-truth node. This provides a more accurate assessment of mapping quality compared to non-exclusive matching, which may overestimate performance when a single merged detection covers multiple ground-truth objects. All methods exhibit decreased recall under this stricter evaluation, but FunFact maintains superior performance across nearly all categories and datasets.

Panel of Figure 8 — Figure 8: Calibration analysis on FunThor dataset. Each subfigure shows a confidence histogram (top) and reliability diagram (bottom).

Panel of Figure 9 — Figure 9: Scene dev/421063/42444511 of SceneFun3D. See Section I for detailed subfigure descriptions. As shown in the top-right subfigure, we predict semantically correct functional relations in this bathroom scene, but none of them are matched to ground truth as indicated by the red arrows. This is due to several annotation issues in the dataset: (1) node label mismatches where “drains” are incorrectly labeled as “button/knob” and the trash bin “pedal” is labeled as “handle” in ground truth; (2) the ground truth label “foucet” is a typo and can never match our correct prediction “faucet”; (3) even when node labels match, our open-vocabulary relation prediction “handle turn or pull to open or close door” cannot be matched to the ground truth annotation “handle rotate to open or close door” because when computing semantic similarity using BERT embeddings, the ground truth relation does not rank in the top-5 most similar relations, despite clearly describing the same interaction to humans. These issues lead to zero matched relations despite semantic correctness. Beyond ground truth, our method identifies additional functional relations, including “towel ring can hold towel” and “towel radiator can dry towel” (highlighted in bottom-left graph and bottom-right point cloud viewer).

Panel of Figure 10 — Figure 10: Scene dev/422007/42446017 of SceneFun3D. See Section I for detailed subfigure descriptions. This baby bedroom scene highlights how the SceneFun3D dataset overly focuses on knob-pull interactions to open cabinets and drawers. Nearly all ground-truth annotations involve such interactions, with only one exception for radiator temperature adjustment. The dataset overlooks inter-object and intra-object functional relations that are crucial for understanding the scene. As shown in the top-right subfigure (predictions) and bottom-left subfigure (functional scene graph), our method identifies these missing relations, including “power button press to turn on/off humidifier” and “baby monitor camera watch crib”, demonstrating our pipeline’s ability to capture diverse functional interactions beyond the narrow focus of existing annotations.

Panel of Figure 11 — Figure 11: Scene 4livingroom of FunGraph3D. See Section I for detailed subfigure descriptions. Our pipeline predicts most ground-truth functional relations, including power outlet-to-kettle connections and oven knob temperature/mode adjustments. We highlight “kettle” in the bottom two subfigures to demonstrate that our pipeline captures both inter-object relations ( e.g . , power outlet provides power to kettle) and intra-object relations ( e.g . , lid flip to open kettle).

Panel of Figure 12 — Figure 12: Scene FloorPlan5 of FunThor . See Section I for detailed subfigure descriptions. We illustrate the dense and complete ground-truth annotations for the FunThor dataset (top-left). Our pipeline accurately detects functional parts of objects ( e.g . , burners and knobs of the stove; buttons of the coffee machine). As only predicted functional relations with confidence scores above $0.5$ are displayed in the functional scene graph, only the two outermost knobs are predicted to functionally connect to their respective outermost burners in the graph.

Methods	Mapping						Functional Graph
	Objects ( $\uparrow$ )		Inter. Elements ( $\uparrow$ )		Overall Nodes ( $\uparrow$ )		Node Assoc. ( $\uparrow$ )		Edge Pred. ( $\uparrow$ )		Overall Triplets ( $\uparrow$ )
	R@3	R@10	R@3	R@10	R@3	R@10	R@5	R@10	R@5	R@10	R@5	R@10
OpenFunGraph	81.1	87.8	71.0	79.5	73.0	82.8	68.3	73.0	88.1	96.2	60.4	70.3
FunFact (Modified)	90.5	95.2	76.4	88.7	81.1	90.9	76.4	83.1	75.8	91.4	57.9	75.9

Appendix

Table 7: Performance after pipeline adaptation on SceneFun3D. To address SceneFun3D’s systematic annotation of knob-shaped and handle-shaped components ( e.g . , drains, valves, lever) with generic “knob” and “handle” labels, we modify our pipeline to always detect knobs and handles when functional parts are identified. The modified pipeline substantially improves both mapping and functional relation metrics, surpassing OpenFunGraph on all mapping metrics and achieving competitive triplet recall, especially at R@10 (75.9% vs. 70.3%).

Methods	Mapping			Functional Graph
	Recall @ 3 $(\uparrow)$			Prec. [%]	Recall [%]	F1 [%]
	Objects	Inter. Elem.	Overall Nodes	$(\uparrow)$	$(\uparrow)$	$(\uparrow)$
OpenFunGraph	54.6	41.1	51.2	23.4	12.2	16.0
FunFact (Original)	68.2	69.5	68.5	31.9	49.3	38.7
FunFact (Modified)	68.2	87.9	73.1	16.0	58.1	25.1

Appendix

Table 8: Evaluating side effects of the pipeline adaptation. To determine whether forcing knob and handle detection negatively affect overall mapping and functional prediction performance, we evaluate both the original and modified pipelines on FunThor . The modified pipeline significantly improves recall for interactive elements (69.5%

\to

87.9%) and functional relations (49.3%

\to

58.1%), but substantially decreases precision (31.9%

\to

16.0%) and F1 score (38.7%

\to

25.1%), confirming that the modification negatively affects the overall performance. Nevertheless, the modified pipeline still outperforms OpenFunGraph in F1 score (25.1% vs. 16.0%).

3D SGG

3D scene graph generation (3D SGG) represents objects, parts, and their spatial or functional relations as structured models of 3D environments.

See all 65 papers

May 15, 2026

Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces

Functional 3D scene graphs offer a versatile and flexible representation for 3D scene understanding and robotic manipulation, defined by object nodes, interactive elements, and functional relationship edges. However, their potential remains underexplored due to the limited coverage of existing benchmarks and the overly straightforward design of previous pipelines, which primarily focus on large-scale furniture but lack of hierarchical structures. Therefore, in this work, we extend the benchmark coverage by introducing dense tabletop objects and explicit multi-level functional relationships. This expansion introduces critical challenges involving small-scale, dense, and similar instances, with lack of visual anchoring in relational reasoning, instance confusion during cross-frame fusion, and attribution uncertainty under dynamic viewpoints. To address these issues, we propose an open-vocabulary pipeline based on 2D visual grounding and 3D graph optimization. Specifically, we anchor fine-grained functional edges from 2D visual evidence, and associate nodes across frames in 3D using multiple cues. Furthermore, edge association is formulated as temporal graph optimization, integrating evidence accumulation, entropy regularization, and temporal smoothing to robustly determine the functional connections of each node. Finally, global hierarchy shaping is performed to recover the hierarchical graph structure. Extensive experiments demonstrate that the proposed method can reliably infer functional 3D scene graphs in challenging real-world scenes, thereby further unlocking their potential for practical applications.

Xinggang Hu, Chenyangguang Zhang, Alexandros Delitzas +4

Tsinghua University, China · ETH Zurich, Switzerland · MPI for Informatics, Germany +3

3D SGG 3D Scene Understanding

Ask PDF

Jun 29, 2026

OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments

3D scene graphs (3DSGs) provide a compact and structured abstraction of 3D environments. Although advances in foundation models have enabled open-vocabulary 3DSG generation, existing approaches remain object-centric and encode limited relational information -- restricting their applicability in real-world scenarios that require fine-grained understanding. We propose OP3DSG, an open-vocabulary part-aware 3DSG generation framework that constructs unified graphs that jointly model objects, interactive parts, spatial relations, functional relations, and affordances. OP3DSG integrates object-part knowledge-guided detection with part-aware 3D fusion to preserve small and interaction-relevant components, and employs a geometry-initialized prior graph with LLM-based refinement to reduce spurious relational predictions while enabling efficient graph construction. To systematically evaluate unified 3D scene graph construction, we introduce UniGraph3D, a benchmark designed for part-aware perception and multi-level relational reasoning. Experimental results show that OP3DSG achieves state-of-the-art performance and demonstrates its effectiveness as a perception backbone in diverse real-world robotics tasks.

Yirum Kim, Ue-Hwan Kim

Gwangju Institute of Science and Technology (GIST), Gwangju, Republic of Korea

3D SGG Robotic Perception 3D Scene Understanding

Ask PDF

Jul 6, 2026

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis

Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet

++

, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage

Xianhao Chen, Jiarui Hu, Yuanbo Yang +5

State Key Lab of CAD&CG, Zhejiang University, China · Zhejiang University, China

Graph Attention Networks Open-Vocabulary 3D Segmentation 3D Scene Understanding

Ask PDF

Abstract

3D SGG

Explore similar work

Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces

OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis