UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation

Jiaying Lin; Dan Xu

cs.CVMar 24, 2026

UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation

Authors: Jiaying Lin, Dan Xu

Organizations: The Hong Kong University of Science and Technology

Abstract

Affordance segmentation in 3D scenes requires an agent to ground implicit natural-language instructions into precise masks of fine-grained interactive elements. Existing training-free methods typically rely on fragmented pipelines, which introduce visual blindness during task parsing and limit accuracy through single-scale spatial and temporal processing. We present UniFunc3D, a unified and training-free framework that treats the multimodal large language model as an active observer. By utilizing a unified MLLM backbone, UniFunc3D performs joint semantic-temporal-spatial reasoning to ground task decomposition in direct visual evidence. Our approach introduces active spatial-temporal grounding with a coarse-to-fine strategy. This allows the model to select correct video frames adaptively and focus on high-detail interactive parts while preserving the global context necessary for disambiguation. On SceneFun3D, our UniFunc3D achieves state-of-the-art performance, surpassing prior training-free methods by a large margin with a relative 59.9% mIoU improvement, and even outperforming training-based methods without any task-specific training. Code is available on our project page: https://jiaying.link/unifunc3d.

Figures & tables

Figure 1 : Overview of UniFunc3D compared to existing fragmented pipelines. (Top) Prior methods like Fun3DU rely on a visually blind text-only LLM for initial task parsing. Coupled with single-scale passive heuristic frame selection, this fragmented approach suffers from three critical failure modes: semantic misinterpretations (Task A $\rightarrow$ Task B), spatial-temporal context inconsistencies (a, b) and imperceptible small target (c), leading to error in the final output. (Bottom) Our proposed UniFunc3D addresses these limitations by utilizing a unified MLLM as an active observer. By employing a coarse-to-fine active spatial-temporal grounding strategy alongside visual mask verification, UniFunc3D consolidates semantic, temporal, and spatial reasoning in a unified MLLM. This allows the model to accurately ground implicit targets and generate precise fine-grained 3D functional masks while preserving necessary global context.

Figure 2 : Method overview. UniFunc3D employs a unified MLLM with active spatial-temporal grounding with joint functional object identification : the coarse stage (Round 1) actively surveys low-resolution video frames across multiple sampling iterations and selects the most informative candidate via visual verification; the fine stage (Round 2) processes a dense temporal window at native high resolution, delivering zoom-in capability while preserving global scene context for precise localization. Visual mask generation and verification uses SAM3 for segmentation with MLLM-based mask verification, then multi-view 3D lifting to obtain the final 3D masks.

	Split0 (30 scenes, val)					Split1 (200 scenes, train)
Method	AP ${}_{50}\uparrow$	AP ${}_{25}\uparrow$	AR ${}_{50}\uparrow$	AR ${}_{25}\uparrow$	mIoU $\uparrow$	AP ${}_{50}\uparrow$	AP ${}_{25}\uparrow$	AR ${}_{50}\uparrow$	AR ${}_{25}\uparrow$	mIoU $\uparrow$
Open-vocabulary 3D segmentation methods:
OpenMask3D Takmaz et al. (2023)	0.2	0.4	24.5	27.0	0.2	0.0	0.0	1.4	2.6	0.1
LERF Kerr et al. (2023)	0.0	0.0	35.1	36.0	0.0	0.0	0.0	24.6	25.1	0.0
OpenIns3D Huang et al. (2024)	0.0	0.0	46.7	51.5	0.1	0.0	0.0	37.1	39.9	0.1
Training-based 3D affordance segmentation methods:

Table 1 : Results on SceneFun3D. UniFunc3D achieves the best performance across all metrics on both split0 and split1, outperforming both training-free and training-based methods. Percentages in parentheses denote relative improvements over Fun3DU.

Panel of Figure 3 — Figure 3 : Qualitative comparison. We show results for representive five queries.

Strategy	2-Stage	Multi-Sam.	Temp.	AP 50	AP 25	AR 50	AR 25	mIoU
Qwen3-VL-8B (High-res)				4.27	7.64	24.27	28.76	3.11
Qwen3-VL-8B (Low-res)				7.42	14.38	33.93	38.65	7.35
Low $\rightarrow$ High (Zoom-in)	✓			15.96	26.29	39.33	48.54	12.71
Two-stage + Multi-sam.	✓	✓		19.33	38.20	44.94	53.71	18.10
Two-stage + Temp. window	✓		✓	17.53	35.28	41.80	51.91	17.26
Ours (Full)	✓	✓	✓	23.82	44.04	46.07	55.51	20.92

Table 2 : Ablation on high-level designs. Comparison of one-stage vs. two-stage approaches. We also evaluate the impact of Multi-Sam. (Multi-sampling), and Temp. (Temporal window).

Configuration	Temp.	Verif.	AP 50	AP 25	AR 50	AR 25	mIoU
$K=1$ (Base)			15.96	26.29	39.33	48.54	12.71
$K=1$ , no verify	✓		15.06	32.58	40.90	50.11	16.00
$K=1$ , verify	✓	✓	17.53	35.28	41.80	51.91	17.26
$K=2$ ,			15.51	28.31	39.33	46.07	14.40
$K=2$ , no window		✓	12.58	27.42	40.00	47.42	13.87
$K=2$ , no verify	✓		21.12	39.10	46.97	56.18	19.57

Table 3 : Ablation on sampling strategies. Results for UniFunc3D-8B evaluating the impact of sampling iterations (

K

), the temporal window and verification.

Panel of Figure 4 — Figure 4 : Visual comparison. Input prompt: open the right door of the wooden display cabinet to the left of the paintings .

Baselines	AP 50	AP 25	AR 50	AR 25	mIoU
TASA-72B † He et al. (2026)	26.9	28.6	–	–	19.7
AffordBot-72B † Wang et al. (2025a)	20.91	24.76	18.99	22.84	14.42
Ours (Qwen2.5-VL-72B)	22.02	40.90	48.09	59.78	20.07
Ours (Qwen3VL-8B)	23.82	44.04	46.07	55.51	20.92
Ours (Qwen3VL-30B)	31.24	51.01	46.97	58.88	24.30

Table 4 : Ablation on base models. Best-performed metrics with Qwen2.5-VL-72B (the top group) are marked in bold . We also provide our models with Qwen3-VL for reference.

Table 9

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Time per scene	Time per query	Speedup
Fun3DU [ 7 ]	$\sim$ 82 min	$\sim$ 332 sec	1 $\times$
UniFunc3D-8B	$\sim$ 16 min	$\sim$ 53 sec	5.2 $\times$
UniFunc3D-30B	$\sim$ 26 min	$\sim$ 99 sec	3.2 $\times$

Appendix

Table 7 : Time comparison. UniFunc3D-8B achieves over 5.2 times speedup compared to Fun3DU while maintaining superior accuracy.

Implementation	AP 50	AP 25	AR 50	AR 25	mIoU
TASA-72B (Reported)	26.9	28.6	–	–	19.7
TASA-72B (Our Reproduced)	19.72	23.58	21.17	25.21	14.61
AffordBot-72B (Reported) [ 35 ]	20.00	23.30	-	-	14.00
AffordBot-72B (Our Reproduced)	20.91	24.76	18.99	22.84	14.42
Fun3DU (Reported) [ 7 ]	16.9	33.3	38.2	46.7	15.2
Fun3DU (Our Reproduced)	12.13	23.15	33.93	40.90	11.34

Appendix

Table 8 : Comparative results of different implementations of baseline methods.

Panel of Table 9 — Table 9 : Prompt used in the coarse grounding stage of our UniFunc3D. T is the input task prompt.

Panel of Table 10 — Table 10 : Prompt used in the fine grounding stage of our UniFunc3D. T is the input task prompt. $o_{\text{func}}^{j}$ is the functional object predicted by the coarse stage.

Panel of Table 11 — Table 11 : Prompt used in the verification stage of our UniFunc3D. $o_{\text{func}}^{j}$ is the functional object predicted by the previous stage.

Panel of Figure 5 — Figure 5 : Visual comparisons for ablation study. From left to right, top to bottom: Base, w/o Multi-view, w/o Temporal, w/o Verification, Ours, and GT. Two different viewpoints are shown for the same test example. Input prompt: Open the top left drawer of the cabinet with the TV on top .

Panel of Figure 6 — Figure 6 : Visual comparisons for ablation study. From left to right, top to bottom: Base, w/o Multi-view, w/o Temporal, w/o Verification, Ours, and GT. Two different viewpoints are shown for the same test example. Input prompt: Open the right door of the wood and glass display cabinet next to the pet transport box .

Panel of Figure 7 — Figure 7 : Qualitative comparison. We show results for five representative queries (columns) across four methods (rows). GT denotes ground truth.

Panel of Figure 8 — Figure 8 : Qualitative comparison. We show results for five representative queries (columns) across four methods (rows). GT denotes ground truth.

Panel of Figure 9 — Figure 9 : Qualitative comparison. We show results for five representative queries (columns) across four methods (rows). GT denotes ground truth.

Explore similar work

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Aug 11, 2026Xinrui Lin, Sha Zhang, Shumin Wang +33D Visual Grounding Affordances
University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China
Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances
May 12, 2026Qirui Wang, Jingyi He, Yining Pan +2Scene Understanding 3D Generation
TUM · A*STAR
SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding
Sep 27, 2026Xiangqi Li, Libo Huang, Jiarui Zhao +43D Scene Understanding State-Tracking
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Aug 11, 2026, cs.CVJ/K move · Enter open · S save

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

Xinrui Lin, Sha Zhang, Shumin Wang, Zenghuan Zhu+2

University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China

3D Visual Grounding Affordances

Open paper AskPDF arXiv

Task-driven 3D affordance grounding aims to localize the functional region in a cluttered 3D scene that enables an action specified by a natural-language instruction. Existing methods either predict 3D masks directly or construct them by selecting and fusing intermediate 2D/3D regions. However, they remain vulnerable to two intertwined failure modes: the predicted or selected regions may miss the target interaction area or have unsuitable granularity, while language grounding may confuse visually similar alternatives under relational instructions. To this end, we introduce ThinkAfford, which decouples high-recall affordance proposal generation from instruction-grounded reasoning. Specifically, the Affordance Proposal Generation module first uses learnable affordance prompts and multi-level visual features to predict interaction-conditioned heatmaps, extracting a variable number of fine-grained proposals without parsed object or part names as segmentation prompts. Visual-Prompted Affordance Reasoning then reasons over labeled proposal overlays using the full instruction, returning identifiers in a structured "think-then-answer" response. Moreover, Group Relative Policy Optimization uses proposal-level rewards from lifted 3D overlap to align VPAR selection with final 3D grounding. On the SceneFun3D validation split, ThinkAfford achieves 10.69% AP50 and 25.46% AP25 under the official evaluator, outperforming comparable 3D open-vocabulary and vision-language-model-based 2D-to-3D baselines. Module-level diagnostics further show that APG attains 77.5% recall at 25% intersection-over-union, while GRPO-trained VPAR achieves 72.1% selection accuracy on APG-covered queries, compared with 63.4% under supervised fine-tuning.

Abstract

Explore similar work

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

Grounding by Remembering: Cross-Scene and In-Scene Memory for 3D Functional Affordances

SceneScaffold: Active Scene-State Construction for Unified 3D Scene Understanding