AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifiers

Nghia Vu; Tuong Do; Khang Nguyen; Baoru Huang; Nhat Le; Binh Xuan Nguyen; Erman Tjiputra; Quang D. Tran

cs.CVMar 30, 2026

AffordMatcher: Affordance Learning in 3D Scenes from Visual Signifiers

Authors: Nghia Vu, Tuong Do, Khang Nguyen, Baoru Huang, Nhat Le, Binh Xuan Nguyen, Erman Tjiputra, Quang D. Tran,

Organizations: AIOZ Ltd., Singapore · University of Liverpool, UK · National Tsing Hua University, Taiwan · MBZUAI ·

Abstract

Affordance learning is a complex challenge in many applications, where existing approaches primarily focus on the geometric structures, visual knowledge, and affordance labels of objects to determine interactable regions. However, extending this learning capability to a scene is significantly more complicated, as incorporating object- and scene-level semantics is not straightforward. In this work, we introduce AffordBridge, a large-scale dataset with 291,637 functional interaction annotations across 685 high-resolution indoor scenes in the form of point clouds. Our affordance annotations are complemented by RGB images that are linked to the same instances within the scenes. Building upon our dataset, we propose AffordMatcher, an affordance learning method that establishes coherent semantic correspondences between image-based and point cloud-based instances for keypoint matching, enabling a more precise identification of affordance regions based on cues, so-called visual signifiers. Experimental results on our dataset demonstrate the effectiveness of our approach compared to other methods.

Figures & tables

Dataset / Attribute	Total Samples	Environment	Form of Interactions		Affordance Annotation	No. Affordances	No. Categories	No. Aff. Actions
Dataset / Attribute	Total Samples	Environment	Implicit	Explicit	Affordance Annotation	No. Affordances	No. Categories	No. Aff. Actions
EPIC-Aff [ 42 ]	38,876	2D Images	–	–	2D masks	–	304	43
AGD20k [ 35 ]	23,816	2D Images	–	–	2D boxes	–	50	36
PartNet [ 39 ]	26,671	3D Objects	–	–	3D masks	573,585	–	24
AffordPose [ 21 ]	641	3D Objects	–	Grasp hand poses	3D masks	26,712	13	8
3DAffordanceNet [ 13 ]	22,949	3D Objects	–	–	3D masks	56,307	23	18

Table 1: Comparisons between AffordBridge and other datasets: Our dataset introduces a large-scale benchmark for spatial affordance identification from visual signifiers through both implicit and explicit human-object interactions. AffordBridge contains

317,844

high-resolution paired samples of 2D-3D representations across

685

scenes. The interacted objects are annotated by 3D masks, yielding

291,637

volumetric masks of interactable regions among

157

object categories through

61

actionable affordances.

Figure 2: Construction of the AffordBridge dataset: Our AffordBridge dataset is built through a semi-supervised pipeline linking visual signifiers with 3D affordances. The building process includes ( i ) 3D scene processing via voxelized point clouds with object-view filtering through visual scanning, ( ii ) visual signifiers processing with human-object interaction extraction with fine-grained captioning, and ( iii ) affordance annotation by matching key views to 3D instances for spatial action labeling.

Panel of Table 2 — Table 2: Train/validation/test split of the AffordBridge dataset.

Figure 3: Dataset statistics: Statistics of objects in human-object interactions yielding affordances in our AffordBridge dataset.

Figure 4: Design architecture of AffordMatcher : Given a high-resolution voxelized scene point cloud and a visual signifier, AffordMatcher reasons over these inputs for zero-shot affordance segmentation. The affordance extractor identifies 3D interactable regions, while the reasoning extractor encodes 2D human-object cues. Cross-modal alignment is achieved via instance matching through a dissimilarity matrix. The features from the dissimilarity matrix are thus optimized through match-to-match attention, followed by a zero-shot affordance optimization to localize actionable spatial regions that align with the given signifier.

Panel of Table 3 — Table 3: Quantitative results: Performance comparisons of AffordMatcher and state-of-the-art methods [ 9 , 21 , 13 , 74 , 30 , 33 ] in terms of mAP, mAP@ $0.25$ , mAP@ $0.50$ , number of parameters (in millions), and inference speed (in milliseconds per sample).

Panel of Figure 5 — Figure 5: Attention visualization: From the visual signifier in the RGB image and the text “ Rest on Pillow ”, AffordMatcher focuses on the pillow area in the RGB image and correctly localizes the corresponding affordance regions in the high-resolution voxelized indoor scene.

Panel of Table 4 — Table 4: Ablation on different input modalities: Removing visual inputs severely degrades performance, followed by inpainting human-object interactions, fine-tuning with PIAD objects, and thus using raw point cloud. The full AffordBridge achieves the highest accuracy through integrated 2D-3D reasoning.

$\mathcal{L}_{\text{align}}$	$\mathcal{L}_{\text{dissim}}$	$\mathcal{L}_{\text{embed}}$	$\mathcal{L}_{\text{bidir}}$	mAP	mAP @0.25	mAP @0.50
				37.3	52.7	42.1
✓				40.9	56.2	45.3
✓	✓			44.1	60.0	48.7
✓	✓	✓		47.8	63.5	53.0
✓	✓	✓	✓	53.4	69.7	59.5

Table 5: Ablation on loss components: Gradual inclusion of each loss enhances performance, with the full objective achieving the best results. The first row indicates the baseline, which corresponds to the semantic affordance objective [ 9 ] .

Panel of Figure 6 — Figure 6: t-SNE visualization: Visual reasoning produces more compact and well-separated clusters among different affordance types compared to without visual reasoning.

Figure 7: Visualization of distinct affordances: “ Sit ” and “ Pull ” cues on the same chair activate different 3D regions, showcasing that AffordMatcher adapts attention to interaction semantics.

Panel of Figure 8 — Figure 8: Qualitative results: Our affordance-mask prediction, compared with other baselines. The first column illustrates the inputs, including visual signifiers and 3D scenes. The remaining columns show the affordance segmentation results, including extracted actions over different methods, where blue texts indicate correct affordance actions and red texts represent wrong ones. In the point clouds, green areas denote correct affordance localization, red and blue areas denote false positives and false negatives, respectively.

Figure 10: User study results in interaction matching criteria.

Figure 11: Affordance prediction for a specific interaction, but results in different objects.

Panel of Figure 12 — Figure 12: Fail cases of our method: the visual signifier provides a complex action, which causes failure in reasoning. Green areas denote correct predictions; red and blue areas are false positives and false negatives, respectively.

3D Scene Understanding

3D scene understanding builds geometric and semantic representations of environments for spatial reasoning, perception, and interaction.

See all 295 papers

BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding
Oct 15, 2026Yijie Su, Yakun Zhu, Yi Bin +53D Affordance Grounding 3D Visual Grounding
Tongji University · University of Electronic Science and Technology of China
ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Aug 11, 2026Xinrui Lin, Sha Zhang, Shumin Wang +33D Affordance Grounding 3D Scene Understanding
University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China
UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception
Sep 29, 2026Yuhao Liu, Yiming Zhong, Hanqing Wang +9Cross-Modal Representation Learning Unified Multimodal Models
ShanghaiTech University · Shandong University · HKUST(GZ) +3

Oct 15, 2026J/K move · Enter open · S save

BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding

Yijie Su, Yakun Zhu, Yi Bin, Pengpeng Zeng+4

Tongji University · University of Electronic Science and Technology of China

3D Affordance Grounding 3D Visual Grounding Neural Network Generalization

Open paper AskPDF arXiv

Language-guided 3D affordance grounding aims to localize functional regions on 3D objects according to interaction intents. However, existing methods exhibit poor generalization to unseen objects. One challenge is the query-target gap: a semantic discrepancy between query features encoding interaction intents and target-region features, leading to inaccurate grounding. End-to-end training favors query representations tailored to seen objects, limiting their compatibility with shifted target-region features on unseen objects. To address this issue, we propose BridgeAfford, a framework that reduces the query-target semantic gap from two complementary perspectives. First, Interaction Region Alignment (IRA) promotes cross-object alignment among geometrically similar interaction regions during training. This enables structures on unseen objects that resemble previously observed ones to produce representations similar to those learned during training, facilitating target-region recognition. Second, Visual Evidence Aggregation (VEA) leverages a 2D affordance model to refine the attention distribution and guide the query toward the correct region, yielding more target-aware queries. Extensive experiments on the LASO and PIAD datasets demonstrate that BridgeAfford outperforms existing methods, particularly on unseen objects, highlighting its robust generalization ability.

Abstract

3D Scene Understanding

Explore similar work

BridgeAfford: Bridging Query-Target Gap for Generalizable 3D Affordance Grounding

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception