HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding

Lei Yao; Yong Chen; Yuejiao Su; Yi Wang; Moyun Liu; Lap-Pui Chau

cs.CVMar 2, 2026

HAMMER: Harnessing MLLM via Cross-Modal Integration for Intention-Driven 3D Affordance Grounding

Authors: Lei Yao, Yong Chen, Yuejiao Su, Yi Wang, Moyun Liu, Lap-Pui Chau

Organizations: The Hong Kong Polytechnic University · Huazhong University of Science and Technology

Abstract

Humans commonly identify 3D object affordance through observed interactions in images or videos, and once formed, such knowledge can be generically generalized to novel objects. Inspired by this principle, we advocate for a novel framework that leverages emerging multimodal large language models (MLLMs) for interaction intention-driven 3D affordance grounding, namely HAMMER. Instead of generating explicit object attribute descriptions or relying on off-the-shelf 2D segmenters, we alternatively aggregate the interaction intention depicted in the image into a contact-aware embedding and guide the model to infer textual affordance labels, ensuring it thoroughly excavates object semantics and contextual cues. We further devise a hierarchical cross-modal integration mechanism to fully exploit the complementary information from the MLLM for 3D representation refinement and introduce a multi-granular geometry lifting module that infuses spatial characteristics into the extracted intention embedding, thus facilitating accurate 3D affordance localization. Extensive experiments on public datasets and our newly constructed corrupted benchmark demonstrate the superiority and robustness of HAMMER compared to existing approaches. All code and weights are publicly available.

Figures & tables

Figure 1: Comparison of current architectures and HAMMER. (a) GREAT [ 33 ] generates texts to assist the fusion process. (b) InteractVLM [ 7 ] first produces 2D masks and back-projects them into 3D space. (c) HAMMER enhances point cloud features with cross-modal information from MLLMs and lifts the extracted intention embedding to 3D for accurate affordance localization.

Figure 2: Overview of our HAMMER. Given a 3D point cloud ${\mathbf{P}}$ and its corresponding interaction image ${\mathbf{I}}$ , our framework first processes ${\mathbf{I}}$ through a pre-trained MLLM ${\mathcal{F}}_{\theta}$ to extract an affordance-guided intention embedding ${\bm{f}}_{c}$ (Sec. 3.2 ). This embedding is then used to enhance point cloud features via a hierarchical cross-modal integration mechanism (Sec. 3.3 ). To imbue ${\bm{f}}_{c}$ with 3D spatial awareness, we apply a multi-granular geometry lifting module that incorporates multi-scale geometric cues (Sec. 3.4 ). Finally, the refined point features $\tilde{{\bm{f}}}_{p}$ and the 3D-aware intention embedding ${\bm{f}}_{c}^{3D}$ are decoded to produce the final affordance map ${\bm{p}}$ (Sec. 3.5 ).

Intention-Driven 3D Affordance Grounding
Methods	Venue	Seen				Unseen
		aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
PMF [ 52 ]	ICCV 21	10.13	75.05	0.425	0.141	4.67	60.25	0.330	0.211
ILN [ 4 ]	SIGGRAPH 22	11.25	75.84	0.427	0.137	4.71	59.69	0.325	0.207
FRCNN [ 42 ]	Remote Sensing 23	11.97	76.05	0.429	0.136	4.71	61.92	0.332	0.195
PFusion [ 41 ]	CVPR 18	12.31	77.50	0.432	0.135	5.33	61.87	0.330	0.193

Table 1: Performance comparison on PIAD [ 44 ] . Best results are highlight in bold . aIOU and AUC are reported in percentage (%).

Methods	Year	Seen				Unseen Object				Unseen Affordance
		aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
OpenAD [ 26 ]	2023	31.88	89.54	0.526	0.104	16.62	73.49	0.339	0.159	8.00	61.22	0.229	0.167
FRCNN [ 42 ]	2023	33.55	87.05	0.600	0.082	18.08	72.20	0.362	0.152	7.96	59.08	0.210	0.156
XMFNet [ 1 ]	2022	33.91	87.39	0.604	0.078	17.40	74.61	0.361	0.126	8.11	60.99	0.225	0.152
IAGNet [ 44 ]	2024	34.29	89.03	0.623	0.076	16.78	73.03	0.351	0.123	8.99	62.29	0.251	0.141
LASO [ 17 ]	2024	34.88	90.34	0.627	0.077	16.05	73.32	0.354	0.123	8.37	64.07	0.228	0.140

Table 2: Performance comparison on PIADv2 [ 33 ] . Best results are highlight in bold . aIOU and AUC are reported in percentage (%).

Figure 3: Qualitative comparison on PIAD [ 44 ] . Our HAMMER generates more precise and complete affordance predictions compared to GREAT [ 33 ] , highlighting its enhanced capability in understanding interaction intentions and reasoning about 3D affordances.

Figure 4: Qualitative comparison on corrupted point clouds. The proposed HAMMER realizes stronger robustness in accurately localizing affordance regions under severe noise corruption, while GREAT [ 33 ] struggles to maintain reliable predictions.

Type	aIOU $\uparrow$		AUC $\uparrow$		SIM $\uparrow$		MAE $\downarrow$
	GREAT	Ours	GREAT	Ours	GREAT	Ours	GREAT	Ours
Scale	13.41	19.18	77.77	85.73	0.490	0.596	0.116	0.093
Jitter	11.67	17.36	75.65	84.57	0.461	0.568	0.115	0.095
Rotate	12.61	18.33	76.99	85.31	0.483	0.591	0.116	0.095
Drop-L	8.67	17.98	72.38	84.59	0.420	0.568	0.122	0.098
Drop-G	13.62	20.38	78.26	86.76	0.494	0.611	0.112	0.089

Table 3: Robustness evaluation. Drop , L and G indicate dropout, local and global corruptions, respectively.

Type	w/ cls	w/ afford	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Seen	✓	✓	22.20	88.43	0.605	0.083
	✗	✓	21.95	87.73	0.594	0.085
	✗	✗	19.72	84.36	0.564	0.094
Unseen	✓	✓	13.71	80.92	0.449	0.109
	✗	✓	12.35	79.28	0.425	0.113
	✗	✗	8.78	66.33	0.345	0.132

Table 4: Ablation study on affordance-guided intention embedding. w/ cls and w/ afford denote including object category in the prompt and predicting affordance label in the output, respectively.

Type	Ablation	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Seen	HAMMER	22.20	88.43	0.605	0.083
	w/o Hier. Integ.	21.15	87.64	0.597	0.085
	w/o Geo. Lifting	20.46	87.25	0.588	0.087
	w/o Both	19.55	86.67	0.587	0.088
	w/o ${\mathcal{L}}_{txt}$	20.69	87.66	0.603	0.086
Unseen	HAMMER	13.71	80.92	0.449	0.109

Table 5: Ablation study of core components. Hier. Integ. and Geo. Lifting mean the hierarchical cross-modal integration and multi-granular geometry lifting, respectively.

Figure 5: Impact of MLLM backbones ( left ) and visualization of point-wise features via PCA ( right ).

Type	Stage I	Stage II	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Seen	✓	✓	22.20	88.43	0.605	0.083
	✗	✓	21.55	87.79	0.598	0.084
	✓	✗	21.70	88.00	0.603	0.084
	✗	✗	21.15	87.64	0.597	0.085
Unseen	✓	✓	13.71	80.92	0.449	0.109
	✗	✓	11.85	79.55	0.433	0.110

Table S.1 : Ablation study on the hierarchical cross-modal integration mechanism. This analysis evaluates the contribution of each integration stage to the overall performance of HAMMER.

Type	Ablation	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Seen	HAMMER	22.20	88.43	0.605	0.083
	Single Lift	21.40	88.17	0.603	0.085
	Concat. Lift	21.05	87.95	0.597	0.084
Unseen	HAMMER	13.71	80.92	0.449	0.109
	Single Lift	12.26	80.64	0.441	0.111
	Concat. Lift	11.45	78.12	0.421	0.110

Table S.2 : Alternative lifting strategies. We compare the performance of our proposed lifting strategy with two alternatives: (1) single-stage lifting, and (2) simple concatenation.

Backbones	PIAD Seen				PIAD Unseen				PIADv2 Seen				PIADv2 Unseen Object				PIADv2 Unseen Affordance
	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
PointNet++	22.20	88.43	0.605	0.083	13.71	80.92	0.449	0.109	40.06	94.19	0.698	0.063	24.28	84.78	0.449	0.112	13.28	72.21	0.302	0.128
Uni3D	24.67	89.77	0.635	0.078	12.28	77.75	0.412	0.102	40.66	93.94	0.690	0.065	24.02	81.07	0.426	0.131	14.49	71.00	0.272	0.128

Table S.3 : Performance analysis with different point cloud backbones. We evaluate the capability of HAMMER using PointNet++ [ 27 ] and Uni3D [ 51 ] as 3D feature extractors.

Type	Ablation	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Seen	w/ Fine-tuning	22.20	88.43	0.605	0.083
	w/o Fine-tuning	19.67	85.53	0.568	0.092
Unseen	w/ Fine-tuning	13.71	80.92	0.449	0.109
	w/o Fine-tuning	7.55	64.89	0.311	0.142

Table S.4 : Comparison of MLLM fine-tuning strategies. We evaluate the performance when applying LoRA fine-tuning to the MLLM versus keeping its weights entirely frozen during training.

Methods	Size	Seen				Unseen
		aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
InteractVLM	13B	21.20	86.47	0.627	0.081	8.50	75.45	0.414	0.099
HAMMER	3B	22.20	88.43	0.605	0.083	13.71	80.92	0.449	0.109

Table S.5 : Comparison to InteractVLM [ 7 ] on PIAD. We evaluate the performance of our HAMMER against InteractVLM across both Seen and Unseen splits of the PIAD dataset.

Method	Metrics	grasp	cont.	lift	open	lay	sit	supp.	wrap.	pour	move	disp.	push	listen	wear	press	cut	stab
XMFNet	aIOU $\uparrow$	12.82	10.41	13.24	9.17	17.00	26.65	7.89	5.24	7.81	6.46	17.25	2.68	5.31	5.34	7.87	6.35	5.88
	AUC $\uparrow$	62.98	79.97	78.93	79.74	88.93	94.91	84.56	56.47	82.15	54.97	82.91	63.74	74.93	70.57	88.18	79.17	69.97
	SIM $\uparrow$	0.415	0.401	0.334	0.184	0.427	0.619	0.659	0.569	0.399	0.391	0.508	0.535	0.433	0.565	0.237	0.427	0.394
	MAE $\downarrow$	0.121	0.131	0.138	0.131	0.129	0.093	0.119	0.128	0.159	0.192	0.127	0.083	0.164	0.129	0.113	0.135	0.152
IAGNet	aIOU $\uparrow$	16.83	17.12	31.95	28.39	31.80	37.72	12.04	6.02	20.33	5.57	30.57	1.79	15.59	6.55	14.42	12.95	9.48
	AUC $\uparrow$	77.53	83.84	95.05	90.89	93.54	95.94	84.58	66.71	86.02	63.09	89.29	84.71	87.13	71.16	89.46	86.69	76.4

Table S.6 : Evaluation Metrics in PIAD Seen. Results of each affordance type in the Seen. cont. , supp. , wrap. , and disp. denote contain , support , wrapgrasp , and display , respectively.

Method	Metrics	cont.	lay	lift	wrap.	open	disp.	stab	grasp	press	cut
XMFNet	aIOU $\uparrow$	6.29	15.10	7.29	1.42	4.32	6.20	6.12	3.97	5.71	13.95
	AUC $\uparrow$	67.98	84.02	68.45	45.74	78.53	62.20	76.92	59.19	69.32	85.87
	SIM $\uparrow$	0.412	0.503	0.403	0.451	0.156	0.075	0.351	0.278	0.270	0.435
	MAE $\downarrow$	0.137	0.135	0.144	0.156	0.094	0.240	0.087	0.117	0.124	0.078
IAGNet	aIOU $\uparrow$	7.24	18.12	8.47	1.89	12.28	16.28	10.39	4.79	4.22	21.47
	AUC $\uparrow$	67.96	84.82	71.10	56.39	90.91	85.51	98.83	78.60	68.07	95.95

Table S.7 : Evaluation Metrics in PIAD Unseen. Results of each affordance type in the Unseen. cont. , wrap. , and disp. denote contain , wrapgrasp , and display , respectively.

Method	Metrics	Carry	Listen	Lay	Pour	Cut	Pull
XMFNet	aIOU $\uparrow$	5.89	3.89	10.93	6.85	5.77	24.52
	AUC $\uparrow$	54.08	56.07	73.16	63.97	44.85	91.40
	SIM $\uparrow$	0.195	0.216	0.399	0.187	0.115	0.349
	MAE $\downarrow$	0.158	0.179	0.130	0.130	0.213	0.050
IAGNet	aIOU $\uparrow$	8.10	3.71	10.47	4.92	4.40	38.27
	AUC $\uparrow$	63.98	54.13	69.94	59.89	49.97	93.72

Table S.8 : Evaluation Metrics in PIADv2 Unseen Affordance. Results of each affordance type in the Unseen Affordance split.

#	Object Category	Affordance Type	Num
1	TrashCan	contain, open, pour	69
2	Door	open, push	47
3	Display	display	52
4	Earphone	grasp, listen	70
5	Vase	contain, wrapgrasp	90
6	Hat	grasp, wear	66

Table S.9 : Statistics of the constructed corrupted benchmark.

Method	Metrics	Base.	Bucket	Clock	Fork	Kettle	Laptop	Mop	Motor.	Refri.	Scis.	Skat.
XMFNet	aIOU $\uparrow$	37.62	10.26	19.13	27.23	5.50	8.66	20.53	5.54	9.41	10.24	18.38
	AUC $\uparrow$	76.75	59.73	77.86	81.93	61.74	73.52	79.09	69.02	84.22	56.31	81.80
	SIM $\uparrow$	0.528	0.169	0.444	0.497	0.107	0.314	0.400	0.162	0.336	0.261	0.482
	MAE $\downarrow$	0.171	0.165	0.137	0.120	0.123	0.128	0.136	0.033	0.094	0.165	0.151
IAGNet	aIOU $\uparrow$	42.84	12.56	13.48	22.60	2.65	6.95	37.66	9.30	14.43	7.37	4.61
	AUC $\uparrow$	85.32	65.52	69.21	73.63	54.38	78.17	92.09	79.74	80.63	56.62	58.84

Table S.10 : Evaluation Metrics in PIADv2 Unseen Object. Results of each object category in the Unseen Object split. Base. , Motor. , Refri. , Scus. and Skat. denote Baseballbat, Motorcycle, Refrigerator, Scissors, and Skateboard, respectively.

Corrupt Type	Level	GREAT				HAMMER
		aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$	aIOU $\uparrow$	AUC $\uparrow$	SIM $\uparrow$	MAE $\downarrow$
Scale	0	13.85	78.63	0.497	0.115	19.76	86.04	0.604	0.091
	1	13.59	77.89	0.495	0.116	19.26	85.75	0.598	0.093
	2	13.50	77.82	0.491	0.116	19.15	85.83	0.594	0.094
	3	13.11	77.21	0.486	0.117	18.94	85.52	0.591	0.095
	4	13.01	77.30	0.483	0.117	18.81	85.51	0.591	0.094

Table S.11 : Detailed robustness performance under different levels of corruptions. We evaluate the robustness of our HAMMER and GREAT [ 33 ] on 7 corruption types. Each corruption type is evaluated under 5 severity levels (0-4). The results demonstrate that HAMMER consistently outperforms GREAT across all corruption types and severity levels, highlighting its superior robustness in handling corrupted 3D objects.

Figure S.1 : Failure cases of our HAMMER model on PIAD.

Figure S.2 : Visualization on PIADv2 Seen. We compare our HAMMER with GREAT [ 33 ] on several examples from the Seen subset of PIADv2.

Figure S.3 : Visualization on PIADv2 Unseen Object and Unseen Affordance subsets. We compare our HAMMER with GREAT [ 33 ] on several examples from the Unseen Object and Unseen Affordance subsets of PIADv2.

Figure S.4 : Visualization of point features with and without hierarchical cross-modal integration. We visualize the PCA-reduced point features for several examples to demonstrate the effectiveness of our proposed hierarchical cross-modal integration mechanism.

Figure S.5 : Visualization of corrupted point clouds. We illustrate two additional examples of corrupted point clouds from our constructed benchmark.

3D Affordance Grounding

See all 15 papers

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment
Mar 18, 2026Dongqiang Gou, Xuming He3D Affordance Grounding 3D Visual Grounding
ShanghaiTech University
ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes
Aug 11, 2026Xinrui Lin, Sha Zhang, Shumin Wang +33D Affordance Grounding 3D Scene Understanding
University of Science and Technology of China, Hefei, China · The Chinese University of Hong Kong, Hong Kong SAR, China
Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
Aug 3, 2025Hanqing Wang, Zhenhao Zhang, Kaiyang Ji +123D Affordance Grounding Human-Object Interaction
The Hong Kong University of Science and Technology, Guangzhou, China · ShanghaiTech University, Shanghai, China · Zhejiang University, Hangzhou, China +3

Mar 18, 2026J/K move · Enter open · S save

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

Dongqiang Gou, Xuming He

ShanghaiTech University

3D Affordance Grounding 3D Visual Grounding Cross-Modal Alignment

Open paper AskPDF arXiv

Grounding natural language questions to functionally relevant regions in 3D objects -- termed language-driven 3D affordance grounding -- is essential for embodied intelligence and human-AI interaction. Existing methods, while progressing from label-based to language-driven approaches, still face challenges in open-vocabulary generalization, fine-grained geometric alignment, and part-level semantic consistency. To address these issues, we propose a novel two-stage cross-modal framework that enhances both semantic and geometric representations for open-vocabulary 3D affordance grounding. In the first stage, large language models generate part-aware instructions to recover missing semantics, enabling the model to link semantically similar affordances. In the second stage, we introduce two key components: Affordance Prototype Aggregation (APA), which captures cross-object geometric consistency for each affordance, and Intra-Object Relational Modeling (IORM), which refines geometric differentiation within objects to support precise semantic alignment. We validate the effectiveness of our method through extensive experiments on a newly introduced benchmark, as well as two existing benchmarks, demonstrating superior performance in comparison with existing methods.

Abstract

3D Affordance Grounding

Explore similar work

Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment

ThinkAfford: Affordance-Centric Reasoning for Fine-Grained 3D Grounding in Cluttered Scenes

Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning