AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis

Xiaofei Wu; Yi Zhang; Yumeng Liu; Yuexin Ma; Yujiao Shi; Xuming He

cs.ROMar 9, 2026

AffordGrasp: Cross-Modal Diffusion for Affordance-Aware Grasp Synthesis

Authors: Xiaofei Wu, Yi Zhang, Yumeng Liu, Yuexin Ma, Yujiao Shi, Xuming He

Organizations: ShanghaiTech University, Shanghai, China · University of Science and Technology of China · Shanghai Engineering Research Center of Intelligent Vision and Imaging

Abstract

Generating human grasping poses that accurately reflect both object geometry and user-specified interaction semantics is essential for natural hand-object interactions in AR/VR and embodied AI. However, existing semantic grasping approaches struggle with the large modality gap between 3D object representations and textual instructions, and often lack explicit spatial or semantic constraints, leading to physically invalid or semantically inconsistent grasps. In this work, we present AffordGrasp, a diffusion-based framework that produces physically stable and semantically faithful human grasps with high precision. We first introduce a scalable annotation pipeline that automatically enriches hand-object interaction datasets with fine-grained structured language labels capturing interaction intent. Building upon these annotations, AffordGrasp integrates an affordance-aware latent representation of hand poses with a dual-conditioning diffusion process, enabling the model to jointly reason over object geometry, spatial affordances, and instruction semantics. A distribution adjustment module further enforces physical contact consistency and semantic alignment. We evaluate AffordGrasp across four instruction-augmented benchmarks derived from HO-3D, OakInk, GRAB, and AffordPose, and observe substantial improvements over state-of-the-art methods in grasp quality, semantic accuracy, and diversity.

Figures & tables

Figure 2: Overview of AffordGrasp. We integrate language instructions with object point clouds and employ the Affordance Generator to predict point-wise confidence features, which are aggregated into the final affordance map to enhance spatial detail and align linguistic semantics with 3D structures.The right part employs a DAM module, which ensures that the synthesized grasping poses generated by the LDM model align with physical constraints and language semantics.

Figure 3: Distribution Adjustment Module (DAM) Architecture. Hand and object features are fused and aligned with language instructions to produce stable, instruction-consistent grasps.

Dataset	Method	Penetration Volume $\downarrow$	Simulation Displacement $\downarrow$	Contact Ratio $\uparrow$	Entropy $\uparrow$	Cluster Size $\uparrow$	ACC $\uparrow$
OakInk [ 33 ]	TTA [ 9 ]	8.21	2.33	97	2.82	2.81	60.83%
	FastGrasp [ 29 ]	7.88	2.27	88	2.88	3.42	78.05%
	D-VQVAE [ 39 ]	7.33	2.44	91	2.88	3.57	76.4%
	Ours(ControlNet)	8.08	3.11	81	2.87	3.44	76.94%
	Ours	7.31	1.43	98	2.94	3.74	80.08%
GRAB [ 25 ]	TTA [ 9 ]	6.51	1.22	91	2.73	1.41	55.00%

Table 1: Quantitative comparison on the OakInk and GRAB dataset. We compare our results with baselines as well as with a framework where the DAM module is replaced by the ControLNet [ 38 ] structure. Our method achieves the best performance on all evaluation metrics.

Dataset	Method	Penetration Volume $\downarrow$	Simulation Displacement $\downarrow$	Contact Ratio $\uparrow$	Entropy $\uparrow$	Cluster Size $\uparrow$	ACC $\uparrow$
HO-3D [ 6 ]	TTA [ 9 ]	12.55	3.22	95	2.55	2.99	66%
	FastGrasp [ 29 ]	14.45	2.73	96	2.81	2.23	52.00%
	D-VQVAE [ 39 ]	13.12	2.33	95	2.78	3.63	64.00%
	Ours(ControlNet)	16.06	2.38	97	2.84	3.52	51.00%
	Ours	7.38	2.33	97	2.85	3.70	72.00%
AffordPose [ 8 ]	TTA [ 9 ]	19.41	4.32	91	2.89	2.97	42.56%

Table 2: Comparison with previous methods on the HO-3D and AffordPose dataset, where our model is trained on the GRAB [ 25 ] dataset. Our model achieves state-of-the-art performance on two out-of-domain dataset, setting new benchmarks.

Figure 4: Affordance Annotation. Implement an automated self-training pipeline that first assigns pseudo-labels to unlabeled data, then iteratively optimizes the model using these refined annotations.

Dataset	Method	Penetration Volume $\downarrow$	Simulation Displacement $\downarrow$	Contact Ratio $\uparrow$	Entropy $\uparrow$	Cluster Size $\uparrow$	ACC $\uparrow$
OakInk [ 33 ]	w/o object affordance	8.27	1.22	97	2.88	3.81	76.56%
	w/o DAM	8.12	1.77	97	2.89	4.07	79.11%
	Whole pipeline	7.31	1.43	98	2.94	3.74	80.08%
GRAB [ 25 ]	w/o object affordance	4.32	1.55	92	2.79	3.61	55.00%
	w/o DAM	4.91	1.71	90	2.84	3.77	63.00%
	Whole pipeline	3.06	1.66	94	2.91	3.53	62.50%

Table 3: Ablation study results on the GRAB, OakInk, HO-3D, AffordPose datasets [ 25 , 33 , 6 , 8 ] . The evaluation of the HO-3D and AffordPose is an out-of-domain generalization test, where the model is trained using the GRAB dataset.

Panel of Figure 5 — Figure 5: Qualitative comparisons with state-of-the-art methods on GRAB, OakInk datasets. Each pair (two columns) visualizes the generated grasps from two different views. Our method demonstrates a significant reduction in object penetration compared to other methods.

Figure 6: Simulation results: grasping a single object under different instructions.

Figure 7: Simulation results: grasping across multiple objects.

Method	Simulation Displacement $\downarrow$	Penetration Volume $\downarrow$	Penetration Distance $\downarrow$	Contact Ratio $\uparrow$
D-VQVAE [ 39 ]	1.61	0.94	5.17	98
FastGrasp [ 29 ]	1.83	0.91	2.39	98

Table 4: Comparative experiments to evaluate the impact of different autoencoder structures.

Method	TTA	Sample optimization	Ours
Time	7.09s	1.73s	0.45s

Table 5: Different optimization method’s inference efficiency.

Figure 8: LLM generates the pipeline of language instructions.

Dataset	train set	val set	test set
Acc	98.11%	97.90%	98.48%

Table 6: Classification accuracy experiment.

Figure 9: Object Affordance Visualization. Left. Grip the handle of the mug. Mid. Wrap your hand around the mug. Right. Support the mug to prevent spills.

Figure 10: Object Affordance Visualization. Left. Press the dispenser to avoid over-pouring. Mid. Wrap your fingers around the dispenser for a secure hold. Right. Support the bottle’s handle to prevent spills.

Figure 11: Object Affordance Visualization. Left. Grip the handle of the bottle. Mid. Wrap-grasp the bottle to prevent spill. Right. twist the bottle to open it.

	bag	bottle	dispenser	earphone	faucet	handle-bottle	jar	keyboard	knife	laptop	mug	pot	scissors
IoU $\uparrow$	0.842	0.866	0.850	0.903	0.867	0.867	0.822	0.764	0.899	0.751	0.905	0.836	0.905
AUC $\uparrow$	0.999	0.988	0.996	1.000	0.999	0.998	0.991	0.986	1.000	0.996	0.999	0.997	0.999
SIM $\uparrow$	0.943	0.953	0.949	0.986	0.965	0.965	0.924	0.885	0.982	0.876	0.979	0.926	0.982
MAE $\downarrow$	0.009	0.025	0.018	0.006	0.010	0.012	0.038	0.061	0.008	0.019	0.009	0.017	0.015

Table 7: Assessment for different object categories.

	Handle-grasp	Press	Lift	Wrap-grasp	Twist	Support	Pull	Lever	OVERALL
IoU $\uparrow$	0.889	0.786	0.872	0.910	0.807	0.723	0.717	0.870	0.855
AUC $\uparrow$	1.000	0.993	1.000	0.991	0.998	0.990	0.999	0.999	0.996
SIM $\uparrow$	0.975	0.903	0.965	0.975	0.923	0.851	0.836	0.967	0.948
MAE $\downarrow$	0.009	0.032	0.001	0.030	0.013	0.029	0.006	0.009	0.019

Table 8: Assessment for different action.

Dataset	Head nums	Penetration Volume $\downarrow$	Simulation Displacement $\downarrow$	Contact Ratio $\uparrow$	Entropy $\uparrow$	Cluster Size $\uparrow$	ACC $\uparrow$
	1	8.32	1.66	96	2.79	4.14	76.33%
OakInk [ 33 ]	2	7.79	1.38	97	2.88	3.66	77.21%
	4	7.32	1.43	98	2.94	3.74	80.08%
	1	5.71	1.25	98	2.79	3.77	62.50%
GRAB [ 25 ]	2	5.11	1.13	100	2.87	3.75	62.50%
	4	3.06	1.66	94	2.91	3.53	62.50%

Table 9: Parameter sensitivity experiment. We performed parameter sensitivity experiments(Tab. 3 ) using the same setting

Figure 12: Visualization of randomly selected grasping poses across different objects, each displayed from two viewpoints.

Figure 13: Visualization of three grasping poses for the same object, each shown from two viewpoints.

Figure 14: Visualizations of affordance generation for multiple objects.

Figure 15: Failure cases of our method. Each pair displays a sample from two views.

Method	Success Rate (%)	MPJPE $\downarrow$	FOL $\downarrow$
CrossDex [ 36 ]	92.90	-	0.260
AffordGrasp(Ours)	92.96	0.161	0.235

Table 10: Quantitative evaluation of AffordGrasp in the simulation environment.

Figure 16: Grasp execution results guided by AffordGrasp in the simulation environment.

Panel of Figure 17 — Figure 17: Real robot visualization.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 80 papers

DCGrasp: Distance-aware Controllable Grasp Generation
Jun 29, 2026Hiroyasu Akada, Jesús Pérez, Emre Aksan +6HOI Generation Robotic Grasping
Google · Max Planck Institute for Informatics, SIC
SECOND-Grasp: Semantic Contact-guided Dexterous Grasping
May 13, 2026Han Yi Shin, Heeju Ko, Jaewon Mun +6Robotic Grasping Language-Conditioned Robot Manipulation
1Korea University · University of Texas at Austin · University of Michigan +1
FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidance
Feb 28, 2025Yongqi Tian, Xueyu Sun, Haoyuan He +2HOI Generation Robotic Grasping
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China

Jun 29, 2026J/K move · Enter open · S save

DCGrasp: Distance-aware Controllable Grasp Generation

Hiroyasu Akada, Jesús Pérez, Emre Aksan, Vasileios Choutas+5

Google · Max Planck Institute for Informatics, SIC

HOI Generation Robotic Grasping

Open paper AskPDF arXiv

Generating 3D hand-object interactions is essential for applications in robotics, XR, and synthetic data generation, where flexible controllability and strong generalization to diverse object geometries are required. However, existing methods rarely satisfy these requirements, limiting their practical applicability. We present DCGrasp, a distance-aware controllable grasp generation system built on a novel grasp energy term. This term computes Distance Profile, a signed distance from each hand vertex to the nearest object point, coupled with distance-aware weighting, effectively capturing the semantically similar hand-object interaction in near-contact regions while remaining invariant to object and hand identity. Given various controllable signals, DCGrasp first generates a Distance Profile based on a Diffusion Transformer, together with a corresponding candidate hand pose. We then refine the candidate pose through optimization, enforcing consistency between the optimized hand pose and the generated Distance Profile in near-contact regions. Our experiments show that DCGrasp produces high-quality, physically plausible grasps with flexible user control, generalizing to diverse object and hand shapes and scales. Our work establishes a robust and versatile pipeline for the synthesis of controllable 3D hand-object interactions.

Abstract

HOI Generation

Explore similar work

DCGrasp: Distance-aware Controllable Grasp Generation

SECOND-Grasp: Semantic Contact-guided Dexterous Grasping

FunHOI: Annotation-Free 3D Hand-Object Interaction Generation via Functional Text Guidance