UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

Linfei Li; Ruining Hu; Lin Zhang; Zhong Wang; Fengyi Zhang; Ying Shen; Binqiang Wang; Xin Zhang

cs.CVOct 21, 2026

UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

Authors: Linfei Li, Ruining Hu, Lin Zhang, Zhong Wang, Fengyi Zhang, Ying Shen, Binqiang Wang, Xin Zhang

Organizations: School of Computer Science and Technology, Tongji University · School of Electrical Engineering and Computer Science, The University of Queensland · IEIT SYSTEMS (Beijing) Co., Ltd · NAIS, Neusoft Corporation

Abstract

Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.

Figures & tables

Figure 1: UniRAP transfers large-scale generic segmentation pretraining to affordance segmentation and spatiotemporally actionable perception, improving language grounding for task-relevant regions and dynamics. It enhances referring segmentation, affordance grounding, and interaction modeling while maintaining strong spatio-temporal consistency.

Figure 2: Overview of UniRAP Model. Given visual input $\mathbf{x}_{v}$ , text prompt $\mathbf{x}_{t}$ , and optional visual reference prompts $\mathbf{x}_{r}$ , a multimodal model $\mathcal{F}$ produces <SEG> -conditioned outputs. When <REF> is present, visual prompts are encoded as embeddings and jointly modeled with text within a shared <SEG> representation space. Concurrently, visual features from $\mathbf{x}_{v}$ are aggregated with memory bank representations via memory attention. The resulting features are aligned with <SEG> -derived embeddings through an MLP projector and decoded by an affordance decoder $\mathcal{D}$ to generate segmentation masks, region detections, and 4-DoF interaction poses for unified vision–language spatiotemporal actionable perception.

Figure 3: Architecture of Unified Affordance Decoder.

Figure 4: Curriculum Transfer Training Pipeline. Details in Appendix B.2 .

Method	Size	RefCOCO [ 118 ]			RefCOCO+ [ 118 ]			RefCOCOg [ 119 ]		ReasonSeg [ 14 ]
Method	Size	val	testA	testB	val	testA	testB	val(U)	test(U)	gIoU	cIoU
Non-LLM-based Specialists
ReLA [ 120 ]	–	73.8	76.5	70.2	66.0	71.0	57.7	65.0	66.0	–	–
X-Decoder [ 121 ]	–	–	–	–	–	–	–	64.6	–	22.6	17.9
SEEM [ 122 ]	–	–	–	–	–	–	–	65.7	–	25.5	21.2
LLM-based Image Generalists

Table 1: Referring Expression and Reasoning Segmentation. cIoU on RefCOCO/+/g.

Method	Size	Referring			Reasoning			Overall			$\mathcal{R}$
Method	Size	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J\&F}$	$\mathcal{R}$
Non-LLM-based Specialists
MTTR [ 127 ]	–	29.8	30.2	30.0	20.4	21.5	21.0	25.1	25.9	25.5	5.6
LMPM [ 128 ]	–	29.0	39.1	34.1	13.3	24.3	18.8	21.2	31.7	26.4	3.2
ReferFormer [ 129 ]	–	31.2	34.3	32.7	21.3	25.6	23.4	26.2	29.9	28.1	8.8
LLM-based Generalists

Table 2: Referring and Reasoning Video Object Segmentation on ReVOS val [ 107 ] split.

Table 7

Method	Size	Seen			Similar			Novel
Method	Size	Bbox	Seg.	Grasp	Bbox	Seg.	Grasp	Bbox	Seg.	Grasp
		gIoU/cIoU	F β /S α	mIoU/gAcc	gIoU/cIoU	F β /S α	mIoU/gAcc	gIoU/cIoU	F β /S α	mIoU/gAcc
Qwen-VL-Max [ 135 ]	-	56.6/57.1	62.8/32.0	2.3/2.4	57.5/57.9	57.6/28.9	2.7/1.3	59.8/60.1	30.3/24.7	1.8/1.2
Gemini2.5-Flash [ 136 ]	-	50.1/51.3	2.8/11.8	3.2/1.2	47.9/49.0	3.1/10.6	2.5/2.2	59.5/60.6	3.7/14.2	3.4/1.9
Qwen2.5VL [ 8 ]	7B	56.8/58.3	80.6/43.8	1.8/0.8	57.1/57.7	77.7/42.2	1.6/0.0	55.6/56.8	48.3/39.2	1.0/0.0
Qwen2.5VL+SFT [ 8 ]	7B	64.8/63.5	82.2/44.1	4.1/2.5	66.7/62.7	79.6/42.6	3.9/2.3	64.9/65.2	48.5/39.4	2.7/1.1

Table 5: Actionable Perception Performance on RealVLG-Benchmark [ 47 ] .

Training Strategy				Prediction Head			Evaluation Benchmark
General Pre-train	Affordance Pre-train	Visual Referring Pre-train	Actionable Affordance Fine-tune	Bbox Head	Heatmap Grasp Head	UniRAP Grasp Head	ReasonSeg (gIoU)	ReVOS (J&F)	3DOI (Ref. gIoU)	3DOI (Reason, gIoU)	RealVLG-Bench (Grasp, gAcc)
✓							55.4	58.3	40.5	37.5	–
✓	✓						56.4	59.2	50.4	52.9	–
✓	✓	✓		✓		✓	57.3	60.2	51.6	54.6	19.6
✓	✓		✓	✓		✓	57.0	59.8	50.8	54.2	26.4
✓	✓	✓	✓			✓	56.8	60.4	51.3	53.8	24.7

Table 6: Ablation Study of different pre-training stages, fine-tuning strategies, and prediction heads.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Stage

Dataset

Inputs

Outputs

#Samples

#Repeat

Text

Image

Video

Point

Box

Text

Mask

Box

Grasp

RefCOCO [ 118 ]

✓

17K

RefCOCO+ [ 118 ]

✓

17K

RefCOCOg [ 119 ]

✓

22K

RefClef [ 118 ]

✓

18K

ReasonSeg [ 14 ]

✓

1.6K

Appendix

Table 8: Distribution of Training Datasets for UniRAP.

\Circle

denotes random selection between two prompt types per sample; † indicates text prompts annotated by RAGNet [ 46 ] . The fourth stage uses the RealVLG dataset [ 47 ] . Stage 1: General Pre-training; Stage 2: Affordance Pre-training; Stage 3: Visual Referring Pre-training; Stage 4: Actionable Affordance Fine-tuning.

Figure 6: Real-World Experimental Setup. A Franka FR3 robotic arm is equipped with an eye-in-hand RealSense D435i camera mounted on the end-effector. The test set consists of 13 common household objects, where objects 1–6 are used to evaluate referring grasping performance, and objects 7–13 are used to assess reasoning-based grasping performance.

Object ID	Object Name	Prompt
Referring
1	Blue Spoon	Please give me the blue spoon.
2	Orange Cup	Please grasp the orange cup.
3	Razor	Please pick up the grey and black razor.
4	Screwdriver	Please give me the screwdriver.
5	Milk Box	Please give me the milk box.

Appendix

Table 9: Object Prompts for Real-World Robotic Vision-Language Grasping.

Task	Referring						Reasoning							Avg.
ObjectID	1	2	3	4	5	6	7	8	9	10	11	12	13	–
RAGNet [ 46 ] + GraspNet [ 133 ]	50	60	40	60	50	60	30	40	30	40	30	40	40	43.8
UniRAP + GraspNet [ 133 ]	70	70	60	70	60	50	40	70	50	50	40	40	50	55.4
RealVLG-R1 [ 47 ]	80	60	70	80	70	70	50	80	40	70	40	20	80	62.3
UniRAP (Ours)	100	90	80	100	90	100	80	80	70	90	80	90	100	88.5

Appendix

Table 10: Grasp Success Rates (%) on 13 Household Objects.

Figure 7: Qualitative Results of Referring Expression and Reasoning Segmentation.

Figure 8: Qualitative Results of Affordance Segmentation.

Figure 9: Qualitative Results of Referring and Reasoning Video Object Segmentation.

Figure 10: Qualitative Results of Actionable Perception on RealVLG-Benchmark [ 47 ] .

Figure 11: Qualitative Results of Spatio-Temporal Actionable Perception. To evaluate the temporal consistency of UniRAP’s actionable perception, we construct pseudo-sequences by randomly sampling 6 frames from each scene in the RealVLG-GraspNet subset, due to the absence of video-based grasping datasets.

Figure 12: Qualitative Results of Real-World Referring Vision-Language Grasping.

Figure 13: Qualitative Results of Real-World Reasoning-Based Vision-Language Grasping. Compared to RealVLG-R1, UniRAP exhibits a stronger affordance modeling capability, enabling it to predict more physically plausible grasp poses, such as correctly identifying appropriate grasp regions on objects like a Shovel . In addition, UniRAP demonstrates more robust localization performance. For instance, RealVLG-R1 fails to accurately localize the Teapot , whereas UniRAP successfully identifies a valid interaction region and produces a stable grasp prediction.

Robotic Affordance Inference

See all 24 papers

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
Jun 4, 2026Qize Yu, Jiadi You, Yuran Wang +12Language-Conditioned Robot Manipulation Robotic Manipulation
Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1
UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation
Mar 24, 2026Jiaying Lin, Dan Xu3D Scene Understanding 3D Visual Grounding
The Hong Kong University of Science and Technology
UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception
Sep 29, 2026Yuhao Liu, Yiming Zhong, Hanqing Wang +9Cross-Modal Representation Learning Unified Multimodal Models
ShanghaiTech University · Shandong University · HKUST(GZ) +3

Jun 4, 2026J/K move · Enter open · S save

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

Qize Yu, Jiadi You, Yuran Wang, Jiaqi Liang+11

Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong+1

Language-Conditioned Robot Manipulation Robotic Manipulation Vision-Language-Action Models Robot Learning

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception-action mappings. To address this challenge, we propose AffordanceVLA, a scalable, unified framework that introduces structured affordance learning as a task-oriented and object-centric intermediate representation to establish a more precise and robust perception-action mapping. Specifically, we progressively model manipulation priors through three complementary components - Which2Act, Where2Act, and How2Act - which jointly produce spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language, and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture and adopt a three-stage progressive training curriculum that leverages 8.8M affordance-centric grounding and pointing samples, translating scalable vision-language supervision into improved robotic manipulation. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on ID & OOD simulation and real-world demonstrate that AffordanceVLA outperforms strong VLA baselines across diverse manipulation scenarios, validating structured affordance learning as a scalable and robust bridge that integrates multi-source vision-language supervision while connecting pretrained VLM semantics to embodied robot control.

Abstract

Robotic Affordance Inference

Explore similar work

AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding

UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation

UniAfford: Token-Routed Multitask Learning for Generalizable 2D-3D Affordance Perception