Organizations: School of Computer Science and Technology, Tongji University · School of Electrical Engineering and Computer Science, The University of Queensland · IEIT SYSTEMS (Beijing) Co., Ltd · NAIS, Neusoft Corporation
Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.
Figures & tables
Figure 1: UniRAP transfers large-scale generic segmentation pretraining to affordance segmentation and spatiotemporally actionable perception, improving language grounding for task-relevant regions and dynamics. It enhances referring segmentation, affordance grounding, and interaction modeling while maintaining strong spatio-temporal consistency.
Figure 2: Overview of UniRAP Model. Given visual input xv , text prompt xt , and optional visual reference prompts xr , a multimodal model F produces <SEG> -conditioned outputs. When <REF> is present, visual prompts are encoded as embeddings and jointly modeled with text within a shared <SEG> representation space. Concurrently, visual features from xv are aggregated with memory bank representations via memory attention. The resulting features are aligned with <SEG> -derived embeddings through an MLP projector and decoded by an affordance decoder D to generate segmentation masks, region detections, and 4-DoF interaction poses for unified vision–language spatiotemporal actionable perception.
Figure 3: Architecture of Unified Affordance Decoder.
Figure 4: Curriculum Transfer Training Pipeline. Details in Appendix B.2 .
Method
Size
RefCOCO [ 118 ]
RefCOCO+ [ 118 ]
RefCOCOg [ 119 ]
ReasonSeg [ 14 ]
val
testA
testB
val
testA
testB
val(U)
test(U)
gIoU
cIoU
Non-LLM-based Specialists
ReLA [ 120 ]
–
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
–
–
X-Decoder [ 121 ]
–
–
–
–
–
–
–
64.6
–
22.6
17.9
SEEM [ 122 ]
–
–
–
–
–
–
–
65.7
–
25.5
21.2
LLM-based Image Generalists
Table 1: Referring Expression and Reasoning Segmentation. cIoU on RefCOCO/+/g.
Method
Size
Referring
Reasoning
Overall
R
J
F
J&F
J
F
J&F
J
F
J&F
Non-LLM-based Specialists
MTTR [ 127 ]
–
29.8
30.2
30.0
20.4
21.5
21.0
25.1
25.9
25.5
5.6
LMPM [ 128 ]
–
29.0
39.1
34.1
13.3
24.3
18.8
21.2
31.7
26.4
3.2
ReferFormer [ 129 ]
–
31.2
34.3
32.7
21.3
25.6
23.4
26.2
29.9
28.1
8.8
LLM-based Generalists
Table 2: Referring and Reasoning Video Object Segmentation on ReVOS val [ 107 ] split.
Table 6: Ablation Study of different pre-training stages, fine-tuning strategies, and prediction heads.
Figure 10
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Dataset
Inputs
Outputs
#Samples
#Repeat
Text
Image
Video
Point
Box
Text
Mask
Box
Grasp
RefCOCO [ 118 ]
✓
✓
✓
✓
17K
5
RefCOCO+ [ 118 ]
✓
✓
✓
✓
17K
5
RefCOCOg [ 119 ]
✓
✓
✓
✓
22K
5
RefClef [ 118 ]
✓
✓
✓
✓
18K
5
ReasonSeg [ 14 ]
✓
✓
✓
✓
1.6K
10
Appendix
Table 8: Distribution of Training Datasets for UniRAP. \Circle denotes random selection between two prompt types per sample; † indicates text prompts annotated by RAGNet [ 46 ] . The fourth stage uses the RealVLG dataset [ 47 ] . Stage 1: General Pre-training; Stage 2: Affordance Pre-training; Stage 3: Visual Referring Pre-training; Stage 4: Actionable Affordance Fine-tuning.
Figure 6: Real-World Experimental Setup. A Franka FR3 robotic arm is equipped with an eye-in-hand RealSense D435i camera mounted on the end-effector. The test set consists of 13 common household objects, where objects 1–6 are used to evaluate referring grasping performance, and objects 7–13 are used to assess reasoning-based grasping performance.
Object ID
Object Name
Prompt
Referring
1
Blue Spoon
Please give me the blue spoon.
2
Orange Cup
Please grasp the orange cup.
3
Razor
Please pick up the grey and black razor.
4
Screwdriver
Please give me the screwdriver.
5
Milk Box
Please give me the milk box.
Appendix
Table 9: Object Prompts for Real-World Robotic Vision-Language Grasping.
Task
Referring
Reasoning
Avg.
ObjectID
1
2
3
4
5
6
7
8
9
10
11
12
13
–
RAGNet [ 46 ] + GraspNet [ 133 ]
50
60
40
60
50
60
30
40
30
40
30
40
40
43.8
UniRAP + GraspNet [ 133 ]
70
70
60
70
60
50
40
70
50
50
40
40
50
55.4
RealVLG-R1 [ 47 ]
80
60
70
80
70
70
50
80
40
70
40
20
80
62.3
UniRAP (Ours)
100
90
80
100
90
100
80
80
70
90
80
90
100
88.5
Appendix
Table 10: Grasp Success Rates (%) on 13 Household Objects.
Figure 7: Qualitative Results of Referring Expression and Reasoning Segmentation.
Figure 8: Qualitative Results of Affordance Segmentation.
Figure 9: Qualitative Results of Referring and Reasoning Video Object Segmentation.
Figure 10: Qualitative Results of Actionable Perception on RealVLG-Benchmark [ 47 ] .
Figure 11: Qualitative Results of Spatio-Temporal Actionable Perception. To evaluate the temporal consistency of UniRAP’s actionable perception, we construct pseudo-sequences by randomly sampling 6 frames from each scene in the RealVLG-GraspNet subset, due to the absence of video-based grasping datasets.
Figure 12: Qualitative Results of Real-World Referring Vision-Language Grasping.
Figure 13: Qualitative Results of Real-World Reasoning-Based Vision-Language Grasping. Compared to RealVLG-R1, UniRAP exhibits a stronger affordance modeling capability, enabling it to predict more physically plausible grasp poses, such as correctly identifying appropriate grasp regions on objects like a Shovel . In addition, UniRAP demonstrates more robust localization performance. For instance, RealVLG-R1 fails to accurately localize the Teapot , whereas UniRAP successfully identifies a valid interaction region and produces a stable grasp prediction.
Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception-action mappings. To address this challenge, we propose AffordanceVLA, a scalable, unified framework that introduces structured affordance learning as a task-oriented and object-centric intermediate representation to establish a more precise and robust perception-action mapping. Specifically, we progressively model manipulation priors through three complementary components - Which2Act, Where2Act, and How2Act - which jointly produce spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language, and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture and adopt a three-stage progressive training curriculum that leverages 8.8M affordance-centric grounding and pointing samples, translating scalable vision-language supervision into improved robotic manipulation. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on ID & OOD simulation and real-world demonstrate that AffordanceVLA outperforms strong VLA baselines across diverse manipulation scenarios, validating structured affordance learning as a scalable and robust bridge that integrates multi-source vision-language supervision while connecting pretrained VLM semantics to embodied robot control.
Qize Yu, Jiadi You, Yuran Wang +12
Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1
Affordance segmentation in 3D scenes requires an agent to ground implicit natural-language instructions into precise masks of fine-grained interactive elements. Existing training-free methods typically rely on fragmented pipelines, which introduce visual blindness during task parsing and limit accuracy through single-scale spatial and temporal processing. We present UniFunc3D, a unified and training-free framework that treats the multimodal large language model as an active observer. By utilizing a unified MLLM backbone, UniFunc3D performs joint semantic-temporal-spatial reasoning to ground task decomposition in direct visual evidence. Our approach introduces active spatial-temporal grounding with a coarse-to-fine strategy. This allows the model to select correct video frames adaptively and focus on high-detail interactive parts while preserving the global context necessary for disambiguation. On SceneFun3D, our UniFunc3D achieves state-of-the-art performance, surpassing prior training-free methods by a large margin with a relative 59.9% mIoU improvement, and even outperforming training-based methods without any task-specific training. Code is available on our project page: https://jiaying.link/unifunc3d.
Jiaying Lin, Dan Xu
The Hong Kong University of Science and Technology
Affordance perception aims to localize actionable regions supporting embodied interaction, yet 2D and 3D affordance grounding have evolved as separate problems, with different task definitions, supervision formats, datasets, and evaluation protocols. This fragmentation limits the learning of transferable object-affordance semantics across visual and geometric spaces. We propose Token Router for Tasks, a multitask training paradigm for MLLM-based systems that routes contextual hidden states to task-specific branches without requiring the language head to generate predefined markers. Routed states are supervised directly by branch-specific objectives, enabling dense prediction losses to shape shared MLLM representations. We instantiate this paradigm as UniAfford, a unified framework for generalizable 2D-3D affordance perception, together with UniAfford-Data, a dataset integrating pixel-level 2D annotations, point-level 3D annotations, and language instructions under a shared object-affordance taxonomy, supporting heterogeneous supervision through semantic-level 2D-3D pairing. UniAfford adopts an MLLM as a shared semantic hub and a modality-aware token router to produce image- and point-cloud-affordance queries. These queries respectively condition a SAM-style pixel decoder and a SONATA-based point decoder, enabling flexible 2D, 3D, and joint affordance inference from image-only, point-cloud-only, or paired multimodal inputs. Experiments demonstrate strong zero-shot generalization across 2D and 3D affordance benchmarks without target-specific fine-tuning, alongside state-of-the-art branch-wise performance under modality-isolated protocols. Ablations validate token routing, joint 2D-3D supervision, and decoder coupling, while language-head diagnostics show that routed latent states carry meaningful object-affordance semantics. Project page: https://4dvlab.github.io/UniAfford
Yuhao Liu, Yiming Zhong, Hanqing Wang +9
ShanghaiTech University · Shandong University · HKUST(GZ) +3