Organizations: School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China
Existing end-to-end approaches of robotic manipulation often lack generalization to unseen objects or tasks due to limited data and poor interpretability. While recent Multimodal Large Language Models (MLLMs) demonstrate strong commonsense reasoning, they struggle with geometric and spatial understanding required for pose prediction. In this paper, we propose RobMRAG, a 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation (MRAG) framework for zero-shot robotic manipulation. Specifically, we construct a multi-source manipulation knowledge base containing object contact frames, task completion frames, and pose parameters. During inference, a Hierarchical Multimodal Retrieval module first employs a three-priority hybrid retrieval strategy to find task-relevant object prototypes, then selects the geometrically closest reference example based on pixel-level similarity and Instance Matching Distance (IMD). We further introduce a 3D-Aware Pose Refinement module based on 3D Gaussian Splatting into the MRAG framework, which aligns the pose of the reference object to the target object in 3D space. The aligned results are reprojected onto the image plane and used as input to the MLLM to enhance the generation of the final pose parameters. Extensive experiments show that on a test set containing 30 categories of household objects, our method improves the success rate by 7.76% compared to the best-performing zero-shot baseline under the same setting, and by 6.54% compared to the state-of-the-art supervised baseline. Our results validate that RobMRAG effectively bridges the gap between high-level semantic reasoning and low-level geometric execution, enabling robotic systems that generalize to unseen objects while remaining inherently interpretable.
Figures & tables
Figure 1: (a) Although multimodal large language models (MLLMs) possess the commonsense knowledge that opening a drawer requires grasping the handle and can roughly localize it, the predicted 2D contact points consistently fall outside the table itself, let alone on the handle. (b) We incorporate contact pose prediction using a multimodal retrieval-augmented generation (MRAG) framework. By retrieving relevant manipulation examples, our method enables more accurate inference of the contact pose on the target object.
Figure 2: Overview of the RobMRAG: a 3D Gaussian splatting-enhanced multimodal retrieval-augmented generation framework.
Method
Test Seen Split
RAM ( Kuang et al. 2024 ) ( Zero-shot ) †
43.14
25.67
14.00
35.29
52.94
35.49
49.02
45.39
52.00
49.23
45.76
36.02
24.00
61.58
24.00
72.00
CrayonRobo ( Li et al. 2025 ) ( Zero-shot ) †
17.65
21.59
28.00
47.06
62.18
33.33
57.45
58.54
48.00
49.19
37.24
57.39
26.00
73.33
12.00
68.00
Ours ( Zero-shot )
62.75
47.45
20.00
66.67
50.98
17.65
59.18
72.05
64.00
43.14
49.02
38.00
22.00
67.85
18.00
78.00
Where2Act ( Mo et al. 2021 )
26.00
36.00
19.00
27.00
23.00
11.00
15.00
47.00
14.00
24.00
13.00
12.00
56.00
68.00
7.00
40.00
UMPNet ( Xu et al. 2022 )
46.00
43.00
15.00
28.00
54.00
32.00
28.00
56.00
44.00
40.00
10.00
23.00
18.00
54.00
20.00
42.00
Table 1: Comparisons of our method against baseline methods, † indicates reproduced results (ASR, %).
Ablation Method
Test seen
Test unseen
Main Component Ablation
w/o Retrieval (only vision input)
19.46
19.01
w/ Textual Retrieval + CosSim only
37.05
34.62
w/ Textual Retrieval + IMD only
45.55
41.27
w/ Textual Retrieval + CosSim + IMD
47.34
42.08
Full MRAG (w/ 3D Pose Align)
49.25
43.53
Table 2: Ablation studies on different components of our MRAG framework (ASR, %).
Figure 3: Ablation study on the number of retrieval candidates (Top-n) and rotation samples ( K ).
Figure 4: Visualization of the three-priority hybrid retrieval strategy: ① sparse retrieval (simulation), ② dense retrieval (simulation), and ③ cross-source retrieval (robotic/Internet).
Figure 5: Visualization of 3D-Aware Pose Refinement.
This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.
Ali Alabbas, Dipshikha Das, Camillo Murgia +3
Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland
Generalizing robotic manipulation to unseen objects remains challenging, as learning-based approaches require many demonstrations and fail in few-shot settings. Prior work transfers affordances through semantic retrieval, but semantics alone neglect geometric similarity, which is critical for manipulation. We propose GRAFT, a geometry-aware correspondence framework for zero-shot manipulation transfer using only one demonstration per object. Objects are represented as part-based graphs, where part-level descriptors support global instance retrieval and part correspondence, and vertex-level descriptors enable fine-grained contact point matching. For an unseen object, our method first retrieves the most functionally and geometrically similar instance from the demonstration buffer with aligned functional parts, and finally propagates the contact points through point-wise correspondence.
Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.
Qiyuan Zhuang, He-Yang Xu, Yijun Wang +3
School of Computer Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, Nanjing 211189, China · Southeast University-Monash University Joint Graduate School, Southeast University, Suzhou 215123, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China