Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China. · State Key Laboratory of Brain Cognition and Brain-Inspired Intelligence Technology, Shanghai 200031, China · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China.
Fine-grained object (FO) manipulation requires robots to distinguish a specified FO from visually similar objects and execute actions reliably despite scene distractors. However, scene-level visual conditioning lacks explicit object selection, while category-level guidance cannot reliably distinguish FOs within the same category. We present a SAM3-guided visuomotor policy framework that addresses these challenges through persistent object memory and focused visual conditioning. First, we introduce FO Memory-driven SAM3 (FOM-SAM3), which learns reusable FO memory tokens from limited multi-view registration images, while keeping SAM3 fully frozen. Through one-vs-rest learning, these tokens encode persistent memories for localizing target FOs and rejecting similar alternatives, which can be stored in a memory bank for reuse. Second, we propose Focused Spatial-Appearance Encoding (FSAE), which combines in-FO appearance features with explicit bounding boxes to condition action policies, including Diffusion Policy (DP) and Action Chunking with Transformers (ACT). The effectiveness of the proposed FOM-SAM3 is validated on the FO-30 dataset comprising 30 physical objects across four coarse categories. Across three real-robot FO manipulation tasks, our FOM-SAM3-guided policies demonstrated robustness against distractors, discriminative ability of similar FOs, and extendibility to new FOs.
Figures & tables
Fig. 1: Fine-grained object (FO) manipulation with FOM-SAM3 guided policy. In contrast to category-level object manipulation, FO manipulation requires identifying and focusing on an object with specified attributes, category, brand, and model. The current methods using scene- and category-level visual conditioning are unable to handle FO manipulation tasks. Our framework leverages SAM3 for FO-level visual guidance, which can be extended to different FOs belonging to the same coarse category via persistent memory learning.
Fig. 2: Persistent FO memory learning for SAM3.
Fig. 3: FOM-SAM3 guided policy with focused spatial-appearance encoding. Two FSAE versions are designed for DP and ACT, respectively. Here, SSP and TP stand for spatial softmax pooling and token projection, respectively.
Fig. 4: FO segmentation results on the FO-30 test set with the original SAM3 text prompting and the proposed FOM-SAM3. The red text indicates that the target FO is absent in the image.
TABLE I: FO Perception Performances Based on Frozen SAM3 (%)
Fig. 5: Real-robot platform, three actions, and experimental objects. DemoFO is the object used in teleoperated demonstration. Distractors are dissimilar objects from different coarse categories. SimilarFOs are visually confusing objects of the same category.
Fig. 6: Success rates of different methods in all real-robot evaluations.
Method
Collect_Can
Push_Box
Lid_Cup
Overall
DP [ 6 ]
15/30
13/30
10/30
38/90
S 2 -Diff. [ 13 ]
22/30
19/30
14/30
55/90
Ours-DP
30/30
27/30
28/30
85/90
ACT [ 5 ]
14/30
18/30
10/30
42/90
Ours-ACT
26/30
28/30
25/30
79/90
TABLE II: Success Rates of DemoFO Manipulation
Method
Collect_Can
Push_Box
Lid_Cup
Overall
Ours-DP
10/10
10/10
8/10
28/30
Ours-ACT
7/10
8/10
8/10
23/30
TABLE III: Success Rates of NewFO Manipulation with Shared Policy
Fig. 7: Visualization of FSAE representations in the left view (a) and the right view (b). In each subfigure, the first row shows the boxes and mask given by FOM-SAM3. The second row visualizes the in-FO appearance features after unified PCA feature dimensionality reduction. The waypoint number on the top-left corner of each picture corresponds to that in the t-SNE map on the far right.
CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan