Detecting and segmenting novel object instances in open-world environments is a fundamental problem in robotic perception. Given only a small set of template images, a robot must locate and segment a specific object instance in a cluttered, previously unseen scene. Existing proposal-based approaches are highly sensitive to proposal quality and often fail under occlusion and background clutter. We propose L2G-Det, a local-to-global instance detection framework that bypasses explicit object proposals by leveraging dense patch-level matching between templates and the query image. Locally matched patches generate candidate points, which are refined through a candidate selection module to suppress false positives. The filtered points are then used to prompt an augmented Segment Anything Model (SAM) with instance-specific object tokens, enabling reliable reconstruction of complete instance masks. Experiments demonstrate improved performance over proposal-based methods in challenging open-world settings.
Figures & tables
Fig. 2 : Overview of our L2G-Det framework for novel instance detection. It consists of a candidate selection module and an augmented SAM module (SAM ∗ ). Only the adapters and object-tokens are learnable, while all other components are frozen.
Fig. 3 : Comparison between the original SAM and our augmented SAM ∗ under candidate points. With all parameters frozen, SAM tends to produce incomplete masks focused on local regions around the prompts. In contrast, SAM ∗ incorporates a learnable instance-specific object token, which guides the decoder to produce more complete object masks improving detection performance.
Method
AP
AP 50
AP 75
avg
hard
easy
small
medium
large
FasterRCNN [ 38 ]
19.5
10.3
23.8
5.0
22.2
38.0
29.2
23.3
RetinaNet [ 24 ]
22.2
14.9
26.5
5.5
25.8
42.7
31.2
25.0
CenterNet [ 50 ]
21.1
11.8
25.7
5.9
24.1
40.4
32.7
23.6
FCOS [ 45 ]
22.4
13.2
28.7
6.2
26.5
38.1
32.8
25.5
DINO [ 49 ]
28.0
17.9
32.6
11.5
31.6
48.3
39.6
32.2
TABLE I : Comparison of detection performance on the HR-InsDet dataset [ 40 ] .
Method
Proposal
AP
AP 50
AP 75
OS2D [ 34 ]
N/A
2.9
6.5
2.0
DTOID [ 29 ]
N/A
3.6
9.0
2.0
OLN Corr. [ 17 ]
OLN
14.4
18.1
15.7
VoxDet [ 22 ]
OLN
18.7
23.6
20.5
OTS-FM [ 41 , 40 ]
GD
56.7
64.8
59.0
IDOW [ 41 ]
GD
59.4
67.8
61.8
TABLE II : Comparison of detection performance on RoboTools dataset [ 22 ] . The Proposal column indicates the proposal generation strategy. Specifically, OLN denotes Object Localization Network [ 17 ] , GD denotes GroundingDINO [ 25 ] .
Fig. 4 : Qualitative results on RoboTools benchmark. From left to right, we show the ground-truth annotations, results produced by NIDS-Net [ 28 ] , and results of our method L2G-Det .
Method
Adapter
Augmented SAM
AP
RoboTools Dataset
w/o Adapter + SAM
✗
✗
64.8
Adapter + SAM
✓
✗
67.5
w/o Adapter + SAM ∗
✗
✓
69
Adapter + SAM ∗
✓
✓
71.9
High-resolution Dataset
TABLE III : Ablation study on the effects of the Adapter in Candidate Selector and Augmented SAM on RoboTools [ 22 ] and High-resolution datasets [ 40 ] .
Figure 7
Method
Backbone
AP
AP 50
AP 75
IDOW [ 41 ]
DINOv2-Small
51.9
63.8
56.5
NIDS-Net [ 28 ]
DINOv2-Large
64.9
79.4
70.8
L2G-Det (Ours)
DINOv2-Small
69.2
81.1
74.3
L2G-Det (Ours)
DINOv2-Base
70.8
82.9
76.1
TABLE VII : Effect of backbone choice on RoboTools benchmark with the adapter and augmented SAM.
Fig. 6 : Overview of the robot experiment process showing the waypoint tracking and target object identification.
Method
NIoU>0.5
NIoU>0.75
NIoU>0.95
NSuccess
L2G-Det with SAM
7
7
7
8
L2G-Det with SAM ∗
8
8
7
8
TABLE VIII : Real-world robotic detection results on 8 objects. NIoU>τ denotes the number of detections whose predicted masks achieve IoU >τ (threshold) w.r.t ground-truth annotations. NSuccess denotes the number of trials in which the robot successfully detects the target object and stops. There are 8 trials in total.
Training-related Cost
Runtime
GPU Memory
Create synthetic images (per object)
3 min
∅
Train object token (per object)
25 min
2400 MiB
Train Adapter (all objects)
2 h
6900 MiB
Inference Cost ( K=4 )
Runtime
GPU Memory
RoboTools (1920 × 1080)
1.10 s
8300 MiB
HR-InsDet (8192 × 6144)
4.10 s
20000 MiB
TABLE IX : Summary of training-related and inference cost. The inference runtime is reported as the end-to-end per-object cost with K=4 template images.
K
Runtime
GPU Memory
4
1.1 s
8300 MiB
8
1.6 s
9000 MiB
12
1.9 s
9800 MiB
16
2.1 s
10500 MiB
TABLE X : Inference runtime and GPU memory with different numbers of template images K on RoboTools.
Method
AP
AP 50
AP 75
AVG_TEMP
61.4
74.2
66.6
Current_TEMP (Ours)
64.8
77.6
70.2
TABLE XI : Ablation study on template-specific similarity computation. Current_TEMP (Ours) computes similarity using the corresponding template embedding for each candidate, while AVG_TEMP uses the averaged embedding over all templates.
Fig. 7 : Qualitative results on HR-InsDet [ 40 ] benchmark. From left to right, we show the ground-truth annotations, results produced by NIDS-Net [ 28 ] , and results of our method L2G-Det .
Fig. 8 : Examples of template-based synthetic training images. Three synthesized scenes are illustrated: 1) Single-object composition, 2) Multi-object composition without overlap, and 3) Multi-object composition with overlap, where the target object may be partially occluded by surrounding objects.
Fig. 9 : Qualitative real-world detection results on 8 target objects with and without instance-specific object tokens.