GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
Authors: Jiaming Li, Zhijia Liang, Weikai Chen, Lin Ma, Guanbin Li
Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Meituan, Shenzhen, China · GuangDong Province Key Laboratory of Information Security Technology, China · Research Institute, Sun Yat-sen University, Shenzhen, China
Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts. While existing open-vocabulary detectors show promise at the base-category level, they underperform in fine-grained settings due to the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings -- leading to over-representation of attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the semantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, HUIDED aligns each subtask with the module best suited for its respective roles. Specifically, given a fine-grained class name, we first use a language model to extract a coarse-grained subject and its descriptive attributes. Then the detector is guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant or overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention-based manner. This mitigates over-representation while preserving discriminative power. Finally, a region-level attribute discrimination module compares each detected region against full fine-grained class names using a refined vision-language model with a projection head for improved alignment. Extensive experiments on FG-OVD and 3F-OVD benchmarks show that GUIDED achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization. Our code will be released at https://github.com/lijm48/GUIDED.
Figures & tables
Figure 1: (a) The t-SNE visualization of CLIP text embeddings on LVIS classes and the fine-grained classes. The figure shows that some CLIP embeddings of fine-grained variants are positioned far apart. (b) The visualization of predictions of an OV detector with different class prompts(A dog vs A dog with a head). The detector focuses on the head instead of the dog, leading to incorrect localization. (c) The mean classification scores and the mean IoU of the prediction box with the ground truth box of OWL-ViT Minderer et al. (2022) under coarse-grained class queries and fine-grained class queries. The detector shows better performance on both classification and localization with coarse-grained queries than with fine-grained queries.
Figure 2: An overview of the proposed GUIDED framework. GUIDED adopts a three-stage pipeline, which consists of subject identification, coarse-grained object detection, and fine-grained attribute discrimination. In subject identification, GUIDED employs an LLM to extract the coarse-grained subject and its attribute embeddings. For detection, coarse-grained subject embeddings are adopted as queries to localize the candidate regions with coarse confidence. An attribute embedding fusion module selectively integrates attribute embeddings into queries. In the discrimination stage, GUIDED estimates the fine-grained score for each detected region with the full fine-grained class names using a refined CLIP with a projection head. The final score is obtained from the weighted multiplication of the detector’s coarse confidence and the attribute similarity scores.
Figure 3: (a) Illustration of the prompt for subject identification. The prompt for extracting the associated attributes is shown in our supplementary document. (b) The architecture of the attribute-fused attention layer.
Detector
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Trans.
Average
OWL-ViT(B/16)
26.2
39.8
38.4
53.9
45.3
37.3
26.6
34.1
37.7
OWLv2(B/16)
25.3
38.5
40.0
52.9
45.1
33.5
19.2
28.5
35.4
OWLv2(L/14)
25.4
41.2
42.8
63.2
53.3
36.9
23.3
12.2
37.3
Detic
11.5
18.6
18.6
69.7
21.5
38.8
30.1
24.6
29.3
ViLD
22.1
36.1
39.9
56.6
43.2
34.9
24.5
30.1
35.9
CORA
13.8
20.0
20.4
35.1
25.0
19.3
22.0
27.9
22.9
Table 1: mAP evaluation results on FG-OVD benchmark (%). The performance in ’Average’ is the average of performance over the 8 sub-datasets. ’Trans.’ denotes the performance on the Transparency subset. ’Finetune’ denotes finetune the LaMI-DETR on FG-OVD training set. Here ∗ denotes the results are from our reproduction.
Table 5
Method
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Transp.
Average
Baseline
29.2
40.6
42.9
63.5
49.5
39.2
34.6
46.2
43.2
+ Train from scratch
31.7
39.6
43.4
42.6
32.4
33.0
30.8
20.4
34.2
+ Finetune
39.5
50.7
54.2
66.0
51.9
53.7
42.1
49.1
50.9
+ GUIDED w/o AEF
53.0
65.2
69.5
72.0
64.1
63.3
49.0
63.4
62.4
+ GUIDED w/o CGOD
48.6
59.7
64.2
64.3
58.9
57.8
45.6
56.6
57.0
+ GUIDED w/o projection
57.5
67.8
70.2
70.8
64.0
66.3
55.4
60.8
64.1
Table 4: The ablation of key factors in the GUIDED framework on the FG-OVD dataset. The results are shown in mAP (%). ’Baseline’ denotes the LaMI-DETR. ’AEF’ denotes the attribute embedding fusion module. ’CGOD’ denotes the coarse-grained object detection.
Table 7Table 8
Method
LLM
Detector
CLIP
Overall
mAP
Baseline
-
198.5
13.2
211.7
43.2
GUIDED with LLaMA-3.1-8B
72.1
198.8
13.3
284.2
65.9
GUIDED with LLaMA-3.3-70B
193.8
198.8
13.3
405.9
66.5
Table 9: The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustration of the LLM prompt for attribute identification.
Method
FG-OVD
LVIS
Average
APr
APc
APf
AP
GUIDED w/o AEF
62.5
43.2
39.3
43.5
41.6
GUIDED w/o Subtract
65.1
43.4
38.4
43.6
41.4
GUIDED
66.4
43.9
39.2
43.7
41.8
Appendix
Table 10: The ablation of attribute embedding fusion on the FG-OVD and the LVIS dataset. The results in the FGOVD dataset are shown in average mAP across the eight FG-OVD subsets. ’AEF’ denoted attribute embedding fusion. ’w/o’ Substract denotes we do not subtract the subject embedding for the key states estimation.
Table 12Table 13
Method
Stage1
Stage2
FG-OVD
LVIS
Average
APr
APc
APf
AP
Baseline
LVIS
-
43.2
43.2
39.3
43.5
41.6
GUIDED
LVIS
-
46.8
43.9
39.2
43.7
41.8
GUIDED
LVIS
FG-OVD
66.4
39.3
32.6
38.7
36.2
GUIDED
LVIS
FG-OVD & LVIS
65.5
44.4
39.4
43.7
41.9
Appendix
Table 15: The comparison on the standard open-vocabulary benchmark LVIS dataset and the FGOVD dataset.
Figure 5: A failure case of GUIDED for our limitations.
Figure 6: Qualitative comparisons against other methods in FG-OVD. Each row juxtaposes four columns: (a) ground-truth boxes, (b) the baseline LaMI-DETR, noted as LaMI (c) LaMI-DETR + HA-FGOVD, and (d) our GUIDED framework. Green text represents positive labels, and red text represents negative labels. Green boxes represent correct classifications, and red boxes represent incorrect classifications.
Beijing University of Posts and Telecommunications · Beijing E-Hualu Information Technology Co., Ltd. · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi, Taiwan. · National Center for High-Performance Computing, Taichung, Taiwan.