GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
Authors: Jiaming Li, Zhijia Liang, Weikai Chen, Lin Ma, Guanbin Li
Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Meituan, Shenzhen, China · GuangDong Province Key Laboratory of Information Security Technology, China · Research Institute, Sun Yat-sen University, Shenzhen, China
Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts. While existing open-vocabulary detectors show promise at the base-category level, they underperform in fine-grained settings due to the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings -- leading to over-representation of attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the semantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, HUIDED aligns each subtask with the module best suited for its respective roles. Specifically, given a fine-grained class name, we first use a language model to extract a coarse-grained subject and its descriptive attributes. Then the detector is guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant or overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention-based manner. This mitigates over-representation while preserving discriminative power. Finally, a region-level attribute discrimination module compares each detected region against full fine-grained class names using a refined vision-language model with a projection head for improved alignment. Extensive experiments on FG-OVD and 3F-OVD benchmarks show that GUIDED achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization. Our code will be released at https://github.com/lijm48/GUIDED.
Figures & tables
Figure 1: (a) The t-SNE visualization of CLIP text embeddings on LVIS classes and the fine-grained classes. The figure shows that some CLIP embeddings of fine-grained variants are positioned far apart. (b) The visualization of predictions of an OV detector with different class prompts(A dog vs A dog with a head). The detector focuses on the head instead of the dog, leading to incorrect localization. (c) The mean classification scores and the mean IoU of the prediction box with the ground truth box of OWL-ViT Minderer et al. (2022) under coarse-grained class queries and fine-grained class queries. The detector shows better performance on both classification and localization with coarse-grained queries than with fine-grained queries.
Figure 2: An overview of the proposed GUIDED framework. GUIDED adopts a three-stage pipeline, which consists of subject identification, coarse-grained object detection, and fine-grained attribute discrimination. In subject identification, GUIDED employs an LLM to extract the coarse-grained subject and its attribute embeddings. For detection, coarse-grained subject embeddings are adopted as queries to localize the candidate regions with coarse confidence. An attribute embedding fusion module selectively integrates attribute embeddings into queries. In the discrimination stage, GUIDED estimates the fine-grained score for each detected region with the full fine-grained class names using a refined CLIP with a projection head. The final score is obtained from the weighted multiplication of the detector’s coarse confidence and the attribute similarity scores.
Figure 3: (a) Illustration of the prompt for subject identification. The prompt for extracting the associated attributes is shown in our supplementary document. (b) The architecture of the attribute-fused attention layer.
Detector
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Trans.
Average
OWL-ViT(B/16)
26.2
39.8
38.4
53.9
45.3
37.3
26.6
34.1
37.7
OWLv2(B/16)
25.3
38.5
40.0
52.9
45.1
33.5
19.2
28.5
35.4
OWLv2(L/14)
25.4
41.2
42.8
63.2
53.3
36.9
23.3
12.2
37.3
Detic
11.5
18.6
18.6
69.7
21.5
38.8
30.1
24.6
29.3
ViLD
22.1
36.1
39.9
56.6
43.2
34.9
24.5
30.1
35.9
CORA
13.8
20.0
20.4
35.1
25.0
19.3
22.0
27.9
22.9
Table 1: mAP evaluation results on FG-OVD benchmark (%). The performance in ’Average’ is the average of performance over the 8 sub-datasets. ’Trans.’ denotes the performance on the Transparency subset. ’Finetune’ denotes finetune the LaMI-DETR on FG-OVD training set. Here ∗ denotes the results are from our reproduction.
Table 5
Method
Hard
Medium
Easy
Trivial
Color
Material
Pattern
Transp.
Average
Baseline
29.2
40.6
42.9
63.5
49.5
39.2
34.6
46.2
43.2
+ Train from scratch
31.7
39.6
43.4
42.6
32.4
33.0
30.8
20.4
34.2
+ Finetune
39.5
50.7
54.2
66.0
51.9
53.7
42.1
49.1
50.9
+ GUIDED w/o AEF
53.0
65.2
69.5
72.0
64.1
63.3
49.0
63.4
62.4
+ GUIDED w/o CGOD
48.6
59.7
64.2
64.3
58.9
57.8
45.6
56.6
57.0
+ GUIDED w/o projection
57.5
67.8
70.2
70.8
64.0
66.3
55.4
60.8
64.1
Table 4: The ablation of key factors in the GUIDED framework on the FG-OVD dataset. The results are shown in mAP (%). ’Baseline’ denotes the LaMI-DETR. ’AEF’ denotes the attribute embedding fusion module. ’CGOD’ denotes the coarse-grained object detection.
Table 7Table 8
Method
LLM
Detector
CLIP
Overall
mAP
Baseline
-
198.5
13.2
211.7
43.2
GUIDED with LLaMA-3.1-8B
72.1
198.8
13.3
284.2
65.9
GUIDED with LLaMA-3.3-70B
193.8
198.8
13.3
405.9
66.5
Table 9: The comparison of inference time(ms) between Baseline(LaMI-DETR) and GUIDED for detecting one class in an image. We also report the average mAP(%) of each method in FGOVD.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustration of the LLM prompt for attribute identification.
Method
FG-OVD
LVIS
Average
APr
APc
APf
AP
GUIDED w/o AEF
62.5
43.2
39.3
43.5
41.6
GUIDED w/o Subtract
65.1
43.4
38.4
43.6
41.4
GUIDED
66.4
43.9
39.2
43.7
41.8
Appendix
Table 10: The ablation of attribute embedding fusion on the FG-OVD and the LVIS dataset. The results in the FGOVD dataset are shown in average mAP across the eight FG-OVD subsets. ’AEF’ denoted attribute embedding fusion. ’w/o’ Substract denotes we do not subtract the subject embedding for the key states estimation.
Table 12Table 13
Method
Stage1
Stage2
FG-OVD
LVIS
Average
APr
APc
APf
AP
Baseline
LVIS
-
43.2
43.2
39.3
43.5
41.6
GUIDED
LVIS
-
46.8
43.9
39.2
43.7
41.8
GUIDED
LVIS
FG-OVD
66.4
39.3
32.6
38.7
36.2
GUIDED
LVIS
FG-OVD & LVIS
65.5
44.4
39.4
43.7
41.9
Appendix
Table 15: The comparison on the standard open-vocabulary benchmark LVIS dataset and the FGOVD dataset.
Figure 5: A failure case of GUIDED for our limitations.
Figure 6: Qualitative comparisons against other methods in FG-OVD. Each row juxtaposes four columns: (a) ground-truth boxes, (b) the baseline LaMI-DETR, noted as LaMI (c) LaMI-DETR + HA-FGOVD, and (d) our GUIDED framework. Green text represents positive labels, and red text represents negative labels. Green boxes represent correct classifications, and red boxes represent incorrect classifications.
Open-vocabulary detectors are proposed to locate and recognize objects in novel classes. However, variations in vision-aware language vocabulary data used for open-vocabulary learning can lead to unfair and unreliable evaluations. Recent evaluation methods have attempted to address this issue by incorporating object properties or adding locations and characteristics to the captions. Nevertheless, since these properties and locations depend on the specific details of the images instead of classes, detectors can not make accurate predictions without precise descriptions provided through human annotation. This paper introduces 3F-OVD, a novel task that extends supervised fine-grained object detection to the open-vocabulary setting. Our task is intuitive and challenging, requiring a deep understanding of Fine-grained captions and careful attention to Fine-grained details in images in order to accurately detect Fine-grained objects. Additionally, due to the scarcity of qualified fine-grained object detection datasets, we have created a new dataset, NEU-171K, tailored for both supervised and open-vocabulary settings. We benchmark state-of-the-art object detectors on our dataset for both settings. Furthermore, we propose a simple yet effective post-processing technique. Our data, annotations and codes are available at https://github.com/tengerye/3FOVD.
Ying Liu, Yijing Hua, Haojiang Chai +2
department of software engineering, Northeastern University, China. · His work was done in Articul8 AI.
Open-Vocabulary Object Detection (OVD) models break the limitations of closed-set detection, enabling the identification of unseen categories through natural language prompts. However, they exhibit notable limitations in fine-grained detection tasks involving attributes like color, material, and texture. We attribute this performance bottleneck in OVD models to a core issue: when category signals dominate, OVD models tend to marginalize attribute information during inference. This leads to incorrect binding between attributes and target objects. To address this, we propose the Dual-Stage Attribute Activation (DSAA) framework, which enhances fine-grained detection capabilities by strengthening attribute semantics at two critical stages. In the text embedding stage, we employ Attribute Prefix Adapter (APA) module to generate attribute prefixes that inject explicit attribute priors. To further amplify the influence of these attributes, our Key/Value (K/V) Modulator module then intervenes during the BERT encoding phase, selectively enhancing the Key and Value vectors of the corresponding attribute tokens. In addition, we introduce an attribute-aware contrastive loss to improve discrimination among same-category instances with different attributes during training. Experimental results on the FG-OVD benchmark demonstrate the effectiveness of our method across various mainstream open-vocabulary models.
Donghong Jiang, Endian Lin, Hanqing Liu +4
Beijing University of Posts and Telecommunications · Beijing E-Hualu Information Technology Co., Ltd. · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
Grounding-based open-vocabulary detectors provide a widely used framework for text-conditioned localization, with a phrase-grounding formulation that aligns object queries to individual text tokens and recovers class scores through token-to-class aggregation. We observe that this token-level scoring design becomes a bottleneck for fine-grained open-vocabulary object detection, where the target category is defined by a complete compositional description rather than by isolated tokens. Based on this observation, we propose SL-Ground, a sentence-level grounding reformulation for grounding-based open-vocabulary detection. Instead of scoring queries against fragmented token evidence, SL-Ground encodes each category description as a complete sentence-level representation and directly scores object queries against these representations. To evaluate fine-grained recognition across label granularity, we construct a multi-granularity Stanford Cars benchmark spanning vehicle type, brand, and car model. At the finest car-model granularity, SL-Ground raises AP over the token-level variant from 14.5 to 70.6 under matched training candidate sampling and to 86.0 with full-vocabulary training, while better preserving open-vocabulary capability and avoiding repeated prompt chunking. Beyond Stanford Cars, SL-Ground also improves FG-OVD performance, achieving the best average AP among compared methods. These results show that sentence-level grounding provides a more suitable scoring unit for fine-grained OVOD and offers a practical path toward fine-grained grounding-based open-vocabulary detectors.
Yang-Syu Li, Cheng-Huan Yang, Chia-Yung Jui +1
Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi, Taiwan. · National Center for High-Performance Computing, Taichung, Taiwan.