SL-Ground: Sentence-Level Grounding for Fine-Grained Open-Vocabulary Object Detection
Authors: Yang-Syu Li, Cheng-Huan Yang, Chia-Yung Jui, Peggy Joy Lu
Organizations: Department of Computer Science and Information Engineering, National Chung Cheng University, Chiayi, Taiwan. · National Center for High-Performance Computing, Taichung, Taiwan.
Grounding-based open-vocabulary detectors provide a widely used framework for text-conditioned localization, with a phrase-grounding formulation that aligns object queries to individual text tokens and recovers class scores through token-to-class aggregation. We observe that this token-level scoring design becomes a bottleneck for fine-grained open-vocabulary object detection, where the target category is defined by a complete compositional description rather than by isolated tokens. Based on this observation, we propose SL-Ground, a sentence-level grounding reformulation for grounding-based open-vocabulary detection. Instead of scoring queries against fragmented token evidence, SL-Ground encodes each category description as a complete sentence-level representation and directly scores object queries against these representations. To evaluate fine-grained recognition across label granularity, we construct a multi-granularity Stanford Cars benchmark spanning vehicle type, brand, and car model. At the finest car-model granularity, SL-Ground raises AP over the token-level variant from 14.5 to 70.6 under matched training candidate sampling and to 86.0 with full-vocabulary training, while better preserving open-vocabulary capability and avoiding repeated prompt chunking. Beyond Stanford Cars, SL-Ground also improves FG-OVD performance, achieving the best average AP among compared methods. These results show that sentence-level grounding provides a more suitable scoring unit for fine-grained OVOD and offers a practical path toward fine-grained grounding-based open-vocabulary detectors.
Figures & tables
Fig. 1: Example of compositional fine-grained labels and token sharing in vehicle categories.
Fig. 2: Overview of SL-Ground. Each class description is independently encoded into one CLIP sentence feature. The feature enhancer fuses visual and class features, class-level proposal scores initialize the decoder, and the prediction outputs boxes and query-to-class logits.
Detector setting
Label granularity (AP ↑ )
Model
Text encoder
Grounding unit
Type
Brand
Car model
GD-BERT
BERT
Token
70.3
40.1
10.3
GD-CLIP
CLIP
Token
80.5
73.3
14.5
SL-Ground
CLIP
Sentence
88.0
92.8
86.0
TABLE I: Stanford Cars detection results across label granularities.
Fig. 3: Epoch-wise Stanford Cars AP over 20 fine-tuning epochs. SL-Ground stays above the token-level variants throughout, with the largest gap at car model granularity.
Fine-tuning setting
Stanford Cars AP ↑
OVOD AP ↑
Fine-tune target
Model
Type
Brand
Car model
COCO
miniLVIS
Zero-shot
GD-BERT
9.6
1.0
0.3
48.4
15.78
GD-CLIP
1.6
0.6
0.1
45.6
26.37
SL-Ground
1.4
0.6
0.9
48.8
25.27
Type
GD-BERT
70.3
3.0
2.6
15.8
6.01
GD-CLIP
80.5
3.6
5.1
21.2
11.36
TABLE II: Cross-granularity transfer and OVOD knowledge preservation after Stanford Cars fine-tuning. Rows indicate the fine-tuning target, and COCO/miniLVIS AP measure retained open-vocabulary performance.
TABLE VI: Fine-grained recognition results on Dog-120, CUB-200, and Car-196. Values are accuracy (%), not detection AP.
Model
Training setting
Ktrain
AP ↑
GD-CLIP
75-token sampling
11.87 ∗
14.5
SL-Ground
75-token sampling
11.87 ∗
70.6
SL-Ground
Fixed size
12
65.5
SL-Ground
Fixed size
32
76.7
SL-Ground
Fixed size
64
83.0
SL-Ground
Full vocabulary
189
86.0
TABLE VII: Effect of training candidate-set size on Stanford Cars car-model detection. Evaluation uses all 189 categories.
Model
Params (M)
COCO
miniLVIS
AP ↑
FPS ↑
Mem. ↓
AP ↑
FPS ↑
Mem. ↓
GD-BERT
173.0
48.4
12.38
1.37
15.78
0.58
1.34
GD-CLIP
187.1
45.6
6.28
1.27
26.37
0.26
1.25
SL-Ground
182.4
48.8
16.01
1.25
25.27
12.28
1.40
TABLE VIII: Accuracy–efficiency comparison on COCO and miniLVIS. Mem. denotes peak allocated GPU memory (GB).
Fig. 4: Stanford Cars margin CDF and vocabulary-scaling latency. Chunk labels denote the number of chunks in the tokenized prompt.
Fig. 5: Stage-level latency on Stanford Cars type labels, where all variants use one prompt chunk. Panel (a) shows stage composition, and panel (b) shows module-level latency savings relative to GD-BERT.
Fig. 6: Controlled memory–latency trade-off with synthetic single-token labels. Token-level variants cap per-pass CUDA memory after chunking but incur repeated detector passes, whereas SL-Ground keeps one pass with lower latency and class-count-dependent memory growth.
Fig. 7: Qualitative comparison on COCO. Grounding DINO is shown in the top row and SL-Ground in the bottom row. SL-Ground reduces class confusion in examples such as skateboard, bird, and horse while preserving comparable box locations.
Fig. 8: Qualitative comparisons on miniLVIS with 1,203 candidate categories. Grounding DINO is shown in the top row and SL-Ground in the bottom row. Grounding DINO shows overlapping predictions with different class labels after outputs from multiple prompt chunks are concatenated without cross-chunk suppression, while SL-Ground produces a less dense set of predictions in a single pass.
Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts. While existing open-vocabulary detectors show promise at the base-category level, they underperform in fine-grained settings due to the semantic entanglement of subjects and attributes in pretrained vision-language model (VLM) embeddings -- leading to over-representation of attributes, mislocalization, and semantic drift in embedding space. We propose GUIDED, a decomposition framework specifically designed to address the semantic entanglement between subjects and attributes in fine-grained prompts. By separating object localization and fine-grained recognition into distinct pathways, HUIDED aligns each subtask with the module best suited for its respective roles. Specifically, given a fine-grained class name, we first use a language model to extract a coarse-grained subject and its descriptive attributes. Then the detector is guided solely by the subject embedding, ensuring stable localization unaffected by irrelevant or overrepresented attributes. To selectively retain helpful attributes, we introduce an attribute embedding fusion module that incorporates attribute information into detection queries in an attention-based manner. This mitigates over-representation while preserving discriminative power. Finally, a region-level attribute discrimination module compares each detected region against full fine-grained class names using a refined vision-language model with a projection head for improved alignment. Extensive experiments on FG-OVD and 3F-OVD benchmarks show that GUIDED achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization. Our code will be released at https://github.com/lijm48/GUIDED.
Jiaming Li, Zhijia Liang, Weikai Chen +2
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Meituan, Shenzhen, China · GuangDong Province Key Laboratory of Information Security Technology, China +1
Open-vocabulary detectors are proposed to locate and recognize objects in novel classes. However, variations in vision-aware language vocabulary data used for open-vocabulary learning can lead to unfair and unreliable evaluations. Recent evaluation methods have attempted to address this issue by incorporating object properties or adding locations and characteristics to the captions. Nevertheless, since these properties and locations depend on the specific details of the images instead of classes, detectors can not make accurate predictions without precise descriptions provided through human annotation. This paper introduces 3F-OVD, a novel task that extends supervised fine-grained object detection to the open-vocabulary setting. Our task is intuitive and challenging, requiring a deep understanding of Fine-grained captions and careful attention to Fine-grained details in images in order to accurately detect Fine-grained objects. Additionally, due to the scarcity of qualified fine-grained object detection datasets, we have created a new dataset, NEU-171K, tailored for both supervised and open-vocabulary settings. We benchmark state-of-the-art object detectors on our dataset for both settings. Furthermore, we propose a simple yet effective post-processing technique. Our data, annotations and codes are available at https://github.com/tengerye/3FOVD.
Ying Liu, Yijing Hua, Haojiang Chai +2
department of software engineering, Northeastern University, China. · His work was done in Articul8 AI.
Vision-language models enable open-vocabulary object grounding through natural language queries, under the implicit assumption that semantically equivalent descriptions yield consistent outputs. We examine this assumption using a controlled pipeline combining DETR for object proposals with CLIP for language-conditioned selection on 263 COCO val2017 images. We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances, with mean instability of 2.11 distinct selections across six prompts. PCA analysis shows this variability is structured and directional, not random. Prompt ensembling does not improve quality and often shifts selections toward generic regions. We further show that text embedding proximity explains only 34% of grounding disagreement (r = -0.58), confirming that instability arises from the argmax selection mechanism rather than text-level distances alone.
Dawar Jyoti Deka, Amit Sethi, Syed Mohammad Ali
Department of Aerospace Engineering, Indian Institute of Technology Bombay, Mumbai, India · Department of Electrical Engineering, Indian Institute of Technology Bombay, Mumbai, India