HDINO: A Concise and Efficient Open-Vocabulary Detector

Hao Zhang; Yiqun Wang; Qinran Lin; Runze Fan; Yong Li

cs.CVMar 3, 2026

HDINO: A Concise and Efficient Open-Vocabulary Detector

Authors: Hao Zhang, Yiqun Wang, Qinran Lin, Runze Fan, Yong Li

Organizations: Chongqing University

Abstract

Despite the growing interest in open-vocabulary object detection in recent years, most existing methods rely heavily on manually curated fine-grained training datasets as well as resource-intensive layer-wise cross-modal feature extraction. In this paper, we propose HDINO, a concise yet efficient open-vocabulary object detector that eliminates the dependence on these components. Specifically, we propose a two-stage training strategy built upon the transformer-based DINO model. In the first stage, noisy samples are treated as additional positive object instances to construct a One-to-Many Semantic Alignment Mechanism(O2M) between the visual and textual modalities, thereby facilitating semantic alignment. A Difficulty Weighted Classification Loss (DWCL) is also designed based on initial detection difficulty to mine hard examples and further improve model performance. In the second stage, a lightweight feature fusion module is applied to the aligned representations to enhance sensitivity to linguistic semantics. Under the Swin Transformer-T setting, HDINO-T achieves 49.2 mAP on COCO using 2.2M training images from two publicly available detection datasets, without any manual data curation and the use of grounding data, surpassing Grounding DINO-T and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively, which are trained on 5.4M and 6.5M images. After fine-tuning on COCO, HDINO-T and HDINO-L further achieve 56.4 mAP and 59.2 mAP, highlighting the effectiveness and scalability of our approach. Code and models are available at https://github.com/HaoZ416/HDINO.

Figures & tables

Figure 1: Comparison of different semantic alignment paradigms. Classifier-based methods align textual features only with their corresponding objects, and fusion-based methods perform alignment on global features; both exhibit limited semantic alignment capability, whereas the proposed HDINO leverages positive noisy samples to facilitate visual–textual alignment.

Figure 2: Overview of the HDINO model. Components in light blue are inherited from the original DINO model, modules in light red are newly introduced in HDINO, and components in light green are shared by both. The Feat Map is a linear layer that projects textual features into a unified embedding dimension, and we adopt the Language-Guided Query Selection strategy proposed in Grounding DINO to select initial anchors and remove the auxiliary queries during inference.

Figure 3: Visualization of positive noisy sample generation. Red boxes indicate ground-truth annotations, and blue boxes denote generated noisy samples obtained by constraining the perturbation of bounding box coordinates.

Model	Backbone	Training Data	Zero-Shot 2017val
DyHead-T Dai et al. (2021)	Swin-T	O365	43.6
GLIP-T (B)	Swin-T	O365	44.9
GLIP-L	Swin-L	FourODs,GoldG,Cap24M	49.8
Grounding-DINO-T	Swin-T	O365,GoldG,Cap4M	48.4
Grounding-DINO-L	Swin-L	O365,OpenImages,GoldG	52.5
T-Rex2-T	Swin-T	O365, GoldG, OpenImages, Bamboo, CC3M, LAION	46.4

Table 1: Zero-shot object detection performance comparison on COCO. All methods are evaluated on the COCO val2017 split under the zero-shot setting. For fair comparison, we report the standard mean Average Precision (mAP) metric. HDINO is trained using only two publicly available detection datasets.

Model	O2M	DWCL	Fusion	AP
DINO+CLIP	✗	✗	✗	46.3
DINO+CLIP	✓	✗	✗	48.3
DINO+CLIP	✓	✓	✗	48.8
DINO+CLIP	✓	✓	✓	49.2

Table 2: Ablation study of (i) the one-to-many semantic alignment mechanism(O2M), (ii) DWCL, and (iii) feature fusion(Fusion), as well as their full combination.

Loss	$\beta_{1}$	$\beta_{2}$	AP
Focal Loss	✗	✗	48.3
DWCL	1	1.5	48.6
DWCL	1	2	48.8
DWCL	1	2.5	48.7
DWCL	2	1	48.5
DWCL	2	1.5	48.6

Table 3: Impact of different

\beta_{1}

and

\beta_{2}

hyperparameters in DWCL. All models are trained with identical pre-training configurations and evaluated on COCO for performance comparison.

Figure 4: Comparison between the standard focal loss and the proposed Difficulty Weighted Classification Loss (DWCL) for positive samples. The loss is plotted as a function of the predicted probability $p$ . For focal loss, $\alpha=0.25$ and $\gamma=2$ are used. For DWCL, the detection difficulty is fixed to $\mathrm{IoU}=0.5$ with $\beta_{1}=1$ and $\beta_{2}=2$ , and the weighting factor $\alpha_{\mathrm{dwcl}}$ is normalized by $\mathbb{E}[1-\mathrm{IoU}]=0.25$ .

Figure 5: 3D visualization of the proposed Difficulty Weighted Classification Loss (DWCL) for positive samples. In this visualization, hyperparameters are fixed to $\beta_{1}=1$ and $\beta_{2}=2$ , and $\alpha_{\mathrm{dwcl}}$ is normalized by $\mathbb{E}[1-\mathrm{IoU}]=0.25$ .

Method	Epochs	AP b	AP ${}^{b_{50}}$	AP ${}^{b_{75}}$
Linear Probing
YOLOE-v8-S	10	35.6	51.5	38.9
YOLOE-v8-M	10	42.2	59.2	46.3
YOLOE-v8-L	10	45.4	63.3	50.0
YOLOE-v11-S	10	37.0	52.9	40.4
YOLOE-v11-M	10	43.1	60.6	47.4

Table 4: Fine-tuning results on COCO.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 87 papers

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
Jun 10, 2026Hao Zhang, Qinran Lin, Linqi Song +1Open-Vocabulary Object Detection Detection Transformer
Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5
Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection
Apr 6, 2026Weihao Cao, Runqi Wang, Xiaoyue Duan +3Open-Vocabulary Object Detection Vision Foundation Model Adaptation
State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University +1

Jun 10, 2026J/K move · Enter open · S save

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

Hao Zhang, Qinran Lin, Linqi Song, Yong Li

Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China

Open-Vocabulary Object Detection Detection Transformer Vision-Language Alignment

Open paper AskPDF arXiv

Vision-language models like CLIP can provide rich semantic priors for open-vocabulary object detection. However, jointly integrating both textual and visual knowledge into detection architectures remains challenging. In this paper, we propose VL-DINO, an open-vocabulary detector that enhances DINO through more effective exploitation of CLIP's vision-language knowledge. Specifically, a Query-guided Positive Sample Construction (QPSC) module is first developed to construct additional high-quality positive samples, enabling the vanilla DINO framework to better accommodate mixed training across heterogeneous data sources while providing more vision-language alignment signals, thereby incorporating richer textual knowledge during training. A Visual Semantic Encoder (VSE) module is then introduced to distill CLIP visual knowledge into backbone-extracted features, producing fused features for subsequent encoder refinement. Based on the fused features, an Object-Region Semantic Alignment (ORSA) module extracts object-centric region features and aligns them with the corresponding textual embeddings, further incorporating textual cues. In the zero-shot setting, VL-DINO-T and VL-DINO-L achieve 36.3 and 38.1 AP on the LVIS benchmark, respectively, consistently outperforming prior advanced approaches. Extensive experiments demonstrate the effectiveness and competitive performance of the proposed design.

Abstract

Open-Vocabulary Object Detection

Explore similar work

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection