cs.CVMar 3, 2026

HDINO: A Concise and Efficient Open-Vocabulary Detector

Authors: Hao Zhang, Yiqun Wang, Qinran Lin, Runze Fan, Yong Li

Organizations: Chongqing University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite the growing interest in open-vocabulary object detection in recent years, most existing methods rely heavily on manually curated fine-grained training datasets as well as resource-intensive layer-wise cross-modal feature extraction. In this paper, we propose HDINO, a concise yet efficient open-vocabulary object detector that eliminates the dependence on these components. Specifically, we propose a two-stage training strategy built upon the transformer-based DINO model. In the first stage, noisy samples are treated as additional positive object instances to construct a One-to-Many Semantic Alignment Mechanism(O2M) between the visual and textual modalities, thereby facilitating semantic alignment. A Difficulty Weighted Classification Loss (DWCL) is also designed based on initial detection difficulty to mine hard examples and further improve model performance. In the second stage, a lightweight feature fusion module is applied to the aligned representations to enhance sensitivity to linguistic semantics. Under the Swin Transformer-T setting, HDINO-T achieves 49.2 mAP on COCO using 2.2M training images from two publicly available detection datasets, without any manual data curation and the use of grounding data, surpassing Grounding DINO-T and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively, which are trained on 5.4M and 6.5M images. After fine-tuning on COCO, HDINO-T and HDINO-L further achieve 56.4 mAP and 59.2 mAP, highlighting the effectiveness and scalability of our approach. Code and models are available at https://github.com/HaoZ416/HDINO.

Figures & tables

Figure 1: Comparison of different semantic alignment paradigms. Classifier-based methods align textual features only with their corresponding objects, and fusion-based methods perform alignment on global features; both exhibit limited semantic alignment capability, whereas the proposed HDINO leverages positive noisy samples to facilitate visual–textual alignment.
Figure 2: Overview of the HDINO model. Components in light blue are inherited from the original DINO model, modules in light red are newly introduced in HDINO, and components in light green are shared by both. The Feat Map is a linear layer that projects textual features into a unified embedding dimension, and we adopt the Language-Guided Query Selection strategy proposed in Grounding DINO to select initial anchors and remove the auxiliary queries during inference.
Figure 3: Visualization of positive noisy sample generation. Red boxes indicate ground-truth annotations, and blue boxes denote generated noisy samples obtained by constraining the perturbation of bounding box coordinates.
ModelBackboneTraining DataZero-Shot 2017val
DyHead-T Dai et al. (2021)Swin-TO36543.6
GLIP-T (B)Swin-TO36544.9
GLIP-LSwin-LFourODs,GoldG,Cap24M49.8
Grounding-DINO-TSwin-TO365,GoldG,Cap4M48.4
Grounding-DINO-LSwin-LO365,OpenImages,GoldG52.5
T-Rex2-TSwin-TO365, GoldG, OpenImages, Bamboo, CC3M, LAION46.4
Table 1: Zero-shot object detection performance comparison on COCO. All methods are evaluated on the COCO val2017 split under the zero-shot setting. For fair comparison, we report the standard mean Average Precision (mAP) metric. HDINO is trained using only two publicly available detection datasets.
ModelO2MDWCLFusionAP
DINO+CLIP✗✗✗46.3
DINO+CLIP✓✗✗48.3
DINO+CLIP✓✓✗48.8
DINO+CLIP✓✓✓49.2
Table 2: Ablation study of (i) the one-to-many semantic alignment mechanism(O2M), (ii) DWCL, and (iii) feature fusion(Fusion), as well as their full combination.
Lossβ1\beta_{1}β2\beta_{2}AP
Focal Loss✗✗48.3
DWCL11.548.6
DWCL1248.8
DWCL12.548.7
DWCL2148.5
DWCL21.548.6
Table 3: Impact of different β1\beta_{1} and β2\beta_{2} hyperparameters in DWCL. All models are trained with identical pre-training configurations and evaluated on COCO for performance comparison.
Figure 4: Comparison between the standard focal loss and the proposed Difficulty Weighted Classification Loss (DWCL) for positive samples. The loss is plotted as a function of the predicted probability pp . For focal loss, α=0.25\alpha=0.25 and γ=2\gamma=2 are used. For DWCL, the detection difficulty is fixed to IoU=0.5\mathrm{IoU}=0.5 with β1=1\beta_{1}=1 and β2=2\beta_{2}=2 , and the weighting factor αdwcl\alpha_{\mathrm{dwcl}} is normalized by E[1−IoU]=0.25\mathbb{E}[1-\mathrm{IoU}]=0.25 .
Figure 5: 3D visualization of the proposed Difficulty Weighted Classification Loss (DWCL) for positive samples. In this visualization, hyperparameters are fixed to β1=1\beta_{1}=1 and β2=2\beta_{2}=2 , and αdwcl\alpha_{\mathrm{dwcl}} is normalized by E[1−IoU]=0.25\mathbb{E}[1-\mathrm{IoU}]=0.25 .
MethodEpochsAP bAP b50{}^{b_{50}}AP b75{}^{b_{75}}
Linear Probing 
YOLOE-v8-S1035.651.538.9
YOLOE-v8-M1042.259.246.3
YOLOE-v8-L1045.463.350.0
YOLOE-v11-S1037.052.940.4
YOLOE-v11-M1043.160.647.4
Table 4: Fine-tuning results on COCO.
More in this topic

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 87 papers

Explore similar work

CardsList
Jun 10, 2026

VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio

Vision-language models like CLIP can provide rich semantic priors for open-vocabulary object detection. However, jointly integrating both textual and visual knowledge into detection architectures remains challenging. In this paper, we propose VL-DINO, an open-vocabulary detector that enhances DINO through more effective exploitation of CLIP's vision-language knowledge. Specifically, a Query-guided Positive Sample Construction (QPSC) module is first developed to construct additional high-quality positive samples, enabling the vanilla DINO framework to better accommodate mixed training across heterogeneous data sources while providing more vision-language alignment signals, thereby incorporating richer textual knowledge during training. A Visual Semantic Encoder (VSE) module is then introduced to distill CLIP visual knowledge into backbone-extracted features, producing fused features for subsequent encoder refinement. Based on the fused features, an Object-Region Semantic Alignment (ORSA) module extracts object-centric region features and aligns them with the corresponding textual embeddings, further incorporating textual cues. In the zero-shot setting, VL-DINO-T and VL-DINO-L achieve 36.3 and 38.1 AP on the LVIS benchmark, respectively, consistently outperforming prior advanced approaches. Extensive experiments demonstrate the effectiveness and competitive performance of the proposed design.

Hao Zhang, Qinran Lin, Linqi Song +1

Chongqing University, Chongqing, China · City University of Hong Kong, Hong Kong, China

Open-Vocabulary Object DetectionDetection TransformerVision-Language Alignment
AskPDF
Apr 3, 2026

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.

Siheng Wang, Yanshu Li, Bohan Hu +12

Jiangsu University · MBZUAI · Brown University +5

Open-Vocabulary Object DetectionDetection TransformerEfficient Object Detection
AskPDF
Apr 6, 2026

Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection

Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.

Weihao Cao, Runqi Wang, Xiaoyue Duan +3

State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University +1

Open-Vocabulary Object DetectionVision Foundation Model AdaptationPrompt TuningParameter-Efficient Fine-Tuning
AskPDF