Organizations: State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University · WeChat AI, China
Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.
Figures & tables
Figure 1: Pre-trained OVOD models perform well on general domain ( e.g. , OV-COCO [ 16 ] ) but fail to generalize to vertical domain ( e.g. , ArTaxOr [ 6 ] , DIOR [ 13 ] , UODD [ 10 ] ) in the zero-shot setting. Although fine-tuning improves performance on vertical domain, it causes a significant degradation on general domain.
Figure 2: Method motivation. (a) Previous methods use predefined templates or learnable vectors prepended to the category label embeddings, ignoring detailed semantics from image features. Our multi-scale prompt bank uses hierarchical semantics from multi-scale feature pyramid to select auxiliary prompts for the category labels. (b) The dynamic routing method [ 32 ] uses the reconstruction error of an input, obtained from multiple autoencoders, as an indicator of its domain. However, the large overlap in reconstruction errors across different domains can confuse the model, leading to incorrect parameter selection. Our method explicitly models both the content and the domain of the inputs, effectively reducing this overlap and enabling more accurate routing decisions.
Figure 3: Overview of the proposed HSA-DINO framework. We incorporate LoRA into the image encoder during training on downstream datasets. We also introduce a multi-scale prompt bank to further enhance the model’s adaptability to downstream tasks. In the test stage, we propose a semantic-aware router that can accurately identify different data distributions, enabling more precise selection between pre-trained semantics and augmented semantics for detection.
Method
ArTaxOr
DIOR
UODD
mAPtgt
mAPcoco
H
mAPtgt
mAPcoco
H
mAPtgt
mAPcoco
H
ZS
GLIP-L [ 14 ]
12.6
51.2
20.2
4.8
51.2
8.8
4.6
51.2
8.4
GroundingDINO-T [ 17 ]
4.6
48.4
8.4
2.9
48.4
5.5
6.3
48.4
11.1
YOLO-World-X [ 1 ]
2.3
46.6
4.4
0.1
46.6
0.2
0.1
46.6
0.2
OV-DINO [ 28 ]
1.4
50.6
2.7
3.0
50.6
5.7
2.9
50.6
5.5
FFT
GLIP-L [ 14 ]
73.1
7.6
13.8
67.5
15.7
25.5
36.6
27.8
31.6
Table 1: Comparison of different methods across various downstream tasks. “ZS” denotes the zero-shot performance of the pre-trained models, while “FFT” and “PEFT” denote the models’ performance after full fine-tuning or parameter-efficient fine-tuning on downstream tasks, respectively. mAPtgt denotes the performance of models on corresponding target downstream datasets ( i.e. , ArTaxOr, DIOR, UODD); mAPcoco denotes the performance of models on the OV-COCO. H denotes the harmonic mean value defined in Eq. 10 .
Method
OV-COCO +
w ArTaxOr
w DIOR
w UODD
ZS
GLIP-L [ 14 ]
48.0
33.2
49.0
GroundingDINO-T [ 17 ]
43.7
36.9
46.8
YOLO-World-X [ 1 ]
42.9
36.3
44.9
OV-DINO [ 28 ]
44.5
37.3
48.7
FFT
GLIP-L [ 14 ]
11.7
13.2
24.5
Table 2: Comparison of mAP across different methods on OV-COCO + . “w” indicates the additional dataset included for joint evaluation ( e.g. , “w ArTaxOr” means OV-COCO evaluated together with ArTaxOr).
Figure 4: Ablation study on (a) bank size N , (b) prompt length M , (c) key matching loss weight λm , and (d) orthogonal loss weight λp .
Method
HArTaxOr
HDIOR
HUODD
Hmean
Predefined + SAR
54.6
47.3
47.7
49.9
CoOp [ 38 ] + SAR
57.1
51.1
48.0
52.1
AttriCLIP [ 29 ] + SAR
58.8
51.6
48.5
53.0
MSPB + SAR (ours)
60.5
53.0
49.6
54.4
Table 3: Comparison of different textual semantic augmentation.
Method
HArTaxOr
HDIOR
HUODD
Hmean
MSPB + DDAS [ 32 ]
49.0
46.4
43.1
46.2
MSPB + SAR (ours)
60.5
53.0
49.6
54.4
Table 4: Comparison of different routing mechanisms.
V-LoRA
MSPB
SAR
mAPtgt
mAPcoco
H
1.4
50.6
2.7
✓
61.6
22.7
33.2
✓
22.5
0.2
0.4
✓
✓
79.1
0.5
1.0
✓
✓
59.5
50.4
54.6
✓
✓
22
50.3
30.6
Table 5: Ablation on different components of our framework on the ArTaxOr dataset.
Lm
Lp
mAPArTaxOr
mAPDIOR
mAPUODD
mAPavg
✓
78.9
57.4
48.1
61.5
✓
✓
79.1
57.7
48.9
61.9
Table 6: Ablation on the orthogonal loss Lp .
Figure 5: Ablation on the routing decision threshold τ .
Figure 6: Visualization of the selected prompts at different textual semantic augmentation using Grad-CAM [ 25 ] .
Figure 7: The probability distribution of reconstruction errors from different domain distributions ( i.e. , OV-COCO and DIOR).
Figure 8: Visualization of detection results on selected samples from the ArTaxOr and COCO datasets using the proposed HSA-DINO and OV-DINO models at different stages.
Despite the growing interest in open-vocabulary object detection in recent years, most existing methods rely heavily on manually curated fine-grained training datasets as well as resource-intensive layer-wise cross-modal feature extraction. In this paper, we propose HDINO, a concise yet efficient open-vocabulary object detector that eliminates the dependence on these components. Specifically, we propose a two-stage training strategy built upon the transformer-based DINO model. In the first stage, noisy samples are treated as additional positive object instances to construct a One-to-Many Semantic Alignment Mechanism(O2M) between the visual and textual modalities, thereby facilitating semantic alignment. A Difficulty Weighted Classification Loss (DWCL) is also designed based on initial detection difficulty to mine hard examples and further improve model performance. In the second stage, a lightweight feature fusion module is applied to the aligned representations to enhance sensitivity to linguistic semantics. Under the Swin Transformer-T setting, HDINO-T achieves 49.2 mAP on COCO using 2.2M training images from two publicly available detection datasets, without any manual data curation and the use of grounding data, surpassing Grounding DINO-T and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively, which are trained on 5.4M and 6.5M images. After fine-tuning on COCO, HDINO-T and HDINO-L further achieve 56.4 mAP and 59.2 mAP, highlighting the effectiveness and scalability of our approach. Code and models are available at https://github.com/HaoZ416/HDINO.
Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.
Open-Vocabulary Object Detection (OVOD) has achieved remarkable success in generalizing to novel categories. However, this success often rests on the implicit assumption of domain stationarity. In this work, we revisit the OVOD paradigm and study a key vulnerability: the fragile coupling between visual manifolds and textual embeddings under distribution shifts. We first formulate Domain-Generalized Open-Vocabulary Object Detection (DG-OVOD) as an evaluation protocol for open-vocabulary recognition under visual shifts. Through empirical analysis, we observe that visual shifts can destabilize the latent cross-modal space, causing novel-category visual signals to drift away from their semantic anchors. Motivated by these observations, we propose Progressive Domain-invariant Cross-modal Alignment (PICA). PICA departs from uniform training by introducing a multi-level curriculum based on ambiguity and signal strength. It constructs a quality-adjusted curriculum over pseudo-word prototypes, refined by sample reliability and visual consistency, to encourage more stable cross-domain modality alignment. Our findings suggest that OVOD robustness under domain shifts is closely linked to the stability of the latent cross-modal alignment space. Our work provides a DG-OVOD evaluation protocol and a practical perspective on building more generalizable open-vocabulary systems beyond static laboratory conditions.
Xiaoran Xu, Xiaoshan Yang, Jiangang Yang +3
the MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · the School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · the Institute of Microelectronics, University of Chinese Academy of Sciences, Beijing, China