Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection

Weihao Cao; Runqi Wang; Xiaoyue Duan; Jinchao Zhang; Ang Yang; Liping Jing

cs.CVApr 6, 2026

Parameter-Efficient Semantic Augmentation for Enhancing Open-Vocabulary Object Detection

Authors: Weihao Cao, Runqi Wang, Xiaoyue Duan, Jinchao Zhang, Ang Yang, Liping Jing

Organizations: State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University · WeChat AI, China

Abstract

Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.

Figures & tables

Figure 1: Pre-trained OVOD models perform well on general domain ( e.g. , OV-COCO [ 16 ] ) but fail to generalize to vertical domain ( e.g. , ArTaxOr [ 6 ] , DIOR [ 13 ] , UODD [ 10 ] ) in the zero-shot setting. Although fine-tuning improves performance on vertical domain, it causes a significant degradation on general domain.

Figure 2: Method motivation. (a) Previous methods use predefined templates or learnable vectors prepended to the category label embeddings, ignoring detailed semantics from image features. Our multi-scale prompt bank uses hierarchical semantics from multi-scale feature pyramid to select auxiliary prompts for the category labels. (b) The dynamic routing method [ 32 ] uses the reconstruction error of an input, obtained from multiple autoencoders, as an indicator of its domain. However, the large overlap in reconstruction errors across different domains can confuse the model, leading to incorrect parameter selection. Our method explicitly models both the content and the domain of the inputs, effectively reducing this overlap and enabling more accurate routing decisions.

Figure 3: Overview of the proposed HSA-DINO framework. We incorporate LoRA into the image encoder during training on downstream datasets. We also introduce a multi-scale prompt bank to further enhance the model’s adaptability to downstream tasks. In the test stage, we propose a semantic-aware router that can accurately identify different data distributions, enabling more precise selection between pre-trained semantics and augmented semantics for detection.

	Method	ArTaxOr			DIOR			UODD
	Method	$\mathrm{mAP}_{\mathrm{tgt}}$	$\mathrm{mAP}_{\mathrm{coco}}$	$H$	$\mathrm{mAP}_{\mathrm{tgt}}$	$\mathrm{mAP}_{\mathrm{coco}}$	$H$	$\mathrm{mAP}_{\mathrm{tgt}}$	$\mathrm{mAP}_{\mathrm{coco}}$	$H$
ZS	GLIP-L [ 14 ]	12.6	51.2	20.2	4.8	51.2	8.8	4.6	51.2	8.4
	GroundingDINO-T [ 17 ]	4.6	48.4	8.4	2.9	48.4	5.5	6.3	48.4	11.1
	YOLO-World-X [ 1 ]	2.3	46.6	4.4	0.1	46.6	0.2	0.1	46.6	0.2
	OV-DINO [ 28 ]	1.4	50.6	2.7	3.0	50.6	5.7	2.9	50.6	5.5
FFT	GLIP-L [ 14 ]	73.1	7.6	13.8	67.5	15.7	25.5	36.6	27.8	31.6

Table 1: Comparison of different methods across various downstream tasks. “ZS” denotes the zero-shot performance of the pre-trained models, while “FFT” and “PEFT” denote the models’ performance after full fine-tuning or parameter-efficient fine-tuning on downstream tasks, respectively.

\mathrm{mAP}_{\mathrm{tgt}}

denotes the performance of models on corresponding target downstream datasets ( i.e. , ArTaxOr, DIOR, UODD);

\mathrm{mAP}_{\mathrm{coco}}

denotes the performance of models on the OV-COCO.

H

denotes the harmonic mean value defined in Eq. 10 .

	Method	OV-COCO +
	Method	w ArTaxOr	w DIOR	w UODD
ZS	GLIP-L [ 14 ]	48.0	33.2	49.0
	GroundingDINO-T [ 17 ]	43.7	36.9	46.8
	YOLO-World-X [ 1 ]	42.9	36.3	44.9
	OV-DINO [ 28 ]	44.5	37.3	48.7
FFT	GLIP-L [ 14 ]	11.7	13.2	24.5

Table 2: Comparison of mAP across different methods on OV-COCO + . “w” indicates the additional dataset included for joint evaluation ( e.g. , “w ArTaxOr” means OV-COCO evaluated together with ArTaxOr).

Figure 4: Ablation study on (a) bank size $N$ , (b) prompt length $M$ , (c) key matching loss weight $\lambda_{\mathrm{m}}$ , and (d) orthogonal loss weight $\lambda_{\mathrm{p}}$ .

Method	$H_{\text{ArTaxOr}}$	$H_{\text{DIOR}}$	$H_{\text{UODD}}$	$H_{\text{mean}}$
Predefined + SAR	54.6	47.3	47.7	49.9
CoOp [ 38 ] + SAR	57.1	51.1	48.0	52.1
AttriCLIP [ 29 ] + SAR	58.8	51.6	48.5	53.0
MSPB + SAR (ours)	60.5	53.0	49.6	54.4

Table 3: Comparison of different textual semantic augmentation.

Method	$H_{\text{ArTaxOr}}$	$H_{\text{DIOR}}$	$H_{\text{UODD}}$	$H_{\text{mean}}$
MSPB + DDAS [ 32 ]	49.0	46.4	43.1	46.2
MSPB + SAR (ours)	60.5	53.0	49.6	54.4

Table 4: Comparison of different routing mechanisms.

V-LoRA	MSPB	SAR	$\mathrm{mAP}_{\mathrm{tgt}}$	$\mathrm{mAP}_{\mathrm{coco}}$	$H$
			1.4	50.6	2.7
✓			61.6	22.7	33.2
	✓		22.5	0.2	0.4
✓	✓		79.1	0.5	1.0
✓		✓	59.5	50.4	54.6
	✓	✓	22	50.3	30.6

Table 5: Ablation on different components of our framework on the ArTaxOr dataset.

$\mathcal{L}_{\mathrm{m}}$	$\mathcal{L}_{\mathrm{p}}$	$\mathrm{mAP}_{\text{ArTaxOr}}$	$\mathrm{mAP}_{\text{DIOR}}$	$\mathrm{mAP}_{\text{UODD}}$	$\mathrm{mAP}_{\mathrm{avg}}$
✓		78.9	57.4	48.1	61.5
✓	✓	79.1	57.7	48.9	61.9

Table 6: Ablation on the orthogonal loss

\mathcal{L}_{\mathrm{p}}

Figure 5: Ablation on the routing decision threshold $\tau$ .

Figure 6: Visualization of the selected prompts at different textual semantic augmentation using Grad-CAM [ 25 ] .

Figure 7: The probability distribution of reconstruction errors from different domain distributions ( i.e. , OV-COCO and DIOR).

Figure 8: Visualization of detection results on selected samples from the ArTaxOr and COCO datasets using the proposed HSA-DINO and OV-DINO models at different stages.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 87 papers

HDINO: A Concise and Efficient Open-Vocabulary Detector
Mar 3, 2026Hao Zhang, Yiqun Wang, Qinran Lin +2Open-Vocabulary Object Detection Multimodal Object Detection
Chongqing University
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5
Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method
Mar 29, 2026Xiaoran Xu, Xiaoshan Yang, Jiangang Yang +3Open-Vocabulary Object Detection VLM Robustness
the MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · the School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China · the Institute of Microelectronics, University of Chinese Academy of Sciences, Beijing, China

Mar 3, 2026J/K move · Enter open · S save

HDINO: A Concise and Efficient Open-Vocabulary Detector

Hao Zhang, Yiqun Wang, Qinran Lin, Runze Fan+1

Chongqing University

Open-Vocabulary Object Detection Multimodal Object Detection Efficient Object Detection Detection Transformer Vision-Language Alignment

Open paper AskPDF arXiv

Despite the growing interest in open-vocabulary object detection in recent years, most existing methods rely heavily on manually curated fine-grained training datasets as well as resource-intensive layer-wise cross-modal feature extraction. In this paper, we propose HDINO, a concise yet efficient open-vocabulary object detector that eliminates the dependence on these components. Specifically, we propose a two-stage training strategy built upon the transformer-based DINO model. In the first stage, noisy samples are treated as additional positive object instances to construct a One-to-Many Semantic Alignment Mechanism(O2M) between the visual and textual modalities, thereby facilitating semantic alignment. A Difficulty Weighted Classification Loss (DWCL) is also designed based on initial detection difficulty to mine hard examples and further improve model performance. In the second stage, a lightweight feature fusion module is applied to the aligned representations to enhance sensitivity to linguistic semantics. Under the Swin Transformer-T setting, HDINO-T achieves 49.2 mAP on COCO using 2.2M training images from two publicly available detection datasets, without any manual data curation and the use of grounding data, surpassing Grounding DINO-T and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively, which are trained on 5.4M and 6.5M images. After fine-tuning on COCO, HDINO-T and HDINO-L further achieve 56.4 mAP and 59.2 mAP, highlighting the effectiveness and scalability of our approach. Code and models are available at https://github.com/HaoZ416/HDINO.

Abstract

Open-Vocabulary Object Detection

Explore similar work

HDINO: A Concise and Efficient Open-Vocabulary Detector

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method