Recent advances in open-vocabulary object detection focus primarily on two aspects: scaling up datasets and leveraging contrastive learning to align language and vision modalities. However, these approaches often neglect internal consistency within a single modality, particularly when background or environmental changes occur. This lack of consistency leads to a performance drop because the model struggles to detect the same object in different scenes, which reveals a robustness gap. To address this issue, we introduce Contextual Consistency Learning (CCL), a novel framework that integrates two key strategies: Contextual Bootstrapped Data Generation (CBDG) and Contextual Consistency Loss (CCLoss). CBDG functions as a data generation mechanism, producing images that contain the same objects across diverse backgrounds. This is essential because existing datasets alone do not support our CCL framework. The CCLoss further enforces the invariance of object features despite environmental changes, thereby improving the model's robustness in different scenes. These strategies collectively form a unified framework for ensuring contextual consistency within the same modality. Our method achieves state-of-the-art performance, surpassing previous approaches by +16.3 AP on OmniLabel and +14.9 AP on D3. These results demonstrate the importance of enforcing intra-modal consistency, significantly enhancing model generalization in diverse environments. Our code is publicly available at: https://github.com/bozhao-li/CCL.
Figures & tables
Figure 1: Performance and robustness comparison of different methods. (a) Our approach, with Contextual Consistency Learning, achieves the best overall results, reaching a normalized score of 1 in all metrics. (b,c) Benchmark backgrounds are altered to test robustness. Tested on D3 BC , baseline methods degrade, while ours remains stable. See Section 4.3 for details.
Figure 2: Overview of our approach. CBDG generates Dj via Categorical Augmentation, Background Generation and Background Replacement. CCL training uses Dj with CCLoss added to total loss.
Figure 3: CBDG Pipeline. We use ChatGPT to generate background prompts for a diffusion model, enabling diverse background synthesis. For single-class images, CBDG augments object categories before background replacement. For multi-class images, CBDG replaces only the background.
Figure 4: Four groups of images are shown, each composed of four sub-images: the leftmost sub-image in every group is the original, while the remaining three display background replacements.
Figure 5: CCL Framework. Visual and textual features are encoded, with regional features pooled into CAAF. Consistency loss is applied within each modality.
OmniLabel
D3
Method
AP
AP-c
AP-d
AP-dP
AP-dS
AP-dM
AP-dL
FULL
PRES
ABS
Detic [ 65 ]
8.0
15.6
5.4
8.0
5.7
5.4
6.2
-
-
-
OFA-DOD [ 52 ]
-
-
-
-
-
-
-
21.6
23.7
15.4
RelationLLM-L [ 51 ]
-
-
-
-
-
-
-
24.3
24.6
23.4
GN-GLIP [ 64 ]
22.2
27.2
18.8
29.0
-
-
-
21.4
20.6
23.7
GN-FIBER [ 64 ]
28.1
32.1
25.1
36.5
-
-
-
26.0
25.2
28.1
Table 1: Performance of our method compared with SOTA methods.
Figure 6: Examples of DC3 Perturbations.
D3 BC
DC3
Method
FULL
PRES
ABS
FULL
mFULL
rFULL
GLIP-T
16.8
15.8
20.4
19.1
13.6
71.2
+ours
29.6
28.9
31.9
30.0
21.7
72.3
FIBER-B
20.1
18.7
24.7
22.7
16.7
73.6
+ours
33.1
32.8
34.0
37.6
27.5
73.1
Table 2: Performance comparison on D3 BC and DC3 .
OmniLabel
D3
Method
AP
AP-c
AP-d
FULL
PRES
ABS
GLIP-T
19.3
23.6
16.4
19.1
18.3
21.5
+data
24.8
29.2
21.8
23.2
22.5
25.3
+ours
32.2
36.1
28.8
30.0
29.2
32.3
FIBER-B
25.7
30.3
22.3
22.7
21.5
26.0
+data
32.7
35.8
29.6
29.1
28.3
31.2
Table 3: Ablation study on the effects of Contextual Bootstrapped Data Generation (CBDG) and Contextual Consistency Loss (CCLoss). “+data” denotes using CBDG without CCLoss, “+ours” combines both components.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Tuning
λI
λT
OmniLabel (AP)
D3 (FULL)
0
0
0
32.7
29.2
1
0.5
0.5
35.8
31.5
2
0.2
0.2
38.2
34.0
3
0.1
0.1
39.9
36.1
4
0.15
0.15
41.2
37.1
5
0.15
0.1
41.7
37.4
Appendix
Table 4: Tuning on the FIBER baseline
Method
AP
Ap-c
Ap-d
FULL
PRES
ABS
FIBER-B+ours
42.0
44.1
39.2
37.6
37.2
38.8
FIBER-B + BBox copy-paste
23.3
27.0
20.6
20.9
20.4
22.6
Appendix
Table 5: Comparison Between Our Method and Simple BBox Copy-Paste Based on the FIBER Baseline
Method
FULL
PRES
ABS
GLIP-T+ours
29.7
28.9
32.1
FIBER-B+ours
37.5
37.1
38.6
Appendix
Table 6: Results on 2,304 D3 Images from Cityscapes and Indoor Urban Environments
Model
Val
Test
R@1
R@5
R@10
R@1
R@5
R@10
GLIP-T
85.7
95.4
96.9
85.7
95.8
97.2
GLIP-T+ours
88.5
95.9
97.2
88.8
96.2
97.2
FIBER-B
87.1
96.1
97.4
87.4
96.4
97.6
FIBER-B+ours
89.9
96.6
97.7
90.0
96.8
97.7
Appendix
Table 7: Recall@K metrics for both GLIP-T and FIBER-B before and after applying our method
Method(Baseline)
OmniLabel
D3
AP
AP-c
AP-d
FULL
PRES
ABS
IAM(FIBER-B)
20.5
24.6
18.0
18.4
17.8
20.3
+ours(FIBER-B)
26.1
31.2
22.2
23.5
22.8
26.1
CBDG+ours(FIBER-B)
32.4
34.6
29.4
30.3
30.0
31.4
Appendix
Table 8: Experiments on different generation method.
Method(Baseline)
OmniLabel
D3
AP
AP-c
AP-d
FULL
PRES
ABS
Matrix(FIBER-B)
29.8
32.4
27.5
27.9
26.8
30.0
Variance(FIBER-B)
30.2
33.2
28.6
28.8
28.1
30.5
CCLoss(FIBER-B)
32.4
34.6
29.4
30.3
30.0
31.4
Appendix
Table 9: Experiments on different consistency loss.
Method(Baseline)
OmniLabel
D3
AP
AP-c
AP-d
FULL
PRES
ABS
GLIP-T
19.3
23.6
16.4
19.1
18.3
21.5
+ours( Kr =0.6)
27.2
30.4
24.7
26.2
25.2
29.3
+ours( Kr =0.8)
30.1
34.1
27.4
28.9
28.1
31.3
FIBER-B
25.7
30.3
22.3
22.7
21.5
26.0
+ours( Kr =0.6)
35.6
38.3
34.0
32.4
32.0
33.6
Appendix
Table 10: Impact of dataset size of the CCL approach, where Kr is the dataset reduction scaling factor.
Figure 7: Three types of background images we generate. Given a prompt from LLM, we use stable diffusion to generate random background images.
Figure 8: Failure cases of background replacement using inpainting method [ 57 ] . Each case consists of paired images: the left image shows the original input, while the right image demonstrates the replaced background with corresponding artifacts. Cases are presented in a two-per-row layout for comparison.
Conventional object detectors typically operate under a closed-set assumption, limiting recognition to a predefined set of base classes seen during training. Open-vocabulary object detection (OVD) addresses this limitation by leveraging vision-language models (VLMs) to generate pseudo labels for novel object classes. However, existing OVD methods suffer from two critical drawbacks: (1) inaccurate class label assignments, as VLMs are optimized for image-level predictions rather than the region-level predictions required for pseudo labeling, and (2) unreliable objectness scores from region proposal networks (RPNs) trained exclusively on base object classes. To address these issues, we propose a novel pseudo labeling framework for OVD. Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation by assessing consistency across hierarchical semantic levels (class, super- and sub-category). We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness token to mitigate base class bias problem of RPNs and provide reliable objectness estimations for novel object classes. Extensive experiments on standard OVD benchmarks, including COCO and LVIS, demonstrate that our approach clearly sets a new state of the art, validating the effectiveness of our approach. Project site: https://cvlab.yonsei.ac.kr/projects/HCC
Sanghoon Lee, Geon Lee, Hyekang Park +1
Yonsei University · Korea Institute of Science and Technology (KIST)
Open-vocabulary object detection (OVOD) enables models to detect any object category, including unseen ones. Benefiting from large-scale pre-training, existing OVOD methods achieve strong detection performance on general scenarios (e.g., OV-COCO) but suffer severe performance drops when transferred to downstream tasks with substantial domain shifts. This degradation stems from the scarcity and weak semantics of category labels in domain-specific task, as well as the inability of existing models to capture auxiliary semantics beyond coarse-grained category label. To address these issues, we propose HSA-DINO, a parameter-efficient semantic augmentation framework for enhancing open-vocabulary object detection. Specifically, we propose a multi-scale prompt bank that leverages image feature pyramids to capture hierarchical semantics and select domain-specific local semantic prompts, progressively enriching textual representations from coarse to fine-grained levels. Furthermore, we introduce a semantic-aware router that dynamically selects the appropriate semantic augmentation strategy during inference, thereby preventing parameter updates from degrading the generalization ability of the pre-trained OVOD model. We evaluate HSA-DINO on OV-COCO, several vertical domain datasets, and modified benchmark settings. The results show that HSA-DINO performs favorably against previous state-of-the-art methods, achieving a superior trade-off between domain adaptability and open-vocabulary generalization.
Weihao Cao, Runqi Wang, Xiaoyue Duan +3
State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University · School of Computer Science & Technology, Beijing Jiaotong University +1
Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.