Partition, Refine, and Detect: Preserving Novel Category Capacity in Open-Vocabulary Object Detection
Authors: Wei Zhang, Zhu Teng, Zhixiang Chi, Junnan Zou, Yi Liu, Jianping Fan
Organizations: the Beijing Jiaotong University, Beijing 100044, China · University of Toronto, Toronto, ON M5S 2E8, Canada · Baidu Inc., Shenzhen, Guangdong 518000, China · institute of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China
Open-vocabulary object detection (OVD) expects the model trained on base classes to recognize both base and novel classes at inference. Typical methods rely on CLIP-style vision-language embeddings, yet the learned space is often biased toward base categories, with limited effective capacity preserved for novel concepts. Such a base-dominated representation can compress novel semantics, leading to semantic collapse and mis-calibrated confidence at inference. To address this issue, we present PR-OVD, a partition-and-refine framework that explicitly reserves capacity for novel categories and strengthens category separation, while being seamlessly integrated into different detector architectures. Specifically, PR-OVD first performs coarse partitioning to build global divisions for base, novel, and an explicit background partition in both visual and textual modalities, preventing object-irrelevant embeddings from leaking into category-specific representations. It then applies category-level refinement within each partition to sharpen decision boundaries and alleviate the lack of fine-grained region-level pretraining in CLIP. To further provide reliable visual anchors for novel categories, we design a cognitively inspired Structure Prompt (SP) that encodes both egocentric attributes and allocentric context, facilitating stable partition-and-refine learning. Extensive experiments on two OVD benchmarks show that PR-OVD delivers an average 6.5% proportional improvement on novel categories over the state-of-the-art methods. Notably, PR-OVD also benefits other open-vocabulary tasks, including segmentation and classification, further confirming its effectiveness and generality.
Figures & tables
Fig. 1: Semantic collapsing and calibration imbalance in OVD. Left: t-SNE visualization and per-class performance on OV-COCO of a SoTA OVD method (CLIPSelf [ 5 ] ). Semantically similar base and novel categories are entangled, resulting in collapsed decision boundaries and calibration imbalance. Right: Our PR-OVD explicitly reserves representational space for novel categories, forming clear base–novel decision boundaries and substantially improving novel performance.
Fig. 2: The architecture of our PR-OVD. Our partition-and-refine paradigm first establishes global partitions in visual and textual embeddings to reserve dedicated space for novel categories, then applies category-level refinement within each partition to sharpen inter-class boundaries. Among them, structure prompt (SP) encodes multi-dimensional category descriptions and guides a generative model to synthesize representative novel images, providing enriched visual anchors for both prototype construction and category-level refinement.
Category
Methods
Venue
Backbone
OV-COCO
OV-LVIS
AP50novel
AP50base
AP50all
mAPnovel
mAPc
mAPf
mAPall
Pseudo Label related
OV-RCNN [ 9 ]
CVPR-21
RN50-C4 (24M)
17.5
41.0
34.9
-
-
-
-
CORA [ 36 ]
CVPR-23
RN50×4 (87M)
35.1
35.5
35.4
22.2
-
-
-
VLDet [ 37 ]
ICLR-23
RN50-C4 (24M)
32.0
50.6
45.8
26.3
39.4
41.9
38.1
SAS-Det [ 18 ]
CVPR-24
RN50-C4 (24M)
37.4
58.5
53.0
20.9
26.1
31.6
27.4
RALF [ 21 ]
CVPR-24
RN50 (24M)
41.3
54.3
50.9
21.9
26.2
29.1
26.6
TABLE I: Comparisons with SoTA OVD methods on OV-COCO [ 9 ] and OV-LVIS [ 44 ] . Methods are categorized and compared according to the taxonomy described in Sec. II . Bolded values indicate the best performance. Following prior works, we focus on the method’s performance on novel categories.
Fig. 3: Qualitative comparison of attention maps on base and novel categories with SoTAs . Compared with CLIPself [ 5 ] and RALF [ 21 ] , PR-OVD produces more precise attention, particularly for novel categories.
Coarse partition
Category-level refine
OV-COCO
AP50novel
AP50base
AP50all
✗
✓
44.8
62.2
57.3
✓
✗
45.9
61.5
58.0
✓
✓
47.4
63.3
59.1
TABLE II: Quantitative analysis of module contributions on the OV-COCO [ 9 ] benchmark.
Fig. 4: Qualitative analysis of module contributions on the OV-COCO benchmark using t-SNE feature visualization.
Language space partition
Vision space partition
OV-COCO
AP50novel
AP50base
AP50all
✗
✗
44.8
62.2
57.3
✗
✓
46.0
61.8
57.7
✓
✗
46.5
62.4
58.1
✓
✓
47.4
63.3
59.1
TABLE III: Ablation of coarse partition module architecture on the OV-COCO benchmark.
BG-N
BG-B
N-B
OV-COCO
AP50novel
AP50base
AP50all
✓
✓
✓
45.6
61.7
57.5
✗
✓
✗
46.2
62.6
58.2
✗
✗
✓
46.4
63.4
58.9
✗
✓
✓
47.4
63.3
59.1
TABLE IV: Ablation of asymmetric prototype learning strategy on the OV-COCO benchmark.
Structure Prompt
OV-COCO
OV-LVIS
AP50novel
AP50base
AP50all
mAPnovel
mAPc
mAPf
mAPall
✗
46.2
62.5
57.8
36.2
34.0
35.2
35.0
✓
47.4
63.3
59.1
37.0
34.0
35.3
35.1
TABLE V: Ablation study on the impact of SP in PR-OVD on the OV-COCO and OV-LVIS benchmarks.
Fig. 5: Qualitative and quantitative evaluation of novel-category image generation strategy. FID measures the distribution gap between generated and real images, with lower values indicating better quality and realism. AP50novel reports the detection performance on novel categories in OV-COCO.
LLM Type
OV-COCO
AP50novel
AP50base
AP50all
GPT-4o [ 31 ]
47.4
63.3
59.1
Qwen2.5-7B-Instruct [ 54 ]
47.2
63.4
59.1
Llama-3.1-8B-Instruct [ 55 ]
47.6
63.1
59.0
TABLE VI: Ablation study on different LLMs for structure prompt generation on the OV-COCO benchmark.
Fig. 6: Sensitivity analysis of the temperature hyperparameter τ . We report AP 50novel , AP 50base , and AP 50all on OV-COCO under different values of τ used in Eq. ( 1 )–Eq. ( 4 ).
Fig. 7: Sensitivity analysis of the number of synthetic images. We evaluate different numbers of synthetic images per novel category on the OV-COCO benchmark. Bar charts report AP 50novel , AP 50base , and AP 50all , while the line curve indicates GPU memory usage.
#
Layer Group
OV-COCO
AP50novel
AP50base
AP50all
1
[6,10,14,23]
47.4
63.3
59.1
2
[6,10,14,20]
45.8
63.1
58.6
3
[2,10,14,23]
46.3
63.5
58.9
4
[2,6,8,10]
44.2
60.7
56.4
5
[14,16,18,23]
45.2
64.8
59.8
TABLE VII: Sensitivity analysis of ViT layer groupings. We evaluate different layer groups used to construct multi-scale FPN features on the OV-COCO benchmark.
Method
COCO
Objects365
AP 50
AP 75
AP
AP 50
AP 75
AP
Supervised [ 44 ]
67.6
50.9
46.5
38.6
28.0
25.6
ViLD [ 44 ]
55.6
39.8
36.6
18.2
12.6
11.8
F-VLM [ 40 ]
53.1
34.6
32.5
19.2
12.6
11.9
BARON [ 56 ]
55.7
39.1
36.2
21.0
14.5
13.6
LP-OVOD [ 57 ]
-
-
-
18.9
13.1
12.6
TABLE VIII: Transferability results across datasets. Methods are trained on OV-LVIS and evaluated on COCO and Objects365.
TABLE IX: Transferability results of integrating our method into other OVD methods on OV-COCO.
Method
A-150
A-847
PC-59
PC-459
CEL-PL TMM’23 [ 58 ]
29.9
9.7
55.6
12.6
EBSeg CVPR’24 [ 59 ]
30.0
11.7
56.7
17.3
SED CVPR’24 [ 60 ]
31.6
11.4
57.3
18.6
CAT-Seg CVPR’24 [ 61 ]
31.8
12.0
57.5
19.0
Mask-adapter CVPR’25 [ 47 ]
35.6
14.2
58.4
17.9
H-CLIP CVPR’25 [ 62 ]
32.4
12.5
57.9
19.4
TABLE X: Performance on Open-Vocabulary Semantic Segmentation. Trained on COCO-Stuff and evaluated on ADE20K and Pascal-Context.
Method
Zero-shot Learning
G-zero-shot Learning
P@3
F1@3
mAP
P@3
F1@3
mAP
MKT AAAI’23 [ 63 ]
27.7
34.1
37.6
35.9
22.0
18.3
OVML ICME’23 [ 64 ]
36.3
40.1
42.6
32.9
18.3
14.3
RAM CVPR’25 [ 51 ]
43.7
48.4
53.2
38.7
23.5
20.7
CoMC ICML’24 [ 65 ]
33.5
41.2
48.2
-
-
-
Our PR-OVMLC
44.4
49.3
53.8
39.2
24.2
21.2
TABLE XI: Performance on Open-Vocabulary Multi-Label Classification on NUS-WIDE. P@3, F1@3 and mAP are reported.
Fig. 8: Visualization of our PR-OVD, CLIPSelf [ 5 ] , and RALF [ 21 ] on the OV-COCO. Red and blue bounding boxes denote novel and base categories.
Fig. 9: Representative challenging scenarios for novel-category detection in OVD. We illustrate two common cases, including tiny novel-category objects and novel objects in crowded scenes, with comparisons between our PR-OVD and existing SoTA method.
Despite the remarkable progress in open-vocabulary object detection (OVD), a significant gap remains between the training and testing phases. During training, the RPN and RoI heads often misclassify unlabeled novel-category objects as background, causing some proposals to be prematurely filtered out by the RPN while others are further misclassified by the RoI head. During testing, these proposals again receive low scores and are removed in post-processing, leading to a significant drop in recall and ultimately weakening novel-category detection performance.To address these issues, we propose a novel training framework-NoOVD-which innovatively integrates a self-distillation mechanism grounded in the knowledge of frozen vision-language models (VLMs). Specifically, we design K-FPN, which leverages the pretrained knowledge of VLMs to guide the model in discovering novel-category objects and facilitates knowledge distillation-without requiring additional data-thus preventing forced alignment of novel objects with background.Additionally, we introduce R-RPN, which adjusts the confidence scores of proposals during inference to improve the recall of novel-category objects. Cross-dataset evaluations on OV-LVIS, OV-COCO, and Objects365 demonstrate that our approach consistently achieves superior performance across multiple metrics.
Yupeng Zhang, Ruize Han, Zhiwei Chen +2
College of Intelligence and Computing, Tianjin University. · Key Research Center for Surface Monitoring and Analysis of Relics, State Administration of Cultural Heritage. · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology. +1
Open-vocabulary object detection (OVOD) aims to detect both seen and unseen categories, yet existing methods often struggle to generalize to novel objects due to limited integration of global and local contextual cues. We propose DetRefiner, a simple yet effective plug-and-play framework that learns to fuse global and local features to refine open-vocabulary detection. DetRefiner processes global image features and patch-level image features from foundational models (e.g., DINOv3) through a lightweight Transformer encoder. The encoder produces a class vector capturing image-level attributes and patch vectors representing local region attributes, from which attribute reliability is inferred to recalibrate the base model's confidence. Notably, DetRefiner is trained independently of the base OVOD model, requiring neither access to its internal features nor retraining. At inference, it operates solely on the base detector's predictions, producing auxiliary calibration scores that are merged with the base detector's scores to yield the final refined confidence. Despite this simplicity, DetRefiner consistently enhances multiple OVOD models across COCO, LVIS, ODinW13, and Pascal VOC, achieving gains of up to +10.1 AP on novel categories. These results highlight that learning to fuse global and local representations offers a powerful and general mechanism for advancing open-world object detection. Our codes and models are available at https://github.com/hitachi-rd-cv/detrefiner.
Soichiro Okazaki, Tatsuya Sasaki, Hiroki Ohashi
Hitachi, Ltd. Research and Development Group, Japan
Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.
Yi Chen, Yinghao Lu, Zhehao Li +4
Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China · Faculty of Computing, Georg-August-Universität Göttingen, Germany · School of Software Technology, Zhejiang University, Ningbo 315000, China +1