Partition, Refine, and Detect: Preserving Novel Category Capacity in Open-Vocabulary Object Detection
Authors: Wei Zhang, Zhu Teng, Zhixiang Chi, Junnan Zou, Yi Liu, Jianping Fan
Organizations: the Beijing Jiaotong University, Beijing 100044, China · University of Toronto, Toronto, ON M5S 2E8, Canada · Baidu Inc., Shenzhen, Guangdong 518000, China · institute of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China
Open-vocabulary object detection (OVD) expects the model trained on base classes to recognize both base and novel classes at inference. Typical methods rely on CLIP-style vision-language embeddings, yet the learned space is often biased toward base categories, with limited effective capacity preserved for novel concepts. Such a base-dominated representation can compress novel semantics, leading to semantic collapse and mis-calibrated confidence at inference. To address this issue, we present PR-OVD, a partition-and-refine framework that explicitly reserves capacity for novel categories and strengthens category separation, while being seamlessly integrated into different detector architectures. Specifically, PR-OVD first performs coarse partitioning to build global divisions for base, novel, and an explicit background partition in both visual and textual modalities, preventing object-irrelevant embeddings from leaking into category-specific representations. It then applies category-level refinement within each partition to sharpen decision boundaries and alleviate the lack of fine-grained region-level pretraining in CLIP. To further provide reliable visual anchors for novel categories, we design a cognitively inspired Structure Prompt (SP) that encodes both egocentric attributes and allocentric context, facilitating stable partition-and-refine learning. Extensive experiments on two OVD benchmarks show that PR-OVD delivers an average 6.5% proportional improvement on novel categories over the state-of-the-art methods. Notably, PR-OVD also benefits other open-vocabulary tasks, including segmentation and classification, further confirming its effectiveness and generality.
Figures & tables
Fig. 1: Semantic collapsing and calibration imbalance in OVD. Left: t-SNE visualization and per-class performance on OV-COCO of a SoTA OVD method (CLIPSelf [ 5 ] ). Semantically similar base and novel categories are entangled, resulting in collapsed decision boundaries and calibration imbalance. Right: Our PR-OVD explicitly reserves representational space for novel categories, forming clear base–novel decision boundaries and substantially improving novel performance.
Fig. 2: The architecture of our PR-OVD. Our partition-and-refine paradigm first establishes global partitions in visual and textual embeddings to reserve dedicated space for novel categories, then applies category-level refinement within each partition to sharpen inter-class boundaries. Among them, structure prompt (SP) encodes multi-dimensional category descriptions and guides a generative model to synthesize representative novel images, providing enriched visual anchors for both prototype construction and category-level refinement.
Category
Methods
Venue
Backbone
OV-COCO
OV-LVIS
AP50novel
AP50base
AP50all
mAPnovel
mAPc
mAPf
mAPall
Pseudo Label related
OV-RCNN [ 9 ]
CVPR-21
RN50-C4 (24M)
17.5
41.0
34.9
-
-
-
-
CORA [ 36 ]
CVPR-23
RN50×4 (87M)
35.1
35.5
35.4
22.2
-
-
-
VLDet [ 37 ]
ICLR-23
RN50-C4 (24M)
32.0
50.6
45.8
26.3
39.4
41.9
38.1
SAS-Det [ 18 ]
CVPR-24
RN50-C4 (24M)
37.4
58.5
53.0
20.9
26.1
31.6
27.4
RALF [ 21 ]
CVPR-24
RN50 (24M)
41.3
54.3
50.9
21.9
26.2
29.1
26.6
TABLE I: Comparisons with SoTA OVD methods on OV-COCO [ 9 ] and OV-LVIS [ 44 ] . Methods are categorized and compared according to the taxonomy described in Sec. II . Bolded values indicate the best performance. Following prior works, we focus on the method’s performance on novel categories.
Fig. 3: Qualitative comparison of attention maps on base and novel categories with SoTAs . Compared with CLIPself [ 5 ] and RALF [ 21 ] , PR-OVD produces more precise attention, particularly for novel categories.
Coarse partition
Category-level refine
OV-COCO
AP50novel
AP50base
AP50all
✗
✓
44.8
62.2
57.3
✓
✗
45.9
61.5
58.0
✓
✓
47.4
63.3
59.1
TABLE II: Quantitative analysis of module contributions on the OV-COCO [ 9 ] benchmark.
Fig. 4: Qualitative analysis of module contributions on the OV-COCO benchmark using t-SNE feature visualization.
Language space partition
Vision space partition
OV-COCO
AP50novel
AP50base
AP50all
✗
✗
44.8
62.2
57.3
✗
✓
46.0
61.8
57.7
✓
✗
46.5
62.4
58.1
✓
✓
47.4
63.3
59.1
TABLE III: Ablation of coarse partition module architecture on the OV-COCO benchmark.
BG-N
BG-B
N-B
OV-COCO
AP50novel
AP50base
AP50all
✓
✓
✓
45.6
61.7
57.5
✗
✓
✗
46.2
62.6
58.2
✗
✗
✓
46.4
63.4
58.9
✗
✓
✓
47.4
63.3
59.1
TABLE IV: Ablation of asymmetric prototype learning strategy on the OV-COCO benchmark.
Structure Prompt
OV-COCO
OV-LVIS
AP50novel
AP50base
AP50all
mAPnovel
mAPc
mAPf
mAPall
✗
46.2
62.5
57.8
36.2
34.0
35.2
35.0
✓
47.4
63.3
59.1
37.0
34.0
35.3
35.1
TABLE V: Ablation study on the impact of SP in PR-OVD on the OV-COCO and OV-LVIS benchmarks.
Fig. 5: Qualitative and quantitative evaluation of novel-category image generation strategy. FID measures the distribution gap between generated and real images, with lower values indicating better quality and realism. AP50novel reports the detection performance on novel categories in OV-COCO.
LLM Type
OV-COCO
AP50novel
AP50base
AP50all
GPT-4o [ 31 ]
47.4
63.3
59.1
Qwen2.5-7B-Instruct [ 54 ]
47.2
63.4
59.1
Llama-3.1-8B-Instruct [ 55 ]
47.6
63.1
59.0
TABLE VI: Ablation study on different LLMs for structure prompt generation on the OV-COCO benchmark.
Fig. 6: Sensitivity analysis of the temperature hyperparameter τ . We report AP 50novel , AP 50base , and AP 50all on OV-COCO under different values of τ used in Eq. ( 1 )–Eq. ( 4 ).
Fig. 7: Sensitivity analysis of the number of synthetic images. We evaluate different numbers of synthetic images per novel category on the OV-COCO benchmark. Bar charts report AP 50novel , AP 50base , and AP 50all , while the line curve indicates GPU memory usage.
#
Layer Group
OV-COCO
AP50novel
AP50base
AP50all
1
[6,10,14,23]
47.4
63.3
59.1
2
[6,10,14,20]
45.8
63.1
58.6
3
[2,10,14,23]
46.3
63.5
58.9
4
[2,6,8,10]
44.2
60.7
56.4
5
[14,16,18,23]
45.2
64.8
59.8
TABLE VII: Sensitivity analysis of ViT layer groupings. We evaluate different layer groups used to construct multi-scale FPN features on the OV-COCO benchmark.
Method
COCO
Objects365
AP 50
AP 75
AP
AP 50
AP 75
AP
Supervised [ 44 ]
67.6
50.9
46.5
38.6
28.0
25.6
ViLD [ 44 ]
55.6
39.8
36.6
18.2
12.6
11.8
F-VLM [ 40 ]
53.1
34.6
32.5
19.2
12.6
11.9
BARON [ 56 ]
55.7
39.1
36.2
21.0
14.5
13.6
LP-OVOD [ 57 ]
-
-
-
18.9
13.1
12.6
TABLE VIII: Transferability results across datasets. Methods are trained on OV-LVIS and evaluated on COCO and Objects365.
TABLE IX: Transferability results of integrating our method into other OVD methods on OV-COCO.
Method
A-150
A-847
PC-59
PC-459
CEL-PL TMM’23 [ 58 ]
29.9
9.7
55.6
12.6
EBSeg CVPR’24 [ 59 ]
30.0
11.7
56.7
17.3
SED CVPR’24 [ 60 ]
31.6
11.4
57.3
18.6
CAT-Seg CVPR’24 [ 61 ]
31.8
12.0
57.5
19.0
Mask-adapter CVPR’25 [ 47 ]
35.6
14.2
58.4
17.9
H-CLIP CVPR’25 [ 62 ]
32.4
12.5
57.9
19.4
TABLE X: Performance on Open-Vocabulary Semantic Segmentation. Trained on COCO-Stuff and evaluated on ADE20K and Pascal-Context.
Method
Zero-shot Learning
G-zero-shot Learning
P@3
F1@3
mAP
P@3
F1@3
mAP
MKT AAAI’23 [ 63 ]
27.7
34.1
37.6
35.9
22.0
18.3
OVML ICME’23 [ 64 ]
36.3
40.1
42.6
32.9
18.3
14.3
RAM CVPR’25 [ 51 ]
43.7
48.4
53.2
38.7
23.5
20.7
CoMC ICML’24 [ 65 ]
33.5
41.2
48.2
-
-
-
Our PR-OVMLC
44.4
49.3
53.8
39.2
24.2
21.2
TABLE XI: Performance on Open-Vocabulary Multi-Label Classification on NUS-WIDE. P@3, F1@3 and mAP are reported.
Fig. 8: Visualization of our PR-OVD, CLIPSelf [ 5 ] , and RALF [ 21 ] on the OV-COCO. Red and blue bounding boxes denote novel and base categories.
Fig. 9: Representative challenging scenarios for novel-category detection in OVD. We illustrate two common cases, including tiny novel-category objects and novel objects in crowded scenes, with comparisons between our PR-OVD and existing SoTA method.
College of Intelligence and Computing, Tianjin University. · Key Research Center for Surface Monitoring and Analysis of Relics, State Administration of Cultural Heritage. · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology. +1
Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China · Faculty of Computing, Georg-August-Universität Göttingen, Germany · School of Software Technology, Zhejiang University, Ningbo 315000, China +1