NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection

Yupeng Zhang; Ruize Han; Zhiwei Chen; Wei Feng; Liang Wan

cs.CVMar 22, 2026

NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection

Authors: Yupeng Zhang, Ruize Han, Zhiwei Chen, Wei Feng, Liang Wan

Organizations: College of Intelligence and Computing, Tianjin University. · Key Research Center for Surface Monitoring and Analysis of Relics, State Administration of Cultural Heritage. · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology. · School of Artificial Intelligence, Nanchang University.

Abstract

Despite the remarkable progress in open-vocabulary object detection (OVD), a significant gap remains between the training and testing phases. During training, the RPN and RoI heads often misclassify unlabeled novel-category objects as background, causing some proposals to be prematurely filtered out by the RPN while others are further misclassified by the RoI head. During testing, these proposals again receive low scores and are removed in post-processing, leading to a significant drop in recall and ultimately weakening novel-category detection performance.To address these issues, we propose a novel training framework-NoOVD-which innovatively integrates a self-distillation mechanism grounded in the knowledge of frozen vision-language models (VLMs). Specifically, we design K-FPN, which leverages the pretrained knowledge of VLMs to guide the model in discovering novel-category objects and facilitates knowledge distillation-without requiring additional data-thus preventing forced alignment of novel objects with background.Additionally, we introduce R-RPN, which adjusts the confidence scores of proposals during inference to improve the recall of novel-category objects. Cross-dataset evaluations on OV-LVIS, OV-COCO, and Objects365 demonstrate that our approach consistently achieves superior performance across multiple metrics.

Figures & tables

Figure 1: Training process for OVD using frozen CLIP. (a) Commonly used training process, (b) Our training process.

Figure 2: Illustration of the training process of NoOVD. We use K-FPN to extract the pyramid embeddings from frozen CLIP to identify latent novel-category objects. Besides the image-text alignment with $\mathcal{L}_{\text{cons}}$ , we also align the features of the RoI head with the features from K-FPN via knowledge self-distillation with $\mathcal{L}_{\text{kd}}$ .

Figure 3: Overall of K-FPN (the CLIP Image Encoder is taken as an example with ViT-B/16).

Method	Backbone	Training Data	$\textit{AP}_{\textit{f}}$	$\textit{AP}_{\textit{c}}$	$\textit{AP}_{\textit{r}}$	AP
RegionCLIP	RN50 ∗	CC3M	34.0	27.4	17.1	28.2
RegionCLIP	RN50x4 ∗	CC3M	36.9	32.1	22.0	32.3
Detic	RN50 ∗	LVIS-base + IN-L	-	-	24.9	32.4
OWL-ViT	ViT-B/16	O365 + VG	-	-	20.6	27.2
OWL-ViT	ViT-L/14	O365 + VG	-	-	31.2	34.6
RKDWTF	RN50 ∗ Base	LVIS-base + IN-L	26.4	19.4	12.2	20.9

Table 1: Comparison with SOTA methods on OV-LVIS (%).

Method	Backbone	Training Data	$\textit{AP}_{\textit{base}}^{\textit{50}}$	$\textit{AP}_{\textit{novel}}^{\textit{50}}$	$\textit{AP}^{\textit{50}}$
RegionCLIP	RN50 ∗	CC3M	57.1	31.4	50.4
RegionCLIP	RN50x4 ∗	CC3M	61.6	39.3	55.7
Detic	RN50 ∗	LVIS-base + IN-L	47.1	27.8	45.0
OWL-ViT	ViT-B/16	O365 + VG	-	-	49.2
RKDWTF	RN50 ∗ Base	LVIS-base + IN-L	53.2	1.7	39.6
	RN50 ∗ RKDPIS		52.8	31.5	47.2

Table 2: Comparison with SOTA methods on OV-COCO (%).

Method	Backbone	Training Data	$\textit{AP}_{\textit{r}}$	AP	$\textit{AP}^{\textit{50}}$
Detic	RN50 ∗	LVIS-all	9.5	13.9	19.7
Detic		LVIS-all + IN-L	12.4	15.6	22.2
MM-OVOD		LVIS-all	10.1	14.8	21.0
MM-OVOD		LVIS-all + IN-L	13.1	16.6	23.1
CLIPSelf + F-ViT	ViT-B/16	LVIS-base	16.8	19.0	32.3
CLIPSelf + F-ViT	ViT-L/14		21.7	23.7	39.2

Table 3: Cross-dataset main results on Objects365 (%).

Baseline	CLIP-top	K-FPN	R-RPN	$\textit{AP}_{\textit{r}}$	AP
✓				25.4	25.4
✓	✓			26.4	26.1
✓	✓	✓		27.5	26.4
✓			✓	26.7	25.9
✓	✓		✓	27.8	26.4
✓	✓	✓	✓	28.3	26.7

Table 4: Effects of each module in NoOVD on the OV-LVIS (%).

$W$	0.2	0.3	0.4	0.5	0.7	0.8
AP	26.2	26.7	26.3	24.8	22.5	19.1

Table 5: Sentitivity analysis of hyperparameter

W

(%).

$\alpha$	0.9	0.7	0.5	0.3	0.1
AP	25.5	25.8	26.7	26.2	25.9

Table 6: Sentitivity analysis of hyperparameter

\alpha

(%).

$\mathcal{L}_{\text{kd}}$	$\mathcal{L}_{\text{1}}$	$\mathcal{L}_{\text{2}}$	$\mathrm{Smooth}\mathcal{L}_{\text{1}}$	$\mathcal{L}_{\text{cons}}$
$AP_{r}$ (%)	27.8	28.3	28.0	17.3

Table 7: Results of different knowledge distillation losses (%).

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 89 papers

Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs
Jun 4, 2026Yi Chen, Yinghao Lu, Zhehao Li +4Open-Vocabulary Object Detection Graph Reasoning
Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China · Faculty of Computing, Georg-August-Universität Göttingen, Germany · School of Software Technology, Zhejiang University, Ningbo 315000, China +1
COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection
May 26, 2026Yupeng Zhang, Ruize Han, Yuzhong Feng +3Open-Vocabulary Object Detection VLM Adaptation
Tianjin University. · Shenzhen University of Advanced Technology.
Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection
Apr 25, 2026Sanghoon Lee, Geon Lee, Hyekang Park +1Open-Vocabulary Object Detection Probability Calibration
Yonsei University · Korea Institute of Science and Technology (KIST)

Jun 4, 2026J/K move · Enter open · S save

Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs

Yi Chen, Yinghao Lu, Zhehao Li, Chenchen Yan+3

Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China · Faculty of Computing, Georg-August-Universität Göttingen, Germany · School of Software Technology, Zhejiang University, Ningbo 315000, China+1

Open-Vocabulary Object Detection Graph Reasoning

Open paper AskPDF arXiv

Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.

Abstract

Open-Vocabulary Object Detection

Explore similar work

Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs

COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection

Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection