Organizations: School of Medical Imaging, Xuzhou Medical University, Xuzhou, 221004, China · School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China · College of Information Science and Technology, Nanjing Forestry University, Nanjing, 210037, China
Open-vocabulary object detection (OVD) is essential for detecting objects beyond the categories seen during training. Establishing vision-language associations is crucial for recognizing unseen categories, while integrating complementary visual features across levels supports detection at different scales. However, existing methods often provide insufficient textual guidance for object regions, rely on limited cross-modal alignment strategies, and underuse complementary features across levels, limiting their ability to detect unseen categories and objects of varying scales. To address these limitations, we propose a Text-Guided Modality Correction Network (TMCNet) that incorporates textual guidance into region feature extraction, cross-modal alignment, and multi-scale decoding to strengthen region-category semantic correspondence and promote complementary feature fusion across levels. First, a text-guided modality encoder incorporates textual semantics into queries to guide regional feature aggregation, reducing semantic ambiguity in object representations. Second, a cross-modal alignment module strengthens the correspondence between region representations and category semantics through interactions between visual and textual features. Finally, a progressive text correction decoder combines a top-down pathway with lateral connections to progressively fuse text-guided deep semantic features with shallow visual features at the corresponding scales, preserving both semantic information and spatial detail for objects of different sizes. Experiments on two public datasets, COCO-OVD and LVIS-OVD, demonstrate that TMCNet achieves strong performance in open-vocabulary object detection using simple text queries as guidance. Extending TMCNet to open-vocabulary segmentation also yields performance gains, demonstrating its cross-task applicability and extensibility.
Figures & tables
Figure 1: Comparison of representative open-vocabulary object detection architectures. (a) Two-stage detectors extract proposal features for region–text classification and bounding-box regression. (b) Single-stage detectors fuse visual and textual features for dense classification and localization. (c) Transformer-based detectors use object queries to aggregate semantically aligned visual features for instance-level prediction. (d) TMCNet incorporates textual priors into query representations and combines cross-modal alignment with progressive multiscale decoding to guide visual feature construction.
Figure 2: Overview of TMCNet. A hierarchical visual encoder and a pretrained text encoder extract visual and category representations, respectively. The text-guided modality encoder (TGME) constructs queries with textual priors. The cross-modal alignment pipeline (CMAP) uses these queries to guide lateral feature correction and generate joint vision-language features. The progressive text correction decoder (PTCD) integrates both outputs through multiscale aggregation, producing features for object classification and localization.
Detection-adapted Baseline
VLM
Feature backbone
TGME
CMAP
PTCD
COCO-OVD(%)
AP 50novel
AP 50base
AP 50all
OVSeg-OVD ∗
ViT-B/16
ResNet-101c
✗
✗
✗
33.70
53.70
49.30
ViT-B/16
ResNet-101c
✓
✗
✗
37.90 {}_{\color[rgb]{1,0,0}\textrm{+4.20}}
58.00
52.80
ViT-B/16
ResNet-101c
✗
✓
✗
35.10 {}_{\color[rgb]{1,0,0}\textrm{+1.40}}
55.20
51.40
ViT-B/16
ResNet-101c
✓
✓
✗
38.50 {}_{\color[rgb]{1,0,0}\textrm{+4.80}}
58.20
53.00
ViT-B/16
ResNet-101c
✓
✓
✓
39.20 {}_{\color[rgb]{1,0,0}\textrm{+5.50}}
58.50
53.30
Table 1: Ablation study of TGME, CMAP, and PTCD on two detection-adapted baselines on COCO-OVD. All results are reported in %. ∗ indicates the detection-adapted variants of OVSeg and SED. ✓ and ✗ indicate the presence and absence of the corresponding module, respectively. Bold indicates the best result for each baseline.
Detection-adapted Baseline
VLM
Feature backbone
TGME
CMAP
PTCD
LVIS-OVD(%)
AP r
AP c
AP f
mAP
OVSeg-OVD ∗
ViT-B/16
ResNet-101c
✗
✗
✗
22.40
28.30
36.00
32.40
ViT-B/16
ResNet-101c
✓
✗
✗
24.90 {}_{\color[rgb]{1,0,0}\textrm{+2.50}}
30.70
38.20
34.50
ViT-B/16
ResNet-101c
✗
✓
✗
23.80 {}_{\color[rgb]{1,0,0}\textrm{+1.40}}
29.50
37.40
33.70
ViT-B/16
ResNet-101c
✓
✓
✗
26.1 {}_{\color[rgb]{1,0,0}\textrm{+3.70}}
31.90
39.50
36.00
ViT-B/16
ResNet-101c
✓
✓
✓
26.70 {}_{\color[rgb]{1,0,0}\textrm{+4.30}}
37.10
40.00
36.50
Table 2: Ablation results for different module configurations on LVIS-OVD. All results are reported in %. ∗ indicates the detection-adapted variants of OVSeg and SED. ✓ and ✗ indicate the presence and absence of the corresponding module, respectively. Bold indicates the best result for each baseline.
Methods
RTM
CAF
TGME
COCO-OVD(%)
LVIS-OVD(%)
AP 50novel
AP 50base
AP 50all
AP r
AP c
AP f
mAP
OVSeg-OVD ∗
✗
✗
✗
33.70
53.70
49.30
22.40
28.30
36.00
32.40
TMCNet OVSeg
✓
✗
✗
37.00 {}_{\color[rgb]{1,0,0}\textrm{+3.30}}
56.50
51.00
23.60 {}_{\color[rgb]{1,0,0}\textrm{+1.20}}
33.70
36.90
33.40
✗
✓
✗
37.70 {}_{\color[rgb]{1,0,0}\textrm{+4.00}}
57.00
51.50
23.90 {}_{\color[rgb]{1,0,0}\textrm{+1.50}}
34.10
37.30
33.80
✗
✗
✓
39.20 {}_{\color[rgb]{1,0,0}\textrm{+5.50}}
58.50
53.30
26.70 {}_{\color[rgb]{1,0,0}\textrm{+4.30}}
37.10
40.00
36.50
SED-OVD ∗
✓
✗
✗
36.10
54.20
49.90
24.80
34.50
38.00
34.00
Table 3: Comparison of text semantic modeling strategies on COCO-OVD and LVIS-OVD. RTM, CAF, and TGME denote Region–Text Matching, Cross-Attention Fusion, and the Text-Guided Modality Encoder, respectively. All results are reported in %. ∗ marks the detection-adapted OVSeg and SED baselines. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding baseline.
Methods
CLIP
LP
CMAP
COCO-OVD(%)
LVIS-OVD(%)
AP 50novel
AP 50base
AP 50all
AP r
AP c
AP f
mAP
OVSeg-OVD ∗
✓
✗
✗
33.70
53.70
49.30
22.40
28.30
36.00
32.40
TMCNet OVSeg
✓
✗
✗
37.50 {}_{\color[rgb]{1,0,0}\textrm{+3.80}}
57.10
51.60
24.00 {}_{\color[rgb]{1,0,0}\textrm{+1.60}}
34.10
37.60
34.10
✗
✓
✗
38.40 {}_{\color[rgb]{1,0,0}\textrm{+4.70}}
57.70
52.30
24.50 {}_{\color[rgb]{1,0,0}\textrm{+2.10}}
34.60
38.00
34.50
✗
✗
✓
39.20 {}_{\color[rgb]{1,0,0}\textrm{+5.50}}
58.50
53.30
26.70 {}_{\color[rgb]{1,0,0}\textrm{+4.30}}
37.10
40.00
36.50
SED-OVD ∗
-
✗
✗
36.10
54.20
49.90
24.80
34.50
38.00
34.00
Table 4: Comparison of cross-modal alignment schemes on COCO-OVD and LVIS-OVD. CLIP, LP, and CMAP denote CLIP feature matching, linear projection, and the proposed Cross-Modal Alignment Pipeline, respectively. All results are reported in %. ∗ marks the detection-adapted OVSeg and SED baselines. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding baseline.
Figure 3: Novel-category pseudo-label coverage on the COCO-OVD validation set across training iterations for different cross-modal alignment schemes. The horizontal axis shows training iterations in thousands, and the vertical axis shows coverage in %.
Detection-adapted Methods
CLIP
Def_DETR
CLIP_Trans
COCO-OVD(%)
LVIS-OVD(%)
AP 50novel
AP 50base
AP 50all
AP r
AP c
AP f
mAP
TMCNet OVSeg
✓
✗
✗
38.00
57.30
52.00
24.00
34.00
37.30
33.50
✗
✓
✗
38.80
58.00
52.80
24.90
35.00
38.20
34.60
✗
✗
✓
39.20 {}_{\color[rgb]{1,0,0}\textrm{+1.20}}
58.50
53.30
26.70 {}_{\color[rgb]{1,0,0}\textrm{+4.30}}
37.10
40.00
36.50
TMCNet SED
✓
✗
✗
39.40
58.50
52.90
25.50
35.20
38.80
34.80
✗
✓
✗
40.70
59.90
54.30
26.00
35.90
39.40
35.50
Table 5: Comparison of detection configurations on COCO-OVD and LVIS-OVD, with TGME, CMAP, and PTCD incorporated in both frameworks. CLIP, Def_DETR, and CLIP_Trans denote the CLIP feature-matching configuration, the Deformable DETR-based configuration, and the adopted query-based CLIP–Transformer head, respectively. ✓ and ✗ indicate whether a configuration is used. All results are reported in %. Bold indicates the best result within each framework; red subscripts report gains in percentage points over the corresponding OVSeg-OVD or SED-OVD baseline.
Methods
Publication
Train Setting
Backbone
detectors
COCO-OVD(%)
AP 50novel
AP 50base
AP 50all
OVR-CNN [ 45 ] †
CVPR’21
V-OVD
RN50-C4
Faster R-CNN
22.80
46.00
39.90
BARON [ 36 ] †
CVPR’23
V-OVD
RN50-C4
Faster R-CNN
33.10
54.80
49.10
BIND [ 46 ] †
CVPR’24
V-OVD
ViT-B/16
Built-in detector
36.30
54.70
50.20
CAKE [ 30 ] †
AAAI’25
V-OVD
RN50
Faster R-CNN
38.20
58.00
52.80
RegionCLIP [ 48 ] †
CVPR’22
C-OVD
RN50-C4
CLIP
26.80
54.80
47.50
Table 6: Comparison of TMCNet with representative open-vocabulary object detection methods on COCO-OVD. † denotes results obtained by retraining and evaluating the models in this work. ∗ indicates the detection-adapted variants of OVSeg and SED. Def_DETR denotes the Deformable DETR-based detection configuration, and CLIP_Trans denotes the query-based CLIP+Transformer detection head adopted in this work. − denotes unreported results
Methods
Publication
Backbone
detectors
LVIS-OVD(%)
Inf. time (ms/image)
AP r
AP c
AP f
mAP
OV-DETR [ 44 ] †
ECCV’22
RN50
Def_DETR
21.00
25.00
32.50
26.60
574.20
BIND [ 46 ] †
CVPR’24
ViT-B/16
Built-in detector
29.40
30.60
33.50
31.40
42.60
SAS-Det [ 47 ] †
CVPR’24
RN50-C4
Faster R-CNN
20.90
26.10
31.60
27.40
71.20
CAKE [ 30 ] †
AAAI’25
RN50
Faster R-CNN
25.00
34.80
38.40
34.90
35.80
MSPL [ 8 ] †
ECCV’26
RN50-FPN
Faster R-CNN
25.00
33.30
37.20
33.50
-
Table 7: Comparison of open-vocabulary object detection methods on LVIS-OVD. † denotes models retrained by the authors of this work. ∗ indicates the detection-adapted variants of OVSeg and SED. Def_DETR denotes the Deformable DETR-based detection configuration, and CLIP_Trans denotes the query-based CLIP+Transformer detection head adopted in this work. − denotes an unreported result.
Figure 4: Qualitative detection results for SED-OVD and its variants on COCO-OVD. From left to right: input images and predictions from SED-OVD, SED-OVD TGME , and TMCNet SED-OVD . Red solid boxes in the input images highlight selected novel-category objects, while white solid boxes in the predictions highlight additional objects detected relative to SED-OVD. Colored bounding boxes and their labels indicate predicted locations, categories, and confidence scores.
Methods
VLM
Feature backbone
Train Dataset
A-847
PC-459
A-150
PC-59
PAS-20
OVSeg [ 20 ] †
ViT-B/16
Swin-B
COCO-Stuff-171
7.10
11.00
24.80
53.30
92.60
TMCNet OVSeg
ViT-B/16
Swin-B
COCO-Stuff-171
9.10 {}_{\color[rgb]{1,0,0}\textrm{+2.00}}
14.20 {}_{\color[rgb]{1,0,0}\textrm{+3.20}}
30.90 {}_{\color[rgb]{1,0,0}\textrm{+6.10}}
55.70 {}_{\color[rgb]{1,0,0}\textrm{+2.40}}
95.40 {}_{\color[rgb]{1,0,0}\textrm{+2.80}}
SED [ 39 ] †
ConvNeXt-B
-
COCO-Stuff-171
8.90
12.40
31.20
47.30
93.70
TMCNet SED
ConvNeXt-B
-
COCO-Stuff-171
11.40 {}_{\color[rgb]{1,0,0}\textrm{+2.50}}
16.70 {}_{\color[rgb]{1,0,0}\textrm{+4.30}}
32.80 {}_{\color[rgb]{1,0,0}\textrm{+1.60}}
57.40 {}_{\color[rgb]{1,0,0}\textrm{+10.10}}
95.30 {}_{\color[rgb]{1,0,0}\textrm{+1.60}}
Table 8: Effectiveness of TMCNet in different open-vocabulary semantic segmentation frameworks. † denotes results obtained by retraining and evaluating the models in this work. − indicates that no additional feature backbone is used. Boldface highlights our methods, and red subscripts indicate absolute gains in percentage points over the corresponding reproduced baseline.
Methods
Publication
VLM
Training Dataset
A-847
PC-459
A-150
PC-59
PAS-20
ZegFormer [ 9 ]
CVPR’22
ViT-B/16
COCO-Stuff-156
4.90
9.10
16.90
42.80
86.20
ZSseg [ 41 ]
ECCV’22
ViT-B/16
COCO-Stuff-171
7.00
–
20.50
47.70
88.40
OVSeg [ 20 ] †
CVPR’23
ViT-B/16
COCO-Stuff-171
7.10
11.00
24.80
53.30
92.60
SED [ 39 ] †
CVPR’24
ConvNeXt-B
COCO-Stuff-171
8.90
12.40
31.20
47.30
93.70
CAT-Seg + DeCLIP [ 35 ] †
CVPR’25
ViT-B/16
COCO-Stuff-171
9.50
15.40
31.30
55.60
92.60
LoGoSeg [ 3 ] †
AAAI’26
ConvNeXt-B
COCO-Stuff-171
10.20
16.30
32.00
55.80
93.00
Table 9: Comparison with representative open-vocabulary semantic segmentation methods. All results are reported in mIoU (%). † denotes results reproduced in this work; other baseline results are taken from the corresponding papers or official implementations. − denotes an unreported result.
Object detection is an important task in computer vision, which aims to detect the objects of interest. through the given category list or query images. In this work, we propose a new problem of language-visual-complementary open-set object detection (LV-OSD), i.e., using the flexible text-based and/or image-based prompts to specify the desired object categories. This setting is more common and practical in real-world applications. For this purpose, we design a dual-branch detection framework, LVDor, which can simultaneously accept both text and image prompts. Specifically, we first build the Multi-modal Prompts (MPr) containing various text descriptions and image samples for each category. Subsequently, to bridge the semantic gap among the input image, text prompts, and image prompts, we design a Target-guided Prompt Dynamic Weighting (TPDW) module. Guided by the prior information of the target image, this module dynamically produces the text and image prompts that best align with the target semantics, achieving precise alignment and effectively reducing the discrepancy between the two modalities, thereby accommodating the LV-OSD setting. We also propose a simple Prompt Random Masking (PRM) mechanism during training to simulate the arbitrary combination of text and/or image prompts in testing. Extensive experimental results verify our problem formulation's reasonability and our method's effectiveness. Prompts and code will be released publicly.
Yupeng Zhang, Ruize Han, Wei Feng +2
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen 518107, Guangdong, China
Open-domain open-vocabulary detection (ODOVD) requires detectors to generalize to both novel categories and unseen domains, making it more challenging than open-vocabulary detection. Existing methods typically train open-vocabulary detectors together with domain generalization modules from scratch, leading to high training cost. we propose ExDet, a lightweight category-domain collaborative generalization framework for ODOVD that enhances the cross-category and cross-domain generalization of existing detectors. ExDet consists of Text-Guided Extrapolation (TGE), a lightweight Detector-Compatible Rectification (DCR) module, and ExRPN. Specifically, TGE exploits the DeltaSpace property of vision-language models (VLMs) to infer category- and domain-aware proxy visual prototypes from text. DCR is learned from the TGE-generated prototypes in a detector training-free and real-data-free manner, and is inserted after the classification head at inference to rectify representations toward a detector-compatible source-domain visual distribution, thereby enhancing classification for targets from novel categories and unseen domains. ExRPN recalibrates proposal scores by combining semantic similarity with RPN confidence, improving recall for novel and domain-shifted objects while providing better support for subsequent classification and DCR. ExDet achieves SOTA performance on OD-LVIS, OV-LVIS, Objects365, and MSOSB.
Yupeng Zhang, Yuzhong Feng, Ruize Han +3
College of Intelligence and Computing, Tianjin University, Tianjin, China · Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology, Shenzhen, China · School of Artificial Intelligence, Nanchang University, Jiangxi, China
Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.
Yi Chen, Yinghao Lu, Zhehao Li +4
Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang 315211, China · Faculty of Computing, Georg-August-Universität Göttingen, Germany · School of Software Technology, Zhejiang University, Ningbo 315000, China +1