DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
Authors: Jing Wang, Huimin Shi, Quan Zhou, Qibo Liu, Suofei Zhang, Huimin Lu
Organizations: National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China · School of Automation, Southeast University, Nanjing 210096, P. R. China
The recent years have witnessed the remarkable development for open-vocabulary semantic segmentation (OVSS) using visual-language foundation models, yet still suffer from following fundamental challenges: (1) insufficient cross-modal communications between textual and visual spaces, and (2) significant computational costs from the interactions with massive number of categories. To address these issues, this paper describes a novel coarse-to-fine framework, called DCP-CLIP, for OVSS. Unlike prior efforts that mainly relied on pre-established category content and the inherent spatial-class interaction capability of CLIP, we dynamic constructing category-relevant textual features and explicitly models dual interactions between spatial image features and textual class semantics. Specifically, we first leverage CLIP's open-vocabulary recognition capability to identify semantic categories relevant to the image context, upon which we dynamically generate corresponding textual features to serve as initial textual guidance. Subsequently, we conduct a coarse segmentation by cross-modally integrating semantic information from textual guidance into the visual representations and achieve refined segmentation by integrating spatially enriched features from the encoder to recover fine-grained details and enhance spatial resolution. In final, we leverage spatial information from the segmentation side to refine category predictions for each mask, facilitating more precise semantic labeling. Experiments on multiple OVSS benchmarks demonstrate that DCP-CLIP outperforms existing methods by delivering both higher accuracy and greater efficiency.
Figures & tables
Fig. 1: Comparison of efficiency and open-vocabulary semantic segmentation performance among CAT-Seg (blue), SED (purple), and our method (green). The left side shows efficiency metrics, and the right side shows segmentation accuracy. GFLOPs and inference time are measured on a single NVIDIA 3090 GPU. Our method achieves superior segmentation accuracy with balanced efficiency.
Fig. 2: Illustration of two representative OVSS methods and our approach. (a) Proposal mask-based method. (b) Cost volume-based method. (c) Our DCP-CLIP, which adopts a coarse-to-fine framework and introduces a dual interaction mechanism to explicitly enhance semantic alignment between textual and visual spaces.
Fig. 3: Overview of our proposed DCP-CLIP. The Dynamic Category Selection module adaptively selects relevant categories based on the image content. Next, we perform cross-modal semantic learning under the guidance of the selected categories, followed by fine-segmentation that restores spatial details and resolution through a Spatial Enhanced Decoder. During training, the fine segmentation output is used for supervision. During inference, Tag Validation module leverages spatial context to refine open-vocabulary predictions.
Fig. 4: The detailed architecture of text-guided alignment. High-level semantics from text are injected via image features as a bridge, and then fused with initial cost volume features through self-attention to produce more expressive representations.
Fig. 5: The detailed architecture of Spatial Enhanced Decoder. Spatial enhancement uses a Swin Transformer to model local context with shallow features and attention maps. Deconvolution and multi-source fusion produce refined semantic maps.
Method
Pub.
VLM
Additional Backbone
Training dataset
A-847
PC-459
A-150
PC-59
PAS-20
SPNet [ 49 ]
CVPR19
-
ResNet-101
PASCAL VOC
-
-
-
24.3
18.3
ZS3Net [ 5 ]
NeurIPS19
-
ResNet-101
PASCAL VOC
-
-
-
19.4
38.3
LSeg [ 38 ]
ICLR22
CLIP ViT-B/32
ResNet-101
PASCAL VOC-15
-
-
-
-
47.4
GroupViT [ 8 ]
CVPR22
ViT-S/16
-
GCC [ 50 ] +YFCC [ 51 ]
4.3
4.9
10.6
25.9
50.7
LSeg+ [ 52 ]
ECCV22
ALIGN
ResNet-101
COCO-Stuff
2.5
5.2
13.0
36.0
-
ZegFormer [ 53 ]
CVPR22
CLIP ViT-B/16
ResNet-101
COCO-Stuff-156
4.9
9.1
16.9
42.8
86.2
TABLE I: Comparison with State-of-the-Art Methods. We use mIoU as the evaluation metric. The best-performing results are highlighted in bold, while the second-best are underlined.
Method
# of learnable params. (M)
# of total params. (M)
Training time (min)
Inference time (s)
Inference GFLOPs
mIoU
ZSSeg [ 56 ]
102.8
530.8
958.5
2.73
22,302.1
20.5
ZegFormer [ 53 ]
103.3
531.2
1,148.3
2.70
19,425.6
16.9
OVSeg [ 13 ]
408.9
532.6
-
2.00
19,345.6
24.8
CAT-Seg [ 19 ]
70.3
433.7
875.5
0.54
2,121.1
27.2
SED [ 20 ]
204.8
762.3
1302.4
0.21
674.8
31.6
DCP-CLIP(Ours)
80.3
443.6
496.4
0.20
616.2
32.4
TABLE II: Efficiency comparison. All results are measured on a single RTX 3090 GPU, with all methods using the base-scale vision-language model to ensure consistency in model size. We report Inference time and mlou on A-150 dataset in this table.
Fig. 6: Visualization of ablation study results on open-vocabulary semantic segmentation. From left to right are the input image, ground-truth segmentation, fine-tuning CLIP, and the predicted results after successively incorporating the DCS, TGA, SED, and TV modules. (Best viewed in color)
DCS
TGA
SED
TV
A-847
PC-459
A-150
PC-59
PAS-20
7.1
14.4
23.1
52.6
93.7
✓
7.4
14.3
23.3
52.9
93.6
✓
✓
10.8
17.7
30.2
56.3
94.6
✓
✓
11.3
18.6
31.5
57.1
94.5
✓
✓
✓
12.1
19.6
32.2
57.9
95.7
✓
✓
✓
✓
12.3
19.7
32.4
58.1
95.9
TABLE III: Ablation study for the contribution of each component of DCP-CLIP. We conduct ablation study by gradually adding DCS, TGA, SED and TV to the Fine-Tuning CLIP baseline.
CA
SA
A-847
PC-459
A-150
PC-59
PAS-20
11.4
18.5
31.2
57.4
94.4
✓
12.1
19.3
31.9
57.6
94.9
✓
11.6
18.7
32.1
57.9
95.5
✓
✓
12.3
19.7
32.4
58.1
95.9
TABLE IV: Ablation study for the text-guided alignment. we gradually remove these components and evaluate their effects on model performance.
Method
A-847
PC-459
A-150
PC-59
PAS-20
full textual guidance (I)
12.3
19.7
32.4
58.1
95.9
(I) w/o Fcls⋅ESC (II)
12.2
19.5
32.5
57.6
95.5
(II) w/o learnable prompt
12.0
19.1
32.0
57.0
95.1
TABLE V: Ablation study for the structure of textual guidance. We gradually ablate components of the proposed method to verify their effectiveness.
(a)
Aggregation
A-847
PC-459
A-150
PC-59
PAS-20
None
11.0
18.1
30.8
56.6
94.7
Spatial-enhance
12.3
19.7
32.4
58.1
95.9
(b)
Decoder Layer
A-847
PC-459
A-150
PC-59
PAS-20
0
11.4
18.8
31.5
57.0
94.9
1
11.8
19.4
31.9
57.6
95.2
2
12.3
19.7
32.4
58.1
95.9
TABLE VI: Ablation study for Spatial Enhanced Decoder. We evaluate the individual contributions of spatial enhancement, decoder layers, and feature fusion by modifying each component separately in (a), (b) and (c).
Fig. 7: Qualitative results on open-vocabulary semantic segmentation benchmarks. Examples are shown from (a) PC-459, (b) A-150, (c) PC-59, and (d) PAS-20 datasets. From top to bottom are the input image, ground-truth segmentation, and our model’s prediction. (Best viewed in color)
θ
A-847
A-150
PAS-20
Memory
Training time
inference time
(GiB)
(min)
(s)
0.9
11.9
32.0
96.0
12.1
461.4
0.19
0.8
12.1
32.2
96.1
12.6
478.9
0.19
0.5
12.3
32.4
95.9
13.2
496.4
0.20
0.2
12.4
32.6
95.6
15.5
528.6
0.22
0
12.4
32.5
95.2
20.1
869.5
0.51
TABLE VII: Analysis of threshold of dynamic category selection. Here, We report the inference time on a single RTX 3090 GPU, evaluated on the A-847, A-150 and PAS-20 dataset. inference time is evaluated on A-150 dataset.
College of Computer and Data Science, Fuzhou University, Fuzhou, China. · School of Computer and Big Data, Minjiang University, Fuzhou, China. · Department of Computer Science and Engineering, European University Cyprus, Nicosia, Cyprus.