Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

Aijing Yu; Zhengbo Wang; Lijun Sheng; Jian Liang; Xiao-Yu Zhang

cs.CVOct 22, 2026

Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

Authors: Aijing Yu, Zhengbo Wang, Lijun Sheng, Jian Liang, Xiao-Yu Zhang

Organizations: School of Cyber Security, University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences · University of Science and Technology of China ·

Abstract

Contrastive Language-Image Pre-training (CLIP) has shown superior performance in the image-level open-vocabulary classification task, benefiting from its powerful alignment of visual features and text embeddings. However, dense prediction tasks require models to perform pixel-level predictions, necessitating modifications to CLIP for region-text alignment. In this paper, we adjust the attention layers within the vanilla visual encoder of CLIP. With a novel attention calculation method for selective attention features, the model retains more regional semantic information. To further mitigate the negative impact of the modality gap, we introduce additional visual prototypes to refine the original predictor solely based on text embeddings. Specifically, patch features with confident predictions are selected to form an additional auxiliary memory-based classifier for more accurate prediction. We further integrate an optimizable feature extractor with an exponential moving average updating strategy to enhance prediction quality. Extensive experiments on $5$ semantic segmentation datasets demonstrate the effectiveness of our method.

Figures & tables

Figure 1: The diagram of the Regional Semantic Preservation Module. We discard Wq and Wk , using only the value feature output by Wv for self-attention calculation. This mechanism yields dense features that better capture regional semantic information.

Figure 2: The pipeline of Visual Feature Memory Module. Given the inputs, we employ the regional semantic preservation module in Section 2.2 to extract the dense features. Then, the text embeddings of CLIP output the class prediction and the corresponding probability confidence. The feature cache is used to select the visual features with Top-K confidence, which are averaged and set as vision embeddings.

Figure 3: Overview of our method. The input image is encoded by the regional semantic preservation module in Section 2.2 , followed by the refined predictor in Section 2.3 for semantic segmentation.

Datasets	ADE	City	COCO	PC	VOC
Top-K	512	128	4,096	1,024	128
$\lambda$	0.5	0.05	0.5	0.5	0.5
Training Iterations †	8,000	4,000	8,000	4,000	2,000

Table 1: Hyper-parameter settings in our method across different datasets. † denotes hyper-parameters in the distillation process.

Method	ADE	City	COCO	PC	VOC	Avg.
CLIP [ 19 ]	3.1	6.5	5.7	11.1	49.4	14.1
MaskCLIP+ † [ 27 ]	10.8	21.6	14.7	23.3	53.7	23.7
MaskCLIP+ [ 27 ]	11.9	24.9	16.7	26.4	74.9	30.3
GroupViT [ 24 ]	10.4	18.5	15.3	23.4	79.7	30.7
ReCLIP [ 21 ]	14.3	19.9	20.2	33.7	75.8	32.8
ReCo [ 20 ]	11.2	21.6	14.8	22.3	57.7	23.5

Table 2: Quantitative results (mIoU,

\%

) comparison on

5

semantic segmentation benchmarks. MaskCLIP † means their baseline without refinement. We highlight the best and the second-best results.

Method	ADE	City	COCO	PC	VOC	Avg.
MaskCLIP	8.3	18.8	10.2	18.5	41.5	19.5
Ours ♢	11.5	22.1	15.0	24.4	59.0	26.4
MaskCLIP+	10.7	23.4	13.6	24.2	57.3	25.8
Ours	13.0	26.9	16.4	28.0	70.2	30.9

Table 3: Quantitative results (mIoU,

\%

) under CLIP-ResNet-50. Ours with

\diamondsuit

symbol denotes the training-free setting.

Figure 4: Qualitative results on all the $5$ datasets. Each $3$ closely arranged images form a group, in which the upper left corner, lower left corner, and right image are respectively the input image, ground truth, and the visualization result of the proposed method.

Panel of Table 4 — Table 4: Ablation results (mIoU, $\%$ ) of each component in our method. A., V., and E. denote the refined self-attention in Section 2.2 , vision embeddings saved by the visual feature memory module in Section 2.3 , and the optimization strategy in Section 2.4 , respectively.

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 87 papers

CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation
Oct 17, 2026Dengke Zhang, Fagui Liu, Quan Tang +1Open-Vocabulary Semantic Segmentation Semantic Segmentation
School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.
SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation
Aug 6, 2026Mohamad Zamini, Diksha ShuklaMemory-Augmented VLMs Open-Vocabulary Semantic Segmentation
University of Wyoming Laramie, WY
DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction
Mar 14, 2026Jing Wang, Huimin Shi, Quan Zhou +3Open-Vocabulary Semantic Segmentation Text-Guided Image Segmentation
National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China +1

Oct 17, 2026J/K move · Enter open · S save

CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

Dengke Zhang, Fagui Liu, Quan Tang, C. L. Philip Chen

School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.

Open-Vocabulary Semantic Segmentation Semantic Segmentation

Open paper AskPDF arXiv

Open-vocabulary semantic segmentation aims to classify pixels without being restricted to a fixed set of categories. Although Contrastive Language-Image Pre-training (CLIP) demonstrates remarkable potential for open-vocabulary recognition, its segmentation capability is impaired by noisy inter-class patch interactions. To this end, this work proposes a unified training-free object-centric framework, named CorrCLIPv2, from three complementary aspects of patch correlation, feature representation, and output regularization. Specifically, for patch correlation, we leverage class-agnostic mask generators to explicitly disentangle inter-class interactions. Moreover, we utilize spatially-coherent features from self-supervised models to compute attention scores to mitigate mask inaccuracies. Regarding feature representation, we devise a refinement mechanism to simultaneously strengthen spatial details and semantic representation. Within the output space, we unify predictions via voting in each mask to enhance spatial consistency. To mitigate contextual bias and enhance prediction accuracy of objects, we further extend the framework by incorporating a cross-image object retrieval paradigm, which incurs minimal computational overhead. We build a memory bank of CorrCLIP-enhanced features from unlabeled images and implement a regime of multi-CLIP ensemble. By aggregating semantically consistent object features across diverse contexts retrieved from the memory bank, CorrCLIPv2 mitigates context-induced interference and obtains more robust predictions. Extensive experiments demonstrate that our approach establishes a new state-of-the-art on 10 standard benchmarks. Remarkably, it even surpasses fully supervised open-vocabulary methods across 22 multi-domain datasets. Code is available at: https://github.com/zdk258/CorrCLIP.

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction