CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

Dengke Zhang; Fagui Liu; Quan Tang; C. L. Philip Chen

cs.CVOct 17, 2026

CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

Authors: Dengke Zhang, Fagui Liu, Quan Tang, C. L. Philip Chen

Organizations: School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.

Abstract

Open-vocabulary semantic segmentation aims to classify pixels without being restricted to a fixed set of categories. Although Contrastive Language-Image Pre-training (CLIP) demonstrates remarkable potential for open-vocabulary recognition, its segmentation capability is impaired by noisy inter-class patch interactions. To this end, this work proposes a unified training-free object-centric framework, named CorrCLIPv2, from three complementary aspects of patch correlation, feature representation, and output regularization. Specifically, for patch correlation, we leverage class-agnostic mask generators to explicitly disentangle inter-class interactions. Moreover, we utilize spatially-coherent features from self-supervised models to compute attention scores to mitigate mask inaccuracies. Regarding feature representation, we devise a refinement mechanism to simultaneously strengthen spatial details and semantic representation. Within the output space, we unify predictions via voting in each mask to enhance spatial consistency. To mitigate contextual bias and enhance prediction accuracy of objects, we further extend the framework by incorporating a cross-image object retrieval paradigm, which incurs minimal computational overhead. We build a memory bank of CorrCLIP-enhanced features from unlabeled images and implement a regime of multi-CLIP ensemble. By aggregating semantically consistent object features across diverse contexts retrieved from the memory bank, CorrCLIPv2 mitigates context-induced interference and obtains more robust predictions. Extensive experiments demonstrate that our approach establishes a new state-of-the-art on 10 standard benchmarks. Remarkably, it even surpasses fully supervised open-vocabulary methods across 22 multi-domain datasets. Code is available at: https://github.com/zdk258/CorrCLIP.

Figures & tables

Figure 1: Impact of intra-class and intra-object correlations and different proportions of inter-class correlations on CLIP’s segmentation performance on COCO Stuff. “Lowest” and “Highest” denote the interactions with the least and most similar inter-class patches, respectively. Intra-Class and Intra-Object correlations achieve superior performance. In contrast, reintroducing inter-class interactions, even with high similarity, results in consistent degradation

Figure 2: Comparison of OVSS performance on ten standard benchmarks. CorrCLIP shows promising results. CorrCLIPv2 further improves it and sets a new state-of-the-art across all benchmarks

Figure 3: The overall framework of CorrCLIP. CLIP’s visual encoder processes the image to obtain value embeddings of image patches, while CLIP’s textual encoder encodes class names to produce class embeddings. A mask generator creates region masks and a feature extractor derives coherent patch embeddings to compute the similarity matrix for value reconstruction. Applying the region masks to the similarity matrix achieves scope reconstruction. The feature map is further refined via spatial and semantic branches, and the refined features are compared with class embeddings to generate the segmentation map, which is finally updated using the region masks

Figure 4: Oracle study on cross-image semantic aggregation. We compare the single-image CorrCLIP baseline against an Oracle CorrCLIPv2 variant that aggregates features from a ground-truth support set. The substantial performance leap, particularly on challenging datasets (e.g., +31.2% on PAS-59 and +27.7% on COCO-80), confirms that semantic prototype distilled from multiple instances effectively mitigates context-induced interference

Figure 5: Evaluation of feature discriminability of different models. We report the 1-NN retrieval accuracy to quantify how well different models maintain semantic consistency. While CorrCLIP improves upon the original CLIP, SSL-based VFMs exhibit significantly higher accuracy across all benchmarks. This observation justifies our design choice of utilizing VFMs as the index to ensure robust and accurate support set retrieval

Figure 6: Construction of the offline feature memory bank. For each image in the unlabeled collection, we extract object embeddings by applying Mask Average Pooling (MAP) to both VFM and CorrCLIP features across the generated region masks. The resulting embeddings populate the bank where VFM features as indexing keys and CorrCLIP features as semantic values

Figure 7: Illustration of the retrieval process during inference. The generated region masks are applied to the VFM feature map via Mask Average Pooling (MAP) to extract object embeddings. For each query embedding, we identify the Top- $K$ nearest neighbors in the VFM bank to obtain the corresponding indices. Then the associated CorrCLIP features are retrieved and averaged to construct the support set

Training-based (Fully Supervised)
Method	V21	V20	PC60	PC59	Object	Stuff	A150	City	A847	PC459	Mean
SAN ( Xu et al., 2023a )	-	95.4	-	57.7	-	-	32.1	-	12.4	15.7	-
CLIPSelf ( Wu et al., 2023 )	-	-	-	62.3	-	-	34.5	-	12.4	-	-
CAT-Seg ( Cho et al., 2024 )	82.5	97.0	-	63.3	-	-	37.9	-	16.0	23.8	-
SED++ ( Zhu et al., 2025 )	-	95.9	-	61.0	-	-	38.4	-	16.0	24.1	-
Training-based (Weakly Supervised)

Table 1: Comparison with state-of-the-art OVSS methods on 10 standard benchmarks. We report the best performance for each method. The best and second-best training-free methods are highlighted in bold and underlined, respectively

Training-based
	General						Earth Monitoring					Medical Sciences				Engineering				Agri. and Bio.
	BDD ( Yu et al., 2020 )	D. Zurich ( Sakaridis et al., 2019 )	MHP ( Li et al., 2017 )	FoodSeg ( Wu et al., 2021 )	ATLANTIS ( Erfani et al., 2022 )	DRAM ( Cohen et al., 2022 )	iSAID ( Waqas Zamir et al., 2019 )	Potsdam ( BSF Swissphoto, 2012 )	W. Floods ( Mateo-Garcia et al., 2021 )	FloodNet ( Rahnemoonfar et al., 2021 )	UAVid ( Lyu et al., 2020 )	Kvasir-I ( Jha et al., 2021 )	CHASE ( Fraz et al., 2012 )	CryoNu. ( Mahbod et al., 2021 )	PAXRay ( Seibold et al., 2022 )	Corrosion ( Bianchi and Hebdon, 2021 )	DeepCrack ( Liu et al., 2019 )	PST ( Shivakumar et al., 2020 )	ZW-f ( Bashkirova et al., 2022 )	SUIM ( Islam et al., 2020 )	CUB ( Welinder et al., 2010 )	CWFID ( Haug and Ostermann, 2014 )	Mean
X-Decoder ( Zou et al., 2023 )	48	25	4	3	28	27	3	32	27	9	26	56	11	12	16	2	25	20	16	25	1	30	20
SAN ( Xu et al., 2023b )	38	25	9	20	37	50	5	38	32	38	42	70	18	12	20	4	51	20	22	23	17	6	27
OVSeg ( Liang et al., 2023 )	46	23	7	17	34	54	9	31	32	36	39	72	22	14	23	7	17	22	12	39	14	34	27
OpenSeeD ( Zhang et al., 2023b )	48	29	3	9	19	30	2	32	31	24	40	60	47	34	38	14	48	3	3	20	1	12	25

Table 2: Comparison with state-of-the-art OVSS methods on the 22 datasets of the MESS benchmark. All methods use their base versions. The best methods are highlighted in bold

Figure 8: Qualitative comparison with state-of-the-art training-free methods. While competitors often exhibit fragmentation or misclassifications, CorrCLIPv2 generates spatially coherent and semantically accurate segmentation maps, closely aligning with the Ground Truth (GT)

Method	V21	PC60	Object	A150	City
Baseline	48.1	29.4	29.4	18.1	32.7
+SR	66.7	40.2	42.0	24.8	43.8
+VR	67.1	40.4	42.5	24.9	44.2
+FR	68.9	42.4	42.5	26.5	45.2
+MC	75.8	45.9	45.4	28.6	53.7

Table 3: Effectiveness of each component

Scale Factor	V21	PC60	Object	A150	City
No SR	48.1	29.4	29.4	18.1	32.7
1	62.6	38.1	39.0	23.3	40.8
2	66.7	40.2	42.0	24.8	43.8
3	68.0	40.8	42.7	25.1	44.8
4	69.2	41.4	43.5	25.6	45.6

Table 4: Impact of mask resolution on Scope Reconstruction. The “Scale Factor” denotes the spatial scaling ratio relative to the size of CLIP feature map

Sem.	Spa.	V21	PC60	Object	A150	City
		67.1	40.4	42.5	24.9	44.2
✓		67.9	41.2	42.6	25.8	43.5
	✓	68.1	41.9	41.9	25.9	45.8
✓	✓	68.9	42.4	42.5	26.5	45.2

Table 5: Effectiveness of the spatial branch and semantic branch

Method	V21	V20	PC60	PC59	Object	Stuff	A150	City	A847	PC459	Mean
CorrCLIP-B	75.8	89.9	45.9	50.8	45.4	32.6	28.6	53.7	10.5	11.8	44.2
+VISR	78.2	92.2	49.3	55.5	48.9	36.9	32.1	57.4	10.9	16.0	47.7
+DMCE	82.2	93.6	50.8	56.3	54.6	38.5	33.2	58.7	11.7	16.8	49.6
CorrCLIP-L	79.5	92.2	47.0	53.1	52.2	35.4	32.9	56.0	14.5	15.2	47.8
+VISR	81.3	93.0	49.8	56.2	56.1	37.4	35.4	58.1	14.8	17.9	50.0
+DMCE	82.3	93.2	51.1	56.8	58.0	39.1	36.5	59.2	14.9	18.6	51.0

Table 6: Effectiveness of VFM-Indexed Semantic Retrieval (VISR) and Decoupled Multi-CLIP Ensemble (DMCE)

VFM	V21	V20	PC60	PC59	Object	Stuff	A150	City	A847	PC459	Mean
SAM	70.8	85.8	41.4	47.3	30.1	27.3	26.3	50.7	6.1	11.6	39.7
DINO	78.0	90.1	46.9	51.9	43.4	32.7	26.7	51.4	7.2	13.8	44.2
DINOv2	82.7	93.8	51.0	56.5	54.3	38.3	32.5	57.7	11.1	15.9	49.4
DINOv3	82.4	93.3	50.4	56.3	54.0	38.7	33.1	58.1	11.7	16.5	49.5
PE-Spatial	80.9	91.0	48.9	53.9	50.9	36.4	30.3	49.3	9.3	14.8	46.6
RADIOv2.5	82.0	93.4	50.6	56.0	54.9	38.7	33.3	59.3	12.0	16.0	49.6

Table 7: Comparison of different VFMs used in CorrCLIPv2. DINOv2 and its distillates demonstrate superior performance due to their robust and generalizable representations

Dataset	Statistics		Performance
Dataset	# Imgs	# Embs	V21	PC60	Object	Stuff	A150	City	A847	PC459
Baseline	-	-	75.8	45.9	45.4	32.6	28.6	53.7	10.5	11.8
Pascal	17k	0.3M	83.1	50.9	47.2	35.5	33.0	60.4	10.9	16.7
ADE20k	20k	0.6M	82.3	50.6	45.3	34.9	33.3	59.7	12.5	16.1
COCO	118k	2.9M	82.2	50.8	54.6	38.5	33.2	58.7	11.7	16.8

Table 8: Comparison of different datasets used to construct memory bank. We report the number of images (Imgs) and embeddings (Embs) in the memory bank

Mask Generator	V21	PC60	Object	A150	City
EoMT	81.9	51.0	56.5	29.2	56.5
SAM	79.7	47.6	51.2	30.2	53.5
SAM2	80.8	48.7	53.2	30.0	56.7
EntitySeg	82.2	50.8	54.6	33.2	58.7
Ground Truth	91.9	59.5	76.1	46.1	77.6

Table 9: Impact of different mask generators

Figure 9: Impact of the fusion weight

\theta

. The original and the retrieved support features are aggregated with weights

\theta

and

1-\theta

, respectively. The optimal performance is observed in the range of

[0.2,0.8]

. We set the default value to 0.5

Figure 10: Impact of the number of retrieved instances for each query (

K

). Performance generally improves with more support instances and saturates around

K=20

Figure 11: Performance of CorrCLIPv2 across diverse CLIP backbones. We evaluate performance using various model capacities and pre-training sources. The consistent improvement over baselines across all configurations demonstrates that our framework is backbone-agnostic, effectively enhancing arbitrary CLIP models regardless of their specific pre-training priors

Method	Latency	Memory	Para.	Perf.
Method	(ms)	(GB)	(M)	(mIoU)
ClearCLIP	19	0.7	150	31.9
ProxyCLIP	69	1.9	235	36.7
Trident	81	2.5	364	41.0
CorrCLIP	177	3.0	458	43.1
CorrCLIPv2	151	4.3	756	49.6

Table 10: Computational costs on RTX 4090 with FP16. Performance is mean mIoU across ten standard benchmarks. CorrCLIPv2-fast uses smaller mask generator and VFM

Figure 12: Demonstration of the memory bank’s dynamic extensibility. By simply augmenting the COCO index with domain-specific unlabeled images (COCO+), CorrCLIPv2 achieves immediate performance gains on specialized benchmarks. This validates our framework’s efficient, plug-and-play adaptation capability

Method	V21	PC60	Object	A150	City
CLIP	11.6	5.0	1.7	1.9	3.0
ClearCLIP	77.5	45.4	53.0	28.3	41.0
ProxyCLIP	78.5	46.8	54.5	30.5	51.1
CorrCLIP	82.1	50.8	56.9	33.4	58.3

Table 11: Comparison of different semantic features for constructing the memory bank. Using CorrCLIP to extract semantic features yields the best results, validating the effectiveness of correlation reconstruction mechanism

Method	PC59	Stuff	A150	A847	PC459
CorrCLIPv2	56.8	39.1	36.5	14.9	18.6
Oracle Prompts	74.0	54.3	58.1	35.8	52.2

Table 12: Performance of CorrCLIPv2 under oracle text prompts

Open-Vocabulary Semantic Segmentation

Open-vocabulary semantic segmentation labels image pixels using text-defined categories beyond a fixed training vocabulary.

See all 87 papers

Mar 14, 2026

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction

The recent years have witnessed the remarkable development for open-vocabulary semantic segmentation (OVSS) using visual-language foundation models, yet still suffer from following fundamental challenges: (1) insufficient cross-modal communications between textual and visual spaces, and (2) significant computational costs from the interactions with massive number of categories. To address these issues, this paper describes a novel coarse-to-fine framework, called DCP-CLIP, for OVSS. Unlike prior efforts that mainly relied on pre-established category content and the inherent spatial-class interaction capability of CLIP, we dynamic constructing category-relevant textual features and explicitly models dual interactions between spatial image features and textual class semantics. Specifically, we first leverage CLIP's open-vocabulary recognition capability to identify semantic categories relevant to the image context, upon which we dynamically generate corresponding textual features to serve as initial textual guidance. Subsequently, we conduct a coarse segmentation by cross-modally integrating semantic information from textual guidance into the visual representations and achieve refined segmentation by integrating spatially enriched features from the encoder to recover fine-grained details and enhance spatial resolution. In final, we leverage spatial information from the segmentation side to refine category predictions for each mask, facilitating more precise semantic labeling. Experiments on multiple OVSS benchmarks demonstrate that DCP-CLIP outperforms existing methods by delivering both higher accuracy and greater efficiency.

Jing Wang, Huimin Shi, Quan Zhou +3

National Engineering Research Center of Communications and Networking, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China. · Institute for Advanced Ocean Research (Nantong), Southeast University, Nantong 226334, P. R. China. · Department of Internet of Things, Nanjing University of Posts & Telecommunications, Nanjing 210003, P. R. China +1

Open-Vocabulary Semantic Segmentation Text-Guided Image Segmentation

Ask PDF

Apr 27, 2026

DouC: Dual-Branch CLIP for Training-Free Open-Vocabulary Segmentation

Open-vocabulary semantic segmentation requires assigning pixel-level semantic labels while supporting an open and unrestricted set of categories. Training-free CLIP-based approaches preserve strong zero-shot generalization but typically rely on a single inference mechanism, limiting their ability to jointly address unreliable local tokens and insufficient spatial coherence. We propose DouC, a training-free dual-branch CLIP framework that decomposes dense prediction into two complementary components. OG-CLIP improves patch-level reliability via lightweight, inference-time token gating, while FADE-CLIP injects external structural priors through proxy attention guided by frozen vision foundation models. The two branches are fused at the logit level, enabling local token reliability and structure-aware patch interactions to jointly influence final predictions, with optional instance-aware correction applied as post-processing. DouC introduces no additional learnable parameters, requires no retraining, and preserves CLIP's zero-shot generalization. Extensive experiments across eight benchmarks and multiple CLIP backbones demonstrate that DouC consistently outperforms prior training-free methods and scales favorably with model capacity.

Mohamad Zamini, Diksha Shukla

Department of Electrical Engineering and Computer Science, University of Wyoming, Laramie, WY, USA.

Image Segmentation Open-Vocabulary Semantic Segmentation

Ask PDF

Mar 24, 2026

Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation

A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.

ByeongCheol Lee, Hyun Seok Seong, Sangeek Hyun +3

Sungkyunkwan University

Open-Vocabulary Semantic Segmentation Sliding-Window Attention Attention Mechanisms VLM Adaptation Vision-Language Alignment

Ask PDF

Abstract

Open-Vocabulary Semantic Segmentation

Explore similar work

DCP-CLIP:A Coarse-to-Fine Framework for Open-Vocabulary Semantic Segmentation with Dual Interaction

DouC: Dual-Branch CLIP for Training-Free Open-Vocabulary Segmentation

Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation