RCPN: towards robust pseudo-labels in unsupervised CLIP adaptation via consistent prototypes and neighborhoods

Eman Ali; Chetan Arora; Muhammad Haris Khan

cs.LGJul 22, 2025

RCPN: towards robust pseudo-labels in unsupervised CLIP adaptation via consistent prototypes and neighborhoods

Authors: Eman Ali, Chetan Arora, Muhammad Haris Khan

Organizations: Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE. · Alexandria University, Alexandria, Egypt. · Indian Institute of Technology Delhi, New Delhi, India.

Abstract

In unsupervised adaptation for vision-language models such as CLIP, pseudo-labels derived from zero-shot predictions often exhibit significant noise, particularly under domain shifts or in visually complex scenarios. Conventional pseudo-label filtering approaches, which rely on fixed confidence thresholds, tend to be unreliable in fully unsupervised settings. In this work, we propose a novel adaptive pseudo-labeling framework that enhances CLIP's adaptation performance by integrating prototype consistency and neighborhood-based consistency. The proposed method comprises two key components: PICS, which assesses pseudo-label accuracy based on in-class feature compactness and cross-class feature separation; and NALR, which exploits semantic similarities among neighboring samples to refine pseudo-labels dynamically. Additionally, we introduce an adaptive weighting mechanism that adjusts the influence of pseudo-labeled samples during training according to their estimated correctness. Extensive experiments on 11 benchmark datasets demonstrate that our method achieves state-of-the-art performance in unsupervised adaptation scenarios, delivering more accurate pseudo-labels while maintaining computational efficiency.

Figures & tables

Fig. 1 : t-SNE visualization of visual embeddings for zero-shot CLIP [ 27 ] and ALPHA -FS on EuroSAT, showing textual (circle $\newmoon$ ) and visual (star $\bigstar$ ) prototypes. Metrics include in-class similarity, $\phi(x_{i})$ , and cross-class separation, $\psi(x_{i})$ . Image $x_{i}$ is near its visual prototype and distant from the most confident samples of other classes.

Fig. 2 : Overview of ALPHA , consisting of two components: (a) PICS : Assesses pseudo-label accuracy via in-class and cross-class metrics. In-class compactness measures similarity between image features and their class prototype, $\mu$ , while cross-class separation evaluates dissimilarity with other classes. (b) NALR : Enhances noisy pseudo-labels by enforcing neighbor consistency, aligning predictions with neighboring samples.

Fig. 3 : Comparison of clean sample counts (left) and test accuracy (right) for FixMatch-based threshold [ 29 ] (threshold=0.95), ALPHA -CS w/o NALR , and ALPHA -CS, evaluated on the EuroSAT dataset.

Zero-shot Methods
Method	Venue	Caltech	DTD	ESAT	Flower	OxPets	UCF	Cars	Food	CIFAR100	CUB	RES45	Avg
CLIP [ 27 ]	ICML’21	90.69	44.42	43.84	66.46	87.46	64.24	58.74	82.40	65.10	51.76	57.59	64.79
CuPL [ 26 ]	ICCV’23	93.92	50.11	50.06	68.05	87.16	66.96	60.80	84.03	65.20	49.71	61.14	67.01
UA Methods
UPL [ 14 ]	-	92.36	45.37	51.88	67.00	83.84	62.04	49.41	84.25	67.41	49.22	57.63	64.58
POUF [ 31 ]	ICML’23	94.10	46.10	62.90	40.00	87.80	61.20	57.70	82.10	62.00	51.59	66.40	64.72

TABLE I : Accuracy (%) of ALPHA compared to SOTA methods across 11 diverse datasets using the ViT-B/32 backbone.

Zero-shot Methods
Method	Caltech	DTD	ESAT	Flower	OxPets	UCF	Avg
Zero-shot CLIP [ 27 ]	90.69	44.42	43.84	66.46	87.46	64.24	66.19
CuPL [ 26 ]	93.92	50.11	50.06	68.05	87.16	66.96	69.38
ALPHA (UA Method)
ALPHA -B	93.48	54.04	61.12	71.78	90.30	68.41	73.19
ALPHA -CS w/o NALR	93.89	57.98	73.86	75.56	89.94	71.42	77.11

TABLE II : Ablation study on ALPHA components. PICS denotes P rototype-based I ntra-class and C ross-class S coring; NALR represents N eighbor-guided A daptive L abel R efinement. CS , RS , and FS indicate cross-class set selection strategies: C onfidence-based top-

k

S election, R andom S ampling, and F usion-based top-

k

S election, respectively. ALPHA -B is a simplified baseline model.

Fig. 4 : Comparison of pseudo-label accuracy (left) and test accuracy (right) on the EuroSAT dataset during training. The proposed methods ( ALPHA -CS, ALPHA -FS, and ALPHA -RS) exhibit greater robustness to confirmation bias than the ALPHA -B baseline.

Fig. 5 : Scalability analysis of ALPHA and DPA on the Flowers dataset, showing accuracy across dataset subsets (

20\%,40\%,60\%,80\%

, and

100\%

Zero-shot Methods
Method	Caltech	DTD	ESAT	Flower	OxPets	UCF	Avg
Zero-shot CLIP [ 27 ]	92.60	44.70	49.00	70.89	89.00	69.10	69.22
CuPL [ 26 ]	95.06	54.31	58.70	73.93	91.17	70.45	73.94
UA Methods
UPL [ 14 ]	95.14	45.90	55.36	73.93	87.98	67.43	70.96
POUF [ 31 ]	95.40	48.60	59.50	72.10	91.80	71.50	73.15

TABLE III : Accuracy (%) comparison of ALPHA against SOTA methods using CLIP-ViT-B/16.

Inductive
Method	Caltech	DTD	ESAT	Flower	OxPets	UCF	Avg
ReCLIP	95.94	53.88	70.80	72.63	87.49	67.01	74.63
ReCLIP *	94.79	55.64	70.64	75.88	90.35	70.64	76.32
DPA	95.94	55.96	79.94	75.56	90.11	66.69	77.37
ALPHA -CS	94.94	57.93	78.18	76.78	90.00	72.61	78.41
ALPHA -RS	94.34	58.67	78.18	76.29	89.92	72.46	78.31

TABLE IV: Accuracy (%) comparison of ALPHA , ReCLIP, and DPA across inductive and transductive settings.

Fig. 6 : Efficiency comparison of ALPHA -CS and SOTA methods on the EuroSAT dataset. Circle radius indicates the number of trainable parameters for each method.

ALPHA (LoRA [ 11 ] )
Method	Caltech	DTD	ESAT	Flower	OxPets	UCF101	Avg
ALPHA -CS	92.11	53.03	54.48	75.48	88.53	69.76	72.23
ALPHA -RS	91.08	54.84	48.76	75.19	89.75	69.68	71.55
ALPHA -FS	91.57	54.89	44.90	74.30	88.63	69.23	70.59
ALPHA (K-Adaptation [ 9 ] )
ALPHA -CS	94.70	58.09	76.80	76.90	89.92	72.03	78.07

TABLE V : Accuracy (%) comparison of CLIP fine-tuning methods on ALPHA , using ViT-B/32 as the backbone for all experiments.

Fig. 7 : Accuracy (%) comparison on CIFAR-10 with added unlabeled samples from unrelated CIFAR-100 classes, restricting target classes to CIFAR-10.

Fig. 8 : Impact of varying

k

and

k_{n}

on classification accuracy (%) for ALPHA -CS on the DTD and Flowers datasets.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation
Oct 20, 2026Hao Yan, Guanghui WangVLM Adaptation Unsupervised Domain Adaptation
Department of Computer Science, Toronto Metropolitan University, Toronto, ON, Canada
Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)
Jun 13, 2026Pengxiao Han, Changkun Ye, Yanshuo Wang +5Source-Free Domain Adaptation Zero-Shot Learning
Australian National University, Canberra, ACT, Australia · China North Vehicle Research Institute, Beijing, China · The Hong Kong Polytechnic University, Hong Kong SAR, China +3
Semi-Supervised Adaptation of Vision-Language Models for Image Classification
Aug 26, 2026Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi +4Remote Sensing Scene Classification VLM Adaptation
Fondazione Bruno Kessler, Via Sommarive 18, 38123, Povo TN, Italy · Applied Computer Science Department, College of Applied Computer Science, King Saud University, 11543, Riyadh, Saudi Arabia · Computer Engineering Department, College of Computer and Information Sciences, King Saud University, 11543, Riyadh, Saudi Arabia +2

Oct 20, 2026J/K move · Enter open · S save

Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation

Hao Yan, Guanghui Wang

Department of Computer Science, Toronto Metropolitan University, Toronto, ON, Canada

VLM Adaptation Unsupervised Domain Adaptation Pseudo-Labeling PEFT of VLMs

Open paper AskPDF arXiv

Pretrained Vision-Language Models (PVLMs) achieve strong zero-shot recognition but often suffer performance degradation under target-domain shift. Unsupervised adaptation provides a practical way to improve target-domain performance without labeled data, but it remains challenging because pseudo-labels generated by the model can be unreliable and no labeled validation set is available for model selection or early stopping. We propose a simple framework for unsupervised PVLM adaptation that combines progressive margin-based subset selection with cascaded consistency learning. The subset selection strategy constructs a class-balanced pseudo-labeled set using prediction margins and gradually expands it during training, enabling adaptation to start from reliable supervision anchors before incorporating broader target-domain data. Cascaded consistency learning further exploits unlabeled samples by using temporally smoothed teacher predictions on weakly augmented views to supervise student predictions on strongly augmented views, with confidence filtering to reduce the influence of uncertain pseudo-labels. To preserve the pretrained embedding structure, the framework updates only normalization parameters in the visual encoder and class prototypes initialized from text prompts. Experiments across multiple recognition benchmarks with both CLIP and SigLIP backbones show consistent improvements over zero-shot baselines and competitive performance against recent unsupervised PVLM adaptation methods. These results suggest that progressive pseudo-label selection and teacher-guided consistency provide an effective and lightweight strategy for adapting PVLM classifiers without labeled target data.

Abstract

VLM Adaptation

Explore similar work

Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation

Label Shift Aware Adaptation for Online Zero-shot Learning with Contrastive Language-Image Pre-Training (CLIP)

Semi-Supervised Adaptation of Vision-Language Models for Image Classification