cs.CVOct 20, 2026

Progressive Subset Selection and Cascaded Consistency for Unsupervised Vision-Language Model Adaptation

Authors: Hao Yan, Guanghui Wang

Organizations: Department of Computer Science, Toronto Metropolitan University, Toronto, ON, Canada

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Pretrained Vision-Language Models (PVLMs) achieve strong zero-shot recognition but often suffer performance degradation under target-domain shift. Unsupervised adaptation provides a practical way to improve target-domain performance without labeled data, but it remains challenging because pseudo-labels generated by the model can be unreliable and no labeled validation set is available for model selection or early stopping. We propose a simple framework for unsupervised PVLM adaptation that combines progressive margin-based subset selection with cascaded consistency learning. The subset selection strategy constructs a class-balanced pseudo-labeled set using prediction margins and gradually expands it during training, enabling adaptation to start from reliable supervision anchors before incorporating broader target-domain data. Cascaded consistency learning further exploits unlabeled samples by using temporally smoothed teacher predictions on weakly augmented views to supervise student predictions on strongly augmented views, with confidence filtering to reduce the influence of uncertain pseudo-labels. To preserve the pretrained embedding structure, the framework updates only normalization parameters in the visual encoder and class prototypes initialized from text prompts. Experiments across multiple recognition benchmarks with both CLIP and SigLIP backbones show consistent improvements over zero-shot baselines and competitive performance against recent unsupervised PVLM adaptation methods. These results suggest that progressive pseudo-label selection and teacher-guided consistency provide an effective and lightweight strategy for adapting PVLM classifiers without labeled target data.

Figures & tables

Figure 1: Overview of the proposed unsupervised vision-language model adaptation framework. The teacher model selects a class-balanced pseudo-labeled subset based on prediction margins. The student is trained with supervised learning on the selected subset and cascaded consistency learning on unlabeled data via weak and strong augmentations. The teacher parameters are updated as an exponential moving average of the student parameters, producing stabilized pseudo-labels that progressively improve subset selection and consistency learning.
MethodsFGVCAircraftDTDCIFAR100Food101OxfordPetsCIFAR10
CLIP ( Radford et al., 2021 )18.944.462.383.387.588.3
UPL ( Huang et al., 2022 )17.145.467.484.383.891.3
POUF ( Tanwisuth et al., 2023 )19.847.068.984.089.592.5
LaFTer ( Mirza et al., 2023 )19.950.369.882.584.994.6
DPA ( Ali et al., 2025 )20.755.776.584.890.796.0
Ours22.957.774.184.890.396.0
Table 1: Comparison with existing unsupervised PVLM adaptation methods using CLIP ViT-B/32 . Accuracy (%) on six benchmark datasets. Baseline adaptation results are taken from published DPA comparisons.
MethodsFGVCAircraftDTDCIFAR100Food101OxfordPetsCIFAR10
CLIP ( Radford et al., 2021 )24.943.966.687.789.090.1
UPL ( Huang et al., 2022 )22.545.970.4–88.092.9
POUF ( Tanwisuth et al., 2023 )24.448.666.3–91.892.3
LaFTer ( Mirza et al., 2023 )22.454.876.3–85.396.5
DPA ( Ali et al., 2025 )25.150.778.1–93.296.4
Ours26.253.476.189.393.597.2
Table 2: Comparison with existing unsupervised PVLM adaptation methods using CLIP ViT-B/16 . Accuracy (%) on the same six benchmark datasets. Food101 for ViT-B/16 is not reported by some baselines
MethodFGVCAircraftDTDCIFAR100Food101OxfordPetsCIFAR10Avg
SigLIP ( Zhai et al., 2023 )43.963.470.889.494.292.575.7
Ours47.467.982.991.695.297.680.4 (+4.7)
Table 3: Additional results using SigLIP ViT-B/16. Accuracy (%) on six benchmark datasets.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Subset Schedule ( KK )Threshold ( τ\tau )FGVCAircraftDTDOxfordPets
Fixed0.950.9521.455.889.3
Progressive0.950.9521.357.189.4
Progressive0.95→0.900.95\rightarrow 0.9021.858.489.9
Progressive0.98→0.950.98\rightarrow 0.9522.957.790.3
Appendix
Table 4: Ablation study on the progressive subset expansion schedule ( KK ) and the confidence threshold schedule ( τ\tau ) in cascaded consistency learning. Accuracy is reported in %.
Figure 2: Training stability of the proposed method. Test accuracy is shown across training epochs starting from the zero-shot initialization for CLIP ViT-B/32 and CLIP ViT-B/16 on DTD and CIFAR100.
Figure 3: Training stability of the proposed method using SigLIP ViT-B/16. Test accuracy across training epochs starting from the zero-shot initialization on DTD, CIFAR100, and CIFAR10.
More in this topic

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Explore similar work

CardsList
Jul 22, 2025

RCPN: towards robust pseudo-labels in unsupervised CLIP adaptation via consistent prototypes and neighborhoods

In unsupervised adaptation for vision-language models such as CLIP, pseudo-labels derived from zero-shot predictions often exhibit significant noise, particularly under domain shifts or in visually complex scenarios. Conventional pseudo-label filtering approaches, which rely on fixed confidence thresholds, tend to be unreliable in fully unsupervised settings. In this work, we propose a novel adaptive pseudo-labeling framework that enhances CLIP's adaptation performance by integrating prototype consistency and neighborhood-based consistency. The proposed method comprises two key components: PICS, which assesses pseudo-label accuracy based on in-class feature compactness and cross-class feature separation; and NALR, which exploits semantic similarities among neighboring samples to refine pseudo-labels dynamically. Additionally, we introduce an adaptive weighting mechanism that adjusts the influence of pseudo-labeled samples during training according to their estimated correctness. Extensive experiments on 11 benchmark datasets demonstrate that our method achieves state-of-the-art performance in unsupervised adaptation scenarios, delivering more accurate pseudo-labels while maintaining computational efficiency.

Eman Ali, Chetan Arora, Muhammad Haris Khan

Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE. · Alexandria University, Alexandria, Egypt. · Indian Institute of Technology Delhi, New Delhi, India.

VLM AdaptationPseudo-LabelingUnsupervised Domain Adaptation
AskPDF
Oct 15, 2025

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.

Dominick Reilly, Manish Kumar Govind, Le Xue +1

University of North Carolina at Charlotte · Elorian AI

Domain AdaptationVLM AdaptationVLMs for RoboticsParameter-Efficient Fine-Tuning
AskPDF
Oct 6, 2026

Unsupervised Long-Tailed Adaptation of Vision-Language Models

Adapting vision-language models to downstream tasks has achieved remarkable success by leveraging pseudo-labels generated from unlabeled data. Existing methods typically assume a uniform unlabeled data distribution, and thus the resulting pseudo-label distribution is likewise uniform. However, real-world data distributions are often long-tailed. To tackle this, we formalize a new scenario termed Unsupervised Long-Tailed Adaptation (ULTA). Under this scenario, existing methods exhibit a contrasting phenomenon: head-class performance drops sharply, which is distinct from supervised long-tailed learning where tail classes suffer the most. In particular, we uncover that the distributional mismatch not only erodes head-class boundaries, but also pushes head samples into confusable classes, reinforcing the model's inherent bias. To address these issues, we propose a novel model called Margin-Aware Refinement with Structural alignment (MARS). Specifically, we mitigate head-class boundary erosion via Boundary-Preserving Alignment, which takes the zero-shot VLM as a fixed visual reference to suppress probability increases that lack visual support in the training targets. Building upon this, we introduce Margin-aware Self-Refinement, which employs a dynamic adjustment strategy to refine tail and confusable classes while preventing prediction bias. Extensive experiments on nine benchmark datasets demonstrate that MARS outperforms state-of-the-art methods, achieving an average accuracy improvement of 4.71 percentage points.

Keliang Chen, Yaxin Hou, Hui Liu +1

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · School of Computing and Information Sciences, Saint Francis University, Hong Kong, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

Long-Tail LearningVLM AdaptationPseudo-Labeling
AskPDF