Semi-Supervised Adaptation of Vision-Language Models for Image Classification
Authors: Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi, Salah E. Khenfer, Mingdeng Shi, Hua Zou, Mansour Zuair
Organizations: Fondazione Bruno Kessler, Via Sommarive 18, 38123, Povo TN, Italy · Applied Computer Science Department, College of Applied Computer Science, King Saud University, 11543, Riyadh, Saudi Arabia · Computer Engineering Department, College of Computer and Information Sciences, King Saud University, 11543, Riyadh, Saudi Arabia · Key Laboratory of Tarim Oasis Agriculture, College of Information Engineering, Tarim University, Aral, China · School of Computer Science, Wuhan University, 430072, Wuhan, China
Vision-language models like CLIP have shown sig- nificant potential in handling natural images, yet their perfor- mance is often limited by the distinct characteristics of satellite imagery. While parameter-efficient adaptation techniques exist, their efficacy is frequently limited by the scarcity of annotated samples. In this letter, we propose Self-Evolutionary CLIP (SE- CLIP), a semi-supervised framework designed for recursive label mining in scene classification. The approach follows a dual-phase pipeline, where an initial warm-up on a few annotated seeds is followed by a recursive discovery phase that iteratively identifies high-confidence samples from unlabeled pools. To maintain the integrity of the evolving support set, we employ a class-balanced selection strategy that prevents the model from being dominated by easily learned categories. Results on the UCM and NWPU benchmarks indicate that SE-CLIP significantly outperforms existing semi-supervised approaches. The framework provides a viable solution for adapting VLMs to the remote sensing domain with minimal human intervention.
Figures & tables
Fig. 1 : Overview of the Low-Rank Adaptation (LoRA) mechanism. Trainable matrices A and B are integrated into the frozen Transformer architecture to efficiently specialize the foundation model for remote sensing scene classification.
Fig. 2 : The SE-CLIP recursive discovery pipeline. The model iteratively synchronizes visual-textual representations on labeled seeds before mining the most similar unlabeled samples to expand the support set.
Dataset
Top- k
1
2
5
7
UCM
98.81 ± 0.69
98.62 ± 0.39
98.62 ± 0.49
98.38 ± 0.78
NWPU
88.56 ± 1.32
91.52 ± 0.48
94.92 ± 0.31
94.77 ± 0.48
TABLE I : SCORES (%) ON UCM AND NWPU DATASETS WITH RANK r=4 ACROSS VARIOUS k VALUES
Dataset
LoRA Rank ( r )
2
4
16
UCM
98.71 ± 0.27
98.81 ± 0.69
99.09 ± 0.54
NWPU
94.41 ± 0.34
94.92 ± 0.31
95.07 ± 0.95
TABLE II : SCORES (%) ON UCM AND NWPU DATASETS ACROSS VARIOUS LoRA RANK ( r ) CONFIGURATIONS
Configuration
UCM
NWPU
Full recursive discovery
99.09 ± 0.54
95.07 ± 0.95
Warm-up only
87.81 ± 2.50
76.34 ± 0.87
TABLE III : ABLATION STUDY COMPARING THE FULL RECURSIVE DISCOVERY MECHANISM AGAINST WARM-UP ONLY
Configuration
UCM
NWPU
Per-class balanced mining
98.62 ± 0.49
94.92 ± 0.31
Global mining
83.19 ± 2.55
42.33 ± 4.69
TABLE IV : ABLATION STUDY COMPARING PER-CLASS BALANCED MINING AGAINST GLOBAL MINING
Fig. 3 : Cumulative pseudo-label standard deviation across iterations for global vs. local selection on UCM and NWPU.
Metric Description
Value / Footprint
Hardware environment
NVIDIA Tesla V100 GPU (32 GB)
Total parameters
152.54 M
Trainable parameters ( r=16 )
1,769,472 (1.16%)
GPU memory consumption
1.12 GB (batch size = 16)
FLOPs
2.95 GFLOPs/image
UCM total training time
26,933.52 sec (7.48 hours)
TABLE V : COMPUTATIONAL EFFICIENCY ANALYSIS
Fig. 4 : Accuracy (%) evolution of SE-CLIP across 200 iterations on the UCM and NWPU benchmarks. The grey dashed vertical line marks the mining onset.
Fig. 5 : Qualitative t-SNE visualization [ 15 ] of the visual embedding space comparing the pre-trained zero-shot CLIP baseline (left panels) against our proposed SE-CLIP method (right panels) across all categories for both UCM (top) and NWPU (bottom) datasets. The proposed SE-CLIP successfully adapts and reorganizes the visual embeddings into tightly bound, well-separated semantic clusters. Distinct marker shapes and color schemes are assigned uniquely to each category to highlight cluster separability. Best viewed in color.
Method
UCM (%)
NWPU (%)
Flexmatch [ 3 ]
82.54
69.16
MSmatch [ 4 ]
85.70
60.00
RSmatch [ 5 ]
84.10
58.30
Freematch [ 6 ]
80.71
69.10
Softmatch [ 7 ]
87.38
70.09
Fixmatch [ 8 ]
84.84
65.89
TABLE VI : COMPARISON WITH STATE-OF-THE-ART SEMI-SUPERVISED METHODS
School of Engineering and Applied Science, Washington State University, Richland, WA 99354, USA · Institute for Software Technology, German Aerospace Center (DLR), Porz, 51147 Cologne, Germany