PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

Seongjae Kang; Dong Bok Lee; Hyungjoon Jang; Dongseop Kim; Sung Ju Hwang

cs.LGJun 1, 2025

PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

Authors: Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Dongseop Kim, Sung Ju Hwang

Organizations: VUNO Inc. · KAIST

Abstract

Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs--i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves every selection method over the no-distillation baseline (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, several widely-used AL selectors underperform Random and none consistently beats it; in contrast, PCoreSet is the only selector that consistently outperforms Random across 5 students and 3 teachers, always achieving the best performance except for the first 2 AL rounds. Our code is available at https://github.com/erjui/PCoreSet.

Figures & tables

Figure 1: Overview of ActiveKD and PCoreSet . (Left) Our two contributions: ActiveKD , a framework that integrates KD from VLM teachers into pool-based active learning, and PCoreSet , a selection strategy that maximizes coverage in the probability space. (Top right) ActiveKD substantially improves every selection method over the no-distillation baseline, so ActiveKD is beneficial regardless of the active selector. (Bottom right) Under ActiveKD , (i) widely used AL selectors ( e.g. , Entropy, Coreset, BADGE) often underperform Random , (ii) no existing selector is a clear winner, and (iii) PCoreSet consistently outperforms all selectors across rounds . Since all methods share an identical training pipeline and differ only in data selection, this gain is significant by active-learning standards. Detailed numbers in Tab. 1 and Fig. 5 .

Figure 2: (Left): Teacher model prediction biases ( $(\mu_{1},r_{1}),\ldots,(\mu_{k},r_{k})$ ) are transferred to student models via distillation, where $\hat{\mu}_{k}=\mu_{k}+y_{c}$ and $\hat{r}_{k}=(1-\lambda)r_{k}$ ( $\mu$ denotes centroids, $r$ denotes radii). (Right): PCoreSet selects samples maximizing distance to labeled points in probability simplex $\Delta^{C-1}$ , uncovering underrepresented regions.

Figure 3

	ImageNet (ResNet-50 student & ResNet-50 teacher)			Avg. over 10 Datasets (ResNet-18 student & ResNet-50 teacher)
Methods	No Distill	ActiveKD (Zero-Shot)	ActiveKD (Few-Shot)	No Distill	ActiveKD (Zero-Shot)	ActiveKD (Few-Shot)
Random	33.36 $\pm$ 0.45	60.69 $\pm$ 0.16 (+27.33)	60.49 $\pm$ 0.12 (+27.13)	63.10 $\pm$ 4.33	76.31 $\pm$ 0.88 (+13.21)	77.48 $\pm$ 0.86 (+14.38)
Entropy	30.76 $\pm$ 0.24	60.43 $\pm$ 0.16 (+29.67)	58.87 $\pm$ 0.14 (+28.11)	64.06 $\pm$ 3.47	75.58 $\pm$ 0.86 (+11.52)	77.47 $\pm$ 0.90 (+13.41)
Coreset	26.61 $\pm$ 0.90	60.58 $\pm$ 0.07 (+33.97)	59.01 $\pm$ 0.42 (+32.40)	64.99 $\pm$ 3.01	76.20 $\pm$ 0.91 (+11.21)	77.63 $\pm$ 0.57 (+12.64)
BADGE	–	–	–	63.14 $\pm$ 4.66	76.18 $\pm$ 0.74 (+13.04)	77.50 $\pm$ 0.67 (+14.36)
ClassBalanced	34.44 $\pm$ 0.60	61.07 $\pm$ 0.11 (+26.63)	61.13 $\pm$ 0.07 (+26.69)	63.57 $\pm$ 4.33	76.28 $\pm$ 0.87 (+12.71)	77.56 $\pm$ 0.66 (+13.99)

Table 1: Final-round accuracy on ImageNet and the 10-dataset average under three AL frameworks. Mean and 95% CI over 5 seeds; gray values in parentheses denote the gain over No Distill. PCoreSet consistently exceeds Random under ActiveKD (+0.47 on ImageNet, +1.13 on the 10-dataset avg.).

Panel of Figure 4 — Figure 4: (Left): The proposed ActiveKD framework consistently improves the final-round accuracy of all selection methods , while PCoreSet further outperforms baselines when combined with ActiveKD. (Right): ActiveKD consistently improves PCoreSet across active learning rounds (No Distill vs. Zero-shot Distill), with further gains when using few-shot teachers (Few-shot Distill). The proposed ActiveKD + PCoreSet even surpasses VLM teachers, outperforming zero-shot teachers on the average of 10 datasets after the first AL round and few-shot teachers after the second .

Figure 5: Results on ImageNet and the average over 8 or 10 datasets across 5 student and 3 teacher architectures under ActiveKD (Zero-Shot) with either 8 or 16 rounds. Under ActiveKD, several widely-used selectors (Entropy, Coreset, BADGE) often underperform Random ; PCoreSet is the only selector that consistently outperforms Random , always achieving the best performance except for the first 2 AL rounds.

Figure 6: Left two panels: KNN loss (L2) and cluster purity of PCoreSet and other selection strategies across 10 datasets; right two panels: performance of few-shot teachers on 11 datasets across rounds.

Figure 7: Comparison of four selection criteria: 1) Uncertainty, 2) Class balance, 3) Feature space diversity, and 4) Probability space diversity. The results are averaged over the 10 datasets excluding ImageNet.

Figure 8: The heatmap of probability vectors in the first AL round using the DTD dataset. See Section F.1 for results on other datasets.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Active Learning on ImageNet
Model Configuration	Training Details
• Student: DINO self-supervised ResNet-50 [ 70 ] • Teacher: CLIP ResNet-50 [ 9 ] • Active learning rounds: 8 • Initial setting: 1-shot (single image per class) • Query size: 1,000 • Unlabeled pool: 100,000 samples randomly selected with seed • Random seeds: 5 different seeds • KD parameters: $\zeta=0.01$ , $\eta=2$ , $\lambda=0.5$ • DHO parameters: $\alpha=0.4$ , $\beta=0.5$ • Validation: No validation split used	• Epochs: 20 for first 7 rounds, 50 for final round • Optimizer: AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ ) • Learning rate: $1\times 10^{-3}$ • Weight decay: $1\times 10^{-2}$ • Batch size: 512 (labeled: 256, unlabeled: 256)
Active Learning on 10 Additional Datasets
Model Configuration	Training Details
• Student: ImageNet pre-trained ResNet-18 [ 7 ] , MobileNetV2 [ 97 ] , and ImageNet-21k [ 104 ] pre-trained ViT-Tiny [ 98 ] • Teacher: CLIP ResNet-50 [ 9 ] • Active learning rounds: 16 • Initial setting: 1-shot (single image per class) • Query size: Equal to number of classes per round • Unlabeled pool: All training samples except labeled set • Random seeds: 5 different seeds • KD parameters: $\zeta=0.01$ , $\eta=2$ , $\lambda=0.5$ • DHO parameters: $\alpha=0.5$ , $\beta=1$ • Validation: No validation split used	• Epochs: 200 for all rounds • Optimizer: AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ ) • Learning rate: $1\times 10^{-3}$ • Weight decay: $1\times 10^{-2}$ • Batch size: 128 (labeled: 64, unlabeled: 64)
Few-Shot Teacher Active Distillation

Appendix

Table 2: Implementation details for our experiments across different settings.

Dataset	#Classes	#Train	#Val	#Test	Domain
Caltech101 [ 88 ]	100	4,128	1,649	2,465	General objects
DTD [ 94 ]	47	2,820	1,128	1,692	Textures
EuroSAT [ 95 ]	10	13,500	5,400	8,100	Satellite
FGVCAircraft [ 91 ]	100	3,334	3,333	3,333	Aircraft
Food101 [ 92 ]	101	50,500	20,200	30,300	Food
Flowers102 [ 90 ]	102	4,093	1,633	2,463	Plants

Appendix

Table 3: Overview of datasets used in our experiments. Note that validation split is not used during the active learning process, and is only shown for completeness.

Dataset	Mean	Min	Max	Std	Total
Caltech101 [ 88 ]	41.28	16	400	56.81	4,128
DTD [ 94 ]	60.00	60	60	0.00	2,820
EuroSAT [ 95 ]	1,350.00	1,000	1,500	174.80	13,500
FGVCAircraft [ 91 ]	33.34	33	34	0.48	3,334
Food101 [ 92 ]	500.00	500	500	0.00	50,500
Flowers102 [ 90 ]	40.13	20	129	22.20	4,093

Appendix

Table 4: Class balance of datasets used in our experiments. The total number specified in the table is the number of samples of training splits as we do active selection on them.

Figure 9: Results on 10 datasets using ResNet-18 with ResNet-50 teacher under ActiveKD (Zero-Shot).

Figure 10: Results on 8 datasets using MobileNetV2 under zero-shot distillation across 16 rounds.

Figure 11: Results on 11 datasets including average of 10 datasets using ResNet-18 without distillation.

Figure 12: Results of ResNet-18 student on 11 datasets including average of 10 datasets under few-shot distillation.

Figure 13: Results of ResNet-50 few-shot teacher on 11 datasets including average of 10 datasets under few-shot distillation.

Strategy	LA	Active Learning Round
Strategy	LA	2	3	4	5	6	7	8
Random	–	65.1	67.4	69.5	71.3	72.4	73.9	74.5
	Hard	55.1 (-10.1)	59.9 (-7.4)	63.1 (-6.5)	66.0 (-5.3)	67.5 (-5.0)	69.4 (-4.5)	70.6 (-3.9)
	Soft	64.4 (-0.7)	67.6 (+0.2)	69.9 (+0.3)	71.5 (+0.2)	72.8 (+0.4)	74.0 (+0.1)	75.0 (+0.5)
Coreset	–	65.7	67.7	69.8	71.5	72.4	73.5	74.8
	Hard	55.5 (-10.2)	60.6 (-7.1)	63.6 (-6.2)	66.6 (-4.9)	68.8 (-3.6)	70.1 (-3.4)	71.9 (-2.9)

Appendix

Table 5: Logit adjustment (Hard / Soft) under ActiveKD on 8 datasets. Mean accuracy across rounds; gray values in parentheses denote the difference from the no-LA baseline. Bold marks the best per round.

Method	KD	Round
		2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Random	No Distill	36.7	42.6	47.6	50.9	53.8	56.7	59.2	60.1	62.5	64.1	65.6	66.5	67.6	68.9	69.5
	ActiveKD (w/o unlabeled)	43.3	49.6	55.2	58.8	61.9	63.9	66.2	67.7	69.3	70.7	71.9	72.8	73.9	74.2	75.0
	ActiveKD	64.5	66.8	69.0	70.8	72.0	73.5	74.1	74.8	75.8	76.2	76.9	77.4	77.6	78.0	78.7
Entropy	No Distill	34.8	40.6	46.4	51.0	54.8	57.2	60.1	61.4	63.1	64.5	66.3	67.6	69.2	70.3	71.3
	ActiveKD (w/o unlabeled)	41.1	48.6	55.8	57.5	61.2	65.1	66.9	68.7	69.8	71.0	72.8	74.1	74.8	76.3	76.7

Appendix

Table 6: Results on 8 datasets (average) when excluding unlabeled data from KD (denoted as w/o unlabeled). Bold marks the best selector per round within each KD configuration.

Figure 14: The heatmap of output probability vectors from different selection strategies in the first active learning round using 10 datasets.

Figure 15: Visualization of samples selected by PCoreSet for Caltech101 dataset.

Figure 16: Visualization of samples selected by PCoreSet for DTD dataset.

Figure 17: Visualization of samples selected by PCoreSet for EuroSAT dataset.

Figure 18: Visualization of samples selected by PCoreSet for Flowers102 dataset.

Figure 19: Visualization of samples selected by PCoreSet for Food-101 dataset.

Figure 20: Visualization of samples selected by PCoreSet for FGVC dataset.

Figure 21: Visualization of samples selected by PCoreSet for Oxford-IIIT Pets dataset.

Figure 22: Visualization of samples selected by PCoreSet for Stanford Cars dataset.

Figure 23: Visualization of samples selected by PCoreSet for SUN397 dataset.

Figure 24: Visualization of samples selected by PCoreSet for UCF101 dataset.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 296 papers

May 1, 2026

Leveraging Vision-Language Models as Weak Annotators in Active Learning

Active learning aims to reduce annotation cost by selectively querying informative samples for supervision under a limited labeling budget. In this work, we investigate how vision-language models (VLMs) can be leveraged to further reduce the reliance on costly human annotation within the active learning paradigm. To this end, we find that the reliability of VLMs varies significantly with label granularity in fine-grained recognition tasks: they perform poorly on fine-grained labels but can provide accurate coarse-grained labels. Leveraging this property, we propose an active learning framework that combines fine-grained human annotations with coarse-grained VLM-generated weak labels through instance-wise label assignment. We further model the systematic noise in VLM-generated labels using a small set of trusted full labels. Experiments on CUB200 and FGVC-Aircraft show that the proposed framework consistently outperforms existing active learning methods under the same annotation budget.

Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise +2

Kyushu University, Japan.

Vision-Language Models Active Learning Fine-Grained Image Classification Image Classification

Ask PDF

Mar 24, 2026

Conformal Cross-Modal Active Learning

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.

Huy Hoang Nguyen, Cédric Jung, Shirin Salehi +3

AIT Austrian Institute of Technology · Automation & Control Institute, Technical University of Vienna · Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University

Active Learning Conformal Prediction

Ask PDF

May 21, 2026

Balancing Uncertainty and Diversity of Samples: Leveraging Diversity of Least, High Confidence Samples for Effective Active Learning

Deep learning models, including Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), have achieved state-of-the-art performance on various computer vision tasks such as object classification, detection, segmentation, generation, and many more. However, these models are data-hungry as they require more training data to learn millions or billions of parameters. Especially for supervised learning tasks, curating a large number of labeled samples for model training is an expensive and time-consuming task. Active Learning (AL) has been used to address this problem for many years. Existing active learning methods aim at choosing the samples for annotation from a pool of unlabeled samples that are either diverse or uncertain. Choosing such samples may hinder the model's performance as we pool based on one dimension, i.e., either diverse or uncertain. In this paper, we propose four novel hybrid sampling methods for pooling both easy and hard samples, which are also diverse. To verify the efficacy of the proposed methods, extensive experiments are conducted using high and low-confidence samples separately. We observe from our experiments that the proposed hybrid sampling method, Least Confident and Diverse (LCD), consistently performs better compared to state-of-the-art methods. It is observed that selecting uncertain and diverse instances helps the model learn more distinct features. The codes related to this study will be available at https://github.com/XXX/LCD.

Vipul Arya, S. H. Shabbeer Basha, Srikrishna U N +2

School of Computer Science and Engineering, RV University, Bengaluru, Karnataka-560059, India · School of Engineering & Technology, Vidyashilp University, Bangalore, India · Shiv Nadar Institution of Eminence, Delhi NCR

Active Learning Uncertainty Sampling Data-Efficient Learning

Ask PDF

Abstract

Active Learning

Explore similar work

Leveraging Vision-Language Models as Weak Annotators in Active Learning

Conformal Cross-Modal Active Learning

Balancing Uncertainty and Diversity of Samples: Leveraging Diversity of Least, High Confidence Samples for Effective Active Learning