PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

Seongjae Kang; Dong Bok Lee; Hyungjoon Jang; Dongseop Kim; Sung Ju Hwang

cs.LGJun 1, 2025

PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

Authors: Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Dongseop Kim, Sung Ju Hwang

Organizations: VUNO Inc. · KAIST

Abstract

Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by transferring the knowledge from teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we first introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs--i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects probabilistically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Extensive evaluations on 11 datasets show that ActiveKD consistently improves every selection method over the no-distillation baseline (e.g., +29.07% on ImageNet, averaged over methods). Under ActiveKD, several widely-used AL selectors underperform Random and none consistently beats it; in contrast, PCoreSet is the only selector that consistently outperforms Random across 5 students and 3 teachers, always achieving the best performance except for the first 2 AL rounds. Our code is available at https://github.com/erjui/PCoreSet.

Figures & tables

Figure 1: Overview of ActiveKD and PCoreSet . (Left) Our two contributions: ActiveKD , a framework that integrates KD from VLM teachers into pool-based active learning, and PCoreSet , a selection strategy that maximizes coverage in the probability space. (Top right) ActiveKD substantially improves every selection method over the no-distillation baseline, so ActiveKD is beneficial regardless of the active selector. (Bottom right) Under ActiveKD , (i) widely used AL selectors ( e.g. , Entropy, Coreset, BADGE) often underperform Random , (ii) no existing selector is a clear winner, and (iii) PCoreSet consistently outperforms all selectors across rounds . Since all methods share an identical training pipeline and differ only in data selection, this gain is significant by active-learning standards. Detailed numbers in Tab. 1 and Fig. 5 .

Figure 2: (Left): Teacher model prediction biases ( $(\mu_{1},r_{1}),\ldots,(\mu_{k},r_{k})$ ) are transferred to student models via distillation, where $\hat{\mu}_{k}=\mu_{k}+y_{c}$ and $\hat{r}_{k}=(1-\lambda)r_{k}$ ( $\mu$ denotes centroids, $r$ denotes radii). (Right): PCoreSet selects samples maximizing distance to labeled points in probability simplex $\Delta^{C-1}$ , uncovering underrepresented regions.

Figure 3

	ImageNet (ResNet-50 student & ResNet-50 teacher)			Avg. over 10 Datasets (ResNet-18 student & ResNet-50 teacher)
Methods	No Distill	ActiveKD (Zero-Shot)	ActiveKD (Few-Shot)	No Distill	ActiveKD (Zero-Shot)	ActiveKD (Few-Shot)
Random	33.36 $\pm$ 0.45	60.69 $\pm$ 0.16 (+27.33)	60.49 $\pm$ 0.12 (+27.13)	63.10 $\pm$ 4.33	76.31 $\pm$ 0.88 (+13.21)	77.48 $\pm$ 0.86 (+14.38)
Entropy	30.76 $\pm$ 0.24	60.43 $\pm$ 0.16 (+29.67)	58.87 $\pm$ 0.14 (+28.11)	64.06 $\pm$ 3.47	75.58 $\pm$ 0.86 (+11.52)	77.47 $\pm$ 0.90 (+13.41)
Coreset	26.61 $\pm$ 0.90	60.58 $\pm$ 0.07 (+33.97)	59.01 $\pm$ 0.42 (+32.40)	64.99 $\pm$ 3.01	76.20 $\pm$ 0.91 (+11.21)	77.63 $\pm$ 0.57 (+12.64)
BADGE	–	–	–	63.14 $\pm$ 4.66	76.18 $\pm$ 0.74 (+13.04)	77.50 $\pm$ 0.67 (+14.36)
ClassBalanced	34.44 $\pm$ 0.60	61.07 $\pm$ 0.11 (+26.63)	61.13 $\pm$ 0.07 (+26.69)	63.57 $\pm$ 4.33	76.28 $\pm$ 0.87 (+12.71)	77.56 $\pm$ 0.66 (+13.99)

Table 1: Final-round accuracy on ImageNet and the 10-dataset average under three AL frameworks. Mean and 95% CI over 5 seeds; gray values in parentheses denote the gain over No Distill. PCoreSet consistently exceeds Random under ActiveKD (+0.47 on ImageNet, +1.13 on the 10-dataset avg.).

Panel of Figure 4 — Figure 4: (Left): The proposed ActiveKD framework consistently improves the final-round accuracy of all selection methods , while PCoreSet further outperforms baselines when combined with ActiveKD. (Right): ActiveKD consistently improves PCoreSet across active learning rounds (No Distill vs. Zero-shot Distill), with further gains when using few-shot teachers (Few-shot Distill). The proposed ActiveKD + PCoreSet even surpasses VLM teachers, outperforming zero-shot teachers on the average of 10 datasets after the first AL round and few-shot teachers after the second .

Figure 5: Results on ImageNet and the average over 8 or 10 datasets across 5 student and 3 teacher architectures under ActiveKD (Zero-Shot) with either 8 or 16 rounds. Under ActiveKD, several widely-used selectors (Entropy, Coreset, BADGE) often underperform Random ; PCoreSet is the only selector that consistently outperforms Random , always achieving the best performance except for the first 2 AL rounds.

Figure 6: Left two panels: KNN loss (L2) and cluster purity of PCoreSet and other selection strategies across 10 datasets; right two panels: performance of few-shot teachers on 11 datasets across rounds.

Figure 7: Comparison of four selection criteria: 1) Uncertainty, 2) Class balance, 3) Feature space diversity, and 4) Probability space diversity. The results are averaged over the 10 datasets excluding ImageNet.

Figure 8: The heatmap of probability vectors in the first AL round using the DTD dataset. See Section F.1 for results on other datasets.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Active Learning on ImageNet
Model Configuration	Training Details
• Student: DINO self-supervised ResNet-50 [ 70 ] • Teacher: CLIP ResNet-50 [ 9 ] • Active learning rounds: 8 • Initial setting: 1-shot (single image per class) • Query size: 1,000 • Unlabeled pool: 100,000 samples randomly selected with seed • Random seeds: 5 different seeds • KD parameters: $\zeta=0.01$ , $\eta=2$ , $\lambda=0.5$ • DHO parameters: $\alpha=0.4$ , $\beta=0.5$ • Validation: No validation split used	• Epochs: 20 for first 7 rounds, 50 for final round • Optimizer: AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ ) • Learning rate: $1\times 10^{-3}$ • Weight decay: $1\times 10^{-2}$ • Batch size: 512 (labeled: 256, unlabeled: 256)
Active Learning on 10 Additional Datasets
Model Configuration	Training Details
• Student: ImageNet pre-trained ResNet-18 [ 7 ] , MobileNetV2 [ 97 ] , and ImageNet-21k [ 104 ] pre-trained ViT-Tiny [ 98 ] • Teacher: CLIP ResNet-50 [ 9 ] • Active learning rounds: 16 • Initial setting: 1-shot (single image per class) • Query size: Equal to number of classes per round • Unlabeled pool: All training samples except labeled set • Random seeds: 5 different seeds • KD parameters: $\zeta=0.01$ , $\eta=2$ , $\lambda=0.5$ • DHO parameters: $\alpha=0.5$ , $\beta=1$ • Validation: No validation split used	• Epochs: 200 for all rounds • Optimizer: AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ ) • Learning rate: $1\times 10^{-3}$ • Weight decay: $1\times 10^{-2}$ • Batch size: 128 (labeled: 64, unlabeled: 64)
Few-Shot Teacher Active Distillation

Appendix

Table 2: Implementation details for our experiments across different settings.

Dataset	#Classes	#Train	#Val	#Test	Domain
Caltech101 [ 88 ]	100	4,128	1,649	2,465	General objects
DTD [ 94 ]	47	2,820	1,128	1,692	Textures
EuroSAT [ 95 ]	10	13,500	5,400	8,100	Satellite
FGVCAircraft [ 91 ]	100	3,334	3,333	3,333	Aircraft
Food101 [ 92 ]	101	50,500	20,200	30,300	Food
Flowers102 [ 90 ]	102	4,093	1,633	2,463	Plants

Appendix

Table 3: Overview of datasets used in our experiments. Note that validation split is not used during the active learning process, and is only shown for completeness.

Dataset	Mean	Min	Max	Std	Total
Caltech101 [ 88 ]	41.28	16	400	56.81	4,128
DTD [ 94 ]	60.00	60	60	0.00	2,820
EuroSAT [ 95 ]	1,350.00	1,000	1,500	174.80	13,500
FGVCAircraft [ 91 ]	33.34	33	34	0.48	3,334
Food101 [ 92 ]	500.00	500	500	0.00	50,500
Flowers102 [ 90 ]	40.13	20	129	22.20	4,093

Appendix

Table 4: Class balance of datasets used in our experiments. The total number specified in the table is the number of samples of training splits as we do active selection on them.

Figure 9: Results on 10 datasets using ResNet-18 with ResNet-50 teacher under ActiveKD (Zero-Shot).

Figure 10: Results on 8 datasets using MobileNetV2 under zero-shot distillation across 16 rounds.

Figure 11: Results on 11 datasets including average of 10 datasets using ResNet-18 without distillation.

Figure 12: Results of ResNet-18 student on 11 datasets including average of 10 datasets under few-shot distillation.

Figure 13: Results of ResNet-50 few-shot teacher on 11 datasets including average of 10 datasets under few-shot distillation.

Strategy	LA	Active Learning Round
Strategy	LA	2	3	4	5	6	7	8
Random	–	65.1	67.4	69.5	71.3	72.4	73.9	74.5
	Hard	55.1 (-10.1)	59.9 (-7.4)	63.1 (-6.5)	66.0 (-5.3)	67.5 (-5.0)	69.4 (-4.5)	70.6 (-3.9)
	Soft	64.4 (-0.7)	67.6 (+0.2)	69.9 (+0.3)	71.5 (+0.2)	72.8 (+0.4)	74.0 (+0.1)	75.0 (+0.5)
Coreset	–	65.7	67.7	69.8	71.5	72.4	73.5	74.8
	Hard	55.5 (-10.2)	60.6 (-7.1)	63.6 (-6.2)	66.6 (-4.9)	68.8 (-3.6)	70.1 (-3.4)	71.9 (-2.9)

Appendix

Table 5: Logit adjustment (Hard / Soft) under ActiveKD on 8 datasets. Mean accuracy across rounds; gray values in parentheses denote the difference from the no-LA baseline. Bold marks the best per round.

Method	KD	Round
		2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Random	No Distill	36.7	42.6	47.6	50.9	53.8	56.7	59.2	60.1	62.5	64.1	65.6	66.5	67.6	68.9	69.5
	ActiveKD (w/o unlabeled)	43.3	49.6	55.2	58.8	61.9	63.9	66.2	67.7	69.3	70.7	71.9	72.8	73.9	74.2	75.0
	ActiveKD	64.5	66.8	69.0	70.8	72.0	73.5	74.1	74.8	75.8	76.2	76.9	77.4	77.6	78.0	78.7
Entropy	No Distill	34.8	40.6	46.4	51.0	54.8	57.2	60.1	61.4	63.1	64.5	66.3	67.6	69.2	70.3	71.3
	ActiveKD (w/o unlabeled)	41.1	48.6	55.8	57.5	61.2	65.1	66.9	68.7	69.8	71.0	72.8	74.1	74.8	76.3	76.7

Appendix

Table 6: Results on 8 datasets (average) when excluding unlabeled data from KD (denoted as w/o unlabeled). Bold marks the best selector per round within each KD configuration.

Figure 14: The heatmap of output probability vectors from different selection strategies in the first active learning round using 10 datasets.

Figure 15: Visualization of samples selected by PCoreSet for Caltech101 dataset.

Figure 16: Visualization of samples selected by PCoreSet for DTD dataset.

Figure 17: Visualization of samples selected by PCoreSet for EuroSAT dataset.

Figure 18: Visualization of samples selected by PCoreSet for Flowers102 dataset.

Figure 19: Visualization of samples selected by PCoreSet for Food-101 dataset.

Figure 20: Visualization of samples selected by PCoreSet for FGVC dataset.

Figure 21: Visualization of samples selected by PCoreSet for Oxford-IIIT Pets dataset.

Figure 22: Visualization of samples selected by PCoreSet for Stanford Cars dataset.

Figure 23: Visualization of samples selected by PCoreSet for SUN397 dataset.

Figure 24: Visualization of samples selected by PCoreSet for UCF101 dataset.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 296 papers

Leveraging Vision-Language Models as Weak Annotators in Active Learning
May 1, 2026Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise +2Vision-Language Models Active Learning
Kyushu University, Japan.
Conformal Cross-Modal Active Learning
Mar 24, 2026Huy Hoang Nguyen, Cédric Jung, Shirin Salehi +3Active Learning Conformal Prediction
AIT Austrian Institute of Technology · Automation & Control Institute, Technical University of Vienna · Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University
Balancing Uncertainty and Diversity of Samples: Leveraging Diversity of Least, High Confidence Samples for Effective Active Learning
May 21, 2026Vipul Arya, S. H. Shabbeer Basha, Srikrishna U N +2Active Learning Uncertainty Sampling
School of Computer Science and Engineering, RV University, Bengaluru, Karnataka-560059, India · School of Engineering & Technology, Vidyashilp University, Bangalore, India · Shiv Nadar Institution of Eminence, Delhi NCR

May 1, 2026J/K move · Enter open · S save

Leveraging Vision-Language Models as Weak Annotators in Active Learning

Phuong Ngoc Nguyen, Kaito Shiku, Ryoma Bise, Seiichi Uchida+1

Kyushu University, Japan.

Vision-Language Models Active Learning Fine-Grained Image Classification Image Classification

Open paper AskPDF arXiv

Active learning aims to reduce annotation cost by selectively querying informative samples for supervision under a limited labeling budget. In this work, we investigate how vision-language models (VLMs) can be leveraged to further reduce the reliance on costly human annotation within the active learning paradigm. To this end, we find that the reliability of VLMs varies significantly with label granularity in fine-grained recognition tasks: they perform poorly on fine-grained labels but can provide accurate coarse-grained labels. Leveraging this property, we propose an active learning framework that combines fine-grained human annotations with coarse-grained VLM-generated weak labels through instance-wise label assignment. We further model the systematic noise in VLM-generated labels using a small set of trusted full labels. Experiments on CUB200 and FGVC-Aircraft show that the proposed framework consistently outperforms existing active learning methods under the same annotation budget.

Abstract

Active Learning

Explore similar work

Leveraging Vision-Language Models as Weak Annotators in Active Learning

Conformal Cross-Modal Active Learning

Balancing Uncertainty and Diversity of Samples: Leveraging Diversity of Least, High Confidence Samples for Effective Active Learning