Organizations: School of EECS, University of Ottawa, Ottawa, ON K1N 6N5, Canada · Research Ireland Lero centre for software, University of Limerick, Ireland
Maintaining or improving the performance of Deep Neural Networks (DNNs) through fine-tuning requires labeling newly collected inputs, a process that is often costly and time-consuming. To alleviate this problem, input selection approaches have been developed in recent years to identify small, yet highly informative subsets for labeling. Diversity-based selection is one of the most effective approaches for this purpose. However, they are often computationally intensive and lack scalability for large input sets, limiting their practical applicability. To address this challenge, we introduce Concept-Based Diversity (CBD), a novel and highly efficient diversity metric for image inputs that leverages Vision-Language Models (VLMs). Our results show that CBD exhibits a strong correlation with Geometric Diversity (GD), an established diversity metric, while requiring only a fraction of its computation time. Building on this finding, we propose a hybrid input selection approach that combines CBD with Margin, a simple uncertainty metric. We conduct a comprehensive evaluation across a diverse set of DNN models, input sets, selection budgets, and six most effective state-of-the-art selection baselines. The results demonstrate that the CBD-based selection consistently outperforms all baselines at guiding input selection to improve the DNN model. Furthermore, the CBD-based selection approach remains highly efficient, requiring selection times close to those of simple uncertainty-based methods such as Margin, even on larger input sets like ImageNet. These results confirm not only the effectiveness and computational advantage of the CBD-based approach, particularly compared to hybrid baselines, but also its scalability in repetitive and extensive input selection scenarios.
Figures & tables
Fig. 1: Calculating similarity score between an image and a natural language concept using CLIP (lower part), and training a linear aligner between a DNN’s internal representation and CLIP’s shared embedding space (upper part).
Fig. 2: The efficient process of finding highly similar concepts for each image using the RCS and the trained linear aligner.
input
Model
Training set
Test set
set
size
size
CIFAR-10
ResNet18
50,000
10,000
ImageNet
ResNet101
1,281,167
50,000
CIFAR-100
DenseNet121
50,000
10,000
TABLE I: Information about the input sets and DNN models
CIFAR-10
ImageNet
CIFAR-100
DNN’s representation size
512
2048
1024
CLIP’s embedding size
512
512
512
SigLIP’s embedding size
768
768
768
R2 (based on CLIP)
0.83
0.72
0.84
R2 (based on SigLIP)
0.78
0.72
0.80
DNN’s execution time
3.5 s
105 s
10 s
TABLE II: Information on the trained linear aligners, including each DNN’s representation size, embedding size of the VLM (including both CLIP and SigLIP) , coefficient of determination ( R2 ), and the time (in seconds) required to extract the related concepts for the entire test set using VLM directly and using the trained aligner with the DNN models
Input set
Input subset size
b =1%
b =3%
b =5%
b =7%
b =10%
CLIP
CIFAR-10
0.90
0.95
0.95
0.95
0.91
ImageNet
0.92
0.92
0.93
0.94
0.96
CIFAR-100
0.97
0.97
0.98
0.98
0.96
SigLIP
CIFAR-10
0.82
0.82
0.94
0.94
0.83
ImageNet
0.90
0.92
0.93
0.94
0.94
TABLE III: Spearman correlation coefficients computed between concept-based diversity and GD
Input set
Metric
Input subset size
b =1%
b =3%
b =5%
b =7%
b =10%
CIFAR-10
CBD
0.3
0.7
1.1
1.53
2.2
GD
1.8
2.5
3.1
5.3
14.6
ImageNet
CBD
1.4
4.2
6.9
9.4
12.5
GD
3.7
27.4
96.7
203.3
437.9
CIFAR-100
CBD
0.3
0.7
1.3
1.7
2.3
TABLE IV: Average diversity calculation time (milliseconds) for a single subset of different sizes using CBD and GD
Input
Selection
CBD-based
Ablation baselines
Baselines
set
size
selecion
Margin
RS+CBD
DATIS
RTS
DeepGD
SETS
GTS
CIFAR-10
b =3%
67.48%
58.12%
27.93%
32.72%
64.68%
66.17%
36.06%
32.16%
b =5%
73.23%
72.63%
50.35%
51.94%
61.11%
54.98%
45.12%
56.10%
b =7%
72.61%
53.76%
40.08%
56.98%
66.4%
45.7%
49.85%
61.41%
b =10%
86.48%
50.48%
36.81%
35.45%
69.36%
55.24%
44.24%
69.79%
b =15%
82.06%
50.42%
32.49%
34.87%
55.8%
54.6%
50.49%
75.17%
TABLE V: Average accuracy improvement percentages achieved by fine-tuning the DNN model with input subsets selected by the CBD-based approach and each baseline (five runs)
Input
Selection
CBD-based
Ablation baselines
Baselines
set
size
selecion
Margin
RS+CBD
DATIS
RTS
DeepGD
SETS
GTS
CIFAR-10
b =3%
2
2
1
5
72
959
0.3
28
b =5%
2
2
1
5
72
1,341
0.5
28
b =7%
2
2
1
5
72
1,966
1
28
b =10%
2
2
1
5
72
3,476
2
28
b =15%
2
2
2
5
72
5,293
5
28
TABLE VI: Selection time (in seconds) of the proposed CBD-based selection and each baseline
Subject
CBD-based
Margin
DATIS
RTS
DeepGD
SETS
GTS
CIFAR-10
155
0
13
293
0
0
100
ImageNet
3,916
0
2,791
15,691
0
0
3,690
CIFAR-100
236
0
65
455
0
0
180
TABLE VII: Required time for initializing the CBD approach and baselines (seconds).
Fig. 3: The average accuracy improvement percentages achieved by the CBD-based approach with different parameter configurations ( binit ).
While open sourced Vision-Language Models (VLMs) have proliferated, selecting the optimal pretrained model for a specific downstream task remains challenging. Exhaustive evaluation is often infeasible due to computational constraints and data limitations in few shot scenarios. Existing selection methods fail to fully address this: they either rely on data-intensive proxies or use symmetric textual descriptors that neglect the inherently directional and model-specific nature of transferability. To address this problem, we propose a framework that grounds model selection in the internal functional dynamics of the visual encoder. Our approach represents each task via layer wise conductance and derives a target-conditioned block importance distribution through entropy regularized alignment. Building on this, we introduce Directional Conductance Divergence (DCD), an asymmetric metric that quantifies how effectively a source task covers the target's salient functional blocks. This allows for predicting target model rankings by aggregating source task ranks without direct inference. Experimental results on 48 VLMs across 21 datasets demonstrate that our method outperforms state-of-the-art baselines, achieving a 14.7% improvement in NDCG@5 over SWAB.
Recent advances have explored visual token pruning to accelerate the inference of large vision-language models (LVLMs). However, existing methods often struggle to balance token importance and diversity: importance-based methods tend to retain redundant tokens, whereas diversity-based methods may overlook informative ones. This trade-off becomes especially problematic under high reduction ratios, where preserving only a small subset of visual tokens is critical. To address this issue, we propose ID-Selection, a simple yet effective token selection strategy for efficient LVLM inference. The key idea is to couple importance estimation with diversity-aware iterative selection: each token is first assigned an importance score, after which high-scoring tokens are selected one by one while the scores of similar tokens are progressively suppressed. In this way, ID-Selection preserves informative tokens while reducing redundancy in a unified selection process. Extensive experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency, especially under extreme pruning ratios. For example, on LLaVA-1.5-7B, ID-Selection prunes 97.2% of visual tokens, retaining only 16 tokens, while reducing inference FLOPs by over 97% and preserving 91.8% of the original performance, all without additional training.
Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.
Clarence Lee, Yejin Choi, Luke Zettlemoyer +2
DSO National Laboratories · Stanford University · University of Washington