BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

Denis Huseljic; Paul Hahn; Marek Herde; Christoph Sandrock; Bernhard Sick

cs.LGMar 13, 2026

BoSS: A Best-of-Strategies Selector as an Oracle for Deep Active Learning

Authors: Denis Huseljic, Paul Hahn, Marek Herde, Christoph Sandrock, Bernhard Sick

Organizations: Intelligent Embedded Systems University of Kassel Kassel, Hesse, Germany · Machine Learning Research Unit TU Wien Vienna, Vienna, Austria

Abstract

Active learning (AL) aims to reduce annotation costs while maximizing model performance by iteratively selecting valuable instances. While foundation models have made it easier to identify these instances, existing selection strategies still lack robustness across different models, annotation budgets, and datasets. To highlight the potential weaknesses of existing AL strategies and provide a reference point for research, we explore oracle strategies, i.e., strategies that approximate the optimal selection by accessing ground-truth information unavailable in practical AL scenarios. Current oracle strategies, however, fail to scale effectively to large datasets and complex deep neural networks. To tackle these limitations, we introduce the Best-of-Strategy Selector (BoSS), a scalable oracle strategy designed for large-scale AL scenarios. BoSS constructs a set of candidate batches through an ensemble of selection strategies and then selects the batch yielding the highest performance gain. As an ensemble of selection strategies, BoSS can be easily extended with new state-of-the-art strategies as they emerge, ensuring it remains a reliable oracle strategy in the future. Our evaluation demonstrates that i) BoSS outperforms existing oracle strategies, ii) state-of-the-art AL strategies still fall noticeably short of oracle performance, especially in large-scale datasets with many classes, and iii) one possible solution to counteract the inconsistent performance of AL strategies might be to employ an ensemble-based approach for the selection.

Figures & tables

Figure 1: Accuracy improvement over random sampling for BoSS and state-of-the-art selection strategies using DINOv2-ViT-S/14.

Table 2

	Time Complexity	Recommended	# Processed Training
Oracle Strategy	per Batch	Hyperparameters	Instances
CDO ( Werner et al., 2024 )	${\mathcal{O}}(m\cdot\sum_{i=1}^{b}\texttt{train-eval}({\bm{\theta}},{\mathcal{L}}^{+i},{\mathcal{E}}))$	$m=20$	$20\cdot(b\cdot\|{\mathcal{L}}\|+\frac{b(b+1)}{2})$
SAS ( Zhou et al., 2021 )	${\mathcal{O}}((s+g)\cdot A\cdot\texttt{train-eval}({\bm{\theta}},{\mathcal{L}}^{+b},{\mathcal{E}}))$	$s=1{,}250,g=250$	$1{,}500\cdot A\cdot(\|{\mathcal{L}}\|+b)$
BoSS	${\mathcal{O}}(T\cdot\texttt{train-eval}({\bm{\theta}},{\mathcal{L}}^{+b},{\mathcal{E}}))$	$T=100$	$10\cdot\|{\mathcal{S}}\|\cdot(\|{\mathcal{L}}\|+b)$

Table 2: Summary of time complexities of oracle strategies.

Dataset	# Classes ( $K$ )	Batch Size ( $b$ )
CIFAR-10 ( 2009 )	10	10
STL-10 ( 2011 )	10	10
Snacks ( 2021 )	20	20
Flowers102 ( 2008 )	102	25
Dopanim ( 2024 )	15	50
DTD ( 2014 )	47	50

Table 3: Overview of datasets, showing number of classes

K

and batch size

b

Oracle	Default	CIFAR-10 ( $b=10$ )	Snacks ( $b=20$ )	Dopanim ( $b=50$ )	DTD ( $b=50$ )
BoSS	$T=100$	$T=100$	$T=100$	$T=100$	$T=100$
CDO	$m=20$	$m=20$	$m=10$	$m=4$	$m=3$
SAS	$s=25{,}000,g=5{,}000$	$s=250,g=10$	$s=225,g=10$	$s=215,g=10$	$s=150,g=10$

Table 4: Hyperparameters of oracle strategies under runtime constraints equivalent to BoSS.

Oracle	Cifar-10	Snacks	Dopanim	DTD
BoSS	10:07	13:19	30:47	22:07
CDO (Adapted)	10:26	14:11	33:25	24:56
SAS (Adapted)	10:20	13:24	31:56	22:14
CDO (Default)	10:26	28:22	2:47:08	2:46:14
SAS (Default)	17:22:39	24:19:42	61:52:42	62:34:13

Table 5: Empirical runtimes of oracle strategies with adapted and default hyperparameters.

Figure 2: Relative learning curves of oracle strategies with aligned runtimes using DINOv2-ViT-S/14.

Panel of Figure 3 — Figure 3: Relative learning curves achieved by BoSS and state-of-the-art selection strategies at each annotation cycle for different pretrained models.

Figure 4: Average relative pick frequency of AL strategies by BoSS across cycles averaged over all datasets.

Figure 5: Relative learning curves of BoSS with naively selected candidate batches vs. our algorithm.

Table 11

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Explanatory plot of the “new” strategy DropQuery in relation to both random and BoSS.

Figure 7: Relative learning curves and pick frequencies with (Vary) and without (No Vary) varying the subset size for candidate batch generation.

Figure 8: Average pick choices of BoSS with three selection strategies on CIFAR-100 and Food101.

Figure 9: Relative pick frequencies of selection strategies by BoSS per dataset averaged over cycles, here given with the numeric value (in %).

Figure 10: Relative learning curves of BoSS and its runtime-optimized variants.

Figure 11: Absolute learning curves of oracle strategies with aligned runtimes using DINOv2-ViT-S/14.

Figure 12: Relative learning curves of oracle strategies with default hyperparameters using DINOv2-ViT-S/14.

Panel of Figure 13 — Figure 13: Absolute learning curves achieved by BoSS and state-of-the-art selection strategies at each annotation cycle for different pretrained models.

	CIFAR-10	STL-10	Snacks	Flowers102	Dopanim	DTD	Food101	CIFAR-100	Tiny ImageNet	ImageNet
Random	0.713	0.763	0.707	0.548	0.612	0.531	0.538	0.531	0.496	0.500
Margin	0.745	0.813	0.740	0.630	0.628	0.550	0.539	0.544	0.513	0.513
CoreSets	0.693	0.759	0.668	0.680	0.525	0.519	0.503	0.543	0.508	0.501
BADGE	0.748	0.810	0.742	0.640	0.626	0.555	0.546	0.550	0.512	0.519
TypiClust	0.777	0.825	0.776	0.721	0.629	0.587	0.571	0.577	0.546	0.503
DropQuery	0.789	0.862	0.792	0.693	0.629	0.584	0.576	0.567	0.546	0.530

Appendix

Table 11: AULC for the low-budget regime across vision datasets using DINOv2 features.

	CIFAR-10	STL-10	Snacks	Flowers102	Dopanim	DTD	Food101	CIFAR-100	Tiny ImageNet	ImageNet
Random	0.893	0.931	0.869	0.889	0.747	0.695	0.725	0.715	0.681	0.662
Margin	0.916	0.979	0.888	0.980	0.765	0.722	0.736	0.732	0.687	0.674
CoreSets	0.886	0.949	0.835	0.967	0.658	0.701	0.671	0.711	0.669	0.647
BADGE	0.919	0.979	0.888	0.981	0.760	0.726	0.744	0.736	0.695	0.677
TypiClust	0.913	0.961	0.882	0.981	0.748	0.721	0.729	0.736	0.693	0.662
DropQuery	0.919	0.981	0.900	0.973	0.756	0.709	0.740	0.731	0.692	0.675

Appendix

Table 12: AULC for the mid-budget regime across vision datasets using DINOv2 features.

	CIFAR-10	STL-10	Snacks	Flowers102	Dopanim	DTD	Food101	CIFAR-100	Tiny ImageNet	ImageNet
Random	0.919	0.959	0.889	0.972	0.774	0.734	0.770	0.759	0.720	0.696
Margin	0.936	0.988	0.910	0.993	0.793	0.760	0.777	0.771	0.726	0.707
CoreSets	0.919	0.977	0.879	0.989	0.710	0.746	0.726	0.751	0.710	0.682
BADGE	0.938	0.987	0.911	0.993	0.789	0.759	0.784	0.775	0.731	0.708
TypiClust	0.930	0.971	0.892	0.990	0.768	0.749	0.770	0.766	0.725	0.695
DropQuery	0.933	0.986	0.911	0.990	0.778	0.740	0.774	0.763	0.722	0.704

Appendix

Table 13: AULC for the high-budget regime across vision datasets using DINOv2 features.

Panel of Figure 14 — Figure 14: Learning curves of BoSS and state-of-the-art selection strategies across annotation cycles on four text classification datasets using MiniLM features.

Panel of Figure 15 — Figure 15: Learning curves of BoSS and its variants with reduced supervision: without supervised strategies (Unsup.) and with approximate supervision (Approx.).

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 288 papers

Soft Strategy Selection for Batch-Mode Active Learning
Oct 4, 2026Rushil Gupta, Romain LopezActive Learning
Department of Computer Science, Courant Institute School of Mathematics, Computing, and Data Science, New York University · Department of Biology, New York University
CAAL: Contextual Bandits based Online Hand-Craft Active Learning Strategy Selection
Jun 6, 2026Shao-An Yin, Jiacong Li, Tianpei Xie +3Active Learning Contextual Bandits
Department of Electrical and Computer Engineering, University of Minnesota, Twin Cities, USA · Jiacong Li, Tianpei Xie, Cecile Levasseur, and Wojciech Kowalinski are with Amazon.
An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations
Apr 25, 2026Varun Totakura, Ankita Singh, Yushun Dong +1Human-in-the-Loop Annotation Active Learning
Department of Computer Science Florida State University Tallahassee, FL, USA

Oct 4, 2026J/K move · Enter open · S save

Soft Strategy Selection for Batch-Mode Active Learning

Rushil Gupta, Romain Lopez

Department of Computer Science, Courant Institute School of Mathematics, Computing, and Data Science, New York University · Department of Biology, New York University

Active Learning

Open paper AskPDF arXiv

Real-world deployment of active learning typically forces practitioners to choose an acquisition strategy before any data is labeled. This is a daunting task: strategy performance varies widely across settings (e.g. datasets, surrogate models) and cannot be assessed without deployment. Existing strategy selection methods explore one strategy from a portfolio at each round and identify the optimal one using bandit feedback or model retraining. Many acquisition rounds are therefore spent exploring strategies rather than collecting the most informative data. Such overhead is a major barrier to AL-driven design of high-throughput experiments, such as genetic perturbation screens and directed evolution, where AL runs consist of only a few rounds with large batch sizes. This regime permits a natural alternative: acquiring data using multiple AL strategies within a single batch. We refer to this as soft strategy selection and introduce FractAL, a method specifically designed for this task. FractAL infers a per-strategy reward using influence-function-based data attribution, which requires no additional retraining, and then computes budget shares for each strategy in the portfolio using online mirror descent. We benchmark FractAL across 7 setups spanning classification, regression, and genetic perturbation effect prediction. The results highlight that strategy selection is a hard problem: every existing method performs worse than random sampling on at least one setup. FractAL, however, matches or outperforms every baseline, including random sampling, on all 7 setups. Its allocations concentrate budget on the strongest strategies in the portfolio while pruning the weakest. FractAL is therefore a reliable choice for real-world deployments, where the optimal strategy is unknown in advance, an important step towards making AL practical for high-throughput experiments and modern scientific discovery.

Abstract

Active Learning

Explore similar work

Soft Strategy Selection for Batch-Mode Active Learning

CAAL: Contextual Bandits based Online Hand-Craft Active Learning Strategy Selection

An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations