Active Learning for Finding a Needle in a Haystack

Emily Silcock; Simon Löwe

cs.CLOct 10, 2026

Active Learning for Finding a Needle in a Haystack

Authors: Emily Silcock, Simon Löwe

Organizations: Harvard University · Burning Glass Institute

Abstract

Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus. Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class. Random sampling cannot supply these at reasonable cost, so they must be selected in some other way, such as keyword search, and any such selection biases the initial set towards part of the target class. We evaluate whether active learning can correct this bias. On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1. On AG News, almost every strategy recovers the omitted subgroup. On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1. Aggregate metrics conceal these differences.

Figures & tables

Dataset	Initial set	Target class	Initial-set target documents from	Train size	Test size	% target (train)
AG News	Unbiased	World	World	90,909	5,757	1.00%
AG News	Biased	World and Sports	World	60,606	3,838	1.00%
Measuring Hate Speech	Unbiased	Religion targeted	All religions	31,587	7,901	1.34%
Measuring Hate Speech	Biased	Religion targeted	Muslims	31,587	7,901	1.34%

Table 1: Tasks used in the experiments. “Initial-set target documents from” gives the part of the target class from which the 50 target-class documents in the initial set are drawn. The AG News target class is down-sampled separately for each random seed (Section 5.3 ); the sizes shown are the same for every seed.

	AG News				Measuring Hate Speech
	Unbiased	Biased			Unbiased	Biased
Method	F1	F1	$R_{U}$	$R_{R}$	F1	F1	$R_{U}$	$R_{R}$
Random	0.52	0.41	0.56	0.78	0.13	0.13	0.17	0.49
LC	0.69	0.73	0.90	0.68	0.05	0.06	0.02	0.10
BALD	0.68	0.68	0.87	0.65	0.08	0.10	0.09	0.23
Core-Set	0.63	0.69	0.90	0.63	0.11	0.10	0.16	0.30

Table 2: Test performance after the final round (1,100 labelled documents), mean over 25 runs. Unbiased and Biased refer to the initial set.

R_{U}

: recall on the subgroups unrepresented in the biased initial set;

R_{R}

: recall on the represented subgroup. Bold: best method in the column; underlined: not significantly worse than the best (paired

t

-test across runs, Holm-corrected over the eight comparisons in the column,

p\geq 0.05

R_{R}

is not tested. Standard errors are in Table 6 .

Figure 1: Recall on the subgroups unrepresented in the initial set (

R_{U}

) against F1 after the final round, with a biased initial set. Points are means over 25 runs, with standard errors. The grey line joins methods not outperformed on both metrics by any other method.

Figure 2: Recall on the represented subgroup (

R_{R}

, open) and on the unrepresented subgroups (

R_{U}

, filled) after the final round, with a biased initial set, mean over 25 runs. A method that corrects the bias of the initial set closes the gap between the two.

Figure 3: Target-class documents from the unrepresented subgroups (top) and from the represented subgroup (bottom) selected for labelling, cumulatively over rounds, with a biased initial set. Mean over 25 runs; excludes the 50 target-class documents in the initial set.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Strategy	Settings
BALD	10 dropout samples
Core-Set	Greedy $k$ -centre; Euclidean distance between unit-normalised embeddings
BADGE	k-means++ initialisation on gradient embeddings
CAL	10 nearest labelled neighbours; unit-normalised embeddings

Appendix

Table 3: Settings of the small-text query strategies. LC and random sampling have no settings.

Hyperparameter	Value
Pretrained model	distilroberta-base
Maximum input length	100 tokens
Training epochs per round	10
Batch size	32
Learning rate	$5\times 10^{-5}$
Loss weighting	Inverse class frequency

Appendix

Table 4: Classifier hyperparameters.

F1
		Epochs
Dataset	Initial set	3	5	8	10
AG News	Unbiased	0.51	0.57	0.47	0.55
AG News	Biased	0.46	0.51	0.44	0.47
Measuring Hate Speech	Unbiased	0.15	0.18	0.10	0.17
Measuring Hate Speech	Biased	0.11	0.10	0.19	0.15

Appendix

Table 5: Epoch tuning: final-round test performance of random sampling, mean over three runs (two runs for AG News unbiased at three epochs and Measuring Hate Speech biased at eight epochs, where a run did not complete).

Figure 4: Test F1 after each round (top) and target-class documents selected for labelling, cumulatively over rounds (bottom), mean over 25 runs. The bottom row excludes the 50 target-class documents in the initial set.

	AG News				Measuring Hate Speech
	Unbiased	Biased			Unbiased	Biased
Method	F1	F1	$R_{U}$	$R_{R}$	F1	F1	$R_{U}$	$R_{R}$
Random	0.52 $\pm$ 0.01	0.41 $\pm$ 0.02	0.56 $\pm$ 0.06	0.78 $\pm$ 0.02	0.13 $\pm$ 0.02	0.13 $\pm$ 0.01	0.17 $\pm$ 0.03	0.49 $\pm$ 0.05
LC	0.69 $\pm$ 0.01	0.73 $\pm$ 0.01	0.90 $\pm$ 0.02	0.68 $\pm$ 0.02	0.05 $\pm$ 0.02	0.06 $\pm$ 0.02	0.02 $\pm$ 0.01	0.10 $\pm$ 0.03
BALD	0.68 $\pm$ 0.01	0.68 $\pm$ 0.03	0.87 $\pm$ 0.04	0.65 $\pm$ 0.04	0.08 $\pm$ 0.02	0.10 $\pm$ 0.02	0.09 $\pm$ 0.03	0.23 $\pm$ 0.05
Core-Set	0.63 $\pm$ 0.03	0.69 $\pm$ 0.01	0.90 $\pm$ 0.01	0.63 $\pm$ 0.03	0.11 $\pm$ 0.02	0.10 $\pm$ 0.02	0.16 $\pm$ 0.04	0.30 $\pm$ 0.06

Appendix

Table 6: The results in Table 2 , with standard errors over the 25 runs.

	AG News, unbiased		Measuring Hate Speech, unbiased		AG News, biased		Measuring Hate Speech, biased
Method	P	R	P	R	P	R	P	R
Random	0.42 $\pm$ 0.02	0.73 $\pm$ 0.02	0.08 $\pm$ 0.01	0.35 $\pm$ 0.05	0.32 $\pm$ 0.02	0.67 $\pm$ 0.03	0.11 $\pm$ 0.01	0.31 $\pm$ 0.04
LC	0.70 $\pm$ 0.01	0.70 $\pm$ 0.01	0.10 $\pm$ 0.04	0.05 $\pm$ 0.02	0.69 $\pm$ 0.02	0.79 $\pm$ 0.02	0.12 $\pm$ 0.03	0.06 $\pm$ 0.02
BALD	0.69 $\pm$ 0.02	0.69 $\pm$ 0.02	0.06 $\pm$ 0.02	0.14 $\pm$ 0.04	0.63 $\pm$ 0.03	0.76 $\pm$ 0.04	0.11 $\pm$ 0.02	0.15 $\pm$ 0.04
Core-Set	0.63 $\pm$ 0.02	0.68 $\pm$ 0.03	0.09 $\pm$ 0.02	0.30 $\pm$ 0.05	0.64 $\pm$ 0.02	0.76 $\pm$ 0.02	0.10 $\pm$ 0.02	0.22 $\pm$ 0.05
BADGE	0.68 $\pm$ 0.02	0.69 $\pm$ 0.03	0.10 $\pm$ 0.02	0.13 $\pm$ 0.03	0.67 $\pm$ 0.02	0.78 $\pm$ 0.02	0.12 $\pm$ 0.02	0.15 $\pm$ 0.03

Appendix

Table 7: Precision (P) and recall (R) of the target class after the final round, mean

\pm

standard error over 25 runs.

	AG News, unbiased		Measuring Hate Speech, unbiased		AG News, biased		Measuring Hate Speech, biased
Method	Macro	Worst	Macro	Worst	Macro	Worst	Macro	Worst
Random	0.73 $\pm$ 0.02	0.73 $\pm$ 0.02	0.30 $\pm$ 0.05	0.03 $\pm$ 0.01	0.67 $\pm$ 0.03	0.54 $\pm$ 0.05	0.20 $\pm$ 0.03	0.00 $\pm$ 0.00
LC	0.70 $\pm$ 0.01	0.70 $\pm$ 0.01	0.05 $\pm$ 0.02	0.00 $\pm$ 0.00	0.79 $\pm$ 0.02	0.68 $\pm$ 0.02	0.04 $\pm$ 0.01	0.00 $\pm$ 0.00
BALD	0.69 $\pm$ 0.02	0.69 $\pm$ 0.02	0.12 $\pm$ 0.03	0.01 $\pm$ 0.01	0.76 $\pm$ 0.04	0.64 $\pm$ 0.04	0.10 $\pm$ 0.03	0.00 $\pm$ 0.00
Core-Set	0.68 $\pm$ 0.03	0.68 $\pm$ 0.03	0.32 $\pm$ 0.06	0.10 $\pm$ 0.04	0.76 $\pm$ 0.02	0.63 $\pm$ 0.03	0.16 $\pm$ 0.04	0.00 $\pm$ 0.00
BADGE	0.69 $\pm$ 0.03	0.69 $\pm$ 0.03	0.10 $\pm$ 0.03	0.01 $\pm$ 0.01	0.78 $\pm$ 0.02	0.65 $\pm$ 0.03	0.11 $\pm$ 0.02	0.00 $\pm$ 0.00

Appendix

Table 8: Recall averaged over the subgroups of the target class (Macro) and on the subgroup with the lowest recall (Worst) after the final round, mean

\pm

standard error over 25 runs. With an unbiased initial set, the AG News target class has a single subgroup, so both equal recall. In Measuring Hate Speech the smallest subgroup has one test document, so worst-group recall is coarse.

Subgroup Robustness

See all 10 papers

Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance
May 30, 2026Jiancheng Zhang, Meiqing Li, Qi Zhang +1Class-Imbalanced Learning Active Learning
University of California, Riverside · 2Carnegie Mellon University · 3Worcester Polytechnic Institute
Testing the Assumptions of Active Learning for Translation Tasks with Few Samples
Apr 10, 2026Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst +2Active Learning Training Data Selection
Mila - Quebec AI Institute · McGill University
An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations
Apr 25, 2026Varun Totakura, Ankita Singh, Yushun Dong +1Human-in-the-Loop Annotation Active Learning
Department of Computer Science Florida State University Tallahassee, FL, USA

May 30, 2026J/K move · Enter open · S save

Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance

Jiancheng Zhang, Meiqing Li, Qi Zhang, Yinglun Zhu

University of California, Riverside · 2Carnegie Mellon University · 3Worcester Polytechnic Institute

Class-Imbalanced Learning Active Learning Learning with Noisy Labels

Open paper AskPDF arXiv

Real-world datasets across image and text domains are often characterized by skewed class distributions and noisy annotations, which jointly degrade model performance, particularly on minority classes. Among existing solutions, active learning offers an effective and efficient paradigm by selectively querying the most informative and balanced samples for annotation. We propose an innovative active learning framework that mitigates class imbalance and selects the most informative samples to annotate. Leveraging foundation model priors, our algorithm enables imbalance-aware co-decisions between foundation model and small model to tackle noisy and imbalanced labels across various domains. We introduce the first study to systematically explore active learning under the dual challenges of label noise and class imbalance across image and text domains. Extensive experiments on imbalanced datasets demonstrate that our method achieves substantial annotation savings-over 50% compared to the best active learning baseline-while preserving performance and robustness to label noise.

Abstract

Subgroup Robustness

Explore similar work

Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance

Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

An Analysis of Active Learning Algorithms using Real-World Crowd-sourced Text Annotations