Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superior. Lastly, to address the limitation of requiring a predefined annotation budget in existing sample selection strategies, we introduce an adaptive stopping criterion that automatically determines the optimal number of samples. We release our code at https://github.com/amazon-science/NLPActiveTesting.
Figures & tables
Figure 1: Annotation cost of using Active Testing vs full test set ( ↓ is better). Each sample has a cost of 0.02(LabelYourData).Theactivecostistheonerequiredtoreachanestimationquality<1%$ wrt. full budget.
Table 1: Class distribution of the selected datasets and time required to compute the embeddings on the full datasets in minutes ( T ). Number of train and test samples and values of the budget are also shown.
Figure 2: Estimation error with Agreement on AgNews using Qwen and Stella embedders ( ↓ is better).
Figure 3: Active Testing results in terms of accuracy on all the datasets by fixing the embedding strategy and the predictor. We observe that the Agreement strategy is performing better than Surrogate on the majority of the datasets. With a reduced budget Agreement is also able to beat Random ( ↓ is better).
Figure 4: Multilingual results. Each row is a value of prior p . Each column refers to a method ( ↓ is better).
Figure 5: Results using the stopping criterion approach in order to avoid annotating all the samples ( ↓ is better). Vertical lines represents when the algorithms stop.
IMDB
QNLI
B
T Agreement
T Random
T Surrogate
Cost
T Agreement
T Random
T Surrogate
Cost
100
5.51
0.01
12.47
2
4.97
0.01
14.31
2
500
5.51
0.001
12.52
10
5.05
0.01
14.32
10
900
5.52
0.01
59.05
18
5.08
0.01
43.06
18
S-Agreement
5.49
-
-
1
5.05
-
-
2
S-Random
-
0.01
-
2
-
NA
-
NA
Table 2: Time (in seconds) and annotation cost (in USD, assuming 0.02persampleasfromLabelYourData)fordifferentActiveTestingstrategies(\downarrow$ is better). S : stopping criterion applied on the strategy; Full : entire dataset annotated; NA : stopping condition not met.
Budget
4 Heads
8 Heads
16 Heads
100
0.0084±0.0058
0.0075±0.0062
0.0077±0.0066
500
0.0047±0.0037
0.0038±0.0034
0.0039±0.0034
900
0.0039±0.0021
0.0034±0.0025
0.0034±0.0026
Table 3: Estimation error in terms of accuracy for Agreement with different numbers of attention heads across annotation budgets. Best results are shown in bold.
Budget
Random Attention
BERT Attention
100
0.0075±0.0012
0.0146±0.0015
500
0.0038±0.0004
0.0038±0.0003
900
0.0034±0.0005
0.0038±0.0006
Table 4: Estimation error in terms of accuracy for Agreement using randomly initialized attention and attention maps from the last layer of a pretrained BERT encoder. Best results are shown in bold.
B
Agreement
Random
Surrogate-SVM
75
0.031 ± 0.006
0.022 ± 0.012
0.022 ± 0.005
150
0.042 ± 0.008
0.036 ± 0.009
0.029 ± 0.006
450
0.044 ± 0.006
0.033 ± 0.007
0.035 ± 0.008
750
0.047 ± 0.003
0.043 ± 0.003
0.044 ± 0.006
Table 5: Per-class F1 on minority class detection ( ↑ is better). Agreement and Surrogate outperform Random. Top: Emotion, bottom: PubMed.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Confusion matrices with the Active Testing Strategies on the x-axis. The top figures have the embedding strategies on the y-axis while the bottom ones the predictors. The best results are marked with *. MAE wrt. optimal curve on the left and Area Under the Estimation Error Curve on the right. ↓ is better.
Claude
Nova Pro
Qwen
Dataset
Accuracy
Recall
F1
Accuracy
Recall
F1
Accuracy
Recall
F1
AG’s News
0.8814
0.8814
0.8813
0.8082
0.8082
0.8066
0.7863
0.7863
0.7886
Banking77
0.7925
0.7925
0.7996
0.6942
0.6942
0.7072
0.6581
0.6581
0.6781
DBPedia
0.9822
0.9822
0.9822
0.9385
0.9385
0.9401
0.9312
0.9312
0.9317
FNC-1
0.4806
0.4806
0.4462
0.2933
0.2933
0.1961
0.4082
0.4082
0.3406
QNLI
0.5072
0.5072
0.5988
0.5017
0.5017
0.5484
0.4882
0.4882
0.5033
Appendix
Table 6: Text classification metrics on the full datasets. For each metric and dataset, the best predictor is bolded while the second best is underlined .
Dataset
Claude
Nova Pro
Qwen
CNN ∗
0.5074
0.5078
0.4827
XLSum ∗
0.4427
0.4543
0.4620
Appendix
Table 7: Summarization results on the full datasets in terms of ROUGE-1. For each metric and dataset, the best predictor is bolded while the second best is underlined .
Figure 7: Active Testing results in terms of accuracy on all the datasets by fixing the predictor (Qwen for classification, DeBERTa finetuned for POS tagging and CNER for NER and the embedding strategy (Qwen).
Large language models (LLMs) require reliable evaluation from pre-training to test-time scaling, making evaluation a recurring rather than one-off cost. As model scales grow and target tasks increasingly demand expert annotators, both the compute and labeling costs needed for each evaluation rise rapidly. Active testing aims to alleviate this bottleneck by approximating the evaluation result from a small but informative subset of the evaluation pool. However, existing approaches primarily target classification and break down on generative tasks. We introduce a novel active testing algorithm tailored to generative tasks. Our method leverages semantic entropy from surrogate models to stratify the evaluation pool and then conducts approximate Neyman allocation based on signals extracted from these surrogates. Across multiple language and multimodal benchmarks and a range of surrogate-target model pairs, our method significantly improves on baselines and closely tracks Oracle-Neyman, delivering up to 28% MSE reduction over Uniform Sampling and an average of 22.9% budget savings.
Active learning (AL) is a training paradigm for selecting unlabeled samples for annotation to improve model performance on a test set, which is useful when only a limited number of samples can be annotated. These algorithms often work by optimizing for the informativeness and diversity of the training data to be annotated. Recent work found that AL strategies fail to outperform random sampling on various language generation tasks when using 100-500 samples. To understand AL's poor performance when only using few samples, we investigate whether the core assumptions underlying AL strategies hold. We find that neither the informativeness nor diversity of the training data, which AL strategies optimize for, are correlated with test set performance. Instead, factors like the ordering of the training samples and interactions with pre-training data have a larger impact on performance. This suggests that future AL methods must take these factors into account in order to work with very few samples.
Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst +2
With the widespread adoption of pre-trained Large Language Models (LLM), there exists a high demand for task-specific test sets to benchmark their performance in domains such as healthcare and biomedicine. However, the cost of labeling test samples while developing new benchmarks poses a significant challenge, especially when expert annotators are required. Existing frameworks for active sample selection offer limited support for generative Question Answering tasks, where option dynamics can affect model decision boundaries. In this paper, we present Generative Active Testing (GAT), an uncertainty-aware acquisition framework leveraging LLMs as surrogates for informing the sample selection process. Using a novel Statement Adaptation Module, we modify generative tasks into a pseudo-classification format, enabling the capture of sample-level uncertainties across unlabeled candidates. Our zero-shot acquisition functions reduce estimation error by ~40% compared to traditional sampling baselines, offering a scalable solution for cost-effective model benchmarking.