Select, Label, Evaluate: Active Testing in NLP

Antonio Purificato; Maria Sofia Bucarelli; Andrea Bacciu; Fabrizio Silvestri; Amin Mantrach

cs.CLMar 23, 2026

Select, Label, Evaluate: Active Testing in NLP

Authors: Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Fabrizio Silvestri, Amin Mantrach

Organizations: Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S

Abstract

Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superior. Lastly, to address the limitation of requiring a predefined annotation budget in existing sample selection strategies, we introduce an adaptive stopping criterion that automatically determines the optimal number of samples. We release our code at https://github.com/amazon-science/NLPActiveTesting.

Figures & tables

Figure 1: Annotation cost of using Active Testing vs full test set (

\downarrow

is better). Each sample has a cost of 0.02

( LabelYourData ). The active cost is the one required to reach an estimation quality

<1%$ wrt. full budget.

Name	#Classes	Class Distribution (%)	$T$ [minutes]	\|Train\|	\|Test\|	$B$
AG’s News	4	[25,25,25,25]	7	120,000	7,600	1,000
Banking77	77	-	2	10,000	3,000	1,000
DBPedia	14	[7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1,7.1]	77	560,000	5,000	1,000
FNC-1	4	[6.72,1.50,17.77,74.01]	2	40,000	4,998	1,000
QNLI	2	[49,51]	3	104,000	5,463	1,000
SST-2	2	[49,51]	1	67,000	872	600

Table 1: Class distribution of the selected datasets and time required to compute the embeddings on the full datasets in minutes (

T

). Number of train and test samples and values of the budget are also shown.

Figure 2: Estimation error with Agreement on AgNews using Qwen and Stella embedders (

\downarrow

is better).

Figure 3: Active Testing results in terms of accuracy on all the datasets by fixing the embedding strategy and the predictor. We observe that the Agreement strategy is performing better than Surrogate on the majority of the datasets. With a reduced budget Agreement is also able to beat Random ( $\downarrow$ is better).

Figure 5: Results using the stopping criterion approach in order to avoid annotating all the samples (

\downarrow

is better). Vertical lines represents when the algorithms stop.

	IMDB				QNLI
$B$	T Agreement	T Random	T Surrogate	Cost	T Agreement	T Random	T Surrogate	Cost
100	5.51	0.01	12.47	2	4.97	0.01	14.31	2
500	5.51	0.001	12.52	10	5.05	0.01	14.32	10
900	5.52	0.01	59.05	18	5.08	0.01	43.06	18
S-Agreement	5.49	-	-	1	5.05	-	-	2
S-Random	-	0.01	-	2	-	NA	-	NA

Table 2: Time (in seconds) and annotation cost (in USD, assuming

0.02 per sample as from LabelYourData ) for different Active Testing strategies (

\downarrow$ is better). S : stopping criterion applied on the strategy; Full : entire dataset annotated; NA : stopping condition not met.

Budget	4 Heads	8 Heads	16 Heads
100	$0.0084\pm 0.0058$	$\mathbf{0.0075\pm 0.0062}$	$0.0077\pm 0.0066$
500	$0.0047\pm 0.0037$	$\mathbf{0.0038\pm 0.0034}$	$0.0039\pm 0.0034$
900	$0.0039\pm 0.0021$	$\mathbf{0.0034\pm 0.0025}$	$0.0034\pm 0.0026$

Table 3: Estimation error in terms of accuracy for Agreement with different numbers of attention heads across annotation budgets. Best results are shown in bold.

Budget	Random Attention	BERT Attention
100	$\mathbf{0.0075\pm 0.0012}$	$0.0146\pm 0.0015$
500	$\mathbf{0.0038\pm 0.0004}$	$0.0038\pm 0.0003$
900	$\mathbf{0.0034\pm 0.0005}$	$0.0038\pm 0.0006$

Table 4: Estimation error in terms of accuracy for Agreement using randomly initialized attention and attention maps from the last layer of a pretrained BERT encoder. Best results are shown in bold.

$B$	Agreement	Random	Surrogate-SVM
75	0.031 $\pm$ 0.006	0.022 $\pm$ 0.012	0.022 $\pm$ 0.005
150	0.042 $\pm$ 0.008	0.036 $\pm$ 0.009	0.029 $\pm$ 0.006
450	0.044 $\pm$ 0.006	0.033 $\pm$ 0.007	0.035 $\pm$ 0.008
750	0.047 $\pm$ 0.003	0.043 $\pm$ 0.003	0.044 $\pm$ 0.006

Table 5: Per-class F1 on minority class detection (

\uparrow

is better). Agreement and Surrogate outperform Random. Top: Emotion, bottom: PubMed.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Confusion matrices with the Active Testing Strategies on the x-axis. The top figures have the embedding strategies on the y-axis while the bottom ones the predictors. The best results are marked with *. MAE wrt. optimal curve on the left and Area Under the Estimation Error Curve on the right. $\downarrow$ is better.

	Claude			Nova Pro			Qwen
Dataset	Accuracy	Recall	F1	Accuracy	Recall	F1	Accuracy	Recall	F1
AG’s News	0.8814	0.8814	0.8813	0.8082	0.8082	0.8066	0.7863	0.7863	0.7886
Banking77	0.7925	0.7925	0.7996	0.6942	0.6942	0.7072	0.6581	0.6581	0.6781
DBPedia	0.9822	0.9822	0.9822	0.9385	0.9385	0.9401	0.9312	0.9312	0.9317
FNC-1	0.4806	0.4806	0.4462	0.2933	0.2933	0.1961	0.4082	0.4082	0.3406
QNLI	0.5072	0.5072	0.5988	0.5017	0.5017	0.5484	0.4882	0.4882	0.5033

Appendix

Table 6: Text classification metrics on the full datasets. For each metric and dataset, the best predictor is bolded while the second best is underlined .

Dataset	Claude	Nova Pro	Qwen
CNN ∗	0.5074	0.5078	0.4827
XLSum ∗	0.4427	0.4543	0.4620

Appendix

Table 7: Summarization results on the full datasets in terms of ROUGE-1. For each metric and dataset, the best predictor is bolded while the second best is underlined .

Figure 7: Active Testing results in terms of accuracy on all the datasets by fixing the predictor (Qwen for classification, DeBERTa finetuned for POS tagging and CNER for NER and the embedding strategy (Qwen).

Human-in-the-Loop Annotation

Human-in-the-loop annotation combines human judgment and AI assistance to create, review, and improve labeled data.

See all 135 papers

Active Testing of Large Language Models via Approximate Neyman Allocation
May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM Evaluation Language Model Generation Evaluation
University of California, Riverside
Testing the Assumptions of Active Learning for Translation Tasks with Few Samples
Apr 10, 2026Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst +2Active Learning Training Data Selection
Mila - Quebec AI Institute · McGill University
Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation
Feb 26, 2026Aashish Anantha Ramakrishnan, Ardavan Saeedi, Hamid Reza Hassanzadeh +2LLM Evaluation Uncertainty Sampling
The Pennsylvania State University · Optum AI

May 11, 2026J/K move · Enter open · S save

Active Testing of Large Language Models via Approximate Neyman Allocation

Zeli Liu, Jiancheng Zhang, Cong Liu, Yinglun Zhu

University of California, Riverside

LLM Evaluation Language Model Generation Evaluation Semantic Entropy Generative AI Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) require reliable evaluation from pre-training to test-time scaling, making evaluation a recurring rather than one-off cost. As model scales grow and target tasks increasingly demand expert annotators, both the compute and labeling costs needed for each evaluation rise rapidly. Active testing aims to alleviate this bottleneck by approximating the evaluation result from a small but informative subset of the evaluation pool. However, existing approaches primarily target classification and break down on generative tasks. We introduce a novel active testing algorithm tailored to generative tasks. Our method leverages semantic entropy from surrogate models to stratify the evaluation pool and then conducts approximate Neyman allocation based on signals extracted from these surrogates. Across multiple language and multimodal benchmarks and a range of surrogate-target model pairs, our method significantly improves on baselines and closely tracks Oracle-Neyman, delivering up to 28% MSE reduction over Uniform Sampling and an average of 22.9% budget savings.

Abstract

Human-in-the-Loop Annotation

Explore similar work

Active Testing of Large Language Models via Approximate Neyman Allocation

Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation