Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

Lorenzo Jaime Yu Flores; Cesare Spinoso di-Piano; Ori Ernst; David Ifeoluwa Adelani; Jackie Chi Kit Cheung

cs.CLApr 10, 2026

Testing the Assumptions of Active Learning for Translation Tasks with Few Samples

Authors: Lorenzo Jaime Yu Flores, Cesare Spinoso di-Piano, Ori Ernst, David Ifeoluwa Adelani, Jackie Chi Kit Cheung

Organizations: Mila - Quebec AI Institute · McGill University

Abstract

Active learning (AL) is a training paradigm for selecting unlabeled samples for annotation to improve model performance on a test set, which is useful when only a limited number of samples can be annotated. These algorithms often work by optimizing for the informativeness and diversity of the training data to be annotated. Recent work found that AL strategies fail to outperform random sampling on various language generation tasks when using 100-500 samples. To understand AL's poor performance when only using few samples, we investigate whether the core assumptions underlying AL strategies hold. We find that neither the informativeness nor diversity of the training data, which AL strategies optimize for, are correlated with test set performance. Instead, factors like the ordering of the training samples and interactions with pre-training data have a larger impact on performance. This suggests that future AL methods must take these factors into account in order to work with very few samples.

Figures & tables

	Flan-T5			Llama 3.1			Gemma 2
	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil
BALD	7.23 ± 1.8	42.72 ± 0.4	19.8 ± 11.3	* 68.29 ± 0.2	74.18 ± 0.6	* 64.52 ± 0.2	* 60.04 ± 0.3	68.58 ± 0.4	59.13 ± 0.6
Core Set	5.03 ± 2.0	43.18 ± 0.6	22.9 ± 6.5	* 68.51 ± 0.4	74.61 ± 0.2	63.85 ± 0.2	58.28 ± 1.2	68.33 ± 0.7	59.04 ± 0.3
DelFy	3.47 ± 0.7	42.10 ± 1.4	27.3 ± 4.0	67.91 ± 0.3	* 74.90 ± 0.1	63.31 ± 0.3	* 59.10 ± 0.4	68.42 ± 0.3	58.18 ± 0.6
Lex. Sim	8.38 ± 0.9	41.14 ± 0.3	20.3 ± 9.8	67.54 ± 0.1	74.69 ± 0.0	0 ± 0	57.01 ± 0.1	68.33 ± 0.2	58.05 ± 0.7
Mean Ent	5.82 ± 1.4	42.81 ± 0.8	18.7 ± 1.5	67.08 ± 1.3	74.53 ± 0.2	63.21 ± 0.1	59.50 ± 0.9	68.27 ± 0.3	59.13 ± 0.8

Table 1: Various AL baselines fail to outperform random sampling when using

b=100

samples (reporting test set ChrF score across three seeds), * indicates significant difference from random (one-way Mann-Whitney,

\alpha=0.05

). The

0\pm 0

entry (Lex. Sim., Llama 3.1, Eng-Fil) is a failed run; see text.

		Flan-T5			Llama 3.1			Gemma 2
		Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil
Diversity	Vocabulary Size	-0.0420	0.0438	0.0162	0.0121	0.0153	-0.0802	-0.0131	-0.0214	0.117
	DelFy (Source)	0.1256	-0.0521	0.0267	-0.0960	0.1124	0.0724	-0.0145	0.0086	-0.2071
	DelFy (Target)	* 0.2924	-0.0819	0.1646	-0.0781	-0.0301	0.0622	0.0459	-0.0479	0.0721
	L2 Distance	-0.0454	0.0693	0.0002	0.0243	0.1683	-0.0300	-0.0366	0.0878	0.1018
Informativeness	Avg Token Ent	-0.0012	-0.0589	-0.1143	0.0237	0.0746	0.0461	-0.1482	0.0855	0.0487

Table 2: Spearman correlation between AL metrics and model performance with 100 samples for training (Test Set ChrF), * displayed for correlations significantly different from zero (

\alpha=0.05

with Bonferroni correction)

Figure 1: Fine-tuning on different subsets of the data yields considerable variance in test set performance; Plotted using 100 subsets with 100 samples each

Panel of Figure 2 — Figure 2: Pre-SFT model performance on the samples chosen by AL strategies for Gemma-2 on Eng-Afr (left) and Eng-Fil (right) shows that AL strategies tend to choose samples in the lower and middle range of pre-SFT performance, rather than samples the model already translates well; strategies vary in where they concentrate

Figure 3: Test set ChrF+ (FLORES Plus) when fine-tuning models on unlabeled data (NLLB) with varying degrees of difficulty (using 500 unlabeled samples, 3 seeds, difficulty measured using pre-SFT model performance)

	Flan-T5	Llama 3.1	Gemma 2
Eng-Afr	0.81	0.71	0.80
Eng-Ger	0.83	0.81	0.53
Eng-Fil	0.75	0.90	0.71
Eng-Afr	0.79	0.92	0.84
Eng-Ger	0.81	0.94	0.65
Eng-Fil	0.93	0.80	0.80

Table 3: Proportion of variance in ChrF from ordering using 100 (top) and 500 (bottom) samples, computed using ten shuffles of ten subsets of data (100 total)

Type	Text	Comment
Source	“We now have 4-month-old mice that are non-diabetic that used to be diabetic,” he added
Target	“Mayroon na tayong 4 na buwang gulang na daga na hindi diabetic na dating diabetic,” dagdag niya
Prediction (Step 23)	“We now have 4-month-old mice na hindi-diabetic,” katanya .	Foreign (Indonesian; katanya : he said)
Prediction (Step 70)	“We ngayon mayroon dalawang buwan gulang na mga maliliit na… katawan ng”	OOD word ( gulang : age/old)

Table 4: Models generate words not in the training data, both correctly ( gulang ) and incorrectly ( katanya )

Figure 4: Plot of % Filipino vocabulary per test example trained on vs. generated at test time (i.e. learned); Training on more vocabulary does not mean the model learns to generate those vocabulary; Each point is generated using a sample in the test set

		% Covered	% Similarity
Flan-T5	Eng-Afr	-0.0854	-0.0994
	Eng-Ger	0.1816	0.1872
	Eng-Fil	0.1124	0.1401
Llama 3.1	Eng-Afr	-0.0119	0.0051
	Eng-Ger	-0.0533	-0.0788
	Eng-Fil	-0.0727	-0.0094

Table 5: Spearman correlation between model test set performance and the similarity between the train and test set, using 100 samples for training, * indicates correlation is significantly different from zero (

\alpha=0.05

with Bonferroni correction)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Flan-T5			Llama-3.1			Gemma-2
	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil
BALD	29.58 ± 1.4	43.88 ± 0.2	29.7 ± 3.3	68.91 ± 0.3	74.53 ± 0.2	63.43 ± 0.4	59.62 ± 1.1	68.38 ± 0.3	60.29 ± 0.4
Core Set	14.11 ± 4.3	44.39 ± 0.4	30.73 ± 3	68.35 ± 0.6	74.56 ± 0.3	63.92 ± 1	59.16 ± 1.2	68.17 ± 0.3	60.32 ± 0.3
DelFy	20.12 ± 1.7	43.51 ± 0.8	28.7 ± 0.8	68.64 ± 0.3	74.48 ± 0.1	63.93 ± 0.8	58.63 ± 1.2	68.49 ± 0.3	59.12 ± 0.8
Lex. Sim	11.15 ± 2.9	43.63 ± 0.2	28.05 ± 1	66.57 ± 0.3	73.72 ± 0.2	0 ± 0	55.2 ± 0.5	68.02 ± 0.4	57.13 ± 0.8
Mean Ent	23.62 ± 2.8	43.12 ± 0.7	29.64 ± 1	64.95 ± 1.3	74.24 ± 0.4	62.97 ± 0.5	60.36 ± 0.7	68.3 ± 0.3	60.15 ± 0.6

Appendix

Table 6: AL baseline performance with 500 samples with st. deviation reported across 3 seeds (Test Set ChrF). The

0\pm 0

entry (Lex. Sim., Llama-3.1, Eng-Fil) is a failed run; see Section 4 .

Figure 5: Fine-tuning on different subsets of the data yields considerable variance in test set performance; Plotted using 100 subsets with 500 samples each

		Flan-T5			Llama 3.1			Gemma 2
		Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil	Eng-Afr	Eng-Ger	Eng-Fil
Diversity	Vocabulary Size	0.0684	0.07	0.0884	-0.1686	0.0822	0.0046	-0.1408	0.0483	0.0072
	DelFy (Source)	-0.076	0.0856	0.0011	-0.0138	0.0186	0.0107	-0.0053	0.0071	0.0144
	DelFy (Target)	-0.0072	-0.0522	-0.0932	-0.0606	0.0674	-0.1436	-0.1091	-0.0656	0.1121
	L2 Distance	-0.1665	0.0208	0.0411	0.0381	-0.0834	0.0388	-0.0736	0.1915	-0.104
Informativeness	Avg Token Entropy	-0.2114	0.0027	-0.0485	0.0555	0.1523	0.0463	-0.1427	-0.1159	0.0715

Appendix

Table 7: Spearman correlation between AL metrics and model performance using 500 samples for training (Test Set ChrF), * displayed for correlations significantly different from zero (

\alpha=0.05

with Bonferroni correction)

		% Vocab Covered in Test	% Vocab Jaccard Similarity to Test
Flan-T5	Eng-Afr	0.1021	0.1051
	Eng-Ger	0.0178	-0.0026
	Eng-Fil	0.2322	0.2287
Llama 3.1	Eng-Afr	-0.0946	0.031
	Eng-Ger	0.0899	0.0537
	Eng-Fil	0.0541	0.0209

Appendix

Table 8: Spearman correlation between model test set performance and the similarity between the training and test set, using 500 samples for training * added if the correlation is significantly different from zero

Panel of Figure 6 — Figure 6: Performance of model pre-SFT on candidates chosen by various strategies across three models and three tasks

Figure 7: Test performance when fine-tuning models on unlabeled data with varying degrees of difficulty (measured using pre-SFT model performance) using 2000 unlabeled samples

Set	Step	Input	Target
Train	91	A prayer for our beloved nation.	Isang panalangin sa aming mahal na nasyon.
Set	Step	Input	Prediction
Test	94	The tenth named storm of the Atlantic Hurricane season, Subtropical Storm Jerry, formed in the Atlantic Ocean today.	Ang lalake ang huling named na na na named na ang panalangin …, ang panalangin sa Atlantic ng Amerika…
	95	The number of people present was so large that it was not possible for everybody to gain access to the funeral in St. Peter’s Square.	Ang mga tao na ito ay hindi posible para sa lahat ng tao ang panalangin sa St. Peter’s Square.
		Prime Minister Stephen Harper has agreed to send the government’s ’Clean Air Act’… for review, before its second reading, after Tuesday’s 25 minute meeting with NDP leader Jack Layton at the PMO.	Stephen Harper ay nag-iisa ang panalangin sa ang lahat ng mga tao para sa panalangin …
	97	The final match of the series will take place at Ellis Park in Johannesburg next week, when the Springboks play Australia.	Ang palangin sa Ellis Park sa Johannesburg, ang panalangin sa Australia ng mga tao ng mga tao ng Australia.

Appendix

Table 9: Models incorrectly generate the word panalangin across various samples after being fine-tuned on an example with the word ( Red indicates wrong usage of the word)

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 288 papers

Select, Label, Evaluate: Active Testing in NLP
Mar 23, 2026Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu +2Human-in-the-Loop Annotation
Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S
Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection
Dec 14, 2025Pouya Ahadi, Blair Winograd, Camille Zaug +3Human-in-the-Loop Annotation Active Learning
School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company
Active Testing of Large Language Models via Approximate Neyman Allocation
May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM Evaluation Language Model Generation Evaluation
University of California, Riverside

Mar 23, 2026J/K move · Enter open · S save

Select, Label, Evaluate: Active Testing in NLP

Antonio Purificato, Maria Sofia Bucarelli, Andrea Bacciu, Fabrizio Silvestri+1

Amazon · Sapienza University of Rome · Université Côte d’Azur, CNRS, Inria, I3S

Human-in-the-Loop Annotation

Open paper AskPDF arXiv

Human annotation cost and time remain significant bottlenecks in Natural Language Processing (NLP), with test data annotation being particularly expensive due to the stringent requirement for low-error and high-quality labels necessary for reliable model evaluation. Traditional approaches require annotating entire test sets, leading to substantial resource requirements. Active Testing is a framework that selects the most informative test samples for annotation. Given a labeling budget, it aims to choose the subset that best estimates model performance while minimizing cost and human effort. In this work, we formalize Active Testing in NLP and we conduct an extensive benchmarking of existing approaches across 18 datasets and 4 embedding strategies spanning 4 different NLP tasks. The experiments show annotation reductions of up to 95%, with performance estimation accuracy difference from the full test set within 1%. Our analysis reveals variations in method effectiveness across different data characteristics and task types, with no single approach emerging as universally superior. Lastly, to address the limitation of requiring a predefined annotation budget in existing sample selection strategies, we introduce an adaptive stopping criterion that automatically determines the optimal number of samples. We release our code at https://github.com/amazon-science/NLPActiveTesting.

Abstract

Active Learning

Explore similar work

Select, Label, Evaluate: Active Testing in NLP

Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

Active Testing of Large Language Models via Approximate Neyman Allocation