Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

Pouya Ahadi; Blair Winograd; Camille Zaug; Karunesh Arora; Lijun Wang; Kamran Paynabar

cs.LGDec 14, 2025

Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

Authors: Pouya Ahadi, Blair Winograd, Camille Zaug, Karunesh Arora, Lijun Wang, Kamran Paynabar

Organizations: School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company

Abstract

Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.

Figures & tables

Figure 1: General framework for AL for classification. The AL cycle will be repeated for $T$ cycles until the total sampling budget is consumed.

Panel of Figure 2 — Figure 2: An example of optimal assignment where $N_{t}=5$ , $M=3$ and $c_{1}=c_{2}=c_{3}=2$ . Red circles represent query points and blue triangles show labelers. Assuming that the points and labelers are sorted, $e_{1}\geq e_{2}\geq\dots\geq e_{5}$ and $a_{1}\geq a_{2}\geq a_{3}$ , the assignment policy can be observed from the figure. Data points with higher entropy will be assigned to the labelers with higher accuracy.

Panel of Figure 3 — Figure 3: Experimental framework for estimating the noise function. Each sample point is evaluated by both an oracle and each labeler. The resulting data is then used to train a logistic regression classifier to estimate the noise function.

Dataset	Number of Features	Number of Instances
Statlog (Heart)	$13$	$270$
Ionosphere	$34$	$351$
Connectionist Bench (Sonar)	$60$	$208$
Spambase	$57$	$4{,}601$
Covertype (subsample)	$54$	$50{,}000$

Table 1: Datasets Used for Experiments

Figure 4: Plot of Noise Model 2 for different values of entropy. Plots show that the function is decreasing in terms of labeler accuracy and increasing in terms of data point entropy.

Dataset	Sampling Budget	$M$	Capacity
Statlog (Heart)	$15$	$5$	$3$
Ionosphere	$20$	$5$	$4$
Connectionist Bench (Sonar)	$12$	$4$	$3$
Spambase	$258$	$17$	$16$
Covertype	$2800$	$54$	$52$

Table 2: Experimental Settings

Dataset	RS+RLA	RS+OLA	ES+RLA	ES+OLA	KC+RLA	MV	BADGE+RLA	OLAS
Statlog (Heart)	$0.404\pm 0.099$	$0.372\pm 0.112$	$0.364\pm 0.100$	$0.358\pm 0.128$	$0.376\pm 0.116$	$0.309\pm 0.135$	$0.366\pm 0.106$	$\mathbf{0.778\pm 0.048}$
Ionosphere	$0.907\pm 0.012$	$0.909\pm 0.014$	$0.915\pm 0.013$	$0.915\pm 0.013$	$\mathbf{0.918\pm 0.012}$	$0.891\pm 0.028$	$0.381\pm 0.100$	$0.916\pm 0.013$
Connectionist Bench (Sonar)	$0.718\pm 0.069$	$0.746\pm 0.053$	$0.706\pm 0.068$	$0.722\pm 0.062$	$0.712\pm 0.070$	$0.711\pm 0.054$	$0.698\pm 0.071$	$\mathbf{0.800\pm 0.035}$
Spambase	$0.904\pm 0.012$	$0.939\pm 0.006$	$0.879\pm 0.012$	$0.889\pm 0.014$	$0.896\pm 0.013$	$0.923\pm 0.008$	$0.915\pm 0.008$	$\mathbf{0.949\pm 0.003}$
Covertype	$0.396\pm 0.019$	$0.409\pm 0.021$	$0.388\pm 0.021$	$0.408\pm 0.030$	$0.398\pm 0.029$	$0.438\pm 0.030$	$0.354\pm 0.021$	$\mathbf{0.452\pm 0.020}$

Table 3: F1 Score Results Based on Noise Model 1 (mean

\pm

standard deviation)

Dataset	RS+RLA	RS+OLA	ES+RLA	ES+OLA	KC+RLA	MV	BADGE+RLA	OLAS
Statlog (Heart)	$0.230\pm 0.052$	$0.211\pm 0.047$	$0.205\pm 0.044$	$0.205\pm 0.045$	$0.235\pm 0.048$	$0.326\pm 0.118$	$0.265\pm 0.082$	$\mathbf{0.707\pm 0.104}$
Ionosphere	$0.099\pm 0.039$	$0.093\pm 0.028$	$0.108\pm 0.033$	$0.107\pm 0.031$	$0.140\pm 0.052$	$0.144\pm 0.075$	$0.129\pm 0.050$	$\mathbf{0.913\pm 0.014}$
Connectionist Bench (Sonar)	$0.302\pm 0.131$	$0.230\pm 0.134$	$0.179\pm 0.103$	$0.162\pm 0.091$	$0.286\pm 0.127$	$0.190\pm 0.139$	$0.130\pm 0.072$	$\mathbf{0.747\pm 0.042}$
Spambase	$0.230\pm 0.025$	$0.230\pm 0.076$	$0.143\pm 0.017$	$0.154\pm 0.027$	$0.219\pm 0.022$	$0.087\pm 0.016$	$0.382\pm 0.030$	$\mathbf{0.949\pm 0.003}$
Covertype	$0.006\pm 0.001$	$0.006\pm 0.002$	$0.009\pm 0.005$	$0.009\pm 0.004$	$0.005\pm 0.002$	$0.105\pm 0.038$	$0.006\pm 0.002$	$\mathbf{0.376\pm 0.032}$

Table 4: F1 Score Results Based on Noise Model 2 (mean

\pm

standard deviation)

Dataset	Noise	RS+RLA	RS+OLA	ES+RLA	ES+OLA	KC+RLA	MV	BADGE+RLA	OLAS
Statlog (Heart)	NM1	$0.191$	$0.191$	$0.197$	$0.192$	$0.194$	$0.182$	$0.200$	$0.181$
Statlog (Heart)	NM2	$0.188$	$0.181$	$0.190$	$0.193$	$0.198$	$0.187$	$0.182$	$0.173$
Ionosphere	NM1	$0.208$	$0.205$	$0.216$	$0.216$	$0.238$	$0.189$	$0.190$	$0.209$
Ionosphere	NM2	$0.212$	$0.213$	$0.219$	$0.219$	$0.243$	$0.191$	$0.188$	$0.209$
Connectionist Bench (Sonar)	NM1	$0.200$	$0.197$	$0.212$	$0.215$	$0.208$	$0.187$	$0.194$	$0.204$
Connectionist Bench (Sonar)	NM2	$0.208$	$0.208$	$0.213$	$0.209$	$0.209$	$0.185$	$0.190$	$0.194$

Table 5: Average Computation Time per AL Cycle (seconds)

Figure 5: Validation of the Section 3.4 estimation pipeline on the Music dataset. The fitted noise function $\hat{\epsilon}(a,e)$ , estimated from real annotator data on the gold held out subset, is shown with the estimated threshold $\beta=0.332$ drawn as a dashed contour.

Figure 6: Median F1 score across 10 active learning cycles on the Music dataset, averaged over 30 replications.

Figure 7: Box plot of final cycle (Cycle 10) macro F1 scores across 30 replications on the Music dataset.

Figure 8: Median F1 score over 30 replications across 10 active learning cycles on the Ford warranty claim dataset. OLAS consistently achieves the highest median F1 score across all cycles.

Figure 9: Box plot of F1 scores over 30 replications for the classifier at the final active learning cycle on the Ford warranty claim dataset. OLAS achieves the highest median with low variability across replications.

Dataset (NM)	Baseline	Mean diff.	95% CI	$p$ ( $t$ )	$p$ (W/MW)	Sig.
Heart Statlog (NM1)	RS+RLA	+0.3738	[0.353, 0.395]	$<0.001$	$<0.001$	Yes
Heart Statlog (NM1)	RS+OLA	+0.4060	[0.383, 0.429]	$<0.001$	$<0.001$	Yes
Heart Statlog (NM1)	ES+RLA	+0.4140	[0.393, 0.435]	$<0.001$	$<0.001$	Yes
Heart Statlog (NM1)	ES+OLA	+0.4198	[0.394, 0.445]	$<0.001$	$<0.001$	Yes
Heart Statlog (NM1)	MV	+0.4691	[0.442, 0.496]	$<0.001$	$<0.001$	Yes
Heart Statlog (NM1)	KC+RLA	+0.4016	[0.378, 0.425]	$<0.001$	$<0.001$	Yes

Table 6: Paired (or unpaired † , where noted) significance tests comparing OLAS to each baseline method, by dataset and noise model. Mean difference is OLAS minus the baseline; positive values favor OLAS.

Explore similar work

Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency
Jun 7, 2026Md Abdullah Al Forhad, Weishi ShiNoisy Labels Human Annotations
Active Machine Learning Lab · Learning · Language · Vision (LLaVi) Lab · Department of Computer Science and Engineering +1
How Many Labels Are Enough? ALDA: Active Learning Deployment Advisor for Medical Image Classification
Aug 4, 2026Julia Machnio, Mads Nielsen, Mostafa Mehdipour GhaziMedical Image Classification Label Propagation
Pioneer Centre for AI, University of Copenhagen, Copenhagen, Denmark
Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance
May 30, 2026Jiancheng Zhang, Meiqing Li, Qi Zhang +1Pool-Based Active Learning Active Learning
University of California, Riverside · 2Carnegie Mellon University · 3Worcester Polytechnic Institute

Jun 7, 2026, cs.LGJ/K move · Enter open · S save

Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency

Md Abdullah Al Forhad, Weishi Shi

Active Machine Learning Lab · Learning · Language · Vision (LLaVi) Lab · Department of Computer Science and Engineering+1

Noisy Labels Human Annotations Efficiency Introspection Adapters

Open paper AskPDF arXiv

While Deep Active Learning (DAL) effectively reduces human annotation costs, its efficacy is constrained by human annotation errors. This is because the data sampled for active learning is assumed to be highly informative for training. When human annotators introduce errors into this informative data at a certain rate, the active learning performance drops significantly and, in some cases, even exhibits worse outcomes than passive learning. In this paper, we first analyze the impact of human annotation errors in the DAL setting. Then we propose a framework to address the human annotation noise problem for DAL. Informed by human learning patterns, the core idea of our proposed solution involves allocating a portion of the human annotation budget to re-annotate data that has already been labeled. Previous theoretical work suggests that when the model possesses a certain level of ability to identify potentially noisy data, even re-labeling a small fraction of the data can effectively remove noise from the active training set. To achieve this, we implement two active noise sampling strategies to detect noise under different circumstances and allocate a part of the annotation budget to re-annotate these instances. Our approach imbues active learning with a revisiting and introspective behavior. Our experiments demonstrate that, under the same annotation budget, our method is more data-efficient and yields a relatively noise-free annotation dataset in the end.

Abstract

Explore similar work

Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency

How Many Labels Are Enough? ALDA: Active Learning Deployment Advisor for Medical Image Classification

Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance