Active learning (AL) reduces annotation by querying informative examples. Continual active learning (CAL) also cuts the training cost, the total number of gradient steps, by training one model across AL rounds with a fixed number of mini-batch steps per round. Pseudo-labels add supervision without annotation, but they increase training cost. In CAL, where every round trains on a fixed number of mini-batches, whether and how pseudo-labeled examples should share these mini-batches with labeled examples is therefore open. We provide the first formalization of this problem as two decisions after each query, namely (i) which unlabeled examples should enter training with a pseudo-label and (ii) what share of each mini-batch they should take. Our method, semi-supervised continual active learning (SSCAL), works with any acquisition function. It admits pseudo-labels based on distances from unlabeled examples to labeled neighbors, caps their share, and reweights the loss on labeled examples to keep their expected gradient contribution unchanged. On frozen embeddings, its pseudo-labeler updates incrementally without extra model forward passes. Across two audio datasets, two frozen encoders, six acquisition functions and ten seeds, SSCAL ranks first on average among four protocols with equal training steps, answering the question affirmatively.
Figures & tables
Setting
uniform
equal
new-only
SSCAL
epoch †
BSD10K / CLAP
57.79 ± 0.67 (2.08)
56.92 ± 0.53 (2.90)
51.20 ± 0.18 (4.00)
59.79 ± 0.53 (1.02) ∗
59.01 ± 0.42 (–)
BSD10K / PANNs
50.70 ± 0.55 (1.65)
49.66 ± 0.39 (2.77)
39.40 ± 0.36 (4.00)
50.86 ± 0.62 (1.58) ns
52.21 ± 0.65 (–)
FSD50K / CLAP
44.40 ± 0.60 (2.27)
43.94 ± 0.55 (2.73)
36.15 ± 0.60 (4.00)
48.95 ± 0.58 (1.00) ∗
56.16 ± 0.33 (–)
FSD50K / PANNs
45.69 ± 0.21 (2.01)
45.42 ± 0.26 (2.88)
36.43 ± 0.38 (4.00)
47.15 ± 0.17 (1.12) ∗
43.46 ± 0.15 (–)
First / 240
34
4
0
202
–
Table 1: AULC ( % , higher is better), mean ± s.d. over ten seeds, each averaging six acquisition functions. Parentheses give mean ranks among four fixed-step protocols within paired runs. ∗ marks p<0.001 and ns marks p=0.42 (two-sided paired Wilcoxon, 60 pairs per setting, unadjusted). The epoch † protocol runs 10.5× as many training steps, for reference only, and has no rank (–).
Acquisition
uniform
equal
new-only
SSCAL
epoch †
random
51.44 ± 0.48 (1.98)
50.84 ± 0.41 (2.90)
44.64 ± 0.34 (4.00)
52.64 ± 0.50 (1.13)
53.00 ± 0.48 (–)
entropy
45.28 ± 0.47 (2.08)
44.77 ± 0.55 (2.68)
33.17 ± 0.39 (4.00)
48.68 ± 0.65 (1.25)
50.32 ± 0.39 (–)
margin
46.20 ± 0.50 (2.15)
45.77 ± 0.51 (2.65)
35.65 ± 0.80 (4.00)
49.64 ± 0.63 (1.20)
51.35 ± 0.63 (–)
Core-set
49.86 ± 0.39 (1.78)
48.60 ± 0.33 (2.98)
40.07 ± 0.28 (4.00)
51.23 ± 0.53 (1.25)
52.36 ± 0.43 (–)
TypiClust
53.22 ± 0.33 (1.85)
52.52 ± 0.37 (3.00)
46.00 ± 0.24 (4.00)
54.35 ± 0.33 (1.15)
54.63 ± 0.33 (–)
BADGE
51.86 ± 0.46 (2.19)
51.42 ± 0.24 (2.71)
45.23 ± 0.24 (4.00)
53.58 ± 0.29 (1.10)
54.60 ± 0.41 (–)
Table 2: AULC ( % ), mean ± s.d. over ten seeds, each averaging four settings. The overall row also averages all six acquisition functions within each seed. Values combine BSD10K macro accuracy and FSD50K macro mAP. Parentheses give mean within-run ranks. The epoch † protocol runs 10.5× as many training steps, for reference only, and has no rank (–).
Figure 1: Aggregated test learning curves for the four settings against the labeled fraction of the training set. Each seed first averages six acquisition functions. Lines and shading show the mean and one standard deviation of these curves across ten seeds. The epoch † protocol runs 10.5× as many training steps and is shown for reference only.
ρ
β=0.1
β=0.2
β=0.4
0.6
+0.15 ± 0.11 (1.25)
+0.16 ± 0.11 (1.38)
+0.12 ± 0.14 (1.38)
0.8
+0.48 ± 0.09 (1.08)
+0.61 ± 0.08 (1.17)
+0.70 ± 0.11 (1.21)
1.0
+1.01 ± 0.12 (1.13)
+1.50 ± 0.15 (1.21)
+2.04 ± 0.15 (1.13)
Table 3: Paired Δ AULC versus uniform (points), mean ± s.d. across ten seeds after averaging all 24 configurations within each seed. Ranks in parentheses compare seed means against uniform, equal and new-only within configurations (Tables 1 – 2 use run ranks). Bold marks the best validation AULC.
Setting
Prec.
Cov.
∣Pt∣
Δ Wall (s)
SSCAL
epoch †
B / CLAP
83.2
52.8
4092
+1.36
+7.43
B / PANNs
75.4
48.0
3730
+0.94
+7.91
F / CLAP
95.2
85.7
28035
+1.36
+9.85
F / PANNs
87.2
71.3
23331
+0.25
+15.95
Table 4: Default SSCAL, with B and F for BSD10K and FSD50K. Prec. is pooled positive-label precision and Cov. is ∣Pt∣/∣Ut∣ , both in % , averaged with ∣Pt∣ over AL rounds and runs. Δ Wall is training-side wall time (training + policy) over initialization and 40 AL rounds in excess of uniform (0.95–2.27 s), averaged over 60 runs. SSCAL and uniform run 820 steps, epoch † 8,610 ( 10.5× , reference only).
Audio active learning (AL) trains a classifier at low annotation cost by labeling only an informative fraction of an unlabeled collection over successive rounds. Each round picks one of the checkpoints saved during training. Mainstream AL protocols do so with an assumed validation set, but in practice its labels also count toward the annotation cost, shrinking the advantage of an already small annotation budget. We propose geometric consistency, a label-free validation rule for checkpoint selection. Both classifiers, nearest-class-mean and k-nearest-neighbor, are built from the labeled set in a frozen pretrained embedding space. They then predict the unlabeled data to model its geometry. The rule scores each checkpoint by its agreement with these pseudo-labels and its cross-entropy on the labeled set. It works with any query strategy, has one hyperparameter, insensitive in practice, and adds only one batched forward pass over the unlabeled data per round. At matched label budgets across datasets, encoders, and query strategies, it attains the best mean rank in area under the labeling curve (AULC) among methods without extra labels and is on par with an out-of-budget validation set. To our knowledge, this is the first systematic study of label-free validation for checkpoint selection in audio AL.
Shiqi Zhang, Tuomas Virtanen
Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland
Over the past couple of decades, many active learning acquisition functions have been proposed, leaving practitioners with an unclear choice of which to use. Bayesian active learning offers principled objectives with explainable intuition, including Expected Error Reduction (EER), Expected Predictive Information Gain (EPIG), and Bayesian Active Learning by Disagreements (BALD). A key challenge of such methods is the difficulty of extending to the batch setting, leading to either computational challenges (BatchBALD) or dramatic performance drops (top-B selection). In this work, we extend active learning's Bayesian Decision Theory framework to derive Partial Batch Label Sampling (ParBaLS) for the EPIG algorithm. We show experimentally that ParBaLS EPIG gives superior performance for a variety of datasets with different budgets using Bayesian logistic regression on embeddings from large pre-trained models. Our code is available at https://github.com/ADDAPT-ML/ParBaLS.
Kangping Hu, Stephen Mussmann
Department of Computer Science, Georgia Institute of Technology, Atlanta, GA 30332
Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.
Pouya Ahadi, Blair Winograd, Camille Zaug +3
School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company