Over the past couple of decades, many active learning acquisition functions have been proposed, leaving practitioners with an unclear choice of which to use. Bayesian active learning offers principled objectives with explainable intuition, including Expected Error Reduction (EER), Expected Predictive Information Gain (EPIG), and Bayesian Active Learning by Disagreements (BALD). A key challenge of such methods is the difficulty of extending to the batch setting, leading to either computational challenges (BatchBALD) or dramatic performance drops (top-B selection). In this work, we extend active learning's Bayesian Decision Theory framework to derive Partial Batch Label Sampling (ParBaLS) for the EPIG algorithm. We show experimentally that ParBaLS EPIG gives superior performance for a variety of datasets with different budgets using Bayesian logistic regression on embeddings from large pre-trained models. Our code is available at https://github.com/ADDAPT-ML/ParBaLS.
Figures & tables
Figure 1: ParBaLS overview, with an example snapshot when the batch size is B=4 , the currently labeled points are L={7,8,9} , and the chosen but unlabeled points in the partial batch are S={2,4} . We see how the point {5} is selected by the inner loop and added to S . After this update, we assume another inner loop to select the point {1} and achieve ∣S∣=4=B . Then we execute the outer loop of sending the batch S to the oracle annotator for ground-truth labels, which are added to L for the next iteration.
Algorithm
Bayesian Tables
Heuristics Tables
Highest Mean (HM)
Ties with HM
Wins − Losses
Highest Loss Rate ↓
Highest Mean (HM)
Ties with HM
Wins − Losses
Highest Loss Rate ↓
Random
-
-
-
-
1.65%
36.09%
-1.08
61.71%
Confidence
-
-
-
-
14.33%
71.35%
+1.56
25.62%
GLISTER
-
-
-
-
2.20%
26.72%
-3.07
75.76%
CoreSet
-
-
-
-
1.93%
28.93%
-4.03
71.90%
BADGE
-
-
-
-
2.75%
39.12%
-0.95
61.98%
Table 1: Leaderboard across all 11 datasets and three budget settings, evaluated separately at each iteration. Highest Mean (HM) and Ties with HM show the percentages of available dataset-budget-iteration evaluations. Wins − Losses is the sum of normalized pairwise win rates in a strategy’s Figure 8 row minus the corresponding column sum. Highest Loss Rate is the worst loss rate of each algorithm in Figure 8 , where lower is better. BatchBALD is evaluated only for the B=10 budget setting.
Figure 2: Learning curves of Bayesian AL algorithms with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Figure 3: Learning curves of ParBaLS and heuristics baselines with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Figure 4: Comparison between batch acquisition and single selection with a total of 300 labeling budget, starting with random initialization of 100 samples.
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: An illustrative diagram for Active Learning.
Figure 6: An illustrative diagram for ParBaLS.
Figure 7: An illustration of the proposed method, ParBaLS.
Dataset
Encoder
Before PCA
After PCA
CIFAR-10
DINOv2-ViT-S/14
384
348
iWildCam
CLIP-ViT-B/32
512
305
fMoW
CLIP-ViT-B/32
512
342
AG News
bert-base-uncased
768
210
Yelp Review Full
bert-base-uncased
768
168
Civil Comments
bert-base-uncased
768
187
Appendix
Table 2: Encoders used for each dataset and the embedding dimensions before and after 99% PCA.
Figure 8: Pairwise statistical-significance win matrices across all acquisition rounds.
Metric
Mean improvement
95% interval
Final test accuracy
+1.54 pp
[+0.46,+2.62]
Iteration-averaged test accuracy
+1.25 pp
[+0.60,+1.91]
Final test NLL reduction
0.0112
[0.0052,0.0172]
Iteration-averaged test NLL reduction
0.0086
[0.0057,0.0115]
Appendix
Table 3: Paired global comparison of ParBaLS with Top- B EPIG across 10 seeds, 11 datasets, and 3 budget settings.
Initial Budget
Acquisition Budget
Bayesian Methods
Heuristic Methods
100
10×20
Table 7
Table 10
20
10×20
Table 8
Table 11
100
10×10
Table 9
Table 12
Appendix
Table 4: Guide to the main active learning result tables. “Initial Budget” denotes the number of randomly labeled samples used for initialization. “Acquisition Budget” gives the number of active learning iterations and the per-iteration labeling budget.
Datasets
Airline Passenger
Credit Card
Algorithm
Satisfaction
Fraud
CIFAR-10
ParBaLS-MAP EPIG
89.34 ± 0.46
93.45 ± 0.29
93.40 ± 1.19
ParBaLS EPIG ( m=5 )
89.45 ± 0.58
93.37 ± 0.13
93.80 ± 1.18
ParBaLS EPIG ( m=10 )
89.35 ± 0.60
93.45 ± 0.22
94.90 ± 0.54
ParBaLS EPIG ( m=20 )
89.70 ± 0.55
93.49 ± 0.20
94.17 ± 0.72
Appendix
Table 5: Final test accuracy of different variants of ParBaLS with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 20 samples, except for the first iteration that starts with 100 samples.
Dataset
Initial Budget
Acquisition Budget
Bayesian Methods
Heuristic Methods
CIFAR-10
20
10×20
Table 13
Table 15
CIFAR-10
100
10×20
Table 14
Table 16
iWildCam
20
10×20
Table 17
Table 19
iWildCam
100
10×20
Table 18
Table 20
fMoW
20
10×20
Table 21
Table 23
fMoW
100
10×20
Table 22
Table 24
Appendix
Table 6: Guide to the one-vs-all active learning result tables. “Initial Budget” denotes the number of randomly labeled samples used for initialization. “Acquisition Budget” gives the number of active learning iterations and the per-iteration labeling budget.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Top- B BALD
87.07 ± 0.83
93.41 ± 0.18
68.73 ± 2.09
82.59 ± 0.28
43.46 ± 0.20
82.19 ± 2.49
80.19 ± 1.80
71.58 ± 1.41
88.25 ± 0.93
78.62 ± 2.83
91.78 ± 3.84
PowerBALD
88.43 ± 0.46
93.18 ± 0.20
65.09 ± 1.33
82.69 ± 0.37
43.64 ± 0.43
86.67 ± 0.89
85.17 ± 1.02
78.51 ± 1.14
90.06 ± 0.75
82.93 ± 4.62
91.42 ± 1.18
SoftmaxBALD
87.42 ± 0.74
92.75 ± 0.24
65.11 ± 2.36
82.03 ± 0.61
43.23 ± 0.68
85.25 ± 0.98
85.29 ± 0.75
78.11 ± 1.22
88.90 ± 0.79
81.99 ± 4.63
90.44 ± 1.51
SoftRankBALD
88.65 ± 0.45
93.34 ± 0.13
65.66 ± 1.82
82.41 ± 0.23
43.59 ± 0.37
86.35 ± 1.64
84.47 ± 1.82
75.39 ± 1.28
91.74 ± 0.86
85.27 ± 1.23
94.12 ± 2.02
Top- B EPIG
89.05 ± 0.44
93.35 ± 0.14
65.45 ± 2.26
82.70 ± 0.19
43.51 ± 0.19
88.41 ± 0.95
87.54 ± 0.79
79.30 ± 0.99
93.82 ± 1.02
85.80 ± 3.00
97.06 ± 0.72
Appendix
Table 7: Bayesian AL comparison: 100+10×20 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Top- B BALD
84.50 ± 2.10
93.07 ± 0.42
67.72 ± 4.32
82.52 ± 0.21
41.30 ± 0.61
70.83 ± 1.87
69.93 ± 2.01
63.60 ± 1.22
77.93 ± 1.06
66.73 ± 4.06
75.47 ± 2.40
PowerBALD
87.62 ± 0.62
93.09 ± 0.33
67.40 ± 1.76
82.29 ± 0.49
41.88 ± 1.90
82.40 ± 1.52
82.21 ± 1.08
77.86 ± 0.89
83.22 ± 0.72
79.84 ± 4.44
86.60 ± 2.14
SoftmaxBALD
86.70 ± 0.59
93.08 ± 0.30
66.66 ± 2.28
81.81 ± 0.66
41.60 ± 1.43
83.03 ± 1.39
81.96 ± 1.41
76.98 ± 1.26
83.86 ± 1.11
81.31 ± 4.93
87.06 ± 1.78
SoftRankBALD
87.88 ± 0.46
93.43 ± 0.28
65.01 ± 2.20
81.71 ± 0.37
41.84 ± 0.74
82.89 ± 2.58
78.80 ± 1.93
69.87 ± 1.31
82.67 ± 0.95
73.53 ± 3.80
82.98 ± 2.15
Top- B EPIG
88.31 ± 0.33
93.26 ± 0.28
64.97 ± 3.08
81.89 ± 0.28
41.28 ± 1.31
86.37 ± 1.20
84.33 ± 1.29
77.31 ± 0.84
87.17 ± 0.97
75.71 ± 6.66
92.03 ± 0.95
Appendix
Table 8: Bayesian AL comparison: 20+10×20 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Top- B BALD
86.94 ± 0.75
93.28 ± 0.20
66.97 ± 1.74
81.44 ± 0.56
41.80 ± 0.87
79.63 ± 1.71
77.04 ± 1.65
69.61 ± 1.46
79.35 ± 1.12
74.41 ± 3.02
81.79 ± 1.46
PowerBALD
87.14 ± 0.54
93.12 ± 0.25
66.66 ± 2.36
81.75 ± 0.48
41.28 ± 1.14
81.18 ± 1.36
81.49 ± 1.13
76.37 ± 1.09
82.20 ± 1.04
79.49 ± 5.06
84.31 ± 1.30
SoftmaxBALD
87.01 ± 0.71
92.69 ± 0.34
65.27 ± 1.18
81.66 ± 0.66
41.65 ± 1.80
81.47 ± 1.69
80.90 ± 1.49
76.16 ± 1.04
82.04 ± 0.95
81.72 ± 4.30
84.18 ± 1.37
SoftRankBALD
87.89 ± 0.51
93.51 ± 0.17
66.05 ± 1.83
81.57 ± 0.39
41.65 ± 1.32
82.56 ± 2.04
80.66 ± 1.70
72.54 ± 1.00
82.62 ± 1.24
77.03 ± 3.68
82.52 ± 1.43
BatchBALD
86.69 ± 0.72
93.27 ± 0.09
67.98 ± 1.89
81.39 ± 0.77
41.80 ± 0.91
82.63 ± 2.68
78.66 ± 1.43
70.82 ± 0.87
81.90 ± 1.39
76.92 ± 3.27
84.52 ± 1.45
Appendix
Table 9: Bayesian AL comparison: 100+10×10 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Random
87.24 ± 0.52
92.84 ± 0.29
65.77 ± 2.10
82.40 ± 0.48
43.55 ± 0.52
84.80 ± 1.25
83.92 ± 0.75
77.61 ± 0.96
88.39 ± 0.48
82.30 ± 3.90
90.02 ± 1.49
Confidence
89.35 ± 0.33
92.71 ± 0.41
66.71 ± 1.74
82.44 ± 0.59
43.33 ± 0.36
89.67 ± 0.74
87.87 ± 0.78
80.11 ± 1.01
96.46 ± 0.44
90.24 ± 3.00
98.96 ± 0.61
GLISTER
87.43 ± 0.73
93.42 ± 0.35
62.25 ± 4.61
80.86 ± 0.65
43.16 ± 0.62
84.13 ± 1.53
82.12 ± 0.89
73.07 ± 1.02
91.65 ± 1.01
79.29 ± 2.99
91.69 ± 3.28
CoreSet
85.87 ± 0.91
93.05 ± 0.24
66.28 ± 1.53
81.02 ± 0.43
43.46 ± 0.32
71.58 ± 1.42
74.59 ± 1.46
69.26 ± 1.12
73.75 ± 0.71
71.95 ± 3.12
73.35 ± 3.33
BADGE
87.88 ± 0.72
92.77 ± 0.30
66.31 ± 1.58
82.35 ± 0.55
43.40 ± 0.24
85.04 ± 1.64
83.84 ± 0.89
77.94 ± 1.23
88.19 ± 0.73
85.95 ± 2.69
92.54 ± 1.46
Appendix
Table 10: Heuristics AL comparison: 100+10×20 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Random
86.17 ± 0.65
92.41 ± 0.38
65.92 ± 1.35
80.99 ± 0.72
42.01 ± 1.34
82.63 ± 1.60
80.69 ± 0.89
76.47 ± 0.83
83.47 ± 1.16
78.58 ± 2.95
85.47 ± 1.33
Confidence
88.61 ± 0.52
92.22 ± 0.49
66.02 ± 2.90
81.50 ± 0.60
43.42 ± 1.40
87.28 ± 1.55
85.34 ± 0.84
78.69 ± 1.09
90.29 ± 1.35
86.65 ± 2.30
94.69 ± 1.61
GLISTER
86.02 ± 1.11
92.40 ± 0.89
65.97 ± 4.94
76.05 ± 2.55
40.54 ± 1.94
72.36 ± 2.50
72.12 ± 2.32
65.88 ± 1.50
78.71 ± 2.03
67.11 ± 3.71
75.89 ± 4.56
CoreSet
80.57 ± 2.04
93.28 ± 0.17
68.86 ± 3.71
80.02 ± 1.14
38.98 ± 0.71
57.48 ± 0.76
63.18 ± 1.11
58.95 ± 1.53
62.60 ± 0.56
60.13 ± 1.95
57.75 ± 2.46
BADGE
87.09 ± 0.58
93.18 ± 0.23
64.85 ± 2.31
81.42 ± 0.52
41.84 ± 1.54
83.07 ± 1.42
81.27 ± 0.85
75.50 ± 0.82
83.59 ± 0.77
83.97 ± 2.22
85.81 ± 1.48
Appendix
Table 11: Heuristics AL comparison: 20+10×20 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Datasets
Tabular Datasets
Text Datasets
Image Datasets
Algorithm
Airline
Credit Card
Magic
Satellite
Vowels
Civil Comments
AG News
Yelp
CIFAR-10
iWildCam
fMoW
Random
86.65 ± 0.81
92.44 ± 0.27
66.16 ± 1.99
81.95 ± 0.41
40.71 ± 2.00
80.88 ± 1.46
80.64 ± 1.41
75.39 ± 1.12
82.78 ± 1.19
78.31 ± 5.05
84.70 ± 1.42
Confidence
88.15 ± 0.66
92.82 ± 0.41
66.48 ± 1.99
81.14 ± 0.66
42.47 ± 1.56
86.43 ± 1.79
83.78 ± 1.09
77.31 ± 1.10
87.31 ± 0.87
83.55 ± 3.11
91.78 ± 1.34
GLISTER
87.14 ± 0.64
93.30 ± 0.39
63.07 ± 4.32
80.88 ± 0.68
39.55 ± 1.64
80.19 ± 2.57
77.92 ± 1.93
71.04 ± 1.19
82.78 ± 1.22
73.29 ± 4.52
81.78 ± 2.11
CoreSet
86.18 ± 0.63
93.32 ± 0.30
64.94 ± 2.37
81.16 ± 0.86
40.71 ± 0.75
72.33 ± 1.45
73.23 ± 1.58
68.88 ± 1.44
72.26 ± 0.78
69.24 ± 2.68
72.79 ± 2.19
BADGE
86.94 ± 0.46
92.88 ± 0.48
65.70 ± 2.09
82.26 ± 0.51
41.80 ± 2.02
79.76 ± 1.79
79.44 ± 1.33
75.19 ± 1.08
81.44 ± 0.75
80.40 ± 3.32
82.56 ± 1.49
Appendix
Table 12: Heuristics AL comparison: 100+10×10 . The reported number is final test accuracy averaged over 10 seeds with ± defining the 95% confidence interval using Student’s t -distribution. The highest mean accuracy is underlined, and all methods with means that are not significantly different ( p -value greater than or equal to 0.05 ) according to Welch’s t -test are bolded.
Figure 9: Learning curves of Bayesian AL algorithms with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Figure 10: Learning curves of ParBaLS and heuristics baselines with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Figure 11: Learning curves of Bayesian AL algorithms with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 10 samples, starting with random initialization of 100 samples.
Figure 12: Learning curves of ParBaLS and heuristics baselines with Bayesian Logistic Regression, where each of the 10 iterations has a labeling budget of 10 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all CIFAR-10
Algorithm
0
1
2
3
4
5
6
7
8
9
Top- B BALD
77.93 ± 1.06
82.13 ± 0.89
76.68 ± 1.52
77.13 ± 1.02
82.07 ± 1.14
79.52 ± 0.91
79.18 ± 0.92
83.30 ± 0.79
80.92 ± 1.11
80.79 ± 0.85
PowerBALD
83.22 ± 0.72
84.28 ± 0.48
84.06 ± 0.94
81.79 ± 0.74
82.14 ± 1.45
82.03 ± 0.69
84.20 ± 1.07
84.97 ± 0.73
84.42 ± 0.64
84.32 ± 0.60
SoftmaxBALD
83.86 ± 1.11
83.43 ± 0.54
83.34 ± 0.72
81.38 ± 1.26
81.64 ± 1.16
82.76 ± 0.76
84.00 ± 0.95
84.07 ± 1.11
84.04 ± 0.88
83.87 ± 0.84
SoftRankBALD
82.67 ± 0.95
85.70 ± 0.94
81.82 ± 0.91
80.57 ± 0.71
84.24 ± 1.02
82.67 ± 0.89
83.24 ± 0.95
86.18 ± 1.19
84.28 ± 1.10
84.96 ± 0.73
Top- B EPIG
87.17 ± 0.97
87.41 ± 0.84
86.06 ± 1.22
85.17 ± 0.91
85.42 ± 0.73
86.60 ± 0.69
88.00 ± 0.75
88.65 ± 1.09
88.25 ± 1.04
87.60 ± 0.83
Appendix
Table 13: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all CIFAR-10, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all CIFAR-10
Algorithm
0
1
2
3
4
5
6
7
8
9
Top- B BALD
88.25 ± 0.93
92.87 ± 0.64
86.98 ± 0.95
85.79 ± 1.10
91.68 ± 1.04
89.07 ± 1.04
89.23 ± 0.59
94.05 ± 1.17
91.70 ± 0.65
91.84 ± 0.83
PowerBALD
90.06 ± 0.75
90.22 ± 0.89
89.17 ± 1.39
87.22 ± 1.09
88.71 ± 0.87
87.95 ± 1.36
89.53 ± 1.15
90.54 ± 0.87
90.95 ± 1.00
89.13 ± 0.70
SoftmaxBALD
88.90 ± 0.79
89.88 ± 1.07
88.78 ± 0.96
87.46 ± 0.71
87.58 ± 1.30
88.07 ± 1.14
89.12 ± 0.54
89.72 ± 1.21
90.37 ± 0.78
89.96 ± 1.07
SoftRankBALD
91.74 ± 0.86
94.65 ± 0.49
91.00 ± 0.92
88.83 ± 0.64
91.84 ± 1.14
91.46 ± 0.99
91.65 ± 0.71
94.99 ± 0.55
93.45 ± 0.98
93.11 ± 0.63
Top- B EPIG
93.82 ± 1.02
94.50 ± 0.95
93.23 ± 1.09
91.24 ± 0.71
92.12 ± 0.85
92.64 ± 0.62
94.84 ± 0.41
94.48 ± 0.88
94.98 ± 0.52
94.74 ± 0.87
Appendix
Table 14: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all CIFAR-10, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all CIFAR-10
Algorithm
0
1
2
3
4
5
6
7
8
9
Random
83.47 ± 1.16
83.93 ± 0.75
82.99 ± 1.16
81.90 ± 0.75
82.58 ± 1.05
82.60 ± 1.21
83.94 ± 0.97
84.36 ± 0.89
83.70 ± 0.90
83.51 ± 1.03
Confidence
90.29 ± 1.35
91.16 ± 0.97
89.48 ± 0.70
87.04 ± 1.07
88.47 ± 0.67
88.55 ± 0.96
90.85 ± 0.73
91.32 ± 0.93
91.66 ± 0.55
90.63 ± 0.82
GLISTER
78.71 ± 2.03
81.66 ± 1.70
77.62 ± 1.38
78.59 ± 0.85
79.34 ± 1.03
79.14 ± 2.04
80.82 ± 2.01
80.57 ± 0.91
81.00 ± 1.89
79.81 ± 2.81
CoreSet
62.60 ± 0.56
63.46 ± 0.62
61.54 ± 0.76
62.10 ± 0.53
63.35 ± 0.64
63.05 ± 0.46
64.33 ± 0.66
62.83 ± 0.32
63.24 ± 0.71
62.94 ± 0.59
BADGE
83.59 ± 0.77
83.15 ± 0.99
82.45 ± 1.21
81.40 ± 0.94
82.07 ± 0.92
82.39 ± 1.08
83.25 ± 1.12
84.31 ± 0.94
84.25 ± 1.02
84.02 ± 1.28
Appendix
Table 15: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all CIFAR-10, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all CIFAR-10
Algorithm
0
1
2
3
4
5
6
7
8
9
Random
88.39 ± 0.48
89.21 ± 0.91
87.62 ± 0.84
86.23 ± 0.55
87.60 ± 0.87
86.86 ± 0.91
88.78 ± 1.09
88.69 ± 0.87
89.14 ± 0.77
88.60 ± 1.06
Confidence
96.46 ± 0.44
97.55 ± 0.43
95.44 ± 0.64
93.21 ± 0.74
94.75 ± 0.53
94.69 ± 0.60
96.97 ± 0.22
97.82 ± 0.44
97.87 ± 0.40
96.82 ± 0.33
GLISTER
91.65 ± 1.01
92.88 ± 0.73
89.61 ± 1.92
89.22 ± 1.21
91.22 ± 1.14
89.46 ± 1.33
92.36 ± 1.12
93.78 ± 1.26
92.01 ± 1.73
92.75 ± 0.99
CoreSet
73.75 ± 0.71
75.39 ± 0.90
73.36 ± 0.74
72.20 ± 0.95
74.11 ± 0.67
74.42 ± 0.93
74.87 ± 0.89
74.44 ± 0.90
74.69 ± 0.67
73.87 ± 0.87
BADGE
88.19 ± 0.73
89.96 ± 0.98
87.66 ± 0.99
86.47 ± 0.93
88.06 ± 0.99
88.12 ± 0.61
88.87 ± 0.67
90.70 ± 0.89
89.95 ± 1.00
89.09 ± 0.44
Appendix
Table 16: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all CIFAR-10, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all iWildCam
Algorithm
0
1
2
3
4
Top- B BALD
66.73 ± 4.06
74.30 ± 2.91
76.66 ± 3.20
78.34 ± 2.10
75.41 ± 3.17
PowerBALD
79.84 ± 4.44
83.63 ± 4.00
82.14 ± 3.03
86.92 ± 4.52
82.66 ± 3.74
SoftmaxBALD
81.31 ± 4.93
82.49 ± 3.18
82.78 ± 2.12
85.52 ± 4.76
82.28 ± 3.63
SoftRankBALD
73.53 ± 3.80
83.37 ± 3.51
83.00 ± 4.28
84.90 ± 3.68
83.49 ± 2.48
Top- B EPIG
75.71 ± 6.66
78.56 ± 5.40
74.86 ± 4.80
83.96 ± 4.15
80.81 ± 4.45
Appendix
Table 17: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all iWildCam, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all iWildCam
Algorithm
0
1
2
3
4
Top- B BALD
78.62 ± 2.83
87.80 ± 2.69
89.33 ± 3.67
93.68 ± 2.35
90.42 ± 1.99
PowerBALD
82.93 ± 4.62
86.18 ± 3.40
83.73 ± 5.29
91.36 ± 2.64
87.60 ± 2.58
SoftmaxBALD
81.99 ± 4.63
85.45 ± 3.94
84.44 ± 5.00
91.02 ± 3.04
87.43 ± 3.46
SoftRankBALD
85.27 ± 1.23
89.87 ± 2.99
90.97 ± 3.45
94.98 ± 4.25
91.25 ± 3.03
Top- B EPIG
85.80 ± 3.00
87.68 ± 2.75
81.24 ± 4.56
91.66 ± 3.95
86.56 ± 4.49
Appendix
Table 18: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all iWildCam, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all iWildCam
Algorithm
0
1
2
3
4
Random
78.58 ± 2.95
81.47 ± 4.14
80.23 ± 5.51
85.29 ± 4.27
83.42 ± 3.90
Confidence
86.65 ± 2.30
92.47 ± 1.76
90.27 ± 2.09
95.02 ± 3.61
92.41 ± 2.02
GLISTER
67.11 ± 3.71
70.73 ± 4.06
72.56 ± 5.24
74.70 ± 3.83
74.15 ± 3.04
CoreSet
60.13 ± 1.95
63.37 ± 3.28
57.61 ± 5.08
60.50 ± 4.09
53.88 ± 3.31
BADGE
83.97 ± 2.22
83.01 ± 2.69
80.29 ± 3.62
87.80 ± 3.63
86.10 ± 3.91
Appendix
Table 19: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all iWildCam, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all iWildCam
Algorithm
0
1
2
3
4
Random
82.30 ± 3.90
83.00 ± 4.90
81.70 ± 5.83
88.41 ± 4.18
85.87 ± 2.98
Confidence
90.24 ± 3.00
96.69 ± 2.03
94.12 ± 1.71
99.78 ± 0.02
96.51 ± 1.68
GLISTER
79.29 ± 2.99
85.97 ± 2.72
88.90 ± 3.35
92.28 ± 2.48
89.59 ± 2.13
CoreSet
71.95 ± 3.12
73.24 ± 3.24
71.84 ± 5.41
73.57 ± 4.55
66.75 ± 4.65
BADGE
85.95 ± 2.69
87.23 ± 4.01
86.00 ± 5.70
92.35 ± 3.98
88.90 ± 3.14
Appendix
Table 20: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all iWildCam, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all fMoW
Algorithm
0
1
2
3
4
Top- B BALD
75.47 ± 2.40
74.41 ± 1.92
73.43 ± 2.37
71.15 ± 1.24
74.55 ± 1.91
PowerBALD
86.60 ± 2.14
84.77 ± 1.35
85.61 ± 1.80
83.91 ± 1.64
86.96 ± 1.52
SoftmaxBALD
87.06 ± 1.78
84.85 ± 1.37
85.04 ± 1.38
83.85 ± 1.40
86.21 ± 1.54
SoftRankBALD
82.98 ± 2.15
81.86 ± 1.66
81.18 ± 2.22
79.76 ± 2.60
83.88 ± 2.52
Top- B EPIG
92.03 ± 0.95
90.12 ± 0.64
91.27 ± 0.99
87.99 ± 1.24
91.91 ± 0.90
Appendix
Table 21: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all fMoW, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all fMoW
Algorithm
0
1
2
3
4
Top- B BALD
91.78 ± 3.84
88.81 ± 2.49
89.80 ± 2.10
84.56 ± 2.98
91.57 ± 2.80
PowerBALD
91.42 ± 1.18
89.52 ± 1.91
90.24 ± 0.94
88.55 ± 1.37
91.11 ± 1.19
SoftmaxBALD
90.44 ± 1.51
88.33 ± 1.36
89.09 ± 1.17
88.08 ± 1.54
90.64 ± 1.06
SoftRankBALD
94.12 ± 2.02
90.59 ± 2.79
92.58 ± 1.52
88.80 ± 2.16
94.44 ± 2.08
Top- B EPIG
97.06 ± 0.72
94.67 ± 0.68
96.25 ± 0.28
93.80 ± 0.88
96.96 ± 0.59
Appendix
Table 22: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all fMoW, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all fMoW
Algorithm
0
1
2
3
4
Random
85.47 ± 1.33
83.47 ± 1.38
84.42 ± 1.09
82.61 ± 1.91
85.39 ± 1.33
Confidence
94.69 ± 1.61
91.84 ± 1.06
93.09 ± 1.44
91.44 ± 1.41
94.65 ± 1.41
GLISTER
75.89 ± 4.56
74.36 ± 4.60
73.75 ± 4.43
71.15 ± 4.58
75.09 ± 3.97
CoreSet
57.75 ± 2.46
56.98 ± 2.28
58.71 ± 2.14
58.43 ± 2.21
57.29 ± 2.11
BADGE
85.81 ± 1.48
83.89 ± 1.11
85.21 ± 1.21
82.97 ± 1.82
86.48 ± 0.94
Appendix
Table 23: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all fMoW, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all fMoW
Algorithm
0
1
2
3
4
Random
90.02 ± 1.49
88.00 ± 2.00
89.04 ± 1.77
86.66 ± 2.10
89.89 ± 1.44
Confidence
98.96 ± 0.61
97.42 ± 0.82
98.20 ± 0.43
95.84 ± 1.38
99.20 ± 0.27
GLISTER
91.69 ± 3.28
89.51 ± 3.01
89.75 ± 3.36
87.01 ± 3.99
92.18 ± 3.25
CoreSet
73.35 ± 3.33
72.14 ± 3.12
73.52 ± 2.69
71.78 ± 3.13
72.91 ± 2.95
BADGE
92.54 ± 1.46
90.66 ± 0.95
91.13 ± 1.01
88.84 ± 1.41
91.50 ± 1.04
Appendix
Table 24: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all fMoW, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all AG News
Algorithm
0
1
2
3
Top- B BALD
69.93 ± 2.01
76.70 ± 1.37
69.21 ± 1.30
68.83 ± 1.10
PowerBALD
82.21 ± 1.08
88.81 ± 1.09
77.43 ± 1.50
78.93 ± 1.16
SoftmaxBALD
81.96 ± 1.41
87.83 ± 1.36
78.30 ± 0.98
79.47 ± 0.85
SoftRankBALD
78.80 ± 1.93
86.64 ± 1.13
75.52 ± 1.11
77.73 ± 1.40
Top- B EPIG
84.33 ± 1.29
91.11 ± 1.02
78.61 ± 0.88
81.27 ± 0.80
Appendix
Table 25: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all AG News, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all AG News
Algorithm
0
1
2
3
Top- B BALD
80.19 ± 1.80
88.53 ± 1.30
78.79 ± 1.17
78.47 ± 1.49
PowerBALD
85.17 ± 1.02
91.46 ± 0.81
80.03 ± 0.96
82.53 ± 0.67
SoftmaxBALD
85.29 ± 0.75
91.45 ± 0.82
79.31 ± 1.14
81.87 ± 1.01
SoftRankBALD
84.47 ± 1.82
92.72 ± 1.27
80.89 ± 1.18
82.63 ± 0.93
Top- B EPIG
87.54 ± 0.79
94.45 ± 0.68
81.79 ± 0.98
84.14 ± 0.71
Appendix
Table 26: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all AG News, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all AG News
Algorithm
0
1
2
3
Random
80.69 ± 0.89
87.59 ± 1.17
76.29 ± 1.03
78.02 ± 1.32
Confidence
85.34 ± 0.84
92.55 ± 0.69
80.02 ± 1.19
82.35 ± 1.22
GLISTER
72.12 ± 2.32
77.76 ± 3.07
71.09 ± 1.86
70.69 ± 2.01
CoreSet
63.18 ± 1.11
65.31 ± 0.79
61.13 ± 1.11
63.44 ± 0.78
BADGE
81.27 ± 0.85
86.75 ± 0.86
75.79 ± 1.54
78.40 ± 0.85
Appendix
Table 27: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all AG News, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all AG News
Algorithm
0
1
2
3
Random
83.92 ± 0.75
90.92 ± 0.77
79.14 ± 1.04
80.90 ± 1.58
Confidence
87.87 ± 0.78
94.82 ± 0.76
82.49 ± 1.35
85.08 ± 0.78
GLISTER
82.12 ± 0.89
90.22 ± 1.06
79.19 ± 1.32
80.24 ± 1.14
CoreSet
74.59 ± 1.46
79.80 ± 1.14
71.15 ± 1.67
73.64 ± 1.42
BADGE
83.84 ± 0.89
90.65 ± 1.06
79.81 ± 0.94
81.15 ± 1.31
Appendix
Table 28: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all AG News, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all Yelp
Algorithm
0
1
2
3
4
Top- B BALD
63.60 ± 1.22
58.83 ± 1.17
60.13 ± 1.04
60.87 ± 1.04
65.27 ± 0.79
PowerBALD
77.86 ± 0.89
68.33 ± 1.43
67.15 ± 1.46
68.47 ± 2.03
74.82 ± 1.17
SoftmaxBALD
76.98 ± 1.26
67.85 ± 1.42
67.68 ± 1.01
68.42 ± 1.10
74.47 ± 0.82
SoftRankBALD
69.87 ± 1.31
64.46 ± 1.81
64.86 ± 1.32
65.79 ± 0.68
73.24 ± 1.63
Top- B EPIG
77.31 ± 0.84
68.76 ± 1.02
69.06 ± 0.83
68.82 ± 0.96
77.09 ± 0.93
Appendix
Table 29: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all Yelp, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all Yelp
Algorithm
0
1
2
3
4
Top- B BALD
71.58 ± 1.41
64.94 ± 1.34
65.26 ± 1.33
65.78 ± 1.07
71.63 ± 0.93
PowerBALD
78.51 ± 1.14
68.51 ± 1.27
69.00 ± 0.83
68.79 ± 1.24
76.16 ± 1.10
SoftmaxBALD
78.11 ± 1.22
70.08 ± 1.19
69.26 ± 0.95
69.58 ± 1.31
76.28 ± 0.68
SoftRankBALD
75.39 ± 1.28
68.48 ± 1.32
68.68 ± 1.30
69.57 ± 1.22
76.63 ± 1.07
Top- B EPIG
79.30 ± 0.99
71.30 ± 0.86
71.07 ± 0.79
71.09 ± 0.63
78.56 ± 0.90
Appendix
Table 30: Final test accuracy of Bayesian AL algorithms with Bayesian Logistic Regression on One-vs-all Yelp, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
Datasets
One-vs-all Yelp
Algorithm
0
1
2
3
4
Random
76.47 ± 0.83
68.73 ± 1.32
67.67 ± 1.42
66.99 ± 1.15
74.00 ± 1.12
Confidence
78.69 ± 1.09
69.85 ± 1.55
70.32 ± 0.87
70.35 ± 0.76
76.80 ± 1.18
GLISTER
65.88 ± 1.50
60.10 ± 2.48
61.08 ± 2.40
62.66 ± 1.59
66.71 ± 1.91
CoreSet
58.95 ± 1.53
55.78 ± 0.62
55.85 ± 1.23
56.79 ± 0.86
58.90 ± 1.12
BADGE
75.50 ± 0.82
68.10 ± 0.96
66.29 ± 0.87
67.83 ± 1.17
74.18 ± 0.90
Appendix
Table 31: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all Yelp, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 20 samples.
Datasets
One-vs-all Yelp
Algorithm
0
1
2
3
4
Random
77.61 ± 0.96
69.03 ± 1.00
68.31 ± 0.95
69.45 ± 1.31
76.51 ± 0.85
Confidence
80.11 ± 1.01
71.63 ± 1.03
71.04 ± 0.84
71.44 ± 1.00
78.72 ± 0.97
GLISTER
73.07 ± 1.02
68.39 ± 1.41
66.46 ± 1.92
68.57 ± 1.01
74.49 ± 1.26
CoreSet
69.26 ± 1.12
63.23 ± 1.27
64.07 ± 1.29
64.56 ± 1.17
69.09 ± 1.23
BADGE
77.94 ± 1.23
68.74 ± 0.80
68.96 ± 1.39
68.40 ± 0.92
75.88 ± 1.29
Appendix
Table 32: Final test accuracy of ParBaLS and heuristic baselines with Bayesian Logistic Regression on One-vs-all Yelp, where each of the 10 iterations has a labeling budget of 20 samples, starting with random initialization of 100 samples.
The central goal of active learning is to gather data that maximises downstream predictive performance, but popular approaches have limited flexibility in customising this data acquisition to different downstream problems and losses. We propose a rigorous loss-driven approach to Bayesian active learning that allows data acquisition to directly target the loss associated with a given decision problem. In particular, we show how any loss can be used to derive a unique objective for optimal data acquisition. Critically, we then show that any loss taking the form of a weighted Bregman divergence permits analytic computation of a central component of its corresponding objective, making the approach applicable in practice. In regression and classification experiments with a range of different losses, we find our approach reduces test losses relative to existing techniques.
Zhuoyue Huang, Freddie Bickford Smith, Tom Rainforth
Batch active learning (BAL) is a crucial technique for reducing labeling costs and improving data efficiency in training large-scale deep learning models. Traditional BAL methods often rely on metrics like Mahalanobis Distance to balance uncertainty and diversity when selecting data for annotation. However, these methods predominantly focus on the distribution of unlabeled data and fail to leverage feedback from labeled data or the model's performance. To address these limitations, we introduce TrustSet, a novel approach that selects the most informative data from the labeled dataset, ensuring a balanced class distribution to mitigate the long-tail problem. Unlike CoreSet, which focuses on maintaining the overall data distribution, TrustSet optimizes the model's performance by pruning redundant data and using label information to refine the selection process. To extend the benefits of TrustSet to the unlabeled pool, we propose a reinforcement learning (RL)-based sampling policy that approximates the selection of high-quality TrustSet candidates from the unlabeled data. Combining TrustSet and RL, we introduce the Batch Reinforcement Active Learning with TrustSet (BRAL-T) framework. BRAL-T achieves state-of-the-art results across 10 image classification benchmarks and 2 active fine-tuning tasks, demonstrating its effectiveness and efficiency in various domains.
Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.
Pouya Ahadi, Blair Winograd, Camille Zaug +3
School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company