Feature Weighting Improves Pool-Based Sequential Active Learning for Regression
Authors: Dongrui Wu
Organizations: Ministry of Education Key Laboratory of Image Processing and Intelligent Control, and Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, 430074 China.
Pool-based sequential active learning for regression (ALR) optimally selects a small number of samples sequentially from a large pool of unlabeled samples to label, so that a more accurate regression model can be constructed under a given labeling budget. Representativeness and diversity, which involve computing the distances among different samples, are important considerations in ALR. However, previous ALR approaches do not incorporate the importance of different features in inter-sample distance computation, resulting in inaccurate distances and hence sub-optimal sample selection. This paper proposes four feature weighted single-task ALR approaches and three feature weighted multi-task ALR approaches, where the ridge regression coefficients trained from a small amount of previously labeled samples are used to weight the corresponding features in inter-sample distance computation. Extensive experiments showed that this intuitive and easy-to-implement enhancement almost always improves the performance of five existing ALR approaches, in both single-task and multi-task regression problems. The feature weighting strategy may also be easily extended to stream-based ALR, and classification algorithms.
Figures & tables
# of
# of
# of
# of
# of
Dataset
Source
samples
raw
numerical
categorical
total
features
features
features
features
Yacht 1
UCI
308
6
6
0
6
autoMPG 2
UCI
392
7
6
1
9
NO2 3
StatLib
500
7
7
0
7
PM10 3
StatLib
500
7
7
0
7
TABLE I: Summary of the 11 single-task regression datasets.
Fig. 1: Performance of the 11 single-task ALR algorithms on the 11 datasets, averaged over 100 runs. (a) Yacht; (b) autoMPG; (c) NO2; (d) PM110; (e) Housing; (f) CPS; (g) EE-Cooling; (h) Concrete; (i) Airfoil; (j) Wine-white; and, (k) BikeSharing.
Dataset
RMSE
CC
FW
FW
FW
FW
FW
FW
FW
FW
BL
EMCM
QBC
RD
-RD
GSx
-GSx
iGS
-iGS
GALR
-GALR
BL
EMCM
QBC
RD
-RD
GSx
-GSx
iGS
-iGS
GALR
-GALR
Yacht
1.00
1.14
1.12
0.97
0.88
1.22
0.97
0.95
0.91
1.17
0.99
1.00
1.02
1.02
1.10
1.07
1.04
1.05
1.06
1.07
1.05
1.05
autoMPG
1.00
0.87
0.84
0.79
0.79
0.73
0.72
0.76
0.75
0.75
0.73
1.00
1.01
1.02
1.03
1.02
1.05
1.05
1.04
1.04
1.05
1.05
NO2
1.00
0.97
0.95
1.00
0.98
0.95
0.93
0.93
0.93
0.94
0.93
1.00
1.00
1.02
1.00
1.01
1.01
1.00
0.99
0.98
1.00
1.01
PM10
1.00
0.93
0.92
0.99
0.97
0.96
0.89
0.89
0.89
0.92
0.89
1.00
1.22
1.26
0.69
0.78
1.34
1.40
1.32
1.27
1.49
1.43
TABLE II: Normalized AUCs of RMSEs and CCs of different algorithms on the 11 datasets.
Approach
RMSE
CC
BL
EMCM
QBC
RD
FW-RD
GSx
FW-GSx
iGS
FW-iGS
GALR
BL
EMCM
QBC
RD
FW-RD
GSx
FW-GSx
iGS
FW-iGS
GALR
EMCM
0.04
0.01
QBC
0.03
0.41
0.02
0.12
RD
0.28
1.00
0.97
0.37
1.00
0.97
FW-RD
0.00
0.21
0.12
0.03
0.17
0.41
0.58
0.58
GSx
0.02
0.17
0.32
0.46
0.70
0.02
0.02
0.05
0.46
0.97
TABLE III: p -values of Wilcoxon paired signed-rank test on the AUCs of RMSEs and CCs across the 11 datasets. p -values smaller than 0.05 are in bold.
Fig. 2: Computational cost of the 11 algorithms on BikeSharing, for M∈[20,60] , averaged over 100 runs. (a) d=39 , while N increases from 100 to 15,000; and, (b) N=10,000 , while d increases from 5 to 39.
Fig. 3: True and estimated feature weights on Housing. Results for first four of the 13 features are shown.
Fig. 4: Performance of the 11 single-task ALR algorithms, averaged over 100 runs. (a) Yacht; and, (b) autoMPG. Two redundant features were added to each dataset.
RMSE
CC
Algorithm
Yacht
AutoMPG
Yacht
AutoMPG
Before
After
Before
After
Before
After
Before
After
BL
1.00
1.18
1.00
0.95
1.00
0.98
1.00
1.00
EMCM
1.14
1.19
0.87
0.85
1.02
1.02
1.01
1.01
QBC
1.12
1.18
0.84
0.81
1.02
1.01
1.02
1.02
RD
0.97
1.00
0.79
0.79
1.10
1.11
1.03
1.02
TABLE IV: Normalized AUCs of RMSE and CC before and after adding correlated/redundant features.
Fig. 5: Performance of the 11 single-task ALR algorithms, averaged over 100 runs. (a) Yacht; and, (b) autoMPG. Two irrelevant features were added to each dataset.
RMSE
CC
Algorithm
Yacht
AutoMPG
Yacht
AutoMPG
Before
After
Before
After
Before
After
Before
After
BL
1.00
1.12
1.00
1.01
1.00
0.93
1.00
0.98
EMCM
1.14
1.12
0.87
0.90
1.02
0.96
1.01
1.01
QBC
1.12
1.11
0.84
0.86
1.02
0.96
1.02
1.01
RD
0.97
1.02
0.79
0.84
1.10
1.05
1.03
1.00
TABLE V: Normalized AUCs of RMSE and CC before and after adding irrelevant features.
Fig. 6: Effect of different number of initially labeled samples, nmin (averaged over 100 runs). (a) Yacht, where d=6 ; and, (b) autoMPG, where d=9 .
Fig. 7: Performance of different initialization approaches on the EE-Cooling dataset, averaged over 100 runs. iGS (iRDM) and FW-iGS (iRDM) used iRDM [ 32 ] to select the first 5 samples to label. iGS (IRD) and FW-iGS (IRD) used IRD [ 33 ] to select the first 5 samples to label.
Fig. 8: Performance of the nine single-task ALR algorithms, averaged over 200 runs. Linear ridge regression was used to estimate the feature weights in FW-RD , FW-GSx , FW-iGS and FW-GALR , and a nonlinear regression tree was trained for new sample prediction. (a) Yacht; (b) autoMPG; and, (c) NO2.
Fig. 9: Performance of the nine single-task ALR algorithms, averaged over 200 runs. A nonlinear regression tree was used in both feature weight estimation in FW-RD , FW-GSx , FW-iGS and FW-GALR , and for new sample prediction. (a) Yacht; (b) autoMPG; and, (c) NO2.
Fig. 10: Performance of the seven multi-task ALR algorithms on the VAM dataset, averaged over 100 runs.
Fig. 11: Performance of the seven multi-task ALR algorithms on the EE dataset, averaged over 100 runs.
Active learning for regression reduces labeling costs by selecting the most informative samples. Improved Greedy Sampling is a prominent method that balances feature-space diversity and output-space uncertainty using a static, multiplicative rule. We propose Weighted improved Greedy Sampling (WiGS), which replaces this framework with a dynamic, additive criterion. We formulate weight selection as a reinforcement learning problem, enabling an agent to adapt the exploration-investigation balance throughout learning. Experiments on 18 benchmark datasets and a synthetic environment show WiGS outperforms iGS and other baseline methods in both accuracy and labeling efficiency, particularly in domains with irregular data density where the baseline's multiplicative rule ignores high-error samples in dense regions.
Simon D. Nguyen, Troy Russo, Kentaro Hoffman +1
Department of Statistics, University of Washington · Department of Sociology, University of Washington
With rapid advancement over the last few years, many different methods are now widely used for classification. However, training these models requires substantial labeled data. Active Learning is a potential solution to this problem. Pool-based active learning minimizes costs by querying only the most informative samples from an unlabeled dataset. Diversity-based approaches, on the other hand, attempt to select a representative subset of the data. There are many different objectives for determining the selection process, including exact K-center, exact K-median, and Greedy K-center. In this paper, we will focus on evaluating the performance of Greedy K-center across a variety of metric spaces: the raw feature space, a Linear Discriminant Analysis (LDA) space, and a model-derived probability space (with and without entropy-based weighting). Using Random Forest classifiers as a baseline evaluator, our empirical results on synthetic and real-world datasets demonstrate that mapping unlabeled instances into a predictive probability space and weighting the result by entropy often dominates the other options for active learning selection with Greedy K-center.
The effectiveness of active learning hinges on the choice of the acquisition criterion by which a learning algorithm selects potentially informative data points whose label is subsequently queried. This paper proposes a novel gradient-based acquisition criterion, derived from a generalization bound introduced by Luo et al. (2022). This criterion can be applied in lieu of uncertainty measures in uncertainty sampling, or incorporated into diversity-based methods that consider the spread of sampled points in addition to the uncertainty of their labels. We provide a theoretical justification of the proposed acquisition criterion, and demonstrate its effectiveness in an empirical evaluation.
Mohamadsadegh Khosravani, Sandra Zilles
Department of Computer Science University of Regina · University of Regina