Active Learning under Simulated Class Imbalance and Label Shift in Binary Semantic Segmentation

The aim of Active Learning is to select the most informative samples from an unlabelled set of data. This is useful in cases where the amount of data is large and labelling is expensive, such as in machine vision or medical imaging. Two particularities of machine vision are first, that most of the images produced are free of defects, and second, that the amount of images produced is so big that we cannot store all acquired images. This results, on the one hand, in a strong class imbalance in defect distribution and, on the other hand, in a potential label shift caused by limited storage. To understand how these two forms of imbalance affect active learning algorithms, we propose a simulation study based on two open-source datasets. Our contribution is not a new acquisition strategy but a controlled and reproducible protocol that synthetically constructs datasets in which the levels of class imbalance and label shift are fixed and vary independently. Four standard active learning selection strategies are compared: random sampling, entropy-based selection, core-set selection and an imbalance aware strategy called galaxy sampling. We empirically find that active learning, and in particular the entropy-based and core-set selections, remains competitive with and typically superior to random sampling even for highly imbalanced datasets, and we quantify the loss of efficiency that occurs under a strong label shift.