How Many Labels Are Enough? ALDA: Active Learning Deployment Advisor for Medical Image Classification
Authors: Julia Machnio, Mads Nielsen, Mostafa Mehdipour Ghazi
Organizations: Pioneer Centre for AI, University of Copenhagen, Copenhagen, Denmark
Abstract
Active learning (AL) promises to reduce the cost of medical imaging projects by lowering the number of clinical labels required. However, practical deployment requires committing to a sampling strategy before the full annotation budget is spent, and choosing the wrong strategy can increase rather than decrease costs. We propose Active-Learning Deployment Advisor (ALDA), a deployment-oriented framework for AL method selection under clinical performance constraints. Given a short pilot phase, ALDA fits a parametric learning-curve model to each candidate strategy, estimates whether that strategy is expected to reach a required clinical performance target, and predicts the number of expert annotations needed to do so. In addition to absolute annotation cost, ALDA introduces a deployment window that quantifies the sensitivity of this cost estimate to uncertainty in the clinical threshold. The final recommendation follows a risk-aware rule: among strategies with near-optimal predicted cost, ALDA prefers the strategy with the narrowest deployment window, the most robust to threshold revisions. Experiments on four medical imaging classification domains show that ALDA predicts the deployment-optimal method from a pilot of 15-30% of the intended budget and reduces annotation costs by up to 82% compared with a poor strategy choice. Rather than introducing a new sampling heuristic, ALDA provides a practical decision layer that answers a deployment-critical question: how many labels are enough?
In practical settings, medical image segmentation models are often developed with limited annotated data rather than fully labeled datasets. Training frequently begins in ultra-low labeled regimes where only a small number of volumes are annotated. In such scenarios, practitioners must simultaneously decide which cases to annotate and how to best use the remaining unlabeled data. Although active learning (AL) and semi-supervised learning (SSL) both target annotation scarcity, they are typically designed and optimized independently, resulting in objective mismatch and unstable training during early-stage "cold start" conditions. We propose RegAL, a unified active semi-supervised framework governed by a shared topology-aware Pareto optimization that couples sample acquisition with unlabeled data utilization. RegAL evaluates images along three complementary axes, voxel-wise uncertainty, feature diversity, and a novel topological consistency metric, to select anatomically informative edge cases for annotation. On the other hand, the same criteria are used to identify geometrically stable atlas candidates for diffeomorphic registration-guided augmentation to train a self-supervised Mean Teacher segmentation network. Across BraTS 2021, dHCP, and ProstateX, RegAL remains stable with few labeled volumes and consistently outperforms state-of-the-art AL, SSL, and active semi-supervised baselines across Dice and boundary-distance (ASD, HD95) metrics under extreme annotation scarcity.
We present FunnelAL, a retrieve-then-rank active learning system for single-class discovery, which adapts the multi-stage funnel architecture of industrial recommender systems to data annotation. Large-scale supervised learning faces two challenges: efficiently finding relevant samples in a massive corpus, and distinguishing true positives from visually confusable negatives when embeddings do not cleanly separate classes. Conventional active learning offers a principled framework for reducing annotation cost, yet it treats sample selection as a single-stage process that addresses neither challenge efficiently. FunnelAL decomposes the problem into cascaded stages. Starting from a single positive and negative example, the system iterates through: (1) embedding-based retrieval scoring that narrows the corpus to a manageable candidate set; (2) a precision-triggered ranking stage that exploits a learned ranker (RankNet) while batch precision remains high, then automatically blends in committee-based exploration (QBC) once returns diminish; and (3) feedback from the annotator's labels that refines both stages in subsequent iterations. We evaluate on three diverse image classification benchmarks. With a perfect annotator, FunnelAL attains the best final F1 on all three benchmarks, the best annotation efficiency (first in AULC), and the fewest annotation rounds. The most recent single-class discovery methods (GAL, PF-MA) at best match its final quality, and only at consistently higher labeling cost. Under annotator labeling errors at realistic rates, FunnelAL remains first or statistically tied for first while classical uncertainty-based methods degrade two to three times faster. Our work provides a concrete bridge between multi-stage recommender systems and active learning.
Active learning aims to reduce annotation cost by selectively querying informative samples for supervision under a limited labeling budget. In this work, we investigate how vision-language models (VLMs) can be leveraged to further reduce the reliance on costly human annotation within the active learning paradigm. To this end, we find that the reliability of VLMs varies significantly with label granularity in fine-grained recognition tasks: they perform poorly on fine-grained labels but can provide accurate coarse-grained labels. Leveraging this property, we propose an active learning framework that combines fine-grained human annotations with coarse-grained VLM-generated weak labels through instance-wise label assignment. We further model the systematic noise in VLM-generated labels using a small set of trusted full labels. Experiments on CUB200 and FGVC-Aircraft show that the proposed framework consistently outperforms existing active learning methods under the same annotation budget.