The increasing size and heterogeneity of exoplanet catalogs have made systematic habitability assessment challenging, particularly given the extreme scarcity of potentially habitable planets and the evolving nature of their labels. In this study, we explore the use of pool-based active learning to improve the efficiency of habitability classification under realistic observational constraints. We construct a unified dataset from the Habitable World Catalog and the NASA Exoplanet Archive and formulate habitability assessment as a binary classification problem. A supervised baseline based on gradient-boosted decision trees is established and optimized for recall in order to prioritize the identification of rare potentially habitable planets. This model is then embedded within an active learning framework, where uncertainty-based margin sampling is compared against random querying across multiple runs and labeling budgets. We find that active learning substantially reduces the number of labeled instances required to approach supervised performance, demonstrating clear gains in label efficiency. To connect these results to a practical astronomical use case, we aggregate predictions from independently trained active-learning models into an ensemble and use the resulting mean probabilities and uncertainties to rank planets originally labeled as non-habitable. This procedure identifies a single robust candidate for further study, illustrating how active learning can support conservative, uncertainty-aware prioritization of follow-up targets rather than speculative reclassification. Our results indicate that active learning provides a principled framework for guiding habitability studies in data regimes characterized by label imbalance, incomplete information, and limited observational resources.
Figures & tables
Figure 1: Lower-triangular Pearson correlation matrix of the final feature set used in this study. Each cell shows the Pearson correlation coefficient between a pair of planetary or stellar properties, with the color scale ranging from −1 (strong negative correlation) to +1 (strong positive correlation). The diagonal elements represent self-correlations. To improve readability, only non-zero correlations are annotated.
Figure 2: UpSet plot showing missing-value patterns across the final feature set. The horizontal bars on the left indicate the total number of missing values for each feature. Each column in the central matrix represents a specific combination of features that are simultaneously missing, marked by filled black circles connected by vertical lines. The vertical bars above the matrix show the number of data instances exhibiting each missingness pattern (intersection size). For example, the tallest bar on the far left corresponds to instances where only orbital eccentricity is missing, while all other features are present. In contrast, columns with multiple connected black circles indicate instances where several stellar and planetary properties are missing together.
Figure 3: Class-conditional distributions of selected planetary and stellar parameters in the final dataset. Kernel density estimates are shown for planet radius, orbital eccentricity, and ESI; and in log scale for planet mass, incident stellar flux, and equilibrium temperature. The figure illustrates the substantial overlap between the two classes across most individual parameters, as well as systematic shifts in location and spread for several features, motivating the need for multivariate classification.
Figure 4: Diagnostics of orbital eccentricity imputation. Panel (a) compares observed and imputed eccentricity distributions; panel (b) shows residuals as a function of true eccentricity; and panel (c) shows the mean residual in bins of true eccentricity, highlighting a systematic bias toward under estimating at higher eccentricities.
Figure 5: Radar plot comparing supervised classifiers across recall, precision, F1-score, balanced accuracy, and AUROC. XGBoost exhibits the strongest overall performance and is selected as the baseline model.
Metric
Value
Recall
0.929
Precision
0.929
F1-score
0.929
Balanced accuracy
0.964
AUROC
0.999
Table 1: Performance of the final XGBoost classifier evaluated on the held-out test set. Metrics are defined in Section II.4 .
Figure 6: Distribution of prediction variability under input perturbations, quantified as the standard deviation of model-predicted habitability probabilities across perturbed realizations.
Figure 7: Global feature importance based on mean absolute SHAP values for the XGBoost classifier. Features are ordered by decreasing contribution to the model output, highlighting the dominant role of thermal and habitability-related parameters.
Figure 8: Permutation feature importance measured as the decrease in recall when individual features are randomly shuffled. Larger values indicate greater importance for identifying potentially habitable planets.
Figure 9: Active learning performance as a function of the number of labeled instances. Mean learning curves over 20 independent runs, evaluated every five queried instances. Shaded regions indicate ±1 standard deviation across runs, illustrating inter-run variability. The horizontal dashed lines indicate the performance of the XGBoost classifier trained on the full labeled training pool. The vertical dashed lines mark the number of labeled instances at which the margin-based strategy approaches this performance level for each metric.
Planetary and orbital parameters
Parameter
Candidate
Median
IQR
Orbital period (days)
636.13
36.77
92.30
Semi-major axis (AU)
1.334
0.165
0.290
Orbital eccentricity
0.160
0.083
0.141
Planet radius ( R⊕ )
1.81
1.78
0.90
Planet mass ( M⊕ )
3.93
4.06
3.48
Table 2: Comparison between the recommended planet ( τ Ceti f) and the distribution of cataloged potentially habitable planets. For each parameter, the value of the recommended candidate is shown alongside the median and interquartile range (IQR) of the potentially habitable sample.
Active Learning (AL) is commonly used in applications where labeling data is expensive or time-consuming. In practice, however, labels are often noisy due to varying labeler expertise and annotation uncertainty, especially for complex or ambiguous samples. Learning from such imperfectly labeled data can degrade classifier performance. We propose an AL framework that explicitly accounts for label noise by optimally assigning labelers and selecting samples to minimize labeling error. Our approach, called OLAS (Optimal Labeler Assignment and Sampling), uses a noise model that depends on both labeler accuracy and model uncertainty to guide these decisions. We develop two tractable optimization formulations: one for assigning samples to labelers to minimize worst-case noise, and another for selecting samples while controlling overall label noise. Theoretical results provide closed-form solutions under mild conditions. Empirical evaluations on benchmark datasets and a real-world warranty claim classification problem show that OLAS achieves the highest or near-highest classification accuracy among existing AL strategies across most settings, using only a single label per sample.
Pouya Ahadi, Blair Winograd, Camille Zaug +3
School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA · Ford Motor Company
Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus. Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class. Random sampling cannot supply these at reasonable cost, so they must be selected in some other way, such as keyword search, and any such selection biases the initial set towards part of the target class. We evaluate whether active learning can correct this bias. On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1. On AG News, almost every strategy recovers the omitted subgroup. On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1. Aggregate metrics conceal these differences.
Active learning is now standard practice in labeling ecological data, enabling ecologists to quickly process large volumes of field data to understand and monitor natural environments. Current practices evaluate active learning inductively, estimating predictive performance on a held-out test set. We argue that this evaluation is misaligned with most ecological tasks, where the goal is to transductively label an entire pool of data as efficiently as possible. We demonstrate that ignoring the human-in-the-loop underestimates the importance of continuing to label, particularly for classes in the long tail which may be of disproportionate ecological importance (rare species, uncommon behaviors, etc.). Our analysis shows that, for this long tail, the transductive objective shifts importance from prediction to discovery: the true challenge becomes finding "needles in the haystack," examples of rare classes that are embedded within dense regions of abundant classes in the latent geometry, which we quantify with a novel metric of sampling difficulty. Finally, to translate these insights to practical ecological workflows, we propose a conservative hybrid stopping criterion inspired by ecological rarefaction curves, and show that combining predictive performance with discovery criteria reduces premature stopping on long-tailed pools, improving rare-class recovery when discovery, not classification, is the limiting factor.
Rupa Kurinchi-Vendhan, Sara Beery
Massachusetts Institute of Technology, Cambridge MA 02139, USA