Anomaly-ranked confirmation slows new-class learning
Organizations: School of Physics and Astronomy, University of Minnesota, Minneapolis, MN 55455, USA
Abstract
Anomaly detectors rank unfamiliar examples for expert confirmation, and the confirmed examples teach a classifier to recognise a new class. Across five benchmarks from physics, medicine and computer vision, the members flagged by the detector are those the classifier learns first. In three systems, confirming the highest-ranked members of a new class gives lower recognition than confirming the same number of random members. Handing subsequent selection to the promoted classifier recovers part of this loss in two systems. A paired intervention further shows that replacing nearby known classes with distant ones sharply reduces the confirmations needed for recognition. These results distinguish finding a new class from selecting examples that teach its extent: within the tested classes, concentrating confirmations at the top of the anomaly ranking makes learning less efficient.
Figures & tables
| Comparison | Result |
|---|---|
| Interpolated , cheaper : dearer | |
| Median , near far | |
| Replicate cheaper : dearer counts | |
| First measured crossing, cheaper : dearer | |
| Median first crossing, near far | |
| Two frequent far classes excluded |
| System | Random | Detector | Least | Alerts | Top | Margin | |
|---|---|---|---|---|---|---|---|
| Gravity Spy (8 classes) | 5 | 0.42 | 0.30 | 0.48 | 0.42 | 0.35 | 0.42 |
| 10 | 0.72 | 0.43 | 0.66 | 0.61 | 0.42 | 0.53 | |
| 20 | 0.89 | 0.67 | 0.79 | 0.82 | 0.73 | 0.83 | |
| CIFAR-100 (13 classes) | 5 | 0.75 | 0.68 | 0.69 | 0.74 | 0.67 | 0.75 |
| 10 | 0.81 | 0.68 | 0.75 | 0.77 | 0.68 | 0.80 | |
| 20 | 0.82 | 0.72 | 0.77 | 0.80 | 0.70 | 0.83 |
| System | Detector | Alerts | Hand-off | |
|---|---|---|---|---|
| Gravity Spy | 5 | 0.23 (3.8) | 0.31 (3.1) | 0.23 (3.8) |
| (8 classes) | 10 | 0.42 (7.8) | 0.42 (6.2) | 0.49 (7.0) |
| 20 | 0.57 (15.6) | 0.66 (12.9) | 0.68 (14.4) | |
| 40 | 0.74 (29.1) | 0.79 (25.2) | 0.88 (31.3) | |
| CIFAR-100 | 5 | 0.54 (3.2) | 0.54 (1.7) | 0.54 (3.2) |
| (13 classes) | 10 | 0.67 (5.9) | 0.67 (3.4) | 0.72 (6.5) |
Active Learning
Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.
See all 330 papers