Positive-First Most Ambiguous: A Simple Active Learning Criterion for Interactive Retrieval of Rare Categories

Kawtar Zaher; Olivier Buisson; Alexis Joly

cs.CVMar 25, 2026

Positive-First Most Ambiguous: A Simple Active Learning Criterion for Interactive Retrieval of Rare Categories

Authors: Kawtar Zaher, Olivier Buisson, Alexis Joly

Organizations: INRIA, LIRMM, Université de Montpellier, France · Institut National de l’Audiovisuel, France

Abstract

Real-world fine-grained visual retrieval often requires discovering a rare concept from large unlabeled collections with minimal supervision. This is especially critical in biodiversity monitoring, ecological studies, and long-tailed visual domains, where the target may represent only a tiny fraction of the data, creating highly imbalanced binary problems. Interactive retrieval with relevance feedback offers a practical solution: starting from a small query, the system selects candidates for binary user annotation and iteratively refines a lightweight classifier. While Active Learning (AL) is commonly used to guide selection, conventional AL assumes symmetric class priors and large annotation budgets, limiting effectiveness in imbalanced, low-budget, low-latency settings. We introduce Positive-First Most Ambiguous (PF-MA), a simple yet effective AL criterion that explicitly addresses the class imbalance asymmetry: it prioritizes near-boundary samples while favoring likely positives, enabling rapid discovery of subtle visual categories while maintaining informativeness. Unlike standard methods that oversample negatives, PF-MA consistently returns small batches with a high proportion of relevant samples, improving early retrieval and user satisfaction. To capture retrieval diversity, we also propose a class coverage metric that measures how well selected positives span the visual variability of the target class. Experiments on long-tailed datasets, including fine-grained botanical data, demonstrate that PF-MA consistently outperforms strong baselines in both coverage and classifier performance, across varying class sizes and descriptors. Our results highlight that aligning AL with the asymmetric and user-centric objectives of interactive fine-grained retrieval enables simple yet powerful solutions for retrieving rare and visually subtle categories in realistic human-in-the-loop settings.

Figures & tables

Figure 1: Comparison of selected samples. MA (left): near-boundary negatives are oversampled. MP (middle): only positives far from the boundary are selected. PF-MA (right): balance between relevant positives and negatives around the boundary.

Figure 2: Interactive retrieval process.

dataset	min	max	mean	median
Cifar100-LT	$0.079\%$	$3.97\%$	$1\%$	$0.56\%$
ImageNet-LT	$0.004\%$	$1.1\%$	$0.1\%$	$0.06\%$
PlantNet300K	$0.001\%$	$2.96\%$	$0.09\%$	$0.01\%$

Table 1: Statistics of class frequencies per dataset.

metric	Rand	DAL	CoreSet	MA
$cov_{25}$	0.111	0.096	0.087	0.899
$f1_{25}$	0.381	0.321	0.164	0.857
$pos_{25}$	0.036	0.032	0.035	0.61

Table 2: Performance comparison of representativeness-based AL methods vs. uncertainty-based selection on CIFAR100-LT at iteration

25

method	Cifar100-LT		ImageNet-LT		PlantNet300K
method	CLIP	DINOv2	CLIP	DINOv2	CLIP	DINOv2
MA	0.89	0.899	0.852	0.804	0.596	0.678
MP	0.875	0.877	0.821	0.776	0.565	0.628
ALAMP	0.811	0.892	0.744	0.771	0.406	0.605
MA-S	0.88	0.895	0.844	0.802	0.583	0.671
MA-D	0.903	0.84	0.842	0.751	0.595	0.679

Table 3: Class coverage scores

cov_{25}

at iteration

25

for different AL methods, datasets and descriptors.

Figure 3: Retrieval results for a bird species at iteration $5$ ( $50$ total annotations). The initial query image is in blue, the positive images in green and the negative ones in red. Top: MA . Middle: MP . Bottom: PF-MA

method	$K$
method	16	32	64
MA	0.848	0.804	0.738
MP	0.776	0.776	0.764
ALAMP	0.808	0.771	0.712
MA-S	0.848	0.802	0.731
MA-D	0.833	0.751	0.648

Table 4: Class coverage scores

cov_{25}

at iteration

25

for different

K

values. Second best is underlined.

Figure 4: Class coverage results per iteration.

Figure 5: Class coverage results at iteration $15$ per range of class size.

Panel of Figure 6 — Figure 6: Distribution of positive ratio among selected samples for different selection strategies.

metric	Rand	DAL	CoreSet	MA
$cov_{5}$	0.05	0.039	0.039	0.504
$f1_{5}$	0.182	0.102	0.07	0.828
$pos_{5}$	0.022	0.019	0.02	0.3
$cov_{15}$	0.081	0.066	0.058	0.839
$f1_{15}$	0.294	0.215	0.099	0.857
$pos_{15}$	0.029	0.024	0.026	0.537

Table 5: Performance comparison of representativeness-based AL methods vs. uncertainty-based selection on CIFAR100-LT at iterations

5

15

and

25

metric	method	$K$
metric	method	16	32	64
$cov_{5}$	MA	0.55	0.423	0.327
	MP	0.413	0.396	0.399
	ALAMP	0.49	0.387	0.315
	MA-S	0.51	0.377	0.28
	MA-D	0.513	0.355	0.246

Table 6: Class coverage scores at iterations

5

15

and

25

for different

K

values. Second best is underlined.

metric	method	Cifar100-LT		ImageNet-LT		PlantNet300K
metric	method	CLIP	DINOv2	CLIP	DINOv2	CLIP	DINOv2
$cov_{5}$	MA	0.402	0.502	0.351	0.423	0.203	0.282
	MP	0.41	0.454	0.368	0.396	0.212	0.29
	ALAMP	0.349	0.471	0.289	0.387	0.148	0.235
	MA-S	0.363	0.442	0.321	0.377	0.188	0.257
	MA-D	0.388	0.407	0.327	0.355	0.193	0.284

Table 7: Class coverage scores at iterations

5

15

and

25

for different AL methods, datasets and descriptors. Second best is underlined.

metric	method	Cifar100-LT		ImageNet-LT		PlantNet300K
metric	method	CLIP	DINOv2	CLIP	DINOv2	CLIP	DINOv2
$f1_{5}$	MA	0.604	0.83	0.504	0.675	0.191	0.413
	MP	0.258	0.597	0.187	0.573	0.077	0.221
	ALAMP	0.56	0.806	0.45	0.646	0.147	0.373
	MA-S	0.57	0.816	0.476	0.667	0.176	0.398
	MA-D	0.517	0.803	0.437	0.661	0.159	0.398

Table 8: F1 scores at iterations

5

15

and

25

for different AL methods, datasets and descriptors. Top-3 results are underlined. A star (*) is used only when our method achieves the top-1 result.

Panel of Figure 7 — Figure 7: Class coverage results per iteration.

Panel of Figure 8 — Figure 8: Class coverage results at iteration $15$ per range of class size.

Panel of Figure 9 — Figure 9: F1 score results per iteration.

Panel of Figure 10 — Figure 10: F1 score results at iteration $15$ per range of class size.

Active Learning

Active learning selects the examples, labels, features, or experiments most useful for improving a model under limited budgets.

See all 288 papers

Jul 28, 2026

FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

We present FunnelAL, a retrieve-then-rank active learning system for single-class discovery, which adapts the multi-stage funnel architecture of industrial recommender systems to data annotation. Large-scale supervised learning faces two challenges: efficiently finding relevant samples in a massive corpus, and distinguishing true positives from visually confusable negatives when embeddings do not cleanly separate classes. Conventional active learning offers a principled framework for reducing annotation cost, yet it treats sample selection as a single-stage process that addresses neither challenge efficiently. FunnelAL decomposes the problem into cascaded stages. Starting from a single positive and negative example, the system iterates through: (1) embedding-based retrieval scoring that narrows the corpus to a manageable candidate set; (2) a precision-triggered ranking stage that exploits a learned ranker (RankNet) while batch precision remains high, then automatically blends in committee-based exploration (QBC) once returns diminish; and (3) feedback from the annotator's labels that refines both stages in subsequent iterations. We evaluate on three diverse image classification benchmarks. With a perfect annotator, FunnelAL attains the best final F1 on all three benchmarks, the best annotation efficiency (first in AULC), and the fewest annotation rounds. The most recent single-class discovery methods (GAL, PF-MA) at best match its final quality, and only at consistently higher labeling cost. Under annotator labeling errors at realistic rates, FunnelAL remains first or statistically tied for first while classical uncertainty-based methods degrade two to three times faster. Our work provides a concrete bridge between multi-stage recommender systems and active learning.

Reihaneh Rostami, Brian Goodwin

RAIC Labs · RAIC Labs USA

Active Learning Classification

Ask PDF

Apr 1, 2026

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

Building on existing approaches, we revisit Human-in-the-Loop Object Retrieval, a task that consists of iteratively retrieving images containing objects of a class-of-interest, specified by a user-provided query. Starting from a large unlabeled image collection, the aim is to rapidly identify diverse instances of an object category relying solely on the initial query and the user's Relevance Feedback, with no prior labels. The retrieval process is formulated as a binary classification task, where the system continuously learns to distinguish between relevant and non-relevant images to the query, through iterative user interaction. This interaction is guided by an Active Learning loop: at each iteration, the system selects informative samples for user annotation, thereby refining the retrieval performance. This task is particularly challenging in multi-object datasets, where the object of interest may occupy only a small region of the image within a complex, cluttered scene. Unlike object-centered settings where global descriptors often suffice, multi-object images require more adapted, localized descriptors. In this work, we formulate and revisit the Human-in-the-Loop Object Retrieval task by leveraging pre-trained ViT representations, and addressing key design questions, including which object instances to consider in an image, what form the annotations should take, how Active Selection should be applied, and which representation strategies best capture the object's features. We compare several representation strategies across multi-object datasets highlighting trade-offs between capturing the global context and focusing on fine-grained local object details. Our results offer practical insights for the design of effective interactive retrieval pipelines based on Active Learning for object class retrieval.

Kawtar Zaher, Olivier Buisson, Alexis Joly

Institut National de l’Audiovisuel INA, Paris, France · Institut National de Recherche et Informatique et Robotique INRIA, Montpellier, France

Human-in-the-Loop AI Human-in-the-Loop Annotation Active Learning

Ask PDF

Oct 10, 2026

Active Learning for Finding a Needle in a Haystack

Training a classifier to find a rare class of documents, such as hate speech directed at a particular religious group, requires labelled examples that are expensive to obtain when the class makes up only a small fraction of a corpus. Active learning reduces this cost, but it must start from an initial labelled set that contains examples of the target class. Random sampling cannot supply these at reasonable cost, so they must be selected in some other way, such as keyword search, and any such selection biases the initial set towards part of the target class. We evaluate whether active learning can correct this bias. On AG News and Measuring Hate Speech, at a prevalence of around 1%, we compare random sampling with eight active learning strategies, starting from initial sets that cover or omit known subgroups of the target class, and measure recall on the omitted subgroups as well as F1. On AG News, almost every strategy recovers the omitted subgroup. On Measuring Hate Speech, uncertainty-based strategies recover less of the omitted subgroups than random sampling, whereas discriminative active learning recovers 55% of them without a significant loss of F1. Aggregate metrics conceal these differences.

Emily Silcock, Simon Löwe

Harvard University · Burning Glass Institute

Subgroup Robustness Active Learning Class-Imbalanced Learning

Ask PDF

Abstract

Active Learning

Explore similar work

FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

Active Learning for Finding a Needle in a Haystack