IntRec: Intent-based Retrieval with Contrastive Refinement

Pourya Shamsolmoali; Masoumeh Zareapoor; Eric Granger; Yue Lu

cs.CVFeb 19, 2026

IntRec: Intent-based Retrieval with Contrastive Refinement

Authors: Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Yue Lu

Organizations: Dept. Computer Science, University of York, York, United Kingdom · Dept. Computer Science, Shanghai Jiao Tong University, Shanghai, China · LIVIA, Dept. of Systems Engineering, École de technologie supérieure, Montreal, Canada · School of Communication and Electronic Eng., East China Normal University, Shanghai, China

Abstract

Retrieving user-specified objects from complex scenes remains a challenging task, especially when queries are ambiguous or involve multiple similar objects. Existing open-vocabulary detectors operate in a one-shot manner, lacking the ability to refine predictions based on user feedback. To address this, we propose IntRec, an interactive object retrieval framework that refines predictions based on user feedback. At its core is an Intent State (IS) that maintains dual memory sets for positive anchors (confirmed cues) and negative constraints (rejected hypotheses). A contrastive alignment function ranks candidate objects by maximizing similarity to positive cues while penalizing rejected ones, enabling fine-grained disambiguation in cluttered scenes. Our interactive framework provides substantial improvements in retrieval accuracy without additional supervision. On LVIS, IntRec achieves 35.4 AP, outperforming OVMR, CoDet, and CAKE by +2.3, +3.7, and +0.5, respectively. On the challenging LVIS-Ambiguous benchmark, it improves performance by +7.9 AP over its one-shot baseline after a single corrective feedback, with less than 30 ms of added latency per interaction.

Figures & tables

Figure 1. Overview of the proposed IntRec. It identifies a user-specified object through an interactive loop. An initial query is encoded to initialize the intent state, which containes both positive exemplars $(Z_{pos}^{(t)})$ and negative exemplars $(Z_{neg}^{(t)})$ . This state guides the contrastive scoring module, which ranks all candidate regions in the target image based on their similarity to the positive exemplars and dissimilarity to the negative ones. The feedback updates the intent state, enabling the model to accurately localize the target object $(b^{\star})$ .

Symbol	Definition
$I_{s},I_{r}$	Target image used for retrieval, and an optional user-provided reference image.
$p_{0}$	Initial multi-modal prompt, consisting of a text description $T_{0}$ and/or a reference image $I_{r}$ .
$b,b^{\star}$	Candidate bounding box and the final confirmed target object.
$R=\{r_{1},\ldots,r_{M}\}$	Set of $M$ candidate region feature vectors extracted from $I_{s}$ .
$E_{T},E_{I}$	Pre-trained text and image encoders (e.g., CLIP).
$z_{T},z_{I},z_{p}$	Embeddings of the text prompt, reference image, and fused query representation.

Table 1. Notation summary for the proposed IntRec.

Method	backbone	AP (r / c / f)
Detic ( Zhou et al., 2022 )	ResNet-50	27.4 (18.6 / - / -)
DetPro ( Du et al., 2022 )	ResNet-50	26.8 (20.3 / 26.5 / 28.9)
RegionCLIP ( Zhong et al., 2022 )	ResNet-50	28.2 (17.1 / 27.4 / 34.0)
ViLD ( Gu et al., 2022 )	ResNet-50	25.8 (16.6 / 24.6 / 30.1)
CCKT-Det ( Zhang et al., 2025 )	ResNet-50	27.1 (18.2 / - / -)
F-VLM ( Kuo et al., 2023 )	ResNet-50	24.2 (18.6 / - / -)

Table 2. Comparison results on LVIS using ResNet-50 detector backbone. AP(r) and AP are the main evaluation metric for this experiment. Rows for our models are highlighted, representing results from text, vision, and multimodal.

Method	object365			COCO
	AP	AP-50	AP(r)	AP	AP-50	AP-75
DetPro ( Du et al., 2022 )	11.7	18.2	9.8	34.9	53.8	37.4
BARON ( Wu et al., 2023 )	12.7	20.3	10.2	36.2	55.7	39.1
CoDet ( Ma et al., 2024a )	13.8	20.6	10.5	37.4	55.1	39.5
CCKT-Det ( Zhang et al., 2025 )	13.4	19.7	11.5	38.5	53.2	42.1
OVMR ( Ma et al., 2024b )	12.3	19.5	10.7	37.6	54.7	40.3

Table 3. Transfer detection result. All models are trained on the LVIS/ImageNet-21K, and evaluated on Object365 and COCO. Our model is shown at both Turn-0 and Turn-1.

Model	AP on LVIS-Ambiguous
CoDet ( Ma et al., 2024a )	13.9
OVMR ( Ma et al., 2024b )	14.5
IntRec (Turn-0)	14.8
IntRec (Turn-1)	22.7

Table 4. Performance on the LVIS-Ambiguous Benchmark. While existing state-of-the-art models perform well on LVIS overall, their performance collapses under ambiguity. In this challenging scenario, our interactive model demonstrates a substantial recovery and clear performance advantage.

Figure 2. Qualitative examples of our proposed model detecting rare categories in the LVIS validation set using textual prompt.

Figure 3. Localization comparison. While baseline models produce diffuse or imprecise heatmaps, our model generates sharp, accurate localizations that correctly ground all semantic components of the prompt. For instance, it correctly distinguishes the green apples from other fruit and localizes the zebra despite the presence of nearby giraffes.

Figure 4. Detection comparison in cluttered scenes. Blue boxes indicate detections of novel/rare categories, red boxes indicate detections of known (base) categories. While the baseline models frequently produce a redundant and overlapping detections for novel objects, our model generates accurate predictions, suppress duplicate detections in dense environments.

Figure 5. Analysis of Local vs. Global alignment on the LVIS-Ambiguous benchmark over two interactive turns. Both models start with a low AP at Turn 0 (initial prompt). However, after the first corrective feedback (Turn 1), our model increases by +7.9, outpacing the +4.3 gain of the global baseline.

Figure 6. Ablation study of our model components on the LVIS benchmark. The results show the final AP after $K=2$ interactive turns. Removing either the intent state (IS) memory or the negative feedback mechanism causes a dramatic drop in performance, demonstrating that both are essential for effective ambiguity resolution. Our full model significantly outperforms all degraded variants.

Figure 7. Visualization of the contrastive refinement process. This shows the normalized scores of 100 candidate regions (y-axis) over 5 interactive steps (x-axis) for an ambiguous query with multiple distractors. (Left) Without our contrastive mechanism $(\lambda=0)$ , the model is confused; scores for many distractor regions stay high throughout the interaction. (Right) Our full model uses negative feedback to suppress the scores of irrelevant regions (light yellow) while specifying the score of the true target (dark blue).

Table 5. Hyperparameter sensitivity on LVIS/ResNet-50.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 89 papers

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers
Apr 1, 2026Kawtar Zaher, Olivier Buisson, Alexis JolyHuman-in-the-Loop AI Human-in-the-Loop Annotation
Institut National de l’Audiovisuel INA, Paris, France · Institut National de Recherche et Informatique et Robotique INRIA, Montpellier, France
AiSearch: Interactive Multi-Modal Search with VLMs
Oct 1, 2026Ali Koksal, Mei Chee Leong, Vicky Sintunata +2Multimodal IR Vision-Language Models
Institute of Advanced Intelligence and Computing (IAIC), Agency for Science, Technology and Research (A*STAR), Singapore
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
May 17, 2026Yuting Yang, Haichao Jiang, Tianming Liang +2Image Segmentation Video Object Segmentation
School of Computer Science and Engineering, Sun Yat-sen University, China · Guangdong Province Key Laboratory of Information Security Technology, China · Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China

Apr 1, 2026J/K move · Enter open · S save

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

Kawtar Zaher, Olivier Buisson, Alexis Joly

Institut National de l’Audiovisuel INA, Paris, France · Institut National de Recherche et Informatique et Robotique INRIA, Montpellier, France

Human-in-the-Loop AI Human-in-the-Loop Annotation Active Learning

Open paper AskPDF arXiv

Building on existing approaches, we revisit Human-in-the-Loop Object Retrieval, a task that consists of iteratively retrieving images containing objects of a class-of-interest, specified by a user-provided query. Starting from a large unlabeled image collection, the aim is to rapidly identify diverse instances of an object category relying solely on the initial query and the user's Relevance Feedback, with no prior labels. The retrieval process is formulated as a binary classification task, where the system continuously learns to distinguish between relevant and non-relevant images to the query, through iterative user interaction. This interaction is guided by an Active Learning loop: at each iteration, the system selects informative samples for user annotation, thereby refining the retrieval performance. This task is particularly challenging in multi-object datasets, where the object of interest may occupy only a small region of the image within a complex, cluttered scene. Unlike object-centered settings where global descriptors often suffice, multi-object images require more adapted, localized descriptors. In this work, we formulate and revisit the Human-in-the-Loop Object Retrieval task by leveraging pre-trained ViT representations, and addressing key design questions, including which object instances to consider in an image, what form the annotations should take, how Active Selection should be applied, and which representation strategies best capture the object's features. We compare several representation strategies across multi-object datasets highlighting trade-offs between capturing the global context and focusing on fine-grained local object details. Our results offer practical insights for the design of effective interactive retrieval pipelines based on Active Learning for object class retrieval.

Abstract

Open-Vocabulary Object Detection

Explore similar work

Revisiting Human-in-the-Loop Object Retrieval with Pre-Trained Vision Transformers

AiSearch: Interactive Multi-Modal Search with VLMs

Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification