WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes

Kishor Datta Gupta; Ahmed Rafi Hasan; Md. Mahfuzur Rahman; Md. Sadman Haque; Mohd Ariful Haque

cs.CVAug 28, 2026

WALDO: One-Shot Exemplar-Conditioned Object Detection in Cluttered Scenes

Authors: Kishor Datta Gupta, Ahmed Rafi Hasan, Md. Mahfuzur Rahman, Md. Sadman Haque, Mohd Ariful Haque

Organizations: Department of Cyber Physical Systems, Clark Atlanta University, Atlanta, GA, USA · Department of Computer Science and Engineering, United International University, Dhaka, Bangladesh

Abstract

Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is absent, large vision-language models usually address this task. We ask whether the same capability is available far more cheaply, from representations already learned by a world-model pretraining objective. We present WALDO, a one-shot exemplar- and language-conditioned detection head with 3.4M trainable parameters that reads frozen V-JEPA 2.1 features to jointly predict object localization and target presence, with no gradient on the backbone. Because exemplar-conditioned supervision is scarce, we synthesize training episodes from instance annotations, mining exemplars from ground-truth boxes and constructing absence cases that exclude the referenced instance while leaving same-category distractors in view. This is easy to get wrong: in the obvious implementation, crop size alone predicts the label, and a head trained on it reaches 0.9998 absence AUROC without ever consulting the exemplar, and we report the negative controls that close the shortcut. On 35 held-out cluttered scenes, WALDO achieves a 0.461 catalogue AP@50, compared to 0.306 for a prompted Grounding DINO baseline under an identical scorer. Substituting DINOv3 for V-JEPA under a matched 576-token grid drops within-category absence AUROC from 0.880 to 0.726 and instance AP@50 from 0.201 to 0.141, isolating the pretraining objective rather than input resolution as the source of the gain. Instance-level Success@1, however, reaches only 0.190 against a 0.190 category-chance floor: world-model features transfer to localization precision and absence detection but not to instance identity.

Figures & tables

Figure 1: WALDO is named after the search-and-find character from the game Where’s Waldo? [ 22 ] , who is likewise found by matching one distinctive figure against a densely cluttered scene.

Figure 2: WALDO architecture. (1) Episodes: category- and instance-level positives, within- and cross-category absences. (2) A shared frozen V-JEPA 2.1 encodes exemplar and query region into $Z_{e}$ and $Z_{q}$ , whose per-patch cosine similarity $c$ is appended to the tokens; a frozen SigLIP2 text tower encodes the description into $u$ . (3) Tokens are FiLM-conditioned on $(Z_{e},u)$ , self-attended, and read out by box, objectness and presence heads.

	Train	Val	Test
Scenes	$150$	$35$	$35$
Instances	$523$	$124$	$116$

Table 1: Experimental setup. Top : splits, by scene, stratified by category at seed

0

. Bottom : backbone arms. Patch size differs across encoders, so we fix the grid at

24\times 24=576

and derive the input as

24\times\text{patch}

; at an input matched to V-JEPA’s

384

px, DINOv3’s patch-

14

grid would be

28\times 28=784

tokens against V-JEPA’s

576

, and the difference could be read as extra resolution rather than better features. This matches the token budget, not the pixel budget: DINOv3 receives

23\%

fewer input pixels and is the only arm run away from its pretraining resolution, both of which cut against it.

		Success@ $K$		AP				Best F1				P / R @ 0.5		Mean IoU
Setting	Backbone	@1	@5	@25	@50	@75	[.5:.95]	F1	P	R	thr	P	R	top-1	matched
Catalogue $n{=}35$	V-JEPA 2.1	.629	.771	.530	.461	.168	.208	.453	.476	.431	.34	1.000	.181	.502	.777
	DINOv3	.600	.886	.492	.428	.021	.139	.471	.452	.491	.34	.857	.103	.425	.719
	SigLIP2	.743	.829	.522	.481	.063	.189	.516	.686	.414	.37	1.000	.121	.530	.750
Instance $n{=}116$	V-JEPA 2.1	.190	.517	.237	.201	.108	.108	.335	.427	.276	.33	–	–	.174	.845
	DINOv3	.190	.474	.179	.141	.020	.049	.282	.259	.310	.28	–	–	.145	n/a

Table 2: Frozen-backbone ablation, single-pass inference. One row per training run; head, episodes, assignment, split and seed are identical across rows and every backbone emits a matched

24\times 24

grid ( Table 1 ), so the pretrained representation is the only free variable. The two blocks are one forward pass scored under two acceptance rules and must not be compared to one another : in the instance block exactly one box per query is correct and every same-category object is a false positive by construction. IoU

{}_{\text{top-1}}

counts misses as zero; IoU

{}_{\text{matched}}

averages over true positives only and is n/a when a backbone yields none at the fixed

0.5

threshold, where P/R is tabulated for the catalogue block alone.

				Best F1		Mean IoU		Can express
Method	Conditioning	S@1	AP@50	F1	R	top-1	matched	Cat.	Inst.	Abs.
WALDO (SigLIP2)	exemplar + text	.743	.481	.516	.414	.530	.750	✓	✓	✓
WALDO (V-JEPA)	exemplar + text	.629	.461	.453	.431	.502	.777	✓	✓	✓
Grounding DINO	category prompt	.429	.306	.357	.543	.335	.770	✓	$\times$	$\times$
Grounding DINO	generic prompt	.114	.101	.162	.543	.093	n/a	✓	$\times$	$\times$
OWLv2	category prompt	.029	.109	.211	.707	.032	.754	✓	$\times^{\dagger}$	$\times$

Table 3: Comparison with frozen open-vocabulary detectors. All rows share test scenes, scorer, NMS (IoU

\,{=}\,0.5

) and detection cap. Scores are restricted to catalogue retrieval, the only setting a prompt-conditioned detector can express; Can express records that interface limitation, not accuracy, and is why Table 4 and the instance block of Table 2 carry no baseline row. Generic prompts use a class-agnostic vocabulary ( object. part. vehicle. … ) naming neither target.

Panel of Figure 3 — Figure 3: One query, five methods, same scene and same target instance. A four-car parking lot; the exemplar is cropped from one specific car. Panels (a) to (c) score under the instance rule: green is that car, red the other three (same-category distractors). Panels (d) and (e) are frozen baselines with no channel to receive which car is meant, so they score under the easier catalogue rule where green marks all four; the two groups do not resolve the same query. Orange is each method’s top-1 box, yellow its next four, cyan in (a) the region examined. Both baselines rank a different car first (IoU $0.000$ and $0.061$ ).

	AUROC		Mean $p(\text{pres.})$		Instance S@1
Backbone	hard	easy	pres.	hard abs.	floor	ratio
V-JEPA 2.1	.880	1.000	.862	.456	.190	1.00 $\times$
DINOv3	.726	1.000	.667	.536	.181	1.05 $\times$
SigLIP2	.831	1.000	.756	.382	.224	1.00 $\times$

Table 4: Per-backbone diagnostics. Top : absence and the instance floor. Hard negatives exclude the exemplar’s object while often retaining a same-category instance (

n{=}46

), easy negatives pair it with a scene of the other category (

n{=}35

), positives

n{=}151

; floor is

\text{S@1}_{\text{cat}}\times\mathbb{E}[1/n_{\text{inst}}]

, recomputed per row, so a ratio near

1.00\times

means the exemplar is not being used. Bottom : the catalogue AP@50 of Table 2 split by object scale and by IoU strictness.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding
Apr 9, 2026Ashutosh Kumar, Rajat Saini, Jingjing Pan +5Vision-Language Pretraining Vision-Language Alignment
Woven by Toyota
Wasserstein-Aligned Localisation for VLM-Based Distributional OOD Detection in Medical Imaging
May 6, 2026Bernhard Kainz, Johanna P Mueller, Matthew Baugh +1Medical Image Anomaly Detection Anomaly Localization
Dept. AIBE, Friedrich–Alexander University Erlangen–Nürnberg, DE · Department of Computing, Imperial College London, UK · Technical University Munich, DE +1
RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection
Oct 7, 2026Yupeng Zhang, Ziyi Zhao, Juntao Cheng +4Open-Vocabulary Object Detection Real-Time Object Detection
Tianjin University. · Shenzhen University of Advanced Technology. · Shanghai Jiaotong University. +1

Apr 9, 2026J/K move · Enter open · S save

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

Ashutosh Kumar, Rajat Saini, Jingjing Pan, Mustafa Erdogan+4

Woven by Toyota

Vision-Language Pretraining Vision-Language Alignment Vision-Language Grounding

Open paper AskPDF arXiv

Current vision-language pre-training (VLP) paradigms excel at global scene understanding but struggle with instance-level reasoning due to global-only supervision. We introduce InstAP, an Instance-Aware Pre-training framework that jointly optimizes global vision-text alignment and fine-grained, instance-level contrastive alignment by grounding textual mentions to specific spatial-temporal regions. To support this, we present InstVL, a large-scale dataset (2 million images, 50,000 videos) with dual-granularity annotations: holistic scene captions and dense, grounded instance descriptions. On the InstVL benchmark, InstAP substantially outperforms existing VLP models on instance-level retrieval, and also surpasses a strong VLP baseline trained on the exact same data corpus, isolating the benefit of our instance-aware objective. Moreover, instance-centric pre-training improves global understanding: InstAP achieves competitive zero-shot performance on multiple video benchmarks, including MSR-VTT and DiDeMo. Qualitative visualizations further show that InstAP localizes textual mentions to the correct instances, while global-only models exhibit more diffuse, scene-level attention.

Abstract

Object Detection

Explore similar work

InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding

Wasserstein-Aligned Localisation for VLM-Based Distributional OOD Detection in Medical Imaging

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection