MLLM-Guided Autonomous Object Detection for Open-World Understanding

Furkan Mumcu; Michael J. Jones; Anoop Cherian; Yasin Yilmaz

cs.CVJul 14, 2025

MLLM-Guided Autonomous Object Detection for Open-World Understanding

Authors: Furkan Mumcu, Michael J. Jones, Anoop Cherian, Yasin Yilmaz

Organizations: University of South Florida · Mitsubishi Electric Research Laboratories (MERL)

Abstract

Object detection typically relies on predefined category vocabularies, while open-world and open-vocabulary approaches remain limited by generic unknown labels or externally specified detection concepts. We propose LAOD, an MLLM-Guided Autonomous Object Detection framework in which a Multimodal Large Language Model (MLLM) generates an image-specific detection vocabulary that is grounded by an open-vocabulary detector. This decomposition separates semantic concept discovery from spatial localization and removes the need to manually enumerate target categories at inference time. We evaluate LAOD using Class-Agnostic Average Precision (CAAP), Semantic Naming Average Precision (SNAP), and Semantic Localization Average Precision (SemLoc-AP), which measure localization, semantic naming, and their correct association, respectively. Experiments on COCO-Val, LVIS-minival, and COCO-OOD across multiple MLLMs and detectors demonstrate strong performance across common, long-tailed, and out-of-distribution objects, while showing that grounding quality remains a major determinant of end-to-end performance. LAOD also substantially outperforms both direct MLLM-based object detection and dedicated models trained specifically for open-ended detection. Overall, the results support a modular formulation of autonomous object detection in which MLLMs determine what to detect and specialized open-vocabulary detectors determine where those concepts occur.

Figures & tables

Figure 1: Comparison of object detection paradigms: traditional object detection, OWOD, OVOD, and our proposed autonomous L A O D framework.

Figure 2: Overview of the M L LM-Guided A utonomous O bject D etection ( L A O D ) framework. Given an input image, the MLLM generates an image-specific detection vocabulary, which is then grounded by an open-vocabulary object detector to localize the corresponding objects.

Situation	CAAP	SNAP	SemLoc-AP
Good localization, poor naming	High	Low	Low
Good naming, poor localization	Low	High	Low
Both good and correctly associated	High	High	High
Both good separately, but associated with different objects	High	High	Low
Poor localization and naming	Low	Low	Low

Table 1: Illustrative interpretation of CAAP, SNAP, and SemLoc-AP. The three metrics capture localization quality, semantic naming quality, and their correct association, respectively.

Figure 3: CAAP and SNAP on COCO-Val for all 28 combinations of seven MLLMs and four open-vocabulary detectors. Each cell reports the macro score averaged over thresholds from $0.50$ to $0.95$ . Darker shading indicates higher performance.

Figure 4: CAAP, SNAP, and SemLoc-AP on LVIS-minival for three MLLMs paired with four open-vocabulary detectors. The three grids are shown vertically for clarity. CAAP and SNAP report macro averages over thresholds from $0.50$ to $0.95$ , while SemLoc-AP averages over the full Cartesian product of the corresponding spatial and semantic thresholds.

Figure 5: CAAP on COCO-OOD for all 28 combinations of seven MLLMs and four open-vocabulary detectors. Each cell reports the macro score averaged over IoU thresholds from $0.50$ to $0.95$ .

Method	U-AP	U-F1	U-PRE	U-REC
Grounding DINO [ 26 , 22 ]	0.26	0.41	0.83	0.28
OW-DETR [ 13 ]	0.03	0.05	0.03	0.38
VOS [ 7 ]	0.20	0.31	0.29	0.34
UnSniffer [ 19 ]	0.45	0.47	0.43	0.53
UN-DETR [ 22 ]	0.47	0.54	0.54	0.55
L A O D (Qwen3.5-9B + YOLO-World)	0.769	0.720	0.787	0.663

Table 2: Unknown-object detection performance on COCO-OOD. For L A O D , we report the main Qwen3.5-9B + YOLO-World configuration selected from the COCO-Val experiments rather than selecting a configuration specifically for COCO-OOD. The standalone Grounding DINO result is reported by UN-DETR [ 22 ] using the generic prompt “unknown object.”

Dataset	Method	CAAP	SNAP	SemLoc-AP	U-AP
COCO-Val	GenerateU (Swin-L, VG+GRIT5M)	0.1976	0.1470	0.0821	–
	L A O D detector avg. range	0.2129–0.2882	0.2097–0.2899	0.1249–0.1779	–
	Qwen2.5-VL-7B Direct	0.1422	0.0842	0.0359	–
	L A O D (Qwen2.5-VL-7B + YOLO-World)	0.3004	0.3110	0.1875	–
	InternVL3-8B Direct	0.1329	0.2072	0.0841	–
	L A O D (InternVL3-8B + YOLO-World)	0.2815	0.2835	0.1716	–

Table 3: Comparison with GenerateU, a task-specific open-ended detector, and direct MLLM detection. For the “ L A O D detector avg.” rows, each range spans the four OVOD-specific detector averages, where each detector is averaged over the evaluated MLLMs. SNAP and SemLoc-AP are not reported on COCO-OOD because all out-of-distribution objects share a single generic unknown label.

Prompt	User prompt
Default	Give me a list of objects that you see in this image. Just give the list of objects comma seperated, don’t explain anything. Do not list sky, street.
Minimal	List the objects that you see in this image.
COCO-specific	Give me a list of objects that you see in this image. Just give the list of objects comma seperated, don’t explain anything. Write only the general name of the objects. Do not list infrastructural objects. Don’t use plural, use singular object names.

Table 4: Prompt variants used in the sensitivity analysis. All variants use the same system prompt; only the user prompt changes.

Dataset	Prompt	CAAP	SNAP	SemLoc-AP	$\Delta$ CAAP vs. Default
COCO-Val	Default	0.2857	0.2955	0.1740	–
	Minimal	0.2855	0.2853	0.1707	$+0.0\%$
	COCO-specific	0.3257	0.4150	0.2204	$+14.3\%$
LVIS-minival	Default	0.1737	0.1821	0.1058	–
	Minimal	0.1718	0.1745	0.1026	$-1.2\%$
	COCO-specific	0.1514	0.1651	0.0983	$-13.1\%$

Table 5: Prompt sensitivity on 500-image subsets of COCO-Val and LVIS-minival. Scores are averaged over the three MLLMs common to both experiments: Gemma4-12B, Gemma4-E4B, and Qwen3.5-9B. The final column reports the mean per-MLLM relative CAAP change with respect to the Default prompt.

Panel of Figure 6 — Figure 6: Qualitative comparison between direct MLLM detection and the two-stage L A O D formulation on COCO-Val. In each row, the same MLLM is used either to directly generate object labels and bounding boxes (middle) or to generate the image-specific vocabulary subsequently grounded by YOLO-World (right). In (a), direct Qwen2.5-VL-7B produces a single prediction for a group of elephants rather than enumerating the individual instances, while the corresponding L A O D configuration localizes them separately. In (b), InternVL3-8B produces a generally plausible set of object detections, but misses an annotated spoon and introduces a spurious chair detection, illustrating that direct MLLM detection can still omit relevant instances and introduce unsupported objects even when the overall scene interpretation is reasonable. Panel (c) shows a case in which direct MLLM detection succeeds, demonstrating that the aggregate performance gap does not imply that MLLMs are incapable of generating accurate object detections. For readability, the L A O D panels show only the highest-ranked predictions; quantitative evaluation uses the complete prediction set subject to the common maximum of 100 detections per image.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

MLLM	Detector	LO	MI	HI	Macro
Gemma4-E4B	YOLO-World	0.4027	0.3354	0.1517	0.3005
	G-DINO-B	0.3762	0.3197	0.1419	0.2833
	G-DINO-T	0.3733	0.3183	0.1496	0.2842
	OWLv2	0.3094	0.2496	0.0945	0.2210
Qwen3.5-9B	YOLO-World	0.4098	0.3389	0.1484	0.3030
	G-DINO-B	0.3762	0.3157	0.1316	0.2786

Appendix

Table 6: Threshold-wise CAAP on COCO-Val. LO, MI, and HI denote the mean AP over IoU thresholds

\{0.50,0.55,0.60\}

\{0.65,0.70,0.75,0.80\}

, and

\{0.85,0.90,0.95\}

, respectively. Macro is averaged over all ten IoU thresholds.

MLLM	Detector	LO	MI	HI	Macro
Gemma4-E4B	YOLO-World	0.3960	0.2927	0.2289	0.3045
	G-DINO-B	0.3158	0.2079	0.1541	0.2241
	G-DINO-T	0.3292	0.2299	0.1762	0.2436
	OWLv2	0.2879	0.2226	0.1579	0.2228
Qwen3.5-9B	YOLO-World	0.4191	0.2989	0.2191	0.3110
	G-DINO-B	0.3255	0.2094	0.1492	0.2262

Appendix

Table 7: Threshold-wise SNAP on COCO-Val using Sentence-BERT for semantic similarity. LO, MI, and HI correspond to similarity thresholds

\{0.50,0.55,0.60\}

\{0.65,0.70,0.75,0.80\}

, and

\{0.85,0.90,0.95\}

, respectively. Macro is averaged over all ten thresholds.

	0.50	0.55	0.60	0.65	0.70	0.75	0.80	0.85	0.90	0.95
$S_{\mathrm{IoU}}(\theta)$	0.1906	0.1861	0.1810	0.1748	0.1667	0.1547	0.1383	0.1140	0.0780	0.0257
$S_{\mathrm{sem}}(\tau)$	0.1875	0.1802	0.1758	0.1483	0.1415	0.1358	0.1300	0.1194	0.1000	0.0913

Appendix

Table 8: SemLoc-AP threshold marginals on COCO-Val, averaged over all 28 MLLM–detector combinations. The first row fixes the IoU threshold

\theta

and averages over all ten semantic thresholds

\tau

; the second fixes

\tau

and averages over all ten IoU thresholds

\theta

. The overall SemLoc-AP averaged over all 100 threshold pairs is

0.1410

		CAAP				SNAP
MLLM	Detector	LO	MI	HI	Macro	LO	MI	HI	Macro
Qwen3.5-9B	YOLO-World	0.2306	0.1867	0.0944	0.1722	0.2628	0.1875	0.0993	0.1836
	G-DINO-B	0.1777	0.1436	0.0632	0.1297	0.2026	0.1431	0.0793	0.1418
	G-DINO-T	0.2025	0.1669	0.0833	0.1525	0.2248	0.1575	0.0886	0.1570
	OWLv2	0.2908	0.2294	0.0915	0.2064	0.3276	0.2427	0.1326	0.2352
Gemma4-12B	YOLO-World	0.2351	0.1904	0.0948	0.1751	0.2623	0.1827	0.0899	0.1787

Appendix

Table 9: Threshold-wise CAAP and SNAP on LVIS-minival. Macro scores are averaged over all ten thresholds rather than over the three interval means.

	0.50	0.55	0.60	0.65	0.70	0.75	0.80	0.85	0.90	0.95
$S_{\mathrm{IoU}}(\theta)$	0.1210	0.1177	0.1138	0.1092	0.1034	0.0959	0.0859	0.0715	0.0513	0.0230
$S_{\mathrm{sem}}(\tau)$	0.1231	0.1175	0.1133	0.1061	0.1005	0.0874	0.0774	0.0707	0.0535	0.0433

Appendix

Table 10: SemLoc-AP threshold marginals on LVIS-minival, averaged over all 12 MLLM–detector combinations. The first row fixes the IoU threshold

\theta

and averages over all ten semantic thresholds

\tau

; the second fixes

\tau

and averages over all ten IoU thresholds

\theta

. The overall SemLoc-AP averaged over all 100 threshold pairs is

0.0893

MLLM	Detector	LO	MI	HI	Macro
Gemma4-E4B	YOLO-World	0.7503	0.6873	0.4267	0.6280
	G-DINO-B	0.6208	0.5739	0.3430	0.5187
	G-DINO-T	0.6624	0.6134	0.3839	0.5592
	OWLv2	0.6945	0.6319	0.3391	0.5628
Qwen2.5-VL-7B	YOLO-World	0.7611	0.6953	0.4310	0.6357
	G-DINO-B	0.5854	0.5431	0.3241	0.4901

Appendix

Table 11: Threshold-wise CAAP on COCO-OOD for all 28 MLLM–detector combinations. Macro is averaged over all ten IoU thresholds from

0.50

0.95

MLLM	Detector	U-AP	U-F1	U-PRE	U-REC
Gemma4-E4B	YOLO-World	0.762	0.721	0.804	0.653
	G-DINO-B	0.629	0.642	0.626	0.658
	G-DINO-T	0.671	0.666	0.634	0.701
	OWLv2	0.703	0.666	0.745	0.602
Qwen3.5-9B	YOLO-World †	0.769	0.720	0.787	0.663
	G-DINO-B	0.591	0.593	0.519	0.692

Appendix

Table 12: Complete unknown-object detection results on COCO-OOD for all 28 MLLM–detector combinations. U-PRE and U-REC are measured at the operating point that maximizes U-F1 for each configuration.

\dagger

denotes the main Qwen3.5-9B + YOLO-World configuration reported in the main-paper comparison.

MLLM	Default	Minimal	COCO-specific
Gemma3-4B	0.2813	0.2871	0.2894
Gemma4-12B	0.2699	0.2799	0.3417
Gemma4-E2B	0.2680	0.2652	0.2886
Gemma4-E4B	0.2928	0.2936	0.3224
InternVL3-8B	0.2785	0.2764	0.2894
Qwen2.5-VL-7B	0.2943	0.2990	0.3190

Appendix

Table 13: CAAP on the 500-image COCO-Val prompt-ablation subset.

MLLM	Default	Minimal	COCO-specific
Gemma3-4B	0.2630	0.2647	0.2897
Gemma4-12B	0.2923	0.2892	0.4681
Gemma4-E2B	0.2596	0.2561	0.3308
Gemma4-E4B	0.2955	0.2897	0.3989
InternVL3-8B	0.2771	0.2696	0.3086
Qwen2.5-VL-7B	0.3031	0.3081	0.3796

Appendix

Table 14: SNAP on the 500-image COCO-Val prompt-ablation subset. Semantic similarity is computed using Sentence-BERT.

MLLM	Default	Minimal	COCO-specific
Gemma3-4B	8.0	8.1	6.8
Gemma4-12B	11.8	17.6	9.3
Gemma4-E2B	3.6	3.8	2.8
Gemma4-E4B	4.3	4.7	3.6
InternVL3-8B	8.6	9.0	7.1
Qwen2.5-VL-7B	8.5	8.2	6.9

Appendix

Table 15: Average number of labels generated per image on the COCO-Val prompt-ablation subset.

MLLM	Default	Minimal	COCO-specific
Gemma4-12B	0.1855	0.1840	0.1597
Gemma4-E4B	0.1548	0.1487	0.1264
Qwen3.5-9B	0.1808	0.1828	0.1681
Mean	0.1737	0.1718	0.1514

Appendix

Table 16: CAAP on the 500-image LVIS-minival prompt-ablation subset.

MLLM	Default	Minimal	COCO-specific
Gemma4-12B	0.1933	0.1863	0.1761
Gemma4-E4B	0.1564	0.1482	0.1279
Qwen3.5-9B	0.1965	0.1891	0.1913
Mean	0.1821	0.1745	0.1651

Appendix

Table 17: SNAP on the 500-image LVIS-minival prompt-ablation subset. Semantic similarity is computed using Sentence-BERT.

MLLM	Default	Minimal	COCO-specific
Gemma4-12B	11.7	17.3	9.0
Gemma4-E4B	4.2	4.6	3.5
Qwen3.5-9B	7.8	9.5	6.7
Mean	7.9	10.5	6.4

Appendix

Table 18: Average number of labels generated per image on the LVIS-minival prompt-ablation subset.

Configuration	Method	Refined (%)	MLLM Calls/Img	OVOD Passes/Img	CAAP	SNAP	SemLoc-AP
Qwen3.5-9B + YOLO-World	Standard L A O D	0	1.00	1.00	0.2809	0.2957	0.1712
	Always Refine	100	2.00	2.00	0.2807	0.3030	0.1730
	Selective Refinement	60	1.60	1.60	0.2844	0.3076	0.1759
Gemma4-E4B + YOLO-World	Standard L A O D	0	1.00	1.00	0.2780	0.2898	0.1741
	Always Refine	100	2.00	2.00	0.2740	0.2910	0.1760
	Selective Refinement	60	1.60	1.60	0.2777	0.2945	0.1782

Appendix

Table 19: Comparison of standard one-pass L A O D , unconditional refinement, and selective MLLM–OVOD refinement on the 500-image COCO-Val analysis subset. Selective refinement applies one additional MLLM interaction and OVOD pass to the

60\%

of images with the lowest first-pass maximum detector confidence. The same refinement budget is used for all configurations. MLLM Calls/Img and OVOD Passes/Img denote the nominal inference requirements; the outputs used in this retrospective analysis were already cached. Bold values indicate the best result within each configuration and metric.

Figure 7: Effect of the refinement rate for the three evaluated MLLM–detector configurations. Images are ordered by first-pass maximum detector confidence, and the lowest-confidence fraction is refined. Qwen3.5-9B + YOLO-World exhibits a broad high-performing region around $55$ – $75\%$ , while Gemma4-E4B + YOLO-World reaches its strongest region earlier, around $40$ – $50\%$ . In contrast, Qwen3.5-9B + OWLv2 degrades as the refinement rate increases. The primary comparison uses a common $60\%$ budget rather than independently optimizing each configuration.

Configuration	CAAP	SNAP	SemLoc-AP
Qwen3.5-9B + YOLO-World	99.2%	99.5%	99.1%
Gemma4-E4B + YOLO-World	88.0%	95.2%	97.1%
Qwen3.5-9B + OWLv2	72.6%	4.4%	4.1%

Appendix

Table 20: Position of low-confidence selective refinement within the 1,000-run random-refinement distribution at the same

60\%

refinement budget. Higher percentiles indicate that selecting low-confidence images is more effective than random selection.

Encoder	LO gain	MI gain	HI gain	Unrelated pairs at $\tau=.50$
CLIP ViT-B/32	$0.0000$	$0.0068$ – $0.0861$	$0.12$ – $0.27$	$100\%$
Sentence-BERT	$0.21$ – $0.36$	$0.17$ – $0.28$	$0.10$ – $0.17$	$2\%$

Appendix

Table 21: Behavior of CLIP and Sentence-BERT for SNAP on COCO-Val. The “gain over shuffle” measures the difference between SNAP using the original predicted labels and SNAP after randomly permuting the predicted label strings while keeping bounding boxes and confidence scores unchanged. Ranges summarize the 28 MLLM–detector configurations.

Detector	Confidence parameter	Selected threshold
YOLO-World	conf	0.001
G-DINO-T	score_threshold	0.03
G-DINO-B	score_threshold	0.03
OWLv2	score_threshold	0.01

Appendix

Table 22: Detector confidence thresholds selected on the hyperparameter holdout. A single threshold is fixed for each detector and used across all evaluated datasets and MLLMs.

Conf.	CAAP	F1	Precision	Recall
0.001	0.3030	0.1770	0.100	0.803
0.005	0.3003	0.2545	0.152	0.767
0.010	0.2974	0.3043	0.192	0.740
0.020	0.2925	0.3611	0.243	0.705
0.030	0.2883	0.3954	0.279	0.678
0.050	0.2816	0.4340	0.329	0.639

Appendix

Table 23: Confidence-threshold sweep for Qwen3.5-9B + YOLO-World on COCO-Val. F1, precision, and recall are computed at IoU

=0.5

. CAAP uses the full IoU range from

0.50

0.95

Model	COCO-Val	COCO-OOD
Qwen2.5-VL-7B	70	0
InternVL3-8B	328	1

Appendix

Table 24: Parsing statistics for the direct-detection baselines. A parsing failure indicates that the generated response could not be converted into the required bounding-box and label representation.

Detector	Vocabulary	CAAP	SNAP	SemLoc-AP	Det./Img Pre-cap	Det./Img Post-cap
Grounding DINO-B	COCO-80	0.5559	0.8803	0.5408	190.1	75.5
Grounding DINO-T	COCO-80	0.4935	0.7815	0.4688	220.8	81.1
YOLO-World	COCO-80	0.4916	0.7958	0.4686	103.8	63.4
OWLv2	COCO-80	0.3289	0.5925	0.3072	227.2	83.3

Appendix

Table 25: Fixed-ontology reference on COCO-Val. Every image is queried using the complete COCO-80 category set. “Pre-cap” reports the mean number of candidate detections per image before the common

\mathrm{maxDets}=100

truncation, while “Post-cap” reports the mean number retained afterward.

Detector	Vocabulary	CAAP	SNAP	SemLoc-AP	Det./Img Pre-cap	Det./Img Post-cap	Chunks
OWLv2	LVIS-1203	0.2461	0.3434	0.1801	603.9	98.7	1
YOLO-World	LVIS-1203	0.1639	0.1731	0.0976	238.0	97.0	1
Grounding DINO-B	LVIS-1203	0.1031	0.1346	0.0769	912.7	99.9	16
Grounding DINO-T	LVIS-1203	0.0689	0.0696	0.0418	11,745.9	100.0	16

Appendix

Table 26: Fixed-ontology reference on LVIS-minival using the complete 1,203-category LVIS ontology. Grounding DINO requires 16 independent text passes because the ontology exceeds its text-encoder capacity. The pre-cap counts for the Grounding DINO variants therefore aggregate detections across all 16 passes and are not directly comparable with the single-pass counts of YOLO-World and OWLv2.

Dataset	Detector	CAAP Auto $\rightarrow$ Fixed	$\Delta$ CAAP	SNAP Auto $\rightarrow$ Fixed	$\Delta$ SNAP	SemLoc Auto $\rightarrow$ Fixed	$\Delta$ SemLoc
COCO	YOLO-World	$0.2882\rightarrow 0.4916$	$+0.2034$	$0.2899\rightarrow 0.7958$	$+0.5059$	$0.1779\rightarrow 0.4686$	$+0.2907$
	G-DINO-B	$0.2634\rightarrow 0.5559$	$+0.2925$	$0.2097\rightarrow 0.8803$	$+0.6706$	$0.1270\rightarrow 0.5408$	$+0.4138$
	G-DINO-T	$0.2522\rightarrow 0.4935$	$+0.2413$	$0.2179\rightarrow 0.7815$	$+0.5636$	$0.1342\rightarrow 0.4688$	$+0.3346$
	OWLv2	$0.2129\rightarrow 0.3289$	$+0.1160$	$0.2159\rightarrow 0.5925$	$+0.3766$	$0.1249\rightarrow 0.3072$	$+0.1823$
LVIS	YOLO-World	$0.1645\rightarrow 0.1639$	$-0.0006$	$0.1687\rightarrow 0.1731$	$+0.0044$	$0.0913\rightarrow 0.0976$	$+0.0063$
	OWLv2	$0.1963\rightarrow 0.2461$	$+0.0498$	$0.2159\rightarrow 0.3434$	$+0.1275$	$0.1172\rightarrow 0.1801$	$+0.0629$

Appendix

Table 27: Comparison between autonomous L A O D detector averages and the fixed-ontology reference. Positive values indicate improvement when the complete benchmark ontology is supplied directly. The Grounding DINO LVIS comparisons are affected by the multi-pass limitation discussed in Section H.5 .

Detector	Boxes/ Chunk	Per-Chunk Max Score	Per-Chunk Median Score
G-DINO-T	732–859	0.247–0.608	0.045–0.066
G-DINO-B	1–18	0.087–0.793	0.034–0.106

Appendix

Table 28: Observed Grounding DINO behavior when the complete LVIS ontology is processed in 16 independent vocabulary chunks.

Panel of Figure 8 — Figure 8: Qualitative examples across the three evaluation settings. On COCO-Val, L A O D localizes the annotated objects while also identifying additional plausible concepts outside the benchmark annotations. On LVIS-minival, it recognizes fine-grained objects and semantically related names, such as cutting board for the annotated chopping board . On COCO-OOD, L A O D assigns meaningful names such as teddy bears , plates , and flowers , whereas the benchmark represents these objects using the generic unknown label.

Open-Vocabulary Object Detection

Open-vocabulary object detection lets models locate and label objects using text categories beyond their fixed training labels.

See all 89 papers

VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection
May 5, 2026Chih-Chung Liu, Zhiwei Lin, Yongtao WangOpen-Vocabulary Object Detection Object Detection
Wangxuan Institute of Computer Technology, Peking University, China
MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection
Oct 16, 2025Hojun Choi, Youngsun Lim, Jaeyo Shin +1Open-Vocabulary Object Detection Visual Reasoning
KAIST AI, South Korea · Boston University, USA
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Apr 3, 2026Siheng Wang, Yanshu Li, Bohan Hu +12Open-Vocabulary Object Detection Detection Transformer
Jiangsu University · MBZUAI · Brown University +5

May 5, 2026J/K move · Enter open · S save

VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection

Chih-Chung Liu, Zhiwei Lin, Yongtao Wang

Wangxuan Institute of Computer Technology, Peking University, China

Open-Vocabulary Object Detection Object Detection

Open paper AskPDF arXiv

Open-world object detection aims to localize and recognize objects beyond a fixed closed-set label space. It is commonly divided into two categories, i.e., open-vocabulary detection, which assumes a predefined category list at test time, and open-ended detection, which requires generating candidate categories during the inference. Existing methods rely primarily on coarse textual semantics and parametric knowledge, which often provide insufficient visual evidence for fine-grained appearance variation, rare categories, and cluttered scenes. In this paper, we propose VL-SAM-v3, a unified framework that augments open-world detection with retrieval-grounded external visual memory. Specifically, once candidate categories are available, VL-SAM-v3 retrieves relevant visual prototypes from a non-parametric memory bank and transforms them into two complementary visual priors, i.e., sparse priors for instance-level spatial anchoring and dense priors for class-aware local context. These priors are integrated with the original detection prompts via Memory-Guided Prompt Refinement, enabling a shared retrieval-and-refinement mechanism that supports open-vocabulary and open-ended inference. Extensive zero-shot experiments on LVIS show that VL-SAM-v3 consistently improves detection performance under both open-vocabulary and open-ended inference, with particularly strong gains on rare categories. Moreover, experiments with a stronger open-vocabulary detector (i.e., SAM3) validate the generality of the proposed retrieval-and-refinement mechanism.

Abstract

Open-Vocabulary Object Detection

Explore similar work

VL-SAM-v3: Memory-Guided Visual Priors for Open-World Object Detection

MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection

DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection