Open-Vocabulary Object Detection

Overview

Open-vocabulary object detection studies how to find and localize objects described by language, including categories that were not labeled during detector training. Common approaches align image regions with text using vision-language models such as CLIP, then improve localization and recognition through grounding, prompting, pseudo-labeling, or adaptation. Methods are commonly evaluated on benchmarks such as COCO and LVIS, with growing attention to fine-grained queries, domain shifts, and open-world settings where unknown objects may appear. A central challenge is making predictions reliable across wording and domains while calibrating confidence so that novel objects are not confused with background or familiar categories.

Latest papers

94 papers5 papers in the last four weeks, down 44% on the four weeks before. 0.0% of all new papers.

Sep 29, 2025

Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detection

Vision-language models (VLMs) have advanced multimodal perception, demonstrated by open-vocabulary object detection with simple language queries. State-of-the-art VLMs still struggle to handle complex queries involving descriptive attributes and relational clauses. To address this problem, we propose restructuring linguistic representations according to the hierarchical relations within sentences for language-based object detection. A key insight is that textual tokens should be disentangled into core components-objects, attributes, and relations-and aggregated into hierarchically structured sentence-level representations. Building on this principle, we introduce the TaSe (Talk in Pieces, See in Whole) framework with three main contributions: (1) a hierarchical synthetic captioning dataset spanning three tiers from category names to descriptive sentences; (2) the three-component disentanglement module guided by a novel disentanglement loss function, transforms text embeddings into subspace compositions; and (3) aggregating disentangled components into hierarchically structured embeddings guided by the proposed hierarchical objectives. Experimental results under the OmniLabel benchmark show a 24% performance improvement, demonstrating the importance of linguistic compositionality.
Aug 15, 2025

Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively. \revise{The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.} Code is available at https://github.com/xiaomoguhz/DeCLIP
Jul 14, 2025

MLLM-Guided Autonomous Object Detection for Open-World Understanding

Object detection typically relies on predefined category vocabularies, while open-world and open-vocabulary approaches remain limited by generic unknown labels or externally specified detection concepts. We propose LAOD, an MLLM-Guided Autonomous Object Detection framework in which a Multimodal Large Language Model (MLLM) generates an image-specific detection vocabulary that is grounded by an open-vocabulary detector. This decomposition separates semantic concept discovery from spatial localization and removes the need to manually enumerate target categories at inference time. We evaluate LAOD using Class-Agnostic Average Precision (CAAP), Semantic Naming Average Precision (SNAP), and Semantic Localization Average Precision (SemLoc-AP), which measure localization, semantic naming, and their correct association, respectively. Experiments on COCO-Val, LVIS-minival, and COCO-OOD across multiple MLLMs and detectors demonstrate strong performance across common, long-tailed, and out-of-distribution objects, while showing that grounding quality remains a major determinant of end-to-end performance. LAOD also substantially outperforms both direct MLLM-based object detection and dedicated models trained specifically for open-ended detection. Overall, the results support a modular formulation of autonomous object detection in which MLLMs determine what to detect and specialized open-vocabulary detectors determine where those concepts occur.
Mar 19, 2025

Fine-Grained Open-Vocabulary Object Detection with Fined-Grained Prompts: Task, Dataset and Benchmark

Open-vocabulary detectors are proposed to locate and recognize objects in novel classes. However, variations in vision-aware language vocabulary data used for open-vocabulary learning can lead to unfair and unreliable evaluations. Recent evaluation methods have attempted to address this issue by incorporating object properties or adding locations and characteristics to the captions. Nevertheless, since these properties and locations depend on the specific details of the images instead of classes, detectors can not make accurate predictions without precise descriptions provided through human annotation. This paper introduces 3F-OVD, a novel task that extends supervised fine-grained object detection to the open-vocabulary setting. Our task is intuitive and challenging, requiring a deep understanding of Fine-grained captions and careful attention to Fine-grained details in images in order to accurately detect Fine-grained objects. Additionally, due to the scarcity of qualified fine-grained object detection datasets, we have created a new dataset, NEU-171K, tailored for both supervised and open-vocabulary settings. We benchmark state-of-the-art object detectors on our dataset for both settings. Furthermore, we propose a simple yet effective post-processing technique. Our data, annotations and codes are available at https://github.com/tengerye/3FOVD.