Object detection research remains constrained by high annotation costs and the closed-set assumptions of supervised models, limiting scalability to novel categories. We present a target-dataset annotation-free object detection framework that eliminates the need for human labeling on the target dataset by leveraging the complementary strengths of pretrained foundation models, including segmentation and vision-language systems, combined with a self-refining training loop. To further reduce reliance on expensive LVLM queries, we introduce a landmark-based clustering mechanism that amortizes labeling across image clusters, improving label consistency while lowering computational cost. Our method autonomously generates segmentation proposals, assigns semantic labels through multimodal reasoning, and iteratively improves both localization and classification using a lightweight open-set detector. This modular pipeline enables adaptation to completely unseen object classes and domains without any human-provided target-dataset annotations, while maintaining compatibility with existing detection architectures. Evaluations on COCO, LVIS, ODinW-35, and Roboflow-100 demonstrate substantial performance gains over state-of-the-art open-vocabulary and long-tail detection methods, achieving 60.8 AP on COCO and 47.4 AP on LVIS under zero-shot settings. Beyond strong benchmark results, our approach provides a scalable foundation for real-world detection systems in unknown environments, where annotation is infeasible.
Figures & tables
Figure 2: Traditional object detection requires annotated paired data and remains closed-set. Our annotator-free pipeline relies only on image scenes, generating proposals via foundational models and refining them iteratively to expand to novel categories.
Figure 3: Proposed annotator-free pipeline. Stage 1: LVLM and SAM generate scene-level labels and object proposals. Stage 2: CLIP encoders assign semantic labels to proposals. Stage 3: an open-set detector trained on these proposals generates refined prompts for SAM, which are re-labeled by the LVLM.
Figure 4: Word clouds of COCO ground-truth instances (top) and our pseudo-labeled annotations (bottom). The proposed annotation scheme preserves relative class frequencies (e.g., person, car, table) while introducing additional categories (e.g., fence, building, window), thereby enriching label diversity. This demonstrates that our method not only captures the underlying distribution but also scales to novel concepts beyond the original dataset.
Model
Backbone
COCO val
LVIS
GLIP-L [ 13 ]
Swin-L
49.8
26.9
DINOv [ 34 ]
Swin-L
46.2
-
GDINO-L [ 15 ]
Swin-L
48.4
27.4
T-Rex2 (Text) [ 10 ]
Swin-L
52.2
45.8
T-Rex2 (Visual) [ 10 ]
Swin-L
46.5
45.3
DetCLIPv3 [ 31 ]
Swin-T
47.2
47.0
Table 1: Annotator-free object detection performance comparison on COCO val and LVIS minival. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.
Model
Backbone
ODinW
Roboflow
GLIP-L [ 13 ]
Swin-L
23.4
8.6
DINOv [ 34 ]
Swin-L
15.7
-
GDINO-L [ 15 ]
Swin-L
22.3
8.3
T-Rex2 (Text) [ 10 ]
Swin-L
22.0
10.5
T-Rex2 (Visual) [ 10 ]
Swin-L
27.8
18.5
Ours
Swin-T
30.1
20.8
Table 2: Annotator-free object detection performance comparison on ODinW-35 val and Roboflow-100 val. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.
Method
Backbone
Novel
Known
All
OVR-CNN [ 33 ]
RN50
22.8
46.0
39.9
ViLD [ 8 ]
ViT-B/32
27.6
59.5
51.3
Detic [ 40 ]
RN50
27.8
47.1
45.0
OV-DETR [ 32 ]
ViT-B/32
29.4
61.0
52.7
BARON [ 29 ]
RN50
34.0
60.4
53.5
CORA [ 30 ]
RN50x4
41.7
44.5
43.8
Table 3: Comparison for novel object detection with other state-of-the-art methods. Performance is evaluated on a novel, known, and all classes on COCO OVD split following a generalized open-vocabulary detection setting.
Figure 5: Ablation of performance improvement across pipeline stages (proposal generation, inference, refinement).
Figure 6: Qualitative results for the detection performance of the proposed annotator-free method. The predicted detection results are on the left column, and the ground truth results are on the right.
Figure 7: Open-world detectors for novel object categories under domain shift. The detection performance for GroundingDINO (left column) are compared with the proposed novel object detection (right column).
Figure 8: Sample images from the simulated dataset.
Method
Backbone
Novel
Known
Overall
GLIP-L [ 13 ]
Swin-L
35.1
12.6
24.9
T-Rex2 [ 10 ]
Swin-L
31.9
21.2
27.1
GDINO-L [ 15 ]
Swin-L
38.4
26.5
33.0
Ours
YOLO
43.7
32.8
38.8
Table 4: Comparison for novel object detection under domain shift with other state-of-the-art methods. Performance is evaluated on novel, known, and overall classes.
Figure 9: Noise reduction steps for SAM proposals.
Open-vocabulary object detection aims to recognize objects from an open set of categories, which leverages vision-language models (VLMs) pre-trained on large-scale image-text data. The cooperative paradigm combines an object detector with a VLM to achieve zero-shot recognition of novel objects. However, VLMs pre-trained on full images often struggle to capture local object details, limiting their effectiveness when applied to region-level detection. We present Decoupled Adaptivity Training (DAT), a self-supervised fine-tuning approach to improve VLMs for cooperative model-based object detection. Given a cooperative model consists of a closed-set detector and a VLM, we first construct a region-aware pseudo-labeled dataset using a pre-trained closed-set object detector, in which regions corresponding to novel objects may be present but remain unlabeled or mislabeled. We then fine-tune the visual backbone of the VLM in a decoupled manner, which enhances local feature alignment while preserving global semantic knowledge via weight interpolation. DAT is a plug-and-play module that requires no inference overhead and fine-tunes less than 0.8M parameters. Experiments on the COCO and LVIS datasets show that DAT consistently improves detection performance on both novel and known categories, establishing a new state of the art in cooperative open-vocabulary detection.
Yazhe Wan, Changjae Oh
Queen Mary University of London · Queen Mary University of London, London, UK
Conventional object detectors typically operate under a closed-set assumption, limiting recognition to a predefined set of base classes seen during training. Open-vocabulary object detection (OVD) addresses this limitation by leveraging vision-language models (VLMs) to generate pseudo labels for novel object classes. However, existing OVD methods suffer from two critical drawbacks: (1) inaccurate class label assignments, as VLMs are optimized for image-level predictions rather than the region-level predictions required for pseudo labeling, and (2) unreliable objectness scores from region proposal networks (RPNs) trained exclusively on base object classes. To address these issues, we propose a novel pseudo labeling framework for OVD. Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation by assessing consistency across hierarchical semantic levels (class, super- and sub-category). We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness token to mitigate base class bias problem of RPNs and provide reliable objectness estimations for novel object classes. Extensive experiments on standard OVD benchmarks, including COCO and LVIS, demonstrate that our approach clearly sets a new state of the art, validating the effectiveness of our approach. Project site: https://cvlab.yonsei.ac.kr/projects/HCC
Sanghoon Lee, Geon Lee, Hyekang Park +1
Yonsei University · Korea Institute of Science and Technology (KIST)
Object detectors are typically trained under closed-set supervision, where unlabeled regions are implicitly treated as background. Under incomplete annotations, this assumption introduces objectness bias: visually valid but unlabeled objects are used as negatives, tying objectness to the annotated taxonomy rather than generic object structure. This limitation is particularly problematic for class-agnostic and open-world detection. This paper proposes Background-Free Objectness Learning (B-FOR), a dense class-agnostic detection framework that learns objectness without explicit background supervision on unlabeled regions. B-FOR formulates detection as the prediction of dense multi-scale object-center and scale fields, from which object hypotheses emerge as local spatial structures. Supervision is confined to reliable annotated regions through spatially structured soft targets, avoiding foreground-background discrimination. To support decoding from emergent local maxima, the paper further introduces displacement-aware scale fields that model object extent as a spatially varying property of the learned objectness field. Experiments on PASCAL VOC, MS-COCO, and Open Images demonstrate strong generalization to unseen categories and cross-dataset object distributions. B-FOR improves recall by more than +10 AR points over prior class-agnostic baselines. Ablation studies show that both localized objectness supervision and displacement-aware scale fields are critical for class-agnostic localization under incomplete annotations. Code available at: https://github.com/Daniaawan/B-FOR.
Dania Batool, Liliana Lo Presti, Marco La Cascia +1
Department of Engineering University of Palermo Palermo, Italy · ICAR National Research Council of Italy Palermo, Italy