Object detection research remains constrained by high annotation costs and the closed-set assumptions of supervised models, limiting scalability to novel categories. We present a target-dataset annotation-free object detection framework that eliminates the need for human labeling on the target dataset by leveraging the complementary strengths of pretrained foundation models, including segmentation and vision-language systems, combined with a self-refining training loop. To further reduce reliance on expensive LVLM queries, we introduce a landmark-based clustering mechanism that amortizes labeling across image clusters, improving label consistency while lowering computational cost. Our method autonomously generates segmentation proposals, assigns semantic labels through multimodal reasoning, and iteratively improves both localization and classification using a lightweight open-set detector. This modular pipeline enables adaptation to completely unseen object classes and domains without any human-provided target-dataset annotations, while maintaining compatibility with existing detection architectures. Evaluations on COCO, LVIS, ODinW-35, and Roboflow-100 demonstrate substantial performance gains over state-of-the-art open-vocabulary and long-tail detection methods, achieving 60.8 AP on COCO and 47.4 AP on LVIS under zero-shot settings. Beyond strong benchmark results, our approach provides a scalable foundation for real-world detection systems in unknown environments, where annotation is infeasible.
Figures & tables
Figure 2: Traditional object detection requires annotated paired data and remains closed-set. Our annotator-free pipeline relies only on image scenes, generating proposals via foundational models and refining them iteratively to expand to novel categories.
Figure 3: Proposed annotator-free pipeline. Stage 1: LVLM and SAM generate scene-level labels and object proposals. Stage 2: CLIP encoders assign semantic labels to proposals. Stage 3: an open-set detector trained on these proposals generates refined prompts for SAM, which are re-labeled by the LVLM.
Figure 4: Word clouds of COCO ground-truth instances (top) and our pseudo-labeled annotations (bottom). The proposed annotation scheme preserves relative class frequencies (e.g., person, car, table) while introducing additional categories (e.g., fence, building, window), thereby enriching label diversity. This demonstrates that our method not only captures the underlying distribution but also scales to novel concepts beyond the original dataset.
Model
Backbone
COCO val
LVIS
GLIP-L [ 13 ]
Swin-L
49.8
26.9
DINOv [ 34 ]
Swin-L
46.2
-
GDINO-L [ 15 ]
Swin-L
48.4
27.4
T-Rex2 (Text) [ 10 ]
Swin-L
52.2
45.8
T-Rex2 (Visual) [ 10 ]
Swin-L
46.5
45.3
DetCLIPv3 [ 31 ]
Swin-T
47.2
47.0
Table 1: Annotator-free object detection performance comparison on COCO val and LVIS minival. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.
Model
Backbone
ODinW
Roboflow
GLIP-L [ 13 ]
Swin-L
23.4
8.6
DINOv [ 34 ]
Swin-L
15.7
-
GDINO-L [ 15 ]
Swin-L
22.3
8.3
T-Rex2 (Text) [ 10 ]
Swin-L
22.0
10.5
T-Rex2 (Visual) [ 10 ]
Swin-L
27.8
18.5
Ours
Swin-T
30.1
20.8
Table 2: Annotator-free object detection performance comparison on ODinW-35 val and Roboflow-100 val. We consider the zero-shot evaluation of state-of-the-art long tail object detectors and open vocabulary object detectors for comparison.
Method
Backbone
Novel
Known
All
OVR-CNN [ 33 ]
RN50
22.8
46.0
39.9
ViLD [ 8 ]
ViT-B/32
27.6
59.5
51.3
Detic [ 40 ]
RN50
27.8
47.1
45.0
OV-DETR [ 32 ]
ViT-B/32
29.4
61.0
52.7
BARON [ 29 ]
RN50
34.0
60.4
53.5
CORA [ 30 ]
RN50x4
41.7
44.5
43.8
Table 3: Comparison for novel object detection with other state-of-the-art methods. Performance is evaluated on a novel, known, and all classes on COCO OVD split following a generalized open-vocabulary detection setting.
Figure 5: Ablation of performance improvement across pipeline stages (proposal generation, inference, refinement).
Figure 6: Qualitative results for the detection performance of the proposed annotator-free method. The predicted detection results are on the left column, and the ground truth results are on the right.
Figure 7: Open-world detectors for novel object categories under domain shift. The detection performance for GroundingDINO (left column) are compared with the proposed novel object detection (right column).
Figure 8: Sample images from the simulated dataset.
Method
Backbone
Novel
Known
Overall
GLIP-L [ 13 ]
Swin-L
35.1
12.6
24.9
T-Rex2 [ 10 ]
Swin-L
31.9
21.2
27.1
GDINO-L [ 15 ]
Swin-L
38.4
26.5
33.0
Ours
YOLO
43.7
32.8
38.8
Table 4: Comparison for novel object detection under domain shift with other state-of-the-art methods. Performance is evaluated on novel, known, and overall classes.
Figure 9: Noise reduction steps for SAM proposals.