Object detection models rely on large-scale annotated datasets that require substantial manual annotation effort. The labeling costs further increase when annotation uncertainty is captured through soft labels, as multiple annotations must be collected for each object. We address this challenge by introducing a semi-supervised framework for the uncertainty-aware annotation of object detection datasets. Our approach leverages open-vocabulary object detectors to generate region proposals and semantically meaningful object representations. Based on these representations, we construct an object-level similarity graph and propagate probabilistic label information. Our framework assigns soft labels to previously unlabeled objects, quantifying both aleatoric and epistemic uncertainty. We evaluate our method on four object detection benchmarks with soft-label annotations and demonstrate that high-quality annotations can be obtained with only a small fraction of manually labeled objects. Furthermore, we show that detectors trained on semi-automatically annotated data achieve performance comparable to models trained on fully manually annotated data. We achieve state-of-the-art downstream detection performance for the task of semi-supervised object detection on the COCO dataset, with 40.5% AP virtually matching the performance of the model trained on fully manually constructed data (40.9%) at only 1% of labeling costs. These results highlight the potential of open-vocabulary models and graph-based label propagation for efficient, accurate and uncertainty-aware pseudo-labeling.
Figures & tables
Fig. 1 : Illustration of label propagation for object detection datasets. Initial region proposals are generated with Grounding DINO [ 6 ] and form a similarity graph (grey lines) over which sparse label information (solid bounding boxes) is propagated. This results in pseudo-labels with soft label estimates based on which false positives are filtered effectively.
Fig. 2 : Method overview: Initial region proposals form a graph based on feature similarity and individual objects are queried for manual inspection. The queried object is assigned either one of the dataset’s classes or the background class. This label information is then propagated over the similarity graph resulting in soft label estimates for each object proposal. The estimate is iteratively refined with each object that is inspected.
Fig. 3 : Average Similarity (AS 50:95 ) [ 2 ] as a function of annotation budget for our propagation method (PLS) on the COCO dataset against Gaussian Kernel Regression, k -NN regression, calibrated Grounding DINO confidence, and Hard-Label Injection. The dotted line marks the AS ceiling where all correctly localized detections perfectly align with the corresponding soft labels.
Fig. 4 : Benchmark results for semi-supervised object detection [ 13 ] . Average precision of a Faster R-CNN model with ResNet-50 backbone is reported on COCO validation data. The model is trained on pseudo-labels which are constructed using the label information available for a fraction of images. At 100% budget, additional unlabeled COCO images are used.
Full
Grounding DINO
Ours
Dataset
Model
100%
1%
10%
1%
10%
COCO
Faster R-CNN
38.6
38.6
38.6
38.7
38.9
YOLOv8-L
47.8
46.8
46.9
46.9
47.9
RT-DETR
47.7
46.7
46.4
46.7
47.0
VOC
Faster R-CNN
44.0
44.3
45.0
44.8
44.8
YOLOv8-L
50.5
51.5
52.0
52.8
53.1
TABLE I : Downstream detection performance (AP in %) for Faster R-CNN, YOLOv8-L, and RT-DETR trained with different annotation sets. We compare fully supervised training with Grounding DINO pseudo-labels and our pseudo-labeling approach, distinguishing two annotation budgets (1% / 10% of labeled images provided). Best (bold) and second best (underlined) values per row are highlighted.
TABLE II : Ablation Results: Adding additional models to Grounding DINO as an ensemble does not improve recall of COCO pseudo-labels but introduces false positives (left). Embedding detected objects with CLIP [ 54 ] for graph construction leads to overall worse results (right).
Fig. 5 : Active querying results: We compare performance in terms of AS (left) and AP (right) on COCO for four querying strategies that choose different nodes to annotate next.
COCO
Cityscapes
KITTI
Pascal VOC
Method
AP
F1
AP
F1
AP
F1
AP
F1
Random
3.50
7.30
1.69
3.36
3.50
6.76
0.65
1.30
GDino
34.51
40.89
19.36
26.99
58.80
57.05
35.91
41.29
Bayesian PLS
34.56
41.16
21.03
26.66
37.62
41.96
33.98
37.88
TABLE III : Comparison of label error detection capabilities. We consider originally false positive predictions as label error proposals and evaluate them on actual label errors using the evaluation protocol of [ 2 ] . We report AP and the highest F1 -score of each method.
Method
1%
5%
10%
100%
Supervised Baseline
10.00 ± 0.26
20.92 ± 0.15
26.94 ± 0.11
40.90
CSD [ 31 ]
10.51
18.63
22.46
38.82
STAC [ 13 ]
13.97 ± 0.35
24.38 ± 0.12
28.64 ± 0.21
39.21
Instant Teaching [ 55 ]
18.05
26.75
30.40
-
Humble Teacher [ 33 ]
16.96
27.70
31.61
-
Unbiased Teacher [ 32 ]
20.75
28.27
31.50
41.30
TABLE A.1 : Comparison of different semi-supervised object detection methods on COCO. The results are reported for different fractions of annotated training data in terms of mean average precision (AP). For annotation budgets below 100%, results are averaged over 5 folds of annotated data and standard deviation is reported, following the evaluation protocol of [ 13 ] . We list the numbers for existing methods as reported by [ 14 ] and [ 36 ] where standard deviations are not always given.
Fig. A.1 : Training curves for Faster R-CNN with ResNet-50 backbone on COCO for the supervised baseline with 10% of labeled images and our pseudo-labels. We observe a fast increase in validation AP that ultimately approaches the performance of the model trained on the entire set of training data and surpasses other pseudo-labeling approaches (dashed lines).
Full
Grounding DINO
Ours
Dataset
Model
100%
1%
10%
1%
10%
COCO
Faster R-CNN
33.9
34.1
34.1
34.1
34.4
YOLOv8-L
44.5
42.9
43.1
42.9
43.6
RT-DETR
42.7
42.0
41.7
42.1
42.2
VOC
Faster R-CNN
37.8
38.2
38.8
39.9
39.0
YOLOv8-L
45.4
46.0
46.5
48.2
46.8
TABLE A.2 : Downstream detection performance (AP in %) for Faster R-CNN, YOLOv8-L, and RT-DETR, on independently constructed annotations of each dataset’s validation set, which were found to cover more object annotations [ 2 ] .
Fig. A.2 : Performance in terms of Average Similarity for parametrized methods depending on hyperparameter values. Hyperparameters were evaluated over a grid for each annotation budget. For readability, we display the three annotation budgets 1% , 10% , and 100% . For PLS, since there are two hyperparameters, we indicate dependence on the parameter k by shading the minimal and maximal obtained value for each spreading intensity α .
Fig. A.3 : Average Precision (AP 50:95 ) [ 9 ] on COCO 2017 validation data with soft labels [ 2 ] depending on annotation budget for our propagation method (PLS) against Gaussian Kernel Regression, k -NN regression, calibrated Grounding DINO confidence, and Hard-Label Injection. We also include a Bayesian approach of our method based on the initial Grounding DINO score. Here, the goal is essentially to resort predictions based on obtained labels such that AP increases.
Fig. A.4 : Similarity-Recall Curve (a) and Precision-Recall Curve (b) on COCO for an IoU threshold of 0.5. To determine similarity in terms of 1-TVD and precision, estimated and true soft labels are compared, where predictions are sorted by confidence as usual in COCO-style evaluation. The displayed curves are the average of all 80 class-specific curves.
Fig. A.5 : Average Similarity (a) and Average Precision (b) depending on the IoU threshold used for GT matching of predictions. A common pattern is that improvements over the Grounding DINO baseline through label propagation are most prominent for lower IoU thresholds.
Fig. A.6 : Performance in terms of Average Similarity depending on the provided annotation budget for three datasets. Hyperparameters were swept through a grid search and optimal values for each annotation budget are displayed.
Fig. A.7 : Performance in terms of Average Precision depending on the provided annotation budget for three datasets. Hyperparameters were swept through a grid search and optimal values for each annotation budget are displayed.
TABLE A.3 : Overview of performance in terms of AS and AP (%) across datasets, methods, and three annotation budgets. Best values per budget is highlighted in bold, second best is underlined.
Fig. A.8 : Hyperparameter analysis of our method for both spreading intensity α and the number of neighbors k . We evaluate Average Similarity on COCO 2017 validation data with soft labels [ 2 ] for each parameter combination.
Fig. A.9 : Hyperparameter analysis of our method for the Grounding DINO score threshold applied as well as the spreading intensity α for a fixed number of neighbors k=20 . We evaluate Average Similarity on COCO 2017 validation data with soft labels [ 2 ] for each parameter combination.
Fig. A.10 : Coverage of labeled nodes during active querying in terms of two metrics where higher indicates better spread of labels across the feature space. In (a), the spread of labeled nodes is captured by the average distance between labeled nodes as a moving average with a window size of 100. Similarly, the redundancy of labeled nodes is captured in (b) by the average distance to any labeled node for all nodes in the graph. We see that querying by epistemic uncertainty leads to better spread of labels whereas this is not the case for aleatoric uncertainty which sits at the random baseline level.
Fig. A.11 : First 200 Queried nodes in two-dimensional UMAP embedding space of Grounding DINO features of COCO validation data for different querying strategies .
Median
Cost at budget (h)
Fully manual
Dataset
answer time (s)
1%
10%
100%
annotation (h)
COCO
1.14
0.3
2.7
27.2
2,005.4
Cityscapes
0.82
0.2
1.8
17.7
992.2
KITTI
1.00
0.3
2.8
27.9
1,306.9
Pascal VOC
0.87
0.1
1.4
14.1
1,016.9
TABLE A.4 : Estimated annotation costs for our approach depending on the dataset and annotation budget. The estimate is based on each dataset’s median answer time per object and is compared to the annotation costs of fully manual annotation with soft labels as done by [ 2 ] .
Fig. A.12 : Examples where our method and Grounding DINO agree on the predicted class.
Fig. A.13 : Examples where our method and Grounding DINO disagree on the predicted class.
Fig. A.14 : Cases where the soft ground truth itself is not fully confident and neither are our method and Grounding DINO.
Fig. A.15 : Examples on Pascal VOC, Cityscapes and KITTI where our method and Grounding DINO agree on the predicted class.
Fig. A.16 : Examples on Pascal VOC, Cityscapes and KITTI where our method and Grounding DINO do not agree on the predicted class.
Fig. A.17 : Uncertain ground truth examples for Pascal VOC, Cityscapes and KITTI.
Reliable object perception is necessary for general-purpose service robots. Open-vocabulary detectors struggle to generalize beyond a few classes and fully supervised training of object detectors requires time-intensive annotations. We present a semi-supervised label propagation approach for household object segmentation. A segment proposer generates class-agnostic masks, and an ensemble of Hopfield networks assigns labels by learning representative embeddings in complementary foundation model embedding spaces (CLIP, ViT, Theia). Our approach scales to 50 object classes with limited annotation overhead and can automatically label 60% of the data in a RoboCup@Home setting, where preparation time is severely constrained. Dataset and code are publicly available at https://github.com/ais-bonn/label_propagation.
Conventional object detectors typically operate under a closed-set assumption, limiting recognition to a predefined set of base classes seen during training. Open-vocabulary object detection (OVD) addresses this limitation by leveraging vision-language models (VLMs) to generate pseudo labels for novel object classes. However, existing OVD methods suffer from two critical drawbacks: (1) inaccurate class label assignments, as VLMs are optimized for image-level predictions rather than the region-level predictions required for pseudo labeling, and (2) unreliable objectness scores from region proposal networks (RPNs) trained exclusively on base object classes. To address these issues, we propose a novel pseudo labeling framework for OVD. Our approach introduces a hierarchical confidence calibration (HCC) technique, which ensures reliable class label estimation by assessing consistency across hierarchical semantic levels (class, super- and sub-category). We also present LoCLIP, a parameter-efficient adaptation of CLIP that incorporates an objectness token to mitigate base class bias problem of RPNs and provide reliable objectness estimations for novel object classes. Extensive experiments on standard OVD benchmarks, including COCO and LVIS, demonstrate that our approach clearly sets a new state of the art, validating the effectiveness of our approach. Project site: https://cvlab.yonsei.ac.kr/projects/HCC
Sanghoon Lee, Geon Lee, Hyekang Park +1
Yonsei University · Korea Institute of Science and Technology (KIST)
Open-vocabulary object detection (OVD) aims to recognize and localize object categories beyond the training set. Recent approaches leverage vision-language models to generate pseudo-labels using image-text alignment, allowing detectors to generalize to unseen classes without explicit supervision. However, these methods depend heavily on single-step image-text matching, neglecting the intermediate reasoning steps crucial for interpreting semantically complex visual contexts, such as crowding or occlusion. In this paper, we introduce MSPL, a framework that incorporates multi-step visual reasoning into the pseudo-labeling process for OVD. It decomposes complex scene understanding into three interpretable steps-object localization, category recognition, and background grounding-where these intermediate reasoning states serve as rich supervision sources. Extensive experiments on standard OVD evaluation protocols demonstrate that MSPL achieves state-of-the-art performance with superior pseudo-labeling efficiency, outperforming the strong baseline by 9.4 AP50 for novel classes on OV-COCO and improving box and mask APr by 3.2 and 2.2, respectively, on OV-LVIS. Code and models are available at https://github.com/hchoi256/mspl.