cs.CVOct 21, 2026
SaveOpen-Vocabulary Models and Label Propagation for Uncertainty-aware Pseudo-Labeling of Object Detection Datasets
Organizations: Institute of Computer Science, Osnabrück University · Department of Mathematics, University of Wuppertal
Abstract
Object detection models rely on large-scale annotated datasets that require substantial manual annotation effort. The labeling costs further increase when annotation uncertainty is captured through soft labels, as multiple annotations must be collected for each object. We address this challenge by introducing a semi-supervised framework for the uncertainty-aware annotation of object detection datasets. Our approach leverages open-vocabulary object detectors to generate region proposals and semantically meaningful object representations. Based on these representations, we construct an object-level similarity graph and propagate probabilistic label information. Our framework assigns soft labels to previously unlabeled objects, quantifying both aleatoric and epistemic uncertainty. We evaluate our method on four object detection benchmarks with soft-label annotations and demonstrate that high-quality annotations can be obtained with only a small fraction of manually labeled objects. Furthermore, we show that detectors trained on semi-automatically annotated data achieve performance comparable to models trained on fully manually annotated data. We achieve state-of-the-art downstream detection performance for the task of semi-supervised object detection on the COCO dataset, with 40.5% AP virtually matching the performance of the model trained on fully manually constructed data (40.9%) at only 1% of labeling costs. These results highlight the potential of open-vocabulary models and graph-based label propagation for efficient, accurate and uncertainty-aware pseudo-labeling.
Figures & tables
Fig. 1 : Illustration of label propagation for object detection datasets. Initial region proposals are generated with Grounding DINO [ 6 ] and form a similarity graph (grey lines) over which sparse label information (solid bounding boxes) is propagated. This results in pseudo-labels with soft label estimates based on which false positives are filtered effectively.
Fig. 2 : Method overview: Initial region proposals form a graph based on feature similarity and individual objects are queried for manual inspection. The queried object is assigned either one of the dataset’s classes or the background class. This label information is then propagated over the similarity graph resulting in soft label estimates for each object proposal. The estimate is iteratively refined with each object that is inspected.
Fig. 3 : Average Similarity (AS 50:95 ) [ 2 ] as a function of annotation budget for our propagation method (PLS) on the COCO dataset against Gaussian Kernel Regression, -NN regression, calibrated Grounding DINO confidence, and Hard-Label Injection. The dotted line marks the AS ceiling where all correctly localized detections perfectly align with the corresponding soft labels.
Fig. 4 : Benchmark results for semi-supervised object detection [ 13 ] . Average precision of a Faster R-CNN model with ResNet-50 backbone is reported on COCO validation data. The model is trained on pseudo-labels which are constructed using the label information available for a fraction of images. At 100% budget, additional unlabeled COCO images are used.
| Full | Grounding DINO | Ours | ||||
| Dataset | Model | 100% | 1% | 10% | 1% | 10% |
| COCO | Faster R-CNN | 38.6 | 38.6 | 38.6 | 38.7 | 38.9 |
| YOLOv8-L | 47.8 | 46.8 | 46.9 | 46.9 | 47.9 | |
| RT-DETR | 47.7 | 46.7 | 46.4 | 46.7 | 47.0 | |
| VOC | Faster R-CNN | 44.0 | 44.3 | 45.0 | 44.8 | 44.8 |
| YOLOv8-L | 50.5 | 51.5 | 52.0 | 52.8 | 53.1 | |
TABLE I : Downstream detection performance (AP in %) for Faster R-CNN, YOLOv8-L, and RT-DETR trained with different annotation sets. We compare fully supervised training with Grounding DINO pseudo-labels and our pseudo-labeling approach, distinguishing two annotation budgets (1% / 10% of labeled images provided). Best (bold) and second best (underlined) values per row are highlighted.
TABLE II : Ablation Results: Adding additional models to Grounding DINO as an ensemble does not improve recall of COCO pseudo-labels but introduces false positives (left). Embedding detected objects with CLIP [ 54 ] for graph construction leads to overall worse results (right).
Fig. 5 : Active querying results: We compare performance in terms of AS (left) and AP (right) on COCO for four querying strategies that choose different nodes to annotate next.
| COCO | Cityscapes | KITTI | Pascal VOC | |||||
| Method | AP | F1 | AP | F1 | AP | F1 | AP | F1 |
| Random | 3.50 | 7.30 | 1.69 | 3.36 | 3.50 | 6.76 | 0.65 | 1.30 |
| GDino | 34.51 | 40.89 | 19.36 | 26.99 | 58.80 | 57.05 | 35.91 | 41.29 |
| Bayesian PLS | 34.56 | 41.16 | 21.03 | 26.66 | 37.62 | 41.96 | 33.98 | 37.88 |
TABLE III : Comparison of label error detection capabilities. We consider originally false positive predictions as label error proposals and evaluate them on actual label errors using the evaluation protocol of [ 2 ] . We report AP and the highest -score of each method.
| Method | 1% | 5% | 10% | 100% |
| Supervised Baseline | 10.00 0.26 | 20.92 0.15 | 26.94 0.11 | 40.90 |
| CSD [ 31 ] | 10.51 | 18.63 | 22.46 | 38.82 |
| STAC [ 13 ] | 13.97 0.35 | 24.38 0.12 | 28.64 0.21 | 39.21 |
| Instant Teaching [ 55 ] | 18.05 | 26.75 | 30.40 | - |
| Humble Teacher [ 33 ] | 16.96 | 27.70 | 31.61 | - |
| Unbiased Teacher [ 32 ] | 20.75 | 28.27 | 31.50 | 41.30 |
TABLE A.1 : Comparison of different semi-supervised object detection methods on COCO. The results are reported for different fractions of annotated training data in terms of mean average precision (AP). For annotation budgets below 100%, results are averaged over 5 folds of annotated data and standard deviation is reported, following the evaluation protocol of [ 13 ] . We list the numbers for existing methods as reported by [ 14 ] and [ 36 ] where standard deviations are not always given.
Fig. A.1 : Training curves for Faster R-CNN with ResNet-50 backbone on COCO for the supervised baseline with of labeled images and our pseudo-labels. We observe a fast increase in validation AP that ultimately approaches the performance of the model trained on the entire set of training data and surpasses other pseudo-labeling approaches (dashed lines).
| Full | Grounding DINO | Ours | ||||
| Dataset | Model | 100% | 1% | 10% | 1% | 10% |
| COCO | Faster R-CNN | 33.9 | 34.1 | 34.1 | 34.1 | 34.4 |
| YOLOv8-L | 44.5 | 42.9 | 43.1 | 42.9 | 43.6 | |
| RT-DETR | 42.7 | 42.0 | 41.7 | 42.1 | 42.2 | |
| VOC | Faster R-CNN | 37.8 | 38.2 | 38.8 | 39.9 | 39.0 |
| YOLOv8-L | 45.4 | 46.0 | 46.5 | 48.2 | 46.8 | |
TABLE A.2 : Downstream detection performance (AP in %) for Faster R-CNN, YOLOv8-L, and RT-DETR, on independently constructed annotations of each dataset’s validation set, which were found to cover more object annotations [ 2 ] .
Fig. A.2 : Performance in terms of Average Similarity for parametrized methods depending on hyperparameter values. Hyperparameters were evaluated over a grid for each annotation budget. For readability, we display the three annotation budgets , , and . For PLS, since there are two hyperparameters, we indicate dependence on the parameter by shading the minimal and maximal obtained value for each spreading intensity .
Fig. A.3 : Average Precision (AP 50:95 ) [ 9 ] on COCO 2017 validation data with soft labels [ 2 ] depending on annotation budget for our propagation method (PLS) against Gaussian Kernel Regression, -NN regression, calibrated Grounding DINO confidence, and Hard-Label Injection. We also include a Bayesian approach of our method based on the initial Grounding DINO score. Here, the goal is essentially to resort predictions based on obtained labels such that AP increases.
Fig. A.4 : Similarity-Recall Curve (a) and Precision-Recall Curve (b) on COCO for an IoU threshold of 0.5. To determine similarity in terms of 1-TVD and precision, estimated and true soft labels are compared, where predictions are sorted by confidence as usual in COCO-style evaluation. The displayed curves are the average of all 80 class-specific curves.
Fig. A.5 : Average Similarity (a) and Average Precision (b) depending on the IoU threshold used for GT matching of predictions. A common pattern is that improvements over the Grounding DINO baseline through label propagation are most prominent for lower IoU thresholds.
Fig. A.6 : Performance in terms of Average Similarity depending on the provided annotation budget for three datasets. Hyperparameters were swept through a grid search and optimal values for each annotation budget are displayed.
Fig. A.7 : Performance in terms of Average Precision depending on the provided annotation budget for three datasets. Hyperparameters were swept through a grid search and optimal values for each annotation budget are displayed.
TABLE A.3 : Overview of performance in terms of AS and AP (%) across datasets, methods, and three annotation budgets. Best values per budget is highlighted in bold, second best is underlined.
Fig. A.8 : Hyperparameter analysis of our method for both spreading intensity and the number of neighbors . We evaluate Average Similarity on COCO 2017 validation data with soft labels [ 2 ] for each parameter combination.
Fig. A.9 : Hyperparameter analysis of our method for the Grounding DINO score threshold applied as well as the spreading intensity for a fixed number of neighbors . We evaluate Average Similarity on COCO 2017 validation data with soft labels [ 2 ] for each parameter combination.
Fig. A.10 : Coverage of labeled nodes during active querying in terms of two metrics where higher indicates better spread of labels across the feature space. In (a), the spread of labeled nodes is captured by the average distance between labeled nodes as a moving average with a window size of 100. Similarly, the redundancy of labeled nodes is captured in (b) by the average distance to any labeled node for all nodes in the graph. We see that querying by epistemic uncertainty leads to better spread of labels whereas this is not the case for aleatoric uncertainty which sits at the random baseline level.
Fig. A.11 : First 200 Queried nodes in two-dimensional UMAP embedding space of Grounding DINO features of COCO validation data for different querying strategies .
| Median | Cost at budget (h) | Fully manual | |||
| Dataset | answer time (s) | 1% | 10% | 100% | annotation (h) |
| COCO | 1.14 | 0.3 | 2.7 | 27.2 | 2,005.4 |
| Cityscapes | 0.82 | 0.2 | 1.8 | 17.7 | 992.2 |
| KITTI | 1.00 | 0.3 | 2.8 | 27.9 | 1,306.9 |
| Pascal VOC | 0.87 | 0.1 | 1.4 | 14.1 | 1,016.9 |
TABLE A.4 : Estimated annotation costs for our approach depending on the dataset and annotation budget. The estimate is based on each dataset’s median answer time per object and is compared to the annotation costs of fully manual annotation with soft labels as done by [ 2 ] .
Fig. A.12 : Examples where our method and Grounding DINO agree on the predicted class.
Fig. A.13 : Examples where our method and Grounding DINO disagree on the predicted class.
Fig. A.14 : Cases where the soft ground truth itself is not fully confident and neither are our method and Grounding DINO.
Fig. A.15 : Examples on Pascal VOC, Cityscapes and KITTI where our method and Grounding DINO agree on the predicted class.
Fig. A.16 : Examples on Pascal VOC, Cityscapes and KITTI where our method and Grounding DINO do not agree on the predicted class.
Fig. A.17 : Uncertain ground truth examples for Pascal VOC, Cityscapes and KITTI.
More in this topic
Pseudo-Labeling
Pseudo-labeling uses a model’s predictions as training targets for unlabeled data, reducing dependence on costly human annotations.
See all 157 papers