Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty
Authors: Sarina Penquitt, Jonathan Klees, Antonia van Betteray, Parssa Jashnieh, Peter Stehr, Matthias Rottmann, Lars Schmarje
Organizations: Institute of Computer Science Osnabrück University Osnabrück, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Wayve London, United Kingdom
While object detection has advanced through improved architectures and open-vocabulary models, we provide strong evidence that benchmark quality is limited by annotation incompleteness. Across four widely used datasets (COCO, Pascal VOC, Cityscapes, KITTI), re-annotation reveals substantial increases in annotated objects (e.g., up to +60% on KITTI and +40% on COCO), driven primarily by previously unlabeled small, occluded, or densely packed instances. While some differences arise from dataset-specific annotation conventions, we consistently find that missing annotations are the main source of label errors across all datasets. To achieve high data quality, we introduce a scalable annotation pipeline that emphasizes high recall and captures ambiguity through soft labels aggregated from at least 11 annotators per object. The resulting annotations improve coverage and align well with human calibration. We show that benchmark performance is highly sensitive to annotation quality, although model rankings remain largely stable. We introduce two large-scale benchmarks: (i) an uncertainty-aware object detection benchmark, and (ii) a label error detection benchmark grounded in real label errors. We show that current detectors are strongly depended on annotation quality and are misaligned with human perception. Current label error detection methods, which have been shown to perform well on synthetic noise, struggle to achieve high recall and precision on real label errors. Our results highlight the need for future object detection benchmarks to move beyond deterministic annotations toward high-recall, uncertainty-aware evaluation that maximizes valid instances and better reflects real-world ambiguity.
Figures & tables
TABLE 1: Summary of related annotation-focused datasets. In Table I(a) , Detection (Det.) refers to object detection datasets, Re-annotation (Re-ann.) denotes efforts to improve annotation completeness through systematic relabeling, Uncertainty (Uncert.) denotes annotation uncertainty captured via multiple annotators and soft labels, and Multi-dataset (Multi-DS) indicates application of a consistent methodology across multiple datasets. Prior work typically addresses either annotation completeness or uncertainty, while our work combines both across multiple object detection benchmarks. Table I(b) summarizes the number of images and classes that were re-annotated in this paper for each dataset.
Fig. 1 : Overview of our annotation pipeline. In stage 1, all potential objects of a given class are marked with keypoints. For each keypoint, a bounding box is then drawn and verified in stage 2. In the third stage, 11 annotators evaluate the proposed object, which results in a soft label. Ambiguous cases receive additional annotations and finally, post-processing removes duplicates and corrects bias. The resulting annotations are our validated ground truth.
Table 3
Less Restrictive
Stricter
Dataset
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Cityscapes
5,249 (8.4%)
2,385 (3.8%)
85 (0.1%)
2,229 (3.6%)
1,318 (2.1%)
56 (0.1%)
COCO
16,219 (44.1%)
7,736 (21.0%)
254 (0.7%)
2,685 (7.3%)
3,497 (9.5%)
98 (0.3%)
KITTI
22,710 (57.4%)
7,167 (18.1%)
1,026 (2.6%)
961 (2.4%)
2,174 (5.5%)
479 (1.2%)
Pascal VOC
3,654 (11.6%)
2,210 (7.0%)
52 (0.2%)
761 (2.4%)
1,290 (4.1%)
25 (0.1%)
TABLE 4: Comparison of identified label errors across datasets for two default label error configurations: a less restrictive setting ( p≥0.5 considering all objects), and a stricter setting ( p≥0.8 considering objects with height ≥40 pixels). Percentages are given relative to the original ground truth. Counts are aggregated over training and validation data, except for COCO (validation only).
Figure 5Table 6
TABLE 7: Performance of object detectors trained on original GT and evaluated on original vs. validated GT. Underlined values indicate improvements >10 points, bold decreases >10 points.
Figure 8
Fig. 5 : Examples of missing label errors across all datasets. Green boxes denote original annotations, red boxes highlight label errors with soft label probability p found based on our validated GT.
Table 10
Appendix figures & tables71 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Question
Description
Repeats
WP
Time [s]/WP
Keypoint
Mark all instances of class that are not already marked with a keypoint. Do not double annotate objects.
If you are not sure, just annotate it. We will check this later.
3
10
1000
Bounding Box
Draw a tight bounding box around the marked object.
Please draw the bounding box as tight as possible. You can also correct it.
2
20
1800
Soft Label
Is the object in the box a class ? If so, click class . If not, select the correct label.
If you are not sure, press “can’t solve”.
11
30
600
Appendix
TABLE 10: Overview of the microtask parameters used during the annotation process. Each task consists of a question and an additional description to guide annotators. The parameter repeats defines how many independent annotators complete the same task, while the working package (WP) specifies how many items (images, keypoints, or bounding boxes depending on the task) are processed per assignment. The time limit indicates the maximum duration available per working package. The placeholder class is dynamically replaced during task execution by the respective dataset class currently under review.
Figure 12
Fig. 8 : Annotator interface for soft label task on a COCO image.
Dataset
Keypoint Time [s]
Bounding Box Time [s]
Per Answer Time [s]
11 Votes [s]
22 Votes [s]
33 Votes [s]
44 Votes [s]
Cityscapes
2.74
11.59
1.73
17.90
51.43
72.06
110.95
Pascal VOC
1.29
6.29
1.84
16.50
49.18
70.07
101.14
KITTI
3.47
7.34
1.99
18.46
46.76
72.11
100.74
COCO
2.50
6.94
2.28
21.11
53.58
79.19
114.04
Appendix
TABLE 11: Average annotation times for the individual annotation microtasks. Keypoint time denotes the average time for a single image-class review by one annotator, and bounding box time denotes the average annotation/review time for a single keypoint by one annotator. Per-answer time denotes the average time for an individual soft label response, while the times for 11, 22, 33, and 44 votes denote the average cumulative annotation time required to obtain the corresponding soft label. All times are reported in seconds.
Fig. 9 : Interface for duplicate removal. Each pair of overlapping bounding boxes is displayed together with the information of the most voted class. The user can decide whether to keep both boxes or merge one of the boxes with the other if they form a duplicate. In this case, the van and the car are two separate objects so both bounding boxes are kept. Candidate pairs are displayed sorted by their IoU and reviewed until convergence.
Fig. 10 : Monitoring results for duplicate removal in COCO. In (a), duplicate rate drops as more cases are reviewed and resolved. In (b), the IoU distribution of duplicate candidates is shown.
Fig. 11 : Monitoring results for duplicate removal in Pascal VOC. In (a), duplicate rate drops as more cases are reviewed and resolved. In (b), the IoU distribution of duplicate candidates is shown.
Fig. 12 : Monitoring results for duplicate removal in KITTI. In (a), duplicate rate drops as more cases are reviewed and resolved. In (b), the IoU distribution of duplicate candidates is shown.
Fig. 13 : Monitoring results for duplicate removal in Cityscapes. In (a), duplicate rate drops as more cases are reviewed and resolved. In (b), the IoU distribution of duplicate candidates is shown.
Class
Train (5717 images)
Val (5823 images)
Original GT
Validated GT
Δ [%]
Original GT
Validated GT
Δ [%]
aeroplane
470
506
+7.7
484
510
+5.4
bicycle
410
436
+6.3
380
416
+9.5
bird
592
673
+13.7
629
642
+2.1
boat
508
602
+18.5
491
575
+17.1
bottle
749
805
+7.5
733
806
+10.0
Appendix
TABLE 12: Class-wise comparison of annotation counts between the original and validated Pascal VOC ground truth for the training and validation splits.
Table 21Table 22
Class
Original GT
Validated GT
Δ[%]
Kitchen
bottle
1,025
1,123
+9.6
wine glass
343
384
+12.0
cup
899
736
-18.1
fork
215
205
-4.7
knife
326
239
-26.7
Appendix
TABLE 17: Comparison of annotation counts between the original and validated COCO ground truth for the validation split across kitchen , food and appliance categories.
Class
Train (2,975 images)
Val (500 images)
Original GT
Validated GT
Δ[%]
Original GT
Validated GT
Δ[%]
person
17,994
18,104
+0.6
3,419
3,559
+4.1
rider
1,807
1,860
+2.9
556
527
-5.2
car
27,155
28,526
+5.0
4,667
4,074
-12.7
truck
489
539
+10.2
93
84
-9.7
bus
385
380
-1.3
98
97
-1.0
Appendix
TABLE 18: Class-wise comparison of annotation counts between the original and validated Cityscapes ground truth for the training and validation splits.
Class
Train (5,984 images)
Val (1,497 images)
Original GT
Validated GT
Δ[%]
Original GT
Validated GT
Δ[%]
car
22,923
36,627
+59.8
5,819
10,357
+78.0
van
2,323
2,196
-5.5
591
733
+24.0
truck
857
1,699
+98.3
237
433
+82.7
pedestrian
3,591
5,912
+64.6
896
1,469
+64.0
person sitting
182
801
+340.1
40
229
+472.5
Appendix
TABLE 19: Class-wise comparison of annotation counts between the original and validated KITTI ground truth for the training and validation splits.
Review
FNR [%]
Relative Reduction [%]
Dataset
Images
Objects
Original GT
Validated GT (Ours)
Pascal VOC
600
2,072
18.6
9.3
50.0
COCO
600
4,666
13.1
6.1
53.2
KITTI
600
5,908
41.5
11.6
72.0
Cityscapes
500
11,594
11.1
3.8
65.8
Appendix
TABLE 20: False negative analysis for the original and validated ground truth. The table reports the number of reviewed images and objects, the corresponding false negative rates (FNRs), and the relative FNR reduction achieved by our validated GT. For Cityscapes, we evaluate the complete validation set of 500 images, since only the validation set was fully re-annotated.
Fig. 14 : Comparison of missing bounding boxes, Rows: Original GT, Validated GT (ours), Columns: COCO, Cityscapes, KITTI, Pascal VOC For KITTI, we found no missing objects thus we are showing a second error of the original GT. We see that all provided examples are clear errors which are missing from the validated or original ground truth, while for the original ground truth way more objects are missing in total.
Dataset
#Boxes
11 [%]
22 [%]
33 [%]
44 [%]
#Ambiguous
[%]
Cityscapes
69,478
92.26
7.33
0.16
0.25
42
0.06
KITTI
65,711
50.12
47.03
1.09
1.75
30
0.05
Pascal VOC
36,134
71.04
27.69
0.48
0.79
30
0.08
COCO
56,067
49.82
46.32
1.25
2.61
350
0.62
Appendix
TABLE 21: Distribution of the number of annotations per bounding box and proportion of ambiguous instances where multiple classes share the highest probability.
Fig. 15 : Examples of ambiguous cases across all four datasets. Each bounding box corresponds to an instance where the soft label has no unique maximum, i.e., multiple classes share the highest probability. The labels shown above each box indicate the classes with equal highest probability.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Considering all objects
Arbitrary
17,371
5,533
829
10,578
4,779
544
≥25 pixels
3,579
3,716
780
2,828
3,391
514
≥40 pixels
944
1,869
570
773
1,709
380
Considering only objects outside of ’Don’t Care’ regions
Appendix
TABLE 22: Number of identified label errors on the train split in the KITTI dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Considering all objects
Arbitrary
5,339
1,634
197
2,527
1,319
141
≥25 pixels
951
1,054
185
674
946
133
≥40 pixels
235
511
138
188
465
99
Considering only objects outside of ’Don’t Care’ regions
Appendix
TABLE 23: Number of identified label errors on the val split in the KITTI dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Arbitrary
4,007
1,130
21
2,798
927
16
≥25 pixels
3,039
1,038
19
2,513
899
15
≥40 pixels
1,895
803
15
1,746
747
14
Appendix
TABLE 24: Number of identified label errors on the train split in the Cityscapes dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Arbitrary
1,242
1,255
64
720
758
47
≥25 pixels
1,002
1,035
64
686
725
47
≥40 pixels
582
698
55
483
571
42
Appendix
TABLE 25: Number of identified label errors on the val split in the Cityscapes dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Arbitrary
16,219
7,736
254
11,855
6,152
146
≥25 pixels
7,224
5,570
226
5,949
4,781
128
≥40 pixels
3,182
3,983
172
2,685
3,497
98
Appendix
TABLE 26: Number of identified label errors on the val split in the COCO dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Arbitrary
1,835
1,120
29
957
856
11
≥25 pixels
1,004
932
29
629
752
11
≥40 pixels
597
780
27
383
644
10
Appendix
TABLE 27: Number of identified label errors on the train split in the Pascal VOC dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Probability threshold
p≥0.5
p≥0.8
Object Height
Missing
Misaligned
Classification
Missing
Misaligned
Classification
Arbitrary
1,819
1,090
23
846
852
17
≥25 pixels
1,030
919
23
574
756
17
≥40 pixels
636
774
21
378
646
15
Appendix
TABLE 28: Number of identified label errors on the val split in the Pascal VOC dataset depending on the probability threshold for soft label annotations and the minimal height of considered objects.
Fig. 16 : Examples of original-only annotations across all datasets. Red bounding boxes correspond to annotations present only in the original ground truth, while green boxes show validated annotations. The examples illustrate that original-only cases arise from both missed objects during re-annotation (often small or occluded instances) and spurious or coarse annotations in the original ground truth, reflecting the inherent ambiguity of these cases.
Dataset
Split
Original-Only
Accepted
Rejected
Accepted [%]
COCO
Val
3,057
2,037
1,020
66.63
Cityscapes
Val
454
242
212
53.30
KITTI
Train
711
501
210
70.46
Val
138
93
45
67.39
Pascal VOC
Train
569
546
23
95.96
Val
506
475
31
93.87
Appendix
TABLE 29: Manual review of original-only annotations across all datasets. For each original-only annotation, we manually review whether the bounding box corresponds to an object that should be included in the dataset. The acceptance rate denotes the fraction of accepted original-only annotations.
Fig. 17 : Distribution of bounding box areas for original-only annotations across all datasets. Each curve shows the frequency of bounding box sizes on a logarithmic scale.
Dataset
Split
Original GT
VGT
ΔVGT [%]
Original-Only Added
VGT + Original-Only
ΔVGT+OO [%]
Cityscapes
Val
10,180
10,383
+1.99
209
10,592
+4.05
KITTI
Train
31,589
49,551
+56.86
376
49,927
+58.05
Val
8,008
13,797
+72.29
72
13,869
+73.19
Pascal VOC
Train
15,774
17,029
+7.96
524
17,553
+11.28
Val
15,787
17,233
+9.16
470
17,703
+12.14
COCO
Val
36,781
51,487
+39.98
1,553
53,040
+44.20
Appendix
TABLE 30: Comparison of the number of annotations in the original ground truth, the validated ground truth (VGT), and the VGT after adding the reviewed original-only (OO) annotations. All counts in this table are based on hard label annotations. The relative increases of the VGT and VGT + original-only are computed with respect to the original GT.
Dataset
Validated GT
Validated GT + Original-Only
Relative Reduction [%]
Pascal VOC
9.3±1.3
7.2±1.1
22.6
COCO
6.1±0.7
3.6±0.5
41.0
KITTI
11.6±0.8
10.6±0.8
8.6
Cityscapes
3.8±0.3
2.8±0.3
26.3
Appendix
TABLE 31: Effect of incorporating reviewed original-only annotations on the false negative rate (FNR). Values are reported in % with 95% Wilson confidence intervals. The validated GT corresponds to the validated annotations before adding original-only annotations, while validated GT + original-only additionally includes the accepted original-only annotations.
Fig. 18 : Annotation Times with proposal (biased, yellow) and without proposal (unbiased, blue). The graphic shows a histogram plot of the raw annotation times and its median.
Fig. 19 : Annotation Times with proposal (biased, yellow) and without proposal (unbiased, blue). The graphic shows a histogram plot of the raw annotation times and its median.
Fig. 20 : Biased minus unbiased probability for proposed label, expect values above 0 which indicate an introduced bias
Fig. 21 : Biased minus unbiased probability for proposed label, expect values above 0 which indicate an introduced bias
Dataset
Method
Pˉunbiased
Pˉbiased
Pˉpost
JS (mean ± std)
ΔJS (95% CI)
Cityscapes
CL δ=0.10 , α=0.5
.827
.888
.882
0.1301 ± 0.1490
−0.00008 ([+0.00035, −0.00053 ])
Kitti
CL δ=0.00
.886
.868
.871
0.1337 ± 0.1579
−0.00193 ([ −0.00170 , −0.00214 ])
Pascal VOC
CL Lite δ=0.50
.845
.915
.877
0.0902 ± 0.1236
−0.00646 ([ −0.00391 , −0.00879 ])
COCO/accessory
CL Lite δ=0.40
.772
.914
.845
0.1626 ± 0.1636
−0.01468 ([ −0.01132 , −0.01824 ])
COCO/animal
CL δ=0.15 , α=0.4
.784
.829
.819
0.0938 ± 0.1222
−0.00089 ([ −0.00012 , −0.00164 ])
COCO/electronic
CL δ=0.40 , α=0.8
.868
.946
.937
0.1159 ± 0.1387
−0.00173 ([ −0.00078 , −0.00273 ])
Appendix
TABLE 32: Post-processing results per dataset. Here, Pˉunbiased denotes the mean probability of the proposed class for unbiased annotations, Pˉbiased the mean probability of the proposed class for the biased annotations, and Pˉpost the mean probability of the proposed class after post processing. CL = CleverLabel with parameters δ and avoid-overcorrection threshold α ; CL Lite uses only δ as a downweight factor. JS denotes the square root of the Jensen–Shannon divergence. ΔJS is the average bootstrapped difference JS(post,unbiased)−JS(biased,unbiased) ; negative values indicate improvement.
Fig. 22 : Alignment of Human Perception with and without proposal and our postprocessing on the dataset COCO - The figure shows the ratings of a expert human labeler if the label is clearly ( ≥0.75 ), maybe ( 0.25−0.75 ) or not ( <0.25 ) correct. The x-axis are the corresponding soft probability for the evaluated class. Median is given as star, mean and std given as line plot.
Fig. 23 : Alignment of Human Perception with and without proposal and our postprocessing on the dataset Cityscapes - The figure shows the ratings of a expert human labeler if the label is clearly ( ≥0.75 ), maybe ( 0.25−0.75 ) or not ( <0.25 ) correct. The x-axis are the corresponding soft probability for the evaluated class. Median is given as star, mean and std given as line plot.
Fig. 24 : Alignment of Human Perception with and without proposal and our postprocessing on the dataset KITTI - The figure shows the ratings of a expert human labeler if the label is clearly ( ≥0.75 ), maybe ( 0.25−0.75 ) or not ( <0.25 ) correct. The x-axis are the corresponding soft probability for the evaluated class. Median is given as star, mean and std given as line plot.
Fig. 25 : Alignment of Human Perception with and without proposal and our postprocessing on the dataset Pascal VOC - The figure shows the ratings of a expert human labeler if the label is clearly ( ≥0.75 ), maybe ( 0.25−0.75 ) or not ( <0.25 ) correct. The x-axis are the corresponding soft probability for the evaluated class. Median is given as star, mean and std given as line plot.
TABLE 33: COCO. Bucket agreement between validated annotations and human expert judgments. Bucket accuracy is 62.8 % vs. 72.4 %, Error 1+ is 37.2 % vs. 27.6 %, and Error 2+ is 3.5 % vs. 0.0 %.
TABLE 34: Cityscapes. Bucket agreement between validated annotations and human expert judgments. Bucket accuracy is 68.0 % vs. 82.7 %, Error 1+ is 32.0 % vs. 17.3 %, and Error 2+ is 3.0 % vs. 0.0 %.
TABLE 35: KITTI. Bucket agreement between validated annotations and human expert judgments. Bucket accuracy is 68.4 % vs. 74.3 %, Error 1+ is 31.6 % vs. 25.7 %, and Error 2+ is 2.6 % vs. 0.0 %.
TABLE 36: Pascal VOC. Bucket agreement between validated annotations and human expert judgments. Bucket accuracy is 77.6 % vs. 82.7 %, Error 1+ is 22.4 % vs. 17.3 %, and Error 2+ is 2.0 % vs. 0.0 %.
Dataset
n
Certainly wrong
COCO
200
6
Cityscapes
200
3
KITTI
200
5
PascalVOC
200
1
Appendix
TABLE 37: Original ground-truth labels: certainly wrong labels only. Counts are based on the “definitely wrong” annotations out of n=200 sampled labels per dataset.
Fig. 26 : Comparison of wrong class errors, Rows: Original GT, Validated GT (ours), Columns: COCO, Cityscapes, KITTI, Pascal VOC For the original GT the ground truth label is given versus the expert reviewed label in bold font. For the validated GT the complete soft distribution is given and in bold the expert reviewed label. We see that the original GT has several very clear errors, especially the human in COCO since the woman holding the umbrella already has a different bounding box (not shown) or truck in KITTI which includes buses. For the validated GT, we always at least some signal for the correct class and for the example in Pascal VOC it is debatable if dog or sofa is actually correct given only one box while the other is not shown.
Model
Original Annotations
Refined Annotations ( p>0.5 )
Refined Annotations ( p>0.8 )
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
Grounding DINO
56.9
74.2
62.6
40.1
61.1
72.6
70.5
77.4
51.2
71.3
55
26.3
50.3
74
61.3
69.6
52.1
71.8
56.1
28
50.3
72.9
63.7
71.7
YOLOv8-World v2
44.5
60.2
48.4
28.9
48.9
57.2
58.5
62.9
42.3
59
45.5
19.4
42.1
60.6
53.8
59.4
43.8
60.5
47.2
21.3
42.8
60.2
56.6
62.3
OWL-ViT
28.7
45.4
30.2
11
29.6
48
44.3
48.5
27.4
43.1
28.5
6.5
22.2
47.5
41
45.2
28.6
44.7
29.8
7.2
22.8
47.3
43.4
47.8
RT-DETR-L
52.4
70.6
56.8
33.6
57
70.5
63.6
69.6
46.1
65.6
48.6
20.9
44.4
68.6
55
62.1
47.2
66.4
50
22.8
44.4
67.2
57.6
64.8
YOLOv8-L
51.7
68.6
56.4
34.3
57.1
68.7
62.4
67
46.4
64.6
49.5
21.3
45.9
67.3
55.1
61
47.8
65.8
51.2
23.4
45.8
66.9
58
64
Appendix
TABLE 38: Performance of object detectors on original vs. refined annotations on COCO.
Model
Original Annotations
Refined Annotations ( p>0.5 )
Refined Annotations ( p>0.8 )
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
Grounding DINO
80
92.4
87.5
33.5
66.5
88
88.2
90.9
72.3
88.6
78.9
30
59.7
84
80.6
84.9
73.8
89.5
80.8
32
60.1
84
82.6
86.5
YOLOv8-World v2
69.6
83.4
75.7
26.8
56.2
77.6
81.9
83.7
63.8
80
69.1
21
50.1
75.4
74.4
77.2
65.4
81.3
71.1
24.1
51.1
75.5
76.6
79.2
OWL-ViT
52.8
72
59.1
11.6
36.3
62.8
74.3
78
48
68
52.9
9.8
32.3
61.6
67.2
71.3
49.2
69.2
54.5
11.4
32.9
61.5
69.4
73.4
RT-DETR-L
70.3
86
76.9
29.3
55.4
77.6
82.3
87.8
62.6
80.8
68.2
21.9
48.3
74.5
73.6
80.3
64.2
82.1
70.2
24.2
49
74.4
76
82.6
YOLOv8-L
74.4
89.2
81.2
33.1
59.2
81.4
82.7
84.6
65.6
83.4
71.4
24.4
51.9
77
74
76.8
67.4
85
73.5
28.2
52.9
77
76.4
79.1
Appendix
TABLE 39: Performance of object detectors on original vs. refined annotations on Pascal VOC.
Model
Original Annotations
Refined Annotations ( p>0.5 )
Refined Annotations ( p>0.8 )
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
Grounding DINO
29.2
51.3
29.5
21.4
30.7
36.2
50.7
56.3
34.5
51.9
38.3
16.8
43.4
67.4
54.2
61.3
39.4
57.3
44.8
20.5
43.5
67.5
61.7
68.2
YOLOv8-World v2
19.9
33.6
21
18.8
18.7
24.7
31.7
33.5
21.3
30.2
22.7
7.7
27
45.2
31.6
34.9
24.4
34.3
26.2
10.5
27.2
44.4
37.7
41.4
OWL-ViT
9.3
20
7.4
1.3
6.8
19.2
22.3
24.4
8.4
16.7
7
1.4
8.5
27.3
18.7
20.3
9.6
19.1
8
1.8
8.5
27.1
22
23.9
RT-DETR-L
78.5
96.2
87.8
76.2
79.6
80.2
82.2
85.5
23.4
43.1
22.9
9.5
34.1
44.6
31.1
34.7
27.3
50.2
26.8
13.2
34.2
44.4
36.9
41.2
YOLOv8-L
74.7
93.5
84.2
67.5
76.7
77.5
78.3
79.2
22.9
42.6
21.8
9.1
33.5
42
29.6
30.1
26.7
49.6
25.4
12.9
33.4
41.9
35.4
36
Appendix
TABLE 40: Performance of object detectors on original vs. refined annotations on KITTI.
Model
Original Annotations
Refined Annotations ( p>0.5 )
Refined Annotations ( p>0.8 )
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
AP 50..95
AP 50
AP 75
AP S
AP M
AP L
AR 10
AR 100
Grounding DINO
34.2
53.3
33.5
7.9
23.7
52.3
51.6
58.8
40.2
57.7
41.9
12.1
27.4
57.8
57.9
66.5
43.8
61.6
46
14.4
28.8
58
63.1
71.1
YOLOv8-World v2
24.1
36.3
24.4
2.2
14.4
41.3
30.6
33.5
26.6
38.8
27.4
7.6
16.1
43.4
33.7
36.7
30.4
44
31.4
4.3
17.4
44.4
38.7
42
OWL-ViT
11.6
21.7
11.1
0.3
4.9
22.1
17.4
18.7
12.2
22.6
11.8
0.2
5.3
22.6
18.5
19.8
13.8
25.5
13.5
0.6
6
23
21.2
22.7
RT-DETR-L
38.2
59.8
38.5
5.7
27.5
57
42.3
47.7
39.1
60.6
40.9
9.1
27.3
57.7
44.9
50.1
42.5
64.2
44.8
11.8
28.8
57.8
50.1
55.4
YOLOv8-L
38.1
58.7
37.4
3.9
26
61.7
43.7
47.4
40.6
60.5
42.5
3.8
26.3
63.3
45.5
49
44.4
65.3
46.8
8.3
27.8
63.2
51.5
55.2
Appendix
TABLE 41: Performance of object detectors on original vs. refined annotations on Cityscapes.
Dataset
Model
ECE (Raw)
ECE (PS)
ECE (IR)
COCO
Grounding DINO [ 54 ]
3.0\times10−02
4.4\times10−03
1.8\times10−04
YOLOv8-World v2 [ 55 ]
6.8\times10−02
2.1\times10−02
1.1\times10−03
OWL-ViT [ 56 ]
1.4\times10−02
2.5\times10−03
1.4\times10−04
RT-DETR-L [ 53 ]
1.0\times10−02
4.2\times10−03
1.7\times10−04
YOLOv8-L [ 52 ]
9.8\times10−02
8.0\times10−03
1.7\times10−03
Faster R-CNN (ResNet 50) [ 35 ]
1.5\times10−02
3.9\times10−03
1.7\times10−04
Appendix
TABLE 42: ECE before and after calibration using Platt Scaling (PS) and Isotonic Regression (IR).
Fig. 27 : Calibration plots for Grounding DINO on COCO before and after post-hoc calibration through isotonic regression.
Model
AUSRC 50..95 ↑
AUSRC 50 ↑
AUSRC 75 ↑
AUSRC S ↑
AUSRC M ↑
AUSRC L ↑
TVD ↓
TVD TP ↓
JS↓
JS TP ↓
Grounding DINO
55.1
79
58
38.4
60.3
78.8
12
38.2
16.4
35
YOLOv8-World v2
48.1
70.1
50.2
30.6
56
70.3
16.4
37.6
22.3
35.7
OWL-ViT
35.8
58.6
35.4
14
42.6
67.8
13.9
75.2
18.4
63
RT-DETR-L
49.1
73.2
50.8
32.3
53.6
74.5
14.1
46.9
19.3
44
YOLOv8-L
46.6
69.1
48.2
30.3
53.4
67.8
17.6
33.1
23.7
31.7
Faster R-CNN (ResNet 50)
44
67.3
45.1
29.6
50.7
61.3
18.2
24.7
21.9
25.5
Appendix
TABLE 43: Soft label-based evaluation of object detector performance on COCO.
Model
AUSRC 50..95 ↑
AUSRC 50 ↑
AUSRC 75 ↑
AUSRC S ↑
AUSRC M ↑
AUSRC L ↑
TVD ↓
TVD TP ↓
JS↓
JS TP ↓
Grounding DINO
81.7
97.4
87.7
58.2
76
90.2
1.7
24.2
2.2
24.8
YOLOv8-World v2
75.2
93.6
81.3
51.3
69.5
82.5
4.4
34.8
7
34.3
OWL-ViT
69.3
92.2
74.6
33.1
60
83.7
0.9
53.5
1.9
48
RT-DETR-L
78.8
95.6
83.7
49.4
71.3
90.5
0.6
36
1.5
36.5
YOLOv8-L
72.2
90.6
78.3
45.9
66.5
82.1
5.9
24.8
8.4
26
Faster R-CNN (ResNet 50)
40
51
44.5
35.6
37.1
31.5
49
71.2
43.6
59.7
Appendix
TABLE 44: Soft label-based evaluation of object detector performance on Pascal VOC.
Model
AUSRC 50..95 ↑
AUSRC 50 ↑
AUSRC 75 ↑
AUSRC S ↑
AUSRC M ↑
AUSRC L ↑
TVD ↓
TVD TP ↓
JS↓
JS TP ↓
Grounding DINO
60.5
87.7
63.4
44.5
73.3
88.2
4.2
40.2
6.8
38.4
YOLOv8-World v2
49.7
70.7
51.8
28.4
68.7
79.1
7.4
49.6
10.9
45
OWL-ViT
29.1
56.2
26.2
9.8
40.5
75.3
2.3
68
4.7
60
RT-DETR-L
40.5
61.9
45.1
19.7
58.1
75.9
1.1
28.4
1.5
27.6
YOLOv8-L
28.5
47.9
29.6
13
42.6
51.6
27.9
6.5
26.4
8.6
Faster R-CNN (ResNet 50)
32.8
50.4
35.8
17.5
45.8
58.8
23.1
34.6
23.8
32.7
Appendix
TABLE 45: Soft label-based evaluation of object detector performance on KITTI.
Model
AUSRC 50..95 ↑
AUSRC 50 ↑
AUSRC 75 ↑
AUSRC S ↑
AUSRC M ↑
AUSRC L ↑
TVD ↓
TVD TP ↓
JS↓
JS TP ↓
Grounding DINO
54.9
79.4
57.1
31.6
54.4
72.7
12.4
35.8
15.3
36.2
YOLOv8-World v2
41.1
62.5
42.2
8.5
40
71
11.9
38.2
16.6
37.2
OWL-ViT
24.2
48.8
21.2
2
17.8
55.7
2.4
54.7
4.8
49.7
RT-DETR-L
53
84.3
53
25.7
53.3
75.4
5.5
33.1
8.6
31.2
YOLOv8-L
45.5
72.4
46.7
13.4
44.9
73.8
8.2
19.8
12.6
21.6
Faster R-CNN (ResNet 50)
12.7
19.9
13.4
6.4
11.8
20.4
73
58.7
60.3
50.8
Appendix
TABLE 46: Soft label-based evaluation of object detector performance on Cityscapes.
COCO
Pascal VOC
KITTI
Cityscapes
Model
AP ↑
AS ↑
ECE ↓
TVD ↓
JS↓
AP ↑
AS ↑
ECE ↓
TVD ↓
JS↓
AP ↑
AS ↑
ECE ↓
TVD ↓
JS↓
AP ↑
AS ↑
ECE ↓
TVD ↓
JS↓
RT-DETR-L
48.7
52.7
3
10.2
15.7
63.2
80.1
3.5
0.6
1.4
56.2
72.2
2.2
1.3
3.1
40.5
53.1
5.6
5.2
8.3
RT-DETR-L ∗
49
52.9
2.4
10
15.4
63.8
80.1
3.3
0.6
1.4
56.8
72
1.7
1.3
3
40.3
52.7
3.4
6.1
9.5
YOLOv8-L
50.2
52
6.7
14.2
20.5
66.6
75.6
1.4
3.8
5.9
50
56.5
4.7
9.9
13.7
40
45.1
1.5
8.6
12.6
YOLOv8-L ∗
50.2
51.8
7.3
14.9
21.2
66.8
75.3
1.2
4.1
6.2
49.9
56.3
4.4
9.9
13.7
39.9
44.6
1.8
8.8
13
Appendix
TABLE 47: Performance and calibration of models trained on hard vs. soft labels ( ∗ ). Values in %.
TABLE 48: Change in AP, when training on our validated annotations, compared to training on original annotations. We evaluate on both sets of annotations and report changes in AP in percentage points. Positive numbers indicate that training on our validated annotations improved AP.
Fig. 28 : Overconfident annotations in the original ground truth. In each case the original hard label was assigned with implicit confidence 1.0, but our crowd-sourced soft-labels indicate ambiguity for these objects.
Fig. 29 : Model predictions vs. soft labels on objects where both disagree. We show illustrative cases with a discrepancy between the YOLOv8 predicted probabilities and the corresponding soft label for each dataset. We display all class probabilities larger than 0.05 in the images instead of the full distribution for readability.
TABLE 49: Detection performance comparison of Grounding DINO and Cascade R-CNN with respect to the original ground truth on the evaluated datasets using COCO evaluation metrics.
Dataset
Batch Size
Image Resolution
# Iterations
Learning Rate
Cityscapes
4
1333×800
120,000
0.01
COCO
12
1000×600
250,000
0.02
KITTI
6
1000×600
70,000
0.01
Pascal VOC
6
1000×600
70,000
0.02
Appendix
TABLE 50: Training parameters of the Cascade R-CNN models used to generate predictions for the loss inspection label error detection method. We report the batch size, input image resolution, number of training iterations, and learning rate for each dataset.
Fig. 30 : Distribution of bounding box areas for validated ground truth annotations compared to Grounding DINO and Cascade R-CNN predictions across the evaluated datasets. For visualization purposes, predictions are filtered using a confidence threshold of 0.2, while the label error detection benchmark itself uses a lower threshold of 0.01 to maximize object coverage.
Cityscapes
AP
Prec
Rec
F1
Naive Baseline
-
0.64
100.00
1.28
Score Baseline
17.16
20.64
47.35
28.75
Loss Inspection
8.83
10.63
35.95
16.41
ObjectLab
13.45
17.25
41.70
24.41
MetaDetect
16.24
21.20
34.58
26.28
Appendix
TABLE 51: Label error detection performance across multiple datasets and representative label error detection methods on the stricter label error ground truth (including only label errors with a minimum height of 40 pixels and soft label probability ≥0.8 ). Evaluation is based on predictions from Grounding DINO. For all score-based methods, Average Precision, Precision, Recall, and F1 are reported at the method-specific threshold that maximizes F1-Score on the corresponding precision-recall curve. The naive baseline has no ranking score and is therefore evaluated as a single operating point; AP is not defined for this baseline. Evaluation is performed using an IoU threshold of 0.1 for all methods.
Fig. 31 : Precision-recall curves of label error detection methods across all datasets, evaluated on the less restrictive label error ground truth. All methods operate on predictions from Grounding DINO, except for the loss-based method, which uses Cascade R-CNN. The cross on each curve indicates the operating point with the highest F1-score, at which all methods are evaluated. The naive baseline is shown for reference but is not directly comparable, as it has no ranking score (and thus no AP) and is evaluated at a single operating point.
Table 76
Cityscapes
AP
Prec
Rec
F1
Naive Baseline
-
3.04
86.86
5.88
Score Baseline
16.77
22.30
30.57
25.79
Loss Inspection
8.83
10.63
35.95
16.41
ObjectLab
15.53
19.89
34.12
25.13
MetaDetect
14.83
24.55
19.71
21.86
Appendix
TABLE 53: Label error detection performance across multiple datasets and representative label error detection methods on the stricter label error ground truth (including only label errors with a minimum height of 40 pixels and soft label probability ≥0.8 ). Evaluation is based on predictions from Cascade R-CNN. For all score-based methods, Average Precision, Precision, Recall, and F1 are reported at the method-specific threshold that maximizes F1-Score on the corresponding precision-recall curve. The naive baseline has no ranking score and is therefore evaluated as a single operating point; AP is not defined for this baseline. Evaluation is performed using an IoU threshold of 0.1 for all methods.
Fig. 32 : Precision-recall curves of label error detection methods across all datasets, evaluated on the less restrictive label error ground truth. All methods operate on predictions from Cascade R-CNN. The cross on each curve indicates the operating point with the highest F1-score, at which all methods are evaluated. The naive baseline is shown for reference but is not directly comparable, as it has no ranking score (and thus no AP) and is evaluated at a single operating point.
Fig. 33 : Examples of missing label errors across all four datasets. The green bounding boxes denote the original ground truth annotations, while the red bounding box highlights the label error with corresponding soft label probability p .
Fig. 34 : Examples of misaligned label errors across all four datasets. The green bounding boxes denote the original ground truth annotations, while the red bounding box highlights the label error with corresponding soft label probability p .
Fig. 35 : Examples of classification label errors across all four datasets. The left (green) bounding box shows the original ground truth annotation with its corresponding label, while the right bounding box and label correspond to the validated ground truth with soft label probability p .
Annotation errors are widespread in computer vision datasets and can significantly degrade the performance of systems trained on them, particularly in complex tasks such as object detection. Several approaches exist to identify annotation errors, including training-free feature-space methods which provide a fast and interpretable way to analyze annotations. However, the behavior on object detection annotations, which include semantic and spatial information, remains largely unexplored. In this work we analyze the applicability of feature-space-based approaches for detecting annotation errors in object detection datasets. By adapting an existing feature-space method, we show that such approaches reliably expose semantic mislabel, while positional errors remain difficult to detect. We evaluate this behavior across multiple pretrained embedding models, synthetic noise types (symmetric, asymmetric, and positional), and real-world annotation errors using VOC2012 and KITTI. All code and real-world corruptions are publicly available at the following repository: https://github.com/ ChristianSieberichs/BoundingBox_corruption_detection
Christian Sieberichs, Simon Geerkens, Thomas Waschulzik +2
University of Applied Sciences Düsseldorf · Siemens Mobility GmbH · Goethe University Frankfurt
Reliable validation of machine learning models is a crucial step in designing robust and trustworthy systems. Yet validation relies on annotated test sets treated as ground truth, even though annotations can be noisy or ambiguous. In object detection, sources may disagree on object presence, box location, or class label. Standard agreement metrics usually binarize box overlap before computing a summary score, obscuring the degree and structure of disagreement between annotation sets. Here, we study an Optimal Transport (OT)-based metric for measuring agreement between object detection annotations. Instead of binarizing bounding box overlap, this metric assigns graded costs to matched and unmatched labeled boxes. Through theoretical analysis and experiments on real and synthetically perturbed annotations, we show that OT captures graded changes in agreement and distinguishes annotation pairs that thresholded metrics score identically. Our results suggest that OT provides a richer measure of annotation agreement in object detection, especially for domains with many small or ambiguous objects. Such agreement estimates can support more reliable validation when labels are noisy or conflicting.
Kirill Prokhorov, Alexandr A. Kalinin
Openface.io, Vilnius, Lithuania · Broad Institute of MIT and Harvard, Cambridge, MA, USA
High degrees of disagreement among annotators can exist for ambiguous objects, e.g. in medical images, underscoring the challenges of establishing ground truth annotations in object detection tasks. Despite this, all existing object detectors implicitly require access to ground truth annotations for either training or evaluation. The fundamental questions we target are: How can we learn an object detector with multiple annotators' annotations but without objective ground truth annotations due to object ambiguity, and how can we enable the learned detector to express meaningful model predictive uncertainties in detecting ambiguous objects? To answer these questions, we present an interpretable approach to calibrate probabilistic object detectors, where the calibration goal is to align the class confidence and bounding box variance estimates to the annotators' annotation distribution. We introduce an efficient yet effective framework to calibrate probabilistic object detectors by designing four evaluation metrics to measure calibration errors regarding classification and localization, and proposing a train-time calibration and post-hoc calibrator, all without the need to access any ground truth. This framework is generalizable to many existing probabilistic object detectors, such as the YOLO families and two-stage detectors. Empirical results with real-world and synthetic datasets of medical and natural images demonstrate the superior performance of the proposed framework with three popular object detectors.
Zhi Qin Tan, Owen Addison, Yunpeng Li
Faculty of Dentistry, Oral & Craniofacial Sciences, King’s College London, London, United Kingdom