cs.CVOct 14, 2026

Quantifying Annotation Agreement in Object Detection with Optimal Transport

Authors: Kirill Prokhorov, Alexandr A. Kalinin

Organizations: Openface.io, Vilnius, Lithuania · Broad Institute of MIT and Harvard, Cambridge, MA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reliable validation of machine learning models is a crucial step in designing robust and trustworthy systems. Yet validation relies on annotated test sets treated as ground truth, even though annotations can be noisy or ambiguous. In object detection, sources may disagree on object presence, box location, or class label. Standard agreement metrics usually binarize box overlap before computing a summary score, obscuring the degree and structure of disagreement between annotation sets. Here, we study an Optimal Transport (OT)-based metric for measuring agreement between object detection annotations. Instead of binarizing bounding box overlap, this metric assigns graded costs to matched and unmatched labeled boxes. Through theoretical analysis and experiments on real and synthetically perturbed annotations, we show that OT captures graded changes in agreement and distinguishes annotation pairs that thresholded metrics score identically. Our results suggest that OT provides a richer measure of annotation agreement in object detection, especially for domains with many small or ambiguous objects. Such agreement estimates can support more reliable validation when labels are noisy or conflicting.

Figures & tables

Figure 1: OT score bounds and corresponding range of values as a function of the ratio between the numbers of boxes in two annotations. When the numbers of boxes coincide, the range equals λ\lambda , while β\beta controls the maximum (minimum) values of the cost (score). All curves asymptotically converge as log⁡2 ⁣(NM)→±∞\log_{2}\!\left(\tfrac{N}{M}\right)\to\pm\infty . The sensitivity of the optimal transport metric to spatial localization gives rise to the full range of possible values.
Figure 2: Probability of real–real matching under the noisy-box assumption. a) Dependence on β\beta for several noise levels γ\gamma . b) Dependence on γ\gamma for fixed β\beta . Theoretical and empirical results diverge at higher noise levels as matches break and the noisy-box assumption becomes less accurate.
Figure 3: OT scores relative to their theoretical bounds. a) ACNE04 scores span much of the feasible range. b) OT scores for several TexBiG classes. c,d) Score distributions between the lower and upper bounds for ACNE04 and TexBiG, respectively. Because TexBiG objects are larger and easier to localize, their spatial costs are generally smaller and their scores lie closer to the upper bound.
Figure 4: Annotator agreement between original and perturbed annotations. a) Agreement versus fraction of swapped labels on AcneSCU dataset. b) Agreement versus ratio of dropped boxes on ACNE04 dataset. c) Agreement versus shift noise level on ACNE04 dataset.
Figure 5: Behavior of agreement metrics under increasing box-shift noise and beyond the mIoU=0mIoU=0 regime. a) Ordering under increasing shift noise. b) Percentage of images with mIoU=0mIoU=0 versus noise level. c) Distribution of OT scores for cases with mIoU=0mIoU=0 .
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Integration regions used in the calculation in Eq. 1 .
Figure 2: a) Two annotations with a moderate spatial shift. b) One annotation is deliberately shifted so that only one pair of boxes overlaps. F1 and Krippendorff’s α\alpha assign the same scores to both examples, whereas OT distinguishes their spatial disagreement. mIoU remains sensitive only while boxes overlap.
Figure 3: Spatial regions in which a translated probe box is eligible for real–real matching under different ρ=β/λ\rho=\beta/\lambda , compared with the region where mIoU>0\mathrm{mIoU}>0 .
Figure 4: TexBiG observed versus expected distances.
Figure 5: Empirical CDF of bounding-box area relative to image area across the datasets and classes used in the experiments.
Figure 6: Example images from the datasets. a,b) ACNE04 with two annotation sets. c,d) AcneSCU; colors denote lesion classes. e,f) TexBiG.
More in this topic

Inter-Rater Reliability

Inter-rater reliability measures how consistently different people or models judge the same items, accounting for chance and task-specific disagreement.

See all 44 papers

Explore similar work

CardsList

  1. Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty

    Sep 18, 2026Sarina Penquitt, Jonathan Klees, Antonia van Betteray +4Object DetectionHuman-in-the-Loop Annotation

    Institute of Computer Science Osnabrück University Osnabrück, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Wayve London, United Kingdom

  2. Calibrating Probabilistic Object Detectors with Annotator Disagreement

    May 23, 2026Zhi Qin Tan, Owen Addison, Yunpeng LiAnnotator DisagreementObject Detection

    Faculty of Dentistry, Oral & Craniofacial Sciences, King’s College London, London, United Kingdom

  3. Differences in Detection: Explainability Where it Matters

    Jun 5, 2026Johannes Theodoridis, Johannes Maucher, Andreas SchillingPairwise ComparisonObject Detection

    University of T ubingen · Institute for Applied AI · Hochschule der Medien Stuttgart