cs.CVMar 28, 2026
SaveKLOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks
Organizations: Bauhaus-Universität Weimar, Germany
Abstract
Progress in object detection benchmarks is stagnating. It is limited not by architectures but by the inability to distinguish model improvements from label noise. To restore trust in benchmarking the field requires rigorous quantification of annotation consistency to ensure the reliability of evaluation data. However, standard statistical metrics fail to handle the instance correspondence problem inherent to vision tasks. Furthermore, validating new agreement metrics remains circular because no objective ground truth for agreement exists. This forces reliance on unverifiable heuristics. We propose KLOS (KALOS), a unified meta-algorithm that generalizes the "Localization First" principle to standardize dataset quality evaluation. By resolving spatial correspondence before assessing agreement, our framework transforms complex spatio-categorical problems into nominal reliability matrices. Unlike prior heuristic implementations, KLOS employs a principled, data-driven configuration; by statistically calibrating the localization parameters to the inherent agreement distribution, it generalizes to diverse tasks ranging from bounding boxes to volumetric segmentation or pose estimation. This standardization enables granular diagnostics beyond a single score. These include annotator vitality, collaboration clustering, and localization sensitivity. To validate this approach, we introduce a novel and empirically derived noise generator. Where prior validations relied on uniform error assumptions, our controllable testbed models complex and non-isotropic human variability. This provides evidence of the metric's properties and establishes KLOS as a robust standard for distinguishing signal from noise in modern computer vision benchmarks.
Figures & tables
Figure 1: Visual Abstract.
Figure 2: The K LOS Framework. (1) Input: The pipeline processes active disagreement between raters. (2) Principled Configuration: Data-driven calibration selects the optimal distance function ( ) and threshold ( ) by maximizing the statistical separation between observed signal ( ) and chance noise ( ). (3) Execution: A solver transforms continuous spatial data into a discrete reliability matrix. Crucially, the completeness assumption encodes missed instances as no_object , treating them as existence disagreement (false positives/negatives) rather than missing data. (4) Downstream Diagnostics: This standardization enables rigorous quality checks, visualizing Localization Sensitivity, Class Difficulty, and Annotator Vitality ( ).
Figure 3: Performance of different correspondence solvers averaged across LVIS, TexBiG and VinDr-CXR. A simple filtered Rand Index variation was used due to the constrained clustering setup, where limited and partially overlapping clusters, together with non-exhaustive ground-truth coverage, rendered more advanced evaluation metrics infeasible or unreliable for assessing correspondence accuracy.
Figure 4: F1-score comparison. Greedy performs best.
Figure 5: Failure-type comparison across correspondence solvers and noise magnitudes using raincloud plots. TP shows correctly matched pairs, Missed Opportunities are correct pairs the solver failed to match, and FP captures spurious matches where no true pair exists. Cuckoo Eggs indicate cases where a correct match was displaced by an incorrect one, effectively combining a FN with a FP and revealing when class disagreement overrides Eq. 6 function.
Figure 6: Return on investment for using more annotators. Majority voting principal applies. Annotators that generally follow the same annotation convention increase consensus, independent of increasing magnitude.
Figure 7: Collaboration-cluster size impact analysis between two groups of A and B raters, denoted as (A–B) for the group sizes. LVIS is used to generate synthetic annotations from two anonymous, interchangeable reference raters.
| Dataset | Task(s) | Amount of Raters per Image/Scene | Annotator Type | Rater Identification + Intra-Rater Evaluation | Guideline and Annotation Process | Automated Proposals |
|---|---|---|---|---|---|---|
| TexBiG [ 40 ] | Instance Segmentation | 3–4 (train), 5 (test) | Mixed (Experts + Non-Experts) | Yes + No | Same | no |
| VinDr-CXR [ 32 ] | Object Detection | 3 (train), 5 (test) | Experts (Radiologists) | Yes + No | Same | no |
| LVIS Consistency Subset [ 16 ] | Instance Segmentation | 2 | Non-Experts | No + No | Same | no |
| COCO Reannotated [ 27 ] | Object Detection | 2 | Non-Experts | No + No | Different | yes |
| Open Images Reannotated [ 27 ] | Object Detection | 2 | Non-Experts | No + No | Different | yes |
| NuCLS [ 2 ] | Instance Segmentation | 11–39 | Mixed (Experts + Non-Experts) | Yes + Yes | Same | yes |
Table 1: Overview of datasets with repeated annotations. Bold names are the ones used for our main analysis.
Figure 8: Three step noise generator: 1) Empirical data is extracted. 2) Data is used to fit parametric models or estimate them using foundational models as validated proxies. 3) The four separate components use a hierarchical composition to generate synthetic annotations based on reference annotations ( e.g . human annotated data).
| Study | Type | Modelling | Hypothesis (Assumptions) |
| Agnew et al . [ 1 ] | Loc. | Uniform expansion (0-10 pixel buffering) of BBox | Noise is primarily expansion (outward shift). |
| Loc. | Radial Gaussian noise ( ) on mask vertices | Vertex precision varies normally; errors localize to edges. | |
| Mathet et al . [ 28 ] (CST) | Unm. | Random dropout & addition (class is based on marginal dist.) | Errors are probabilistic; False Positives follow dataset statistics. |
| Loc. | Random uniform shift | Shift magnitude is proportional to object length. | |
| Cat. | Confusion matrix (prevalence & overlap based) | Errors correlate with class frequency and semantic overlap. | |
| Fra. | Randomly select element and split it | Annotators mistakenly split single entities. |
Table 2: Comparison of Synthetic Noise Generation Methods. We use the taxonomy of Mathet et al . [ 28 ] to model four channels: (i) Loc. (localization: shift, scale), (ii) Unm. (unmatched instances: missing, extra), (iii) Cat. (category mismatches), and (iv) Fra./Com. (fragmentation/combination). Without a reference, unmatched instances are sampled symmetrically. The study of Hu et al . [ 17 ] is excluded, since we didn’t have any specifications of the generated noise. Note that Mathet et al . originally applied these concepts to text segmentation, not object detection, we present their idea for noise generation here as well since it’s the most comprehensive.
Figure 9: Impact of IoU threshold on error distribution. The plots illustrate the trade-off between unmatched instances (FP/FN) and conditionally matched instances (localization shifts, classification mistakes, combinations, and fragmentations) across the Combined, LVIS, TexBiG, and VinDr-CXR datasets.
Figure 10: Analysis of directional bias in bounding box shifts. The gray histogram represents the empirical distribution of shift angles. We tested four parametric models: Unimodal, Axial Unimodal, Axis-Centered Mixture, and Standard Quadrant Mixture. The Axis-Centered Mixture Model (brown line) yields the lowest Akaike Information Criterion (AIC 319,623), validating the hypothesis that human error is biased toward cardinal axes ( ).
Figure 11: Relationship between relative scale error and object size. The hexbin plots visualize the density of scale errors across four datasets. A clear trend is visible where smaller objects (left side of x-axis) exhibit larger relative size differences (top of the y-axis), confirming the need for size-dependent noise modeling. The (thin) dashed white vertical lines divide into the COCO object size categories (adjusted to actual image size): Small ( ), Medium ( to ), and Large ( ) for an image of .
Figure 12: Analysis of classification co-occurrence for VinDr-CXR. Left: The symmetric (empirical) count of class confusions between annotators. Right: The conditional probability matrix, highlighting that confusion is often clustered among specific, semantically related pathologies ( e.g . , Pulmonary fibrosis vs. Lung Opacity).
Figure 13: Cumulative Distribution Function (CDF) of instance areas for matched (blue) versus unmatched (orange) instances across all datasets. The distinct leftward shift of the unmatched curve indicates that missed or spurious annotations are systematically smaller than agreed-upon annotations.
Figure 14: Cumulative Distribution Function (CDF) comparing the size of "parent" (blue) instances (involved in fragmentation/combination) versus all other instances. Parent instances are systematically larger, confirming that larger objects are more prone to split/merge errors.
Figure 15: Distribution of the angular difference between two child fragments relative to their parent centroid. The peak near indicates that fragments tend to be located on opposing sides of the object, rather than clustered together.
Figure 16: Visual comparison of synthetic (solid lines) and reference (dashed lines) annotations in LVIS. Matched pairs are shown in light orange (synthetic) and green (reference). Unmatched synthetic annotations (false positives) are navy, while unmatched reference annotations (false negatives) are brown. Images are cropped to a quadratic format.
| Magnitude | 0.25 | 0.50 | 1.00 | 2.00 | 5.00 |
|---|---|---|---|---|---|
| Signal-loss ratio |
Table 3: Signal Loss Analysis (LVIS). As noise magnitude increases, the competition for finite candidates leads to saturation, where the effective error rate diverges from the theoretical sampling rate.
Figure 17: Rater-level annotations for LVIS image No. 467776 (shows only the region of interest) illustrating a conceptual edge case where class information must influence the correspondence cost. The two panels on the left show annotations from rater r1, and the two on the right from rater r2. Both raters provide two labels (cow and calf) for the very same physical instance, and both classes occupy effectively identical spatial regions. Because object detection allows overlapping instances, the spatial component alone provides no discriminative signal: all four boxes overlap nearly perfectly, and localization-based matching is therefore ambiguous. A class-aware cost function resolves this ambiguity in a principled way by preferring correspondences that preserve each rater’s intended class assignment rather than artificially swapping them. This example illustrates why incorporating class information into the matching cost is conceptually necessary for certain real-world annotation patterns, even before considering any quantitative evaluation.
Figure 18: Comparison of the two correspondence cost functions on LVIS using the Greedy solver across noise magnitudes and rater configurations. The category-lenient cost consistently achieves slightly higher filtered Rand Index and F1-scores than the baseline . While the differences are small, they are systematic and stable across all settings, indicating that incorporating class information into the cost improves correspondence robustness.
| Dataset | Annotations | Mean (95% CI) | KS Mean (95% CI) |
|---|---|---|---|
| LVIS [ 16 ] | ( ) | ( ) | |
| TexBiG [ 40 ] | ( ) | ( ) | |
| VinDr-CXR [ 32 ] | ( ) | ( ) |
Table 4: Global robustness analysis of the calibration anchor ( ) and KS statistic across 100 bootstrap iterations (All Sizes). CI denotes the 95% Confidence Interval.
| Dataset | Size | Target Anns | Mean (95% CI) | KS Mean (95% CI) |
|---|---|---|---|---|
| LVIS [ 16 ] | Small | ( ) | ( ) | |
| Medium | ( ) | ( ) | ||
| Large | ( ) | ( ) | ||
| TexBiG [ 40 ] | Small | ( ) | ( ) | |
| Medium | ( ) | ( ) | ||
| Large | ( ) | ( ) |
Table 5: Size-stratified robustness analysis using asymmetric matching. Small objects require highly lenient (higher) distance thresholds compared to large objects, exposing the scale bias of IoU.
| Distance Metric | KS Statistic | Calibration Anchor |
|---|---|---|
| Polygon IoU | 0.82 | |
| L2 Centroid | 0.78 | |
| Mask GIoU | 0.78 |
Table 6: TexBiG results for principled configuration.
Figure 19: Distribution of Observed ( ) vs. Expected ( ) disagreement for instance segmentation on TexBiG using .
Figure 20: Per-Image Agreement. The left-skewed distribution reveals that while the median image achieves perfect consensus ( ), a long tail of disagreement exists, identifying specific ambiguous samples for review.
Figure 21: Localization Sensitivity Analysis (LSA). Agreement remains stable ( ) up to but collapses at ( ). This “Cliff of Precision” quantifies the limit of human spatial consistency.
Figure 22: Class Recognition Difficulty. Structural elements like header ( ) are robust, whereas semantic definitions for equation ( ) and sub-heading ( ) require guideline refinement.
Figure 23: Annotator Vitality. coder_d ( ) acts as a strong consensus builder, while coder_e ( ) introduces systematic noise, suggesting a need for retraining.
Figure 24: Collaboration Clusters. The heatmap reveals distinct “schools of thought.” coder_f and coder_d achieve higher pairwise agreement ( ) than the global average, indicating shared implicit conventions.
Figure 25: Signal vs. Chance in 3D. The distribution of Expected Disagreement ( ) converges to a Dirac delta at . This reflects the sparsity of nodules within the vast 3D volume; unlike in 2D images, the probability of two random annotations overlapping in a voxel grid is statistically negligible.
Figure 26: Per-Image Agreement Distribution. The distribution shows a significant spread (Mean ). While a subset of scans achieves perfect consensus ( ), the heavy tail indicates that medical annotation involves inherent ambiguity not present in structural tasks like document layout.
Figure 27: Localization Sensitivity (LSA). The agreement exhibits a “Plateau” behavior, remaining stable ( ) up to . Agreement starts to drop when leaving the calibration anchors confidence interval and drops significantly at .
Figure 28: High Noise Floor. The distribution of Expected Disagreement ( ) centers at a remarkably low distance of . Due to the constrained cage environment and fixed camera, a random mouse pose spatially resembles the target pose, necessitating a strict threshold ( or as distance ) to statistically isolate the annotator signal ( ).
Figure 29: Per-Image Agreement. The high median ( ) indicates that for most frames, annotators achieve perfect consensus. However, the distribution tail reveals specific frames where occlusion or motion blur degrades reliability.
Figure 30: Localization Sensitivity (LSA). The metric remains stable across standard thresholds but drops at , revealing the upper bound of inter-annotator precision.
More in this topic
Inter-Rater Reliability
Inter-rater reliability measures how consistently different people or models judge the same items, accounting for chance and task-specific disagreement.
See all 45 papers