K$α$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks

David Tschirschwitz; Volker Rodehorst

cs.CVMar 28, 2026

K $α$ LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks

Authors: David Tschirschwitz, Volker Rodehorst

Organizations: Bauhaus-Universität Weimar, Germany

Abstract

Progress in object detection benchmarks is stagnating. It is limited not by architectures but by the inability to distinguish model improvements from label noise. To restore trust in benchmarking the field requires rigorous quantification of annotation consistency to ensure the reliability of evaluation data. However, standard statistical metrics fail to handle the instance correspondence problem inherent to vision tasks. Furthermore, validating new agreement metrics remains circular because no objective ground truth for agreement exists. This forces reliance on unverifiable heuristics. We propose K $α$ LOS (KALOS), a unified meta-algorithm that generalizes the "Localization First" principle to standardize dataset quality evaluation. By resolving spatial correspondence before assessing agreement, our framework transforms complex spatio-categorical problems into nominal reliability matrices. Unlike prior heuristic implementations, K $α$ LOS employs a principled, data-driven configuration; by statistically calibrating the localization parameters to the inherent agreement distribution, it generalizes to diverse tasks ranging from bounding boxes to volumetric segmentation or pose estimation. This standardization enables granular diagnostics beyond a single score. These include annotator vitality, collaboration clustering, and localization sensitivity. To validate this approach, we introduce a novel and empirically derived noise generator. Where prior validations relied on uniform error assumptions, our controllable testbed models complex and non-isotropic human variability. This provides evidence of the metric's properties and establishes K $α$ LOS as a robust standard for distinguishing signal from noise in modern computer vision benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Visual Abstract.

Figure 2: The K $\alpha$ LOS Framework. (1) Input: The pipeline processes active disagreement between raters. (2) Principled Configuration: Data-driven calibration selects the optimal distance function ( $d_{\text{loc}}$ ) and threshold ( $\tau^{*}$ ) by maximizing the statistical separation between observed signal ( $D_{o}$ ) and chance noise ( $D_{e}$ ). (3) Execution: A solver transforms continuous spatial data into a discrete reliability matrix. Crucially, the completeness assumption encodes missed instances as no_object , treating them as existence disagreement (false positives/negatives) rather than missing data. (4) Downstream Diagnostics: This standardization enables rigorous quality checks, visualizing Localization Sensitivity, Class Difficulty, and Annotator Vitality ( $V_{r}$ ).

Figure 3: Performance of different correspondence solvers averaged across LVIS, TexBiG and VinDr-CXR. A simple filtered Rand Index variation was used due to the constrained clustering setup, where limited and partially overlapping clusters, together with non-exhaustive ground-truth coverage, rendered more advanced evaluation metrics infeasible or unreliable for assessing correspondence accuracy.

Figure 4: F1-score comparison. Greedy performs best.

Figure 5: Failure-type comparison across correspondence solvers and noise magnitudes using raincloud plots. TP shows correctly matched pairs, Missed Opportunities are correct pairs the solver failed to match, and FP captures spurious matches where no true pair exists. Cuckoo Eggs indicate cases where a correct match was displaced by an incorrect one, effectively combining a FN with a FP and revealing when class disagreement overrides

\psi_{\text{soft}}

Eq. 6 function.

Figure 6: Return on investment for using more annotators. Majority voting principal applies. Annotators that generally follow the same annotation convention increase consensus, independent of increasing magnitude.

Figure 7: Collaboration-cluster size impact analysis between two groups of A and B raters, denoted as (A–B) for the group sizes. LVIS is used to generate synthetic annotations from two anonymous, interchangeable reference raters.

Dataset	Task(s)	Amount of Raters per Image/Scene	Annotator Type	Rater Identification + Intra-Rater Evaluation	Guideline and Annotation Process	Automated Proposals
TexBiG [ 40 ]	Instance Segmentation	3–4 (train), 5 (test)	Mixed (Experts + Non-Experts)	Yes + No	Same	no
VinDr-CXR [ 32 ]	Object Detection	3 (train), 5 (test)	Experts (Radiologists)	Yes + No	Same	no
LVIS Consistency Subset [ 16 ]	Instance Segmentation	2	Non-Experts	No + No	Same	no
COCO Reannotated [ 27 ]	Object Detection	2	Non-Experts	No + No	Different	yes
Open Images Reannotated [ 27 ]	Object Detection	2	Non-Experts	No + No	Different	yes
NuCLS [ 2 ]	Instance Segmentation	11–39	Mixed (Experts + Non-Experts)	Yes + Yes	Same	yes

Table 1: Overview of datasets with repeated annotations. Bold names are the ones used for our main analysis.

Figure 8: Three step noise generator: 1) Empirical data is extracted. 2) Data is used to fit parametric models or estimate them using foundational models as validated proxies. 3) The four separate components use a hierarchical composition to generate synthetic annotations based on reference annotations ( e.g . human annotated data).

Study	Type	Modelling	Hypothesis (Assumptions)
Agnew et al . [ 1 ]	Loc.	Uniform expansion (0-10 pixel buffering) of BBox	Noise is primarily expansion (outward shift).
	Loc.	Radial Gaussian noise ( $1-5\mu$ ) on mask vertices	Vertex precision varies normally; errors localize to edges.
Mathet et al . [ 28 ] (CST)	Unm.	Random dropout & addition (class is based on marginal dist.)	Errors are probabilistic; False Positives follow dataset statistics.
	Loc.	Random uniform shift	Shift magnitude is proportional to object length.
	Cat.	Confusion matrix (prevalence & overlap based)	Errors correlate with class frequency and semantic overlap.
	Fra.	Randomly select element and split it	Annotators mistakenly split single entities.

Table 2: Comparison of Synthetic Noise Generation Methods. We use the taxonomy of Mathet et al . [ 28 ] to model four channels: (i) Loc. (localization: shift, scale), (ii) Unm. (unmatched instances: missing, extra), (iii) Cat. (category mismatches), and (iv) Fra./Com. (fragmentation/combination). Without a reference, unmatched instances are sampled symmetrically. The study of Hu et al . [ 17 ] is excluded, since we didn’t have any specifications of the generated noise. Note that Mathet et al . originally applied these concepts to text segmentation, not object detection, we present their idea for noise generation here as well since it’s the most comprehensive.

Figure 9: Impact of IoU threshold on error distribution. The plots illustrate the trade-off between unmatched instances (FP/FN) and conditionally matched instances (localization shifts, classification mistakes, combinations, and fragmentations) across the Combined, LVIS, TexBiG, and VinDr-CXR datasets.

Figure 10: Analysis of directional bias in bounding box shifts. The gray histogram represents the empirical distribution of shift angles. We tested four parametric models: Unimodal, Axial Unimodal, Axis-Centered Mixture, and Standard Quadrant Mixture. The Axis-Centered Mixture Model (brown line) yields the lowest Akaike Information Criterion (AIC

\approx

319,623), validating the hypothesis that human error is biased toward cardinal axes (

0^{\circ},90^{\circ},180^{\circ},270^{\circ}

Figure 11: Relationship between relative scale error and object size. The hexbin plots visualize the density of scale errors across four datasets. A clear trend is visible where smaller objects (left side of x-axis) exhibit larger relative size differences (top of the y-axis), confirming the need for size-dependent noise modeling. The (thin) dashed white vertical lines divide into the COCO object size categories (adjusted to actual image size): Small ( $<32^{2}$ ), Medium ( $32^{2}$ to $96^{2}$ ), and Large ( $>96^{2}$ ) for an image of $640\times 480$ .

Figure 12: Analysis of classification co-occurrence for VinDr-CXR. Left: The symmetric (empirical) count of class confusions between annotators. Right: The conditional probability matrix, highlighting that confusion is often clustered among specific, semantically related pathologies ( e.g . , Pulmonary fibrosis vs. Lung Opacity).

Figure 13: Cumulative Distribution Function (CDF) of instance areas for matched (blue) versus unmatched (orange) instances across all datasets. The distinct leftward shift of the unmatched curve indicates that missed or spurious annotations are systematically smaller than agreed-upon annotations.

Figure 14: Cumulative Distribution Function (CDF) comparing the size of "parent" (blue) instances (involved in fragmentation/combination) versus all other instances. Parent instances are systematically larger, confirming that larger objects are more prone to split/merge errors.

Figure 15: Distribution of the angular difference between two child fragments relative to their parent centroid. The peak near

180^{\circ}

indicates that fragments tend to be located on opposing sides of the object, rather than clustered together.

Panel of Figure 16 — Figure 16: Visual comparison of synthetic (solid lines) and reference (dashed lines) annotations in LVIS. Matched pairs are shown in light orange (synthetic) and green (reference). Unmatched synthetic annotations (false positives) are navy, while unmatched reference annotations (false negatives) are brown. Images are cropped to a quadratic format.

Magnitude $\lambda$	0.25	0.50	1.00	2.00	5.00
Signal-loss ratio	$0.2\%$	$1.3\%$	$4.2\%$	$10.1\%$	$28.3\%$

Table 3: Signal Loss Analysis (LVIS). As noise magnitude

\lambda

increases, the competition for finite candidates leads to saturation, where the effective error rate diverges from the theoretical sampling rate.

Figure 17: Rater-level annotations for LVIS image No. 467776 (shows only the region of interest) illustrating a conceptual edge case where class information must influence the correspondence cost. The two panels on the left show annotations from rater r1, and the two on the right from rater r2. Both raters provide two labels (cow and calf) for the very same physical instance, and both classes occupy effectively identical spatial regions. Because object detection allows overlapping instances, the spatial component alone provides no discriminative signal: all four boxes overlap nearly perfectly, and localization-based matching is therefore ambiguous. A class-aware cost function resolves this ambiguity in a principled way by preferring correspondences that preserve each rater’s intended class assignment rather than artificially swapping them. This example illustrates why incorporating class information into the matching cost is conceptually necessary for certain real-world annotation patterns, even before considering any quantitative evaluation.

Figure 18: Comparison of the two correspondence cost functions on LVIS using the Greedy solver across noise magnitudes and rater configurations. The category-lenient cost

\psi_{\text{soft}}

consistently achieves slightly higher filtered Rand Index and F1-scores than the baseline

\psi_{\text{neg}}

. While the differences are small, they are systematic and stable across all settings, indicating that incorporating class information into the cost improves correspondence robustness.

Dataset	Annotations	$\tau^{*}$ Mean (95% CI)	KS Mean (95% CI)
LVIS [ 16 ]	$100,480$	$0.654$ ( $[0.643,0.668]$ )	$0.798$ ( $[0.786,0.807]$ )
TexBiG [ 40 ]	$52,372$	$0.435$ ( $[0.420,0.455]$ )	$0.803$ ( $[0.796,0.810]$ )
VinDr-CXR [ 32 ]	$36,096$	$0.996$ ( $[0.996,0.997]$ )	$0.684$ ( $[0.675,0.692]$ )

Table 4: Global robustness analysis of the calibration anchor (

\tau^{*}

) and KS statistic across 100 bootstrap iterations (All Sizes). CI denotes the 95% Confidence Interval.

Dataset	Size	Target Anns	$\tau^{*}$ Mean (95% CI)	KS Mean (95% CI)
LVIS [ 16 ]	Small	$35,574$	$0.976$ ( $[0.973,0.978]$ )	$0.773$ ( $[0.750,0.795]$ )
	Medium	$38,558$	$0.748$ ( $[0.725,0.771]$ )	$0.863$ ( $[0.855,0.870]$ )
	Large	$26,348$	$0.488$ ( $[0.473,0.501]$ )	$0.889$ ( $[0.883,0.894]$ )
TexBiG [ 40 ]	Small	$12,769$	$0.695$ ( $[0.644,0.764]$ )	$0.904$ ( $[0.894,0.918]$ )
	Medium	$15,376$	$0.607$ ( $[0.570,0.651]$ )	$0.880$ ( $[0.870,0.889]$ )
	Large	$24,227$	$0.117$ ( $[0.113,0.121]$ )	$0.843$ ( $[0.834,0.851]$ )

Table 5: Size-stratified robustness analysis using asymmetric matching. Small objects require highly lenient (higher) distance thresholds compared to large objects, exposing the scale bias of IoU.

Distance Metric	KS Statistic $\uparrow$	Calibration Anchor $\tau^{*}$
Polygon IoU	0.82	$0.53$
L2 Centroid	0.78	$0.02$
Mask GIoU	0.78	$0.28$

Table 6: TexBiG results for principled configuration.

Figure 19: Distribution of Observed (

D_{o}

) vs. Expected (

D_{e}

) disagreement for instance segmentation on TexBiG using

1-\text{IoU}

Figure 20: Per-Image Agreement. The left-skewed distribution reveals that while the median image achieves perfect consensus (

\alpha=1.0

), a long tail of disagreement exists, identifying specific ambiguous samples for review.

Figure 21: Localization Sensitivity Analysis (LSA). Agreement remains stable (

\alpha\approx 0.94\to 0.90

) up to

\tau_{s}=0.5

but collapses at

\tau_{s}=0.9

(

\alpha=0.35

). This “Cliff of Precision” quantifies the limit of human spatial consistency.

Figure 22: Class Recognition Difficulty. Structural elements like header (

\alpha=0.99

) are robust, whereas semantic definitions for equation (

\alpha=0.45

) and sub-heading (

\alpha=0.46

) require guideline refinement.

Figure 23: Annotator Vitality. coder_d (

V_{r}=+0.04

) acts as a strong consensus builder, while coder_e (

V_{r}=-0.02

) introduces systematic noise, suggesting a need for retraining.

Figure 24: Collaboration Clusters. The heatmap reveals distinct “schools of thought.” coder_f and coder_d achieve higher pairwise agreement ( $0.95$ ) than the global average, indicating shared implicit conventions.

Figure 25: Signal vs. Chance in 3D. The distribution of Expected Disagreement (

D_{e}

) converges to a Dirac delta at

1.0

. This reflects the sparsity of nodules within the vast 3D volume; unlike in 2D images, the probability of two random annotations overlapping in a voxel grid is statistically negligible.

Figure 26: Per-Image Agreement Distribution. The distribution shows a significant spread (Mean

\alpha=0.37

). While a subset of scans achieves perfect consensus (

\alpha=1.0

), the heavy tail indicates that medical annotation involves inherent ambiguity not present in structural tasks like document layout.

Figure 27: Localization Sensitivity (LSA). The agreement exhibits a “Plateau” behavior, remaining stable (

\alpha\approx 0.37

) up to

\tau_{s}=0.50

. Agreement starts to drop when leaving the calibration anchors confidence interval and drops significantly at

\tau_{s}=0.8

Figure 28: High Noise Floor. The distribution of Expected Disagreement (

D_{e}

) centers at a remarkably low distance of

0.18

. Due to the constrained cage environment and fixed camera, a random mouse pose spatially resembles the target pose, necessitating a strict threshold (

\tau_{s}^{*}=0.92

or as distance

\tau^{*}=0.08

) to statistically isolate the annotator signal (

D_{o}

Figure 29: Per-Image Agreement. The high median (

\alpha=1.0

) indicates that for most frames, annotators achieve perfect consensus. However, the distribution tail reveals specific frames where occlusion or motion blur degrades reliability.

Figure 30: Localization Sensitivity (LSA). The metric remains stable across standard thresholds but drops at

\tau_{s}=0.95

, revealing the upper bound of inter-annotator precision.

Inter-Rater Reliability

Inter-rater reliability measures how consistently different people or models judge the same items, accounting for chance and task-specific disagreement.

See all 45 papers

Oct 14, 2026

Quantifying Annotation Agreement in Object Detection with Optimal Transport

Reliable validation of machine learning models is a crucial step in designing robust and trustworthy systems. Yet validation relies on annotated test sets treated as ground truth, even though annotations can be noisy or ambiguous. In object detection, sources may disagree on object presence, box location, or class label. Standard agreement metrics usually binarize box overlap before computing a summary score, obscuring the degree and structure of disagreement between annotation sets. Here, we study an Optimal Transport (OT)-based metric for measuring agreement between object detection annotations. Instead of binarizing bounding box overlap, this metric assigns graded costs to matched and unmatched labeled boxes. Through theoretical analysis and experiments on real and synthetically perturbed annotations, we show that OT captures graded changes in agreement and distinguishes annotation pairs that thresholded metrics score identically. Our results suggest that OT provides a richer measure of annotation agreement in object detection, especially for domains with many small or ambiguous objects. Such agreement estimates can support more reliable validation when labels are noisy or conflicting.

Kirill Prokhorov, Alexandr A. Kalinin

Openface.io, Vilnius, Lithuania · Broad Institute of MIT and Harvard, Cambridge, MA, USA

Inter-Rater Reliability Object Detection Optimal Transport

Ask PDF

May 11, 2026

Task-Agnostic Noisy Label Detection via Standardized Loss Aggregation

Noisy labels are common in large-scale medical imaging datasets due to inter-observer variability and ambiguous cases. We propose a statistically grounded and task-agnostic framework, Standardized Loss Aggregation (SLA), for detecting noisy labels at the sample level. SLA quantifies label reliability by aggregating standardized fold-level validation losses across repeated cross-validation runs. This formulation generalizes discrete hard-counting schemes into a continuous estimator that captures both the frequency and magnitude of performance deviations, yielding interpretable and statistically stable noisiness scores. Experiments on a public fundus dataset demonstrate that SLA consistently outperforms the hard-counting baseline across all noise levels and converges substantially faster, especially under low noise ratios where subtle loss variations are informative. Samples with high SLA scores indicate potentially ambiguous or mislabeled cases, guiding efficient re-annotation and improving dataset reliability for any classification task.

Inhyuk Park, Doohyun Park

VUNO Inc.

Noisy-Label Learning Learning with Noisy Labels

Ask PDF

Oct 27, 2026

Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability

Reliable evaluation of inter-annotator agreement (IAA) is important for assessing the consistency of annotation processes. While established IAA metrics such as Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are commonly applied in single-label settings, multi-label tasks require additional decisions about how full label sets, joint selections, and joint omissions should be compared. In this work, we present Multi-Label Agreement (MLA), a flexible framework for evaluating agreement in multi-label annotation tasks. MLA preserves the full binary label vectors assigned to each item and uses average pairwise Hamming agreement, thereby accounting for both joint selections and joint omissions. Its pooled-marginal form is not a mathematically independent coefficient: it is a chance-disagreement-weighted average of label-wise Fleiss/Scott-type coefficients and is closely related to Krippendorff's alpha with normalized Hamming distance. The framework supports pooled-marginal, Gwet AC1, and fixed-baseline normalizations, yields item-level, label-level, and corpus-level results, and includes a conditional minimum-agreement diagnostic. Through a worked example and Monte Carlo simulations, we show how the resulting values depend on label prevalence, the declared label inventory, and the selected model of expected agreement. We also provide a Python implementation. MLA thus offers a transparent and interpretable framework for multi-label agreement analysis without claiming that one normalization is universally preferable.

Tim Menzner, Jochen L. Leidner

Center for Responsible Artificial Intelligence (CRAI), Coburg University of Applied Sciences

Inter-Rater Reliability

Ask PDF

Abstract

Inter-Rater Reliability

Explore similar work

Quantifying Annotation Agreement in Object Detection with Optimal Transport

Task-Agnostic Noisy Label Detection via Standardized Loss Aggregation

Multi-Label Agreement: A Flexible Framework for Inter-Annotator Reliability