From Ground Truth to Measurement: A Statistical Framework for Human Labeling

Robert Chew; Stephanie Eckman; Christoph Kern; Frauke Kreuter

stat.MEApr 8, 2026

From Ground Truth to Measurement: A Statistical Framework for Human Labeling

Authors: Robert Chew, Stephanie Eckman, Christoph Kern, Frauke Kreuter

Organizations: Department of AI, Data, and Product Engineering, RTI International · Social Data Science Center, University of Maryland · Department of Statistics, LMU Munich · Munich Center for Machine Learning (MCML)

Abstract

Supervised machine learning assumes that labeled data provide accurate measurements of the concepts models are meant to learn. Yet in practice, human labeling introduces systematic variation arising from ambiguous items, divergent interpretations, and simple mistakes. Machine learning research commonly treats all disagreement as noise, which obscures these distinctions and limits our understanding of what models actually learn. This paper reframes annotation as a measurement process and introduces a statistical framework for decomposing labeling outcomes into interpretable sources of variation: instance difficulty, annotator bias, situational noise, and relational alignment. The framework extends classical measurement-error models to accommodate both shared and individualized notions of truth, reflecting traditional and human label variation interpretations of error, and provides a diagnostic for assessing which regime better characterizes a given task. Applying the proposed model to a multi-annotator natural language inference dataset, we find empirical evidence for all four theorized components and demonstrate the effectiveness of our approach. We conclude with implications for data-centric machine learning and outline how this approach can guide the development of a more systematic science of labeling.

Figures & tables

Figure 1: Illustration of error components in the Global Ground Truth model . Each row demonstrates a different source of annotation error using a claim verification task. Instance-level error ( $\beta_{j}$ ): Both annotators make the same mistake on an ambiguous claim where multiple interpretations are justifiable. Between-person error ( $\rho_{i}$ ): Annotators differ consistently across trials due to stable individual characteristics (here, medical expertise). Within-person error ( $\sigma_{ijt}$ ): The same annotator produces inconsistent labels across trials due to transient factors such as fatigue or distraction.

Figure 2: Measurement model structure. The conceptual error decomposition (top) is operationalized as a latent propensity

Y^{*}_{ijt}

(middle), which is transformed via link function

\Phi

to generate observed categorical labels (bottom). The additive structure on the latent scale implies multiplicative effects on the probability/odds scale.

Figure 3: Illustration of variation components in the Individual Ground Truth model. Interpretive variation ( $\delta_{ij}$ ): Both annotators are internally consistent but apply different offensiveness thresholds. Instance ambiguity ( $\beta_{j}$ ): Context-dependence creates genuine uncertainty; neither annotator can consistently self-validate any interpretation across trials, demonstrating that some instances resist stable judgment even within individual frameworks. Between-person error ( $\rho_{i}$ ): Both annotators believe sarcasm is offensive, but Annotator 2 has a stable tendency to miss subtle cues, consistently failing self-validation. Within-person error ( $\sigma_{ijt}$ ): Annotator 2 makes a transient error in Trial 1 but shares Annotator 1’s interpretive standard. Note: “Validates?” indicates whether the annotator, upon later review, affirms their own prior judgment ( $\checkmark$ = self-consistent; $\times$ = self-contradictory). Each annotator is evaluated against their own interpretive framework, not an external ground truth.

	Random-only	Baseline Features	+ Ambiguity
Fixed Effects (OR [95% CI])
Intercept	0.22 [0.14, 0.34] ∗∗∗	0.25 [0.16, 0.37] ∗∗∗	0.06 [0.04, 0.10] ∗∗∗
Ambiguity (TRUE)	—	—	9.05 [7.31, 11.20] ∗∗∗
Lexical Overlap (1 SD)	—	0.99 [0.91, 1.07]	1.02 [0.93, 1.11]
Avg. Toks/Sent (1 SD)	—	0.98 [0.90, 1.06]	1.06 [0.96, 1.17]
Neg. Presence Flip	—	1.22 [1.02, 1.47] ∗	0.97 [0.80, 1.18]

Table 1: Mixed-effects logistic regression models predicting annotation error under the global ground truth definition.

	Random-only	Baseline Features	+ Ambiguity
Fixed Effects (OR [95% CI])
Intercept	0.08 [0.02, 0.26] ∗∗∗	0.07 [0.02, 0.24] ∗∗∗	0.09 [0.03, 0.29] ∗∗∗
Ambiguity (TRUE)	—	—	0.68 [0.50, 0.92] ∗
Lexical Overlap (1 SD)	—	1.03 [0.88, 1.20]	1.02 [0.87, 1.19]
Avg. Toks/Sent (1 SD)	—	1.14 [1.01, 1.29] ∗	1.12 [0.99, 1.28] †
Neg. Presence Flip	—	1.14 [0.81, 1.62]	1.21 [0.85, 1.71]

Table 2: Mixed-effects logistic regression models predicting annotation error under the individual ground truth definition.

	Estimate
Fixed Effect (OR [95% CI])
Intercept	9.57 [3.20, 28.59] ∗∗∗
Random Effects (Var / SD)
Document	0.94 / 0.97
Judge	1.16 / 1.08
Labeler	0.02 / 0.13

Table 3: Mixed-effects logistic regression model predicting pairwise validation outcomes (

V_{kij}

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Variable	Description and expected relation to error
ambiguity	If an instance was had more than one class deemed as valid by the annotators. High ambiguity increase error likelihood.
similarity	Semantic proximity between premise and hypothesis (semantic similarity score) using the ms-marco-MiniLM-L6-v2 cross-encoder embeddings within SentenceBERT ( Reimers and Gurevych, 2019 ) . Mid-range similarity indicates ambiguity and increases error likelihood; very high or very low similarity generally reduces disagreement.
lexical_overlap	Proportion of shared unigrams between premise and hypothesis. Medium or low overlap raises error risk because inference is required; very high overlap can also mislead annotators on non-entailments.
avg_toks_per_sent	Average sentence length across premise and hypothesis. Longer sentences impose greater cognitive load and slightly increase error rates.
fk_grade	Flesch–Kincaid readability grade ( Kincaid et al., 1975 ) . Harder or less readable text modestly increases the chance of labeling mistakes.
neg_presence_flip	Binary indicator for negation or polarity mismatch between premise and hypothesis. Presence of a negation flip increases error likelihood, especially for contradictions.

Appendix

Table 4: Instance-level covariates and their hypothesized relation to labeling error.

Error Source	Design-time Mitigation	Post-hoc Modeling / Adjustment	Interpretation under Global vs. HLV
Instance-level ( $\beta_{j}$ )	Clarify task framing and guidelines ( Parrish et al., 2023 ) ; pilot ambiguous items ( Pyatkin et al., 2023 ) ; collect rationales to separate semantic ambiguity from misunderstanding ( Jiang et al., 2023 ) .	Model instance difficulty ( Whitehill et al., 2009 ) ; down-weight/flag high-variance items; use probabilistic or soft labels for ambiguous cases ( Wu et al., 2023 ) .	Global: shared confusion to reduce. HLV: meaningful interpretive variation—retain/stratify.
Between-person ( $\rho_{i}$ )	Recruit/screen for expertise ( Beck, 2023 ) ; calibration rounds ( Klie et al., 2024 ) ; if HLV is of interest, sample diverse perspectives ( Plank, 2022 ) .	Model annotator behavior ( Yan et al., 2010 ) ; bias adjustment or clustering into interpretive communities; condition models on annotator embeddings ( Deng et al., 2023 ) .	Global: unwanted annotator bias—correct/reweight. HLV: stable, meaningful differences across individuals/groups.
Within-person ( $\sigma_{ijt}$ )	Shorter sessions/breaks ( Pandey et al., 2022 ) ; randomize item order ( Beck et al., 2024 ) ; monitor response time for fatigue/inattention ( Chew et al., 2019 ) .	Filter rushed/inconsistent responses ( Northcutt et al., 2021b ) ; include session-level effects; treat residual as baseline noise ( Lin et al., 2024 ) .	Global: random error to minimize. HLV: still noise—distinguish from systematic variation.

Appendix

Table 5: Strategies for addressing sources of labeling error across task design and modeling.

Annotator Disagreement

Annotator disagreement studies why human labels differ, how to measure and model that variation, and how it should shape AI training and evaluation.

See all 91 papers

Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data
Oct 3, 2026Marharyta ShvetsAnnotator Disagreement LLM Reliability
Eötvös Loránd University (ELTE), Budapest, Hungary
Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements
Aug 18, 2026Naoki Egami, Sooahn ShinSemiparametric Inference Latent Variable Models
Department of Political Science and Statistics and Data Science Center, Massachusetts Institute of Technology, Cambridge, MA 02139 · Department of Political Science, Massachusetts Institute of Technology, Cambridge, MA 02139
From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review
May 12, 2026Moussa Kassem Sbeyti, Joshua Holstein, Philipp Spitzer +2Uncertainty Quantification Human-in-the-Loop Annotation
1Scientific Computing Center, Karlsruhe Institute of Technology. · Institute for Information Systems, Karlsruhe Institute of Technology.

Oct 3, 2026J/K move · Enter open · S save

Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data

Marharyta Shvets

Eötvös Loránd University (ELTE), Budapest, Hungary

Annotator Disagreement LLM Reliability

Open paper AskPDF arXiv

Data quality now matters as much as compute for training language models. Much training data comes from human annotation of text, and interpretive annotation has no ground truth that could settle what is "accurate". Two lines of work respond to this. One combines annotators into a "ground truth" and measures how well they agree with each other; the other treats their disagreement as a signal. Both compare different people at one point in time. We measure something else: how well one reader reproduces their own reading of the same text over time. One expert human reader and three LLM families segmented three Sumerian myths and labelled the causal function of each segment with one of seven states. Across runs months apart, the human cut the text in much the same places but named the segments differently, in every myth. The models show no such consistent pattern: their gap between the two layers is positive in some myths and negative in others, and its size varies. The human's label changes are not random: the runs go through much the same functions but start them one step apart, while model runs start them at the same places. We argue that this pattern is a usable measure of data quality and a contamination check: a "human" annotation whose labels are as stable as its boundaries, and whose functions start in sync, looks like a model's.

Abstract

Annotator Disagreement

Explore similar work

Boundaries Agree, Labels Do Not: Intra-Annotator Dynamics as a Kind of Training Data

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

From Model Uncertainty to Human Attention: Localization-Aware Visual Cues for Scalable Annotation Review