Position: Every Ground Truth is a Human Construction, not an Objective Truth

Charlotte Högberg; Ericka Johnson; Kiri L. Wagstaff

cs.LGMay 28, 2026

Position: Every Ground Truth is a Human Construction, not an Objective Truth

Authors: Charlotte Högberg, Ericka Johnson, Kiri L. Wagstaff

Abstract

Ground truth datasets play a fundamental role as reference values in the training and evaluation of machine learning models. This position paper argues that ground truths are not neutral objective measurements that are naturally given, but instead that they are constructed by arrangements of humans and technologies. We argue that the ML community will benefit from articulating and discussing these often invisible or unreported choices and acknowledging that reference data sets are contingent, not universal. Focusing on the situated and context-dependent nature of ground truths can improve reliability by enabling a better informed perspective on where, when, and how the datasets, and the models they have shaped, can best be used. We argue for increasing `situated reliability' which includes articulating the limits and strengths of models and their truth claims. Finally, paying more attention to the construction of ground truths can support transparency, accountability, and interdisciplinary work.

Explore similar work

Position: State-of-the-Art Claims Require State-of-the-Art Evidence
May 17, 2026YongKyung OhArtificial Intelligence Benchmarks State-Of-The-Art
Dataset Usage Inference without Shadow Models or Held-out Data
Jun 24, 2026Wojciech Łapacz, Stanisław Pawlak, Jan Dubiński +2Training Data Private Data
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3Graphical User Interface Spatial Reasoning

May 17, 2026 · cs.LGJ/K move · Enter open · S save

Position: State-of-the-Art Claims Require State-of-the-Art Evidence

YongKyung Oh

Medical & Imaging Informatics (MII), University of California, Los Angeles (UCLA)

State-of-the-Art (SOTA) claims pervade Artificial Intelligence (AI) and Machine Learning (ML) research. These claims rest on benchmark evaluations, where models are ranked by aggregate scores across tasks. Public benchmarks or leaderboards are the most visible instance, but the same structure appears in paper tables throughout the literature. However, such minimal evidence often cannot support these strong claims. We identify a widespread claim-evidence gap in AI benchmarking. Claiming SOTA carries implicit assumptions beyond mean score superiority, suggesting that a model meaningfully outperforms alternatives across most tasks. However, a marginal improvement in the mean score merely indicates a top average rank rather than true superiority. Analyzing ten cross-domain benchmarks from public leaderboards, we found that in more than half of top-model comparisons, at least one commonly assumed property of superiority does not hold. These properties include meaningful effect size, consistency across tasks, or robustness to dataset removal. Instead, aggregate gains are frequently driven by outlier datasets. This fragility persists even in benchmarks with many tasks. We argue that claim language should reflect the strength of the underlying evidence. This requires no additional experiments, only honest reporting of what results actually show, enabling more precise and interpretable comparisons across models.

Artificial Intelligence Benchmarks State-Of-The-Art Position

Open paper Ask about it PDF

Abstract

Explore similar work

Position: State-of-the-Art Claims Require State-of-the-Art Evidence

Dataset Usage Inference without Shadow Models or Held-out Data

GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models