cs.LGApr 12, 2025

Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotation

Authors: Hsuan Wei Liao, Christopher Klugmann, Daniel Kondermann, Rafid Mahmood

Organizations: Telfer School of Management, University of Ottawa · Quality Match · NVIDIA

Abstract

Data annotation is an essential but costly aspect of developing machine learning-based technology, requiring multiple human workers to repeatedly label each data point in a data set. We explore the tradeoff between annotation accuracy and cost by identifying minority reports---worker responses disagreeing with the reference majority-vote label---that may indicate unnecessary redundancy, and can therefore be proactively pruned before they are executed. We develop a theoretical framework for choosing the number of repeated annotations required for a task and pruning policies at a target label quality. We estimate minority-report risk on two computer vision datasets re-annotated through a professional platform, then evaluate a rolling policy on the assignment stream to prune over 35% of repeats while preserving more than 99.9% of reference majority-vote labels.

Figures & tables

Explore similar work

CardsList
  1. A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection

    Jul 27, 2026Gabriel Singer, Samuel Gruffaz, Olivier Vo Van +2Class-Imbalanced LearningClassification

  2. Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation

    May 15, 2026Sunil Kothari, Sumukha Sharma Thoppanahalli Chandramouli, Naman Khandelwal +8Human-in-the-Loop Annotation

  3. From Ground Truth to Measurement: A Statistical Framework for Human Labeling

    Apr 8, 2026Robert Chew, Stephanie Eckman, Christoph Kern +1Annotator DisagreementHuman-in-the-Loop Annotation