math.STJun 24, 2022

How many labelers do you have? A closer look at gold-standard labels

Authors: Chen Cheng, Hilal Asi, John Duchi

Organizations: Department of Statistics, Stanford University · Department of Electrical Engineering, Stanford University · Departments of Statistics and Electrical Engineering, Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of "true" label. We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models more feasible than it is with cleaned labels. The entire story, however, is subtle, and the contrasts between aggregated and fuller label information depend on the particulars of the problem, where estimators that use aggregated information exhibit robust but slower rates of convergence, while estimators that can effectively leverage all labels converge more quickly if they have fidelity to (or can learn) the true labeling process. The theory makes several predictions for real-world datasets, including when non-aggregate labels should improve learning performance, which we test to corroborate the validity of our predictions.

Figures & tables

 MLE-type estimator ( 2 )majority-vote estimator ( 3 )
single linkClassification : r(m)≍1mr(m)\asymp\frac{1}{m}Classification: r(m)≍1mr(m)\asymp\frac{1}{\sqrt{m}}
(well-specified)Calibration : t(m)=1t(m)=1Calibration: t(m)≍mt(m)\asymp\sqrt{m}
mis-specified orCalibration: t(m)≍1t(m)\asymp 1Calibration: t(m)≍mt(m)\asymp\sqrt{m}
multiple linksClassification: r(m)≍1r(m)\asymp 1Classification : r(m)≍1/mr(m)\asymp 1/\sqrt{m}
Table 1: Comparison of multi-label and majority-vote estimators for X∼N(0,Id)X\sim\mathsf{N}(0,I_{d}) according to the scalings in classification error r(m)r(m) and calibration error t(m)t(m) that the limits ( 3 ′ ) identify. If t(m)≫1t(m)\gg 1 , the model is uncalibrated; if the rate r(m)r(m) is small, the model achieves lower asymptotic variance. Boldface indicates the associated estimator is (rate) optimal.
Figure 1: Experiments on BlueBirds dataset. (a) Classification error. (b) Calibration error ∣logit(p~)−logit(p)∣|\mathop{\rm logit}(\widetilde{p})-\mathop{\rm logit}(p)| with ResNet features reduced via PCA to dimension d=25d=25 . Error bars show 2 standard error confidence bands over T=100T=100 trials.
Figure 2: Experiments on CIFAR-10H dataset. (a) Classification error. (b) Calibration error ∣logit(p~)−logit(p)∣|\mathop{\rm logit}(\widetilde{p})-\mathop{\rm logit}(p)| with ResNet features reduced via PCA to dimension d=40d=40 . Error bars show 2 standard error confidence bands over T=100T=100 trials.
Figure 3: Experiments on the semisynthetic CIFAR-10 dataset. The median labeler accuracy represents the median pp of pi=max⁡ysoftmax(αf(xi))yp_{i}=\max_{y}\textup{softmax}(\alpha f(x_{i}))_{y} on the training data. Results are averaged over 2020 trials, and vertical axes give mis-classification rate from (synthetic) ground-truth labels on held-out test set. Legend keys correspond to maximum likelihood θ^n,mmle\widehat{\theta}^{\textup{mle}}_{n,m} (MLE), majority vote θ^n,mmv\widehat{\theta}^{\textup{mv}}_{n,m} (MV), hard-labeled Dawid-Skene (DS) and GLAD crowdsourced estimators θ^n,m\textscds\widehat{\theta}^{\textsc{ds}}_{n,m} and θ^n,m\textscglad\widehat{\theta}^{\textsc{glad}}_{n,m} , and soft-labeled Dawid-Skene (DS prob) and GLAD (GLAD prob) estimators θ^n,m\textscds−p\widehat{\theta}^{\textsc{ds-p}}_{n,m} and θ^n,m\textscglad−p\widehat{\theta}^{\textsc{glad-p}}_{n,m} . (a) Comparison of methods using hard labels. (b) Comparison with crowdsourced estimated soft-labels. (c) Number of labelers mm versus test error for fixed median accuracy p=.105p=.105 (noisy labelers). (d) Number of labelers mm versus test error for fixed median accuracy p=.42p=.42 . Both (c) and (d) report 95% error bars over the trials.

Explore similar work

CardsList

  1. How Many Labels Does Model Choice Need? Certificates and Budgets for Selective Prediction

    Sep 16, 2026Tetsuji KuboyamaModel SelectionConfidence Intervals

    Computer Centre, Gakushuin University

  2. The Label Defines the Timescale: Trait-State Limits of Temporal-Aggregate Learning

    Aug 3, 2026Xizhe ZhangTimescaleTraining Time

  3. No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference

    May 26, 2025Pranav Mani, Peng Xu, Zachary C. Lipton +1Finite-SamplePrediction Error

    Abridge AI, San Francisco, CA, USA · Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA