cs.LGNov 17, 2026

How Many Training Labels Are Wrong? Auditing a Fitted Model through Gradient Allocation

Authors: Yuchen Li, Zongqi Fan, Xintong Li, Nguyen H. Tran, Ken-Tye Yong

Organizations: The University of Sydney

Abstract

How many labels in a training set are wrong, and which ones? The usual answers read a held-out set or the model's outputs. A held-out reading depends on how the held-out set was drawn, and an output-based reading fails once the model has fitted its wrong labels, because the loss of a fitted wrong label is small and the prediction agrees with it. We read the answer from the per-example gradients of a single trained model. Each example, together with its label, asks the network to change in a particular way. We normalise the magnitudes of its gradient into a gradient allocation and compare it with the mean allocation of every class. An example is flagged when relabelling it as its nearest other class makes its allocation fit better than under its own label, and the share of flagged examples is the reading of the noise rate. The rule has no threshold and needs no clean data, no held-out set and no further training. After the model has fitted 99-100% of its wrong labels, the allocation still separates wrong from right labels with an AUC of 0.94-0.96 on the noisiest human labels of CIFAR-10N, where a feature-space margin falls to 0.79-0.84 and the loss, the gradient norm and the self-influence of an example to 0.73-0.76. On such a fitted model, readings of the noise rate from predictions or feature margins fall to 0-6% at true rates of 8-58% on CIFAR-10 with ViT-S, while the audit reads 54-96% of the true rate, and at least 93% of the examples it counts carry wrong labels. On a briefly trained model, before the wrong labels are fitted, the reading lies within 1.7 percentage points of the true noise rate on CIFAR-10 from 0% to 58% noise. Confident learning, which trains four further models, separates wrong labels about as well (AUC 0.925-0.998 against 0.900-0.995) and mostly reads the noise rate more closely, at a lower precision. An audit of 5,000 examples takes 14 to 27 seconds on one GPU.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. When Can Old Evaluations Certify a New Model? Label-Efficient Release Decisions under Evaluator Drift

    Sep 26, 2026Joyanta Jyoti Mondal, Mridul Banik, Md. Shifatul Ahsan Apurba +1Sequential Hypothesis TestingLLM Auditing

  2. Bias Leaves a Gradient Trail: Label-Free Bias Identification via Gradient Probes on Concept Decompositions

    May 27, 2026Thomas Vitry, Kieran Edgeworth, Stefan Wermter +1Debiased MLSpurious Correlation Robustness