Hard or Mislabeled? Failure Screens Enrich Vision Benchmarks for Label Errors
Organizations: Meta
Abstract
Hard vision benchmarks often keep images that strong classifiers get wrong. Since a classifier is also marked wrong when the label is wrong, such failure screens may collect label errors. We measure this on ImageNet's twice-relabeled validation set with 103 classifiers. Among the 1,892 images that all 66 supervised classifiers get wrong, 61% have a wrong label, versus 11.1% overall. The fewer images a screen lets through, the more of them are mislabeled, along approximately one curve. Fit on one model family, this curve predicts the screens of two held-out model families with a mean error of 2.4 percentage points, and replays of ImageNet-D's and ImageNet-Hard's screens within 0.1 and 6.3 points. On new pools, ImageNet-V2 and CIFAR-10, enrichment at the strictest screens is stronger than the curve predicts: on ImageNet-V2, by human relabeling, 67% of the images that all 66 classifiers get wrong are mislabeled, against 46% predicted. Screens that let models choose among only some of the classes, as ImageNet-A's does (200 of 1,000), fall below the curve. By ReImageNet, 32% of ImageNet-Hard's ImageNet images have no correct label. ScreenAudit, which comprises an audit tool, a script that rebuilds corrected answer sets for the 5,464 decided ImageNet-Hard images (of 5,646) from ReImageNet's gated release, and per-image audits of 1,198 benchmark images, will be released later.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Fam. | Checkpoint | Released by | Training data | Top-1 |
|---|---|---|---|---|---|
| tv_alexnet | A | alexnet / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 56.5 |
| tv_vgg11_bn | A | vgg11_bn / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 70.4 |
| tv_vgg16_bn | A | vgg16_bn / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 73.4 |
| tv_vgg19_bn | A | vgg19_bn / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 74.2 |
| tv_squeezenet1_1 | A | squeezenet1_1 / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 58.2 |
| tv_resnet18 | A | resnet18 / IMAGENET1K_V1 | PyTorch/torchvision | IN-1k | 69.7 |
Benchmark Auditing
Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.
See all 109 papers