Doomed to Re-Annotate, Forever: The ImageNet Story

Illia Volkov; Nikita Kisel; Tetiana Mishkina; Klara Janouskova; Jiri Matas

cs.CVAug 13, 2026

Doomed to Re-Annotate, Forever: The ImageNet Story

Authors: Illia Volkov, Nikita Kisel, Tetiana Mishkina, Klara Janouskova, Jiri Matas

Organizations: Visual Recognition Group, Faculty of Electrical Engineering Czech Technical University in Prague

Abstract

Top-1 accuracy on ImageNet-1k remains the most commonly reported metric in visual recognition. Quality issues with the dataset have been repeatedly reported, yet the original 2012 noisy labels are still predominantly used. The paper presents a comprehensive effort, which goes well beyond prior correction attempts, towards obtaining accurate and complete ImageNet-1k validation set annotations. The result, ReImageNet, includes multilabel correction, object localization, revised class definitions, and semantic attributes (text-recognition, rendition, reflection, crowd, dominant). The reannotation reveals that approximately 12% of the original ImageNet-1k labels are incorrect, 33.3% of images are multilabel and 3.8% contain no object from an ImageNet-1k class. With the new labels, top-1 accuracy increases by up to 1.2% for supervised models and by 5-6% for MLLMs. We argue that annotation at ImageNet scale cannot realistically be completed in one pass, as errors and definitional issues are discovered only through annotating, and we build our pipeline around repeated refinement and error checking. We observed that human and LLM collaboration with appropriate tooling represents the current quality ceiling for annotation at this scale. ImageNet-1k issues propagate into its derivative test sets, indicating that the problem is structural rather than specific to any single benchmark. All annotations, class definitions, guidelines, and analysis code have been publicly released. Project page: https://vrg.fel.cvut.cz/reimagenet Annotations: https://huggingface.co/datasets/vrg-prague/ReImageNet Code: https://github.com/klarajanouskova/ImageNet

Figures & tables

Panel of Figure 1 — Figure 1: ImGT – original single-label annotations (left) and ReGT – new multilabel localized reannotations (right). For both (a) and (b), ImGT is correct but incomplete – ReGT labels at least all objects a person notices within 5 seconds. In (b), television is a reflection visible in the mirror (see Fig. 2 ). Image (c) shows an obvious labelling error. Image (d) needed a fine-grained correction: the two snake species are visually near-identical to non-experts due to convergent evolution; the ReGT was confirmed via reverse image search, tracing the photograph to a Utah-based wildlife biologist.

Panel of Figure 2 — Figure 2: Annotation extensions: attributes . Objects may have multiple, only one is visualized here.

Panel of Figure 3 — Figure 3: Small-scale objects and their zoomed-in crops (right). Such objects are recognizable only from the context. Initially, we aimed to annotate every visible object in the image; small objects, as shown in (a) and (b), right image, discovered during annotation, led to a revision of the definition of what to annotate, see Sec. 3 . The minivan bounding box has the crowd attribute.

Figure 5: Distributions of the number of labels and bounding boxes per image, log-scaled.

	ImGT	ReGT
	$\mathrm{{A{\setminus}N}_{48114}}$	$\mathrm{{A{\setminus}N}_{48114}}$	$\mathrm{S_{31448}}$	$\mathrm{{S+}_{28685}}$	$\mathrm{{S-}_{2763}}$	$\mathrm{M_{16666}}$	$\mathrm{{M+}_{15279}}$	$\mathrm{{M-}_{1387}}$
GPT-5.4 [ 35 ]	72.54	77.64 +5.10	77.86	80.92	46.11	77.22	80.59	40.16
Qwen3-VL [ 57 ]	70.95	77.04 +6.09	75.97	78.88	45.71	79.08	82.26	44.05
Gemma 4 [ 10 ]	69.48	74.87 +5.39	74.48	77.23	45.93	75.60	78.63	42.25
SigLIP [ 59 ]	84.87	87.86 +2.99	88.54	92.86	43.68	86.60	91.09	37.06
SigLIP2 [ 43 ]	85.82	88.43 +2.61	89.11	93.55	42.96	87.16	91.75	36.63

Table 1: Recognition accuracy

\mathrm{Acc}

on image categories defined in Fig. 4 for mllm (top), vlm (middle) and supervised models (bottom). Subscripts denote the number of images in a category. Images in

\mathrm{N}

contain no valid ImageNet-1k class and are excluded from evaluation. Increase in accuracy on ReGT with respect to ImGT is shown for

\mathrm{{A{\setminus}N}_{48114}}

Model	$\mathrm{S}_{31448}$	$\mathit{Crowd}_{1931}$	$\mathit{Refl}_{228}$	$\mathit{Rend}_{410}$	$\mathit{Text}_{284}$	$\mathit{Mix}_{88}$	$\mathit{\emptyset}_{28507}$
GPT-5.4	77.86	77.94 +0.08	75.44 -2.42	75.61 -2.25	87.32 +9.46	71.59 -6.27	77.83 -0.03
Qwen3-VL	75.97	78.72 +2.75	77.19 +1.22	74.63 -1.34	91.55 +15.58	79.55 +3.58	75.62 -0.35
Gemma 4	74.48	77.63 +3.15	74.12 -0.36	72.93 -1.55	87.32 +12.84	67.05 -7.43	74.18 -0.30
SigLIP	88.54	89.95 +1.41	90.35 +1.81	84.88 -3.66	95.42 +6.88	90.91 +2.37	88.40 -0.14
SigLIP2	89.11	89.59 +0.48	91.23 +2.12	84.88 -4.23	96.48 +7.37	89.77 +0.66	89.04 -0.07
SigLIP2-g	89.69	89.23 -0.46	93.42 +3.73	85.85 -3.84	96.13 +6.44	94.32 +4.63	89.67 -0.02

Table 2: Accuracy

\mathrm{Acc}

(with ReGT) for images in the

\mathrm{S}

category per visual attribute . Subscripts denote the number of images in a subset. Increase or decrease in accuracy with respect to

\mathrm{S}

are shown.

\mathit{Mix}

and

\emptyset

denote images containing multiple attributes, and no attributes, respectively.

Table 7

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Multilabel objects in ImageNet-1k, i.e. objects with multiple valid labels. Some objects exhibit strict hierarchy, where a class is a subclass of another: e.g. (a) all agaric are mushroom . Super-classes are added automatically, so annotators only need to label the most specific class. Other classes show more complex relations: e.g. (b) this wall clock is also an analog clock , but not all wall clocks are analog, (c) this specific cardigan is made of wool , but not all cardigans are, (d) this CRT screen is a monitor , but not all CRTs are. In these cases, the annotators need to label all of the classes.

Panel of Figure 7 — Figure 7: Images containing objects with the rendition attribute. The attribute is used when the depicted object visually mimics an instance of the original class, but appears as a toy, drawing, sculpture, or other representation. In some categories ( e.g. triceratops ), real photographs cannot exist, and all images are renditions. In (d), cucumber is a multilabel object, see Fig. 6 .

Panel of Figure 8 — Figure 8: Images containing bounding boxes with the crowd attribute. The attribute is used when the image depicts five or more instances of the same class, all of which share identical values for the other (optional) attributes, to simplify the annotation process.

Panel of Figure 9 — Figure 9: Images containing objects with the text-recognition attribute, where image classification relies on reading and interpreting text.

Panel of Figure 10 — Figure 10: Images containing objects with the reflection attribute. This attribute is used regardless of whether the reflected object is visible (b) or not (a, c). Compact camera is not an ImageNet-1k class.

Panel of Figure 11 — Figure 11: Examples illustrating the dominant attribute. Dominance is entirely context-dependent and does not correlate with object size: (a) doormat occupies most of the image yet is not dominant , (b) hockey puck is small but immediately draws attention and is marked dominant , (c) lens cap is large yet not dominant given the surrounding scene, and (d) Persian cat is marked dominant despite sharing the image with carton .

Panel of Figure 12 — Figure 12: The role of context in image recognition. Both images show a cream-like white substance on skin. In (a), the festival-like background and characteristic stripes on the face suggest it is body paint. In (b), a woman in a bikini on the beach and wearing a sun hat suggests it is sunscreen . Either interpretation could apply to both images. We can only make an educated guess based on the context.

Figure 13: Annotation application workflow. Screenshots illustrate the annotation loop supported by the application. (1) Select an image from the task queue, organized by class and browsable in a grid view for class-level inspection and definition review. (2) Switch to the image-level mode to draw bounding boxes, add attributes , edit existing annotations, and reference model predictions. (3) After completing the current image, proceed to the next, repeating the loop.

Panel of Figure 14 — Figure 14: Annotation application update for the verification phase. The core functionality remains the same as in the main phase; the key changes are the replacement of OWLv2/OpenCLIP predictions with mllm +SAM3 proposals and the addition of class definitions directly in the prediction panel.

Panel of Figure 15 — Figure 15: Examples of annotation changes made during the second verification phase. Each row shows a before (left) and after (right) pair. Colors denote objects with: no attributes, rendition attribute, reflection attribute. Annotators add previously unnoticed objects, verify that existing annotations correspond to finalised class definitions, and assign attributes that were introduced or clarified mid-annotation.

	$i$ -th largest bounding box						top $i$ largest bounding boxes
Model	$Dom_{668}$	$1_{16666}$	$2_{15754}$	$3_{9343}$	$4_{5867}$	$5_{3840}$	$\mathrel{\scriptstyle\leq}\!2_{16666}$	$\mathrel{\scriptstyle\leq}\!3_{16666}$	$\mathrel{\scriptstyle\leq}\!4_{16666}$	$\mathrel{\scriptstyle\leq}\!5_{16666}$	$\mathrm{M}_{16666}$
GPT-5.4	78.14	51.23	26.51	16.80	11.22	7.71	72.13	75.83	76.69	77.00	77.22
Qwen3-VL	75.60	52.78	26.62	16.63	11.04	8.02	73.67	77.47	78.34	78.70	79.08
Gemma 4	79.34	49.50	26.27	16.76	11.42	7.97	70.18	74.12	74.99	75.32	75.60
SigLIP	76.35	57.04	30.48	18.78	12.63	8.65	81.10	85.11	85.95	86.28	86.60
SigLIP2	78.74	57.28	30.84	18.68	12.43	8.98	81.60	85.58	86.46	86.82	87.16

Appendix

Table 5: Top-1 ReGT accuracy on

M

by ground-truth bounding box area. Index

i

denotes the

i

-th largest box. Columns

i_{n}

report accuracy for the

i

-th largest box over

n

images. Columns

\mathrel{\scriptstyle\leq}i_{n}

report ReaL accuracy, treating the top-

i

boxes as valid labels.

\lim_{i\to\infty}\mathrm{Acc}_{\leq i}(subset)=\mathrm{Acc}_{\mathrm{ReGT}}(M)

Table 19

	ImGT	ImageNet ReaL [ 2 ]
	$\mathrm{A_{50000}}$	$\mathrm{{A{\setminus}N}_{46837}}$	$\mathrm{S_{39394}}$	$\mathrm{{S+}_{35811}}$	$\mathrm{{S-}_{3583}}$	$\mathrm{M_{7443}}$	$\mathrm{{M+}_{6495}}$	$\mathrm{{M-}_{948}}$
GPT-5.4	71.32	77.53	78.83	82.96	37.54	70.68	74.67	43.35
GPT-5.4 ReGT	71.32	77.64	81.91	84.10	58.39	65.23	67.11	52.15
Qwen3-VL	69.77	76.11	76.43	80.49	35.86	74.43	78.60	45.89
Qwen3-VL ReGT	69.77	77.04	80.38	82.44	58.27	69.18	71.05	56.11
Gemma 4	68.33	74.69	75.65	79.41	38.12	69.57	73.49	42.72

Appendix

Table 8: Model accuracy evaluated against four prior reannotation datasets, each shown with original labels (top row per model) and with ReGT label categories applied (bottom row). Subsets are defined relative to each dataset own annotations. Images with no valid label are excluded from evaluation, according to the label set of each row (top: images unlabeled in the prior dataset; bottom:

\mathrm{N}

category images in ReGT).

Panel of Figure 16 — Figure 16: Alluvial diagrams illustrating how proposed labels from prior ImageNet-1k reannotation efforts map to our ReGT labels. Each image is assigned to a category as defined in Fig. 4 , and the flows reveal the degree of agreement and disagreement.

Image Classification

Image classification assigns semantic labels to images, enabling recognition of objects, scenes, conditions, and fine-grained categories.

See all 381 papers

Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation
Mar 5, 2026Junyu Chen, Md Yousuf Harun, Christopher KananUnsupervised Visual Concept Discovery Multi-Label Classification
University of Rochester · Rochester Institute of Technology
Multimodal Large Language Models as Image Classifiers
Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language Models Multimodal Classification
Visual Recognition Group, Czech Technical University in Prague
Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty
Sep 18, 2026Sarina Penquitt, Jonathan Klees, Antonia van Betteray +4Object Detection Human-in-the-Loop Annotation
Institute of Computer Science Osnabrück University Osnabrück, Germany · Department of Mathematics University of Wuppertal Wuppertal, Germany · Wayve London, United Kingdom

Mar 5, 2026J/K move · Enter open · S save

Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation

Junyu Chen, Md Yousuf Harun, Christopher Kanan

University of Rochester · Rochester Institute of Technology

Unsupervised Visual Concept Discovery Multi-Label Classification Image Classification

Open paper AskPDF arXiv

The original ImageNet benchmark enforces a single-label assumption, despite many images depicting multiple objects. This leads to label noise and limits the richness of the learning signal. Multi-label annotations more accurately reflect real-world visual scenes, where multiple objects co-occur and contribute to semantic understanding, enabling models to learn richer and more robust representations. While prior efforts (e.g., ReaL, ImageNetv2) have improved the validation set, there has not yet been a scalable, high-quality multi-label annotation for the training set. To this end, we present an automated pipeline to convert the ImageNet training set into a multi-label dataset, without human annotations. Using self-supervised Vision Transformers, we perform unsupervised object discovery, select regions aligned with original labels to train a lightweight classifier, and apply it to all regions to generate coherent multi-label annotations across the dataset. Our labels show strong alignment with human judgment in qualitative evaluations and consistently improve performance across quantitative benchmarks. Compared to traditional single-label scheme, models trained with our multi-label supervision achieve consistently better in-domain accuracy across architectures (up to +2.0 top-1 accuracy on ReaL and +1.5 on ImageNet-V2) and exhibit stronger transferability to downstream tasks (up to +4.2 and +2.3 mAP on COCO and VOC, respectively). These results underscore the importance of accurate multi-label annotations for enhancing both classification performance and representation learning. Project code and the generated multi-label annotations are available at https://github.com/jchen175/MultiLabel-ImageNet.

Abstract

Image Classification

Explore similar work

Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation

Multimodal Large Language Models as Image Classifiers

Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty