Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation

Junyu Chen; Md Yousuf Harun; Christopher Kanan

cs.CVMar 5, 2026

Unlocking ImageNet's Multi-Object Nature: Automated Large-Scale Multilabel Annotation

Authors: Junyu Chen, Md Yousuf Harun, Christopher Kanan

Organizations: University of Rochester · Rochester Institute of Technology

Abstract

The original ImageNet benchmark enforces a single-label assumption, despite many images depicting multiple objects. This leads to label noise and limits the richness of the learning signal. Multi-label annotations more accurately reflect real-world visual scenes, where multiple objects co-occur and contribute to semantic understanding, enabling models to learn richer and more robust representations. While prior efforts (e.g., ReaL, ImageNetv2) have improved the validation set, there has not yet been a scalable, high-quality multi-label annotation for the training set. To this end, we present an automated pipeline to convert the ImageNet training set into a multi-label dataset, without human annotations. Using self-supervised Vision Transformers, we perform unsupervised object discovery, select regions aligned with original labels to train a lightweight classifier, and apply it to all regions to generate coherent multi-label annotations across the dataset. Our labels show strong alignment with human judgment in qualitative evaluations and consistently improve performance across quantitative benchmarks. Compared to traditional single-label scheme, models trained with our multi-label supervision achieve consistently better in-domain accuracy across architectures (up to +2.0 top-1 accuracy on ReaL and +1.5 on ImageNet-V2) and exhibit stronger transferability to downstream tasks (up to +4.2 and +2.3 mAP on COCO and VOC, respectively). These results underscore the importance of accurate multi-label annotations for enhancing both classification performance and representation learning. Project code and the generated multi-label annotations are available at https://github.com/jchen175/MultiLabel-ImageNet.

Figures & tables

Figure 1: Comparison of Existing ImageNet Train-split Relabeling Strategies with Ours. Original ImageNet [ 25 ] annotations assume a single label per image. (a) MIIL [ 23 ] adds hierarchical labels from ImageNet-21K but lacks object-level distinctions. (b) ImageNet-Segments [ 11 ] (IN-Seg) offers pixel masks for 9k training images with single object annotation. (c) ReLabel [ 37 ] assigns soft labels via a $15^{2}$ patch map, requiring crop coordinates to extract local supervision. (d) In contrast, our method generates explicit multi-label annotations with corresponding spatial masks, offering true multi-object labeling for the entire training set.

Figure 2: Overview of our relabeling pipeline. (a) We apply MaskCut [ 34 ] on DINOv3 [ 27 ] ViT features to generate object proposals. ReLabel [ 37 ] maps are used to filter proposals most aligned with the original ground-truth label, which supervise a lightweight labeler. (b) At inference, the labeler predicts class scores for each proposal, enabling spatially grounded multi-label annotations. (c) Compared to a global classifier (e.g., EVA02 [ 10 ] ), ReLabel improves proposal filtering while can still produce high-confidence false positives. (d) Visualization of top-1 predictions per region shows our labeler better disambiguates multiple objects than ReLabel, avoiding context bias and recognizing distinct object categories.

Figure 3: Qualitative examples comparing our multi-label annotations against ImageNet and ReaL [ 4 ] . (a) Our method successfully corrects missing or incorrect labels from ReaL by identifying additional objects and providing improved grounding. (b) Representative failure cases, including ambiguity (e.g., notebook vs. laptop) and missed object proposals.

Method	Top-1 Acc $\uparrow$					Multi-Label: mAP $\uparrow$
Method	IN-Val	ReaL	IN-Seg	IN-v2	INv2-ML	ReaL	IN-Seg	INv2-ML
Original Label	77.6	84.0	84.3	65.4	77.4	87.1	87.8	73.0
Original + Label Smooth	78.2	84.1	84.4	66.1	78.2	87.0	87.7	72.3
SCL [ 32 ] (reported)	76.9	83.4	-	-	-	82.2	-	-
LL [ 15 ]	77.8	84.2	84.3	65.7	77.7	87.2	87.7	72.7
ReLabel [ 37 ]	78.9	85.0	84.8	6 7.3	79.4	87.9	88.2	7 4.8

Table 1: Top‑1 accuracy (%) on original ImageNet (IN) and ImageNet‑V2 (INv2), and mAP (%) on multi‑label validation sets (ReaL, IN-Seg and INv2‑ML) for ResNet‑50. The best and 2nd best performances are highlighted by bold and underline, respectively. Our method consistently outperforms single‑label training and single‑positive methods.

Method	ReaL: mAP $\uparrow$
	k=all:	k=1:	k=2:	k=3:	k $\geq$ 4:
	46837	39394	5408	1319	716
Original Label	87.1	90.6	71.1	61.7	60.8
Original + Label Smooth	87.0	90.5	70.8	60.3	59.7
SCL [ 32 ] (reported)	82.2	88.8	61.4	36.8	21.1
LL [ 15 ]	87.2	90.7	71.4	61.4	61.5

Table 2: Subgroup analysis of multi-label classification performance on ReaL. We report mAP overall and stratified by number of ground-truth labels (k) per image. The best and 2nd best performances are highlighted by bold and underline, respectively.

Model	Method	Top-1 Acc $\uparrow$					Multi-Label: mAP $\uparrow$			mAP $\uparrow$
		ImageNet Evaluation								Transfer Learning
		IN-Val	ReaL	IN-Seg	INv2	INv2-ML	ReaL	IN-Seg	INv2-ML	COCO	VOC
	Single-label E2E	78.2	84.1	84.4	66.1	78.2	87.0	87.7	72.3	77.0	89.2
	+Multi-label FT	78.7	85.4	85.9	67.3	80.4	88.2	89.1	76.1	78.7	90.5
ResNet-50	Multi-label E2E	78.7	85.6	85.5	67.4	81.0	88.2	88.8	76.2	78.9	90.7
	Single-label E2E	79.3	84.7	85.3	68.2	80.0	87.1	88.2	72.7	79.5	90.4

Table 3: End-to-end training and transfer performance with our multi-label annotations. We compare single-label training (Single-label E2E), fine-tuning with our multi-labels (+ Multi-label FT), and end-to-end multi-label training (Multi-label E2E) across various model architectures. Best results are highlighted in bold. Our multi-label supervision improves in-domain performance on ImageNet and its variants, and yields consistent gains in downstream multi-label transfer to COCO and VOC.

Method	Top-1 Acc $\uparrow$					Multi-Label: mAP $\uparrow$			mAP $\uparrow$
	ImageNet Evaluation								Transfer Learning
	IN-Val	ReaL	IN-Seg	INv2	INv2-ML	ReaL	IN-Seg	INv2-ML	COCO	VOC
IN21k-Mul Pretrain	-	-	-	-	-	-	-	-	82.8	92.1
+IN1k-Sig FT	84.3	88.7	88.8	74.0	86.4	90.9	91.4	80.7	82.3	90.5
+IN1k-Mul FT	83.6	88.9	89.2	73.8	86.9	91.0	91.8	82.6	82.5	90.9
IN1k-Mul E2E	83.4	88.8	89.1	73.9	86.8	90.7	91.5	81.8	84.7	94.5

Table 4: Comparison with MIIL [ 23 ] , which uses hierarchical multi-labels from ImageNet-21K for pretraining. While MIIL fine-tunes on ImageNet-1K (IN1k) using single-label (Sig) supervision, fine-tuning MIIL with our multi-labels (Mul) improves all multi-label benchmarks. our end-to-end multi-label (Mul E2E) training—without 21K pretraining—achieves comparable in-domain performance and better downstream transfer results. Results use ViT-B/16 at

224

resolution. Best results are highlighted in bold.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Comparison of mask proposals from SAMv2 [ 21 ] (two configurations) and MaskCut [ 34 ] . While SAM can produce fine-grained masks under certain settings, it often over-segments or misses key objects depending on the image, highlighting its sensitivity to hyperparameters. In contrast, MaskCut generates more consistent, instance-level masks across diverse images using a fixed configuration, making it more suitable for large-scale object proposal generation.

$\mathbf{k=0}$	$\mathbf{k=1}$	$\mathbf{k=2}$	$\mathbf{k=3}$	$\mathbf{k\geq 4}$	Avg.
0.59%	78.9%	17.8%	2.5%	0.28%	1.23

Appendix

Table 5: Distribution of the number of unique labels (

\mathbf{k}

) predicted per image in the relabeled ImageNet training set, using a softmax confidence threshold of

\tau=0.5

. The last column (Avg.) reports the average number of labels per image across the full training set.

Local Label	Global Label	Top-1 Acc $\uparrow$					mAP $\uparrow$
Local Label	Global Label	IN-Val	ReaL	IN-Seg	IN-v2	INv2-ML	ReaL	IN-Seg	INv2-ML
Original Label		77.6	84.1	84.1	65.7	77.7	87.1	87.6	72.7
Hard ( $\tau=0.5$ )	None	77.3	84.6	85.0	66.0	79.2	86.9	88.1	73.4
Soft	None	77.4	84.3	84.5	66.2	78.4	87.2	88.1	73.0
Hard ( $\tau=0.8$ )	Pred	77.7	84.6	84.9	66.1	78.7	87.0	88.0	72.3
Soft	Pred	77.6	84.5	85.0	66.5	79.3	87.6	88.2	74.5

Appendix

Table 6: Evaluation of different relabeling configurations on multi-label benchmarks. We sweep thresholding strategies (probability vs. fixed threshold) and choice of global supervision (original vs. predicted) using a ResNet-50 trained on ImageNet for 100 epochs. The best and second best performance are highlighted by bold and underline, respectively.

Figure 5: Additional qualitative comparisons between our multi-label annotations and those from ImageNet and ReaL [ 4 ] . Blue panels (a-e): Examples categorized by the degree of label overlap with ReaL. Labels pruned due to low confidence are indicated by strikethrough ( deleted ). Green panels (f): Common failure modes, including missed part–whole relations, fine-grained confusion, and proposals outside the label space (out-of-vocabulary, OOV).

Co-occurrence	Class $\mathbf{A}$	Class $\mathbf{B}$	Freq( $\mathbf{A}$ )	Freq( $\mathbf{B}$ )	Conf( $\mathbf{A\|B}$ )	Conf( $\mathbf{B\|A}$ )
153	sunglass	sunglasses, dark glasses, shades	175	168	0.91	0.87
135	laptop, laptop computer	notebook, notebook computer	157	151	0.89	0.86
88	monitor	screen, CRT screen	227	101	0.87	0.39
87	maillot	maillot, tank suit	106	119	0.73	0.82
84	car wheel	grille, radiator grille	232	143	0.58	0.36
55	missile	projectile, missile	70	66	0.83	0.79

Appendix

Table 8: List of identified ambiguous class pairs in ImageNet. For each pair, we report class-wise occurrence counts, as well as conditional label confidence:

\text{Conf}(A|B)

is the proportion of times class A appears given B is present, and vice versa.

Image	Supervision	Top-1 Acc $\uparrow$					Multi-Label: mAP $\uparrow$			mAP $\uparrow$
		ImageNet Evaluation								Transfer Learning
		IN-Val	ReaL	IN-Seg	INv2	INv2-ML	ReaL	IN-Seg	INv2-ML	COCO	VOC
	Single-label E2E	81.4	87.0	87.2	70.7	83.1	89.0	89.7	75.6	79.1	91.0
	+Multi-label FT	81.5	87.7	87.8	70.8	84.1	90.1	90.9	80.2	81.0	91.9
$224^{2}$	Multi-label E2E	82.0	88.1	88.3	72.2	85.1	90.3	91.1	80.7	83.3	93.3
	Single-label E2E	83.2	88.2	88.2	72.6	85.1	90.5	91.0	79.6	84.5	92.8

Appendix

Table 10: End-to-end training and transfer performance with our multi-label annotations. We compare single-label (Single-label E2E) training, fine-tuning with our multi-labels (+ Multi-label FT), and end-to-end multi-label training (Multi-label E2E) across various backbones and input sizes. Best results are highlighted in bold. Our multi-label supervision improves in-domain performance on ImageNet and its variants, and yields consistent gains in downstream multi-label transfer to COCO and VOC.

Model	Supervision	Entropy $\uparrow$
Model	Supervision	ImageNet	VOC	COCO
ResNet-50	Rand. Init.	1114	1540	1509
	IN1k-Sig	1705	2582	2310
	IN1k-Mul	2034	2663	2426
ViT-B	Rand. Init.	0.0	137	19
	IN1k-Sig	562	910	810

Appendix

Table 11: Evaluation of

k

-NN-based entropy of penultimate-layer features (Euclidean distance). Models trained with our multi-label supervision (IN1k-Mul) consistently exhibit higher entropy across datasets, indicating greater feature diversity and reduced representation collapse compared to single-label training (IN1k-Sig). This suggests improved representational quality and helps explain the observed gains in transfer learning performance.

Method	100 Epoch		300 Epoch
Method	IN-Val	ReaL	IN-Val	ReaL
Original	77.5	83.5	78.2	84.1
ReLabel [ 37 ]	71.9	78.6	78.9	85.0
Classifier Map (Ours)	74.9	82.1	78.9	84.9

Appendix

Table 12: Top-1 accuracy comparison of soft-label training under different supervision strategies. We follow the ReLabel setup to generate a label map for each image and assign soft labels to random crops based on patchwise logits. All methods use ResNet-50 trained on ImageNet-1K with either 100 or 300 epochs. Our spatial label map achieves comparable or better classification accuracy, while requiring less labeled data.

Config	# Masks	AP $\uparrow$	AP50 $\uparrow$	AP75 $\uparrow$	APs $\uparrow$	APm $\uparrow$	APl $\uparrow$
Full	1.93M (100%)	8.71	17.50	7.78	1.91	7.33	22.52
$\tau=0.3$	1.81M (94.0%)	8.81	17.80	7.90	1.87	7.49	22.52
$\tau=0.5$	1.70M (88.0%)	8.82	17.80	7.92	1.94	7.63	22.35

Appendix

Table 13: Zero-shot segmentation performance on COCO using CutLER masks filtered by our classification head. We remove pseudo-masks whose top-1 softmax confidence falls below threshold

\tau

. Despite excluding up to

12\%

of masks (at

\tau=0.5

), segmentation performance improves across most metrics, suggesting that filtering out low-confidence proposals strengthens the quality of supervision for unsupervised segmentation. Metrics follow COCO-style evaluation: AP is the mean average precision over IoU thresholds, AP50/AP75 are AP at IoU 0.50/0.75, and APs, APm, APl represent AP for small, medium, and large objects, respectively.

Figure 6: Example of interactive annotation using our classification head. Given arbitrary region proposals (highlighted in red), the model predicts ImageNet class labels with confidence scores. Even for out-of-vocabulary objects that are not explicitly covered by the label set (e.g., hiking boot), our classifier predicts semantically related categories (e.g., running shoe) with lower confidence, enabling efficient and flexible region-level labeling.

Unsupervised Visual Concept Discovery

See all 13 papers

Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels
Jun 10, 2026Cheng Chen, Jingyu Zhou, Yifan Zhao +1Multi-Label Classification VLM Adaptation
State Key Laboratory of Virtual Reality Technology and Systems, SCSE & QRI, Beihang University, Beijing 100191, China.
Doomed to Re-Annotate, Forever: The ImageNet Story
Aug 13, 2026Illia Volkov, Nikita Kisel, Tetiana Mishkina +2Image Classification Human-in-the-Loop Annotation
Visual Recognition Group, Faculty of Electrical Engineering Czech Technical University in Prague
Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning
Jul 23, 2026Zhihua Xu, Zhijing Yang, Yufeng Yang +1Contrastive Learning Multi-Label Classification
aSun Yat-Sen University, Guangzhou, 510006, GuangDong, China · bGuangdong University of Technology, Guangzhou, 510006, GuangDong, China

Jun 10, 2026J/K move · Enter open · S save

Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels

Cheng Chen, Jingyu Zhou, Yifan Zhao, Jia Li

State Key Laboratory of Virtual Reality Technology and Systems, SCSE & QRI, Beihang University, Beijing 100191, China.

Multi-Label Classification VLM Adaptation Zero-Shot Image Classification

Open paper AskPDF arXiv

Understanding multi-label images remains a challenging task in computer vision. With the rapid progress of vision-language multimodal learning, vision-language models (VLMs) enable zero-shot recognition without labeled data. However, due to their intrinsic design, these models often prioritize the most iconic object and omit other contextual positives. This intrinsic bias conflicts with the nature of multi-label learning, thereby limiting their applicability. In this work, we propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding, enabling label-free multi-label image recognition. Our approach consists of two key stages, cutting'' and sewing'': In the cutting stage, we present the multi-sampling response estimator to prevent the model from concentrating only on one single object. In the second sewing stage, the multi-object blend adaptation is introduced to adjust the labels to better conform to the multi-label distribution while preserving the intrinsic characteristics of the original model within only one epoch. Extensive experiments show that our framework significantly outperforms existing unsupervised approaches on four public datasets, even surpassing several representative weakly supervised baselines. These results demonstrate the potential of adapting pre-trained VLMs for more comprehensive visual understanding without manual annotations. Our code is publicly available at https://github.com/iCVTEAM/TailorCLIP.

Abstract

Unsupervised Visual Concept Discovery

Explore similar work

Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels

Doomed to Re-Annotate, Forever: The ImageNet Story

Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning