Are Multimodal Large Language Models Good Annotators for Image Tagging?

Ming-Kun Xie; Jia-Hao Xiao; Zhiqiang Kou; Zhongnian Li; Gang Niu; Masashi Sugiyama

cs.CVFeb 24, 2026

Are Multimodal Large Language Models Good Annotators for Image Tagging?

Authors: Ming-Kun Xie, Jia-Hao Xiao, Zhiqiang Kou, Zhongnian Li, Gang Niu, Masashi Sugiyama

Organizations: RIKEN Center for Advanced Intelligence Project, Japan · Southeast University, China · China University of Mining and Technology, China · The University of Tokyo, Japan

Abstract

Image tagging, a fundamental vision task, traditionally relies on human-annotated datasets to train multi-label classifiers, which incurs significant labor and costs. While Multimodal Large Language Models (MLLMs) offer promising potential to automate annotation, their capability to replace human annotators remains underexplored. This paper aims to analyze the gap between MLLM-generated and human annotations and to propose an effective solution that enables MLLM-based annotation to replace manual labeling. Our analysis of MLLM annotations reveals that, under a conservative estimate, MLLMs can reduce annotation cost to as low as one-thousandth of the human cost, mainly accounting for GPU usage, which is nearly negligible compared to manual efforts. Their annotation quality reaches about 50% to 80% of human performance, while achieving over 90% performance on downstream training tasks.Motivated by these findings, we propose TagLLM, a novel framework for image tagging, which aims to narrow the gap between MLLM-generated and human annotations. TagLLM comprises two components: Candidates generation, which employs structured group-wise prompting to efficiently produce a compact candidate set that covers as many true labels as possible while reducing subsequent annotation workload; and label disambiguation, which interactively calibrates the semantic concept of categories in the prompts and effectively refines the candidate labels. Extensive experiments show that TagLLM substantially narrows the gap between MLLM-generated and human annotations, especially in downstream training performance, where it closes about 60% to 80% of the difference.

Figures & tables

Figure 1: An example of MLLM-generated annotations using different prompting methods. Green denotes true labels. Red denotes missing labels. Open-ended and multi-option prompting require only a single inference per image, whereas binary prompting requires $q$ inferences per image, where $q$ is the number of classes.

Panel of Figure 2 — Figure 2: The quality of annotations generated by Qwen3-VL-8B using different prompt formats and prompt styles.

Panel of Figure 3 — Figure 3: The quality of annotations and performance of training model obtained by using Qwen3-VL-8B.

Panel of Figure 4 — Figure 4: Comparison of three prompting strategies on COCO and Objects365. Left: Precision; middle: Recall; right: average number of candidate labels per image. DP (Disco-occurrence Partition) prompting yields the highest recall.

Panel of Figure 5 — Figure 5: Comparison of three prompting strategies on COCO and Objects365.

	GPU Time	OP	OR	OF1	CP	CR	CF1	mAP
NXTP	4	62.78	55.94	59.16	57.89	50.43	53.91	57.38
CLIP	$<1$	59.30	59.20	59.25	65.00	61.65	63.28	66.85
TagCLIP	4	69.06	68.74	68.90	67.98	65.29	66.61	73.61
CaSED	3	86.22	23.85	37.36	83.30	28.70	42.69	54.42
RAM++	$<1$	89.23	55.08	68.12	89.53	61.66	73.03	79.05
	Qwen3-VL-8B-Instruct

Table 1: The results of annotation quality (OP, OR, OF1, CP, CR, CF1) and model performance (mAP) on COCO 2014.

	GPU Time	OP	OR	OF1	CP	CR	CF1	mAP
NXTP	4	34.71	25.11	29.14	41.72	18.52	25.65	23.81
CLIP	$<1$	39.60	40.87	40.22	31.90	29.94	30.89	27.75
TagCLIP	9	51.71	50.53	51.11	38.01	31.99	34.74	33.34
CaSED	3	63.84	6.41	11.65	48.02	11.12	18.05	22.08
RAM++	$<1$	66.11	23.54	34.72	54.71	31.25	39.78	39.43
	Qwen3-VL-8B-Instruct

Table 2: The results of annotation quality (OP, OR, OF1, CP, CR, CF1) and model performance (mAP) on O365.

Panel of Figure 6 — Figure 6: The performance on COCO 2014 validation set of ResNet-101 trained on annotations generated by difference methods on COCO 2017 unlabeled set.

Method					COCO 2014			O365
MOP	BP	Two-Stage	DCP	CAD	OF1	CF1	mAP	OF1	CF1	mAP
✓					77.54	78.57	78.80	54.01	46.61	43.25
	✓				82.97	83.18	80.76	57.97	49.71	43.45
✓	✓	✓			84.82	84.91	81.58	63.05	54.86	45.22
✓	✓	✓	✓		85.56	85.68	81.71	65.29	57.57	46.19
✓	✓	✓	✓	✓	87.92	87.43	82.76	66.90	58.91	46.50

Table 3: Ablation Studies on COCO 2014 and O365.

Figure 7: Visualization of annotations generated by different methods on COCO 2017.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Simple	Carefully-Designed
Binary Prompt	Is there a {object} in this image?. Answer with only yes or no.	Carefully examine the image and decide if it contains a {object}. Answer with only yes or no.
Multi-Option Prompt	What objects are in this image? Candidates: {cls_list}. From this list, output only the names of the objects that are present, separated by commas. Do not include any object that is not in the candidate list. If none of the candidate objects are present, output exactly NO.	Carefully examine the image and decide which of the following candidate objects are present in the image. Candidates: {cls_list}. From this list, output only the names of the objects that are present, separated by commas. Do not include any object that is not in the candidate list. If none of the candidate objects are present, output exactly NO.

Appendix

Table 4: Prompt templates for binary prompting and multi-option prompting

The Pompt for CooP	The Pompt for DiscP
Based on the co-occurrence relationships among the categories, please divide these n categories into m groups, ensuring that the categories within each group frequently co-occur.	Based on the co-occurrence relationships among the categories, please divide these n categories into m groups, ensuring that the categories within each group do not co-occur.

Appendix

Table 5: Prompt templates for CooP and DiscP.

	Definition	Examples
Sense Ambiguity	A category name is semantically unclear and may carry multiple interpretations, potentially referring to distinct types of objects.	In COCO 2014, the category name orange , which typically refers to the fruit but can also denote the color and apple , which usually refers to the fruit, but may also be interpreted as the technology brand.
Hypernym Overreach	A category name covers an excessively wide range of meanings, often extending beyond what the user originally intended to refer to.	In COCO 2014 and O365, the category name tie typically refers to a necktie . However, the word tie can also broadly denote any knot-like object.
Misnomer/ Underspecification	A category name fails to precisely capture the intended semantic meaning, making them unable to properly represent what the user originally intended to annotate.	In O365, the category names mask and marker are used to denote face mask and marker pen , respectively.

Appendix

Table 6: Definitions and examples of three types of category-concept misalignment.

Dataset	# CLS	# Train	# Val	# Labels / Image
Dataset	# CLS	# Train	# Val	Min $\sim$ Max	Avg
COCO-2014	80	82,081	40,137	1 $\sim$ 18	2.9
Objects365-filtered	251	104,395	78,303	1 $\sim$ 30	6.3
COCO-2017-unlabeled	80	123,403	-	-	-

Appendix

Table 7: The statics of benchmark datasets used.

COCO			Objects365
airplane	$\rightarrow$	plane	sneakers	$\rightarrow$	running shoe
fire hydrant	$\rightarrow$	hydrant	other shoes	$\rightarrow$	shoe
suitcase	$\rightarrow$	luggage	desk	$\rightarrow$	table
skis	$\rightarrow$	ski	street lights	$\rightarrow$	street light
potted plant	$\rightarrow$	plant	cabinet/shelf	$\rightarrow$	cabinet
dining table	$\rightarrow$	table	handbag/satchel	$\rightarrow$	handbag

Appendix

Table 8: Category difference comparison: Original dataset category to (

\rightarrow

) RAM category.

LLM-Assisted Annotation

Large language model-assisted annotation uses AI to label, classify, and structure data, often with human review to improve scale and consistency.

See all 202 papers

Multimodal Large Language Models as Image Classifiers
Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language Models Multimodal Classification
Visual Recognition Group, Czech Technical University in Prague
AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
Aug 17, 2026Yan Ma, Lizhuo ZhangMultimodal Model Evaluation LLM-Assisted Annotation
School of Foreign Studies, Changsha University of Science and Technology, Changsha, China and School of Education, Hunan Agricultural University, Changsha, China · School of Information and Intelligence, Hunan Agricultural University, Changsha, China
Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking
Mar 20, 2026Tomas Ruiz, Tanalp Agustoslu, Carsten SchwemmerMultimodal Model Evaluation Multimodal Large Language Models
Computational Social Sciences LMU Munich Munich, Germany · Computational Linguistics LMU Munich Munich, Germany

Mar 6, 2026J/K move · Enter open · S save

Multimodal Large Language Models as Image Classifiers

Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas

Visual Recognition Group, Czech Technical University in Prague

Multimodal Large Language Models Multimodal Classification Multimodal Model Evaluation Image Classification

Open paper AskPDF arXiv

Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.

Abstract

LLM-Assisted Annotation

Explore similar work

Multimodal Large Language Models as Image Classifiers

AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty

Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking