Multimodal Large Language Models as Image Classifiers

Nikita Kisel; Illia Volkov; Klara Janouskova; Jiri Matas

cs.CVMar 6, 2026

Multimodal Large Language Models as Image Classifiers

Authors: Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas

Organizations: Visual Recognition Group, Czech Technical University in Prague

Abstract

Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.

Figures & tables

Figure 1 : Challenging visual recognition cases from ImageNet-1k. ImGT: original single ImageNet label, ReGT: our reannotations. Predictions of representative models, including DINOv3, EfficientNetV2, EfficientNet-L2, EVA-02 (self-)supervised on ImageNet, are often wrong on such data. Correct predictions in green .

Figure 2 : The three evaluated classification tasks — OW, MC, and CW(+) — described in Sec. 2.3

Cat.	Definition	Cat.	Definition
$A$	$N\cup S\cup M$	$S^{+}$	$S\cap\{i:gt_{i}\in L_{i}\}$
$N$	$\{i:\|L_{i}\|=0\}$	$S^{-}$	$S\cap\{i:gt_{i}\notin L_{i}\}$
$S$	$\{i:\|L_{i}\|=1\}$	$M^{+}$	$M\cap\{i:gt_{i}\in L_{i}\}$
$M$	$\{i:\|L_{i}\|>1\}$	$M^{-}$	$M\cap\{i:gt_{i}\notin L_{i}\}$

Table 1: Definitions of ImageNet-1k label categories based on ReGT and ImGT differences. For image

i

gt_{i}

is ImGT label, and

L_{i}

the set of ReGT labels.

A,S,M

correspond to all, single-label, and multi-label respectively, +/- denotes agreement/disagreement of ReGT and ImGT.

	$\mathrm{A}$	$\mathrm{S}$	$\mathrm{{S+}}$	$\mathrm{{S-}}$	$\mathrm{M}$	$\mathrm{{M+}}$	$\mathrm{{M-}}$	$\mathrm{{N}}$
#	31250	18071	16177	1894	11834	10756	1078	1345
%	100	57.8	51.8	6.1	37.9	34.4	3.5	4.3

Table 2: The number of images in each label category (#) and their fraction among reannotated images (%). Images with multiple labels (

\mathrm{M}

) account for roughly 40 % of image in the selected 625 classes. Approximately 5 % of the images have no correct corresponding ImageNet-1k label (

\mathrm{N}

Model	Task	$\mathrm{A_{31250}}$		$\mathrm{A_{31250}}$	$\mathrm{S_{18071}}$	$\mathrm{{S+}_{16177}}$	$\mathrm{{S-}_{1894}}$	$\mathrm{M_{11834}}$	$\mathrm{{M+}_{10756}}$	$\mathrm{{M-}_{1078}}$	Im $\cap$ Re
		ImGT	ReGT								Im $\cap$ Re
PaliGemma2-mix-28B/448	OW	37.11 (13)		47.94 +10.8 , (13)	39.74 (13)	41.49 (13)	24.76 (13)	54.55 (13)	56.24 (13)	37.66 (5)	35.78
LLaVA-OneVision-72B-Chat	OW	62.00 (12)		70.58 +8.6 , (12)	67.11 (12)	70.67 (12)	36.69 (4)	72.52 (12)	75.87 (12)	39.05 (3)	59.39
InternVL3.5-38B	CW+	64.31 (11)		72.57 +8.3 , (11)	67.69 (11)	71.05 (11)	39.02 (1)	76.91 (11)	80.43 (11)	41.74 (1)	61.53
Qwen3-VL-235B-A22B-Inst	OW	68.74 (10)		76.68 +7.9 , (10)	73.61 (10)	77.79 (10)	37.91 (3)	78.71 (10)	82.46 (10)	41.28 (2)	65.87
GPT-4o-2024-08-06	CW+	76.40 (9)		82.36 +6.0 , (9)	81.11 (9)	86.12 (9)	38.28 (2)	82.27 (9)	86.74 (9)	37.66 (5)	72.79

Table 3 : Recognition accuracy on label categories of ImageNet-1k defined in Tab. 1 . The highest-performing task (OW/CW+) is selected for the mllm . Increase or decrease in classification accuracy on ReGT w.r.t. ImGT. Model (rank)s are also shown. All models achieve higher accuracy on single-label images. The more dependence there is on the training data, the less the model benefits from the reannotation. The Im

\cap

Re column reports the intersection of the two correctness sets: images on which the model is correct under ImGT and images on which it is correct under ReGT. This value indicates the overlap between these two correctness subsets, which are largely different.

Figure 3 : Classification accuracy on ImageNet-1k, change from the original (ImGT) to reannotated labels (ReGT).

Figure 4 : To address mllm out-of-prompt “OOP” predictions, often referred to as hallucinations [ 22 , 45 ] , we map model outputs that fall outside the provided class list to the nearest in-prompt class using the best model-specific encoder. Examples where the mapping is correct are shown. Columns correspond to the label categories from which the images were sampled. The row shows Qwen3-VL predictions, for which 38.75% OOP predictions are correctly mapped. Correct or incorrect image label.

PaliGemma 2	OW	37.11	47.94	39.74	41.49	24.76	54.55	56.24	37.66
		ImGT	ReGT
	Task	$\mathrm{A_{31250}}$	$\mathrm{A_{31250}}$	$\mathrm{S_{18071}}$	$\mathrm{{S+}_{16177}}$	$\mathrm{{S-}_{1894}}$	$\mathrm{M_{11834}}$	$\mathrm{{M+}_{10756}}$	$\mathrm{{M-}_{1078}}$
LLaVA-OV	OW	62.00 +18.6	70.58 +17.8	67.11 +17.5	70.67 +18.7	36.69 +7.1	72.52 +20.3	75.87 +22.0	39.05 +4.1
	CW+	52.66 +9.3	62.82 +10.1	59.54 +10.0	62.32 +10.4	35.80 +6.2	63.61 +11.4	65.84 +11.9	41.37 +6.4
	CW	43.40	52.75	49.59	51.94	29.57	52.20	53.92	34.97
InternVL3.5	OW	59.23 -4.9	68.18 -4.2	62.44 -5.1	65.78 -5.1	33.90 -5.1	73.33 -3.5	76.66 -3.7	40.07 -1.4

Table 4: Recognition accuracy for CW(+) and OW tasks. PaliGemma 2 results are reported only for the OW setup, since it is not feasible to perform CW with all class names included in a single prompt. Changes relative to the Closed World (CW) baseline are indicated as increase or decrease . The CW+ setup extends the CW task by applying the same text-embedding-space output mapping used in the Open World (OW) task, addressing OOP predictions in CW that would otherwise be considered incorrect. Results are obtained using the best-performing model-specific encoder; for a comparison of all encoders, see Tab. 17 . As expected, accuracy in CW+ is higher than in standard CW. Notably, LLaVA-OV and Qwen3-VL outperform their CW results in the OW setup, an unexpected trend that contrasts prior findings on mllm [ 22 , 45 ] . In contrast, InternVL3.5 and GPT-4o achieve the highest performance in CW+.

Model	$\mathrm{A}$	$\mathrm{S}$	$\mathrm{{S+}}$	$\mathrm{{S-}}$	$\mathrm{M}$	$\mathrm{{M+}}$	$\mathrm{{M-}}$	$\mathrm{{N}}$
InternVL3.5	0.9	0.8	0.8	0.8	0.8	0.7	1.2	2.5
GPT-4o	5.3	4.0	3.4	8.7	6.0	5.6	9.9	16.4
Qwen3-VL	10.5	9.3	8.7	13.8	10.9	10.6	14.8	24.2
LLaVA-OV	26.8	24.1	23.6	28.1	29.0	28.8	31.2	42.5

Table 5: mllm out-of-prompt (OOP) rate, the (%) of cases when the output is none of the classes specified, per label categories. Models hallucinate more on 1. multilabel images than single label ones, 2. images that do not contain their ImGT in ReGT (S- and M-), 3. images from N category (contain no ImageNet labels in ReGT). On the last, the hallucination rate is highest for all models.

		ImGT	ReGT
	Distractors	$\mathrm{A_{625}}$	$\mathrm{A_{625}}$	$\mathrm{S_{352}}$	$\mathrm{{S+}_{316}}$	$\mathrm{{S-}_{36}}$	$\mathrm{M_{240}}$	$\mathrm{{M+}_{221}}$	$\mathrm{{M-}_{19}}$
GPT-4o	ImGT + random	99.62 ±0.07	90.95 ±0.05	89.49 ±0.07	99.67 ±0.07	0.09 ±0.18	91.85 ±0.10	99.75 ±0.11	0.00 ±0.00
	ImGT + confEVA(ImGT)	90.66 ±0.22	85.12 ±0.21	84.27 ±0.26	93.87 ±0.29	0.00 ±0.00	84.31 ±0.39	91.56 ±0.42	0.00 ±0.00
	ReGT + confEVA(ReGT)	51.35 ±0.21	84.26 ±0.37	92.45 ±0.36	93.90 ±0.31	79.66 ±1.50	70.09 ±0.89	70.78 ±0.83	62.14 ±3.36
	ImGT + ReGT + random	91.77 ±0.26	94.89 ±0.13	94.79 ±0.16	99.67 ±0.07	51.97 ±1.35	94.31 ±0.27	99.80 ±0.09	30.56 ±2.97
	ImGT + ReGT + confEVA(ImGT,ReGT)	88.31 ±0.29	92.52 ±0.15	91.59 ±0.20	96.36 ±0.15	49.73 ±1.27	92.86 ±0.31	98.16 ±0.20	31.24 ±3.30

Table 6 : 4-way multiple-choice (MC) task results on a randomly sampled subset of 625 images (one per class) with 95 % confidence intervals (CI). The function confEVA() generates challenging distractors based on the confusion matrix of EVA-02. The Closed World result corresponds to 999 distractors. Only GPT-4o results are shown here, as the trend of gradually declining accuracy with more challenging distractors is similar across models. Full multiple-choice task results for all evaluated mllm are provided in Tab. 18 .

Figure 5 : Images from the second annotation pass, where annotators 1. preferred GPT-4o over ReGT (left), 2. kept part of ReGT without adding GPT-4o (right), and 3. combined ReGT with GPT-4o predictions (middle). “ReReGT” indicates second pass reannotations. Annotators in the second pass preserved correct ReGT or added GPT-4o predictions while also correcting erroneous first-pass labels.

Figure 6 : Challenging classification cases. (a) Dominant objects do not always match the target class; e.g., a “lens” $\notin$ ImageNet-1k labels. (b) Appearance variations hinder recognition of long-tail content, e.g., espresso served in a goblet vs. a cup. (c) Fixed label sets force assignment even for out-of-distribution objects (e.g., “seagull” $\notin$ ImageNet-1k). (d) Recognizability depends on scale and context; small or ambiguous objects are harder to identify.

Figure 7 : Second annotation pass results on images where GPT-4o gave a valid prediction and disagreed with ReGT for $\mathrm{S-}$ and $\mathrm{M-}$ label categories (52.32% and 49.16% of images in these categories, respectively; challenging since ImGT $\notin$ ReGT). Color coding: Annotators preferred the GPT-4o prediction (ReGT was incorrect); preserved correct ReGT without adding GPT-4o prediction; combined GPT-4o prediction with ReGT (GPT-4o $\notin$ ReGT before); made other changes (GPT-4o & ReGT $\notin$ ReReGT).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Vision Backbone	Lang. Encoder	Training Strategy
PaliGemma2-mix-28B/448	SigLIP so400M	Gemma2 27B	Joint multimodal pretraining + finetuning
LLaVA-OneVision-72B-Chat	SigLIP so400M	Qwen2 72B	Projector-based alignment + supervised finetuning
InternVL3.5-38B	InternViT 6B	Qwen3 32B	Progressive multimodal scaling + instruction tuning
Qwen3-VL-235B-A22B-Inst	SigLIP 2 so400M	Qwen3 MoE	Large-scale multimodal pretraining + instruction tuning
GPT-4o-2024-08-06	(undisclosed)	(undisclosed)	Proprietary multimodal pretraining + post-training alignment (RLHF)

Appendix

Table 7: Vision backbones and language models for all five evaluated mllm . GPT-4o-2024-08-06 is a closed-source model; architectural details are not publicly disclosed.

Panel of Figure 8 — Figure 8 : Weasel family case study: Correctly labeled ImageNet-1k images for “weasel”, “mink”, “polecat”, and “black-footed ferret”. (a) Weasel often refers broadly to mustelidae family, but images mainly depict least weasels. (b) Minks are easier to recognize but still confused with similar species. (c) European polecat is the ancestor of domestic ferret and they are extremely hard to classify. Often, they are recognized by the environment, as opposed by the animals’ features. (d) Black-footed ferret images are frequently domestic ferrets, leading to renaming. See [ 16 ] for details.

		ImGT		ReGT	WeaselGT
Model	Task	$\mathrm{A_{31250}}$	$\mathrm{A_{159}}$	$\mathrm{A_{31250}}$	$\mathrm{A_{159}}$	$\mathrm{{S+_{112}}}$	$\mathrm{{S-_{47}}}$
ImGT	CW	100.00	100.00	91.20 -8.8	70.44 -29.6	100.00	00.00
LLaVA-OV	CW	42.56	25.79	52.00 +9.4	28.93 +3.1	35.71	12.77
LLaVA-OV	MC	82.17	45.91	90.01 +7.8	61.01 +15.1	59.82	63.83
Intern-VL3.5	CW	63.68	44.65	72.96 +9.3	61.01 +16.4	60.71	61.70
Intern-VL3.5	MC	85.89	55.97	90.53 +4.6	75.47 +19.5	75.89	74.47

Appendix

Table 8: Accuracy on the full 31250-image ReGT set and on the WeaselGT 159-image subset . Increase or decrease indicates the accuracy change under ReGT vs. ImGT. For WeaselGT, deltas denote the accuracy difference between WeaselGT vs. ImGT on the same 159-image subset . Gains are consistently higher on WeaselGT, especially for vlm and mllm —indicating greater sensitivity to fine-grained label corrections than supervised models.

	weasel		mink		polecat		domestic ferret
	ImGT	WeaselGT	ImGT	WeaselGT	ImGT	WeaselGT	ImGT	WeaselGT
ImGT	100.00	84.62 -15.4	100.00	85.00 -15.0	100.00	57.14 -42.9	100.00	61.54 -38.5
LLaVA-OV	40.00	76.92 +36.9	60.00	65.00 +5.0	0.00	0.00 +0.0	0.00	0.00 +0.0
Intern-VL3.5	30.00	65.38 +35.4	32.00	37.50 +5.5	0.00	0.00 +0.0	92.00	100.00 +8.0
Qwen3-VL	36.00	76.92 +40.9	72.00	80.00 +8.0	0.00	3.57 +3.6	42.00	43.08 +1.1
GPT-4o	46.00	100.00 +54.0	94.00	95.00 +1.0	10.00	17.86 +7.9	92.00	100.00 +8.0

Appendix

Table 9: Recall comparison between ImGT and WeaselGT. mllm (top) show large gains, often reaching perfect recall, indicating sensitivity to noisy or ambiguous ImGT labels. vlm (center) also improve, reflecting reduced fine-grained sensitivity. Supervised models (bottom) sometimes drop, suggesting overfitting to ImGT. Note: mllm recall is computed only for the CW setup.

		ImGT	ReGT
	Batch	$\mathrm{A_{625}}$	$\mathrm{A_{625}}$	$\mathrm{S_{352}}$	$\mathrm{{S+}_{316}}$	$\mathrm{{S-}_{36}}$	$\mathrm{M_{240}}$	$\mathrm{{M+}_{221}}$	$\mathrm{{M-}_{19}}$	$\mathrm{OOP}$
LLaVA-OV	1	42.56	52.00	46.31	48.10	30.56	53.75	55.20	36.84	168
	5	34.88	42.56	37.78	40.19	16.67	41.67	43.44	21.05	233
	10	27.20	35.36	30.97	32.28	19.44	32.92	34.39	15.79	258
Qwen3-VL	1	64.96	73.12	69.32	73.10	36.11	75.00	77.83	42.11	28
	5	63.52	72.00	70.45	74.37	36.11	70.42	72.85	42.11	44

Appendix

Table 10: Image batch size comparison for mllm on 625 sampled images (one per class) using batches of 1, 5, and 10. Qwen3-VL and GPT-4o show consistent performance across batch sizes, while LLaVA-OV accuracy drops with larger batches. Thus, batch size 10 is used for Qwen3-VL and GPT-4o, and 1 for LLaVA-OV.

Model	In-Batch Ordering	$\mathrm{A_{6250}}$	$\mathrm{A_{6250}}$	$\mathrm{OOP}$
		ImGT	ReGT
Qwen3-VL	Random	63.47	70.83	734
Qwen3-VL	Same-Class	76.96	78.19	470
GPT-4o	Random	75.78	80.82	324
GPT-4o	Same-Class	86.03	84.61	155

Appendix

Table 11: We compare random class mixtures to same-class batches on 6250 images (10 per class). Grouping by class causes mllm to assign identical labels within a batch, inflating ImGT accuracy and agreement with ReGT. Random ordering reduces this bias and is used in all experiments with image-batched inputs.

			ImGT	ReGT
	Img Pos.	Batch	$\mathrm{A_{63(62)}}$	$\mathrm{A_{63(62)}}$	$\mathrm{OOP}$
LLaVA-OV	1st	1	44.44	60.32	18
	1st	5	42.86 -1.6	58.73 -1.6	20 +2
	1st	10	33.33 -11.1	52.38 -7.9	23 +5
	5th	1	55.56	69.84	11
	5th	5	26.98 -28.6	34.92 -34.9	28 +17

Appendix

Table 12: Accuracy of the 1st, 5th, and 10th images from request batches of size 1, 5, and 10, evaluated on 625 sampled images (one per class). Image positions vary by batch size. Qwen3-VL and GPT-4o show stable accuracy across positions, while LLaVA-OV’s accuracy declines for later images, consistent with its sensitivity to larger batches (see Tab. 10 ).

Model	Response	$\mathrm{A_{625}}$	$\mathrm{A_{625}}$	$\mathrm{S_{352}}$	$\mathrm{{S+}_{316}}$	$\mathrm{{S-}_{36}}$	$\mathrm{M_{240}}$	$\mathrm{{M+}_{221}}$	$\mathrm{{M-}_{19}}$	$\mathrm{OOP}$
		ImGT	ReGT
LLaVA-OV	ID	7.68	14.24	9.38	9.81	5.56	9.58	9.95	5.26	0
LLaVA-OV	Class name	42.56 +34.9	52.00 +37.8	46.31	48.10	30.56	53.75	55.20	36.84	168
InternVL3.5	ID	57.60	67.20	62.22	64.24	44.44	70.00	72.85	36.84	6
InternVL3.5	Class name	63.68 +6.1	72.96 +5.8	67.05	69.30	47.22	77.92	80.54	47.37	6
Qwen3-VL	ID	48.00	57.28	54.55	56.96	33.33	55.42	58.37	21.05	0

Appendix

Table 13: Comparison of two response formats for mllm on 625 sampled images (one per class): ID : predicting a class ID via a mapping ( e.g . 0 – “tench”, …) and Class Name : selecting directly from class names. Accuracy deltas relative to the ID format. While accuracy gains over the ID format vary by model, Class Name consistently performs better and can produce OOP outputs useful for CW+. We therefore use Class Name in all experiments.

		OOP		Correctly Mapped
		#	%	#	%
InternVL3.5	$\mathrm{A_{31250}}$	271	0.87	83	30.63
	$\mathrm{S_{18071}}$	145	0.80	34	23.45
	$\mathrm{{S+}_{16177}}$	130	0.80	34	26.15
	$\mathrm{{S-}_{1894}}$	15	0.79	0	0.00
	$\mathrm{M_{11834}}$	92	0.78	15	16.30

Appendix

Table 14: Overview of mllm out-of-prompt across label categories. Counts (#) indicate predictions falling outside the provided class names in the CW setup and the number correctly mapped (#) in CW+. Harder label categories: M– and S–, which do not contain the ImGT label, and N, which contains no label at all – exhibit a higher ratio of OOP predictions.

Model	Partial	ImageNet	Abstain	Wrong
InternVL3.5	21.77	0.0	1.11	77.12
GPT-4o	35.34	3.22	6.38	55.07
Qwen3-VL	26.81	1.67	0.0	71.52
LLaVA-OV	36.29	2.59	0.04	61.08

Appendix

Table 15: Distribution of out-of-prompt mllm predictions. “Partial” denotes cases where the predicted string is a subset of any of the provided class names, regardless of word order. “ImageNet” corresponds to predictions that exactly match the commonly used OpenAI ImageNet-1k class names. “Abstain” refers to predictions in the form of “I don’t know” or similar expressions (including LLM-generated variations). “Wrong” includes all remaining predictions.

Panel of Figure 9 — Figure 9 : Correlation matrices across models evaluated on reannotated labels (ReGT). Left: Computes per-class recall for single-label images only and measures cross-model similarity using Spearman correlation. Right: Builds an image correctness vector including all images with valid label(s) and computes cross-model similarity using the Phi coefficient. The blocks indicate mllm , vlm , supervised models and original ground-truth (ImGT).

		ImGT	ReGT
Model	Batch	$\mathrm{A_{625}}$	$\mathrm{A_{625}}$	$\mathrm{S_{352}}$	$\mathrm{{S+}_{316}}$	$\mathrm{{S-}_{36}}$	$\mathrm{M_{240}}$	$\mathrm{{M+}_{221}}$	$\mathrm{{M-}_{19}}$	$\mathrm{OOP}$
GPT-4o	10	74.69 ±0.19	81.32 ±0.18	79.87 ±0.19	84.49 ±0.21	39.25 ±0.82	80.89 ±0.35	84.72 ±0.36	36.33 ±0.58	30.13 ±1.11

Appendix

Table 16 : Accuracy deviation over 31 repeated evaluations of a 625 sampled images (one per class) using GPT-4o at temperature 0. Despite deterministic settings, small non-zero variability appears, shown by 95 % confidence intervals. LLaVA-OV, InternVL3.5, and Qwen3-VL are fully deterministic and omitted.

		ImGT	ReGT
	Emb. Space	$\mathrm{A_{31250}}$		$\mathrm{A_{31250}}$	$\mathrm{S_{18071}}$	$\mathrm{{S+}_{16177}}$	$\mathrm{{S-}_{1894}}$	$\mathrm{M_{11834}}$	$\mathrm{{M+}_{10756}}$	$\mathrm{{M-}_{1078}}$
PaliGemma 2	Sentnce-BERT	30.60		41.41	32.58	34.01	20.38	48.24	49.58	34.88
	Sentence-BERT †	31.89		42.81	33.86	35.32	21.44	49.98	51.33	36.55
	Qwen3-Embedding-8B	33.24		43.53	35.49	37.05	22.23	49.37	50.99	33.21
	Qwen3-Embedding-8B †	36.52		47.51	39.05	40.71	24.92	54.45	56.16	37.38
	SigLIP 2	34.56		45.04	36.45	38.23	21.28	51.90	53.51	35.81

Appendix

Table 17: Results for the OW setup with different language models (LMs) used for prediction mapping are reported.

\dagger{}

denotes the use of templating following Radford et al. [ 28 ] . Sentence-BERT is included, as it is used for semantic similarity in Conti et al. [ 7 ] . Templating improves performance for all LMs. SigLIP 2 encoding performs best for PaliGemma 2 and GPT-4o, while Qwen3-Embedding-8B encoding achieves the highest performance for LLaVA-OV, InternVL3.5, and Qwen3-VL. The model-specific LM selected based on OW accuracy is used for evaluation in both the OW and CW+ setups in the main paper.

		ImGT	ReGT
	Distractors	$\mathrm{A_{625}}$	$\mathrm{A_{625}}$	$\mathrm{S_{352}}$	$\mathrm{{S+}_{316}}$	$\mathrm{{S-}_{36}}$	$\mathrm{M_{240}}$	$\mathrm{{M+}_{221}}$	$\mathrm{{M-}_{19}}$
LLaVA-OV	ImGT + random	99.28 ±0.15	90.75 ±0.11	89.36 ±0.13	99.52 ±0.14	0.18 ±0.25	91.52 ±0.18	99.39 ±0.20	0.00 ±0.00
	ImGT + confEVA(ImGT)	79.93 ±0.32	76.02 ±0.30	73.13 ±0.45	81.46 ±0.50	0.00 ±0.00	76.96 ±0.47	83.58 ±0.51	0.00 ±0.00
	ImGT + confBERT(ImGT)	88.59 ±0.29	82.65 ±0.26	79.66 ±0.42	88.73 ±0.46	0.00 ±0.00	84.65 ±0.32	91.93 ±0.35	0.00 ±0.00
	ReGT + confEVA(ReGT)	44.28 ±0.26	78.36 ±0.39	81.42 ±0.43	81.49 ±0.48	80.83 ±1.40	70.90 ±0.69	71.77 ±0.64	60.78 ±3.86
	ReGT + confBERT(ReGT)	49.19 ±0.25	84.92 ±0.42	89.08 ±0.42	89.21 ±0.47	87.99 ±1.24	76.75 ±0.82	76.57 ±0.91	78.78 ±2.66

Appendix

Table 18 : 4-way multiple-choice (MC) results across 625 sampled images (one per class), reported with 95 % confidence intervals. confEVA() and confBERT() denote functions that return challenging distractors based on the confusion matrix of EVA-02 and the distance between class names in the BERT embedding space, respectively. The Closed World result corresponds to using 999 distractors.

GPT-4o Prompt
You are an image classifier. You will receive up to 50 images in order (image "1" = first, "2" = second, etc.). You are also provided with a list of class names: {class_list}. Classification Rules: - For each image, return the single class name that best represents the main subject of the image. - Choose only one class per image - the most relevant or dominant one. - Only return classes from the provided list. Output Rules: - Return exactly one output per image. - Each output must be only a single class name (no separators or lists). - Do not include explanations, confidence scores, or extra text.


LLaVA-OV & InternVL3.5 & Qwen3-VL Prompt
You are an image classifier. You will receive up to 50 images in order (image "1" = first, "2" = second, etc.). You are also provided with a list of class names: {class_list}. Your output will be automatically structured as JSON with keys "1", "2", "3", etc. corresponding to the order of images in the request. Each value should be the predicted class name for that image. Classification Rules: - For each image, return the class name only from the provided list. - Only return classes from the provided list. Output Rules: - Return exactly one JSON key per image ("1", "2", "3", etc.). - Each value must be only class names. - Do not include explanations, confidence scores, or extra text.

Appendix

Table 19 : Prompts for the Closed World setup.

GPT-4o Prompt
You are an open-set fine-grained image classifier. You will receive up to 50 images in order (image "1" = first, "2" = second, etc.). Classification Rules: - For each image, identify the dominant object. - Return the most fine-grained, specific label that accurately describes that object (e.g., "golden retriever puppy", "1950s red convertible", "blue morpho butterfly", "ceramic coffee mug with floral pattern"). - Use natural-language labels that reflect detailed visual distinctions such as species, make/model, style, color, or material. - Avoid generic terms like "dog", "car", or "bird" when a more specific subtype or description is visually inferable. - If the dominant object cannot be clearly identified, return a concise descriptive label of its appearance (e.g., "abstract metal sculpture", "blurry human silhouette"). - Focus only on the dominant object, even if multiple are present. Output Rules: - Return exactly one output per image. - The output must contain only the final label (no punctuation beyond normal text, no explanations, confidence scores, or extra text).


LLaVA-OV & InternVL3.5 & Qwen3-VL Prompt
You are an open-set fine-grained image classifier. You will receive up to 50 images in order (image "1" = first, "2" = second, etc.). Your output will be automatically structured as JSON with keys "1", "2", "3", etc. corresponding to the order of images in the request. Each value should be the predicted label for that image. Classification Rules: - For each image, identify the dominant object. - Return the most fine-grained, specific label that accurately describes that object (e.g., "golden retriever puppy", "1950s red convertible", "blue morpho butterfly", "ceramic coffee mug with floral pattern"). - Use natural-language labels that reflect detailed visual distinctions such as species, make/model, style, color, or material. - Avoid generic terms like "dog", "car", or "bird" when a more specific subtype or description is visually inferable. - If the dominant object cannot be clearly identified, return a concise descriptive label of its appearance (e.g., "abstract metal sculpture", "blurry human silhouette"). - Focus only on the dominant object, even if multiple are present. Output Rules: - Return exactly one output per image. - The output must contain only the final label (no punctuation beyond normal text, no explanations, confidence scores, or extra text).

Appendix

Table 20 : Prompts for the Open World setup.

LLaVA-OV & InternVL3.5 & Qwen3-VL & GPT-4o Prompt

You are an image classifier. You will receive one image. You are also provided with four multiple-choice options (A, B, C, D). What is the main object in this image? {dynamic_choices} Classification Rules: - For the image, return the letter (A, B, C, or D) that corresponds to the correct option. - Only return one letter. Output Rules: - Return exactly one letter (A, B, C, or D). - Do not include explanations, the class name, or extra text. - Your answer must be only the letter.

Appendix

Table 21 : Prompt for the Multiple-Choice setup. The same prompt is used for all mllm .

Panel of Figure 10 — Figure 10 : Class pairs (part 1/2) considered equivalent during evaluation ( i.e . for a pair of classes $\{c_{1},c_{2}\}$ and an image labeled $c_{1}$ , a predicted label $c_{2}$ is also considered correct). Each pair is accompanied by a brief note explaining the decision.

Panel of Figure 11 — Figure 11 : Class pairs (part 2/2) considered equivalent during evaluation ( i.e . for a pair of classes $\{c_{1},c_{2}\}$ and an image labeled $c_{1}$ , a predicted label $c_{2}$ is also considered correct). Each pair is accompanied by a brief note explaining the decision.

Explore similar work

May 14, 2026cs.CV

MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models

This paper introduces a multi-label visual emotion analysis benchmark dataset for comprehensively evaluating the ability of multimodal large language models (MLLMs) to predict the emotions evoked by images. Recent user studies report an unintuitive finding: humans may prefer the predictions of MLLMs over the labels in existing datasets. We argue that this phenomenon stems from the suboptimal annotation scheme used in existing datasets, where each annotator is shown a single candidate emotion for each image and judges whether it is evoked or not. This approach is clearly limited because a single image can evoke multiple emotions with varying intensities. As a result, evaluations based on these datasets may underestimate the capabilities of MLLMs, yet an appropriate benchmark for evaluating such models remains lacking. To address this issue, we introduce a new multi-label benchmark dataset for visual emotion analysis toward MLLMs evaluation. We hire

20

annotators per image and ask them to select all emotions they feel from an image. Then, we aggregate the votes across all annotators, providing a more reliable and representative dataset labeled with a distribution of emotions. The resulting dataset contains

10,344

images with

236,998

valid votes across eight emotions. Based on this benchmark dataset, we evaluate several recent models, including Qwen3-VL, OpenAI's GPT, Gemini, and Claude. We assess model performance on both dominant emotion prediction and emotion distribution prediction. Our results demonstrate the progress achieved by recent MLLMs while also indicating that substantial room for improvement remains. Furthermore, our experiments with LLM-as-a-judge show that the method does not consistently improve MLLMs' performance, indicating its limitations for the subjective task of visual emotion analysis.

Tianwei Chen, Takuya Furusawa, Yuki Hirakawa +3

ZOZO NEXT Inc.

Multi-Label Classification Multimodal Large Language Models Multimodal Emotion Recognition Multimodal Model Evaluation Emotion Recognition

Ask PDF

May 24, 2026cs.CV

Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities across a wide range of vision language tasks. However, when applied to large scale image classification, their performance degrades significantly as the label space expands a phenomenon we define as Performance Collapse in Long Sequence Recognition. Through an information theoretic analysis, we reveal that this collapse stems from a fundamental conflict between the escalating information entropy and the prominent attention dilution and decay within attention mechanisms, which impairs the model's ability to maintain a sufficient signal-to-noise ratio when processing extremely long prompts. To mitigate this, we propose Divide-and-Conquer Inference (DCI), a novel test-time scaling strategy for visual recognition with MLLMs. DCI recursively decomposes complex global classification tasks into multiple simpler, localized subproblems and employs a dynamic pruning mechanism to compress the search space. This method effectively improves the local signal to noise ratio and model accuracy by mitigating the inherent weight dilution issues in long-sequence inference. Moreover, while traditional self-attention incurs a prohibitive quadratic computational complexity, DCI achieves more favorable scaling behavior and substantially accelerates inference in large scale classification scenarios. Extensive experiments on benchmarks such as ImageNet-1K and ImageNet-21K demonstrate that DCI consistently improves classification accuracy. This enables lightweight open-source models to rival or even surpass frontier closed-source giants without any additional training or fine-tuning. As a model-agnostic, plug-and-play paradigm, DCI offers an efficient approach for scaling the inferential precision of MLLMs in large-scale scenarios.

Zhipeng Ye, Jiaqi Huang, Feng Jiang +5

Taizhou Institute of Science and Technology, Nanjing University of Science and Technology, Taizhou, 225300, Jiangsu, China · Department of Intelligence Science, Xi’an Jiaotong-Liverpool University, Suzhou, 215123, Jiangsu, China · Department of Statistical Sciences, University of Toronto, Toronto, M5S 1A1, Ontario, Canada +1

Efficient Multimodal Inference Multimodal Large Language Models Image Classification

Ask PDF

Feb 13, 2025cs.CV

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.

Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31

University of Cambridge

Visual Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Foundation Models Multimodal Model Evaluation Multimodal Understanding

Ask PDF