Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

Tomas Ruiz; Tanalp Agustoslu; Carsten Schwemmer

cs.CLMar 20, 2026

Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

Authors: Tomas Ruiz, Tanalp Agustoslu, Carsten Schwemmer

Organizations: Computational Social Sciences LMU Munich Munich, Germany · Computational Linguistics LMU Munich Munich, Germany

Abstract

Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.

Figures & tables

Question	is_political			is_saxony
Model	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
Gemma-3 4B It	0.84	1.00	0.91	0.27	0.93	0.42
Gemma-3 12B It	0.93	0.97	0.95	0.71	0.81	0.76
Gemma-3 27B It	0.92	0.98	0.95	0.66	0.84	0.74
Qwen 2.5 VL 3B Instruct	0.91	0.97	0.94	0.36	0.91	0.52
Qwen 2.5 VL 7B Instruct	0.99	0.72	0.83	0.31	0.56	0.40

TABLE I: Agreement Subset Evaluation Higher is better. Bold: best value by model family.

Question	is_intolerant		is_hedonic		is_eudaimonic
Model	Brier $\downarrow$	JSD $\downarrow$	Brier $\downarrow$	JSD $\downarrow$	Brier $\downarrow$	JSD $\downarrow$
Gemma-3 4B It	0.37	0.26	0.38	0.27	0.27	0.20
Gemma-3 12B It	0.17	0.13	0.29	0.21	0.26	0.19
Gemma-3 27B It	0.31	0.22	0.37	0.26	0.25	0.19
Qwen 2.5 VL 3B Instruct	0.23	0.17	0.29	0.22	0.25	0.19
Qwen 2.5 VL 7B Instruct	0.12	0.10	0.28	0.21	0.25	0.19

TABLE II: Disagreement Subset evaluation Lower is better. Bold: best value by model family.

Question	$\alpha$	Subset
is_political	0.81	Agree
is_saxony	0.74	Agree
is_hedonic_entertainment	0.55	Disagree
is_intolerant	0.48	Disagree
is_eudaimonic_entertainment	0.38	Disagree

TABLE III: Questions by Evaluation Subset Sorted by descending Krippendorff’s Alpha (

\alpha

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Fig. 1: (Top) Correlation of runtime, output tokens generated and input frames per video. (Bottom) GPU memory usage across models. All models fit on a single H100 (96GB) except Qwen2.5-VL 72B, which required 3× H100 GPUs. We use HuggingFace to run our experiments. Total compute is approximately 80 GPU hours.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

Multimodal Large Language Models as Image Classifiers
Mar 6, 2026Nikita Kisel, Illia Volkov, Klara Janouskova +1Multimodal Large Language Models Multimodal Classification
Visual Recognition Group, Czech Technical University in Prague
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
May 16, 2025Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie +6Algorithmic Fairness Multimodal Large Language Models
Vector Institute for Artificial Intelligence, Toronto, Canada · University of Central Florida, Orlando, USA
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
Jun 1, 2026Mirko Lai, Alessandra Urbinati, Simona Frenda +2Annotator Disagreement Conformal Prediction
Heriot-Watt University, Edinburgh, Scotland · aequa-tech, Torino, Italy · Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA +2

Mar 6, 2026J/K move · Enter open · S save

Multimodal Large Language Models as Image Classifiers

Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas

Visual Recognition Group, Czech Technical University in Prague

Multimodal Large Language Models Multimodal Classification Multimodal Model Evaluation Image Classification

Open paper AskPDF arXiv

Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and Vision-Language Models (VLMs) report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLM underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLM predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.

Abstract

Multimodal Model Evaluation

Explore similar work

Multimodal Large Language Models as Image Classifiers

HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction