MLLM-as-a-Judge Exhibits Model Preference Bias

Shuitsu Koyama; Yuiga Wada; Daichi Yashima; Komei Sugiura

cs.CVApr 13, 2026

MLLM-as-a-Judge Exhibits Model Preference Bias

Authors: Shuitsu Koyama, Yuiga Wada, Daichi Yashima, Komei Sugiura

Organizations: Keio University, Japan

Abstract

Automatic evaluation using multimodal large language models (MLLMs), commonly referred to as MLLM-as-a-Judge, has been widely used to measure model performance. If such MLLM-as-a-Judge methods were biased, they could distort model comparisons and benchmark-driven scientific progress. However, it remains unclear to what extent MLLM-as-a-Judge methods favor or disfavor text generated by specific MLLMs. In this study, we propose Philautia-Eval to investigate such model-specific preference bias. Philautia-Eval quantifies the degree of the bias by disentangling preference tendencies from differences in generation quality. Using 1.29M caption-score pairs collected from 12 MLLMs, we found that representative MLLMs tend to exhibit self-preference bias. Moreover, experimental results indicate mutual preference bias within particular model families, which is potentially driven by reused connectors and overlapping instruction-tuning resources. Finally, we introduce a simple ensemble of MLLMs, Pomms. Our results demonstrated that Pomms effectively mitigated the model-specific preference bias while maintaining performance.

Figures & tables

Figure 1: Schematic of our approach for investigating model-specific preference bias in MLLM-as-a-Judge. As described in Section 6 , each MLLM typically favors its own generations (self-preference bias), whereas LLaVA-1.5 [ 35 ] favors text generated by other models within the LLaVA family (cross-model preference bias). Our P anel o f M LL M Evaluator s ( Pomms ) exhibits less model-specific preference bias.

Figure 2: An example of self-preference bias in MLLM-as-a-Judge. The figure shows captions generated by Gemini 2.5 Pro (top) and the average scores given to these captions by Evaluators (bottom). In this example, Gemini 2.5 Pro may exhibit self-preference bias because it gives higher scores on average to its own captions than other Evaluators do.

Figure 3: Pipeline of the proposed method. Generators generate image captions, which are then given evaluation scores by Evaluators. From these scores, we construct a matrix $\mathrm{\Phi}$ , whose rows and columns correspond to the Generators and the Evaluators, respectively. We then standardize $\mathrm{\Phi}$ column-wise and subsequently row-wise to obtain $\tilde{\mathrm{\Phi}}$ . The diagonal entries indicate the degree of self-preference bias, which we name the philautia score . In the figure, “std.” stands for standardization.

Figure 4: Visualization of $\tilde{\mathrm{\Phi}}$ in the (i) reference-based and (ii) reference-free settings. All philautia scores (diagonal items) were greater than zero, indicating the presence of self-preference bias within the MLLMs used in our experiments.

Figure 5: Example of self-preference bias. The bar chart shows the scores given to a caption generated by Gemini-2.5-Pro. Gemini-2.5-Pro exceptionally gave high scores to its own generations compared with the other Evaluators. The symbol $\blacklozenge$ represents the mean value of the scores by each Evaluator. Red text within $\hat{\bm{y}}_{g}$ highlights hallucination.

Figure 6: Visualization of preference bias within model families. (i) Submatrix for Qwen-based models: nine of the 12 off-diagonal entries were positive, suggesting a preference bias within the model family. (ii) Submatrix for LLaVA-family models: LLaVA-1.5-13B tends to favor its successor models (e.g., LLaVA-NeXT-Vicuna-7B, LLaVA-OneVision-7B).

Metrics		Nebula		Flickr8k-Ex		SelfEval-Cap
		$\tau_{b}$ $\uparrow$	$\tau_{c}$ $\uparrow$	$\tau_{b}$ $\uparrow$	$\tau_{c}$ $\uparrow$	$\mathrm{\Phi}$ -score
G-VEval	GPT-4o	56.1	53.2	61.5	59.7	1.08
	Qwen2.5-VL-7B	55.3	52.4	54.6	54.0	1.12
	InternVL2.5-8B	54.1	51.3	54.6	52.9	3.03
Pomms	(i) GPT-4o and InternVL2.5-8B	56.4	53.5	61.5	59.7	1.31
	(ii) + Eagle2-9B	56.6	53.6	62.7	60.8	0.45

Table 1: Quantitative comparison between Pomms and the baselines. For Pomms , each row sequentially adds a model to the ensemble. The philautia scores for G-VEval are obtained from the

\tilde{\mathrm{\Phi}}

matrix in Fig. 4 (ii); therefore, the values are not identical to those calculated from

\tilde{\mathrm{\Phi}}

containing Pomms . The details are provided in Appendix I . Bold font indicates the best results, and underlined font indicates the second-best results. “

\mathrm{\Phi}

-score” represents the philautia score.

Evaluator	Generator
Evaluator	GPT-4o	Gemini	Qwen	Molmo	Eagle	LV-OV	DeepSeek	Gemma	Phi	LV-NV	LV	Intern
Self	1.08	1.84	1.12	0.86	1.05	1.83	2.00	1.10	1.26	1.09	1.26	3.03
Pomms	0.23	0.18	0.67	-0.26	0.01	-0.90	0.67	-0.33	0.11	0.14	-0.18	0.34

Table 2: Quantitative comparison between

\tilde{\mathrm{\Phi}}_{E^{(i)}}(G^{(i)})

and

\tilde{\mathrm{\Phi}}_{\text{{Pomms}}}(G^{(i)})

. Bold font indicates the best results. The philautia scores are computed in the same manner as in Table 1 . The full model names for the abbreviations are as follows: Gemini: Gemini-2.5-Pro; Qwen: Qwen2.5-VL-7B; Molmo: Molmo-7B-D; Eagle: Eagle2-9B; LV-OV: LLaVA-OneVision-7B; DeepSeek: DeepSeek-VL2; Gemma: Gemma3-4B-IT; Phi: Phi-3.5-Vision; LV-NV: LLaVA-NeXT-Vicuna-7B; LV: LLaVA-1.5-13B; Intern: InternVL2.5-8B.

Model	#Param	Vision Encoder	LLM backbone
GPT-4o [ 21 ]	-	-	-
Gemini 2.5 Pro [ 13 ]	-	-	-
Qwen2.5-VL-7B [ 6 ]	8.29B	ViT	Qwen2.5-7B
Molmo-7B-D-0924 [ 14 ]	8.02B	CLIP ViT-L/14	Qwen2-7B
Eagle2-9B [ 31 ]	8.93B	Siglip-400M+ConvNext	Qwen2.5-7B-Instruct
LLaVA-OneVision-7B [ 29 ]	7.06B	SigLIP-SO400M-384	Qwen2-7B

Table C : Details of the MLLMs used in our experiments. “#Param” indicates the number of parameters.

Figure G : (i) Visualization of $\mathrm{\Phi}$ in the reference-based setting. The vertical and horizontal axes represent the Generators and Evaluators, respectively. High scores are shown in red and low scores in blue. (ii) Visualization of $\mathrm{\Phi}$ after Evaluator-wise standardization. Because the means and standard deviations differ across Evaluators, the matrix entries should be standardized in the columnwise direction. Similarly, they should be standardized in the row-wise direction.

Figure H : Additional example of self-preference bias. DeepSeek-VL2 exceptionally gave high scores to its own generations compared with the other Evaluators. The symbol $\blacklozenge$ represents the mean value of scores for each Evaluator. Red text within $\hat{\bm{y}}_{g}$ highlights hallucination.

Figure I : Example of self-preference bias. LLaVA-1.5 exceptionally gave high scores to its own generations compared with the other Evaluators.

Figure J : Visualization of a full matrix $\tilde{\mathrm{\Phi}}$ reconstructed by incorporating Pomms . The scores given by Pomms were significantly lower than the corresponding philautia scores of the individual constituent MLLMs.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 115 papers

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
Jun 6, 2026Songeun Chae, Min Kim, Donghoon Jung +2Bias in LLM-as-a-Judge Language Model Bias Evaluation
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Apr 20, 2026J/K move · Enter open · S save

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

Sua Lee, Sanghee Park, Jinbae Im

Seoul National University · NAVER Cloud AI · KAIST AI

LLM-as-a-Judge Multimodal Large Language Models Bias in LLM-as-a-Judge Vision-Language Model-as-a-Judge Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

Self- and Other-Labels Induce Bidirectional Bias in LLM Judges