Automatic evaluation using multimodal large language models (MLLMs), commonly referred to as MLLM-as-a-Judge, has been widely used to measure model performance. If such MLLM-as-a-Judge methods were biased, they could distort model comparisons and benchmark-driven scientific progress. However, it remains unclear to what extent MLLM-as-a-Judge methods favor or disfavor text generated by specific MLLMs. In this study, we propose Philautia-Eval to investigate such model-specific preference bias. Philautia-Eval quantifies the degree of the bias by disentangling preference tendencies from differences in generation quality. Using 1.29M caption-score pairs collected from 12 MLLMs, we found that representative MLLMs tend to exhibit self-preference bias. Moreover, experimental results indicate mutual preference bias within particular model families, which is potentially driven by reused connectors and overlapping instruction-tuning resources. Finally, we introduce a simple ensemble of MLLMs, Pomms. Our results demonstrated that Pomms effectively mitigated the model-specific preference bias while maintaining performance.
Figures & tables
Figure 1: Schematic of our approach for investigating model-specific preference bias in MLLM-as-a-Judge. As described in Section 6 , each MLLM typically favors its own generations (self-preference bias), whereas LLaVA-1.5 [ 35 ] favors text generated by other models within the LLaVA family (cross-model preference bias). Our P anel o f M LL M Evaluator s ( Pomms ) exhibits less model-specific preference bias.
Figure 2: An example of self-preference bias in MLLM-as-a-Judge. The figure shows captions generated by Gemini 2.5 Pro (top) and the average scores given to these captions by Evaluators (bottom). In this example, Gemini 2.5 Pro may exhibit self-preference bias because it gives higher scores on average to its own captions than other Evaluators do.
Figure 3: Pipeline of the proposed method. Generators generate image captions, which are then given evaluation scores by Evaluators. From these scores, we construct a matrix Φ , whose rows and columns correspond to the Generators and the Evaluators, respectively. We then standardize Φ column-wise and subsequently row-wise to obtain Φ~ . The diagonal entries indicate the degree of self-preference bias, which we name the philautia score . In the figure, “std.” stands for standardization.
Figure 4: Visualization of Φ~ in the (i) reference-based and (ii) reference-free settings. All philautia scores (diagonal items) were greater than zero, indicating the presence of self-preference bias within the MLLMs used in our experiments.
Figure 5: Example of self-preference bias. The bar chart shows the scores given to a caption generated by Gemini-2.5-Pro. Gemini-2.5-Pro exceptionally gave high scores to its own generations compared with the other Evaluators. The symbol ⧫ represents the mean value of the scores by each Evaluator. Red text within y^g highlights hallucination.
Figure 6: Visualization of preference bias within model families. (i) Submatrix for Qwen-based models: nine of the 12 off-diagonal entries were positive, suggesting a preference bias within the model family. (ii) Submatrix for LLaVA-family models: LLaVA-1.5-13B tends to favor its successor models (e.g., LLaVA-NeXT-Vicuna-7B, LLaVA-OneVision-7B).
Metrics
Nebula
Flickr8k-Ex
SelfEval-Cap
τb↑
τc↑
τb↑
τc↑
Φ -score
G-VEval
GPT-4o
56.1
53.2
61.5
59.7
1.08
Qwen2.5-VL-7B
55.3
52.4
54.6
54.0
1.12
InternVL2.5-8B
54.1
51.3
54.6
52.9
3.03
Pomms
(i) GPT-4o and InternVL2.5-8B
56.4
53.5
61.5
59.7
1.31
(ii) + Eagle2-9B
56.6
53.6
62.7
60.8
0.45
Table 1: Quantitative comparison between Pomms and the baselines. For Pomms , each row sequentially adds a model to the ensemble. The philautia scores for G-VEval are obtained from the Φ~ matrix in Fig. 4 (ii); therefore, the values are not identical to those calculated from Φ~ containing Pomms . The details are provided in Appendix I . Bold font indicates the best results, and underlined font indicates the second-best results. “ Φ -score” represents the philautia score.
Evaluator
Generator
GPT-4o
Gemini
Qwen
Molmo
Eagle
LV-OV
DeepSeek
Gemma
Phi
LV-NV
LV
Intern
Self
1.08
1.84
1.12
0.86
1.05
1.83
2.00
1.10
1.26
1.09
1.26
3.03
Pomms
0.23
0.18
0.67
-0.26
0.01
-0.90
0.67
-0.33
0.11
0.14
-0.18
0.34
Table 2: Quantitative comparison between Φ~E(i)(G(i)) and Φ~Pomms(G(i)) . Bold font indicates the best results. The philautia scores are computed in the same manner as in Table 1 . The full model names for the abbreviations are as follows: Gemini: Gemini-2.5-Pro; Qwen: Qwen2.5-VL-7B; Molmo: Molmo-7B-D; Eagle: Eagle2-9B; LV-OV: LLaVA-OneVision-7B; DeepSeek: DeepSeek-VL2; Gemma: Gemma3-4B-IT; Phi: Phi-3.5-Vision; LV-NV: LLaVA-NeXT-Vicuna-7B; LV: LLaVA-1.5-13B; Intern: InternVL2.5-8B.
Model
#Param
Vision Encoder
LLM backbone
GPT-4o [ 21 ]
-
-
-
Gemini 2.5 Pro [ 13 ]
-
-
-
Qwen2.5-VL-7B [ 6 ]
8.29B
ViT
Qwen2.5-7B
Molmo-7B-D-0924 [ 14 ]
8.02B
CLIP ViT-L/14
Qwen2-7B
Eagle2-9B [ 31 ]
8.93B
Siglip-400M+ConvNext
Qwen2.5-7B-Instruct
LLaVA-OneVision-7B [ 29 ]
7.06B
SigLIP-SO400M-384
Qwen2-7B
Table C : Details of the MLLMs used in our experiments. “#Param” indicates the number of parameters.
Figure G : (i) Visualization of Φ in the reference-based setting. The vertical and horizontal axes represent the Generators and Evaluators, respectively. High scores are shown in red and low scores in blue. (ii) Visualization of Φ after Evaluator-wise standardization. Because the means and standard deviations differ across Evaluators, the matrix entries should be standardized in the columnwise direction. Similarly, they should be standardized in the row-wise direction.
Figure H : Additional example of self-preference bias. DeepSeek-VL2 exceptionally gave high scores to its own generations compared with the other Evaluators. The symbol ⧫ represents the mean value of scores for each Evaluator. Red text within y^g highlights hallucination.
Figure I : Example of self-preference bias. LLaVA-1.5 exceptionally gave high scores to its own generations compared with the other Evaluators.
Figure J : Visualization of a full matrix Φ~ reconstructed by incorporating Pomms . The scores given by Pomms were significantly lower than the corresponding philautia scores of the individual constituent MLLMs.
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.
Sua Lee, Sanghee Park, Jinbae Im
Seoul National University · NAVER Cloud AI · KAIST AI
As LLMs rapidly saturate existing benchmarks, automated benchmark creation using LLMs (LLM as a benchmark) where a model generates test inputs (LLM as a testset) and evaluates outputs (LLM as an evaluator) has gained traction as a cheap alternative to human curation. We show that this paradigm has a fundamental problem: LLM-generated benchmarks systematically favor the model that created them. Using machine translation as our primary testbed, we find that self bias arises from two compounding sources, LLM as a testset and LLM as an evaluator, and their combination amplifies the effect. Crucially, even when test data is generated with explicit diversity controls, each modelś implicit stylistic tendencies produce homogeneous, model-specific outputs that inflate its own scores. Increasing source text diversity, using our proposed diversity metric, partially mitigates this bias. Self bias is strong enough to cause each model to rank itself first, overriding the peer consensus ordering. We confirm that the phenomenon extends to open-ended generation on the Chatbot Arena task.
As LLM-as-a-judge becomes increasingly widespread, self-preference -- the tendency of a judge to favor its own outputs -- raises growing concerns about evaluation reliability. However, this bias has been studied predominantly on generated text, where stylistic features and response quality are inevitably conflated. As a result, existing measurements cannot separate genuine self-preference from these confounds. We address this limitation by changing the object of evaluation: instead of judging generated text, ten LLMs assess sets of narrative constraints selected from a shared pool, which carry no stylistic fingerprint yet retain a recoverable model-specific signature. Two experiments on this task yield complementary findings. Under blind evaluation, self-preference disappears, with a small effect remaining in the opposite direction once selection quality and judge severity are controlled. Under matched quality, however, self- and other-labels alone -- without naming any model -- shift scores bidirectionally. LLM judges inflate scores for self-labeled selections and deflate those for other-labeled ones regardless of the selection's actual source. We make two contributions: 1) authorship attribution is a distinct driver of evaluation bias, and 2) ground-truth-free tasks can serve as controlled instruments for studying LLM judge behavior.
Songeun Chae, Min Kim, Donghoon Jung +2
School of Digital Humanities and Computational Social Sciences, KAIST, South Korea