Multimodal large language models (MLLMs) increasingly serve as judges in training and evaluation pipelines, often assessing answers together with explanations. Although AI judges are sensitive to textual factors, it remains unclear whether ordinary plausible explanations can sway judgments requiring visual verification. We introduce JUPE, a visual-question-answering (VQA)-style benchmark that fixes answers while varying explanations. Across 14 MLLM judges, ordinary explanations that sound plausible mislead judges, with effects comparable on average to deliberately persuasive arguments. Plain justifications and chain-of-thought rationales reduce rejection of incorrect answers by 12.7 and 13.8 percentage points, respectively. We call this failure plausibility capture. Using signal detection theory, we find a significant mean shift in decision criterion toward acceptance, without a clear change in mean discriminability. Explanations sway judgments across domains, explanation generators, and judging protocols, while tested mitigations offer limited, model-dependent protection. Such shifts can inflate performance estimates when judges evaluate answers with explanations.
Figures & tables
Figure 1: With the image, question, and candidate answer fixed, adding a plausible explanation changes the judge’s verdict on the incorrect answer from definitely incorrect to definitely correct.
Role
Condition
Img.
CoT
Tech.
Main n
Ind. n
Baseline
No explanation
n/a
n/a
n/a
180
70
Plausible
CoT
✓
joint
-
180
-
Plausible
Plain
✓
-
-
180
-
Plausible
Persuasion (VI)
✓
✓
✓
7,380
2,870
Plausible
Persuasion (TO)
-
✓
✓
7,380
2,870
Control
Vacuous
-
-
-
180
-
Table 1: Explanation conditions and judgment-item counts in JUPE , before judge-model replication. Main and Ind. denote the 180-query main and 70-query induced-wrong sets. Each query yields one item per non-persuasion condition and 41 per persuasion variant. Img., CoT, and Tech. denote generator access to the image, CoT trace, and persuasion-technique definition. † Mismatch uses a CoT trace from another same-domain item. VI = vision-informed; TO = text-only.
Judge
Acc.
CRR ↑
IRR ↓
NoExp
Plain
CoT
Pers.
NoExp
Plain
CoT
Pers.
Random Predictor
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
Closed-Source Models
Gemini 3 Flash
73.9
40.6
33.0
30.2
30.9
6.1
7.1
3.6
4.8
Gemini 3.1 Flash-Lite
65.6
41.1
35.4
28.1
34.5
13.3
9.6
11.9
12.7
Gemini 3.1 Pro
65.0
61.5
67.7
63.5
64.6
9.5
10.7
13.1
12.5
Table 2: Judge model performance across explanation conditions. Acc. is the model’s accuracy on VQA questions. Correct Rejection Rate (CRR %) and Incorrect Rejection Rate (IRR %) are reported as absolute values for each condition: no explanation (NoExp), plain explanation (Plain), chain-of-thought (CoT), and persuasive explanation (Pers.). The darker shades of Blue or Red indicate larger rejection rate.
Figure 2: Judgment behavior relative to no explanation. (a) Decision-criterion shift ( Δc ) and (b) discriminability change ( Δd′ ) per judge (dots), with the cross-judge mean (diamonds) and item-level bootstrap 95% CI. (c) Four-label verdict shares on incorrect-candidate items, averaged over the 14 judges.
Figure 3: Each of the 41 persuasion techniques under VI and TO, by mean persuasive-rhetoric rating and mean CRR drop from no explanation over the 14 judges. Plain and CoT are condition-level references.
Figure 4: Per-judge CRR on the induced-wrong set and VQA accuracy on the same items (diamonds).
Figure 5: Judge reliability with real images (y-axis) versus blank inputs (x-axis), across 11 judges and four explanation conditions. Panels show CRR, IRR, and their gap; points above the diagonal in (c) indicate a larger gap with real images.
CRR ↑
IRR ↓
ρ
Setting
NoExp
Δ Plain
Δ Pers.
NoExp
Δ Plain
Δ Pers.
Plain
Pers.
In-distribution (main set)
All
42.4
− 12.7
− 12.5
23.3
− 10.7
− 10.7
ref
ref
Chart
43.8
− 8.2
− 11.1
25.0
− 12.0
− 12.7
-
-
Medical
36.5
− 9.4
− 10.4
24.8
− 11.0
− 11.7
-
-
Hallucination
50.9
− 20.9
− 16.9
20.1
− 9.3
− 7.6
-
-
Table 3: Plausibility capture across deployment settings. NoExp: mean CRR and IRR without explanation (%); Δ Plain and Δ Pers.: paired change (pp) under Plain and Persuasion . ρ : Spearman correlation of per-judge CRR drops with the ref row. All rows use 14 judges except Grounding (12) and image quality (9).
Figure 6: CRR (a) and IRR (b) under Persuasion -VI, alone or with conservative judging or self-verification.
Figure 7: Mean normalized ratings of incorrect candidates by judging protocol, displayed as percentages (main: Output scale, 4-label). Stems connect no-explanation means (grey) to explanation means; dots show variation across judges.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Δ CRR ↑
Δ IRR
ID
Technique
Strategy
Eth.
VI
TO
VI
TO
32
Threats
Threat
×
15.2
14.0
11.3
11.1
1
Evidence-based Persuasion
Information-based
✓
14.9
14.0
12.4
11.7
41
Information-dense
Information-based
✓
14.4
14.0
12.4
11.9
40
Discouragement
Social-sabotage
×
14.4
15.2
12.4
10.8
12
Complimenting
Relationship-based
✓
14.3
12.8
11.4
11.1
Appendix
Table 4: The 41 persuasion techniques instantiated in the Persuasion conditions (§ 2.3 ) and their per-technique persuasion effect. Strategy and ethical labels follow the source taxonomy of Zeng et al. (2024) ; technique 41 ( Information-dense ) is adapted from Hackenburg et al. (2025) ; ✓and × mark techniques the taxonomy labels ethical and unethical. Effect columns give the mean drop in correct rejection rate ( Δ CRR, ↑ larger is stronger degradation) and incorrect rejection rate ( Δ IRR), in percentage points relative to the no-explanation baseline, averaged over the 14 judges, under vision-informed (VI) and text-only (TO) persuasion. Techniques are sorted by Δ CRR under VI.
Condition
n
Perplexity ↓
Readability
Distinct-1
Distinct-2
NLI Ent. ↑
Vacuous
ctrl
32.25
16.66
.818
.987
.002
Mismatch
ctrl
52.75
10.55
.731
.963
.032
Plain
180
45.03
9.86
.712
.949
.695
CoT
180
53.33
10.12
.726
.960
.489
Persuasion (VI)
7,380
60.15
11.11
.740
.969
.595
Persuasion (TO)
7,380
59.05
11.16
.735
.967
.597
Appendix
Table 5: Text-quality metrics across explanation conditions on the main set. Perplexity and NLI Entailment arrows indicate the preferred direction. Vacuous and Mismatch serve as control conditions with fixed or shuffled text, hence n is marked “ctrl.”
Condition
Q3 Readability
Q4 Coherence
CoT
4.81
4.67
Persuasion -VI
4.67
4.38
Persuasion -TO
4.77
4.57
Mismatch
4.08
1.88
Vacuous
3.23
1.96
Plausible (mean)
4.75
4.54
Appendix
Table 6: Human audit: mean readability (Q3) and coherence (Q4) by condition over 59 sound items ( 11 - 12 per condition).
Main set
Induced-wrong set
CRR ↑
IRR ↓
CRR ↑
Judge
Vac.
Mis.
VI
TO
Vac.
Mis.
VI
TO
Acc.
NoExp
VI
TO
Closed-Source Models
Gemini 3 Flash
41.7
67.4
31.4
30.5
7.2
16.7
4.6
4.9
97.0
83.3
81.9
83.5
Gemini 3.1 Flash-Lite
44.8
62.5
34.3
34.6
20.5
39.8
13.1
12.4
100.0
86.4
79.5
80.8
Gemini 3.1 Pro
69.8
76.0
64.9
64.3
9.5
16.7
12.4
12.5
92.4
93.9
94.1
94.1
Appendix
Table 7: CRR and IRR of each judge in percent on the main set under the Vacuous and Mismatch controls, shown as Vac. and Mis., and under vision-informed and text-only persuasion, shown as VI and TO. The right block gives accuracy and CRR on the induced-wrong set, where Acc. is the accuracy of the judge when it answers the questions itself. Darker shades mark higher rates.
Δc
Δd′
Judge
Plain
CoT
Pers.
Vac.
Mis.
Plain
CoT
Pers.
Vac.
Mis.
Closed-Source Models
Gemini 3 Flash
-0.06
-0.27
-0.19
+0.06
+0.63
-0.28
-0.03
-0.14
-0.06
+0.11
Gemini 3.1 Flash-Lite
-0.17
-0.21
-0.10
+0.19
+0.70
+0.04
-0.29
-0.15
-0.19
-0.31
Gemini 3.1 Pro
+0.12
+0.12
+0.12
+0.11
+0.38
+0.10
-0.13
-0.08
+0.23
+0.07
GPT-5.4
-0.16
-0.16
-0.14
+0.05
+0.78
-0.16
-0.16
-0.11
-0.16
-0.06
Appendix
Table 8: Signal-detection decomposition of judge behavior: per-judge change relative to no explanation in decision criterion ( Δc ; positive = shifts toward rejection) and discriminability ( Δd′ ), computed from the CRR and IRR in Tables 2 and 7 . The last rows give the cross-judge mean and its item-level bootstrap 95% CI.
Incorrect
Correct
Judge
Def.
Prob.
Prob.
Def.
No Expl.
Gemini 3 Flash
34.4
6.2
2.1
57.3
Gemini 3.1 Flash-Lite
40.0
1.1
8.4
50.5
Gemini 3.1 Pro
61.5
0.0
1.0
37.5
GPT-5.4
20.8
14.6
35.4
29.2
Appendix
Table 9: Share of each rating in percent for each judge on items with an incorrect candidate answer. Each block shows one explanation condition. Def. means definitely and Prob. means probably. Closed-source judges are listed first.
Figure 8: Verdict transition matrices on incorrect-candidate items, pooled over the 14 judges: the verdict a judge gives without explanation (rows) versus its verdict under each explanation condition (columns), row-normalized (%).
CRR ↑
IRR ↓
Judge
NoExp
Plain
Pers.
Plain+Box
NoExp
Plain
Pers.
Plain+Box
Closed-Source Models
Gemini 3.1 Pro
92.5
90.8
93.0
91.7
9.2
7.5
10.0
10.0
GPT-5.4
85.8
81.7
83.2
83.3
14.2
7.5
6.2
6.7
GPT-5.4 mini
65.0
51.7
55.3
44.2
10.0
1.7
2.7
0.8
GPT-5.6 Luna
75.0
70.8
69.2
71.7
14.2
7.5
8.2
11.7
Appendix
Table 10: CRR and IRR of each judge in percent on the independent grounding set without explanation, under Plain , under Persuasion -VI, and under the multimodal explanation that adds a red box on the candidate’s object to Plain , shown as Plain+Box. The last rows give the cross-judge mean change from no explanation and its query-level bootstrap 95% CI.
Table 19
CRR ↑
IRR ↓
Judge
NoExp
Plain
CoT
Pers.
NoExp
Plain
CoT
Pers.
Chart
Gemini 3 Flash
31.6
38.9
26.3
26.8
0.0
7.4
0.0
4.2
Gemini 3.1 Flash-Lite
33.3
31.6
21.1
30.8
7.7
7.7
7.4
8.1
Gemini 3.1 Pro
52.6
52.6
42.1
41.2
3.7
11.1
14.8
11.6
GPT-5.4
52.6
31.6
26.3
33.4
14.8
7.4
11.1
9.6
Appendix
Table 13: CRR and IRR of each judge in percent in each task domain. Closed-source judges are listed first.
Table 21
NoExp
Plain
CoT
Pers.-VI
Judge
Clean
JPEG
Noise
Clean
JPEG
Noise
Clean
JPEG
Noise
Clean
JPEG
Noise
CRR ↑
Gemini 3 Flash
40.6
36.5
35.8
33.0
26.3
32.3
30.2
23.2
23.2
30.9
23.7
25.7
Gemini 3.1 Flash-Lite
41.1
41.7
41.7
35.4
31.2
31.2
28.1
30.2
27.1
34.4
31.9
30.7
GPT-5.4 mini
17.7
27.1
21.9
13.5
13.5
14.6
13.5
11.5
12.5
14.4
16.9
14.6
InternVL3.5-8B
41.7
43.8
43.8
10.4
9.4
7.3
12.5
7.3
10.4
12.1
12.1
11.5
Appendix
Table 16: CRR and IRR of each judge in percent on the clean image, under JPEG compression at quality 5, and under Gaussian noise with σ=25 . Closed-source judges are listed first.
NoExp
Plain
CoT
Pers.-VI
Judge
Real
Blank
Real
Blank
Real
Blank
Real
Blank
CRR ↑
Gemini 3 Flash
40.6
52.1
33.0
38.9
30.2
27.1
30.9
29.0
Gemini 3.1 Flash-Lite
41.1
31.2
35.4
22.9
28.1
24.0
34.4
27.5
GPT-5.4 mini
17.7
58.3
13.5
51.0
13.5
42.7
14.4
49.6
InternVL3.5-14B
33.3
71.9
28.4
60.4
25.0
51.0
26.2
51.9
Appendix
Table 17: CRR and IRR of each judge in percent with the real image and with a blank image. Closed-source judges are listed first.
NoExp
Baseline
Conservative
Self-Verify
Expl.-Aware
Pers.-Aware
Self-Summ.
Judge
CRR
IRR
CRR
IRR
CRR
IRR
CRR
IRR
CRR
IRR
CRR
IRR
CRR
IRR
Persuasion-VI explanations
Gemini 3 Flash
40.6
6.1
30.8
5.2
32.5
4.9
37.4
5.4
33.6
5.4
35.0
5.5
41.7
10.8
Gemini 3.1 Flash-Lite
41.1
13.3
33.8
15.1
35.9
12.1
33.6
12.2
37.6
14.2
33.6
12.0
36.2
10.1
GPT-5.4 mini
17.7
7.1
13.4
3.7
15.2
4.4
14.8
3.3
13.4
4.7
14.3
4.6
14.8
3.0
InternVL3.5-14B
33.3
22.6
26.0
13.7
30.0
14.0
29.8
13.9
26.8
13.4
25.6
11.5
23.0
12.0
Appendix
Table 18: CRR and IRR of each judge in percent under each prompt-level mitigation. The top block uses Persuasion -VI explanations and the bottom block uses CoT explanations. NoExp is the rate without explanation, and Baseline is the rate with the explanation and no mitigation. Each mitigation adds one sentence to the judging instruction. Conservative asks the judge to be conservative in accepting the candidate answer as correct. Self-Verify asks the judge to determine its own answer before verifying the candidate. Expl.-Aware warns that the explanation may bias the judge, and Pers.-Aware warns that the explanation may be persuasive. Self-Summ. asks the judge to summarize the explanation before evaluating the candidate. Closed-source judges are listed first.
Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.
Seojeong Park, Jiho Choi, Junyong Kang +3
Graduate School of Artificial Intelligence, KAIST, Republic of Korea · KRAFTON
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.
Sua Lee, Sanghee Park, Jinbae Im
Seoul National University · NAVER Cloud AI · KAIST AI
Multimodal judges struggle to ground decisions in visual evidence. We present MJ1, a multimodal judge trained with reinforcement learning that enforces visual grounding through a structured grounded verification chain (observations → claims → verification → evaluation → scoring) and a counterfactual consistency reward that penalizes position bias. Even without training, our mechanism improves base-model accuracy on MMRB2 by +3.8 points on Image Editing and +1.7 on Multimodal Reasoning. After training, MJ1, with only 3B active parameters, achieves 77.0% accuracy on MMRB2 and surpasses orders-of-magnitude larger models like Gemini-3-Pro. These results show that grounded verification and consistency-based training substantially improve multimodal judgment without increasing model scale.