A Plausible Explanation Is Enough to Mislead Multimodal Judges

Anglin Cai; Zhiqi Wang; Xiaozhi Yang; Yurun Lin; Dongwon Lee; Yuchen Yang

cs.CLOct 15, 2026

A Plausible Explanation Is Enough to Mislead Multimodal Judges

Authors: Anglin Cai, Zhiqi Wang, Xiaozhi Yang, Yurun Lin, Dongwon Lee, Yuchen Yang

Organizations: Northwestern University · The Pennsylvania State University · University of Pennsylvania · University of Washington

Abstract

Multimodal large language models (MLLMs) increasingly serve as judges in training and evaluation pipelines, often assessing answers together with explanations. Although AI judges are sensitive to textual factors, it remains unclear whether ordinary plausible explanations can sway judgments requiring visual verification. We introduce JUPE, a visual-question-answering (VQA)-style benchmark that fixes answers while varying explanations. Across 14 MLLM judges, ordinary explanations that sound plausible mislead judges, with effects comparable on average to deliberately persuasive arguments. Plain justifications and chain-of-thought rationales reduce rejection of incorrect answers by 12.7 and 13.8 percentage points, respectively. We call this failure plausibility capture. Using signal detection theory, we find a significant mean shift in decision criterion toward acceptance, without a clear change in mean discriminability. Explanations sway judgments across domains, explanation generators, and judging protocols, while tested mitigations offer limited, model-dependent protection. Such shifts can inflate performance estimates when judges evaluate answers with explanations.

Figures & tables

Figure 1: With the image, question, and candidate answer fixed, adding a plausible explanation changes the judge’s verdict on the incorrect answer from definitely incorrect to definitely correct.

Role	Condition	Img.	CoT	Tech.	Main $n$	Ind. $n$
Baseline	No explanation	n/a	n/a	n/a	180	70
Plausible	CoT	✓	joint	-	180	-
Plausible	Plain	✓	-	-	180	-
Plausible	Persuasion (VI)	✓	✓	✓	7,380	2,870
Plausible	Persuasion (TO)	-	✓	✓	7,380	2,870
Control	Vacuous	-	-	-	180	-

Table 1: Explanation conditions and judgment-item counts in JUPE , before judge-model replication. Main and Ind. denote the 180-query main and 70-query induced-wrong sets. Each query yields one item per non-persuasion condition and 41 per persuasion variant. Img., CoT, and Tech. denote generator access to the image, CoT trace, and persuasion-technique definition. † Mismatch uses a CoT trace from another same-domain item. VI = vision-informed; TO = text-only.

Closed-Source Models
Judge	Acc.	CRR $\uparrow$				IRR $\downarrow$
		NoExp	Plain	CoT	Pers.	NoExp	Plain	CoT	Pers.
Random Predictor	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0
Gemini 3 Flash	73.9	40.6	33.0	30.2	30.9	6.1	7.1	3.6	4.8
Gemini 3.1 Flash-Lite	65.6	41.1	35.4	28.1	34.5	13.3	9.6	11.9	12.7
Gemini 3.1 Pro	65.0	61.5	67.7	63.5	64.6	9.5	10.7	13.1	12.5

Table 2: Judge model performance across explanation conditions. Acc. is the model’s accuracy on VQA questions. Correct Rejection Rate (CRR %) and Incorrect Rejection Rate (IRR %) are reported as absolute values for each condition: no explanation (NoExp), plain explanation (Plain), chain-of-thought (CoT), and persuasive explanation (Pers.). The darker shades of Blue or Red indicate larger rejection rate.

Figure 2: Judgment behavior relative to no explanation. (a) Decision-criterion shift (

\Delta c

) and (b) discriminability change (

\Delta d^{\prime}

) per judge (dots), with the cross-judge mean (diamonds) and item-level bootstrap 95% CI. (c) Four-label verdict shares on incorrect-candidate items, averaged over the 14 judges.

Figure 3: Each of the 41 persuasion techniques under VI and TO, by mean persuasive-rhetoric rating and mean CRR drop from no explanation over the 14 judges. Plain and CoT are condition-level references.

Figure 4: Per-judge CRR on the induced-wrong set and VQA accuracy on the same items (diamonds).

Figure 5: Judge reliability with real images (y-axis) versus blank inputs (x-axis), across 11 judges and four explanation conditions. Panels show CRR, IRR, and their gap; points above the diagonal in (c) indicate a larger gap with real images.

	CRR $\uparrow$			IRR $\downarrow$			$\rho$
Setting	NoExp	$\Delta$ Plain	$\Delta$ Pers.	NoExp	$\Delta$ Plain	$\Delta$ Pers.	Plain	Pers.
In-distribution (main set)
All	42.4	$-$ 12.7	$-$ 12.5	23.3	$-$ 10.7	$-$ 10.7	ref	ref
Chart	43.8	$-$ 8.2	$-$ 11.1	25.0	$-$ 12.0	$-$ 12.7	-	-
Medical	36.5	$-$ 9.4	$-$ 10.4	24.8	$-$ 11.0	$-$ 11.7	-	-
Hallucination	50.9	$-$ 20.9	$-$ 16.9	20.1	$-$ 9.3	$-$ 7.6	-	-

Table 3: Plausibility capture across deployment settings. NoExp: mean CRR and IRR without explanation (%);

\Delta

Plain and

\Delta

Pers.: paired change (pp) under Plain and Persuasion .

\rho

: Spearman correlation of per-judge CRR drops with the ref row. All rows use 14 judges except Grounding (12) and image quality (9).

Figure 6: CRR (a) and IRR (b) under Persuasion -VI, alone or with conservative judging or self-verification.

Figure 7: Mean normalized ratings of incorrect candidates by judging protocol, displayed as percentages (main: Output scale, 4-label). Stems connect no-explanation means (grey) to explanation means; dots show variation across judges.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Technique	Strategy	Eth.	VI	TO	VI	TO
				$\Delta$ CRR $\uparrow$		$\Delta$ IRR
32	Threats	Threat	$\times$	15.2	14.0	11.3	11.1
1	Evidence-based Persuasion	Information-based	✓	14.9	14.0	12.4	11.7
41	Information-dense	Information-based	✓	14.4	14.0	12.4	11.9
40	Discouragement	Social-sabotage	$\times$	14.4	15.2	12.4	10.8
12	Complimenting	Relationship-based	✓	14.3	12.8	11.4	11.1

Appendix

Table 4: The 41 persuasion techniques instantiated in the Persuasion conditions (§ 2.3 ) and their per-technique persuasion effect. Strategy and ethical labels follow the source taxonomy of Zeng et al. (2024) ; technique 41 ( Information-dense ) is adapted from Hackenburg et al. (2025) ; ✓and

\times

mark techniques the taxonomy labels ethical and unethical. Effect columns give the mean drop in correct rejection rate (

\Delta

CRR,

\uparrow

larger is stronger degradation) and incorrect rejection rate (

\Delta

IRR), in percentage points relative to the no-explanation baseline, averaged over the 14 judges, under vision-informed (VI) and text-only (TO) persuasion. Techniques are sorted by

\Delta

CRR under VI.

Condition	$n$	Perplexity $\downarrow$	Readability	Distinct-1	Distinct-2	NLI Ent. $\uparrow$
Vacuous	ctrl	32.25	16.66	.818	.987	.002
Mismatch	ctrl	52.75	10.55	.731	.963	.032
Plain	180	45.03	9.86	.712	.949	.695
CoT	180	53.33	10.12	.726	.960	.489
Persuasion (VI)	7,380	60.15	11.11	.740	.969	.595
Persuasion (TO)	7,380	59.05	11.16	.735	.967	.597

Appendix

Table 5: Text-quality metrics across explanation conditions on the main set. Perplexity and NLI Entailment arrows indicate the preferred direction. Vacuous and Mismatch serve as control conditions with fixed or shuffled text, hence

n

is marked “ctrl.”

Condition	Q3 Readability	Q4 Coherence
CoT	4.81	4.67
Persuasion -VI	4.67	4.38
Persuasion -TO	4.77	4.57
Mismatch	4.08	1.88
Vacuous	3.23	1.96
Plausible (mean)	4.75	4.54

Appendix

Table 6: Human audit: mean readability (Q3) and coherence (Q4) by condition over

59

sound items (

11

12

per condition).

Closed-Source Models
	Main set								Induced-wrong set
	CRR $\uparrow$				IRR $\downarrow$					CRR $\uparrow$
Judge	Vac.	Mis.	VI	TO	Vac.	Mis.	VI	TO	Acc.	NoExp	VI	TO
Gemini 3 Flash	41.7	67.4	31.4	30.5	7.2	16.7	4.6	4.9	97.0	83.3	81.9	83.5
Gemini 3.1 Flash-Lite	44.8	62.5	34.3	34.6	20.5	39.8	13.1	12.4	100.0	86.4	79.5	80.8
Gemini 3.1 Pro	69.8	76.0	64.9	64.3	9.5	16.7	12.4	12.5	92.4	93.9	94.1	94.1

Appendix

Table 7: CRR and IRR of each judge in percent on the main set under the Vacuous and Mismatch controls, shown as Vac. and Mis., and under vision-informed and text-only persuasion, shown as VI and TO. The right block gives accuracy and CRR on the induced-wrong set, where Acc. is the accuracy of the judge when it answers the questions itself. Darker shades mark higher rates.

Closed-Source Models
	$\Delta c$					$\Delta d^{\prime}$
Judge	Plain	CoT	Pers.	Vac.	Mis.	Plain	CoT	Pers.	Vac.	Mis.
Gemini 3 Flash	-0.06	-0.27	-0.19	+0.06	+0.63	-0.28	-0.03	-0.14	-0.06	+0.11
Gemini 3.1 Flash-Lite	-0.17	-0.21	-0.10	+0.19	+0.70	+0.04	-0.29	-0.15	-0.19	-0.31
Gemini 3.1 Pro	+0.12	+0.12	+0.12	+0.11	+0.38	+0.10	-0.13	-0.08	+0.23	+0.07
GPT-5.4	-0.16	-0.16	-0.14	+0.05	+0.78	-0.16	-0.16	-0.11	-0.16	-0.06

Appendix

Table 8: Signal-detection decomposition of judge behavior: per-judge change relative to no explanation in decision criterion (

\Delta c

; positive = shifts toward rejection) and discriminability (

\Delta d^{\prime}

), computed from the CRR and IRR in Tables 2 and 7 . The last rows give the cross-judge mean and its item-level bootstrap 95% CI.

	Incorrect		Correct
Judge	Def.	Prob.	Prob.	Def.
No Expl.
Gemini 3 Flash	34.4	6.2	2.1	57.3
Gemini 3.1 Flash-Lite	40.0	1.1	8.4	50.5
Gemini 3.1 Pro	61.5	0.0	1.0	37.5
GPT-5.4	20.8	14.6	35.4	29.2

Appendix

Table 9: Share of each rating in percent for each judge on items with an incorrect candidate answer. Each block shows one explanation condition. Def. means definitely and Prob. means probably. Closed-source judges are listed first.

Figure 8: Verdict transition matrices on incorrect-candidate items, pooled over the 14 judges: the verdict a judge gives without explanation (rows) versus its verdict under each explanation condition (columns), row-normalized (%).

Closed-Source Models
	CRR $\uparrow$				IRR $\downarrow$
Judge	NoExp	Plain	Pers.	Plain+Box	NoExp	Plain	Pers.	Plain+Box
Gemini 3.1 Pro	92.5	90.8	93.0	91.7	9.2	7.5	10.0	10.0
GPT-5.4	85.8	81.7	83.2	83.3	14.2	7.5	6.2	6.7
GPT-5.4 mini	65.0	51.7	55.3	44.2	10.0	1.7	2.7	0.8
GPT-5.6 Luna	75.0	70.8	69.2	71.7	14.2	7.5	8.2	11.7

Appendix

Table 10: CRR and IRR of each judge in percent on the independent grounding set without explanation, under Plain , under Persuasion -VI, and under the multimodal explanation that adds a red box on the candidate’s object to Plain , shown as Plain+Box. The last rows give the cross-judge mean change from no explanation and its query-level bootstrap 95% CI.

Table 19

Chart
	CRR $\uparrow$				IRR $\downarrow$
Judge	NoExp	Plain	CoT	Pers.	NoExp	Plain	CoT	Pers.
Gemini 3 Flash	31.6	38.9	26.3	26.8	0.0	7.4	0.0	4.2
Gemini 3.1 Flash-Lite	33.3	31.6	21.1	30.8	7.7	7.7	7.4	8.1
Gemini 3.1 Pro	52.6	52.6	42.1	41.2	3.7	11.1	14.8	11.6
GPT-5.4	52.6	31.6	26.3	33.4	14.8	7.4	11.1	9.6

Appendix

Table 13: CRR and IRR of each judge in percent in each task domain. Closed-source judges are listed first.

Table 21

CRR $\uparrow$
	NoExp			Plain			CoT			Pers.-VI
Judge	Clean	JPEG	Noise	Clean	JPEG	Noise	Clean	JPEG	Noise	Clean	JPEG	Noise
Gemini 3 Flash	40.6	36.5	35.8	33.0	26.3	32.3	30.2	23.2	23.2	30.9	23.7	25.7
Gemini 3.1 Flash-Lite	41.1	41.7	41.7	35.4	31.2	31.2	28.1	30.2	27.1	34.4	31.9	30.7
GPT-5.4 mini	17.7	27.1	21.9	13.5	13.5	14.6	13.5	11.5	12.5	14.4	16.9	14.6
InternVL3.5-8B	41.7	43.8	43.8	10.4	9.4	7.3	12.5	7.3	10.4	12.1	12.1	11.5

Appendix

Table 16: CRR and IRR of each judge in percent on the clean image, under JPEG compression at quality 5, and under Gaussian noise with

\sigma=25

. Closed-source judges are listed first.

CRR $\uparrow$
	NoExp		Plain		CoT		Pers.-VI
Judge	Real	Blank	Real	Blank	Real	Blank	Real	Blank
Gemini 3 Flash	40.6	52.1	33.0	38.9	30.2	27.1	30.9	29.0
Gemini 3.1 Flash-Lite	41.1	31.2	35.4	22.9	28.1	24.0	34.4	27.5
GPT-5.4 mini	17.7	58.3	13.5	51.0	13.5	42.7	14.4	49.6
InternVL3.5-14B	33.3	71.9	28.4	60.4	25.0	51.0	26.2	51.9

Appendix

Table 17: CRR and IRR of each judge in percent with the real image and with a blank image. Closed-source judges are listed first.

Persuasion-VI explanations
	NoExp		Baseline		Conservative		Self-Verify		Expl.-Aware		Pers.-Aware		Self-Summ.
Judge	CRR	IRR	CRR	IRR	CRR	IRR	CRR	IRR	CRR	IRR	CRR	IRR	CRR	IRR
Gemini 3 Flash	40.6	6.1	30.8	5.2	32.5	4.9	37.4	5.4	33.6	5.4	35.0	5.5	41.7	10.8
Gemini 3.1 Flash-Lite	41.1	13.3	33.8	15.1	35.9	12.1	33.6	12.2	37.6	14.2	33.6	12.0	36.2	10.1
GPT-5.4 mini	17.7	7.1	13.4	3.7	15.2	4.4	14.8	3.3	13.4	4.7	14.3	4.6	14.8	3.0
InternVL3.5-14B	33.3	22.6	26.0	13.7	30.0	14.0	29.8	13.9	26.8	13.4	25.6	11.5	23.0	12.0

Appendix

Table 18: CRR and IRR of each judge in percent under each prompt-level mitigation. The top block uses Persuasion -VI explanations and the bottom block uses CoT explanations. NoExp is the rate without explanation, and Baseline is the rate with the explanation and no mitigation. Each mitigation adds one sentence to the judging instruction. Conservative asks the judge to be conservative in accepting the candidate answer as correct. Self-Verify asks the judge to determine its own answer before verifying the candidate. Expl.-Aware warns that the explanation may bias the judge, and Pers.-Aware warns that the explanation may be persuasive. Self-Summ. asks the judge to summarize the explanation before evaluating the candidate. Closed-source judges are listed first.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 118 papers

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
Jun 1, 2026Seojeong Park, Jiho Choi, Junyong Kang +3Reward Modeling LLM-as-a-Judge
Graduate School of Artificial Intelligence, KAIST, Republic of Korea · KRAFTON
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
MJ1: Multimodal Judgment via Grounded Verification
Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMs MLLM Evaluation
Haize Labs

Jun 1, 2026J/K move · Enter open · S save

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee+2

Graduate School of Artificial Intelligence, KAIST, Republic of Korea · KRAFTON

Reward Modeling LLM-as-a-Judge Multimodal Model Evaluation

Open paper AskPDF arXiv

Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

MJ1: Multimodal Judgment via Grounded Verification