Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

Shuwei Yue; Kaida Xiao; Zhihua Wang; Qiao Chen

cs.CVOct 29, 2026

Benchmarking color judgment in multimodal LLMs with CIE-computable ground truth

Authors: Shuwei Yue, Kaida Xiao, Zhihua Wang, Qiao Chen

Organizations: Shenzhen Polytechnic University, Shenzhen, China · University of Leeds, Leeds, United Kingdom · Sun Yat-sen University, Guangzhou, China

Abstract

Multimodal large language models (MLLMs) are now used as zero-shot judges of image quality, but their color judgments are rarely tested with colorimetric ground truth. We present a five-task benchmark based on the CIE colorimetric framework: AWB pair judgment, skin-tone bias direction, CCT estimation, CIE Delta E 2000 pair ordering, and color naming under changing illuminants. We test eight MLLM endpoints, including four Qwen3-VL variants, GLM-4.5V, GPT-5.4, Claude-Opus-4.7, and Gemini-3.1-Pro. The models often solve coarse binary AWB judgments, but they remain weak on numeric CCT estimation and on some fine color-difference judgments. On ColorChecker-24 stimuli, the three closed-weight frontier models reach CCT MAE of 715-1056 K, compared with 1358-2133 K for the five open-weight models. A non-ColorChecker control shows that this advantage is partly template-dependent. The results suggest that current MLLMs have useful color words and coarse visual priors, but their color judgments are not reliably calibrated to CIE-computable color differences.

Figures & tables

Figure 1: Headline CCT result. Closed-weight frontier endpoints reach lower CCT MAE on ColorChecker-24 than the tested open-weight endpoints. The dashed 1000 K line is a visual reference only; Kelvin is not perceptually uniform [ 11 ] .

Figure 2: Representative stimuli for the five tasks. (A) Task A AWB pair (correct vs. wrong). (B) Task B skin-tone bias direction. (C) Task C ColorChecker rendered at 2700 and 10000 K. (D) Task D $\Delta E$ ordering triplet. (E) Task E color constancy — the same patch under 2700 and 9000 K. All stimuli are 480 $\times$ 320 PNGs.

Qwen3-VL family — open-weight, scale + reasoning axis
Model	A which	A bias	B dir $\pm$ 1	C MAE K	C $\leq$ 500K	D order	E raw const.
	$n{=}24,\ \uparrow$	$n{=}24,\ \uparrow$	$n{=}48,\ \uparrow$	$n{=}27,\ \downarrow$	$n{=}27,\ \uparrow$	$n{=}24,\ \uparrow$	$n{=}8$ patch, $\uparrow$
8B-Instruct	0.58 [.38,.79]	0.25 [.08,.46]	0.60 [.48,.73]	2052 [1522,2537]	0.11 [.00,.26]	0.79 [.62,.92]	0.78 [.65,.90]
30B-A3B-Instruct	0.67 [.46,.83]	0.25 [.08,.46]	0.52 [.39,.67]	2133 [1511,2774]	0.15 [.04,.30]	0.79 [.62,.92]	0.68 [.55,.80]
32B-Instruct	1.00 [.86,1]	1.00 [.86,1]	0.60 [.48,.75]	1922 [1459,2411]	0.22 [.07,.41]	0.79 [.62,.92]	0.85 [.73,.95]
32B-Thinking	1.00 [.86,1]	1.00 [.86,1]	0.67 [.54,.81]	1358 [ 0 950,1808]	0.33 [.17,.54]	0.75 [.58,.92]	0.85 [.73,.95]

Table 1: Headline accuracy on the five color tasks for eight MLLMs, grouped by access mode. Each cell: mean with 95% bootstrap CI (1000 resamples). Bold = best, underline = 2nd-best per column. Chance: A which = 0.50, A bias = 0.25 (4-way), B dir

\pm

1 = 0.375 (3 of 8 clock positions), D order = 0.50. “1.00 [.86,1]” uses the Clopper-Pearson 95% lower bound for 24/24 successes (vs. the zero-variance bootstrap [1,1]). Task E “raw constancy” rewards stable-but-wrong names; read together with the conditional analysis (FM4).

Figure 3: Task C — CCT estimation across the full 9-CCT, 27-stimulus task. (A) Predicted vs. true CCT scatter. Five models (the four Qwen variants + GLM-4.5V) cluster around a dashed $5500$ K daylight-band anchor (closer to D55 than to D65) regardless of the true illuminant; GPT-5.4, Claude-Opus-4.7, and Gemini-3.1-Pro track the identity line. (B) Aggregate mean absolute error (lower is better, blue, left axis) and within-500 K rate (higher is better, orange, right axis). GPT-5.4 cuts MAE by $2.9\times$ over Qwen-8B; Claude-Opus-4.7 reaches the highest within-500 K rate of any model.

Figure 4: Tasks D and E, eight models. (A) $\Delta E_{2000}$ pair-ordering accuracy (chance $=0.5$ ). The four Qwen variants cluster in a tight $0.75$ – $0.79$ plateau; GLM-4.5V, Claude-Opus-4.7, Gemini-3.1-Pro all reach $0.875$ and GPT-5.4 reaches $0.917$ . The plateau is Qwen-specific. (B) Matches-D65-name accuracy (blue) and cross-illuminant constancy index (orange) per model. Constancy clusters in a $0.675$ – $0.850$ band across all eight models; GLM-4.5V tops matches-D65 at 0.375.

MLLM Evaluation

See all 40 papers

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models
Jun 29, 2026Sajjad Ghiasvand, Maryam Amirizaniani, Haniyeh Ehsani Oskouie +2Language Model Generation Evaluation Multimodal Reasoning
UCSB · University of Washington · UCLA
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM Robustness LLM-as-a-Judge
Northeastern University · Mohamed bin Zayed University of Artificial Intelligence

Apr 20, 2026J/K move · Enter open · S save

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

Sua Lee, Sanghee Park, Jinbae Im

Seoul National University · NAVER Cloud AI · KAIST AI

LLM-as-a-Judge Multimodal Large Language Models Bias in LLM-as-a-Judge Vision-Language Model-as-a-Judge Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.

Abstract

MLLM Evaluation

Explore similar work

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation