cs.CVOct 1, 2026

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

Authors: Yuan Huang, Zirui Song, Xiuying Chen

Organizations: Northeastern University · Mohamed bin Zayed University of Artificial Intelligence

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model training. However, systematically auditing whether these judges are influenced by cues irrelevant to editing quality is challenging because visual interventions may themselves alter the quality being evaluated. A judgment shift can therefore be attributed to bias only when the intervention is verified to preserve the underlying editing quality. To address this challenge, we introduce EditJudgeBias, a counterfactual benchmark with verified quality preservation, comprising 1,196 real editing samples and 13 cues injected across four evaluation sites. We verify quality preservation for the requested edit using calibrated multimodal validators, controls, and human inspection. We then audit five MLLM judges along three complementary dimensions: invariance to quality-preserving cues, agreement with human judgments, and stability of pairwise preferences. Importantly, observed shifts are evaluated against each judge's own zero-dose and re-query noise floors rather than against zero. Experiments show that quality-preserving cues move every judge beyond its own noise. Fabricated majority opinions increase ratings, irrelevant visual elements cause larger shifts than whole-image manipulations, and swapping candidate order reverses up to 60.9% of pairwise decisions. Edit-region cues also tend to reduce human agreement. The three measures characterize judges differently, showing that robustness cannot be captured by a single metric.

Figures & tables

Figure 1: Bias-relevant cues occur naturally in image-editing benchmarks. Four real examples illustrate framing, typographic, provenance, and colorfulness cues. Human ratings may not penalize these cues, yet MLLM judges explicitly attend to them; EditJudgeBias tests these effects under controlled interventions
Figure 2: Overview of EditJudgeBias. (a) Evaluation framework for measuring judge invariance, agreement with human judgments, and stability under quality-preserving cues. (b) Composition of 1,196 editing triplets drawn from five public benchmarks spanning three independent content pools. (c) Human supervision used to assess agreement with human judgments and verify edit quality.
Figure 3: Overview of the thirteen bias cues and the zero-dose control. Each row identifies a cue, its injection site in the judging pipeline, and its intervention and dose. All image-side cues are illustrated using the same edited I2EBench example. position is illustrated by the same judged pair under both display orders; it changes neither pixels nor prompt text.
 gpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Bias (short name)IAEQDPIAEQDPIAEQDPIAEQDPIAEQDP
Bandwagon8.211.59.0 ▼\scriptscriptstyle\blacktriangledown18.5 ▼\scriptscriptstyle\blacktriangledown18.118.55.26.04.73.56.05.77.27.3 ▼\scriptscriptstyle\blacktriangledown5.0 ▼\scriptscriptstyle\blacktriangledown
Authority (model name)6.47.96.617.815.117.32.92.93.33.24.74.54.44.94.2
Colorfulness (saturation)5.87.36.08.2 ▼\scriptscriptstyle\blacktriangledown7.57.63.02.73.22.43.94.34.24.73.8
Aesthetic (aesthetic filter)6.6 ▼\scriptscriptstyle\blacktriangledown8.5 ▼\scriptscriptstyle\blacktriangledown7.812.010.812.23.83.84.64.15.35.46.16.05.3
Provenance (watermark)6.87.9 ▼\scriptscriptstyle\blacktriangledown7.4 ▼\scriptscriptstyle\blacktriangledown7.7 ▼\scriptscriptstyle\blacktriangledown8.07.63.13.33.64.66.96.27.18.2 ▼\scriptscriptstyle\blacktriangledown6.2
Table 1: Invariance and human agreement across cues and judges. Cells report 10 times the mean absolute per-item change in each 1–10 rating dimension (IA: instruction adherence; EQ: editing quality; DP: detail preservation). Bold indicates cells whose 95% bootstrap lower bound exceeds both the sham and retest floors; plain cells exceed one floor and gray cells exceed neither. Superscripts indicate significant changes in agreement with human ratings ( ▼\blacktriangledown decrease; ▲\blacktriangle increase).
Figure 4: Main results of the EditJudgeBias audit. (a) Pairwise preference shifts induced by quality-preserving cues. (b) Verdict reversal rates under display-order swaps, compared with identical re-queries. (c) Calibration of the preservation validators using controlled edit degradation and the zero-dose control. (d) Comparison of judges across invariance, agreement, and stability, showing that judge rankings differ across the three robustness measures.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Cuegemini-3.5gpt-4o-miniglm-4v
Colorfulness (saturation)100.097.394.5
Aesthetic (aesthetic filter)99.195.587.3
Provenance (watermark)100.098.291.8
Luminance (brightness)99.197.393.6
Framing (padding)100.094.593.6
Typographic (text overlay)100.090.092.7
Appendix
Table 2: Preservation validation across cues. Panel A reports validator pass rates; Panel B reports detection of controlled quality degradations among 93 evaluable examples per condition.
Cuegpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Bandwagon++ 0.45++ 0.44++ 0.34++ 0.24++ 0.45
Authority (model name)−- 0.04++ 0.12++ 0.05++ 0.13−- 0.10
Colorfulness (saturation)++ 0.00−- 0.09++ 0.08++ 0.01++ 0.10
Aesthetic (aesthetic filter)++ 0.10++ 0.05++ 0.06++ 0.09++ 0.12
Provenance (watermark)−- 0.20−- 0.19++ 0.02−- 0.29++ 0.01
Luminance (brightness)−- 0.03−- 0.11−- 0.03−- 0.02++ 0.08
Appendix
Table 3: Signed shifts in instruction adherence. Cells report paired mean changes on the 1–10 scale; bold indicates Benjamini–Hochberg significance. ‡ marks the borderline cue.
Cuegpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Bandwagon++ 0.62++ 0.41++ 0.50++ 0.42++ 0.48
Authority (model name)−- 0.13−- 0.05++ 0.10++ 0.18++ 0.01
Colorfulness (saturation)−- 0.10−- 0.17−- 0.07−- 0.08++ 0.02
Aesthetic (aesthetic filter)−- 0.09−- 0.18−- 0.12−- 0.02−- 0.09
Provenance (watermark)−- 0.32−- 0.28++ 0.02−- 0.38−- 0.25
Luminance (brightness)−- 0.22−- 0.25−- 0.22−- 0.25−- 0.17
Appendix
Table 4: Signed shifts in editing quality. Cells report paired mean changes on the 1–10 scale; bold indicates Benjamini–Hochberg significance. ‡ marks the borderline cue.
Cuegpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Bandwagon++ 0.51++ 0.69++ 0.22++ 0.21++ 0.30
Authority (model name)++ 0.02++ 0.46−- 0.01++ 0.03−- 0.12
Colorfulness (saturation)−- 0.06++ 0.03−- 0.05−- 0.08++ 0.02
Aesthetic (aesthetic filter)−- 0.17++ 0.16−- 0.24−- 0.10−- 0.14
Provenance (watermark)−- 0.36−- 0.04++ 0.02−- 0.25++ 0.19
Luminance (brightness)−- 0.30−- 0.09−- 0.32−- 0.25−- 0.16
Appendix
Table 5: Signed shifts in detail preservation. Cells report paired mean changes on the 1–10 scale; bold indicates Benjamini–Hochberg significance. ‡ marks the borderline cue.
JudgeInvarianceAgreementStability
gpt-5.59.67 (4)0.0474 (4)0.8458 (2)
gemini-3.512.94 (5)0.0402 (3)0.8994 (1)
kimi-k2.55.29 (1)0.0272 (1)0.7955 (3)
qwen3.57.14 (2)0.0348 (2)0.3523 (5)
VIEScore7.32 (3)0.0615 (5)0.6883 (4)
Appendix
Table 6: Robustness rankings across judges. Panel A gives the primary rankings used in Figure 4 d; Panel B summarizes rank inversions under alternative aggregation schemes.
  clean itemscollision items
CueJudgeΔ\Delta95% CIΔ\Delta95% CI
Luminance (brightness)gpt-5.5−- 0.67[ −- 0.89, −- 0.45]−- 0.06[ −- 0.65, ++ 0.52]
 gemini-3.5−- 0.16[ −- 0.35, ++ 0.04]−- 1.65[ −- 2.49, −- 0.82]
 kimi-k2.5−- 0.69[ −- 0.86, −- 0.52]−- 0.12[ −- 0.43, ++ 0.20]
 qwen3.5−- 0.66[ −- 0.95, −- 0.39]++ 0.07[ −- 0.72, ++ 0.92]
 VIEScore−- 0.33[ −- 0.54, −- 0.12]++ 0.02[ −- 0.41, ++ 0.47]
Appendix
Table 7: Photometric controls for instruction collisions. Cells report paired mean score shifts for clean and collision items with 95% confidence intervals; bold indicates Benjamini–Hochberg significance.
Cuegpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Bandwagon++ 2.65++ 3.89++ 1.36++ 0.74++ 1.30
Luminance (brightness)−- 0.28++ 2.67−- 0.59−- 1.28−- 0.57
Framing (padding)−- 2.83++ 1.48−- 1.17−- 1.59−- 0.89
Typographic (text overlay)−- 3.28−- 0.30−- 1.87−- 3.61−- 1.48
Attention guidance (box)−- 4.22−- 1.09−- 2.33−- 2.91−- 2.33
Distraction (sticker)−- 4.20−- 2.26−- 4.13−- 5.69−- 2.50
Appendix
Table 8: Cue effects on high-quality edits. Cells report mean signed score shifts within the highest human-MOS tertile; bold indicates Benjamini–Hochberg significance.
CueAnchorgpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
Luminance (brightness)EBench-18K−- 0.006−- 0.008−- 0.010++ 0.013−- 0.001
 ImagenHub++ 0.060−- 0.049−- 0.028−- 0.035++ 0.019
Framing (padding)EBench-18K−- 0.105−- 0.044−- 0.004−- 0.018−- 0.002
 ImagenHub++ 0.014−- 0.018−- 0.045−- 0.017++ 0.014
Typographic (text overlay)EBench-18K−- 0.025−- 0.007++ 0.008++ 0.012−- 0.037
 ImagenHub++ 0.011−- 0.055−- 0.037−- 0.066−- 0.066
Appendix
Table 9: Cross-benchmark agreement and selection-corrected inference. Panel A reports Δρ\Delta\rho for the four cues evaluated on both anchor benchmarks; bold indicates that the uncorrected turn-clustered 95% confidence interval excludes zero. Panel B reports permutation tests with pselp_{\mathrm{sel}} correcting for selection of the most extreme cue.
Cuegpt-5.5gemini-3.5kimi-k2.5qwen3.5VIEScore
EBench-18K, 17 editors
Luminance (brightness)1.7  ∙\,\bullet0.5  ∙\,\bullet0.80.30.8  ∙\,\bullet
Framing (padding)1.7  ∙\,\bullet1.9  ∙\,\bullet1.11.61.9  ∙\,\bullet
Typographic (text overlay)3.6  ∙\,\bullet3.1  ∙\,\bullet2.2  ∙\,\bullet2.5  ∙\,\bullet1.5  ∙\,\bullet
Attention guidance (box)1.8  ∙\,\bullet2.6  ∙\,\bullet2.22.20.7  ∙\,\bullet
ImagenHub, 8 editors
Appendix
Table 10: Single-entrant leaderboard perturbations. Cells report the cued editor’s mean rank change; ∙\bullet indicates that the top-ranked editor changes.
JudgeRuleCoverageAcc. decidedChanged
gpt-5.5base order only0.9270.8450
 reconciled, lenient0.8700.88419
 reconciled, strict (MT-Bench)0.8080.8900
gemini-3.5base order only0.8360.8850
 reconciled, lenient0.8520.89317
 reconciled, strict (MT-Bench)0.7670.9140
Appendix
Table 11: Two-order reconciliation in pairwise judging. Strict follows the MT-Bench two-order rule ( Zheng et al., 2023 ) and retains only pairs whose two orders agree, whereas lenient additionally adopts the non-Tie choice when exactly one order returns Tie; pairs with opposite winners remain unresolved. The table reports coverage, accuracy on decided pairs, and the number of verdicts changed from the base-order verdict.
CueMembersΔ\Delta95% CIqq
Bandwagon5/5 up++ 0.157[ ++ 0.132, ++ 0.183]<< 0.001
Authority (model name)4/5, split−- 0.002[ −- 0.021, ++ 0.019]0.750
Colorfulness (saturation)1/5 down−- 0.021[ −- 0.040, −- 0.004]0.001
Aesthetic (aesthetic filter)1/5 down−- 0.032[ −- 0.053, −- 0.011]<< 0.001
Provenance (watermark)3/5 down−- 0.067[ −- 0.086, −- 0.047]<< 0.001
Luminance (brightness)5/5 down−- 0.069[ −- 0.089, −- 0.050]<< 0.001
Appendix
Table 12: Standardized-median ensemble across cues. Each judge is standardized on its un-cued distribution before aggregation; bold indicates Benjamini–Hochberg significance. “Members” summarizes significant effects in the five individual judges.
CueΔ\Delta95% CIFive judges
Bandwagon++ 1.27[ ++ 1.11, ++ 1.44]5/5, all up
Authority (model name)−- 0.09[ −- 0.21, ++ 0.04]4/5, split
Colorfulness (saturation)−- 0.22[ −- 0.38, −- 0.06]1/5, all down
Aesthetic (aesthetic filter)−- 0.20[ −- 0.40, −- 0.02]1/5, all down
Provenance (watermark)−- 0.08[ −- 0.32, ++ 0.15]3/5, all down
Luminance (brightness)−- 0.43[ −- 0.59, −- 0.27]5/5, all down
Appendix
Table 13: Replication of cue effects. Cells report paired mean score shifts for qwen3-vl-32b-instruct with 95% confidence intervals; bold indicates Benjamini–Hochberg significance within the replication family. “Five judges” summarizes significant effects among the five primary judges.
Figure 5: Scoring judge prompt. The shaded VIEScore paragraph is included only for the VIEScore scoring condition. The three dimension ratings are used individually or through their 3–30 sum in the reported analyses; overall_score and reason are required output fields but are not analyzed.
Figure 6: Pairwise judge prompt. All pairwise judges receive the same prompt template. The position intervention swaps candidates A and B without changing the prompt text.
Figure 7: Prompt-site cue templates. Tinted boxes contain the exact inserted strings. Braced fields are filled from the corresponding evaluation instance; the bandwagon target in pairwise evaluation is assigned with a fixed seed.
Figure 8: Preservation-validator prompt. The same prompt template is used for all three validators; only the evaluated images and editing instruction vary across instances.
Figure 9: Human preservation-annotation instructions. The card reproduces the written instructions for the two annotation packages and the additional independence instruction given to the second annotator. Presentation-only headings and rules are identified in the card.

Explore similar work

CardsList

  1. MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

    Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLlm-As-A-JudgeMultimodal Large Language Models

    Seoul National University · NAVER Cloud AI · KAIST AI

  2. On the Adversarial Robustness of Multimodal LLM Judges

    Jun 14, 2026Zihan Wang, Guansong Pang, Zelin Liu +3Large Language Model JudgesLlm-As-A-Judge

    School of Computer Science and Engineering, Beihang University, Beijing, China · School of Computing and Information Systems, Singapore Management University, Singapore · State Key Laboratory of Virtual Reality Technology and System, Beihang University, Beijing, China +1

  3. Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

    Jun 1, 2026Seojeong Park, Jiho Choi, Junyong Kang +3Llm-As-A-JudgeJudges

    Graduate School of Artificial Intelligence, KAIST, Republic of Korea · KRAFTON