cs.CVNov 9, 2026

EditJudge-Bench: Auditing VLM Image-Edit Judges with Synthetic Ground Truth

Authors: Jens Parslov, Marco Schouten, Dim P. Papadopoulos

Organizations: Technical University of Denmark · Pioneer Centre for AI

Abstract

Vision-language models (VLMs) are increasingly used as automated judges for image-editing benchmarks, yet their reliability is poorly understood. Auditing these judges is difficult because real edited images lack controlled ground truth and entangle the requested changes with model artefacts, unintended side effects, and ambiguous degrees of success. In this paper, we introduce EditJudge-Bench, a diagnostic benchmark of 1,382 synthetic before/after image pairs rendered in Blender, with edits parameterised across nine categories derived from real user requests and exact ground truth from known 3D scene parameters. Because every edit is generated by a known intervention on a controlled scene, EditJudge-Bench evaluates VLM judges as binary verifiers of whether an instruction was correctly executed, isolating judge perception from editor quality. Across 25 judge configurations, we find substantial reliability gaps: (a) the widely adopted VIEScore template reaches only 69.0% AUROC with GPT-4o; (b) fine-tuned judges score 51.7-66.3%; spatial edits such as rotation and shape remain challenging; (c) prompt-formatting choices like image-text ordering shift performance by up to 14.1 points. Despite using only synthetic data, judge rankings on EditJudge-Bench correlate strongly with two real-image human-preference benchmarks (ρρ=0.95 and ρρ=0.71), indicating that controlled audits surface failure modes relevant to real evaluation.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

    Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM RobustnessLLM-as-a-Judge

  2. GEditBench v2: A Human-Aligned Benchmark for General Image Editing

    Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing EvaluationText-Guided Image Editing