Vision-Language Model-as-a-Judge

Overview

Vision-language model-as-a-judge studies how multimodal models can evaluate visual outputs and tasks, including image and video generation, image editing, and agent actions. Common approaches use pairwise comparisons, explicit rubrics, step-by-step visual verification, or learned reward models trained on human preferences. Benchmarks compare judgments with human ratings and test whether evaluators notice specific errors, follow instructions, and remain consistent across tasks and languages. A central open challenge is that strong average agreement can coexist with grounding failures, position and authority biases, uncertainty, and weak performance on unfamiliar or fine-grained criteria.

Latest papers

CardsList

70 papers13 papers in the last four weeks, down 13% on the four weeks before. 0.1% of all new papers.

  1. EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing

    Mar 16, 2026Zitong Xu, Huiyu Duan, Zhongpeng Ji +9Image Editing EvaluationText-Guided Image Editing

  2. Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge

    Mar 12, 2026Junjie Wu, Xuan Kan, Zihao He +3Vision-Language Model-as-a-JudgeMultimodal Model Evaluation

  3. HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation

    Mar 11, 2026Hongji Yang, Yucheng Zhou, Wencheng Han +3Vision-Language Model-as-a-JudgeVLM Evaluation

  4. MJ1: Multimodal Judgment via Grounded Verification

    Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMsMLLM Evaluation

  5. TIIF-Bench: How Does Your T2I Model Follow Your Instructions?

    Jun 2, 2025Xinyu Wei, Jinrui Zhang, Zeqing Wang +4Vision-Language Model-as-a-JudgeT2I Generation Evaluation

  6. Multimodal Language Models as Text-to-Image Model Evaluators

    May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-JudgeBenchmark Design