Vision-Language Model-as-a-Judge

Overview

Vision-language model-as-a-judge studies how multimodal models can evaluate visual outputs and tasks, including image and video generation, image editing, and agent actions. Common approaches use pairwise comparisons, explicit rubrics, step-by-step visual verification, or learned reward models trained on human preferences. Benchmarks compare judgments with human ratings and test whether evaluators notice specific errors, follow instructions, and remain consistent across tasks and languages. A central open challenge is that strong average agreement can coexist with grounding failures, position and authority biases, uncertainty, and weak performance on unfamiliar or fine-grained criteria.

Latest papers

CardsList

70 papers13 papers in the last four weeks, down 13% on the four weeks before. 0.1% of all new papers.

  1. Stress Testing a Multimodal Large Language Model Grading Assistant on Kinematics Graph Reading

    Nov 17, 2026Ons Zammel, Carlos Soares, Flávia Carvalhido +1Automated GradingMLLM Evaluation

  2. LEDGE: Separating Critic and Editor Failures in Text-to-Image Self-Correction

    Nov 16, 2026Vyom Agarwal, Mokshda Gangrade, Samet Ayhan +1Self-Correction in VLMsT2I Generation

  3. StyleJudge: A Human-Validated, In-Context-Learning Vision-Language Evaluator for Artistic Style Transfer

    Nov 14, 2026Lokesh, Pramukto Mandal, Somanshu Juneja +6Vision-Language Model-as-a-JudgeImage Style Transfer

  4. Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models

    Nov 11, 2026Shurun Li, Michael Hua Wang, Haibo Zeng +1Automated EvaluationT2I Generation Evaluation

  5. Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics

    Nov 10, 2026Qinan Zhang, Xinyu Wang, Qihang Jin +6VLM EvaluationVision-Language Model-as-a-Judge

  6. EditJudge-Bench: Auditing VLM Image-Edit Judges with Synthetic Ground Truth

    Nov 9, 2026Jens Parslov, Marco Schouten, Dim P. PapadopoulosVLM EvaluationVision-Language Model-as-a-Judge

  7. ScreenProof: Where Is This Screen Wrong? Finding UI Faults Without a Reference

    Nov 6, 2026Huiming Sun, Yue Li, Kangning Yang +6Fault DetectionFault Localization

  8. Grounding-Aware Judge Reward Reinforcement Learning for Out-of-Domain Face Anti-Spoofing

    Nov 6, 2026Xiaoyong Yu, Xinge YuFace Presentation Attack DetectionOOD Generalization

  9. VLM-as-a-Judge for Person Re-Identification

    Nov 2, 2026Bessie Dominguez-Dager, Miguel Cazorla, Vincent LepetitPerson Re-IdentificationVision-Language Model-as-a-Judge

  10. EditAgent: Rethinking Agentic Image Editing

    Oct 29, 2026Haiyang Xu, Tianyu Wang, Jialu Li +5Image EditingTool-Using Vision-Language Agents

  11. CropReward: A Human-Preference Benchmark and Reward Model for Aesthetic Image Cropping

    Oct 25, 2026Xiangfei Sheng, Xiaofeng Pan, Leida Li +4Reward ModelingHuman Preference Modeling

  12. Execution Logs Sway VLM Judges of Robot Progress

    Oct 25, 2026Weiyan Zhao, Sunny YangVLM RobustnessVision-Language Model-as-a-Judge

  13. ImageRSI: Verification-Driven Recursive Self-Improvement for Agentic Image Generation

    Oct 23, 2026Shuo Han, Zhilin Wang, Peng Ling +5Agentic Image GenerationImage Generation

  14. RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

    Oct 20, 2026Siyuan Ma, Yu Qi, Cheng Ye +8Vision-Language Model-as-a-JudgeVideo Generation Evaluation

  15. A Plausible Explanation Is Enough to Mislead Multimodal Judges

    Oct 15, 2026Anglin Cai, Zhiqi Wang, Xiaozhi Yang +3Bias in LLM-as-a-JudgeVision-Language Model-as-a-Judge

  16. OmniVerifier-C1: Decomposed Graded Verifier for Rule-Governed Generative World Modeling

    Oct 15, 2026Bowei Liu, Xinchen Zhang, Xuhuan Li +10Multimodal Model EvaluationVision-Language Model-as-a-Judge

  17. VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

    Oct 12, 2026Subhadeep Roy, Christoph Leiter, Steffen EgerVLM EvaluationImage Captioning Evaluation

  18. Visual Jev Rewards: Reference-Bound Verification for Multi-Subject Image Generation

    Oct 7, 2026Baoteng Li, Wenzhuo Wu, Kongming Liang +1Multi-Subject Image GenerationReward Modeling

  19. DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

    Oct 2, 2026Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes +2Text-Image AlignmentT2I Generation Evaluation

  20. Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

    Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM RobustnessLLM-as-a-Judge

  21. VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

    Sep 30, 2026Vu Dinh Xuan, Duc-Hai Nguyen, Minh-Dung Dao +6VLM EvaluationVision-Language Model-as-a-Judge

  22. It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

    Sep 29, 2026Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem +1VLM EvaluationFigurative Language Understanding

  23. Undertone: Personalizing Frozen VLM Aesthetic Judges from Few Ratings

    Sep 22, 2026Amit Jadhav, Shaurya Beriwala, Beomjin KimVLM EvaluationVision-Language Models

  24. RULER: Instance-aware Rubric Rewards for SVG Generation

    Sep 21, 2026Hangyu Ran, Yuhao Zheng, Yingying Zhang +2Text-to-SVG GenerationVector Graphics Generation

  25. ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

    Sep 21, 2026Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang +3VLM EvaluationLLM Evaluation