Vision-Language Model-as-a-Judge
Overview
Vision-language model-as-a-judge studies how multimodal models can evaluate visual outputs and tasks, including image and video generation, image editing, and agent actions. Common approaches use pairwise comparisons, explicit rubrics, step-by-step visual verification, or learned reward models trained on human preferences. Benchmarks compare judgments with human ratings and test whether evaluators notice specific errors, follow instructions, and remain consistent across tasks and languages. A central open challenge is that strong average agreement can coexist with grounding failures, position and authority biases, uncertainty, and weak performance on unfamiliar or fine-grained criteria.
Latest papers
70 papers13 papers in the last four weeks, down 13% on the four weeks before. 0.1% of all new papers.
Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation
CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
MJ1: Multimodal Judgment via Grounded Verification
DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation
E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought
Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervision
30x larger baselines. The reasoning judge attains very high correlation with human ratings and 98.9% percent pairwise agreement on the human-annotated meta-evaluation subset. These results establish reasoning fidelity as a quantifiable dimension of deepfake detection and demonstrate scalable supervision for interpretable deepfake reasoning. Our user study shows that participants preferred the reasonings generated by our framework 70% of the time, in terms of faithfulness, groundedness, and usefulness, compared to those produced by other models and datasets. All of our datasets, models, and codebase are open-sourced.