Vision-Language Model-as-a-Judge

Follow this topic Ask about this topic

Vision-language model-as-a-judge studies how multimodal models can evaluate visual outputs and tasks, including image and video generation, image editing, and agent actions. Common approaches use pairwise comparisons, explicit rubrics, step-by-step visual verification, or learned reward models trained on human preferences. Benchmarks compare judgments with human ratings and test whether evaluators notice specific errors, follow instructions, and remain consistent across tasks and languages. A central open challenge is that strong average agreement can coexist with grounding failures, position and authority biases, uncertainty, and weak performance on unfamiliar or fine-grained criteria.

Latest papers

59 papers15 papers in the last four weeks, up 88% on the four weeks before. 0.1% of all new papers.

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation
Jun 18, 2026Ali Asaria, Tony Salomone, Deep GandhiVLM Evaluation 3D Asset Generation
Transformer Lab
TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting
Jun 15, 2026Zhi Chen, Yuxuan Wang, Jialong Wu +5VLM Evaluation Forecasting Benchmarks
School of Software, BNRist, Tsinghua University, Beijing 100084, China
TECCI: Tricky Edits of Collected and Curated Images
May 31, 2026Aishwarya Agrawal, Roy Hirsch, Yasumasa Onoe +2VLM Evaluation Image Editing Evaluation
Google Research · Google DeepMind
A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation
May 29, 2026Yi Zhao, Siqi Wang, Zhe Hu +2VLM Evaluation LLM-as-a-Judge
Department of Computing, The Hong Kong Polytechnic University
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
May 10, 2026Rui Liu, Dian Yu, Zhenwen Liang +6Multimodal Understanding Multimodal Reward Modeling
1Tencent Hunyuan · University of Maryland, College Park · University of North Carolina, Chapel Hill
WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models
May 5, 2026Karthik Inbasekar, Guy Rom, Omer ShlomovitsHuman Preference Evaluation Video Quality Assessment
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
Apr 27, 2026Arya Shah, Deepali Mishra, Chaklam SilpasuwanchaiVLM Evaluation Vision-Language Models
IIT Gandhinagar Gandhinagar, India · IIT Kanpur Kanpur, India · Asian Institute of Technology Bangkok, Thailand
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM Evaluation VLM Robustness
♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad
Lost in Translation: Do LVLM Judges Generalize Across Languages?
Apr 21, 2026Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem +5Multimodal Model Evaluation Multilingual VLM Evaluation
York University · University of Alberta · Nanyang Technological University +1
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
Apr 20, 2026Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1VLM Evaluation Vision-Language Alignment
The Pennsylvania State University · Oracle AI
Redefining Quality Criteria and Distance-Aware Score Modeling for Image Editing Assessment
Apr 14, 2026Xinjie Zhang, Qiang Li, Xiaowen Ma +3Image Editing Evaluation Image Quality Assessment
Northwestern Polytechnical University, China · Shenzhen Research Institute of Northwestern Polytechnical University, China
Imperfect Visual Verification for Code Edition : A Case Study on TikZ
Apr 9, 2026Charly Reux, Mathieu Acher, Djamel Eddine Khelladi +2LLM Self-Refinement Code Generation
UR, INSA Rennes, DiverSe · Univ Rennes, Inria, IRISA, INSA, Rennes, France · CNRS, IUF, IRISA, UR, DiverSe +7
Vision-Guided Iterative Refinement for Frontend Code Generation
Apr 7, 2026Hannah Sansford, Derek H. C. Law, Wei Liu +3Code Generation Vision-Language Model-as-a-Judge
School of Mathematics, University of Bristol, UK · Amazon AGI
LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspection
Apr 7, 2026Akram Hossain, Rabab Abdelfattah, Xiaofeng Wang +1LLM Reliability Vision-Language Model-as-a-Judge
School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406 USA. · Electrical Engineering Department, University of South Carolina, Columbia, SC, USA. · Computer Science Department, Faculty of Computers & Artificial Intelligence, Fayoum University, Fayoum, Egypt.
GEditBench v2: A Human-Aligned Benchmark for General Image Editing
Mar 30, 2026Zhangqi Jiang, Zheng Sun, Xianfang Zeng +7Image Editing Evaluation Text-Guided Image Editing
Nanyang Technological University · StepFun · Southeast University
AIBench: Evaluating Visual-Logical Consistency in Academic Illustration Generation
Mar 30, 2026Zhaohe Liao, Kaixun Jiang, Zhihang Liu +11Image Generation Evaluation Image Generation
Tongyi Lab, Alibaba Group · SJTU · FDU +3
Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approach
Mar 20, 2026Shiqi Gao, Zitong Xu, Kang Fu +3Image Editing Evaluation Text-Guided Image Editing
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
EditHF-1M: A Million-Scale Rich Human Preference Feedback for Image Editing
Mar 16, 2026Zitong Xu, Huiyu Duan, Zhongpeng Ji +9Image Editing Evaluation Text-Guided Image Editing
Shanghai Jiao Tong University · Vivo Mobile Communication Co., Ltd · University of Electronic and Science Technology of China +2
Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
Mar 12, 2026Junjie Wu, Xuan Kan, Zihao He +3Vision-Language Model-as-a-Judge Multimodal Model Evaluation
Meta AI · Hong Kong University of Science and Technology · Emory University
HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation
Mar 11, 2026Hongji Yang, Yucheng Zhou, Wencheng Han +3Vision-Language Model-as-a-Judge VLM Evaluation
SKL-IOTSC, CIS, University of Macau · CSE, Shandong University · Online-Video BU, Tencent
CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
Mar 11, 2026Marta Sumyk, Oleksandr KosovanComputer-Use Agent Evaluation AI Agent Auditing
Ukrainian Catholic University Lviv, Ukraine
MJ1: Multimodal Judgment via Grounded Verification
Mar 9, 2026Bhavesh Kumar, Dylan Feng, Leonard TangPosition Bias in VLMs MLLM Evaluation
Haize Labs
DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation
Feb 26, 2026Varun Gopal, Rishabh Jain, Aradhya Mathur +6Human Preference Modeling Reward Modeling
MDSR, Adobe
E-comIQ-ZH: A Human-Aligned Dataset and Benchmark for Fine-Grained Evaluation of E-commerce Posters with Chain-of-Thought
Feb 25, 2026Meiqi Sun, Mingyu Li, Junxiong ZhuText Rendering in T2I Generation Automated Evaluation
Taobao & Tmall Group, Alibaba Group
Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervision
Feb 23, 2026Kartik Kuckreja, Parul Gupta, Muhammad Haris Khan +1Deepfake Detection Explainable Deepfake Detection
MBZUAI · Monash University *
TIIF-Bench: How Does Your T2I Model Follow Your Instructions?
Jun 2, 2025Xinyu Wei, Jinrui Zhang, Zeqing Wang +4Vision-Language Model-as-a-Judge T2I Generation Evaluation
The Hong Kong Polytechnic University · OPPO Research Institute · Tsinghua University
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3

Jun 18, 2026J/K move · Enter open · S save

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

Ali Asaria, Tony Salomone, Deep Gandhi

Transformer Lab

VLM Evaluation 3D Asset Generation Vision-Language Model-as-a-Judge

Open paper AskPDF arXiv

A companion study established a de-biased, cross-model VLM-as-3D-judge that reliably ranks single-image-to-3D mesh quality where cheap geometry and CLIP proxies fall short. This paper asks: can that judge's preferences specialize a strong open generator, TRELLIS, on one asset class (furniture), cheaply and without human labels? Taking the judge from ranking to optimization is where the work lives. Pushing a VLM judge into the training and evaluation loop exposes failure modes ranking never triggered, so our contribution is an optimization-grade hardening of the judge: a training judge (Qwen2.5-VL-7B) held distinct from an evaluation judge (InternVL3-8B) to break circularity; position-bias correction; and fixes for three failure modes (image overload, geometry-hiding splat renders, and reference-free judging that rewards clean-but-wrong outputs), with calibration evidence (clear-gap win-rate 0.83-1.0; base-vs-base ~0.5). Using this protocol as an independent evaluator, and working only from public models and data with lightweight parameter-efficient adaptation, we find our methods match the strong base rather than exceed it. Independent base samples carry essentially no learnable preference (0.94 order-flip rate), so signal must be engineered by quality-contrastive construction. Across six adaptation methods, two input regimes, and a severity sweep, the most targeted - conditioner repair under severe degradation - reaches parity (0.50) with the base, while no method clears the >=65% win-rate target. The result is mechanistic: clean inputs saturate the judge, flow-DIT fine-tuning washes out through the sampler, and conditioning repair is the locus that moves geometry. Win-rates are directional at n=8 objects. Matching a strong public-data base with cheap adaptation is itself informative: exceeding it needs more than lightweight PEFT on public data, and the judge protocol is reusable.