cs.AINov 10, 2026

Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics

Authors: Qinan Zhang, Xinyu Wang, Qihang Jin, Tianyuan Wang, Hongbing Luo, Pengfei Li, Wenjun Lv, Yu Kang, 

Organizations: University of Science and Technology of China

Abstract

Vision-language models (VLMs) are increasingly used as automated judges for multimodal and embodied tasks, yet aggregate accuracy or consistency alone does not reveal whether a judge changes its verdict for the right reason. We propose State-Conditional Invariance (SCI), a reliability principle requiring a judge to remain stable under semantics-preserving changes in presentation while responding to causal interventions that change the task outcome. We instantiate SCI in GridWM-Judge, a simulator-grounded diagnostic benchmark built from deterministic MiniGrid trajectories with aligned Full, evidence-removal, and counterfactual variants, plus presentation and representation probes. Across five diagnostic tasks, GridWM-Judge tests local perception, action-conditioned transitions, presentation stability, representation exchangeability, and Full-CF causal discrimination. Our diagnostic evaluations reveal heterogeneous failure modes, including sensitivity to evidence presentation and encoding, low activation on successful trajectories, and weak paired counterfactual discrimination. Perception-controlled analyses further indicate that visual grounding and objective recovery are substantial, model-dependent bottlenecks. These results motivate jointly reporting correctness, presentation stability, and causal sensitivity instead of relying on a single score.

Figures & tables

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

    Sep 29, 2026Nagham Omar, Mahmoud Jabarin, Kinan Ibraheem +1VLM EvaluationFigurative Language Understanding

  2. Evaluating the Diagnostic Robustness of Vision-Language Models Under Visual and Textual Perturbations

    Aug 5, 2026Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian +1VLM EvaluationVLM Robustness

  3. When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias

    Apr 20, 2026Xiaohan Zou, Roshan Sridhar, Mohammadtaher Safarzadeh +1VLM EvaluationVision-Language Alignment