Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics
Organizations: University of Science and Technology of China
Abstract
Vision-language models (VLMs) are increasingly used as automated judges for multimodal and embodied tasks, yet aggregate accuracy or consistency alone does not reveal whether a judge changes its verdict for the right reason. We propose State-Conditional Invariance (SCI), a reliability principle requiring a judge to remain stable under semantics-preserving changes in presentation while responding to causal interventions that change the task outcome. We instantiate SCI in GridWM-Judge, a simulator-grounded diagnostic benchmark built from deterministic MiniGrid trajectories with aligned Full, evidence-removal, and counterfactual variants, plus presentation and representation probes. Across five diagnostic tasks, GridWM-Judge tests local perception, action-conditioned transitions, presentation stability, representation exchangeability, and Full-CF causal discrimination. Our diagnostic evaluations reveal heterogeneous failure modes, including sensitivity to evidence presentation and encoding, low activation on successful trajectories, and weak paired counterfactual discrimination. Perception-controlled analyses further indicate that visual grounding and objective recovery are substantial, model-dependent bottlenecks. These results motivate jointly reporting correctness, presentation stability, and causal sensitivity instead of relying on a single score.
Figures & tables
| Condition | Question | Instantiation | Metrics |
|---|---|---|---|
| C1: Presentation Invariance | Stable under semantics-preserving presentation changes? | Task C: temporal, visual, and framing probes | NS, JCR, LES |
| C2: Action-conditioned Dynamics | Do transition predictions depend on actions? | Task A: next-state prediction; action shuffle/mask | , , |
| C3: Representation Exchangeability | Consistent across Composite/Split interfaces? | Task C: Composite/Split comparison | |
| C4: Causal Sensitivity | Does the judge distinguish matched Full/CF outcomes? | Task C: Full/CF pairs; Task D: reference-aided Full/CF | , , |
| Model | Deg. (C/S) | ||||||
|---|---|---|---|---|---|---|---|
| C/S | C | S | C | S | |||
| Claude Sonnet 4.6 † | 0.06/0.11 | 0.06 | 0.12 | 0.06 | 0.12 | 0.924 | AF/— |
| GPT-4o | 0.28/0.78 | 0.28 | 0.78 | 0.22 | 0.72 | 0.706 | —/— |
| GPT-5.4 | 0.06/0.06 | 0.06 | 0.06 | 0.06 | 0.06 | 1.000 | AF/AF |
| Gemini 3 Flash | 0.72/0.67 | 0.72 | 0.67 | 0.67 | 0.67 | 0.707 | —/— |
| Gemini 2.5 Flash Lite | 0.50/0.67 | 0.50 | 0.67 | 0.28 | 0.56 | 0.796 | —/— |
| Metric | Composite Mean | Split Mean | Mean | Median | W/T/L | -value |
|---|---|---|---|---|---|---|
| 0.433 | 0.494 | +0.061 | +0.028 | 5/2/3 | 0.617 | |
| 0.444 | 0.493 | +0.048 | +0.031 | 5/2/3 | 0.641 | |
| 0.163 | 0.337 | +0.173 | +0.031 | 5/5/0 | 0.063 | |
| 0.624 | 0.720 | +0.096 | +0.038 | 4/4/0 | 0.125 | |
| *For , models with undefined or low-activation conditional IDR are excluded from W/T/L counting according to the reporting rule in Appendix A . | ||||||
| Model | B_Acc | E_Acc | Acc_A1 | ||
|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.000 | 0.477 | 0.252 | +0.039 | +0.068 |
| GPT-4o | 0.000 | 0.495 | 0.214 | -0.039 | -0.068 |
| GPT-5.4 | 0.111 | 0.615 | 0.252 | +0.000 | -0.019 |
| Gemini 2.0 Flash | 0.007 | 0.274 | 0.175 | -0.038 | -0.025 |
| Gemini 2.5 Flash Lite | 0.000 | 0.345 | 0.204 | -0.039 | -0.029 |
| Gemini 3 Flash | 0.090 | 0.702 | 0.282 | +0.019 | +0.010 |
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Formula sketch | Task(s) | Interpretation |
|---|---|---|---|
| C/D | Neutral full-trajectory correctness anchor | ||
| C/D | Variant-wise full success acceptance | ||
| C/D | NoCue success accuracy; evidence-removal diagnostic | ||
| D; C if CF probes are included | Counterfactual-failure rejection accuracy | ||
| C/D | Overall success-verdict rate; label-bias indicator | ||
| C/D | Rate at which the judge accepts genuine full successes |
| Environment | Causal chain and success condition | NoCue evidence removal | CF intervention |
|---|---|---|---|
| DoorKey | Pick up key unlock/open matching door reach goal. Success requires positive reward and terminal goal entry. | Mask early frames in which the key is visible before pickup. | Relocate or remove the goal at so that the unchanged action sequence no longer reaches the true goal. |
| KeyCorridor | Pick up color-matched key return to locked door unlock door reach the mission target object. | Mask the door-matched key in the pre-pickup window. | Remove the mission target object before rollout, so the same action sequence cannot satisfy the pickup objective. |
| RedBlueDoor | Open red door open blue door pass through the ordered-door sequence. Opening blue before red is invalid. | Mask the red door before the first red-door interaction; blue-door evidence is retained. | Replace or block the blue-door passage before rollout, preserving the early red-door subtask but preventing final success. |
| LavaGap | Identify the single safe gap in the lava barrier cross through the gap reach goal without stepping on lava. | Mask visible lava evidence in early pre-crossing frames. | Relocate or remove the goal outside the planned path while keeping the safe-crossing behavior physically valid. |
| Memory | Observe cue object traverse corridor choose the matching terminal object. | Mask the cue object before corridor entry. | Swap the correct and distractor terminal objects before rollout, so the same remembered choice becomes wrong. |
| MultiRoom | Traverse a sequence of rooms, opening intervening doors, then enter the goal room and reach the goal. | Mask early door/goal evidence before entering the goal-containing room. | Remove the final goal object before rollout, preserving room traversal but preventing terminal success. |
| Environment | Masked target evidence | Max masked frames | Alignment threshold |
|---|---|---|---|
| DoorKey | Key before pickup | 5 | |
| KeyCorridor | Color-matched key before pickup | 5 | |
| RedBlueDoor | Red door before first red-door toggle | 5 | |
| LavaGap | Lava evidence before safe crossing | 4 | |
| Memory | Cue object before corridor entry | 10 | |
| MultiRoom | Door/goal evidence before goal-room entry | 12 |
| Gate | Purpose | Rule | Scope |
|---|---|---|---|
| D1 | Action identity | The action IDs are identical across matched variants in the same group. | all variants |
| D2 | Length consistency | Frames and states align with the action sequence: . | all variants |
| D3 | Input isolation | Model-facing inputs contain only rendered evidence, mission/action text, and task instructions; labels, rewards, states, and intervention metadata remain audit-only. | all tasks |
| D4 | Outcome contract | Full succeeds, NoCue preserves the Full outcome, and CF fails without truncation artifacts. | C/D |
| D5 | No interaction masking | NoCue never masks the frame in which the agent is facing or interacting with the target evidence. | NoCue |
| D6 | Physics validity | NoCue preserves the exact state trajectory; CF is a valid rerollout under the same action sequence, with divergence allowed only as a consequence of the declared intervention. | all variants |
| Task | Diagnostic role | Environments | Exam rows | Variant/probe structure |
|---|---|---|---|---|
| A | Action-conditioned next-observation prediction | Six core environments | 103 | Local transitions from 18 source trajectories; 103 baseline questions |
| B | Single-frame perception and interface control | Six core environments | 144 | : 12 source groups per environment, two frames per group; structured scene parsing |
| C | Presentation invariance and representation sensitivity | KeyCorridor, MultiRoom, RedBlueDoor | 486 | 18 groups: temporal/visual items, plus framing items |
| D | Causal sensitivity under matched intervention | Six core environments | 144 | : two query groups per environment, three variants, four presentation conditions; successful Full references |
| E | Prefix/checkpoint perception control | UnlockPickup, BlockedUnlockPickup | 325 | 48 source trajectories (24 per environment); 136 identity and 189 checkpoint-status questions |
| Task | Model-facing evidence | Required answer | Diagnostic role |
|---|---|---|---|
| B | Single MiniGrid frame with a generic scene-description instruction | Structured JSON scene description or component labels | Perception and interface control |
| E | Prefix-only sparse storyboard before the terminal state | JSON label for the first picked object or checkpoint status | Object/state tracking control |
| C | Sparse trajectory storyboard with presentation, framing, and representation probes | Binary success/failure verdict | Presentation invariance; representation exchangeability |
| A | Current observation, action condition, and four candidate next observations | Multiple-choice next-state label | Action-conditioned transition diagnostic |
| D | Full-trajectory evidence for matched Full and CF variants | Binary success/failure verdict | Causal sensitivity under intervention |
| Axis | Values | Interpretation |
|---|---|---|
| Trajectory condition | Full , NoCue , CF | Changes the evidence condition or the underlying outcome; not all levels are semantics-preserving. |
| Temporal order | original, reversed | C1 presentation probe over frame ordering. |
| Visual style | clean, noisy, style-transformed | C1 presentation probe over low-level rendering perturbations. |
| Prompt framing | neutral, positive, negative | C1 presentation probe over wording or prior-framing changes, without changing the task goal or evidence. |
| Representation | composite storyboard, split images | C3 equivalent-evidence-format probe for representation exchangeability. |
| Model | |
|---|---|
| Claude Sonnet 4.6 | 0.69 |
| Gemini 2.0 Flash | 0.66 |
| Gemini 2.5 Flash Lite | 0.66 |
| Gemini 3 Flash | 0.73 |
| GPT-4o | 0.71 |
| GPT-5.4 | 0.66 |
| Model | Group 95% CI | Env. 95% CI | ||
|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.250 | 0.250 | [0.000,0.500] | [0.000,0.583] |
| Gemini 2.0 Flash | 0.667 | 0.583 | [0.333,0.833] | [0.250,0.917] |
| Gemini 2.5 Flash Lite | 0.750 | 0.167 | [0.000,0.417] | [0.000,0.333] |
| Gemini 3 Flash | 0.833 | 0.167 | [0.000,0.417] | [0.000,0.500] |
| GPT-4o | 0.750 | 0.500 | [0.250,0.750] | [0.167,0.833] |
| GPT-5.4 | 0.750 | 0.333 | [0.083,0.583] | [0.083,0.667] |
| Model | Parsed | API | Trunc. | Other unparse. | [email protected] | Mean-all / parsed |
|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 141 | 0 | 0 | 3 | 0.000 | 0.186 / 0.190 |
| Gemini 2.0 Flash | 63 | 81 | 0 | 0 | 0.007 | 0.099 / 0.226 |
| Gemini 2.5 Flash Lite | 140 | 0 | 0 | 4 | 0.000 | 0.157 / 0.161 |
| Gemini 3 Flash | 30 | 0 | 0 | 114 | 0.090 | 0.168 / 0.807 |
| GPT-4o | 144 | 0 | 0 | 0 | 0.000 | 0.223 / 0.223 |
| GPT-5.4 | 144 | 0 | 0 | 0 | 0.111 | 0.616 / 0.616 |
| Model | Position | Carrying | Front cell | Object Jaccard | |
|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 141 | 0.064 | 0.709 | 0.035 | 0.182 |
| Gemini 2.0 Flash | 63 | 0.206 | 0.714 | 0.095 | 0.052 |
| Gemini 2.5 Flash Lite | 140 | 0.043 | 0.721 | 0.007 | 0.111 |
| Gemini 3 Flash | 30 | 1.000 | 0.867 | 0.533 | 0.817 |
| GPT-4o | 144 | 0.201 | 0.708 | 0.090 | 0.058 |
| GPT-5.4 | 144 | 0.903 | 0.847 | 0.444 | 0.228 |
| Model | Evidence | Task | Objective | Verdict | Joint |
|---|---|---|---|---|---|
| GPT-5.4 | Pixel | 0.442 | 0.405 | 0.731 | 0.320 |
| GPT-5.4 | State | 0.905 | 0.905 | 0.895 | 0.820 |
| GPT-5.4 | Pixel+State | 0.854 | 0.874 | 0.881 | 0.765 |
| Gemini 3.1 Pro | Pixel | 0.704 | 0.395 | 0.755 | 0.347 |
| Gemini 3.1 Pro | State | 0.949 | 0.912 | 0.973 | 0.895 |
| Gemini 3.1 Pro | Pixel+State | 0.983 | 0.929 | 0.997 | 0.929 |
| Model | Evidence | Group 95% CI | |||
|---|---|---|---|---|---|
| GPT-5.4 | Pixel | 0.745 | 0.469 | [0.367,0.571] | 0.630 |
| GPT-5.4 | State | 0.929 | 0.765 | [0.684,0.847] | 0.824 |
| GPT-5.4 | Pixel+State | 0.918 | 0.765 | [0.684,0.847] | 0.833 |
| Gemini 3.1 Pro | Pixel | 0.612 | 0.602 | [0.500,0.694] | 0.983 |
| Gemini 3.1 Pro | State | 1.000 | 0.929 | [0.878,0.980] | 0.929 |
| Gemini 3.1 Pro | Pixel+State | 1.000 | 0.990 | [0.969,1.000] | 0.990 |
| Probe | Exact pairs | Degradation | Rescue | Acc. | Correctness flip |
|---|---|---|---|---|---|
| Noise | 1080 | 59 | 75 | +0.015 | 0.124 |
| Style | 1080 | 46 | 55 | +0.008 | 0.094 |
| Temporal reversal | 1080 | 136 | 92 | -0.041 | 0.211 |
| Variant | Pairs | Verdict flip | Gain / Loss | Accuracy | 95% CI |
|---|---|---|---|---|---|
| Full | 1620 | 35.7% | 277 / 274 | +0.002 | [-0.173,0.167] |
| NoCue | 1620 | 33.8% | 235 / 293 | -0.036 | [-0.198,0.122] |
| CF | 1620 | 25.6% | 328 / 84 | +0.151 | [0.028,0.272] |
| Model | Prompt | Full accept | CF reject | [95% CI] | |
|---|---|---|---|---|---|
| GPT-5.4 | Direct | 0.827 | 0.541 | 0.367 [0.276,0.459] | 0.444 |
| GPT-5.4 | Structured verdict | 0.704 | 0.520 | 0.265 [0.184,0.357] | 0.377 |
| GPT-5.4 | Structured rationale | 0.449 | 0.878 | 0.398 [0.306,0.490] | 0.886 |
| Claude Sonnet 4.6 | Direct | 0.255 | 0.939 | 0.224 [0.143,0.306] | 0.880 |
| Claude Sonnet 4.6 | Structured verdict | 0.694 | 0.816 | 0.541 [0.439,0.643] | 0.779 |
| Claude Sonnet 4.6 | Structured rationale | 0.520 | 0.796 | 0.367 [0.276,0.459] | 0.706 |
| Task | Base rows | Expanded rows | Ratio | Source groups | Env. configs |
|---|---|---|---|---|---|
| A | 103 | 1,030 | 76 | 9 | |
| C | 486 | 1,326 | 442 | 10 | |
| D | 144 | 1,296 | 432 | 9 |
| Task | Model | Groups | CF rej. | Group 95% CI | Env. 95% CI | ||
|---|---|---|---|---|---|---|---|
| C | GPT-5.4 | 442 | 0.523 | 0.629 | 0.314 | [0.271,0.357] | [0.249,0.372] |
| C | Claude Sonnet 4.6 | 442 | 0.043 | 0.993 | 0.043 | [0.025,0.063] | [0.009,0.107] |
| D | GPT-5.4 | 432 | 0.720 | 0.530 | 0.322 | [0.278,0.366] | [0.226,0.461] |
| D | Claude Sonnet 4.6 | 432 | 0.113 | 0.933 | 0.088 | [0.062,0.116] | [0.037,0.189] |
| D | Gemini 3 Flash | 432 | 0.972 | 0.637 | 0.613 | [0.567,0.660] | [0.335,0.767] |
| D | Qwen3-32B | 432 | 1.000 | 0.000 | 0.000 | [0.000,0.000] | [0.000,0.000] |
| Model | Task C Acc. | Task A Acc. | Task E Acc. | Strict Rate |
|---|---|---|---|---|
| LLaVA-Critic-R1-7B | 0.33 | 0.28 | 0.11 | 0.27 |
| LLaVA-Critic-7B | 0.65 | 0.25 | 0.00 | 0.00 |
| LLaVA-OneVision-7B | 0.63 | 0.27 | 0.00 | 0.00 |
| LLaVA-OneVision-Chat | 0.63 | 0.24 | 0.00 | 0.00 |
| Mean | 0.56 | 0.26 | 0.03 | – |
| Component | Diagnostic role | Model-facing evidence | Reported output |
|---|---|---|---|
| MiniWorld suite | 3D external-validity probe | First-person RGB storyboards; simulator state traces retained as audit metadata | Env./group/item counts |
| Task A-style | C2: action-conditioned evidence use | Current 3D observation, action text, and candidate next observations or trajectory slices | |
| Task C-style | C1: presentation-sensitive trajectory judgment | Composite 3D storyboard with mission and action context | |
| Framing probe | C1: prompt-framing sensitivity | Alternative framing prompt applied to the same 3D trajectory evidence | |
| Task D-style | C4: causal sensitivity | Successful trajectory evidence versus matched failure or counterfactual evidence | , |
| Model | |||||
|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 0.50 | 0.38 | 0.68 | ||
| Gemini 2.0 Flash | 0.40 | 0.43 | 0.20 | ||
| Gemini 2.5 Flash Lite | 0.48 | 0.35 | 0.50 | ||
| Gemini 3 Flash | 0.55 | 0.81 | 0.78 | ||
| GPT-4o | 0.55 | 0.75 | 0.50 | ||
| GPT-5.4 | 0.50 | 0.51 | 0.66 |
| Model | Task | Scenes | Acc. | CF rej. | Group 95% CI | |||
|---|---|---|---|---|---|---|---|---|
| GPT-5.4 | A | 150 | 10 | 0.447 | – | – | – | – |
| Gemini 3 Flash | A | 150 | 10 | 0.553 | – | – | – | – |
| GPT-5.4 | C | 100 | 18 | 0.590 | 0.280 | 0.900 | 0.230 | [0.150,0.320] |
| Gemini 3 Flash | C | 100 | 18 | 0.720 | 0.870 | 0.570 | 0.470 | [0.370,0.570] |
| GPT-5.4 | D | 100 | 18 | 0.550 | 0.580 | 0.520 | 0.160 | [0.090,0.240] |
| Gemini 3 Flash | D | 100 | 18 | 0.590 | 0.970 | 0.210 | 0.200 | [0.120,0.280] |
| Dependency | Version | License | Purpose |
|---|---|---|---|
| MiniGrid | 3.0.0 | Apache 2.0 | Grid-world environments |
| Gymnasium | 1.2.3 | MIT | Environment interface |
| OpenAI SDK | 2.29.0 | Apache 2.0 | API inference client |
| MiniWorld | 2.1.0 | Apache 2.0 | 3D environments |
| Pygame | 2.6.1 | LGPL | Rendering support |
| Pillow | 12.1.1 | HPND | Image processing |
| Alias in paper | Model family / provider | Inference route |
|---|---|---|
| Claude Sonnet 4.6 | Claude Sonnet family | API |
| GPT-4o | GPT-4o family | API |
| GPT-5.4 | GPT-5-family judge run | API |
| Gemini 3 Flash | Gemini Flash family | API |
| Gemini 3.1 Pro | Gemini Pro family (Oracle analysis) | API |
| Gemini 2.5 Flash Lite | Gemini Flash Lite family | API |
| Item | Documentation / release location |
|---|---|
| Code | public GitHub repository |
| Dataset | public Hugging Face dataset |
| Exam files | JSONL files in dataset release |
| Rendered images | image folders referenced by JSONL records |
| Prompt templates | repository and dataset metadata |
| Ground-truth labels | audit-only fields in JSONL records |
| Axis | What the paper supports | What remains outside scope |
|---|---|---|
| Domain | Controlled MiniGrid diagnostics with auditable states, actions, and success predicates. | Direct deployment claims for real robots, photorealistic video, or open-ended embodied tasks. |
| Agent behavior | Deterministic rollouts that isolate judge-side reliability failures. | Noisy control, stochastic policies, suboptimal exploration, and long-horizon recovery behavior. |
| Presentation | Temporal, visual, prompt-framing, and representation-format probes over matched trajectory semantics. | All possible UI formats, natural-language task framings, or video encodings. |
| Models | A finite set of model-based judges under documented task-specific evaluation protocols. | Universal claims about all VLMs, future models, or heavily optimized prompting/training regimes. |
| Metrics | Diagnostic evidence for correctness, presentation stability, representation exchangeability, and causal sensitivity. | A single deployment-readiness score or safety certificate. |
| Consideration | Potential concern | Mitigation in this paper / release |
|---|---|---|
| Overclaiming | Treating SCI scores as deployment-readiness guarantees. | Explicitly state the controlled diagnostic scope and report limitations alongside results. |
| Benchmark gaming | Tuning models to public prompts or examples without improving real reliability. | Document prompts and probes; plan versioned splits and held-out diagnostic slices for future releases. |
| Misuse of judge diagnostics | Crafting inputs that exploit known judge weaknesses. | Frame results as reliability audits and avoid presenting the benchmark as an adversarial attack toolkit. |
| Privacy and data rights | Releasing sensitive images or trajectories. | Use simulator-generated synthetic data with no human subjects or private data. |
| Compute cost | Repeated VLM inference across many variants. | Provide modular subsets, deterministic scripts, and transparent compute reporting. |
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 1,050 papers