Model judges now supervise multimodal systems at scale, filtering training data, selecting outputs, and supplying the reward that shapes multimodal reasoning models. Trusting one means first checking that it uses its evidence, and that check is itself worth scrutinizing, so we ask whether a counterfactual probe of visual grounding measures what it claims to. The probe edits the image so the ground truth flips, holds the reasoning trace fixed, and asks whether the verdict follows. We formalize it as the Verdict Grounding Score and show it cannot be read the way such scores are read. A verdict responds only to an edit that reaches the judge's decision-relevant reading, so the score is capped by how perceptible the edit is, and unless editing makes the attribute easier to read, the error is one-sided: the score can only make a judge look less grounded than it is. The practical failure is therefore a false alarm, an auditor discarding a usable overseer. Under assumptions we state, we show this missing quantity is not merely bounded but identified from three quantities the same audit protocol already collects, which makes the false-alarm rate directly measurable rather than merely a concern. Auditing nine judges, we find the predicted ordering holds strictly across our entire primary pool, and the typical judge there acts on only about half of the edits whose attribute it can otherwise resolve. Applying a conservative rejection threshold certifies several cells as false alarms outright, the clearest being a judge that detects the injected error essentially every time while still scoring as if it had not used the image at all. The rule that follows is that an image-side counterfactual score should never be reported alone: a detection probe on the unedited image upper-bounds it, certifies its false alarms, and costs nothing extra to run.
Figures & tables
Figure 1: The reading confound and the ordering it predicts. (a) A verdict that does not follow a truth-flipping edit is consistent with two judges, one that ignores the image and one whose decision-relevant reading the edit never reached. VGS cannot separate them; GEDR can, because a GEDR above the image-blind rate excludes the first ( Proposition 8 ). (b) All 36 measured cells, plotted as (VGS,GEDR) . Corollary 6 predicts every cell falls on or above the diagonal; 35 do, and all 18 primary cells (ChartQA, HallusionBench) do so strictly. Primary cells are filled, secondary open. Bars are 95% paired bootstrap intervals on VGS; the exception’s interval spans the diagonal. Vertical distance to the diagonal is the shortfall GEDR−VGS , which Theorem 5 decomposes into missed edits and verdict instability. The banded region is where Corollary 7 certifies a false alarm ( VGS<0.20 with GEDR>21 ); the ringed cell in it is Llama-3.2-Vision on ScienceQA, at GEDR=1.00 and VGS=0.00 .
Table 1: One row per judge, averaged over ChartQA ( n=100 ), HallusionBench ( n=104 ), MathVista ( n=16 ), and ScienceQA ( n=22 ). All 36 cells are in Fig. 1(b) and Table A5 . Bars span each judge’s range on a common axis from −0.12 to 1.02 , ticked at the image-blind anchor 0 , with a dot at the mean. Columns are rounded independently.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Judge
Dataset
VGS(X+)
VGS(X†)
diff
LLaVA-OneVision
ChartQA
0.21
0.20
−0.01
Qwen2.5-VL-7B
ChartQA
0.49
0.41
−0.08
InternVL3-8B
ChartQA
0.29
0.27
−0.02
InternVL3-8B
MathVista
0.06
0.19
0.12
Llama-3.2-Vision
ChartQA
−0.05
0.01
0.06
GPT-4o
MathVista
0.12
0.19
0.06
Appendix
Table A1 : Artifact-matched control. The ten of 32 cells where the X† -based and X+ -based VGS differ by at least 0.01 ; the other 22 match to two decimals.
Figure A1 : The four image interventions. The reference cell is the unedited image x , about which the injected claim is false; here the trace claims that the tallest bar is blue and it is not. GroundJudge forms a truth-preserving re-encoding x+ , a blank x∅ and a mismatched x′ , and a counterfactual x~ edited so that the claim becomes true and the ground-truth verdict flips.
Figure A2 : The magnitude sweep, and its null result. Each curve is the per-magnitude VGS-like quantity of Sec. C.13 ( Sec. 4.2 , “What would close this, and what we ran”), for the four open-weight judges on ChartQA’s 511 magnitude-tagged counterfactual instances. The prediction was a monotone rise with ∣magnitude∣ on both sides of zero (shown schematically, low and unfit to any curve); three of four judges instead fall or stay flat, with monotone permutation p=0.80 to 0.98 . Llama-3.2-11B-Vision alone reaches significance ( p=0.019 ), on a curve whose seven points span only 0.027 , consistent with the verdict-binarization insensitivity this judge shows elsewhere in the audit ( Sec. C.15 ). Recomputed directly from the shipped per-instance verdicts by make_figS2.py , which asserts these exact figures at build time and fails if they drift.
Dataset
VGS
GEDR
Δnull
shortfall
edit-miss
ChartQA
0.040
0.210
0.110
0.170
0.060
HallusionBench
0.308
0.452
0.038
0.144
0.106
MathVista
0.000
0.375
0.125
0.375
0.250
ScienceQA
0.364
0.727
0.045
0.363
0.318
Appendix
Table A2 : Held-out judge (Qwen3-VL-8B-Instruct). The ordering holds strictly on all four cells, and the edit-miss rate is non-negative throughout.
Non-clean selection rate
Reward ρ
Judge
ChartQA
HallB.
MathV.
SciQA
mean
LLaVA-OneVision
0.20
0.27
0.75
0.05
0.40
Qwen2.5-VL-7B
0.23
0.11
0.06
0.00
0.34
InternVL3-8B
0.25
0.18
0.19
0.09
0.31
Llama-3.2-Vision
0.55
0.37
0.00
0.00
0.15
GPT-4o
0.20
0.07
0.38
0.00
0.32
Appendix
Table A3 : Downstream selection and reward-groundedness correlation, all eight judges, all four datasets. The non-clean selection rate is the rate at which the judge’s argmax pick against the true image is not the clean trace, from an independent candidate pool built from each dataset’s own ground-truth vocabulary; task accuracy is one minus this rate by construction, so only one column is given per dataset. The reward column is the Pearson correlation between each judge’s own main-audit pointwise score (as a reward) and ground-truth groundedness, averaged over the four per-dataset values reported in Table A4 . Every judge selects a non-clean candidate at a nonzero rate on at least one dataset; on both MathVista and ScienceQA several judges (four of eight on ScienceQA) never do, and those nulls are reported as found. Neither column orders judges the way mean VGS does: the reward correlation does not significantly correlate with mean VGS (Spearman ρ=0.62 , p=0.10 , n=8 ) or with mean selection rate ( ρ=−0.26 , p=0.53 ) across the eight-judge pool.
Judge
ChartQA
HallB.
MathV.
SciQA
Qwen2.5-VL-7B
0.33
0.31
0.17
0.54
InternVL3-8B
0.33
0.26
0.21
0.46
LLaVA-OneVision
0.40
0.22
0.00
0.47
Llama-3.2-Vision
0.02
0.00
0.00
0.56
VisualPRM-8B-v1.1
0.34
0.45
0.53
0.46
GPT-4o
0.30
0.35
0.12
0.50
Appendix
Table A4 : Reward-correlation analysis: reward-groundedness correlation, per dataset. Pearson correlation between each judge’s own main-audit pointwise score, used as a reward, and ground-truth groundedness (whether the trace is clean), reusing the main audit’s scores with no new judge calls. Row means appear in Table A3 ’s Reward column.
Grounding and detection
Gap
Ablation Δ
Judge
Dataset
VGS (95% CI)
GEDR
LEDR
Δnull
GEDR − VGS
blank
mismatch
Open-weight judges
LLaVA-OneVision
ChartQA
0.21 [0.13,0.30]
0.65
0.57
0.03
0.44
0.19
0.49
HallusionBench
0.15 [0.09,0.23]
0.56
0.26
0.01
0.41
0.31
0.22
MathVista
0.31 [0.12,0.56]
0.50
0.38
0.00
0.19
0.19
0.44
ScienceQA
0.59 [0.41,0.77]
0.86
0.36
0.00
0.27
0.18
0.50
Appendix
Table A5 : The full GroundJudge audit, all 36 judge × dataset cells. Instance counts are ChartQA n=100 , HallusionBench n=104 , MathVista n=16 , ScienceQA n=22 . ChartQA and HallusionBench are the primary pool of the main paper ( 18 cells); MathVista and ScienceQA are secondary and carry nothing in the headline. VGS carries a 95% paired nonparametric bootstrap interval ( 2,000 resamples over instances). Δnull is the flip rate under the truth-preserving re-encoding x+ . The gap is the quantity Corollary 6 predicts non-negative under its side condition; columns are rounded independently, so it can differ from the rounded difference by 0.01 . The ablation columns are binarized-verdict disagreement rates, not mean absolute change on the raw score. Section C.9 gives the full inferential detail: per-cell CIs for the shortfall, the pooled instance-level test, and per-cell permutation tests with Holm–Bonferroni correction.