Multimodal large language models (MLLMs) are typically evaluated by whether they produce correct answers, but correctness alone does not reveal whether their outputs are supported by the right visual evidence. We introduce VIGOR, a benchmark for evaluating VIsually GrOunded Rationales in MLLMs. VIGOR represents visual rationales as explicit concepts paired with pixel-level groundings, covering multiple abstraction levels including low-level visual properties, intermediate structures, objects, parts and high-level motion, relations. To construct VIGOR at scale, we develop an automatic annotation framework that reuses existing dense supervision when available and generates missing concept groundings through proposal-based annotation followed by conservative multimodal verification. Using VIGOR, we evaluate whether MLLMs "look at what they say" by comparing model-derived visual evidence for each expressed concept against annotated ground-truth rationales. Our results show that current MLLMs exhibit limited visual rationale correctness and reveal distinct grounding failures across concept levels. VIGOR provides a benchmark and scalable data construction pipeline for studying whether MLLM outputs are supported by appropriate visual evidence. Code is available at https://github.com/deep-real/VIGOR
Figures & tables
Figure 1: Illustration of the double correctness problem. Although the model produces a correct description by mentioning “traffic lights,” its visual rationale is misaligned with the actual traffic light regions, showing the gap between output correctness and rationale correctness .
Figure 2: Overview of the VIGOR annotation pipeline. VIGOR builds a hierarchical concept vocabulary from heterogeneous source data, proposes masks with multiple annotation tools when source masks are unavailable, verifies candidate masks through MLLM voting, and combines verified masks with reusable source annotations to produce pixel-level visual rationales.
Figure 3
Figure 4: Human evaluation of VIGOR annotation quality across concept levels. Most annotations are rated as Agree or Strongly Agree , indicating consistently high reliability of the propose-then-verify pipeline across different concept levels.
Model
Overall
Primitive
Intermediate
Object
Scene
AP
Pointing
AP
Pointing
mAP
Pointing
mAP
Pointing
mAP
Pointing
Random
5.8
5.7
4.0
3.9
1.9
1.8
5.0
4.9
17.7
17.3
InternVL3-1B
19.6
27.7
28.7
40.7
9.9
14.1
22.3
32.9
24.9
30.5
InternVL3.5-1B
15.0
13.7
14.5
12.3
8.0
9.0
17.0
15.6
22.4
18.1
InternVL3-2B
20.1
27.7
26.6
39.0
10.9
15.3
23.1
32.8
25.0
28.9
InternVL3.5-2B
28.8
36.5
36.9
46.9
16.8
21.3
34.0
43.7
31.0
37.2
Table 2: Main results on VIGOR benchmark. We report AP and Pointing Accuracy for overall grounding and for primitive, intermediate, object, and scene-level concepts. The best performing model in each column is bolded and the second best in underlined .
Figure 5: AP and Recall distributions across concept levels on VIGOR. The large recall gap, especially for scene concepts, shows that model rationales often cover only partial supporting evidence.
Figure 6: Layer-wise overall AP on VIGOR by normalized layer depth, averaged across all models.
Figure 7: Relationship between rationale correctness and hallucination benchmark performance across 16 MLLMs. The positive correlation indicates that models with better visual rationale grounding also tend to achieve stronger hallucination robustness. Hallucination performances are sourced from Wang et al. (2025) .
Model
AP
Pointing
Qwen2.5-VL-3B
27.5
34.6
+VLM-R1
27.7
35.0
Qwen2.5-VL-7B
28.4
35.6
+Vision-R1
28.6
35.8
Qwen2.5-VL-7B
28.4
35.6
+ViGoRL
28.6
35.9
Table 3: Effect of grounding-oriented fine-tuning on VIGOR. Existing visual reasoning and grounding methods yield only marginal improvements in AP and Pointing Accuracy.
Figure 8: Qualitative examples of rationale failures on VIGOR. Models can attend to incorrect regions, cover only part of the ground-truth evidence, or produce diffuse attention maps, showing that generated concepts are often not fully grounded in the appropriate visual rationale.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Prompt for filtering candidate concepts (part 1)
Figure 10: Prompt for filtering candidate concepts (part 2)
Figure 11: Prompt for filtering candidate concepts (part 3)
Figure 12: Prompt for cleaning candidate concepts (part 1)
Figure 13: Prompt for cleaning candidate concepts (part 2)
Figure 14: Prompt for Identifying Missing Concepts from Images
Overall
Primitive
Intermediate
Object
Scene
Qwen / InternVL
0.321
0.311
0.303
0.330
0.301
Qwen / GLM
0.445
0.467
0.488
0.424
0.481
InternVL / GLM
0.274
0.241
0.262
0.287
0.280
Appendix
Table 4: Pairwise verifier agreement by concept level.
Agreement
Overall
0.183
Primitive
0.175
Intermediate
0.180
Object
0.184
Scene
0.201
Appendix
Table 5: Three-way agreement among MLLM verifiers by concept level. Agreement denotes the proportion of annotations for which all three verifiers selected the same decision.
Figure 15: Prompt for mask proposal verification
Figure 16: Example of composite panel of mask proposals.
Agreement
Overall
0.778
Primitive
0.685
Intermediate
0.868
Object
0.801
Scene
0.735
Appendix
Table 7: Inter-reviewer agreement by concept level. We treat “Strongly Agree” and “Agree” as positive, “Strongly Disagree”, “Disagree”, “Neutral” as negative, and calculate the agreement rate across all three human reviewers.
Figure 17: Prompt for constructing captions from annotated concepts
Evaluator
Faithfulness
Fluency
Evaluator A
4.86 ± 0.04
4.79 ± 0.06
Evaluator B
4.91 ± 0.06
4.85 ± 0.04
Evaluator C
4.82 ± 0.07
4.82 ± 0.05
Average
4.86
4.82
Appendix
Table 8: Human validation of the fixed VIGOR captions. Three evaluators independently rated 500 randomly sampled gpt-5.4 captions for faithfulness to the image and annotated concepts, and for fluency, on a five-point Likert scale. We report the mean rating ± standard error.
Model
Output correctness
Generated caption
Fixed caption
CIDEr
ROUGE-L
AP
Pointing
AP
Pointing
Qwen3-VL-2B
0.494
0.229
34.2
42.0
25.2
31.9
Qwen3-VL-4B
0.466
0.216
37.0
44.0
28.3
34.5
Qwen3-VL-8B
0.468
0.217
34.2
40.4
26.3
32.2
InternVL3.5-2B
0.613
0.257
40.6
50.3
28.8
36.5
InternVL3.5-4B
0.551
0.249
41.8
52.1
29.7
37.2
Appendix
Table 9: Evaluation with model-generated captions. Output correctness (CIDEr, ROUGE-L) and rationale correctness (AP, Pointing Accuracy) are computed on each model’s own caption; fixed-caption scores are from Table 2 .
Model
Overall
Primitive
Intermediate
Object
Scene
AP
Pointing
AP
Pointing
mAP
Pointing
mAP
Pointing
mAP
Pointing
Random
5.8
5.7
4.0
3.9
1.9
1.8
5.0
4.9
17.7
17.3
InternVL3-1B
12.8
17.4
10.1
16.0
6.4
9.0
13.9
20.6
23.3
23.7
InternVL3-2B
12.9
18.4
12.1
17.9
5.8
7.5
14.2
21.2
23.2
31.0
InternVL3.5-2B
17.4
20.5
15.5
19.9
8.8
11.7
21.2
25.7
22.5
20.8
Qwen2-VL-2B
16.0
17.3
15.8
18.5
8.7
10.0
18.5
20.2
22.4
21.7
Appendix
Table 10: Benchmark results of GradCAM on VIGOR benchmark. We report AP and Pointing Accuracy for overall grounding and for primitive, intermediate, object, and scene-level concepts. The best performing model in each column is bolded and the second best in underlined .
Model
Overall
Primitive
Intermediate
Object
Scene
AP
Pointing
AP
Pointing
mAP
Pointing
mAP
Pointing
mAP
Pointing
Random
5.8
5.7
4.0
3.9
1.9
1.8
5.0
4.9
17.7
17.3
InternVL3-1B
11.7
13.6
9.2
10.2
5.8
6.8
13.1
16.1
20.2
21.1
InternVL3.5-1B
10.5
12.0
10.4
12.2
4.4
5.5
11.3
13.8
19.9
18.2
InternVL3-2B
13.8
17.6
16.1
23.0
6.7
8.9
15.6
21.3
20.5
20.1
InternVL3.5-2B
13.0
15.1
13.3
17.7
6.7
8.0
14.8
18.0
19.4
18.1
Appendix
Table 11: Benchmark results of LogitLens on VIGOR benchmark. We report AP and Pointing Accuracy for overall grounding and for primitive, intermediate, object, and scene-level concepts. The best performing model in each column is bolded and the second best in underlined .
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1