Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3--70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9--12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.
Figures & tables
Figure 1: VRIQ overview. Five reasoning categories are instantiated in abstract and natural-image domains following the same category definitions. The domains are matched at the category level rather than item by item.
Benchmark
Abstract items
Natural-image domain
How perception is assessed
Perceptual information supplied
Interventions
PGM / RAVEN
✓
–
Not assessed
None
–
MM-IQ
✓
–
Not assessed
None
–
PuzzleVQA
✓
–
Staged diagnostic hints
Explanations including the rule
Staged hints
MARVEL
✓
–
Separate perception questions
None
–
VRIQ (ours)
✓
✓ category-matched
Model-generated descriptions
Verified rule-neutral descriptions
GT-description augment and tool-based inference
Table 1: Comparison with selected closely related abstract visual reasoning benchmarks. VRIQ’s natural-image domain is matched to the abstract domain by category, not item by item.
Domain
3D
FR
MP
OO
SC
Total
Abstract
210
198
141
106
133
788
Natural
111
97
146
149
100
603
Total
321
295
287
255
233
1,391
DiagVRIQ (natural subset)
–
–
98
51
60
209
Table 2: VRIQ composition by category and domain. DiagVRIQ items are a subset of the natural domain.
Model
3D
FR
MP
OO
SC
Avg
Open-weight
Qwen2.5-VL-3B-Instruct
28.1
23.2
17.0
25.5
20.3
22.8
Qwen2.5-VL-3B-Instruct-AWQ
30.0
17.2
19.1
19.8
20.3
21.3
Qwen2.5-VL-7B-Instruct
32.9
25.3
31.2
26.4
40.6
31.3
InternVL3-9B
31.9
20.2
31.2
16.0
24.1
24.7
LLaVA-v1.6-Mistral-7B
28.6
20.2
19.9
27.4
15.8
22.4
Table 3: Abstract-domain accuracy (%) on VRIQ. 3D = 3D Visualization, FR = Figure Rotation, MP = Matrix Prediction, OO = Odd One Out, and SC = Sequence Completion. The 2026 frontier group contains the most recent high-capability model evaluated from each of OpenAI, Google, and Anthropic at the time of our experiments; the remaining 17 open-weight and proprietary models are non-frontier models. The frontier mean uses GPT-5.6 Sol (high). Avg is the unweighted mean over the five categories. Bold marks the best model result in each column. Human accuracy is measured on a stratified 150-item abstract subset.
Model
3D
FR
MP
OO
SC
Avg
Open-weight
Qwen2.5-VL-3B-Instruct
29.7
40.2
43.2
40.9
34.0
37.6
Qwen2.5-VL-3B-Instruct-AWQ
25.2
35.1
28.8
36.2
37.0
32.5
Qwen2.5-VL-7B-Instruct
29.7
35.1
39.7
49.7
25.0
35.8
InternVL3-9B
39.6
45.4
34.9
40.3
40.0
40.0
LLaVA-v1.6-Mistral-7B
27.0
17.5
24.7
30.2
20.0
23.9
Table 4: Natural-domain accuracy (%) on VRIQ. Category abbreviations and model groups follow Table 3 . The frontier mean uses GPT-5.6 Sol (high). Avg is the unweighted mean over the five categories. Bold marks the best model result in each column. Human accuracy is measured on a stratified 150-item natural subset.
Model
End-to-end
Description success
Acc. ∣P✓
LLaVA-v1.6-Mistral-7B
26.8 [21.2, 33.2]
12.0 [8.2, 17.1]
52.0 [33.5, 70.0]
Qwen2.5-VL-7B
51.7 [44.9, 58.4]
62.2 [55.5, 68.5]
69.2 [60.8, 76.5]
GPT-4o
84.7 [79.2, 88.9]
92.3 [87.9, 95.2]
90.7 [85.7, 94.0]
Claude Opus 5
90.9 [86.2, 94.1]
85.6 [80.2, 89.8]
91.1 [86.0, 94.4]
GPT-5.6 Sol
88.0 [82.9, 91.8]
84.7 [79.2, 88.9]
88.1 [82.5, 92.1]
Table 5: Pooled DiagVRIQ metrics over all 209 items, with 95% Wilson intervals. Description success is the fraction of generated descriptions satisfying the perception-success criterion. Acc. ∣P✓ denotes answer accuracy conditional on the generated description satisfying that criterion.
Matrix Prediction
Odd One Out
Sequence Completion
Model
Base
GT
Δ
Base
GT
Δ
Base
GT
Δ
LLaVA-v1.6-Mistral-7B
21.4
36.7
+15.3
39.2
49.0
+9.8
25.0
66.7
+41.7
Qwen2.5-VL-7B
59.2
69.4
+10.2
52.9
56.9
+4.0
38.3
66.7
+28.4
GPT-4o
87.8
89.8
+2.0
84.3
90.2
+5.9
80.0
83.3
+3.3
Table 6: DiagVRIQ accuracy (%) without (Base) and with (GT) ground-truth description augmentation, and the change Δ (points). Every model–category cell improves, although the GPT-4o gains are small.
Natural domain
Abstract domain
Category
With
Without
Δ
With
Without
Δ
3D Visualization
96.4
48.6
+47.8
35.2
23.8
+11.4
Figure Rotation
91.8
40.2
+51.6
68.2
41.9
+26.3
Matrix Prediction
75.3
62.3
+13.0
46.8
35.5
+11.3
Odd One Out
69.8
59.1
+10.7
52.8
31.1
+21.7
Sequence Completion
80.0
69.0
+11.0
48.9
36.8
+12.1
Table 7: Accuracy (%) of o3 with and without tool access on all VRIQ items. Natural-domain category sizes are 111 (3D), 97 (FR), 146 (MP), 149 (OO), and 100 (SC); abstract-domain category sizes are 210, 198, 141, 106, and 133, respectively. Overall is the unweighted mean over the five categories.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: 3D abstract examples built by Tinkercad.
Model
Params
Type
Citation
Small open-weight (3B–9B)
Qwen2.5-VL-3B-Instruct
3B
Instruction-tuned
( Hugging Face, 2025f )
Qwen2.5-VL-3B-Instruct-AWQ
3B
Quantized
( Hugging Face, 2025e )
Qwen2.5-VL-7B-Instruct
7B
Instruction-tuned
( Hugging Face, 2025g )
InternVL3-9B
9B
Instruction-tuned
( Hugging Face, 2025d )
LLaVA-v1.6-Mistral-7B
7B
Instruction-tuned
( Hugging Face, 2025b )
Appendix
Table 8: Vision–language models evaluated on VRIQ. Open-weight models accessed via Hugging Face; proprietary models accessed via official APIs. “Thinking” indicates reasoning-optimized variants; “Tools” indicates models with image-manipulation tool access during inference.
Model
N
Agreement (%)
LLaVA-1.6-Mistral-7B
209
97.1
Qwen2.5-VL-7B
209
89.0
GPT-4o
209
92.3
Claude Opus 5
209
93.8
GPT-5.6 Sol
209
91.9
Combined
1,045
92.8
Appendix
Table 9: Human–judge agreement on the perception-success criterion.
Figure 4: Representative LLaVA-1.6 examples in which the baseline response contains an incorrect visual description and an incorrect answer, while supplying the verified ground-truth description changes the final answer to the correct option.
Figure 5: Representative residual reasoning failure from Qwen2.5-VL-7B. The model correctly identifies the visual entities and panel structure but applies an incorrect relational rule, producing the wrong answer even after GT augmentation.
Figure 6: Additional residual reasoning failure under GT augmentation. The model correctly identifies all entities but constructs an invalid grouping strategy that cannot uniquely determine the odd-one-out item.
Model
Cat.
End-to-end
Description success
Acc. ∣P✓
LLaVA-v1.6-Mistral-7B
MP (98)
21/98 (21.4)
6/98 (6.1)
2/6 (33.3)
OO (51)
20/51 (39.2)
17/51 (33.3)
10/17 (58.8)
SC (60)
15/60 (25.0)
2/60 (3.3)
1/2 (50.0)
Qwen2.5-VL-7B
MP (98)
58/98 (59.2)
69/98 (70.4)
51/69 (73.9)
OO (51)
27/51 (52.9)
43/51 (84.3)
26/43 (60.5)
SC (60)
23/60 (38.3)
18/60 (30.0)
13/18 (72.2)
Appendix
Table 10: Per-category DiagVRIQ results (counts and %). Description success is the fraction of generated descriptions that satisfy the perception-success criterion. Acc. ∣P✓ denotes answer accuracy conditional on the generated description satisfying that criterion. All quantities are computed from the same describe-then-solve generation.
Model
3D
FR
MP
OO
SC
Open-weight
Qwen2.5-VL-3B-Instruct
28.1 [22.5, 34.5]
23.2 [17.9, 29.6]
17.0 [11.7, 24.1]
25.5 [18.1, 34.5]
20.3 [14.3, 27.9]
Qwen2.5-VL-3B-Instruct-AWQ
30.0 [24.2, 36.5]
17.2 [12.6, 23.0]
19.1 [13.5, 26.4]
19.8 [13.3, 28.4]
20.3 [14.3, 27.9]
Qwen2.5-VL-7B-Instruct
32.9 [26.9, 39.5]
25.3 [19.7, 31.7]
31.2 [24.1, 39.3]
26.4 [19.0, 35.5]
40.6 [32.6, 49.1]
InternVL3-9B
31.9 [26.0, 38.5]
20.2 [15.2, 26.3]
31.2 [24.1, 39.3]
16.0 [10.3, 24.2]
24.1 [17.6, 32.0]
LLaVA-v1.6-Mistral-7B
28.6 [22.9, 35.0]
20.2 [15.2, 26.3]
19.9 [14.1, 27.2]
27.4 [19.8, 36.5]
15.8 [10.6, 22.9]
Appendix
Table 11: Abstract-domain accuracy (%) on VRIQ with 95% Wilson confidence intervals. Category sizes are 210, 198, 141, 106, 133 for 3D, FR, MP, OO, and SC, respectively. Each item is evaluated once. Point estimates match Table 3 .
Model
3D
FR
MP
OO
SC
Open-weight
Qwen2.5-VL-3B-Instruct
29.7 [22.0, 38.8]
40.2 [31.0, 50.2]
43.2 [35.4, 51.3]
40.9 [33.4, 49.0]
34.0 [25.5, 43.7]
Qwen2.5-VL-3B-Instruct-AWQ
25.2 [18.1, 34.0]
35.1 [26.3, 45.0]
28.8 [22.0, 36.6]
36.2 [29.0, 44.2]
37.0 [28.2, 46.8]
Qwen2.5-VL-7B-Instruct
29.7 [22.0, 38.8]
35.1 [26.3, 45.0]
39.7 [32.2, 47.8]
49.7 [41.7, 57.6]
25.0 [17.5, 34.3]
InternVL3-9B
39.6 [31.0, 48.9]
45.4 [35.8, 55.3]
34.9 [27.7, 43.0]
40.3 [32.7, 48.3]
40.0 [30.9, 49.8]
LLaVA-v1.6-Mistral-7B
27.0 [19.6, 36.0]
17.5 [11.2, 26.3]
24.7 [18.4, 32.2]
30.2 [23.4, 38.0]
20.0 [13.3, 28.9]
Appendix
Table 12: Natural-domain accuracy (%) on VRIQ with 95% Wilson confidence intervals. Category sizes are 111, 97, 146, 149, 100 for 3D, FR, MP, OO, and SC, respectively. Each item is evaluated once. Point estimates match Table 4 .
Figure 7: Prompt used for the GT-augmented condition in Section 4.3 .
Figure 8: Input sequence with the missing final image and multiple-choice options (A–D).
Figure 9: Reference image after 270 ∘ rotation produced by the tool, used for comparison with the answer options.
Figure 10: Natural-domain sequence completion task with missing final image and answer options (A–D).
Figure 11: Cropped candidate options used by o3 for visual inspection and counting.
Figure 12: Abstract matrix reasoning task with one missing cell (bottom-right).
Figure 13: Tool execution trace showing repeated circle-detection attempts. Despite systematic cropping, accurate enumeration and spatial localization remain unreliable under clutter.
Figure 14: Sample VRIQ abstract Matrix Prediction items. Each puzzle requires identifying the missing panel in a 3×3 matrix based on a transformation rule applied across rows, columns, or both.
Figure 15: Sample VRIQ abstract Sequence Completion items. Each puzzle requires identifying the next element in a sequence governed by a positional, structural, or arithmetic rule.
Figure 16: Sample VRIQ abstract Odd One Out items. Each puzzle requires identifying the candidate that does not share a common visual property with the others.
Figure 17: Sample VRIQ abstract Figure Rotation items. Each puzzle requires identifying a rotated version of a geometric figure.
Figure 18: Sample VRIQ natural Sequence Completion questions, with the correct option marked by ✓.
Figure 19: Sample VRIQ natural Sequence Completion items. Each puzzle requires identifying the next element in a sequence of real-world entities or scenes that follow an underlying rule.
Figure 20: Sample VRIQ natural Odd One Out items. Each puzzle requires identifying the entity that does not belong to a shared category.
Figure 21: Sample VRIQ natural 3D Visualization items. Each puzzle requires identifying an alternate viewpoint of a real-world object.
Figure 22: Sample VRIQ natural Figure Rotation items. Each puzzle requires identifying a rotated version of a real-world scene.
Figure 23: Three representative DiagVRIQ Matrix Prediction items paired with their human-verified ground-truth descriptions.
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Large Vision Language Models have integrated reasoning capabilities, elevating cognitive performance to new levels. However, existing evaluations either focus solely on perception or rely on specific domains such as maths or coding. Evaluation for reasoning capabilities that align with an open-world environment is still required, especially one that considers perception and reasoning jointly. To bridge this gap, we propose to evaluate LVLMs by exploiting visual illusions as a diagnostic tool. Visual illusions are phenomena in which the human visual system misinterprets objective signals, resulting in an understanding that deviates from reality. We constructed IllusionReasoning, a benchmark of illusion images collected from the real world, incorporating diverse annotated question-answer pairs. Based on IllusionReasoning, we show that the reasoning capabilities of a wide range of LVLMs are not as advanced as claimed. Our work provides new insights into LVLMs and offers future direction for optimisation.
Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5
Adelaide University · Tsinghua University · Amap, Alibaba Group +3
Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.
Rong Yu Xu, Prayag Tiwari, Shaolei Zhang
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China