VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

Tina Khezresmaeilzadeh; Jike Zhong; Konstantinos Psounis

cs.CVFeb 5, 2026

VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

Authors: Tina Khezresmaeilzadeh, Jike Zhong, Konstantinos Psounis

Organizations: University of Southern California, Los Angeles, USA

Abstract

Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3--70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9--12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors. VRIQ and DiagVRIQ provide a framework for studying visual reasoning through complementary end-to-end, description-based, and intervention-based analyses rather than treating it as a single monolithic capability.

Figures & tables

Figure 1: VRIQ overview. Five reasoning categories are instantiated in abstract and natural-image domains following the same category definitions. The domains are matched at the category level rather than item by item.

Benchmark	Abstract items	Natural-image domain	How perception is assessed	Perceptual information supplied	Interventions
PGM / RAVEN	✓	–	Not assessed	None	–
MM-IQ	✓	–	Not assessed	None	–
PuzzleVQA	✓	–	Staged diagnostic hints	Explanations including the rule	Staged hints
MARVEL	✓	–	Separate perception questions	None	–
VRIQ (ours)	✓	✓ category-matched	Model-generated descriptions	Verified rule-neutral descriptions	GT-description augment and tool-based inference

Table 1: Comparison with selected closely related abstract visual reasoning benchmarks. VRIQ’s natural-image domain is matched to the abstract domain by category, not item by item.

Domain	3D	FR	MP	OO	SC	Total
Abstract	210	198	141	106	133	788
Natural	111	97	146	149	100	603
Total	321	295	287	255	233	1,391
DiagVRIQ (natural subset)	–	–	98	51	60	209

Table 2: VRIQ composition by category and domain. DiagVRIQ items are a subset of the natural domain.

Model	3D	FR	MP	OO	SC	Avg
Open-weight
Qwen2.5-VL-3B-Instruct	28.1	23.2	17.0	25.5	20.3	22.8
Qwen2.5-VL-3B-Instruct-AWQ	30.0	17.2	19.1	19.8	20.3	21.3
Qwen2.5-VL-7B-Instruct	32.9	25.3	31.2	26.4	40.6	31.3
InternVL3-9B	31.9	20.2	31.2	16.0	24.1	24.7
LLaVA-v1.6-Mistral-7B	28.6	20.2	19.9	27.4	15.8	22.4

Table 3: Abstract-domain accuracy (%) on VRIQ. 3D = 3D Visualization, FR = Figure Rotation, MP = Matrix Prediction, OO = Odd One Out, and SC = Sequence Completion. The 2026 frontier group contains the most recent high-capability model evaluated from each of OpenAI, Google, and Anthropic at the time of our experiments; the remaining 17 open-weight and proprietary models are non-frontier models. The frontier mean uses GPT-5.6 Sol (high). Avg is the unweighted mean over the five categories. Bold marks the best model result in each column. Human accuracy is measured on a stratified 150-item abstract subset.

Model	3D	FR	MP	OO	SC	Avg
Open-weight
Qwen2.5-VL-3B-Instruct	29.7	40.2	43.2	40.9	34.0	37.6
Qwen2.5-VL-3B-Instruct-AWQ	25.2	35.1	28.8	36.2	37.0	32.5
Qwen2.5-VL-7B-Instruct	29.7	35.1	39.7	49.7	25.0	35.8
InternVL3-9B	39.6	45.4	34.9	40.3	40.0	40.0
LLaVA-v1.6-Mistral-7B	27.0	17.5	24.7	30.2	20.0	23.9

Table 4: Natural-domain accuracy (%) on VRIQ. Category abbreviations and model groups follow Table 3 . The frontier mean uses GPT-5.6 Sol (high). Avg is the unweighted mean over the five categories. Bold marks the best model result in each column. Human accuracy is measured on a stratified 150-item natural subset.

Model	End-to-end	Description success	Acc. $\mid P^{\checkmark}$
LLaVA-v1.6-Mistral-7B	26.8 [21.2, 33.2]	12.0 [8.2, 17.1]	52.0 [33.5, 70.0]
Qwen2.5-VL-7B	51.7 [44.9, 58.4]	62.2 [55.5, 68.5]	69.2 [60.8, 76.5]
GPT-4o	84.7 [79.2, 88.9]	92.3 [87.9, 95.2]	90.7 [85.7, 94.0]
Claude Opus 5	90.9 [86.2, 94.1]	85.6 [80.2, 89.8]	91.1 [86.0, 94.4]
GPT-5.6 Sol	88.0 [82.9, 91.8]	84.7 [79.2, 88.9]	88.1 [82.5, 92.1]

Table 5: Pooled DiagVRIQ metrics over all 209 items, with 95% Wilson intervals. Description success is the fraction of generated descriptions satisfying the perception-success criterion. Acc.

\mid P^{\checkmark}

denotes answer accuracy conditional on the generated description satisfying that criterion.

	Matrix Prediction			Odd One Out			Sequence Completion
Model	Base	GT	$\Delta$	Base	GT	$\Delta$	Base	GT	$\Delta$
LLaVA-v1.6-Mistral-7B	21.4	36.7	+15.3	39.2	49.0	+9.8	25.0	66.7	+41.7
Qwen2.5-VL-7B	59.2	69.4	+10.2	52.9	56.9	+4.0	38.3	66.7	+28.4
GPT-4o	87.8	89.8	+2.0	84.3	90.2	+5.9	80.0	83.3	+3.3

Table 6: DiagVRIQ accuracy (%) without (Base) and with (GT) ground-truth description augmentation, and the change

\Delta

(points). Every model–category cell improves, although the GPT-4o gains are small.

	Natural domain			Abstract domain
Category	With	Without	$\Delta$	With	Without	$\Delta$
3D Visualization	96.4	48.6	+47.8	35.2	23.8	+11.4
Figure Rotation	91.8	40.2	+51.6	68.2	41.9	+26.3
Matrix Prediction	75.3	62.3	+13.0	46.8	35.5	+11.3
Odd One Out	69.8	59.1	+10.7	52.8	31.1	+21.7
Sequence Completion	80.0	69.0	+11.0	48.9	36.8	+12.1

Table 7: Accuracy (%) of o3 with and without tool access on all VRIQ items. Natural-domain category sizes are 111 (3D), 97 (FR), 146 (MP), 149 (OO), and 100 (SC); abstract-domain category sizes are 210, 198, 141, 106, and 133, respectively. Overall is the unweighted mean over the five categories.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: 3D abstract examples built by Tinkercad.

Model	Params	Type	Citation
Small open-weight (3B–9B)
Qwen2.5-VL-3B-Instruct	3B	Instruction-tuned	( Hugging Face, 2025f )
Qwen2.5-VL-3B-Instruct-AWQ	3B	Quantized	( Hugging Face, 2025e )
Qwen2.5-VL-7B-Instruct	7B	Instruction-tuned	( Hugging Face, 2025g )
InternVL3-9B	9B	Instruction-tuned	( Hugging Face, 2025d )
LLaVA-v1.6-Mistral-7B	7B	Instruction-tuned	( Hugging Face, 2025b )

Appendix

Table 8: Vision–language models evaluated on VRIQ. Open-weight models accessed via Hugging Face; proprietary models accessed via official APIs. “Thinking” indicates reasoning-optimized variants; “Tools” indicates models with image-manipulation tool access during inference.

Model	$N$	Agreement (%)
LLaVA-1.6-Mistral-7B	209	97.1
Qwen2.5-VL-7B	209	89.0
GPT-4o	209	92.3
Claude Opus 5	209	93.8
GPT-5.6 Sol	209	91.9
Combined	1,045	92.8

Appendix

Table 9: Human–judge agreement on the perception-success criterion.

Figure 4: Representative LLaVA-1.6 examples in which the baseline response contains an incorrect visual description and an incorrect answer, while supplying the verified ground-truth description changes the final answer to the correct option.

Figure 5: Representative residual reasoning failure from Qwen2.5-VL-7B. The model correctly identifies the visual entities and panel structure but applies an incorrect relational rule, producing the wrong answer even after GT augmentation.

Figure 6: Additional residual reasoning failure under GT augmentation. The model correctly identifies all entities but constructs an invalid grouping strategy that cannot uniquely determine the odd-one-out item.

Model	Cat.	End-to-end	Description success	Acc. $\mid P^{\checkmark}$
LLaVA-v1.6-Mistral-7B	MP (98)	21/98 (21.4)	6/98 (6.1)	2/6 (33.3)
	OO (51)	20/51 (39.2)	17/51 (33.3)	10/17 (58.8)
	SC (60)	15/60 (25.0)	2/60 (3.3)	1/2 (50.0)
Qwen2.5-VL-7B	MP (98)	58/98 (59.2)	69/98 (70.4)	51/69 (73.9)
	OO (51)	27/51 (52.9)	43/51 (84.3)	26/43 (60.5)
	SC (60)	23/60 (38.3)	18/60 (30.0)	13/18 (72.2)

Appendix

Table 10: Per-category DiagVRIQ results (counts and %). Description success is the fraction of generated descriptions that satisfy the perception-success criterion. Acc.

\mid P^{\checkmark}

denotes answer accuracy conditional on the generated description satisfying that criterion. All quantities are computed from the same describe-then-solve generation.

Model	3D	FR	MP	OO	SC
Open-weight
Qwen2.5-VL-3B-Instruct	28.1 [22.5, 34.5]	23.2 [17.9, 29.6]	17.0 [11.7, 24.1]	25.5 [18.1, 34.5]	20.3 [14.3, 27.9]
Qwen2.5-VL-3B-Instruct-AWQ	30.0 [24.2, 36.5]	17.2 [12.6, 23.0]	19.1 [13.5, 26.4]	19.8 [13.3, 28.4]	20.3 [14.3, 27.9]
Qwen2.5-VL-7B-Instruct	32.9 [26.9, 39.5]	25.3 [19.7, 31.7]	31.2 [24.1, 39.3]	26.4 [19.0, 35.5]	40.6 [32.6, 49.1]
InternVL3-9B	31.9 [26.0, 38.5]	20.2 [15.2, 26.3]	31.2 [24.1, 39.3]	16.0 [10.3, 24.2]	24.1 [17.6, 32.0]
LLaVA-v1.6-Mistral-7B	28.6 [22.9, 35.0]	20.2 [15.2, 26.3]	19.9 [14.1, 27.2]	27.4 [19.8, 36.5]	15.8 [10.6, 22.9]

Appendix

Table 11: Abstract-domain accuracy (%) on VRIQ with 95% Wilson confidence intervals. Category sizes are 210, 198, 141, 106, 133 for 3D, FR, MP, OO, and SC, respectively. Each item is evaluated once. Point estimates match Table 3 .

Model	3D	FR	MP	OO	SC
Open-weight
Qwen2.5-VL-3B-Instruct	29.7 [22.0, 38.8]	40.2 [31.0, 50.2]	43.2 [35.4, 51.3]	40.9 [33.4, 49.0]	34.0 [25.5, 43.7]
Qwen2.5-VL-3B-Instruct-AWQ	25.2 [18.1, 34.0]	35.1 [26.3, 45.0]	28.8 [22.0, 36.6]	36.2 [29.0, 44.2]	37.0 [28.2, 46.8]
Qwen2.5-VL-7B-Instruct	29.7 [22.0, 38.8]	35.1 [26.3, 45.0]	39.7 [32.2, 47.8]	49.7 [41.7, 57.6]	25.0 [17.5, 34.3]
InternVL3-9B	39.6 [31.0, 48.9]	45.4 [35.8, 55.3]	34.9 [27.7, 43.0]	40.3 [32.7, 48.3]	40.0 [30.9, 49.8]
LLaVA-v1.6-Mistral-7B	27.0 [19.6, 36.0]	17.5 [11.2, 26.3]	24.7 [18.4, 32.2]	30.2 [23.4, 38.0]	20.0 [13.3, 28.9]

Appendix

Table 12: Natural-domain accuracy (%) on VRIQ with 95% Wilson confidence intervals. Category sizes are 111, 97, 146, 149, 100 for 3D, FR, MP, OO, and SC, respectively. Each item is evaluated once. Point estimates match Table 4 .

Panel of Figure 7 — Figure 7: Prompt used for the GT-augmented condition in Section 4.3 .

Figure 8: Input sequence with the missing final image and multiple-choice options (A–D).

Figure 9: Reference image after 270 ∘ rotation produced by the tool, used for comparison with the answer options.

Figure 10: Natural-domain sequence completion task with missing final image and answer options (A–D).

Figure 11: Cropped candidate options used by o3 for visual inspection and counting.

Figure 12: Abstract matrix reasoning task with one missing cell (bottom-right).

Figure 13: Tool execution trace showing repeated circle-detection attempts. Despite systematic cropping, accurate enumeration and spatial localization remain unreliable under clutter.

Figure 14: Sample VRIQ abstract Matrix Prediction items. Each puzzle requires identifying the missing panel in a $3\times 3$ matrix based on a transformation rule applied across rows, columns, or both.

Figure 15: Sample VRIQ abstract Sequence Completion items. Each puzzle requires identifying the next element in a sequence governed by a positional, structural, or arithmetic rule.

Figure 16: Sample VRIQ abstract Odd One Out items. Each puzzle requires identifying the candidate that does not share a common visual property with the others.

Figure 17: Sample VRIQ abstract Figure Rotation items. Each puzzle requires identifying a rotated version of a geometric figure.

Figure 18: Sample VRIQ natural Sequence Completion questions, with the correct option marked by ✓.

Figure 19: Sample VRIQ natural Sequence Completion items. Each puzzle requires identifying the next element in a sequence of real-world entities or scenes that follow an underlying rule.

Figure 20: Sample VRIQ natural Odd One Out items. Each puzzle requires identifying the entity that does not belong to a shared category.

Figure 21: Sample VRIQ natural 3D Visualization items. Each puzzle requires identifying an alternate viewpoint of a real-world object.

Figure 22: Sample VRIQ natural Figure Rotation items. Each puzzle requires identifying a rotated version of a real-world scene.

Figure 23: Three representative DiagVRIQ Matrix Prediction items paired with their human-verified ground-truth descriptions.

Visual Reasoning

Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.

See all 159 papers

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visual Reasoning VLM Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities
Jul 30, 2026Liangjie Zhao, Jiaqing Lyu, Kexin Tang +5VLM Evaluation Visual Reasoning
Adelaide University · Tsinghua University · Amap, Alibaba Group +3
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

May 25, 2026J/K move · Enter open · S save

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Longteng Guo, Yifan Wang, Pengkang Huo, Tailai Chen+3

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visual Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding Reasoning for VLMs

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Abstract

Visual Reasoning

Explore similar work

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models