CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Yongkang Du; Xiaohan Zou; Minhao Cheng; Lu Lin

cs.AIMar 30, 2026

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Authors: Yongkang Du, Xiaohan Zou, Minhao Cheng, Lu Lin

Organizations: Pennsylvania State University

Abstract

Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.

Figures & tables

Figure 1: The single-step analogy (left) delivers one transformation ( color ) from input to target, while the compositional analogy (right) delivers the union of color (wood to red) and number (one to two) transformations.

Task	#Input	#Atomic	Count
Single	1	2	500
Compos.
Shared	2	2,3,4	3,500
Diff.	2	2	1,500
Total			5,500

Table 1: Distribution of the dataset. #Input: number of input image pairs. #Atomic: number of atomic transformations in each pair.

Figure 2: Overview of Diagnosis Pipeline. Given two image pairs, the model is instructed to describe the transformations, decompose the transformations into atomic transformations, perform the set operation, and apply the resulting transformation on the query image. Then we apply an evaluator model to check the correctness of each step.

Model	Single	Union		Intersection		Difference
Model	Single	SS	DS	SS	DS	SS	DS
Closed-source models:
Gemini-2.5 Flash	75.0	49.4	37.4	60.2	64.2	41.4	33.2
Gemini-2.5 Pro	79.2	51.6	40.4	62.8	67.2	59.4	55.6
GPT-5.1	81.2	64.6	51.0	75.0	73.0	59.0	62.4
GPT-4o	44.8	27.2	9.0	34.0	36.0	25.6	21.2

Table 2: Performance Comparison on Compositional Analogical Reasoning Tasks. SS stands for same source setting and DS stands for different source setting. The reasoning models are highlighted with green.

Panel of Figure 3 — Figure 3: Detailed analysis of failure distributions. (a) Across different models, the major bottleneck for closed-source models is decomposition, while for open-source models, it is perception. (b) As we scale the number of atomic transformations ( $N$ ) in image pairs, the portion of decomposition failure significantly increases.

Figure 4: Failure Contribution by Property Combinations. For most models, combinations among subject, number, and position contribute most to the failure.

Figure 5: Error Distribution by Property Combination of Gemini-2.5 Flash. We study the most challenging combinations, the failure mainly concentrates in decomposition.

Figure 6: Accuracy by Number of Atomic Transformations. The performance drops as the number of atomic transformations (

N

) increases.

Model	$\|T\|{=3}$	$\|T\|{=4}$
GPT-4o	37.0%	46.0%
GPT-5.1	85.0%	88.0%
Gemini-2.5 Flash	85.0%	74.0%
Gemini-2.5 Pro	82.0%	91.0%

Table 3: Experiment on output cardinality. By increasing the output cardinality, there is no clear trend about models’ performance.

Model	With Format	Without Format	Oracle
GPT-4o	42.0%	51.0%	86.0%
GPT-5.1	60.0%	78.0%	94.0%
Gemini-2.5 Flash	77.0%	90.0%	99.0%
Gemini-2.5 Pro	86.0%	86.0%	97.0%

Table 4: Task performance in ablation study. The results strengthen our claim that models struggle at the decomposition step and that the task format only slightly affects performance.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Task	I-S	I-D	U-S	U-D
Accuracy	37%	38%	20%	8%

Appendix

Table 5: Performance of GPT-4o with neuro-symbolic prompting

Panel of Table 6 — Table 6: Template for Guided Prompting

Panel of Table 7 — Table 7: Template for Diagnosis Prompting

Model	Run 1	Run 2	Run 3	Run 4	Mean $\pm$ SD
Gemini-2.5 Pro	41.8	40.2	42.2	40.4	41.15 $\pm$ 0.86
GPT-5.1	50.2	49.8	50.4	51.6	50.50 $\pm$ 0.67

Appendix

Table 8: Accuracy (%) across 4 independent runs on a 200-question subset from the Union, Different Source setting. The low standard deviation confirms the stability of our main results.

Analogical Reasoning

Analogical reasoning studies how systems transfer relations and solutions between situations with shared structure, enabling inference beyond direct examples.

See all 28 papers

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visual Reasoning VLM Reasoning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation
Apr 12, 2026Bingzhe Wu, Haotian Lu, Yuchen MouAnalogical Reasoning for Language Models Content Moderation
Shenzhen University · Tsinghua University · National University of Singapore

Apr 17, 2026J/K move · Enter open · S save

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti, Vineeth N Balasubramanian+1

CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India

Spatial Reasoning Benchmarks Visual Spatial Reasoning Visual Reasoning Multimodal Large Language Models Visually Grounded Reasoning Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains less understood. We introduce "Mind's Eye", a multiple-choice benchmark of eight visuo-cognitive tasks inspired by classic human intelligence tests and organized under a novel "A-R-T" taxonomy: Abstraction, Relation, and Transformation. The tasks probe core processes of fluid intelligence such as pattern induction, analogical relation mapping, and mental transformation. We evaluate a diverse suite of closed-source and open-source MLLMs and compare their performance with human participants. Humans achieve 80% accuracy, while top performing MLLMs remain below 50%. Error analysis reveals failures in: (i) visual attention allocation, (ii) internal perceptual manipulation, and (iii) weak abstraction of underlying visual concepts. Our findings suggest that current MLLMs exhibit limited visuospatial reasoning capabilities, when compared with human participants, highlighting the need for more cognitively grounded evaluation frameworks.

Abstract

Analogical Reasoning

Explore similar work

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

CARO: Chain-of-Analogy Reasoning Optimization for Robust Content Moderation