Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.
Figures & tables
Figure 1: The single-step analogy (left) delivers one transformation ( color ) from input to target, while the compositional analogy (right) delivers the union of color (wood to red) and number (one to two) transformations.
Task
#Input
#Atomic
Count
Single
1
2
500
Compos.
Shared
2
2,3,4
3,500
Diff.
2
2
1,500
Total
5,500
Table 1: Distribution of the dataset. #Input: number of input image pairs. #Atomic: number of atomic transformations in each pair.
Figure 2: Overview of Diagnosis Pipeline. Given two image pairs, the model is instructed to describe the transformations, decompose the transformations into atomic transformations, perform the set operation, and apply the resulting transformation on the query image. Then we apply an evaluator model to check the correctness of each step.
Model
Single
Union
Intersection
Difference
SS
DS
SS
DS
SS
DS
Closed-source models:
Gemini-2.5 Flash
75.0
49.4
37.4
60.2
64.2
41.4
33.2
Gemini-2.5 Pro
79.2
51.6
40.4
62.8
67.2
59.4
55.6
GPT-5.1
81.2
64.6
51.0
75.0
73.0
59.0
62.4
GPT-4o
44.8
27.2
9.0
34.0
36.0
25.6
21.2
Table 2: Performance Comparison on Compositional Analogical Reasoning Tasks. SS stands for same source setting and DS stands for different source setting. The reasoning models are highlighted with green.
Figure 3: Detailed analysis of failure distributions. (a) Across different models, the major bottleneck for closed-source models is decomposition, while for open-source models, it is perception. (b) As we scale the number of atomic transformations ( N ) in image pairs, the portion of decomposition failure significantly increases.
Figure 4: Failure Contribution by Property Combinations. For most models, combinations among subject, number, and position contribute most to the failure.
Figure 5: Error Distribution by Property Combination of Gemini-2.5 Flash. We study the most challenging combinations, the failure mainly concentrates in decomposition.
Figure 6: Accuracy by Number of Atomic Transformations. The performance drops as the number of atomic transformations ( N ) increases.
Model
∣T∣=3
∣T∣=4
GPT-4o
37.0%
46.0%
GPT-5.1
85.0%
88.0%
Gemini-2.5 Flash
85.0%
74.0%
Gemini-2.5 Pro
82.0%
91.0%
Table 3: Experiment on output cardinality. By increasing the output cardinality, there is no clear trend about models’ performance.
Model
With Format
Without Format
Oracle
GPT-4o
42.0%
51.0%
86.0%
GPT-5.1
60.0%
78.0%
94.0%
Gemini-2.5 Flash
77.0%
90.0%
99.0%
Gemini-2.5 Pro
86.0%
86.0%
97.0%
Table 4: Task performance in ablation study. The results strengthen our claim that models struggle at the decomposition step and that the task format only slightly affects performance.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Task
I-S
I-D
U-S
U-D
Accuracy
37%
38%
20%
8%
Appendix
Table 5: Performance of GPT-4o with neuro-symbolic prompting
Table 6: Template for Guided Prompting
Table 7: Template for Diagnosis Prompting
Model
Run 1
Run 2
Run 3
Run 4
Mean ± SD
Gemini-2.5 Pro
41.8
40.2
42.2
40.4
41.15 ± 0.86
GPT-5.1
50.2
49.8
50.4
51.6
50.50 ± 0.67
Appendix
Table 8: Accuracy (%) across 4 independent runs on a 200-question subset from the Union, Different Source setting. The low standard deviation confirms the stability of our main results.
Analogical reasoning studies how systems transfer relations and solutions between situations with shared structure, enabling inference beyond direct examples.
Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains less understood. We introduce "Mind's Eye", a multiple-choice benchmark of eight visuo-cognitive tasks inspired by classic human intelligence tests and organized under a novel "A-R-T" taxonomy: Abstraction, Relation, and Transformation. The tasks probe core processes of fluid intelligence such as pattern induction, analogical relation mapping, and mental transformation. We evaluate a diverse suite of closed-source and open-source MLLMs and compare their performance with human participants. Humans achieve 80% accuracy, while top performing MLLMs remain below 50%. Error analysis reveals failures in: (i) visual attention allocation, (ii) internal perceptual manipulation, and (iii) weak abstraction of underlying visual concepts. Our findings suggest that current MLLMs exhibit limited visuospatial reasoning capabilities, when compared with human participants, highlighting the need for more cognitively grounded evaluation frameworks.
Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India
Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.
Longteng Guo, Yifan Wang, Pengkang Huo +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Current large language models (LLMs), even those explicitly trained for reasoning, often struggle with ambiguous content moderation cases due to misleading "decision shortcuts" embedded in context. Inspired by cognitive psychology insights into expert moderation, we introduce \caro (Chain-of-Analogy Reasoning Optimization), a novel two-stage training framework to induce robust analogical reasoning in LLMs. First, \caro bootstraps analogical reasoning chains via retrieval-augmented generation (RAG) on moderation data and performs supervised fine-tuning (SFT). Second, we propose a customized direct preference optimization (DPO) approach to reinforce analogical reasoning behaviors explicitly. Unlike static retrieval methods, \caro dynamically generates tailored analogical references during inference, effectively mitigating harmful decision shortcuts. Extensive experiments demonstrate that \caro substantially outperforms state-of-the-art reasoning models (DeepSeek R1, QwQ), specialized moderation models (LLaMA Guard), and advanced fine-tuning and retrieval-augmented methods, achieving an average F1 score improvement of 24.9% on challenging ambiguous moderation benchmarks.
Bingzhe Wu, Haotian Lu, Yuchen Mou
Shenzhen University · Tsinghua University · National University of Singapore