Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. Current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks. Does it guarantee their performance on these foundational visual capabilities? To systematically investigate this gap, we introduce VisFacor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFacor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 55.9%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition.
Figures & tables
Figure 1: A stacked bar chart ranks all models by overall score, with each bar decomposed into the four cognitive domains. There is still a gap between the best model and our human baseline (average performance of undergraduate students). Figure 5 further shows model scores vs. their release dates.
Figure 2: VisFactor comprises 20 vision-centric cognitive subtests. Each task is designed to isolate core factors of human visual cognition, covering 10 distinct factors in total. Example stimuli, questions, and ground-truth answers are shown for each task. The subtests are converted into either yes/no questions or fill-in-the-blank questions according to § A.3 . Prompts are shown in § E .
Table 1: The performance of 39 models on VisFactor . The rightmost column shows the total score. Darker scores show higher scores. The best model is GPT-5.5, achieving 55.9%.
Table 2: The performance of the GPT-5.5 model on the generated subsets in VisFactor . The “Original” row reports performance on the original FRCT questions. The “Normal” row uses the same configuration as the original questions. The “Easy” and “Hard” rows correspond to questions that are modified to be easier and more difficult, respectively.
Figure 3: An example of our generated MA1 image-number pairs using CF2 and MV1 figures.
Number of Pairs
10
20
40
80
MA1
CF2
MA1
CF2
MA1
CF2
MA1
CF2
GPT-4.1-2025-04-14
90.48
78.57
83.33
57.14
88.10
52.38
92.86
33.33
Claude-3.7-Sonnet
97.62
73.81
73.81
45.24
85.71
38.10
85.71
9.52
Qwen-VL-Max-250408
97.62
83.33
88.10
47.62
90.48
2.38
73.81
7.14
Table 3: MA1 performance of three models using different image sources and pair numbers.
Benchmarks
#T
#Q
P
G
D
M
I
PR
BM
MM
RT
MZ
PZ
ARC-AGI-2 Chollet et al. (2025)
1
240
✓
✓
N/A
S
✓
✓
✓
✓
BlindTest Rahmanzadehgervi et al. (2024)
7
9,720
✓
✓
✓
✓
S
BLINK Fu et al. (2024)
14
3,807
✓ IQ
NS
✓
✓
✓
✓
Bongard-HOI Jiang et al. (2022)
4
53K
✓ BP
N
✓
Bongard-LOGO Nie et al. (2020)
3
12K
✓ BP
✓
✓
S
✓
Bongard-OpenWorld Wu et al. (2024b)
10
1,010
✓ BP
✓
N
✓
Table 4: Comparison of VisFactor and other vision benchmarks. #T: number of tasks; #Q: number of queries; P: psychological grounding; G: generation of new test cases; D: different difficulties for a task; M: rigorous measurement; I: N atural or S ynthetic images; PR: pattern recognition; BM: Bongard/matrix reasoning; MM: memory; RT: rotation; MZ: maze; PZ: puzzle.
Appendix figures & tables55 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: VisFactor integrates 20 subtests across 10 factors from four domains.
Figure 5: Performance vs. release date. Plotting each model’s overall score against its release date shows scores plateauing near 25 through most of 2025, then rising sharply from Gemini-3.0 onward to exceed 40, with GPT-5.5 peaking at 55.9.
Figure 6: The “Middle Score Anomaly” on the P3 (Identical Pictures Test). We plot the distribution of model scores on P3, along with the chance level (3.13) and human performance (91.7). Model scores cluster in the 30–60 band, far from both chance and the human near-ceiling.
Table 6: The performance of GPT-4.1-2025-04-14, GPT-4o-2024-11-20, and GPT-4o-Mini-2024-07-18 in VisFactor using different temperatures of {0.0, 0.5, 1.0} .
Table 7: The performance of GPT-4.1-2025-04-14, GPT-4o-2024-11-20, and GPT-4o-Mini-2024-07-18 with and without CoT. We also test GPT-5.1-2025-11-13 with different levels of reasoning efforts.
Figure 7: Pearson correlation between VisFactor (10 factors) and existing benchmarks. The numbers after each benchmark in the parentheses represent the sample size. Note that some of them are relatively small due to availability in the original papers.
Figure 8: Pearson correlation between VisFactor (20 subtests) and existing benchmarks. The numbers after each benchmark in the parentheses represent the sample size. Note that some of them are relatively small due to availability in the original papers.
Figure 9: Pearson correlation between all subtests in VisFactor .
Table 8: Additional results on 39 models.
Table 9: The performance of the GPT-4.1 model on the generated subsets in VisFactor . The “Original” row reports performance on the original FRCT questions. The “Normal” row uses the same configuration as the original questions. The “Easy” and “Hard” rows correspond to questions that are modified to be easier and more difficult, respectively.
Figure 19Figure 20Figure 21
Figure 10: Samples of our generated images. We can dynamically adjust test difficulties in VisFactor . For example, the grid size of CF3 is changed to 6×6 instead of 5×5 , 8×9 instead of 7×8 for SS3, and more folding steps for VZ2.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguistic priors to compensate for their fragile visual understanding. We uncovered a crucial fact: state-of-the-art MLLMs consistently fail on basic visual tasks that humans, even 3-year-olds, can solve effortlessly. To systematically investigate this gap, we introduce BabyVision, a benchmark designed to assess core visual abilities independent of linguistic knowledge for MLLMs. BabyVision spans a wide range of tasks, with 388 items divided into 22 subclasses across four key categories. Empirical results and human evaluation reveal that leading MLLMs perform significantly below human baselines. Gemini3-Pro-Preview scores 49.7, lagging behind 6-year-old humans and falling well behind the average adult score of 94.1. These results show despite excelling in knowledge-heavy evaluations, current MLLMs still lack fundamental visual primitives. Progress in BabyVision represents a step toward human-level visual perception and reasoning capabilities. We also explore solving visual reasoning with generation models by proposing BabyVision-Gen and automatic evaluation toolkit. Our code and benchmark data are released at https://github.com/UniPat-AI/BabyVision for reproduction.
Liang Chen, Weichu Xie, Yiyan Liang +27
1UniPat AI · 6Peking University · 3Alibaba Group +8
We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.
Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and visuospatial reasoning remains less understood. We introduce "Mind's Eye", a multiple-choice benchmark of eight visuo-cognitive tasks inspired by classic human intelligence tests and organized under a novel "A-R-T" taxonomy: Abstraction, Relation, and Transformation. The tasks probe core processes of fluid intelligence such as pattern induction, analogical relation mapping, and mental transformation. We evaluate a diverse suite of closed-source and open-source MLLMs and compare their performance with human participants. Humans achieve 80% accuracy, while top performing MLLMs remain below 50%. Error analysis reveals failures in: (i) visual attention allocation, (ii) internal perceptual manipulation, and (iii) weak abstraction of underlying visual concepts. Our findings suggest that current MLLMs exhibit limited visuospatial reasoning capabilities, when compared with human participants, highlighting the need for more cognitively grounded evaluation frameworks.
Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India