cs.CVFeb 23, 2025

Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs

Authors: Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang, Youliang Yuan, Xiaoyuan Liu, Wenxuan Wang, Wenxiang Jiao, Pinjia He, 

Organizations: CUHK · PKU · CUHKSZ · RUC · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. Current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks. Does it guarantee their performance on these foundational visual capabilities? To systematically investigate this gap, we introduce VisFacor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFacor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 55.9%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition.

Figures & tables

Figure 1: A stacked bar chart ranks all models by overall score, with each bar decomposed into the four cognitive domains. There is still a gap between the best model and our human baseline (average performance of undergraduate students). Figure 5 further shows model scores vs. their release dates.
Figure 2: VisFactor comprises 20 vision-centric cognitive subtests. Each task is designed to isolate core factors of human visual cognition, covering 10 distinct factors in total. Example stimuli, questions, and ground-truth answers are shown for each task. The subtests are converted into either yes/no questions or fill-in-the-blank questions according to § A.3 . Prompts are shown in § E .
Table 1: The performance of 39 models on VisFactor . The rightmost column shows the total score. Darker scores show higher scores. The best model is GPT-5.5, achieving 55.9%.
Table 2: The performance of the GPT-5.5 model on the generated subsets in VisFactor . The “Original” row reports performance on the original FRCT questions. The “Normal” row uses the same configuration as the original questions. The “Easy” and “Hard” rows correspond to questions that are modified to be easier and more difficult, respectively.
Figure 3: An example of our generated MA1 image-number pairs using CF2 and MV1 figures.
Number of Pairs10204080
MA1CF2MA1CF2MA1CF2MA1CF2
GPT-4.1-2025-04-1490.4878.5783.3357.1488.1052.3892.8633.33
Claude-3.7-Sonnet97.6273.8173.8145.2485.7138.1085.719.52
Qwen-VL-Max-25040897.6283.3388.1047.6290.482.3873.817.14
Table 3: MA1 performance of three models using different image sources and pair numbers.
Benchmarks#T#QPGDMIPRBMMMRTMZPZ
ARC-AGI-2 Chollet et al. (2025)1240✓ ✓N/AS✓✓ ✓ ✓
BlindTest Rahmanzadehgervi et al. (2024)79,720✓✓✓✓S      
BLINK Fu et al. (2024)143,807✓ IQ   NS✓✓ ✓ ✓
Bongard-HOI Jiang et al. (2022)453K✓ BP   N ✓    
Bongard-LOGO Nie et al. (2020)312K✓ BP✓✓ S ✓    
Bongard-OpenWorld Wu et al. (2024b)101,010✓ BP ✓ N ✓    
Table 4: Comparison of VisFactor and other vision benchmarks. #T: number of tasks; #Q: number of queries; P: psychological grounding; G: generation of new test cases; D: different difficulties for a task; M: rigorous measurement; I: N atural or S ynthetic images; PR: pattern recognition; BM: Bongard/matrix reasoning; MM: memory; RT: rotation; MZ: maze; PZ: puzzle.
Appendix figures & tables55 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: VisFactor integrates 20 subtests across 10 factors from four domains.
CategoryNameID#Questions#QueriesGuess Acc.
PerceptualHidden Figures TestCF1321603.13%
& ClosureHidden Patterns TestCF2804003.13%
Copying TestCF364644.00%
Gestalt Completion TestCS120200.00%
Concealed Words TestCS250500.00%
Snowy PicturesCS324240.00%
Appendix
Table 5: VisFactor includes 3,046 queries (808 questions), providing sufficient statistical power.
Figure 5: Performance vs. release date. Plotting each model’s overall score against its release date shows scores plateauing near 25 through most of 2025, then rising sharply from Gemini-3.0 onward to exceed 40, with GPT-5.5 peaking at 55.9.
Figure 6: The “Middle Score Anomaly” on the P3 (Identical Pictures Test). We plot the distribution of model scores on P3, along with the chance level (3.13) and human performance (91.7). Model scores cluster in the 30–60 band, far from both chance and the human near-ceiling.
Table 6: The performance of GPT-4.1-2025-04-14, GPT-4o-2024-11-20, and GPT-4o-Mini-2024-07-18 in VisFactor using different temperatures of {0.0, 0.5, 1.0} .
Table 7: The performance of GPT-4.1-2025-04-14, GPT-4o-2024-11-20, and GPT-4o-Mini-2024-07-18 with and without CoT. We also test GPT-5.1-2025-11-13 with different levels of reasoning efforts.
Figure 7: Pearson correlation between VisFactor (10 factors) and existing benchmarks. The numbers after each benchmark in the parentheses represent the sample size. Note that some of them are relatively small due to availability in the original papers.
Figure 8: Pearson correlation between VisFactor (20 subtests) and existing benchmarks. The numbers after each benchmark in the parentheses represent the sample size. Note that some of them are relatively small due to availability in the original papers.
Figure 9: Pearson correlation between all subtests in VisFactor .
Table 8: Additional results on 39 models.
Table 9: The performance of the GPT-4.1 model on the generated subsets in VisFactor . The “Original” row reports performance on the original FRCT questions. The “Normal” row uses the same configuration as the original questions. The “Easy” and “Hard” rows correspond to questions that are modified to be easier and more difficult, respectively.
Figure 19
Figure 20
Figure 21
Figure 10: Samples of our generated images. We can dynamically adjust test difficulties in VisFactor . For example, the grid size of CF3 is changed to 6×66\times 6 instead of 5×55\times 5 , 8×98\times 9 instead of 7×87\times 8 for SS3, and more folding steps for VZ2.
Figure 23
Figure 24
Figure 25
Figure 26
Figure 27
Figure 28
Figure 29
Figure 30
Figure 31
Figure 32
Figure 33
Figure 34
Figure 35
Figure 36
Figure 37
Figure 38
Figure 39
Figure 40
Figure 41
Figure 42
Figure 43
Figure 44
Figure 45
Figure 46
Figure 47
Figure 48
Figure 49
Figure 50
Figure 51
Figure 52
Figure 53
Figure 54
Figure 55
Figure 56
Figure 57
Figure 58
Figure 59
Figure 60
Figure 61
Figure 62
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 986 papers

Explore similar work

CardsList

  1. BabyVision: Visual Reasoning Beyond Language

    Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM EvaluationVisual Reasoning

    1UniPat AI · 6Peking University · 3Alibaba Group +8

  2. PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

    Jul 27, 2026Zichao Lin, Yifeng Xie, Bowen Qu +30Object Hallucination in VLMsMultimodal Large Language Models

    Moonshot AI

  3. Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

    Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning BenchmarksVisual Spatial Reasoning

    CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India