Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

Yinghui Li; Jiayi Kuang; Peng Xing; Daixian Liu; Yongheng Zhang; Junnan Dong; Shu-Yu Guo; Yangning Li

cs.AIMar 19, 2026

Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding

Authors: Yinghui Li, Jiayi Kuang, Peng Xing, Daixian Liu, Yongheng Zhang, Junnan Dong, Shu-Yu Guo, Yangning Li,

Organizations: Tsinghua University, China · Sun Yat-sen University, China · Independent Researcher · The Hong Kong Polytechnic University, China ·

Abstract

Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains unclear. We present a multi-domain benchmark spanning language, culture, mathematics, physics and chemistry, organized into three cognitive levels: perception and recognition, combination and reasoning, and association and critical thinking. Across leading MLLMs, we observe a consistent cognitive mismatch. Models frequently underperform on elementary symbol recognition while appearing relatively competent on more complex reasoning tasks. This recognition-reasoning inversion indicates that current systems often compensate with linguistic priors, template retrieval or procedural reasoning instead of robust visual grounding. The pattern is especially clear for sparse, low-redundancy symbols such as handwritten characters, formula graphs, circuit diagrams and chemical structures. These results show that symbolic understanding remains a major bottleneck for multimodal intelligence and motivate training and evaluation schemes that prioritize grounded perception in discrete semantic spaces.

Figures & tables

Figure 1: Continuous and discrete semantic spaces differ fundamentally in how visual information is organized. Discrete symbol understanding requires precise recognition of individual symbolic units and their structured relations.

Figure 2: Performance and representative cases for language-symbol understanding across three cognitive levels. Models are evaluated on faked-character detection, misspelled-character detection and visual-semantic correction.

Figure 3: Performance and representative cases for cultural-symbol understanding. The tasks progress from lexical grounding of emojis to idiomatic composition and culturally mediated interpretation.

Figure 4: Performance and representative cases for mathematical-symbol understanding, covering function graphs and geometric figures across recognition, reasoning, and verification tasks.

Figure 5: Performance and representative cases for physical-symbol understanding across mechanics and electromagnetism. Sparse physical symbols remain difficult for most models, especially in visually grounded mapping tasks.

Figure 6: Performance and representative cases for chemical-symbol understanding. The benchmark examines molecular structure recognition, reaction reasoning and higher-level correction or prediction.

Figure 7: Overview of the benchmark task design framework and illustrative examples. (a) The instantiation of the three-level symbolic understanding hierarchy across five distinct domains: Language, Cultural, Mathematical, Physical, and Chemical symbols. (b) Representative examples of the tasks designed for our benchmark.

Figure 8: Cross-domain summary of benchmark results. (a) Radar charts illustrate the fine-grained performance of models across General, Language, Culture, Math, Physics and Chemistry domains. (b) Global performance aggregated by difficulty, with accuracy averaged across five symbolic domains. (c) Scatter plots exploring interrelationships between domains.

Figure 9: Human performance on the benchmark compared with the best-performing model Gemini.

Figure 10: Case studies from the comprehensive analysis, highlighting model failures in recognizing and integrating symbols.

Figure 11: Overall performances of different models across the five symbolic domains.

Figure 12: Overview of the benchmark task design, including language symbols, cultural symbols, math symbols, chemistry symbols and physical symbols.

Figure 13: The data construction pipeline of our benchmark.

Field	Sub-field	Task ID	Task Name	Level	Instances	Level Total
Language	Character	T1	Faked Character Detection	Perception and Recognition	526	526
	Character	T2	Contextual Character Misuse Identification	Combination and Reasoning	488	488
	Character	T3	Visual-Semantic Character Correction	Association and Critical Thinking	824	824
Culture	Emoji	T1	Emoji Semantic Grounding	Perception and Recognition	843	843
	Emoji	T2	Emoji-Based Idiomatic Reasoning	Combination and Reasoning	783	783
	Emoji	T3	Cross-Cultural Emoji-to-Idiom Mapping (4-word)	Association and Critical Thinking	1,360	1,549

Table 1: Distribution of tasks across domains, sub-fields, and hierarchical levels.

Figure 14: Dataset distribution. (a) Overall data distribution across five domains. (b) Detailed task distribution specifically for Mathematical symbols.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
Sep 26, 2026Jintao Tong, Yujing Lou, Zhanming Shen +6Multimodal Large Language Models Multimodal Reasoning
Huazhong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Zhejiang University
Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1

Sep 26, 2026J/K move · Enter open · S save

MM-OPD: Towards One More Bottleneck Between Perception and Reasoning

Jintao Tong, Yujing Lou, Zhanming Shen, Jiaqi Gu+5

Huazhong University of Science and Technology · Alibaba Token Hub, Alibaba Group · Zhejiang University

Multimodal Large Language Models Multimodal Reasoning Knowledge Distillation

Open paper AskPDF arXiv

Recent multimodal large language models (MLLMs) advance visual reasoning by strengthening both perception and reasoning, implicitly assuming a process that transitions seamlessly from perception to reasoning. However, we observe a counterintuitive phenomenon that challenges this assumption: holding the model, question, and decoding fixed, we replace images with their caption or code representations (symbolic views), which seems to be redundant given the clear image structures, but the performance surprisingly improves by 10.2% to 23.6% across model scales and datasets. We term this performance gap as the Symbolic Visual Gap and then take a closer look at it. Through experiments, we find that although the visual evidence can already appear in the reasoning trace for the image-input model, the symbolic-view-input model shows much higher attention to the correct evidence than the image-input model. This suggests that despite good capabilities from current works in perception and reasoning themselves, another bottleneck exists between perception and reasoning in selecting perceived visual information as appropriate evidence for subsequent reasoning. To handle this bottleneck, since the symbolic view steers attention toward correct evidence and is readily obtained at scale, it provides supervision for evidence selection without manually labeled evidence. Building on this, we introduce MM-OPD, a multimodal on-policy self-distillation framework for symbolic-to-visual correction that transfers guidance from symbolic-conditioned behavior to the image-conditioned policy through residual token-level targets, steering the model toward correct visual evidence. Experiments across benchmarks and model scales show that MM-OPD improves a broad range of multimodal abilities, with gains in visual perception, chart and document understanding, mathematical reasoning, and general VQA.

Abstract

Multimodal Large Language Models

Explore similar work

MM-OPD: Towards One More Bottleneck Between Perception and Reasoning

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs